DEV Community: Denis Augsburger

Multilingual Markdown Documentations and Posts in Seconds

Denis Augsburger — Wed, 05 May 2021 09:30:56 +0000

The translation of Markdown files is commonly needed in technical documentations and headless content management systems, where you want to reach a target audience that speaks different languages. I'm gonna show you how you can translate Markdown easy and fast without compromising on quality. If you'd like to get a head start and try out the Markdown Translator just sign up.

Markdown Translation Tool

More and more tools use Markdown to structure their content. Some examples are:

Docusaurus, Gitbook for documentations
Hugo, Jekyll, GatsbyJS as static site generators (SSG)
Contentful, Strapi, SquareSpace as content management systems (CMS)

Depending on the project, it is necessary to generate multilingual content and update it regularly. The traditional translation process can be time-consuming and waiting on (human) translations can block your release cycles. Therefore we were looking for a fast and reliable solution.

Common Challenges

We've tried out several translation tools and inserted Markdown but we were not satisfied with the translation results they provided.
Common problems we encountered:

Broken Markdown Syntax
Translation of things that should not be translated, like Code Snippets, Emoji's
Different styles in translation results
Setup/Installion necessary

Let's take a look at how a simple Markdown file is translated from English to German if you use it directly in DeepL or Google Translate and compare it to the Simpleen Markdown Translator

The file contains a list, some emoji's and headers. Dev.to doesn't support fenced code within their blog posts, therefore the tripple ticks are substituted by html code tags.

## Setup

Install the CLI to **translate** files from source to target path.

<code language="shell">
yarn add simpleen
yarn run simpleen init
</code>

You can search for files in `./blog/posts/en/*.md` and translate them to `./blog/posts/$locale/$FILE.md`.

## Additional support :smile:

- PO-Files
- JSON
- YAML

DeepL Markdown Example

With DeepL the result looks like the following.

## Einrichtung

Installieren Sie die CLI zum **Übersetzen** von Dateien vom Quell- in den Zielpfad.

<code language="Shell">
yarn add simpleen
yarn run simpleen init
</code>

Sie können nach Dateien in `./blog/posts/de/*.md` suchen und sie in `./blog/posts/$locale/$FILE.md` übersetzen.

## Zusätzliche Unterstützung :smile:

- PO-Dateien
- JSON
- YAML

As you can see the code snippet is broken because the fenced code block now starts with two backticks instead of three. Also, the name of the language Shell is now upper-cased. The list, the emoji and the paths are handled correctly in this simple case. The bold text is also marked correctly.

Google Translate Markdown Example

Let's compare this with Google Translate:

## Einrichten

Installieren Sie die CLI, um Dateien von der Quelle in den Zielpfad zu übersetzen.

<brokencode language="shell">
Garn hinzufügen einfach
Garn laufen einfach init
</brokencode>

Sie können nach Dateien in "./blog/posts/en / *. Md" suchen und diese in ". / Blog / posts / $ locale / $ FILE.md" übersetzen.

## Zusätzliche Unterstützung: smile:

- PO-Dateien
- JSON
- YAML

The result with Google Translate is worse than DeepL. The code snippet is broken because the backticks of the code snippet are seperated with a space. Also, the content is translated with is not desirable. The paths are splitted and differently marked. The Emoji is also broken.

Simpleen Markdown Translator

Let's see how Simpleen handles this Markdown example in comparison to DeepL and Google Translate (in this case with DeepL).

## Einrichtung

Installieren Sie die CLI, um Dateien vom Quell- in den Zielpfad **zu übersetzen**.

<code language="shell">
yarn add simpleen
yarn run simpleen init
</code language="shell">

Sie können nach Dateien in `./blog/posts/de/*.md` suchen und sie in `./blog/posts/$locale/$FILE.md` übersetzen.

## Zusätzliche Unterstützung :smile:

- PO-Dateien
- JSON
- YAML

Because we love Markdown we wanted to deliver better and more consistent results with an online translator that let's you translate Markdown into many languages.

Simpleen provides better results because we handle Markdown differently than other services. Instead of just handling Markdown as Text or convert it to HTML, which is supported by most MT services, we go deeper to understand the whole document structure of your Markdown files.

Furthermore, Simpleen understands the most common Markdown extensions and flavors and applies the provided styles from your file to the translation result. For example, if you use two spaces at the end of a line to break a line, we also use two spaces in the translated result.

Supported Flavors & Extensions

Markdown comes in different flavors, and therefore supports different syntax to write your documentations, blog posts and more.
The most common flavors that are used and supported for translations by Simpleen are:

CommonMark
GFM Github Flavor Markdown

with the following extensions:

Emoji's (😄 or 👍)
Footnotes (partial)
Frontmatter
Math

CommonMark is a Markdown flavor that many frameworks and libraries support or build upon, for example GatsbyJS with their remark transformer. Also many headless content management system do support CommonMark.

Better Style Support

There are different valid ways to mark your headers, bold text, lists and more. Simpleen detects your style and reproduces the translated
Markdown file in a consistent way. For example if you use a dash for your lists

My shopping list:

- Dictionary
- Paper
- Pencil

then this Markdown example is translated to German like this:

Meine Einkaufsliste:

- Wörterbuch
- Papier
- Bleistift

Or if you use a star for your list instead it's getting translated to:

Meine Einkaufsliste:

* Wörterbuch
* Papier
* Bleistift

Both results are valid in most Markdown flavors, but we want to consistently apply the styles from the provided Markdown file.
As a result you can use the translated Markdown file directly in your Markdown documentation tool. Furthermore, the editor or human translator is not getting confused by different styles in case of post-editing.

Translate .md/.mdx Files

A Markdown file contains multiple parts that need to be localized. Other parts - like code segments and frontmatter fragments (meta data) - need to be excluded from translation.

Not translated:

Code Fences
Emoji's
Frontmatter
Math Expressions
MDX (not yet supported, drop us a line if you like to use it)

Translated:

Headers (atx, setext)
Paragraphs with bold, italic styles, links, images
List Items
Table Headers
Table Entries
ToDo List Entries
Footnotes (partial, #fn-1 instead of ^1)

We have plans to improve the Markdown translation tool even more. Quick Roadmap:

We want to handle internal links correctly (adapt to translated result)
Handle footnote links
Adapt the Simpleen CLI to support Markdown files in your local project

Online JSON Machine Translator: The Fastest Way to Localize

Denis Augsburger — Fri, 26 Jun 2020 00:00:00 +0000

The translation of JSON is commonly used for software localization and the translation of data. In this blog post we will check out an easy and fast way to translate locales as well as JSON data. We will cover different aspects that are relevant for these use cases. If you like to get a head start just sign up.

Online JSON Translator

The Simpleen online translator supports JSON (and other common formats like YAML) with variable/interpolation handling. You can translate from and to 12 languages which are supported by DeepL like Spanish, English, French or Portuguese. Just copy/paste your locale files or JSON data and insert it in the translator. After a few seconds you can copy the translated result.

Configuration of Translator

Basically you configure a translator with a Format like JSON, a source- and target language and the corresponding interpolation for localization. Additional translation options are the formality to get more or less formal results. This is very useful if you translate from english to other languages because in english the formality cannot always be determined.
For the translation of JSON data you can add selections to partially translate your structure and exclude others from getting translated.
The configuration is saved in your account and can be reused easily.

Adapt Translations With Custom Glossary

With your glossary you can define words that should be excluded from getting translated. This is really useful for brands and product names. You can also define some words that should always be translated to specific words. Depending on how you use your glossary they can be target language specific or generally used for multiple configurations.

Limits of the Online Translator

The Online Translator is a good starting point to try out Simpleen. You don’t have any vendor lock-in or an additional dependency in your project. But there are limits for this approach. Especially for a continuous localization process, where you also adapt your translations and handle multiple versions we recommend to use the CLI. Also if you have many JSON files that you want to get translated.

Continuous Translations With the CLI

When to Use the CLI

The Simpleen CLI brings machine translations into your development workflow. You create your multilingual app/game and can enjoy the results in other languages while developing. This is the fastest way to check and improve your app for a multilingual audience. It's a suitable approach for quality assurance during internationalization (i18n). It's also an alternative approach to pseudo localization.

Lock Translations

The translation results will evolve over time. Therefore, adapted translations should not be overwritten once you adapted the localization result. To handle this situation, the CLI can create a lock file which should be put under version control. After a manual change or a merge from your translation management system, simply run simpleen lock. It considers the source translation file, so that keys can be reused in larger projects.

Translate JSON to Localize

A variety of i18n libraries use JSON locales by default to support localizing your app into multiple languages. Depending on the library they use flat (i.e. react intl) or nested JSON structures (i.e. i18next). These can be stored in multiple files or extracted into a single one. If you want to localize your software, game or app you will need to translate your locale files into your desired language.

Variables (Interpolation)

You probably want to insert some data into your translation text. These variables are specifically marked and replaced with data later on. Here is a list of common JSON based i18n libraries with their default interpolations:

Library	Variable Interpolation
Angular ngx-translate, react intl, vue i18n	{ variable }
i18next, Transloco	{{ variable }}
Polyglot.js	%{ variable }

The translation of such interpolation variables would break the implementation at runtime. Therefore they need to be handled carefully and not to be translated by mistake. For example the sentence: There are {{count}} similar recipes{lang=en} becomes to 似た {{count}} ようなレシピがあります。{lang=ja}.

Additional Formats in JSON Locales

To correctly translate JSON locales you can handle embedded formats as well. Some common ones of i18n libraries are HTML and Markdown inside the translation text. You can highlight certain words/phrases or use some links directly in your translations. You don’t want to break your formatted strings while translating, otherwise your app shows some markup in your text (i.e. some **) or the whole phrase gets bold.

Multilingual Software

Multilingual apps need more attention on formats, i.e. currencies, number formats or date formats. Try to imagine which languages you want to support. Depending on the selection of the language pluralization rules or text floating (right to left) can become more challenging. With the translation of your application as early as possible you will be able to determine implementation problems and solve them.

Translation of JSON Data

JSON is also a common data format outside of software localization. The JSON structure consists of more information that should not be altered, i.e. some ids, a creation date or an author name. With Simpleen you can selectively translate your JSON data.
For example, let's take the following recipe:

{
   "title": "World's Best Lasagna",
   "instructions": [
     "Heat up the oven", 
     "Cut the onions", 
     "Cook the ground beef, onions, and garlic over medium heat until well browned"
   ],
   "notes": "<p class=\"info\">One of my most favorite dishes</p>",
   "ingredients": [{
     "amount": "2 pcs",
     "name": "onions"
   }, {
     "amount": "1 tbsp",
     "name": "salt"
   }],
   "published": "2020-07-19",
   "author": "@denisaugsburger"
}

When translating this kind of data, we may only want to translate parts of the provided recipe.

Keep Your Structure

To do this, we want to keep our existing structure as well as not corrupt any data in it. Also, our styles and data structure in HTML need to stay intact.

Keys should not be translated otherwise your code doesn't work anymore. Some values should stay as is, i.e. the amount of the ingredient, because it is handled by a special conversion function. The same applies to the published date which is handled by localization afterwards.

Also, some of the included formats like HTML or Markdown should be translated without losing their formatting.
Simpleen makes it easy to selectively translate some content in your JSON structure. Just define your selections and formats in your Simpleen translator.

Implement the Recipe Example

So let’s see how this works with the example from above and translate our recipe into Japanese.

First: Create a Simpleen Account and configure your Translator. Your selections for the example above could be as follows:

Path	Format
title	Text
instructions[*]	Text
notes	HTML
ingredients[*].name	Text

Your created translators can be used independently with their ID. They are configured with your expected format, selections, languages as well as a glossary.

Second you can use your translator to translate (some of) your JSON data.

/* Post-Request to use your translator for translations */
axios.post("https://api.simpleen.io/translators/{ID}/translate", {
  // You can select which part of the JSON should be translated via selections
  text: JSON.stringify(recipe),
}, 
options: {
  params: {
    auth_key: '{YOUR_API_AUTHENTICATION_TOKEN}'
  }
});

Our result with a configured target language of Japanese looks like this:

  {
    "title": "世界一おいしいラザニア",
    "instructions": [
        "オーブンを温める",
        "玉ねぎを切る",
        "牛ひき肉、玉ねぎ、にんにくを中火でよく焦げ目がつくまで煮る。"
    ],
    "notes": "<p class=\"info\">一番好きな食べ物の一つ</p>",
    "ingredients": [
        {
            "amount": "2 pcs",
            "name": "玉ねぎ"
        },
        {
            "amount": "1 tbspn",
            "name": "塩"
        }
    ],
    "published": "2020-07-19",
    "author": "@denisaugsburger"
}

DeepL for Better Translation Results

Currently we support the translation with DeepL, which results in better translation results in comparison to Google Translate. Unfortunately DeepL currently support a dozen languages with some varieties like brazilian portuguese or british english. Google translate supports much more languages but with less accuracy. Therefore we have decided to build Simpleen with DeepL in the background but we will consider more options in the future.

Start JSON Translating

You can start now localizing & translating by creating a Simpleen account. If you have any questions you can reach out to me via Twitter or Mail.

Combine Apollo Tooling and Gatsby for Type Safety

Denis Augsburger — Wed, 24 Jul 2019 00:00:00 +0000

Apollo Tooling is a command line tool for generating static types from a GraphQL backend. In combination with GatsbyJS it can be used to generate TypeScript definitions and validate the queries. To support a faster, safer and more reliable way to write these queries I highly recommend using the Apollo VSCode Plugin. It highlights your graphql queries and provides you with a useful autocomplete.

Download schema and install VSCode Plugin

The CLI looks for a config file and additionallly provided command line flags. It is possible to use different config files for specific tasks. First we want to download the schema, therefore you Start your gatsby project

gatsby developer

Create your config as apollo.config.download.js

module.exports = { 
  client: {    
    service: {      
      name: "gatsby-schema",
      url: "http://localhost:8000/___graphql"
    },
    tagName: "graphql"  
  }
};

Download your schema to schema.json

apollo service:download -c apollo.config.download.js

Install Apollo GraphQL VSCode Plugin

Install the plugin

ext install apollographql.vscode-apollo

Next you create your apollo.config.js file. This one is used for code generation as well as the VSCode plugin.

module.exports = {
  client: {    
    addTypename: false,
    includes: [
      "./src/**/*.tsx",
      "./src/**/*.ts",
      "./node_modules/gatsby-transformer-sharp/src/fragments.js",
      "./node_modules/gatsby-source-contentful/src/fragments.js"
    ],    
    service: {      
      name: "gatsby-schema",
      localSchemaFile: "./schema.json"
    },
    tagName: "graphql"
  }
};

GatsbyJS uses graphql as their default tag name for their queries while apollo uses gql as their default. This gives us the possibility to also use it with apollo client. Some GatsbyJS Plugins provide some global GraphQL fragments, which are used but not referenced in your queries. Therefore you need to include them in your apollo.config.js, otherwise you will get errors like

Unknown fragment “GatsbyImageSharpFixed”

Now we can enjoy the autocomplete and syntax highlighting of the apollo vscode plugin as you can see in the animated gif below.

Generate TypeScript Definitions from your queries

As a final step you combine the provided schema information and your GatsbyJS queries to create client side type definitions for TypeScript.

apollo client:codegen -c apollo.config.json --target typescript __generated__

A new subdirectory will be created for each page and component with a GraphQL query - it is named __generated__. From there you can import your TypeScript definitions. Tip:

type PageProps<TData> = {
 data: TData;
} & MatchRenderProps<{}>;

For your pages you can create a reusable type with a generic that can be extended for cross cutting concerns. Then you can use it as follows:

type BlogProps = PageProps<BlogPageQuery>
export const MyPage: React.FC<BlogProps> = ({data}) => {
    return ()
}
export const pageQuery = graphql`
  query BlogPageQuery {
    contentfulImages(name: { eq: “testImage” }) {
      id
      description   
    }
  }
`

In your components you can also use the generic version directly.

const data = useStaticQuery<TestPageQuery>(
  graphql`
    query TestPageQuery {
      contentfulImages(name: { eq: "testImage" }) {
        id
        description
        name
        fluid(maxWidth: 1000) {
          ...GatsbyContentfulFluid
        }
      }
    }
  `
)

Current limitations

In some cases we experienced that the schema.json file is not correctly reloaded into the Apollo VS Code Plugin after some changes. Therefore a restart of VS Code is necessary
The null checks can be extensive, so better introduce some helper function to get the related data
With strict null checks you can't use the contentful images in gatsby images because of the different handling of null and undefined