MDX with Translations 🐠

In recent years the need for some easy yet flexible format for simple static content has grown. Originally, HTML was meant for that - but let's be honest: it is neither simple, nor flexible. There have been plenty replacement proposals - as pretty much every CMS comes with its own dialect.

One good contender for static content that can be read and rendered nicely is Markdown. However, it has quite a bit of shortcomings. Most notably, Markdown is not fully specified - leading to multiple dialects that exist. Furthermore, many necessary things are only available in form of non-standard extensions. Finally, using custom components within Markdown is not possible.

This is where MDX comes into play. It's essentially a hybrid of JSX / React and a well-specified dialect of Markdown. Essentially, this allows writing texts that are as simple as plain Markdown, as well as creating complex components as you might have done in full JSX.

Website Structure

We build upon the structure set up in our recently published article faster pages with React and our original blog post from 2019. In this structure we have all the pages placed in their path-relative order using React for each page:

Now there are two things that we want to take care of:

1. Improve the content writing (and reading) for each employee at smapiot
2. Simplify translations; right now the structure requires duplicating pages (placing them with different / translated content in the "de" and "en" folders).

For the former we want to leverage MDX - for the latter we need to come up with a solution that allows us to have a flat structure.

Integrating MDX

Bringing MDX into the solution is actually quite straight forward. The @mdx-js/mdx package contains everything to compile and work with MDX files. However, as we are already using a bundler (Vite), we can just rely on the specific plugin that exists for that bundler.

npm i @mdx-js/rollup --save-dev

Why are we installing a package called rollup if our bundler is Vite? It turns out, that Vite is actually just a build-tool - for the actual bundling two other things are used:

• esbuild to bundle/process anything in the third-party packages (as well as use it, e.g., on the config file)
• rollup to bundle/process the user code

Hence the rollup plugin - which is all we need.

The integration into our vite.config.mjs file looks like this:

import codegen from 'vite-plugin-codegen';
import mdx from '@mdx-js/rollup';
import { resolve } from 'path';

export default {
  build: {
    assetsInlineLimit: 0,
  },
  resolve: {
    alias: {
      '@': resolve(__dirname, 'src'),
    },
  },
  plugins: [
    codegen(),
    mdx(),
  ],
};

Note that we are not providing any options right now. It just works!

So what can we do here? Consider one of our previous pages, the legal disclaimer (who doesn't love legal pages?!):

import * as React from 'react';
import { Page } from '@smapiot/components/lib/basic/Page';

export const meta = {
  title: 'Legal Disclaimer',
  legal: 'Legal Disclaimer',
};

const title = 'smapiot - Legal Disclaimer';

export default () => (
  <Page title={title}>
    <section className="container">
      <h1>Legal Disclaimer</h1>
      <h3>Liability for Contents</h3>
      <p>
        The contents of our pages have been compiled with the greatest care. However, we cannot guarantee for accuracy,
        completeness or topicality of the contents. As service providers, we are liable for own contents of these
        websites according to sec. 7, paragraph 1 German Telemedia Act (TMG). However, according to sec. 8 to 10 German
        Telemedia Act (TMG), service providers are not obligated to permanently monitor submitted or stored information
        or to search for evidences that indicate illegal activities. Legal obligations to removing information or to
        blocking the use of information remain unchallenged. In this case, liability is only possible at the time of
        knowledge about a specific violation of law. Illegal contents will be removed immediately at the time we get
        knowledge of them.
      </p>
      <h3>Liability for Links</h3>
      <p>
        Our web pages include links to external third party websites. We have no influence on the contents of those
        external websites, therefore we cannot guarantee for those contents. Providers or administrators of linked
        websites are always responsible for their own contents. The linked websites had been checked for possible
        violations of law at the time of the establishment of the link. Illegal contents were not detected at the time
        of the linking. A permanent monitoring of the contents of linked websites cannot be imposed without reasonable
        indications that there has been a violation of law. Illegal links will be removed immediately at the time we get
        knowledge of them.
      </p>
      <h3>Copyright</h3>
      <p>
        Contents and compilations published on these websites by the providers are subject to German copyright laws.
        Reproduction, editing, distribution as well as the use of any kind outside the scope of the copyright law
        require a written permission of the author or originator. Downloads and copies of these websites are permitted
        for private use only. The commercial use of our contents without permission of the originator is prohibited.
        Copyright laws of third parties are respected as long as the contents on these websites do not originate from
        the provider. Contributions of third parties on this site are indicated as such. However, if you notice any
        violations of copyright law, please inform us. Such contents will be removed immediately.
      </p>
    </section>
  </Page>
);

This isn't too bad! What we can now do is to rename it from Disclaimer.tsx to Disclaimer.mdx and change its content to become:

export const meta = {
  title: 'Legal Disclaimer',
  legal: 'Legal Disclaimer',
};

<section className="container">
  # Legal Disclaimer

  ### Liability for Contents

  The contents of our pages have been compiled with the greatest care. However, we cannot guarantee for accuracy, completeness or topicality of the contents. As service providers, we are liable for own contents of these websites according to sec. 7, paragraph 1 German Telemedia Act (TMG). However, according to sec. 8 to 10 German Telemedia Act (TMG), service providers are not obligated to permanently monitor submitted or stored information or to search for evidences that indicate illegal activities. Legal obligations to removing information or to blocking the use of information remain unchallenged. In this case, liability is only possible at the time of knowledge about a specific violation of law. Illegal contents will be removed immediately at the time we get knowledge of them.

  ### Liability for Links

  Our web pages include links to external third party websites. We have no influence on the contents of those external websites, therefore we cannot guarantee for those contents. Providers or administrators of linked websites are always responsible for their own contents. The linked websites had been checked for possible violations of law at the time of the establishment of the link. Illegal contents were not detected at the time of the linking. A permanent monitoring of the contents of linked websites cannot be imposed without reasonable indications that there has been a violation of law. Illegal links will be removed immediately at the time we get knowledge of them.

  ### Copyright

  Contents and compilations published on these websites by the providers are subject to German copyright laws. Reproduction, editing, distribution as well as the use of any kind outside the scope of the copyright law require a written permission of the author or originator. Downloads and copies of these websites are permitted for private use only. The commercial use of our contents without permission of the originator is prohibited. Copyright laws of third parties are respected as long as the contents on these websites do not originate from the provider. Contributions of third parties on this site are indicated as such. However, if you notice any violations of copyright law, please inform us. Such contents will be removed immediately.
</section>

Much easier to read! Note that unnecessary imports such as react could be removed, while the boilerplate wrapper component Page will now be automatically inserted.

But there was a second thing we needed, right? What about improvements for the localization / page structure?

Let's see what we can do here!

Providing Translations

What if we would just put the localization strings all in a little sidecar? For instance, for a file like Disclaimer.mdx we'd also have a file Disclaimer.yml. In this file we could place translations according to the languages.

The whole structure could then look like this:

Where an individual file, e.g., for Disclaimer.yml could be written as:

de:
  content$: |
    # Haftungsausschluss

    ### Haftung für Inhalte

    Die Inhalte unserer Seiten wurden mit größter Sorgfalt erstellt. Für die Richtigkeit, Vollständigkeit und Aktualität der Inhalte können wir jedoch keine Gewähr übernehmen. Als Diensteanbieter sind wir gemäß § 7 Abs.1 TMG für eigene Inhalte auf diesen Seiten nach den allgemeinen Gesetzen verantwortlich. Nach §§ 8 bis 10 TMG sind wir als Diensteanbieter jedoch nicht verpflichtet, übermittelte oder gespeicherte fremde Informationen zu überwachen oder nach Umständen zu forschen, die auf eine rechtswidrige Tätigkeit hinweisen. Verpflichtungen zur Entfernung oder Sperrung der Nutzung von Informationen nach den allgemeinen Gesetzen bleiben hiervon unberührt. Eine diesbezügliche Haftung ist jedoch erst ab dem Zeitpunkt der Kenntnis einer konkreten Rechtsverletzung möglich. Bei Bekanntwerden von entsprechenden Rechtsverletzungen werden wir diese Inhalte umgehend entfernen.

    ### Haftung für Links

    Unser Angebot enthält Links zu externen Webseiten Dritter, auf deren Inhalte wir keinen Einfluss haben. Deshalb können wir für diese fremden Inhalte auch keine Gewähr übernehmen. Für die Inhalte der verlinkten Seiten ist stets der jeweilige Anbieter oder Betreiber der Seiten verantwortlich. Die verlinkten Seiten wurden zum Zeitpunkt der Verlinkung auf mögliche Rechtsverstöße überprüft. Rechtswidrige Inhalte waren zum Zeitpunkt der Verlinkung nicht erkennbar. Eine permanente inhaltliche Kontrolle der verlinkten Seiten ist jedoch ohne konkrete Anhaltspunkte einer Rechtsverletzung nicht zumutbar. Bei Bekanntwerden von Rechtsverletzungen werden wir derartige Links umgehend entfernen.

    ### Urheberrecht

    Die durch die Seitenbetreiber erstellten Inhalte und Werke auf diesen Seiten unterliegen dem deutschen Urheberrecht. Die Vervielfältigung, Bearbeitung, Verbreitung und jede Art der Verwertung außerhalb der Grenzen des Urheberrechtes bedürfen der schriftlichen Zustimmung des jeweiligen Autors bzw. Erstellers. Downloads und Kopien dieser Seite sind nur für den privaten, nicht kommerziellen Gebrauch gestattet. Soweit die Inhalte auf dieser Seite nicht vom Betreiber erstellt wurden, werden die Urheberrechte Dritter beachtet. Insbesondere werden Inhalte Dritter als solche gekennzeichnet. Sollten Sie trotzdem auf eine Urheberrechtsverletzung aufmerksam werden, bitten wir um einen entsprechenden Hinweis. Bei Bekanntwerden von Rechtsverletzungen werden wir derartige Inhalte umgehend entfernen.
en:
  content$: |
    # Legal Disclaimer

    ### Liability for Contents

    The contents of our pages have been compiled with the greatest care. However, we cannot guarantee for accuracy, completeness or topicality of the contents. As service providers, we are liable for own contents of these websites according to sec. 7, paragraph 1 German Telemedia Act (TMG). However, according to sec. 8 to 10 German Telemedia Act (TMG), service providers are not obligated to permanently monitor submitted or stored information or to search for evidences that indicate illegal activities. Legal obligations to removing information or to blocking the use of information remain unchallenged. In this case, liability is only possible at the time of knowledge about a specific violation of law. Illegal contents will be removed immediately at the time we get knowledge of them.

    ### Liability for Links

    Our web pages include links to external third party websites. We have no influence on the contents of those external websites, therefore we cannot guarantee for those contents. Providers or administrators of linked websites are always responsible for their own contents. The linked websites had been checked for possible violations of law at the time of the establishment of the link. Illegal contents were not detected at the time of the linking. A permanent monitoring of the contents of linked websites cannot be imposed without reasonable indications that there has been a violation of law. Illegal links will be removed immediately at the time we get knowledge of them.

    ### Copyright

    Contents and compilations published on these websites by the providers are subject to German copyright laws. Reproduction, editing, distribution as well as the use of any kind outside the scope of the copyright law require a written permission of the author or originator. Downloads and copies of these websites are permitted for private use only. The commercial use of our contents without permission of the originator is prohibited. Copyright laws of third parties are respected as long as the contents on these websites do not originate from the provider. Contributions of third parties on this site are indicated as such. However, if you notice any violations of copyright law, please inform us. Such contents will be removed immediately.

For this file a set of conventions exist:

• The top-level keys must be valid languages (right now only de and en)
• The keys can be nested, i.e., objects / arrays are allowed
• When a key ends with a $ sign it will be interpreted as Markdown (specifically, MDX)

How does the MDX file look with such a sidecar file?

export const meta = {
  title: 'Legal Disclaimer',
  legal: 'Legal Disclaimer',
};

<section className="container">
  {locale.content$}
</section>

Now this looks appealing!

For integrating the localization we still need to do something. See the use of locale in the code above. But how does locale enter the landscape in the file? After all, we don't see it!

Turns out we can manipulate the generated code by MDX. Let's configure the tool a bit by adding a custom plugin to the recmaPlugins option. This allows us to have full control over the generated abstract syntax tree (AST), which is then used to generate the actual code representing the MDX file.

import codegen from 'vite-plugin-codegen';
import mdx from '@mdx-js/rollup';
import { resolve } from 'path';
import localize from './src/tools/localize.mjs';

export default {
  build: {
    assetsInlineLimit: 0,
  },
  resolve: {
    alias: {
      '@': resolve(__dirname, 'src'),
    },
  },
  plugins: [
    codegen(),
    mdx({
      recmaPlugins: [localize],
    }),
  ],
};

The imported localize function is the actual plugin - which just yields another function - the so-called transformer. The transformer is responsible for manipulating the AST.

The transform function in the next snippet receives the AST and the virtual file (i.e., it's name, content, ...) of the MDX file that is currently being handled.

Let's see the code before we go over what it does:

import { compileSync } from '@mdx-js/mdx';
import { loadLocale } from './localization.mjs';

function fromMarkdown(content) {
  let result = {};
  compileSync(content, {
    development: false,
    recmaPlugins: [
      () => (ast) => {
        const [, fn] = ast.body;
        result = {
          type: 'CallExpression',
          optional: false,
          callee: {
            ...fn,
            id: null,
          },
          arguments: [
            {
              type: 'ObjectExpression',
              properties: [],
            },
          ],
        };
      },
    ],
  });

  return result;
}

function getExpression(content, key) {
  switch (typeof content) {
    case 'object':
      if (Array.isArray(content)) {
        return {
          type: 'ArrayExpression',
          elements: content.map((item, i) => getExpression(item, `${i}`)),
        };
      }

      return {
        type: 'ObjectExpression',
        properties: Object.entries(content).map(([name, value]) => ({
          type: 'Property',
          key: {
            type: 'Identifier',
            name,
          },
          value: getExpression(value, name),
          kind: 'init',
        })),
      };
    case 'number':
    case 'boolean':
      return {
        type: 'Literal',
        value: content,
      };
    case 'string':
      if (key.endsWith('$')) {
        // uses potentially markdown
        return fromMarkdown(content);
      }

      return {
        type: 'Literal',
        value: content,
      };
    case 'undefined':
    default:
      return {
        type: 'Literal',
        value: null,
      };
  }
}

async function transform(ast, vfile) {
  const language = process.env.WEBSITE_LOCALE || 'en';

  const source = vfile.history[0];
  const locale = await loadLocale(source, language);

  const idx = ast.body.findLastIndex((m) => m.type === 'ImportDeclaration');
  const imprt = ast.body.find((node) => node.type === 'ImportDeclaration' && node.source.value === 'react/jsx-runtime');

  const requiredImports = [
    ['_jsx', 'jsx'],
    ['_jsxs', 'jsxs'],
    ['_Fragment', 'Fragment'],
  ];

  for (const requiredImport of requiredImports) {
    const [alias, original] = requiredImport;

    if (!imprt.specifiers.find((node) => node.type === 'ImportSpecifier' && node.imported.name === original)) {
      imprt.specifiers.push({
        type: 'ImportSpecifier',
        imported: {
          type: 'Identifier',
          name: original,
        },
        local: {
          type: 'Identifier',
          name: alias,
        },
      });
    }
  }

  ast.body.splice(idx + 1, 0, {
    type: 'VariableDeclaration',
    kind: 'const',
    declarations: [
      {
        type: 'VariableDeclarator',
        id: {
          type: 'Identifier',
          name: 'locale',
        },
        init: getExpression(locale, language),
      },
    ],
  });
}

const plugin = () => transform;

export default plugin;

Looks more complicated than it needs to be! In the end, it all boils down to changing the AST to

• obtain the locale for the document
• insert potentially missing named imports (to fully support the generated Markdown / content, if any) into the react/jsx-runtime import
• insert the locale declaration - first thing after the imports (this way you can use locale pretty much everywhere in the document
• initialize the locale variable to be the object we know; with exception of Markdown strings (suffixed with $): these are transformed into an IIFE keeping the original MDX code

A good way to see what we are doing can be observed by using the MDX playground.

As you can see the compiled code is a bit strange, but can be mapped nicely to our original MDX file.

In contrast, the view that we are actually most interested about is the stage before the code is in that compiled state. It's the "esast" view in the playground (i.e., the AST of the ESTree stage).

What we want to achieve with our plugin is that we manipulate the start. Unfortunately, directly in MDX code we'd need to use an export for that, but by manipulating the AST directly we actually don't need it.

To obtain the locale variable / object we the loadLocale function from a dedicated module.

This module does:

1. Read and parse the local / file specific translations
2. Obtain the language-specific translations of the local file
3. Read and parse the global translations
4. Obtain the language-specific translations of the global file
5. Merge with local translations being regarded higher

For obtaining the language-specific translations we choose a base translation (in our case en) and "walk" through the object. The current language (let's say de) then either overrides the base translation or just uses the snippet obtained from the base translation.

You can think of this process as a deep merge with the base translation being the original.

import { readFile } from 'fs/promises';
import { resolve } from 'path';
import { parse } from 'yaml';

async function getLocalization(path) {
  try {
    const content = await readFile(path, 'utf8');
    return parse(content);
  } catch (e) {
    console.error('Error reading YAML file:', e);
    return {};
  }
}

function mergeLocale(result, baseLocale, newLocale) {
  if (!newLocale) {
    Object.assign(result, baseLocale);
  } else {
    Object.entries(baseLocale).forEach(([name, value]) => {
      const c = newLocale[name];

      if (typeof value === 'object') {
        mergeLocale((result[name] = {}), value, c || {});
      } else if (typeof c === 'string' || c) {
        result[name] = c;
      } else {
        result[name] = value;
      }
    });
  }

  return result;
}

function getLocale(locales, lang) {
  const baseLocale = locales.en || {};

  if (lang !== 'en' && lang in locales) {
    const newLocale = locales[lang] || {};
    return mergeLocale({}, baseLocale, newLocale);
  }

  return baseLocale;
}

export async function loadLocale(source, language) {
  if (source.endsWith('.mdx')) {
    const globalFn = resolve(import.meta.dirname, '..', 'global.yml');
    const targetFn = source.replace('.mdx', '.yml');

    const globals = await getLocalization(globalFn);
    const locales = await getLocalization(targetFn);

    return {
      ...getLocale(globals, language),
      ...getLocale(locales, language),
      language,
    };
  }

  return { language };
}

The global translations are placed in the folder above the pages. The name of the file is global.yml.

For the global.yml we only have some very common translation terms stored. An example:

de:
  germany: Deutschland
en:
  germany: Germany

Being able to just change a single file when new translations come (or just changing the structure in one place independent of language changes) is a big deal. For us this was quite important and the new structure reflects that.

Conclusion

With the new approach the pages are much easier to write than beforehand. Also, since we now have a static translation system in place we do not need to duplicate the pages or come up with some complicated system for intermediate components. It's all already done at compile-time in the given structure.