Posted on Jul 25, 2022

Creating a script to sync internationalisation encodings between JS and Dart in Flutter

Created: March 20, 2022 9:58 PM
Published: Yes

I was recently tasked with setting implementing localisation for the flutter app we’ve been building for the past few months.

For a bit of background, we have had a Progressive Web App (PWA) version of our app for a number of years now and we had decided to make it a native app to take more advantage of mobile technology mostly with regards to file storage (with PWA’s you are essentially using browser storage which is limited to 50% of the full disk space and each origin having only 20% of that or max 2GB).

The PWA is built with React (JS duh) has been localised to support 4 languages. The [i18n-js](https://www.npmjs.com/package/i18n-js) npm was used to provide provide translated messages at run time based on the locale provided.

Localised strings are provided as JSON structures with keys as translation identifiers and values as the actual translated strings

//en.js
account: {
        language: 'Language',
        settings: {
            label: 'Settings'
        },
        about: 'About',
        consent: 'Consent',
        support: {
            title: 'Support Lines',
            subtitle: 'You can contact us through these methods for further help:'
        }
    }

//fr.js
account: {
        language: 'Langue',
        settings: {
            label: 'Réglages'
        },
        about: 'À propos',
        consent: 'Consentement',
        support: {
            title: 'Assistance',
            subtitle: 'Vous pouvez nous contacter comme qui suit si vous avez besoin d’aide:'
        }
    }

Strings are then inserted at run time with the target locale (fr in this case)

<ListItemText
    primary={I18n.t('account.language', { locale })}
/>

This enables us to localise the app for as many locales we can create translations for.

We planned to still maintain the PWA as some for some users the browser would still be their access point. Additionally we had not set up iOS deployment for the native app so it would only be available for Android. users.

Our goal was to use the same translation files as in the PWA for the flutter. We also did not want to be manually updating both translation files as there was bound to be missing translations whenever we modified any one of them. The key here was maintainability of the translations across both the PWA in Javascript and the Flutter app in dart.

There were a few problems:

Flutter converts all keys into getters for a class created through code generation. JSON allows string keys that can contain numbers and special characters like hyphen (-), all of which are illegal characters for variable names in flutter.
The intl package which is responsible for the code generation in flutter, requires the JSON file to be flat with no nested structures.

Placeholders for inserting dynamic strings require a different structure in Flutter than in the PWA with the I18n package.

The JSON for the I18n package for dynamic strings are structured as below:

numericalRangeFeedback:
            'The correct answer is a number between {lowerNumber} and {higherNumber} inclusive',

The corresponding JSON structure of the above suitable for Flutters intl is shown below:

"@question_numericalRangeFeedback": {
    "placeholders": {
      "higherNumber": {
        "type": "String"
      },
      "lowerNumber": {
        "type": "String"
      }
    }
  }

This meant we had to do some extra work to modify the translation files to be used in flutter and we had to do it in a way that was easy to maintain. Hello Automation,

First of all we decided to create a script on the pwa repo build process to:

Insert keywords in respective languages into the JSON structure
Replace all hyphens in keys with underscores which are allowed in Flutter
Flatten all keys.
Replace dynamic string braces with a new json entry with a placeholders key
Upload the processed JSON translation to a remote location. Firebase is used in this case.

Ideally this transformation should be done on the flutter repo as the processing of the JSON structure into an acceptable format for flutter is more of a concern of the flutter repo. However we didn’t fuss too much about this we found it easier to use javascript and npm packages to make the translation scripts.

const dynamicRegex = /{(\w+)}/gim;

const getDynamicStrings = (str) => {
    return [...str.match(dynamicRegex)];
}

const removeBraces = (str) => {
    return str.replace(/{|}/gmi, "");
}

const formatWithSingleBraces = (multiBracedText) => {
    const leftRegex = /{+/gm;
    const rightRegex = /}+/gm;
    return multiBracedText.replace(leftRegex, '{').replace(rightRegex, '}')
}

const flattenObjectStructure = (originalObject) => {
    return flatten(originalObject, {
        delimiter: '_',
        transformKey: function(key){
            return key.replace(new RegExp('-'), "_")
        }})
}
const addAnnotatedDynamicEntry = (translationObject, key) => {
    let newEntry = {};
    newEntry[`@${key}`] =  {placeholders: {}};
    const dynamicStrings = getDynamicStrings(translationObject[key]);
    for (const dynamicString of dynamicStrings){
        const dynamicKey = removeBraces(dynamicString);
        newEntry[`@${key}`].placeholders[dynamicKey] = {
            "type": "String"
        }
    }
    return newEntry;
}

const processDynamicStrings = ( flatTranslation ) => {
    const processedTranslation = {};
    for(let key in flatTranslation){
        processedTranslation[key] = formatWithSingleBraces(flatTranslation[key]);
        if(dynamicRegex.test(flatTranslation[key])){
            const newEntry = addAnnotatedDynamicEntry(flatTranslation, key)
            Object.assign(processedTranslation, newEntry)
        }
    }
    return processedTranslation;
}

for(let locale in availableLocales){
    if(!(locale in translationMap)){
        throw Error('Unsupported locale')
    }
    const translation = translationMap[locale](keywords[locale]);
    const flattenedTranslation = flattenObjectStructure(translation);

    const finalTranslation = processDynamicStrings(flattenedTranslation);
    const docRef = db.collection('translations').doc(locale);
    docRef.set(finalTranslation)
        .then(()  => console.info(`${locale} locale uploaded`))
        .catch((err) => {
            console.error(err)
        });
}

With this process, whenever the base JSON translation files are changed and committed to production a new build is triggered and the files are processed and uploaded to Firebase.

Additionally tests were added to check all translations against the english translations to identify any missing translations and fail if any are found.

During the Flutter build process, a dart file is run to download all the JSON from the remote repository, clean the strings (remove any unwanted characters) and create new .arb files (based on JSON) which intl uses to generate classes and getters for translations. The beauty of intl creating classes and getters means if any of the previously used translations is mistakenly removed, flutters static type checker would point this out for us.

Note that I dont fetch directly from firebase because setting up firebase on flutter was not as straightforward as I would like and it appeared a lot of work just to pull json strings from a remote. So I created a proxy server using expressjs that does the work and set up firebase on that.

void main() {
  getTranslations();
}

void getTranslations() async {
  var requestBody = {
    'translationKeys': jsonEncode(['en', 'rw', 'es', 'fr', 'sw'])
  };
  var url =
      Uri.parse('remote server with translations url');
  var response = await http.post(url,
      headers: {'Content-Type': 'application/json'},
      body: json.encode(requestBody));
  var translations = jsonDecode(response.body);
  translations.forEach((key, value) async {
    var filename = 'lib/l10n/intl_$key.arb';
    await File(filename)
        .writeAsString(removeHTMLTags(jsonEncode(value)).replaceAll('%', ""));
  });
}

So there we have it, an automated maintainable way to ensure translation files are consistent across both repos.

The greater topic of interest is internationalisation and localisation of your applications for different areas you serve or might serve. If you intend on going for growth you should always have in mind the visual strings in your app should be easy to replace with different localised versions for when you eventually tackle internationalisation. Of course there is also architecting your app component to support Right-To-Left languages like Arabic and Urdu but that requires a bit more than described above and maybe would be tackled in a separate blog post

You can read more about internationalisation in flutter here.