DEV Community: Ilya Krukowski

Lokalise custom processors: Ruby on Rails

Ilya Krukowski — Thu, 25 Aug 2022 14:41:00 +0000

In this article, you will learn about a new Lokalise feature called Custom processor. The Custom processor app enables you to intercept translation keys uploaded and downloaded on Lokalise, then analyze or transform those as needed.

Today I'll show you how to create a Custom processor using Ruby on Rails framework, set it up on Lokalise, and test it.

If you'd like to know how to build a Custom processor with Node and Fastify, check out the corresponding tutorial. You can find even more code samples in our DevHub.

What is a Custom processor?

The Custom processor app enables you to utilize your own server or a third-party service to fetch translation keys imported to and exported from Lokalise, analyze, format, transform, or even remove those as needed.

To better understand the idea, let's see how a regular upload is performed on Lokalise:

You choose one or more translation files to upload.
Lokalise detects the translation file languages automatically, but you can adjust those as needed.
You adjust any additional uploading options and click Upload.
Lokalise parses the uploaded data, and extracts keys and translation values.
Extracted data is displayed in the Lokalise TMS translation editor.

However, if you have a Custom processor set up, after performing step 4 Lokalise will automatically send the parsed data to your processor. There you can perform any additional actions, such as analyzing the uploaded data, transforming your translations, adding special formatting, removing unnecessary special characters or certain banned words, restructuring translation keys, and so on. The only requirement is that the data returned by your processor must preserve the initial structure.

Now let's briefly cover the download process:

You choose file formats to download.
You adjust any additional options as needed and click Build and download.
Lokalise collects the chosen keys and translations.
Lokalise generates translation files based on the download options and zips those into an archive.

So, a Custom processor will intercept the data collected in step 3 and once again it can perform any additional modifications before these data are zipped into an archive.

As you can see, the idea is quite simple, however with this feature you can automate repetitive manual work.

Custom processor use cases

There are various use cases:

Translate the uploaded content using your own or third-party service.
Perform placeholder replacement.
Clean up the imported translations by removing unnecessary special characters.
Restructure or rename the exported translation keys.
Perform text analysis for your localization workflow needs.
Apply special formatting to the imported or exported translations.
Remove or replace banned or undesired words.

Now, let's see how to code a custom processor.

Creating a Custom processor

In this section, you'll learn how to create a script performing pre-processing and post-processing of your translation data.

Pre-processing happens when you are uploading translation files. We are going to use a third-party API called funtranslations.com to translate regular English texts into pirate speech.
Post-processing is performed when you are downloading your translation files back. We'll create a script to remove a banned phrase from all our translations.

Preparing the app

To get started, let's create a new Rails app as usual:

rails new CustomProcessors

I will be using Rails 7 but all the code samples should be relevant for Rails 5 and 6 as well.

Create a new controller in the app/controllers/processors_controller.rb file:

require 'uri'
require 'net/http'
require 'openssl'
require 'json'

class ProcessorsController < ApplicationController
  skip_before_action :verify_authenticity_token, only: %i[preprocess postprocess]
end

First, we import all the necessary modules that will be used to perform the API request.

Next, we also have to skip authenticity token verification for preprocessing and postprocessing because these actions will be called from another resource.

Let's also add two routes in the config/routes.rb file:

post '/preprocess', to: 'processors#preprocess'
post '/postprocess', to: 'processors#postprocess'

That's it!

Performing pre-processing

Now let's discover how to perform data pre-processing. I would like to find all English translations and turn them into funny pirate speech using funtranslations.com. To get started, let's iterate over translation keys:

def preprocess
  payload = params

  payload[:collection][:keys].each do |key|
  end
end

Next, we have to modify translations for every key, but only if the language ID is equal to 640 (which is English), and respond with a JSON containing the uploaded data:

def preprocess
  payload = params

  payload[:collection][:keys].each do |key|
    key[:translations].map! do |trans|
      if trans[:languageId] == 640
        trans[:translation] = translate(trans[:translation])
      end
      trans
    end
  end

  render json: payload
end

Let's also add the translate private method:

private

def translate(initial_text)
  url = URI("https://api.funtranslations.com/translate/pirate.json")
  url.query = URI.encode_www_form({text: initial_text})

  http = Net::HTTP.new(url.host, url.port)

  http.use_ssl = true

  request = Net::HTTP::Get.new(url)

  request["Accept"] = 'application/json'

  response = http.request(request)

  JSON.parse(response.read_body)['contents']['translated']
end

There's nothing complex really: we generate a proper URL, add our initial text to the query, send the request, and then get the translated content. Great job!

Performing post-processing

Post-processing is performed in a very similar way. Let's create a new action which is going to remove a banned phrase from all translations. Specifically, pirates fear the Flying Dutchman so let's get rid of it:

def postprocess
  payload = params

  payload[:collection][:keys].each do |key|
    key[:translations].map! do |trans|
      trans[:translation].gsub!(/flying\sdutchman/i, '')
      trans
    end
  end

  render json: payload
end

So, we are simply using gsub! to remove all occurrences of the given phrase. Once again, make sure to respond with a JSON containing all translations data.

Setting up a Custom processor

Once you have created a custom processor, it's time to enable it! Therefore, perform the following steps:

Proceed to lokalise.com, log in to the system, and open your translation project.
Proceed to Apps, find Custom processor in the list, and click on it.

Then click Install and configure the app by entering a pre-process and a post-process URL.
If needed, choose a file format to run this processor for.
When you are ready, click Save changes.

That's it, great job!

Testing it out

Now that everything is ready, you can test your new processor. To do that, proceed to the Upload page, choose an English translation file, and click Upload.

Return to the project editor and observe the results:

To test the post-processing feature, simply add the "Flying Dutchman" phrase to any translation, proceed to the Download page, and click Build and download. You should not see the banned phrase in the resulting translations.

en:
  password: "Say th' secret phrase and enter. It's  over there",
  welcome: "Welcome, me bucko!"

Congratulations!

Conclusion

And this concludes our tutorial. As you can see, the Custom processor is a very powerful feature that can be used to build custom workflows, so be sure to check it out. You can find even more code samples in our DevHub. However, if you have any additional questions, please don't hesitate to drop us a line.

Thank you for sticking with me today, and happy coding!

Lokalise custom processors: Node and Fastify

Ilya Krukowski — Fri, 19 Aug 2022 09:50:00 +0000

In this article, we will discuss a new Lokalise feature called Custom processor. The Custom processor app enables you to intercept translation keys uploaded and downloaded on Lokalise, then analyze or transform those as needed.

Here we’ll show you how to create a Custom processor using Node and Fastify, set it up on Lokalise, and test it.

You can find even more code samples in our DevHub.

What is a Custom processor?

To better understand the idea, let's see how a regular upload is performed on Lokalise:

You choose one or more translation files to upload.
Lokalise detects the translation file languages automatically, but you can adjust those as needed.
You adjust any additional uploading options and click Upload.
Lokalise parses the uploaded data, and extracts keys and translation values.
Extracted data is displayed in the translation editor.

Now let's briefly cover the download process:

You choose file formats to download.
You adjust any additional options as needed and click Build and download.
Lokalise collects the chosen keys and translations.
Lokalise generates translation files based on the download options and zips those into an archive.

So, a Custom processor will intercept the data collected in step 3 and once again it can perform any additional modifications before these data are zipped into an archive.

As you can see, the idea is quite simple, however with this feature you can automate repetitive manual work.

Why would you need a Custom processor?

There are various use cases:

Translate the uploaded content using your own or third-party service.
Perform text analysis for your localization workflow needs.
Apply special formatting to the imported or exported translations.
Clean up the imported translations by removing unnecessary special characters.
Remove or replace banned or undesired words.
Perform placeholder replacement.
Restructure or rename the exported translation keys.

So, as you can see, there are numerous potential use cases and you can think of very complex scenarios! Therefore, let's see how to code a custom processor.

Creating a Custom processor

In this section, you'll learn how to create a script performing pre-processing and post-processing of your translation data.

Pre-processing happens when you are uploading translation files. We are going to use a third-party API called funtranslations.com to translate regular English texts into pirate speech.
Post-processing is performed when you are downloading your translation files back. We'll create a script to remove a banned phrase from all our translations.

Preparing the app

So, let’s get started by creating a new project directory (for example, custom_processor) with a package.json file inside:

{
  "name": "custom-processors",
  "main": "server.js",
  "scripts": {
    "start": "node server.js",
    "dev": "fastify start -w -l info -P app.js",
  },
  "dependencies": {
    "config": "^3.3.7",
    "fastify": "^3.29.0",
    "fastify-autoload": "^3.13.0",
    "fastify-cli": "^3.1.0",
    "fastify-plugin": "^3.0.1"
  },
  "devDependencies": {
    "nodemon": "^2.0.16"
  }
}

Run:

npm install

Then, create a server.js file with the following contents:

require('dotenv').config()
const Fastify = require('fastify')
const app = Fastify({
  logger: true,
  pluginTimeout: 30000,
})
app.register(require('./app.js'))
app.listen(process.env.PORT || 3000, '0.0.0.0', (err) => {
  if (err) {
    app.log.error(err)
    process.exit(1)
  }
})

Finally, add an app.js file:

const path = require('path')
const AutoLoad = require('fastify-autoload')
module.exports = async function (fastify, opts) {
  fastify.register(AutoLoad, {
    dir: path.join(__dirname, 'plugins'),
    options: Object.assign({}, opts),
  })
  fastify.register(AutoLoad, {
    dir: path.join(__dirname, 'routes'),
    options: Object.assign({}, opts),
  })
}

Nice!

Performing pre-processing

Now let’s see how to perform pre-processing. I’d like to find all English translations and turn them into pirate speech using funtranslations.com. To achieve that, create a new file called routes/preprocess.js. Inside it we’ll iterate over translation keys:

module.exports = async function (fastify, opts) {
  fastify.register(require('fastify-http-client'))
  fastify.post('/preprocess', async function (request, reply) {
    if (!request.body) {
      return { "error" : "Invalid request: Missing body"};
    }
    const payload = request.body;
    for (const [keyId, keyValue] of Object.entries(payload.collection.keys)) {
    }
    return payload;
  })
}

Then we have to check the language ID; it should be equal to 640 which is English. If the language ID is the proper one, we’ll send an API request to translate our texts:

module.exports = async function (fastify, opts) {
  fastify.register(require('fastify-http-client'))
  fastify.post('/preprocess', async function (request, reply) {
    if (!request.body) {
      return { "error" : "Invalid request: Missing body"};
    }
    const payload = request.body;
    for (const [keyId, keyValue] of Object.entries(payload.collection.keys)) {
      for (const [lang, v] of Object.entries(keyValue.translations)) {
        if(v.languageId === 640) {
          const res = await fastify.curl('https://api.funtranslations.com/translate/pirate.json&text=' + v.translation).then((result) => {
            const translated = JSON.parse(result.data.toString())
            payload.collection.keys[keyId].translations[lang].translation = translated.contents.translated
          }).catch((err) => {
            payload.collection.keys[keyId].translations[lang].translation = v.translation
          })
        }
      }
    }
    return payload;
  })
}

Basically, this is it.

Performing post-processing

Post-processing is performed in a very similar way. Let’s create a new route which is going to remove a banned phrase, “Flying Dutchman”, from all the translations (after all, pirates are usually afraid of this mystical ship). Populate the routes/postprocess.js file with the following code:

module.exports = async function (fastify) {
  fastify.post('/postprocess', async function (request, reply) {
    if (!request.body) {
      return { error: 'Invalid request: Missing body' }
    }
    const payload = request.body
    for (const [keyId, keyValue] of Object.entries(payload.collection.keys)) {
      for (const [lang, v] of Object.entries(keyValue.translations)) {
        payload.collection.keys[keyId].translations[lang].translation = v.translation.replace(
          /flying\sdutchman/gi,
          '',
        )
      }
    }
    await reply.send(payload)
  })
}

Here we are simply removing the banned phrase from all translations.

Great job!

Setting up a Custom processor

Once you have created a Custom processor, it’s time to enable it! Therefore, perform the following steps:

Proceed to lokalise.com, log in to the system, and open your translation project.
Proceed to Apps, find Custom processor in the list, and click on it.

Then click Install and configure the app by entering a preprocess and a postprocess URL.
If needed, choose a file format to run this processor for.
When you are ready, click Save changes.

That’s it, great job!

Testing it out

Now that everything is ready, you can test your new processor. To do that, proceed to the Upload page, choose an English translation file, and click Upload.

Return to the project editor and observe the results:

To test the postprocessing feature, simply add the "Flying Dutchman" phrase to any translation, and proceed to the Download page.

Click Build and download.

You should not see the banned phrase in the resulting translations.

{
  "password": "Say th' secret phrase and enter. It's  over there",
  "welcome": "Welcome, me bucko!"
}

Congratulations!

Conclusion

Thank you for sticking with me today, and happy coding!

Introducing Developer Hub: A new resource for developers

Ilya Krukowski — Mon, 08 Aug 2022 10:19:00 +0000

We’d like to tell you about a new resource we’ve created, the Lokalise Developer Hub. It’s a single place to host all relevant information for developers including guides, sample apps written in different languages, API explorer where you can test endpoints in real time, tips, docs, API and SDK changelogs. So, let’s see what it has to offer.

REST API docs and explorer

We have a dedicated section providing comprehensive documentation for all the API endpoints that Lokalise supports. Here you can also find information on authentication (with API tokens and OAuth 2), pagination, rate limiting and branching support.

And last but not the least is the ability to test our API by sending real requests using the API explorer. To get started, you’ll only require an API token and a project ID (for certain endpoints you might require a team ID instead):

Log into your Lokalise account or create a new one.
Click on the avatar in the bottom left corner, choose Profile settings, proceed to the API tokens tab and generate a new token. Make sure to never publicly expose it and treat it as your password.
Next, open your translation project, click More > Settings, and under the General tab find your project ID.
Next, return to the API explorer, choose one of the endpoints from the left menu and, paste your API token into the X-Api-Token header text field, fill in the other request params as needed, and press Try It. Then you’ll be able to observe the response.

To learn more about the API explorer, you can watch the short video tutorial found on the “getting started” page.

Guides and tutorials

Lokalise apps and engines

Apart from the API docs, we also provide additional guides and articles related to apps and engines. So, what's an app, you ask? In terms of Lokalise, we consider an app any piece of code that expands Lokalise's functionality. An app can come in handy when you want to automate your localization process and workflows, customize and extend Lokalise core capabilities, or connect Lokalise with other systems. For example, we already have apps to send notifications via Slack or Discord and to automatically create tasks on Asana or Jira. There are various types of apps available, and you can read about those in the corresponding Developer Hub article.

An engine, in turn, facilitates the content exchange between Lokalise and ecommerce, PIM systems, or other content management systems, for example, Contentful, Storyblok, or Wordpress.

A custom processor is one example of a Lokalise engine that we’ve introduced recently. To put it simply, a custom processor is a script that runs on a third-party service and manages translations that are uploaded and downloaded on Lokalise.

To better understand the idea, let's see how a regular upload is performed on Lokalise:

You choose one or more translation files to upload.
Lokalise detects the translation file languages automatically, but you can adjust those as needed.
You adjust any additional uploading options and click Upload.
Lokalise parses the uploaded data, and extracts keys and translation values.
Extracted data is displayed in the translation editor.

However, if you have a custom processor set up, after performing step 4 Lokalise will automatically send the parsed data to your service. There you can perform any additional actions, such as analyzing the uploaded data, transforming your translations, adding special formatting, removing unneeded special characters or certain banned words, restructuring translation keys, and so on. The only requirement is that the data returned by your processor must preserve the initial structure. A custom processor can also intercept the downloaded data in the same way.

This is very convenient, for example, when you would like to perform complex text analysis, remove banned words from your translations, add custom formatting, and so on.

Sample apps

Of course you might want to build a custom app to work with Lokalise, and to help you get started we’ve prepared a collection of tutorials with app samples. Specifically, you can learn how to work with the Lokalise API, implement an OAuth 2 flow and act on the user's behalf, utilize webhooks to react to certain events happening in Lokalise. On top of that, we provide boilerplate examples for custom processors showing how exactly you can process received data.

Currently we provide code samples in the PHP, JS, Python, and Ruby languages.

App publishing process

After you’ve built an app, you might want to feature it on Lokalise so that other customers can take advantage of it (however, it’s not mandatory). You can refer to the app publishing section to learn about requirements regarding the apps, and what the overall publishing process looks like. Also, you can find an app review checklist and some useful tips on building successful apps.

I18n tutorials

If you are not looking to build a custom Lokalise app but rather want to know how to implement internationalization (i18n) in your project, this section is for you. We have a vast collection of I18n tutorials explaining how to build multilingual apps with different technologies. Our articles cover dozens of technologies including React, Angular, Django, Rails, Laravel, and many, many others.

Changelogs

Finally, changelogs for all our official API SDKs and the API itself can be found on the DevHub. You’ll never miss an update!

Conclusion

So, in this article we’ve briefly explored the Lokalise DevHub and checked out what it has to offer. We sincerely hope this new resource will help developers build something truly wonderful.

If you have any feedback for us, please don’t hesitate to fill out this form – we would love to hear from you.

That’s all for now. Stay tuned and happy coding!