Localizing Image Text Overlays using Cloudinary + 8base

Imagine that you're a social media intern. Please, bear with me! You're a social media intern and your manager, instead of including you in on strategy meetings, hands you a laundry list of image-related tasks. It includes:

1. Crop this.
2. Brighten that.
3. Overlay "New arrivals!" on the Twitter header image.
4. Resize the Facebook share card.
5. Get me coffee.
6. Translate "New arrivals!" to Russian, German, and Swahili.

...You get the point

Now, you're a smart cookie. There's no way you want to spend your day wasting away on the computer having to MANUALLY manipulate all these images. So, you set out to find a better way.

⏸ Story over, for now

That scenario is no made-up story. It's real! Every day, whether at work or for personal projects, millions of images get created, edited, updated, hosted, taken down, lost, and so on. Services that help manage the chaos or streamline the process can be incredibly helpful.

The other day, a friend shared with me Cloudinary's URL API. Immediately, I saw it as being an answer to so many image-related inefficiencies suffered by companies and people. Very quickly, I'll mention a why and a how.

Why

From a single image, dozens of tweaked versions might need to get created (faded, text-overlay, black-white, etc.). Each of those versions takes time to create, update, and organize.

How

Cloudinary's URL API takes a dynamic parameter that applies transformations to the image upon retrieval. Think of it like on-demand Photoshop!

Personally, this got me excited. Notably, the Text Overlay transformation. After spending a little time playing with it, I wanted to see if it could be extended to incorporate localization (translation) of image text.

A working demo came out of the exercise. You can play with it here, or keep reading and learn how it works!

First off, let's take a quick look at the anatomy of the URL API. A large section of parameters exists between the upload/ and /horse.jpg sections. These are a list of image transformations that get executed when the image is requested. Pretty cool! Right? The documentation is right here if you'd like to dive deeper.

https://res.cloudinary.com/demo/image/upload/c_crop,g_face,ar_16:9,w_1200,h_600/e_auto_contrast/b_rgb:00000099,e_gradient_fade,y_-0.4/co_white,fl_relative,l_text:Times_100_bold_italic:I%20am%20a%20unicorn!,w_0.95/co_black,e_shadow,x_2,y_1/fl_layer_apply,g_south_west,x_20,y_25/dpr_auto,q_auto,f_auto/horse.jpg

Now, the image you see below gets rendered using the link above. Moreover, - this is the crucial part - if you change the transformation, a brand new image gets returned!

The l_text:Times_100_bold_italic:I%20am%20a%20unicorn! is easily visible when inspecting the URL. While we can't add a custom transformation tags (that is, on Cloudinary's side), we do have the ability to apply transformations to the URL. Meaning that, in the case of localizing our image overlays, we can coerce the URL before requesting the image.

A serverless GraphQL resolver function can get deployed to an 8base workspace to accomplish this. It can handle the parsing of the URL and translation. There are many ways to deploy a serverless function. However, 8base made it super simple and straight forward.

As a quick specification, let's design the function to behave as follows.

1. If a local_[2-char-lang-code] tag precedes the text, translate the text, and update the URL.
2. If a local code does not precede the text, return the original URL.

Enough talk, let's do it

1. Create a new 8base project

If you have an existing projected, you can always add a new function to it.

# Install the CLI globally
npm install -g 8base-cli

# Initialize a new project with a GraphQL resolver called "localizer."
8base init my-project --functions="resolver:localizer"

These commands create a new project with all the files and code we need to start invoking our GraphQL resolver function. We'll need to make a few changes though before it's translating our Cloudinary URL's 😉

2. Update the resolver's graphql.schema

Open up the file at src/resolvers/localizer/schema.graphql. We need to define our query operation and response. In this case, we'll be returning an object with the updated url after having received the cloudinaryUrl. Update the file with the following snippet.

type LocalizeResult {
  url: String!
}

extend type Query {
  localize(cloudinaryUrl: String!): LocalizeResult
}

3. Update the mock for invoke-local

Update src/resolvers/localizer/mocks/request.json so that the function can get invoked locally with data. The mock file generated has the same schema as what gets passed to the function in production.

{
  "data": {
    "cloudinaryUrl": "https://res.cloudinary.com/cdemo/image/upload/c_crop,g_face,ar_16:9,w_1200,h_600/e_auto_contrast/b_rgb:00000099,e_gradient_fade,y_-0.4/co_white,fl_relative,l_text:Times_100_bold_italic:local_es:Breaking%20news:%208base%20solves%20all%20your%20image%20related%20needs!,w_0.95/co_black,e_shadow,x_2,y_1/fl_layer_apply,g_south_west,x_20,y_25/dpr_auto,q_auto,f_auto/dosh1/img-0.jpg"
  },
  "headers": {
    "x-header-1": "header value"
  },
  "body": "{\"cloudinaryUrl\":\"https://res.cloudinary.com/cdemo/image/upload/c_crop,g_face,ar_16:9,w_1200,h_600/e_auto_contrast/b_rgb:00000099,e_gradient_fade,y_-0.4/co_white,fl_relative,l_text:Times_100_bold_italic:local_es:Breaking%20news:%208base%20solves%20all%20your%20image%20related%20needs!,w_0.95/co_black,e_shadow,x_2,y_1/fl_layer_apply,g_south_west,x_20,y_25/dpr_auto,q_auto,f_auto/dosh1/img-0.jpg\"}"
}

4. The function

We're going to need a translation engine. I chose AWS Translate, which offers 2-million free characters per month. Let's add the required library and config to the project.

# Install AWS SDK
npm install --save aws-sdk

Update src/resolvers/localizer/handler.ts

const AWS  = require('aws-sdk');

AWS.config.update({
    region: 'us-east-1', 
    credentials: {
      accessKeyId: process.env.AWS_IAM_SECRET_KEY, 
      secretAccessKey: process.env.AWS_IAM_ACCESS_KEY 
    }
});

const translate = new AWS.Translate({ apiVersion: '2017-07-01' });

/* Other code ... */

When developing locally, you'll need to set your AWS credentials as environment variables or static values. The example you see above is what works when the function gets deployed to 8base. Here's the documentation on accessing 8base environment variables.

Since we're using TypeScript, the function response needs a type. This type must match the structure and name of that added to the graphql.schema file. For our scenario, prepend the following to the function body.

type LocalizeResult = {
  data: {
    url: string
  }
};

The function body is pretty self-explanatory. Instead of describing it here and then showing it there, please read the inline comments for clarification on what's happening.

export default async (event: any, ctx: any) : Promise<LocalizeResult> => {
  /**
   * Regex Statement for matching our custom local_tag and preceeding text
   */
  const REG_EX = /(local_[a-z]{2})\:(.*?)([,\/])/g
  /**
   * Pull the given cloudinary url from our function arguments 
   */
  let url = event.data.cloudinaryUrl
  /**
   * Execute our Regex statement returning a match object
   */
  const matchObj = REG_EX.exec(url);
  /**
   * If a local tag is matched, we're in business! If not,
   * we're simply returning the passed url.
   */
  if (matchObj) {
    /**
     * Pull out the matched local and text values from
     * the matchObj array.
     */
    let local = matchObj[1], text  = matchObj[2];

    try {
      /**
       * Make the request to AWS Translate after decoding the given text
       * and slicing the last two characters from the local tag (e.g. local_es)
       */
      let request = translate.translateText({
        TargetLanguageCode: local.slice(-2),
        SourceLanguageCode: 'auto',
        Text: decodeURI(text)
      }).promise();

      let data = await request;
      /**
       * The ACTUAL cloudinary url will break if it has our custom tag. Plus, we
       * need to update the text with the translation! So, let's replace the previously
       * matched locale and text with our tranlsated text, that needs to be escaped.
       */
      url = url.replace(`${local}:${text}`, data.TranslatedText.replace(/[.,%\`\s]/g,'%20'))
    } 
    catch (err) {
      console.log(err, err.stack);
    }    
  } 
  /**
   * Return the final result.
   */
  return {
    data: {
      url
    }
  }
};

5. Run it!

Done! Let's prove it by invoking our function locally. The returned URL's text section translates to the locale specified language! Copy the link and throw it in a browser to see the magic.

8base invoke-local localize -p src/resolvers/localize/mocks/request.json
invoking...

Result:
{
  "data": {
    "localize": {
      "url": "https://res.cloudinary.com/demo/image/upload/c_crop,g_face,ar_16:9,w_1200,h_600/e_auto_contrast/b_rgb:00000099,e_gradient_fade,y_-0.4/co_white,fl_relative,l_text:Times_100_bold_italic:¡Soy%20un%20unicornio%20genial!,w_0.95/co_black,e_shadow,x_2,y_1/fl_layer_apply,g_south_west,x_20,y_25/dpr_auto,q_auto,f_auto/horse.jpg"
    }
  }
}

🏁 Wrap up

Sorry, we're going back to storytime. Remember back when you were a social media intern? Well, you ended up finding and using Cloudinary for all your on-the-fly image transformation and 8base for lightening fast serverless deployment of serverless GraphQL functions.

Excited by the chance to become "Employee of the Month", you approach your boss and share with him the big news by saying:

"I was able to apply dynamic URL transformations to our images using a URL API and extend its functionality to support real-time translations of text overlay!"

Seemingly confused, your manager looks at your hands and responds:

"You forgot my coffee?"

Cloudinary and 8base both do A LOT more than what is in this post. I highly recommend you check them out!

Comments

[Tammy Lee]

Great post, Seb! I've been curious about localization and it hadn't even occurred to me to use Cloudinary's text overlay like this!