Using Meta-Programming For Internationalization and Localization in Nim

Hein Thant Maung Maung — Fri, 03 Mar 2023 15:23:24 +0000

TL;DR

Check this out: https://github.com/heinthanth/ni18n

What is Nim

The Nim Programming Language is a Efficient, Expressive, Elegant programming language that is transpiled to different backend ( C, C++, ObjC, JavaScript, etc. ). Nim compiler has Nim VM, embedded in it, which allow us to execute code, generate code at compile-time.

The Problem

Managing translations for internationalizing or localizing an app was never easy. For example, We have to worry about:

missing translation for certain locales
having typo in translation name ( key )
invalid substitution or interpolation, etc.

These are some problems or difficulties I found while developing internationalized web apps and things like that.

ni18n to the rescue

What if we can validate our translations and their usage at compile-time? Here's my approach to solve those problems.

DSL

I created a custom DSL to write translations. So that I can traverse that DSL to validate translations and generate code for it.

type
    Locale = enum
        English
        Chinese
        Myanmar

i18nInit Locale, true:
    hello:
        # translations can be string literal
        English = "Hello, $name!"
        Chinese = "你好, $name!"
        Myanmar = "မင်္ဂလာပါ၊ $name ရေ။"

With this DSL, I'm forcing myself to:

define a enum of supported locales
write translation for every locale
every translation signature / method must be the same for all locale for a given translation name ( key )

Code Generation

Nim has an awesome feature called macro. With this, you can write Nim AST and generate Nim code at compile-time.

Let's say, we want to generate a function that return a string like this:

proc someFn(): string {.inline.} =
    return "HELLO, WORLD!"

Then, we can write Nim AST like this:

nnkProcDef.newTree(
    ident("someFn")           # function name
    newEmptyNode()            # not interested in here
    newEmptyNode()            # no generic parameter
    nnkFormalParams.newTree(  # normal parameter
        ident("string")       # return type
    ),
    nnkPragma.newTree(        # pragma
        ident("inline")
    ),
    newEmptyNode(),           # reserved slot ( not interested in here )
    newStmtList(              # function body      
        nnkReturnStmt.newTree(
            newLit("HELLO, WORLD!")
        )
    )
)

So, with the same idea, we can convert a piece of the following DSL:

hello:
    English = "Hello"

into Nim function by emitting Nim AST like this:

nnkProcDef.newTree(
    newIdentNode("hello_English"),
    newEmptyNode(),
    newEmptyNode(),
    nnkFormalParams.newTree(
        newIdentNode("string"),
        nnkIdentDefs.newTree(
            newIdentNode("args"),
            nnkBracketExpr.newTree(
                newIdentNode("varargs"),
                newIdentNode("string")
            ),
            nnkBracket.newTree()
        )
    ),
    nnkPragma.newTree(
        newIdentNode("inline")
    ),
    newEmptyNode(),
    nnkStmtList.newTree(
        nnkReturnStmt.newTree(
            nnkCall.newTree(
                newIdentNode("format"),
                newLit("Hello"),
                newIdentNode("args")
            )
        )
    )
)

Now, we got the following Nim function generated at compile-time.

proc hello_English(args: varargs[string] = []): string {.inline.} =
    return format("Hello", args)

The same strategy goes for Chinese. We have another function generated like this:

proc hello_Chinese(args: varargs[string] = []): string {.inline.} =
    return format("你好", args)

What next? Well, we have to generate a lookup function that matches runtime locale value and call correct locale-suffixed function ( like hello_Chinese, hello_English, etc. ).

proc hello(locale: Locale, args: varargs[string] = []): string {.inline.} =
   case locale
   of English:
       return hello_English(args)
   of Chinese:
       return hello_Chinese(args)

Cool, right?

Validation

To check if we're missing certain locale, we can track the number of generated function like this:

# convert Locale enum into HashSet like {"English", "Chinese", "Myanmar"}
var missingLocales = allowedEnums.toHashSet()

for locale in generatedFns.keys:
    # then, remove locale that has generated function from the set
    missingLocales.excl(locale)

# so, locales that are left in missingLocales don't have translation
if missingLocales.len() > 0:
    # now, we know that some locale are missing

So, we can tell compiler to stop compiling and show error message like this:

let missing = missingLocales.toSeq().map(l => escape(l)).join(", ")
error("missing $# translations for $#" % [missing, escape(identToDotNotation(curIdent))], namePair)

So, If I write some code like below:

type
    Locale = enum
        English
        Chinese
        Myanmar

i18nInit Locale, true:
    hello:
        # translations can be string literal
        English = "Hello, $name!"
        Chinese = "你好, $name!"

Then, Nim compiler will complain me about translation of hello is missing for Myanmar locale.

For validating function signature, we can do something like this:

proc sameSignatureProc(a, b: NimNode): bool {.inline, compileTime.} =
    ## Checks if two proc definitions have the same signature.
    expectKind(a, nnkProcDef)
    expectKind(b, nnkProcDef)
    result = true

    # see above comment for ProcDef structure
    # to check if two procs have the same signature,
    # we need to check nnkGenericParams, nnkFormalParams and nnkPragma.
    # But generic can't be used in Lambda and not supported in ni18n,
    # so we will check just nnkFormalParams and nnkPragma
    if a[3] != b[3] or a[4] != b[4]: return false

proc sameSignatureProcs(fns: varargs[NimNode]): bool {.inline, compileTime.} =
    for i in 1 ..< fns.len(): (if not sameSignatureProc(fns[0], fns[i]): return false)
    result = true

sameSignatureProcs(generatedFns.values.toSeq())

So, if I write code like below:

type
    Locale = enum
        English
        Chinese
        Myanmar

i18nInit Locale, true:
    hello:
        English = proc(name: string): string =
            return "Hello, " & name & "!"
        Chinese = proc(name: bool): string =
            return "Hello, " & name & "!"
        Myanmar = proc(name: int): string =
            return "Hello, " & name & "!"

Then, Nim compiler will complain me about procedure ( function ) signature mismatch across locales.

Conclusion

Phewwww, we're done ... sounds complex, right? But that's pretty simple. I'd recommend you to read this article along with ni18n source code and Nim macro docs. It will help you understand a bit more.

The main idea is to:

define a DSL
traverse a DSL
generate a function if we find = assign operator
if we find : call operator, we will recursively traverse the RHS of : operator ( go back to step 2 )
Keep track of generated functions from step 3.
Validate the number and signature of generated functions

That's it! Hope you learn something new or did I blow your mind?

Thanks for reading till the end. I appreciate your curiosity!
Don't forget to check ni18n at https://github.com/heinthanth/ni18n and please consider giving a star if that's useful for you!

DEV Community: Hein Thant Maung Maung