Simple way to make i18n support in Rust with with examples and tests

Localization is crucial for modern software development. By supporting multiple languages, applications can reach a wider audience and become more inclusive, aligning with OnlyCoiners' mission.

However, managing translations efficiently in a multi-threaded environment can be challenging. In this post, we'll explore how to leverage Rust's OnceCell and Mutex to handle translations stored in JSON files, caching them in memory to facilitate effective localization throughout the application.

You might be curious why we chose not to use an established solution like rust-i18n. What we want was to create a version equivalent to the Python code snippet implemented in our FastAPI server, as described in this post so we can reuse translation files and ease the process of rewriting some Python code in our Rust OnlyCoiners API server.

You can test it working here in production.

You can create an API token here first after you make an account at OnlyCoiners.

If your company is looking to hire a Rust developer or provide support to another organization using Rust in production, consider posting a job at OnlyCoiners.

Find and post Rust jobs at our job board.

Contact us for inquiries, and we’ll be happy to offer you and your company exclusive discounts and other benefits if you are hiring or want to have a partnership with us!

Feel free to join our platform if you are a Rustacean and we welcome all of you and hire as well!

You can see the version ES ou PT also.

You can see the original post at our website.

This week in Rust team, if you share this post, please use the original post as a way to support a company use Rust in production.

Complete Code Snippet for translator.rs

Before we proceed, we like to present the complete Rust code snippet for the Translator struct, designed to facilitate internationalization within your Rust codebase.

use once_cell::sync::OnceCell;
use serde_json;
use std::collections::HashMap;
use std::sync::Mutex;
use std::{env, fs};

// 1. Global static storage for translation modules
static TRANSLATIONS: OnceCell<Mutex<HashMap<String, HashMap<String, HashMap<String, String>>>>> =
    OnceCell::new();

pub struct Translator {
    lang: String,
}

impl Translator {
    // 2. Initialize the translator with a language
    pub fn new(lang: &str) -> Self {
        // Ensure translations are loaded once
        let _ = TRANSLATIONS.get_or_init(|| Mutex::new(HashMap::new()));
        Translator {
            lang: lang.to_string(),
        }
    }

    // 3. Load translations from files or other sources
    fn load_translation_module(
        &self,
        file_key: &str,
    ) -> Option<HashMap<String, HashMap<String, String>>> {
        let mut translations = TRANSLATIONS.get().unwrap().lock().unwrap();

        // Get the current working directory and construct the full path dynamically
        let current_dir = env::current_dir().unwrap();
        let module_path =
            current_dir.join(format!("src/translations/{}/{}.json", self.lang, file_key));

        // If translation is already loaded, return a cloned version
        if let Some(file_translations) = translations.get(file_key) {
            return Some(file_translations.clone());
        }

        // Load the translation file - error.json
        match fs::read_to_string(module_path) {
            Ok(content) => {
                // Parse the JSON into a nested HashMap - error -> common -> internal_server_error
                let file_translations: HashMap<String, HashMap<String, String>> =
                    serde_json::from_str(&content).unwrap_or_default();

                translations.insert(file_key.to_string(), file_translations.clone());
                Some(file_translations)
            }
            Err(e) => {
                tracing::error!("Error loading translation file - {}", e);
                None
            }
        }
    }

    // 4. Translate based on key and optional variables
    pub fn t(&self, key: &str, variables: Option<HashMap<&str, &str>>) -> String {
        let parts: Vec<&str> = key.split('.').collect();
        let file_key = parts.get(0).unwrap_or(&""); // "error"
        let section_key = parts.get(1).unwrap_or(&""); // "common"
        let translation_keys = &parts[2..]; // "INTERNAL_SERVER_ERROR"

        // Load the correct translation module (e.g., "error.json")
        if let Some(translation_module) = self.load_translation_module(file_key) {
            if let Some(section) = translation_module.get(*section_key) {
                let mut current_value: Option<&String> = None;

                // Traverse the translation keys to get the final string value
                for translation_key in translation_keys {
                    if current_value.is_none() {
                        // At the beginning, current_value is None, so we access the section (a HashMap)
                        if let Some(next_value) = section.get(*translation_key) {
                            current_value = Some(next_value);
                        } else {
                            return format!("Key '{}' not found in '{}' locale", key, self.lang);
                        }
                    }
                }

                // At this point, current_value should be a &String
                if let Some(translation_string) = current_value {
                    let mut translated_text = translation_string.clone();

                    // Handle variables if present
                    if let Some(variables) = variables {
                        for (variable, value) in variables {
                            let variable_format = format!("{{{}}}", variable);
                            translated_text = translated_text.replace(&variable_format, value);
                        }
                    }
                    translated_text
                } else {
                    format!("Key '{}' not found in '{}' locale", key, self.lang)
                }
            } else {
                format!(
                    "Section '{}' not found in '{}' locale",
                    section_key, self.lang
                )
            }
        } else {
            format!("Module '{}' not found for '{}' locale", file_key, self.lang)
        }
    }
}

Why Use Static Storage for Translations?

When working on multi-threaded applications, handling global data requires careful thought. Without proper synchronization, you may encounter data races, crashes, or other issues. Rust provides tools like OnceCell and Mutex to solve these problems safely.

OnceCell ensures that a value is initialized only once and provides access to it across threads. Mutex guarantees safe, mutable access to shared data between threads by locking access when one thread is reading or writing.

By combining these two, we can create a static global storage that caches translation files in memory, so they are loaded once and then reused throughout the lifetime of the program. This approach avoids repeatedly loading files from disk and ensures translations are handled safely in a concurrent environment.

The Code Walkthrough

Let’s dive into the code that powers this translation system. It leverages a combination of OnceCell, Mutex, and a nested HashMap to load and store translations from JSON files. Once a file is loaded, it’s cached in memory and reused for subsequent requests.

1. Global Translation Storage

The translation data is stored in a global static variable, TRANSLATIONS, which uses OnceCell and Mutex to ensure the data is thread-safe and only initialized once. The structure of the HashMap allows for organizing translations in a hierarchical way:

• The first level stores translations by file key like error.json.

• The second level groups translations by section key like common.

• The third level stores the actual translation key-value pairs.
  

use once_cell::sync::OnceCell;
use std::collections::HashMap;
use std::sync::Mutex;

static TRANSLATIONS: OnceCell<Mutex<HashMap<String, HashMap<String, HashMap<String, String>>>>> =
    OnceCell::new();

Here’s how the nested HashMap works:

• File key like "error" points to a map of section keys.
• Each section key like "common" contains the translation strings, organized by keys like "internal_server_error", with corresponding messages like "Internal server error" as you can see below in json file used in production for OnlyCoiners API server.

src/translations/en/error.json

{
  "common": {
    "internal_server_error": "Internal server error",
    "not_authorized": "You are not authorized to use this resource",
    "not_found": "{resource} not found"
  },
  "token": {
    "no_api_user_token": "API-USER-TOKEN header is not included",
    "invalid_api_user_token": "API-USER-TOKEN header is not valid",
    "no_api_admin_token": "API-ADMIN-TOKEN header is not included",
    "unable_to_read_api_token": "Unable to read API Token"
  },
  "database": {
    "unable_to_query_database": "Unable to query database"
  }
}

2. Initializing the Translator

The Translator struct represents an object that is tied to a specific language like "en" for English or "pt" for Portuguese. When we create a Translator instance, the TRANSLATIONS global variable is initialized if it hasn’t been already.

pub struct Translator {
    lang: String,
}

impl Translator {
    pub fn new(lang: &str) -> Self {
        let _ = TRANSLATIONS.get_or_init(|| Mutex::new(HashMap::new()));
        Translator {
            lang: lang.to_string(),
        }
    }
}

This ensures that the global storage for translations is set up and ready to be used. The lang field in the Translator struct stores the language code, such as "en" for English or "es" for Spanish, and is used when loading translation files.

3. Loading Translation Files

The load_translation_module function is responsible for loading translation data from a file like src/translations/en/error.json. It reads the JSON file, parses it, and stores the data in the global TRANSLATIONS map for future use. If the file has already been loaded, it simply returns the cached version.

use std::{env, fs};

fn load_translation_module(
    &self,
    file_key: &str,
) -> Option<HashMap<String, HashMap<String, String>>> {
    let mut translations = TRANSLATIONS.get().unwrap().lock().unwrap();
    let current_dir = env::current_dir().unwrap();
    let module_path =
        current_dir.join(format!("src/translations/{}/{}.json", self.lang, file_key));

    if let Some(file_translations) = translations.get(file_key) {
        return Some(file_translations.clone());
    }

    match fs::read_to_string(module_path) {
        Ok(content) => {
            let file_translations: HashMap<String, HashMap<String, String>> =
                serde_json::from_str(&content).unwrap_or_default();
            translations.insert(file_key.to_string(), file_translations.clone());
            Some(file_translations)
        }
        Err(e) => {
            tracing::error!("Error loading translation file - {}", e);
            None
        }
    }
}

This function does the following.

1. Check if the file is already loaded: If it is, it returns the cached data from the TRANSLATIONS map.
2. Load the translation file: If the file hasn’t been loaded yet, it reads the JSON file from the src/translations/{lang}/{file}.json path, parses the content into a HashMap, and stores it in memory.
3. Handle errors: If the file cannot be read for exmaple, if it doesn’t exist, an error message is logged, and the function returns None.

4. Translating Keys with Variables

Once the translations are loaded, you can retrieve them using the t function. This function takes a key, which is a dot-separated string. For example, "error.common.internal_server_error", and retrieves the corresponding translation string. It also supports variable replacement, allowing you to insert dynamic values into the translation.

use serde_json;

pub fn t(&self, key: &str, variables: Option<HashMap<&str, &str>>) -> String {
    let parts: Vec<&str> = key.split('.').collect();
    let file_key = parts.get(0).unwrap_or(&"");
    let section_key = parts.get(1).unwrap_or(&"");
    let translation_keys = &parts[2..];

    if let Some(translation_module) = self.load_translation_module(file_key) {
        if let Some(section) = translation_module.get(*section_key) {
            let mut current_value: Option<&String> = None;

            for translation_key in translation_keys {
                if current_value.is_none() {
                    if let Some(next_value) = section.get(*translation_key) {
                        current_value = Some(next_value);
                    } else {
                        return format!("Key '{}' not found in '{}' locale", key, self.lang);
                    }
                }
            }

            if let Some(translation_string) = current_value {
                let mut translated_text = translation_string.clone();
                if let Some(variables) = variables {
                    for (variable, value) in variables {
                        let variable_format = format!("{{{}}}", variable);
                        translated_text = translated_text.replace(&variable_format, value);
                    }
                }
                translated_text
            } else {
                format!("Key '{}' not found in '{}' locale", key, self.lang)
            }
        } else {
            format!(
                "Section '{}' not found in '{}' locale",
                section_key, self.lang
            )
        }
    } else {
        format!("Module '{}' not found for '{}' locale", file_key, self.lang)
    }
}

This function does these.

1. Splits the key into parts: file_key, section_key, and the actual translation key(s).
2. Loads the translation file: It calls load_translation_module to ensure that the correct file is loaded.
3. Traverses the keys: It navigates through the file’s HashMap to find the desired translation string.
4. Handles dynamic variables: If the translation contains placeholders like {username}, they are replaced with the values passed in the variables map.

For example, if the translation string is "{username}, Create, Earn and Network with OnlyCoiners!" and you provide {"username": "Rust"}, the final result will be "Rust, Create, Earn and Network with OnlyCoiners!".

Error Handling

The system is designed to provide useful error messages when translations cannot be found. For instance, if a section or key is missing, it returns a message like:

Key 'error.common.INTERNAL_SERVER_ERROR' not found in 'en' locale

This ensures that developers can easily spot missing translations during development.

Usage examples in production

The translator module is used in production at OnlyCoiners API server.

We will give you a few code snippets you can use as a reference. You can first make a mdware like this for axum.

// #[derive(Clone)]
// pub struct Language(pub String);

use std::collections::HashSet;

use crate::{constants::language::{ALLOWED_LANGUAGE_LIST, EN}, schemas::language::Language};
use axum::{extract::Request, middleware::Next, response::Response};

pub async fn extract_client_language(
    mut request: Request, // mutable borrow for later modification
    next: Next,
) -> Result<Response, String> {
    let accept_language = {
        // Temporarily borrow the request immutably to get the header
        request
            .headers()
            .get("Accept-Language")
            .and_then(|value| value.to_str().ok())
            .unwrap_or("")
            .to_string() // convert to String to end the borrow
    };

    let mut locale = accept_language.split(',').next().unwrap_or(EN).to_string();
    // Remove any region specifier like en-US to en
    locale = locale.split('-').next().unwrap_or(EN).to_string();

    // Create a set of allowed languages for quick lookup
    let allowed_languages: HashSet<&str> = ALLOWED_LANGUAGE_LIST.iter().cloned().collect();

    // Verify if the extracted locale is allowed; if not, default to the default language
    if !allowed_languages.contains(locale.as_str()) {
        locale = EN.to_string();
    }

    // Insert the language into request extensions with mutable borrow
    request.extensions_mut().insert(Language(locale));

    // Proceed to the next middleware or handler
    let response = next.run(request).await;

    Ok(response)
}

You can include that to your axum app.

let app = Router::new()
        .route("/", get(root))
        // Attach `/api` routes
        .nest("/bot", bot_routes)
        .nest("/admin", admin_routes)
        .nest("/api", api_routes)
        .layer(from_fn(extract_client_language))

Then, use it inside your handler.

pub async fn find_user_list(
    Extension(session): Extension<SessionData>,
    Extension(language): Extension<Language>,
) -> Result<Json<Vec<UserListing>>, (StatusCode, Json<ErrorMessage>)> {
    let translator = Translator::new(&language.0);
    let not_authorized = translator.t("error.common.not_authorized", None);

    Err((
        StatusCode::UNAUTHORIZED,
        Json(ErrorMessage {
            text: not_authorized,
        }),
    ))
}

Optionally, you can make tests for Translator module and use $cargo test to test.

#[cfg(test)]
mod tests {
    use crate::translations::translator::Translator;

    use super::*;
    use std::collections::HashMap;

    #[test]
    fn test_translation_for_english_locale() {
        let translator = Translator::new("en");

        let translation = translator.t("error.common.internal_server_error", None);
        assert_eq!(translation, "Internal server error");

        let not_found = translator.t("error.common.non_existent", None);
        assert_eq!(not_found, "Key 'error.common.non_existent' not found in 'en' locale");
    }

    #[test]
    fn test_translation_for_portuguese_locale() {
        let translator = Translator::new("pt");

        // Test known translation
        let translation = translator.t("error.common.internal_server_error", None);
        println!("translation {}", translation);
        assert_eq!(translation, "Erro interno no servidor");

        // Test key not found
        let not_found = translator.t("error.common.non_existent", None);
        assert_eq!(not_found, "Key 'error.common.non_existent' not found in 'pt' locale");
    }

    #[test]
    fn test_translation_with_variables() {
        let translator = Translator::new("en");

        let mut variables = HashMap::new();
        variables.insert("resource", "User");

        let translation_with_vars = translator.t("error.common.not_found", Some(variables));
        assert_eq!(translation_with_vars, "User not found");
    }

    #[test]
    fn test_translation_module_not_found() {
        let translator = Translator::new("es");

        // Test loading a non-existent module
        let translation = translator.t("non_existent_module.common.internal_server_error", None);
        assert_eq!(
            translation,
            "Module 'non_existent_module' not found for 'es' locale"
        );
    }

    #[test]
    fn test_translation_section_not_found() {
        let translator = Translator::new("en");

        // Test section not found in translation file
        let translation = translator.t("error.non_existent_section.internal_server_error", None);
        assert_eq!(
            translation,
            "Section 'non_existent_section' not found in 'en' locale"
        );
    }
}

You can also test the mdware.

pub async fn test_handler(Extension(language): Extension<Language>) -> Json<serde_json::Value> {
    // Return a JSON response with the extracted language
    let response_message = match language.0.as_str() {
        "en" => "en",
        "pt" => "pt",
        "es" => "es",
        _ => "deafult",
    };

    Json(json!({ "message": response_message }))
}

#[cfg(test)]
mod tests {
    use crate::{
        constants::language::{EN, EN_US, ES, PT}, mdware::language::extract_client_language, tests::{test_handler, TRANSLATOR}  
    };
    use axum::{
        body::{to_bytes, Body}, http::Request, middleware::from_fn, Router
    };
    use hyper::StatusCode;
    use tower::ServiceExt; 
    use serde_json::{json, Value};

    #[tokio::test]
    async fn test_with_valid_accept_language_header() {
        let app = Router::new()
            .route("/", axum::routing::get(test_handler))
            .layer(from_fn(extract_client_language)); 

        // Simulate a request with a valid Accept-Language header like
        let request = Request::builder()
            .header("Accept-Language", EN_US) // "en-US"
            .body(Body::empty())
            .unwrap();

        let response = app.oneshot(request).await.unwrap();
        assert_eq!(response.status(), StatusCode::OK);

        let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
        let body_str = std::str::from_utf8(&body).unwrap();
        // println!("Response Body: {:?}", body_str);

        let body_json: Value = serde_json::from_str(body_str).unwrap();
        assert_eq!(body_json, json!({ "message": EN }));
    }

    #[tokio::test]
    async fn test_with_valid_accept_language_header_wiht_pt() {
        let app = Router::new()
            .route("/", axum::routing::get(test_handler))
            .layer(from_fn(extract_client_language)); 

        let request = Request::builder()
            .header("Accept-Language", "pt")
            .body(Body::empty())
            .unwrap();

        let response = app.oneshot(request).await.unwrap();
        assert_eq!(response.status(), StatusCode::OK);

        let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
        let body_str = std::str::from_utf8(&body).unwrap();

        let body_json: Value = serde_json::from_str(body_str).unwrap();
        assert_eq!(body_json, json!({ "message": PT }));
    }

    #[tokio::test]
    async fn test_with_valid_accept_language_header_wiht_pt_br() {
        let app = Router::new()
            .route("/", axum::routing::get(test_handler))
            .layer(from_fn(extract_client_language)); 

        let request = Request::builder()
            .header("Accept-Language", "pt-BR")
            .body(Body::empty())
            .unwrap();

        let response = app.oneshot(request).await.unwrap();
        assert_eq!(response.status(), StatusCode::OK);

        let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
        let body_str = std::str::from_utf8(&body).unwrap();

        let body_json: Value = serde_json::from_str(body_str).unwrap();
        assert_eq!(body_json, json!({ "message": PT }));
    }

    #[tokio::test]
    async fn test_with_valid_accept_language_header_wiht_es() {
        let app = Router::new()
            .route("/", axum::routing::get(test_handler))
            .layer(from_fn(extract_client_language)); 

        let request = Request::builder()
            .header("Accept-Language", "es")
            .body(Body::empty())
            .unwrap();

        let response = app.oneshot(request).await.unwrap();
        assert_eq!(response.status(), StatusCode::OK);

        let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
        let body_str = std::str::from_utf8(&body).unwrap();
        let body_json: Value = serde_json::from_str(body_str).unwrap();
        assert_eq!(body_json, json!({ "message": ES }));
    }

    #[tokio::test]
    async fn test_with_unsupported_language() {
        let app = Router::new()
            .route("/", axum::routing::get(test_handler))
            .layer(from_fn(extract_client_language)); 

        let request = Request::builder()
            .header("Accept-Language", "fr")
            .body(Body::empty())
            .unwrap();

        let response = app.oneshot(request).await.unwrap();
        assert_eq!(response.status(), StatusCode::OK);

        let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
        let body_str = std::str::from_utf8(&body).unwrap();

        let body_json: Value = serde_json::from_str(body_str).unwrap();
        assert_eq!(body_json, json!({ "message": EN }));
    }

    #[tokio::test]
    async fn test_without_accept_language_header() {
        let app = Router::new()
            .route("/", axum::routing::get(test_handler))
            .layer(from_fn(extract_client_language)); 

        let request = Request::builder()
            .body(Body::empty())
            .unwrap();

        let response = app.oneshot(request).await.unwrap();
        assert_eq!(response.status(), StatusCode::OK);

        let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
        let body_str = std::str::from_utf8(&body).unwrap();

        let body_json: Value = serde_json::from_str(body_str).unwrap();
        assert_eq!(body_json, json!({ "message": EN }));
    }
}

You can use these json translation files as references.

en.json
{
  "common": {
    "internal_server_error": "Internal server error",
    "not_authorized": "You are not authorized to use this resource",
    "not_found": "{resource} not found"
  },
}
pt.json
{
  "common": {
    "internal_server_error": "Erro interno no servidor",
    "not_authorized": "Você não está autorizado a usar este recurso",
    "not_found": "{resource} não encontrado"
  },
}
es.json
{
  "common": {
    "internal_server_error": "Error interno del servidor",
    "not_authorized": "No estás autorizado para usar este recurso",
    "not_found": "{resource} no encontrado"
  },
}

Conclusion

This translation system efficiently handles translations in a Rust application using static storage and thread-safe access. By leveraging OnceCell and Mutex, we can ensure that translation files are loaded once and cached, improving performance and reducing disk access. The t function allows for flexible translation retrieval with support for dynamic variables, making it a powerful tool for localization.

If you're building an application that requires localization, this approach provides a simple, scalable and efficient solution for managing translations. By using Rust’s powerful memory safety features, you can ensure that your translations are handled securely and efficiently across multiple threads.

We hope this post has helped you implement a simple translation system using Rust. We actively use Rust in production and are looking to hire more Rust developers.

Feel free to join our platform if you liked this post and support a company using Rust in production!