DEV Community: spyke

A Look Back at Outsourcing in the Age of AI

spyke — Mon, 02 Mar 2026 22:05:36 +0000

Now that I am no longer working in outsourcing, it feels like the right moment to reflect on that industry (and IT consultancy more broadly). The timing couldn't be better -- we've entered the Age of Agentic AI. The similarities are so striking that it’s almost amusing.

The Promise of AI-Native Companies

In 2025, companies around the world began declaring themselves AI-native and AI-first. Token farms started consuming the majority of new investments. Then came the agents.

Why would anyone need an agent?

Because it lets you shift your position on the output–cost curve:

Increase output without hiring, reducing costs per unit of work.
Maintain output with fewer employees, reducing labor costs.
Scale output quickly by investing capital rather than waiting through the long hiring cycle.

Hiring takes time. Building a team takes even longer. Now you can transfer money and get a ready-to-work "team." Sounds amazing.

But we've had this capability for decades. It's called outsourcing and outstaffing.

You can buy services that range from a black-box "click a button to build a website" solution to having a person working in your office.

The Reputation Gap

Unlike AI agents, which are currently surrounded by optimism, outsourcing has long been infamous -- especially among smaller companies.

Startup incubators and accelerators tell founders to build teams strictly in-house. Investors push management to get rid of contractors to improve valuation.

Looking for a job with only outsourcing experience can be an uphill battle. I've heard countless variations of:

"I liked your answers, but I don't think you'd be able to do this job because you don't have product experience."

Why does this perception persist?

Were there bad outsourcing companies that damaged the industry's reputation? Probably. But is that so different from today's "slope"?

There are bad engineers and there are bad models. We still assume that AI agents are useful. So logically, outsourcing must also have good players.

Life as a "Third Party"

Here's what I saw over more than a decade on the other side.

In theory, you're helping others succeed. In practice, you're often treated as a third party.

Remember investors encouraging companies to build only in-house teams? That mindset permeates daily operations.

Examples:

Limited permissions: Sometimes it's forbidden to even message another employee on Slack to resolve an issue.
Tools: Many licenses are reserved for employees.
Production access: Connect to the DB, run a pipeline? No -- security concerns.
KPIs and goal meetings: Not for contractors.
Infrastructure cost data: "We appreciate your interest in optimization, but that’s internal information."

Trying to improve efficiency?

"You want to refactor that? Sounds like you're trying to scam us into paying your company more."

Coding standards inconsistently applied?

"Bob doesn't need to follow them -- his team is knows what they are doing."

Timesheets?

Contractors: Mandatory. And increase granularity to 3-hour maximums.
Employees: No need.

Team building?

"We'd love to invite you, but we can't."

Promotions?

"Senior roles are for employees only. Talk to your manager. Forgot to mention, Bob, the junior developer you onboarded three years ago now would be your tech lead. You did all the architectural work, so please help him."

Stock options? Bonuses?

"Your performance is outstanding even without them."

The Parallel with Agentic Teams

This is what the opportunity to get a "remote agentic team" often looked like from my perspective.

So I have questions:

Is it really that different to grant build pipeline access to a third-party user versus a third-party agent?
Could AI providers to raise prices just as an outsourcing company would?
When an LLM suggests refactoring or using a new tool, is that a scam or an improvement?
Is AGENTS.md fundamentally different from the project guidelines that were supposed to be written anyway?

And the bigger question:

If a company couldn't effectively utilize a team it could talk to in person, will it be able to utilize a team it can't?

Closing Thought

Outsourcing wasn't just about cost arbitrage. It was about elasticity -- about converting capital into capacity.

AI agents promise the same thing.

The real question is not whether agents will work. It's whether organizations that struggled with human third parties are structurally ready for non-human ones.

Localization Keys vs. Direct Text Keys

spyke — Mon, 03 Mar 2025 21:45:35 +0000

This is a Deep Research report on the topic of the advantages and disadvantages of using keys vs. direct English text for localization, considering your priorities of maintainability and ease of development in the context of i18next localization in a TypeScript + React.js project.

Internationalizing a React application involves deciding how to reference UI strings in code. Two common approaches are using abstract localization keys (e.g. UNT:BannerBody_NACS_adapter_required) and using the actual English text as the key (e.g. "You need to use a charging adapter on this route."). Each approach has its own advantages and disadvantages. This guide compares them across several factors — maintainability, scalability, ease of development, translation workflow, developer readability, cross-language consistency, and handling of plurals/dynamic content — and provides recommendations for using i18next in a TypeScript + React project. We’ll also cover best practices to keep translations maintainable and developer-friendly.

Approaches to Referencing Translation Strings

Key-Based Localization (Abstract Keys)

Description: In this approach, strings in the code are referenced by a key that acts as an identifier. The key is typically a short code or path (often in English or an encoded form) that maps to the actual text in a translation file. For example, t('errors.networkTimeout') might look up a JSON entry like "errors": { "networkTimeout": "The request timed out." }. The key itself is not shown to users; it’s only used to retrieve the correct localized text.

Pros (Key-Based):

Maintainable & Flexible Content: You can change the actual displayed text without altering application code – only the translation files need updates (Key based i18n vs default language i18n · Issue #50 · nodejs/i18n · GitHub). This means minor copy changes or tweaks in English (or any language) don’t require code deployments. The key stays the same (stable), so other languages’ translations remain linked to it even if the English phrasing changes (you’d simply update the English translation for that key) (Key based i18n vs default language i18n · Issue #50 · nodejs/i18n · GitHub). This stability is great for long-term maintainability when text may evolve.
Structured Organization: Keys can be organized hierarchically (e.g. grouped by screen or feature), improving manageability in large apps (8 Advantages of using translation keys in your localization files - POEditor Blog) (8 Advantages of using translation keys in your localization files - POEditor Blog). For instance, keys might be prefixed with the page or component name ("Checkout.PaymentError.CardDeclined"), making it clear where they are used. This organization helps keep hundreds or thousands of strings scalable and easy to find.
Context & Uniqueness: Well-chosen keys can encode context or meaning. This avoids collisions and ambiguity. For example, you might use separate keys like button.openFile vs status.fileOpen to distinguish the word “Open” as an action vs. a state (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). Dedicated identifiers provide extra context about where/how the text is used (Using english text as the translation csv file's keys. Good or terrible idea? : r/godot), helping translators choose the right translation. Keys are unique IDs, so the same English word in different contexts can have different translations.
Pluralization & Dynamic Content: Key-based catalogs integrate naturally with i18n features for plurals and interpolation. You define base keys and variant forms in translation files (e.g. "itemCount_one": "1 item", "itemCount_other": "{{count}} items"), and i18next will select the correct form when you call t('itemCount', { count }) (Plurals - i18next documentation). This keeps plural logic in the localization system. Similarly, keys with placeholders (e.g. "welcomeMessage": "Hello, {{name}}!") allow i18next to handle dynamic insertion. In code you just use the key with variables (no need to build strings manually).
Type Safety (with TypeScript): It’s possible to generate or maintain a TypeScript type for all valid keys, so the t(…) function only accepts known keys. This prevents typos or missing key errors at compile time. Many teams auto-generate TypeScript definitions from the JSON translation files. This benefit is more feasible with short keys than with long sentence keys.

Cons (Key-Based):

Development Overhead: Developers must define and maintain keys. Adding a new UI message means coming up with a new key and writing the English text in a translation file, effectively writing the message twice (once as key, once as value) (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). Without tooling, this extra step can slow down development and invite mistakes (e.g. forgetting to add the key to the translations). It’s more work upfront compared to just writing the string directly in the component.
Reduced Readability: Code is less self-explanatory. A key like UNT:BannerBody_NACS_adapter_required or even errors.networkTimeout doesn’t immediately tell a developer or designer what text the user will see. Developers may need to look up the key in the locale file to know the actual message. Poorly named keys make this worse (e.g. a generic error_42 gives no clue). This context switching can hinder quick understanding of the UI logic (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs). However, using descriptive keys (containing hints of the content) can mitigate this. For example, a key login.error.invalidPassword is somewhat readable, and some teams use “natural language” phrases as part of keys for clarity (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs) (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs).
Synchronization with Design/Copy Changes: If keys are very abstract or developer-oriented, it might be harder for non-developers (like PMs or copywriters) to see where text lives or to ensure updates are applied in all places. A robust process or tools (like a translation management system) is needed to keep keys and actual content in sync.
Potential for Outdated Keys: Because keys are meant to be stable, there’s a risk that the English translation for a key changes over time but the key name stays the same. For example, the key label.submit might have originally been “Submit Order” and later changed to “Place Order” in English. The code still uses label.submit, which could confuse developers if they assume the key matches the old wording. This outdated key issue is mostly a naming concern — using clear, semantic keys (like action.placeOrder instead of the generic label.submit) can help, as can documentation or even renaming keys when necessary (though renaming keys requires updating all translations).

Using English Text as Keys (Natural Keys)

Description: This approach uses the actual English string (or a close variant of it) as the key in code. In other words, the default language text itself serves as the identifier. For example, a developer writes t("You need to use a charging adapter on this route.") directly. With i18next, this typically relies on treating the key literally as the English source string and using it as a fallback if no translation is provided (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). The translation files might even have the English sentence as both the key and the value for the base language.

Pros (Natural Keys):

Immediate Readability & Less Guesswork: The code contains human-readable sentences, so any developer browsing the JSX/TSX can understand what message will appear in the UI (Step by step guide (v9) | react-i18next documentation). This makes the interface logic clear without opening a separate file. It’s also easier to search the codebase for a phrase you see in the app and find where it’s used (since the same text is in the code) (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs). Debugging is more straightforward — seeing t("File not found.") in code is obvious, whereas t("error.fileNotFound") requires an extra lookup.
Faster Development Workflow: Using English strings as keys means developers don’t have to create separate key names or initially populate translation files for the default language. You “write it once” – just put the English text in the t() call, and you instantly have a working UI in English (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). This minimizes the impact of i18n on development, keeping things as simple as writing normal strings (which is a design goal of frameworks like gettext (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow)). In practice, this can speed up prototyping and reduce the cognitive load of naming things. It also avoids the scenario of seeing empty or placeholder text in the UI before translations are added – the English text serves as a placeholder that is user-readable from the start (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow).
Better Context for Translators: If you use a system like gettext or i18next with natural keys, translators effectively see the full English sentence as the source text to translate. This provides more context than a cryptic key would (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). Translators can often produce more accurate translations when they see a complete English phrase (especially if the phrase itself is self-explanatory). While key-based workflows typically also show the English source (as a separate field), the natural-key approach guarantees that the key is exactly the English source. There’s no risk of a missing or outdated developer comment – the sentence itself carries meaning. This can simplify the translation workflow (fewer lookup steps to see context).
No English “Translation” Needed: The default language (English) is inherently covered because the app will display the key string if no other translation is loaded (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). You might not even need an English JSON file for i18next if you configure it to fall back to the key. This reduces duplication of having the same English text in both code and an English resource file. (However, for consistency, many teams still maintain an English translation file to use in the translation workflow, as discussed later.)

Cons (Natural Keys):

Maintenance on Content Changes: Changing the English text means changing the key, which requires code changes and re-translating into all other languages (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow) (Using english text as the translation csv file's keys. Good or terrible idea? : r/godot). This is the biggest drawback. For example, if the key/text "You need to use a charging adapter on this route." needs to be rephrased to "A charging adapter is required for this route.", you can’t simply edit a translation file – you must update every place that calls t("You need to use a charging adapter on this route.") to use the new string (and update every translation memory). Essentially, the old key is deprecated and a new key is introduced. All existing translations for the old sentence either become invalid or must be copied over if still applicable. This can be cumbersome and error-prone for large projects (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). Some workflows mitigate this by treating the old phrase as an identifier (never changing it in code) and only updating the displayed text via translation files (i.e. an "override" in the English translation) (Key based i18n vs default language i18n · Issue #50 · nodejs/i18n · GitHub). While that preserves code stability, it means the code’s string no longer matches the actual UI text, potentially causing confusion.
Context Collisions: Using full sentences as keys can backfire if the same English sentence appears in different contexts requiring different translations (Using english text as the translation csv file's keys. Good or terrible idea? : r/godot). For instance, the word “Archive” could be a noun in one place and a verb in another, and English might use the same word for both. If you used "Archive" as the key in two contexts, i18n will treat them as one entry – both instances will get the same translation in other languages, which may be incorrect. With abstract keys, you would have defined two distinct keys (like folder.archive.label vs action.archive) to distinguish them. Natural keys make it harder to manage these nuances unless you deliberately alter the English phrasing to differentiate keys, which is not ideal. This can lead to inconsistent or wrong translations if not carefully handled. It’s safer when your phrases are long/unique, but common short phrases can collide.
Loss of Systematic Organization: While you can still organize natural keys by namespace or comments, you lose some inherent structure that coded keys offer. The translation files might end up as a flat list of English sentences. It can be harder to see at a glance which part of the app a string belongs to just from the key. Additional metadata or naming conventions (like prefixing keys with a context manually) might be needed to maintain order. Without that, a large project using plain sentences as keys can become messy, with duplicates or very similar sentences scattered around.
Pluralization & Dynamic Content Challenges: If you rely on the English sentence as the key, handling plural forms and injected variables requires care. You might end up writing multiple full-sentence keys for singular and plural versions, which can be repetitive. For example, you’d have one key "You have one new message." and another key "You have {{count}} new messages." as separate entries, since i18next’s automatic pluralization rules expect a base key to modify (they work more naturally with abstract base keys like inbox.count_one/inbox.count_other). It’s possible to use ICU syntax or similar with natural keys, but at that point you’re almost treating the sentence like a mini-format rather than a literal key. In short, automatic plural handling is less straightforward with natural keys – you might end up managing plural variants manually, whereas with key-based approach you’d typically use i18next’s built-in plural keys or context for gender, etc. Similarly, for dynamic content, you must ensure the key is written as a template string (with placeholders) and hope it’s not misinterpreted. (Usually i18next will still replace {{placeholder}} in the fallback key text, so it does work, but you have to include those placeholders in the key exactly.) This is doable, but some teams find it cleaner to use a concise key and keep the full sentence (with placeholders) in the translations instead.
Potential Performance/Size Concerns: Each key in your translation resources might be a lengthy sentence. This can marginally increase the size of your translation files and memory usage, since the key is duplicated as the English default. In practice this is usually negligible and not a huge factor, but it’s less efficient than short keys (especially if a sentence is repeated identically in many places – you’d have the same long key string stored multiple times unless you refactor to reuse it). Given the question’s focus on maintainability over performance, this is likely a minor point, but worth noting for very resource-constrained contexts.

Comparison by Key Factors

Below is a factor-by-factor comparison summarizing how each approach fares:

Maintainability

Key-Based: High maintainability for evolving content. Since the key is decoupled from the actual phrase, you can adjust messaging without touching code. This makes updates and A/B testing of text much easier – only the localization files change. It also means you won’t force re-translation of other languages unless the meaning truly changed. However, maintainability depends on disciplined key naming. If keys are poorly named or if the English text changes meaning but you forget to notify translators (because the key didn’t change), you can end up with outdated translations. Overall, using stable keys acts as a layer of indirection that supports maintainability in the long run (Key based i18n vs default language i18n · Issue #50 · nodejs/i18n · GitHub).
English Text as Key: More brittle for maintenance. The English text is directly wired into code, so any text change is a code change that cascades to all translations (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). This tight coupling can hinder quick content tweaks – minor rephrasing becomes a development task and requires updating every other language’s entry (or doing a find-and-replace on keys). In a fast-paced project, this can accumulate overhead or discourage refining copy. Without careful management, you risk stale keys (if you avoid changing the key and thus display a different string than the key suggests) or inconsistent updates (if one instance of a sentence was changed but another identical one wasn’t, you might inadvertently fork what should be a single translation entry). Maintenance is simpler only as long as the English never needs to change. For a static content app this might be fine, but for most projects text does evolve. As one developer cautioned about using English strings as IDs: “it’s just a matter of time until you hit a corner case which will require you to redo 2/3 of the project” (Using english text as the translation csv file's keys. Good or terrible idea? : r/godot).

Verdict: Key-based wins on maintainability for apps where text is updated or refined frequently. The indirection adds initial work but saves effort in the long run when copy changes. Natural keys can be maintainable in scenarios where messages are essentially final or very unlikely to change, but that’s rare. Even then, you should be prepared for the eventual maintenance burden if a change is needed.

Scalability

Key-Based: Designed for scalability. Large applications with many pages and languages benefit from the structured approach of keys. You can categorize and partition translation files by feature or module (e.g. using i18next namespaces per page or section). This modularization means as the app grows, translations remain organized rather than one gigantic flat file. Keys also promote reusability: if the same concept appears in multiple places, you can use the same key everywhere and translate it once, ensuring consistency and saving translator effort (8 Advantages of using translation keys in your localization files - POEditor Blog). Version control of translations is clearer with keys too – diffs show which keys changed, and you can track changes in meaning. Overall, key-based catalogs handle growth in both content and number of languages gracefully (8 Advantages of using translation keys in your localization files - POEditor Blog) (8 Advantages of using translation keys in your localization files - POEditor Blog).
English Text as Key: Simpler in small scale, but can get unwieldy as things grow. In a small app, it’s straightforward – each string is just itself. But as the number of strings increases, managing them without a key hierarchy might become difficult. You might end up with duplicates (the same sentence written slightly differently in two places) which could have been a single reusable key in a key-based approach. Also, if multiple developers are adding strings, slight variations in phrasing could slip in, making translations inconsistent or redundant. There is also a potential performance consideration at scale: if each key is a long sentence, your translation lookups and memory usage might be marginally heavier (though in practice i18n libraries handle thousands of keys fine). Another scalability issue is maintaining consistency – with natural keys, ensuring that common phrases are translated uniformly across the app relies on humans noticing the repetition, whereas keys could enforce reuse. In translation workflow terms, if you have a lot of strings, having unique identifiers (keys) might help translators and tools manage the content (for example, identifying when one English phrase is just a duplicate entry vs a new context).

Verdict: Key-based is more scalable and easier to keep organized as projects grow in size and languages. It’s the preferred approach in large-scale applications and by most localization platforms (8 Advantages of using translation keys in your localization files - POEditor Blog). Natural keys can work for smaller projects or those with very limited dynamic text, but risk becoming chaotic at scale without strict conventions.

Ease of Development

Key-Based: Initially, a bit more work for the developer. You have to decide on a key name, add it to the translations, and then call t('your.key.name'). This two-step process (write key in code, write text in file) can slow down iteration (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). However, modern workflows mitigate this: for example, i18next lets you pass a default value inline (t('error.networkTimeout', 'The request timed out.')) which can auto-fill missing translations during development (Essentials | i18next documentation). There are also tools that can extract keys and strings from code to generate translation files, or even live-reload missing keys. Once these tools or patterns are in place, adding new strings becomes more seamless. Another aspect is cognitive: thinking of a short key name that conveys the purpose is an extra mental step for the developer. Some find this trivial, others find it disruptive when writing a lot of UI text. On the positive side, key-based approach forces developers to think about reuse and context, which can be good for consistency. And with TypeScript, if you have types for keys, the IDE auto-completion can actually aid development by suggesting available keys.
English as Key: Extremely straightforward for developers, especially at the start. You simply write the message as you want the user to see it (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). This means you get immediate UI text without worrying about missing translation entries or naming. It’s very WYSIWYG: what you put in t("...") is what shows up (at least for English). This often results in faster prototyping. As one developer put it, using English strings directly gave “meaningful output from the start and [you] don't have to think about naming placeholders”, resulting in “less work for the developers.” (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). You also avoid the scenario of forgetting to add a translation and seeing a raw key or no text – here, the fallback is the English string which is perfectly readable. This approach aligns well with agile development where you might not have all copy finalized but need something on screen now. The drawback is that down the line, if those strings need to change or be referenced elsewhere, the lack of an abstract key can slow you (as discussed in maintainability). But from a pure coding perspective, many find this approach easier and more intuitive.

Verdict: For development speed and simplicity, using English strings as keys has an edge, especially early in a project or for less experienced i18n developers (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). Key-based can be nearly as smooth with good tooling (like default values, extraction scripts, etc.), but it has a learning curve and initial overhead. If developer productivity and minimizing steps is the priority (and you’re willing to accept some technical debt in translations), the natural key approach is attractive. Just weigh this against future maintenance costs.

Translation Workflow

Key-Based: This is the traditional workflow expected by most localization teams and tools. Developers maintain an English resource (e.g. an en.json file) where each key has an English string. Translators use a translation management system (TMS) or files where they see something like: errors.networkTimeout ⇒ "The request timed out." as the source and then provide, say, French "La requête a expiré." for that key. Context for translators can be provided via the key name (if it’s descriptive) and additional comments or screenshots. One advantage is that keys can be accompanied by developer comments explaining usage (most TMS support a comments field per key). Translators typically see the English (or source language) text alongside the key, so they know what to translate (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). The key itself might not mean much to them, but it doesn’t have to — it’s just an identifier. As long as the English (source) string and any notes are clear, translation is straightforward. Another advantage: if the English text changes (but the key stays same for semantic reasons), the system can flag that the translations for that key might need updating (since the "source" changed). Many TMS support versioning or a “re-translate” flag when source text updates. This approach scales to many languages easily and avoids duplication in translation work because each key is translated once and reused. The downside is that without a good tool, a translator working from raw files might see a key like BTN_CONFIRM and not know what it means without context. But in a robust workflow, context is managed.
English as Key: This can simplify or complicate the translator’s job depending on the tooling. In a basic scenario, you might not have a separate English source file at all – the English text is the key. If using something like gettext, the .pot file (catalog of source strings) is basically a list of English phrases. Translators translate those into target languages. This is a very direct workflow: the English text is the source text. Translators definitely have context because they see the full string they need to translate (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). In our example, the translator would directly see “You need to use a charging adapter on this route.” and provide a translation. This is efficient and avoids the need for an English key description. However, there are a few pitfalls:
- If the same English string is used in two different places for different purposes, a translator might only see one entry and provide a single translation, which could be wrong for one of the contexts (because the system doesn’t know they were meant to be distinct). Gettext mitigates this with context flags (msgctxt) if needed, but that requires developers to mark contexts. Without such measures, the translator might not even know that one English phrase actually appears twice in different contexts with potentially different meanings.
- If you change an English sentence slightly, you’ll generate a new key and the translator will see it as a new string to translate, with the old one potentially removed. Unless the translation memory matches it, they might have to translate a very similar sentence again. In contrast, a stable key would have shown the English change and possibly allowed the translator to update the existing translation with minimal effort.
- Some translation platforms prefer having stable keys as identifiers. If you integrate with a service that expects keys, using the full sentence as the key is still possible but sometimes the tooling around comments, screenshots, etc., might assume keys are not huge text. Generally, though, gettext-based workflows and many modern TMS do support using the source string as key (often called “source string-based localization”).

In summary, translators will find natural keys very straightforward for understanding context, since the source is explicit (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). But project managers and engineers managing translations might find it harder to track changes and maintain consistency without the abstraction layer. Also, coordinating large changes (like renaming a term across the app) is easier with keys (one key, many languages to update) versus searching through many strings.

Verdict: Both approaches can fit into standard translation workflows, but key-based is generally more robust for large teams and external translation services. English-as-key can be perfectly fine when using tools designed for gettext or similar paradigms – it’s essentially what many open-source projects do. The key is to ensure translators have context for each string. If using English keys, you might rely less on separate comments since the string is there, but you should still provide notes if something isn’t obvious or if a placeholder like {count} appears. If using coded keys, providing the English source string and context in the TMS is essential so translators aren’t guessing (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow).

Readability for Developers

Key-Based: Requires developers (and anyone reading the code) to mentally map keys to actual content. If keys are descriptive (e.g. user.profile.saveSuccess), they at least hint at meaning, but they might still be somewhat abstract. New developers might have a harder time understanding the UI text without running the app or checking the translation files. This can slow down code reviews or debugging. That said, consistent naming conventions help a lot – if your team agrees on how keys are structured, a developer can often infer the message. For example, seeing error.payment.declined in code clearly is an error message about payment declined, which is almost as good as seeing the full sentence. Some IDE integrations or simple editor tricks (like searching the JSON) can also bridge the gap. In terms of code aesthetics, keys keep the code compact and language-agnostic, which some prefer (the code isn’t cluttered with long text strings). But the main drawback is the cognitive load: it's an indirection that can make the code less immediately readable (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs), especially if keys are not well-chosen.
English as Key: Very high readability in code – essentially self-documenting. When you read t("Delete user profile"), you know exactly what the UI is conveying. This can make development and code reviews quicker in terms of understanding intent. It also reduces context switching: you don’t need to open another file or rely on memory for what a key means (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs). Additionally, searching the repository for a phrase you see in the UI will directly lead you to where it's implemented, which is great for debugging (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs). On the flip side, the presence of large chunks of text in the code can be a bit distracting, and if you have multi-sentence strings, it might affect code formatting or line length. But generally, developers value clarity, and there’s not much ambiguity when the actual sentence is in the code. The only time this can become confusing is if, as mentioned, the code’s text is no longer the real text (if you’ve overridden English in the translation file or something). In that case, the code might lie about what’s shown, which is arguably worse than an abstract key – so one must avoid that scenario or comment it clearly.

Verdict: Using English (natural language) as keys provides better readability and transparency in the codebase (Step by step guide (v9) | react-i18next documentation) (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs). It lowers the barrier for any contributor to understand the interface text. Key-based references make the code a bit more opaque, though good naming conventions can alleviate this. If developer readability is a top concern (e.g., in open source projects or teams where developers themselves change text often), natural keys shine. If keys are used, invest in making them as semantic as possible (some teams even include part of the phrase in the key for readability (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs) (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs), as a compromise).

Consistency Across Languages

Key-Based: This approach inherently treats each key as a single concept that should be consistently translated in each language. It promotes consistency because all languages are keyed off the same identifier. If you use one key in multiple places, all languages will use their translation of that key, keeping the messaging uniform. It also allows enforcing consistency: if two English strings should be translated the same in French, you can deliberately use one key for both so that translators only provide one translation. Additionally, keys with contextual info ensure that translators know the intent (so they don’t mistakenly translate two identical English words the same when in context they should differ). When maintaining multiple languages, key-based setups make it easier to spot when one language is out-of-date (e.g., the English text changed but French still has the old translation – the key is the same, but you can mark the French entry as needing update). Another consistency benefit is the ability to do things like “pseudo-localization” or automated checks – keys make it clear which strings correspond across languages. Overall, a structured key system acts as a backbone aligning all languages to the same set of messages.
English as Key: In terms of consistency, using the English string as the pivot can work, but it has some risks. If the English phrasing is the single source of truth, other languages will attempt to mirror that meaning. As long as that holds, translations will be consistent with English. However, if English phrasing is changed (creating a new key), there’s a chance not all languages update simultaneously, leading to a period where English has one message and others still reflect the old message (since the link was broken by changing the key). Also, if two languages need slightly different nuance, you don’t have a straightforward way to handle that via keys – though that’s more a translation issue than a key issue (usually handled via separate keys or context). Another subtle consistency issue: two different English strings that mean the same thing (synonyms) will produce two separate keys that could be translated differently in some language. For example, if one part of the app says "Close" and another says "Exit" (English synonyms) and you used both as keys, a translator might not realize they should use the same word in their language for both. Key-based approach could have enforced a single term by using one key for both or by at least making the relationship obvious. Essentially, natural keys might make it harder to detect and enforce consistency when the English vocabulary varies. From a process standpoint, ensuring consistency across languages with natural keys means diligently updating every language when English changes (since the key changes), and keeping an eye on identical or similar English phrases that should perhaps be unified. Translation memory tools can help by suggesting translations for identical phrases, but they won’t inherently link them like a key would.

Verdict: Key-based approach provides a clearer path to consistency across locales, since each key anchors a concept in all languages. It’s easier to maintain parity and detect divergences. Using English strings as keys can still yield consistency if managed well, but it relies more on discipline and translation memory to keep things aligned. If consistency and tight control over wording across languages is paramount (as it often is for branding or legal text), key-based is safer.

Handling Pluralization and Dynamic Content

Key-Based: Internationalization frameworks like i18next offer robust pluralization support when using keys. Typically, you define a base key and multiple forms, for example:
```
{
  "mail": {
    "unread_one": "You have 1 unread message.",
    "unread_other": "You have {{count}} unread messages."
  }
}
```
In code you call t('mail.unread', { count: messagesCount }) and i18next will pick the _one or _other form based on the count (Plurals - i18next documentation). This system relies on keys and suffixes to differentiate plural forms. It cleanly separates singular vs plural logic and lets translators handle language-specific pluralization (even for languages with multiple plural forms). Similarly, for gendered or contextual variants, you might use keys like welcome_user_male vs welcome_user_female or use i18next’s context feature, again leveraging the key system. For dynamic content, keys work in tandem with interpolation: the key identifies the sentence template, and the translation string includes placeholders (like {{username}}) that get replaced at runtime. i18next encourages this approach as it keeps grammatical order correct per language and avoids string concatenation in code (Best Practices | i18next documentation) (Best Practices | i18next documentation). Overall, key-based translations handle plurals and variables in a structured way defined in the localization files (which is important because different languages have different grammar around those).
English as Key: It’s possible to manage plurals and dynamic content, but you often end up embedding the logic in the English strings themselves. For plurals, one way is to just treat the singular and plural as two separate keys (as mentioned earlier), e.g. t("You have 1 unread message.") and t("You have {{count}} unread messages.") depending on the count. This works, but it means the onus is on the developer to pick the right key for singular vs plural. You lose the automatic pluralization rules that i18n can provide, which can be error-prone when expanding to languages with more complex plural rules than English. Alternatively, you could still use i18next’s pluralization by setting a custom key or enabling ICU syntax. For instance, i18next supports ICU message format: you could have one key with an ICU string like "You have {count, plural, one{# unread message} other{# unread messages}}" as the English translation. In code you’d call that key with a count. But note, here we introduced a synthetic key (or we treat the entire ICU string as the key which is awkward). Generally, using natural keys for singular/plural will push you toward writing more code logic to handle plurals, or leveraging ICU in translations (which is fine, but that’s essentially moving away from pure “English as the key” to “English as the default translation with a symbolic key”). For dynamic content, if you include placeholders in the key string, i18next will interpolate them even in the fallback. For example, t("Hello, {{name}}!", { name: userName }) would likely show "Hello, John!" by replacing {{name}} even if it's using the key as the output (since i18next treats the key as a default value here). This is convenient – you still get variable substitution. The caution is that the presence of placeholders might necessitate some context for translators (they need to know what {{name}} represents), but that’s true in both approaches. In short, pluralization is the one area where key-based approach has a clear structural advantage, while dynamic content (interpolation) is handled similarly by both, with just some careful attention needed when using raw strings as keys.

Verdict: Key-based approach is more aligned with how i18n frameworks handle plural and dynamic content out-of-the-box. It enables you to use the library’s full capabilities (plural rules per locale, etc.) without hacking the keys. The natural key approach can still work, but you may end up writing more conditional code for plurals or using more complex translation strings (ICU) to achieve the same result. If your application has a lot of pluralization or gendered phrases, you might lean towards key-based for clarity and correctness. For simple cases (like just two forms, English-like), natural keys won’t pose much issue as long as you’re careful.

Recommendations for When to Use Each Approach

Both approaches can be made to work with i18next in a TypeScript + React stack, but the best choice depends on your project’s priorities and team workflow. Here are some recommendations:

Use Key-Based Localization when:

Your project is large or growing – If you anticipate a lot of UI text or many languages, a key-based approach will scale better and remain manageable (8 Advantages of using translation keys in your localization files - POEditor Blog). The structure helps avoid duplication and inconsistencies as the app expands.
Text will be iterated on – If copy is likely to change due to UX research, marketing input, or frequent tweaks, it’s safer to use abstract keys so that changing the English (or any base language) doesn’t require code changes (Key based i18n vs default language i18n · Issue #50 · nodejs/i18n · GitHub). This decoupling means product or content teams can suggest wording changes that developers implement just by updating translation files.
Multiple contexts for similar text – When the same or similar English words appear in different contexts (and might need different translations), key-based is preferable. You can encode context in keys (e.g. menu.open vs action.open) to guide translators (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow), whereas using the raw text “Open” for both would create ambiguity. If your app has a lot of short phrases or reused terms, key-based gives you finer control.
You want consistency and reusability – If the goal is to have a single source of truth for each piece of text and reuse translations across the app, use keys. For example, the text for “Cancel” button can be one key reused everywhere, ensuring every “Cancel” is translated identically. This also reduces translation workload (each language translates “Cancel” once) (8 Advantages of using translation keys in your localization files - POEditor Blog). With natural strings, a dev might accidentally use “Cancel” in one place and “Abort” in another, leading to duplicate entries for translators.
Your team uses a TMS or formal process – Most localization platforms (Locize, Transifex, Lokalise, etc.) and professional translators are very comfortable with key-based workflows. It’s often easier to integrate keys with things like screenshot management, context descriptions, and versioning of source text. If you’ll have dedicated translators or a localization team, they might prefer keys + English source strings as a clear separation of concerns (identifier vs content). Also, if your organization has a glossary or requires consistent terminology, managing that by key is more straightforward.
Type safety and refactoring are important – In a TypeScript project, you might want to leverage type checking for translations. With key-based approach, you can use packages or scripts to generate a union of all keys, allowing t() to be typed. This means if a developer uses a non-existent key, TypeScript will error – preventing runtime translation misses. It also makes refactoring keys (renaming) easier with find-and-replace, since keys are usually simpler strings without spaces/punctuation. Using entire sentences as keys is harder to type-check (the union of all sentences is huge and not practical to maintain manually).
Examples of when to choose key-based: A complex enterprise app with forms, messages, tooltips across dozens of screens; a product where English copy is still being refined over time; an app where future translation to 5+ languages is planned; any scenario where you have translators working in parallel and you need strict control over context.

Use English (Natural) Keys when:

Rapid prototyping or early development – If you’re at an early stage and need to get the UI up quickly in the default language, using the text directly can save time (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). You can worry about abstracting keys later (or not at all if it suffices). This is common in hackathons, MVPs, or internal tools where initial speed is valued over long-term elegance. i18next allows this by setting fallbackLng to English and using the key as default text (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow).
Small or one-off projects – For a simple app or a static website with limited text, the overhead of managing separate keys might not be worth it. If there are only, say, 20 strings and they’re not likely to change much, using natural keys is fine and makes the code very clear. You could even decide to not have a separate JSON file for English at all – just use the strings in code and maintain translation files for other languages. The simplicity can trump the theoretical downsides since scope is limited.
Projects with gettext or similar i18n heritage – If your team or tools come from a gettext background, using the string as the message ID is a normal practice. Libraries like react-i18next can support this mode (by disabling key separators and using the key as fallback text) to mimic gettext style. In such cases, you might have workflows to extract strings from code into a .pot file and merge translations. Sticking to English as keys could integrate better with those existing processes.
When developer clarity outweighs future proofing – In some teams, especially those without dedicated localization folks, having the actual text in code can reduce misunderstandings. For example, a developer sees t("Server error") and knows what it is, whereas t("ERR_42_MSG") might lead them down a documentation hunt. If the team is small and you’re confident you can manage the occasional text change manually, the convenience for devs might win. Just be aware of the debt you incur (as discussed). This is often a judgment call. Some maintain that the time saved during development and debugging using natural keys more than offsets the time spent later if a change is needed.
Content is truly fixed or auto-generated – If the text is unlikely to ever need editing (perhaps regulatory text, or content that comes from elsewhere and is stable), using it directly is reasonable. Also, if keys would end up as nearly the full sentence anyway (because you’d make them descriptive to the point of being the sentence), then you might ask: why not just use the sentence? For example, if your key naming policy would produce something like instructions.pleasePlaceDeviceOnFlatSurface for the text “Please place the device on a flat surface.”, some might prefer to avoid the indirection and just use the sentence.
Examples of when to choose natural keys: A prototype or proof-of-concept where i18n is needed but likely only one language initially; a simple marketing site with a few translatable phrases; a backend-rendered template system where gettext is already used; a plugin or module intended to be easy for others to read and maybe contribute to without needing to understand a separate translation file.

Hybrid Approaches: It’s worth noting you don’t strictly have to choose one style for everything, but you should avoid mixing styles arbitrarily within the same project (Step by step guide (v9) | react-i18next documentation). Some teams adopt a hybrid: for UI labels and short text they use natural keys (for readability), but for longer or reused text, they use semantic keys. If doing this, you must be disciplined to prevent confusion. Another variant is the “engineering English” approach (Key based i18n vs default language i18n · Issue #50 · nodejs/i18n · GitHub) – you use English phrases as keys initially, but once set, the key is treated as immutable. If the English needs to change, you change the English translation for that key instead of the key itself. Essentially the key becomes an identifier that looks like English. For example, you might have t("You need to use a charging adapter on this route.") as a key. Later, if you want to reword it, you keep the key string the same but in the English resource file you map that key to a new sentence (so the UI shows the new sentence). This preserves code and translation links at the cost of the key being somewhat misleading. This method can be useful if you started with natural keys and later realized you need stability – but it requires clear communication that the “key” is not literally the displayed text anymore. Generally, it’s cleaner to either use keys from the start or accept that changing natural keys will be a bit of work.

For i18next in a React+TypeScript project, either approach can be configured:

If you go key-based, define your namespaces and use t('namespace:key') or structured keys with dots. Maintain an en.json file with all keys and English strings (this acts as the source for translators). You might set up TypeScript types for the resources (using react-i18next type augmentation or a codegen tool) to ensure keys exist at compile time.
If you go natural keys, initialize i18next with keySeparator: false (and nsSeparator: false if you plan to include : in keys or use one global namespace) so that your sentence strings are not split on . or : (Step by step guide (v9) | react-i18next documentation). You can still use an English JSON file if you want, but it may be redundant – instead, you might rely on t(key, { defaultValue: key }) or simply t(key) with fallback to show the key when untranslated (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). Ensure all developers know not to change key strings lightly. Also, consider using the saveMissing option in i18next during development – it can collect any keys (in this case full sentences) that aren’t in the translation files and save them, which helps populate your base language file automatically (Essentials | i18next documentation).
In either case, make use of i18next’s features like t function options for plurals (count) and interpolation ({{var}}) rather than concatenating strings. For React specifically, the <Trans> component can be useful for complex markup within translations, and it works with both styles of keys (you either pass a key or the default text inside the component). Just remain consistent with whichever key style you choose.

Best Practices for Maintainable and Developer-Friendly Translations

No matter which approach you choose, the following best practices will help keep your internationalization workflow smooth and your codebase clean:

Establish a Naming Convention (if using keys): Decide on a clear scheme for your keys and stick to it project-wide (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs) (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs). For example, you might use a <scope>.<subscope>.<description> format (as in "checkout.payment.errorCardDeclined"). Consistent patterns make keys easier to decode and avoid collisions. Include context in keys where needed (e.g., "label.save" vs "verb.save" if a word could be a noun or verb) (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow). Document this convention so all contributors use it. Also, avoid overly generic keys (like "message1"); keys should be descriptive enough to identify the content.
Keep Translation Files Organized: Use i18next namespaces or separate JSON files per module/page. This prevents one huge file and reduces merge conflicts when multiple people add strings. For example, have auth.json for authentication screens, common.json for reusable common phrases, etc. In React, you can lazy-load namespaces as needed (i18next supports dynamic loading), but given maintainability is prioritized over performance here, it’s also fine to load all at once for simplicity. Organizing by feature also helps translators know where text is used, especially if your keys or source strings are not self-explanatory.
Use Default Values and Missing Key Logging: During development, take advantage of i18next’s default value feature to avoid double-writing strings in code and files. For instance, t('errors.networkTimeout', { defaultValue: 'The request timed out.' }) will display the default text if that key isn’t yet in your translation file (Essentials | i18next documentation). Coupled with saveMissing: true in i18next config, it can even send that default text to your translation storage (or console log it) so you know to add it (Essentials | i18next documentation). This way, developers can add new translations in one step. Just remember to remove default values in production or ensure your localization process captures them.
Avoid Hard-Coding Text: Apart from perhaps some very static content, all user-facing strings should go through the i18n system. This ensures nothing is missed when translating. Run ESLint or other linters to catch accidental hard-coded literals in the code. For example, there are ESLint plugins for i18next or for frameworks (like vue-i18n) that warn if you have raw text in JSX that isn’t wrapped in a <Trans> or t() call (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs). This enforces consistency and makes future maintenance easier (no “forgotten” string that only exists in code).
Provide Context to Translators: If using key-based, always supply the translators with an English reference for each key (usually your English JSON) and any necessary comments (many TMS allow commenting on keys). Even if using natural keys, add context if the usage isn’t obvious. For example, if a string is “Draft”, clarify whether it’s a noun or verb, or maybe provide a screenshot of the screen. Context can also include notes about placeholders (e.g., “{{count}} is a number of items”) or character limits if any. This extra effort prevents mis-translations and reduces back-and-forth. Remember, even with natural keys, a phrase taken out of UI context can be interpreted in multiple ways, so a brief note can help.
Handle Plurals and Gender Properly: Use i18next’s pluralization features rather than writing logic in code to pick singular/plural strings. This means defining plural forms in your translation files (or using ICU messages) so that languages with complex plural rules are supported. Test your pluralization with a language like Russian or Arabic (which have 3-4 plural forms) to ensure your approach covers those cases (Plurals - i18next documentation). Similarly, if your app needs gendered terms or other contextual variants, consider using the i18next context feature (keys with suffixes like _male _female) or separate keys for each. The goal is to let the translation system, not the application logic, handle these variations as much as possible (Best Practices | i18next documentation) (Best Practices | i18next documentation).
Use Descriptive Variable Names in Strings: When you have dynamic content, prefer named interpolation like {{username}} over positional or generic placeholders. This makes the meaning clear to translators. For instance, "Hello {{username}}, you have {{count}} new messages" is better than "Hello %s, you have %d new messages". It reduces confusion and allows changing word order if needed (translators can move the {{count}} placeholder as appropriate for their grammar). i18next by default uses the {{var}} syntax which is good (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs). Just ensure the variable names are meaningful ({{count}}, {{name}}, {{totalPages}}, etc.). This is a general i18n best practice that improves localization quality.
Consistent Formatting and Punctuation: Decide whether keys should include punctuation or not, and be consistent. For example, some teams omit the period at the end in the key and only include it in the translation, to avoid keys differing only by punctuation. Others include full punctuation in keys for clarity. If using natural keys, you might run into issues with certain characters (like colons, dots) because i18next might interpret them as separators. You can escape them or adjust the config (e.g., set nsSeparator: false if you have : in keys like UNT:BannerBody…) (Step by step guide (v9) | react-i18next documentation). The key point is to standardize how you handle it so you don’t accidentally have two keys where one has a punctuation or newline difference. For maintainability, treat the text consistently.
Automate Where Possible: Utilize tools to lighten the translation maintenance burden. For example, use the i18next parser or similar to extract strings/keys from code into translation files, so nobody forgets to add them. Set up CI checks to warn if a translation key is missing in any language (or at least in the default language). If using TypeScript with key-based, set up the type generation so that as soon as you add a key to en.json, it’s available in the t() function types (there are community tools for this, or you can write a simple script reading JSON and generating a d.ts file). Automation ensures your code and translation files don’t drift out of sync and reduces manual errors.
Review and Refactor: Treat your i18n content as part of the codebase that needs occasional refactoring. For key-based projects, that might mean cleaning up unused keys (over time, some keys might no longer be used in code – remove them to avoid confusion). For natural key projects, it could mean identifying and merging duplicate strings or clarifying ambiguous ones (maybe by switching a couple to use a context key instead). Also, periodically review if the chosen approach is still serving the project well. It’s possible to migrate from natural keys to structured keys later (via script or gradually) if you outgrow the initial method. It’s harder to go the other way around though. In any case, keeping the translation files tidy and logically structured improves maintainability significantly.
Test in Multiple Languages: Don’t wait until the last minute to see your app in other languages. Even if you only have English initially, try adding a second locale (even a dummy or pseudo-translation) early on. This will help catch issues like keys not being found, text concatenation problems, or layout issues with longer text. It also forces developers to think about translation impact. If you’re using English as keys, switch the locale to something else (say French) and see if everything still makes sense (likely you’ll see English because it’s falling back to keys, which is fine). If using keys, maybe do a fake locale where every string is prefixed to ensure all keys are going through translation. The point is to integrate i18n testing into your routine so that when real translations come in, the system is robust.
Prioritize Clarity over Premature Optimization: Since the focus is maintainability and dev ease over performance, it’s okay to choose the approach that makes the code cleaner and the team happier, even if it’s not the most CPU or memory efficient. For example, having longer, descriptive keys or default texts might use a bit more memory, but it reduces confusion. Loading all translations at once might use more bandwidth, but simplifies development – you don’t have to worry about splitting or lazy loading messages. You can always optimize later if needed (like extracting only used keys per page), but a correct and clear implementation is the first goal. i18next is quite fast for reasonable numbers of keys, so optimize for developer productivity first.

In conclusion, using localization keys vs. direct text keys each has trade-offs. Key-based catalogs offer stronger long-term maintainability, scalability, and flexibility for changes (Using english text as the translation csv file's keys. Good or terrible idea? : r/godot) (Using english text as the translation csv file's keys. Good or terrible idea? : r/godot), making them ideal for larger projects and collaborative translation workflows. Direct text keys provide simplicity, immediacy, and code readability that can accelerate development and reduce friction (internationalization - Why do people use plain english as translation placeholders? - Stack Overflow) (i18n: Do you prefer using one big translation dictionary for your whole app, or sprinkle translation strings inside your components? : r/vuejs), which is attractive for small projects or early stages. Evaluate your project’s needs: if you foresee lots of evolution and numerous locales, lean towards key-based with good practices; if you need to move fast and the content domain is fairly static, natural keys can serve you well initially.

For an i18next + TypeScript + React setup, either route can be implemented – just keep consistency and use i18next’s features to your advantage. Whichever approach you choose, apply the best practices above to maintain a clean, efficient, and developer-friendly localization codebase. Internationalization is as much about process as technology, so choose the approach that best aligns with your team’s workflow and the app’s future requirements, and be ready to adapt as the project grows.

Making API Calls from Node.js

spyke — Thu, 18 May 2023 07:46:57 +0000

Calling another API from Node.js is straightforward.

Starting with Node.js version 18, a web-compatible fetch() function is available out of the box. However, as it is still marked as experimental, there are battle-tested libraries like Axios suitable for production environments, offering convenient APIs and a comprehensive feature set:

import axios from 'axios';

const http = axios.create({
  baseURL: 'https://example.com/api'
});

export function getUser(userId: string) {
  // An equivalent to `GET /users?id=12345`
  return http.get('/user', {
    params: {
      id: 12345
    }
  });
}

That's it. Happy coding!

Expanding the Scope

With the basics covered, let's discuss how to scale this solution. Not in terms of algorithmic or space complexity - which is often the focus of job interviews - but in terms of scaling it for a large application that will be worked on by a team of developers over several years. This is the kind of scaling we deal with in day-to-day work.

To illustrate this, we'll use a real-world API from a digital bank with public documentation: https://docs.solarisgroup.com/api-reference

Disclaimer: I am not affiliated with Solaris SE. I just happened to be familiar with this API. Using realistic examples can be beneficial.

Fetching Person Data

Let's begin with a basic getPerson call:

import axios from 'axios';

const http = axios.create({
  baseURL: process.env.API_BASE_URL
});

export function getPerson_v0(personId: string) {
  return http.get(`v1/persons/{id}`);
}

As you can see, it's not significantly different from the code in the introductory section. However, considering the returned value (see the response object in the docs), you might spot some issues.

Firstly, axios returns its own response object. Passing this up the stack isn't ideal as our callers (users of the API client we make) don't need to know what HTTP library we're using. We want to retain the flexibility to replace it later. Moreover, the users are primarily interested in the Person object itself, which is contained in the data property:

export function getPerson_v1(personId: string) {
  return http
    .get(`v1/persons/{id}`)
    .then((response) => response.data);
}

Now, let's consider the Person value itself:

Keys are in snake_case, which can lead to warnings from ESLint and complaints from other developers because our standard is camelCase.
Dates are represented as strings.

export async function getPerson_v2(personId: string) {
  return http
    .get<Person>(`v1/persons/{id}`)
    .then((response) => response.data)
    .then(snakeToCamelCase)
    .then(restoreDates);
}

In the above code, snakeToCamelCase and restoreDates are utility functions that iterate through the object's keys and map either the key or the value, depending on what needs to be corrected. The details of their implementation are not the focus of our discussion.

I've also added <Person> to the type parameters. Naturally, we need to create interfaces and enumerations for all data that we send or retrieve. These could be generated from an OpenAPI spec or created manually, depending on what the docs provide.

An Unexpected Value

We restore date values, but we don't need to restore enumerations like employment_status as they are just strings. But what if a new unexpected value appears in the response that we don't have in our EmploymentStatus? Or even worse, some other field becomes a string instead of a number. We'll probably corrupt the application state at runtime.

An API is a living creature; it changes over time. It also has bugs (everything has bugs). We need to defend our code from invalid response data.

Schema Validation

Schema validation can ensure that the data your application is receiving is of the correct type and format. This can help you catch bugs and discrepancies in the API, as well as protect your application from potential crashes or corrupt states due to unexpected data.

For Node.js, libraries such as Joi, Yup, Zod are often used. I will be using Runtypes.

import * as t from 'runtypes';

export const PersonSchema = t.Record({
  // Other fields are omitted for simplicity
  id: t.String,
  first_name: t.String,
  last_name: t.String,
  email: t.String,
  birth_date: t.InstanceOf(Date),
  employment_status: EmploymentStatusSchema,
});

export type Person = t.Static<typeof PersonSchema>;

export async function getPerson_v3(personId: string) {
  return http
    .get(`v1/persons/{id}`)
    .then((response) => response.data)
    .then(snakeToCamelCase)
    .then(restoreDates)
    .then((person) => PersonSchema.check(person));
}

Calling .check(person) either returns back a typed Person or fails with an error.

Authentication

We need to address authentication as well. The API is protected by a bearer token, which you can get via the oAuth endpoint. The token has an expiration time, and we will need to refresh it from time to time. I'm going to encapsulate all business logic about tokens in the TokenManager component. Read as "I will mostly skip it":

export class TokenManager {
  getToken(): Promise<string> {
    // ...
    return token;
  }

  close(): void {
    // ...
  }
}

It is really a separate client, which knows how to call the token endpoint and is using setTimeout to refresh the token prematurely. getToken returns a promise with the token value.

Axios provides interceptors as a mechanism, which we will utilize for injecting the token:

const tokenManager = new TokenManager();

async function injectNewToken<
  C extends InternalAxiosRequestConfig | AxiosRequestConfig
>(config: C): Promise<C> {
  const token = await tokenManager.getToken();
  if (typeof config.headers?.setAuthorization == "function") {
    config.headers.setAuthorization(`Bearer ${token}`);
    return config;
  }
  return {
    ...config,
    headers: {
      ...config.headers,
      Authorization: `Bearer ${token}`,
    },
  };
}

httpClient.interceptors.request.use(async (config) => {
  return injectNewToken(config);
});

httpClient.interceptors.response.use(undefined, async (error) => {
  if (isAxiosError(error) && error.response?.status === 401 && error.config) {
    return injectNewToken(error.config).then((c) => httpClient.request(c));
  }
  throw error;
});

The code is simplified, but what we're doing is:

Before actually dispatching a request, we ask TokenManger to give us a valid token, then we add it as the Authorization header to the request config.
In case of an error response with a 401 status code, we know from the documentation that the token was expired by the time it arrived at the API, and we need to get a new one and repeat the request.

Errors

The API can respond with more than just 401 errors. Even more importantly, there's a whole page describing the error response format. At a bare minimum, we need to parse the response and throw something like new Error(responseError.code). But we will need the error ID when we write to their support (and we will, a lot). Error description is helpful as well for debugging the issues.

The users of our client library should not dig into Axios errors and responses to understand that the person is not found. They should be able to easily pattern match on different error situations.

Introducing custom errors and error mapping. This is where I will be simplifying even more. Otherwise, it will be a thousand lines of code. For example, I will assume that there could be only 1 error in the response. And we will map only error codes, ignoring status codes and possible combinations.

Errors begin with making types and schemas for server error responses:

export enum SolarErrorCode {
  METHOD_NOT_ALLOWED = "method_not_allowed",
  MODEL_NOT_FOUND = "model_not_found",
  UNAUTHORIZED_ACTION = "unauthorized_action",
}

export const SolarErrorDataSchema = t.Record({
  // Other fields are omitted for simplicity
  id: t.String,
  status: t.Number,
  code: t.String,
  detail: t.String,
});

export const SolarErrorResponseSchema = t.Record({
  // Other fields are omitted for simplicity
  errors: t.Array(SolarErrorDataSchema),
});

After that we can define our own error class, which will be used by the code calling our client:

export enum ClientErrorCode {
  CONFIG_ISSUE = "config_issue",
  MODEL_NOT_FOUND = "model_not_found",
  NO_CONNECTION = "no_connection",
  UNAUTHORIZED = "unauthorized",
  UNEXPECTED_ERROR = "unexpected_error",
}

export class ClientError extends Error {
  static is(value: unknown): value is ClientError {
    return value instanceof ClientError;
  }

  static mapCode<R>(codeMap: { [key: string]: (error: ClientError) => R }) {
    return function clientErrorMapper(error: unknown) {
      if (ClientError.is(error)) {
        const remapper = codeMap[error.code] ?? codeMap["_"];
        if (typeof remapper === 'function') {
          return remapper(error);
        }
      }
      throw error;
    };
  }

  id?: string;
  code: string;

  constructor(
    errorData: SolarErrorData | Pick<SolarErrorData, "detail" | "code">
  ) {
    super(errorData.detail);
    if ("id" in errorData) {
      this.id = errorData.id;
    }
    this.code = errorData.code;
  }
}

This is how to pattern match a ClientError:

const person = await getPerson("id").catch(
  ClientError.mapCode({
    [ClientErrorCode.CONFIG_ISSUE]: (e) => {
      throw new Error(`Another error: ${e.code}`);
    },
    [ClientErrorCode.UNEXPECTED_ERROR]: () => 123 as const,
  })
);

Here users would be able to remap client errors to their own internal errors. Also, if you look at the inferred type of the person, it would be Person | undefined | 123.

But where does this ClientError come from? We parse error responses and map them into appropriate values (like undefined instead of 404) or ClientError instances depending on what is required. The goal is to hide the transport or API-specific stuff.

Here is the helper. Most of the conditions (if blocks) come from how you should handle Axios errors:

export function mapAxiosError<R>(errorMap: {
  [key: string]: (() => R) | ClientErrorCode;
}) {
  return function axiosErrorMapper(error: unknown): R {
    if (!isAxiosError(error)) {
      throw error;
    }
    if (error.response) {
      const data = SolarErrorResponseSchema.check(error.response.data);
      const firstError = data.errors[0];
      const valueMapper = errorMap[firstError.code] ?? errorMap["_"];
      if (typeof valueMapper === "function") {
        return valueMapper();
      }
      throw new ClientError({
        id: firstError.id,
        code:
          typeof valueMapper === "string"
            ? valueMapper
            : ClientErrorCode.UNEXPECTED_ERROR,
        detail: firstError.detail,
        status: firstError.status,
      });
    } else if (error.request) {
      throw new ClientError({
        code: ClientErrorCode.NO_CONNECTION,
        detail: "Connection failed, please check your network",
      });
    } else {
      throw new ClientError({
        code: ClientErrorCode.CONFIG_ISSUE,
        detail: "Request configuration is invalid",
      });
    }
  };
}

The helper again, but applied to getPerson:

export async function getPerson_v4(personId: string) {
  return httpClient
    .get(`v1/persons/${personId}`)
    .then((response) => response.data)
    .then(snakeToCamelCase)
    .then(restoreDates)
    .then((person) => PersonSchema.check(person))
    .catch(
      mapAxiosError({
        [SolarErrorCode.MODEL_NOT_FOUND]: () => undefined,
        [SolarErrorCode.UNAUTHORIZED_ACTION]: ClientErrorCode.UNAUTHORIZED,
        _: ClientErrorCode.UNEXPECTED_ERROR,
      })
    );
}

Now, half of the code in the getPerson would be repeated in other API functions, so we need to extract that code:

export interface TypedRequestConfig<T> extends AxiosRequestConfig<T> {
  schema?: Runtype<T>;
}

export async function makeRequest<T>(
  config: TypedRequestConfig<T>
): Promise<T> {
  const response = await httpClient.request(config);

  const result = snakeToCamelCase(restoreDates(response.data));

  if (config.schema) {
    return config.schema.check(result);
  }

  return result;
}

Which removes a lot of duplication:

export async function getPerson_v5(personId: string) {
  return makeRequest({
    method: "GET",
    url: `v1/persons/${personId}`,
    schema: PersonSchema,
  }).catch(
    mapAxiosError({
      [SolarErrorCode.MODEL_NOT_FOUND]: () => undefined,
      [SolarErrorCode.UNAUTHORIZED_ACTION]: ClientErrorCode.UNAUTHORIZED,
      _: ClientErrorCode.UNEXPECTED_ERROR,
    })
  );
}

Wrapping up

I bet you are already tired of writing code. But there's still so much to do:

A circuit breaker and retries. If the network fails, it is convenient to automatically retry. But you need to control this behavior and prevent infinite loops, especially with 401 responses and invalid tokens.
Logs for debugging. Bad things happen, and you will need to debug the app, probably even in a live environment. It is nice to have some way to enable logs showing requests and responses.
Telemetry. You probably want to collect response times and other data and sent it to Prometheus or another system.
Request overrides. There should be a way to attach additional headers, set a specific timeout, etc., for every API function individually.
Distributed tracing. This could be done through request overrides, but you probably want to set it up once during client initialization.
Get rid of process.env. Use a reliable config component like envalid. Use lodash or another library instead of typeof.
Who knows what else.

What I want you to understand is that making a production-grade client is difficult. You can definitely call an API from Node.js as shown at the beginning of this post. But after a couple of years of work of multiple developers or even teams, the app and the client will become enormous. Unmanaged and accidental complexity will silently waste the time of every developer when they add a new function, update it, or fix a bug. And it will not be visible on Jira charts. So, at some point in time you will need to refactor and you may use this post for inspiration.

Bonus Content

If you look at the getPerson_v4, you might notice that response processing is like a data pipeline with multiple steps. A successful response has its own steps, while an error response has its own. These steps are simple.

Speaking of steps, if we were to create a function for something like POST (where we send data to the API), we would have one more pipeline. This pipeline would process the request body: converting camelCase to snake_case, and so on. It's similar to the response pipeline, but in the opposite direction.

You've seen data pipelines before. In express, request processing is a data pipeline. Every middleware somehow affects or transforms the request until it reaches your handler (controller).

Many of the recommendations I made are applicable to frontend and universal clients too.

The full source code is available on CodeSandbox.

Searching through JSON logs with Node.js

spyke — Sat, 04 Mar 2023 20:29:44 +0000

I had to do a search through logs from our backend on my laptop. We have about 40 various web services that write logs in JSON format. And these logs together with logs from sidecars then streamed to Sematext (which is a managed ELK service). After a retention period logs got chunked into LZ4 compressed text files and backed up to an AWS S3 bucket.

The problem? There're a lot of logs. Every such archive is ~1 MB is size. It contains about 5,000 of JSON records. 1 month of logs is about 100k of files and about 100 GB in size.

Unfortunately, there was no established process on the project. A couple of team mates had experience with logs a year ago and all they did was: download, unpack, grep.

In my case grepping wasn't the best option as I needed not only to find the specific request (log), but also the review its distributed trace and context.

I had 2 options on my mind: local ELK and lnav. I tried to google about how to quickly run ELK in Docker locally (I had no prior experience in setting ELK up), but haven't found good tutorials. I believe they exist, but I also was thinking about possible caveats.

Our logs are not unified: some services populate different field names like status vs. statusCode, and they may also be of a different type (number vs. string). Writing ElasticSearch preprocessors in painless is not so painless. And not everything could be done there. I know this as I was setting up Sematext Pipelines before.

So, I've downloaded 112GB of logs for 1 month and tried to tame lnav instead. Unfortunately, lnav couldn't open my lz4 files because of some unexpected end of stream issues. lz4 command line utility produces such errors as well, but still decompresses fine. But uncompressed files are 8 MB instead of 1 MB (112GB --> 896GB?). I tried on an uncompressed directory with just couple of files and it was overwhelming. I was reading through lnav docs when saw that you may also define a scheme and load logs into an internal SQLite database.

This is also where I saw a new shiny bicycle 🚲 But at least it could be fun in comparison with boring docs...

I'm a JavaScript dev: loading/parsing files is easy. I also could easily transform all logs to a unified format and throw away fields that I don't need like Kubernetes labels, etc. And our team is used to Postgres. If I upload all the data into a local DB, it might be very convenient. And I could sort everything properly, because logs in files are sorted in order of streaming from multiple sources, not by actual timestamps. So many benefits!

I settled on a very simple streamed pipeline:

glob --> lz4 --> createReadStream --> readline
  --> transform --> filter --> insert

Creating CLIs with yargs is a piece of cake. For LZ4 there is a streaming implementation on npm which is a single function and also happened to decompress my files without any errors. Everything else is out-of-box Node.js.

Transformation itself is just creating a new object based on parsed JSON. I know all 12 fields I will need. Creating a DDL for Postgres table didn't took much time as it's structure is the same as TS interface I already defined.

Interestingly, every log has a natural Primary Key, because logs are in files, which are static. Backups only add up, no files are modified. We could use file_name + line_number to identify any log record. Why do we need this? For an incremental workflow. Inserting into a DB is very slow and is the main bottleneck. Of course, we could optimize it by using multi-value insert, disable triggers and constraints, etc. But I only disabled WAL for the table (as I will not modify row and can restore data) and decided to load data gradually:

Filter incoming records in JS to find the place in time where your issue(s) happened. For example, load only DELETE requests.
Use SQL GUI to view/filter this data, sort by time.
When time spans of interest are identified, tweak glob expression to load additional files for these time spans without filtering or with a different filer.
Repeat.

Filter is a function in the TS file, which gets a transformed record and returns a boolean. If it's true, the tool executes an INSERT command with Postgres' extension ON CONFLICT DO NOTHING. So, even if you try to load the same file multiple times, you would not get duplicate records thanks to the natural PK.

It looks like this:

$ yarn ts-node ./log-loader.ts './aws/2021/02/**/*.lz4'
...
Loading file [159432/159438]: ./aws/2021/02/24/21/20210224T214041.318Z_5950e47a96ee741360bedbf6153ab069.json.lz4
...
Done, 159,438 files loaded, 1,095,380,480 lines parsed, 241,584 records added
✨  Done in 7572.51s

Time varies greatly with the number of inserts. Still, it's not exactly instant, but is fast enough for my use case. After a couple of iterations I got proper distributed traces for several requests of my interest in the DB and found who called what and why. I saved time not in grepping, but in a nice GUI to look at final logs and understanding call stacks.

A couple of month gave me 397,785 final rows. SQL queries work quite fast even without indexes: a search by URL prefix took only 2.278 sec.

Actually, loading local files is not enough. I have only 512 GB on my laptop SSD and all it can fit is 3 month. So, for some cases I had to load data directly from AWS S3. It is slower for multiple runs, but I couldn't use external drive or other computer.

Adding S3 support was very easy too. I just extracted fs/glob code into a fsAdapter.ts and added awsAdapter.ts. If you specify --aws option, then the positional argument of CLI is treated as bucket + prefix and the AWS adapter is used. From AWS client you only need listObjects and getObject, after which you obtain the same Readable stream as you had with the file system.

Next time I might try to set up a local ELK to get a similar experience to not yet backed up logs (nice UI). As for the current project... I'm curious how many years the log-loader will be in use by my colleagues if the manual way lasted for almost 6 years.

Readying JavaScript packages for better future

spyke — Mon, 30 Nov 2020 14:28:21 +0000

Modern web applications consist of hundreds and thousands of files of different kinds. To manage this complexity and streamline app delivery developers started using the same approach as on desktops: compile & link. In relation to a web application compilation usually means transpiling, while linking means bundling.

Same goes for libraries and other packages that web applications depend on and install from registries like npm. But the actual code in these registries is almost always exists in a form of the old ES5. It is harder to read and debug such code, it may run worse on modern engines, and it will be down compiled one more time during the app build process.

Publishing ES5 code isn't that necessary anymore and may be avoided. An app could rely on the code in it's original form while providing better efficiency, dead code elimination, and easier debugging experience.

Babel became a de facto tool for down compiling JavaScript code to older versions and could be used as an integration point. Combining it with ECMAScript Modules and staged JavaScript development process governed by TC39 group it is possible to define a set of rules for how to publish and consume packages in their original form.

Raw Module Specification or RMS does exactly that. It is a convention for modern JavaScript packages and modules aiming to avoid excess code recompilation and deoptimization, retaining code readability and ease of debugging.

Specification

Package requirements

Package MUST follow Node.js ESM package format and have the module type in its package.json.
Package MUST contain valid ESM modules.
Package MUST NOT contain any code using features unsupported in the latest stable @babel/preset-env or core-js. Usually this means simply not using unfinished proposals.
Upgrading to a newer major Babel version IS a breaking change.

Installing a package

Install the latest stable @babel/preset-env and core-js.
Install the package.
At build time compile the package with Babel using @babel/preset-env preset and load stable polyfills from core-js (this could be done in preset options setting corejs and useBuiltIns props).

Upgrading the package

Upgrade @babel/preset-env and core-js to their latest stable versions.
Upgrade the package.

the-spyke / rms

Raw Module Specification is a convention for modern JavaScript packages and modules aiming to avoid excess code recompilation, deoptimization, retaining code readability and ease of debugging.

If you're interested in this approach and want to help with developing and improving the specification, please join the discussion here or open an issue on GitHub.

* Photo by Kelli McClintock on Unsplash

Wait, React isn't about virtual DOM?

spyke — Wed, 28 Oct 2020 15:43:03 +0000

Short answer for those who don't want to read:

Functional programming is the true theme of React. Virtual DOM is just one of its consequences and isn't really a benefit. It is slower by definition, and there could be no DOM at all if you're using React for sound or 3D rendering. Choose React if it matches your mindset, not because it was faster a couple of years ago.

Let's start with the opposite of virtual DOM: the real DOM. We're going to use an uncomplicated Counter component, who's content HTML may look like this:

<div>
  Count: 123
</div>
<div>
  <button type="button">Increment</button>
  <button type="button">Decrement</button>
<div>

Imaging how would you build it using plain JavaScript. Probably you'll go by one of these 2 ways: createElement or innerHTML.

Creating elements manually is time consuming. Just buttons section is almost screen height:

class Counter {
  /* rest of the code */

  renderButton(text, handleClick) {
    const button = document.createElement("button");

    button.setAttribute("type", "button");
    button.textContent = text;
    button.addEventListener("click", handleClick);

    return button;
  }

  renderButtons() {
    const buttons = document.createElement("div");

    buttons.append(
      renderButton("Increment", this.handleIncrement),
      renderButton("Decrement", this.handleDecrement),
    );

    return buttons;
  }
}

We need a createElement call per every node, to append all required children, etc. But having an element reference allows easy attaching event listeners.

innerHTML may look less, but needs ids/classes to assign listeners:

class Counter {
  /* rest of the code */

  render() {
    this.container.innerHTML = `
      <div>
       Count: <span id="label">${this.count}</span>
      </div>
      <div>
       <button type="button" id="btn-inc">Increment</button>
       <button type="button" id="btn-dec">Decrement</button>
      <div>
    `;

    this.label = document.getElementById("label");
    this.btnIncrement = document.getElementById("btn-inc");
    this.btnDecrement = document.getElementById("btn-dec");

    this.btnIncrement.addEventListener("click", this.handleIncrement);
    this.btnDecrement.addEventListener("click", this.handleDecrement);
  }
}

We use less lines on setting attributes, but more on searching for elements for future updates and adding excess classes.

Of course, no one wants to do such work manually. That's why we've got UI libraries like Angular, Vue, Svelte, and others. These 2 options of building a Counter are roughly what we get in a template-based library.

The innerHTML is somewhat the original AngularJS: our bundle contains the template string and the engine runs on the client by parsing this template, finding slots for data and expressions inside it, inserting it into the page, and attaching methods as listeners. Larger bundle size and additional load on the browser are downsides of this approach.

The createElement is like modern Svelte/Ivy, where the template is parsed/compiled build time into a set of document manipulation commands, so no string embedding or runtime is required. We get less bundle overhead and the code is optimized specifically for our component, but at a cost of loosing features on the client.

Looks not that complicated, right?

That's because we forgot the part with the template language: conditions and repeaters. All the good stuff anyone can't really use templates without. Imagine adding that to our Counter code: instead of a simple innerHTML we need to parse the string and "run" dynamic parts. What if condition changes later, how we're going to find out about that? Will we re-render only dynamic parts or the entire component? The codebase will be complicated and much larger.

But there's more. What if we need to use a custom Button component?

<div
  component="Button"
  label="Increment"
  onclick="this.handleIncrement"
></div>

It's doable. Just create this div element and pass it as a container to a class registered as Button. But it must be registered in advance:

const Button = require("../components/button.js");

UI.registerComponent("Button", Button);

Attributes should be parsed to distinguish between div's HTML attributes and arguments to the Button. Basically the div is now a sub-tree and should operate on its own.

But what if we want to use not just a Button, but one of several components conditionally?

<div
  components="this.isLoading ? 'Button' : 'Image'"
  label="Increment"
  onclick="this.handleIncrement"
></div>

It isn't a simple mapping anymore, but an expression, which needs to be compiled appropriately with JS executed at right times and the component instances destroyed/created. And those attributes may be re-parsed every time, because label could be an argument for a Button, but not for an Image.

Think about the original AngularJS with all its scopes, hierarchies, transclusion, etc. Complexity goes nuts with dynamically nested templates. That's why ng-include was static and we couldn't just render any template based on business logic.

But there's more. What if we need to build a component on the fly? Is it even possible, if template parsing and code emitting happens at build time?

We could get a team of super-stars and try to build an engine or a compiler providing all those features, but the point is that almost every feature influences the rules by which you will write template and/or logic because of it's complexity. And you're still somewhat restricted by a template.

Now, let's abstract away and get into a functional data driven land.

Everything in the world could be represented as a result of a function call and its arguments:

function(args) ⟶ anything

Inside a function you can do any kind of things including calling other functions (composition). We had functions (methods) before in the Counter class too, but with different insides.

Instead of only producing a result, methods alter existing state (in our case document elements with append or innerHTML), especially on counter updates. In functional world it is forbidden and passed arguments are immutable. Even if we pass a container div into a function, it can't add nodes here. Instead, we should rely only on the returned value. And in case of an update, to re-execute the function and to get next result out of it.

As we draw a UI, return values should describe it somehow. We could return an HTMLElement, but it has imperative mutable interface. Anyway, manually using document APIs is time-consuming as we know. Let's revisit HTML of our component:

<div>
  Count: 123
</div>

It's not that different from a JavaScript object.

const html = { element: "div", children: [
  "Count: 123"
] }

An object notation is more verbose for sure, as a general language should be to a DSL. But we could easily build such objects ourselves without mutating anything (and parsing a template). We could even reduce boilerplate by implementing a little helper:

function element(name, ...children) {
  return { element: name, children };
}

const ui = element("div",
  "Count: 123"
)

Moreover, objects can reference functions, so we don't need a map of pre-registered components:

function CounterLabel(children) {
  return element("div",
    "Count is ",
    element("span", ...children)
  );
}

const ui = element(CounterLabel, 0);

And the result would be:

const counterLabelResult = {
  element: "div",
  children: [
    "Count is ",
    { element: "span", children: [0] }
  ]
};

const ui = { element: CounterLabel, children: [0] };

Now we need somebody to recursively go through this object tree (UI description) calling functions (our components) inside element properties.

One more thing. A real-world UI needs to react on events like button click. How would we know to re-execute the function? Let's just pass a callback for this, which could be used, for example, as a click handler:

function FancyButton(children, refresh) { ... }

Assume that we've made such function that processes the object tree recursively, simultaneously passing the callback. We will call it getDescriber:

function getDescriber(component) {
  /*
   const describeUI = ...
   ...
  */
  return refresh => describeUI(component, refresh);
}

const describer = getDescriber(Counter);

describer accepts a refresh callback and outputs a complete UI description as a nested object of strings, numbers, and arrays (basically, a JSON).

The only part missing is a function to read this description and emit DOM elements into the document. We will call it render, and assume that we have its implementation already done by somebody:

function render(describer, mountNode) { ... }

render(describer, document.getElementById("root"));

Let's recap. We have 2 parts and just 3 functions:

element(name, ...children) and getDescriber(component) [react]
render(describer, mountNode) [react-dom]

Part #1 consists of element and getDescriber used together to make a description. Part #2 is only render, which is used exclusively when you need to get actual HTML elements. Both parts are independent. The only thing that connects them together is the structure of the description. render expects a nested object with element and children properties. That's all.

Part #1 could do whatever it wants: generate functions/closures of the fly and execute them, check conditions of any complexity... Instead of adding another complicated template language syntax you just use the whole power of JavaScript. As long as it outputs required objects, no downsides or limits of template engines exist.

You can call this object description a virtual DOM, but only if you're using that particular render function from above. We can make render that instead of calling document.createElement will... play sounds! We may interpret the description as we want. Is it DOM anymore?

As you might guess Part #1 is react and Part #2 is react-dom.

React isn't about virtual DOM. It's about abstracting away physical body of your structured data and helping you to update that structure over time. You work on the structure and data with React, some one else will materialize that structure later. Web pages do have a structure, so it's convenient for React to have a materializer for DOM. If Facebook was a music company, maybe React would have shipped with react-midi instead.

React is about functional approach, abstraction, flexibility, and unidirectional flow. Virtual DOM is a consequence of using it in a browser. Reconciliation and partial updates aren't fast. Manually crafted set of DOM manipulations is more effective by definition, and compilators can do this for templates. But React allows you to think different about UI, not as about strings and markup. React allows you to use functional composition for UI structure and a real language for UI logic. It's a mindset thing.

Building a logger with Undercut

spyke — Fri, 24 Jul 2020 14:00:59 +0000

In this tutorial we will use Undercut to build a non-complicated logger. The simplest one you can think of is just a console.log(), but we need more:

Ability to disable logging where not needed without commenting out lines.
Severity levels support with logger outputting only severe enough entries.
Log entries processing and transformation (like middlewares).
Support for custom destinations and outputting into other loggers.
Less code.

Let's start with use case example:

function myAction(userId, postId) {
  // code
  log.info(() => `User ${userId} requesting post ${postId}`);
  // code
  log.debug(() => [`Current DB context is`, { dbContext }]);
  // code
}

Severity levels are embedded into method names, and we use Arrow Functions for building messages (and template literals instead of formatted strings for simplicity). If we need to pass some context among the message, we could do this as a Tuple (an array).

Of course, we want to have more data than just a message. A log entry should be more robust and could be an object with various meaningful properties:

{
  severity: 3,
  level: 'debug',
  timestamp: '2020-07-23T13:56:19.325Z',
  message: 'Current DB context is',
  context: { },
}

Such object could be processed by middlewares, which may add more information to it, format some values, filter excess entries, etc. Basic process may look like this:

[log_entry] => [middleware_1] => [middleware_2] => ... => [done]

A middleware could also output an entry somewhere as Browser's console or a remote web server. It should be configurable too.

As we're going to utilize Undercut for this task, let's think what would be the best choice for our requirements. Log entries come with time. If we can't iterate synchronously, the best option would be to use Observers (Push Lines).

After slight modifications the diagram from above may look like this:

[log_entry] => [------------- push_line--------------]
               [operation_1] => [operation_2] => ... ]

Before codding the Logger itself, we need to define severity levels:

// level.js

export const ERROR = 0;
export const WARNING = 1;
export const INFO = 2;
export const DEBUG = 3;

We will use a factory function approach for the Logger. Start with options:

// logger.js_1

import { filter, pushLine, toNull } from "@undercut/push";
import { isString } from "@undercut/utils";

import * as Level from "./level.js";

export function createLogger(pipeline, options = {}) {
  const isEnabled = options.isEnabled ?? true;
  const lowestSeverity = options.severity ?? Level.INFO;

The function requires a pipeline (a list of operations-middlewares) and an options object. isEnabled allows disabling logger entirely, level defines lowest allowed severity level (all levels lower will be skipped).

  const isAllowed = severity => severity <= lowestSeverity && isEnabled;
  const observer = pushLine(pipeline, toNull());

The observer will represent a chain of middlewares. We're using toNull target because there can be more than 1 destination to write logs and users will specify destinations within the pipeline, so toNull is just a placeholder.

  function log(severity, messageFactory) {
    if (!isAllowed(severity)) return;

    const timestamp = Date.now();
    const message = messageFactory();
    const entry = isString(message)
      ? { severity, timestamp, message }
      : { severity, timestamp, message: message[0], context: message[1] };

    observer.next(entry);
  }

The log function represents the whole logic of the Logger. Firstly we need to check severity level as soon as possible to lower performance penalty. Then we call the messageFactory (an arrow function where you specify the message) and look if it returns a tuple of [message, context] or just a message string. These values represent initial log entry, which we pass to the observer.

  return {
    error: m => log(Level.ERROR, m),
    warning: m => log(Level.WARNING, m),
    info: m => log(Level.INFO, m),
    debug: m => log(Level.DEBUG, m),
    observer: filter(e => isAllowed(e.severity))(observer)
  };
}

The logger object has methods for each severity level. This could be done automatically with a helper like collectProps and going through the Level enumeration, but manual way is the simplest one to get type ahead in IDEs.

Notice the observer property. It could be used by a middleware to pass entries from one logger to another. The observer is wrapped into a filter with severity level check. This check is done only in the log function for performance, so we need to add it here too.

The Logger is finished, but we need to provide a built-in middleware for connecting loggers together.

Every middleware is just a Push Operation. Knowing this and we could reuse operations from Undercut instead of writing our own from scratch.

import { filter, forEach, map } from "@undercut/push";

export function toLogger({ observer }) {
  return forEach(entry => observer.next(entry));
}

All toLogger does is getting that observer from a logger logger and passing all incoming log entries to it.

Let's add more middlewares.

export const convertTimestampToISO = map(entry => ({
  ...entry,
  timestamp: new Date(entry.timestamp).toISOString()
}));

convertTimestampToISO maps incoming log entry to a new object (a clone) with timestamp set to an ISO string instead of original Unix Time number.

export const filterNoContext = filter(entry => entry.context);

filterNoContext is even more simple and filters log entries with no context property.

export const toConsole = prefix =>
  forEach(entry => console.log(prefix + JSON.stringify(entry)));

toConsole outputs every log entry as a JSON string to browser's console adding text prefix at the beginning.

Testing time:

import * as Level from "./level.js";
import { createLogger, toLogger } from "./logger.js";
import {
  addLevelName,
  addProps,
  convertTimestampToISO,
  filterNoContext,
  toConsole,
  uppercaseMessage
} from "./middlewares.js";

const logger1 = createLogger(
  [
    convertTimestampToISO,
    addLevelName,
    addProps({ test: true }),
    toConsole("logger1: ")
  ],
  { severity: Level.WARNING }
);

logger1 processes only log entries with levels WARNING and ERROR. Processing of entries looks like this:

Create base entry { severity, timestamp, message, context }.
Convert timestamp to ISO string.
Add level prop.
Add test prop with value true.
Output entry to the console prefixed by "logger1: ".

We could create another logger and connect it to the prevoius:

const logger2 = createLogger(
  [
    filterNoContext,
    toLogger(logger1),
    uppercaseMessage,
    toConsole("logger2: ")
  ],
  { severity: Level.INFO }
);

logger2 processes severity levels INFO, WARN, ERROR, only DEBUG will be skipped.

Create base entry { severity, timestamp, message, context }.
Filter out entry without contexxt.
Pass entry to logger1.
Make message prop uppercase.
Output entry to the console prefixed by "logger2: ".

Now we can log some stuff:

const context = { username: "root" };
let counter = 100;

logger1.debug(() => `Debug message: ${counter++}`);
// No output from logger1 because of low severity.

logger1.info(() => `Info message: ${counter++}`);
// No output from logger1 because of low severity.

logger1.warning(() => `Warning message: ${counter++}`);
// logger1: {"severity":1,"timestamp":"2020-07-24T12:34:58.894Z","message":"Warning message: 100","level":"warning","test":true}

logger1.error(() => `Error message: ${counter++}`);
// logger1: {"severity":0,"timestamp":"2020-07-24T12:34:58.895Z","message":"Error message: 101","level":"error","test":true}

logger2.info(() => [`Info message: ${counter++}`, context]);
// No output from logger1 because of low severity.
// logger2: {"severity":2,"timestamp":1595594098895,"message":"INFO MESSAGE: 102","context":{"username":"root"}}

logger2.error(() => `Error message: ${counter++}`);
// No output from logger1 because was filtered by logger2.
// No output from logger2 because of missing context.

logger2.error(() => [`Error message: ${counter++}`, context]);
// logger1: {"severity":0,"timestamp":"2020-07-24T12:34:58.895Z","message":"Error message: 104","context":{"username":"root"},"level":"error","test":true}
// logger2: {"severity":0,"timestamp":1595594098895,"message":"ERROR MESSAGE: 104","context":{"username":"root"}}

That's it. Try yourself to make some cool middlewares or use the knowledge in other projects. Feel free to ask your questions in comments.

Visit GitHub Undercut repository for code and undercut.js.org website for documentation.

Photo by NeONBRAND on Unsplash

What's new in Undercut 0.6.0

spyke — Wed, 22 Jul 2020 08:18:25 +0000

✂ We continue to add handy utilities into Undercut with release 0.6.0 while polishing its API. But this time there are also many internal changes making maintenance easier and users happier.

Code coverage

Code coverage is back where it should be above 80% mark, so using Undercut shouldn't be so scary anymore :) In fact, coverage was good from the beginning, but our monorepo structure were interfering with Jest/Istanbul processing. As result, most operation tests weren't considered as hits.

Website

Website have moved to Netlify. In past we were using GitHub Pages for hosting, but it isn't that convenient.

For example, one of the issues with GHP was the requirement to have full write access on CI as it needs to push files to the gh-pages branch on build. With Netlify you can continue using read-only keys. On top of that Netlify allows you to have a preview for Pull Requests and even separate deployments for non-main branches.

The setup was as easy as specifying yarn build command and a path to a directory to deploy.

Conventional Commits

Some time ago we started to adopt Conventional Commits and 0.6.0 release is the first one having its changelog generated from commit messages. Lerna supports Conventional Commits out of the box, which is nice.

Release Highlights

getObjectType for retrieving more detailed type name like AsyncFunction from Object's toString method.
Classic helpers in a form of head/tail utilities for working with Iterables. Unfortunately, the head utility have replaced former peekIterable, so we have a breaking change here.
A pack of utilities to quickly filter/map/collect object keys and values. You were able to do this with Pull already, but sometimes you only want a short single operation instead of the full power and ceremony of the pipelines.
A set of randomized functions like randomDecimal or randomIndex utilities.
Somehow missing isPromise utility.

Lots of other code and documentation improvements. Check out the list commits from previous version or view our full release notes.

Visit GitHub for code and undercut.js.org for documentation.

Photo by Florian Cario on Unsplash.

Reusing operations for cleaner code

spyke — Tue, 12 May 2020 14:57:55 +0000

Imagine we're getting a list of users from an API:

const users = getUsers();
// [{ id, name, location, ... }, ...]

We can do quite a complex processing using Undercut pipelines, but what if we need only to do something quick: like grouping users by a location and just passing the result further? Even calling pull may be too much:

sendPreview(pullArray([groupBy(u => u.location)], users));

A case with a variable may be distracting too:

const usersByLocation = pullArray([groupBy(u => u.location)], users);

sendPreview(usersByLocation);

Fortunately, Undercut operations are framework-agnostic, which means they work by themselves. An operation is just a functions doing something on an Iterables. Arrays are Iterables, so our users fit just right and groupBy(u => u.location)(users) will return grouped users:

sendPreview(groupBy(u => u.location)(users));

Now it's better as we have no noise like pullArray and square brackets.

There's another use case: you can define selectors using operations like so:

const groupUsers = groupBy(u => u.location);

// And use later:
const usersByLocation = groupUsers(users);

I have specifically chosen groupBy because with even shorter actions you will likely go with plain Array methods and Arrow Functions:

render(users.filter(u => u.active));

But if you do need more serious job like flatMap/groupBy/orderBy/etc or already using Undercut in other places, consider reusing from the Undercut or your own custom operations.

Undercut docs: undercut.js.org
Previous post: Lazy data processing using Undercut

Pitfalls of Flux Dispatcher

spyke — Fri, 07 Feb 2020 13:47:18 +0000

Flux was presented in May 2014 and quickly become a new movement in web development. Today Flux isn't that widely used. The driver seat was taken by its offspring Redux. Anyway, it's still interesting to discuss some of the issues with Flux's architecture about which you don't even think in Redux.

This one was famous:

"Cannot dispatch in the middle of a dispatch"

This error had to mean that you did dispatch() in a wrong time and need to move it somewhere else. The most brave people just ignored it by wrapping the dispatch into setTimeout(). But there were many other hacks to avoid it.

Flux's official website and issue tracker have no good explanation of how to deal with this problem only recommending not to dispatch. Unfortunately, there're too many scenarios when it's unavoidable. As you'll see later this error is only a symptom of a much larger issue.

Flux describes a store as a state manager of a domain. That means you will have more stores than 1. Same time, some stores may depend on another, what is described by calling waitFor() method.

Imagine a basic app of two components:

<App>
    <Posts />
</App>

The App is the root and shows a login screen instead of its children while user isn't authenticated. The Posts component starts loading its data in componentDidMount() hook what is the recommended practice. Both these components depend on different stores: AppStore and PostsStore. The PostsStore may also depend on the AppStore too, but it isn't important.

Let's look at the time when user just authenticated and the app got a positive answer from the server with user's session:

Actions are represented as arrow-like blocks. Let's follow the diagram:

AUTH_SUCCESS is dispatched. Flux Dispatcher starts calling stores' callbacks and does this in order.
AppStore's callback is called first, the store recalculates its state.
All AppStore subscribers start to update. We have only one subscriber in our case -- the App component.
The state was updated, and the App starts to re-render.
This time isAuth is true and we start rendering Posts (this happens synchronously).
componentDidMount() also happens synchronously. So, just right after the initial Posts render we start loading actual posts (Posts shows a Loader).
Loading posts means dispatching LOAD_POSTS_STARTED first.
What means we're back in the Flux Dispatcher, which will throw the nasty error.

Now look at the #5. When render happens we're still in the middle of dispatch. That means that only a part of stores were updated and we're looking at inconsistent state. Not only we're getting errors in totally normal scenarios, but even without errors the situation is hardly better.

The most popular solution to this entire scope of issues is to fire change event in setTimeout(). But this removes synchronicity of React rendering. Theoretically, event subscribers may be called in different order, because order of execution of setTimeout callbacks is unspecified (even if we know that browsers just add them to a queue).

I like another solution which isn't that well known, but lies on surface. Redux works this way and is consistent, error-less, and synchronous. The whole dispatch process inside Redux may be written as such:

dispatch(action) {
    this.$state = this.$reducer(this.$state, action);
    this.$emit();
}

It calculates the new state and only then calls subscribers. The state is always consistent, and the entire process is like an atomic DB transaction.

In Flux this approach would be more verbose, but still doable. Stores manage their subscribers individually, but they could return a function to dispatcher. This function will call store's emit(). Most of the time stores don't pass event arguments, so they would just return the emit itself. In case if you want to optimize some things and filter events based on args, a store may return a custom callback.

Taking Flux Dispatcher as the base only a few places require tweaks:

dispatch(payload){
    // No more "Cannot dispatch..."
    this._startDispatching(payload);
    // Same try/finally as before.
    // After state calculation notify all subscribers.
    this._notifyAll();
}

_notifyAll() {
    // In case of a nested dispatch just ignore.
    // The topmost call will handle all notifications.
    if (!this._isNotifying) {
        this._isNotifying = true;
        while (this._notifyQueue.length > 0) {
            const notify = this._notifyQueue.shift();
            notify();
        }
        this._isNotifying = false;
    }
}

_invokeCallback(id) {
    this._isPending[id] = true;
    // Save callback from the store to the queue.
    const notify = this._callbacks[id](this._pendingPayload);
    if (notify) {
        this._notifyQueue.push(notify);
    }
    this._isHandled[id] = true;
}

It requires some error handling code, but the idea should be clear. This is how a store may look like:

class PostsStore extends EventEmitter {
    constructor(dispatcher) {
        this.$emit = this.$emit.bind(this);
        this.$posts = {};
        this.dispatchToken = dispatcher.register(payload => {
            switch (payload.actionType) {
                case "LOAD_POSTS_SUCCESS":
                    // Don't forget to "return" here
                    return this.$loadPosts(payload);
            }
        };
    }

    $loadPosts(payload) {
        this.$posts[payload.userId] = payload.posts;
        return this.$emit; // A generic case with no args;
    }

    $clearPosts(userId) {
        delete this.$posts[userId];
        // When only a part of subscribers should update.
        return () => this.$emit(userId);
    }
}

The rest of the app's code stays the same.

This solution has not that big refactoring penalty, but gives you state consistency, removes unnecessary errors, and keeps update-render process synchronous and simple to follow.

Atomicity is a nice property which we hadn't in Flux and not always notice in Redux. Redux is also more simple, maybe that's why we (the community) haven't seen implementations like Atomic Flux Dispatcher and moved forward straight to Redux.

Originally posted on Medium in 2019.

Async Redux doesn't exist

spyke — Wed, 05 Feb 2020 08:32:53 +0000

Sometimes people ask what is the best way to handle asynchronicity in Redux? There is official documentation about it, but I suggest revisiting some basic concepts to see if it's really that simple.

The basics

A state is an object. It's used as a value somewhere on UI or for its rendering:

{
    username: "zerocool"
}

An action is an object too. It describes an event (or a command) happened in app's world. By convention it must have the "type" property containing event name and may have some other data:

{
    type: "ADD_TODO",
    text: "Hello"
}

A reducer is a function. Its signature is

(state, action) => state

The following example has a function with similar signature and even a comparable method name "reduce":

[1, 2, 3].reduce((acc, item) => acc + item, 0)

In fact, this is exactly what happens in Redux, but instead of an array of numbers Redux gets an infinite array (stream) of events (actions), and its reduction spans the life-time of the app. Of course, state and action could be primitive types in Redux too, but in real world apps it isn't super useful.

A reducer is all about computation. Nothing more, nothing less. It is synchronous, pure, and simple like a sum.

Developers use Redux through a store. It is an object that remembers the computation (reducer) and its first argument (state) freeing you from passing it every time. Interactions are based on calling dispatch() method to run the computation and accessing the last computed value by calling getState(). Parameter types are irrelevant to dispatch() because it simply passes them to reducer, dispatch() doesn't return a value either. This is how a simple Redux store may look and work like:

// Instead of manually implementing store subscriptions we could use EventEmitter.
class Store extends EventEmitter {
    constructor(fn, value) {
        super();
        this.$fn = fn;
        this.$value = value;        
    }

    getState() {
        return this.$value;
    }

    dispatch(data) {
        // This is the only thing happening inside a store.
        this.$value = this.$fn(this.$value, data);
        this.emit("change");
    }
}

// Let's try the store on numbers.
const store1 = new Store((acc, item) => acc + item, 0);

// And output its state to the console on every dispatch.
// "on()" is similar to "subscribe()" in the Redux and comes from EventEmitter.
store1.on("change", () => console.log(store1.getState()));

[1, 2, 3].forEach(item => store1.dispatch(item));
// 1
// 3
// 6

// Now let's try a more real-world reducer.
const store2 = new Store((state, action) => {
    switch (action.type) {
        case "ADD_ITEM":
            return { ...state, items: [...(state.items || []), action.item] };
        default:
            return state;
    }
}, {});

// Outputting the state as a JSON.
store2.on("change", () => console.log(JSON.stringify(store2.getState())));

store2.dispatch({ type: "ADD_ITEM", item: "Hello" });
// {"items":["Hello"]}
store2.dispatch({ type: "ADD_ITEM", item: "World" });
// {"items":["Hello","World"]}

It looks KISSish and complies with the Single responsibility principle. The example is so simple that it's hard to imagine where to put asynchronicity into. As you will see later, attempts to add asynchronicity will break some of the definitions written above.

By the way, the original Redux isn't that small. Why? Because it provides various utilities: middlewares, store enhancement, etc. More on this later.

Asynchronicity

If you try to read Redux docs about asynchronicity, the first page you'll encounter is the Async Actions page. Its title looks rather strange because we know that actions are objects and objects can't be async. Reading further down you see Async Action Creators and middlewares for them.

Let's look at what are regular synchronous Action Creators first. From the docs:

Action creators are exactly that - functions that create actions.

function addTodo(text) {
    return {
        type: "ADD_TODO",
        text
    }
}

dispatch(addTodo("Finish the article"));

A factory function for reducing code duplication in creating action objects, cool. If there're dispatches of same actions in different parts of the app, Action Creators may help.

Middlewares. They are utilities to override store's behavior in more functional style (like Decorators in OOP). So, you don't have to write this by hand if you want to log every dispatched action to the console:

const originalDispatch = store.dispatch;

store.dispatch = function myCustomDispatch(action) {
    console.log(`action : ${action.type}`);
    originalDispatch.call(this, action);
};

In reality it looks more like a chain of dispatch functions calling each other in order with the original one in the end. But the idea is similar. Async Action Creators require specific middlewares to work, let's check them out.

Redux Thunk

The first one on the list is redux-thunk. This is how a thunk may look like:

function addTodo(text) {
    return dispatch => {
        callWebApi(text)
        .then(() => dispatch({ type: "ADD_TODO", text }))
        .then(() => sendEmail(text));
    };
}

dispatch(addTodo("Finish the article"));

From the description of the library:

Redux Thunk middleware allows you to write action creators that return a function instead of an action.

Returning a function from Action Creators? Actions Creators create actions (objects), it's obvious from their name. There should be a new term instead.

Google says that by returning functions you may continue to dispatch normally and components will not depend on Action Creators' implementation. But dispatching "normally" means running the computation of the new state and doing it synchronously. With this new "normal" dispatch you can't check getState() to see the changes right after the call, so the behavior is different. It's like patching Lodash.flatten() to allow you to continue "normally" flattening Promises instead of Arrays. Action Creators return objects, so there's no implementation either. Same time, presentational components don't usually know about dispatch(), they operate with available handlers (passed as React props). Buttons are generic. It's Todo page who decides what a button does, and this decision is specified by passing the right onClick handler.

The thunk can be used to delay the dispatch of an action, or to dispatch only if a certain condition is met.

A dispatch() is a function call, just like sum(). How to delay sum() in JavaScript? By using setTimeout(). How to delay a button click? With setTimeout(), but inside a handler. It is unlikely that patching a button to know how to delay clicks (if it is not a button animating delay countdown, which is different) is necessary. How to call a function if certain conditions are met? By adding an "if-then-else" block inside a handler. Plain JS.

Looking closer at the proposed dispatch call itself. Not only it changes dispatch's interface:

dispatch(dispatch => { … });

But we're passing a function expecting dispatch as an argument into a function called dispatch. This is quite confusing 🤷‍♂️ Melding together different concepts removes simplicity and raises contradictions. But what is the problem that Redux Thunk is trying to solve in the first place?

function handleAddTodo() {
    dispatch(addTodo(text));
}

<Button onClick={handleAddTodo}>Add Todo</Button>

Adding some async calls turns into:

function handleAddTodo() {
    callWebApi(text)
        .then(() => dispatch(addTodo(text)));
}

<Button onClick={handleAddTodo}>Add Todo</Button>

Nothing has changed for the button, but there is a problem indeed if you have several identical handleAddTodo() implementations in different parts of the app. Cutting corners with Redux Thunk may look like a solution, but still will add all downsides this middleware introduce. It can be avoided by having only one implementation somewhere on upper level and passing it down or by extracting dispatch() calls into external functions (basically moving handleAddTodo() to another file).

Redux Promise

Redux Promise encourages you to dispatch Promises. It is very similar by effect to Redux Thunk, so I'll skip it.

There is also another way encouraged by subsequent middlewares, but let's step aside from thunks and asynchronicity for a second and talk about processes happening inside apps.

Business Logic

Apps react on users and environment. Complexity of reactions grows with app's complexity. Instead of simple things like changing button's color on a click, apps starts to execute rather complex scenarios. For example, adding a Todo record to the state is simple. Adding it also to the local storage, syncing it to a backend, showing a notification on the screen… is not so. Somewhere between those steps may be even a user interaction.

Such groups of actions are usually represented by flow charts and have many names: flows, workflows, control flows, business processes, pipelines, scenarios, sagas, epics, etc. I will use the term "workflow". A simple money transfer between two bank accounts internally may be a huge operation involving distributed transactions between multiple independent parties. But the workflow from the image above may be a simple function:

function addTodoWorkflow(text) {
    dispatch(addTodo(text));
    saveToLocalStorage(text);

    if (isSignedIn) {
        const response = syncWithServer(text);

        if (response.code === OK) {
            showSuccess();
            dispatch(todoSynced());
        } else {
            showError();
        }
    }
}

It looks like and totally is a regular function composition. I made it sync, but it will be the same with promises.

From the point of workflow's view dispatch(), syncWithServer(), and Lodash.groupBy() are the same.

Browser APIs, web clients, libraries, triggering UI changes, coming from imports or arriving in arguments, sync or async. They all are just some services that were composed into a workflow to do the job. Even if a workflow is asynchronous, you still run it like this:

addTodoWorkflow(args...);

If you have a button submitting a Todo, just call it in the event handler. In more advanced scenarios you will have tons of async stuff, cancellation, progress reporting, etc. Achieving this is possible with extended promises, generators, streams, and other libraries and techniques (such as reactive programming).

Workflows exist is many areas of software development, and they aren't tied to UI state management. They may also call dispatch() several times with completely different action types or not to have UI indication and state change at all. Workflows may be composable just like functions in JS. Similar concepts exist even high in the clouds and in IoT.

Understanding that workflows are a separate concern is important. By moving business logic into Action Creators this separation starts to vanish. Redux doesn't require special treatment, nor it is more important than other subsystems in the app.

There two ways of executing workflows: directly and indirectly.

The direct way is the simplest: you call the workflow directly in a handler. This way you have a good visibility of what will happen and control right in the code:

function onAddTodoClick() {
    addTodoWorkflow(text);
}

The indirect way is opposite. You start with a dummy action like ADD_TODO that must not change any state, but there is another system subscribed to Redux actions. This system will launch a workflow defined for this specific action. This way you may add functionality without updating UI components' code. But now you have no idea what will happen after a dispatch. Let's look on the middlewares.

Redux Saga

Redux Saga isn't really about the Saga pattern.

The Distributed Saga pattern is a pattern for managing failures, where each action has a compensating action for rollback.

It doesn't help you dealing with state rollbacks. Instead it allows you to write workflows in a CSP-style manner, but with power of generators (which is great). There're very few mentions of Redux in the docs. 99% of Redux Saga are about sagas themselves hidden in sub-packages.

Sagas are pure workflows, and the docs teach you to manage running tasks, doing effects, and handling errors. The Redux part only defines a middleware which will repost actions to the root saga. Instead of manually building a map [Action → Saga] you need to compose all sagas into a tree similar to reducers composition in Redux. UI code remains the same:

function addTodo(text) {
    return {
        type: "ADD_TODO",
        text
    }
}

function handleAddTodo() {
    dispatch(addTodo(text));
}

<Button onClick={handleAddTodo}>Add Todo</Button>

Changes happen only in the corresponding saga:

function* addTodoSaga(action) {
    yield takeEvery("ADD_TODO", function* (action) {
        const user = yield call(webApi, action.text);
        yield put({ type: "ADD_TODO_SUCCEEDED" });
    });
}

function* rootSaga() {
    yield all([
      ...,
      addTodoSaga()
    ]);
}

It is dramatically different to Redux Thunk: the dispatch() hasn't changed, Action Creators stay sync and sane, Redux continues to be simple and clear.

Redux Observable

Redux Observable is identical to Redux Sagas, but instead of CSP and Sagas you work with Observables and Epics leveraging RxJS (more difficult, but even more powerful).

Retrospective

So, what is the best way to handle asynchronicity in Redux?

There is no asynchronicity in Redux. You should not build a facade with middlewares like Thunk hiding the real Redux behind it. It couples the knowledge of workflow execution with UI state management and makes the terminology complicated.

There are ways to react on actions in a better manner. You may choose a direct approach of calling workflows manually and/or going by indirect path of binding workflows to actions. Both ways have their own strengths and weaknesses.

Sagas provide a nice balance in ease of use, functionality, testability and may be a good starting point. Same time, choosing Sagas over calling workflows directly is like choosing between Redux and React State: you don't always need the former.

In advanced scenarios with async modules you may want to register new sagas/epics on demand instead of a prebuilt root saga/epic. But usually it's better not to overthink.

Originally posted on Medium in 2019.

Lazy data processing using Undercut

spyke — Mon, 20 Jan 2020 12:31:42 +0000

Undercut is a JavaScript library for processing data in a lazy or deferred manner by building pipelines.

The focus of the library is on leveraging existing JavaScript features like Iterators/Generators while having balanced API: not being Java/C# influenced or heavily functional. Undercut also aims to avoid prototype extension and a situation, where you need to name a method as flat instead of flatten. You may also use it as an alternative to Lodash's chain functionality with support for lazy execution, tree shaking, etc.

Imagine a conveyor on a car factory: a chain of operations from welding body parts and painting doors to gluing a logo and inflating wheels. Every operation is independent and is based only on a protocol: a car comes from this side and goes to that side after the operation is complete.

In JavaScript we may represent this as an array of functions:

const pipeline = [
    skip,
    map,
    filter,
    take,
];

Of course, those operations have some input data: this car should have 17" wheels, that car should have 16" wheels. We can do this too:

const pipeline = [
    skip(1),
    map(x => x - 3),
    filter(x => x !== 4),
    take(100),
];

Calling skip(1) creates a function (operation) that knows how to skip exactly 1 item (car).

Sometimes you need to make a new model with additional equipment package. It may be as simple as adding a couple of steps to the conveyor:

const pipeline_2 = [
    ...pipeline,
    filter(x => x < 1000)
];

Or replacing some steps in existing:

pipeline[1] = map(x => x - 4);

Arrays give you this flexibility to concatenate, merge, copy, and modify existing pipelines.

To finish the conveyor there should be some mechanism like moving belt that will transport a car from one operation from another. This is where Undercut tries to help (not mentioning a pack of 40+ prebuilt common operations like filter/map/skip/etc).

Core pull functions allow you to quickly run a pipeline and acquire the result or combine it into something self-contained and reusable like an Iterable.

Having a list of numbers called source:

const source = [1, 2, 3, 4, 5, 6, 7];

And a pipeline of operations:

const pipeline = [
    skip(1),
    map(x => x - 3),
    filter(x => x !== 4),
    take(100),
];

We could pull items out of the source through the pipeline and get an array of result items:

const result = pullArray(pipeline, source);

In our case result will be:

[ -1, 0, 1, 2, 3 ]

All is done lazily, so map won't run for the skipped item. There're also pullValue, if your result is a single value (not a sequence). Or more generic pull, where you pass target function getting result items and converting it into whatever you want.

As pull is built around Iterables, and many native objects are Iterable out of the box (arrays, strings, maps, sets, etc), you can easily transform a Map of Usernames-by-Id into an Object of Ids-by-Username.

const namesById = new Map([
    ["root", 0],
    ["sam", 1000],
    ["kate", 1004],
]);

const pipeline = [
    filter(entry => entry[0] > 0),
    map(entry => [entry[1], entry[0]]),
];

const idsByNameObj = pull(Object.fromEntries, pipeline, namesById);

// idsByNameObj == Object {"1000":"sam","1004":"kate"}

Moreover, you may create a reusable view of this data:

const idsByName = pullLine(pipeline, source);

The pullLine function binds together a pipeline and a source into an Iterable. Every time you iterate over it, the pipeline will be executed again, giving you a fresh view on processed data.

namesById.set("sam", 1111);

console.log(Object.fromEntries(idsByName)); // Object {"1111":"sam","1004":"kate"}

Every operation is just a function, so you can create your own. Or even create a whole library of your own operations and reuse in different projects. The protocol, operations rely on, is similar to car-in/car-out, but instead of cars there're Iterables. An operation get an Iterable of items to process and return an Iterable of processed items. Returning an Iterable sounds complicated, but it isn't with JavaScript Generators.

Let's build a pow operation:

function* powOperation(iterable) {
    for (const item of iterable) {
        const newItem = Math.pow(item, exponent);

        yield newItem;
    }
}

Get an Iterable, go by its items, calculate new values, put them into another iterable with yield.

If you aren't familiar with generators (functions marked with * asterisk). Basically, the return value of such function will be not what you return, but an implicit Iterable you can put items into with the yield keyword. Please read MDN for more detailed decsription. I also recommend reading an awesome book Exploring ES6 by Dr. Axel Rauschmayer.

Actually, one important aspect is missing. The exponent value isn't defined and should be assigned in a pipeline like those 17" wheels. To fix this just add another function around:

function pow(exponent) {
    function* powOperation(iterable) {
        for (const item of iterable) {
            const newItem = Math.pow(item, exponent);

            yield newItem;
        }
    }
}

And this pow we can actually use:

const source = [0, 1, 2];
const pipeline = [
    map(x => x + 1),
    pow(2),
];

const result = pullArray(pipeline, source);

console.log(result); // [1, 4, 9]

It was only a brief review of the Undercut, but should be enough for basic use cases. If you want to learn more, please with undercut.js.org for documentation and tutorials.