<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Joung Park</title>
    <description>The latest articles on DEV Community by Joung Park (@joungpark).</description>
    <link>https://dev.to/joungpark</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F716955%2Fffdd6810-23b5-413d-94c2-d9830183b088.jpeg</url>
      <title>DEV Community: Joung Park</title>
      <link>https://dev.to/joungpark</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/joungpark"/>
    <language>en</language>
    <item>
      <title>AWS AppConfig: Feature Flags, Configuration, and How Services Read Them</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Wed, 11 Nov 2026 00:12:01 +0000</pubDate>
      <link>https://dev.to/joungpark/aws-appconfig-feature-flags-configuration-and-how-services-read-them-2l49</link>
      <guid>https://dev.to/joungpark/aws-appconfig-feature-flags-configuration-and-how-services-read-them-2l49</guid>
      <description>&lt;p&gt;We centralized feature flags and service configuration in &lt;strong&gt;AWS AppConfig&lt;/strong&gt; instead of a monolith flag store and S3 env files baked in by CI/CD. &lt;strong&gt;Amazon’s AppConfig Agent or extension&lt;/strong&gt; delivers settings on &lt;strong&gt;&lt;code&gt;localhost:2772&lt;/code&gt;&lt;/strong&gt;; backend services use one &lt;strong&gt;shared library&lt;/strong&gt; with the same read pattern on every stack. Fictional examples: &lt;code&gt;payments-api&lt;/code&gt;, &lt;code&gt;client-app&lt;/code&gt;, flag &lt;code&gt;enable_new_checkout_flow&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This post covers generic flags and configuration, &lt;strong&gt;AWS AppConfig&lt;/strong&gt; and delivery on port 2772, how we layout and merge profiles for services, and local development against a mock.&lt;/p&gt;

&lt;h2&gt;
  
  
  Feature flags and configuration
&lt;/h2&gt;

&lt;p&gt;Feature flags and configuration answer different questions. Teams usually store them separately, but services often need both at runtime. Shipping safely means knowing which is which before you wire a new key.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Feature flags&lt;/th&gt;
&lt;th&gt;Configuration&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Purpose&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Turn behavior on/off; optional &lt;strong&gt;attributes&lt;/strong&gt; for sub-toggles&lt;/td&gt;
&lt;td&gt;Strings, numbers, URLs—settings with more than two states&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Shape&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Named flags with an &lt;code&gt;enabled&lt;/code&gt; switch (and optional fields)&lt;/td&gt;
&lt;td&gt;Key–value (or structured) settings per service or environment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;When to use&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Gate code paths (web, mobile, microservices)&lt;/td&gt;
&lt;td&gt;Connection strings, feature thresholds, UI copy, service knobs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Typical change&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Flip a boolean or attribute without redeploying (if you read at runtime)&lt;/td&gt;
&lt;td&gt;Adjust a value; may still require deploy if config is baked at build time&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  How we did before
&lt;/h3&gt;

&lt;p&gt;Flags lived in a central database behind a PHP monolith and cache; Lambdas, ECS, and mobile often called a monolith HTTP API. Configuration was a &lt;strong&gt;&lt;code&gt;.env&lt;/code&gt; on disk&lt;/strong&gt; on laptops; in AWS, &lt;strong&gt;CI/CD&lt;/strong&gt; pulled env files from &lt;strong&gt;S3&lt;/strong&gt; into task definitions or instances at deploy time.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Area&lt;/th&gt;
&lt;th&gt;Before&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Feature flags&lt;/td&gt;
&lt;td&gt;Central DB + cache behind the monolith&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Application config&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;.env&lt;/code&gt; locally; S3 env file wired in by CI/CD at deploy&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That split hurt in a growing fleet: no shared flag source without the monolith, weak audit on config changes, and many knob turns required redeploy or restart. What follows is the centralized store and delivery model we use today, and how services read configuration from it.&lt;/p&gt;

&lt;h2&gt;
  
  
  AWS AppConfig
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;AWS AppConfig&lt;/strong&gt; is the managed store we use for both feature flags and configuration. In our setup it replaces the monolith flag database and the S3 env file pattern with versioned definitions and explicit deployment to each environment.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;What it provides&lt;/strong&gt; — Hosted configuration and feature flags, validators on config, version history, and &lt;strong&gt;per-environment deployment&lt;/strong&gt; (a console edit is not live until you deploy that version).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Why it fit&lt;/strong&gt; — One audit-friendly place for flags and config; services can read at runtime instead of redeploying for every toggle, when they use the delivery path below.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;How objects nest&lt;/strong&gt; — An &lt;strong&gt;application&lt;/strong&gt; (for example &lt;code&gt;global-feature-flags&lt;/code&gt; or &lt;code&gt;payments-api-config&lt;/code&gt;) contains &lt;strong&gt;environments&lt;/strong&gt; (&lt;code&gt;dev&lt;/code&gt;, &lt;code&gt;qa&lt;/code&gt;, &lt;code&gt;prod&lt;/code&gt;) and &lt;strong&gt;configuration profiles&lt;/strong&gt; (for example &lt;code&gt;feature-flags&lt;/code&gt;, &lt;code&gt;private-config&lt;/code&gt;). The layout we provision is spelled out in the next major section.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;How runtimes reach it&lt;/strong&gt; — Our workloads do not embed AppConfig API clients. &lt;strong&gt;AppConfig Agent&lt;/strong&gt; on ECS and EC2 and the &lt;strong&gt;AppConfig extension&lt;/strong&gt; on Lambda cache configuration locally, poll the &lt;strong&gt;data plane&lt;/strong&gt;, and expose &lt;strong&gt;&lt;code&gt;localhost:2772&lt;/code&gt;&lt;/strong&gt; inside the process network.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Feature flags use AppConfig’s feature-flag profile type; other settings use freeform JSON with schema validation at deploy time (details in the layout section below).&lt;/p&gt;

&lt;h3&gt;
  
  
  Agent and extension on &lt;code&gt;localhost:2772&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;On each stack, the listener on &lt;strong&gt;port 2772&lt;/strong&gt; is &lt;strong&gt;published by AWS&lt;/strong&gt;. Application processes issue HTTP &lt;strong&gt;GET&lt;/strong&gt; requests on localhost; they do not implement polling or sync to AppConfig themselves.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stack&lt;/th&gt;
&lt;th&gt;Component on 2772&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;ECS Fargate&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;AppConfig Agent&lt;/strong&gt; container image (&lt;code&gt;public.ecr.aws/aws-appconfig/aws-appconfig-agent:2.x&lt;/code&gt;) in the task&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;EC2 / PM2&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;AppConfig Agent&lt;/strong&gt; on the host (e.g. installed from RPM on the AMI)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Lambda&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;AppConfig extension&lt;/strong&gt; layer (&lt;code&gt;AWS-AppConfig-Extension&lt;/code&gt;) in the execution environment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Public config Lambda&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Same &lt;strong&gt;extension&lt;/strong&gt; pattern; mobile and web clients have no agent on the device&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every stack uses the same URL pattern:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;http://localhost:2772/applications/&amp;lt;application&amp;gt;/environments/&amp;lt;env&amp;gt;/configurations/&amp;lt;profile&amp;gt;&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;The workload GETs that URL; the agent or extension returns its in-memory cache. &lt;a href="https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-agent.html" rel="noopener noreferrer"&gt;Agent overview&lt;/a&gt; and &lt;a href="https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-integration-lambda-extensions.html" rel="noopener noreferrer"&gt;Lambda extensions&lt;/a&gt; cover polling intervals and tuning (for example &lt;code&gt;AWS_APPCONFIG_EXTENSION_POLL_INTERVAL_SECONDS&lt;/code&gt; on Lambda).&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
  AppConfig[AWS AppConfig data plane]

  subgraph ecsPath [ECS Fargate]
    WorkloadE[your workload]
    ECSAgent[AppConfig Agent container]
    WorkloadE --&amp;gt;|"GET localhost:2772"| ECSAgent
  end

  subgraph ec2Path [EC2 PM2]
    WorkloadH[your workload]
    EC2HostAgent[AppConfig Agent on host]
    WorkloadH --&amp;gt;|"GET localhost:2772"| EC2HostAgent
  end

  subgraph lambdaPath [Lambda service]
    WorkloadL[your workload]
    LambdaExtension[AppConfig extension layer]
    WorkloadL --&amp;gt;|"GET localhost:2772"| LambdaExtension
  end

  subgraph publicConfigPath [Public config API]
    PublicHandler[handler]
    PublicExtension[AppConfig extension layer]
    PublicHandler --&amp;gt;|"GET localhost:2772"| PublicExtension
  end

  ECSAgent --&amp;gt;|poll| AppConfig
  EC2HostAgent --&amp;gt;|poll| AppConfig
  LambdaExtension --&amp;gt;|poll| AppConfig
  PublicExtension --&amp;gt;|poll| AppConfig&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;&lt;strong&gt;Deploy, poll, and application cache&lt;/strong&gt; are three different clocks. A configuration edit in the console does not affect the data plane until you &lt;strong&gt;deploy&lt;/strong&gt; that version to an environment. After deploy, the agent or extension &lt;strong&gt;refreshes on its poll interval&lt;/strong&gt; (often on the order of 45 seconds). Even an up-to-date agent cannot help if the application read flags once at boot and never asked again.&lt;/p&gt;

&lt;p&gt;Config read &lt;strong&gt;at build time&lt;/strong&gt;—for example when generating a task definition or &lt;code&gt;.env&lt;/code&gt; file—&lt;strong&gt;bakes&lt;/strong&gt; values into artifacts; changing AppConfig still requires redeploying those artifacts for those keys. &lt;strong&gt;Run-time&lt;/strong&gt; GETs to 2772 can pick up flag changes without a new image, as long as deploy, poll, and re-read behavior align. IAM on the task or instance role allows the AWS listener to reach AppConfig; your process only talks to localhost.&lt;/p&gt;

&lt;p&gt;When a flag change does not show up, confirm deployment finished for the right environment, allow time for the agent or extension to poll, and check whether the service re-fetches flags after startup.&lt;/p&gt;

&lt;h2&gt;
  
  
  How services consume AppConfig
&lt;/h2&gt;

&lt;p&gt;As sketched above, an &lt;strong&gt;infrastructure repository&lt;/strong&gt; (CDK) provisions AppConfig applications, environments, profiles, and a &lt;strong&gt;public HTTPS config API&lt;/strong&gt; per account and environment. The layout is &lt;strong&gt;centralized flags, federated per-app config&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;We use one &lt;strong&gt;shared library&lt;/strong&gt; on ECS, EC2, Lambda, and local mock: same localhost contract on every stack, independent of which agent or extension sits on 2772. It does not poll AppConfig or replace those AWS components.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;AppConfig application&lt;/th&gt;
&lt;th&gt;Profiles&lt;/th&gt;
&lt;th&gt;Role&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;global-feature-flags&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;feature-flags&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Shared flag namespace&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;global-config&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;public-config&lt;/code&gt;, &lt;code&gt;private-config&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Defaults for every app&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;payments-api-config&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;public-config&lt;/code&gt; / &lt;code&gt;private-config&lt;/code&gt; (if defined)&lt;/td&gt;
&lt;td&gt;Service-specific overrides&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;client-app-config&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Often &lt;code&gt;public-config&lt;/code&gt; only&lt;/td&gt;
&lt;td&gt;Mobile/web-facing settings&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The &lt;strong&gt;app registry&lt;/strong&gt; describes allowed keys per logical app; infrastructure turns those definitions into JSON Schema validators on each profile. Registering a &lt;strong&gt;new service&lt;/strong&gt; means a registry entry and a deploy; registering a &lt;strong&gt;new flag&lt;/strong&gt; updates the canonical flag definition without touching every app entry.&lt;/p&gt;

&lt;h3&gt;
  
  
  How profiles merge when you read them
&lt;/h3&gt;

&lt;p&gt;Which profiles exist is a layout concern; &lt;strong&gt;who merges what&lt;/strong&gt; depends on the caller.&lt;/p&gt;

&lt;p&gt;For a service such as &lt;code&gt;payments-api&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Flags&lt;/strong&gt; come only from the global feature-flag application and profile—not from per-app config.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Config&lt;/strong&gt; is a &lt;strong&gt;shallow merge&lt;/strong&gt;: global defaults, then service-specific overrides; the app wins on key clashes. A missing app profile is treated as empty.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Public vs private&lt;/strong&gt; profiles control which HTTP API may return them—not a second merge algorithm. Private values must not appear on the mobile-facing route.
&lt;/li&gt;
&lt;/ul&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
  subgraph store [AppConfig applications]
    GF[global-feature-flags]
    GC[global-config]
    AppApp[payments-api-config]
  end
  subgraph read [Merged response for payments-api]
    FF[featureFlags]
    CFG[config object]
  end
  GF --&amp;gt; FF
  GC --&amp;gt; CFG
  AppApp --&amp;gt; CFG&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;Two consumption patterns sit on top of that layout:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mobile and web&lt;/strong&gt; call a &lt;strong&gt;public config Lambda&lt;/strong&gt; behind API Gateway. That component reads through 2772 and returns one JSON object &lt;code&gt;{ featureFlags, config }&lt;/code&gt; suited to clients.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backend services&lt;/strong&gt; use the shared library with &lt;strong&gt;two read patterns&lt;/strong&gt;: the &lt;strong&gt;shared flag namespace&lt;/strong&gt; only (no other apps’ config), and &lt;strong&gt;this app’s config only&lt;/strong&gt; (global private defaults plus that app’s private profile—never another service’s profiles). The library does not assemble flags and config into one in-process payload the way the public API does.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Environment names in URLs must match the AppConfig &lt;strong&gt;environment&lt;/strong&gt; the workload runs in (&lt;code&gt;dev&lt;/code&gt;, &lt;code&gt;qa&lt;/code&gt;, &lt;code&gt;prod&lt;/code&gt;, …).&lt;/p&gt;

&lt;h2&gt;
  
  
  Local development
&lt;/h2&gt;

&lt;p&gt;The infrastructure repo runs a &lt;strong&gt;mock listener&lt;/strong&gt; on &lt;strong&gt;2772&lt;/strong&gt; so laptops hit the same URL shapes as production. A lightweight &lt;strong&gt;public API on 2773&lt;/strong&gt; stands in for the mobile route during quick tests.&lt;/p&gt;

&lt;p&gt;The same shared library and URL contract apply on a laptop: only the process behind 2772 is a fixture instead of the AppConfig Agent container, host agent, or extension.&lt;/p&gt;

&lt;p&gt;Edit a &lt;strong&gt;local overrides file&lt;/strong&gt; to flip flags and attributes; you do not need to restart services for those fixture updates. Example:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;http://localhost:2772/applications/global-feature-flags/environments/dev/configurations/feature-flags&lt;/code&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Caveats
&lt;/h2&gt;

&lt;p&gt;Migration is incremental: older flag and env patterns still appear on some services. AWS supports multi-variant flags; our deployment tooling focuses on booleans and boolean attributes.&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;public config Lambda&lt;/strong&gt; and backend services both read through 2772, but only the public API merges &lt;code&gt;{ featureFlags, config }&lt;/code&gt; for HTTP clients; the shared library keeps flags and per-service config as separate reads for in-process use.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-agent.html" rel="noopener noreferrer"&gt;What is AWS AppConfig Agent?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-creating-configuration-and-profile-feature-flags.html" rel="noopener noreferrer"&gt;Creating a feature flag configuration profile&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-integration-ec2.html" rel="noopener noreferrer"&gt;AppConfig on EC2&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-integration-lambda-extensions.html" rel="noopener noreferrer"&gt;AppConfig Lambda extensions&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>architecture</category>
      <category>aws</category>
      <category>backend</category>
      <category>infrastructure</category>
    </item>
    <item>
      <title>Bulk email without burning reputation</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Tue, 10 Nov 2026 11:34:35 +0000</pubDate>
      <link>https://dev.to/joungpark/bulk-email-without-burning-reputation-1p1e</link>
      <guid>https://dev.to/joungpark/bulk-email-without-burning-reputation-1p1e</guid>
      <description>&lt;p&gt;Large-scale reminder campaigns needed to reach a large eligible audience without a single-day blast that would spike SES throttling, bounces, and complaints. Reputation here was mostly an operations problem: control &lt;strong&gt;how much mail leaves per day&lt;/strong&gt; and &lt;strong&gt;how fast it hits SES&lt;/strong&gt;, then watch Virtual Deliverability Manager (VDM) while sending.&lt;/p&gt;

&lt;p&gt;The code does not score reputation. It paces volume, targets an average send rate within each batch, deduplicates by send date, and filters eligibility in SQL. Below: the two guardrails, how &lt;code&gt;createBatches&lt;/code&gt; and &lt;code&gt;runBatch&lt;/code&gt; enforce them, and the path through Kafka and the messenger Lambda to the messaging API and SES.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture overview
&lt;/h2&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
  subgraph prep [Mail per day]
    cb[createBatches]
    file1[batch-1.jsonl]
    file2[batch-2.jsonl]
    filen[batch-n.jsonl]
    cb --&amp;gt; file1
    cb --&amp;gt; file2
    cb --&amp;gt; filen
  end

  subgraph sendPath [Rate within a day]
    rb[runBatch]
    dedup[check duplication]
    kfk[Kafka topic]
    lam[Lambda]
    ses[AWS SES]
    rb --&amp;gt; dedup --&amp;gt; kfk --&amp;gt; lam --&amp;gt; ses
  end

  sendPath2[Run on 2nd day]
  sendPathn[Run on n-th day]

  file1 --&amp;gt; rb
  file2 --&amp;gt; sendPath2
  filen --&amp;gt; sendPathn&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;&lt;code&gt;createBatches&lt;/code&gt; can produce many JSONL files; cron runs &lt;strong&gt;&lt;code&gt;runBatch&lt;/code&gt; once per day&lt;/strong&gt;, drains the next pending file, then follows the same send path (check duplication, Kafka, Lambda, SES). Later files run on later cron days—the labels are shorthand for that schedule, not separate pipelines.&lt;/p&gt;

&lt;p&gt;The automation path is &lt;code&gt;npm run createBatches -- --email-id &amp;lt;campaign&amp;gt;&lt;/code&gt; ahead of send windows, then a daily cron &lt;code&gt;npm run runBatch&lt;/code&gt;. The same emit logic also supports a direct &lt;code&gt;emitBulkEmails --input&lt;/code&gt; JSONL workflow without the &lt;code&gt;email_reminder_automation_batches&lt;/code&gt; table.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Reputation guardrail — how much mail per day
&lt;/h2&gt;

&lt;p&gt;A sudden jump from zero to millions of messages in one day is what SES and mailbox providers notice. We spread work across &lt;strong&gt;calendar days&lt;/strong&gt; by splitting recipients into batch files and draining &lt;strong&gt;at most one pending batch per cron run&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Knob&lt;/th&gt;
&lt;th&gt;Role&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;batchSize&lt;/code&gt; in &lt;code&gt;configuration.json&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Max recipients per &lt;code&gt;batch-{n}.jsonl&lt;/code&gt;; one row in &lt;code&gt;email_reminder_automation_batches&lt;/code&gt; per file&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;eligible-user.sql&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Who gets mail; stable &lt;code&gt;ORDER BY&lt;/code&gt; for reproducible splits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;{emailId}/{YYYY-MM-DD}/batch-{n}.jsonl&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;On-disk snapshot for a send date (UTC at create time)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;email_reminder_automation_batches&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;pending&lt;/code&gt; → &lt;code&gt;processing&lt;/code&gt; → &lt;code&gt;completed&lt;/code&gt; or &lt;code&gt;failed&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;createBatches&lt;/strong&gt; loads config and SQL, snapshots eligible users into a MySQL temp table so paging is stable while export runs, then writes JSONL in chunks of &lt;code&gt;batchSize&lt;/code&gt; and inserts a &lt;code&gt;pending&lt;/code&gt; row per file. One run can create several files: for example, 2M users with &lt;code&gt;batchSize: 500000&lt;/code&gt; yields four files and four pending rows—four cron days of work at one batch per day.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;runBatch&lt;/strong&gt; selects the first pending row (&lt;code&gt;ORDER BY send_date_key, batch_number LIMIT 1&lt;/code&gt;), marks it &lt;code&gt;processing&lt;/code&gt;, runs the bulk emit script on that file, then &lt;code&gt;completed&lt;/code&gt; or &lt;code&gt;failed&lt;/code&gt;. Production cron runs daily (&lt;code&gt;0 1 * * *&lt;/code&gt; UTC) with &lt;code&gt;flock&lt;/code&gt; so only one batch drains per day. That is how “500K per day” is enforced: operational schedule, not an in-process daily cap inside the emitter.&lt;/p&gt;

&lt;p&gt;Layout on disk:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{emailId}/                          e.g. monthly-reminder
├── configuration.json
├── eligible-user.sql
└── 2026-10-01/
    ├── batch-1.jsonl
    └── batch-2.jsonl
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Example JSONL line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"…"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user@example.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"reminderData"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"firstName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Alex"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"gameKey"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"fortnite"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  2. Reputation guardrail — how fast mail hits SES within a day
&lt;/h2&gt;

&lt;p&gt;While a batch file is emitting, the goal is to keep &lt;strong&gt;average&lt;/strong&gt; recipient throughput near an approved target and stay under the account SES &lt;code&gt;MaxSendRate&lt;/code&gt;. Bursts into the shared messaging SMTP path cause throttling and noisy delivery patterns.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Knob&lt;/th&gt;
&lt;th&gt;Role&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;maxSendRatePerSecond&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Target &lt;strong&gt;average&lt;/strong&gt; recipients per second; used to compute &lt;code&gt;minMessageDurationMs&lt;/code&gt; on each Kafka payload (not a hard per-second token bucket by itself)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;emailsPerKafkaMessage&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Recipients bundled per Kafka message; &lt;code&gt;runBatch&lt;/code&gt; passes this as emit &lt;code&gt;batchSize&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;minMessageDurationMs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Producer-set floor on wall time per message; the Lambda waits any &lt;strong&gt;remaining&lt;/strong&gt; time after messaging sends complete&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For each Kafka record, the producer computes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;minMessageDurationMs = ceil(recipientCount * partitionKeyShards * 1000 / maxSendRatePerSecond)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Tune &lt;strong&gt;&lt;code&gt;maxSendRatePerSecond&lt;/code&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;emailsPerKafkaMessage&lt;/code&gt;&lt;/strong&gt; for reputation and quota. &lt;code&gt;partitionKeyShards&lt;/code&gt; appears in &lt;code&gt;configuration.json&lt;/code&gt; so the duration math matches deployed consumer parallelism.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Consumer pacing:&lt;/strong&gt; The &lt;code&gt;on-email-bulk-send&lt;/code&gt; Lambda consumes the &lt;code&gt;email-bulk-send&lt;/code&gt; topic with MSK &lt;code&gt;batchSize: 1&lt;/code&gt; (one Kafka record per invocation). It calls the messaging API to send, then applies a minimum message duration wait in a &lt;code&gt;finally&lt;/code&gt; block. If SES still complains, operator docs point at lowering &lt;code&gt;maxSendRatePerSecond&lt;/code&gt;, &lt;code&gt;emailsPerKafkaMessage&lt;/code&gt;, or the messenger env var for per-recipient concurrency inside one invocation—not Kafka key count.&lt;/p&gt;

&lt;p&gt;Example with campaign-style settings (&lt;code&gt;maxSendRatePerSecond: 20&lt;/code&gt;, &lt;code&gt;emailsPerKafkaMessage: 10&lt;/code&gt;, &lt;code&gt;partitionKeyShards: 2&lt;/code&gt; in config only for the formula): &lt;code&gt;ceil(10 × 2 × 1000 / 20) = 1000&lt;/code&gt; ms minimum per message, i.e. roughly 20 recipients per second on average across active consumers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Resume and dedup:&lt;/strong&gt; Send state keys &lt;code&gt;{ sendDateKey, userId }&lt;/code&gt; (the recipient &lt;code&gt;id&lt;/code&gt; from the batch file). The producer skips users already sent for that date; re-run the same &lt;code&gt;sendDateKey&lt;/code&gt; after an interrupt. Do not change &lt;code&gt;sendDateKey&lt;/code&gt; when resuming or users can receive the same reminder twice.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Monitoring:&lt;/strong&gt; Script pauses for VDM review during large direct sends; watch SES configuration-set metrics, bounces, and complaints. The system paces and observes—it does not auto-adjust reputation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pipeline walkthrough
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Create batch
&lt;/h3&gt;

&lt;p&gt;Inputs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;configuration.json&lt;/code&gt; — template, query path, &lt;code&gt;batchSize&lt;/code&gt;, rate fields (see configuration reference).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;eligible-user.sql&lt;/code&gt; — who is eligible. Supports &lt;code&gt;${emailId}&lt;/code&gt; and &lt;code&gt;${sendDateKey}&lt;/code&gt; placeholders.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Outputs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;JSONL under &lt;code&gt;{emailId}/{sendDateKey}/batch-{n}.jsonl&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;One &lt;code&gt;pending&lt;/code&gt; insert per file in &lt;code&gt;email_reminder_automation_batches&lt;/code&gt; (&lt;code&gt;email_id&lt;/code&gt;, &lt;code&gt;send_date_key&lt;/code&gt;, &lt;code&gt;batch_number&lt;/code&gt;, &lt;code&gt;status&lt;/code&gt;, &lt;code&gt;when_completed&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Run batch and publish
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Fetch first &lt;code&gt;pending&lt;/code&gt; batch row.&lt;/li&gt;
&lt;li&gt;Resolve &lt;code&gt;batch-{batchNumber}.jsonl&lt;/code&gt;; fail the row if the file is missing.&lt;/li&gt;
&lt;li&gt;The emit script reads the file, checks duplication against send state, publishes to Kafka.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Headers include &lt;code&gt;message-type&lt;/code&gt; (template id, e.g. &lt;code&gt;monthly-reminder&lt;/code&gt;) and &lt;code&gt;message-id&lt;/code&gt; (&lt;code&gt;{sendDateKey}-batch-{batchStartIndex}&lt;/code&gt;). Body includes &lt;code&gt;emailTemplate&lt;/code&gt;, &lt;code&gt;recipients&lt;/code&gt;, and &lt;code&gt;minMessageDurationMs&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"emailTemplate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"monthly-reminder"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"minMessageDurationMs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"recipients"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"userId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"…"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user@example.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"reminderData"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"firstName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Alex"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"gameKey"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"fortnite"&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Consume and send
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;on-email-bulk-send&lt;/code&gt; dispatches by &lt;code&gt;message-type&lt;/code&gt;, validates the payload, calls messaging, and delivers through SES (configuration set and message tags on the SMTP path). Pacing is described in section 2.&lt;/p&gt;

&lt;h2&gt;
  
  
  Configuration reference
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;Reputation-facing?&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;emailTemplate&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;Messaging template name&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;query&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Hygiene&lt;/td&gt;
&lt;td&gt;Path to &lt;code&gt;eligible-user.sql&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;batchSize&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Daily volume&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Recipients per JSONL file&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;maxSendRatePerSecond&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Intraday rate&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Average target; drives &lt;code&gt;minMessageDurationMs&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;emailsPerKafkaMessage&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Intraday rate&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Bundle size per Kafka message&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;partitionKeyShards&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Fixed in JSON&lt;/td&gt;
&lt;td&gt;Used in duration formula only&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Campaign automation&lt;/strong&gt; (e.g. monthly reminder): large &lt;code&gt;batchSize&lt;/code&gt; (500000), &lt;code&gt;maxSendRatePerSecond: 20&lt;/code&gt;, modest &lt;code&gt;emailsPerKafkaMessage&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"emailTemplate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"monthly-reminder"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"query"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"./eligible-user.sql"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"batchSize"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;500000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"partitionKeyShards"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"emailsPerKafkaMessage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"maxSendRatePerSecond"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Direct emit / throughput test&lt;/strong&gt; (operator README): smaller per-message bundles and higher target rate, e.g. &lt;code&gt;--batch-size 14&lt;/code&gt; and &lt;code&gt;--max-send-rate-per-second 70&lt;/code&gt; against an exported JSONL—same Kafka pacing math, without the batch table queue.&lt;/p&gt;

&lt;h2&gt;
  
  
  Operations and observability
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concern&lt;/th&gt;
&lt;th&gt;Where to look&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Batch queue&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;email_reminder_automation_batches.status&lt;/code&gt;, &lt;code&gt;when_completed&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Producer progress&lt;/td&gt;
&lt;td&gt;Emit script logs (processed / successful / failed / skipped)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Consumer errors&lt;/td&gt;
&lt;td&gt;CloudWatch for &lt;code&gt;on-email-bulk-send-&amp;lt;env&amp;gt;&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SES health&lt;/td&gt;
&lt;td&gt;VDM, configuration set metrics, suppression list&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Send state&lt;/td&gt;
&lt;td&gt;By &lt;code&gt;sendDateKey&lt;/code&gt; and user id&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stuck sends&lt;/td&gt;
&lt;td&gt;Pending batches with missing JSONL; SES throttling vs &lt;code&gt;maxSendRatePerSecond&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;Reputation for this system boils down to two operational guardrails: &lt;strong&gt;emails per day&lt;/strong&gt; (&lt;code&gt;batchSize&lt;/code&gt; + daily &lt;code&gt;runBatch&lt;/code&gt;) and &lt;strong&gt;send rate within a day&lt;/strong&gt; (&lt;code&gt;maxSendRatePerSecond&lt;/code&gt;, &lt;code&gt;emailsPerKafkaMessage&lt;/code&gt;, Lambda &lt;code&gt;minMessageDurationMs&lt;/code&gt;, plus SES quota and VDM). Everything else—Kafka, send-state dedup, SQL eligibility—implements those limits so a large reminder campaign does not become a single spike in the eyes of SES or recipients.&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>aws</category>
      <category>backend</category>
      <category>devops</category>
    </item>
    <item>
      <title>Microservice Patterns: Rules, Boundaries, and Commands</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Thu, 05 Nov 2026 07:14:28 +0000</pubDate>
      <link>https://dev.to/joungpark/microservice-patterns-rules-boundaries-and-commands-ih4</link>
      <guid>https://dev.to/joungpark/microservice-patterns-rules-boundaries-and-commands-ih4</guid>
      <description>&lt;p&gt;We run a fleet of Nest services on ECS and PM2, with Kafka for domain integration and Firebase at the API gateway. That stack only works if we agree on a few defaults: who owns data, when to use events versus HTTP, how contracts are defined, and where auth stops. This post is our &lt;strong&gt;engineering playbook&lt;/strong&gt; for those defaults—not a claim that every service already matches them. Some rules describe &lt;strong&gt;where we want to be&lt;/strong&gt; (for example strict one-database-per-service everywhere); others we apply today with known gaps (consumer backoff patterns still being defined).&lt;/p&gt;

&lt;p&gt;The post has three parts: &lt;strong&gt;ten foundation principles&lt;/strong&gt;, &lt;strong&gt;patterns for crossing domain boundaries&lt;/strong&gt; (with Order and User as the running example), and a &lt;strong&gt;payment debit/credit case study&lt;/strong&gt; on why implicit event listeners are painful and what explicit commands buy us.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Each microservice owns its data
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rule:&lt;/strong&gt; one domain, one database. No service reads or writes another service’s store.&lt;/p&gt;

&lt;p&gt;If you need someone else’s data, use a &lt;strong&gt;versioned REST call&lt;/strong&gt; or subscribe to &lt;strong&gt;Kafka events&lt;/strong&gt; and keep a local copy. The only exception we allow is &lt;strong&gt;temporary shared access during a migration&lt;/strong&gt; between services—and even then we treat it as debt with an exit date.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Use Kafka for state changes, not queries
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rule:&lt;/strong&gt; publish when domain state &lt;strong&gt;changes&lt;/strong&gt;—&lt;code&gt;user-created&lt;/code&gt;, &lt;code&gt;order-placed&lt;/code&gt;, &lt;code&gt;payment-success&lt;/code&gt;—not when another service needs a &lt;strong&gt;read&lt;/strong&gt; or a &lt;strong&gt;real-time decision&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Kafka is an integration log, not a query engine. Using it to “fetch” user status at decision time turns consumers into brittle RPC clients and couples lag to business logic.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sagas&lt;/strong&gt; are the intentional exception in shape, not in spirit: a long-running flow may touch several domains and emit &lt;strong&gt;multiple state-change events&lt;/strong&gt; along the way. That is still “something happened,” not “give me the current row.”&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Expose a versioned REST API
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rule:&lt;/strong&gt; public and partner-facing HTTP APIs live under &lt;strong&gt;&lt;code&gt;/v1/...&lt;/code&gt;&lt;/strong&gt; (and future &lt;code&gt;/v2/&lt;/code&gt; when we break compatibility on purpose).&lt;/p&gt;

&lt;p&gt;Services at the edge expose REST. &lt;strong&gt;Internal&lt;/strong&gt; traffic can be REST or Kafka events, depending on freshness and coupling—we pick deliberately in § Crossing domain boundaries. Versioning at the path keeps gateway routing and client expectations explicit.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Define clear API contracts
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rule:&lt;/strong&gt; every service ships an &lt;strong&gt;OpenAPI (Swagger)&lt;/strong&gt; spec that matches what it actually runs.&lt;/p&gt;

&lt;p&gt;Payloads stay &lt;strong&gt;flat and consistent&lt;/strong&gt; across services: prefer clear field names and shallow objects over deep nesting that mirrors every join in the database. The spec is the handshake for humans, codegen, and reviews—not documentation we write after the fact.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Validate contracts
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rule:&lt;/strong&gt; validate &lt;strong&gt;HTTP bodies and event payloads&lt;/strong&gt; at the boundary with shared schemas—we standardize on &lt;strong&gt;zod&lt;/strong&gt; in Node services.&lt;/p&gt;

&lt;p&gt;Parsing once at the edge of the handler (or consumer) gives us &lt;strong&gt;fail fast&lt;/strong&gt; behavior (see §6) and keeps invalid messages out of domain code. OpenAPI describes the contract; zod enforces it at runtime. Consumer-driven contract testing (for example Pact) is on our roadmap as a complement, not a replacement for schema validation at the service.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Fail fast, retry smart
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rule:&lt;/strong&gt; reject bad input and unprocessable state &lt;strong&gt;early&lt;/strong&gt; with a clear error. &lt;strong&gt;Retry&lt;/strong&gt; only for &lt;strong&gt;transient&lt;/strong&gt; failures—network blips, broker hiccups, throttling—not for validation or business rule violations.&lt;/p&gt;

&lt;p&gt;Kafka consumers in our stack already retry poison-adjacent cases with operational playbooks; we still need &lt;strong&gt;shared backoff and dead-letter patterns&lt;/strong&gt; documented and implemented consistently. That work is deliberate follow-up: the principle is decided; the library conventions are not finished yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Trace everything
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rule:&lt;/strong&gt; one logical operation should be followable across HTTP, Kafka, Lambdas, and long-running workers.&lt;/p&gt;

&lt;p&gt;We are standardizing on &lt;strong&gt;&lt;a href="//./2026-10-29-open-telemetry.md"&gt;OpenTelemetry&lt;/a&gt;&lt;/strong&gt; and W3C &lt;code&gt;traceparent&lt;/code&gt; propagation. That &lt;strong&gt;supersedes&lt;/strong&gt; ad-hoc &lt;code&gt;x-correlation-id&lt;/code&gt; headers for new work: any hop that understands OTEL continues the same trace. Until every path is instrumented, we may still see correlation IDs in older services; new code should inject and extract trace context on produce/consume and outbox commit.&lt;/p&gt;

&lt;p&gt;Application logs go to &lt;strong&gt;Elastic Cloud&lt;/strong&gt; as structured JSON. Besides trace and span IDs, we include &lt;strong&gt;Kafka message identifiers&lt;/strong&gt; (topic, partition, offset, and our internal message id when present) so on-call can jump from a log line to a broker record without guessing.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Keep logic close to the domain
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rule:&lt;/strong&gt; services are not thin CRUD wrappers over a table unless the domain truly is that simple.&lt;/p&gt;

&lt;p&gt;Where it pays off we use &lt;strong&gt;Domain-Driven Design&lt;/strong&gt; building blocks—aggregates, value objects, explicit invariants—so business rules live in the domain layer, not scattered across controllers and ORM callbacks. The goal is code that reads like the business, not like a generic data access layer.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Avoid shared libraries for business logic
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rule:&lt;/strong&gt; share &lt;strong&gt;infrastructure&lt;/strong&gt; helpers (logging wrappers, OTEL bootstrap, HTTP client defaults); do &lt;strong&gt;not&lt;/strong&gt; share &lt;strong&gt;domain&lt;/strong&gt; rules across services.&lt;/p&gt;

&lt;p&gt;A shared “UserValidator” npm package feels convenient until two services need slightly different eligibility rules and you ship a breaking version to everyone. Duplication of domain logic is cheaper than hidden coupling. Copy the pattern, not the rule, when domains diverge.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. Auth at the edge, AuthZ in the service
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rule:&lt;/strong&gt; &lt;strong&gt;authentication&lt;/strong&gt; happens at the API gateway or entry point (&lt;strong&gt;Firebase&lt;/strong&gt; verifies the token before traffic hits Nest). &lt;strong&gt;Authorization&lt;/strong&gt;—roles, ownership, step-up, country eligibility—runs &lt;strong&gt;inside each service&lt;/strong&gt; that owns the resource.&lt;/p&gt;

&lt;p&gt;That split matches what I documented for Second-Memory in &lt;a href="//./2026-10-10-why-i-chose-third-party-firebase-authentication.md"&gt;Why I Chose Third-Party (Firebase) Authentication&lt;/a&gt;: Firebase proves identity; our services decide what that identity may do on &lt;em&gt;this&lt;/em&gt; aggregate. No service should trust another service’s “already authorized” flag without its own check when the resource is theirs.&lt;/p&gt;




&lt;h2&gt;
  
  
  Crossing domain boundaries
&lt;/h2&gt;

&lt;p&gt;Business rules often need facts from another bounded context. Example: &lt;strong&gt;Order&lt;/strong&gt; must confirm &lt;strong&gt;User&lt;/strong&gt; is in good standing and in an allowed country before accepting a new order. User owns that data; Order owns the placement rule. How do we bridge the gap?&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TD
  needData[Need foreign domain fact]
  needData --&amp;gt; stalenessOK{Minor staleness OK?}
  stalenessOK --&amp;gt;|yes| snapshot[Kafka snapshot in local DB]
  stalenessOK --&amp;gt;|no| rest[REST query at runtime]
  snapshot --&amp;gt; hybrid{Cache miss or suspect?}
  hybrid --&amp;gt;|yes| rest
  hybrid --&amp;gt;|no| useLocal[Use local copy]&lt;/code&gt;&lt;/pre&gt;



&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Pattern&lt;/th&gt;
&lt;th&gt;Freshness&lt;/th&gt;
&lt;th&gt;Coupling&lt;/th&gt;
&lt;th&gt;When to use&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;A — Cached snapshot (Kafka)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Eventually consistent (often seconds)&lt;/td&gt;
&lt;td&gt;Low; no runtime dependency on User API&lt;/td&gt;
&lt;td&gt;Performance matters; minor staleness is acceptable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;B — REST query&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Up to date at call time&lt;/td&gt;
&lt;td&gt;Higher latency; temporal coupling to User&lt;/td&gt;
&lt;td&gt;Real-time accuracy is essential&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;C — Hybrid&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Usually local; fresh on miss&lt;/td&gt;
&lt;td&gt;Medium; fallback path must be tested&lt;/td&gt;
&lt;td&gt;Default for many flows: fast path + safety net&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Option A — Cached snapshot (preferred default)
&lt;/h3&gt;

&lt;p&gt;At runtime, User owns the canonical row; Order keeps a &lt;strong&gt;local user-data snapshot&lt;/strong&gt;—only the fields Order needs for placement (for example country, state, status)—fed by Kafka so the placement path never calls User’s HTTP API. Firebase auth runs at the gateway on both the profile write and the order request (see §10).&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TD
    subgraph TOP [API Gateway Layer]
        GW[API Gateway]
    end

    subgraph LEFT [User Domain]
        US[User Service]
        DB[(User DB)]
        US --&amp;gt; DB
    end

    subgraph CENTER [Message Broker]
        K["Kafka topic user.updated"]
    end

    subgraph RIGHT [Order Domain]
        OS[Order Service]
        UDS[(User data snapshot)]
        OS --&amp;gt; UDS
    end

    GW --&amp;gt;|"1. updateUserProfile"| US
    US --&amp;gt;|"2. emit message"| K
    K -.-&amp;gt;|"3. async update"| OS
    GW --&amp;gt;|"4. createOrder"| OS
    OS --&amp;gt;|"5. retrieve data"| UDS&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;The numbered edges in the diagram:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;updateUserProfile&lt;/strong&gt; — The client calls User through the gateway to change profile, status, address, or anything else that affects order eligibility.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;emit message&lt;/strong&gt; — User writes to &lt;strong&gt;User DB&lt;/strong&gt; and publishes a &lt;strong&gt;state-change&lt;/strong&gt; event (not a query).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;async update&lt;/strong&gt; — Order’s consumer upserts the &lt;strong&gt;user-data snapshot&lt;/strong&gt; (subset of User fields Order cares about); this path is eventually consistent.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;createOrder&lt;/strong&gt; — A separate order request hits Order only. Order does &lt;strong&gt;not&lt;/strong&gt; call User synchronously on this path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;retrieve data&lt;/strong&gt; — Eligibility checks read the &lt;strong&gt;snapshot&lt;/strong&gt;, not User Service.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;User publishes those changes—for example &lt;code&gt;user-updated&lt;/code&gt;, &lt;code&gt;user-status-changed&lt;/code&gt;, or &lt;code&gt;user-address-changed&lt;/code&gt;—often on a single topic such as &lt;code&gt;user.updated&lt;/code&gt;. Order subscribes and persists only what it needs in its own database (orders plus the local user snapshot):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// OrderService Kafka handler&lt;/span&gt;
&lt;span class="p"&gt;@&lt;/span&gt;&lt;span class="nd"&gt;KafkaListener&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;user.updated&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="nf"&gt;handleUserUpdate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;UserUpdatedEvent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;userRepo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;saveOrUpdate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;country&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;country&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Validation for new orders uses that &lt;strong&gt;snapshot&lt;/strong&gt;, not a full User record. Fast and decoupled; accept that a brand-new status change may lag briefly behind User. Order must not read User DB or call User HTTP on the placement path under Option A alone—if the snapshot row is missing or too stale, use &lt;strong&gt;Option C (hybrid)&lt;/strong&gt; below.&lt;/p&gt;

&lt;h3&gt;
  
  
  Option B — REST API call (query pattern)
&lt;/h3&gt;

&lt;p&gt;When the decision must reflect User &lt;strong&gt;right now&lt;/strong&gt;, call User at runtime:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// OrderService: validateUserForOrder&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;user&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;httpService&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`https://user-service.internal/v1/users/&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toPromise&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;good_standing&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;country&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;AU&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ForbiddenException&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;User is not eligible to place an order&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Always current; you pay in latency and in &lt;strong&gt;availability coupling&lt;/strong&gt;—if User is down, Order cannot complete the rule check unless you have a fallback.&lt;/p&gt;

&lt;h3&gt;
  
  
  Option C — Hybrid
&lt;/h3&gt;

&lt;p&gt;Use the snapshot by default; on cache miss or suspicious staleness, call User and optionally refresh the local row (still persisting only the fields Order needs).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;user&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;localUserSnapshotRepo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;user&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;apiUser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;userApiClient&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getUser&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;localUserSnapshotRepo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;apiUser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;country&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;apiUser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;country&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;apiUser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;apiUser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;apiUser&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;user&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We use this when the happy path should stay off the critical path of User’s API, but we cannot afford silent wrong decisions on empty cache.&lt;/p&gt;




&lt;h2&gt;
  
  
  When events are the wrong shape — payment debit and credit
&lt;/h2&gt;

&lt;p&gt;Sometimes integration should be modeled as a &lt;strong&gt;&lt;a href="https://event-driven.io/en/whats_the_difference_between_event_and_command/" rel="noopener noreferrer"&gt;command&lt;/a&gt;&lt;/strong&gt;—an intention directed at a specific handler—not as a domain event every downstream service must interpret. Payment is our clearest example.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Payment&lt;/strong&gt; debits and credits user balances. Many originating services need those operations. We &lt;strong&gt;used to&lt;/strong&gt; model it like this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Originating services emit &lt;strong&gt;their&lt;/strong&gt; domain events that &lt;em&gt;implicitly&lt;/em&gt; mean “take money” or “refund money.”&lt;/li&gt;
&lt;li&gt;Payment registers &lt;strong&gt;a listener per event type&lt;/strong&gt; and encodes knowledge of each scenario: which event implies debit, how to derive transaction source and request id.&lt;/li&gt;
&lt;li&gt;Payment performs the transaction and emits success or failure events keyed by transaction source and request id.&lt;/li&gt;
&lt;li&gt;Originating services must &lt;strong&gt;implicitly&lt;/strong&gt; know that those fields map back to &lt;em&gt;their&lt;/em&gt; workflow.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That design spreads payment semantics into every producer’s event vocabulary. Payment becomes a central interpreter of foreign languages instead of a service with a narrow, explicit API.&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;sequenceDiagram
  participant Orig as OriginatingService
  participant Kafka
  participant Pay as PaymentService

  Orig-&amp;gt;&amp;gt;Kafka: domain_event_A
  Orig-&amp;gt;&amp;gt;Kafka: domain_event_B
  Kafka-&amp;gt;&amp;gt;Pay: listener maps A to debit
  Kafka-&amp;gt;&amp;gt;Pay: listener maps B to credit
  Pay-&amp;gt;&amp;gt;Kafka: transaction_success
  Kafka-&amp;gt;&amp;gt;Orig: must decode source plus requestId&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  Alternative — synchronous API commands
&lt;/h3&gt;

&lt;p&gt;Model debit and credit as &lt;strong&gt;well-known HTTP commands&lt;/strong&gt; with a stable schema. The caller supplies &lt;strong&gt;transaction source&lt;/strong&gt; and &lt;strong&gt;transaction request id&lt;/strong&gt;; Payment does not care which feature invoked it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pros:&lt;/strong&gt; Payment stays ignorant of callers; immediate success/failure in the response.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cons:&lt;/strong&gt; &lt;strong&gt;Temporal coupling&lt;/strong&gt;—if Payment is down, the caller must retry until it recovers. In an event-first codebase that often means: emit a local event, handle it in a listener, and loop retries from there—more moving parts than a direct call with idempotency keys.&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;sequenceDiagram
  participant Orig as OriginatingService
  participant Pay as PaymentService

  Orig-&amp;gt;&amp;gt;Pay: POST /v1/debit
  Pay--&amp;gt;&amp;gt;Orig: 200 or 4xx&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  Alternative — asynchronous commands
&lt;/h3&gt;

&lt;p&gt;Publish commands to a &lt;strong&gt;well-known Kafka topic&lt;/strong&gt; Payment owns. Callers produce &lt;code&gt;DebitPayment&lt;/code&gt; / &lt;code&gt;CreditPayment&lt;/code&gt; messages; Payment consumes and executes. No synchronous response—outcomes appear as &lt;strong&gt;domain events&lt;/strong&gt; Payment already owns, for example &lt;code&gt;user-transaction-succeeded&lt;/code&gt; / &lt;code&gt;user-transaction-failed&lt;/code&gt;, which callers subscribe to if they need completion.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pros:&lt;/strong&gt; Same decoupling as explicit API (Payment does not know the caller’s event names); &lt;strong&gt;no&lt;/strong&gt; temporal coupling on the produce path; callers choose their own correlation ids.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cons:&lt;/strong&gt; No immediate feedback; originating workflows must be &lt;strong&gt;saga-shaped&lt;/strong&gt;—listen for outcome events or poll a read model. That aligns with how we already name work on the Event Modeling board (&lt;a href="//./2026-10-26-from-event-modeling-to-a-kafka-saga/index.md"&gt;From Event Modeling to a Kafka Saga&lt;/a&gt;): &lt;code&gt;Payment.debitPayment&lt;/code&gt; as a command with &lt;code&gt;payment_success&lt;/code&gt; / &lt;code&gt;payment_failure&lt;/code&gt; as outcomes, rather than Payment guessing from unrelated domain events.&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;sequenceDiagram
  participant Orig as OriginatingService
  participant Kafka
  participant Pay as PaymentService

  Orig-&amp;gt;&amp;gt;Kafka: DebitPayment command
  Kafka-&amp;gt;&amp;gt;Pay: consume command
  Pay-&amp;gt;&amp;gt;Kafka: user_transaction_succeeded
  Kafka-&amp;gt;&amp;gt;Orig: resume workflow&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;We are not flipping every flow overnight. The direction is clear: &lt;strong&gt;local snapshots and explicit commands&lt;/strong&gt; beat &lt;strong&gt;REST spaghetti&lt;/strong&gt; and &lt;strong&gt;payment listeners that speak every team’s event dialect&lt;/strong&gt;. Events stay for &lt;strong&gt;things that happened&lt;/strong&gt;; commands carry &lt;strong&gt;work we want Payment to do&lt;/strong&gt;; REST fills the gaps when freshness beats cache.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Prefer a denormalized read model fed by Kafka when staleness is acceptable; prefer explicit payment commands over payment inferring debits from someone else’s domain events; use REST when the decision must be correct to the second—and document which choice you made.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

</description>
      <category>architecture</category>
      <category>backend</category>
      <category>microservices</category>
      <category>systemdesign</category>
    </item>
    <item>
      <title>OpenTelemetry in Production: When Auto-Instrumentation Stops (MSK, Outbox, Cron)</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Mon, 02 Nov 2026 04:25:49 +0000</pubDate>
      <link>https://dev.to/joungpark/opentelemetry-in-production-when-auto-instrumentation-stops-msk-outbox-cron-3c8f</link>
      <guid>https://dev.to/joungpark/opentelemetry-in-production-when-auto-instrumentation-stops-msk-outbox-cron-3c8f</guid>
      <description>&lt;p&gt;We already had &lt;strong&gt;distributed logging&lt;/strong&gt;—enough to grep across services when something broke. What we did not have was a &lt;strong&gt;single picture of a request&lt;/strong&gt; as it crossed HTTP, Kafka, Lambdas, and long-running Node workers on PM2. &lt;strong&gt;Tracing&lt;/strong&gt; was the next pillar; we chose &lt;strong&gt;&lt;a href="https://opentelemetry.io/" rel="noopener noreferrer"&gt;OpenTelemetry&lt;/a&gt;&lt;/strong&gt; so instrumentation stays &lt;strong&gt;vendor-neutral&lt;/strong&gt; while we export spans to &lt;strong&gt;Elastic APM&lt;/strong&gt; today and can change backends later without rewriting apps.&lt;/p&gt;

&lt;p&gt;This post follows how we rolled it out: &lt;strong&gt;Collector topology&lt;/strong&gt;, &lt;strong&gt;auto-instrumentation on the happy path&lt;/strong&gt;, then &lt;strong&gt;where auto stops&lt;/strong&gt; and &lt;strong&gt;how we fix it&lt;/strong&gt; (&lt;code&gt;withTraceSpan&lt;/code&gt; at entry points, &lt;strong&gt;&lt;code&gt;propagation.inject&lt;/code&gt;&lt;/strong&gt; on outbox commit)—not a full CDK or Dockerfile runbook.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Observability and OpenTelemetry
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Observability&lt;/strong&gt; means you can answer &lt;em&gt;what is the system doing, and why?&lt;/em&gt; Three signals work together:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Pillar&lt;/th&gt;
&lt;th&gt;Question it answers&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Logging&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;What happened, when, where, and with what context?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Tracing&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;How did one logical operation flow through services and queues?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Metrics&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;How is the system behaving over time—and when should we alert?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;We added &lt;strong&gt;traces first&lt;/strong&gt;; logs stayed as-is. Metrics and alert routing (Slack, on-call) are next on the same OTLP path.&lt;/p&gt;

&lt;p&gt;Proprietary APM agents work until they do not. OpenTelemetry gives &lt;strong&gt;shared APIs and OTLP export&lt;/strong&gt; so the same spans can feed Elastic, Jaeger, or Prometheus/Grafana. &lt;strong&gt;TL;DR:&lt;/strong&gt; instrument once at the app; let a &lt;strong&gt;Collector&lt;/strong&gt; handle batching, secrets, and vendor-specific export.&lt;/p&gt;

&lt;h3&gt;
  
  
  Traces, spans, context, and W3C
&lt;/h3&gt;

&lt;p&gt;A &lt;strong&gt;trace&lt;/strong&gt; is the full story of one logical operation (for example “place tournament entry”) across every service it touches. A &lt;strong&gt;span&lt;/strong&gt; is one unit of work—HTTP handler, DB query, Kafka consume.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt; ties spans together. Cross-service work uses a &lt;strong&gt;carrier&lt;/strong&gt; (HTTP headers, Kafka/AMQ message headers, outbox JSON stored in MySQL/Mongo):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Extract&lt;/strong&gt; on inbound: read the carrier into the active OpenTelemetry context before starting spans.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Inject&lt;/strong&gt; on outbound: write the current span’s identifiers into the carrier.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We use &lt;strong&gt;&lt;a href="https://www.w3.org/TR/trace-context/" rel="noopener noreferrer"&gt;W3C Trace Context&lt;/a&gt;&lt;/strong&gt; (&lt;code&gt;OTEL_PROPAGATORS=tracecontext&lt;/code&gt;). The &lt;code&gt;traceparent&lt;/code&gt; header encodes &lt;code&gt;version-trace_id-parent_span_id-trace_flags&lt;/code&gt; (for example &lt;code&gt;00-aed57677…-2f208df4…-00&lt;/code&gt;). That is the same problem space as &lt;strong&gt;correlation IDs&lt;/strong&gt;, but standardized so any OTEL-aware hop can continue the trace.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Span names&lt;/strong&gt; should name a &lt;em&gt;class&lt;/em&gt; of work, not one instance—&lt;code&gt;get_user&lt;/code&gt; not &lt;code&gt;get_user/314159&lt;/code&gt; (&lt;a href="https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#span" rel="noopener noreferrer"&gt;spec guidance&lt;/a&gt;).&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;sequenceDiagram
  participant Client
  participant Nest as Nest_HTTP_ECS
  participant Kafka
  participant Lambda as MSK_Lambda

  Client-&amp;gt;&amp;gt;Nest: HTTP plus traceparent
  Nest-&amp;gt;&amp;gt;Nest: SERVER span auto
  Nest-&amp;gt;&amp;gt;Kafka: outbox or produce with traceparent
  Kafka-&amp;gt;&amp;gt;Lambda: MSK event
  Lambda-&amp;gt;&amp;gt;Lambda: CONSUMER span manual entry&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  What the OpenTelemetry Collector does
&lt;/h3&gt;

&lt;p&gt;The &lt;strong&gt;SDK in your app&lt;/strong&gt; creates spans, attaches attributes, batches them, and sends &lt;strong&gt;OTLP&lt;/strong&gt; (gRPC or HTTP) to an endpoint—usually on localhost.&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;&lt;a href="https://opentelemetry.io/docs/collector/" rel="noopener noreferrer"&gt;OpenTelemetry Collector&lt;/a&gt;&lt;/strong&gt; is a separate process that:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;Role&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Receivers&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Accept OTLP (and other formats) from apps&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Processors&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Batch, sample, filter, enrich attributes, cap memory&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Exporters&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Forward to Elastic APM, logging, another OTLP endpoint, etc.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Why not export straight from the app to Elastic?&lt;/strong&gt; Collector config holds &lt;strong&gt;API keys and endpoints&lt;/strong&gt;; you can change backends or sampling without redeploying every service; batching and retries reduce load on the app and on APM. Symfony, Nest on ECS, PM2 workers, and Lambda can all speak the same OTLP dialect to a Collector (&lt;a href="https://www.elastic.co/guide/en/apm/guide/current/open-telemetry.html" rel="noopener noreferrer"&gt;Elastic + OTEL&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Apps still set &lt;strong&gt;&lt;code&gt;OTEL_SERVICE_NAME&lt;/code&gt;&lt;/strong&gt;, &lt;strong&gt;&lt;code&gt;OTEL_PROPAGATORS=tracecontext&lt;/code&gt;&lt;/strong&gt;, and &lt;strong&gt;&lt;code&gt;OTEL_RESOURCE_ATTRIBUTES&lt;/code&gt;&lt;/strong&gt; (including &lt;code&gt;deployment.environment.name=dev|qa|prod&lt;/code&gt; so environments do not merge in APM).&lt;/p&gt;

&lt;h3&gt;
  
  
  Sidecar, daemon, and Lambda layer (same Collector, different placement)
&lt;/h3&gt;

&lt;p&gt;These are &lt;strong&gt;deployment patterns&lt;/strong&gt;, not different products:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Placement&lt;/th&gt;
&lt;th&gt;Where we use it&lt;/th&gt;
&lt;th&gt;App sends OTLP to&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Sidecar&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Nest HTTP on ECS&lt;/strong&gt; — second container in the &lt;strong&gt;same task&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;http://127.0.0.1:4317&lt;/code&gt; (or 4318) on the task network&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Daemon / agent&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;PM2 on EC2&lt;/strong&gt; — Kafka listeners, AMQ watchers, cron, legacy routers&lt;/td&gt;
&lt;td&gt;Host-local Collector on &lt;code&gt;127.0.0.1&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Layer&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Lambda&lt;/strong&gt; — Collector bundled in the OTEL &lt;strong&gt;layer&lt;/strong&gt; (not a separate sidecar container)&lt;/td&gt;
&lt;td&gt;Localhost inside the execution environment; config often from S3&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart LR
  AppContainer["Nest ECS task app"]
  SidecarCollector["Collector sidecar"]
  ElasticAPM["Elastic APM"]
  AppContainer --&amp;gt;|OTLP| SidecarCollector
  SidecarCollector --&amp;gt;|OTLP plus auth| ElasticAPM&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;Until first-class OTEL logging is everywhere, we &lt;strong&gt;enrich application logs&lt;/strong&gt; with trace and span IDs from the active span (Winston instrumentation with log shipping disabled worked for us).&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Auto-instrumentation: the happy path
&lt;/h2&gt;

&lt;p&gt;We enabled &lt;strong&gt;&lt;a href="https://www.npmjs.com/package/@opentelemetry/auto-instrumentations-node" rel="noopener noreferrer"&gt;auto-instrumentations&lt;/a&gt;&lt;/strong&gt; on Node via &lt;code&gt;NODE_OPTIONS&lt;/code&gt; (&lt;code&gt;--require dotenv/config --require ./util/otel/register.js&lt;/code&gt;) and PHP via the &lt;strong&gt;OpenTelemetry extension&lt;/strong&gt; plus Composer auto packages on Symfony (HTTP, PDO, Guzzle). That gave quick wins: HTTP &lt;strong&gt;SERVER&lt;/strong&gt; spans, many DB and outbound HTTP &lt;strong&gt;CLIENT&lt;/strong&gt; spans, and sometimes &lt;strong&gt;KafkaJS&lt;/strong&gt; produce/consume when the client runs &lt;strong&gt;in the same process&lt;/strong&gt; with an active context.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stack snapshot&lt;/strong&gt; — same SDK ideas, different &lt;strong&gt;process role&lt;/strong&gt; and &lt;strong&gt;Collector placement&lt;/strong&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Runtime&lt;/th&gt;
&lt;th&gt;Where it runs&lt;/th&gt;
&lt;th&gt;Collector&lt;/th&gt;
&lt;th&gt;Auto covers (typical)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Symfony&lt;/td&gt;
&lt;td&gt;EC2 / Docker&lt;/td&gt;
&lt;td&gt;Agent or container on host&lt;/td&gt;
&lt;td&gt;HTTP, SQL, Guzzle&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nest &lt;strong&gt;HTTP API&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;ECS&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Sidecar&lt;/strong&gt; in task&lt;/td&gt;
&lt;td&gt;HTTP, many DB/HTTP clients&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nest / Node &lt;strong&gt;Kafka listener&lt;/strong&gt;, &lt;strong&gt;AMQ&lt;/strong&gt;, &lt;strong&gt;cron&lt;/strong&gt;, legacy routers&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;EC2 + PM2&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Daemon&lt;/strong&gt; on host&lt;/td&gt;
&lt;td&gt;Pure consumers: no HTTP entry; KafkaJS &lt;strong&gt;sometimes&lt;/strong&gt;; not MSK Lambda&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lambda&lt;/td&gt;
&lt;td&gt;AWS&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Layer&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;HTTP varies; MSK needs manual extract&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Happy path we validated first:&lt;/strong&gt; client → &lt;strong&gt;Nest HTTP on ECS&lt;/strong&gt; → auto SERVER span → auto DB children → optional &lt;strong&gt;direct&lt;/strong&gt; KafkaJS produce while the request span is still active (inject on broker headers). Export: app → &lt;strong&gt;sidecar&lt;/strong&gt; → Elastic. Nest &lt;strong&gt;Kafka listeners&lt;/strong&gt; on PM2 were often fine with auto-instrumentation; &lt;strong&gt;MSK Lambda&lt;/strong&gt;, &lt;strong&gt;outbox&lt;/strong&gt;, and &lt;strong&gt;cron&lt;/strong&gt; were not (section 3).&lt;/p&gt;

&lt;p&gt;Useful env vars (all Node roles): &lt;code&gt;OTEL_TRACES_EXPORTER=otlp&lt;/code&gt;, &lt;code&gt;OTEL_EXPORTER_OTLP_ENDPOINT&lt;/code&gt; (sidecar or host Collector), &lt;code&gt;OTEL_NODE_DISABLED_INSTRUMENTATIONS=fs&lt;/code&gt; when needed, &lt;code&gt;OTEL_NODE_RESOURCE_DETECTORS=env,host,os&lt;/code&gt; on bare EC2.&lt;/p&gt;

&lt;p&gt;Symfony local dev used a &lt;strong&gt;Collector container&lt;/strong&gt; on 4317/4318; production stayed “OTLP to nearest Collector.” Where auto stopped on Symfony we used PHP &lt;strong&gt;&lt;code&gt;#[WithSpan]&lt;/code&gt;&lt;/strong&gt;—same entry-span idea as the Node wrapper in section 3.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Where auto stops—and how we fix
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Auto-instrumentation&lt;/strong&gt; worked well for us—HTTP, SQL, and many clients inside a process. We kept it enabled everywhere.&lt;/p&gt;

&lt;p&gt;Our stack also uses patterns auto does not cover end-to-end: &lt;strong&gt;MSK-triggered Lambdas&lt;/strong&gt;, &lt;strong&gt;transactional outbox&lt;/strong&gt;, and &lt;strong&gt;cron jobs that emit events&lt;/strong&gt;. Three &lt;strong&gt;boundaries&lt;/strong&gt;, two &lt;strong&gt;mechanisms&lt;/strong&gt; (&lt;code&gt;withTraceSpan&lt;/code&gt;, &lt;strong&gt;&lt;code&gt;propagation.inject&lt;/code&gt;&lt;/strong&gt;)—different &lt;strong&gt;runtime shapes&lt;/strong&gt; from section 2’s in-process &lt;strong&gt;KafkaJS&lt;/strong&gt; happy path, not a publish-versus-consume checklist.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where auto stops (and why)
&lt;/h3&gt;

&lt;h4&gt;
  
  
  1. MSK-triggered Lambda (not KafkaJS consume)
&lt;/h4&gt;

&lt;p&gt;&lt;strong&gt;Why auto stops:&lt;/strong&gt; The handler receives &lt;strong&gt;AWS event records&lt;/strong&gt; from MSK (headers as &lt;strong&gt;binary&lt;/strong&gt; fields), not a KafkaJS &lt;strong&gt;&lt;code&gt;EachMessagePayload&lt;/code&gt;&lt;/strong&gt;. &lt;strong&gt;&lt;code&gt;@opentelemetry/instrumentation-kafkajs&lt;/code&gt;&lt;/strong&gt; never runs its consume hook—you may still model a &lt;strong&gt;CONSUMER&lt;/strong&gt; span manually, but this is &lt;strong&gt;not&lt;/strong&gt; a KafkaJS consumer for auto. Nothing auto-&lt;strong&gt;extract&lt;/strong&gt;s &lt;code&gt;traceparent&lt;/code&gt; from record headers or opens that entry span, so Lambda work often gets a &lt;strong&gt;second trace id&lt;/strong&gt; while the producer/outbox half stays on the first—not assuming context from the message body alone.&lt;/p&gt;

&lt;p&gt;One user action can look like &lt;strong&gt;two traces&lt;/strong&gt; in APM:&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
  subgraph traceAAA ["Trace AAA"]
    httpSpan[HTTP place-entry]
    outboxSpan[mysql outbox insert]
    httpSpan --&amp;gt; outboxSpan
  end
  subgraph traceBBB ["Trace BBB - orphan"]
    lambdaSpan[mysql update in Lambda]
  end
  outboxSpan -.-&amp;gt;|"Kafka headers not extracted"| lambdaSpan&lt;/code&gt;&lt;/pre&gt;



&lt;h4&gt;
  
  
  2. Transactional outbox
&lt;/h4&gt;

&lt;p&gt;&lt;strong&gt;Why auto stops:&lt;/strong&gt; The &lt;strong&gt;writer&lt;/strong&gt; commits a &lt;strong&gt;DB row&lt;/strong&gt; (header JSON + payload)—&lt;strong&gt;not&lt;/strong&gt; a Kafka produce. There are no broker headers until the &lt;strong&gt;relay&lt;/strong&gt; publishes; auto’s KafkaJS &lt;strong&gt;produce&lt;/strong&gt; hook runs on direct &lt;strong&gt;&lt;code&gt;send&lt;/code&gt;&lt;/strong&gt;, not on outbox insert. Auto does not write OTEL fields into that header JSON. Without &lt;strong&gt;&lt;code&gt;inject&lt;/code&gt; at commit&lt;/strong&gt; (snapshot into the row, not “make outbox a KafkaJS producer”), there is nothing for relay to copy and nothing for consumers to &lt;strong&gt;extract&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;If you &lt;strong&gt;&lt;code&gt;inject&lt;/code&gt; only at relay&lt;/strong&gt;, &lt;strong&gt;&lt;code&gt;context.active()&lt;/code&gt;&lt;/strong&gt; is the &lt;strong&gt;relay process&lt;/strong&gt; (relay worker trace id), not the &lt;strong&gt;producer-side trace&lt;/strong&gt; from commit—publish-time inject &lt;strong&gt;mis-links&lt;/strong&gt;. If the row has &lt;strong&gt;no&lt;/strong&gt; stored context and relay does not inject, Kafka may have &lt;strong&gt;no&lt;/strong&gt; &lt;code&gt;traceparent&lt;/code&gt; and consumers often start a &lt;strong&gt;separate trace&lt;/strong&gt;. Relay delay (minutes or days) is fine; &lt;strong&gt;&lt;code&gt;traceparent&lt;/code&gt; is a snapshot&lt;/strong&gt;, not a live session.&lt;/p&gt;

&lt;p&gt;With &lt;strong&gt;direct&lt;/strong&gt; KafkaJS produce—the section 2 pattern—the &lt;strong&gt;same instrumented Node process&lt;/strong&gt; produces and consumes while &lt;strong&gt;active context in the writer&lt;/strong&gt; still matches the logical operation. No outbox hop, so auto can inject and extract on broker headers without a DB snapshot. When &lt;strong&gt;MSK Lambda&lt;/strong&gt; is downstream of outbox, you typically need &lt;strong&gt;inject on the writer&lt;/strong&gt; (here) &lt;strong&gt;and&lt;/strong&gt; manual extract on Lambda (Stop 1).&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;sequenceDiagram
  participant Writer
  participant DB as Outbox_DB
  participant Relay
  participant Kafka

  Writer-&amp;gt;&amp;gt;DB: commit business plus outbox row
  Note over Writer,DB: inject traceparent at commit producer context
  Relay-&amp;gt;&amp;gt;Kafka: publish later
  Note over Relay: copy stored headers no re-inject for linking&lt;/code&gt;&lt;/pre&gt;



&lt;h4&gt;
  
  
  3. Cron job — simple entry that emits events
&lt;/h4&gt;

&lt;p&gt;&lt;strong&gt;Why auto stops:&lt;/strong&gt; There is no inbound &lt;strong&gt;carrier&lt;/strong&gt; / &lt;code&gt;traceparent&lt;/code&gt; (no browser, API Gateway, or Kafka consume on this process), so auto never opens an &lt;strong&gt;entry&lt;/strong&gt; span. The script &lt;strong&gt;emits events&lt;/strong&gt; that start a long downstream flow—you need a &lt;strong&gt;root for this run&lt;/strong&gt; plus &lt;strong&gt;inject on emit&lt;/strong&gt; so Kafka and consumers join that trace (see Fix 1 and Fix 2).&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart LR
  CronRun[Cron run]
  Emit[emit event]
  Kafka[Kafka]
  Down[downstream consumers]
  CronRun --&amp;gt; Emit --&amp;gt; Kafka --&amp;gt; Down&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  What we had to do
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;MSK Lambda&lt;/strong&gt; — MSK record headers are &lt;strong&gt;binary&lt;/strong&gt; → build a string-key &lt;strong&gt;carrier map&lt;/strong&gt; → &lt;strong&gt;&lt;code&gt;propagation.extract&lt;/code&gt;&lt;/strong&gt; → start a &lt;strong&gt;CONSUMER&lt;/strong&gt; entry span (&lt;code&gt;withTraceSpan&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Transactional outbox&lt;/strong&gt; — Persist &lt;strong&gt;W3C context&lt;/strong&gt; (&lt;code&gt;traceparent&lt;/code&gt; via &lt;strong&gt;&lt;code&gt;propagation.inject&lt;/code&gt;&lt;/strong&gt;) on the outbox row &lt;strong&gt;in the same transaction as the business write&lt;/strong&gt;, while &lt;strong&gt;commit-time active context&lt;/strong&gt; still reflects the &lt;strong&gt;producer-side trace&lt;/strong&gt;; relay &lt;strong&gt;replays&lt;/strong&gt; that carrier onto Kafka headers (does not re-derive from relay &lt;strong&gt;&lt;code&gt;context.active()&lt;/code&gt;&lt;/strong&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cron (and similar entrypoints)&lt;/strong&gt; — No inbound &lt;code&gt;traceparent&lt;/code&gt; → &lt;strong&gt;&lt;code&gt;withTraceSpan&lt;/code&gt;&lt;/strong&gt; opens a &lt;strong&gt;new root trace&lt;/strong&gt; per run; &lt;strong&gt;inject on emit&lt;/strong&gt; (Fix 2 below) when writing outbox or publishing.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  How we fix
&lt;/h3&gt;

&lt;h4&gt;
  
  
  Fix 1 — Wrapper function (&lt;code&gt;withTraceSpan&lt;/code&gt;)
&lt;/h4&gt;

&lt;p&gt;&lt;strong&gt;Maps to:&lt;/strong&gt; MSK Lambda (Stop 1), cron and similar entrypoints (Stop 3); AMQ / API Gateway in the aside below.&lt;/p&gt;

&lt;p&gt;Call sites pass &lt;strong&gt;&lt;code&gt;{ name, options, carrier }&lt;/code&gt;&lt;/strong&gt; when wrapping—the wrapper is intentionally small; each entry point chooses span name, &lt;strong&gt;&lt;code&gt;SpanKind&lt;/code&gt;&lt;/strong&gt; / attributes, and the W3C &lt;strong&gt;carrier&lt;/strong&gt; map.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Inbound carrier (MSK, AMQ, API Gateway):&lt;/strong&gt; &lt;strong&gt;&lt;code&gt;carrier&lt;/code&gt;&lt;/strong&gt; = decoded headers (after binary → map for MSK). &lt;strong&gt;&lt;code&gt;options&lt;/code&gt;&lt;/strong&gt; sets &lt;strong&gt;CONSUMER&lt;/strong&gt; or &lt;strong&gt;SERVER&lt;/strong&gt; and messaging/HTTP attributes. &lt;strong&gt;&lt;code&gt;propagation.extract(context.active(), carrier)&lt;/code&gt;&lt;/strong&gt; runs before &lt;strong&gt;&lt;code&gt;startSpan&lt;/code&gt;&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No inbound carrier (cron):&lt;/strong&gt; &lt;strong&gt;&lt;code&gt;carrier: {}&lt;/code&gt;&lt;/strong&gt; → &lt;strong&gt;new root trace&lt;/strong&gt; per run; &lt;strong&gt;&lt;code&gt;name&lt;/code&gt;&lt;/strong&gt; = job module name. With that context active, &lt;strong&gt;inject on emit&lt;/strong&gt; (Fix 2 for outbox rows, or direct publish).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On failure the span gets &lt;strong&gt;&lt;code&gt;recordException&lt;/code&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;SpanStatusCode.ERROR&lt;/code&gt;&lt;/strong&gt;. Auto-instrumentation inside the handler is unchanged. PHP &lt;strong&gt;&lt;code&gt;#[WithSpan]&lt;/code&gt;&lt;/strong&gt; on Symfony follows the same entry-span idea.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cron:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nf"&gt;runJobWithOtelThenExit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nf"&gt;withTraceSpan&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;__filename&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;carrier&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt; &lt;span class="p"&gt;})(&lt;/span&gt;&lt;span class="nx"&gt;run&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Linked trace after extract + entry span:&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
  subgraph traceOne ["Single trace AAA"]
    http2[HTTP place-entry]
    outbox2[mysql outbox]
    kafkaHop[Kafka]
    consumerSpan["CONSUMER span topic-name"]
    db2[mysql update]
    http2 --&amp;gt; outbox2 --&amp;gt; kafkaHop --&amp;gt; consumerSpan --&amp;gt; db2
  end&lt;/code&gt;&lt;/pre&gt;



&lt;h4&gt;
  
  
  Trace vs tracer
&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Trace&lt;/strong&gt; — End-to-end journey; one &lt;strong&gt;trace id&lt;/strong&gt; in &lt;code&gt;traceparent&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tracer&lt;/strong&gt; — &lt;code&gt;trace.getTracer(serviceName)&lt;/code&gt; scopes spans to this &lt;strong&gt;service&lt;/strong&gt; in APM; spans still join the same trace when parent context is passed to &lt;code&gt;startSpan&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Span&lt;/strong&gt; — One unit of work; auto-instrumented children nest under the entry span when the handler runs inside &lt;code&gt;context.with(trace.setSpan(...))&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;MSK Lambda still chains &lt;strong&gt;&lt;code&gt;createTransformMskRecord&lt;/code&gt;&lt;/strong&gt; (binary headers → &lt;strong&gt;&lt;code&gt;carrier&lt;/code&gt;&lt;/strong&gt; map) via &lt;strong&gt;&lt;code&gt;composeHandler&lt;/code&gt;&lt;/strong&gt;. Middleware builds &lt;strong&gt;&lt;code&gt;name&lt;/code&gt;&lt;/strong&gt; (topic), &lt;strong&gt;CONSUMER&lt;/strong&gt; &lt;strong&gt;&lt;code&gt;options&lt;/code&gt;&lt;/strong&gt;, and &lt;strong&gt;&lt;code&gt;carrier&lt;/code&gt;&lt;/strong&gt; per record, then applies &lt;strong&gt;&lt;code&gt;withTraceSpan&lt;/code&gt;&lt;/strong&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nf"&gt;composeHandler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nf"&gt;createTransformMskRecord&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="nx"&gt;withTraceSpanForMskRecord&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nf"&gt;createCaptureCorrelationIds&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;correlationIds&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="nx"&gt;recordHandler&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;// withTraceSpanForMskRecord (per record): withTraceSpan({&lt;/span&gt;
&lt;span class="c1"&gt;//   name: topic,&lt;/span&gt;
&lt;span class="c1"&gt;//   options: { kind: SpanKind.CONSUMER, attributes: { 'messaging.destination': topic, ... } },&lt;/span&gt;
&lt;span class="c1"&gt;//   carrier: decodedHeaders,&lt;/span&gt;
&lt;span class="c1"&gt;// })(handler)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Lifecycle:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;propagation.extract(context.active(), carrier)&lt;/code&gt; from props&lt;/li&gt;
&lt;li&gt;&lt;code&gt;trace.getTracer(serviceName)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;tracer.startSpan(name, options, activeContext)&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;context.with(trace.setSpan(activeContext, span), () =&amp;gt; original(...args))&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;On catch: &lt;code&gt;recordException&lt;/code&gt;, &lt;code&gt;setStatus(ERROR)&lt;/code&gt;, rethrow&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;span.end()&lt;/code&gt; in &lt;code&gt;finally&lt;/code&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;propagation&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;SpanStatusCode&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@opentelemetry/api&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;serviceName&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;./config&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;withTraceSpan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;props&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;options&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;carrier&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;props&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;wrap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;original&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;wrapped&lt;/span&gt;&lt;span class="p"&gt;(...&lt;/span&gt;&lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;activeContext&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;propagation&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;extract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;active&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="nx"&gt;carrier&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;tracer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getTracer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;serviceName&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;span&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;tracer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startSpan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;options&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;activeContext&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

      &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;with&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setSpan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;activeContext&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;original&lt;/span&gt;&lt;span class="p"&gt;(...&lt;/span&gt;&lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;recordException&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setStatus&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
          &lt;span class="na"&gt;code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;SpanStatusCode&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ERROR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="na"&gt;message&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;message&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;});&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;finally&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h4&gt;
  
  
  Fix 2 — &lt;code&gt;propagation.inject&lt;/code&gt; (outbox headers in DB)
&lt;/h4&gt;

&lt;p&gt;&lt;strong&gt;Maps to:&lt;/strong&gt; Transactional outbox (Stop 2).&lt;/p&gt;

&lt;p&gt;Build the header object (Content-Type, message-&lt;em&gt;), then *&lt;/em&gt;&lt;code&gt;propagation.inject(context.active(), headers)&lt;/code&gt;** in the &lt;strong&gt;same DB transaction&lt;/strong&gt; as the business write (&lt;a href="https://opentelemetry.io/docs/languages/php/propagation/#manual-context-propagation" rel="noopener noreferrer"&gt;manual propagation&lt;/a&gt;). &lt;strong&gt;&lt;code&gt;traceparent&lt;/code&gt;&lt;/strong&gt; is what inject adds for &lt;code&gt;tracecontext&lt;/code&gt;; optional &lt;strong&gt;&lt;code&gt;tracestate&lt;/code&gt;&lt;/strong&gt; if non-empty. The JSON column is the &lt;strong&gt;carrier&lt;/strong&gt;; relay copies it to &lt;strong&gt;Kafka record headers&lt;/strong&gt;. &lt;strong&gt;KafkaJS&lt;/strong&gt; listeners often &lt;strong&gt;auto-extract&lt;/strong&gt;; &lt;strong&gt;MSK Lambda&lt;/strong&gt; still needs Fix 1.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;headers&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Content-Type&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;application/json&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;message-type&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;messageType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;message-schema-version&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;schemaVersion&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;message-id&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;messageId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="nx"&gt;propagation&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;inject&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;active&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="c1"&gt;// persist headers with outbox row in same transaction&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Example stored with the outbox row:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"Content-Type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"application/json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"traceparent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"00-aed57677a060346c7850ad43d8a95dc6-2f208df453f6743a-00"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"message-type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"tournament-entered"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"message-schema-version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"message-id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"c7bf7357-a32e-4b36-9f8c-1ea057220184"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After relay copies &lt;strong&gt;&lt;code&gt;traceparent&lt;/code&gt;&lt;/strong&gt; to record headers, an in-process &lt;strong&gt;KafkaJS listener on PM2&lt;/strong&gt; can often &lt;strong&gt;extract&lt;/strong&gt; with auto; &lt;strong&gt;MSK-triggered Lambda&lt;/strong&gt; still needs Fix 1.&lt;/p&gt;

&lt;h3&gt;
  
  
  Aside — other wrapper uses
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;AMQ:&lt;/strong&gt; Auto does not hook our queue manager like KafkaJS → &lt;strong&gt;&lt;code&gt;withTraceSpan&lt;/code&gt;&lt;/strong&gt; with queue name, &lt;strong&gt;CONSUMER&lt;/strong&gt; &lt;strong&gt;&lt;code&gt;options&lt;/code&gt;&lt;/strong&gt;, message-header &lt;strong&gt;&lt;code&gt;carrier&lt;/code&gt;&lt;/strong&gt; on inbound; &lt;strong&gt;&lt;code&gt;propagation.inject&lt;/code&gt;&lt;/strong&gt; on outbound publish.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;API Gateway Lambda:&lt;/strong&gt; &lt;strong&gt;&lt;code&gt;withTraceSpan&lt;/code&gt;&lt;/strong&gt; with &lt;strong&gt;SERVER&lt;/strong&gt; &lt;strong&gt;&lt;code&gt;options&lt;/code&gt;&lt;/strong&gt;, span name &lt;code&gt;httpMethod&lt;/code&gt; + resource path, &lt;strong&gt;&lt;code&gt;carrier&lt;/code&gt;&lt;/strong&gt; from API Gateway headers (including &lt;code&gt;x-correlation-id&lt;/code&gt;).
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nf"&gt;addItemToOutboundQueue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;outboundQueue&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;item&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;customHeaders&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;...)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;headers&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;customHeaders&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="nx"&gt;propagation&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;inject&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;active&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;super&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;addItemToOutboundQueue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;outboundQueue&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;item&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;...);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Footnotes
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Lambda:&lt;/strong&gt; OTEL layers, feature flags, handler export &lt;code&gt;{ handler }&lt;/code&gt;, optional KafkaJS/Winston via post-wrapper &lt;code&gt;registerInstrumentations&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Unrecoverable Kafka consumer crashes can block process exit—shut down explicitly so the supervisor can restart.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;We got value fast from &lt;strong&gt;auto-instrumentation&lt;/strong&gt; and a consistent &lt;strong&gt;Collector&lt;/strong&gt; story (&lt;strong&gt;ECS sidecar&lt;/strong&gt;, &lt;strong&gt;EC2 daemon&lt;/strong&gt;, &lt;strong&gt;Lambda layer&lt;/strong&gt;). &lt;strong&gt;Continuous traces&lt;/strong&gt; at async boundaries needed &lt;strong&gt;two mechanisms&lt;/strong&gt; across &lt;strong&gt;three patterns&lt;/strong&gt;: &lt;strong&gt;&lt;code&gt;withTraceSpan&lt;/code&gt;&lt;/strong&gt; for MSK (and AMQ/API Gateway) &lt;strong&gt;extract&lt;/strong&gt; plus &lt;strong&gt;cron root&lt;/strong&gt; when there is no inbound carrier; &lt;strong&gt;&lt;code&gt;propagation.inject&lt;/code&gt;&lt;/strong&gt; to snapshot &lt;strong&gt;producer-side context&lt;/strong&gt; on outbox commit—direct &lt;strong&gt;KafkaJS produce and consume in one process&lt;/strong&gt; often stayed on auto; &lt;strong&gt;outbox snapshot on the writer plus manual extract on MSK Lambda&lt;/strong&gt; when that is the downstream path. Auto unchanged inside handlers.&lt;/p&gt;

&lt;p&gt;For a similar rollout: enable &lt;strong&gt;tracecontext&lt;/strong&gt;, put a Collector in front of Elastic, prove one &lt;strong&gt;ECS HTTP&lt;/strong&gt; path, then fix &lt;strong&gt;one async boundary&lt;/strong&gt; (outbox + one consumer) end-to-end before calling tracing “done.” Metrics and unified OTEL logging can follow once traces read as one story.&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>devops</category>
      <category>monitoring</category>
    </item>
    <item>
      <title>From Event Modeling to a Kafka Saga: Tournament Entry Case Study</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Wed, 28 Oct 2026 07:25:34 +0000</pubDate>
      <link>https://dev.to/joungpark/from-event-modeling-to-a-kafka-saga-tournament-entry-case-study-jd8</link>
      <guid>https://dev.to/joungpark/from-event-modeling-to-a-kafka-saga-tournament-entry-case-study-jd8</guid>
      <description>&lt;p&gt;We Event-Modeled a &lt;strong&gt;multi-team tournament product&lt;/strong&gt; to keep delivery &lt;strong&gt;scalable&lt;/strong&gt;: domain squads own slices on one board, and later features can often &lt;strong&gt;subscribe to events already on the timeline&lt;/strong&gt; instead of rewiring core flows.&lt;/p&gt;

&lt;p&gt;This &lt;strong&gt;case study&lt;/strong&gt; follows three beats:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Why Event Modeling&lt;/strong&gt; — product-wide board, Conway-aligned teams, event contracts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Event Modeling in practice&lt;/strong&gt; — the &lt;a href="https://eventmodeling.org/posts/what-is-event-modeling/#seven-steps" rel="noopener noreferrer"&gt;&lt;strong&gt;seven steps&lt;/strong&gt;&lt;/a&gt; on the &lt;strong&gt;tournament entry&lt;/strong&gt; process (diagrams, abbreviated tickets).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Running tournament entry&lt;/strong&gt; — &lt;strong&gt;Saga&lt;/strong&gt; and &lt;strong&gt;CQRS&lt;/strong&gt;-style read paths as the runtime shape that matches the model.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;We modeled the &lt;strong&gt;whole tournament&lt;/strong&gt;; this post &lt;strong&gt;zooms into tournament entry&lt;/strong&gt; for depth on fan-out, orchestration, and compensation. Simplified example; reflects how we modeled, sliced, and shipped the work.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Why Event Modeling
&lt;/h2&gt;

&lt;p&gt;The tournament product had &lt;strong&gt;many processes&lt;/strong&gt; and &lt;strong&gt;domain-based teams&lt;/strong&gt; in parallel—for example &lt;strong&gt;Tournament (Team A)&lt;/strong&gt; and &lt;strong&gt;Payment (Team B)&lt;/strong&gt;. We needed one &lt;strong&gt;shared&lt;/strong&gt; picture of workflows, not a separate integration design per feature.&lt;/p&gt;

&lt;p&gt;Event Modeling gave us a &lt;strong&gt;scalable&lt;/strong&gt; way to deliver:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Parallel ownership&lt;/strong&gt; — swimlanes aligned to Conway’s law; each team implements workflow steps from the same board.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stable contracts&lt;/strong&gt; — event payloads define what you consume and publish; backlog items snap to adjacent steps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Easy extension&lt;/strong&gt; — a later feature (for example notifications) can often be a new consumer on an event the model already defines, without changing tournament entry’s core design.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We modeled the &lt;strong&gt;full tournament&lt;/strong&gt; on one board. This article &lt;strong&gt;goes deep on tournament entry&lt;/strong&gt; because that slice best shows &lt;strong&gt;Saga&lt;/strong&gt; and &lt;strong&gt;CQRS&lt;/strong&gt; at runtime in §3—not because Event Modeling is only for tournament entry. Tournament entry is &lt;strong&gt;async and multi-service&lt;/strong&gt; on the board; &lt;strong&gt;§3&lt;/strong&gt; shows how we ran it.&lt;/p&gt;

&lt;p&gt;Using the seven-step workshop &lt;strong&gt;only&lt;/strong&gt; for a tiny feature would be over-engineering; at &lt;strong&gt;product&lt;/strong&gt; scope it paid off.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Event Modeling: tournament entry
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Part I&lt;/strong&gt; — how we modeled &lt;strong&gt;tournament entry&lt;/strong&gt; on the product board (other tournament processes used the same seven steps; we do not repeat them here).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tournament entry scenario:&lt;/strong&gt; a player enters an online tournament from the app. One user action must &lt;strong&gt;charge an entry fee&lt;/strong&gt;, &lt;strong&gt;verify game rank&lt;/strong&gt;, and &lt;strong&gt;record a baseline&lt;/strong&gt; before play starts—across Team A/B services and external APIs.&lt;/p&gt;

&lt;p&gt;The PO had PRD and wireframes for the &lt;strong&gt;full tournament&lt;/strong&gt;; we aligned the model to them and elaborated &lt;strong&gt;tournament entry&lt;/strong&gt; below using the &lt;a href="https://eventmodeling.org/posts/what-is-event-modeling/#seven-steps" rel="noopener noreferrer"&gt;&lt;strong&gt;seven steps&lt;/strong&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;At product scale the board was much larger than the diagrams in this post. The export below is the &lt;strong&gt;whole-tournament&lt;/strong&gt; model we worked from (company security limits how crisp we can share it—expect a wide, low-resolution image). Everything from step 1 onward is &lt;strong&gt;tournament entry&lt;/strong&gt; only.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6lczuruhb1mmjmc7s5ms.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6lczuruhb1mmjmc7s5ms.png" alt="Full tournament Event Modeling board (low-resolution workshop export)" width="800" height="107"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1 — Brainstorm
&lt;/h3&gt;

&lt;p&gt;We collected &lt;strong&gt;state-changing&lt;/strong&gt; outcomes only—not read-only actions like “user viewed status.” Participants named happy paths, failures, and compensation. Not every idea stayed: &lt;strong&gt;if there was no place to put an event&lt;/strong&gt; yet (no owner, no &lt;strong&gt;when X → publish Y&lt;/strong&gt; hook), we &lt;strong&gt;ignored it&lt;/strong&gt; until a later step gave it a home. Each kept name is the &lt;strong&gt;Y&lt;/strong&gt; you &lt;strong&gt;publish&lt;/strong&gt; when a step’s outcome is known. The list is &lt;strong&gt;unordered&lt;/strong&gt;; nothing goes on the timeline until step 2.&lt;/p&gt;

&lt;p&gt;For tournament entry, the brainstorm list included:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;tournament_enter_requested&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;payment_success&lt;/code&gt; / &lt;code&gt;payment_failure&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;game_rank_verified&lt;/code&gt; / &lt;code&gt;game_rank_verification_failed&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;baseline_captured&lt;/code&gt; / &lt;code&gt;baseline_capture_failed&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;&lt;code&gt;payment_refunded&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;tournament_entry_confirmed&lt;/code&gt; / &lt;code&gt;tournament_entry_rejected&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Step 2 — The plot
&lt;/h3&gt;

&lt;p&gt;Put the brainstorm list on a &lt;strong&gt;timeline&lt;/strong&gt; and check the story holds. Commands, wireframes, swimlanes, and payloads come in later steps.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://mermaid.live/edit#pako:eNp1kUtugzAURbeC3hhFBvML03YVDQi58CA0wdCHHTWNsvcaqwn5kAGSfS_n-HeCsq8QUsADStWZ8b6VTSYdR9UO8xw8KEf1mqToTF-YD6kg_NY4KqyczeZ1mecXjW81gzja30ZdljiOhn1IZoDfAbVo95rwBvhPZiCwQGP2UZCQu-KA1Nat3eBzOGPhIlYK1fbSLrJouOtnWWRln2KcrhCLUgzK7HESPGUzFC9C89ovmlmQ3F0VYa1lZcnHaEbWC69Kx6LsZd1St_Cqt-VV47FlDeEXlmrZcunyHFxoqK0gVaTRhQ6pE9MUTpM-A7XFDjNIzbAStMsgk2fDDEJ-9H0HaS32o-Go1832OtNDJRS-t6Ih0V3UhOb09NZrqSD1POuA9AQ_kPoBXyWB5_tRwuK1F4aJC8cpZiseBCyMPc44j6L47MKvXZatImZyHvJ1zHmSmOoPkxwpJA" rel="noopener noreferrer"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNp1kV1vgyAUhv-KOdemQfGr3m6_YrUxTI_WtaI7QrOu6X8fknX0w12YwPvyPCCcoRpqhBzwiFL1ZnzoZFtIz1ONxwIPj8pTgyYpetOX5kMqCT81Tgprb7P5v9xur5rQakZxsssmXVU4TYZ9SBzA74BGdAdNeAP8Jg6ILNCac5Qk5L48InVNZw_4HDosXsQqobpB2k0WDXe9kyVW9i6m-QqxrMSozBlnwVPmoHQRcnv_0zhBdndVhI2WtSUfI4esF16VTmU1yKajfuFVb8s_TcCWNYQfWKlly7XbbsGHlroackUafeiRejFP4TzrC1A77LGA3AxrQfsCCnkxzCjk2zD0V4wG3e4gb8RhMjM91kLhaydaEm4Jmp-nl0FLBXkQWwXkZ_iCPIz4KouCMEwylq6DOM58OM0xW_EoYnEacMZ5kqQXH77trmyVMJPzmK9TzrPMVD9pnijd%3Ftype%3Dpng" width="1904" height="113"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3 — Storyboard
&lt;/h3&gt;

&lt;p&gt;The PO’s wireframes sit at the top of the board: screens, fields, and what the user triggers. That anchors commands and read models to real UI—who clicks what, and what they see while the workflow runs. We did not redraw UX in the workshop; we aligned the model to wireframes already in the PRD package.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4 — Identify inputs
&lt;/h3&gt;

&lt;p&gt;We tied each timeline step to &lt;strong&gt;what triggers it&lt;/strong&gt; and &lt;strong&gt;what data flows in&lt;/strong&gt;: UI actions, commands, and event payloads upstream handlers consume.&lt;/p&gt;

&lt;p&gt;A &lt;strong&gt;command&lt;/strong&gt; is an intention to do work—it can be rejected or fail before anything is recorded as an event. Examples on our board:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Tournament.enterTournament&lt;/code&gt; (user action in the UI)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Payment.debitPayment&lt;/code&gt; (often triggered after &lt;code&gt;tournament_enter_requested&lt;/code&gt; arrives)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;GameRank.checkGameRank&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;Baseline.captureBaseline&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, &lt;code&gt;Payment.debitPayment&lt;/code&gt; expresses what we want the payment service to attempt.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://mermaid.live/edit#pako:eNp1kV1vgyAUhv-KOdemQfGr3m6_YrUxTI_WtaI7QrOu6X8fknX0w12YwPvyPCCcoRpqhBzwiFL1ZnzoZFtIz1ONxwIPj8pTgyYpetOX5kMqCT81Tgprb7P5v9xur5rQakZxsssmXVU4TYZ9SBzA74BGdAdNeAP8Jg6ILNCac5Qk5L48InVNZw_4HDosXsQqobpB2k0WDXe9kyVW9i6m-QqxrMSozBlnwVPmoHQRcnv_0zhBdndVhI2WtSUfI4esF16VTmU1yKajfuFVb8s_TcCWNYQfWKlly7XbbsGHlroackUafeiRejFP4TzrC1A77LGA3AxrQfsCCnkxzCjk2zD0V4wG3e4gb8RhMjM91kLhaydaEm4Jmp-nl0FLBXkQWwXkZ_iCPIz4KouCMEwylq6DOM58OM0xW_EoYnEacMZ5kqQXH77trmyVMJPzmK9TzrPMVD9pnijd" rel="noopener noreferrer"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNp9TztvwjAQ_iunW1ggsuMEggeWdunWoVMJQiY-QgRxqLFRKeK_1zGPdmkX67vvdeczVp0mlEhHMq4NeNeYujQAbg2Mg2_grfPWqDbIcIbwkoWVd64zUO2aaksaLqW5J1KoWv0rksTAz_zjzICODtxDWUbn0tKHp4MLraPZrO-bz__2LBaPPs5gX1l4Vad4aQxnd43Hq25aomnVuNvQ57Vy6r9Dzn2NP5B90RIGV8AHPRmc9hTZiEY8FZF3ytbklnXoW1pltsFQdzsdtTWRhDxhrDQXHGJtG43SWU9DbMm2qh8xrizRbailEmWAWtltidfMXpn3rmvvMdv5eoNyrXaHMPl9-A89N6q2qn2wlowm-9R541DyaexAecZPlGkmkiLjaTou2GTK87wY4qmnWSKyjOUTLpgQ4_HkMsSvuJYlYxZ4kYvpRIiiCNI30CzBfw%3Ftype%3Dpng" width="1374" height="635"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5 — Identify outputs
&lt;/h3&gt;

&lt;p&gt;We filled in &lt;strong&gt;published event payloads&lt;/strong&gt; and &lt;strong&gt;read models&lt;/strong&gt; (views)—the fields other swimlanes and the UI need after each step completes.&lt;/p&gt;

&lt;p&gt;Outcome events from the payment command, for example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;payment_success&lt;/code&gt; — fee, transaction id, entry id&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;payment_failure&lt;/code&gt; — entry id and failure status&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Read models such as &lt;code&gt;entry_status&lt;/code&gt; aggregate those outcomes for queries and the frontend.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://mermaid.live/edit#pako:eNp9TztvwjAQ_iunW1ggsuMEggeWdunWoVMJQiY-QgRxqLFRKeK_1zGPdmkX67vvdeczVp0mlEhHMq4NeNeYujQAbg2Mg2_grfPWqDbIcIbwkoWVd64zUO2aaksaLqW5J1KoWv0rksTAz_zjzICODtxDWUbn0tKHp4MLraPZrO-bz__2LBaPPs5gX1l4Vad4aQxnd43Hq25aomnVuNvQ57Vy6r9Dzn2NP5B90RIGV8AHPRmc9hTZiEY8FZF3ytbklnXoW1pltsFQdzsdtTWRhDxhrDQXHGJtG43SWU9DbMm2qh8xrizRbailEmWAWtltidfMXpn3rmvvMdv5eoNyrXaHMPl9-A89N6q2qn2wlowm-9R541DyaexAecZPlGkmkiLjaTou2GTK87wY4qmnWSKyjOUTLpgQ4_HkMsSvuJYlYxZ4kYvpRIiiCNI30CzBfw" rel="noopener noreferrer"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNp9T11PwjAU_SvNfeFlzG3dF3sgQSRqYsAHfZGRUbcyprQlXUfEhf9uNzajkvjUe87t-bg1pCKjEAE9UK6YnncFz2OOkNog20Ypy9AjOTK9NDP6WqgOxLz_4yB6UGh_ppOySlNalmi5_MOsVr0A_xJsSLGrJP0h6JhG0EtcJJlAeimPSamIqko0HI-b7PbBaP1WCr6uUQydSwyRBl32VWcZAzo1phlR5KJx3YS1EfdZhAbtNLQdPGhLSMJLkqpC8HarpK35S7P-mn_NJCW6rabni6dkNl88394l15OHyXw6a0zBgFwWGURKVtQARiUjDYTWNAa1pYye78uIfI_hrNkT_iIE62VSVPkWog3ZlRpVe12T3hQkl4R9s5LyjMqpqLiCyMGtB0Q1fGjkYjN0bcfxQysY2Z4XGnBsaMvErmt5gY0tjH0_OBnw2cZapm9pHnt4FGAchnr1BbV_x84%3Ftype%3Dpng" width="1458" height="478"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 6 — Swimlanes (Conway’s law)
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Swimlanes&lt;/strong&gt; assign commands, events, and views to owning components so teams can own autonomous parts. The timeline explains &lt;em&gt;what happens&lt;/em&gt;; swimlanes clarify &lt;em&gt;who implements it&lt;/em&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Typical owning team&lt;/th&gt;
&lt;th&gt;Responsibility&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Web application&lt;/td&gt;
&lt;td&gt;Tournament (A) / frontend&lt;/td&gt;
&lt;td&gt;Submit tournament entry request and display status&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tournament Entry API&lt;/td&gt;
&lt;td&gt;Tournament (A)&lt;/td&gt;
&lt;td&gt;Create a pending tournament entry&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Payment Service&lt;/td&gt;
&lt;td&gt;Payment (B)&lt;/td&gt;
&lt;td&gt;Process payment and refunds&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rank Service&lt;/td&gt;
&lt;td&gt;Tournament (A) or game integration&lt;/td&gt;
&lt;td&gt;Verify player eligibility&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Baseline Service&lt;/td&gt;
&lt;td&gt;Tournament (A) or game integration&lt;/td&gt;
&lt;td&gt;Capture game statistics&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Saga Orchestrator&lt;/td&gt;
&lt;td&gt;Tournament (A) or platform&lt;/td&gt;
&lt;td&gt;Track operation results and coordinate the overall outcome&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Unlike a sequential workflow, payment processing, rank verification, and baselining can run independently once &lt;code&gt;tournament_enter_requested&lt;/code&gt; exists. Final confirmation still depends on all three results—the orchestrator combines them in §3.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://mermaid.live/edit#pako:eNqNVF1v2jAU_SuWX_pC09gOheahUttV3aSJTlP3MlKlJrmkWYnDHAeNRf3vc-w4QPnQeADfc-85vvfYpsFJmQIOMaxAqEKvF7nIIoGQmiOfoDpHT2UtBS90GjVIf4NEs1qpUqBkkSdvkKL3SDgGRUmRblE8Q9jEro4hWZStmlzHleKqrtD59XXLb9ASRKqbaGVtcYBgpZDqRWIjGkv4XUOldAMddTo9XvP83DdJfLRMJPrG12YoQw5cjpgBupyXwixXXeAqqOlmacG4qpMEqkrv_QFpN7QEtkOY83xRS9CETUVw2A29lflh6OVXVYqXBkW4k4lwqINuq4tOM8JbR0HtlA_aje9cvO2OSe2YLuklr5C8ucjV2EEzjcZSw_EKZD7Ptd3T6T7YD0PZQVrCVV4KM_1BhZ38RuyIM9Q6Q7ed6RWtN66xCyu54w2z3tzyqr3usOsNs964pJfwpdLmuthVWXdmHRp3Ve1oe1g_DmMHSRtXjmQ2Akf8YNYPtu2H07J2uF627ED_87GepVzxUw-waYvqCuSXNERndkHOWtC0alCzOieUGVxxmYGK-0PTBVm5SE1uDhCioef7kTBnZnb_-OCak-pasuJJe6VMVsm2m30x9xhPikng2lMNTx6f4vvJ44-Hz_Htzdebyd19K4oHOJN5ikMlaxjgAmTB2xAb0QirVyi6Q0i51LfTcpZc_CzLwtFkWWevOJzzRaWjeqnbhE85zyQvelTqP0aQd2UtFA7ppdHAYYP_6IheeSQIrhgdjwkdD0kwwGsckrHvjZhPhwGjARld-ux9gP-abYnHfOIHo7FPrkYkYMPh-z_dZAQP" rel="noopener noreferrer"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNqNVNFu2jAU_RXLL32haWyHQfNQqe2qbtJEp6l7GalSk1zSrMRhjoNGI_59jh0HaAGNB_A9957je49tGpyUKeAQwwqEKvR6kYssEgipOfIJqnP0WNZS8EKnUYP0N0g0q5UqBUoWefIKKdpEwjEoSop0h-IZwjZ2dQzJomzV5DquFFd1hc6vrlp-g5YgUt1EK2uLAwQrhVQvEhvRWMKfGiqlG-io0-nxmqenvknio2Ui0Xe-NkMZcuByxAzQ5bwUZrnqAldBTTdLC8ZVnSRQVXrvd0i7oSWwPcKc54tawg6hQ7aE4LA5emfzw9Dz76oUzw2KcKcR4VAH3c4XnWCEd06G2qHvtTk_uHjdn5raqV3SS14geXWRq7FzZxqNpYbjFch8nmv3p9OPYD8MZQdpCVd5KczoBxX28luxI85Q6wzddaZXtN64xi6s5J43zHpzw6v29sO-N8x645JewpdKm-tiV2XdmXVo3FW1o33A-nEYO0jaunIksxU44gezfrBdP5yWtcP1smMH-p-P9Szlip96j01bVFcgv6YhOrMLctaCplWDmtU5oczgissMVNwfmi7IykVqcnOAEA0934-EOTOz-_v315xU15IVT9orZbJKtt18FHNv86SYBK491fDk4TG-mzz8vP8S31x_u57c3rWieIAzmac4VLKGAS5AFrwNsRGNsHqBojuElEt9Oy1nycWvsiwcTZZ19oLDOV9UOqqXuk34nPNM8qJHpf6fBHlb1kLhkPlGA4cN_otDSi89EgSXjI7HhI6HJBjgNQ7J2PdGzKfDgNGAjD75bDPAb2Zb4jGf-MFo7JPLEQnYcLj5B4ZiCk8%3Ftype%3Dpng" width="1904" height="451"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;(click the image above to see it in a full screen in mermaid live)&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 7 — Elaborate scenarios (Task creation)
&lt;/h3&gt;

&lt;p&gt;Each workflow step on the board became concrete backlog work: handle a command, publish an event, subscribe and react, or update a read model. Specs are &lt;strong&gt;Given–When–Then&lt;/strong&gt; (or Given–Then for views), tied to &lt;strong&gt;one&lt;/strong&gt; command or read model on the board. In the backlog, we wrote &lt;strong&gt;detailed&lt;/strong&gt; tickets—schema fields, topic names, error cases, and dependencies copied from the model. The snippet below is an &lt;strong&gt;abbreviated&lt;/strong&gt; PAY-1 in that format, not the full ticket we shipped.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://mermaid.live/edit#pako:eNp9T11PwjAU_SvNfeFlzG3dF3sgQSRqYsAHfZGRUbcyprQlXUfEhf9uNzajkvjUe87t-bg1pCKjEAE9UK6YnncFz2OOkNog20Ypy9AjOTK9NDP6WqgOxLz_4yB6UGh_ppOySlNalmi5_MOsVr0A_xJsSLGrJP0h6JhG0EtcJJlAeimPSamIqko0HI-b7PbBaP1WCr6uUQydSwyRBl32VWcZAzo1phlR5KJx3YS1EfdZhAbtNLQdPGhLSMJLkqpC8HarpK35S7P-mn_NJCW6rabni6dkNl88394l15OHyXw6a0zBgFwWGURKVtQARiUjDYTWNAa1pYye78uIfI_hrNkT_iIE62VSVPkWog3ZlRpVe12T3hQkl4R9s5LyjMqpqLiCyMGtB0Q1fGjkYjN0bcfxQysY2Z4XGnBsaMvErmt5gY0tjH0_OBnw2cZapm9pHnt4FGAchnr1BbV_x84" rel="noopener noreferrer"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fmermaid.ink%2Fimg%2Fpako%3AeNp9jj9PwzAQxb-KdQtLGtlx_tVDF1jYmGmqysSXNKJ2gmNXlKrfHSdtkBhgsPTe753f3QXqXiEIwBMap4M-dqatDCGuITQleHLE9d4aqUO-Dw_t3uKHx9GhItvt3-FuV5l7EaNkqC15kedpkKw2m9C9ZIzUWi1ZrPCtc3ezTCTzGcMN7kdf1ziOS8h_hY3sjt7itFlJJ_-7_TI1-BHtsxLk4SbYwwTDpD3PdFYrlvCZN4iCZDGllblCBK3tFAhnPUag0Wo5WZhrK3AH1FiBCFJJ-17B7c8gzWvf6-Wb7X17ANHI4xicH8LN-NTJ1kr9Qy0ahfax98aBSIq5A8QFPoNLeVymLEnykhZrlmVlBOcJ05inKc0KxinneV5cI_ia19I4p4HzjK8LzssyRN-yManz%3Ftype%3Dpng" width="1371" height="478"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h4&gt;
  
  
  Example ticket (abbreviated): PAY-1 — &lt;code&gt;Payment.debitPayment&lt;/code&gt; on &lt;code&gt;tournament_enter_requested&lt;/code&gt;
&lt;/h4&gt;

&lt;p&gt;&lt;strong&gt;Scenario: successful debit&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Given&lt;/strong&gt; a consumed &lt;code&gt;tournament_enter_requested&lt;/code&gt; with valid &lt;code&gt;entryId&lt;/code&gt;, &lt;code&gt;userId&lt;/code&gt;, and &lt;code&gt;fee&lt;/code&gt;, and no successful debit already recorded for that &lt;code&gt;entryId&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When&lt;/strong&gt; the Payment service runs &lt;code&gt;Payment.debitPayment&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Then&lt;/strong&gt; &lt;code&gt;payment_success&lt;/code&gt; is published (including &lt;code&gt;transactionId&lt;/code&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Scenario: debit fails&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Given&lt;/strong&gt; a consumed &lt;code&gt;tournament_enter_requested&lt;/code&gt; with valid &lt;code&gt;entryId&lt;/code&gt;, &lt;code&gt;userId&lt;/code&gt;, and &lt;code&gt;fee&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When&lt;/strong&gt; the Payment service runs &lt;code&gt;Payment.debitPayment&lt;/code&gt; and the provider rejects the debit (for example insufficient balance)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Then&lt;/strong&gt; &lt;code&gt;payment_failure&lt;/code&gt; is published for that &lt;code&gt;entryId&lt;/code&gt; (including &lt;code&gt;reason&lt;/code&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Scenario: idempotent retry&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Given&lt;/strong&gt; a successful debit was already recorded for &lt;code&gt;entryId&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When&lt;/strong&gt; the same &lt;code&gt;tournament_enter_requested&lt;/code&gt; is delivered again&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Then&lt;/strong&gt; the service does not double-charge and does not publish a second &lt;code&gt;payment_success&lt;/code&gt; for the same logical debit&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;PAY-1 is one slice among many on the board. &lt;strong&gt;§3&lt;/strong&gt; shows how tournament entry runs at runtime.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Tournament entry at runtime: Saga and CQRS
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Part II&lt;/strong&gt; — implement what the board drew for tournament entry: async fan-out, &lt;strong&gt;Saga&lt;/strong&gt; orchestration, and &lt;strong&gt;CQRS&lt;/strong&gt;-style separation between accepting the request and querying status.&lt;/p&gt;

&lt;p&gt;The swimlanes already showed parallel work after one initiating event; the failure tree already implied refunds and rejection. Architecture did not replace the model—it implemented it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why tournament entry needs async orchestration
&lt;/h3&gt;

&lt;p&gt;In a monolith, fee, rank check, and baseline might share one database &lt;strong&gt;transaction&lt;/strong&gt;. Split into microservices—and external payment/game APIs—that pattern breaks: &lt;strong&gt;partial failure&lt;/strong&gt; (payment succeeds, rank fails), &lt;strong&gt;no rollback&lt;/strong&gt; on external charges, &lt;strong&gt;long or fragile&lt;/strong&gt; sync HTTP chains from the Tournament Entry API, and &lt;strong&gt;unclear recovery&lt;/strong&gt; if the caller crashes mid-flow.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Operation&lt;/th&gt;
&lt;th&gt;Owning area&lt;/th&gt;
&lt;th&gt;Typical failure&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Payment&lt;/td&gt;
&lt;td&gt;Team B&lt;/td&gt;
&lt;td&gt;Insufficient balance, provider error&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rank verification&lt;/td&gt;
&lt;td&gt;Team A / game API&lt;/td&gt;
&lt;td&gt;Rank mismatch, API unavailable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Baselining&lt;/td&gt;
&lt;td&gt;Team A / game API&lt;/td&gt;
&lt;td&gt;Timeout, missing data&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Those constraints are why &lt;strong&gt;tournament entry&lt;/strong&gt; maps to &lt;strong&gt;Saga + events&lt;/strong&gt; in this section.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fan-out processing with Kafka
&lt;/h3&gt;

&lt;p&gt;The Tournament Entry API creates a pending tournament entry and publishes a single event—the same &lt;code&gt;tournament_enter_requested&lt;/code&gt; the Payment task subscribes to. Multiple services consume it independently.&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TD
  E[tournament_enter_requested] --&amp;gt; K1[Kafka]

  K1 --&amp;gt; PS[Payment Service]
  K1 --&amp;gt; RS[Rank Service]
  K1 --&amp;gt; BS[Baseline Service]

  PS --&amp;gt; PR[Payment Result]
  RS --&amp;gt; RR[Rank Result]
  BS --&amp;gt; BR[Baseline Result]

  PR --&amp;gt; K2[Kafka]
  RR --&amp;gt; K2
  BR --&amp;gt; K2

  K2 --&amp;gt; SO[Saga Orchestrator]&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;This is a &lt;strong&gt;fan-out/fan-in pattern&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fan-out
&lt;/h3&gt;

&lt;p&gt;A single event is distributed to multiple independent consumers.&lt;/p&gt;

&lt;p&gt;Each service handles its own responsibility:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The Payment Service processes the entry fee.&lt;/li&gt;
&lt;li&gt;The Rank Service verifies player eligibility.&lt;/li&gt;
&lt;li&gt;The Baseline Service captures the player's initial statistics.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each service uses its own Kafka consumer group so that it receives the event independently.&lt;/p&gt;

&lt;p&gt;The operations don't need to complete in a particular order.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fan-in
&lt;/h3&gt;

&lt;p&gt;After processing, each service publishes a result event.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kr"&gt;interface&lt;/span&gt; &lt;span class="nx"&gt;payment_success&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;entryId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;transactionId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kr"&gt;interface&lt;/span&gt; &lt;span class="nx"&gt;game_rank_verified&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;entryId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;rank&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kr"&gt;interface&lt;/span&gt; &lt;span class="nx"&gt;baseline_captured&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;entryId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;baselineId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These are simplified event payloads. A production implementation would typically include event identifiers, timestamps, correlation information, and schema versions.&lt;/p&gt;

&lt;p&gt;The Saga orchestrator consumes the result events and updates the status of each operation.&lt;/p&gt;

&lt;p&gt;Kafka provides asynchronous communication, but it doesn't determine whether the tournament entry should be confirmed or rejected.&lt;/p&gt;

&lt;p&gt;That decision belongs to the Saga orchestrator.&lt;/p&gt;

&lt;h3&gt;
  
  
  Saga orchestration with status aggregation
&lt;/h3&gt;

&lt;p&gt;The Saga Pattern coordinates a distributed business transaction through multiple local operations and compensating actions.&lt;/p&gt;

&lt;p&gt;There are two common approaches:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Choreography:&lt;/strong&gt; Services react to each other's events without a central workflow coordinator.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Orchestration:&lt;/strong&gt; A central coordinator maintains workflow state and determines the overall outcome, including any required compensation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We'll use orchestration, but rather than executing every step sequentially, the orchestrator aggregates the results of independently running operations.&lt;/p&gt;

&lt;h3&gt;
  
  
  Tracking individual operation statuses
&lt;/h3&gt;

&lt;p&gt;Each operation has its own status.&lt;/p&gt;

&lt;p&gt;A simplified Saga state might look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;StepStatus&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;PENDING&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;SUCCESS&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;FAILED&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;EntryStatus&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;PENDING&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;CONFIRMED&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;COMPENSATING&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;REJECTED&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;REQUIRES_ATTENTION&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kr"&gt;interface&lt;/span&gt; &lt;span class="nx"&gt;TournamentEntrySaga&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;entryId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="nl"&gt;payment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;StepStatus&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;rankVerification&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;StepStatus&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;baselining&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;StepStatus&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="nl"&gt;refundStatus&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="nx"&gt;StepStatus&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="nl"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;EntryStatus&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the tournament entry is created, all operations start as &lt;code&gt;PENDING&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"entryId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"entry-123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"payment"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PENDING"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"rankVerification"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PENDING"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"baselining"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PENDING"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PENDING"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;As events arrive, the orchestrator updates the corresponding statuses.&lt;/p&gt;

&lt;p&gt;For example, after receiving &lt;code&gt;payment_success&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"entryId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"entry-123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"payment"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"SUCCESS"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"rankVerification"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PENDING"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"baselining"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PENDING"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PENDING"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When all operations succeed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"entryId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"entry-123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"payment"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"SUCCESS"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"rankVerification"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"SUCCESS"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"baselining"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"SUCCESS"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"CONFIRMED"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The main status is determined by the combined results rather than by any single service.&lt;/p&gt;

&lt;h3&gt;
  
  
  Evaluating the overall status
&lt;/h3&gt;

&lt;p&gt;A simplified evaluation function might look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;evaluateEntryStatus&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nx"&gt;saga&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;TournamentEntrySaga&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nx"&gt;EntryStatus&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;steps&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="nx"&gt;saga&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;payment&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nx"&gt;saga&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;rankVerification&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nx"&gt;saga&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;baselining&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;];&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;every&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;SUCCESS&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;CONFIRMED&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;FAILED&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;saga&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;payment&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;SUCCESS&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;saga&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;refundStatus&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;SUCCESS&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;REJECTED&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;

      &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;COMPENSATING&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;saga&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;payment&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;PENDING&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;PENDING&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;REJECTED&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;PENDING&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This illustrates the basic aggregation rules. A production implementation would also distinguish retryable technical failures from definitive business failures and handle refund initiation separately.&lt;/p&gt;

&lt;p&gt;The important point is that &lt;strong&gt;the orchestrator evaluates the overall state whenever an operation result arrives&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;It doesn't assume events will arrive in a particular order.&lt;/p&gt;

&lt;h3&gt;
  
  
  Handling out-of-order events and compensation
&lt;/h3&gt;

&lt;p&gt;Independent services can complete at different times.&lt;/p&gt;

&lt;p&gt;Suppose rank verification fails before the Payment Service publishes its result.&lt;/p&gt;

&lt;p&gt;The events might arrive in this order:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. game_rank_verification_failed
2. baseline_captured
3. payment_success
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the first event arrives, the orchestrator knows the tournament entry cannot be confirmed.&lt;/p&gt;

&lt;p&gt;However, it doesn't yet know whether payment has succeeded.&lt;/p&gt;

&lt;p&gt;It cannot simply reject the tournament entry and forget about the remaining operations.&lt;/p&gt;

&lt;p&gt;When &lt;code&gt;payment_success&lt;/code&gt; eventually arrives, the orchestrator must recognise that the tournament entry is already ineligible and initiate a refund.&lt;/p&gt;

&lt;p&gt;This is why maintaining individual operation statuses is important.&lt;/p&gt;

&lt;h3&gt;
  
  
  Compensating the payment
&lt;/h3&gt;

&lt;p&gt;Once the orchestrator knows that a tournament entry has failed and payment succeeded, it requests a refund.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;game_rank_verification_failed
           |
           v
   Tournament entry cannot be confirmed
           |
           v
     payment_success
           |
           v
   Payment.refundPayment
           |
           v
      payment_refunded
           |
           v
   tournament_entry_rejected
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The refund is not a database rollback.&lt;/p&gt;

&lt;p&gt;It is a separate business transaction that compensates for the successful payment.&lt;/p&gt;

&lt;p&gt;The orchestrator should wait for the refund result before considering the failed tournament entry fully resolved.&lt;/p&gt;

&lt;h3&gt;
  
  
  What if the refund fails?
&lt;/h3&gt;

&lt;p&gt;Compensating transactions can fail just like normal operations.&lt;/p&gt;

&lt;p&gt;For example, the payment provider might be temporarily unavailable, or a refund request might time out.&lt;/p&gt;

&lt;p&gt;In that situation, the orchestrator should retain enough state to retry the refund safely.&lt;/p&gt;

&lt;p&gt;If automatic recovery is exhausted, the Saga may need to enter a state such as &lt;code&gt;REQUIRES_ATTENTION&lt;/code&gt; for operational intervention.&lt;/p&gt;

&lt;p&gt;The key principle is that &lt;strong&gt;a failed business workflow must not silently leave a successful payment uncompensated&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Handling duplicate events
&lt;/h3&gt;

&lt;p&gt;Kafka consumers can process the same message more than once.&lt;/p&gt;

&lt;p&gt;For example, if a consumer processes an event but crashes before committing its offset, the event may be delivered again.&lt;/p&gt;

&lt;p&gt;Without idempotency, duplicate processing could trigger repeated operations or refunds.&lt;/p&gt;

&lt;p&gt;A common approach is to assign each event a unique identifier and track processed events.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kr"&gt;interface&lt;/span&gt; &lt;span class="nx"&gt;DomainEvent&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;eventId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;entryId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;T&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When handling an event, the orchestrator can check whether it has already been processed.&lt;/p&gt;

&lt;p&gt;State transitions should also be protected against concurrent updates. A duplicate event must not cause a completed Saga to return to a pending state or issue another refund.&lt;/p&gt;

&lt;p&gt;For payment and refund operations, stable idempotency keys are especially important.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;refundIdempotencyKey&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="s2"&gt;`entry:&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;entryId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:refund:&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;transactionId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;All retries of the same logical refund should use the same key.&lt;/p&gt;

&lt;h3&gt;
  
  
  Persisting state reliably
&lt;/h3&gt;

&lt;p&gt;The Saga's state needs to survive service restarts.&lt;/p&gt;

&lt;p&gt;The orchestrator must be able to recover the current operation statuses and determine whether compensation is still required.&lt;/p&gt;

&lt;p&gt;Another important consideration is the consistency between database updates and Kafka message publication.&lt;/p&gt;

&lt;p&gt;For example, if the orchestrator records that compensation is required but crashes before publishing the refund command, the system needs a recovery mechanism.&lt;/p&gt;

&lt;p&gt;The Transactional Outbox Pattern is one possible solution. It stores the state change and outgoing message in the same database transaction, then publishes the message asynchronously.&lt;/p&gt;

&lt;p&gt;This is an optional implementation technique rather than an inherent part of the Saga Pattern, but it addresses an important reliability concern.&lt;/p&gt;

&lt;h3&gt;
  
  
  CQRS: exposing tournament entry status to the frontend
&lt;/h3&gt;

&lt;p&gt;The backend workflow is asynchronous, but the user still needs immediate feedback.&lt;/p&gt;

&lt;p&gt;Rather than keeping the HTTP request open until payment, rank verification, and baselining finish, the API can create a pending tournament entry and return immediately.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;POST /tournament-entries

&lt;/span&gt;&lt;span class="k"&gt;HTTP&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="m"&gt;1.1&lt;/span&gt; &lt;span class="m"&gt;201&lt;/span&gt; &lt;span class="ne"&gt;Created&lt;/span&gt;
&lt;span class="na"&gt;Content-Type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;application/json&lt;/span&gt;

&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"entryId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"entry-123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PENDING"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The frontend then polls tournament entry status.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;GET /tournament-entries/entry-123
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;While the Saga is running:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"entryId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"entry-123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PENDING"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once all operations succeed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"entryId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"entry-123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"CONFIRMED"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a required operation fails and any necessary compensation completes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"entryId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"entry-123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"REJECTED"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is an example of eventual consistency.&lt;/p&gt;

&lt;p&gt;The initial request creates a tournament entry, but confirmation happens later when the distributed workflow reaches a final outcome.&lt;/p&gt;

&lt;p&gt;This is the &lt;strong&gt;read model&lt;/strong&gt; from step 5 in concrete form: &lt;strong&gt;commands and events&lt;/strong&gt; advance the workflow; &lt;strong&gt;&lt;code&gt;entry_status&lt;/code&gt;&lt;/strong&gt; (and the GET API) are what the UI polls while the Saga runs. We did not split write and read databases for every field—the important decision is &lt;strong&gt;request acceptance vs business completion&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;201 Created&lt;/code&gt; is appropriate when a tournament entry resource has been created in a pending state.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Lessons learned
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Part III&lt;/strong&gt; — reflections from the case study above.&lt;/p&gt;

&lt;h3&gt;
  
  
  The model, tasks, and architecture should stay aligned
&lt;/h3&gt;

&lt;p&gt;When swimlanes match service ownership and tasks match publish/subscribe boundaries, the Kafka topology and orchestrator responsibilities largely fall out of the board. Payment's task diagram and the fan-out diagram describe the same edges with different detail. When the model and the code drift apart, the cost shows up in integration tests and cross-team PRs—not in the tutorial diagrams.&lt;/p&gt;

&lt;h3&gt;
  
  
  Event Modeling makes failure scenarios visible
&lt;/h3&gt;

&lt;p&gt;A successful workflow is usually easy to describe.&lt;/p&gt;

&lt;p&gt;The difficult part is identifying what happens when one operation fails while others are still running.&lt;/p&gt;

&lt;p&gt;Event Modeling helps expose these scenarios before implementation, making it easier to define compensation requirements and state transitions.&lt;/p&gt;

&lt;h3&gt;
  
  
  Independent operations require careful coordination
&lt;/h3&gt;

&lt;p&gt;Fan-out processing allows multiple services to work independently, potentially reducing overall processing time.&lt;/p&gt;

&lt;p&gt;However, it also means result events can arrive in any order.&lt;/p&gt;

&lt;p&gt;The orchestrator must combine operation statuses without assuming a particular completion sequence.&lt;/p&gt;

&lt;h3&gt;
  
  
  Saga orchestration is more than sequencing commands
&lt;/h3&gt;

&lt;p&gt;An orchestrator doesn't necessarily need to execute every operation one after another.&lt;/p&gt;

&lt;p&gt;In this example, its primary responsibilities are maintaining workflow state, evaluating the combined results, and coordinating compensation.&lt;/p&gt;

&lt;p&gt;The Saga's correctness depends on how these responsibilities are implemented, not simply on having a central service.&lt;/p&gt;

&lt;h3&gt;
  
  
  Compensation is a business operation
&lt;/h3&gt;

&lt;p&gt;A refund is not equivalent to rolling back a database transaction.&lt;/p&gt;

&lt;p&gt;It can fail, require retries, and need operational intervention.&lt;/p&gt;

&lt;p&gt;Compensating transactions deserve the same attention to reliability and idempotency as the original operations.&lt;/p&gt;

&lt;h3&gt;
  
  
  Kafka doesn't guarantee business consistency
&lt;/h3&gt;

&lt;p&gt;Kafka provides reliable event transport and supports asynchronous processing, but it doesn't automatically solve duplicate processing, distributed transactions, or business-level consistency.&lt;/p&gt;

&lt;p&gt;Those concerns must be addressed in the application design.&lt;/p&gt;

&lt;h3&gt;
  
  
  Eventual consistency is also a frontend concern
&lt;/h3&gt;

&lt;p&gt;The frontend must understand that an accepted request isn't necessarily a completed business operation.&lt;/p&gt;

&lt;p&gt;Pending states, status polling, meaningful error messages, and recovery from long-running operations are all part of the user experience.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;We Event-Modeled the &lt;strong&gt;tournament product&lt;/strong&gt; to &lt;strong&gt;scale delivery&lt;/strong&gt; across domain teams; new features can often attach to events already on the board without replatforming tournament entry.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;§2:&lt;/strong&gt; the seven steps on &lt;strong&gt;tournament entry&lt;/strong&gt; (board, swimlanes, PAY-1). &lt;strong&gt;§3:&lt;/strong&gt; the same slice as &lt;strong&gt;Saga&lt;/strong&gt; and &lt;strong&gt;CQRS&lt;/strong&gt;-style reads at runtime.&lt;/p&gt;

&lt;p&gt;Reliable tournament entry still needs persisted saga state, idempotent handlers, and a frontend that treats acceptance and confirmation as different states.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The most important design choice is not the happy path. It is what must happen when only part of the workflow succeeds—and making sure the board, the backlog, and the runtime all tell that story the same way.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>backend</category>
      <category>softwareengineering</category>
      <category>systemdesign</category>
    </item>
    <item>
      <title>Migrate AMQ to Kafka</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Tue, 27 Oct 2026 02:41:16 +0000</pubDate>
      <link>https://dev.to/joungpark/migrate-amq-to-kafka-2j9e</link>
      <guid>https://dev.to/joungpark/migrate-amq-to-kafka-2j9e</guid>
      <description>&lt;p&gt;As our system moved from a monolith towards microservices, services needed a way to communicate without depending on one another directly. We used ActiveMQ, managed through Amazon MQ, for that messaging. Later, we moved to Kafka on Amazon MSK.&lt;/p&gt;

&lt;p&gt;This was not simply a change of hosting provider. ActiveMQ and Kafka have different messaging models, so the migration made us revisit how we publish, consume, and operate events. Here is why we made the move, what changed, and what I would keep in mind when planning a similar migration.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why move from ActiveMQ to Kafka?
&lt;/h2&gt;

&lt;p&gt;Amazon MQ gave us a managed ActiveMQ broker and a familiar message-broker model. That was a reasonable starting point: services could send messages through a broker instead of calling each other directly.&lt;/p&gt;

&lt;p&gt;As we moved further towards event-driven microservices, Kafka became a better fit for the kind of communication we wanted to build around. Kafka stores records in durable, partitioned logs. Consumers read those records at their own pace, and separate consumer groups can each process the same stream independently.&lt;/p&gt;

&lt;p&gt;That model is useful when an event should be available to more than one downstream service, or when we want consumers to be able to catch up by reading retained events. It also gives us a common platform for event-driven workflows and stream processing.&lt;/p&gt;

&lt;p&gt;The key point is that Kafka was not automatically better for every message. We chose it because its log-and-consumer model fit the direction of the architecture. A traditional broker can still be the right choice for workloads that need its queueing and delivery semantics.&lt;/p&gt;

&lt;h2&gt;
  
  
  The important difference: queue versus log
&lt;/h2&gt;

&lt;p&gt;The biggest conceptual change was moving from thinking of messages as work handed to a broker queue to thinking of events as records in a log.&lt;/p&gt;

&lt;p&gt;With a queue, a message is typically delivered to a consumer, acknowledged, and removed from the work waiting to be processed. With Kafka, a record remains in a topic according to its retention policy. Each consumer group tracks its own position, so multiple groups can independently process the same records.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ActiveMQ queue
Producer ──► Queue ──► Consumer

Kafka topic
Producer ──► Topic (retained log) ──► Consumer group A
                              └─────► Consumer group B
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This changes what consumers are responsible for. A Kafka consumer tracks offsets and must handle the possibility of seeing a record again, for example after a restart or a failure before its progress is committed. In practice, consumers need to be designed for safe retries and duplicate delivery; moving to Kafka does not make processing exactly-once by default.&lt;/p&gt;

&lt;p&gt;Kafka also orders records within a partition, not across an entire topic. If related events need to be processed in order, the producer needs to use a consistent key so those events are routed to the same partition. Ordering is a design decision, not something to assume globally.&lt;/p&gt;

&lt;h2&gt;
  
  
  Considerations when moving from ActiveMQ to Kafka
&lt;/h2&gt;

&lt;p&gt;Two requirements shaped our migration: &lt;strong&gt;keep the live system running without downtime&lt;/strong&gt; and &lt;strong&gt;preserve the ordering our scoring logic depends on&lt;/strong&gt;. Changing the broker was only useful if we could maintain both.&lt;/p&gt;

&lt;h3&gt;
  
  
  Migrating a live system without downtime
&lt;/h3&gt;

&lt;p&gt;Our system was already live, so we needed to move traffic without interrupting message processing. We brought Kafka online, switched the producer over, and kept the ActiveMQ listener available during the monitoring and rollback period:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Create the Kafka topic&lt;/strong&gt; with the required configuration.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Start the Kafka listener&lt;/strong&gt; and confirm it is healthy before sending it production messages.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stop publishing to ActiveMQ and start publishing to Kafka.&lt;/strong&gt; This switches new messages to the new path without intentionally sending each message to both brokers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Monitor the ActiveMQ queue&lt;/strong&gt; to confirm no new messages are arriving and that any messages already queued are draining.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep the ActiveMQ listener running while monitoring Kafka.&lt;/strong&gt; If an issue appears during the cutover, switch publishing back to ActiveMQ while investigating and debugging the Kafka path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stop the ActiveMQ listener&lt;/strong&gt; only after the Kafka path has been verified and the rollback window has passed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Continue monitoring Kafka.&lt;/strong&gt; Once the new path is stable, remove the old ActiveMQ queue and publishing/listener logic.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Keeping the ActiveMQ listener available gives us a way to back out while the old path is still available. Before stopping it, confirm that the old queue has drained and no messages remain in flight. Monitor Kafka consumer lag, processing errors, and successful downstream score updates—not just whether the listener is running. Growing lag can mean scores are delayed even when the brokers appear healthy.&lt;/p&gt;

&lt;h4&gt;
  
  
  Tip: keep message handling separate from business logic
&lt;/h4&gt;

&lt;p&gt;One thing that made this migration easier was separating the &lt;strong&gt;service layer&lt;/strong&gt; from the &lt;strong&gt;controller/presentation layer&lt;/strong&gt;. Our scoring logic lived in the service layer, while the message listener acted as an entry point that translated an incoming message into a service call.&lt;/p&gt;

&lt;p&gt;Instead of changing the scoring service, we added a Kafka listener alongside the existing ActiveMQ listener. Both called the same service:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ActiveMQ listener ──┐
                    ├──► Scoring service
Kafka listener ─────┘
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This kept the migration focused on the messaging boundary:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Controller/presentation layer:&lt;/strong&gt; receive and translate messages, and handle broker-specific acknowledgements or offset commits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Service layer:&lt;/strong&gt; apply the scoring rules without depending on whether the request came from ActiveMQ or Kafka.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We could deploy the Kafka entry point before switching publishers, keep the ActiveMQ entry point available for rollback, and remove it after the cutover without rewriting the scoring logic. The practical tip is to &lt;strong&gt;add a new entry point, not a second copy of the business logic&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Ordering and grouping messages
&lt;/h3&gt;

&lt;p&gt;Our scoring flow needs related messages to be processed in strict order. A delta between snapshots needs the previous state, and a win streak depends on the order of outcomes. Processing those messages concurrently or out of order could cause race conditions and inconsistent score calculations.&lt;/p&gt;

&lt;p&gt;The grouping key must identify the &lt;strong&gt;scoring subject&lt;/strong&gt;—for example, a player ID, match ID, or another stable identifier for the state being calculated. We used:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;ActiveMQ:&lt;/strong&gt; &lt;code&gt;JMSXGroupID&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Kafka:&lt;/strong&gt; the message key&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The technology changed, but the reason for grouping did not: all events contributing to the same score must be processed sequentially.&lt;/p&gt;

&lt;h4&gt;
  
  
  How it works under the hood
&lt;/h4&gt;

&lt;h5&gt;
  
  
  ActiveMQ (&lt;code&gt;JMSXGroupID&lt;/code&gt;)
&lt;/h5&gt;

&lt;p&gt;&lt;code&gt;JMSXGroupID&lt;/code&gt; is a message property that enables JMS message grouping. The broker keeps messages with the same group ID associated with one consumer at a time and delivers them in order within that group.&lt;/p&gt;

&lt;p&gt;The broker maintains the group-to-consumer association. If a consumer disconnects, the group can be reassigned. This provides consumer affinity for a group, while allowing different groups to be processed by different consumers.&lt;/p&gt;

&lt;h5&gt;
  
  
  Kafka (message key)
&lt;/h5&gt;

&lt;p&gt;The message key determines which partition receives a record. With the same partitioning strategy and partition count, records with the same key are routed to the same partition, where Kafka preserves their order:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AMQ:    JMSXGroupID = player-123  ──► same consumer at a time, in order
Kafka:  key = player-123          ──► same partition, in order
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Within a Kafka consumer group, each partition is assigned to only one consumer at a time. A rebalance can assign that partition to a different consumer, so Kafka provides partition-level ordering rather than permanent key-to-consumer affinity.&lt;/p&gt;

&lt;h4&gt;
  
  
  Implications
&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ordering is scoped differently:&lt;/strong&gt; AMQ groups messages by consumer affinity; Kafka orders records within a partition.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Parallelism is across groups or partitions:&lt;/strong&gt; one group is processed sequentially, while separate groups or partitions can be processed concurrently.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hot keys can limit throughput:&lt;/strong&gt; a busy scoring subject can become a hotspot. In Kafka, adding consumers does not increase parallelism beyond the number of partitions, and uneven key distribution can create hot partitions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Choose the key from the state being updated:&lt;/strong&gt; use the same stable scoring-subject identifier for all events that contribute to that calculation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Preserve order in the handler too:&lt;/strong&gt; do not process events for the same scoring subject concurrently. Partition ordering does not prevent application code from completing work out of order.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Design retries for stateful processing:&lt;/strong&gt; duplicate delivery must not apply a score change twice. A failed event also needs an explicit recovery path; skipping it and processing the next snapshot or outcome could produce the wrong score.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The cutover needs the same care. An older event still being processed from ActiveMQ could finish after a newer event from Kafka. Keeping the same scoring identifier in both systems does not create ordering across the two brokers. Before allowing newer events to update a subject's score, ensure its earlier ActiveMQ work has completed. The same concern applies when switching publishing back during rollback.&lt;/p&gt;

&lt;p&gt;If the Kafka message key is not the right fit, alternatives depend on the required guarantee:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Use a separate topic&lt;/strong&gt; when the message flow needs independent scaling, retention, or failure handling—not just because it has a different message type.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Choose an explicit partition&lt;/strong&gt; only when fixed placement is required and the coupling to the topic's partition layout is acceptable. A stable key is generally easier to maintain as the topic scales.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Handle ordering in the application&lt;/strong&gt; with sequence numbers or version checks when strict broker-level ordering is not required. Consumers must then detect and deal with out-of-order or duplicate events.&lt;/li&gt;
&lt;/ul&gt;

&lt;h4&gt;
  
  
  Key takeaway
&lt;/h4&gt;

&lt;p&gt;In both systems, correctness depends on choosing a stable grouping identifier that matches the state being calculated. In our scoring flow, we used &lt;code&gt;JMSXGroupID&lt;/code&gt; with ActiveMQ and the same scoring-subject value as the Kafka message key, so events for that subject stay ordered.&lt;/p&gt;

&lt;h2&gt;
  
  
  The outcome
&lt;/h2&gt;

&lt;p&gt;Moving from ActiveMQ on Amazon MQ to Kafka on Amazon MSK was part of a broader shift in how our services communicate. Kafka gave us a durable event log that multiple consumer groups can read independently, which fits the event-driven architecture we were moving towards.&lt;/p&gt;

&lt;p&gt;The two requirements stayed the same throughout: keep processing live traffic and keep score calculations correct. Separating the listeners from the service layer let us change the messaging entry point without rewriting the scoring logic. Keeping the old path available gave us a rollback option, while choosing the scoring subject as the Kafka key carried our grouping requirement into the new system.&lt;/p&gt;

&lt;p&gt;The main lesson for me is that a messaging migration is not just a broker change. It is about preserving the behavior the application depends on while changing how messages reach it.&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>aws</category>
      <category>backend</category>
      <category>microservices</category>
    </item>
    <item>
      <title>Step-Up Verification: OTP, Magic Link</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Mon, 26 Oct 2026 04:50:35 +0000</pubDate>
      <link>https://dev.to/joungpark/step-up-verification-otp-magic-link-20p3</link>
      <guid>https://dev.to/joungpark/step-up-verification-otp-magic-link-20p3</guid>
      <description>&lt;p&gt;We already use &lt;strong&gt;Firebase Authentication&lt;/strong&gt; for primary login—ID tokens on the client, verified on the backend with the Admin SDK. That answers &lt;strong&gt;who the user is&lt;/strong&gt;. It does not always prove they &lt;strong&gt;still control the account&lt;/strong&gt; at the moment they confirm a payment, change payout details, or export sensitive data. &lt;strong&gt;Step-up verification&lt;/strong&gt;—email &lt;strong&gt;one-time passwords (OTP)&lt;/strong&gt; or magic links—closes that gap.&lt;/p&gt;

&lt;p&gt;Step-up does not replace Firebase. Users keep sending the same ID token; sensitive routes additionally require recent proof via email. Because Firebase is already in the stack, &lt;a href="https://firebase.google.com/docs/auth/web/email-link-auth" rel="noopener noreferrer"&gt;Firebase email-link sign-in&lt;/a&gt; was the obvious first idea. This post compares that shortcut with a &lt;strong&gt;custom OTP&lt;/strong&gt; service layered on the same auth.&lt;/p&gt;

&lt;h2&gt;
  
  
  Primary login vs step-up
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Responsibility&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Firebase (existing)&lt;/td&gt;
&lt;td&gt;Sign-in, session, ID token, &lt;code&gt;firebaseUid&lt;/code&gt; → internal user&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Step-up (new)&lt;/td&gt;
&lt;td&gt;Short-lived proof before payments, billing changes, exports, etc.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The backend pattern stays familiar: verify the Firebase token first, resolve the user, then—for protected actions—check a step-up credential or reject with &lt;code&gt;403&lt;/code&gt; and prompt for verification.&lt;/p&gt;

&lt;h2&gt;
  
  
  When step-up matters
&lt;/h2&gt;

&lt;p&gt;Typical triggers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Confirming or changing &lt;strong&gt;payment&lt;/strong&gt; methods and billing settings&lt;/li&gt;
&lt;li&gt;Approving a &lt;strong&gt;transfer or purchase&lt;/strong&gt; above a threshold&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Exporting&lt;/strong&gt; or deleting account-held data&lt;/li&gt;
&lt;li&gt;Any action where a stolen session should not be enough&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For those flows the API usually needs two answers:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Is the caller authenticated (session or ID token)?&lt;/li&gt;
&lt;li&gt;Have they &lt;strong&gt;recently&lt;/strong&gt; verified via a channel you control (email OTP or equivalent)?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The shortcut we tried first: trigger Firebase email-link sign-in, then read &lt;strong&gt;last sign-in time&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;sign_in_provider&lt;/code&gt;&lt;/strong&gt; from the ID token or Admin SDK user record. If the provider indicated email link and sign-in fell within a window (for example 15 minutes), allow the action—no new OTP tables or send endpoints.&lt;/p&gt;

&lt;p&gt;That only works—&lt;strong&gt;if&lt;/strong&gt; the metadata matches what the user actually did.&lt;/p&gt;

&lt;h2&gt;
  
  
  Options considered
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Firebase email-link step-up&lt;/th&gt;
&lt;th&gt;Custom OTP&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Implementation&lt;/td&gt;
&lt;td&gt;Reuse Firebase send/link APIs + session metadata&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;POST /otp/send&lt;/code&gt;, &lt;code&gt;POST /otp/verify&lt;/code&gt;, DB, email&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;User steps&lt;/td&gt;
&lt;td&gt;Open email, tap link (device matters)&lt;/td&gt;
&lt;td&gt;Open email, type or paste code (any device)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Proof on backend&lt;/td&gt;
&lt;td&gt;Last login time + &lt;code&gt;sign_in_provider&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;OTP record + short-lived step-up token&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ops burden&lt;/td&gt;
&lt;td&gt;Low if metadata is trustworthy&lt;/td&gt;
&lt;td&gt;You own security, delivery, and abuse controls&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Security concerns
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Firebase email-link step-up
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Weak backend proof&lt;/strong&gt; — If you gate on &lt;code&gt;sign_in_provider&lt;/code&gt; and last login, a normal password sign-in can look like step-up; policy mistakes can skip re-verification entirely.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Link as bearer secret&lt;/strong&gt; — Anyone with the magic link can consume it; forwarded email or a shared inbox completes the flow.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Full sign-in side effects&lt;/strong&gt; — Email link is real authentication, not a scoped step-up token—sessions refresh, audit trails mix login with “confirm payment,” and revocation semantics are coarser.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Device binding&lt;/strong&gt; — Links opened on the wrong device can strand or confuse users; support pressure rises before security does.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Custom OTP
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Your threat model&lt;/strong&gt; — Brute force, credential stuffing on &lt;code&gt;/otp/verify&lt;/code&gt;, and OTP spraying via &lt;code&gt;/otp/send&lt;/code&gt; need rate limits, lockouts, and monitoring you implement and operate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Storage and logs&lt;/strong&gt; — Plaintext OTPs in the database or logs are a common failure mode; hash at rest and never log codes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Email channel&lt;/strong&gt; — Same mailbox risks as a magic link; a short numeric code can be easier to phish or shoulder-surf if UX is careless.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Step-up token lifecycle&lt;/strong&gt; — Issue tight TTL, bind to user (and optionally device/session), and reject replay; otherwise a verified OTP window becomes a long-lived bypass.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Option 1: Firebase sign-in via email link
&lt;/h2&gt;

&lt;p&gt;Firebase documents this flow as passwordless email-link authentication on the web—see &lt;a href="https://firebase.google.com/docs/auth/web/email-link-auth" rel="noopener noreferrer"&gt;Authenticate with Firebase Using Email Link&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why it is attractive
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Reuse existing Firebase auth&lt;/strong&gt; — same project, Admin SDK, and email patterns as primary login.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No OTP product to build&lt;/strong&gt; — Firebase generates and validates the link; you focus on when to require step-up.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deep links&lt;/strong&gt; — mobile clients can resume in-app after the user taps the link.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Continue URLs / query params&lt;/strong&gt; — return the user to the payment or settings screen that triggered verification.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Where it breaks down
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cross-device UX&lt;/strong&gt; — magic links often complete on the device that opened the email. Starting checkout on phone and opening mail on a laptop breaks the flow unless you invest in handoff.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Session edge cases&lt;/strong&gt; — expired links, stale tabs, and multiple active sessions need explicit testing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;POC friction&lt;/strong&gt; — a quick spike often surfaces integration details that are not obvious from the docs; Firebase handles sign-in, not necessarily your step-up policy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The blocker: &lt;code&gt;sign_in_provider&lt;/code&gt;&lt;/strong&gt; — after email-link sign-in, Firebase reports &lt;code&gt;sign_in_provider&lt;/code&gt; as &lt;strong&gt;&lt;code&gt;password&lt;/code&gt;&lt;/strong&gt;, not a distinct email-link provider. You cannot tell “just completed email link step-up” from “signed in with email/password” using provider alone. &lt;strong&gt;Last login time + provider is an unreliable gate&lt;/strong&gt; before a payment or similar action.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Unless you add another durable signal (custom claims from a Cloud Function on link completion, a dedicated step-up flag, etc.), Option 1 does not give the backend clean, auditable step-up proof.&lt;/p&gt;

&lt;h2&gt;
  
  
  Option 2: Custom OTP
&lt;/h2&gt;

&lt;p&gt;Firebase stays primary auth. Step-up is a &lt;strong&gt;narrow API&lt;/strong&gt; beside it: send a code to the email on the Firebase user (or your internal profile), verify it, issue a &lt;strong&gt;short-lived step-up token&lt;/strong&gt; (or a server-side “verified until” timestamp keyed by user id).&lt;/p&gt;

&lt;h3&gt;
  
  
  Pros
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Explicit proof&lt;/strong&gt; — the backend knows an OTP was sent, verified, consumed, and when.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;UX control&lt;/strong&gt; — codes work when email is read on another device; resend, cooldown, and validation UI are yours.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One pattern, many gates&lt;/strong&gt; — the same flow can guard payments, billing changes, and exports.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No Firebase metadata quirks&lt;/strong&gt; — step-up does not depend on &lt;code&gt;sign_in_provider&lt;/code&gt; semantics.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Cons
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Build and maintain&lt;/strong&gt; — generation, hashing, expiry, single-use, templates, monitoring (see security notes above).&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flow
&lt;/h3&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;sequenceDiagram
    participant U as User
    participant C as Web / Mobile Client
    participant A as Backend API
    participant D as Database
    participant E as Email Service

    Note over U,E: Send OTP
    U-&amp;gt;&amp;gt;C: Start protected action (e.g. confirm payment)
    C-&amp;gt;&amp;gt;A: POST /otp/send (Firebase ID token)
    A-&amp;gt;&amp;gt;A: Verify token, resolve user
    A-&amp;gt;&amp;gt;A: Generate OTP, store hash + expiry
    A-&amp;gt;&amp;gt;E: Send email with code
    E--&amp;gt;&amp;gt;U: Email delivered
    A--&amp;gt;&amp;gt;C: 200 OK

    Note over U,E: Verify OTP
    U-&amp;gt;&amp;gt;C: Enter code
    C-&amp;gt;&amp;gt;A: POST /otp/verify (code)
    A-&amp;gt;&amp;gt;D: Load pending OTP for user
    alt Valid and not expired
        A-&amp;gt;&amp;gt;D: Mark OTP used
        A-&amp;gt;&amp;gt;A: Issue step-up token or session flag
        A--&amp;gt;&amp;gt;C: 200 OK
        C--&amp;gt;&amp;gt;U: Continue protected action
    else Invalid or expired
        A--&amp;gt;&amp;gt;C: 401 / 400
        C--&amp;gt;&amp;gt;U: Retry or resend
    end&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;In words:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Send&lt;/strong&gt; — Client calls &lt;code&gt;/otp/send&lt;/code&gt; with the usual Firebase ID token. Backend verifies it, resolves the user, stores a &lt;strong&gt;hashed&lt;/strong&gt; OTP with expiry, sends email. Never store plaintext codes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verify&lt;/strong&gt; — Client submits the code. Backend checks hash, expiry, and single-use, marks the OTP consumed, returns a &lt;strong&gt;step-up credential&lt;/strong&gt; (JWT claim, cookie, or server-side “verified until”).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Protected APIs&lt;/strong&gt; — Require a fresh step-up credential within the policy window in addition to normal auth—for example before submitting a charge or updating a payout account.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Implementation notes (custom OTP)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Hash OTPs&lt;/strong&gt; at rest; short TTL (for example 10–15 minutes); one active send per user with resend cooldown.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rate limit&lt;/strong&gt; send and verify; cap attempts per OTP before invalidation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Email&lt;/strong&gt; — state the action (“Confirm payment”, “Update billing”) so users know why they received a code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Audit&lt;/strong&gt; — log send/verify outcomes without logging the code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Clients&lt;/strong&gt; — one backend contract; web and mobile differ only in how they collect the code.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaway
&lt;/h2&gt;

&lt;p&gt;We wanted step-up without ripping out Firebase. Email-link step-up is the natural fit, but &lt;strong&gt;&lt;code&gt;sign_in_provider&lt;/code&gt; reporting &lt;code&gt;password&lt;/code&gt; breaks the simple “provider + last login” rule&lt;/strong&gt;. Patching that with custom claims or Cloud Functions adds moving parts—and still blurs sign-in with step-up.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Custom OTP&lt;/strong&gt; is more code, but it sits cleanly &lt;strong&gt;on top of&lt;/strong&gt; existing Firebase auth: same ID token at the edge, plus an explicit step-up proof the API can enforce before payments and other sensitive work.&lt;/p&gt;

&lt;p&gt;A practical rollout: ship a thin vertical slice (send + verify + one protected endpoint, such as payment confirmation) behind a feature flag, and load-test rate limits before widening coverage.&lt;/p&gt;

</description>
      <category>authentication</category>
      <category>backend</category>
      <category>security</category>
    </item>
    <item>
      <title>Share a Kafka Topic Only When Order Matters</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Mon, 19 Oct 2026 05:16:34 +0000</pubDate>
      <link>https://dev.to/joungpark/share-a-kafka-topic-only-when-order-matters-4dlp</link>
      <guid>https://dev.to/joungpark/share-a-kafka-topic-only-when-order-matters-4dlp</guid>
      <description>&lt;p&gt;We merged Kafka topics because the same services consumed the messages together. It felt simpler. Two incidents showed what we had really done: one shared topic means one shared queue, one failure domain, and one on-call problem—not a tidy folder in a repo.&lt;/p&gt;

&lt;p&gt;My default rule: &lt;strong&gt;share a topic only when order matters&lt;/strong&gt; for a clear stream (same partition key, same lifecycle). If urgency, retries, batch vs live traffic, or ownership differ, split the topics. Below I explain why we merged, what broke in production, when one topic is still right, and defaults that help teams avoid the same mistakes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why we merged topics
&lt;/h2&gt;

&lt;p&gt;We had several topics. The same services read the events. Dashboards grouped them. We used the same domain words (“orders,” “billing,” “notifications”). Merging felt reasonable:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fewer topic names and fewer ACL/schema subjects to manage&lt;/li&gt;
&lt;li&gt;One obvious place per business area&lt;/li&gt;
&lt;li&gt;One mental pipeline for producers and consumers&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We also followed a familiar rule of thumb: &lt;strong&gt;combine topics that are mostly consumed together.&lt;/strong&gt; If the same consumers normally subscribe to all of them, one topic can be simpler. Consumers can skip message types they do not handle—that is acceptable. Fetching from Kafka is cheap; ignoring even a large share of messages on a topic is often not a big cost. You only need a separate topic when consumers would ignore the &lt;em&gt;vast majority&lt;/em&gt; of messages.&lt;/p&gt;

&lt;p&gt;We put &lt;strong&gt;message type in the header&lt;/strong&gt; so each record could be parsed and routed to the right handler. That made “one topic, many processes” feel safe: separate code paths without separate topic names.&lt;/p&gt;

&lt;p&gt;That made merging feel low-risk. The incidents below showed we had still missed something.&lt;/p&gt;

&lt;h2&gt;
  
  
  What that advice does not cover
&lt;/h2&gt;

&lt;p&gt;Co-consumption answers &lt;em&gt;who reads the stream&lt;/em&gt;. It does not spell out &lt;em&gt;what guarantees the log must provide&lt;/em&gt;—ordering, retention, retries, and what happens when one message type misbehaves.&lt;/p&gt;

&lt;p&gt;You can join streams in application code, materialized views, or analytics. A Kafka topic is not a join key. It is an append-only log with contracts for ordering and retention. Headers that distinguish message type help you &lt;strong&gt;dispatch&lt;/strong&gt; inside the consumer; they do not give you separate partitions, consumer lag, or retry boundaries. When message types need different contracts, one topic still ties them together. You often only notice that under load or during failure—not when consumers happily skip or route messages in code.&lt;/p&gt;

&lt;h2&gt;
  
  
  Incident 1: Bulk traffic blocked important work
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Setup.&lt;/strong&gt; Topic &lt;code&gt;billing.events&lt;/code&gt;, partition key &lt;code&gt;accountId&lt;/code&gt;, already handled live &lt;code&gt;payment-captured&lt;/code&gt; and &lt;code&gt;invoice-issued&lt;/code&gt;. Later we added &lt;code&gt;ledger-export-row&lt;/code&gt; to the same topic so the export job could reuse the same pipeline and &lt;code&gt;accountId&lt;/code&gt; keying. One consumer group still processed every message type in partition order.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What happened.&lt;/strong&gt; After &lt;code&gt;ledger-export-row&lt;/code&gt; joined &lt;code&gt;billing.events&lt;/code&gt;, a heavy export run—or backlog after an outage—filled partitions for busy &lt;code&gt;accountId&lt;/code&gt; values. &lt;code&gt;payment-captured&lt;/code&gt; and &lt;code&gt;invoice-issued&lt;/code&gt; kept flowing as before, but new records queued behind export rows on the same partitions. The consumer could not prefer the live types that had been fine on their own; offset order now applied to &lt;code&gt;ledger-export-row&lt;/code&gt; too. Lag on &lt;code&gt;payment-captured&lt;/code&gt; and &lt;code&gt;invoice-issued&lt;/code&gt; climbed while the group chewed through export rows at a fixed rate.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Symptoms.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;End-to-end latency alerts on &lt;code&gt;payment-captured&lt;/code&gt; / &lt;code&gt;invoice-issued&lt;/code&gt; flows while &lt;code&gt;ledger-export-row&lt;/code&gt; lag was high on &lt;code&gt;billing.events&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Support reports: payments or invoices “stuck for hours” with no export job visibly “broken”&lt;/li&gt;
&lt;li&gt;Healthy brokers, growing consumer lag on the shared group consuming &lt;code&gt;billing.events&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Mitigation.&lt;/strong&gt; We stopped the &lt;code&gt;ledger-export-row&lt;/code&gt; producer, watched lag on &lt;code&gt;billing.events&lt;/code&gt;, and waited until the consumer drained enough backlog that &lt;code&gt;payment-captured&lt;/code&gt; and &lt;code&gt;invoice-issued&lt;/code&gt; caught up. It worked, but only as a manual lever—identify the bulk source, turn it off, hope you chose the right one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lesson.&lt;/strong&gt; Live types on a topic do not stay safe when you add bulk later. Do not park &lt;code&gt;ledger-export-row&lt;/code&gt;-class export on the same partition-ordered topic as &lt;code&gt;payment-captured&lt;/code&gt; and &lt;code&gt;invoice-issued&lt;/code&gt; unless you accept that slow export blocks fast billing for that key. Move export back to its own topic (for example &lt;code&gt;billing.export-replay&lt;/code&gt;) or isolate it with a separate consumer group.&lt;/p&gt;

&lt;h2&gt;
  
  
  Incident 2: One broken process blocked everything on the topic
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Setup.&lt;/strong&gt; Topic &lt;code&gt;order.events&lt;/code&gt;, partition key &lt;code&gt;orderId&lt;/code&gt;. A &lt;strong&gt;message-type&lt;/strong&gt; header routed each record—for example &lt;code&gt;order-placed&lt;/code&gt;, &lt;code&gt;payment-authorized&lt;/code&gt;, and &lt;code&gt;return-initiated&lt;/code&gt;—to its own handler with different validation and idempotency rules. Separate code paths, but still &lt;strong&gt;one topic&lt;/strong&gt; and &lt;strong&gt;one consumer group&lt;/strong&gt; per partition, processed in offset order.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What happened.&lt;/strong&gt; A &lt;code&gt;return-initiated&lt;/code&gt; message was built from bad data in the database (invalid refund amount on the return record). The returns handler rejected it, the message retried, and the consumer could not commit past that offset on the partition. Later records on the same partition—including &lt;code&gt;order-placed&lt;/code&gt; and &lt;code&gt;payment-authorized&lt;/code&gt; for other orders that hashed to that partition—waited behind the failing &lt;code&gt;return-initiated&lt;/code&gt;. Headers picked the right handler; they did not unblock the shared queue.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Symptoms.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Log spikes for &lt;code&gt;return-initiated&lt;/code&gt; validation failures on &lt;code&gt;order.events&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Stalled processing for &lt;code&gt;order-placed&lt;/code&gt; / &lt;code&gt;payment-authorized&lt;/code&gt; on the same partition despite healthy payloads&lt;/li&gt;
&lt;li&gt;Retries on the bad &lt;code&gt;return-initiated&lt;/code&gt; record amplifying load; on-call debating skip, seek, or redeploy on &lt;code&gt;order.events&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Mitigation.&lt;/strong&gt; We fixed the invalid data in the database—the source of the bad refund amount—then replayed or retried until the &lt;code&gt;return-initiated&lt;/code&gt; handler could succeed and the partition offset could move. &lt;code&gt;order-placed&lt;/code&gt; and &lt;code&gt;payment-authorized&lt;/code&gt; traffic resumed on that partition. Patching DB rows to unblock a Kafka partition was slow, risky, and coordination-heavy; it should not be the routine way to recover &lt;code&gt;order.events&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lesson.&lt;/strong&gt; Routing &lt;code&gt;return-initiated&lt;/code&gt; by header is not the same as isolating it. One topic still shares retry policy and poison-message risk across types—and a bad row in the database can stall unrelated message types on the same partition. A dedicated topic (for example &lt;code&gt;order.returns&lt;/code&gt;) or an isolated consumer + DLQ for &lt;code&gt;return-initiated&lt;/code&gt; shrinks blast radius so returns work does not hold up fulfillment or payment events.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rule: share a topic only when order matters
&lt;/h2&gt;

&lt;p&gt;Share a topic when all of this is true:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Events form one logical sequence for one id (aggregate, entity, or workflow instance).&lt;/li&gt;
&lt;li&gt;Consumers must process that sequence in order to stay correct—not just for easier debugging.&lt;/li&gt;
&lt;li&gt;You can name the partition key for that sequence (&lt;code&gt;orderId&lt;/code&gt;, &lt;code&gt;accountId&lt;/code&gt;, &lt;code&gt;shardId&lt;/code&gt;, …).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Kafka orders messages within a partition. It does not order across keys. It does not let you prioritize message type A over type B on the same partition.&lt;/p&gt;

&lt;h3&gt;
  
  
  When one topic is the right model
&lt;/h3&gt;

&lt;p&gt;Example: account lifecycle for one account.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Topic:&lt;/strong&gt; &lt;code&gt;account.events&lt;/code&gt; (or your environment prefix)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Key:&lt;/strong&gt; &lt;code&gt;accountId&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Events:&lt;/strong&gt; &lt;code&gt;account-opened&lt;/code&gt; → &lt;code&gt;limit-set&lt;/code&gt; → &lt;code&gt;debit-applied&lt;/code&gt; → &lt;code&gt;account-closed&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you rebuild account state from the log, you need order per account. One topic, stable keys, and documented schema changes fit well.&lt;/p&gt;

&lt;h3&gt;
  
  
  When the same domain still needs multiple topics
&lt;/h3&gt;

&lt;p&gt;You may still say “account” or “order” in conversation—but need different guarantees:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concern&lt;/th&gt;
&lt;th&gt;Split because&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Live lifecycle vs bulk backfill&lt;/td&gt;
&lt;td&gt;Different urgency and lag tolerance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Commands vs analytics fan-out&lt;/td&gt;
&lt;td&gt;Strict order for state; best-effort for metrics/search&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Process A vs process B handlers&lt;/td&gt;
&lt;td&gt;Different failures, retries, and owners&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Notifications vs ledger&lt;/td&gt;
&lt;td&gt;Different SLOs and idempotency rules&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Example split:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;order.lifecycle&lt;/code&gt; — key = &lt;code&gt;orderId&lt;/code&gt;, state machine, strict order&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;order.backfill&lt;/code&gt; — bulk replay, isolated consumers, not on the live path&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;order.notifications&lt;/code&gt; — key = &lt;code&gt;orderId&lt;/code&gt; or &lt;code&gt;userId&lt;/code&gt;, own retry/DLQ&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;For experienced teams:&lt;/strong&gt; Draw topic boundaries around SLOs and failure domains. DDD boxes can guide naming; the log contract is what you must get right.&lt;/p&gt;

&lt;h2&gt;
  
  
  Recommendations
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Do not share a topic when…
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Interactive and batch/export/replay traffic need different latency.&lt;/li&gt;
&lt;li&gt;One message type can poison the queue or retry forever while others are fine.&lt;/li&gt;
&lt;li&gt;Different teams ship on different cycles—you do not want one bad deploy to block another team’s events.&lt;/li&gt;
&lt;li&gt;One event type can spike 10× and define lag for everything else.&lt;/li&gt;
&lt;li&gt;“We read them together” is only true in a UI or report—join in the app, not in the broker.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;message-type header&lt;/strong&gt; (or similar) routes work in code—it does not replace separate topics when lag, retries, or failure domains must differ.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Creating a new topic per feature is usually correct
&lt;/h3&gt;

&lt;p&gt;Teams ask: “Will we drown in topics?” Creating topics is cheap. Coupled outages are expensive. A feature with its own producers, consumers, retention, and on-call owner deserves its own topic name and a clear ordering story—not a slot on a crowded legacy topic because the noun matches.&lt;/p&gt;

&lt;p&gt;What grows with topic count is governance: ACLs, Schema Registry subjects, dashboards, naming rules—not a cliff where the cluster stops accepting topics. Invest there instead of minimizing topic count for its own sake.&lt;/p&gt;

&lt;h3&gt;
  
  
  Defaults worth putting in team standards
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Document what the partition key means for “order.”&lt;/li&gt;
&lt;li&gt;One consumer group per service that owns processing; do not share groups to save machines.&lt;/li&gt;
&lt;li&gt;DLQ and retry policy per pipeline—not one policy for unrelated handlers on a mega-topic.&lt;/li&gt;
&lt;li&gt;Backfill on dedicated topics or controlled replay; do not mix with live traffic without a design review.&lt;/li&gt;
&lt;li&gt;Runbooks: owner, retention, key, ordering guarantee, and what to do when lag spikes—without “patch the database to unblock the partition” as step one.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Request/reply and channel separation
&lt;/h3&gt;

&lt;p&gt;In many microservice setups (including NestJS/Kafka-style patterns), request and reply use two topics. That is on purpose: different consumers, correlation rules, and scaling. Same idea—order may matter per correlation id on each channel, not “one topic for all RPC-shaped traffic.”&lt;/p&gt;

&lt;h2&gt;
  
  
  Closing
&lt;/h2&gt;

&lt;p&gt;We merged topics because consumption looked shared. Incident 1: bulk and urgent traffic must not share an ordered queue if recovery means stopping producers and waiting. Incident 2: unrelated processes on one topic share retries and poison messages—and fixing bad rows in the database to unblock the log is a bad habit, not a strategy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Default:&lt;/strong&gt; Share a topic only when order matters for a defined key and lifecycle. Split on urgency, failure isolation, and ownership before the next 2 a.m. drain.&lt;/p&gt;

&lt;p&gt;This week, pick one busy shared topic and ask: mixed urgency? mixed handlers? one retry policy for everything? If yes, plan a split—the incidents above are what that review prevents.&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>backend</category>
      <category>devops</category>
      <category>systemdesign</category>
    </item>
    <item>
      <title>Find Similar Sentences Using Embeddings and a Vector Database</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Thu, 15 Oct 2026 12:05:56 +0000</pubDate>
      <link>https://dev.to/joungpark/find-similar-sentences-using-embeddings-and-a-vector-database-mej</link>
      <guid>https://dev.to/joungpark/find-similar-sentences-using-embeddings-and-a-vector-database-mej</guid>
      <description>&lt;h2&gt;
  
  
  Why I Started Exploring Semantic Search
&lt;/h2&gt;

&lt;p&gt;I'm building Second Memory, a personal memory application that allows users to save their memories and retrieve them later through natural-language questions.&lt;/p&gt;

&lt;p&gt;One challenge is that users don't necessarily remember the exact words they used when saving a memory. They might remember its context or meaning instead.&lt;/p&gt;

&lt;p&gt;For example, a user might have saved these memories:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;"I forgot my umbrella and got soaked in the rain."&lt;/li&gt;
&lt;li&gt;"I lifted weights at the gym after work."&lt;/li&gt;
&lt;li&gt;"I caught up with John over coffee yesterday."&lt;/li&gt;
&lt;li&gt;"I stayed up late debugging my application."&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Later, the user asks:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Who did I meet recently?"&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The most relevant memory would be:&lt;/p&gt;

&lt;p&gt;"I caught up with John over coffee yesterday."&lt;/p&gt;

&lt;p&gt;A traditional keyword search might miss this because "meet" and "caught up" are different expressions.&lt;/p&gt;

&lt;p&gt;This is where &lt;strong&gt;semantic search&lt;/strong&gt; comes in. Instead of relying on exact keyword matches, we can search for text with similar meanings using embeddings.&lt;/p&gt;

&lt;p&gt;In this post, I'll explore how to generate embeddings, calculate cosine distance, find relevant sentences, and perform vector searches using PostgreSQL with pgvector.&lt;/p&gt;

&lt;p&gt;The examples use a locally runnable embedding model, so no API key or paid embedding service is required.&lt;/p&gt;

&lt;h2&gt;
  
  
  Generate Embeddings
&lt;/h2&gt;

&lt;p&gt;An embedding is a numerical representation of text. An embedding model converts text into a vector, allowing us to compare sentences mathematically.&lt;/p&gt;

&lt;p&gt;For this demonstration, I'll use &lt;code&gt;sentence-transformers/all-MiniLM-L6-v2&lt;/code&gt;, a small, open-source embedding model that runs locally and produces 384-dimensional vectors.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sentence_transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SentenceTransformer&lt;/span&gt;

&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;SentenceTransformer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sentence-transformers/all-MiniLM-L6-v2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;sentences&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I forgot my umbrella and got soaked in the rain.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I lifted weights at the gym after work.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I caught up with John over coffee yesterday.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I stayed up late debugging my application.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;embeddings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sentences&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shape&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# (4, 384)
&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# [[ 0.04035296  0.10288464  0.09306896 ... -0.00502765 -0.13130508  0.05253862]
#  [-0.07009698  0.04319851  0.00160278 ... -0.06155823 -0.09415536 -0.01941661]
#  [-0.04802262 -0.01395144  0.12706237 ... -0.0639021  -0.03755342 -0.06052431]
#  [ 0.00379441  0.01192298 -0.02876158 ... -0.02021407 -0.0892337   0.05982817]]
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The model converts each sentence into a vector containing 384 floating-point values.&lt;/p&gt;

&lt;p&gt;The shape &lt;code&gt;(4, 384)&lt;/code&gt; represents four sentences, each with a 384-dimensional embedding.&lt;/p&gt;

&lt;p&gt;Each number is a coordinate in a high-dimensional vector space. Individual dimensions don't necessarily correspond to specific concepts such as people, places, or activities. Instead, semantic information is distributed across the vector.&lt;/p&gt;

&lt;p&gt;The output is abbreviated with &lt;code&gt;...&lt;/code&gt;, but each embedding contains all 384 values.&lt;/p&gt;

&lt;p&gt;Looking at these numbers alone doesn't tell us which sentences are semantically related. To find that out, we need to compare their vectors.&lt;/p&gt;
&lt;h3&gt;
  
  
  Using an Embedding API
&lt;/h3&gt;

&lt;p&gt;Embeddings can also be generated through an API rather than running a model locally.&lt;/p&gt;

&lt;p&gt;For example, OpenAI provides an embeddings endpoint:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl https://api.openai.com/v1/embeddings &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Authorization: Bearer &lt;/span&gt;&lt;span class="nv"&gt;$OPENAI_API_KEY&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{
    "model": "text-embedding-3-small",
    "input": "I caught up with John over coffee yesterday."
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The response contains an embedding vector, just like the local model.&lt;/p&gt;

&lt;p&gt;The concept is the same, although different models produce different vector dimensions and values. Embeddings generated by different models generally shouldn't be mixed in the same similarity search.&lt;/p&gt;

&lt;p&gt;For the rest of this post, I'll continue using the local model so the examples remain reproducible without an API key.&lt;/p&gt;
&lt;h2&gt;
  
  
  Calculate Cosine Distance
&lt;/h2&gt;

&lt;p&gt;Now that we have embeddings, we need a way to measure how closely two sentences are related.&lt;/p&gt;

&lt;p&gt;One commonly used metric is &lt;strong&gt;cosine distance&lt;/strong&gt;, which compares the directions of two vectors rather than their magnitudes.&lt;/p&gt;

&lt;p&gt;Let's calculate the distance between two sentences.&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sentence_transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SentenceTransformer&lt;/span&gt;

&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;SentenceTransformer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sentence-transformers/all-MiniLM-L6-v2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;embeddings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I forgot my umbrella and got soaked in the rain.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Who did I meet recently?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;normalize_embeddings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Cosine distance using normalized vectors
&lt;/span&gt;&lt;span class="n"&gt;distance&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Cosine distance: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;distance&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# Cosine distance: 0.7981
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The &lt;code&gt;normalize_embeddings=True&lt;/code&gt; option scales each vector to unit length.&lt;/p&gt;

&lt;p&gt;For normalized vectors, the dot product (&lt;code&gt;np.dot()&lt;/code&gt;) is equivalent to cosine similarity. Subtracting it from 1 gives us cosine distance:&lt;/p&gt;

&lt;p&gt;

&lt;/p&gt;
&lt;div class="katex-element"&gt;
  &lt;span class="katex-display"&gt;&lt;span class="katex"&gt;&lt;span class="katex-mathml"&gt;&lt;/span&gt;&lt;span class="katex-html"&gt;&lt;span class="base"&gt;&lt;span class="strut"&gt;&lt;/span&gt;&lt;span class="mord text"&gt;&lt;span class="mord"&gt;cosine&amp;nbsp;distance&lt;/span&gt;&lt;/span&gt;&lt;span class="mspace"&gt;&lt;/span&gt;&lt;span class="mrel"&gt;=&lt;/span&gt;&lt;span class="mspace"&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="base"&gt;&lt;span class="strut"&gt;&lt;/span&gt;&lt;span class="mord"&gt;1&lt;/span&gt;&lt;span class="mspace"&gt;&lt;/span&gt;&lt;span class="mbin"&gt;−&lt;/span&gt;&lt;span class="mspace"&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="base"&gt;&lt;span class="strut"&gt;&lt;/span&gt;&lt;span class="mord text"&gt;&lt;span class="mord"&gt;cosine&amp;nbsp;similarity&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;/div&gt;



&lt;p&gt;Sentence Transformers also provides &lt;code&gt;model.similarity()&lt;/code&gt;, which calculates cosine similarity by default. The equivalent calculation would be:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;distance&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;similarity&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;item&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;However, using NumPy keeps the distance calculation independent of Sentence Transformers. The same approach works with embeddings returned by OpenAI or other embedding APIs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A smaller cosine distance indicates greater similarity.&lt;/strong&gt; A distance of 0 means the vectors point in the same direction.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why Cosine Distance?
&lt;/h3&gt;

&lt;p&gt;There are several ways to compare vectors:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Description&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Cosine distance&lt;/td&gt;
&lt;td&gt;Compares vector directions, ignoring magnitude&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Euclidean distance&lt;/td&gt;
&lt;td&gt;Measures straight-line distance between vectors&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Inner product&lt;/td&gt;
&lt;td&gt;Measures vector alignment and is affected by magnitude&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For text embeddings, we're generally interested in semantic relationships rather than differences in vector magnitude, making cosine distance a practical default.&lt;/p&gt;

&lt;p&gt;When embeddings are L2-normalized, cosine distance, Euclidean distance, and inner product produce equivalent rankings. The choice of metric matters more when embeddings aren't normalized or when the model recommends a particular metric.&lt;/p&gt;

&lt;p&gt;We'll use cosine distance throughout this post because pgvector supports it directly through the &lt;code&gt;&amp;lt;=&amp;gt;&lt;/code&gt; operator.&lt;/p&gt;

&lt;h2&gt;
  
  
  Find the Most Relevant Memory
&lt;/h2&gt;

&lt;p&gt;Now let's apply cosine distance to our original example and find the memory most relevant to the question:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Who did I meet recently?"&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sentence_transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SentenceTransformer&lt;/span&gt;

&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;SentenceTransformer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sentence-transformers/all-MiniLM-L6-v2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;texts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I forgot my umbrella and got soaked in the rain.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I lifted weights at the gym after work.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I caught up with John over coffee yesterday.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I stayed up late debugging my application.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Who did I meet recently?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="n"&gt;memory_embeddings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;texts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;normalize_embeddings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;query_embedding&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;normalize_embeddings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Cosine distance using normalized vectors
&lt;/span&gt;&lt;span class="n"&gt;distances&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;memory_embeddings&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;query_embedding&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;argsort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;distances&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;distances&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; - &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;texts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# 0.6097 - I caught up with John over coffee yesterday.
# 0.7981 - I forgot my umbrella and got soaked in the rain.
# 0.8151 - I lifted weights at the gym after work.
# 0.8454 - I stayed up late debugging my application.
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This time, we're comparing a single query against multiple candidate memories.&lt;/p&gt;

&lt;p&gt;The results are sorted in ascending order because &lt;strong&gt;smaller cosine distance indicates greater relevance&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;As expected, the memory about meeting John ranks first, even though the question uses "meet" while the memory uses "caught up".&lt;/p&gt;

&lt;p&gt;This demonstrates semantic retrieval: finding related meanings rather than matching exact words.&lt;/p&gt;

&lt;p&gt;However, the closest result isn't guaranteed to be relevant in every case. The ranking depends on the embedding model and the available memories.&lt;/p&gt;

&lt;p&gt;In a production system, we'd need to evaluate retrieval quality using representative queries and data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Introducing pgvector
&lt;/h2&gt;

&lt;p&gt;So far, we've generated embeddings and calculated distances in Python. But generating embeddings for every stored memory on each query would be inefficient.&lt;/p&gt;

&lt;p&gt;A better approach is to generate embeddings when memories are created or updated, persist them, and search against the stored vectors.&lt;/p&gt;

&lt;p&gt;For this, I'll use &lt;strong&gt;pgvector&lt;/strong&gt;, a PostgreSQL extension that adds vector storage and similarity search capabilities.&lt;/p&gt;

&lt;p&gt;Before using it with text embeddings, let's look at a simple example with two-dimensional vectors.&lt;/p&gt;

&lt;h3&gt;
  
  
  Store Vectors
&lt;/h3&gt;

&lt;p&gt;First, enable the extension and create a table:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="n"&gt;EXTENSION&lt;/span&gt; &lt;span class="n"&gt;IF&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;EXISTS&lt;/span&gt; &lt;span class="n"&gt;vector&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;examples&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="nb"&gt;SERIAL&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;embedding&lt;/span&gt; &lt;span class="n"&gt;VECTOR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Insert three vectors:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;examples&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;embedding&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;VALUES&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'[1,0]'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'[0.8,0.6]'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'[0,1]'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Using two-dimensional vectors makes it easier to see how pgvector calculates distances.&lt;/p&gt;

&lt;h3&gt;
  
  
  Search by Cosine Distance
&lt;/h3&gt;

&lt;p&gt;Let's find the vectors closest to &lt;code&gt;[1,0]&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;
    &lt;span class="n"&gt;embedding&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;embedding&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'[1,0]'&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;distance&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;examples&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;distance&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;&amp;lt;=&amp;gt;&lt;/code&gt; operator calculates cosine distance.&lt;/p&gt;

&lt;p&gt;The results are:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; embedding | distance
-----------+----------
 [1,0]     | 0
 [0.8,0.6] | 0.2
 [0,1]     | 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first vector has a distance of 0 because it points in the same direction as the query.&lt;/p&gt;

&lt;p&gt;The second vector has a distance of 0.2, while the third is orthogonal to the query and has a distance of 1.&lt;/p&gt;

&lt;p&gt;The important point is that &lt;strong&gt;pgvector doesn't understand the meaning of text&lt;/strong&gt;. It stores vectors and performs mathematical operations on them.&lt;/p&gt;

&lt;p&gt;The embedding model is responsible for producing vectors that represent semantic relationships.&lt;/p&gt;

&lt;h3&gt;
  
  
  Vector Indexes
&lt;/h3&gt;

&lt;p&gt;Without an index, PostgreSQL performs an exact nearest-neighbor search.&lt;/p&gt;

&lt;p&gt;For larger datasets, pgvector supports approximate nearest-neighbor indexes such as HNSW and IVFFlat.&lt;/p&gt;

&lt;p&gt;For example, we can create an HNSW index:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;examples_embedding_hnsw_idx&lt;/span&gt;
&lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;examples&lt;/span&gt;
&lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="n"&gt;hnsw&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;embedding&lt;/span&gt; &lt;span class="n"&gt;vector_cosine_ops&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The search query itself doesn't need to change.&lt;/p&gt;

&lt;p&gt;PostgreSQL can use the index to accelerate searches, although the query planner may choose a sequential scan for small datasets.&lt;/p&gt;

&lt;p&gt;Approximate indexes can improve search performance by trading off some retrieval accuracy.&lt;/p&gt;

&lt;h3&gt;
  
  
  Embedding Dimensions and Index Limitations
&lt;/h3&gt;

&lt;p&gt;One practical consideration when using pgvector is the number of dimensions in an embedding.&lt;/p&gt;

&lt;p&gt;pgvector supports storing vectors with up to 16,000 dimensions, but standard HNSW and IVFFlat indexes using the &lt;code&gt;vector&lt;/code&gt; type support up to 2,000 dimensions.&lt;/p&gt;

&lt;p&gt;This becomes relevant when using models such as OpenAI's &lt;code&gt;text-embedding-3-large&lt;/code&gt;, which produces 3,072-dimensional embeddings by default.&lt;/p&gt;

&lt;p&gt;One option is to use a shorter embedding supported by the model.&lt;/p&gt;

&lt;p&gt;OpenAI's &lt;code&gt;text-embedding-3&lt;/code&gt; models use &lt;strong&gt;Matryoshka Representation Learning (MRL)&lt;/strong&gt;, which allows shorter embeddings to retain useful semantic information.&lt;/p&gt;

&lt;p&gt;For example, we can request a 1,536-dimensional embedding from &lt;code&gt;text-embedding-3-large&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl https://api.openai.com/v1/embeddings &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Authorization: Bearer &lt;/span&gt;&lt;span class="nv"&gt;$OPENAI_API_KEY&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{
    "model": "text-embedding-3-large",
    "input": "I caught up with John over coffee yesterday.",
    "dimensions": 1536
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This reduces storage requirements and allows the embedding to fit within pgvector's standard vector index dimension limit.&lt;/p&gt;

&lt;p&gt;Another option is pgvector's &lt;code&gt;halfvec&lt;/code&gt; type, which supports indexing up to 4,000 dimensions using half-precision values.&lt;/p&gt;

&lt;p&gt;The right approach depends on retrieval quality, storage, precision, and performance requirements.&lt;/p&gt;

&lt;p&gt;For our example using 384-dimensional embeddings, the standard &lt;code&gt;vector&lt;/code&gt; type is sufficient.&lt;/p&gt;

&lt;h2&gt;
  
  
  Store and Search Embeddings with pgvector
&lt;/h2&gt;

&lt;p&gt;Now that we've covered embedding generation, cosine distance, and pgvector, let's bring everything together.&lt;/p&gt;

&lt;p&gt;We'll store the four memories with their embeddings in PostgreSQL and retrieve them using cosine distance.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pgvector.psycopg&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;register_vector&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sentence_transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SentenceTransformer&lt;/span&gt;

&lt;span class="n"&gt;DB_URL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;postgresql://user:password@localhost:5432/mydatabase&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;SentenceTransformer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sentence-transformers/all-MiniLM-L6-v2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;texts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I forgot my umbrella and got soaked in the rain.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I lifted weights at the gym after work.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I caught up with John over coffee yesterday.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;I stayed up late debugging my application.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Who did I meet recently?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="n"&gt;DIMENSIONS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;384&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;DB_URL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CREATE EXTENSION IF NOT EXISTS vector&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;register_vector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;
        CREATE TABLE IF NOT EXISTS journal (
            id BIGSERIAL PRIMARY KEY,
            content TEXT NOT NULL,
            embedding VECTOR(&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;DIMENSIONS&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;) NOT NULL
        )
    &lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;embeddings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;texts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# Reset the demo data
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;TRUNCATE TABLE journal RESTART IDENTITY&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;embedding&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;texts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;
            INSERT INTO journal (content, embedding)
            VALUES (%s, %s)
            &lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;embedding&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;commit&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;query_embedding&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;
        SELECT
            content,
            embedding &amp;lt;=&amp;gt; %s AS distance
        FROM journal
        ORDER BY embedding &amp;lt;=&amp;gt; %s ASC
        LIMIT 4
        &lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query_embedding&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;query_embedding&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;fetchall&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Query: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;distance&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;distance&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; - &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# Query: Who did I meet recently?
# 0.6097 - I caught up with John over coffee yesterday.
# 0.7981 - I forgot my umbrella and got soaked in the rain.
# 0.8151 - I lifted weights at the gym after work.
# 0.8454 - I stayed up late debugging my application.
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There are two key differences from the in-memory implementation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Embeddings are persisted&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;pgvector provides the &lt;code&gt;VECTOR(384)&lt;/code&gt; data type, allowing embeddings to be stored alongside their original text.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;register_vector(conn)&lt;/code&gt; integrates pgvector with psycopg so NumPy arrays can be passed directly as parameters.&lt;/p&gt;

&lt;p&gt;In a real application, embeddings would be generated when memories are created or updated, rather than on every query.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Similarity search moves into PostgreSQL&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Instead of loading every embedding into Python to calculate distances, we let PostgreSQL perform the search.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;&amp;lt;=&amp;gt;&lt;/code&gt; operator calculates cosine distance, and the query returns memories ordered by their distance from the query embedding.&lt;/p&gt;

&lt;p&gt;As shown in the output, the ranking matches our earlier Python example, with the memory about meeting John appearing first.&lt;/p&gt;

&lt;p&gt;For larger datasets, vector indexes can improve search performance without changing the core retrieval logic.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;We've explored how embeddings represent sentence meanings numerically, how cosine distance helps identify related text, and how to retrieve relevant sentences even when their wording differs.&lt;/p&gt;

&lt;p&gt;We started with a locally runnable embedding model, calculated distances in Python, and then introduced pgvector using simple two-dimensional vectors.&lt;/p&gt;

&lt;p&gt;Finally, we combined everything into a PostgreSQL-backed semantic search implementation.&lt;/p&gt;

&lt;p&gt;The same retrieval approach can be used in a Retrieval-Augmented Generation (RAG) system, where retrieved text is provided to an LLM as context for generating an answer.&lt;/p&gt;

&lt;p&gt;There are further considerations for production systems, including vector indexing, hybrid search, relevance thresholds, and retrieval evaluation.&lt;/p&gt;

&lt;p&gt;But the core mechanism is straightforward: &lt;strong&gt;generate embeddings, calculate distances, and retrieve the closest matches.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>embedding</category>
      <category>pgvector</category>
    </item>
    <item>
      <title>Why I Chose Third-Party (Firebase) Authentication</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Fri, 09 Oct 2026 06:12:00 +0000</pubDate>
      <link>https://dev.to/joungpark/why-i-chose-third-party-firebase-authentication-43hn</link>
      <guid>https://dev.to/joungpark/why-i-chose-third-party-firebase-authentication-43hn</guid>
      <description>&lt;p&gt;Authentication is essential to Second-Memory, but it isn't its core business.&lt;/p&gt;

&lt;p&gt;The goal of Second-Memory is to help users capture their thoughts and retrieve meaningful information from them. Building an authentication system doesn't directly advance that goal.&lt;/p&gt;

&lt;p&gt;There's also a user-experience consideration. People already manage many online accounts. Asking them to create and remember another password adds friction, while social login lets them sign in with an account they already have.&lt;/p&gt;

&lt;h2&gt;
  
  
  Build vs. Buy
&lt;/h2&gt;

&lt;p&gt;I considered two approaches: building authentication in-house or using an existing service.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Consideration&lt;/th&gt;
&lt;th&gt;In-house authentication&lt;/th&gt;
&lt;th&gt;Firebase Authentication&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Initial development&lt;/td&gt;
&lt;td&gt;Build authentication flows and supporting infrastructure&lt;/td&gt;
&lt;td&gt;Integrate an existing service&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Security&lt;/td&gt;
&lt;td&gt;Implement and maintain security controls&lt;/td&gt;
&lt;td&gt;Firebase manages core authentication infrastructure; the application still has security responsibilities&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Maintenance&lt;/td&gt;
&lt;td&gt;Handle security updates and operational issues&lt;/td&gt;
&lt;td&gt;Google maintains the service&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;User experience&lt;/td&gt;
&lt;td&gt;Users may need another account and password&lt;/td&gt;
&lt;td&gt;Users can sign in with a supported existing identity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Engineering cost&lt;/td&gt;
&lt;td&gt;More development and ongoing maintenance&lt;/td&gt;
&lt;td&gt;Less implementation and maintenance work&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For a project like Second-Memory, building authentication from scratch would mean taking on additional complexity without providing much differentiation.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Does In-House Authentication Involve?
&lt;/h2&gt;

&lt;p&gt;Building authentication is more than verifying a username and password. It means taking responsibility for several areas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Security&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Store passwords using an appropriate password-hashing algorithm.&lt;/li&gt;
&lt;li&gt;Protect against brute-force attacks and credential stuffing.&lt;/li&gt;
&lt;li&gt;Implement secure session management, including expiration and revocation.&lt;/li&gt;
&lt;li&gt;Protect account recovery and verification flows against account takeover.&lt;/li&gt;
&lt;li&gt;Manage secrets, security logging, monitoring, and incident response.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;2. Account lifecycle&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Registration and email verification&lt;/li&gt;
&lt;li&gt;Login and logout&lt;/li&gt;
&lt;li&gt;Account recovery and password resets&lt;/li&gt;
&lt;li&gt;Account deletion and related data cleanup&lt;/li&gt;
&lt;li&gt;Session revocation and suspicious-login handling&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;3. Social login integration&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Supporting social login involves integrating with external identity providers and handling their authentication flows. Standards such as OpenID Connect (OIDC) and OAuth 2.0 are commonly involved, depending on the provider and flow.&lt;/p&gt;

&lt;p&gt;The application must also handle identity mapping, account linking where appropriate, and the relationship between an external identity and its own user records.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Ongoing operations&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Authentication requires continued attention after launch. Security vulnerabilities emerge, dependencies need updates, and authentication flows must be tested as the application evolves.&lt;/p&gt;

&lt;p&gt;The effort doesn't end when login starts working.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Decision
&lt;/h2&gt;

&lt;p&gt;I don't have specific authentication requirements that justify building and maintaining my own system.&lt;/p&gt;

&lt;p&gt;I'll use Firebase Authentication and make social login the preferred way to sign in. This avoids asking users to create another password and lets me rely on an established authentication service.&lt;/p&gt;

&lt;p&gt;There may be usage costs and some provider dependency, but building authentication in-house also has costs in development time, maintenance, and security responsibilities.&lt;/p&gt;

&lt;p&gt;For Second-Memory, Firebase Authentication is a practical choice because it meets my needs without requiring me to build and operate an authentication system myself.&lt;/p&gt;

&lt;p&gt;However, using a third-party authentication service doesn't eliminate application security responsibilities. Second-Memory still needs to validate identities correctly, enforce authorization in the backend, and ensure users can access only their own data.&lt;/p&gt;

&lt;h2&gt;
  
  
  How Authentication Works in Second-Memory
&lt;/h2&gt;

&lt;p&gt;Second-Memory has web and mobile clients, plus backend services. Both clients use Firebase Authentication, while the backend verifies each request and maps the Firebase identity to an internal user record.&lt;/p&gt;

&lt;h3&gt;
  
  
  The authentication flow
&lt;/h3&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;sequenceDiagram
    participant C as Web / Mobile Client
    participant F as Firebase Authentication
    participant A as Backend API
    participant D as Second-Memory Database

    C-&amp;gt;&amp;gt;F: Sign in
    F--&amp;gt;&amp;gt;C: Firebase ID token
    C-&amp;gt;&amp;gt;A: API request + Bearer token
    A-&amp;gt;&amp;gt;A: Verify token and extract uid
    A-&amp;gt;&amp;gt;D: Find user by firebaseUid
    D--&amp;gt;&amp;gt;A: Return internal user record
    A-&amp;gt;&amp;gt;A: Authorize request
    A--&amp;gt;&amp;gt;C: API response&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;The flow has six steps:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Sign in:&lt;/strong&gt; The client signs in with Firebase Authentication and obtains a Firebase ID token.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Call the API:&lt;/strong&gt; The client sends the token in the &lt;code&gt;Authorization: Bearer &amp;lt;token&amp;gt;&lt;/code&gt; header.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verify the token:&lt;/strong&gt; The backend verifies the token using the Firebase Admin SDK. It must not trust a token simply because it can decode it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Extract the Firebase UID:&lt;/strong&gt; After verification, the backend reads the user's Firebase UID from the token.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Resolve the internal user:&lt;/strong&gt; The backend looks up the user record using &lt;code&gt;firebaseUid&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Authorize the request:&lt;/strong&gt; The backend uses the internal user ID to enforce access controls and scope data operations to that user.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Why maintain our own user table?
&lt;/h3&gt;

&lt;p&gt;Firebase provides authentication, but Second-Memory still needs its own application-level user record.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;id&lt;/th&gt;
&lt;th&gt;firebaseUid&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;usr_123&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;firebase-uid-abc&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;usr_456&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;firebase-uid-xyz&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;id&lt;/code&gt; is Second-Memory's internal user ID, while &lt;code&gt;firebaseUid&lt;/code&gt; links the record to Firebase Authentication. The &lt;code&gt;firebaseUid&lt;/code&gt; column should have a unique constraint.&lt;/p&gt;

&lt;p&gt;This separation has a few advantages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Domain independence:&lt;/strong&gt; Memories and other application entities can reference Second-Memory's internal user ID.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Consistent authorization:&lt;/strong&gt; Backend services can use the same internal identity when checking data access.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Flexibility:&lt;/strong&gt; If the authentication provider changes in the future, the application's domain model doesn't need to use provider-specific identifiers everywhere.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The backend must still enforce authorization on every relevant request. A valid Firebase token proves the caller's authenticated identity; it does not automatically grant access to a particular memory or resource. The backend should derive identity from the verified token, not trust a user ID supplied by the client.&lt;/p&gt;

&lt;h3&gt;
  
  
  How this fits the client and backend stack
&lt;/h3&gt;

&lt;p&gt;The same authentication principle applies across the different parts of Second-Memory, although their implementations differ.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Next.js web app:&lt;/strong&gt; Configure Firebase for the web client, implement sign-in, and attach the ID token to API requests.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Expo mobile app:&lt;/strong&gt; Configure Firebase Authentication for React Native, handle session persistence, and attach the ID token to API requests.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backend services:&lt;/strong&gt; Use the Firebase Admin SDK to verify ID tokens, resolve the internal user, and enforce authorization.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The important distinction is that the clients obtain tokens, but the backend is responsible for verifying them and deciding what the authenticated user is allowed to access.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;For Second-Memory, authentication is necessary infrastructure, not a feature I need to reinvent.&lt;/p&gt;

&lt;p&gt;Firebase Authentication reduces the amount of authentication code I need to build and maintain. Keeping an internal user record also lets Second-Memory separate external identity verification from application-level identity and authorization.&lt;/p&gt;

&lt;p&gt;The goal isn't to eliminate security responsibilities. It's to use an established service for authentication while keeping control of the application's own data and access rules.&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>authentication</category>
      <category>startup</category>
      <category>webdev</category>
    </item>
    <item>
      <title>From Monorepo to Turborepo: Managing Multiple Projects</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Fri, 09 Oct 2026 04:42:43 +0000</pubDate>
      <link>https://dev.to/joungpark/from-monorepo-to-turborepo-managing-multiple-projects-6e3</link>
      <guid>https://dev.to/joungpark/from-monorepo-to-turborepo-managing-multiple-projects-6e3</guid>
      <description>&lt;h2&gt;
  
  
  Why a Monorepo?
&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;A monorepo is a single repository containing multiple distinct projects, with well-defined relationships. — &lt;a href="https://monorepo.tools" rel="noopener noreferrer"&gt;Monorepo Explained&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A monorepo makes sense when multiple projects have meaningful relationships:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Shared libraries and utilities&lt;/strong&gt; — reuse common code across projects.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shared contracts&lt;/strong&gt; — keep API types consistent between producers and consumers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cross-project changes&lt;/strong&gt; — update related projects in a single change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shared tooling and rules&lt;/strong&gt; — maintain consistent development practices.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If projects have no meaningful relationships, separate repositories may be simpler.&lt;/p&gt;

&lt;p&gt;For Second-Memory, Web, Mobile, and Backend need to agree on API contracts. Sharing TypeScript API types helps keep those contracts consistent.&lt;/p&gt;

&lt;p&gt;Web uses React/Next.js, while Mobile uses React Native/Expo, so their UI implementations remain separate. Code is shared only where it genuinely works across both platforms.&lt;/p&gt;

&lt;p&gt;Keeping related projects together makes coordinating changes and managing shared contracts easier.&lt;/p&gt;

&lt;p&gt;How do we manage their packages and dependencies?&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Workspaces?
&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Workspaces are a set of features in the npm CLI that provide support for managing multiple packages from a local file system within a single top-level root package. — &lt;a href="https://docs.npmjs.com/cli/v8/using-npm/workspaces" rel="noopener noreferrer"&gt;npm Workspaces&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A monorepo can work without workspaces. Each project can have its own &lt;code&gt;package.json&lt;/code&gt; and manage dependencies independently.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;second-memory/
├── apps/
│   ├── web/
│   └── mobile/
├── services/
│   ├── memory-service/
│   ├── ask-service/
│   └── embedding-worker/
└── packages/
    ├── api-types/
    └── ui-component/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Suppose both Web and Memory Service depend on &lt;code&gt;api-types&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Without workspaces, each project needs to manage its dependencies and local references separately. A project might reference the shared package using a local file dependency:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"dependencies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"@second-memory/api-types"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"file:../../packages/api-types"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The relative path depends on the consuming project's location. Dependencies must be installed and managed in each project, and keeping local references and updates consistent requires additional care.&lt;/p&gt;

&lt;p&gt;With workspaces, the root configuration defines which directories contain packages.&lt;/p&gt;

&lt;p&gt;For example, with pnpm:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;pnpm-workspace.yaml&lt;/code&gt;&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;packages&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;apps/*"&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;services/*"&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;packages/*"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each package declares its own dependencies in its &lt;code&gt;package.json&lt;/code&gt;. For example, Web can depend on the shared API types package:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;apps/web/package.json&lt;/code&gt;&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"dependencies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"@second-memory/api-types"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"workspace:*"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The shared package identifies itself in its own &lt;code&gt;package.json&lt;/code&gt;:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;packages/api-types/package.json&lt;/code&gt;&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"@second-memory/api-types"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"1.0.0"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;workspace:*&lt;/code&gt; protocol explicitly tells pnpm to use the local workspace package.&lt;/p&gt;

&lt;p&gt;Dependencies can then be installed from the repository root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pnpm &lt;span class="nb"&gt;install&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;pnpm recognises the workspace packages and links local dependencies, making it easier to manage relationships between projects.&lt;/p&gt;

&lt;h3&gt;
  
  
  Without Workspaces vs With Workspaces
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Area&lt;/th&gt;
&lt;th&gt;Without workspaces&lt;/th&gt;
&lt;th&gt;With workspaces&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Install dependencies&lt;/td&gt;
&lt;td&gt;Install separately in each project&lt;/td&gt;
&lt;td&gt;Install from the root&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reference shared packages&lt;/td&gt;
&lt;td&gt;Configure local file dependencies or links&lt;/td&gt;
&lt;td&gt;Declare dependencies between workspace packages&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Manage package relationships&lt;/td&gt;
&lt;td&gt;Maintain references across projects&lt;/td&gt;
&lt;td&gt;Define participating packages in the root configuration&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Develop shared packages&lt;/td&gt;
&lt;td&gt;Manage local references and updates carefully&lt;/td&gt;
&lt;td&gt;Work with linked local packages&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Run builds and tests&lt;/td&gt;
&lt;td&gt;Run scripts manually or maintain root scripts&lt;/td&gt;
&lt;td&gt;Still requires scripts or task orchestration&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A workspace reduces manual package and dependency management. It provides a way to define packages and their local dependencies within the monorepo.&lt;/p&gt;

&lt;p&gt;However, managing packages is only part of the problem. As the number of projects and dependencies grows, coordinating builds, tests, and other tasks becomes another challenge.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Task Orchestration?
&lt;/h2&gt;

&lt;p&gt;With only a few projects, running build and test commands manually may be enough. As the repository grows, two problems become more noticeable.&lt;/p&gt;

&lt;h3&gt;
  
  
  Problem 1: Running Only the Necessary Tasks
&lt;/h3&gt;

&lt;p&gt;The dependencies in Second-Memory look like this:&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TD
    AT[api-types] --&amp;gt; W[apps/web]
    AT --&amp;gt; M[apps/mobile]
    AT --&amp;gt; MS[memory-service]
    AT --&amp;gt; AS[ask-service]

    UI[ui-component] --&amp;gt; W
    UI --&amp;gt; M

    EW[embedding-worker]&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;Suppose I change &lt;code&gt;packages/ui-component&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Web and Mobile depend on it, but Memory Service, Ask Service, and Embedding Worker do not.&lt;/p&gt;

&lt;p&gt;Ideally, I want to run checks for the affected projects without wasting time checking unrelated services.&lt;/p&gt;

&lt;p&gt;With workspaces alone, I can run individual commands using package filters:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pnpm &lt;span class="nt"&gt;--filter&lt;/span&gt; web &lt;span class="nb"&gt;test
&lt;/span&gt;pnpm &lt;span class="nt"&gt;--filter&lt;/span&gt; mobile &lt;span class="nb"&gt;test&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But I still need to know which projects depend on the changed package and decide which commands to run.&lt;/p&gt;

&lt;p&gt;As the dependency graph grows, maintaining this selection manually becomes more difficult.&lt;/p&gt;

&lt;p&gt;Task orchestration can automate the selection of relevant tasks based on package relationships, configured task dependencies, and changes.&lt;/p&gt;

&lt;h3&gt;
  
  
  Problem 2: Managing Dependencies Between Tasks
&lt;/h3&gt;

&lt;p&gt;Some tasks need to run before others.&lt;/p&gt;

&lt;p&gt;For example, if a service consumes a shared package's compiled output, that package needs to be built before the service.&lt;/p&gt;

&lt;p&gt;I could maintain a script to enforce the order:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"scripts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"build:memory"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"pnpm --filter api-types build &amp;amp;&amp;amp; pnpm --filter memory-service build"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This works for a simple case, but the scripts become harder to maintain as more packages and task dependencies are added.&lt;/p&gt;

&lt;p&gt;Ideally, I want to define task relationships once and let a tool determine the correct execution order.&lt;/p&gt;

&lt;h2&gt;
  
  
  How Task Orchestration Helps
&lt;/h2&gt;

&lt;p&gt;Task orchestration adds a layer above workspace package management. It coordinates tasks such as builds, tests, and linting across packages.&lt;/p&gt;

&lt;h3&gt;
  
  
  Task Graph
&lt;/h3&gt;

&lt;p&gt;A task graph describes dependencies between tasks and helps determine their execution order.&lt;/p&gt;

&lt;p&gt;For example, if &lt;code&gt;api-types&lt;/code&gt; must be built before its consumers:&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TD
    A[Build api-types] --&amp;gt; B[Build memory-service]
    A --&amp;gt; C[Build ask-service]
    A --&amp;gt; D[Build web]
    A --&amp;gt; E[Build mobile]&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;The graph should reflect the actual build requirements. If consumers compile the shared package's source directly, a separate build step may not be necessary.&lt;/p&gt;

&lt;p&gt;Turborepo uses the workspace package dependency graph together with configured task dependencies to determine execution order. The package relationships provide the foundation; task configuration defines how tasks depend on one another.&lt;/p&gt;

&lt;h3&gt;
  
  
  Affected Tasks
&lt;/h3&gt;

&lt;p&gt;When a package changes, Turborepo can use Git change information and the dependency graph to select affected packages and their relevant tasks.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Change &lt;code&gt;api-types&lt;/code&gt;:&lt;/strong&gt; Web, Mobile, Memory Service, and Ask Service may be affected.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Change &lt;code&gt;ui-component&lt;/code&gt;:&lt;/strong&gt; Web and Mobile may be affected.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Change &lt;code&gt;embedding-worker&lt;/code&gt;:&lt;/strong&gt; Its checks can run independently of the other projects.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This reduces the need to maintain manual build and test sequences as the repository grows. The result depends on the configured task graph, available Git change information, and the tasks defined by each package.&lt;/p&gt;

&lt;h3&gt;
  
  
  Additional Benefits
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Technique&lt;/th&gt;
&lt;th&gt;Benefit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Caching&lt;/td&gt;
&lt;td&gt;Reuse valid task results instead of repeating unchanged work&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Parallel execution&lt;/td&gt;
&lt;td&gt;Run independent tasks concurrently while respecting dependencies&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Remote caching&lt;/td&gt;
&lt;td&gt;Share cached results across developers and CI when configured&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These capabilities can improve development and CI performance, although the primary goal is to manage task relationships and execution more reliably.&lt;/p&gt;

&lt;h2&gt;
  
  
  Comparing the Approaches
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Capability&lt;/th&gt;
&lt;th&gt;Workspaces alone&lt;/th&gt;
&lt;th&gt;Turborepo&lt;/th&gt;
&lt;th&gt;Nx&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Manage packages and local dependencies&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Uses a workspace package manager&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Run scripts across packages&lt;/td&gt;
&lt;td&gt;Yes, using scripts and filters&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Coordinate task dependencies&lt;/td&gt;
&lt;td&gt;Manual scripts or additional tooling&lt;/td&gt;
&lt;td&gt;Built-in task orchestration&lt;/td&gt;
&lt;td&gt;Built-in task orchestration&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Identify affected tasks&lt;/td&gt;
&lt;td&gt;Manual selection or custom tooling&lt;/td&gt;
&lt;td&gt;Supported&lt;/td&gt;
&lt;td&gt;Supported&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cache task results&lt;/td&gt;
&lt;td&gt;Requires additional tooling&lt;/td&gt;
&lt;td&gt;Built-in&lt;/td&gt;
&lt;td&gt;Built-in&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Run independent tasks in parallel&lt;/td&gt;
&lt;td&gt;Manual or script-based&lt;/td&gt;
&lt;td&gt;Built-in&lt;/td&gt;
&lt;td&gt;Built-in&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Broader project tooling and plugins&lt;/td&gt;
&lt;td&gt;Limited to package management&lt;/td&gt;
&lt;td&gt;Task-focused&lt;/td&gt;
&lt;td&gt;More extensive options&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Workspaces alone can be sufficient for a small repository with simple build and test requirements. Turborepo and Nx become useful when coordinating tasks across project dependencies requires more automation.&lt;/p&gt;

&lt;p&gt;Both provide task orchestration, affected-task capabilities, caching, and parallel execution. Nx offers a broader set of project tooling and plugins, while Turborepo focuses more directly on task orchestration.&lt;/p&gt;

&lt;p&gt;For Second-Memory, I wanted the task orchestration capabilities without needing a broader project-management system.&lt;/p&gt;

</description>
      <category>monorepo</category>
      <category>turborepo</category>
      <category>workspaces</category>
      <category>pnpm</category>
    </item>
    <item>
      <title>Monorepo or Polyrepo?</title>
      <dc:creator>Joung Park</dc:creator>
      <pubDate>Wed, 07 Oct 2026 05:52:16 +0000</pubDate>
      <link>https://dev.to/joungpark/monorepo-or-polyrepo-1gib</link>
      <guid>https://dev.to/joungpark/monorepo-or-polyrepo-1gib</guid>
      <description>&lt;p&gt;After deciding to use microservices, the next question was:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How should I organise the codebase?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I chose a &lt;strong&gt;monorepo&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Second-Memory has several applications and services:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;apps/
  web/
  mobile/

services/
  memory-service/
  ask-service/
  embedding-worker/

packages/
  api-types/
  ui-components/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;They are independently deployable, but they are all part of the same product.&lt;/p&gt;

&lt;h2&gt;
  
  
  Monorepo vs Polyrepo
&lt;/h2&gt;

&lt;p&gt;There is no universally better choice.&lt;/p&gt;

&lt;p&gt;The trade-offs become clearer when looking at common development tasks:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Area&lt;/th&gt;
&lt;th&gt;Monorepo&lt;/th&gt;
&lt;th&gt;Polyrepo&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Share code&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Easy&lt;/strong&gt; — reusable code can live in shared packages&lt;/td&gt;
&lt;td&gt;Harder — publish packages or duplicate code&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Share types&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Easy&lt;/strong&gt; — TypeScript types can be shared directly&lt;/td&gt;
&lt;td&gt;Harder — types need separate versioning&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cross-project changes&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Easy&lt;/strong&gt; — related changes can be made in one PR&lt;/td&gt;
&lt;td&gt;Harder — changes may require multiple PRs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Dependency management&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Easy&lt;/strong&gt; — dependencies can be managed consistently&lt;/td&gt;
&lt;td&gt;More complex — each repo manages its own dependencies&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Local development&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Easy&lt;/strong&gt; — one repository contains the whole system&lt;/td&gt;
&lt;td&gt;More setup — multiple repositories need to be configured&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;CI/CD&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;More complex — need to detect affected projects&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Easy&lt;/strong&gt; — each repository naturally has its own pipeline&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Service isolation&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Harder — boundaries need to be enforced within the repository&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Easy&lt;/strong&gt; — repositories provide natural isolation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Access control&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Harder — repository access generally covers the whole codebase&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Easy&lt;/strong&gt; — permissions can be managed per repository&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Independent releases&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Possible — CI/CD can deploy projects independently&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Easy&lt;/strong&gt; — each repository has its own release lifecycle&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Neither approach is inherently better.&lt;/p&gt;

&lt;p&gt;For Second-Memory, I valued &lt;strong&gt;easy development across the whole system&lt;/strong&gt; more than repository-level isolation.&lt;/p&gt;

&lt;h2&gt;
  
  
  When to Choose Which
&lt;/h2&gt;

&lt;p&gt;A monorepo can be a good fit when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Small team or solo developer&lt;/strong&gt; — one repository is easier to manage than several.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Projects share contracts or code&lt;/strong&gt; — changes can be made and tested together.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cross-project changes are common&lt;/strong&gt; — one PR can update multiple applications or services.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The whole system is closely related&lt;/strong&gt; — one repository makes it easier to understand and work across the product.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Polyrepo can be a better fit when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Large or independent teams&lt;/strong&gt; need clear ownership boundaries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Strong repository isolation&lt;/strong&gt; is important.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Projects have different release cycles&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Access control&lt;/strong&gt; needs to differ between projects.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For Second-Memory, being a &lt;strong&gt;sole developer&lt;/strong&gt; was one of the main reasons I chose a monorepo.&lt;/p&gt;

&lt;p&gt;Managing several repositories would add overhead without giving me much benefit. I didn't need repository-level ownership boundaries between teams.&lt;/p&gt;

&lt;p&gt;Instead, I wanted to make it easy to work across the web app, mobile app, backend services, and shared contracts.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Should Be Shared?
&lt;/h2&gt;

&lt;p&gt;A monorepo makes sharing easy.&lt;/p&gt;

&lt;p&gt;But &lt;strong&gt;easy to share doesn't mean everything should be shared&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;I wanted to keep the shared surface area small.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Shared Part&lt;/th&gt;
&lt;th&gt;Used By&lt;/th&gt;
&lt;th&gt;Why Share It?&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;API types&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Web, Mobile, Backend&lt;/td&gt;
&lt;td&gt;Keep API contracts consistent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Web-specific code&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Web&lt;/td&gt;
&lt;td&gt;Keep it inside &lt;code&gt;apps/web&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mobile-specific code&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Mobile&lt;/td&gt;
&lt;td&gt;Keep it inside &lt;code&gt;apps/mobile&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Service domain logic&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Individual service&lt;/td&gt;
&lt;td&gt;Keep ownership within the service&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Service implementation&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Individual service&lt;/td&gt;
&lt;td&gt;Preserve service boundaries&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So instead of creating shared packages for everything, I only create a package when there is a genuine reason to share something.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;apps/
  web/
  mobile/
services/
  memory-service/
  ask-service/
  embedding-worker/
packages/
  api-types/
  ui-components/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The API types are shared because they represent a contract between different parts of the system.&lt;/p&gt;

&lt;p&gt;But a Web-only component stays in &lt;code&gt;apps/web&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A Mobile-only utility stays in &lt;code&gt;apps/mobile&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;And Memory Service domain logic stays inside &lt;code&gt;memory-service&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Share the contract, not the ownership
&lt;/h3&gt;

&lt;p&gt;The fact that services live in the same repository doesn't mean they should share their internal implementation.&lt;/p&gt;

&lt;p&gt;The API contract can be shared:&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    T[packages/api-types]

    W[Web] --&amp;gt; T
    M[Mobile] --&amp;gt; T
    B[Backend Services] --&amp;gt; T&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;while the services remain independently owned:&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    MS[Memory Service]
    AS[Ask Service]

    MS --&amp;gt; MD[Memory Domain]
    AS --&amp;gt; AD[AI Interaction]

    MD --&amp;gt; DB1[(Memory DB)]
    AD --&amp;gt; DB2[(Ask Data)]&lt;/code&gt;&lt;/pre&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Share contracts where it makes sense, but don't share service ownership.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Keeping the Shared Surface Small
&lt;/h2&gt;

&lt;p&gt;There is another practical reason for limiting shared code.&lt;/p&gt;

&lt;p&gt;The more projects depend on a shared package, the larger the potential impact of a change.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    S[packages/shared]
    S --&amp;gt; W[Web]
    S --&amp;gt; M[Mobile]
    S --&amp;gt; ME[Memory]
    S --&amp;gt; A[Ask]&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;A change to &lt;code&gt;shared&lt;/code&gt; may require all projects to be tested.&lt;/p&gt;

&lt;p&gt;But a change here:&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart LR
    C[Change in apps/web] --&amp;gt; W[Web]
    W --&amp;gt; CI[Web CI/CD]&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;only affects Web.&lt;/p&gt;

&lt;p&gt;This has implications for both &lt;strong&gt;CI/CD&lt;/strong&gt; and &lt;strong&gt;application dependencies&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;I don't want a large &lt;code&gt;shared&lt;/code&gt; package that contains unrelated functionality used by almost every project.&lt;/p&gt;

&lt;p&gt;Instead, I prefer small, focused shared packages with clear reasons for existing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The CI/CD Trade-off
&lt;/h2&gt;

&lt;p&gt;CI/CD is one area where polyrepo can be simpler.&lt;/p&gt;

&lt;p&gt;With polyrepo, a change to &lt;code&gt;memory-service&lt;/code&gt; naturally belongs to the &lt;code&gt;memory-service&lt;/code&gt; repository.&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    R[memory-service repo]
    R --&amp;gt; CI[CI/CD]
    CI --&amp;gt; T[Test]
    CI --&amp;gt; B[Build]
    CI --&amp;gt; D[Deploy]&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;In a monorepo, multiple deployable units live in the same repository.&lt;/p&gt;

&lt;p&gt;A change to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;services/memory-service/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;shouldn't necessarily rebuild and deploy everything.&lt;/p&gt;

&lt;p&gt;The pipeline needs to understand:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;what changed&lt;/li&gt;
&lt;li&gt;what depends on it&lt;/li&gt;
&lt;li&gt;what needs to be tested&lt;/li&gt;
&lt;li&gt;what needs to be built&lt;/li&gt;
&lt;li&gt;what needs to be deployed&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This becomes particularly important when shared packages are involved.&lt;/p&gt;

&lt;p&gt;A change to &lt;code&gt;ui-components&lt;/code&gt; may affect only web and mobile projects, while a change isolated to &lt;code&gt;services/embedding-worker/&lt;/code&gt; should ideally affect only the relevant pipeline.&lt;/p&gt;

&lt;p&gt;The repository structure itself is simple, but the build and deployment system needs to become more intelligent as the project grows.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Monorepo for Second-Memory?
&lt;/h2&gt;

&lt;p&gt;The main benefits for me were:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Easy&lt;/strong&gt; sharing of API types&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Easy&lt;/strong&gt; cross-project changes&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Easy&lt;/strong&gt; local development&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Easy&lt;/strong&gt; dependency management&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Easy&lt;/strong&gt; visibility of the whole product&lt;/li&gt;
&lt;li&gt;Less overhead for a &lt;strong&gt;solo developer&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The trade-offs were:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;More complex&lt;/strong&gt; CI/CD&lt;/li&gt;
&lt;li&gt;More tooling needed as the repository grows&lt;/li&gt;
&lt;li&gt;Less repository-level isolation&lt;/li&gt;
&lt;li&gt;The need to carefully control shared dependencies&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For Second-Memory, I was willing to accept these trade-offs.&lt;/p&gt;

&lt;p&gt;The goal wasn't to share as much code as possible.&lt;/p&gt;

&lt;p&gt;The goal was to get the convenience of a monorepo while keeping &lt;strong&gt;ownership and dependencies clear&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That led to the next question:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I manage builds, dependencies, task execution, and caching across all these projects?&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>microservices</category>
      <category>monorepo</category>
      <category>polyrepo</category>
    </item>
  </channel>
</rss>
