DEV Community: Gouranga Das Samrat

Illegal Instruction: How I Woke Up a Year-Old 'Not Planned' Bug and (Almost) Fixed Vite on Android

Gouranga Das Samrat — Fri, 07 Aug 2026 09:34:42 +0000

My dev server didn't start. It just died.

$ pnpm create vite test-react-app --template react-compiler-ts
$ cd test-react-app && pnpm run dev
Illegal instruction        pnpm run dev

No stack trace a human can read. No hint about what to fix. Just Illegal instruction, and the terminal handing control back like nothing happened. Same story for create-vue. Same story for a plain vite dev. Three different starter templates, one identical crash.

This was on a Termux install on an Android tablet, not some exotic setup. And that detail matters more than it sounds like it should, because a lot of people's only computer is the phone in their pocket. I'm one of them some days. If you're learning to code from Bangladesh, Nigeria, Indonesia, or a dozen other places, a laptop isn't a given — Termux on an old Android phone is often the whole dev environment. When the standard npm create vite path doesn't work there, it's not a minor inconvenience. It's a wall.

Chasing the actual cause

I started where anyone would: assume it's a Termux problem. Filed termux/termux-packages#30841 with logs for Next.js, Vite+React, and Vue, all crashing or silently falling back to a slower path.

Termux maintainer robertkirkman pointed out that pkg install turbopack fixes the Next.js case, since Turbopack is packaged for Termux directly. Tried it — true, but only for Next.js. Vite and Vue don't touch Turbopack at all, and they were still crashing exactly the same way.

That's when it became clear this wasn't a packaging gap. Vite 6 and up ships Rolldown as its default bundler, a Rust-based, N-API bundler that loads a native binary — @rolldown/binding-android-arm64 for our case. Something in that binary was blowing up the moment it loaded.

truboxl, another Termux maintainer, asked for a logcat tombstone instead of just terminal output, and that's where the real evidence showed up:

pid: 32487, tid: 32487, name: MainThread  >>> node <<<
ABI: 'arm64'
signal 4 (SIGILL), code 1 (ILL_ILLOPC), fault addr 0x6eb3132d64 (*pc=0xc8e9fe93)

backtrace:
  #00 pc 00000000006e2d64  .../rolldown-binding.android-arm64.node
  #01 pc 00000000006e2474  .../rolldown-binding.android-arm64.node
  #02 pc 000000000004a1d8  linker64 (call_array+288)
  #03 pc 000000000004a3d4  linker64 (soinfo::call_constructors+380)
  #04 pc 0000000000035a9c  linker64 (do_dlopen+2076)

That's a crash happening during dlopen, inside the native .node binary itself, before a single line of JS runs. The classic fingerprint of a native binary compiled with instruction-set assumptions that don't hold on the chip actually running it. Not a Termux bug at all. truboxl confirmed it and labeled it an upstream issue.

The ghost in the closet

Naturally, the next move was to go report this to Rolldown. Except somebody already had.

Buried in the Rolldown repo was rolldown/rolldown#6342 — "Illegal instruction on android arm64" — opened back in September 2025 by a user named hyperz111. Same crash signature, same platform. He'd shared a screen recording and full CPU info because he couldn't put together a minimal reproduction repo. The auto-reply bot wanted a reproduction within 14 days or the issue would close. Nobody circled back in time, and it closed as not planned in October 2025.

Ten months of silence followed. Nearly a year of Vite quietly not working on Android, and basically nobody talking about it.

Here's the part that bugged me most: "not planned" reads like a policy decision — we've decided Android isn't a target we support. But that's not actually what was true. Rolldown was already building and publishing @rolldown/binding-android-arm64 on every release. The platform wasn't unsupported in theory; it was broken in practice, and a bot closed the thread because a repro checkbox never got ticked. A build bug got mistaken for a feature request nobody asked for, and then everyone — including me, at first — treated "closed as not planned" as a dead end instead of what it actually was: an unfinished conversation.

In the meantime, I filed termux/termux-packages#30852 asking Termux to package Rolldown directly, the same way it already packages Turbopack, since going through upstream looked like it was going nowhere.

Going back with receipts

But a package request downstream doesn't fix the actual bug upstream. So robertkirkman went back to the closed Rolldown issue with the tombstone trace in hand and reframed the ask. This wasn't "please make Android a fully maintained target." It was narrower: the android-arm64 build already exists and already ships, it's crashing at load on real silicon, and that's a cross-compilation flag problem, not a support-commitment problem.

That reframing is what actually moved it. Rolldown maintainer shulaoda reopened #6342 and was refreshingly honest about the gap: the team doesn't have a shelf of random Android devices to test against, so verification from someone with the actual crashing hardware was exactly what they needed.

Then sapphi-red spotted the likely root cause, and it's a satisfying one: Rolldown depends on mimalloc, and its no_opt_arch feature — the thing that stops mimalloc from assuming newer ARM instructions are present — was only being enabled when target_os = "linux". Android runs on Linux under the hood but reports a different target_os, so the Android build skipped that safety flag entirely and got compiled assuming ARMv8.1 atomic instructions that a chip like a Cortex-A53 simply doesn't have. Since mimalloc initializes inside a library constructor, that lines up exactly with a SIGILL at dlopen, before any of Rolldown's own code even runs.

shulaoda shipped a preview build in rolldown/rolldown#10638 within the same day and asked for someone with the affected hardware to verify it.

Testing it on the actual crashing device

So I did. Cortex-A53, MediaTek MT6762V, the same tablet that had been throwing Illegal instruction this whole time.

$ npx rolldown@1.2.3 --version
Illegal instruction

$ pnpm i https://pkg.pr.new/rolldown@10638
$ npx rolldown --version
rolldown v1.2.3+commit.a7ba7ad

No crash. Then the real test: a fresh Vite + React project.

$ pnpm create vite test-react-app --template react-compiler-ts
$ pnpm run dev
  VITE v6.x.x  ready in 412 ms
  ➜  Local:   http://localhost:5173/

It just worked. Ten months of Illegal instruction, gone with one removed target_os condition.

Then I tweeted it

I posted the whole trail on X, tagging Evan You, the Rolldown team, and Vite. Rolldown's shulaoda — Jerry Zhao — replied publicly the next day, confirmed the team was taking it seriously, and pointed back at the reopened issue and the preview build. I replied that I was already testing it on the affected device and would post results on the issue. Small exchange, but it's the kind of public back-and-forth that keeps an issue from quietly dying a second time.

The side quest nobody asked for

While all this was happening upstream, the Termux side didn't just sit around waiting for a release. Contributor DevGitPit opened termux/termux-packages#30887 to package Rolldown 1.2.3 downstream anyway, so Termux users wouldn't be stuck waiting on Rolldown's release cycle. That PR turned into its own small collaboration — robertkirkman debugging build errors alongside DevGitPit, and at one point asking the Rolldown team directly whether they'd consider supporting more Android ABIs (32-bit ARM, x86, x86_64) beyond just arm64, since some Termux users are on hardware Rolldown doesn't build for at all.

Two separate open-source projects, two separate maintainer teams, ended up helping each other out over the same crash. Termux gave Rolldown a real device to test against and a trace nobody on the Rolldown side could have reproduced. Rolldown gave Termux users a real fix instead of a permanent downstream patch to maintain forever.

Where it stands right now

The fix works. I've confirmed it on the exact hardware that was crashing. It hasn't landed in a release yet — the preview build is verified, the PR is open, and it's waiting on the Rolldown team to merge and cut a new version. Once that happens, npm install -g rolldown (or just running create vite) should work on Android arm64 without anyone needing a Termux-specific package at all.

Why I bothered writing this up

"Closed as not planned" isn't always a wall. Sometimes it's a bot doing its job on an issue that never got the follow-up it needed. The difference between a bug staying closed for another year and getting fixed in a week wasn't cleverness — it was a device to reproduce it on, a tombstone instead of a vague description, and enough persistence to go back and ask for the closed issue to be looked at differently instead of filing yet another duplicate.

If you're running Termux on Android arm64 and hit this, or you're on different silicon — an older Snapdragon, an Exynos, anything that isn't a Cortex-A53 — go pull the fix from #10638 and drop a comment on #6342 with what you find. More devices means more confidence before it ships, and right now that's the only thing standing between this bug and actually being closed for good.

API Versioning

Gouranga Das Samrat — Sun, 02 Aug 2026 02:00:00 +0000

One-liner: API versioning lets you evolve your API without breaking existing clients — you maintain multiple contract versions simultaneously until clients migrate.

❓ Why Version Your API?

Once a public API has consumers, you can't change it freely:

Renaming a field breaks every client using the old name
Removing an endpoint causes 404s for clients that depend on it
Changing response format silently corrupts client logic

Versioning gives clients time to migrate at their own pace.

🏗️ 4 Versioning Strategies

1. URI Path Versioning ✅ (Most Common)

GET /api/v1/users/42
GET /api/v2/users/42   ← new version with different response shape

Pros: Obvious, easy to route, cacheable by CDN

Cons: Pollutes URLs, clients must change URLs to upgrade

2. Query Parameter Versioning

GET /api/users/42?version=1
GET /api/users/42?version=2

Pros: URL stays stable

Cons: Easy to forget, doesn't work well with caching

3. Header Versioning

GET /api/users/42
Accept: application/vnd.myapi.v2+json
              OR
X-API-Version: 2

Pros: Clean URLs, aligns with REST purists

Cons: Not visible in browser, harder to test/share URLs

4. Content Negotiation (Accept header)

GET /api/users/42
Accept: application/vnd.company.user-v2+json

Used by GitHub API and Stripe. Powerful but complex.

🔀 Routing Versions

At the API Gateway or router level:

// Express routing example
const v1Router = require('./routes/v1');
const v2Router = require('./routes/v2');

app.use('/api/v1', v1Router);
app.use('/api/v2', v2Router);

// Or via API Gateway (AWS API Gateway, Kong, Nginx):
# nginx.conf
location /api/v1/ { proxy_pass http://service-v1/; }
location /api/v2/ { proxy_pass http://service-v2/; }

📋 Breaking vs Non-Breaking Changes

Change	Breaking?	Strategy
Add new optional field to response	❌ No	Safe — clients ignore unknown fields
Add new optional request param	❌ No	Safe — use defaults for old clients
Remove a field	✅ Yes	New version required
Rename a field	✅ Yes	New version required
Change field type (string→int)	✅ Yes	New version required
Change HTTP method	✅ Yes	New version required
Add required request param	✅ Yes	New version required

Rule: If an existing client would break without code changes, it's breaking.

🔄 Migration Lifecycle

v1 released
    ↓
v2 released (both live simultaneously)
    ↓
v1 deprecated — notice given (3-6 months typical)
    ↓
v1 sunset — clients that haven't migrated break
    ↓
v1 removed from infrastructure

Deprecation headers (tell clients they're on old version):

HTTP/1.1 200 OK
Deprecation: true
Sunset: Sat, 01 Jan 2026 00:00:00 GMT
Link: <https://api.example.com/v2/users>; rel="successor-version"

🏷️ Semantic Versioning for Internal APIs

MAJOR.MINOR.PATCH

v2.0.0 → breaking change (new MAJOR version)
v2.1.0 → non-breaking additions (new MINOR version)
v2.1.1 → bug fix (PATCH)

For public REST APIs, only expose MAJOR version in the URL (v1, v2). Minor/patch changes are transparent.

🗂️ Version Coexistence Patterns

Shared Code, Version-Specific Transform

// Core business logic shared
const userService = require("./services/userService");

// v1 controller — old response shape
router.get("/v1/users/:id", async (req, res) => {
  const user = await userService.getUser(req.params.id);
  res.json({ user_id: user.id, user_name: user.name }); // v1 shape
});

// v2 controller — new response shape
router.get("/v2/users/:id", async (req, res) => {
  const user = await userService.getUser(req.params.id);
  res.json({ id: user.id, name: user.name, email: user.email }); // v2 shape
});

✅ Pros

Clients can migrate at their own pace
Enables continuous API evolution without coordination
Clear contracts per version — easy to document
URI versioning is CDN-cacheable and easy to debug

❌ Cons

Maintaining multiple versions increases codebase complexity
Duplicate code / logic divergence over time
Clients often stay on old versions longer than expected
Sunset enforcement requires communicating with all API consumers

⚖️ When to Use / When NOT to Use

✅ Use when:

Public APIs with external consumers
Mobile apps (you can't force users to update immediately)
B2B APIs where clients control their upgrade schedule
Any breaking change to a live API

❌ Avoid / not needed when:

Internal APIs where you control all consumers (coordinate and deploy together)
GraphQL APIs (schema evolution handles this differently)
Early product with no external consumers yet (iterate freely)

Data Consistency in Microservices — Saga Pattern

Gouranga Das Samrat — Sat, 01 Aug 2026 02:00:00 +0000

One-liner: In microservices, you can't use a single database transaction across services — the Saga pattern breaks it into a sequence of local transactions with compensating rollbacks on failure.

❓ The Problem: Distributed Transactions

In a monolith, you can wrap everything in one DB transaction:

BEGIN;
  UPDATE orders SET status='confirmed' WHERE id=1;
  UPDATE inventory SET stock=stock-1 WHERE product_id=5;
  INSERT INTO payments (order_id, amount) VALUES (1, 99.99);
COMMIT; -- all or nothing

In microservices, each service owns its own DB. You cannot do a cross-service ACID transaction. If the payment service fails after inventory was deducted, you're in an inconsistent state.

🎭 The Saga Pattern

A Saga is a sequence of local transactions. Each step publishes an event triggering the next. If any step fails, compensating transactions undo the previous steps.

Example: E-commerce Order Flow

Step 1: Order Service      → Create order (PENDING)
Step 2: Inventory Service  → Reserve items
Step 3: Payment Service    → Charge customer
Step 4: Shipping Service   → Schedule delivery
Step 5: Order Service      → Mark order CONFIRMED

Happy path ✅:

Order Created → Items Reserved → Payment Charged → Shipping Scheduled → Done ✅

Failure at Payment ❌:

Order Created → Items Reserved → Payment FAILED
                                       ↓
                          Compensate: Release inventory
                          Compensate: Cancel order

🏗️ Two Implementation Styles

1. Choreography (Event-driven, decentralized)

Each service listens for events and reacts. No central coordinator.

OrderService    ──publishes──► "OrderCreated"
InventoryService ──listens──► reserves stock, publishes "StockReserved"
PaymentService  ──listens──► charges card, publishes "PaymentProcessed"
ShippingService ──listens──► schedules delivery, publishes "Shipped"

On failure:
PaymentService publishes "PaymentFailed"
InventoryService listens, releases stock, publishes "StockReleased"
OrderService listens, marks order CANCELLED

Pros: Simple, decoupled, no single point of failure

Cons: Hard to track overall saga state, cyclic dependencies possible

2. Orchestration (Centralized coordinator)

A dedicated Saga Orchestrator tells each service what to do.

                    ┌─────────────────────┐
                    │   Saga Orchestrator  │
                    │   (Order Saga)       │
                    └──────────┬──────────┘
                               │
         ┌─────────────────────┼──────────────────────┐
         ▼                     ▼                       ▼
  InventoryService       PaymentService          ShippingService
  "Reserve stock"        "Charge $99"            "Ship to addr"
  → "Reserved" ✅        → "Failed" ❌           (not called)
         │                     │
         └─────────────────────┘
                    Orchestrator receives "Failed"
                    → tells InventoryService: "Release stock"
                    → marks saga FAILED

Pros: Clear saga state, easy to audit and debug

Cons: Orchestrator is a bottleneck / SPOF if not designed carefully

🆚 Choreography vs Orchestration

	Choreography	Orchestration
Control	Decentralized	Centralized
Coupling	Loose (events)	Tighter (direct calls)
Visibility	Hard to trace end-to-end	Easy — one place to look
Complexity	Low per-service, high overall	High orchestrator logic
Best for	3-4 step sagas	Complex, many-step flows

💾 Saga State Storage

Track saga state to survive crashes:

CREATE TABLE sagas (
  id UUID PRIMARY KEY,
  type VARCHAR(50),          -- 'order_saga'
  current_step VARCHAR(50),  -- 'awaiting_payment'
  status VARCHAR(20),        -- 'IN_PROGRESS', 'COMPLETED', 'FAILED'
  payload JSONB,             -- order details
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

✅ Pros

Maintains data consistency without distributed transactions
Works with polyglot persistence (each service uses its own DB)
Choreography scales well horizontally
Orchestration provides clear audit trail

❌ Cons

Eventual consistency — data temporarily inconsistent between steps
Compensating transactions are complex to implement correctly
Hard to debug when something fails mid-saga
Idempotency required for every step (retries happen)

⚖️ When to Use / When NOT to Use

✅ Use when:

Multi-service workflows that must be atomic (order → payment → shipping)
Different services own different databases
Long-running business processes (insurance claims, loan approvals)

❌ Avoid when:

Single service with one DB (just use a DB transaction)
Simple async operations that don't need rollback
You can redesign to avoid the distributed write entirely

Shipping the Chaos: One Contract, Three Codegens, and a Docker Gotcha Hiding in Plain Sight

Gouranga Das Samrat — Tue, 28 Jul 2026 04:00:00 +0000

This is the final post in "The Muse of Microservices," a series on the system design behind Polyhymnia. Parts 1–3 covered the architecture, the Go gateway, and the Rust/C++ backends. This post covers what holds it all together and how it ships — plus one bug hunt you can go do yourself.

How Polyhymnia's single protobuf contract turns into three different generated codebases, how the project ships to Docker, and a loopback-address gotcha worth hunting down yourself.

One file, three languages, zero hand-written serialization

Everything about how these services talk to each other traces back to a single 40-line file: proto/quote.proto. It defines two services, five messages, and three RPCs, and it is the only place any of that is defined. Nobody hand-writes a JSON schema for the Rust service and a separate struct for the Go client and hopes they stay in sync — that's exactly the kind of manual synchronization that silently rots the moment someone forgets to update one side.

Instead, each language generates its own stubs from the same source of truth, through three genuinely different pipelines:

Rust does it automatically, on every cargo build, via a build.rs script that calls tonic_build::compile_protos(...). You never run a separate codegen step — it's baked into the normal build.
C++ does it through a CMake add_custom_command that shells out to protoc with the gRPC C++ plugin, generating .pb.cc/.pb.h files into the build directory before the actual source is even compiled.
Go is the odd one out — its stubs are generated explicitly via a just proto-go recipe that invokes protoc-gen-go and protoc-gen-go-grpc, rather than as an automatic build step.

Three different mechanisms, one contract, zero drift possible between them — because there's nothing to drift. Change a field in the .proto file and every language's next build simply reflects it, or fails to compile until you update the code that used the old shape. That failure mode, "won't compile" instead of "silently breaks in production," is the entire value proposition of contract-first API design in one sentence.

A small but telling asymmetry: reflection

The C++ service turns on gRPC server reflection:

grpc::reflection::InitProtoReflectionServerBuilderPlugin();

which means a tool like grpcurl can introspect the Randomizer service's methods and message shapes at runtime, with no .proto file in hand. The Rust service doesn't do this — tonic-reflection isn't even in its dependency list. Neither choice is wrong; reflection is a debugging convenience, not a requirement, and it's genuinely useful to notice that two services in the same codebase made different calls about it. It's a good habit to carry into your own projects: reflection on your internal services costs you almost nothing and saves a .proto file scavenger hunt the next time you're debugging with grpcurl at 2am.

`justfile`: one command runner for three build systems

Cargo, CMake, and the Go toolchain each have their own build model, their own flags, and their own idea of what "clean" means. Polyhymnia flattens all three behind a single justfile:

just build        # build-rust + build-cpp + build-go
just run           # build everything, then launch all four processes
just proto-go       # regenerate Go stubs
just clean          # wipe every language's build artifacts

This is a small but real answer to a real problem: polyglot repositories have an onboarding cost that monoglot repositories simply don't, because every contributor has to context-switch between build systems just to get the thing running. A single command runner doesn't eliminate that cost, but it hides it behind one consistent vocabulary — just build, just run, just clean — regardless of which language is underneath. It's the kind of unglamorous tooling decision that never shows up in an architecture diagram and saves everyone real time anyway.

Packaging: four Dockerfiles, three of them multi-stage

Each service ships with its own Dockerfile, and three of the four follow the same shape: a fat builder stage with the full compiler toolchain, followed by a slim runtime stage that copies out only the finished binary.

The Rust image, for instance, compiles inside rust:latest (with protobuf-compiler installed for tonic-build), then copies just the resulting binary into a bare debian:bookworm-slim image. The Go image does the same trick even more aggressively — it builds with CGO_ENABLED=0 and ships the result into alpine:3.18, ending up with a runtime image that has no build tooling, no source code, and a dramatically smaller attack surface than the image that built it. The frontend, having nothing to compile, skips all of this and is just static files dropped straight into nginx:stable-alpine.

Multi-stage builds like this are standard practice for a reason: your production image should contain exactly what's needed to run the service and nothing that was needed to build it. Every extra tool left in a runtime image is one more thing that can have a vulnerability, and one more reason the image is bigger than it needs to be.

Two compose files, two audiences

The repo ships two separate docker-compose files, and the distinction between them is worth calling out because it maps to two genuinely different users:

docker-compose.yml builds every image locally from source — the right choice for a contributor who's actively changing code.
docker-compose.images.yml instead points at pre-built images on Docker Hub (with a <VERSION> placeholder to fill in) — the right choice for someone who just wants to run the finished thing without owning a full multi-language toolchain.

Same four services, same ports, same topology — just a different answer to "where does the image come from." It's a small piece of design that quietly respects that "contributor" and "user" are different personas with different needs.

The gotcha: what "127.0.0.1" means depends on where you're standing

Here's where it gets genuinely interesting, and it connects directly back to something we flagged in Part 2.

Recall that the Go gateway dials its two backends at hardcoded addresses:

rustDbAddr    = "127.0.0.1:50051"
cppEngineAddr = "127.0.0.1:50052"

and the Rust service binds to:

const LISTEN_ADDR: &str = "127.0.0.1:50051";

Both of those are entirely correct for the workflow the docs lead with first — just run, where every process launches directly on your machine and shares one network stack. On localhost, 127.0.0.1 unambiguously means "this machine," and every service can reach every other service through it without issue.

Docker changes the ground under that assumption. Each container in a Compose stack gets its own network namespace — its own private, isolated 127.0.0.1 that refers only to itself. When the go-gateway container dials 127.0.0.1:50051, it is not reaching the rust-db container at all; it's trying to reach port 50051 on itself, where nothing is listening. Cross-container traffic in Compose is meant to travel over the service name instead — rust-db:50051, resolved through Docker's built-in DNS on the shared Compose network — which is precisely what depends_on in the compose file gestures at but doesn't, on its own, wire up.

Interestingly, the codebase already contains a working example of the container-friendly pattern, sitting right next to the problem: the C++ engine binds to

constexpr char kListenAddress[] = "0.0.0.0:50052";

0.0.0.0 means "listen on every network interface," which is reachable from sibling containers, unlike Rust's loopback-only bind. So the two backend services in this project actually demonstrate both the wrong pattern and the right one for containerized binding, and the fix on the Go side would follow the same idea: make the dial addresses configurable — an environment variable like RUST_DB_ADDR, defaulting to 127.0.0.1:50051 for local runs but overridable to rust-db:50051 inside Compose — rather than baking one deployment mode's assumption into a compile-time constant.

This is genuinely worth doing yourself as an exercise if you clone the repo: spin up docker-compose up, watch what happens, and then go fix the addressing to make it actually work across containers. It's a small, self-contained, extremely realistic bug — the exact category of "worked perfectly on my machine, broke the second it left it" issue that shows up constantly in real infrastructure work, just shrunk down to a size you can fully understand and fix in one sitting.

(And it's a nice full-circle moment for the series: the fixed-interval retry loop we covered in Part 2 is precisely the mechanism that would cover the timing half of a multi-container startup — Compose's depends_on only waits for a container to start, not for the service inside it to actually be ready, and the gateway's own retry-and-block dial logic is what papers over that gap. It just needs to be dialing the right address to begin with.)

What you'd change to make it "real"

None of this needs fixing to serve its actual purpose — Polyhymnia is upfront about being satire with genuinely good bones, not a production template. But it's worth closing the series by naming, plainly, what separates "excellent teaching example" from "thing you'd actually run at scale," because that list is itself a useful mental checklist:

Externalize configuration. Addresses, ports, and timeouts as environment variables instead of constants, so the same binary behaves correctly across just run, Compose, and a real orchestrator without a recompile.
Add a circuit breaker. Right now a slow or failing backend just eats into the gateway's fixed timeout budget on every request; a breaker would stop hammering a backend that's already down instead of retrying it into the ground.
Move from fixed-interval to backoff-with-jitter retries once more than one caller could plausibly be retrying against the same dependency at once.
Pool database connections instead of guarding a single one behind a mutex, once concurrent load is real rather than theoretical.
Encrypt service-to-service traffic. Every gRPC channel here uses insecure.NewCredentials() — completely fine on a trusted loopback interface, and the first thing to change the moment these services cross a network you don't fully control.

That list isn't a criticism of the project — it's the reward for having read it closely enough to write it. A system small enough to fully understand is also a system small enough to see clearly what growing it up would actually require.

Closing the loop

Four services, one contract, five hops, and one very committed random number generator — all in service of a button that returns a sentence and a name. That gap between the size of the problem and the size of the solution was always the point, and if you've followed this series through, you now know exactly what's happening at every one of those hops and why.

Go clone it, click the button a few times, and then go fix that Docker networking issue yourself: GourangaDasSamrat/Polyhymnia.

Trust No One: Rust's Safety Layer and C++'s Beautifully Unnecessary Randomness

Gouranga Das Samrat — Mon, 27 Jul 2026 04:00:00 +0000

This is Part 3 of "The Muse of Microservices," a series on the system design behind Polyhymnia. Part 2 covered the Go gateway that calls both services you're about to meet.

Two backend services, two philosophies: how Polyhymnia's Rust database owner protects a single SQLite file, and how its C++ engine turns picking one array index into a systems-programming set piece.

Two services live behind the Go gateway, and they couldn't have more different personalities. One holds the only piece of mutable state in the entire system and treats that responsibility like a vault. The other holds nothing at all and treats a coin flip like a research paper. Put side by side, they're a compact lesson in two of the most fundamental axes of service design: who owns data, and what it costs to keep a service stateless.

Rust: the database's only bodyguard

The Rust service's full title in the README is "Database Manager & Safety Layer," and both halves of that name are backed by real code, not vibes.

One owner, no exceptions

Exactly one service in Polyhymnia is allowed to open quotes.db. Not the Go gateway, not the C++ engine — only Rust, and every other service reaches the data exclusively through two gRPC methods: GetAllIds and GetQuoteById. This is the database-per-service principle in its purest, smallest possible form: shared databases create a hidden coupling between every service that touches them, because a schema change in one now has to be coordinated with every other reader and writer, whether or not they ever talk to each other directly. Give exactly one service exclusive ownership of the data, and that coupling problem simply doesn't exist. Everyone else negotiates through an API instead of a schema.

Parameters, not string concatenation

The "safety" in the service's name isn't just branding — it shows up directly in how every query is built:

conn.query_row(
    "SELECT quote, author FROM quotes WHERE id = ?1",
    params![id],
    |row| { /* ... */ },
)

That ?1 placeholder, filled in via rusqlite::params!, is a parameterized query — the value is bound separately from the SQL text rather than spliced into a string. It's a one-line habit that closes off SQL injection as a category entirely, because the database driver never has to guess where a query ends and untrusted data begins. Every query in the service follows this pattern; there isn't a single interpolated string anywhere in the SQL layer. For a service whose entire reason for existing is to be the trustworthy custodian of the one file everyone else depends on, that's not a stylistic footnote — it's the whole job description.

The concurrency model, and its ceiling

The connection itself is wrapped like this:

struct QuoteDbService {
    conn: Arc<Mutex<Connection>>,
}

One SQLite connection, shared across every request, guarded by a tokio::sync::Mutex. In practice, that means every database operation the service performs — every GetAllIds, every GetQuoteById — is serialized. Only one query runs at a time, full stop, no matter how many requests arrive concurrently.

That's a completely reasonable choice for this project, and it's worth understanding why it's reasonable rather than just accepting it. Arc<Mutex<T>> is the simplest possible way to share mutable state safely across async tasks in Rust — it trades throughput for correctness and simplicity, which is exactly the right trade when your query load is "occasionally, one at a time, from a demo app." It would become the ceiling on this service's scalability the moment real concurrent load showed up — a connection pool (handing out several connections instead of guarding one) is the natural next step if that ever became a problem. Knowing which lever to pull, and when it's actually worth pulling, is most of what capacity planning is.

Interestingly, the service still turns on SQLite's write-ahead log:

let _: String = conn.query_row("PRAGMA journal_mode = WAL", [], |row| row.get(0))?;

Given that the app itself already serializes every query through the mutex, that might look redundant — and for traffic that arrives only through this service, it is. But WAL mode also changes how SQLite behaves when something outside the app touches the same file — a sqlite3 CLI session opened for debugging, for instance, which the project's own troubleshooting docs describe doing. WAL lets readers avoid blocking on a writer at the file level, not just at the application level, which is a nice bit of defense-in-depth for a database file that's meant to have exactly one legitimate writer but might occasionally get an uninvited guest.

Idempotent bootstrapping

On first launch, the service creates its own schema and seeds itself with ten quotes — but only if the table is empty:

let count: i64 = conn.query_row("SELECT COUNT(*) FROM quotes", [], |row| row.get(0))?;
if count == 0 {
    /* seed */
}

CREATE TABLE IF NOT EXISTS plus a count-gated seed step means the service can be started, killed, and restarted indefinitely without ever duplicating data or crashing on an already-initialized database. It's a self-contained, self-healing bootstrap — a tiny version of what a real migrations framework buys you at scale, running here with nothing but a conditional and a transaction.

C++: a coin flip with a PhD's worth of ceremony

Then there's the Randomizer. Its interface is one RPC — SelectRandomId(IdList) -> SelectedId — and its job, stated as plainly as possible, is: given a list of numbers, return one of them.

Here's how it actually does that:

std::random_device rd;
const uint64_t hi = static_cast<uint64_t>(rd());
const uint64_t lo = static_cast<uint64_t>(rd());
const uint64_t seed = (hi << 32) | lo;

std::mt19937_64 engine(seed);
std::uniform_int_distribution<std::size_t> dist(0, count - 1);

Two independent pulls from the OS's entropy source, bit-shifted together into a single 64-bit seed, fed into a Mersenne Twister engine, which is then sampled through a uniform distribution to get an index. And then, instead of just calling .at(index) like a person in a hurry:

const int64_t* base = request->ids().data();
const int64_t* target = base + index;
const int64_t selected_id = *target;

...it walks a raw pointer to the target element by hand. This is, transparently, a joke — the README says as much, and one call to dist(engine) alone would have been more than sufficient to pick a number out of a list of a dozen IDs. But the ceremony is worth taking seriously for a second, because every individual piece of it is a real pattern lifted from a context where it genuinely matters:

random_device seeding a fast PRNG is exactly how you'd build a cryptographically-aware random number generator: pull true entropy once, expensively, from the OS, then use it to seed a much faster deterministic generator for everything after. It's the right shape, applied to a problem that never needed it.
One footnote worth knowing regardless: the C++ standard never actually guarantees std::random_device is non-deterministic. Some standard library implementations fall back to a deterministic PRNG when no hardware entropy source is available on the platform. It's the kind of assumption that's safe 99% of the time and exactly the kind of thing worth double-checking before you build something security-sensitive on top of it.
The pointer arithmetic is functionally identical to operator[] — the compiler will very likely generate the same instructions either way — but it's a nice reminder that in C++, "correct" and "idiomatic" are two different bars, and this project is having fun clearing the first one while cheerfully ignoring the second.

Stateless, and proud of it

The one thing the Randomizer is genuinely, unironically good architecture for: it holds no state whatsoever. It doesn't know what a quote is, doesn't know what a database is, and doesn't remember anything between calls. Its only input is a list of integers; its only output is one of them.

That statelessness is the real payoff, and it's worth contrasting directly against the Rust service sitting right next to it. Because the Randomizer carries no data between requests, you could run ten copies of it behind a load balancer tomorrow with zero coordination between them — no shared cache to invalidate, no lock to contend over, nothing to keep in sync. The Rust service, by contrast, can't be casually replicated at all without first solving the much harder problem of who owns the one SQLite file. Two services, two completely different scaling stories, sitting three lines of Go apart in the gateway's handler. That contrast — stateless services scale by copying, stateful services scale by coordinating — is one of the most load-bearing ideas in distributed systems, and Polyhymnia hands it to you as a two-service case study you can read in an afternoon.

It also validates its own input, which is easy to overlook: an empty ID list returns INVALID_ARGUMENT rather than crashing or silently returning garbage. Small, boring, and exactly what you want from a service sitting at a trust boundary.

What's next

We've now met all four services and the one contract that lets them talk to each other. Part 4 zooms out to that contract itself — one .proto file feeding three independent codegen pipelines — and follows the project the rest of the way to a running container stack, including a networking subtlety in the Docker setup that's easy to miss until you go looking for it.

Containers & Docker

Gouranga Das Samrat — Sun, 26 Jul 2026 02:00:00 +0000

One-liner: Containers package your application and all its dependencies into one portable unit that runs identically on any machine — eliminating "works on my machine" problems.

📌 The Problem Before Containers

Dev machine:    Node 18, npm 9, Ubuntu 22
Staging server: Node 16, npm 8, CentOS 7
Production:     Node 14, npm 6, Amazon Linux 2

Result: App works in dev, breaks in prod 🔥

💡 Containers vs Virtual Machines

Virtual Machine (VM)

[App A] [App B] [App C]
[Guest OS] [Guest OS] [Guest OS]   ← each VM has full OS (GBs)
[Hypervisor (VMware, KVM)]
[Host OS]
[Hardware]

Heavy: ~1 GB+ per VM, slow to start (minutes)

Container

[App A] [App B] [App C]
[Container Runtime (Docker)]       ← share host OS kernel
[Host OS]
[Hardware]

Lightweight: ~10-100 MB per container, starts in milliseconds

🐳 Docker Core Concepts

Image

A read-only template — like a blueprint/snapshot of your app + dependencies.

# Dockerfile — recipe to build an image
FROM node:18-alpine          # base image
WORKDIR /app                 # working directory
COPY package*.json ./        # copy dependency files
RUN npm ci --production      # install dependencies
COPY . .                     # copy source code
EXPOSE 3000                  # document port
CMD ["node", "server.js"]    # start command

Build: docker build -t my-app:1.0 .

Container

A running instance of an image.

docker run -d \
  -p 3000:3000 \
  -e DATABASE_URL=postgres://... \
  --name my-app \
  my-app:1.0

Registry

Where images are stored and shared.

docker push myrepo/my-app:1.0   # push to Docker Hub / ECR
docker pull myrepo/my-app:1.0   # pull on another machine

Popular registries: Docker Hub, AWS ECR, GitHub Container Registry, GCR

📁 Dockerfile Best Practices

Layer Caching — Order Matters!

# ❌ Bad — any code change invalidates npm install layer
FROM node:18-alpine
COPY . .
RUN npm ci

# ✅ Good — npm install layer cached unless package.json changes
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./   ← copy just package files first
RUN npm ci              ← this layer cached if package.json unchanged
COPY . .                ← code changes only bust this layer
CMD ["node", "server.js"]

Multi-Stage Build (Small Final Image)

# Stage 1: Build
FROM node:18 AS builder
WORKDIR /app
COPY . .
RUN npm ci && npm run build

# Stage 2: Production (only compiled output)
FROM node:18-alpine AS production
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
CMD ["node", "dist/server.js"]

Result: Final image is tiny (no dev tools, no source, no test files).

.dockerignore

node_modules
.git
*.test.js
.env
dist

🔧 Docker Compose — Multi-Container Local Dev

# docker-compose.yml
version: "3.8"
services:
  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgres://postgres:pass@db:5432/mydb
      - REDIS_URL=redis://redis:6379
    depends_on:
      - db
      - redis

  db:
    image: postgres:15
    environment:
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: mydb
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine

volumes:
  postgres_data:

Start everything: docker compose up -d

Stop: docker compose down

☸️ Container Orchestration: Kubernetes (K8s)

Docker runs containers on ONE machine. In production you need many machines.

Kubernetes orchestrates containers across a cluster of machines.

Key K8s Concepts

Concept	What It Is
Pod	Smallest unit — one or more containers
Deployment	Manages pod replicas, rolling updates
Service	Stable network endpoint for a set of pods
Ingress	HTTP routing into the cluster (like an API Gateway)
ConfigMap	Non-sensitive config (env vars)
Secret	Sensitive config (passwords, API keys)
Namespace	Logical isolation within a cluster

# Deployment — run 3 replicas of my-app
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    spec:
      containers:
        - name: my-app
          image: myrepo/my-app:1.0
          ports:
            - containerPort: 3000
          resources:
            requests:
              memory: "128Mi"
              cpu: "250m"
            limits:
              memory: "256Mi"
              cpu: "500m"

K8s Benefits

Auto-scaling — scale pods based on CPU/memory
Self-healing — restart failed containers automatically
Rolling updates — deploy new version with zero downtime
Load balancing — distribute traffic across pods
Secret management — inject secrets as env vars

Managed K8s Services

Cloud	Service
AWS	EKS (Elastic Kubernetes Service)
Google	GKE (Google Kubernetes Engine)
Azure	AKS (Azure Kubernetes Service)

🔄 Container in CI/CD Pipeline

Developer pushes code →
[GitHub Actions / Jenkins]
  1. Run tests
  2. Build Docker image: docker build -t app:$GIT_SHA .
  3. Push to ECR: docker push app:$GIT_SHA
  4. Deploy to K8s: kubectl set image deployment/app app=app:$GIT_SHA
  5. K8s rolling update → zero-downtime deploy

🎨 Diagram

The diagram shows:

VM vs Container architecture layers
Docker image → container lifecycle
Docker Compose multi-service local setup
K8s cluster with nodes, pods, services

🔑 Key Takeaways

Containers solve the environment consistency problem — build once, run anywhere
Always use multi-stage builds to keep images small
Docker Compose for local dev; Kubernetes for production orchestration
Container images should be immutable — no SSH into containers to fix things; rebuild and redeploy

Microservices

Gouranga Das Samrat — Sat, 25 Jul 2026 02:00:00 +0000

One-liner: Instead of one giant application (monolith), microservices split the system into small, independent services — each owning its own data and deployed separately.

📌 Monolith vs Microservices

Monolith

[One Big App]
 ├── User Module
 ├── Order Module
 ├── Payment Module
 ├── Notification Module
 └── [One Database]

Deploy anything → redeploy everything. One bug → whole app down.

Microservices

[User Service]         → [Users DB]
[Order Service]        → [Orders DB]
[Payment Service]      → [Payments DB]
[Notification Service] → [Notif DB]

Each service: independent codebase, independent deployment, independent database.

✅ Pros of Microservices

Benefit	Detail
Independent deployment	Deploy Payment without touching User Service
Independent scaling	Scale only the Order Service during sales
Tech diversity	Order Service in Go, ML Service in Python
Fault isolation	Notification Service down → Orders still work
Small teams	Each team owns one service (Conway's Law)
Easier to understand	Each service is small and focused

❌ Cons of Microservices

Problem	Detail
Network overhead	In-process calls → network calls (latency + failure)
Distributed tracing	Debugging a request across 10 services is hard
Data consistency	No cross-service ACID transactions
Operational complexity	20 services = 20 deployments, 20 logs, 20 monitors
Service discovery	How does Service A find Service B?
Testing	Integration testing is much harder

🔄 Inter-Service Communication

Synchronous (Request-Response)

Order Service → HTTP/gRPC → Payment Service → response

✅ Simple, immediate response

❌ Tight temporal coupling — both must be up

❌ Cascading failures

When to use: When the caller needs the result immediately (e.g., payment confirmation before showing success page)

Asynchronous (Event-Driven)

Order Service → [Event Bus / Queue] → Payment Service
                                    → Email Service

✅ Decoupled — services can be down

✅ Natural fan-out

❌ Eventual consistency

❌ Harder to debug

When to use: Notifications, analytics, side effects that don't block the user

🏗️ Service Communication Patterns

API Gateway Pattern

Client → [API Gateway] → routes to appropriate service
                       → handles auth, rate limiting for all

Service Mesh (Advanced)

[Service A] ←──[sidecar proxy]──[sidecar proxy]──► [Service B]
                  (Envoy)              (Envoy)

Sidecar handles: mTLS, retries, timeouts, circuit breaking, observability.
Examples: Istio, Linkerd, AWS App Mesh.

BFF (Backend for Frontend)

[Mobile App] → [Mobile BFF] → aggregates calls to services
[Web App]    → [Web BFF]    → different data shape per client

🗄️ Database Per Service

Each service owns its data. No shared DB.

❌ Wrong:
[User Service]  ──┐
[Order Service] ──┼──► [Shared DB]  ← tight coupling at data level
[Payment Service] ┘

✅ Right:
[User Service]    → [User DB (PostgreSQL)]
[Order Service]   → [Order DB (MongoDB)]
[Payment Service] → [Payment DB (PostgreSQL)]

But what about joins?

Denormalize data (copy what you need)
Use API calls to get data from another service
Use event-driven data replication (each service maintains its own read model)

🔒 Saga Pattern — Distributed Transactions

No cross-service ACID transactions. Use Sagas instead.

Choreography Saga (Event-Driven)

Order Created →
  Payment Service listens → charges card → Payment Succeeded event
  Inventory Service listens → reserves stock → Stock Reserved event
  Email Service listens → sends confirmation

If payment fails → Payment Failed event →
  Order Service listens → cancels order

Orchestration Saga (Central Coordinator)

[Saga Orchestrator]
  1. → tell Payment Service to charge
  2. → tell Inventory to reserve
  3. → tell Email to send confirmation

If step 2 fails:
  → tell Payment Service to refund (compensating transaction)

🌡️ Circuit Breaker Pattern

Prevent cascading failures:

Normal:   Service A → Service B (works fine)
Degraded: Service A → Service B (Service B is slow/erroring)
Open:     Service A → [Circuit Open] → return fallback (don't call B)
          Wait 30 seconds → try again (half-open)
          If OK → close circuit → normal again

States: Closed (normal) → Open (failing, reject calls) → Half-Open (testing recovery)

Libraries: Hystrix (Java), resilience4j, Polly (.NET), opossum (Node.js)

🔍 Observability in Microservices

Distributed Tracing

Request ID: abc-123
→ API Gateway     [10ms]
→ Order Service   [50ms]
→ Payment Service [200ms]  ← slow here!
→ Email Service   [30ms]
Total: 290ms

Tools: Jaeger, Zipkin, AWS X-Ray, Datadog APM

Centralized Logging

All services ship logs to one place: ELK Stack (Elasticsearch + Logstash + Kibana), Datadog, CloudWatch.

Metrics & Alerting

Prometheus + Grafana: CPU, memory, request rate, error rate, latency (p50, p95, p99).

🎨 Diagram

The diagram shows:

Monolith vs microservices comparison
API Gateway + service mesh layout
Saga orchestration flow
Circuit breaker states

🔑 Key Takeaways

Don't start with microservices — start with a well-structured monolith and extract services when you hit scaling/team pain
Database per service is non-negotiable for true independence
Sagas replace distributed transactions (embrace eventual consistency)
Circuit breakers are essential — one slow service will take down the whole system without them
Observability (tracing, logging, metrics) is twice as important in microservices

The Go Gateway: Confessions of a Professional Middleman

Gouranga Das Samrat — Fri, 24 Jul 2026 04:00:00 +0000

This is Part 2 of "The Muse of Microservices," a series on the system design behind Polyhymnia, a deliberately overengineered random quote generator. If you haven't met the whole cast yet, Part 1 covers the architecture end to end — worth reading first.

Inside Polyhymnia's single HTTP endpoint — orchestration vs choreography, deadline budgets, retry loops, and the art of turning three gRPC calls into one honest JSON response.

One endpoint, zero excuses

The entire public surface of Polyhymnia is a single route:

GET /api/quote

No path parameters, no query strings, no body. And yet everything the system does routes through it, because the Go service in front of it isn't a proxy — it's an orchestrator. It's the one component in the whole stack that knows, and enforces, the correct order of operations: fetch every ID, hand them to the randomizer, fetch the winning quote. Nothing downstream is trusted to figure that sequence out on its own, and nothing downstream is reachable from outside the cluster at all. Every other service in Polyhymnia speaks gRPC exclusively; Go is the only translator standing between "the internet" and "the mesh."

That's a real, deliberate pattern with a name: the API gateway. Its value isn't glamorous — it's a bulkhead. The public contract (plain HTTP, JSON) is decoupled from the internal contract (gRPC, protobuf), which means the internal services are free to change their wire format, their language, even their topology, without a single line of frontend code ever noticing.

Orchestration, chosen on purpose

There are two broad ways to coordinate a multi-step workflow across services. Choreography has each service react to events as they arrive — nobody's in charge, the sequence emerges from a chain of triggers. Orchestration puts one component in the driver's seat, explicitly calling each step and deciding what happens if one fails.

Polyhymnia picks orchestration, and the gateway's handler reads almost like pseudocode because of it: fetch the ID list, pass it to the randomizer, fetch the chosen quote, encode the response. Three sequential RPCs, one after another, all driven from a single function.

That choice has real tradeoffs, and Polyhymnia's small enough that you can see both sides clearly:

You get it: a request's entire lifecycle is traceable in one place. Reading the handler top-to-bottom tells you exactly what happened for any given request — there's no event bus to reconstruct, no implicit ordering to reverse-engineer.
You pay for it: the gateway is now coupled to the internal shape of the workflow. If a third backend service joined the pipeline tomorrow, the gateway's code — not some independent event contract — is what has to change.

For a three-step, single-consumer pipeline like this one, orchestration is the right call. It stops being the right call once you have a dozen services and every team wants to own their slice of the sequence independently — which is exactly the kind of judgment call real system design is made of.

A deadline budget, not a stopwatch per hop

Here's a detail that's easy to skim past and genuinely worth sitting with:

ctx, cancel := context.WithTimeout(r.Context(), callTimeout) // 5 seconds
defer cancel()

That one context is threaded through all three downstream calls. It is not "5 seconds per RPC" — it's a single 5-second budget for the entire request, shared across GetAllIds, SelectRandomId, and GetQuoteById combined. If the first call to Rust takes 4.5 seconds for whatever reason, the C++ engine and the second Rust call are left splitting the remaining half-second between them, whether they like it or not.

This is deadline propagation, and it's one of those concepts that looks academic right up until you've been paged for a service that kept retrying a downstream call for 30 seconds because nobody told it the caller had already given up 29 seconds ago. Polyhymnia's version is about as simple as deadline propagation gets — one shared context.Context, no per-hop sub-budgets — but the underlying idea, that a deadline should travel with a request rather than being reinvented at every hop, is exactly the idea production systems lean on.

Retrying like you mean it (sort of)

Before the gateway serves a single request, it has to actually connect to both backends:

const (
    maxRetries    = 30
    retryInterval = 500 * time.Millisecond
)

dialGrpc loops up to 30 times, sleeping 500ms between attempts, using grpc.WithBlock() so the dial call won't return until a connection actually succeeds (or every attempt is exhausted, at which point the gateway logs a fatal error and exits). That's roughly a 15-second window for the Rust and C++ services to become reachable before the gateway gives up on life entirely.

It's worth naming precisely what this is and isn't. It's a fixed-interval retry with a blocking dial — simple, predictable, and completely adequate for a small number of startup-order dependencies. It is not exponential backoff, and it has no jitter. That's a fine tradeoff here: with only two dependencies and a bounded startup window, a fixed 500ms cadence isn't going to hurt anyone. The moment you're retrying against a service under real load with many callers doing the same thing at once, a fixed interval turns into a synchronized herd hammering the same endpoint every 500ms — which is precisely the scenario exponential backoff with jitter exists to prevent. Good instinct to have in your back pocket even when, like here, the simpler tool is the right one for the job.

An error taxonomy that actually means something

The gateway's error handling is short, but it draws a distinction a lot of production code gets lazy about:

if len(idList.GetIds()) == 0 {
    writeError(w, http.StatusNotFound, "no quotes available")
    return
}

...versus:

if err != nil {
    writeError(w, http.StatusBadGateway, "failed to fetch quote ids")
    return
}

An empty result and a failed call are not the same kind of problem, and the gateway refuses to collapse them into one generic "something went wrong" response. A database with zero rows is a legitimate state of the world — the request succeeded, there was simply nothing to return, so it's a 404. A gRPC call that errors out means a dependency is broken or unreachable — that's a 502 Bad Gateway, correctly blaming the infrastructure rather than the caller. A request with the wrong HTTP method gets its own 405. Three distinct failure modes, three distinct status codes, and a frontend that can actually tell them apart if it ever needs to.

It's a small thing. It's also exactly the kind of small thing that separates an API that's pleasant to build against from one that returns 500 for everything and makes every client guess.

The localhost trap

Both the gateway's dial addresses and the Rust service's listen address are pinned to explicit IPv4:

rustDbAddr    = "127.0.0.1:50051"
cppEngineAddr = "127.0.0.1:50052"

The code comment explains why, and it's a genuinely useful bit of networking trivia: on some systems, localhost resolves to 127.0.0.1 (IPv4) while a server that bound to [::1] (IPv6-only) is listening on a different address entirely. The result is a maddening "connection refused" error against a server you can plainly see is running. Pinning both sides to the same explicit address family sidesteps the ambiguity completely.

File that one away — it's the kind of bug that costs an afternoon the first time and takes ten seconds to recognize every time after. It'll also come back with a twist in Part 4, once these same services move into separate Docker containers and "127.0.0.1" stops meaning what everyone assumed it meant.

What's next

The gateway's job is to ask the right questions in the right order and translate the answers into JSON. It doesn't know how a random ID actually gets picked, and it definitely doesn't know how the database it's querying decided to keep itself safe. Part 3 goes downstream into both of those services — Rust's data-ownership model and C++'s spectacularly unnecessary approach to picking a number — where the real personality of this project lives.

Meet Polyhymnia: The Random Quote Generator That Took the Scenic Route

Gouranga Das Samrat — Thu, 23 Jul 2026 04:00:00 +0000

Why a one-line SQL query became a four-language, five-hop distributed system — and why that's secretly a great way to learn system design.

The most over-qualified button you'll ever click

Somewhere in this project's proto file lives a service called Randomizer. Its entire job — its only job — is to pick one number out of a short list. That's it. That's the service. It has its own gRPC port, its own protobuf messages, and its own opinions about entropy.

That single fact tells you everything about Polyhymnia: a random quote generator that could have been a SELECT quote, author FROM quotes ORDER BY RANDOM() LIMIT 1; and one HTML page, and instead is four services, three languages, one build system juggling three different toolchains, and a network hop count that would make a CDN blush.

This is the first post in a four-part series pulling apart why it's built this way and what it's quietly teaching you about real distributed systems along the way. No PhDs required — just a tolerance for services that do less work than the plumbing required to reach them.

What it does vs. how it does it

Functionally, Polyhymnia is almost insultingly simple: you click a button, you get a quote and its author. There is no user data, no auth, no pagination, no edge cases worth mentioning.

Architecturally, that click sets off a small relay race:

┌────────────┐   HTTP GET    ┌──────────────┐
│  Frontend  │ ────────────▶ │  Go Gateway  │
│ (JS/HTML)  │ ◀──────────── │ (Orchestrator│
└────────────┘   JSON quote  └──────┬───────┘
                                     │
                 ┌───────────────────┼───────────────────┐
                 │ 1. GetAllIds       │ 3. GetQuoteById    │
                 ▼                    │                    ▼
          ┌─────────────┐             │             ┌─────────────┐
          │  Rust "db"  │◀────────────┘             │  Rust "db"  │
          │  service    │                            │  service    │
          └──────┬──────┘                            └─────────────┘
                 │ 2. SelectRandomId
                 ▼
          ┌─────────────┐
          │  C++ engine │
          │ (Randomizer)│
          └─────────────┘

Five network hops. Three separate gRPC calls. Two backend services that only exist to serve one HTTP endpoint. The gap between "what this does" and "what it took to do it" is the entire point — and it's a deliberate constraint, not an accident of scope creep.

The name is doing a lot of work

Polyhymnia was the muse of sacred poetry and song in Greek mythology — a fitting name for something that ships one-liners. But the real pun is etymological: Poly (many) + Hymnia (words/song). Swap "many words" for "many languages," and you get the actual joke — this is a message (a quote) delivered through a genuinely polyglot pipeline: Go, C++, Rust, and JavaScript, four languages for one payload.

It's a rare case of a project name being both a classics reference and an architecture diagram.

The tech stack, with a straight face

Layer	Language	Responsibility
Frontend	JavaScript / HTML / CSS	One button, one quote, zero frameworks
API Gateway / Orchestrator	Go	HTTP entrypoint, drives the 3-step gRPC workflow
Mathematical Engine	C++	Picks one ID at random, with unnecessary ceremony
Database Manager & Safety Layer	Rust	Owns `quotes.db` — the only service allowed to touch SQLite
Database	SQLite	Three columns: `id`, `quote`, `author`
Inter-service transport	gRPC + Protobuf	One shared contract, three independent codegens

Each row on that table is a genuine, deliberate architectural decision, not a random language pull from a hat. Go fronts the request because it's comfortable being a thin, concurrent I/O layer. Rust owns the only piece of mutable state in the system because "safety layer" is basically Rust's job title. C++ shows up purely to demonstrate that even picking a random array index can be turned into a systems-programming set piece, complete with hand-mixed entropy and pointer arithmetic. We'll get intimate with both of those services in Part 3.

Overengineering as a teaching tool, not a punchline

It would be easy to wave this project off as a joke and move on — and it is a joke, proudly, in the README's own words. But strip away the comedy and what's left underneath is a genuinely well-formed distributed system, built at a scale small enough to actually read in one sitting:

A single ingress point. Only the Go gateway speaks plain HTTP; everything downstream is gRPC-only. That's a real API gateway pattern, not a toy version of one.
Contract-first design. One .proto file is the single source of truth for every message and RPC. Three languages generate their stubs from it independently — nobody hand-writes serialization code.
Strict data ownership. Exactly one service is allowed to open the SQLite file. Nothing else touches it, ever, directly or otherwise.
A stateless compute service sitting next to a stateful one. The C++ engine holds no data and could be replicated infinitely with zero coordination. The Rust service holds the only state in the system and has to be far more careful about it. That contrast is a genuinely useful thing to internalize, and Polyhymnia hands it to you on a plate.
Orchestration, not choreography. The Go gateway explicitly sequences three calls rather than having services react to each other's events. That's a real, debatable system design choice with real tradeoffs — not an accident.

None of that is complicated in isolation. What makes Polyhymnia worth studying is that it's small enough to hold the entire system in your head at once, while still containing every major decision point a much bigger distributed system has to make. It's a system design course condensed into something you can git clone and run on a laptop in a few minutes.

Where this series goes from here

Over the next three posts, we're taking the project apart layer by layer:

Part 2 — the Go gateway: the art of being a professional middleman, and what its retry loop and error-mapping logic quietly teach about deadline budgets and failure taxonomy.
Part 3 — Rust's safety layer and C++'s beautifully unnecessary randomness: data ownership, parameterized queries, and what "overengineered" entropy actually looks like under the hood.
Part 4 — shipping the chaos: the shared protobuf contract, three independent codegen pipelines, Docker packaging, and one networking gotcha hiding in plain sight in the compose files.

If you want to follow along with the actual code, the repository is here: GourangaDasSamrat/Polyhymnia. Clone it, click the button a few times, and then come back for Part 2 — where we put the Go gateway under a microscope.

Fan-out Architecture

Gouranga Das Samrat — Sun, 19 Jul 2026 02:00:00 +0000

One-liner: Fan-out means one event triggers multiple parallel downstream processes — like dropping a pebble in water and watching rings spread outward.

📌 The Pattern

[One Event] ──► [Fan-out Layer] ──► [Process A]
                                 ──► [Process B]
                                 ──► [Process C]

All processes run simultaneously and independently.

🔄 Fan-out on Write vs Fan-out on Read

This distinction is critical for feed/timeline systems (Twitter, Instagram).

Fan-out on Write (Push)

When a user posts, immediately write to all followers' timelines:

User posts → find 1,000 followers → push to each follower's feed cache
Read: O(1) — just read pre-computed feed
Write: O(followers) — expensive for celebrities

Fan-out on Read (Pull)

When a user opens their feed, compute it on the fly:

User opens feed → fetch followed users → merge their posts → sort
Read: O(following × posts) — slow at scale
Write: O(1) — just save the post

Hybrid (Best for large systems)

Normal users → fan-out on write
Celebrities (>1M followers) → fan-out on read
Merge both at read time

🌍 SNS → SQS Fan-out (AWS)

See Pub-Sub (SNS) for the full implementation.

The canonical pattern:

[Event] → [SNS Topic] → [SQS Queue A] → [Consumer A]
                      → [SQS Queue B] → [Consumer B]
                      → [SQS Queue C] → [Consumer C]

SQS between SNS and consumers adds durability and independent retry per consumer.

Pub-Sub (SNS) & Fan-out Architecture

Gouranga Das Samrat — Sat, 18 Jul 2026 02:00:00 +0000

One-liner: Pub-Sub lets publishers broadcast events to all interested subscribers without knowing who they are. Fan-out ensures one event triggers many parallel actions.

📌 Pub-Sub Model

In traditional queues: one message → one consumer.

In Pub-Sub: one message → all subscribers receive a copy.

[Publisher] → [Topic] → [Subscriber A]
                      → [Subscriber B]
                      → [Subscriber C]

☁️ AWS SNS (Simple Notification Service)

SNS is AWS's managed Pub-Sub service.

Key Concepts

Term	Meaning
Topic	A named channel that messages are published to
Publisher	Sends messages to a topic
Subscriber	Receives copies of messages
Subscription	A subscriber's registration to a topic

Supported Subscriber Types

SQS Queue — most common (durable, retry-able)
Lambda — serverless processing
HTTP/HTTPS endpoint — webhooks
Email / SMS — direct notifications
Mobile Push (FCM, APNs) — push notifications

🔄 SNS Message Flow

Order Placed Event
      ↓
[SNS Topic: order-events]
      ├──► [SQS Queue] → [Payment Lambda]
      ├──► [SQS Queue] → [Email Service]
      ├──► [SQS Queue] → [Inventory Service]
      └──► [Lambda]    → [Analytics Ingestion]

All four happen simultaneously and independently.

🌟 Fan-out Architecture

Fan-out = one event → many parallel processes

SNS → SQS Fan-out Pattern (Most Robust)

[Event Source]
      │
      ▼
[SNS Topic]
   │    │    │
   ▼    ▼    ▼
[SQS] [SQS] [SQS]
  │     │     │
  ▼     ▼     ▼
[Svc A][Svc B][Svc C]

Why SQS in between SNS?

Durability — if Service A is down, message waits in SQS
Rate control — each service processes at its own pace
Retry — SQS handles retries independently per service
Decoupling — SNS doesn't care about downstream state

Direct SNS Fan-out (No SQS buffer)

[SNS Topic] → [Lambda A]  (immediate, no buffer)
            → [Lambda B]

Risk: If Lambda B fails, message is lost (no retry queue).

🏗️ Real-World Example: User Registration

User signs up → [Auth Service] → publishes to SNS: "user.created"
                                          │
                              ┌──────────┼──────────┐
                              ▼          ▼          ▼
                         [SQS]→     [SQS]→     [SQS]→
                         [Welcome   [Analytics  [Referral
                          Email]     Track]      Credit]

Each service is independent. If email service is down → message waits → sent later.
Adding a 4th service (e.g., Slack alert)? Just add a new SNS subscription. Zero code change in Auth Service.

🆚 SNS vs SQS vs Kafka

Feature	SNS	SQS	Kafka
Pattern	Pub-Sub (broadcast)	Queue (single consumer)	Log (many consumers, replay)
Message retention	None (fire and forget)	Up to 14 days	Configurable (forever)
Fan-out	✅ Native	❌ (one consumer)	✅ (consumer groups)
Replay	❌	❌	✅
Ordering	Best effort	FIFO option	Per-partition
Durability	With SQS subscription	✅	✅
Managed	✅ AWS	✅ AWS	Self-hosted / Confluent

🔔 SNS for Push Notifications

SNS handles mobile push at scale:

[Your Server] → SNS publishes to platform endpoint
                     ↓
             [FCM] (Android) → User's phone
             [APNs] (iOS) → User's phone

SNS manages platform credentials, token management, delivery.

📊 Message Filtering (SNS)

Subscribers can filter messages by attributes:

// SNS message with attributes
{
  "order_type": "premium",
  "region": "us-east",
  "amount": 999
}

// Subscription filter policy (Premium SQS only gets premium orders)
{
  "order_type": ["premium"]
}

Only premium orders go to the premium processing queue — others are filtered out.

🎨 Diagram

The diagram shows:

Publisher → SNS Topic → multiple SQS queues → multiple consumers
Fan-out flow on user.created event
Message filtering at subscription level
DLQ for each consumer independently

🔑 Key Takeaways

Pub-Sub (SNS) = broadcast; Queue (SQS) = point-to-point
Always use SNS → SQS fan-out for durability (SQS buffers failures)
Pub-Sub enables zero-code service addition — new subscribers need no changes to publisher
Message filtering reduces unnecessary processing across services

Message Queues (SQS)

Gouranga Das Samrat — Sun, 12 Jul 2026 02:00:00 +0000

One-liner: A message queue decouples services by letting a producer drop messages into a queue, where consumers pick them up asynchronously — neither needs to be available at the same time.

📌 The Problem: Tight Coupling

User places order →
  [Order Service] → calls [Payment Service] → waits →
                  → calls [Email Service] → waits →
                  → calls [Inventory Service] → waits →
  All done! (3s total, 3 failure points)

Problems:

High latency — user waits for every downstream service
Cascading failure — if Email service is down, order fails
Tight coupling — Order service knows about all downstream services
No retry logic — failures are permanent

💡 The Solution: Message Queue

User places order → [Order Service] → DROP message in queue → RESPOND (fast!)
                                              ↓
                                   [Queue]
                                   /    |    \
                          [Payment] [Email] [Inventory]  ← process at their own pace

Benefits:

Decoupling — services don't know about each other
Resilience — downstream service can be down; message waits in queue
Load leveling — queue absorbs traffic spikes
Retry — failed processing retried automatically

🔄 Queue Concepts

Producer & Consumer

[Producer] → push message → [Queue] → pull message → [Consumer]

Producer: Creates and sends messages
Consumer: Reads and processes messages
Queue: Durable buffer between them

Message Lifecycle

1. Producer sends message → Queue stores it
2. Consumer polls queue → Queue marks message as "in-flight" (invisible to others)
3. Consumer processes → Success: Consumer deletes message
                     → Failure: After visibility timeout, message reappears

Visibility Timeout

Consumer pulls message → message hidden for 30 seconds
If consumer crashes → 30 seconds later, message reappears → another consumer picks it up
If consumer succeeds → consumer deletes message → gone forever

☁️ AWS SQS (Simple Queue Service)

Standard Queue

Unlimited throughput
At-least-once delivery — message may be delivered more than once!
Best-effort ordering — no strict FIFO

FIFO Queue

Exactly-once processing — deduplication
Strict ordering — first in, first out
Limited throughput — 300 msg/sec (or 3000 with batching)

SQS Key Concepts

Message retention:  Up to 14 days (default: 4 days)
Visibility timeout: How long consumer has to process (default: 30s)
Dead Letter Queue:  Where failed messages go after N retries
Max message size:   256 KB

SQS + Lambda (Serverless Consumer)

[SQS Queue] ──triggers──► [Lambda Function]
                           (auto-scales to process backlog)

🔀 Queue Patterns

Work Queue (Task Distribution)

[Producer] → [Queue] → [Worker 1]
                     → [Worker 2]
                     → [Worker 3]

Each message processed by exactly ONE worker

Use case: Image resizing, email sending, PDF generation

Fan-out via Multiple Queues

[Event] → [SNS Topic] → [Queue A] → [Consumer A]
                      → [Queue B] → [Consumer B]
                      → [Queue C] → [Consumer C]

Each consumer gets a copy of the message. (See: SNS/Pub-Sub)

Priority Queue

High priority queue → processed first by consumers
Low priority queue  → processed when high is empty

♻️ Dead Letter Queue (DLQ)

When a message fails to process after N retries → move to DLQ:

[Queue] → [Consumer fails 3 times] → [DLQ]
                                          ↑
                                    Ops team inspects,
                                    fixes bug, redrives

// SQS Redrive Policy
{
  "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123:my-dlq",
  "maxReceiveCount": 3 // after 3 failures, move to DLQ
}

⚠️ Idempotency — Critical Requirement

Standard queues deliver at-least-once. Your consumer MUST be idempotent:

Message: "charge user 42 for $99"

❌ Non-idempotent: Just run the charge every time → user charged twice!
✅ Idempotent: Check if orderId already processed → skip if yes

async function processPayment(message) {
  const { orderId, userId, amount } = message;

  // Check idempotency key
  const alreadyProcessed = await redis.get(`payment:${orderId}`);
  if (alreadyProcessed) return; // skip duplicate

  await chargeUser(userId, amount);
  await redis.setex(`payment:${orderId}`, 86400, "1"); // mark done
}

📊 SQS vs Other Queues

Feature	AWS SQS	RabbitMQ	Apache Kafka
Type	Managed cloud	Self-hosted broker	Distributed log
Retention	14 days	Until consumed	Configurable (forever)
Ordering	Best-effort/FIFO	Per-queue	Per-partition
Replay	❌	❌	✅ (replay from offset)
Throughput	Very high	High	Very high
Use case	Decoupling tasks	Complex routing	Event streaming

🎨 Diagram

The diagram shows:

Producer → Queue → Multiple consumers
Visibility timeout flow (in-flight → reappear)
Dead Letter Queue after max retries
Fan-out via SNS → multiple queues

🔑 Key Takeaways

Queues decouple services and absorb traffic spikes
Visibility timeout ensures messages aren't lost if a consumer crashes
Always implement idempotent consumers with SQS Standard
DLQ is essential for debugging failed messages
Use FIFO when order matters; Standard when throughput matters

DEV Community: Gouranga Das Samrat

Illegal Instruction: How I Woke Up a Year-Old 'Not Planned' Bug and (Almost) Fixed Vite on Android

Chasing the actual cause

The ghost in the closet

Going back with receipts

Testing it on the actual crashing device

Then I tweeted it

The side quest nobody asked for

Where it stands right now

Why I bothered writing this up

API Versioning

❓ Why Version Your API?

🏗️ 4 Versioning Strategies

1. URI Path Versioning ✅ (Most Common)

2. Query Parameter Versioning

3. Header Versioning

4. Content Negotiation (Accept header)

🔀 Routing Versions

📋 Breaking vs Non-Breaking Changes

🔄 Migration Lifecycle

🏷️ Semantic Versioning for Internal APIs

🗂️ Version Coexistence Patterns

Shared Code, Version-Specific Transform

✅ Pros

❌ Cons

⚖️ When to Use / When NOT to Use

Data Consistency in Microservices — Saga Pattern

❓ The Problem: Distributed Transactions

🎭 The Saga Pattern

Example: E-commerce Order Flow

🏗️ Two Implementation Styles

1. Choreography (Event-driven, decentralized)

2. Orchestration (Centralized coordinator)

🆚 Choreography vs Orchestration

💾 Saga State Storage

✅ Pros

❌ Cons

⚖️ When to Use / When NOT to Use

Shipping the Chaos: One Contract, Three Codegens, and a Docker Gotcha Hiding in Plain Sight

One file, three languages, zero hand-written serialization

A small but telling asymmetry: reflection

justfile: one command runner for three build systems

Packaging: four Dockerfiles, three of them multi-stage

Two compose files, two audiences

The gotcha: what "127.0.0.1" means depends on where you're standing

What you'd change to make it "real"

Closing the loop

Trust No One: Rust's Safety Layer and C++'s Beautifully Unnecessary Randomness

Rust: the database's only bodyguard

One owner, no exceptions

Parameters, not string concatenation

The concurrency model, and its ceiling

Idempotent bootstrapping

C++: a coin flip with a PhD's worth of ceremony

Stateless, and proud of it

What's next

Containers & Docker

📌 The Problem Before Containers

💡 Containers vs Virtual Machines

Virtual Machine (VM)

Container

🐳 Docker Core Concepts

Image

Container

Registry

📁 Dockerfile Best Practices

Layer Caching — Order Matters!

Multi-Stage Build (Small Final Image)

.dockerignore

🔧 Docker Compose — Multi-Container Local Dev

☸️ Container Orchestration: Kubernetes (K8s)

Key K8s Concepts

K8s Benefits

Managed K8s Services

🔄 Container in CI/CD Pipeline

🎨 Diagram

🔑 Key Takeaways

Microservices

📌 Monolith vs Microservices

Monolith

`justfile`: one command runner for three build systems