DEV Community: Qihu Zhang

Six services, 6 GB, one command: what it takes to make a microservice stack shippable

Qihu Zhang — Sun, 30 Aug 2026 04:37:16 +0000

I spent a week turning a working six-service system into something a stranger could run. The code barely changed. Almost all of the work was in the gap between "it runs on my machine" and "it runs on yours, on the first try, without me."

That gap is where most side projects quietly die, and it is much wider than it looks from the inside.

The memory budget is a product decision, not an ops detail

Six JVMs, Kafka, PostgreSQL, Redis, a tracing collector, a server-rendered storefront and two static consoles. Thirteen containers. Someone has to run this on a box they pay for monthly, and the size of that box is the first thing they decide — before they read a line of my code.

So I capped the Docker VM at 6 GB and measured a cold start, sampling every two seconds:


Peak, all thirteen containers	3.35 GiB
Steady state after the full product walk	3.26 GiB
Containers killed for memory	0
Building the jars (`mvn clean package`, peak RSS)	530 MiB

The peak matters more than the steady state, and it is the number nobody publishes. Steady state is six JVMs that have finished waking up. The peak is six JVMs initialising at once while Flyway migrates five schemas and Kafka creates its topics — the exact moment a 4 GB box dies. If I had only measured the calm afterwards, I would have been advertising a floor I had never tested.

Three decisions came out of that budget rather than out of taste:

-Xmx256m on every service, and Kafka's heap capped explicitly. Kafka's default heap will happily take a gigabyte it does not need on a single-broker demo.
One PostgreSQL instance, one database per service. Five containers of Postgres would have been more purist and would have cost about 400 MB more for nothing. The isolation that matters is schema ownership, not process count.
I stopped at six services. Splitting further was tempting and would have pushed the floor to 16 GB — which does not make the design better, it makes the product unsellable.

Cold start is the gate

The single highest-value thing I built was not a feature. It is one script that:

builds the jars,
tears the stack down including volumes,
brings all thirteen containers up from nothing,
waits for every health check,
then walks the whole product path with 32 assertions — a buyer applies to sell, an admin approves them, the shop is created, a cross-shop order splits, stock is reserved, payment moves it, each shop fulfils its own slice, one is refunded, the rest is billed with commission and paid out.

One command, exit code 0 or a specific failed assertion. It runs in about ninety seconds to all-green.

This turned out to matter more than the tests. Unit and integration tests told me my code was right given my machine's state. The gate tells me a stranger's first command works. Those are different claims, and only the second one is what shipping means.

It caught things nothing else did. Deleting three products from a seed file left another service's seed generating stock rows for SKUs that no longer existed — orphan ids that then collided with the next id the platform handed out. Every test passed. The gate failed on a duplicate key, in a step unrelated to the change.

And near the end I moved the gate one step further out: unzip the actual distributable into a clean directory and run it from there. A working tree that runs proves nothing about the archive you upload — missing files, an over-eager ignore rule, a generated file that never got committed, all of it only surfaces after extraction.

Say what it does not do, early and in the same voice

The payment gateway is mocked. There is one interface with charge and refund, a mock implementation, and documentation for swapping in a real processor including the part people skip — the webhook is not optional, events arrive twice, and you move the order to paid from the webhook rather than from the browser's return trip.

I could have buried that. Instead it is in the second paragraph of the product description, next to the memory floor and next to "this will not run on shared hosting." My reasoning is not nobility, it is arithmetic: a buyer who discovers the limit after purchase writes a refund request and a one-star review, and both cost more than the sale.

The same applies to single currency (amounts are integer cents in one currency throughout — multi-currency is a project, not a setting) and to there being no email at all. Writing those down was uncomfortable for about ten minutes and then it made the documentation better, because each limit has an obvious next question and answering it is most of a good manual.

What actually consumed the week

Not the services. The list looked like this:

A demo mode that answers the whole app from inside itself. One build flag and the mobile app serves its own catalogue, splits its own orders and runs its own fulfilment clock. Three jobs from one implementation: a reviewer walks the product without deploying anything, the live preview becomes a static file, and screenshots come out of the same journey script the real stack uses.
Documentation for the person who does not use a terminal. Every command explained, and a section on doing database work through pgAdmin instead. My previous product's review cycle was three rounds of "please cover this for non-technical buyers." Cheaper to write it up front.
The platform I had never actually built for. I had shipped four milestones on an iOS simulator. The first Android build crashed instantly: the package rename had left MainActivity in the old Kotlin folder, so the class in the manifest did not exist. It had been broken for months. iOS has no Kotlin, so nothing ever noticed. The app name on the launcher was still the project slug, and Android has blocked plain HTTP since API 28, which no amount of iOS testing would have told me.

That last one is the lesson I would keep if I could only keep one: a platform you have never actually run is entirely unverified, and the defect density there is high. Six real defects in one afternoon, in code that had passed every gate I had.

The part that generalises

Making something shippable is mostly the work of removing yourself from the instructions. Every step that only works because of what you already know — a port that happens to be free, a JDK that happens to be 21, an emulator that happens to reach your laptop, a file that happens to exist because you made it once by hand — is a step that fails for everyone else.

The way to find those steps is not to think harder. It is to start from nothing, in a clean directory, and let a script tell you where you were leaning on yourself.

What may degrade, and what must fail fast

Qihu Zhang — Sat, 29 Aug 2026 04:09:47 +0000

Six services sit between a buyer clicking "buy" and a vendor eventually getting paid: auth, gateway, catalog, inventory, order, and now settlement. Any one of them can be slow, restarting, or fully down at any moment — that's the normal condition of a distributed system, not an incident you fix once and move past. The question worth writing down isn't "how do we prevent that." You can't, fully. It's: when it happens, which requests should degrade to a worse-but-safe answer, and which must refuse outright rather than guess?

The instinct most people reach for first is a single global rule — either "always degrade gracefully" or "always fail fast." Both are wrong the moment you apply them uniformly across services that don't carry the same kind of risk. I found the actual criterion isn't "how important is this service." It's whether the business cost of guessing wrong is reversible.

The criterion, not the checklist

A service whose worst-case wrong guess is "the buyer sees something slightly stale" may degrade — the guess self-corrects the moment the dependency comes back, and nothing irreversible happened while it was wrong. A service whose worst-case wrong guess is "we sold something we don't have" or "money moved that shouldn't have" must fail fast, because there is no UPDATE statement that un-ships a package or un-spends a vendor's payout. That's the whole rule. Everything below is just applying it, service by service, to real code that's actually running.

`catalog` down → degrade

The storefront can serve a cached snapshot of what it last saw, or an empty result for an uncached query, and let the buyer retry. Worst case: a product looks unavailable that's actually fine, or a price is a few seconds stale. Both are cosmetic and self-correct the instant catalog comes back. Nothing about "can't browse right now" commits the business to anything.

`inventory` down → fail fast

order.create()'s call to inventory's reserve endpoint is synchronous, and it has no fallback path — deliberately. If the fallback were "let the order through and reconcile inventory later," the failure mode is overselling the last unit of a scarce SKU to five buyers at once, discovered only when four of them can't be shipped what they paid for. Checkout returning a 503 and telling the buyer to retry is strictly better than checkout succeeding and lying. This is the same conditional-update invariant I wrote about in the inventory-reservation piece — WHERE available >= ? either commits or it doesn't, and there's no version of "degrade gracefully" that doesn't mean "sell something you can't deliver."

`settlement` down → degrade to zero visible effect

This is the newest row in the table, and the one that only exists because M3 added a sixth service. order never calls settlement synchronously — the only coupling is order publishing order-paid, sub-order-delivered, and sub-order-refunded to Kafka and moving on. Settlement being down means those events queue up unconsumed. Checkout, payment, and fulfillment are entirely unaffected, because nothing on the buyer path is waiting for settlement to acknowledge anything. The cost lands entirely on vendors getting paid later than usual — bounded and recoverable, because Kafka retains the backlog and settlement's own idempotent consumer means catching up on a day of queued events produces the same end state as processing them on time. I didn't just argue this — the M3 gate exercises it directly: place an order, split it, pay, mark it delivered, and only afterward bring the billing sweep around to consume the resulting events. Nothing in the buyer-facing steps has ever depended on settlement being reachable, by construction, not by the gate happening to get lucky with timing.

`auth` down → degrade for issued tokens, fail fast for new ones

The gateway validates JWTs locally against a cached JWKS — a request carrying a still-valid signed token is authorized without calling auth at all, so a buyer mid-session keeps shopping through an auth outage with zero visible effect. A brand-new login does have to reach auth, and that has to fail rather than accept a token it can't verify. Skipping that check isn't a degradation choice, it's a decision to stop checking signatures — a security regression wearing a resilience costume.

The table, for anyone paging through an incident

Dependency down	Buyer-facing behavior	Why
`catalog`	Cached/stale/empty results	Wrong guess is cosmetic and reversible
`inventory`	Checkout fails fast (503)	Wrong guess is overselling — irreversible
`settlement`	No visible effect at all	Not on the buyer path; backlog drains later
`auth` (existing token)	Unaffected (local JWT verify)	No call needed
`auth` (new login/JWKS miss)	Fails fast	Can't verify what we can't check

The value of writing this down isn't the table itself — it's that "which side of this table a service belongs on" is a property of what the wrong guess costs, not of how central the service feels in an architecture diagram. Add a seventh service later, and the question to ask is the same one this table already answers: can a wrong-guess response be undone by a later correction, or does it commit the business to something — stock shipped, money moved — that can't be clawed back cleanly? The answer decides the failure mode before a single line of fallback code gets written, not after an incident reveals it was wrong.

It's also worth naming what this framework is not saying. It's not "settlement doesn't matter" — money not reaching vendors on time matters a great deal to the vendors waiting on it. It's that the recovery is cheap and bounded (drain the backlog; reconciliation catches anything that still doesn't add up), so it belongs in the "degrade" column even though the thing it's protecting is high-stakes. Reversibility, not stakes, is the axis this sorts on.

Where a circuit breaker fits, and where it doesn't

Resilience4j sits at the gateway, and it's tempting to treat "add a circuit breaker" as a solved, mechanical step once you've written a resilience ADR. It isn't, because a circuit breaker's failure-rate threshold and open-state duration are business judgments wearing config-file syntax, not defaults you tune once and forget. Set the threshold too sensitive and you trip the breaker on an inventory service having one slow GC pause, converting a 20ms blip into checkout being unavailable for the whole open-state window — you've just built your own worse incident on top of a fake one. Set it too lax and the breaker never opens before a genuinely struggling dependency has already caused the actual harm you built the thing to prevent.

The number that should set that threshold isn't a Resilience4j tutorial's example config — it's the answer this ADR already worked out for each dependency. A breaker in front of inventory should trip fast and stay open long enough that a struggling instance gets a real chance to recover, because every request that gets through to a failing inventory call is a request that either fails fast (correct) or, worse, times out slowly while a buyer stares at a spinner. A breaker in front of catalog, by contrast, can afford to be far more patient, because a slightly-too-eager open state there just means a few extra buyers see a stale result they'd have seen anyway. Same library, same default config shape, opposite tuning — because the two dependencies sit on opposite sides of the reversibility line this whole piece is about. The circuit breaker doesn't replace the judgment call. It just automates executing whichever judgment call you already made.

This is part of a series on building a multi-vendor commerce platform. The open-source half, stallora-cloud-starter, carries the gateway and auth services this piece describes. Next up: shipping a six-service stack that buyers can actually run on a 6 GB VPS.

Never split the money on payment success

Qihu Zhang — Wed, 19 Aug 2026 06:18:45 +0000

Here's a line of code that looks completely reasonable: the moment a buyer's card is charged, credit the vendor's share to their payout balance. It's the obvious anchor — paid_at is already a timestamp sitting right there on the order, the money genuinely did just arrive, and "bill the vendor when we got paid" reads like the simplest possible rule.

It is also, in a marketplace with real buyers and real returns, the most expensive line of code you can write, because it commits you to a promise you can't always keep: that a refund is just a database correction. It isn't. I built the settlement service for a multi-vendor platform around the opposite anchor — bill a vendor only after the order is delivered and the return window has closed — and the reason isn't academic purity about escrow. It's that the alternative turns every post-payment refund into a collections problem.

What "bill on payment" actually commits you to

Walk through what a refund has to do under a paid_at anchor. The vendor's cut was billed the instant the charge cleared. In this system, "billed" is what makes a bill eligible to be locked into a payout and paid out — so by the time a buyer returns the item three days later, that money may already have left the platform and landed in the vendor's bank account. The refund can't just flip a status column anymore. Now you're chasing the vendor for money back, netting it against whatever they're owed next cycle, or eating the loss yourself. None of those are database operations. They're accounts-receivable operations, and they involve a second party who didn't do anything wrong and may not appreciate a clawback request.

The fix people reach for is "well, don't pay out until some grace period after billing." But that's just re-deriving a return-window anchor through the back door, badly — now you have two clocks (billing and payout-eligibility) that both have to track the same real-world fact (has the buyer's refund window closed), and they can drift out of sync. Easier to have one clock.

The anchor: delivered, plus the return window

So the actual anchor is delivered_at + RETURN_WINDOW. A sub-order becomes billable only once it's been marked delivered and the configured return window has elapsed with nothing excluding it. Here's the query that decides what's billable, straight from the running service:

SELECT id, sub_order_id, order_id, vendor_id, vendor_name, gross_cents, currency, lines::text AS lines
FROM pending_settlement
WHERE bill_id IS NULL AND excluded = FALSE AND delivered_at IS NOT NULL
  AND delivered_at < now() - CAST(? AS interval)
ORDER BY id LIMIT ? FOR UPDATE SKIP LOCKED

Three conditions have to hold before a row is even a candidate: nothing has billed it yet (bill_id IS NULL), nothing has excluded it (excluded = FALSE), and the return window has actually passed since delivery. FOR UPDATE SKIP LOCKED means a second sweeper running concurrently splits the batch instead of fighting over the same rows — the same posture the inventory service uses for its own timeout scanner, because both are the same shape of problem: a scheduled job claiming a batch of rows without a coordinator arbitrating between instances.

Until all three conditions hold, the platform is just... holding the money. Not processing it, not provisionally crediting it, just holding it. That's what escrow actually means: cash in, held, released to the vendor only once the window has run out with no refund. It isn't a side effect of the data model. It's the point of the data model.

What a refund actually does under this anchor

Here's the part that makes the whole design earn its keep. A refund inside the window doesn't reach into a payout and try to reverse it. It flips one boolean on a row that was never going to be billed anyway:

public int markExcluded(long subOrderId) {
    // bill_id IS NULL guard: a refund event racing past an already-generated bill must not
    // silently exclude it — that case is reconciliation's to flag, not this consumer's to hide.
    return jdbc.update("UPDATE pending_settlement SET excluded = TRUE "
            + "WHERE sub_order_id = ? AND bill_id IS NULL AND excluded = FALSE", subOrderId);
}

Read the WHERE clause carefully, because the guard is doing real work, not just being defensive for its own sake. bill_id IS NULL means: only exclude a sub-order from billing if it hasn't been billed yet. If a refund event somehow arrives after the sweep already generated a bill for that sub-order — a misconfigured return window, a slow consumer, some edge case nobody designed for — this update quietly does nothing. markExcluded returns 0 rows affected, and the caller only writes the REFUND_EXCLUDED ledger entry when the row count says the exclusion actually took effect. The already-generated bill sits there, untouched, and becomes reconciliation's problem to surface — not something this consumer tries to paper over by pretending it can undo a bill that already exists. A refund's job is to prevent a debt from ever being incurred. It is explicitly not the job of stopping a debt that's already been recorded; that's a different, harder problem, and the code refuses to pretend otherwise.

That's the whole trick, and it's why "refund = exclusion, not a negative bill" is the right way to describe it rather than "refund = accounting adjustment." There's no -$40 line item anywhere. There's a sub-order that simply never crosses into settlement_bill at all. The BillingSweep that turns pending rows into bills is the only writer of that table, and it only ever reads rows where excluded = FALSE — so an excluded row isn't reversed, it's never admitted into the billing story in the first place.

The cost, stated plainly

This isn't free. The vendor gets paid one full return-window later than they would under a paid_at anchor. For a marketplace, that delay is the actual price of being the party holding the bag during the window a buyer is entitled to change their mind. It's not a performance bug to be optimized away — it's what escrow costs, and pretending otherwise (by billing early and hoping refunds stay rare) just moves the cost from "vendor payout is delayed" to "vendor payout occasionally has to be un-done," which is a strictly worse failure mode because it involves a third party's bank account.

I did consider anchoring on paid_at and clawing back via a receivable on refund. It's not a strawman — it's how a lot of systems are built, often because settlement gets bolted onto an order model after the fact and paid_at is the timestamp that's already there. I rejected it here because the fix has to happen eventually anyway once someone experiences a payout that's already spent when a refund shows up, and building the receivable-clawback machinery now, only to replace it with a return-window anchor once that incident happens, is strictly more work than building the right anchor from the start.

What this bought, in numbers

The commission calculation happens once, at billing time, and gets frozen into the bill row. Change a commission rule after that and history doesn't move — I proved this in a test that mutates a category's rate mid-run and asserts already-generated bills keep their original commission amount. Against the running stack: two sub-orders get delivered, the return window (compressed to two minutes for the demo, seven days by default) passes, and the sweep produces two bills where net_cents + commission_cents == gross_cents on every row, using at least two distinct commission-rule scopes to prove the vendor/category/default priority actually resolves differently per line. A second order gets refunded inside its window, and the gate asserts three things directly: the refunded sub-order never gets a bill, its stock comes back, and the still-active pending rows are unaffected. Nothing rolled back. Nothing clawed back. There was simply nothing to bill.

This is part of a series on building a multi-vendor commerce platform. The open-source half, stallora-cloud-starter, carries the outbox library the settlement service consumes events through. Next up: which of this platform's five services are allowed to degrade gracefully when they go down, and which one has to refuse outright — and why that split isn't about which service feels the most important.

The outbox pattern is four sentences in a blog post. Here are three incidents from running it.

Qihu Zhang — Mon, 17 Aug 2026 05:01:42 +0000

Write the event into the same transaction as the row. Poll the table. Send to Kafka. Mark it sent. That's the whole pattern, and every version of it you'll find online stops right there, because at that level of description it's obviously correct — one commit, so a published fact can never disagree with the row that caused it.

I built one, ran it for real between two services, and hit three separate ways for "obviously correct" to still go wrong in practice. None of them are exotic. All three are the kind of thing that only shows up once something is actually polling a table and actually talking to a broker instead of living in a diagram.

Pitfall 1 — the poison message that never gets counted

The relay's job is: claim a batch of unsent rows, send each one, mark it SENT on a broker ack. The first version of that loop looked reasonable:

} catch (Exception e) {
    log.warn("Outbox relay failed for message {}; row stays NEW and will be retried", messageId, e);
    break;   // keep ordering; remaining rows retried next pass
}

break instead of continue is deliberate and correct on its own — you want per-partition ordering preserved, so if row 3 fails, rows 4 through 20 shouldn't jump ahead of it. The bug is what's missing: nothing counts how many times row 3 has failed. If the reason it's failing is permanent — a broker that's actually down for that topic, a message the producer can't serialize, anything that isn't going to resolve itself — the relay retries the same row forever, on every poll, and every row behind it queues up behind a jam that will never clear on its own. No counter, no ceiling, no way to notice except staring at a growing NEW count in the table.

The fix is the boring one: give it a number to compare against.

int attempts = ((Number) row.get("attempts")).intValue() + 1;
if (attempts >= maxAttempts) {
    jdbc.update("UPDATE outbox_message SET status='FAILED', attempts=? WHERE id=?", attempts, row.get("id"));
    log.error("Outbox message {} parked as FAILED after {} attempts", messageId, attempts, e);
    continue;
}
jdbc.update("UPDATE outbox_message SET attempts=? WHERE id=?", attempts, row.get("id"));
break;

After maxAttempts (default 10), the row is parked as FAILED and the loop moves past it — continue, not break, specifically for the row that's given up, so the jam it was causing clears while the ones behind it that might genuinely still succeed keep trying. A FAILED row is something a human can query, alert on, and retry by hand once the root cause is fixed. An infinitely-retried NEW row that never surfaces anywhere is not.

The part worth sitting with: the code that shipped without a counter passed every test that existed at the time. Tests exercised the happy path and one transient failure that resolved itself. Nothing exercised "this row will never succeed," because writing that test requires first believing a message can be permanently unsendable — which is exactly the assumption that's easy to skip when you're picturing the pattern as four sentences.

Pitfall 2 — the trace dies at the queue

HTTP calls carry a trace context automatically. order calls inventory's reservation endpoint, and whatever traced the request into order traces it straight through to inventory — that's what tracing instrumentation for HTTP clients is for.

A polled outbox message gets none of that. By the time the relay picks a row off the table, the HTTP request that originally caused the write is long finished, its context long gone. Kafka doesn't know what a trace is. Left alone, every event on the far side of a KafkaListener starts a trace of its own — inventory deducting a hold shows up in Jaeger as an orphan, with no link back to the checkout that paid for it.

The fix has to be manual, because there's no automatic HTTP-style propagation to lean on. The outbox row carries the trace id as a column, written at insert time from whatever's currently in MDC:

jdbc.update("INSERT INTO outbox_message (message_id, topic, message_key, payload, trace_id) VALUES (?,?,?,?,?)",
        messageId, topic, key, jsonPayload, MDC.get("traceId"));

The relay reads that column back and attaches it as a Kafka header alongside the message id used for deduplication:

String traceId = (String) row.get("trace_id");
if (traceId != null) {
    rec.headers().add(OutboxHeaders.TRACE_ID, traceId.getBytes(StandardCharsets.UTF_8));
}

And the consumer restores it before doing anything else, so every log line and span for the duration of processing that message is tagged with the trace that started at checkout:

var traceHeader = record.headers().lastHeader(OutboxHeaders.TRACE_ID);
String traceId = traceHeader == null ? null : new String(traceHeader.value(), StandardCharsets.UTF_8);
if (traceId != null) {
    MDC.put("traceId", traceId);
}
try {
    // ... process
} finally {
    MDC.remove("traceId");
}

Three lines of column, three lines of header, three lines of MDC — and the payoff shows up as one line grepped out of two different services' logs:

traceId bd1a34b2a4665b240cf284c31db76708 crossed Kafka into the consumer log

Same id, order's side and inventory's side, proving the bridge actually held. Skip this and the trace doesn't error out or warn you — it just quietly stops, and "quietly stops" is the worst failure mode a debugging tool can have, because you don't find out until you're already looking for something and it isn't there.

Pitfall 3 — at-least-once means you will get it twice

Kafka's delivery guarantee is at-least-once, not exactly-once, and that isn't a rare edge case you might hit — it's the normal operating mode. A broker retry, a consumer rebalance, a producer resending after a slow ack: any of them redelivers a message the consumer already handled. The second copy of order-paid has to be a no-op. If it isn't, "convert the hold into a sale" runs twice and the stock ledger disagrees with reality.

The dedup row and the side effect it's guarding both have to commit in the same transaction, or the fix doesn't actually fix anything:

public boolean processOnce(UUID messageId, String consumerGroup, Runnable action) {
    return Boolean.TRUE.equals(tx.execute(status -> {
        int inserted = jdbc.update(
                "INSERT INTO processed_message (message_id, consumer_group) VALUES (?,?) ON CONFLICT DO NOTHING",
                messageId, consumerGroup);
        if (inserted == 0) {
            return false;
        }
        action.run();   // same transaction: the dedup row and the side effect commit together
        return true;
    }));
}

Splitting that into two transactions — insert the dedup row, commit, then run the side effect — reopens the exact gap this exists to close: a crash between the two either loses the effect (dedup row committed, side effect never ran) or blocks it forever (some other bug makes the side effect always fail, but the dedup row already says "done"). One transaction, both or neither.

processed_message is the fast path, and it's not the only layer here — the ledger's own unique constraint on (order_id, sku_id, type) is a second, independent check at the data level, so even a message that somehow slipped past deduplication can't double-insert a movement. Testing this for real means actually delivering the same message twice and watching nothing move the second time, not asserting it in the abstract:

// deliver the same messageId twice; Awaitility.during(2s) — not a snapshot —
// confirms the ledger row count and available/reserved stay flat across the whole window

A one-shot assertion right after the second delivery can pass by accident if the second message just hasn't been processed yet. Watching a window is the difference between "looks idempotent" and "is idempotent."

When you'd reach for Debezium instead

Everything above is the polling-outbox version: a scheduled query with FOR UPDATE SKIP LOCKED instead of a CDC connector tailing the write-ahead log. Debezium never misses a row and adds essentially no latency, and it costs you Kafka Connect — another JVM, another thing that can fall over, another gigabyte you don't get back. On a budget where the whole stack has to cold-start on a 6 GB box, that trade isn't close.

The honest trigger for switching: sustained throughput in the thousands of events per second, a latency requirement tighter than "within about a second," or an operations team that's already running Kafka Connect for something else, so the marginal cost is close to zero. None of those describe this system today. The schema doesn't care which way you go — outbox_message looks the same either way, which is the actual point of keeping the contract in the database instead of in whichever tool happens to be reading it.

This is part of a series on building a multi-vendor commerce platform. The transactional outbox and the idempotent-consumer helper both live in stallora-cloud-starter, Apache-2.0, and every pitfall above was found by actually running the thing against real Postgres and real Kafka, not by reading the pattern's description one more time.

The textbook says cross-service inventory needs a coordinator. This checkout has none.

Qihu Zhang — Sat, 15 Aug 2026 06:34:47 +0000

Ask how to reserve stock across two services and keep it correct, and the textbook answer arrives fast: you need a distributed transaction coordinator. Two-phase commit, or a try/confirm/cancel (TCC) protocol run by something like Seata, sitting between the services and making sure both sides agree.

I placed an order that splits across two vendors, reserves stock in a separate inventory service, and survives two buyers racing for the last unit of the same SKU — with no coordinator anywhere in the request path. Not "eventually add one." None, by design. I want to show you why that holds up, because the reasoning generalises past inventory.

The invariant is smaller than the transaction

Here is the constraint that actually has to hold: stock must never go negative. That's it. Not "the order and the reservation must be created atomically" — just that one row, available, cannot drop below zero no matter how many requests hit it at once.

That is single-row arithmetic, and Postgres already has an answer for single-row arithmetic under concurrency: a conditional update.

UPDATE stock_item SET available = available - ?, reserved = reserved + ?
WHERE sku_id = ? AND available >= ?

If the row's available is high enough, the update commits and the hold exists. If it isn't, zero rows are affected, StockService.reserve throws, and order.create() never persists anything. There is no tentative state, no half-reserved row waiting for a second message to confirm it. The WHERE clause is the entire oversell defence, and it's a property of one row in one service's own database — nothing about it requires a second service to agree on anything, in the same instant or otherwise.

The habit that leads people to a coordinator is conflating "this spans two services" with "this needs cross-service atomicity." Placing an order spans two services. The part that must never be wrong — the stock count — does not. Once you separate those two questions, most of the case for a coordinator goes away on its own.

Try/confirm/cancel survive, the coordinator doesn't

The reservation lifecycle really is shaped like TCC. I didn't argue that away — I kept the shape and cut the coordinator out of the middle of it:

TCC step	What actually runs	Where the correctness lives
try	`order.create()` calls `inventory`'s reserve endpoint synchronously, one HTTP call, no saga framework	The conditional `UPDATE` above, inside `inventory`'s own local transaction
confirm	`order` writes `order-paid` to its outbox in the same transaction as the `PAID` state change; `inventory` consumes it later and deducts the hold	Kafka's at-least-once delivery + an idempotent consumer
cancel	Same shape, on `order-cancelled` — buyer cancel or the timeout sweep	Same idempotent consumer, same code path

try is synchronous because the caller is still on the line and the answer is needed right now: either the hold exists or the order was never worth creating. confirm and cancel are asynchronous because they're genuinely eventual — payment takes real time, might never happen, and nothing about "eventually tell inventory to convert or release the hold" needs to happen at the same instant as anything else. A coordinator's whole value is enforcing that two updates happen in the same instant. Nothing here asks for that.

What crosses the service boundary isn't a joint decision, it's a sequence: reserve now, then confirm or cancel later, exactly once. Outbox-plus-Kafka already delivers that sequence. Adding a coordinator on top wouldn't remove the "later, eventually" step — it would add a second system whose only job is remembering where in the sequence everyone currently is, duplicating the job Kafka and an idempotent consumer already do.

If the reservation succeeds but the order fails to persist right after — a duplicate Idempotency-Key loses a race, or the insert throws for an unrelated reason — that's the one compensation that runs synchronously: order.create()'s catch block calls inventory's release endpoint before returning the error to the caller, because the failure is already known and there's no reason to hand it to an async pipeline for something the request thread can just fix immediately.

The unglamorous parts that make it safe

None of the above works without three pieces of plumbing that will never make it onto a conference slide.

Idempotency, four layers deep. Kafka promises at-least-once, which means the consumer will eventually see the same message twice, or — worse — see order-cancelled arrive after it already applied order-paid for the same order because something upstream misbehaved. Four checks catch that, cheapest first:

processed_message (message_id, consumer_group) — a redelivered message is recognized before any business logic runs.
stock_ledger's UNIQUE (order_id, sku_id, type) — a second RESERVE/RELEASE/DEDUCT row for the same order and SKU can't insert, so even a message that slipped past layer 1 (a crash between marking it processed and committing) is caught at the data level.
The conditional arithmetic itself, unconditionally guarded against going negative.
Terminal-movement mutual exclusion — nothing above stops release from firing on an order that was already deducted. release and deduct each check the ledger for the opposite terminal movement first and refuse if it's already there. This is the layer that turns "our own producer emitted the wrong event" into a loud no-op instead of a silent double-adjustment of available. It also happens to be the layer I added a fix round after the first version shipped — the original code idempotency-checked duplicate deliveries but never checked whether the opposite terminal state had already landed by a different route. It's the kind of gap that four layers of "handle duplicates" doesn't catch, because a bypassed state machine isn't a duplicate.

Timeouts found by scanning, not by waiting. Kafka has no delayed-delivery primitive, so "cancel this order if nobody pays within 30 minutes" can't be a message with a fuse sitting in a queue. Instead a scheduled job selects expired PENDING orders with SELECT ... FOR UPDATE SKIP LOCKED against a partial index on status = 'PENDING' — the locking means running the scanner on more than one instance is safe, and the partial index means it never touches rows that were never candidates. The cancel and its outbox write commit together, so there's no window where the sweep decided to cancel but never told anyone.

Reconciliation as a feature, not an afterthought. The four idempotency layers guarantee duplicates and bypasses are caught. They don't guarantee anyone finds out. A scheduled job diffs stock_ledger movements against live order state and files a row for a human to look at on any mismatch. Without it, "eventually consistent" is a hope, not a property — a system can be silently a little bit wrong forever if nothing is watching for it.

One more small, unglamorous fact worth naming: the Idempotency-Key on POST /orders is scoped to the buyer, not global. The first version wasn't — a header collision between two different buyers would have returned buyer A's order to buyer B. UNIQUE (buyer_id, idempotency_key) instead of UNIQUE (idempotency_key) closed it. Small column, real information leak if it's missing.

When you actually need a coordinator

I don't think "you never need one" is true, and saying so out loud is the point of writing this down as an ADR rather than a blog opinion. A coordinator earns its cost when one business operation requires two or more services to update local state in the same instant, where neither side has a meaningful "happens now, the other happens eventually" split — the canonical case is a ledger transfer that must debit one account and credit another with no window where the money exists in neither or in both.

Reservation doesn't have that shape. try fully completes before the order exists at all; confirm/cancel have a real business reason to be delayed — waiting for a human to pay. The other trigger for a coordinator is a compensation that can't be expressed in business terms — if "give the stock back" weren't a meaningful operation, Saga-style compensation would have no move to make. Every compensation here — release a hold, cancel an order, retry a payment — is an ordinary write to ordinary data. None of that applies.

Numbers

Two buyers race for the last unit of a SKU with available = 1, fired from two threads released by the same latch so they genuinely overlap rather than queue politely:

Tests run: 1  -- ConcurrencyTest
  exactly 1 winner (200), exactly 1 loser (INSUFFICIENT_STOCK)

No lock taken in application code, no coordinator consulted — Postgres's own row-level locking on the conditional UPDATE decides the winner.

Against the running stack (verify-m2.sh, not a unit test): a buyer places one order across two vendors and it becomes one Order with two SubOrders, each line carrying its price and name snapshot. A second buyer tries to over-order a scarce SKU (one in stock) and gets a fast 409 with the stock count untouched. A paid order's hold converts to a deduction the moment the event is consumed — same traceId visible on both sides of the Kafka hop. An unpaid order, given a 15-second timeout for the demo, gets cancelled by the sweep and its hold comes back. Four sentences, four real outcomes, no coordinator watching any of them.

This is part of a series on building a multi-vendor commerce platform. The open-source half, stallora-cloud-starter, carries the outbox library this reservation flow is built on. Next up: three production incidents that the outbox pattern's four-sentence blog-post version doesn't warn you about.

Your product search does not need Elasticsearch

Qihu Zhang — Thu, 13 Aug 2026 00:55:52 +0000

I added multi-word product search to a marketplace this week. The new infrastructure it required: one column.

Not a container. Not a sync pipeline. A generated column and an index, in the migration that already existed. Search queries now go to the same database that owns the products, inside the same transaction domain, and a product is findable the instant its edit commits.

I want to lay out what that actually buys you, what it costs you, and — the part I think gets skipped — why the consistency story is the strongest argument, not the resource savings everyone leads with.

The default answer, and what it really costs

Ask how to add product search and the answer arrives before the question finishes: Elasticsearch. Or Meilisearch, or Typesense, if the person is being budget-conscious. All three are good at what they do. All three also mean the same three things:

Another always-on process. My whole stack has a hard ceiling: it has to cold-start on a 6 GB VPS, because that is what someone running this will actually rent. Six JVMs at -Xmx256m, Postgres, Redis, Kafka and tracing already claim most of it. A search engine's few hundred megabytes is not a rounding error at that scale — it is the difference between "runs on the box you have" and "buy a bigger box."

Another sync pipeline. Products live in Postgres. The engine needs its own copy. Something has to move rows across: a change-data-capture stream, an outbox consumer, a cron reindex, application-level dual writes. Whatever you pick, you have written a distributed system whose only job is to make two stores agree.

Another way to be wrong. That pipeline can lag, die quietly, or apply events out of order. Now "the product page shows the new price but search shows the old one" is a bug class you own forever. Ask anyone who has run a search cluster in production what fraction of incidents were the engine itself versus the thing feeding it.

None of that is an argument against search engines. It is an argument for knowing what you are buying before you buy it — and for checking whether the database you already run can do the job.

What Postgres gives you already

Three pieces, all built in.

A generated tsvector column, maintained by the database rather than by your code:

CREATE TABLE product (
    id          BIGSERIAL PRIMARY KEY,
    slug        VARCHAR(160) NOT NULL UNIQUE,
    name        VARCHAR(240) NOT NULL,
    description TEXT,
    -- ...
    search_vec  tsvector GENERATED ALWAYS AS (
                  to_tsvector('english', name || ' ' || coalesce(description, ''))) STORED
);
CREATE INDEX idx_product_search ON product USING GIN (search_vec);

GENERATED ALWAYS AS ... STORED is the important half. There is no trigger to write, no UPDATE ... SET search_vec = ... to remember in every write path, no chance of an application forgetting. Insert or update a row and the search vector is already correct before the statement returns. It is not eventually consistent with the row; it is part of the row.

The GIN index makes the lookup fast. And then the query:

SELECT p.id, p.slug, p.name, p.price_cents
  FROM product p
 WHERE p.status = 'ACTIVE'
   AND p.search_vec @@ websearch_to_tsquery('english', ?)

websearch_to_tsquery is the function to know. Its predecessor, to_tsquery, demands real tsquery syntax — wireless & noise & cancelling — which means you get to write a parser that turns human input into operators and escapes everything that would otherwise be a syntax error. websearch_to_tsquery accepts what a search box actually receives: wireless noise cancelling, quoted phrases, -excluded terms. Bad input yields no results, not an exception.

So a shopper types three words into a form, the string goes to the API as a parameter, and the parameter reaches websearch_to_tsquery. There is no sanitising step in the middle, and no injection surface, because it never stops being a bound parameter.

English stemming comes along for free: "cancelling" matches "cancel", "headphones" matches "headphone". The analyzer is a per-column choice, so a second language is a second expression, not a second system.

What you give up

Being honest about this matters more than the pitch, because the failure mode of "just use Postgres" advice is someone adopting it for a problem it does not fit.

No typo tolerance. "hedphones" returns nothing. Postgres has pg_trgm for fuzzy matching and you can bolt similarity search on as a fallback, but it is not the same thing as a real engine's fuzziness, and combining the two into one ranked result set gets fiddly quickly. If "did you mean" is a product requirement, this is the wrong tool.

No faceting. Counts per category, per brand, per price bucket, all computed alongside the result set — engines do this natively. In SQL each facet is another aggregate over the filtered set, which is fine for a handful of dimensions and unpleasant beyond that.

Relevance tuning stops at ts_rank. You can weight fields and boost by recency. You cannot ship the kind of tuned scoring, synonym dictionaries and per-market rules that a mature search team lives in.

Scale has a ceiling. For a catalog in the thousands, GIN lookups are not the slow part of your page. In the hundreds of thousands, with heavy filtering and real relevance needs, you will feel it.

My catalog is small, the requirement is "find products by words in their name or description", and every one of those limits is one I can state out loud without wincing. That is the actual test.

The part nobody mentions: consistency

Here is the argument I find most convincing, and it is not about memory.

With a search engine, your search index is a second copy of your data with its own clock. The write path becomes: commit to Postgres, then somehow get it to the engine. Even done well — outbox table, event stream, idempotent consumer — there is a window where the two disagree. You do not get to eliminate the window; you get to make it small and observable.

With a generated column, there is no second copy and no window. The vector is computed by the same statement that writes the row, under the same transaction. Roll the transaction back and the search state rolls back with it. Restore from a backup and the index comes back consistent, because it is the data. There is no reindex job, no "search is behind again" dashboard, no bootstrap procedure for a new environment.

That property is worth real money in operational calm, and it is invisible in a feature comparison table. Feature tables compare fuzziness and faceting. They do not have a row for "cannot drift."

Where this leaves you

I wrote the decision down as an ADR, and the part I care most about is the last section: what would make me change my mind. Typo-tolerant search as a stated product requirement. Faceted navigation. A catalog two orders of magnitude larger with real relevance work. Multi-language analyzers beyond what per-column configuration handles.

Notice that none of those are "we grew." They are all "the requirement changed." And when one does, the blast radius is small on purpose: search is behind one endpoint, GET /catalog/products?q=. Swapping the executor behind it does not touch the storefront, the mobile client, or the admin console. The contract does not know what runs underneath — which is the same reason it was cheap to start here.

The pattern generalises past search. Every "you'll need X for this" reflex is worth one question: what does the thing I already run do here, and what exactly would I be buying? Sometimes the answer is that you genuinely need X. Often the answer is a column.

This is part of a series on building a multi-vendor commerce platform — Spring Boot services, an event-driven consistency story, and a hard rule that the whole thing cold-starts on a 6 GB box. The shared infrastructure lives in stallora-cloud-starter under Apache-2.0. Next up: splitting one shopper's order across several vendors without a distributed transaction coordinator.

Publishing a Spring Cloud starter to Maven Central in 2026

Qihu Zhang — Tue, 11 Aug 2026 03:28:00 +0000

Half the guides still point at a JIRA board that no longer exists.

If you search for how to publish to Maven Central, you will find detailed walkthroughs telling you to open a ticket on issues.sonatype.org, wait for a human to approve your group id, then deploy to OSSRH and "close" a staging repository. That flow is gone. Since the Central Portal took over new registrations, the ticket step does not exist, the endpoint is different, and the plugin is different. The old posts are not wrong so much as archaeological.

Here is what the path actually looks like now, including the four places I lost time. I was publishing two small libraries — a conventions module and a transactional-outbox module — out of a Spring Cloud starter I am building in public.

Four prerequisites, all of which must be true at once

1. A verified namespace. In the Portal you claim a namespace rather than a group id. If you own no domain, io.github.<your-github-username> is free: the Portal verifies it by checking a GitHub repository you create with a name it dictates. Mine came out as io.github.danzizhangdev, and the Publishing Settings page shows it with a green Verified badge. That badge is the thing to look for; nothing downstream works without it.

2. A Portal user token. Generate it under your account, not your password, and drop it into ~/.m2/settings.xml as a <server> entry.

3. A GPG key pair, published to a keyserver. Central rejects unsigned artifacts. You need the private half locally and the public half discoverable.

4. Release plugins in the pom — sources jar, javadoc jar, GPG signing, and the Portal's own publishing plugin.

Each is simple. The interesting part is how they fail.

Trap 1: `${server}` is a placeholder, not a variable

The Portal shows you a ready-made settings.xml snippet to paste. It looks like this:

<server>
  <id>${server}</id>
  <username>...</username>
  <password>...</password>
</server>

${server} is not something Maven interpolates. It is a fill-in-the-blank, and the value has to match the publishingServerId of the publishing plugin — central by default. Paste it verbatim and your deploy fails to find credentials, with an error that does not point anywhere near your settings file.

Trap 2: "different login methods are different accounts, even with the same email"

That sentence is in Sonatype's own documentation, and it is the most expensive sentence on the page. Sign up with GitHub, come back later and sign in with email, and you are a different principal: your namespace is not there and your token belongs to someone else. The symptom is a 401 or 403 at deploy time, which reads like a credential problem, so you regenerate the token — and it happens again.

The rule that follows: always sign in the same way you signed in when you got the verified namespace. If the Namespace page ever looks empty, suspect the login method before you suspect the namespace.

Trap 3: `gpg --full-generate-key` times out and never says why

I answered every prompt — RSA, 4096, expiry, identity — watched it print "We need to generate a lot of random bytes", and then it sat there until:

gpg: agent_genkey failed: Timeout
Key generation failed: Timeout

Nothing in that message mentions the actual cause: gpg-agent had no way to ask me for a passphrase. I had gnupg and a terminal-only pinentry installed, no ~/.gnupg/gpg-agent.conf, and no GPG_TTY exported — so the agent could neither open a dialog nor draw a prompt in the terminal, and waited until it gave up.

Three fixes, and you want all three:

brew install pinentry-mac
cat >> ~/.gnupg/gpg-agent.conf <<'EOF'
pinentry-program /opt/homebrew/bin/pinentry-mac
allow-loopback-pinentry
default-cache-ttl 600
EOF
echo 'export GPG_TTY=$(tty)' >> ~/.zshrc     # GnuPG asks for this explicitly
gpgconf --kill gpg-agent

Note the tutorials that use %no-protection (a passphraseless key in a batch file) sail straight past this. That is why so few of them mention it.

Trap 4: `dirmngr` does not read your proxy environment variables

Uploading the public key failed with something that looks like a network outage:

gpg: sending key ... failed: No route to host

http_proxy was set and working — curl was fine. But keyserver traffic goes through dirmngr, a separate daemon that does not inherit http_proxy. It needs its own file:

cat >> ~/.gnupg/dirmngr.conf <<'EOF'
honor-http-proxy
http-proxy http://127.0.0.1:7897
EOF
gpgconf --kill dirmngr

One more thing that looks like a failure and is not: after uploading to keys.openpgp.org, fetch your key back and gpg --show-keys prints nothing. The key is there — that server deliberately strips every user id until you click the link in the verification email it sent. Central only needs the key material, so this does not block a release. Check keyserver.ubuntu.com alongside it and you will see the same key with its identity attached.

The pom side

Two plugin groups, and one attribute that matters more than the rest.

<plugin>
  <groupId>org.sonatype.central</groupId>
  <artifactId>central-publishing-maven-plugin</artifactId>
  <version>0.8.0</version>
  <extensions>true</extensions>
  <configuration>
    <publishingServerId>central</publishingServerId>
    <autoPublish>true</autoPublish>
    <waitUntil>published</waitUntil>
  </configuration>
</plugin>

autoPublish plus waitUntil=published is what replaces the old "close the staging repository in the web UI" dance: the build blocks until the Portal says the deployment is live, so a green build means a real release.

Signing, sources and javadoc go in a profile, so day-to-day builds stay fast:

<profile>
  <id>release</id>
  <build><plugins>
    <plugin><artifactId>maven-source-plugin</artifactId>
      <executions><execution><goals><goal>jar-no-fork</goal></goals></execution></executions></plugin>
    <plugin><artifactId>maven-javadoc-plugin</artifactId>
      <configuration><doclint>none</doclint></configuration>
      <executions><execution><goals><goal>jar</goal></goals></execution></executions></plugin>
    <plugin><artifactId>maven-gpg-plugin</artifactId>
      <configuration><keyname>${gpg.keyname}</keyname></configuration>
      <executions><execution><phase>verify</phase><goals><goal>sign</goal></goals></execution></executions></plugin>
  </plugins></build>
</profile>

My repository is a multi-module build where only two modules are meant to be consumed as libraries; the three runnable services are reference implementations. Those get one line:

<plugin>
  <groupId>org.sonatype.central</groupId>
  <artifactId>central-publishing-maven-plugin</artifactId>
  <configuration><skipPublishing>true</skipPublishing></configuration>
</plugin>

Publishing the whole reactor by accident is easy, and unpublishing is impossible.

Verify like you mean it

A release is not verified until you have pulled it back from Central on a machine that could not have had it cached. The trap is your own local repository:

rm -rf ~/.m2/repository/io/github/danzizhangdev
mvn dependency:get -Dartifact=io.github.danzizhangdev:starter-common:0.1.0

Without the rm, Maven answers from disk and you have proven nothing.

There is a second false positive that bit me for a different reason. My network cannot reliably download Spring Boot's dependency tree straight from Central — parallel downloads get their TLS connections cut — so I resolve through a mirror. A mirror silently invalidates the check above: "pulled it from Central" becomes "pulled it from whatever the mirror had". So I keep a second settings file with no mirrors at all, purely for release verification:

mvn -s ~/.m2/settings-nomirror.xml dependency:get -Dartifact=...:0.1.0

Mirrors affect resolution, never publication — the publishing plugin talks to the Portal API directly. But if you do not separate the two, your verification step is measuring your mirror.

What I would tell someone starting today

Ignore anything that mentions a JIRA ticket; you want the Central Portal documentation.
Do the GPG work first, with a passphrase and a GUI pinentry, before you touch the pom.
Keep one login method for the Portal and write down which one it was.
Treat skipPublishing and a no-mirror verification profile as part of the release setup, not as polish.

The starter itself — gateway, JWT auth with a JWKS endpoint, a transactional outbox library, and both Compose and Helm deployments of the same images — is Apache-2.0 here:

https://github.com/danzizhangdev/stallora-cloud-starter

Next in this series: what the memory budget actually looks like when you measure it, and why Kafka's default heap is the first thing to pin on a small box.

I split a commerce backend into 6 services for a shop with zero users. On purpose.

Qihu Zhang — Sun, 09 Aug 2026 05:09:55 +0000

There is a genre of blog post where someone explains that they moved off microservices and everything got better. Those posts are usually right. If you are building a product and your only goal is to ship it, a modular monolith will beat what I am about to describe on almost every measure that matters: build time, deploy time, cognitive load, your evenings.

I am doing the opposite anyway, and I want to be precise about why — because "learning" is the kind of reason people give when they have not actually thought about the trade.

The four goals, in order

Stallora is a multi-vendor commerce platform: a Next.js storefront, a Flutter app, an admin panel, and a backend split into a gateway plus five services. I build it about ten hours a week, alone. It has four goals, and I ranked them before writing any code:

Learn how distributed systems are actually run in production.
Write about it in public.
Leave behind a reusable starter other people can pick up.
Sell the business half as a code product.

The order is the interesting part. Goal 4 pays money and goal 1 does not, and goal 1 still wins. Any time the two conflict — a shortcut that would ship faster but hide the distributed problem — the shortcut loses. That rule is written into the repository's constraints, because six weeks from now, tired on a Wednesday night, I will want to take the shortcut and call it pragmatism.

If your ranking is different, most of what follows does not apply to you. That is fine. This is not advice. It is a decision record with its reasoning exposed.

What you cannot learn in a monolith

Here is the honest core of it. In a monolith there is no network between your modules. Every call either returns or throws, in-process, in your transaction. That is a feature for shipping and a problem for learning, because three things simply do not exist:

Partial failure. In a monolith, inventoryService.reserve() cannot half-happen. In Stallora, order calls inventory over HTTP, and that call can time out after inventory has already committed the reservation. Now two services disagree about reality and nobody threw an exception. Every design decision downstream — synchronous reserve, compensating release, timeout sweep — exists to answer that one sentence. You cannot practise this against a method call.

Messages that arrive twice. Once state changes travel as events, at-least-once delivery is the default and exactly-once is a marketing term. So consumers must be idempotent: the second copy of OrderPlaced has to be a no-op, not a second shipment. In-process, this problem is invisible. Over Kafka it is Tuesday.

Two writes that must agree. Committing a row and publishing an event are two systems. Do them in the wrong order and you either publish a fact that was rolled back, or commit a fact nobody hears about. The fix — write the event into the same database transaction as the row, then relay it — is the transactional outbox, and it only makes sense once the boundary is real.

Those three constraints produce the parts of this project I actually want on my résumé: inventory reservations that fail fast under concurrency, sagas that compensate instead of a distributed lock, consumers with deduplication tables. In a monolith I would be simulating all of it, and I would know I was simulating it.

The 6 GB constraint is the best thing in the design

The other reason this is not a toy: someone has to be able to run it. The business half ships to buyers who will deploy it on a cheap VPS, so the whole stack — six JVMs, a broker, a database, a cache, tracing — has to fit in about 4 GB, on a 6 GB machine, and start with one command.

That single number killed more architecture than any principle did. It is why there is no service registry, no config server, no separate tracing stack of its own:

Concern	Choice	Instead of
Discovery	Kubernetes Service DNS / compose service names	Eureka
Configuration	env vars + ConfigMap/Secret, Spring profiles	Config Server
Service calls	Spring HTTP Interface (`@HttpExchange`) + RestClient	OpenFeign
Tracing	Micrometer Tracing + OTLP → Jaeger	Brave + Zipkin
Messaging	Kafka (KRaft mode)	RabbitMQ
Database	PostgreSQL (`SKIP LOCKED`, JSONB)	MySQL
Consistency	Saga + transactional outbox	a transaction coordinator

Every row on the right is a JVM, a container, or a dependency I would have to explain to a buyer and pay for in RAM. Every row on the left is something the platform already does. The pleasant surprise is that the application ends up with no deployment dependencies at all: because discovery is DNS and configuration is environment variables, the same images run under Docker Compose and under Helm on Kubernetes, with nothing but different env values. Compose stays the buyer's path; Helm is the production-shaped path. I did not have to choose.

A follow-up post will show the measured numbers per container, including what Kafka does to your budget if you forget to cap its heap. (It defaults to a 1 GB heap. On a 6 GB box that is not a detail.)

When you should not do this

I would not build this shape on a team with these requirements. If Stallora were a funded product with a deadline, the right call would be a modular monolith: one deployable, clean module boundaries, extract a service only when scaling or team ownership forces it. Most teams that split early pay the operational tax for years and collect none of the benefit, because their bottleneck was never the runtime — it was the deadline.

The distinction I would draw is this: optimise for learning surface or for shipping speed, and know which one you picked. A repository that optimises for learning surface should say so out loud, in a file, where future-me can be held to it. Mine says so in docs/adr/0001-deliberately-over-decomposed.md, along with the sentence "This repository optimises for learning surface, not for shipping speed."

If you read that as an admission of overengineering — yes. That is the word. It is deliberate, bounded by a memory budget, and documented, which is the difference between overengineering and an accident.

What's next

The open-source half is stallora-cloud-starter: gateway, JWT auth with a JWKS endpoint, a transactional outbox library, and both Docker Compose and Helm deployments of the same images. It is Apache-2.0.

Repo: https://github.com/danzizhangdev/stallora-cloud-starter
Next post: the memory budget in detail — measured per container, and the four infrastructure swaps that paid for themselves.

If you have run something like this on a small box, I would like to hear what your first OOM was. Mine has not happened yet, which mostly means I have not finished.