iCe Gaming

Posted on Oct 19

Keeping checkout alive when the store's internet dies

#rust #architecture #programming #systemdesign

A store's internet connection is the least reliable part of its checkout. The fibre gets cut, the router reboots in the middle of the lunch rush, or the ISP has a bad afternoon. If your tills call a cloud API to look up a price, take a payment or sign a receipt, the store stops selling until the link comes back.

The usual fix is to put a small server in the store and have the tills talk to that instead. I've spent the last few months building one in Rust with SQLite. This post covers the design decisions that mattered, and two places where my first version was wrong.

The shape of it

The rule that makes everything else work: tills only ever talk to the hub. Nothing a till does goes straight to the internet.

That only holds if the hub can finish a sale with no outside calls. Everything a sale needs (catalog, price books, promotions, coupon definitions, tax rules, store config) is synced down from HQ ahead of time, with a checkpoint per entity so a sync interrupted halfway can resume. When the uplink drops, the data on the hub is a few minutes stale at worst, and that is fine for pricing a basket.

The hard part is the other direction: getting sales out reliably, exactly once, after an outage of unknown length.

Pick your commit point, then order everything around it

When a till sends "finalize", a lot happens. The order of those steps matters more than any individual step.

The question to answer first: at what moment does the sale exist? For me it's the write to the local order ledger (step 3), done in the same SQLite transaction as the outbox row that queues the sale for HQ. Everything is arranged around that line:

Before it go the steps that can still refuse the sale: capturing payment and getting a fiscal signature. If one of them fails hard, nothing has been written yet, so there's nothing to clean up except the card payment, and that gets flagged for automatic reversal.
After it go the steps that must never undo the sale: committing reserved stock, earning loyalty points, building the receipt. They're best-effort. A loyalty storage error is logged and counted, but the customer still walks out with a receipt.

The code comment at that line says it plainly:

// The order is durable, so the money is earned. Past this point the sweeper
// must never touch these payments, even if a later best-effort step fails.

Thinking about it this way removed a whole class of "what if step 4 fails after step 2 succeeded?" questions. Each step is either before the line (it can abort) or after it (it can't).

The till will retry. Plan for it.

On a flaky store network a till sends a request, the response gets lost, and the till sends it again. If "again" means "charge the card twice", you have a real problem.

Every command from a till carries an idempotency key. The hub handles it at two levels.

Whole-command replay. A successful response is stored under its key. A repeat of the same key gets the stored response back without running the command again:

if let Ok(Some(stored_response)) =
    apex_edge_storage::get_response(&app.pool, envelope.idempotency_key).await
{
    if let Ok(replayed) =
        serde_json::from_str::<PosResponseEnvelope<serde_json::Value>>(&stored_response)
    {
        return Json(replayed);
    }
}

Payment-level protection. Command replay isn't enough for money, because the stored response is only written on success. A retry after a failure that happened after the card was charged would otherwise charge it again. So before talking to a payment provider, the hub records a payment intent keyed by (idempotency_key, provider, tender_id). A retry finds the intent already captured and answers from the local ledger instead of asking the provider for the money again:

// A retried command finds its intent already captured. Answer from the ledger rather
// than asking the provider for money a second time.
if intent.state == PaymentIntentState::Captured || intent.state == PaymentIntentState::Settled {
    return Ok(CapturedTender { /* from the stored intent */ });
}

The idempotency key sent on to the provider is derived from the till's key plus the tender and the operation, so the provider deduplicates too.

Fiscal signing is a network call too

In several European countries every receipt must be cryptographically signed by a certified device or service, for example the TSE in Germany. Often that signer is a cloud API, which brings back the exact problem the hub exists to solve.

The hub splits signer failures into two kinds:

Transient (timeout, connection refused, 5xx): the sale goes through, and the unsigned transaction goes into a sign-later queue. A background job keeps retrying it until the authority accepts it or rejects it outright, and a rejection lands in a dead-letter state for a person to deal with.
Anything else, typically misconfiguration: the sale fails, before the ledger is written. A wrongly configured fiscal setup should stop the till loudly, not quietly print non-compliant receipts all day.

Whether "sign later" is legal for you, and for how long, depends on the jurisdiction. Check before you copy this.

One outbox, many destinations

The sale lands in an outbox table as a single row, written in the same transaction as the ledger. A background dispatcher delivers it.

The first version had one destination: HQ. Then e-invoicing came along (a Peppol access point that wants invoices but not shift reports), then a webhook for analytics. The tempting shortcut is one status column on the outbox row. That breaks as soon as destinations fail independently: if the Peppol endpoint is down, HQ shouldn't stop receiving sales, and the hub must not forget that Peppol is still owed them.

So delivery state lives per destination:

Each dispatcher cycle does three things:

Fan out. For each pending submission and each enabled destination that wants that kind of payload, make sure a delivery row exists.
Deliver. Attempt the deliveries that are due. Each has its own attempt count, backoff and error message.
Settle. A submission is finished only when no destination is still owed it. If one destination gave up, the submission counts as dead-lettered even though the others took it, because half-delivered is a failure someone has to see.

A few smaller decisions that held up:

Filter by payload kind, and treat an empty filter as "everything". A destination config of {"payload_kinds": []} delivers everything rather than nothing. A typo that sends too much is much easier to spot than one that silently sends nothing.
Sign the exact bytes you send. Destinations can opt into an HMAC-SHA256 signature over "{timestamp}.{body}". The body is serialized once and those same bytes are both signed and sent, so the signature always covers what the receiver reads. The secret stays in an environment variable. The destinations table only stores the name of that variable.
A 2xx with an unreadable body counts as accepted. If HQ replied 200 but the body didn't parse, sending the sale again risks a duplicate at a destination that already stored it. I'd rather log it than resend.

The part I got wrong: the retry budget

The backoff policy looked sensible: 5 seconds, doubling each attempt, capped at 320 seconds, dead-letter after 10 attempts. Then I worked out how long that actually is.

About 21 minutes. A connection error counted as an attempt just like an HTTP 500 did, so during a one-hour outage every sale from the first ~39 minutes ran out of attempts and ended up in the dead-letter queue. Nothing was lost, since they were still stored and an operator could requeue them. But the whole point of the hub is to ride out exactly this situation without a person stepping in, and it didn't.

The lesson generalizes: your retry budget has to be longer than your longest normal outage, and "the network is down" is a different kind of failure from "the server said no." The fix: a try that never reaches the destination (connection refused, DNS failure, connect timeout) now bumps a separate counter that only paces the backoff. Those keep retrying at the capped interval and drain by themselves once the link is back. The dead-letter queue is saved for destinations that answered and refused.

Err(e) if e.is_connect() => {
    // The request never reached the destination. That is not a verdict on
    // the payload, so it must not use up attempts.
    record_attempt(&delivery.destination_code, OUTCOME_UNREACHABLE);
    let next = Utc::now()
        + Duration::seconds(policy.backoff_delay_seconds(delivery.unreachable_attempts));
    schedule_unreachable_delivery_retry(pool, delivery.attempt_id, next, &e.to_string())
        .await?;
    Ok(false)
}

The other thing I got wrong: two writes instead of one

My first version wrote the ledger row and the outbox row as two separate writes. If the process died between them, the sale existed locally but HQ never heard about it. The window was tiny, and on SQLite the fix was cheap: both now go in one transaction, and a test checks that a failed outbox insert rolls the ledger write back too. That's the classic transactional outbox, and I should have started there.

What I'd still change

Real sequence numbers. Each sale submission has a sequence_number field, but every sale is currently sent with 1. With a per-register counter, HQ could detect gaps itself instead of trusting the hub to have sent everything.

What to watch in production

Three numbers tell you whether a store hub is healthy, and none of them involve the tills:

Outbox queue depth, by state. Pending rising while the store is online means a destination has quietly stopped accepting.
Dead-letter count. It should be zero. Anything non-zero is a sale someone has to look at.
Fiscal queue depth. Unsigned receipts waiting on the signer.

All three are Prometheus metrics on the hub. The runbook linked below has alert queries for the outbox ones.

DEV Community