DEV Community: 137Foundry

How to Write Readiness Checks That Survive Real Production Traffic

137Foundry — Mon, 22 Jun 2026 10:10:15 +0000

Readiness checks are deceptively easy to write and surprisingly easy to write badly. A readiness check that works in development can become a load-bearing piece of your production infrastructure in ways the author never intended. This guide walks through how to write a readiness check that holds up under real production traffic and does not become its own reliability liability.

The audience for the readiness check is the load balancer in your orchestrator (typically Kubernetes or a similar platform) and any front-end reverse proxy like Nginx or an edge service like Cloudflare. The job of the check is to tell the load balancer whether this instance can serve real production traffic right now.

Step 1: Define what "ready" means for this service

Before writing any code, write down the dependencies your service must have available to serve a real production request. The list should be short and specific. For most web services, it looks like:

The primary database (read and write).
The primary cache.
The message broker, if the request path produces messages synchronously.
Any internal upstream service called synchronously in the request handler.

Things that should not be on the list:

Asynchronous background dependencies. If your service writes events to a queue but the request can complete without confirming queue health, the queue is not a readiness dependency.
Non-critical caches. If a cache outage degrades performance but does not break the request, it should be reported in detail status but not block readiness.
External APIs called outside the synchronous request path. Same reasoning.
Disk space, memory, CPU. These are metrics for Prometheus or another monitoring tool, not binary readiness signals.

This list is the entire substance of the readiness check. Writing it down explicitly is the most important step.

Photo by panumas nikhomkhai on Pexels

Step 2: Implement each dependency check as a fast, isolated function

Each dependency check should be a small function that takes no parameters, runs a single fast query against the dependency, and returns true/false plus a brief reason string.

A reasonable database readiness check looks like:

async function checkPrimaryDb() {
  const start = performance.now();
  try {
    await db.query("SELECT 1");
    return { ok: true, latency_ms: performance.now() - start };
  } catch (e) {
    return { ok: false, error: e.message, latency_ms: performance.now() - start };
  }
}

Notes on each part:

The query is SELECT 1. Not a real query. Not a table scan. The check is testing connectivity, not performance.
The timing is captured. You will want it for the detailed status endpoint.
The function does not throw; it returns a structured result. The caller decides what to do with a failure.

Repeat this pattern for each dependency. Resist the urge to write a "generic" check that takes a dependency name and runs a different query per name. Specific is better than generic here; every dependency has its own quirks.

Step 3: Run the checks in parallel and with a timeout

The readiness handler should call every dependency check in parallel and apply a hard timeout (typically 500 milliseconds to 2 seconds). The timeout matters because a slow dependency that does not return in time should fail the readiness check fast, not hang the probe.

A reasonable handler in pseudocode:

async function readyz(req, res) {
  const checks = await Promise.all([
    withTimeout(checkPrimaryDb(), 1000),
    withTimeout(checkPrimaryCache(), 500),
    withTimeout(checkMessageBroker(), 1000),
  ]);
  const allOk = checks.every(c => c.ok);
  res.status(allOk ? 200 : 503).json({ checks });
}

The parallel execution means the overall check completes in roughly the time of the slowest dependency, not the sum of all dependencies. The timeout means a stuck dependency does not stick the probe.

Step 4: Cache the result briefly

The orchestrator polls the readiness endpoint frequently. If every probe ran a fresh database query, the database would experience a constant trickle of health-check load. Multiply by every pod, and the health check becomes a load source.

The fix is to cache the readiness result for a short window (typically 1 to 2 seconds). The handler:

Returns the cached result if it is less than the cache TTL old.
Otherwise runs the checks fresh, caches the result with the timestamp, and returns it.

A 1-second cache means each pod runs at most 1 set of dependency checks per second, regardless of how often the orchestrator polls. This is the right shape: the readiness result is fresh enough to be meaningful but not so fresh that it becomes a workload.

Photo by Al Nahian on Pexels

Step 5: Distinguish between hard and soft failures

Not every dependency is equally critical. A reasonable readiness handler distinguishes:

Hard dependencies. If any of these is down, the service cannot serve requests. The readiness check should fail. Database, primary cache, message broker.
Soft dependencies. If these are degraded, the service can serve requests with degraded behavior. The readiness check should pass; the degraded state should be visible in detailed status.

For most web services, the soft category includes secondary caches, analytics endpoints, and any non-critical external integration. The boundary depends on the service.

A common bug: classifying a soft dependency as hard, then having a non-critical service outage take the whole production cluster out of rotation. The fix is to be explicit about the boundary and to test failure modes for each.

Step 6: Handle graceful shutdown

When the process receives SIGTERM (a deploy, a scale-down, an orchestrator-initiated restart), the readiness check should immediately start failing so the load balancer stops routing new requests. Meanwhile in-flight requests continue to be served until they complete.

In practice this looks like:

let shuttingDown = false;
process.on("SIGTERM", () => { shuttingDown = true; });

async function readyz(req, res) {
  if (shuttingDown) {
    return res.status(503).json({ shutting_down: true });
  }
  // ... normal check logic
}

The liveness probe should keep passing during this window so the orchestrator does not SIGKILL the process before in-flight work has drained.

Step 7: Confirm the failure modes in a test environment

The most important step. In a non-production environment, simulate each dependency outage and confirm:

Killing the database causes readiness to fail within the timeout window.
Killing the cache causes readiness to fail.
Killing the message broker causes readiness to fail.
Killing a soft dependency does not cause readiness to fail but does show up in detail status.
SIGTERM causes readiness to fail immediately.
After dependency recovery, readiness recovers automatically.

Run these tests during the initial implementation and after any meaningful change. They catch the four lies a health endpoint can tell, all in about an hour of work.

Step 8: Watch the probe under real load

Once deployed, monitor the readiness probe over time. Watch for:

Frequent flaps (rapid 200/503/200/503 cycles). Usually indicates a timeout that is too aggressive or a dependency that is genuinely flaky.
Slow response times. If the probe takes more than 500ms on average, something is wrong with the parallelization or the timeout configuration.
Correlation with downstream incidents. Genuine readiness failures should correspond to real downstream issues.

Tools like Prometheus can scrape the probe over time and graph these signals. The graphs will tell you whether the readiness check is doing its job or has become noise.

Common mistakes and how to fix them

A few patterns we have seen in audits at 137Foundry's web development service:

Sequential dependency checks. Each dependency is checked in series. The total probe time is the sum of all checks, which slows the probe response. Fix: parallelize.

No timeout. The check awaits each dependency without a timeout. A slow dependency hangs the probe indefinitely. Fix: add a timeout per dependency.

No caching. Every probe runs fresh checks. Probe load becomes meaningful at scale. Fix: cache for 1-2 seconds.

Catch-all success. A try/catch around the whole handler returns 200 on any exception. Real failures get hidden. Fix: return 503 on any caught error and surface the error in the response body.

Missing graceful shutdown. The probe does not respect SIGTERM. Deploys lose in-flight requests. Fix: add the shuttingDown flag.

Checking the wrong dependency. The check queries a stub or a deprecated endpoint that no longer reflects production state. Fix: trace every check to a real production dependency.

The takeaway

A readiness check that survives production traffic is a small piece of code that defines the dependency contract explicitly, runs checks in parallel with timeouts, caches briefly to avoid becoming load, distinguishes hard and soft failures, and respects graceful shutdown. None of these requirements is hard. All of them are easy to miss.

For the longer write-up on how readiness sits inside the three-endpoint health check design and how the orchestrator, load balancer, and detailed status views all coordinate, the hub article on health check endpoint design is the place to land. The services hub is where the longer engagements live.

Most teams ship a v1 readiness check in a single afternoon. Most teams do not revisit it for two years. The audit that the section above describes usually catches at least one bug in the v1 implementation; running it once is a strong upgrade for any production web service.

How to Tell If Your Health Endpoint Is Lying To You

137Foundry — Mon, 22 Jun 2026 10:09:54 +0000

A lying health endpoint is not a fictional problem. Almost every long-running web application accumulates one over time: a /health or /healthz route that returns 200 OK even when the application is, by any reasonable definition, broken. The team trusts the dashboard. The dashboard trusts the endpoint. The endpoint trusts a route handler that was scaffolded three years ago and has not been audited since.

This is a guide to figuring out whether your own health endpoint is telling the truth. It is the audit that every web service should have run at least once and that very few have.

The four ways a health endpoint lies

A health endpoint can lie in four distinct ways. Knowing which one applies tells you what to fix.

Lie #1: It returns 200 OK because the route handler is hard-coded to do so.

The route handler reads, in essence:

app.get("/health", (req, res) => res.send("ok"));

There is no check. There is no dependency probe. The endpoint reports the process can serve HTTP, and nothing more. For some teams this is fine (this is what a liveness probe should look like). For most teams who think they have a real health endpoint, it is not fine.

How to spot it: read the route handler. If it has no awaits, no database calls, no external checks, no try/catch around dependencies, it is this kind of lie.

Lie #2: It returns 200 OK because the check inside it is catching all errors.

The route handler tries to check the database, the cache, and maybe an upstream service, but every check is wrapped in a try block whose catch returns 200 anyway. The original intent may have been "do not let the health endpoint itself crash," but the implementation hides every dependency error behind a green light.

How to spot it: search the route handler for catch blocks. If any catch returns success, the endpoint can lie about that path.

Lie #3: It returns 200 OK because it checks the wrong dependency.

The handler checks the database connection pool by querying a stub or a status table that does not actually exercise the database. Or it checks a cache that has been replaced by a new cache, but the old check still works. Or it checks an upstream service that has been deprecated and now returns 200 from a maintenance page regardless of state.

How to spot it: trace every check inside the handler back to a real production dependency. If any of them are vestigial, they are lying.

Lie #4: It returns 200 OK because the process is fine but the parts that serve real traffic are not.

This is the subtlest lie. The route handler runs on a worker pool, an event loop, or a thread that is completely separate from the workers serving real requests. When the request-serving workers wedge, the health-endpoint worker is unaffected. The endpoint passes; users see latency and errors.

How to spot it: look at the framework's worker configuration. If the health endpoint runs on its own loop or worker pool, the endpoint is structurally unable to detect a wedge in the request-serving path.

Photo by Brett Sayles on Pexels

A four-step audit you can run today

The audit takes an hour or two for most web applications and produces concrete output: either the endpoint is telling the truth, or you have a short list of things to fix.

Step 1: Read the route handler.

Open the source file containing the health endpoint. Read it line by line. Write down:

What checks does it run?
What dependencies does each check exercise?
What does it return on success, on partial failure, on total failure?
Is the route served by the same worker pool as real traffic?

If the answers are surprising (you thought it checked the cache and it does not), you have already found one of the lies above.

Step 2: Run a fault-injection test.

In a non-production environment, stop the database. Call the health endpoint. Does it return 503? If it returns 200, the endpoint is lying. Repeat for the cache, for any internal upstream services in the request path, and for the message broker.

This is the single most valuable test you can run on a health endpoint, and it is surprisingly underused. Teams audit code; very few teams audit signals.

Step 3: Probe under realistic load.

In a non-production environment, generate traffic that mimics production. Watch the health endpoint while doing so. Does it stay snappy? Does it ever return 503 spuriously? Tools like Prometheus scraping the endpoint over time will surface slow checks and intermittent failures that hand-testing misses.

A health endpoint that takes 2 seconds to respond is not a health endpoint; it is a latency liability. If the orchestrator polls it every 5 seconds, you have just added meaningful background load to your own database.

Step 4: Trace the dashboard's chain.

The dashboard says the service is healthy. Where does that signal come from? Usually it is an external uptime checker, a monitoring tool, or the orchestrator's own status. Confirm what each of them is actually checking. Often the dashboard is summarizing the orchestrator's view, which in turn is summarizing the liveness probe. If the liveness probe is the always-200 lie from above, the entire dashboard chain is built on a lie.

Common discoveries from the audit

In our experience auditing client systems at 137Foundry, the patterns recur:

Pattern 1: The handler was written before the service used a database. The endpoint is a leftover from when the application was a single in-memory service. The original handler was honest at the time and has never been updated.

Pattern 2: The handler checks an old dependency that has been replaced. The application now uses Redis for caching, but the health check still queries a stub that pretends a Memcached instance is up.

Pattern 3: The handler is the same route for liveness, readiness, and detail. It tries to satisfy three different audiences and ends up lying to at least one of them.

Pattern 4: The handler catches exceptions silently. Someone added a catch block years ago to prevent the endpoint from 500ing, and now any internal error is masked as 200.

The fixes are usually 10 to 50 lines of code each, plus a configuration change to the orchestrator's probe settings.

Photo by Keysi Estrada on Pexels

The three-endpoint solution

The longer-term fix is the three-endpoint pattern: split the single /health into /livez (liveness), /readyz (readiness), and /statusz (detailed JSON for humans). Each endpoint has a clear scope and a clear audience.

Liveness checks only that the process can serve HTTP at all. Cheap, fast, no dependencies.
Readiness checks every dependency the application needs to serve real requests. Slightly slower, parallelized, cached briefly.
Detailed status returns a JSON payload with per-dependency information for dashboards and on-call engineers.

This pattern is described fully in the hub article on health check endpoint design worth trusting. Container orchestrators like Kubernetes are designed around this separation; load balancers like Nginx and edge proxies like Cloudflare all respect the readiness signal as a routing primitive. The split is not invented; it is what the underlying infrastructure already expects.

What the audit will surface

After the audit, you will land on one of three outcomes.

Outcome A: The endpoint is honest. The route handler runs real checks, returns 503 on real failures, runs on the production worker pool, and the orchestrator's view matches reality. You have spent two hours and confirmed a piece of your stack is working as intended. Worth knowing.

Outcome B: The endpoint is structurally lying. It is hardcoded to 200, or it is checking the wrong things, or it runs on the wrong worker pool. You have a short list of changes to ship. Usually one afternoon of work.

Outcome C: The endpoint is mostly honest but has gaps. It checks the database but not the cache. It catches one exception silently but not the others. It checks an upstream that no longer matters. You have a slightly longer list, but each item is mechanical.

In our consulting work, Outcome B is the most common. Teams add real checks year by year, and the endpoint grows organically into something that mostly works but has at least one significant lie buried in it.

The takeaway

A lying health endpoint is the most expensive piece of seemingly trivial code in your stack. The reason it is expensive is that every alert, dashboard, runbook, and on-call escalation depends on it. When it lies, every downstream signal is poisoned, and the team trusts a green dashboard while users are unhappy.

The audit is short. The fix is mechanical. The cost of doing it once is a fraction of the cost of a single 2 AM page caused by a probe that should have surfaced the problem ten minutes earlier. If you have not audited your health endpoint in the last year, you are almost certainly running on at least one of the four lies above.

For a longer reference and the full three-endpoint design pattern, the hub article covers the rest. The services hub is where the audit conversation usually begins.

How to Add Idempotency Keys to an Existing Integration Without Breaking Live Traffic

137Foundry — Sun, 21 Jun 2026 10:06:57 +0000

Most idempotency retrofit work is on integrations that have been running in production for months or years, processing live traffic, with downstream consumers that depend on the existing message format. The challenge is not the cryptography or the design pattern. It is the deployment sequencing that lets you add keys, change behavior, and clean up edge cases without breaking anything in flight.

This guide walks through a six-step deployment sequence that works for most production retrofits. The whole sequence takes one to three weeks of calendar time per integration, depending on how many consumers you have and how aggressive your retry windows are.

Photo by Giant Asparagus on Pexels

Step 1: Add the Key Column Without Using It

The first deployment is a no-op from a behavioral standpoint. You add a UUID column to the outgoing event table, populate it for new rows, and leave existing rows with NULL.

The sender does not change yet. The receiver does not change yet. The only difference is that new rows now have a UUID that future code can use.

This step takes a few hours to ship and prove. You verify by inspecting the database that new rows are getting UUIDs and old rows still have NULL. If anything is wrong, you roll back the column add (or just leave it; an unused column is harmless).

This step is important because it lets you backfill the new column for existing rows during the next quiet window without time pressure. The Wikipedia overview of idempotence covers the property you are ultimately enforcing, but the deployment work is what makes the enforcement reliable in production.

Step 2: Backfill UUIDs for Existing Rows

For rows that already exist with NULL UUIDs, generate UUIDs and populate the column. This is a one-time backfill, usually run as a batch job.

The UUIDs for backfilled rows do not need to be deterministic, since these rows correspond to events that have already been processed. The only requirement is that future code can rely on every row having a non-NULL UUID.

The backfill is usually fast (a single UPDATE statement for small tables, a chunked job for large ones). Verify that no rows remain with NULL UUIDs before moving to the next step. Wikipedia's overview of extract, transform, load processes covers the broader pattern of backfilling state without disrupting live processing.

Step 3: Modify the Sender to Include the UUID

The sender now reads the UUID from the row and includes it in the outgoing message (header or body, depending on the message format).

The receiver still does not use the UUID. It simply receives it and ignores it. This step is also behaviorally a no-op, but it gets the data flowing through the pipeline so that downstream changes can rely on it being present.

You verify by tailing the receiver's request log and confirming that every incoming request includes a UUID.

Step 4: Add Receiver Dedup Without Enforcing It

The receiver now records every incoming UUID in a dedup table but does not yet refuse duplicates. If a duplicate UUID arrives, the receiver logs the event but processes the request as normal.

This step is critical. It lets you observe what the actual duplicate rate looks like in production before flipping the enforcement switch. The expected hit rate is between 0.1 and 1 percent. Wikipedia's entry on the saga pattern and writing from practitioners like Martin Fowler at martinfowler.com both reinforce why observation before enforcement is the safer sequencing.

Two outcomes are possible:

The hit rate is in the expected range. Move to step 5.
The hit rate is 0 percent or 10+ percent. There is a problem worth understanding before enforcement: either the keys are not stable across retries (move to investigating sender behavior) or the keys are colliding across distinct operations (move to investigating key generation logic).

Spend at least one full week in this step before proceeding. Production traffic patterns vary day to day, and a week of data is the minimum to be confident the hit rate is representative.

Step 5: Flip the Enforcement Switch

Once you are confident that the dedup mechanism is recognizing duplicates correctly, change the receiver to actually return the original response (without re-applying side effects) on duplicate UUIDs.

This is the moment when the integration becomes truly idempotent. From this deployment forward, retries are safe to perform.

The change is small and behaviorally observable: duplicate-side-effect incidents should drop to zero from this point on. The dedup hit rate stays roughly stable; the difference is that the hits now actually prevent the duplicate work.

Step 6: Tighten Up the Sender Retry Logic

Once enforcement is on, the sender can be more aggressive about retries because the receiver is now safe under duplicates. Increase the retry count, shorten the backoff interval, or both, depending on the operational characteristics you want.

This is also the step where you switch the sender's retry logic from "create new row" to "update existing row" if it was not already. With enforcement on at the receiver, a new row would just produce a different UUID and a fresh side effect. The retry has to reuse the existing row's UUID.

After this step, the integration is fully retrofitted. You should observe:

A non-zero, roughly stable dedup hit rate on the receiver (proof that retries are working).
Zero new duplicate-side-effect incidents in the receiver's data.
Faster recovery from broker outages or network instability, because the sender can retry more aggressively without risking corruption.

What to Verify Before Each Step

A few specific checks save a lot of grief:

Before step 1: Confirm you have permission to alter the outgoing events table schema. Some legacy integrations have this table managed by a separate team.

Before step 3: Confirm that the message format allows adding a new field without breaking existing consumers. For some legacy formats (fixed-length records, strictly schemaed protocol buffers), this requires a separate compat-layer change first.

Before step 4: Confirm you have a place to put the receiver's dedup table and have provisioned enough storage for at least the retention window times the expected message rate.

Before step 5: Confirm the receiver's response cache returns identical results for duplicate UUIDs. If the cached response includes a timestamp or other "now" field, duplicate responses might look different from each other even though the side effect was skipped, which can confuse downstream consumers that check response equality.

Common Retrofit Failure Modes

A few patterns that reliably cause problems:

Skipping the observation phase. Going from "no UUIDs" to "enforce on receiver" in two deploys means you find out about key generation bugs only after enforcement turns them into visible errors. The observation phase exists to find these bugs in a safe window.

Generating UUIDs at send time instead of event time. The retrofit hooks the UUID into the wrong place, and the dedup hit rate stays at zero even after step 4. The fix is to move generation upstream to the source-of-truth event. The 137Foundry data integration practice treats event-time generation as the default for any new integration we ship, because the retrofit cost when this is wrong is meaningfully larger than the design cost to get it right from the start.

Backfilling UUIDs for already-processed rows in a way that collides with future generation. Make sure the backfill uses a UUID scheme that cannot accidentally produce the same value as future event-time generation. Standard random UUIDs handle this; sequential or time-based schemes can collide if not handled carefully.

Forgetting to clean up the dedup table. Without TTL or partitioning, the dedup table grows unbounded and eventually becomes the bottleneck. Add cleanup as part of step 4, not as a follow-up.

Why the Sequencing Matters

The six-step sequence works because each step is independently shippable and observably safe. Any single step can be rolled back without affecting the others. The pipeline can sit indefinitely in step 4 (recording but not enforcing) if the team needs more time to validate.

The alternative (a big-bang retrofit that adds UUIDs, enforces dedup, and changes retry behavior in one deploy) is much more dangerous because the failure modes interact in ways that are hard to debug in production.

For more depth on the underlying patterns, the longer reference on how to handle idempotency in data integration pipelines covers the design patterns the retrofit is aiming to deliver, and the 137Foundry services overview covers how this work fits into broader integration engineering.

The Honest Framing

A retrofit done right is invisible to upstream and downstream consumers. The pipeline just becomes more resilient over the course of a few weeks, without any breaking changes or visible behavior shifts.

A retrofit done in a rush usually breaks something in the middle and produces a postmortem that lasts longer than the deployment itself. The slow path is the fast path.

The six-step sequence is not the only valid approach, but it has worked across enough production retrofits to recommend as a default. Variations on timing or step ordering are fine. Skipping the observation phase is not.

Five Patterns for Making Data Integration Operations Safe to Retry

137Foundry — Sun, 21 Jun 2026 10:06:57 +0000

Every data integration pipeline has to handle retries, because every network boundary eventually produces a duplicate delivery. The patterns below are the five that show up most often in production-grade integration code, each with a clear use case and a clear set of trade-offs. The right choice depends on the operation shape and the cost structure of the integration.

This is a roundup of the patterns, with notes on when each one fits best and when to reach for a different tool.

Photo by Tom Fisk on Pexels

1. The Idempotency Key Pattern

The most common and most general pattern. The sender generates a stable UUID for each logical operation, includes it with every delivery attempt, and the receiver records processed UUIDs to dedupe duplicates.

Use case: any operation that produces side effects (creates, charges, sends, allocates) and needs to be safe under retries.

Trade-off: requires a dedup store at the receiver with a retention window longer than the maximum retry window. Storage cost scales with message volume.

The Wikipedia overview of idempotence covers the underlying property. The canonical implementation in production code is Stripe's "Idempotency-Key" header, which is the model most homegrown implementations follow.

The key implementation detail: generate the UUID at the source-of-truth event, not at send time. A UUID generated inside the sender function is unique per call, not unique per logical operation, which defeats the entire pattern.

2. The Absolute-State Pattern

Convert operations from relative changes ("increment count by 3") to absolute state ("set count to 7"). Absolute-state operations are naturally idempotent because re-applying produces the same result.

Use case: operations where you can express the desired end state as a value rather than a delta. Inventory levels, status flags, role assignments, configuration values.

Trade-off: messages carry more state, which costs bandwidth and may require sending fields the receiver does not change. For 50-field records where one field moved, sending all 50 is wasteful.

The Wikipedia overview of extract, transform, load processes covers the broader pattern of moving state to a target system in a retry-safe way. Absolute state is the easiest mental model for ETL because each batch overwrites the previous state.

The pattern works best for low-frequency state synchronization (nightly inventory sync) and worst for high-frequency event-stream integrations where bandwidth matters.

3. The Optimistic Concurrency / Version Check Pattern

The entity being modified carries a version number. Each write includes the expected current version, and the receiver only applies the write if the version matches.

A retry that arrives after the original applied finds a higher version on the entity and is harmlessly rejected. The sender treats the rejection as a successful retry resolution (the operation has already been applied).

Use case: relative operations on entities that can carry version metadata. The Wikipedia entry on database transactions covers the broader concurrency model this pattern fits into, and standard databases like PostgreSQL support row-level version semantics via xmin or explicit version columns.

Trade-off: requires a read to find the current version before each write, which doubles the network cost. For high-throughput integrations where reads are expensive, this becomes the bottleneck.

The pattern is more common in OLTP-style integrations (updating user records, modifying inventory) than in event-stream integrations (replicating a changelog), because OLTP already has version metadata on most entities.

4. The Saga Pattern for Multi-Step Operations

For operations that span multiple receivers and cannot be wrapped in a single transaction, the saga pattern uses local idempotent operations at each step with compensating actions to handle partial failures.

Each step in the saga is independently retry-safe via one of the other patterns (idempotency key, absolute state, version check). The saga adds a layer above the individual steps: if a later step fails, the orchestrator triggers compensating actions on the earlier steps to roll back.

Use case: any integration that touches more than one receiver and where partial application has to be cleaned up. Common in workflow orchestration, order processing across multiple services, multi-region replication.

Trade-off: significantly more design work than single-step patterns. Compensating actions are real code that has to be written and tested. The saga state machine has to be persisted and resumable.

Wikipedia's overview of the saga pattern covers the trade-offs against the simpler but more fragile two-phase commit protocol. Writing from practitioners like Martin Fowler at martinfowler.com covers the broader pattern in the context of microservice architectures.

5. The Broker-Level Dedup Pattern

Some message brokers (notably Apache Kafka with idempotent producer mode) handle a subset of the dedup problem at the transport layer. The producer attaches a sequence number to each message, and the broker rejects out-of-order or duplicate sequence numbers from the same producer.

Use case: producer-to-broker duplicate prevention. Useful as a foundation layer, not a complete solution.

Trade-off: only covers the producer-to-broker hop. Consumer-side duplicates (from rebalances or consumer crashes after processing but before offset commit) still happen and still need application-level idempotency.

The right framing is "necessary but not sufficient." Broker-level dedup eliminates one class of duplicate and reduces the volume the application-level dedup has to handle, but the consumer side of the integration still needs one of the other patterns.

How to Choose

A rough decision flow:

State-setting operation with infrequent updates and small messages: Absolute-state. Simplest pattern, naturally idempotent, no dedup store needed.

Side-effect operation (create, charge, send) with high volume: Idempotency key with event-time generation. Most general, requires receiver dedup store but scales well.

Relative operation on an entity that already has version metadata: Optimistic concurrency with version checks. Cleaner than idempotency keys when version is already present.

Multi-step operation across multiple receivers: Saga with idempotent steps. Required when partial failure cleanup is necessary.

Producer-to-broker messaging with consumer crash risk: Broker-level dedup plus application-level idempotency at the consumer. Layered defense.

Most production integrations use a mix. A typical pipeline might use absolute-state for inventory syncs, idempotency keys for outgoing webhooks, version checks for updates to user records, and broker-level dedup as a foundation underneath all of them.

Common Mistakes Across Patterns

A few patterns that reliably cause problems regardless of which idempotency pattern is in use:

Generating idempotency keys at send time. The key has to be stable across all retries of the same logical operation, which means generating it at the source-of-truth event, not at delivery attempt.

Unbounded dedup stores. Without TTL or partitioning, the dedup table grows until it becomes the bottleneck. Add cleanup as a baseline requirement, not as a follow-up.

Treating partial-success responses as success. A receiver that returns 200 OK after the side effect but before recording the dedup key opens a window where retries duplicate the side effect. The dedup record has to be written atomically with the side effect.

Skipping the observation phase during retrofit. Adding idempotency to an existing integration in one big-bang deploy is much riskier than the multi-step sequence of "record but do not enforce" then "enforce after observation."

Why Pattern Choice Matters

The choice of pattern shapes the cost structure of the integration over its lifetime. Idempotency keys with a well-managed dedup store have low ongoing cost but require disciplined design at the start. Absolute-state has minimal ongoing cost but constrains the message format. Sagas have high design cost but produce the cleanest behavior for multi-step operations.

The wrong pattern usually shows up as either ongoing operational pain (cleaning up dedup tables that grew too large) or design pain (trying to add a saga compensation flow after the integration is in production). The right pattern shows up as the absence of incidents.

For more depth on these patterns and the specific design choices that make each one production-grade, https://137foundry.com covers the broader engineering practice these patterns plug into. The 137Foundry services overview covers how integration design fits with the rest of the platform work, and the longer reference on idempotency in data integration pipelines walks through the unified theory that ties the five patterns together.

The Take

There is no single right pattern. There are five well-understood ones, each with a clear fit and a clear cost. The integrations that survive in production are the ones where the pattern was chosen deliberately for the operation shape, not the ones where the team picked one pattern and tried to make every operation fit it.

The five patterns above are the toolkit. Picking the right one is the engineering work.

Why the Empty States in a Data Table Deserve Three Separate Designs Instead of One Generic Message

137Foundry — Sat, 20 Jun 2026 10:04:27 +0000

Most data tables ship with a single empty state: a centered "No data" message, sometimes with a small illustration, sometimes with a "create new" button. It looks fine in design review. It produces support tickets in production.

The problem is that "the table is empty" can mean three completely different things, and the user's right next action is different in each case. A single generic message hides that distinction and leaves the user guessing.

The three cases are: first-use (no data exists yet), filtered-empty (data exists but the current filters exclude everything), and load-failure (the request to fetch data did not succeed). Each one needs its own copy, its own actionable element, and its own visual treatment. Designing them all the same is one of the more common UX failures in data-heavy tools.

Case 1: First-use empty

The user has just landed on a screen where no data exists yet, because no one on the team has created anything that would appear here.

The user's question: "What goes here, and how do I make it?"

The right response: an explanation of what would appear in the table, plus a primary call to action for creating the first item, plus optional secondary actions for related onboarding paths.

A good first-use empty state for a "Customer accounts" table:

No customer accounts yet.

Customer accounts let you track your client relationships and link them to invoices and notes.

Create your first account
Or import from CSV

The copy explains the purpose of the table in one sentence. The primary action is the most common path forward. The secondary action acknowledges an alternative path (importing from another system).

The mistake to avoid: a generic "No data" message in the first-use case. The user does not know what data should be there or how to create it. The message is a dead end.

The Nielsen Norman Group has consistently reported that first-use empty states are one of the highest-leverage moments in onboarding new users to a tool: a good empty state can compress the time-to-first-value from days to minutes. A bad one extends it indefinitely.

Case 2: Filtered-empty

Data exists in the table, but the user has applied filters that exclude all of it. The "no results matching" case.

The user's question: "Why is nothing showing? Did the tool break?"

The right response: an explicit message that filters are active and excluding all rows, plus an action to clear or modify the filters.

A good filtered-empty state:

No accounts match the current filters.

You're filtering for: Status = Closed AND Created after 2025-01-01.

Clear all filters
Or Edit filters

The copy names the filters that are active. This is the most common cause of "why is the table empty" support tickets, and surfacing the cause in the empty state eliminates most of them.

The mistake to avoid: a generic "No data" message that does not mention filters. The user has forgotten which filters they applied (or has navigated back to a screen that preserved the filters from earlier) and assumes the data is gone. They click around, they file a ticket, they have a bad experience that was entirely preventable.

A subtler version of this failure: showing the filter chips above the empty state but not in the empty state itself. The filter chips become visual noise above an empty area; the user is staring at the empty area, not at the chips. The empty state needs to surface the filter status itself.

Case 3: Load-failure

The fetch to load data failed. The server returned an error, the network is down, the auth token expired, the API rate-limited the user.

The user's question: "Is this temporary? Should I try again or do I need to do something?"

The right response: an explicit message that loading failed, plus context about the failure (when it happened, what error code, how to report it), plus a retry action.

A good load-failure empty state:

Could not load accounts.

Last attempt: 2 minutes ago. The server returned a 503 error, which usually means the service is temporarily unavailable.

Retry
If the problem persists, contact support and reference error code REF-7281.

The copy distinguishes the failure mode (503 versus 401 versus network error versus timeout) when possible. The user can self-diagnose whether to retry or to act on the underlying cause (re-authenticate, contact support, wait).

The mistake to avoid: showing the same "No data" message for load-failure as for the other cases. The user assumes the data is empty, makes wrong decisions based on that assumption, and may discover hours later that the load was failing silently.

The Web Content Accessibility Guidelines cover error message standards that apply to load-failure cases: the message should be perceivable, the cause should be identifiable, and the user should be able to recover. Generic messages fail all three.

Why teams design only one empty state

A few patterns I have observed repeatedly:

Empty states get designed late. The table is built, data flows in during development, and no one notices that the empty case has not been designed until QA or production. By then, the team is shipping something and reaches for a generic placeholder.

Empty states get designed during design review, when the table has demo data. The designer adds an empty state, the team reviews it in the context of the demo data, and the three different empty cases never come up because the data is there.

Empty states are seen as edge cases. The team thinks empty states are uncommon and not worth the polish. They are wrong: filtered-empty happens routinely in production, load-failure happens whenever the network has a hiccup, first-use happens for every new user the tool ever onboards.

Empty states are conflated with placeholder states. A "skeleton loader" while data loads is different from an empty state after data has loaded. Teams sometimes design only the skeleton and forget that the post-load empty case still needs its own treatment.

What good empty state coverage looks like

A practical checklist for empty state design in a data-heavy tool:

First-use empty: explains the table's purpose, primary CTA to create the first item, secondary CTA for alternative paths (import, link existing data).
Filtered-empty: names the active filters in the empty message, primary CTA to clear filters, secondary CTA to edit filters.
Load-failure: distinguishes the failure type when possible, includes a timestamp, primary CTA to retry, optional secondary CTA to contact support with an error reference.
Partial-load: when some rows loaded but the request was incomplete or paginated and the next page failed. Shows what loaded plus a load-failure message for the missing portion. This is a sub-case of load-failure but worth designing separately.
Permission-denied: when the user is authenticated but does not have access to view the data. Different from load-failure: the right action is to request access, not to retry. This is sometimes overlooked.

The pattern at 137Foundry for clients building data-heavy interfaces is to surface all five cases in design review explicitly, before the table goes to QA. The cost of designing them separately is small; the cost of shipping a generic placeholder is paid forever in support volume. Designers and developers benefit from looking at all five together because the cases share visual treatment but differ in copy and action.

A practical implementation note

The three (or five) empty states are usually conditional renders in the table component. A common implementation pattern:

function TableBody({ data, loading, error, filters }) {
  if (loading) return <SkeletonLoader />;
  if (error) return <LoadFailureState error={error} onRetry={refetch} />;
  if (data.length === 0 && hasActiveFilters(filters)) {
    return <FilteredEmptyState filters={filters} onClear={clearFilters} />;
  }
  if (data.length === 0) {
    return <FirstUseEmptyState />;
  }
  return <DataRows data={data} />;
}

The conditional structure makes it clear which case is which, and prevents the "fall through to generic empty" failure that happens when there is only one empty state component for all cases.

The Mozilla Developer Network covers the underlying fetch and error-handling patterns that the load-failure case needs to detect specifically. Building empty states well is partly a design question and partly a question of having reliable error context to display, which depends on the data-fetching layer surfacing useful errors.

What the user experiences when this is done right

A user encountering a first-use table sees a welcoming explanation of what goes there and a clear next action. They create the first item and the table populates.

A user who filters to an empty state sees an explicit "filters are excluding everything" message and clears the filters. They never wonder if the tool is broken.

A user who hits a load-failure sees a clear failure message with a retry button and an error reference. They retry, the load succeeds, they continue. If retries keep failing, they have a reference to give support.

In all three cases, the time the user spends being confused is short, the next action is clear, and the support tickets do not get filed. That is what good empty state coverage actually produces.

The longer treatment of data table design at scale covers how empty states pair with the rest of the design decisions in a production data table.

How to Design Keyboard Navigation for a Data Grid So Power Users Stop Reaching for the Mouse

137Foundry — Sat, 20 Jun 2026 10:04:27 +0000

A well-designed data grid lets power users navigate, edit, and act on rows without ever touching the mouse. They tab into the grid, use arrow keys to move between cells, hit Enter to activate the row's primary action, hit Escape to cancel, and tab back out. Their hands stay on the home row. Their throughput is two to three times higher than a mouse-driven user on the same interface.

A poorly-designed data grid forces those same users to mouse-click every action. Even with keyboard shortcuts that exist somewhere in the documentation, the in-grid navigation is broken enough that the keyboard path is slower than the mouse path. Power users learn to live with it but they hate the tool.

The difference between these two outcomes is roughly a hundred lines of focus management code, plus deliberate design choices about which keys do what. Here is the design and what to watch for.

The standard keyboard model

A reasonable default for data grid keyboard navigation:

Tab moves focus into the grid from outside, and out of the grid to the next focusable element. Once in the grid, Tab moves between focusable controls within the active row (action buttons, edit controls). Tab does NOT move between cells in the grid; arrow keys do that.
Arrow keys move between cells. Up and Down between rows in the same column. Left and Right between columns in the same row. Holding Shift extends the selection.
Enter activates the primary action for the current row. Usually this is "open the row's detail view" or "edit this row inline." The primary action is the one a mouse user would click on by clicking the row identity column.
Space toggles the row's checkbox in tables with bulk-select. In single-select tables, Space is the same as Enter.
Escape cancels any in-progress edit, closes any open menu, or moves focus back to the parent element if no other escape action applies.
Home / End move to the first / last column in the current row. Ctrl+Home / Ctrl+End move to the first / last cell in the grid.
Page Up / Page Down move one viewport of rows up / down.

This model matches the WAI-ARIA Authoring Practices for grids, and screen reader users and pure-keyboard users have been trained on it across most enterprise tools. Diverging from this pattern means asking users to relearn the grid, which they will resist.

The W3C Web Accessibility Initiative maintains the authoring practices documentation, which is the canonical reference for the standard grid keyboard model.

Why Tab versus arrow keys matters

The most common keyboard-navigation failure in data grids: using Tab to move between cells.

This makes intuitive sense (Tab moves focus through page elements), but breaks at scale. A grid with 7 columns means Tab cycles through 7 focusable elements per row. A 20 row visible viewport means 140 Tab presses to traverse one screen of data. Compare to arrow keys, where a single Down press moves to the next row regardless of column count.

The standard model uses Tab for entering and exiting the grid (a single focusable composite control), and arrow keys for navigation within the grid. This is called "composite widget" focus management, and it is the right pattern for any grid-shaped control.

Implementing this requires tabindex="-1" on cells (so they are programmatically focusable but not in the Tab order) and tabindex="0" on the grid container (so Tab enters the grid). The active cell within the grid receives tabindex="0" when focused, with all other cells getting tabindex="-1". The grid manages which cell is the current focus target.

function handleKeyDown(event, grid) {
  const { key } = event;
  const current = grid.activeCell;

  if (key === 'ArrowDown') {
    grid.activeCell = nextRow(current);
    event.preventDefault();
  }
  if (key === 'ArrowUp') {
    grid.activeCell = prevRow(current);
    event.preventDefault();
  }
  if (key === 'ArrowRight') {
    grid.activeCell = nextColumn(current);
    event.preventDefault();
  }
  if (key === 'ArrowLeft') {
    grid.activeCell = prevColumn(current);
    event.preventDefault();
  }
  if (key === 'Enter') {
    activatePrimaryAction(current);
    event.preventDefault();
  }
  if (key === 'Home') {
    grid.activeCell = firstColumnInRow(current);
    event.preventDefault();
  }
  if (key === 'End') {
    grid.activeCell = lastColumnInRow(current);
    event.preventDefault();
  }
  // etc.
}

The event.preventDefault() calls are important: without them, arrow keys would also scroll the page, which fights the in-grid navigation.

Inline editing patterns

Tables with inline editing need an additional keyboard model layered on top of navigation.

The common pattern: a cell has two states, navigation (the default) and edit. Pressing Enter on a navigation-state cell switches to edit state. The edit state is a text input, dropdown, date picker, or other interactive control. Pressing Enter on an edit-state cell commits the change. Pressing Escape on an edit-state cell cancels and returns to navigation state.

In edit state, arrow keys move within the editor (typing left and right in a text field, opening dropdown options) rather than moving between cells. This is what users expect when actively editing a field.

The transition between states is what teams get wrong. A few rules that help:

Tab in edit state commits the change and moves to the next cell in edit state, if the next cell is editable. Otherwise commits and moves focus to the next cell in navigation state. This matches spreadsheet patterns and is what most users expect.
Enter in edit state on the last editable cell of a row commits and moves to the first editable cell of the next row. Again, spreadsheet-style.
Click outside the cell in edit state commits the change. This matches most modern editor patterns. Some tools cancel on click-outside; this is harder to recover from and frustrates users who accidentally click elsewhere.
Escape in edit state cancels and returns the value to what it was before editing. This is the standard escape behavior across operating system controls.

Bulk-select with Shift and Ctrl

For grids with bulk operations, the standard keyboard model layers on selection patterns from file managers and spreadsheets:

Space toggles the selection state of the focused row.
Shift+Space selects from the last-selected row to the current focus, inclusive.
Ctrl+A (or Cmd+A on Mac) selects all rows. This should respect any active filters: select-all-filtered, not select-all-data.
Ctrl+Click in a selection mouse interaction toggles individual rows; the keyboard equivalent is just Space.

The Shift+Click range-selection pattern is the one most worth implementing for power users: it cuts the time to select 200 rows from 200 clicks to 2 clicks. The keyboard equivalent (navigate to first, Space, navigate to last, Shift+Space) is the same pattern via the keyboard.

The Nielsen Norman Group has published longitudinal research on bulk-selection patterns in enterprise tools, and the Shift-range pattern consistently shows up as the highest-impact selection feature for power users.

Discoverability without clutter

Power users learn the keyboard shortcuts. Casual users may never learn them. The question is how to make the shortcuts discoverable without cluttering the interface.

A few patterns work:

A "?" or "Shift+?" keyboard shortcut that opens a help overlay. The overlay lists all keyboard shortcuts for the current view. Most modern web applications use this pattern, and users have been trained to try it.

Inline tooltips on action buttons that include the shortcut. A button labeled "Edit" with a tooltip "Edit (Enter)" teaches the shortcut every time the user hovers.

Onboarding mention. A one-time tooltip during first-use that says "Use arrow keys to navigate and Enter to act." Most users dismiss it, but the small fraction who do not pick up enough of the model to discover the rest.

Documentation page. A dedicated keyboard shortcuts page in the help section, linked from the help menu. This is the fallback for users who think to look.

The combination of these (help overlay, tooltips, onboarding hint, docs) covers different discovery paths without forcing the shortcuts onto every user.

Common failures to avoid

A few patterns reliably break keyboard navigation in data grids:

Capturing too many keys. A grid that intercepts every keypress (including text input) breaks accessibility tools and surprises users. Capture only the keys the grid needs (arrows, Enter, Escape, Tab, Space, Home, End, PageUp, PageDown), and let everything else pass through to the browser.

Inconsistent focus between row click and keyboard nav. Clicking a row should put focus in a predictable cell (usually the identity column). Arrow keys should then move from that cell. If clicking jumps the focus to one cell but arrow keys treat a different cell as the starting point, users get confused.

Tab order that includes hidden controls. Columns that are not visible (because of column customization or virtualization) sometimes accidentally remain in the Tab order. Tab presses go to invisible elements, which is disorienting. Use tabindex="-1" on anything not visually present.

Modal interactions that trap focus incorrectly. A row-detail panel that opens on Enter should trap focus within itself until the user closes it. Returning from the modal should restore focus to the row that opened it, not to the first cell of the grid.

No focus indicator. A focused cell that does not have a visible focus ring is invisible to keyboard users. Make the focus indicator prominent. The default browser focus ring is often inadequate against busy table backgrounds; design a custom one that is clearly visible.

A practical implementation order

For adding keyboard navigation to an existing data grid:

Add Tab-in / Tab-out at the grid level (composite widget pattern).
Wire arrow key navigation between cells, with preventDefault on the events.
Add Enter for primary row action, Escape for cancel.
Add Home / End / Ctrl+Home / Ctrl+End / PageUp / PageDown.
Add Space for bulk-select toggle, Shift+Space for range select.
Add Ctrl+A for select-all-filtered.
Layer the inline edit state on top, with Enter / Escape / Tab transitions.
Add the "?" help overlay listing all shortcuts.
Add tooltips on action buttons that include the shortcut.
Test with actual keyboard-only usage and with screen readers.

The longer write-up of 137Foundry on data table design at scale covers how keyboard navigation fits with the rest of the table's design decisions (defaults, column widths, virtualization). For the formal authoring practices that define the standard model, the W3C Web Accessibility Initiative is the canonical reference, and the Mozilla Developer Network covers the underlying browser focus management APIs.

What good keyboard navigation actually looks like

A power user lands in a 5,000 row data grid via a link from another screen. The grid loads, Tab moves focus into the grid, the focus indicator appears on a sensible default cell (usually the first cell of the first row). The user uses Down arrow to move to the row they want. They press Enter to open the row detail panel. They close it with Escape and return to the grid with focus restored to the row they opened. They Shift+Space to extend selection to the row above. They use a custom shortcut (Ctrl+E in this hypothetical tool) to bulk-edit the selected rows. They commit the edit with Enter. They Tab out of the grid and continue with their workflow.

The whole sequence is keyboard-only and fast. The mouse path for the same workflow would take 2 to 3 times as long. Multiplied across a workday and a user population of power users, the keyboard model produces a meaningful productivity difference in the tool.

That outcome is what good keyboard navigation actually buys, and the implementation cost (a hundred lines of focus management, a help overlay, well-tested transitions) is small compared to the value.

Why Your Service Worker Cache Is Silently Breaking Your Offline Mode

137Foundry — Fri, 19 Jun 2026 10:09:26 +0000

A Progressive Web App that promises offline support and only mostly delivers is worse than one that promises nothing. Users learn to distrust the offline indicator after the first time it lies to them, and the entire feature stops being a feature. The most common cause of this kind of silent failure is a service worker cache that does not actually contain what the app needs to operate offline.

This piece walks through the patterns that produce the silent break and the techniques that prevent it.

Photo by İsmail Enes Ayhan on Unsplash

The silent break, in one sentence

Your service worker caches what the user has already visited, which is usually less than what the user will try to visit while offline. The mismatch is silent because there are no errors; the app simply fails to load assets it needs and falls back to whatever skeleton or error state it has, often without the user understanding why.

The fix is not to cache more aggressively. The fix is to be deliberate about what gets cached, when, and from what trigger. Most service worker caches end up reactive: the first time the user visits a route, that route's assets are cached. The second time, they are served from cache. The user never visits the third route offline because they have not visited it online yet either.

Step one: precache the critical shell

The most reliable pattern is to precache the application shell at install time. The shell is the set of assets the app needs to start: the entry HTML, the main JavaScript bundle, the main CSS, any fonts, and any images that are part of the always-visible UI.

Precaching is done in the service worker's install event:

self.addEventListener('install', (event) => {
  event.waitUntil(
    caches.open('app-shell-v1').then((cache) => {
      return cache.addAll([
        '/',
        '/main.js',
        '/main.css',
        '/fonts/inter.woff2',
        '/icons/logo.svg',
      ])
    })
  )
})

The shell is loaded once, kept in cache forever (or until the version is bumped), and serves as the foundation for offline operation. Users visiting any route within the app can get to a working shell even on a cold offline visit, because the shell was cached at install time, not on first visit.

For the deeper offline-tolerant data fetching that needs to layer on top of this, the broader pattern lives in the Workbox documentation and the MDN service worker reference, which together cover the most common runtime caching strategies.

Step two: distinguish between cache-first and network-first routes

Once the shell is in place, runtime caching of API responses and dynamic content has to choose a strategy per route. The two that come up most often:

Cache-first means the service worker checks the cache first and only falls back to the network on miss. Good for assets that rarely change (avatars, product images, font files). Bad for API responses where staleness matters.

Network-first means the service worker tries the network first and falls back to the cache only on network failure. Good for API responses where freshness matters more than offline support. The user gets fresh data when online and the last-known data when offline.

A third pattern, stale-while-revalidate at the service worker layer, returns the cached response immediately and fires a background fetch that updates the cache for next time. This is the same idea as the HTTP-level stale-while-revalidate directive but implemented in JavaScript with more control.

The trap is using cache-first for API responses without thinking. The user updates their profile, the server stores the change, the service worker keeps returning the cached old profile because the cache hit comes back before the network attempt. The offline mode "works" in that the UI loads, but the data is wrong.

Step three: handle the cache version bump

Service worker caches are versioned. When you ship a new version of the app, the new service worker installs alongside the old one and the old caches stay around until something explicitly cleans them up. If you do not handle this, the user's browser accumulates dead caches forever, and storage quota eventually evicts good caches before bad ones.

The pattern is to clean up old caches in the activate event:

self.addEventListener('activate', (event) => {
  const allowList = ['app-shell-v2', 'api-cache-v1']
  event.waitUntil(
    caches.keys().then((keys) =>
      Promise.all(
        keys.filter((key) => !allowList.includes(key)).map((key) => caches.delete(key))
      )
    )
  )
})

Bump the version string on every release that changes the shell. The activate handler deletes everything that is not on the current allowlist, freeing storage for whatever the new version needs.

Photo by Anete Lusina on Pexels

Step four: detect quota pressure before it bites

Browsers limit the total storage a single origin can use. The limit varies by browser and by available disk space, but it is rarely as much as the application author assumes. When the limit is reached, the browser silently evicts entries. The app stops working offline, and the team has no signal that anything is wrong.

The fix is to estimate quota usage at runtime and log when it approaches the limit:

if ('storage' in navigator && 'estimate' in navigator.storage) {
  navigator.storage.estimate().then((estimate) => {
    const percent = (estimate.usage / estimate.quota) * 100
    if (percent > 80) {
      console.warn(`Storage at ${percent.toFixed(1)}% of quota`)
    }
  })
}

In production, this signal should go to your error tracking. Quota pressure is a leading indicator of impending offline breakage; catching it early lets the team prune caches before the user notices.

"Offline mode is not a feature you ship and forget. It is an ongoing operational commitment, and the silent failure modes deserve as much attention as the active ones." - Dennis Traina, founder of 137Foundry

Step five: test offline mode in CI

The hardest part of shipping reliable offline support is testing it. Manual testing is unreliable because developers always remember to visit the route online first, which warms the cache, which masks the bug.

The fix is to run an automated test that exercises the offline flow from a cold state. Playwright and similar tools let you simulate an offline network condition and verify that key routes still work. The test should:

Launch a fresh browser context (no cached state).
Navigate to the app once online to install the service worker and precache the shell.
Force the browser into offline mode.
Navigate to each route that should work offline and verify the rendered output.

If any route fails, the precache list is incomplete or a runtime caching strategy is wrong. The test catches the bug before users see it.

The web development team at 137Foundry treats this as a default part of any PWA we ship with offline support. Without it, the offline mode is theoretical, and the user complaints arrive eventually.

A small observability note

One more layer worth mentioning: instrument the service worker itself. Add logging to every fetch event handler that records whether the response came from cache or network, and what the cache name was. Aggregate these logs to your observability backend.

The metric to watch is the cache hit rate per route. A healthy precache hit rate is close to 100% for shell routes; a healthy runtime cache hit rate is 50% or more for routes the user visits repeatedly. A sudden drop indicates a regression, usually that a recent code change has broken the cache key or the precache list.

The service worker context is awkward for logging because it does not have direct access to the rest of the app's logging infrastructure. The pattern that works is to post a message from the service worker to the main thread with the log payload, and let the main thread forward it to the observability backend. A few lines of code, and the offline mode now has the same observability story as the rest of the app.

For the broader caching architecture, including how the service worker cache layer interacts with HTTP-level caching, the 137Foundry article on browser API caching walks through the decision matrix in detail. The web development service page covers the related architectural work we do for clients building PWAs.

How to Use ETags for Cheap Revalidation in a React App

137Foundry — Fri, 19 Jun 2026 10:07:37 +0000

ETags are one of the cheapest performance wins available to a React app talking to a JSON API. The browser does most of the work, the server returns a body-less 304 when the data has not changed, and the user sees response times that drop by a large fraction without any visible cost.

This guide walks through the practical steps to wire ETags into a React application end to end, with the patterns that work and the gotchas to know about.

Photo by Brett Sayles on Pexels

Step 1: Make sure the server returns ETags

ETags are server-driven. The server attaches an ETag: "some-value" header to each response, and the browser stores it alongside the body. On subsequent requests, the browser sends If-None-Match: "some-value" and the server either returns 304 with no body (cache stays valid) or 200 with a new body and a new ETag.

The implementation on the server can be one of two patterns:

Content-hash ETag. The server hashes the response body (SHA-1, MD5, anything stable). Useful for any response, but does not save the server-side work of generating the response.
Version-based ETag. The server uses a version or updated_at field from the underlying entity. Saves both the database read and the response generation, because revalidation can check the version alone.

Express, Next.js, and most modern Node.js frameworks support both patterns out of the box. Frameworks like Fastify and Hono make this even easier with first-class ETag middleware. Verify your server is actually sending ETags by inspecting the response headers in the browser dev tools network panel.

Step 2: Confirm the browser is sending If-None-Match on revalidation

If your Cache-Control headers are set to allow caching (e.g., Cache-Control: private, max-age=0, must-revalidate), the browser should automatically include If-None-Match on every subsequent request. Open the network panel, find your request, and check the request headers.

If the header is missing, the most likely cause is a Cache-Control: no-store directive somewhere in your response chain that is preventing the browser from storing the response at all. A no-store response cannot be revalidated, because there is nothing to revalidate against. Change to no-cache (the response is stored but must be revalidated on every use) if you want ETag-driven freshness.

Step 3: Pick a React data-fetching library that respects HTTP caching

Most modern React data-fetching libraries layer their own cache on top of the browser cache, and not all of them play nicely with HTTP-level caching. The two that handle this well:

TanStack Query (formerly React Query) honors HTTP caching by default if you use the fetch API directly inside your query function. The library's own staleTime and cacheTime are layered on top.
SWR does the same. The library's own cache is keyed by URL, and the underlying fetch respects browser HTTP caching transparently.

The library-level cache is configured to be aggressive by default (stale data is served while a background revalidation fires). The browser-level cache adds another layer underneath, providing the 304-based optimization without any extra code.

import { useQuery } from '@tanstack/react-query'

function UserProfile({ userId }) {
  const { data, isLoading } = useQuery({
    queryKey: ['user', userId],
    queryFn: () => fetch(`/api/users/${userId}`).then(r => r.json()),
    staleTime: 30 * 1000,
  })

  if (isLoading) return <Skeleton />
  return <ProfileCard user={data} />
}

The fetch call here automatically picks up the browser's cached response when revalidation produces a 304. No additional configuration needed.

Photo by fauxels on Pexels

Step 4: Set up Cache-Control to allow the browser to cache

The Cache-Control header decides whether the browser will store the response at all. For ETag-driven revalidation to work, the response must be storable. The directive that works for most user-specific API endpoints:

Cache-Control: private, max-age=0, must-revalidate

This says: store the response, but always revalidate before serving it. The revalidation uses the ETag. On 304, the cache is served. On 200, the cache is replaced. Both outcomes are correct.

For endpoints where stale-while-revalidate is appropriate, add it:

Cache-Control: private, max-age=0, stale-while-revalidate=60, must-revalidate

This serves the cached response immediately while firing a background revalidation, giving the user a fast load and converging to fresh within sixty seconds.

Step 5: Verify the round trip with the dev tools

A working ETag round trip shows up in the network panel as:

First request: 200 OK, full response body, ETag header in response.
Subsequent requests within the cache window: 304 Not Modified, no response body, If-None-Match header in request.

If you see 200 every time, the browser is not honoring the cache. Check Cache-Control on the response. If you see 304 but the response body in the panel is still the full body, that is the dev tools showing the cached body for clarity, not the actual network response. The actual network transfer is just the 304 headers.

A common gotcha: the browser's network panel has a "Disable cache" checkbox that, when on, prevents any cache from being used. Make sure it is off when testing.

"The 304 round trip is the single cheapest thing you can do to make a React app feel snappy. Almost no code changes; almost all the win comes from the server doing its part." - Dennis Traina, founder of 137Foundry

Step 6: Handle the invalidation case

When the underlying data changes, the server's next response will be 200 with a new ETag. The browser stores the new response and the new ETag. The next revalidation uses the new ETag, gets a 304, and the cycle continues.

There is one case where this falls down: if the underlying data changed but the user's open tab does not know about it. Until the user takes an action that triggers a refetch, the UI continues to show the previous response.

The pattern that solves this is to combine ETag revalidation with library-level staleness. TanStack Query and SWR will refetch on window focus and on network reconnect by default. These triggers, combined with the 304 round trip when nothing has changed, give you fast loads and reasonably fresh data without explicit work.

For real-time updates where the user must see changes immediately, the right pattern is server-pushed invalidation via WebSocket or Server-Sent Events. ETags are for the case where freshness within a few seconds of revalidation is good enough, which covers most dashboards and admin interfaces.

The architecture team at 137Foundry defaults to the ETag-plus-library-revalidation pattern for any React app with significant API traffic. It is the strategy with the best performance-per-line-of-code ratio.

Step 7: Layer in a small monitoring habit

The last piece is a monitoring habit that catches regressions when the ETag pipeline degrades. Add a small instrumented wrapper around your fetch calls that logs the response status, the elapsed time, and whether the response was a 304 or a 200. Aggregate these metrics in your observability tool of choice.

The metric to watch is the 304 rate over time. Healthy ETag-driven caching produces a 304 rate of 40% to 70% on typical user sessions. If the rate drops suddenly, something has changed about the cache directives or the ETag implementation. Catching this within a day saves the team from a slow degradation that nobody notices until the API costs start climbing.

For an app at modest scale, this monitoring is two lines of code wrapped around the fetch call. For an app at larger scale, an APM tool like Datadog or Sentry's performance monitoring handles it more cleanly. The discipline matters more than the tool.

For the deeper version of this guide, including the broader browser caching architecture and where the Cache API fits in, see the 137Foundry article on browser API caching. The web development service page covers some of the related architectural work as well.

How to Wire aria-describedby and aria-invalid on Form Errors Without Breaking Your Styles

137Foundry — Thu, 18 Jun 2026 10:02:55 +0000

Most existing forms have a validation experience that visually works and is invisible to users on assistive technology. The fix is usually two ARIA attributes (aria-describedby and aria-invalid) plus a small live region on the error element. The total code change for a typical form is under 50 lines and produces a meaningful improvement in accessibility scores.

This is a step-by-step walkthrough of the wiring, with the specific gotchas that come up when you retrofit it onto an existing form.

Photo by Yusuf Çelik on Pexels

The current state of most forms

A typical existing form has this kind of structure:

<div class="form-field">
  <label for="email-input">Email</label>
  <input type="email" id="email-input" name="email" />
  <div class="error" id="email-error"></div>
</div>

The error div is in the DOM. The validation JavaScript writes a message into it when something is wrong. The CSS styles the error red. From a sighted user's perspective, it works.

From a screen reader user's perspective, the input and the error are unrelated DOM siblings. Focusing the input announces only the label, not the error. The error appears and a user not navigating to it never knows.

The fix is connecting the input to the error programmatically.

Step 1: add aria-describedby

The first attribute. The input gets aria-describedby pointing to the id of the error element.

<input
  type="email"
  id="email-input"
  name="email"
  aria-describedby="email-error"
/>

This is what tells the screen reader that when the user focuses the input, the content of the #email-error element is part of the description. The same attribute can take multiple ids (aria-describedby="email-error email-hint") if you also have a static help text element below the input.

The attribute can be on the input from the start; you do not have to add and remove it based on whether an error is present. The screen reader announces the description only when there is content in the referenced element. An empty error div produces no announcement.

Step 2: add aria-invalid dynamically

aria-invalid is the programmatic signal that the input has a validation error. Set it to true when the error shows, false (or remove it) when the error clears.

function showError(input, errorElement, message) {
  errorElement.textContent = message;
  input.setAttribute('aria-invalid', 'true');
}

function clearError(input, errorElement) {
  errorElement.textContent = '';
  input.setAttribute('aria-invalid', 'false');
}

aria-invalid is separate from the visual styling. Screen readers announce it as "invalid entry" or similar; sighted users see the red border via CSS. Both signals point at the same state but go to different users.

A common mistake: setting aria-invalid on the input element only, while the visual red border is on a wrapping div. Either move the styling to the input itself (so the visual and the ARIA share a target), or add a corresponding visual class on the input in the same function. Drift between the two is the root cause of most "the form looks valid but the screen reader says invalid" bugs.

Step 3: make the error announce when it appears

The aria-describedby connection is sufficient if the user navigates back to the field after the error appears. It is not sufficient if the error appears while the user is still focused on the field (live re-validation on keystroke after an initial error).

A live region on the error element fixes this:

<div class="error" id="email-error" aria-live="polite" role="status"></div>

aria-live="polite" tells the screen reader to announce changes at the next natural pause, not interrupt the user mid-typing. The role="status" is a redundant signal (many implementations require both for cross-browser support; you can drop role and test with target screen readers if you want minimal markup).

Polite is the right choice for form validation. The assertive variant (aria-live="assertive") interrupts and is reserved for emergencies, which a form error is not.

Step 4: handle the case where the input itself has the busy state

For async validation (username check, address lookup), the user expects feedback that the field is being verified. Without it, the spinner is the only signal, and the screen reader user sees neither.

Two patterns:

Pattern A: Use aria-busy="true" on the input or its wrapper while the check is in flight. Most screen readers will announce that the field is busy.

Pattern B: Use a separate live region for status updates ("Verifying username...") that updates when the check starts and clears when it completes.

Pattern B works more reliably across screen readers, at the cost of slightly more markup. Pattern A is simpler when it works but inconsistent across implementations.

<div class="form-field">
  <label for="username-input">Username</label>
  <input
    type="text"
    id="username-input"
    name="username"
    aria-describedby="username-error username-status"
  />
  <div class="error" id="username-error" aria-live="polite"></div>
  <div class="status visually-hidden" id="username-status" aria-live="polite"></div>
</div>

The status element is visually hidden (via a standard visually-hidden CSS utility) but exposed to screen readers. It is the audio-only counterpart to the spinner.

Step 5: handle keyboard focus correctly

A field in an error state still needs to be reachable by keyboard. Two patterns that get broken in retrofits:

Focus stops working because of overflow rules. Some forms hide error states by clipping with overflow: hidden, which can also clip the focus outline. Make sure the focus indicator on an error field is still visible.
Tab order skips the error element. This is usually fine because the error is not interactive (no need to tab into it), but if you have an "edit" or "fix" link in the error message, make sure it is in the tab order with tabindex="0".

The general rule: keyboard navigation through the form should be unchanged by the presence of errors. The errors are descriptive, not interactive.

Photo by Letícia Alvares on Pexels

Step 6: verify with the standards

Once the wiring is in place, the W3C Web Content Accessibility Guidelines success criteria most relevant to form validation are:

3.3.1 Error Identification: Errors must be identified in text. Color alone is not sufficient.
3.3.3 Error Suggestion: When the error has known fixes, suggest them. "Add the @ to make this a valid email" is a suggestion; "Invalid email" is not.
4.1.3 Status Messages: Status messages (including errors) must be programmatically determined and announced to assistive tech without receiving focus.

The WAI-ARIA Authoring Practices Guide has worked code samples for accessible form validation patterns. If you are unsure about a specific wiring choice, the examples in the APG are the authoritative reference.

Step 7: test with a real screen reader

ARIA wiring that looks correct in code does not always announce correctly. Test with at least one screen reader before merging. NVDA on Windows is free; VoiceOver on macOS is built in. Ten minutes of testing on a real screen reader catches more bugs than any automated audit.

The minimum test pass:

Tab into the field. Label announces.
Type an invalid value. Tab away. Error announces.
Tab back to the field. Label announces with the error as the description.
Fix the input. Error clears, optionally with an announcement.
Trigger an async check. Pending status announces. Result announces when ready.

Tools that complement manual testing:

WAVE browser extension surfaces ARIA wiring gaps inline on the page.
axe DevTools browser extension catches ARIA misuse and color contrast issues; usable in CI for regression testing.
WebAIM contrast checker verifies error text and input border colors meet WCAG thresholds.

Step 8: keep the existing styles intact

The whole point of this retrofit is to avoid rebuilding the form. The styles already work for sighted users; the ARIA additions are about adding signal for users on assistive tech.

Specific things to avoid changing:

The visual layout of the form. The error div stays where it is.
The color of the error text or the input border. If the colors meet WCAG, leave them.
The trigger logic (when validation runs). If the timing is wrong, fix it as a separate change, not as part of the accessibility retrofit. Bundling the two makes both harder to review.

The retrofit is a small targeted change. The design improvements come next.

The longer read on the full design

This piece is the wiring mechanics. The full design of inline validation (when to trigger, what to say, async patterns, visual treatment, success states) sits in a longer guide on designing inline form validation that actually helps users on https://137foundry.com. The two pieces stack: this one wires the accessibility, the longer one improves the design that the accessibility now exposes correctly.

The right order is: ARIA retrofit first (cheap, high impact), then the design improvements on top of an accessibility-correct baseline. Skipping the retrofit and only doing the design work means the improved design is still invisible to users on assistive technology, which defeats the point.

Why the Async Username Check Is the Worst Part of Most Signup Forms

137Foundry — Thu, 18 Jun 2026 10:02:54 +0000

I have looked at a lot of signup forms in the last year. Most of them have one specific bug that the team probably knows about but has not prioritized: the async username check.

The pattern shows up in slightly different shapes. Sometimes the field reports "available" for half a second before the server responds with "taken." Sometimes the spinner runs for two seconds and the user has already moved to the next field. Sometimes the submit button does not block while the check is pending, so a user who mashes submit can get past the validation entirely. Sometimes the check fails on a network timeout and the user sees a red error that says "username is required" when actually their network is just slow.

Each of these is the same underlying problem: the synchronous validation patterns are well-trodden, but the way forms handle a server roundtrip in the middle of inline feedback is where users get most confused. This is a writeup of how to get it right.

Photo by Sanket Mishra on Pexels

What the user experience is supposed to be

Strip out the engineering. From the user's perspective:

User types a desired username.
User pauses (or tabs away) and the form does a quick server check.
The form tells the user either "this username is available" or "this is taken, here is what is wrong" within a second or so.
The user fixes the username if needed and the field updates the moment they have a working one.
The user submits and the username is reserved atomically with the rest of the form.

That is the spec. Now look at what most signup forms actually do and you will see at least one step where the implementation drifts.

Where the implementations actually fail

Fail mode 1: no debounce, one request per keystroke. The user types "j-o-h-n-s-m-i-t-h" and the form fires nine requests. Eight of them return results that no longer match the current input. The form has to track which request was the latest and ignore the older ones, which it usually does not, so the user sometimes sees "available" flash because an earlier in-flight request resolved last.

The fix is debouncing. Wait 300 to 500 ms after the user stops typing before firing the request. The user perception is "the form checks when I am done with the field," not "the form checks every letter."

Fail mode 2: no pending state. The form fires the request, the spinner is missing or hidden, and the user tabs away expecting that the field is now valid. Half a second later the server returns "taken" and the field flips to red while the user is already in the next field.

The fix is a visible pending state. A small spinner inside or next to the field, plus disabling the submit button. The user understands "this is still being checked" and waits, or at least is not surprised when the result comes in.

Fail mode 3: submit not blocked during pending check. The user types a username, mashes submit before the check completes, and the form sends with username_valid: true based on the assumption that the field looked fine at submit time. The server then either accepts the submission (creating a duplicate username state) or rejects it with a generic 400 that does not surface in the inline UI.

The fix is to disable submit while the check is pending. The button is gray, with a tooltip explaining why ("Verifying username...") until the check completes. The user sees the friction and waits a beat instead of submitting blind.

Fail mode 4: network errors masquerading as validation errors. The server timeout returns nothing or returns a 500. The form's error handler is wired to treat any non-success response as "username is invalid" and shows a red error. The user thinks they did something wrong; the actual problem is the network or the server.

The fix is to distinguish the two cases. A successful server response that says "taken" is a validation error and should show as one. A network timeout or a 5xx is an infrastructure error and should show as one ("we could not verify this right now, please try again"). The two messages are different because the user actions to fix them are different.

The pattern that handles all four

A small piece of pseudocode that captures the right shape:

let activeRequest = null;
let debounceTimer = null;

input.addEventListener('input', () => {
  // Clear any pending debounce and previous error
  clearTimeout(debounceTimer);
  clearError(input, errorElement);

  // Start a new debounce window
  debounceTimer = setTimeout(() => {
    runCheck(input.value);
  }, 400);
});

async function runCheck(value) {
  // Mark request as in flight and disable submit
  setPendingState(input, true);
  submitButton.disabled = true;

  const requestId = ++latestRequestId;
  try {
    const result = await fetch(`/api/usernames/check?value=${encodeURIComponent(value)}`);

    // If a newer request has started, ignore this one
    if (requestId !== latestRequestId) return;

    if (result.ok) {
      const data = await result.json();
      if (data.available) {
        clearError(input, errorElement);
      } else {
        showError(input, errorElement, `${value} is taken. Try another.`);
      }
    } else {
      showInfrastructureMessage(input, errorElement, 'We could not verify this right now. Please try again.');
    }
  } catch (e) {
    showInfrastructureMessage(input, errorElement, 'We could not verify this right now. Please try again.');
  } finally {
    setPendingState(input, false);
    submitButton.disabled = false;
  }
}

The four fail modes map to four patterns in this code: debounce, in-flight tracking with a request id, distinguishing OK from network failure, and disabling submit.

The accessibility piece nobody hand-codes

Once the validation logic is correct, the announcement needs to be wired so screen readers convey the right state. The minimum:

aria-describedby="username-error" on the input pointing to the error element.
aria-invalid="true" set when the error is showing, cleared when it clears.
aria-live="polite" on the error element so the message is announced when it changes.
A separate live region or aria-busy="true" on the input while the check is pending, so the screen reader user knows the field is still being evaluated.

The W3C Web Content Accessibility Guidelines cover the formal requirements; the WAI-ARIA Authoring Practices Guide has worked examples of accessible form validation patterns. The pattern is stable across screen readers; the work is making sure your code actually emits the ARIA states consistently.

Photo by Campaign Creators on Unsplash

The atomic uniqueness check on submit

Even with a perfect inline check, the server still needs to be authoritative on uniqueness. Two users picking the same username at the same time will both pass the inline check; only one will pass the server transaction.

The server-side pattern: the actual username reservation happens in a database transaction with a unique constraint. If the insert fails because of a conflict, the server returns a structured error that the form maps back to a friendly message ("That username was just taken. Try another."). The inline check is a UX optimization; the server is the source of truth.

This is the pattern that makes the form survive the rare race condition without confusing the user. The form respects the inline result for the common case and trusts the server for the conflict case.

How to test it

Three tests that catch most of the failure modes:

Throttle the network. Use the browser dev tools to add 1 to 2 seconds of latency on the username-check endpoint. Confirm the pending state is visible, submit is disabled, and the eventual result lands correctly.
Type fast and stop. Type a username quickly and stop. Confirm only one request fires (debounce working). Confirm the result is the result for the final input, not an earlier intermediate value.
Test a network failure. Use the dev tools to block the username-check endpoint. Confirm the form shows the infrastructure message, not a validation error.

Add an automated test for the in-flight tracking specifically: fire two checks with a small delay, return the first response after the second response, and confirm the form ignores the stale result. The bug where the wrong result wins because requests resolve out of order is one of the most common in real signup flows and is invisible without a test for it.

For a broader accessibility audit, the WAVE browser extension catches missing ARIA attributes; running it on the form during testing surfaces wiring gaps that manual testing misses.

The longer read on the full validation design

This piece focuses on the async username check because it is the part most teams ship wrong. The full design of inline validation (when to trigger sync checks, what the error message should say, how to handle paste and autofill, when success states earn their place) sits in a longer guide on designing inline form validation that actually helps users on 137Foundry's web development service.

The async case is where validation systems leak. Getting the four patterns above right (debounce, pending state, submit blocked, distinguish network from validation) is the difference between a form that feels smooth and one that feels broken.

How to Handle Late-Arriving Events in Apache Flink With Side Outputs

137Foundry — Wed, 17 Jun 2026 13:17:45 +0000

The biggest single-line bug in a Flink job is usually the watermark. The second biggest is what happens when an event arrives past the watermark. The defaults Flink ships with are sensible for the synthetic examples in the documentation but wrong for nearly every real-world integration.

This walks through how to wire up Flink's side-output mechanism for late events, and how to feed the side output into a correction process that keeps downstream consumers honest.

Photo by Quoc Anh Tran Duong on Pexels

What Flink does by default

A Flink keyed window with a tumbling window assigner emits a final result when the watermark passes the window's end time. By default, any event arriving with an event time at or before the window's end but after the watermark has passed is dropped silently.

The behavior is documented in the Apache Flink windowing documentation, and the relevant API surface is allowedLateness and sideOutputLateData.

The mental model that helps: the watermark is the pipeline's promise about "I have seen everything up to time T." allowedLateness is a grace period during which the window stays alive and accepts late events. sideOutputLateData is the escape hatch for events arriving past the grace period.

Setting up the side output

OutputTag<MyEvent> lateOutputTag = new OutputTag<MyEvent>("late-events") {};

SingleOutputStreamOperator<Aggregate> mainResult = events
    .keyBy(e -> e.getKey())
    .window(TumblingEventTimeWindows.of(Time.minutes(15)))
    .allowedLateness(Time.minutes(5))
    .sideOutputLateData(lateOutputTag)
    .process(new MyAggregator());

DataStream<MyEvent> lateEvents = mainResult.getSideOutput(lateOutputTag);
lateEvents.addSink(new LateEventSink());

The pattern: a fifteen-minute tumbling window, five-minute grace period, late events captured to a named side output and shipped to a sink. The LateEventSink should be durable; a Kafka topic or a database table both work.

Why allowedLateness alone is not enough

allowedLateness is necessary but not sufficient. It only widens the window's acceptance period to the value you set. Events arriving past that period are still dropped unless sideOutputLateData is also configured.

Setting allowedLateness to a very large value is a common workaround that does not work. The window state has to stay open for the full lateness period, so a one-hour allowed lateness on a fifteen-minute window means Flink holds state for one hour and fifteen minutes per window key. That is a lot of state. The recommended pattern is to set allowedLateness to a tight value (matching the realistic late-event distribution observed in production) and use the side output to absorb the longer-tail late events.

What goes in the LateEventSink

The sink should persist enough context to reconstruct the affected window for correction.

For each late event, write a row with: the original event payload, the event's timestamp, the window key, the window's start time, the window's end time, and the time the late event was received by the pipeline.

That last field, the receive time, is what your correction job uses to do incremental processing. A correction job that runs every fifteen minutes only needs to look at side-output rows with receive_time newer than its last successful run.

The correction job

The correction job is a separate Flink job (or batch job) that reads the side output, recomputes the aggregate for affected windows, and writes corrected aggregates back to the target table.

The mechanics:

# Pseudocode
last_run_ts = read_last_run_timestamp()
late_events = read_side_output(since=last_run_ts)
affected_windows = group_by_window_key_and_window_start(late_events)

for window_key, window_start, events in affected_windows:
    original_events = read_original_events_in_window(window_key, window_start)
    corrected_aggregate = aggregate(original_events + events)
    write_versioned_fact(window_key, window_start, corrected_aggregate, now())

write_last_run_timestamp(now())

The key piece is the versioned-fact write: instead of mutating the original aggregate row, write a new row with a later emitted_at timestamp. Downstream consumers query the latest version per window key.

Idempotent emit

The correction job has to be idempotent. If it crashes mid-run and restarts, it should not produce duplicate corrections.

The cleanest way to achieve idempotency is to make the target table's primary key (window_key, window_start, emitted_at), with emitted_at populated from a deterministic source like "the timestamp of the latest event in the correction batch." Then re-running the same batch produces the same emitted_at and the same primary key, which collides on insert and either no-ops or updates.

For PostgreSQL targets, INSERT ... ON CONFLICT DO UPDATE handles this in one statement. For BigQuery or Snowflake, MERGE does the same.

If your target is a data lake using Apache Iceberg, the Iceberg specification supports merge operations natively and the streaming engine can write through Iceberg's merge-on-read or copy-on-write modes depending on read pattern.

The view that downstream consumers query

CREATE VIEW window_aggregates_current AS
SELECT window_key, window_start, value
FROM (
  SELECT
    window_key,
    window_start,
    value,
    emitted_at,
    ROW_NUMBER() OVER (
      PARTITION BY window_key, window_start
      ORDER BY emitted_at DESC
    ) AS rn
  FROM window_aggregates
) ranked
WHERE rn = 1;

This is the only thing the dashboard, the BI tool, or any downstream consumer should query. They see the latest version per window automatically. The underlying table keeps every version for audit and reconstruction.

Metrics to add

numLateEvents is a counter you can register from inside the ProcessWindowFunction using Flink's metrics API. It increments every time a late event arrives. Tag it by window-key partition if your job has skewed partitions. The Prometheus docs cover the metric-collection layer once Flink is emitting these.

watermarkLag is the gap between current processing time and current watermark, in milliseconds. Flink exposes this as a built-in metric; just enable it in your metrics config.

correctionsEmittedPerHour is a counter from the correction job. Spikes here correlate with upstream incidents.

Wire all three into Prometheus or whatever your stack uses. The point is to have a visible signal when late-event volume changes, so you can retune the grace period or chase down a producer regression.

Tradeoffs to acknowledge

The side-output pattern is not free.

State management costs more because allowedLateness keeps windows open longer. For high-cardinality keys, this can be significant. Watch your state size and adjust if it grows beyond your operational budget.

The correction job adds latency. Downstream consumers see the "almost final" aggregate when the window closes, and the corrected aggregate fifteen minutes to one hour later. For most business reporting this is acceptable; for sub-second reporting it is not.

The versioned-fact pattern adds storage cost because every correction is persisted. Old versions are usually compacted on a quarterly schedule.

These costs are usually worth paying. The alternative is silent drift, which costs more in trust and rework.

What this looks like at 137Foundry

137Foundry has retrofitted this pattern onto live pipelines for clients integrating Stripe webhooks, Salesforce event streams, and internal CDC streams to Snowflake. The typical retrofit is a one- to two-week project, not a full rewrite.

The longer article that frames this in context, including the versioned-fact pattern in more detail and the lambda-architecture alternative, is How to Handle Late-Arriving Data in a Streaming Integration Pipeline Without Corrupting Downstream Reports. The piece above is the Flink-specific recipe; the article is the design rationale.

A note on testing

The hardest part of validating late-event handling is constructing a realistic test case. Most unit tests for streaming jobs assume events arrive in order. The late-event path only fires when events arrive out of order.

The cleanest test pattern: build a fixture stream that emits events with deliberately scrambled event times, including some events past the watermark, and assert that the side output contains the late events and the corrected aggregate matches the expected total. Run the test against an embedded Flink mini-cluster.

Without this test, the late-event handling path is essentially untested in CI, and regressions slip through unnoticed. With it, you can iterate on grace periods and side-output logic without fear.

How to Implement the Versioned-Fact Pattern for Streaming Corrections

137Foundry — Wed, 17 Jun 2026 13:16:39 +0000

The versioned-fact pattern is the most useful tool for handling corrections from a streaming pipeline without forcing every downstream consumer to learn how the pipeline works internally. This walks through how to implement it on a target table, what the view looks like, how to handle compaction, and what breaks if you skip steps.

The use case: a streaming pipeline emits a windowed aggregate (sum of transactions per minute, count of events per user per hour, whatever). Late events arrive after the initial aggregate is emitted. You need a way to publish corrected aggregates without breaking dashboards or BI tools that already read the original numbers.

Photo by Seraphfim Gallery on Pexels

The basic schema

The target table has the window key, the value, an emitted_at timestamp, and a primary key built from (window_key, emitted_at) or some equivalent uniqueness constraint.

CREATE TABLE window_aggregates (
  window_key TEXT NOT NULL,
  window_start TIMESTAMPTZ NOT NULL,
  window_end TIMESTAMPTZ NOT NULL,
  value NUMERIC NOT NULL,
  emitted_at TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (window_key, window_start, emitted_at)
);

The original emit goes in with the time the window closed. Each correction goes in with a later emitted_at. The table holds the entire history.

The current-view layer

Downstream consumers should never query the raw table. They query a view that selects the latest version per window key.

CREATE VIEW window_aggregates_current AS
SELECT window_key, window_start, window_end, value
FROM (
  SELECT
    window_key,
    window_start,
    window_end,
    value,
    emitted_at,
    ROW_NUMBER() OVER (
      PARTITION BY window_key, window_start
      ORDER BY emitted_at DESC
    ) AS rn
  FROM window_aggregates
) ranked
WHERE rn = 1;

The view is the contract with downstream. Anything that needs "the current truth" reads from the view. Anything that needs historical reconstruction reads from the table.

The emit side

The streaming pipeline emits the original aggregate with emitted_at = window_end + grace_period. The correction job emits corrected aggregates with emitted_at = correction_job_start_time.

def emit_correction(window_key, window_start, corrected_value, batch_id):
    insert(
        window_aggregates,
        {
            "window_key": window_key,
            "window_start": window_start,
            "window_end": window_start + window_duration,
            "value": corrected_value,
            "emitted_at": batch_id,
        },
    )

batch_id is a deterministic timestamp tied to the correction batch (typically the cron schedule's wall-clock time, or the timestamp of the latest event in the batch). This makes re-runs idempotent: the same batch produces the same (window_key, window_start, emitted_at) and collides cleanly on insert.

Idempotent inserts

For PostgreSQL, INSERT ... ON CONFLICT DO UPDATE makes re-runs safe:

INSERT INTO window_aggregates (window_key, window_start, window_end, value, emitted_at)
VALUES (...)
ON CONFLICT (window_key, window_start, emitted_at)
DO UPDATE SET value = EXCLUDED.value, window_end = EXCLUDED.window_end;

For BigQuery or Snowflake, MERGE is the equivalent:

MERGE INTO window_aggregates t
USING corrections_batch c
ON t.window_key = c.window_key
   AND t.window_start = c.window_start
   AND t.emitted_at = c.emitted_at
WHEN MATCHED THEN UPDATE SET t.value = c.value, t.window_end = c.window_end
WHEN NOT MATCHED THEN INSERT (window_key, window_start, window_end, value, emitted_at)
                     VALUES (c.window_key, c.window_start, c.window_end, c.value, c.emitted_at);

Either pattern lets the correction job crash and restart without producing duplicate rows.

Compaction strategy

The versioned-fact table grows linearly with the number of corrections per window. After enough time, the table is mostly historical versions and the view is doing a lot of work to find the latest per window.

The compaction strategy: on a quarterly schedule, run a job that copies the latest version per window (the view's output) into a new table, deletes everything from the original, and writes the latest versions back. This collapses the table to "latest only" while preserving the schema.

For pipelines that need audit history, keep the pre-compaction table as a snapshot in cheap storage (object store, archive table). For pipelines that only care about the current value, drop it after compaction.

When the view stops being enough

For very high-cardinality tables, the window function in the view becomes a hot spot. Two mitigations:

The first is to materialize the view as a table that the correction job updates instead of computing on read. The downside is the materialization layer adds latency and complexity. The upside is read performance is fixed.

The second is to use a database feature like PostgreSQL's DISTINCT ON, which the planner can optimize better than ROW_NUMBER() in some cases. The query becomes:

SELECT DISTINCT ON (window_key, window_start)
  window_key, window_start, window_end, value
FROM window_aggregates
ORDER BY window_key, window_start, emitted_at DESC;

For tables in the hundreds of millions of rows, the DISTINCT ON pattern is often two to three times faster than the row-number version, depending on indexes.

Indexes

The primary key alone is usually enough for the OLTP write path. For the read path through the view, two indexes help:

A composite index on (window_key, window_start, emitted_at DESC) makes the latest-per-window lookup almost free.

A separate index on emitted_at makes the correction job's "find corrections newer than my last run" query fast without scanning the whole table.

Validation

After the pattern is in place, validate two things every day.

The reconciliation against the source system should show near-zero variance after the correction window has closed (typically the day after). If the variance is non-zero after corrections have had time to land, the late-event handling is incomplete and you have a producer regression or a side-output bug.

The compaction job's row counts should match the view's row counts at the moment of compaction. A mismatch means the table is missing windows or has phantom rows. Either is a bug in the emit logic.

What to avoid

Mutating the aggregate row in place. This makes the dashboard correct but destroys the audit trail and breaks idempotency for the correction job.

Using natural keys (window start time alone) as the primary key. This forces UPSERT semantics on every emit, which is harder to debug than INSERT with conflict handling.

Treating the latest-version view as a transient construct. It is the contract with downstream consumers; document it, version it, and treat changes to it like API changes.

How this fits into the larger pattern

This pattern is one piece of a larger late-data-handling design. The other pieces are watermarks, grace periods, side outputs, and reconciliation against the source system. All five together produce a streaming pipeline that downstream consumers can rely on.

The end-to-end design is covered in How to Handle Late-Arriving Data in a Streaming Integration Pipeline Without Corrupting Downstream Reports. The versioned-fact pattern is the last mile before the downstream dashboard; the article walks through everything upstream of it.

When to ask for help

If you are retrofitting this pattern onto an existing pipeline with downstream consumers already in production, the migration plan matters more than the implementation. You usually want to deploy the view alongside the old table, gradually move downstream consumers to the view, validate against historical reports, then turn on the correction job. Skipping the gradual migration is how you accidentally republish incorrect numbers to people who were not expecting them.

137Foundry's data integration service covers this kind of retrofit work, including the migration playbook and the reconciliation reporting that proves the change did not regress anything. The Apache Flink docs and the Iceberg spec are the canonical references for the underlying engine and storage-layer details. The Prometheus docs cover the metrics layer that makes the whole thing operationally observable.

Closing thought

The versioned-fact pattern is more bookkeeping than algorithm. The algorithmic insight is "do not mutate, append with a version." Everything else is execution detail. The pattern's value is that it lets the streaming pipeline correct itself without breaking the contract with every dashboard, BI tool, and report that already consumes its output. That contract is what makes the pipeline trustworthy.