Somebody on r/sre asked the question out loud this week: “Our staging load tests do not match production. How do I fix this?” The replies all said the same thing in different words: they never match. Nobody had a fix, because a match is the wrong thing to chase. Here is what to chase instead.
First, the scene. It is invented — no names, no real incident — but you have lived a version of it. A team load-tests its new release in staging on Tuesday. A load test means sending simulated traffic at a system to see how much it can carry before it slows down or falls over. Tuesday’s results: latency low and flat, errors zero, servers loafing. Green across the board. Friday, the same build launches. Real users arrive, latency climbs like a lift with the cable cut, errors bloom, and somebody rolls the release back while the launch announcement is still warm. Same code. Same test. Different planet.
The test did not malfunction. It answered the question it was asked — “how does this code behave in staging?” — and the team heard a different answer: “how will this code behave in production?” The gap between those questions has a shape, and it is made of three lies.
Lie one: your data has the wrong shape
When engineers say data shape, they mean more than how much data you have — they mean how it is spread. Staging data is usually generated on purpose: fresh rows, similar sizes, reads and writes sprinkled evenly across all of them, like practice questions in a textbook.
Production data has years of weather in it. A tiny share of keys — one celebrity product, one enterprise customer, one cursed account — takes a giant share of the traffic. This unevenness is called skew. Picture a supermarket with twenty checkouts open and every customer queuing at the same two, because those two have the chocolate at the till. The other eighteen are not slow. They are irrelevant.
Skew changes how the database behaves underneath your code. Hot rows sit in memory and get hammered; the cold majority sits on disk. Indexes — the database’s lookup shortcuts — and query plans, the route the database picks for each question, are chosen against the real distribution, not your tidy one. A query that flies over ten thousand uniform rows can crawl over millions of skewed ones, queuing behind everyone else’s requests for the same hot rows. Your staging test never met those rows. It timed the motorway at 5 AM and reported it as the road. Production drives it at 6 PM on a festival weekend.
Lie two: your script only visits the happy path
A load-test script is a polite guest. It calls the five endpoints somebody remembered, in a fixed, sensible mix, with well-formed requests that mostly succeed. Real traffic is the whole extended family arriving at once: the long tail of rarely used endpoints, the legacy URL nobody dares delete, the mobile app from three versions ago — plus bots, plus impatient humans hammering refresh. Worst of all, real traffic retries. When responses slow down, clients try again, apps re-fire, users tap twice. Load multiplies at exactly the moment the system is least able to carry it. Your polite script, meeting a slow response, usually just waits — and reports the wait as calm.
Temperature is the other quiet liar here. A cache is fast memory that keeps hot data close; a connection pool is a set of ready-made database connections kept open so requests do not pay setup costs each time. A staging run warms both up as it goes — by minute ten, everything it measures is pre-heated. A production launch starts cold: empty caches, empty pools, every first request paying full price. The test measured the restaurant at steady service. The launch is the doors opening.
There is a trap inside the tool itself, too. Most scripts are written as a fixed crowd of virtual users, each sending the next request only after the previous one comes back. Load testers call this a closed model — and its dirty secret is that when the system slows, the crowd politely slows with it, sending less load exactly when you wanted more pressure. The honest alternative is an open model: an arrival rate fixed no matter how slow responses get. Real users do not stop arriving because your service got slow. In k6, the open model is the constant-arrival-rate executor:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
steady_load: {
executor: 'constant-arrival-rate',
rate: 300, // iterations started per second...
timeUnit: '1s', // ...no matter how slow the system gets
duration: '10m',
preAllocatedVUs: 100,
maxVUs: 300,
},
},
thresholds: {
http_req_duration: ['p(95)<500', 'p(99)<1000'],
http_req_failed: ['rate<0.01'],
dropped_iterations: ['count<10'],
},
};
export default function () {
const res = http.get('https://staging.example.com/api/products');
check(res, { 'status is 200': (r) => r.status === 200 });
}
Read the thresholds as the test’s promise. p95 under 500 ms means 95 of 100 requests must finish within half a second; failures stay under 1%. And the honest one: dropped_iterations near zero proves the generator itself kept up — the load you aimed for is the load that arrived.
Lie three: a scale model is not the building
Staging is smaller. Two replicas where production runs twenty — the numbers are illustrative, pick your own disappointment. Scale changes behaviour, not just capacity. Queues form differently when twenty small servers share the work than when two big ones do. Garbage collection — the runtime’s housekeeping, which pauses the program to discard dead objects — pauses differently at different memory sizes and work rates. And production machines are shared: other services, backup jobs and log shippers quietly eat CPU and disk. These are the noisy neighbours. Staging sits alone in a quiet room and reports the silence as headroom.
Then there are the third parties. Payment gateways and outside APIs have rate limits — quotas on how many calls you may make. Staging never sends enough to trip one. Production trips it at the worst possible minute, and the provider’s slowness becomes your latency and your errors. Add config drift: a feature flag on in staging and off in production means you load-tested a building with a different floor plan. Side by side, the two runs share almost nothing:
Data — Staging: small, fresh, evenly spread. Production: years deep, with a few hot keys carrying most of the load.
Traffic — Staging: a tidy script cycling the same few endpoints. Production: long tail, retries, bots, refresh-hammering.
Temperature — Staging: warm, heated by the test itself. Production at launch: cold caches, empty connection pools.
Topology — Staging: a handful of replicas on quiet machines. Production: the full fleet on shared machines, neighbours included.
Third parties — Staging: quotas never approached. Production: quotas armed and waiting.
Config — Staging: flags as the test team left them. Production: flags as the launch left them.
What staging can honestly tell you
Do not throw the test away in a huff. Used honestly, it answers three questions very well:
Per-request cost. Does this build make each request cheaper or costlier than the last — in CPU, memory, database queries? That comparison survives the environment gap, because both builds suffered the same staging.
Where each component saturates. Saturation means how full your tightest resource is. Push one tier at a time — web, app, database, cache — and note which tops out first. The order of bottlenecks transfers to production even when the numbers do not.
Relative comparisons. Build A against build B, same test, same staging: “B is slower per request” or “B halves the error rate” are claims you can defend in a review.
What it can never hand you is the absolute number. “Staging held 800 requests per second” does not mean production holds 800, or 400, or 1,200. A scaled-down, warmed-up, quiet-room environment cannot issue production capacity figures. A launch plan built on one is fiction with a chart attached.
Staging tells you the shape of the cliff. Production tells you where the cliff is.
Make the test honest
Feed it production-shaped data. Take a real subset of the production database — the most recent slice, say — keep the skew exactly as it is, and scramble only the personal fields: names, emails, anything identifying a human. The hot keys stay hot, the fat rows stay fat, and the database finally behaves like itself.
Replay real traffic. Record a slice of actual production requests and play them at staging — ideally as shadowing, where live requests are copied to staging in parallel and staging’s answers are thrown away. Users never see those responses, but staging feels the real mix: the long tail, the retries, the weird payloads. Scripts guess at user behaviour. Replays remember it.
Scale the bottleneck tier, not the world. Cloning all of production is how budgets die. If saturation tests keep fingering the database tier, run that one tier at production size and keep the rest small — on purpose, written down, so nobody mistakes it for a full-scale run.
Change the shapes. A gentle ramp hides two failures: the slow rot that appears only after hours under load (a soak test — hold steady load for hours, watch for leaks and creeping pauses) and the sudden burst (a spike test — jump to peak in seconds, the way a launch actually arrives). In k6, a spike is a ramping arrival rate with honest stages:
export const options = {
scenarios: {
spike: {
executor: 'ramping-arrival-rate',
startRate: 10,
timeUnit: '1s',
preAllocatedVUs: 50,
maxVUs: 400,
stages: [
{ target: 10, duration: '2m' }, // a normal morning
{ target: 600, duration: '30s' }, // the launch post lands
{ target: 600, duration: '3m' }, // hold the spike
{ target: 10, duration: '1m' }, // back down again
],
},
},
};
Measure the same four signals in both places. Google’s SRE book boils service health down to four golden signals: latency (how long requests take), traffic (how much demand is arriving), errors (how much of it is failing) and saturation (how full the tightest resource is). Capture those four, on the same dashboards, in staging and production alike. The numbers will differ; the shapes of the curves become comparable — and shapes are where the truth lives.
Keep a correction factor. After every launch, divide what production actually did by what staging predicted, and keep the ratio per tier. Next launch, staging’s numbers get multiplied by the factor before anyone quotes them. The test stops being a liar and becomes an instrument — calibrated against reality, like any honest instrument.
Pin the whole discipline in one small file next to the test, and review it like code:
# load-test-profile.yaml - the honesty settings for this test
environment:
data: production-subset # skew preserved, personal fields scrambled
traffic_mix: replay-capture # real requests from a busy day, not hand-written scripts
start_state: cold # caches cleared, pools empty - launch conditions
shapes:
- steady: 30m at expected peak
- soak: 4h at 70% of peak # slow rot only shows up late
- spike: 10x in 30s # the launch moment, not a gentle ramp
measure: [latency, traffic, errors, saturation] # same four signals as production
after_launch:
record: staging number vs production number
update: correction factor per tier
Staging warning: replay copies of production traffic only into staging, and never point a spike test at production or a shared environment without warning everyone who shares it first. A load test aimed at the wrong target is just a self-inflicted outage with unusually good documentation.
The make-it-honest checklist
Data carries production’s skew — hot keys stay hot, fat rows stay fat; only personal fields are scrambled.
Traffic mix matches the real logs — long tail, error paths, retries and bots are in, not just the top five endpoints.
The test starts cold — caches cleared, connection pools empty — or the report says plainly that it did not.
Load is arrival-rate based — when the system slows, the test keeps arriving instead of politely backing off.
Shapes include a soak and a spike — hours at steady load, and a jump to peak in seconds. Not only a gentle ramp.
The bottleneck tier runs at production scale — the rest may stay small, on purpose, with the choice written down.
Third parties are stubbed with their real rate limits — or the plan states what happens when the quota trips.
Feature flags and config are diffed against production before the run, not discovered after it.
The same four golden signals are captured in both environments, on the same dashboards.
Last launch’s correction factor is applied to this run’s numbers before anyone quotes them in the launch review.
The honest sentence
So, back to that r/sre thread. The fix is not a bigger staging or a cleverer script. It is a change of job description. Stop asking the test for a verdict — “will it hold?” — and start asking it for evidence: what does each request cost, which tier saturates first, is this build better or worse than the last, and by what factor has staging been wrong before?
Say the honest sentence in the launch review, out loud, before anyone quotes a staging number as a promise: staging tells you the shape of the cliff; production tells you where the cliff is. Plan for the cliff being closer than the test says — because it is — and let every launch teach the test a little more truth.
Disclaimer
This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.
© 2026 Honey Kiran. All rights reserved.
Honey Kiran · DevOps & Platform Engineering
DevOps, SRE, Terraform, and platform engineering — explained the practical way.
What is your staging-to-production correction factor — have you ever measured it, or is your team still quoting staging numbers raw? Tell me in the comments; the honest factors are the useful ones.
Examples are illustrative — validate in your own environment before running in production.
Top comments (0)