A betting platform our team has worked on since 2018 expected to hold 5,000+ concurrent users. The first honest measurement put the ceiling at 717. Not a slow endpoint, not a p99 outlier — 717 was where the platform stopped. No amount of functional coverage would have said so: the suite was green, the flows worked, and the number nobody had measured was the one that mattered.
Finding it took a dedicated test farm — 200+ machines (200 c5d.18xlarge nodes, 72 cores and 144 GB RAM each) driving 5,000+ threads, with Gatling generating load and Grafana reading the system underneath. That run cannot live in a pipeline. It takes hours to provision, it costs real money per execution, and its whole purpose is to break the system.
And here is the part teams get wrong: the run that belongs in a pipeline would never have found the 717. These are two different tests. They share a tool, a vocabulary and nothing else. Capacity discovery answers where does this break. Performance regression answers did this commit make it worse. Trying to put the first inside CI produces a job too slow and too noisy to gate anything, which is why most teams end up muting it — and a muted performance job is worse than no performance job, because it looks like coverage.
Two tests that share a tool and nothing else
Capacity discovery
Discovery needs realism, and realism is expensive. On the betting platform we built a dynamic test-data suite sized to the browser count of the Kubernetes cluster, so the data volume scaled with the generator instead of becoming the bottleneck. Testing against production meant tuning request timeouts specifically to avoid tripping the platform's own DDoS protection — a detail that only exists because the target was real infrastructure. The engagement identified 20+ critical performance issues alongside 50+ critical/major functional ones, and the capacity figure was eventually validated at 10,000+ concurrent users. None of that came from a pipeline job. It came from dedicated infrastructure, run deliberately, a few times.
Discovery runs when something changes structurally: a new region, a schema migration, a caching layer, a traffic forecast the business wants to trust. Monthly at most. Quarterly for many teams. On demand, always.
Performance regression
Regression needs the opposite. It needs stability, not realism — the same scenarios, the same data volume, the same environment, the same load, every run, so that a difference in the numbers means a difference in the code. It does not need to find the ceiling. It needs to notice that a p95 that sat at 240 ms for three weeks is now 610 ms, and to stop the merge that did it.
Two shapes from projects our engineers work on show what that looks like in practice. The Mortgage Office runs a suite of 10+ load scenarios integrated into its CI pipelines, with k6 for load and Playwright for functional automation on Azure Pipelines. A childcare management platform — after, in the client's words, "a few failed attempts to create a load testing framework" — got a scalable k6 framework built from scratch: 20+ end-to-end load tests wired both into CI and into the monitoring stack (Grafana, New Relic).
One caveat before the how-to sections: these case studies document setups, not the decision logic behind them. The split above is my model for reasoning about them, evidenced by the fact that the two shapes are so obviously different — 200+ machines on one side, a k6 job in Azure Pipelines on the other.
Picking the scenarios: four to ten, not fifty
The instinct is to rank endpoints by traffic share and test the top ten. That gives you a suite dominated by read-heavy endpoints that hit cache, which are the worst canaries in the system — they stay fast right up until nothing else does.
Pick by write path, fan-out and business cost instead:
- Write paths first. Anything that takes a lock, opens a transaction, or writes to a queue. These are where connection pools saturate and where an added index or an N+1 shows up as latency rather than as a failing assertion.
- Fan-out next. A request that triggers five downstream calls degrades non-linearly. The cryptocurrency wallet platform our engineers work on scoped its ~20 JMeter load tests specifically at the RPC proxies covering ~20 chains — the dependency most likely to degrade and the one least under the team's control.
- Business cost, not request count. Checkout, bet placement, payment authorization. A 300 ms regression there is worth more attention than a 2-second regression on an admin report.
Two mechanical rules that save a lot of debugging later. Include the login/auth path exactly once — it is a shared dependency and it will otherwise dominate every scenario's numbers. And keep the request mix of a real session rather than hammering one endpoint: Datasport built its load tests around 4 main user workflows containing 60+ different requests, which is the right shape — few journeys, many requests each, in realistic order.
For scale: The Mortgage Office's 10+ scenarios, the childcare platform's 20+, the wallet platform's ~20. That is observed range, not a recommended count, and none of these sources publishes its selection criteria — the criteria above are mine.
What "load" means inside the pipeline
Here is the assumption worth killing: a CI load test should simulate realistic peak traffic. It should not. Peak traffic is a discovery question. In the pipeline you want the smallest constant load that produces a stable signal.
Constant is the operative word, and it is where most VU-based scripts quietly lie. In a closed model, a new iteration starts only when the previous one finishes, so when the system slows down, your load goes down with it. k6's own documentation puts numbers on it: 1 VU making 6-second requests over 60 seconds completes about 10 iterations; if those requests take 10 seconds, you get 6. The system degraded and your test politely backed off. Gatling's docs are blunter — "if you're using a closed workload model in your load tests while your system actually is an open one, your test is broken."
So use a fixed arrival rate where the tool supports it. In k6 that is the constant-arrival-rate executor:
export const options = {
scenarios: {
checkout_regression: {
executor: 'constant-arrival-rate',
rate: 20, // 20 iterations started per timeUnit,
timeUnit: '1s', // regardless of how slow the system gets
duration: '90s',
preAllocatedVUs: 40, // too low and k6 can't hold the rate
maxVUs: 120,
},
},
};
In JMeter, the equivalent lever is a throughput timer pacing samples to a target requests-per-minute rather than letting thread count define the load.
The rest of the load model is budget arithmetic:
- Short ramp, short steady state, no soak. 15–30 seconds of ramp, 60–120 seconds of measurement. Soak tests belong to discovery.
- Budget against the rest of the pipeline. If your PR checks run in 8 minutes, a 3-minute performance job is a tax people will vote to remove.
- Identical data volume every run. A table that grew 40% since last week will move your p95 on its own, and you will spend a day blaming a commit.
- Don't generate load from the CI runner. GitHub's standard hosted Ubuntu runner gives 2 CPUs and 8 GB RAMon private repos; Microsoft-hosted Azure Pipelines agents are 2 cores and 7 GB. Past a modest rate you are measuring your generator.
For contrast on where "realistic" load actually sits: the childcare platform's k6 work stands at roughly 1,000 concurrent users in distributed execution. That is a discovery-scale figure. It is not a per-commit figure, and nothing in that case study suggests it was used as one.
The honest counterargument: some regressions only appear under contention. Connection pool exhaustion, lock waits, GC pressure, thread starvation — a 20 rps run will miss all of them. That is a real gap, and it is why the nightly and on-demand tiers below are not optional extras.
Thresholds that can fail a build
This is the whole thing. A load test without an explicit pass/fail rule is a chart nobody opens. The threshold is the deliverable; the script is just how you produce the number.
In k6, thresholds are first-class and the tool exits non-zero when one fails, which is all a pipeline needs:
export const options = {
thresholds: {
// hard fail — blocks the merge
http_req_failed: [{ threshold: 'rate<0.01', abortOnFail: true, delayAbortEval: '15s' }],
'http_req_duration{scenario:checkout_regression}': ['p(95)<450', 'p(99)<1200'],
// warn only — reported, does not gate
'http_req_duration{endpoint:search}': ['p(95)<800'],
},
};
Four rules I hold to, stated plainly as craft rather than case evidence — none of the projects above publishes its thresholds, SLOs or performance budgets, so anyone quoting a number here would be making it up:
Assert on percentiles and error rate. Never on the mean. An average hides exactly the tail you are hunting; Gil Tene's How NOT to Measure Latency is still the clearest explanation of why. p95 for the routine gate, p99 for the hard one.
Set the first thresholds from your measured baseline plus headroom, not from an SLO nobody has verified. Run the suite ten times on an unchanged main branch, take the worst p95, add 15–25%. That number is defensible because it came from your system. An aspirational 200 ms lifted from a blog post will fail on day one and be deleted on day two.
Separate hard fail from warn. Error-rate breaches and p99 blowouts stop the build. A 10% p95 drift on a secondary endpoint gets reported and looked at. If everything is a hard fail, the job gets bypassed within a month.
Version thresholds next to the tests, in the repo. A threshold change should show up in a diff and get reviewed like any other change. Thresholds stored in a dashboard UI get quietly relaxed by whoever is on call at 2 a.m.
For comparing against a baseline rather than a static number, export the summary and diff it in the job:
k6 run --summary-export=current.json checkout.js
jq -e --slurpfile base baseline.json '
.metrics.http_req_duration["p(95)"] <= ($base[0].metrics.http_req_duration["p(95)"] * 1.15)
' current.json || { echo "p95 regressed >15% vs baseline"; exit 1; }
Tooling is not the interesting variable here — k6 shows up in the CI and release flows at The Mortgage Office, QIMA and the childcare platform; JMeter at the wallet platform; Gatling at the betting platform; Datasport uses both k6 and Apache JMeter. Teams run several generators for different jobs, and no case study attributes an outcome to the choice.
Where it runs, and how often
Three tiers, and the placement matters more than the scripts:
Per-commit: the two or three cheapest write-path scenarios, under three minutes total, gated on hard thresholds only.
Nightly or per-release: the full set. QIMA's k6-based autotests were integrated into the release flow, not into every commit — for a suite of that size, that is the correct placement. (Release testing there went from 3–4 days to 1–2 days, but that engagement rebuilt functional automation at the same time — ~1,000 new automated scripts, ~90% API-generated test data, a 2.5x execution-time reduction. The release-time figure belongs to the whole program, not to the performance suite.)
On demand: discovery, on dedicated infrastructure, with a human watching.
Two placement constraints that decide whether any of this works:
A dedicated environment, or the numbers are noise. On shared staging, every run competes with someone's migration, someone's nightly job, someone's manual exploration. You will chase phantom regressions and then stop trusting the gate. If your team does not have an environment it can hold still for three minutes, fix that before writing your first threshold — the threshold is worthless without it.
Results next to infrastructure metrics. A p95 with no CPU, memory, pool-utilization or GC data beside it tells you something got slower and nothing about why. The childcare platform's load tests are integrated with Grafana and New Relic for exactly this reason; the betting platform paired Gatling with Grafana. SoftNAS is the other useful data point — a performance suite added to a CI process that previously had no performance testing at all. (Reported alongside 45% faster delivery of automation results, though that engagement also built 1,000+ automated tests to 80% coverage, so the number is not attributable to the performance suite either.)
What this doesn't buy you
Be precise about the scope of the signal: a CI load test catches relative degradation, on the paths you chose, in an environment that is not production. That is genuinely valuable and it is also all it is.
- False confidence is the main risk. A green pipeline says none of your 4–10 scenarios got measurably slower at 20 rps. It says nothing about capacity, about the paths you didn't pick, or about behavior under contention.
- Data setup is the real maintenance cost. Not the scripts. Keeping seed data identical run over run, across schema changes, is where the hours go — and it is why the childcare platform needed a framework built from scratch after earlier attempts failed.
- Flaky performance jobs get muted faster than flaky functional ones, because everyone already half-suspects the numbers. Google's testing team reported about 1.5% of all test runs flaky and ~16% of tests with some flakiness — dated 2016, and functional rather than performance, but performance jobs are strictly more sensitive to environment noise than functional ones. Budget for the environment or expect the mute.
- It will not find your 717. The betting platform's 20+ critical performance issues and its real capacity ceiling came from dedicated distributed infrastructure. A pipeline job was never going to surface either.
One more thing worth saying out loud: every engagement referenced here is multi-year with dedicated engineers — Datasport since 2020 with 3 QA engineers, the childcare platform since 2021 with 6, the betting platform since 2018 with 2 senior automation engineers. Load testing in CI is not a Friday afternoon setup task that then runs itself.
The smallest version that works
Pick two write-path scenarios. Run them at a fixed arrival rate against an environment nobody else is touching. Assert p95 and error rate against last week's measured baseline plus headroom. Let it fail the build. That is a working performance gate, and it fits in an afternoon once the environment exists.
Then be explicit about what still has to happen off the pipeline: capacity discovery on real infrastructure before every structural change, soak tests where memory and connection leaks live, and a nightly run of the full scenario set to catch the contention-only regressions the small run misses.
And if you don't have an environment you can hold still — fix that first. Everything above is downstream of it.
Top comments (0)