Load-testing a production app on Azure: how I ran it, measured it, and what it taught me

#loadtesting #testing #jmeter

A field note on proving a freshly-migrated web app could take real traffic — the plan, the tooling, the numbers, and the one result that almost fooled me.

Why I did this

We'd just moved a line-of-business web application into production on Azure — an Angular front end, a set of .NET 8 APIs, an Azure SQL Managed Instance behind them, and a separate integration API that a downstream system hammers in batches. Before anyone signed off on "yes, this is production-grade," I wanted evidence, not vibes. Specifically:

Does the backend stay fast as real users pile on?
Does the database cope, or does it become the bottleneck?
Can the integration API absorb a large batch run without falling over?

Load testing is how you answer those with numbers instead of hope.

How I directed it

I treated the app as three separate things, because they fail in different ways and you learn nothing by blending them:

The interactive app — a recorded user journey replayed by more and more virtual users. I ramped it in steps: 50 → 75 → 100 concurrent users. Steps matter. A single "throw 100 users at it" run tells you pass/fail; a ramp shows you the shape of degradation, which is where the insight lives.
The integration API — driven as a volume test, not a concurrency test: 500 → 1,000 → 5,000 → 10,000 → 50,000 → 100,000 calls at a fixed concurrency. The question here isn't "how fast is one call" but "does it stay correct and stable across a huge batch."
The database — not driven directly, but observed during both of the above.

The golden rule I set for myself: decide the pass/fail line before running anything. A test with no pre-agreed threshold produces data but no decision. Microsoft's own Well-Architected guidance says the same — define clear pass/fail criteria on latency, error rate and throughput up front, or a run just proves "it didn't crash," not "it performs."

What I used

Nothing exotic — all standard kit:

Apache JMeter — to generate load and produce the HTML dashboards (per-request percentiles, error counts, throughput).
Azure Application Insights — the Failures, Performance, Application Map and Live Metrics blades, one instance per service, to see server-side latency, dependencies and error codes.
Log Analytics + KQL — to query across all services at once and slice failures by service, endpoint and response code.
SQL MI dynamic management views (DMVs) — sampled every 5 seconds into a spreadsheet, to watch active requests, sessions, blocking and CPU while the load ran.
A spreadsheet — genuinely, for the DMV export and quick pivots. Not everything needs a dashboard.

How I measured

A few principles did most of the work:

Percentiles, never averages. An average hides the users having a bad time. The industry standard is p95 (95% of requests are faster than this) with p99 for the unlucky tail. I reported p50 / p90 / p95 / p99 / max for every step. This isn't my preference — it's the consensus across Google's SRE practice and every serious performance guide: averages "obscure the long tail."

Isolate the backend. My first pass mixed in the front-end static assets (a big JavaScript bundle, login redirects, telemetry beacons). Those dominated the worst-case numbers and made the API look terrible. Stripping everything except the actual API calls gave the honest picture — the app's own latency, not the browser's download time.

Watch the layers together. The JMeter numbers, the App Insights traces and the DMV samples all covered the same clock. When something spiked, I could ask "was it the app, a dependency, or the database?" and actually answer it.

The results, in short: the backend median held flat (~180 ms) as users tripled, p95 landed around 0.7 s, worst case ~3 s. The database never blocked a single request — it sat at 2–5 active requests the whole time. The integration API processed 100,000 calls at ~99.5% success.

The result that almost fooled me

The raw error rate came back at ~18–22%. That looks like a failing app. It wasn't.

Every failure was an HTTP 401 Unauthorized — the API correctly refusing a request that arrived with no login token. The tell: I broke the errors down by test step and found the recorded script had 71 API calls — 38 always succeeded, 33 always failed, and zero were "sometimes." If it were a load or capacity problem, the same call would sometimes pass and sometimes fail. Instead the outcome was fixed per call, because 33 calls had been recorded without the auth header attached. Even a single user got the exact same 401s.

The lesson: authentication over HTTP is per-request, not per-session. Every call carries its own token or it gets refused — and a "high error rate" can be entirely a test-harness artifact. Always find out what the errors are before you panic about how many. The fix was one line of test config (attach the token at the top so every call inherits it), not an app change.

How the numbers stack up (external benchmarks)

It's easy to declare victory against your own expectations. Better to check against what the wider field considers "good":

Metric	My result	External benchmark	Read
Median (p50)	~180 ms	< 200 ms "feels instant" (Nielsen / HCI research)	Excellent
p95	~0.7 s	< 500 ms is a strong public-API target; < 800 ms is a common load-test pass gate; 1 s is where users notice (Nielsen)	Acceptable — a touch above "great," well under "bad"
Worst case	~3 s	> 3 s is where a real share of users abandon	At the edge — worth trimming
Real error rate (after test fix)	~0%	SLOs typically want < 1%, high-traffic sites < 0.1%	Comfortable
Integration success	99.0–99.5%	99% is a common floor; many teams target 99.9%	Good, not yet "three nines"

So: the backend is genuinely fast for typical requests, the tail is fine but has room, and the integration path clears the usual bar. Grounding it in outside numbers turns "looks good to me" into "meets recognised targets," which is what a sign-off actually needs.

What I covered vs. what's still missing

Being honest about the gaps is half the value of a report.

Covered

✅ Backend latency under a stepped concurrency ramp
✅ Percentile-based measurement (p50–p99), not averages
✅ Database behaviour under load (concurrency, blocking, CPU)
✅ Integration/batch throughput at volume (to 100k)
✅ End-to-end trace correlation across services
✅ Failure root-cause isolated (auth, not capacity)
✅ Baseline metric alerts on one service

Still missing / next

⬜ A clean re-run after fixing the test auth — to capture the true error rate
⬜ Soak / endurance test (hours at steady load) — catches memory leaks and token-expiry over long runs
⬜ Spike test (sudden surge, not a gentle ramp)
⬜ Failover / HA test — the app currently runs on a single instance with no load balancing; killing an instance is untested
⬜ Autoscale validation — once the plan moves to a tier that supports scale-out
⬜ Front-end / CDN performance — deliberately excluded here, but real users feel it
⬜ Alerts on every service, not just one
⬜ DR drill — replica failover and point-in-time restore, actually exercised
⬜ Production-realistic data and think-times — the recorded script had duplication artifacts

Transferable learnings

Decide the pass/fail threshold before you run. Otherwise you generate data, not decisions.
Ramp, don't dump. Steps reveal the degradation curve; a single big run only says crashed/didn't.
Report percentiles. The average is the metric that lies.
Isolate the layer you're judging. Front-end noise made my API look 100× worse than it was.
Investigate errors before counting them. A scary error rate was a one-line test-config bug.
A test that stays green forever is suspicious. Watch for the opposite too — mine was red for the wrong reason.
Single instance = untested failover. "It held" and "it's resilient" are different claims.