Most of us have heard the term performance testing, whether before releasing a big new feature, launching a new application or simply checking whet...
For further actions, you may consider blocking this person and/or reporting abuse
I usually follow the 80/20 rule: 80% of traffic arrives within 20% of the time window. I use this to design my load test scenarios. Writing the test code is the easiest part. The most annoying work is resource coordination before the load test and root-cause analysis afterward — that stuff is brutal 😂
Interesting, I usually approach it more from the business side, so I calculate the expected traffic and time window based on things like the number of orders in e-commerce or the expected number of operations within a specific period.
And yes, I agree, writing the tests is usually the easiest part. 😄
Btw, what do you use for creating and executing the tests? Most of the time, I’ve been pretty satisfied with JMeter.
Back in the day I used LoadRunner — it was so heavy and costly. Then I moved to JMeter. Once AI took off, I started having it write Python load test scripts with Locust, and it works surprisingly well.
Hmm, sounds interesting. I’ll check it out.
One benefit of JMeter is that testers don’t really need to know how to program, since most of it is GUI-based. But you’re right in the AI age it can feel a bit heavy.
Last time I had to edit something, I used Copilot on our ~35k-line JMeter XML file and it ate through a lot of tokens. I actually had to change my Copilot settings so it wouldn’t burn all my monthly credits. 😅
JMeter’s XML is its biggest flaw, lol 😂
Yeah, fair point. 😄 To be honest, I never really needed to touch the raw JMeter XML before AI, so it never bothered me much.
It’s not completely unreadable, it just gets insanely verbose once the test grows, which becomes a bit of a bottleneck in the AI age 🤣
Thanks, this was genuinely interesting! I especially liked the point about making tests reflect real user behavior instead of just throwing a huge number of requests at the API.
At work, we've generated so many tests with AI across different projects that it's honestly getting a little scary 😄 I wouldn't be surprised if somewhere there's already a test checking whether the tests themselves can actually run. 😂
Thanks, glad you liked it!
Yeah, realistic user behavior vs. just throwing requests at the API is really important and in my opinion a lot of performance-testing articles don’t highlight it enough.
That sounds funny and scary at the same time. 😂 But now I’m curious with so many AI-generated tests, who is actually responsible for them? Testers or developers? I’m asking because I’m also curious about the code quality and long-term maintainability. From my experience, a lot of testers can vibe-code tests that work initially, but maintaining them later can become a real problem.
I guess the responsible person is whoever still remembers where the tests are 😄
The good news is that every company will probably need at least one person who remembers that. So AI won't replace all of us after all 😂
Now it’s even scarier. 🤣
😂😂😂
Great post about API performance testing! 🤗 First, explaining the basics and later explaining an example test using a user simulation helped me understand it better. Also, the images helped me understand it too. Every developer using APIs should read this post! 💯
Oh, thanks! I really appreciate that! 😊
I was trying to show how to think when designing performance tests, because some of the things I describe in the article I had to learn the hard way. 😅
Ah! Because you experienced it the hard way, now you can avoid the bug beforehand. Nice tackle! 💪
The load generator bottleneck is the one that catches everyone sooner or later. We hit this in a monitoring stack where the "test" ran as a sidecar on the same box as the thing being measured — the generator and the target starved each other for CPU, and the p95 that came out told us more about the scheduler than the service.
The part I'd underline is defining the acceptable result before running the test. We have a watchdog that asserts against concrete SLOs (thresholds, error rate, max consecutive misses with backoff) rather than p95-vs-vibes. When the numbers disagree with the SLO you wrote down beforehand, you actually investigate. When you only have "it seems slow," everything looks slow.
Yeah, running the generator on the same box as the target is a great example of how easily the test itself can influence the result.
And I agree, defining the acceptable result before the test is probably one of the most important parts. Otherwise, you can easily focus on numbers that don’t really matter. It’s a bit like shooting in the dark. 😄
"Shooting in the dark" is exactly the right phrase — the acceptance value is what forces you to say before the run what an acceptable result is. We now keep it in the task contract rather than in the summary, so the number you show at the end has to match the promise you made at the start. And for the generator: we treat "the test box itself got slower" as a valid finding of its own, not an excuse to rerun. If the tooling is drifting, that's a fact about the loop, and it's cheaper to know it early than to discover it in a baseline that suddenly no longer matches.
Exactly. We define what success looks like before the test runs and then compare the actual results against those criteria.
When the load generator itself starts degrading, for example when it can no longer maintain the requested RPS, it becomes a bit more situational. Depending on the circumstances, we may stop and rerun with more resources, or let the test finish and use the run for diagnostics. But once the generator becomes the bottleneck, I wouldn’t treat that run as a reliable performance result for the application itself.
That distinction is the one I'd codify: the load generator gets its own health gate. If achieved RPS sits below ~95% of target for more than a few seconds, the run is void for pass/fail and only useful as a diagnostic — otherwise you're quietly measuring the harness's saturation point instead of the app. Predefined acceptance criteria plus that abort rule together kill most of the shooting-in-the-dark problem.
Great breakdown on designing realistic user journeys rather than just hammering a single
/pingendpoint!One critical addition to the Metrics section: Always avoid reporting Mean (Average) Latency.
Averages hide tail latency outliers completely. If 95 out of 100 requests return in 10ms, but 5 requests hang for 10 seconds (due to GC pauses or DB thread pool exhaustion), your average latency will sit at a misleading ~500ms.
In production microservices:
Also, when calculating throughput vs. response time, watch out for the Coordinated Omission problem (a term coined by Gil Tene). If your test harness blocks on a slow response before sending the next request, your load generator is accidentally hiding the queueing delay that real-world concurrent users would actually experience.
Thanks, I appreciate it! And yes, relying on average latency alone can definitely hide important problems, that’s why I prefer looking at p95/p99 together with the other metrics.
Coordinated Omission is actually a new concept to me. That’s a really interesting problem and something I hadn’t considered before. Thanks for bringing it up! I’ll definitely dig into it more and I might update the article itself.
Daniel, the external services section is where I would add one layer. LLM calls do not have fixed latency like a normal REST call.
A prompt can return in 200ms or take 8 seconds, depending on output length and provider load. A single mock delay hides that variance completely.
I test with a latency distribution instead of one fixed number, so the p99 still means something once an LLM sits behind the endpoint.
That’s a really good point. With LLM calls, simulating one fixed latency would definitely hide a big part of the real behavior. Using a latency distribution makes much more sense, especially when we care about p95/p99. Thanks for adding this.
LLM endpoints would probably deserve their own performance-testing section. I’ve worked with AI endpoints and their unpredictable latency, but I don’t have enough hands-on experience with performance testing them in real production yet, so anything deeper from me would mostly be theory or observations from hobby projects.
Great article — especially the focus on realistic user flows instead of isolated endpoints helped me identify a few gaps in my own approach.
My existing suite already measures p95/p99, throughput, error rates, warm-up, and stepped load. What I’m taking from this article is the need for role-based journey profiles, realistic think time, a mix of different user paths, and load targets derived more clearly from business requirements or production telemetry. I also want to correlate results more closely with resources such as database connections, queues, and worker capacity.
Thank you for this practical perspective — connecting test design, environment, and metrics is what makes performance testing genuinely useful.
Thanks, I really appreciate this!
I’m genuinely happy that the article gave you a few concrete ideas to improve your existing suite. Your current setup already sounds solid, so adding more realistic user journeys should make the results even more useful.
Really like the focus on realistic workflows rather than testing endpoints in isolation. I’m exploring something related with Kaktoos, an open-source tool for API verification and engineering change impact. The workflow-based approach is very much aligned with what I’m building
Thanks! Yeah, the workflow-based approach is really important if you want the results to reflect actual usage.
Really enjoyed this perspective on API performance testing. One thing that stands out is how easy it is to focus on sending more requests and forget that real users don’t interact with APIs as isolated endpoints. They follow journeys, create different workloads, and experience the system as a whole.
The idea of designing tests around realistic user behavior is something more teams should adopt. A test that only measures numbers but ignores actual usage patterns can give a false sense of confidence.
Performance testing is not just about finding the maximum load an API can handle, it’s about understanding how the system behaves under real-world conditions and where improvements will have the biggest impact.
Great article and a good reminder that quality testing starts with understanding users, not just tools and metrics.
Thanks, glad you liked it!
Exactly, the tools and metrics are important, but if the workload doesn’t reflect real user behavior, the test results won’t tell us much about how the system will behave in production.
Excellent guide on realistic API testing! The point about simulating real user behavior instead of just hammering endpoints is crucial. I've found that adding think time between requests and using realistic payload sizes makes a huge difference in getting actionable results. What tools do you recommend for load testing in a CI/CD pipeline?
To be honest, I mostly used JMETER, usually run from a script in the CI/CD pipeline. It works and was sufficient in most cases, so from my own experience that’s still the tool I can recommend and it also integrates well with BlazeMeter.
That said, JMETER does feel a bit heavy nowadays, especially with its verbose XML when AI is involved. 😄
@xulingfeng suggested Python with Locust, which I think could be a cleaner option and should also be easy to run in a pipeline. But I haven’t used Locust yet, so I can’t really recommend it.
The part about external services is where most performance tests fall apart. I ran load tests against 6 APIs last month and 4 of them had mock external dependencies that responded in 5ms — the real third-party calls averaged 340ms. Your test numbers look great until production hits and 60% of your latency is someone else's server. If your test environment doesn't replicate the real latency of external calls, you're not testing performance, you're testing your own optimism.
Yep, agree. That’s actually a really good real-world example of the problem. Thanks for sharing!
Explain the topic in a simple and practical way for beginners. Cover performance testing types, realistic user traffic, load and stress testing, RPS, response time, p95/p99, error rate, external APIs, test environment, and performance metrics.
Use clear headings, short paragraphs, simple examples, and a conclusion. Keep the content original and easy to understand. Do not copy the reference article.
I’m not fully sure I understand this comment. Is this a prompt to summarize or rewrite the article in a simpler way?
A good performance test is really about realism, not just pushing the highest possible request count. Modeling real user behavior, traffic patterns, dependencies, and production-like conditions makes the results much more actionable.
Exactly. The closer the test is to real production behavior, the more useful the results become.
Good rundown, especially the bit about mocking to control cost. One thing worth adding for anyone load testing something that calls out to an LLM API specifically: a WireMock stub with fixed latency will hide the actual shape of the problem. Real LLM API latency isn't a flat round trip, it's a wide distribution with a long tail that gets worse under load on the provider's own side, and that tail is often exactly what blows out your own p99 in production. A fixed-latency mock passes every load test and then the real integration falls over the first time traffic spikes. If the external call matters to the result at all, sampling real latency percentiles into the mock (or at least injecting jitter matching them) gets you a lot closer to the truth than a constant delay does.
Agree, that’s a really good point. Someone in the comments already mentioned something similar.
And yes, a fixed delay can be too simplistic for mocked services in general, not just LLM APIs. If we want the test to reflect real behavior, sampling real latency percentiles or introducing realistic jitter in the mock is a much better approach.
Thanks for pointing it out!
Great article! One aspect I’d add from a distributed systems perspective is that realistic performance testing should also evaluate how the system behaves when its dependencies start degrading.
An API might maintain excellent p95/p99 latency under normal load, but what happens when a downstream service becomes slow, Redis experiences a cache miss spike, or the database connection pool reaches saturation?
In my experience working with .NET and cloud-native architectures, combining load tests with distributed tracing, metrics, and controlled failure scenarios helps reveal bottlenecks that traditional endpoint-level testing often misses.
I’d also pay close attention to retry amplification, circuit breakers, queue backlogs, and recovery time after a traffic spike.
Ultimately, passing a load test is not just about handling the expected RPS. It’s about maintaining predictable behavior and recovering gracefully when the system is under pressure.
Thanks! Yeah, a lot of this is closely related to what I tried to cover with external dependencies, queues/connections, and looking at multiple metrics together.
I like your point about going one step further. I’d summarize it as: don’t only test happy-path dependency behavior under load, test how the system behaves when dependencies start degrading while the load is still there.
Thanks for bringing it up, you’ve given me something to think about.
This is a very practical and valuable guide to API performance testing. I especially liked the focus on simulating realistic user journeys instead of simply sending a large number of requests to individual endpoints. Think time, different user paths, gradual ramp-up, realistic data volume, and production-like cache and database conditions can make a huge difference in the accuracy of test results.
The section about external services is also very important. Mocking dependencies is useful for isolating our own application, but real network latency, rate limits, and third-party failures should be tested separately or simulated when they affect the user experience.
I also agree that average response time alone is not enough. p95/p99 latency, error rate, throughput, resource usage, database connections, and recovery after load should be analyzed together. A CI/CD example with performance thresholds and automatic build failure would be a great follow-up. Great article!
Thanks, I really appreciate it! The suggestion about a CI/CD example is actually a great idea. I’ll add it to my already pretty long TODO list. 😅
An exceptional article that hits the nail on the head! 👏
What caught my attention the most—and what I strongly agree with—is your point about "simulating realistic user behavior instead of randomly blasting endpoints." Many teams fall into the trap of measuring vanity metrics (fake RPS) without replicating actual user journeys, leading to a false sense of security that ends in a production disaster.
Another crucial point you highlighted is "database size and configuration." Far too many test environments run against nearly empty databases, only for real-world performance to crash once they hit millions of records.
Thank you for such a practical, real-world approach far away from abstract theory! 🚀
"This is a stellar guide! 🎯 The emphasis on simulating realistic user workflows (rather than just spamming isolated endpoints) is the most crucial part of performance testing that many developers overlook.Also, watching out for database size, realistic data distribution, and environment parity is so important to avoid misleading metrics. (And amen to the JMeter XML verbosity comment in the thread—keeping test scripts lightweight is a lifesaver for both version control and AI token limits! 😂) Thanks for sharing these hard-earned insights!"
Thanks, really appreciate it. Hope it helps others at least a little bit. 😄
This is the kind of performance-testing article I love because it moves the conversation away from “How much traffic can we throw at the API?” and toward the much more important question: “Will this test tell us anything meaningful about production?”
✨ “The goal is not just to generate a large number of requests, but to create performance tests that actually tell us something useful.”
That sentence should probably be the opening principle for every performance-testing strategy. A huge load against an unrealistic scenario can produce impressive graphs while telling us very little about how real users will experience the system.
✨ “What has worked well for me over time is identifying typical user behavior and simulating it in performance tests.”
This is such an important shift. APIs don't exist in isolation. Users move through workflows. Search leads to product selection, product selection leads to cart activity, cart activity leads to checkout, and checkout leads to payment. Testing those relationships gives us a much more realistic picture of system behavior.
✨ “Real users don't click as fast as a performance test can send requests.”
Obvious when stated, but surprisingly easy to forget.
A load generator can produce traffic at machine speed. Humans cannot. Adding realistic think time and different user journeys is what turns a request generator into a behavioral model.
✨ “Not every user follows the same path.”
This is another detail that can dramatically change the usefulness of a test. Real traffic is heterogeneous. Some users browse. Some purchase. Some abandon carts. Some repeatedly refresh pages. Some trigger entirely different backend workflows.
A single perfectly synchronized journey isn't reality.
✨ “It does not mean that 28 RPS should automatically become our test target.”
I especially appreciate this clarification. The article doesn't treat a calculated number as magical. It recognizes that averages can hide peaks, abandoned flows, background traffic, and other workload sources.
The number is a starting point—not the answer.
✨ “External services require special attention during performance testing.”
Absolutely. A performance test can accidentally become a test of someone else's API limits, pricing model, latency, or availability instead of your own application.
The discussion around mocking versus including external dependencies is particularly useful because it frames the decision around what you're actually trying to measure.
✨ “We should also avoid testing against an almost empty database.”
This is one of those details that separates a synthetic benchmark from a useful performance test.
A query against a tiny dataset can behave beautifully. Add millions of realistic rows, different distributions, indexes, statistics, joins, sorting, caching behavior, and suddenly the performance profile can look completely different.
✨ “The load generator itself.”
This might be the easiest bottleneck to overlook. If the machine generating the traffic runs out of CPU, memory, network capacity, or connections, you've just measured the limitations of your test infrastructure instead of the application.
And I really like the final philosophy:
✨ “The final report should therefore not contain only a single number such as average response time.”
Exactly.
Average latency can look fantastic while a meaningful percentage of users are having a terrible experience. Percentiles, error rate, throughput, resource utilization, queues, connections, and recovery behavior all tell different parts of the story.
The biggest takeaway for me is that realistic performance testing is essentially an exercise in respecting causality.
If the workload isn't realistic, the result isn't trustworthy.
If the environment isn't representative, the bottleneck may be misleading.
If external dependencies aren't controlled, attribution becomes difficult.
If the metrics aren't connected, you know something went wrong but not necessarily why.
The best performance test isn't the one that produces the scariest number.
It's the one that lets you look at the result and confidently say:
“This is what our system is likely to experience—and here's the evidence showing why.”
Oh, that’s probably one of the longest comments I’ve ever got here. 😄 Thanks for taking the time to write it! You basically summarized the whole article in one comment. 😂
“The biggest takeaway for me is that realistic performance testing is essentially an exercise in respecting causality.”
That’s actually an interesting way to put it, I didn’t think about performance testing as “respecting causality,” but it fits really well. 👍️
Easy to blame the API when the load generator runs out of resources first. At that point, you’re testing the machine, not the application.
Yep, agree. Easy thing to overlook when the focus is only on the application.
Very important take always. Thanks
What frameworks are you using for the tests and how is it integrated in you CI/CD?
Example in GitHub actions
Thanks! I actually answered a very similar question in another comment here.
From my own experience, I mostly used JMeter, usually run from a script in the CI/CD pipeline and it also works well with BlazeMeter. Someone here suggested Locust, but I haven’t used it enough yet to recommend it over JMeter.
Thanks
Another question if I may, is the CI CD on resources similar to production? Or just arbitrary GitHub action machine?
This is actually a bit tricky and depends on the load you want to generate, but the CI/CD runner does not need to be similar to production. The system under test should be production-like, the load generator just needs enough resources to generate the intended workload without becoming the bottleneck.
You can recognize that it’s too small when the generator itself starts saturating: CPU stays near 100%, memory pressure increases, the network becomes saturated or you ask for more RPS but the generated throughput no longer increases.
Good reminder that realistic traffic matters more than just chasing benchmark numbers.
Exactly. Benchmark numbers can look great, but they don’t mean much if the traffic pattern itself isn’t realistic.
Interesting read! I find Spike (Peak) tests the most interesting, especially when testing APIs myself.
Thanks, glad you liked it.
Which type of test to use really depends on the application. For example, on one e-commerce app I used load and spike tests, while on a more business-oriented application I used load and volume tests.
We always try to test what actually makes sense for the system.
Nice write-up Daniel!!! :D
Thanks! 😄 Glad you liked it!
Thanks, glad you liked it!