DEV Community: Liudas

Idempotency and Concurrency in APIs: API Testing Rule 7/10

Liudas — Fri, 02 Oct 2026 09:45:25 +0000

A payment request that works once may still charge the customer twice.

State-changing API operations must be tested beyond the first successful request. Creates, updates, transfers, and deletes can behave incorrectly when clients retry after a timeout, replay a completed request, or send identical operations concurrently.

API testing should verify repeated requests after success, retries after lost responses, concurrent requests using the same idempotency key, different updates against the same resource, and updates based on stale data. The expected behavior may be deduplication, conflict detection, serialization, or returning the original result, depending on the API contract.

For payment APIs, reusing the same idempotency key and request must not create a second transaction. The same protection should work when identical requests reach different service instances at the same time. Similarly, concurrent updates must not silently overwrite valid changes made by another client.

Do not verify only the HTTP response. Inspect the final resource state, transactions, emitted events, audit records, external side effects, and asynchronous jobs. Two successful responses must not create duplicate payments, repeated orders, lost updates, or partially completed operations.

Testing idempotency, retry safety, replay protection, and concurrency is essential for reliable APIs and distributed systems. Rentgen is an API discovery tool built to test what an endpoint actually does beyond one successful request.

Read the complete API testing white paper: https://qaontime.com/research/the-power-of-ten-rules-for-testing-http-apis.html

Automation Before Automation.

API Testing Rule 6/10: APIs Must Fail Predictably

Liudas — Wed, 30 Sep 2026 12:40:07 +0000

API failures are inevitable, but they should never be unpredictable. Invalid requests, missing resources, authentication failures, rate limits, dependency outages, and internal errors are different conditions. They should return different and meaningful HTTP status codes instead of collapsing into a generic 500 Internal Server Error.

Predictable API error handling allows clients to understand what happened and decide whether to fix the request, authenticate, retry later, or stop. Similar failures should produce consistent status codes, stable error structures, and useful messages without exposing sensitive implementation details.

For example, a missing resource should return 404 Not Found, a malformed request should return 400 Bad Request, and a rate-limited client should receive 429 Too Many Requests. A 500 Internal Server Error should indicate that the server failed while processing an otherwise valid request—not that the client submitted invalid data.

API testing should cover validation errors, missing resources, authentication and authorization failures, unsupported operations, rate-limit violations, timeouts, dependency failures, and unexpected internal errors. Testers should verify that error responses are consistent, actionable, secure, and easy for API clients to interpret.

Any reproducible request that consistently generates a 5xx response should be investigated as a potential defect. Reliable APIs are not defined only by successful responses. Predictable failure handling is a core part of API reliability, REST API design, and software quality.

Rentgen is an API discovery tool built to explore failure paths beyond the expected request and expose assumptions before they become production incidents.

Read the complete API testing white paper: https://qaontime.com/research/the-power-of-ten-rules-for-testing-http-apis.html

Automation Before Automation.

API Testing Rule 5/10: Every Collection Endpoint Must Have Bounded Results

Liudas — Tue, 29 Sep 2026 10:16:12 +0000

GET /customers may work perfectly with 100 records. But what happens when the dataset grows to one million?

Pagination alone is not enough. Collection endpoints should enforce a maximum page size and predictable defaults. API testing should verify what happens when pagination parameters are omitted, extremely large limits are requested, complex filtering and sorting are combined with pagination, or large collections are retrieved repeatedly.

Without server-enforced limits, a single valid request can consume excessive database, memory, network, and client resources. An endpoint that returns unbounded results is a scalability and reliability defect waiting to become a production incident.

In the complete white paper, we explain how to test pagination, maximum page sizes, cursor-based navigation, filtering, sorting, performance under large datasets, and protection against excessive requests.

Read the complete white paper: https://qaontime.com/research/the-power-of-ten-rules-for-testing-http-apis.html

Automation Before Automation.

API Testing Rule 4/10: Authentication Authorization

Liudas — Mon, 28 Sep 2026 06:53:05 +0000

A valid token proves identity, not permission.

Testing whether a user can access their own profile is not enough. Repeat the same request with another user’s resource ID and verify that access is denied.

Check different roles, owned and unowned resources, cross-tenant access, read/update/delete operations, and predictable IDs.

Authentication can work perfectly while authorization fails completely.

Rentgen helps discover these issues by testing API behavior beyond the expected authenticated request.

Read the complete white paper:https://qaontime.com/research/the-power-of-ten-rules-for-testing-http-apis.html

Automation Before Automation.

I Made a Physical Notebook for Software Testers. Yes, in 2026.

Liudas — Fri, 18 Sep 2026 17:51:40 +0000

We have Jira. Notion. Confluence. Test management systems. AI assistants. Automated test suites running in CI/CD. So naturally, I made a paper notebook.

Not because I suddenly became nostalgic for the 1990s, and definitely not because I think QA teams should start documenting test cases with a pen. The reason is much simpler: after many years in software testing, I still find that some of my best testing happens before I open another tool.

When I start testing something new, I usually need to understand the problem first. What changed? What can go wrong? What assumptions are we making? What don't I understand yet? What did I just observe that looks strange?

And somewhere between Slack, Jira, browser tabs, API clients, DevTools, and documentation, those thoughts disappear surprisingly quickly. So I created the QA Notebook.

This is not a test case repository. That was probably the most important rule when I started designing it. I didn't want another place where a tester writes:

Step 1: Click Login
Step 2: Enter username
Step 3: Enter password
Expected result: User is logged in

We already have enough tools for storing things like that. I wanted something for the part of testing that happens before everything becomes structured and repeatable. Each testing session starts with a few simple questions: What am I testing? What changed? What can go wrong? What are the known risks? What did I learn? What is still uncertain? And finally: Do I have enough confidence to move forward?

Yes. No. Or with known risks.

The page next to it is intentionally mostly empty. Because testing doesn't always fit into predefined fields. Sometimes you need to draw a flow. Sometimes you write down three API responses. Sometimes you discover a weird state transition and draw arrows everywhere. Sometimes you have a hypothesis that turns out to be completely wrong ten minutes later. That's fine. The notebook should help the investigation, not force the investigation to fit the notebook. Good testing starts with questions, not test cases

One thing I've tried to put throughout the notebook is small reminders that interrupt the usual testing routine.

For example: A 200 OK response can be a bug.
Test what developers didn't think about.
Every assumption is a potential test case.
If the UI prevents something, check whether the backend prevents it too. A passing test only tells you what happened under one set of conditions.

These aren't supposed to be profound quotes hanging on an office wall. They're there because sometimes one question is enough to send your testing in a completely different direction.

You might be testing a perfectly normal form, see a reminder about boundaries and suddenly try the value just above the documented maximum.

You might be testing permissions and notice: A disabled button is not an authorization control. So instead of stopping at the UI, you inspect the API. That is exactly what I wanted this notebook to do: occasionally give the tester one more idea. Bugs need investigation, not just tickets. I also added dedicated bug investigation pages.

When something strange happens, my first instinct isn't necessarily to immediately create a Jira ticket. First I want to understand what I actually found.

What happened? What did I expect? Can I reproduce it? What changed? What evidence do I have? What am I assuming? What is the smallest reproducible case? That last question is particularly useful.

A giant broken workflow involving twelve steps, three users and two services isn't very helpful if you can reduce the same problem to one request. So the investigation pages have space to work through that before turning the finding into a polished bug report.

I also wanted the notebook to generate testing ideas. There are reference sections scattered through it rather than putting 150 pages of blank templates together and calling it a QA notebook.

There's SFDIPOT for when you think you've run out of things to test: Structure, Function, Data, Interfaces, Platform, Operations and Time.

There are reminders to look beyond functional behavior into performance, security, reliability, usability, accessibility, compatibility, scalability, recoverability, availability, localization and observability.

There are sections specifically about API testing: authentication, authorization, input validation, headers, CORS, status codes, rate limits, oversized requests and other things that are very easy to forget when the happy path works perfectly.

Toward the end I added practical reminders about decision table testing and state transition testing, because interesting bugs often don't live in a single condition or state. They live in combinations and transitions.

The notebook ended up being 162 pages, but the goal was never to create a QA textbook. It's something that should sit next to the laptop while you're actually testing.

Why paper? This is probably the strange part.

I work in IT. I build automation. I use AI. I write Playwright tests. I work with APIs and CI/CD. And I still like writing things down.

There is something different about having a notebook open next to the keyboard while investigating a system. I don't have to decide where the note belongs. I don't need to create a ticket. I don't need to format anything. I don't need another browser tab. I can just write: "Why did this return 200?" Circle it. Test something else. Come back to it twenty minutes later. Maybe it becomes a bug. Maybe it becomes a test. Maybe it becomes nothing because my assumption was wrong. That's testing.

The notebook doesn't need to preserve every thought forever. It needs to help me think while the investigation is happening. Automation Before Automation. There's a line in the notebook that probably summarizes the whole idea better than anything else: Think first. Test second. Automate when it makes sense. I'm obviously not against automation. Quite the opposite.

But automation becomes much more useful after we understand what behavior matters, what risks we're protecting against and what conditions are worth checking repeatedly.

Before asking "How do we automate this?", somebody still has to ask: "What should we test?" That's the space I wanted the QA Notebook to live in. Not instead of Jira. Not instead of Playwright. Not instead of AI. Just next to the keyboard. With a pen. And preferably a tester who keeps asking one more question.

QA Notebook — Think. Test. Question. Write it down.

by Liudas Jankauskas
https://www.amazon.com/QA-Notebook-Liudas-Jankauskas/dp/B0HH6MNG4Z

API Testing Rule #3: Response Codes Are Part of the Functionality

Liudas — Thu, 17 Sep 2026 05:12:27 +0000

An API does not communicate only through its response body. The HTTP status code is also part of its contract.

A successful request should not return the same status code as an authentication failure, invalid input, a missing resource, or an unsupported HTTP method. Clients, monitoring tools, and other services rely on these distinctions.

For example:

201 Created for a successfully created resource
400 Bad Request for invalid input
401 Unauthorized for missing or invalid authentication
403 Forbidden for insufficient permissions
404 Not Found for an unavailable resource
405 Method Not Allowed for an unsupported method
429 Too Many Requests when rate limits are exceeded
500 Internal Server Error only for unexpected server failures
Incorrect response codes can hide bugs, confuse API consumers, and expose information about protected resources.

Response code testing should cover both successful and unsuccessful scenarios. An API that returns the wrong status code is not behaving correctly, even if the response body looks reasonable.

Rentgen helps explore these cases automatically from a working cURL request, including invalid input, authentication failures, unsupported methods, and unexpected responses.

HTTP status codes are not decoration. They are functionality.

Full white paper:
https://qaontime.com/research/the-power-of-ten-rules-for-testing-http-apis.html

API Testing Rule #2: Every Endpoint Must Be Tested With Invalid Input

Liudas — Tue, 15 Sep 2026 11:41:41 +0000

A successful request proves only that the happy path works. It does not tell you how the API behaves when the input is incomplete, malformed, unexpected, or too large.

Most real defects appear when application assumptions are violated. That is why every endpoint should be tested not only with valid data, but also with invalid input.

For a user creation endpoint, a valid request might look like this:
{ "name": "John Smith", "email": "john@example.com", "phone": "+971501234567" }

That request confirms that valid data is accepted. However, it does not answer more important questions:

What happens when a required field is missing?
What happens when email contains an invalid value?
What happens when a string is replaced with a number?
Are null values accepted?
Are empty strings rejected?
Are unexpected properties ignored or stored?
What happens when the payload is extremely large?
Does malformed JSON result in a controlled error?

A common API defect is returning HTTP 500 Internal Server Error when the client sends invalid input. The server should reject malformed data with a clear client error, not fail internally.

Oversized values are another important example. An API may accept a user name containing several megabytes of text and return a successful response. The problem might appear later in a report, mobile application, search index, integration, or analytics pipeline.

This makes invalid input testing more than a validation check. It is also a reliability, security, and operational stability check.

At minimum, every endpoint should be tested with:

Missing required fields
Invalid formats
Incorrect data types
null values
Empty values
Boundary values
Oversized strings
Oversized payloads
Malformed JSON
Unexpected properties

The expected result should normally be a controlled 4xx response with a meaningful validation message.

Invalid input should never cause application crashes, database corruption, service instability, unbounded memory consumption, or unexpected HTTP 500 responses.

With Rentgen, this type of testing does not require a large test suite to be written manually. Start with a working cURL request, describe what the fields represent, and let the tool explore variations around the expected input.

The goal is not only to prove that valid requests work. The goal is to discover what happens when the assumptions behind those requests are wrong.

That is where many serious API defects begin.

This is the second rule from The Power of Ten – Rules for Testing HTTP APIs:
https://qaontime.com/research/the-power-of-ten-rules-for-testing-http-apis.html

API Testing Rule #1: Treat Every Endpoint as an Attack Surface

Liudas — Sun, 06 Sep 2026 08:06:03 +0000

“Internal” describes an endpoint’s current audience. It is not a security boundary.

Internal APIs often become dependencies for mobile apps, other teams, partners, automation scripts, and future services. If a request can reach an endpoint, that endpoint must be tested independently.

At minimum, verify:

Authentication requirements
Authorization and ownership boundaries
Invalid input handling
Supported HTTP methods
Response code correctness
Rate limiting
Sensitive information disclosure

CORS should not be treated as protection for the API. It restricts certain browser requests but does not stop scripts, custom clients, or server-to-server communication.

Rentgen is not another general-purpose API client. It is an API discovery tool that starts with one working cURL request and systematically explores behavior beyond the expected path.

This is the first rule from The Power of Ten – Rules for Testing HTTP APIs: https://qaontime.com/research/the-power-of-ten-rules-for-testing-http-apis.html

StackHawk and Rentgen: Same API World, Different Layers

Liudas — Thu, 27 Aug 2026 04:22:50 +0000

StackHawk and Rentgen can look similar because both test APIs, send imperfect input, and analyze backend behavior. There is some real overlap, especially when malformed requests expose weak validation, unexpected server errors, or incorrect authentication behavior. However, the tools begin at different moments and answer different questions.

Rentgen starts with one working cURL request and turns it into structured API hygiene checks. It explores missing fields, wrong data types, boundary values, malformed payloads, invalid authentication, unsupported methods, and inconsistent responses. The goal is to give developers and QA engineers a fast reality check before a test suite, CI pipeline, or formal security process exists.

StackHawk approaches the same API from an application security perspective. It provides structured DAST across running applications and APIs, integrates with development pipelines, and helps teams identify, prioritize, and manage security findings. This becomes especially valuable when security testing needs to be systematic and repeatable across the wider application.

These tools do not need to compete. Rentgen can expose fragile API behavior while an endpoint is still being built, helping remove basic validation and error-handling problems early. StackHawk can then focus on broader and deeper security weaknesses as part of a mature DevSecOps workflow.

This is Automation Before Automation.

The full comparison is available on Rentgen API Stories: https://rentgen.io/api-stories/StackHawk-and-Rentgen-API-security-scanning-and-API-hygiene-before-automation.html

ffuf vs Rentgen: Same HTTP Space, Completely Different Job

Liudas — Wed, 26 Aug 2026 07:20:05 +0000

I’ve seen ffuf and Rentgen mentioned in the same breath a few times, which kind of makes sense on the surface: both deal with HTTP, both involve changing inputs, both can produce weird responses.

But they’re not really for the same job.

ffuf is mainly a discovery tool. You use it when you want to find things: paths, subdomains or vhosts, parameters, files, endpoints, response differences that might point to something interesting. It’s great when the question is basically, “what’s here?”

Rentgen starts from a different place. You already have a working API request. You paste in the cURL, and then the tool starts checking how that endpoint behaves when the input stops being clean or expected.

Things like:

missing fields
wrong data types
broken or invalid auth
malformed JSON
oversized values
unsupported methods
other not-quite-valid input
So even though both tools touch HTTP and mutate requests in some way, the goal is different.

With ffuf, you’re exploring the surface area.

With Rentgen, you’re poking at the behavior of something you already know exists.

That’s why comparing them directly never felt quite right to me. One helps you figure out where to look. The other helps you understand how a known API handles bad input.

Same general space, different phase of the work.

Full story here: https://rentgen.io/api-stories/ffuf-and-Rentgen-web-fuzzing-and-API-hygiene-are-not-same-planet.html

Tusk Drift vs Rentgen: Traffic Replay vs Early API Discovery

Liudas — Tue, 04 Aug 2026 06:00:59 +0000

API quality is improving rapidly, but different tools solve different problems.

Recently I looked at Tusk Drift, and I think it introduces an interesting idea: recording real production traffic and replaying it as regression tests. At first glance it may seem similar to Rentgen, but after looking closer, the two tools actually sit at different stages of the API lifecycle.

They complement each other rather than compete.
Tusk Drift starts with production traffic
Tusk Drift records real API requests generated by users or integrations and later replays them as regression tests.
The goal is straightforward: Did today’s code change break behavior that already existed yesterday?
This is extremely valuable because production traffic often exercises scenarios nobody thought to automate manually. Instead of maintaining hundreds of regression tests yourself, you can reuse what users are already doing.
For mature systems, that’s a powerful safety net.

Rentgen starts with a single request

Rentgen works much earlier.
It doesn’t need production traffic.
It doesn’t need historical traces.
It doesn’t even need an existing test suite.
It starts from one working cURL request (or a request imported from Postman, Swagger, logs, or another client).
From there Rentgen automatically generates hundreds of API checks by mutating that request:

missing fields
invalid data types
boundary values
oversized payloads
malformed JSON
unsupported HTTP methods
invalid enums
whitespace problems
authentication scenarios
protocol validation The question Rentgen asks is completely different: How does this endpoint behave when the input stops being clean?

Different questions

Tusk Drift asks: Has behavior changed?
Rentgen asks: Is behavior robust?
Those are not the same thing.
An API can replay production traffic perfectly while still crashing on unexpected input.
Likewise, an API may survive every malformed request Rentgen generates but still introduce regressions after future code changes.
Both are important.

Different timing

This is probably the biggest distinction.
Tusk Drift
Works best when:
the service already has users
production traffic exists
behavior is worth protecting
Rentgen
Works even if:
the endpoint was created five minutes ago
there are no users
there is no production traffic
there is no regression suite
One working request is enough.

A practical workflow

These tools actually fit together quite naturally.
Build a new endpoint.
Run Rentgen locally from a single cURL request.
Fix validation gaps, unexpected 500 errors, inconsistent responses, and weak input handling.
Deploy.
Let users generate real traffic.
Use Tusk Drift to replay that traffic as regression tests.
One discovers problems before release.
The other makes sure they never silently return.

No replacement story

I don’t see these tools replacing each other.
Tusk Drift is not trying to be an API mutation engine.
Rentgen is not trying to record and replay production traffic.
They’re solving different problems at different moments in the software lifecycle.
That’s why comparing them is interesting. Not because one wins. Because together they cover more of the API quality journey.

The complete comparison, including diagrams and examples, is available here: https://rentgen.io/api-stories/Tusk-Drift-and-Rentgen-replaying-real-traffic-vs-breaking-one-request-early.html

Apigee and Rentgen: API Management Is Not API Testing

Liudas — Wed, 29 Jul 2026 17:41:25 +0000

If you work with APIs, you've probably heard of Apigee. You may also have come across Rentgen. Since both tools operate in the API space, it's easy to assume they solve similar problems. They don't.

They work at different layers of the API lifecycle.

Apigee manages APIs. Apigee is an API management platform. It helps organisations expose, secure, monitor and govern APIs at scale.

Typical responsibilities include:

API Gateway
Authentication & Authorization
Rate limiting
Traffic management
Policies
Analytics
Developer portal
API product management

When your APIs become products consumed by customers, partners or internal teams, this layer becomes essential.

Rentgen tests API behaviour. Rentgen starts with a single working cURL request. Instead of managing traffic, it asks a different question: How does this endpoint behave when the input is no longer perfect?

It automatically generates request variations such as:

Missing required fields
Invalid data types
Boundary values
Empty strings
Malformed JSON
Unsupported HTTP methods
Invalid headers

The goal isn't automation. The goal is understanding endpoint behaviour before automation begins. Different layers.

One of the biggest misconceptions is believing that an API gateway somehow guarantees API quality. It doesn't.

A gateway can enforce authentication, quotas and routing, while the backend may still:

return unexpected 500 errors
accept invalid values
expose inconsistent validation
produce confusing error responses

These are implementation problems, not infrastructure problems. That's where endpoint-level testing becomes valuable.

Rather than choosing one tool over the other, they fit naturally into different stages of API development.

Build the endpoint.
Test its behaviour with Rentgen.
Fix validation and reliability issues.
Publish and manage the API with Apigee.

One focuses on implementation quality. The other focuses on operating APIs at scale. Both are important.

If you'd like the full comparison with more examples and discussion, you can read the original article here: https://rentgen.io/api-stories/Apigee-and-Rentgen-managing-API-traffic-is-not-same-as-testing-API-behavior.html