DEV Community: Jeff

MCP Tool Versioning and Deprecated Tools

Jeff — Sat, 14 Nov 2026 15:27:21 +0000

Quick answer: MCP does not define a standard version or deprecated field for a tool, so you evolve tools by convention: keep breaking changes under a new name (such as search_v2), mark the old tool as deprecated in its description, keep it working with a warning, and remove it only after a fixed sunset window. Additive changes can stay on the same name; anything that changes the input shape or the meaning of output needs a new version.

Why tools need versioning

An MCP server keeps changing after clients connect: a tool gains a parameter, a field changes type, or a value changes unit. Saved agents and prompts still call the old name, and a silent change can produce wrong results with no signal. Versioning makes the change visible so callers move on their own schedule rather than getting a new contract on an old name.

What the protocol gives you

A tool returned from tools/list carries a name, an optional description, and an input schema. There is no required semantic-version field and no boolean that marks a tool as deprecated; the optional tool annotations cover hints such as read-only or idempotent, not retirement. That means versioning and deprecation are conventions you enforce, not fields the host will act on automatically. Treat the tool name as the stable contract the model relies on.

How teams version a tool

Approach	What it looks like	When to use it
Versioned name	`search_v1`, `search_v2`	Any breaking change; the model sees a distinct tool
Description note	Version stated in the description	Communicating revision without a new name
Schema field	Optional `version` input parameter	Caller needs to select behavior explicitly
Server metadata	Server version in the init handshake	Tracking the whole server, not one tool

The versioned name is the most reliable lever because the model chooses tools primarily by name and description. A new name is a new tool, so an old agent that named search_v1 does not silently get v2 behavior.

How to deprecate without breaking callers

Do not delete the old tool on the day the new one ships. Saved prompts and long-running agents still reference the old name, and a missing tool becomes a hard failure. First, prefix the old description with a clear DEPRECATED note and name the replacement. Keep the tool functioning but return a warning alongside its result so the model is nudged toward the new one. Publish a sunset date, then remove the name only after that window; logging calls to the old tool shows whether anything still depends on it.

Compatible versus breaking changes

Change	Same name or new	Reason
Add optional input parameter	Same	Old callers omit it and still work
Add a response field	Same	Callers that ignore unknown fields are fine
Change a field type or unit	New	Old parsing and assumptions break
Remove or rename a parameter	New	Existing calls fail or misbehave
Change what a value means	New	Correct output silently becomes wrong

The last row is the dangerous one: a field that keeps its name but changes meaning passes every schema check and still breaks reasoning. To confirm a v2 output really differs from v1, send both requests side by side and compare the schemas; Powerduck keeps the two responses visible so the difference is evidence rather than a guess.

What the model actually sees

The model reads the tool name and description at selection time; it does not run a version resolver, and it is not guaranteed to honor a deprecation note the way a compiled client would honor a typed header. A warning in the result helps, but the host cannot assume every call migrates by the sunset date. That is why the old name must stay functional through the window rather than throwing.

Honest limits

These patterns are conventions, not protocol guarantees, and different hosts may surface descriptions differently. A deprecation note does not prove the model switches tools, and naming alone does not prevent collisions across many servers, which needs separate prefixing. Registry-level lifecycle, where a published package is listed, retired, or replaced across an ecosystem, is a separate concern from versioning a tool inside your own server. Importing or connecting a server also does not deploy a backend, and the protocol does not validate that a new tool's behavior is correct.

Frequently asked questions

Does MCP have a built-in tool version field? No. A tool exposes name, description, and input schema; teams add versioning through naming and descriptions.

How do I mark a tool as deprecated? Add a clear DEPRECATED note to its description, name the replacement, and return a warning while keeping the tool working until a sunset date.

Can I change an existing tool's input schema in place? Only for additive, optional changes. Anything that breaks old callers should ship under a new versioned name.

When can I remove the old tool? After the published sunset window, once logs show no remaining calls to the old name.

The bottom line

Because MCP leaves versioning and deprecation to you, use the tool name as the contract: ship breaking changes under a new versioned name, keep the old one alive with a visible warning and a fixed sunset, and reserve same-name edits for purely additive changes. For more on building and operating these servers, see the MCP server guide hub.

What Is the Model Context Protocol?

Jeff — Sat, 14 Nov 2026 13:30:31 +0000

Quick answer: The Model Context Protocol (MCP) is an open, client-server protocol that standardizes how an AI application connects to external tools and data sources. Instead of building a separate custom integration for every tool, an app talks to lightweight MCP servers over one common interface, and each server exposes a set of tools, resources, and prompts the model can use. It defines the connection and the message format, not the model itself.

The problem it solves

Without a shared protocol, every AI application needs its own connector for every data source and service it wants to reach. As the number of apps and services grows, the number of one-off integrations multiplies, and each one handles auth, error handling, and data shapes differently. MCP collapses that into a single client-server contract. A server written once can be used by any compatible host, and a host can support many servers without learning each vendor's private API. The common analogy is a universal port: one connector shape on both sides.

The roles

The architecture has three parts that are easy to conflate. The host is the application the user runs and in which the model lives. Inside the host, a client manages the connection to a single server, with one client per server. The server is a separate, small program that exposes capabilities and runs outside the model, often as a local process.

Role	What it is	Responsibility
Host	The AI application	Runs the model, holds context, enforces consent
Client	Connector inside the host	One per server, manages the session
Server	External program	Exposes tools, resources, prompts

The three primitives

A server can offer three kinds of capability. Tools are functions the model can invoke, such as running a query or creating a record; they may have side effects, so a host normally asks for user approval before one runs. Resources are readable data, like a file or a database row, that the model or user can pull into context. Prompts are reusable templates and slash commands that shape how a task is started. Tools are actions the model takes, resources are data it reads, and prompts are authored starting points.

How they talk

Messages use JSON-RPC 2.0, with requests, responses, and notifications flowing in both directions. When a session starts, the client and server exchange capabilities so each side knows what the other supports. Two transports are common: stdio, where the host launches a local server as a subprocess and communicates over standard input and output, and a remote HTTP transport for servers reached over a network. The same primitives work over either transport; only the carrying mechanism changes. If you want to see what a tool call actually does before wiring it into an agent, you can exercise the underlying HTTP request in a local workspace; Powerduck keeps that request and its response visible rather than hiding behavior behind a tool name.

Security and consent

MCP does not grant a server automatic trust. The host stays in control of which tools run and can require confirmation for actions with side effects, and the model cannot reach a resource the user has not authorized. Remote servers use standard authorization such as OAuth, and the principle is least privilege: a server gets access to only what it needs. Connecting a server does not make its output correct, and a malicious or poorly written server is still a risk the host and user must evaluate.

Honest limits

The protocol standardizes the connection, not the quality or safety of what sits behind it. It does not validate that a tool behaves correctly, that returned data is accurate, or that a server handles errors well, and it does not solve naming collisions, versioning, or rate limits across many servers on its own. Those remain engineering and operational choices. MCP is also not a model and does not run inference, and importing or connecting a server does not deploy a backend. Treat it as a well-defined transport for context and actions, with the real guarantees coming from how each server is built and reviewed.

Frequently asked questions

Is MCP an AI model? No. It is a protocol that connects an AI application to external tools and data; the model stays inside the host.

What is the difference between a tool and a resource? A tool performs an action and may have side effects; a resource is data that is read into context.

Does an MCP server run locally or remotely? It can do either. Local servers commonly use stdio, while remote servers use an HTTP transport.

Does connecting a server make it safe? No. The host still controls consent and authorization, and the server's behavior and security need separate review.

The bottom line

MCP gives AI applications one standard client-server interface for reaching tools, resources, and prompts over JSON-RPC, replacing a sprawl of custom connectors, while leaving trust, consent, and data quality to the host and the server authors. For more on building and operating these servers, see the MCP server guide hub.

Testing Bulk Endpoint Partial Failure

Jeff — Sat, 14 Nov 2026 03:26:23 +0000

Quick answer: A bulk endpoint accepts many items in one request and can succeed on some while rejecting others, so the top-level status code does not tell you what happened. First pin down whether the endpoint is atomic (all items commit or all roll back) or non-atomic (each item is independent), then test the per-item result array across all-success, all-fail, and mixed cases, match every result back to its request item by identifier, and confirm the client lets the user retry only the failed entries without replaying the ones that already landed. Asserting on the single outer status is the mistake that hides a half-applied batch.

First pin down the contract

The two designs behave differently under failure and the tests follow from which one you have. An atomic endpoint treats the whole batch as one transaction: if any item is invalid it returns a single 4xx and nothing is written, so there is no partial state to reconcile. A non-atomic endpoint processes each item independently and returns a result per entry, so a batch can leave some resources created and others rejected. The documentation should state which model applies; if it does not, send a mixed batch and observe whether the valid items persist before you write the suite. Do not assume the presence of a bulk endpoint means a transaction.

The response shapes

The explicit form is 207 Multi-Status, whose body lists each entry with its own status, identifier, and error detail. Other APIs return 200 with a result array where every item carries a success flag, or 202 Accepted and process the batch asynchronously with a status endpoint to poll. In every non-atomic shape the meaningful result is the per-entry outcome, not the outer code. A 207 can contain both 201 and 409 entries, and a 200 can carry failed items, so a client that branches on the outer status alone will misread the batch.

The test matrix

Case	Request	Expected per-item result
All valid	Clean items	Every entry 2xx, all side effects present
All invalid	Bad items	Every entry 4xx, nothing written
Mixed	Valid and bad items	Success and failure entries both returned
Empty or single	Zero or one item	Handled, no crash or false result
Duplicate id	Same identifier twice	Conflict flagged, not silently merged
Over size limit	More than the cap	Rejected or bounded, never truncated silently
Partial then retry	Re-send failed items	Failed items fixed, landed items not duplicated

Matching results and retrying

Match each result to its request by the client-supplied identifier or the resource id, not by array position, because the server may reorder or drop entries. For the mixed case, assert the successful items exist and the rejected ones have no side effect, then check the UI renders the two groups separately so the user can act on the failures. A retry should send only the failed entries and rely on idempotency so the already-created items are not duplicated; re-sending the entire batch risks a second set of resources. In a desktop API workspace you can run the bulk request and the follow-up retry as an ordered scenario and inspect each sub-result. Powerduck supports this flow, keeping the per-item sequence deterministic rather than eyeballing a large response.

Honest limits

These tests prove the client handles per-item results and retries correctly, but they do not prove the real server's transaction boundaries, partial-commit behavior, database constraints, or how it processes a large batch under load. A local mock cannot reproduce an async worker that fails after accepting the job, race conditions between concurrent batches, or the latency of a queue, so confirm the production path with a genuine mixed batch and verify which items actually persisted. A passing run says nothing about whether the server stays consistent when a worker crashes mid-batch, and importing a schema does not deploy a backend.

Frequently asked questions

Does 207 mean the request failed? No. It means the response carries multiple independent statuses; read each entry rather than treating the whole call as an error.

How do I know if the endpoint is atomic? Send a mixed batch and check whether the valid items persist. If they do, it is non-atomic; if nothing is written, it is atomic.

What should I do after a partial failure? Retry only the failed items by id, confirm the landed items are not re-sent, and verify the retry does not create duplicates.

Can I trust result order? No. Match by identifier, since the server may process or return items in a different order.

The bottom line

Decide atomicity first, then drive the per-item result array across all-success, all-fail, mixed, duplicate, and retry cases, matching entries by identifier and asserting both the response and the side effects. Confirm the real server's partial-commit behavior separately. For more on this approach, see the API testing guide hub.

Testing Cache Invalidation Headers

Jeff — Fri, 13 Nov 2026 19:26:31 +0000

Quick answer: Cache invalidation is controlled by response headers, not by a separate delete call. Cache-Control sets how long a response stays fresh and whether it can be stored, and ETag plus a conditional request handles revalidation once that window passes. Test the fresh window, the stale window with a 304, a changed ETag returning 200 with a new body, no-store never being cached, and Vary splitting the cache by the right dimension. The assertions should cover both whether the client sends a request and which conditional headers it carries, since a stale response that silently serves old data is the failure you are hunting.

What the headers mean

Cache-Control: max-age=60 marks the response fresh for 60 seconds; no-cache forces revalidation before reuse even though the response may be stored; no-store forbids storage entirely; private limits the response to a single user's cache while public allows shared caches. The server also sends an ETag, and on the next request the client echoes it in If-None-Match. If the resource is unchanged the server returns 304 with no body and the client keeps its copy; if it changed, the server returns 200 with a new body and a new ETag. Last-Modified and If-Modified-Since do the same job at one-second precision.

The test matrix

Case	Response headers	Next request	Expected
Fresh window	`max-age=60`	None sent	Cached copy used
Stale, unchanged	`no-cache`, same ETag	`If-None-Match` sent	304, copy reused
Stale, changed	New ETag	`If-None-Match` sent	200, new body
Never stored	`no-store`	Full request each time	200, no reuse
Split cache	`Vary: Authorization`	Separate request per identity	No cross-user copy
Private data	`private, max-age=0`	Revalidated per user	Not shared

Deterministic timing and revalidation

Real wall-clock waits make these tests slow and flaky, so drive the clock or set short, fixed max-age values against a mock you control. Assert the request count: inside the fresh window the client should not call the network at all, and after expiry it should send a conditional request rather than a blind full fetch. When the ETag matches, confirm the client treats 304 as a valid response and does not reset state or flash a loading state; when it differs, confirm the new body replaces the old one and the stored ETag updates. A client that ignores 304 and shows an empty view, or one that never revalidates and serves stale data forever, both fail here.

Vary and cache busting

Vary tells the cache which request headers change the response, so a response that differs per user or per encoding must list those dimensions or one user gets another's copy. Assert that two identities with the same URL do not share a stored response. For resources that change on deploy, fingerprinting the URL with a content hash is more reliable than waiting for expiry: a new file name is a cache miss by definition, while the old URL stays cached. In a desktop API workspace you can run each header combination as an ordered scenario and inspect the exact conditional request and response. Powerduck works this way, making the revalidation sequence repeatable instead of relying on browser cache behavior.

Honest limits

These tests prove the client and HTTP layer honor the headers you configure, but they do not prove how a real browser, CDN, or corporate proxy will cache the same response. A local mock cannot reproduce multi-layer caches, clock skew, purge propagation delay, or a CDN's default rules, so confirm the production path with a small set of genuine requests and check the Age header. The checks do not cover server-side cache invalidation or a purge that fails silently, and a passing run says nothing about whether the stored content is correct. Importing a schema does not deploy a backend, and a green run is not evidence about the upstream service.

Frequently asked questions

What is the difference between no-cache and no-store? no-cache allows storage but requires revalidation before reuse; no-store forbids storing the response at all.

Does a 304 return a body? No. The server sends headers only, and the client keeps using its stored body when the ETag still matches.

Why does my changed response still show old data? The max-age window has not expired, the ETag did not change, or Vary is missing. A fingerprinted URL avoids the first two.

Should I test real wall-clock expiry? Use short fixed ages or a controllable clock for determinism, then confirm one genuine end-to-end path against the real CDN.

The bottom line

Treat invalidation as explicit branches: fresh reuse, stale revalidation with 304, changed ETag with 200, no-store, and Vary-split entries, asserting the request count and conditional headers rather than only the rendered view. Confirm the real CDN behavior separately. For more on this approach, see the API testing guide hub.

Testing Webhook Signature Verification

Jeff — Fri, 13 Nov 2026 17:31:08 +0000

Quick answer: A webhook signature is an HMAC computed over the raw request body with a shared secret, sent in a header the receiver checks before doing any work. Test it with a fixed set of cases: a valid signature accepted, a tampered body rejected, a wrong secret rejected, a missing or malformed header rejected, and a stale timestamp rejected as a replay. On every rejection the receiver should return 401 and skip the side effect, and the signature comparison should run in constant time. The signature must be computed on the original bytes, not on JSON re-serialized by the framework, since reordering fields changes the digest.

How the signature is built

The sender concatenates a timestamp and the raw body, hashes the result with the shared secret, and puts the digest in a header, often in a form such as t=1700000000,v1=.... The receiver rebuilds the same digest from the bytes it received and compares it against the header. Because the secret never travels in the request, an attacker who cannot produce the digest cannot forge a call, which is why unsigned or badly signed requests must be dropped before the handler runs. Pin the exact scheme and header format your provider uses rather than assuming a generic shape.

The test matrix

Case	Signature header	Expected	Side effect
Valid	Correct digest, fresh timestamp	200	Handler runs once
Tampered body	Digest from the original body	401	None
Wrong secret	Digest built with another key	401	None
Missing header	Absent or empty	401	None
Stale timestamp	Valid digest, old timestamp	401	None
Rotated secret	New `v1` alongside old `v0`	200	Handler runs once

The tampered-body case is the one that matters most: keep the signature from the original payload while changing one byte, so the digest no longer matches. The rotated-secret case covers key rollover, where the receiver accepts more than one version for a window.

Use the raw body, not re-serialized JSON

A common failure is verifying the signature against a parsed and re-serialized object instead of the original bytes. JSON allows different field order and whitespace, so the same logical object re-serialized by the framework produces a different digest and rejects genuine calls, or worse, a loose parser accepts a body the signature never covered. Capture the raw request buffer and run the HMAC over those exact bytes, then parse the body only after the check passes. The test should include a payload with reordered or extra whitespace to confirm the receiver hashes what actually arrived.

Constant-time comparison and replay window

Do not compare digests with a plain equality check that exits on the first differing byte, since the response time leaks how many leading bytes matched. Use a constant-time comparison provided by the platform so every signature takes the same path regardless of how close it is. The timestamp gives a separate replay control: define a tolerance window, reject signatures outside it even if the digest is valid, and confirm the test for an old timestamp returns 401. In a desktop API workspace you can run the whole matrix as an ordered scenario that builds each header variant and asserts the receiver status and recorded side effects. Powerduck works this way, turning each signature branch into a repeatable request instead of a handcrafted curl.

Honest limits

These tests prove the receiver follows the verification logic you wrote, but they do not prove the shared secret is stored safely, rotated in production, or protected from logs and crash dumps. A local mock can stand in for the provider, but it cannot confirm the real provider's header scheme, clock skew, or retry behavior, so pin the scheme against genuine documentation and send a small number of real webhooks. The checks do not cover TLS termination, network-level attackers, or a secret that has already leaked, and a passing signature says nothing about whether the payload content is correct. Importing a schema does not deploy a backend, and a passing run is not evidence about the upstream service.

Frequently asked questions

Why verify the raw body instead of parsed JSON? Parsing and re-serializing can change field order and whitespace, which changes the digest. The HMAC has to match the exact bytes the sender signed.

What does the timestamp protect against? It limits replay. A captured request with a valid signature is rejected once the timestamp falls outside the tolerance window.

Is a 401 the only rejection response? Most receivers use 401, but pin the status your framework returns. The key assertion is that no handler side effect runs on any rejected branch.

Why use constant-time comparison? A normal string comparison returns early on the first mismatch, leaking timing information. A constant-time check avoids that signal.

The bottom line

Treat webhook verification as a set of explicit branches and run the valid, tampered, wrong-secret, missing-header, stale-timestamp, and key-rotation cases, hashing the raw bytes and comparing in constant time. Assert that every rejection returns 401 and performs no side effect, and confirm the real provider scheme separately. For more on this approach, see the API testing guide hub.

Testing File Upload Size Boundaries

Jeff — Fri, 13 Nov 2026 15:28:06 +0000

Quick answer: A file upload limit is a byte boundary, so test it with files deliberately placed at the boundary rather than roughly sized samples: empty, one byte, limit minus one, the exact limit, and limit plus one. The exact-limit case settles whether the rule is "at or below" or strictly below, and a rejected file must never be stored. Per-file and total caps are separate checks, and a local mock can return a fixed 413 for each threshold while you confirm the client message and cleanup without moving real large files.

The limit is a byte boundary, not a rough size

A cap such as 5 MB is enforced on a precise count, so a file that looks close tells you nothing until you know its exact bytes. Generate deterministic buffers of a chosen length instead of picking documents by eye, and pin down whether the rule is less than or equal, since an off-by-one at the boundary either accepts a file the server meant to reject or blocks a valid one. Add one file far above the cap to confirm the rejection is not a fluke, and give every case a known expected result before running it.

Per-file and total request limits

Case	Byte size	Expected	Side effect
Empty file	0	Accepted or clear rule	No stored content
Just under	limit - 1	200/201	File stored
Exact limit	limit	Pin accept or reject	Matches the rule
One over	limit + 1	413	Nothing stored
Two valid files	Each under, sum over total	413	Neither stored

The last row catches a common gap: each file passes its own cap while the combined body exceeds the request-level limit. Servers often enforce both, so the suite needs a per-file threshold and a total threshold treated as distinct assertions rather than one "large upload" check.

Watch how the size is measured

The number the client displays and the number the server counts can differ. Base64 encoding grows payloads by about a third, and a multipart body adds boundary and per-part header bytes on top of the file, so the raw file size, the encoded size, and the Content-Length are not the same value. State the unit plainly, because MB can mean either a million bytes or a mebibyte, and the error message should show both the limit and the actual size. A client-side pre-check gives fast feedback, but the server stays the authority, and both should use the same exact threshold rather than rounded numbers.

Assert beyond the status code

A 413 Payload Too Large is common, though some APIs use 422, so pin the status your API returns and check Retry-After if present. Then confirm the rejected file was not stored or processed, because a status code alone does not prove the side effect was avoided. The upload should stop cleanly, the progress state reset, and the client should not resend the same oversized file in a retry loop. For the accepted boundary case, verify the file really lands and its recorded size matches. In a desktop API workspace, run the byte-sized cases as an ordered scenario against one local mock with a size-keyed response, and inspect the recorded requests. Powerduck works this way, turning each boundary into a repeatable fixed response instead of a slow manual upload.

Honest limits

A local mock proves the client handles each threshold, maps the message, and cleans up, but it does not prove the real server's byte cap. A reverse proxy or load balancer often rejects an oversized body before the application ever sees it, sometimes with its own error page, and gateway settings such as a maximum body size need a small set of genuine uploads to confirm. The mock does not store files, run malware scans, enforce disk quotas, or handle chunked and resumable uploads, and it says nothing about streaming memory use or upload timeouts on very large files. Importing a schema does not deploy a backend, and a passing mock run is not evidence about the upstream service or the network in front of it.

Frequently asked questions

Why test the exact limit instead of just one large file? The exact-limit case determines whether the rule accepts or rejects a file at the cap, which is the off-by-one a rough sample never reaches.

What is the difference between per-file and total limits? A per-file cap checks each part, while a total cap checks the whole body. Two valid files can each pass individually yet fail when combined.

Does a 413 mean the file was not saved? Not by itself. Assert against storage or a follow-up read to confirm nothing was stored, since an application can reject the request after writing part of it.

Why might the client and server disagree on size? Encoding overhead, multipart header bytes, and MB versus mebibyte rounding produce different numbers, so the test should use exact byte counts and a shared unit.

The bottom line

Treat the upload cap as an exact byte boundary and run the empty, under, exact, and over cases against both per-file and total limits, then assert storage side effects and clean failure handling rather than trusting the status code. Verify the real gateway and server caps with a few genuine uploads, since the mock only covers client behavior. For more on this approach, see the API testing guide hub.

Testing Backwards-Compatible Additive Changes

Jeff — Fri, 13 Nov 2026 13:30:38 +0000

Quick answer: A change is backwards compatible only when a client that knows nothing about it keeps working, so test it by sending the old request against the new server and confirming every old field behaves exactly as before. Add the new field in a separate case, and keep the two directions distinct: the new client using the new field, and the old client ignoring it. The failures that slip through are not deletions, they are changes that look additive, such as a new enum value or a shifted default, that a strict parser rejects.

What additive actually means

An additive change adds without removing or redefining: a new optional request parameter, a new response field, a new endpoint, or a new header. The new request parameter must be optional with a defined default so an old client that never sends it gets the old behavior. A new response field is safer because an old client is expected to skip what it does not recognize, but that only holds if the client tolerates unknown fields. The moment you delete a field, rename it, change its type, or tighten its validation, the change is no longer additive and needs a version or a deprecation path instead.

Run the old client against the new server

Capture the pre-change contract as a baseline, then drive the new server with two identities. The old client sends a payload without the new field and receives a response that still contains the new field; assert that the old fields are unchanged and that the extra field is ignored rather than causing a parse failure. The new client sends the field and asserts it takes effect. Keep both sequences in the same suite so a server that only works for the newest client is caught immediately. A desktop API workspace can hold the two payloads as variables and run them as ordered scenarios against one local mock, with a response override for the new field. Powerduck works this way, keeping the old-and-new comparison local and repeatable instead of relying on a deployed server.

Changes that look additive but break

Change	Why it is not safe	Test setup	Expected result
New enum value	Strict enum parser rejects the unknown value	Return the new value to the old client	Client tolerates or maps it
New required request field	Old client cannot send it	Call without the field	Old default path still works
Tightened validation	Inputs once accepted now fail	Replay a previously valid request	Same success, not a 400
Changed default	Omitting the field changes behavior	Send no field, compare to baseline	Old behavior preserved
New response field	Whole-body exact match now fails	Assert only the old subset	Old fields unchanged

Assert on the subset, not the whole body

Do not compare the entire response to a stored literal, because a correctly added field turns that assertion into a false failure. Instead, validate that the old fields still satisfy the old schema and check the new field on its own, confirming it is optional and never alters the old path. Verify the parser's tolerance for unknown fields explicitly rather than assuming it, and pin the default the server uses when the new request parameter is absent. When a new enum value is involved, the old client should ignore or map it rather than crash, and the server should keep accepting the old values. These checks turn compatibility from a hope into a named set of assertions.

Honest limits

A local mock proves field-level compatibility and the client's tolerance for unknown data, but it does not exercise the real server's serialization, data migration, or gateway layers. It cannot cover every old client in the wild, including third-party integrations and already shipped versions, so the test represents a chosen baseline rather than the full population. Semantic and enum-level breakage also needs real traffic and monitoring to confirm. Calling a change backwards compatible is a statement about the known clients, not a guarantee that no consumer depends on behavior the test did not model. Importing a schema does not deploy a backend, and a passing mock run says nothing about production.

Frequently asked questions

Is adding a response field always safe? It is safe only if the client ignores unknown fields. A client that validates the whole body against a strict schema can fail on the new field.

Why is a new enum value risky? A parser that expects a closed set of values can reject the new one even though nothing was removed. Test that the old client tolerates or maps it.

What proves the old default still works? Send the request without the new parameter and compare the result to the pre-change baseline. The server must preserve the old behavior, not just return 200.

Should I assert the full response equals a saved example? No. Assert the old field subset against the old schema and check the new field separately, so a valid addition does not break the test.

The bottom line

Backwards compatibility is proven by the old client working against the new server, not by the new field functioning in isolation. Run both directions, treat new enum values and shifted defaults as the real risks, and assert the old subset while tolerating unknown fields, then verify real server behavior and shipped clients separately. For more on this approach, see the API testing guide hub.

Testing the Permission Matrix

Jeff — Fri, 13 Nov 2026 03:31:19 +0000

Quick answer: A permission matrix is a table of roles, resources, and actions where every cell is either allow or deny, and it should be tested as a data-driven set rather than a handful of hand-written calls. Encode the matrix as the expected result, then for each role send every action against every resource and assert the exact status code, the response body, and the side effect. The cases that expose real holes are the deny cells, vertical and horizontal escalation, and the default-deny rule for anything the matrix does not name.

The matrix is the oracle, not the happy path

Write the matrix down first: list the principals such as anonymous, viewer, editor, and owner, the resources they touch, and the actions each one performs. That table is the oracle your tests compare against, so a missing cell is a gap in the test before it is a bug in the code. Most teams only exercise the admin path, which is all allow, plus one obvious deny. That leaves the combinations where a lower role reaches a high-priv action or one user reaches another user's object. A matrix also fixes the default rule: any combination not explicitly allowed is denied, and that default needs its own assertion rather than being assumed.

Drive every cell instead of hand-writing cases

The number of cells grows quickly, so hand-writing each request is how cases get skipped. Build a parameterized list from the matrix and loop over it, selecting the token for the role, issuing the action against the resource, and comparing the result to the expected cell. Keep one token per role so the identity under test is never in doubt, and keep the expected allow or deny next to the inputs so a wrong result is easy to read. A desktop API workspace can store each role's token in a variable and run an ordered scenario that walks the matrix, asserting status, headers, and body at each step. Powerduck works this way, with a local mock returning the allow or deny conditionally so the full matrix runs without a live authorization server.

The privilege failures that matter

Failure class	Definition	Test setup	Expected result
Vertical escalation	Lower role calls a high-priv action	Viewer token against an admin endpoint	403, no side effect
Horizontal (IDOR)	Same role, another user's resource	User A token with user B's resource id	403, no data leaked
Anonymous	No identity at all	Request without a token	401
Default deny	Combination not in the matrix	Action left unlisted	403
Owner versus member	Ownership changes the result	Owner and collaborator tokens	Allow only where the matrix says

Assert beyond the status code

A correct 403 is not enough on its own. On a deny, confirm the response body does not include the protected fields and that no side effect occurred, because an endpoint can return 403 while still writing a row. On an allow, confirm the action truly executes rather than only that the status is 200. Watch the distinction between a missing resource and a forbidden one: returning 404 for a real object the caller cannot see can hide the resource, while 403 reveals it exists, so choose deliberately and test it. Sub-resources should inherit the parent's permission, and a batch call needs a rule for partial authorization rather than silently applying the forbidden items.

Honest limits

A local mock returns allow or deny from rules you wrote, so it proves the matrix is fully covered and the client handles each result, but it does not verify the real authorization engine, row-level database permissions, middleware order, or policy caching. The mock is not the source of truth for access, and a rule there can quietly differ from the live policy. A passing run only shows the implementation matches the matrix you wrote; it does not prove the matrix captured every business rule. Confirm genuine escalation and object-level access against a real environment with realistic data, and treat the matrix itself as something a reviewer checks.

Frequently asked questions

What is the difference between vertical and horizontal escalation? Vertical is a lower role reaching a higher-priv action. Horizontal is a user reaching another user's resource at the same level, often called IDOR.

Why use a data-driven test instead of writing each call? The matrix has many cells, and a loop generated from the table covers every one without skipping a combination or letting a case drift from the expected result.

Is a 403 enough to prove a deny? No. Also assert the body leaks no protected fields and no side effect occurred, since a write can happen before the status is returned.

What should happen to a combination not in the matrix? It should be denied by default. Add an explicit case for an unlisted action so the default-deny rule is verified rather than assumed.

The bottom line

Treat the permission matrix as the test oracle and drive every role, resource, and action cell with a parameterized run. Cover vertical and horizontal escalation, anonymous access, and default deny, then assert the body and side effect on each deny and verify the real authorization engine separately. For more on this approach, see the API testing guide hub.

Testing Client Timeout Behavior

Jeff — Fri, 13 Nov 2026 01:26:35 +0000

Quick answer: A timeout is the client deciding to stop waiting, not a response from the server, so test it with fixed mock latency that lands just before and just after the threshold. Assert that the call aborts with the right error, the loading state resets, timers and in-flight work are cleaned up, retries follow a backoff schedule, and a response that arrives after the abort is ignored rather than applied twice. If you only test the fast, successful call, the slow path is still untested.

A timeout is a client decision, not a server error

When the threshold passes, the server may still be processing the request, which is why a blind retry can create a duplicate side effect. Three distinct decisions get conflated under the word timeout. A connect timeout means the TCP connection never established. A read or request timeout means the connection opened but the full response did not arrive in time. An idle timeout means the response started and then stalled, such as headers arriving with no body. Each needs its own case because the cleanup and the error the caller sees are different. Default values in an HTTP library are generic and rarely match your real latency distribution, so leaving them untouched either makes users stare at a spinner or kills slow but successful calls.

Pin the latency to the threshold

Use a local mock with a fixed, known delay rather than a real network, and set the delay to the threshold value and to one millisecond on either side. The boundary case is the one that reveals off-by-one configuration: a response that resolves at the limit should still succeed, while one a moment later should abort. Keep the delays large and stable so a busy machine cannot blur them, and run the near and far cases in the same test so the threshold is judged against both. A desktop API workspace can set mock latency directly and drive conditional overrides, so the timing is chosen in advance and the same run reproduces every time. Powerduck works this way, keeping the timeout test local and repeatable instead of dependent on live network conditions.

The timeout types worth covering

Timeout type	Trigger	Mock setup	Expected client behavior
Connect	TCP connection never opens	Endpoint that refuses or drops the connection	Fail fast, retry with backoff
Read/request	Connected, full response too slow	Fixed latency just over the threshold	Abort, show error, reset loading
Idle/stream	Response stalls after headers	Send headers, then hold the body	Abort the partial read, clean up
Late response	Result arrives just after abort	Latency just past the threshold	Ignore the late result, no double apply

What to assert after the abort

Check the error type and message rather than only that a promise rejected, and confirm the loading flag returns to idle so the UI is not stuck. Verify the timer that drove the timeout is cleared and the in-flight request is cancelled rather than left running. For retries, assert the count and spacing against a backoff schedule and an overall deadline, and treat a non-idempotent POST carefully because an automatic retry can repeat a charge or a create. The late-response row matters most: when the answer arrives after the abort, the client must not apply it to state that already moved on, and must not process the same result twice. A 200 that lands late is still a response the client should treat as stale.

Honest limits

A local mock runs in the desktop main process and reproduces waiting time, but it does not model the full shape of a real failure. It cannot reproduce a half-open TCP connection, a network reset, packet loss, or a server that keeps processing after the client leaves, and application-level fixed delays do not exercise the operating system's real socket behavior. The mock proves the named timeout paths and the client's handling of them; it does not replace genuine network tests or a check of what the server does when the caller disconnects. Importing a schema does not deploy a backend, and a passing mock run is not evidence about production connectivity.

Frequently asked questions

What is the difference between a connect and a read timeout? A connect timeout means the connection never opened. A read timeout means it opened but the response did not finish in time, so the cleanup and error differ.

Why test a response that arrives after the timeout? The server may still answer. The client must ignore that late result instead of applying it to stale state or handling the same response twice.

Should a timed-out POST be retried automatically? Only if the operation is idempotent or carries an idempotency key. Otherwise the retry can repeat a side effect that the first request already performed.

How should I choose the threshold? Base it on the real latency distribution rather than a local round trip, set separate connect and read values, and confirm the choice against live measurements.

The bottom line

Treat timeout as client behavior and pin it with fixed mock latency around the exact threshold, covering the connect, read, idle, and late-response cases. Assert on the error, loading reset, cleanup, backoff-driven retries, and the late response that gets ignored, then verify real socket behavior and server-side handling separately. For more on this approach, see the API testing guide hub.

Testing Concurrent Requests and Race Conditions

Jeff — Thu, 12 Nov 2026 23:37:25 +0000

Quick answer: Concurrent bugs depend on timing, so ordinary tests see them at random and a green run proves little. Make the interleaving deterministic by releasing every request from one start barrier and using fixed delays to control completion order, then assert on the final state, looking for lost updates, duplicate side effects, and stale results from out-of-order responses. If the same test passes or fails without a code change, the test has not yet pinned down the race.

Why concurrency slips past tests

A loop that fires several requests is not truly concurrent, because each call starts a few instructions apart and the scheduler decides the rest. The failure only appears under one particular ordering, which means the bug hides on most runs and shows up in production under load. Two things need coverage: how the client handles overlapping calls, and how the shared resource behaves when more than one request touches it. On the client side, responses can resolve in a different order than the requests were sent, and shared in-memory state can be updated by callbacks that no longer reflect the latest call. On the resource side, read-modify-write and check-then-act sequences can interleave so the last writer quietly undoes the first.

Make the interleaving deterministic

Use a start barrier, a latch or a promise gate, so every request is built first and released at the same instant instead of trickling out of a loop. Then use fixed, known delays rather than real network timing to choose which response lands first. Hold one request at a synchronization point while the other runs to reproduce a specific crossing, and run both orderings, A first and B first, as separate cases. Assert on the resulting state rather than on timing, and keep the delay values large enough that a slow machine cannot blur them. A desktop API workspace lets you drive these calls against a local mock with fixed latency and conditional responses, so the interleaving is chosen in advance and the same run reproduces every time. Powerduck works this way, keeping the concurrency test local and repeatable rather than dependent on live scheduling.

The race patterns worth pinning down

Race pattern	Interleaving	Result without a guard	Typical guard
Lost update	Both read, then both write	Last write wins, first update lost	Version or ETag, atomic update
Check-then-act	Both pass the check, both act	Duplicate side effect	Unique constraint, lock
One-time initialization	Both see it missing, both initialize	Double setup	Once primitive, atomic create
Out-of-order completion	B resolves before A	Stale state shown to the user	Sequence check, not arrival order

For each row, assert the final resource state and the side effect count, not just the status codes. A 200 from each request does not show that one update vanished or that an effect ran twice. When the client aggregates calls, also pin down the failure contract: whether one rejected request rejects the whole set and whether cancellation stops the in-flight calls.

Honest limits

A local mock runs in the desktop main process and verifies the request ordering and the client's handling of it, but it does not exercise the real server's locks, transaction isolation, or database constraints. It cannot reproduce true thread scheduling, kernel-level contention, or the full range of network jitter across separate machines. Deterministic local tests prove the code paths you can name; they do not replace load testing against a real system or a check of the actual isolation level. Importing a schema does not deploy a backend, and a passing local run is not evidence about production behavior under contention.

Frequently asked questions

Why does my concurrent test fail only sometimes? The ordering is left to the scheduler, so the dangerous interleaving appears on only a fraction of runs. A start barrier and fixed delays remove that randomness.

What is the difference between a lost update and a duplicate effect? A lost update comes from read-modify-write and overwrites a change. A duplicate effect comes from check-then-act and runs an action twice, such as a double charge.

Do I need to test both completion orders? Yes, if the client holds shared state. Responses can arrive out of order, and the client should apply the latest sequence rather than whichever call happened to resolve last.

Can a mock replace a load test? No. The mock proves the named interleavings and client logic, while a real load test checks server-side locking, isolation, and contention that the local process cannot model.

The bottom line

Treat flaky concurrent tests as a signal that the timing is still uncontrolled, then use a start barrier and fixed delays to make each interleaving a deliberate case. Assert on final state and side-effect counts across the lost-update, check-then-act, initialization, and out-of-order patterns, and verify real server-side locking and isolation separately. For more on this approach, see the API testing guide hub.

Testing Pagination Boundaries: Catching Off-by-One Errors

Jeff — Thu, 12 Nov 2026 21:25:33 +0000

Quick answer: Pagination off-by-one bugs show up at the edges, where a row is duplicated across two pages or skipped entirely. Test them with fixed datasets sized around exact multiples of the page size, walk every page, and collect the ids into one set so a duplicate or missing row cannot hide behind a correct count. Cover the zero-item, exact-multiple, last-page, and out-of-range cases rather than only the happy path.

Where the off-by-one hides

A page-based API takes a page number and a size, and the offset is usually (page - 1) * size. The errors cluster around a few comparisons. Treating the first page as zero instead of one shifts every offset by a full page. Using floor instead of ceiling for the page count drops the final partial page. When the total is an exact multiple of the size, a wrong loop condition can produce an extra empty page. The comparison at the boundary, using greater-than versus greater-than-or-equal, decides whether the row at the edge lands on one page or both. Limit and offset endpoints have the same problem written differently, and cursor endpoints can repeat or skip the last seen id.

Build boundary-sized datasets

Use a stable, ordered list of known ids and choose totals that sit on and around the boundaries, rather than testing one arbitrary count. With a size of 10, use totals of 0, 1, 9, 10, 11, and 20.

Input	Expected behavior	Common off-by-one
25 items, size 10	3 pages, last page holds 5	Floor gives 2 pages
20 items, size 10	2 pages, no empty page 3	Extra empty page returned
Page 1	Offset 0, first 10 rows	Offset 10 skips the first page
Page past the last	Empty list or a 400	Repeats the last page
Size 0 or negative	A 400	Division error or all rows
Edge ids 10 and 11	One id per page	Duplicate or missing row

Assert the response metadata, total and page count and next and previous links, matches the rows actually returned, since the two can drift independently.

Assert on ids, not just counts

Walk every page and gather the ids into a single set, then compare that set against the full dataset. A count alone cannot tell you that two pages each returned id 10 while id 11 never appeared. Check that the set has no duplicates, covers every id exactly once, and that each full page has the right size and the last page has the expected remainder. Add explicit calls for page zero, the page after the last, and an oversized or zero size so the error contract is pinned down. A desktop API workspace lets you run these pages as an ordered scenario, carry the collected ids through the steps, and assert on each response against a fixed local list. Powerduck works this way, so the boundary walk is repeatable without depending on changing live data.

Honest limits

A local mock runs in the desktop main process and returns a fixed ordered list, which is ideal for the arithmetic and comparison bugs but does not verify the real database query, index, or sort stability. It cannot reproduce rows moving while a client pages through, since inserts and deletes during a real walk can cause duplicates or gaps, and it does not prove ties in the sort key are ordered consistently. Confirm those behaviors against genuine data with a stable sort that includes a unique key. Importing a schema does not deploy a backend, and a passing mock run is not evidence about production query behavior.

Frequently asked questions

Why does a correct total count still fail? Counting returned rows does not detect a duplicate id paired with a missing id. Collect the ids into a set and compare it with the source list to catch both.

Why test exact multiples of the page size? That is where an extra empty page or a missing final page appears, because the loop boundary lands exactly on the last row.

What should a page past the last return? Either an empty list or a documented 400, applied consistently. The bug is silently repeating the last page so the client never stops.

Do cursor APIs have the same problem? Yes, around the last seen id. Repeating the cursor returns the same row, and advancing past it skips a row, so assert on the edge id.

The bottom line

Pagination bugs live at the page edges, so size the test data around exact multiples and the zero and one-item cases, then collect ids across every page and assert on the set rather than a count. Pin down the out-of-range and invalid-size contracts, and verify real database sorting and concurrent changes separately. For more on this approach, see the API testing guide hub.

Mocking Correlation IDs for Request Tracing

Jeff — Thu, 12 Nov 2026 19:25:46 +0000

Quick answer: A correlation ID is a single value that follows a request through several services, so their logs can be joined into one chain. Mock the header exchange by having the mock echo the ID, generate one when the client sends none, and return a mismatch or a missing value on demand. This lets you verify that the client generates, sends, reads, and logs the ID correctly without depending on a real distributed system.

What a correlation ID does

A request often crosses a gateway, one or more services, and a queue, and each component writes its own log. Without a shared identifier, those entries are hard to connect when something fails. The correlation ID, carried in a header such as X-Correlation-ID or X-Request-ID, gives every hop the same value. It has two common origins: the client generates one and sends it, or the entry gateway assigns one when the header is absent and returns it in the response. The response header usually carries the same value, so a support engineer can take the ID shown to the user and pull the matching logs.

Mock the header exchange

Use conditional responses to control what comes back, and inspect the recorded requests to see exactly what the client sent. Cover the cases below.

Scenario	Request header	Mock response header	Client expectation
Client generated	ID present	Echoes the same ID	Log that ID
Gateway generated	No ID	A newly generated ID	Adopt the returned ID
Downstream differs	ID present	A different ID	Follow the documented rule
Missing on response	ID present	No ID header	Keep its own ID, no crash
Retry	Same logical call	Echoes the value	Reuse or renew per convention

The mismatch row matters because a real downstream service can replace the value, and the client needs one clear rule for which ID wins rather than silently logging two unrelated values.

What the client must handle

Put ID generation and reading in one middleware layer so the behavior is consistent instead of being rebuilt per request. Generate a unique, well-formed value for each new logical request, send it on the header, and record it alongside the call. When the response carries an ID, read it and follow the agreed precedence, typically adopting the gateway value or keeping the one the client sent. Decide the retry convention up front: reusing the same ID keeps the attempts tied together, while a new ID distinguishes each attempt, and either is valid if the team applies it consistently. A desktop API workspace lets you set these conditional responses, inspect the recorded headers, and assert on each case in the same place. Powerduck works this way, so the header exchange is a fixed set of local responses.

Honest limits

A local mock runs in the desktop main process and verifies one hop of header handling, not the full behavior of a distributed trace. It does not model propagation across many real services, sampling, or a trace standard such as W3C Trace Context, and it does not prove the live services forward the ID or that the logging backend can actually query it. The echo and precedence rules are ones you define, while a real downstream service can truncate, replace, or drop the value. Confirm end-to-end propagation with a small set of genuine requests. Importing a schema does not deploy a backend, and a passing mock run is not evidence about production observability.

Frequently asked questions

What is the difference between a correlation ID and a trace ID? A correlation ID joins logs for one request across services. A trace ID is part of a tracing standard and often carries span and sampling context. They overlap, but the correlation ID is the simpler header-level value.

Who should generate the ID? Either the client or the entry gateway. If the client sends one, the gateway keeps it; if not, the gateway assigns one and returns it. The mock covers both paths.

What happens when the response has no ID? The client should keep the ID it sent and continue without crashing, rather than failing the request or logging an empty value.

Should a retry reuse the same ID? It depends on the convention. Reusing it links the attempts; renewing it separates them. Pick one and apply it consistently so the logs stay interpretable.

The bottom line

Use a correlation ID to connect the logs for a request, and mock the header exchange to prove the client handles the generated, gateway-assigned, mismatched, and missing cases. Keep generation and reading in one layer, define the precedence and retry rules explicitly, and verify real cross-service propagation separately. For more on this approach, see the API mocking guide hub.