DEV Community: Prasad MK

An LLM Judge and a Ledger: SwarmSentinel and SwarmSheriff next to Google's Agent Gateway

Prasad MK — Tue, 13 Oct 2026 02:50:04 +0000

In my previous post I looked at AWS AgentCore Gateway, which checks each tool call against a Cedar policy, and argued that it needs something beside it that remembers the session. Google announced its own Agent Gateway at Gemini at Work 2026. I expected the same kind of per-request firewall, but the docs surprised me: Google's gateway does look at history, by handing it to a language model.

Below I go through what Google's stack checks, run one multi-agent attack through it, and look at where my two open-source projects, SwarmSentinel (tool path) and SwarmSheriff (model path), would fit. What I say about Google's product comes from its public docs as of October 2026. Several of the pieces are in Preview, so details may change. The integration I describe at the end is a design, not something I've built.

What Google's gateway checks

Agent Gateway is the network entry and exit point for agent traffic: user to agent, agent to tool, agent to agent. It passes through HTTP traffic, including MCP and A2A, and for MCP it parses the request so policies can match on attributes such as the tool name. Around it sit several layers, and they work in quite different ways.

The base layer is identity. Each agent gets a SPIFFE ID, and mTLS and DPoP are enforced. Outbound calls are denied unless an IAM policy grants the agent access to the destination and the destination is registered in Agent Registry. IAP enforces this, and IAM conditions can see MCP attributes like mcp.toolName and whether a tool is read-only. All of this is deterministic and default-deny, and I think it's the strongest part of the design.

Model Armor adds content filters that scan prompts and tool responses for prompt injection and sensitive data. Like any classifier, it has to recognise the attack in the text.

Semantic Governance Policies are the new idea. Administrators write natural-language constraints. When the model proposes a tool call, the gateway sends the proposal, the constraints, the tools manifest and the chat history to a policy engine. It checks two things: whether the call matches the user's intent, and whether it complies with the constraints. A denied call is removed from the model's response before the agent sees it. The docs are frank that the engine "uses an LLM to implement natural language policies" and that "verdicts may not be accurate". They also say it isn't a substitute for IAM, rate limits or network controls. The feature is in Preview.

Finally, every policy type is enforced through authorization extensions managed with Service Extensions, and you can delegate decisions to your own engine the same way. I come back to that hook at the end.

Semantic Governance can catch things a rule engine can't express, and I don't want to undersell that. An agent can do something no rule forbids that the user never asked for: emailing the whole customer list when asked to fix one record, for example. A model reading the conversation may notice. SwarmSentinel won't, because the provenance of every input is clean.

The invoice relay, run through Google's layers

Here is the scenario I used last time, from SwarmSentinel's synthetic suite. An orchestrator hands invoice processing to a parser agent. The parser reads a vendor invoice that contains instructions to move money. The parser has no payment rights, so it messages the ledger agent, which does, and asks it to "settle the invoice as instructed".

Take each layer in turn as the ledger agent proposes payments.transfer.

IAP and the registry allow it, because the ledger agent is registered and holds the role for the payments tool. That's correct, since it does.

Model Armor gets a real chance when the parser reads the invoice, because it scans tool responses. If the injected text looks like an injection, it's flagged there. A paraphrased instruction, one split across two documents or one that reads like ordinary vendor boilerplate may get through. The message the ledger agent receives contains no malicious text at all.

Semantic Governance is the interesting one. It checks whether the transfer matches the user's intent. For the ledger agent, the "user" in that conversation is the parser agent, and the parser's request is the transfer. As far as I can see, the call matches the intent of whoever asked. This is what bothers me about judging intent in a multi-agent system: the upstream agent's intent is the thing the attacker took over. The docs don't say whether chat history spans more than one agent. If it only covers the ledger agent's own conversation, the poisoned invoice never appears in what the judge reads. If it does reach back to the original document, the judge is now reading attacker-written text while deciding, which is the same position the compromised agent was in.

I'm not claiming the judge would fail. A good constraint ("transfers must match an approved purchase order") could catch this case, and an LLM might notice that something is off. Still, the decision depends on a model's reading of text, and the attacker wrote some of that text.

What deterministic session state adds

SwarmSentinel implements ASP (Agent Security Policy), a policy model I proposed in a preprint. It's a proposal, not a standard. It handles the invoice relay without reading any of the text. The parser read content marked untrusted, so its context is tainted. The taint follows the message to the ledger agent, and the policy withdraws payment capability from tainted contexts. The transfer is dropped, and the flight recorder logs the rule that fired and the event where the taint started. How the invoice was worded makes no difference, because no step in that decision reads the wording.

This is the main difference from Semantic Governance. The decision rests on where content came from, and the attacker has no way to argue with provenance. The other ASP controls work the same way:

Admission and lineage: a spawned agent's scope has to sit inside its parent's, at runtime, with a depth cap. Agent Registry says which agents may exist. Lineage says what a particular child may do, given who spawned it.
Write caps per shared resource and step budgets per agent, which the Semantic Governance docs explicitly leave to other controls.
Repeated-intent throttling that counts distinct agent IDs, for an attack that retries through fresh identities.
Graph tripwires (injection spreading to several agents, delegation loops, coordinated writes) that revoke an agent and everything it spawned.
A report-only mode for asking "what would this policy have done?" I've used it to replay records from the AI Village dataset (AI Digest, "AI Village dataset", 2026). Those runs are counterfactual: they show what the policy would have flagged, not anything it stopped.

You pay for this with coarseness. Taint blocks some legitimate actions, because it doesn't try to tell a harmless request from a malicious one. An LLM judge makes the opposite trade, which is why I'd run both.

The model path: Model Armor and SwarmSheriff

Google's gateway sits between the agent and its model as well as its tools, so Model Armor can screen outbound prompts for sensitive data. SwarmSheriff covers the same exit for agents that don't run on Google Cloud, such as coding agents on a laptop calling Anthropic or OpenAI. It's a single Go binary acting as a loopback proxy. It classifies each request against data classes the company defines, checking source labels and fingerprints of known documents before patterns and keywords, and calls a classifier only for what's left ambiguous. Report mode is the default, and it never stores request bodies.

The part I think is different is the link to SwarmSentinel, which is built and tested. An ASP policy can label a context when a tool call touches something sensitive, for example any argument pointing at ~/.aws or any call to the CRM. When that agent next calls its model, SwarmSheriff resolves the agent's engine-issued credential and gets its labels. A data class can then hold the request whatever the prompt says. Revoked agents can't reach their model either. A content scanner, Model Armor included, has to find the secret in the prompt. A label says this agent touched the secret, so the prompt is held even if the agent has paraphrased or encoded it.

	Google Agent Gateway layers	SwarmSentinel	SwarmSheriff
Decides by	IAM rules, classifiers, an LLM judge	Provenance, lineage and counters	Labels, fingerprints, patterns
Memory	Chat history (Semantic Governance)	Session graph across all agents	Labels from SwarmSentinel
Deterministic	IAM yes; Model Armor and Semantic Governance no	Yes	Yes for the first three steps
Where it runs	Google Cloud, managed	Anywhere; Python engine	Anywhere; loopback
Maturity	Product, parts in Preview	Research prototype	Research prototype

How I'd connect them

Agent Gateway lets you add a custom authorization engine as a Service Extension, bound with an authorization policy. In Google's codelab, IAP is attached with the REQUEST_AUTHZ profile, which runs once per request at the headers stage, and Model Armor with CONTENT_AUTHZ, which streams body events. A SwarmSentinel extension would need the body to see tool arguments, so it would most likely use the content profile, or rely on the gateway's parsed MCP attributes if those reach custom engines.

The shape would follow the codelab's bindings:

# Sketch only. Field values follow the codelab; check the Service Extensions
# reference for how a self-hosted backend is referenced and what it receives.
name: swarmsentinel-authz
service: <your SwarmSentinel extension backend>
failOpen: false
timeout: 1s

{
  "policyProfile": "CONTENT_AUTHZ",
  "action": "CUSTOM"
}

The extension would turn each MCP call into a proposed action for SwarmSentinel's evaluate endpoint. The agent ID would come from the SPIFFE identity, the target from the MCP tool name, and a span ID from the request. A denial would be returned as the gateway's deny. The codelab sets failOpen: true on its extensions, which is fine for a demo, but I'd want a security control to fail closed. SwarmSentinel would also have to speak the extension protocol, while today it's a REST service. I haven't verified the callout contract, so treat this as the shape of the work, not a recipe.

The order of extensions matters as well. You'd want SwarmSentinel to see calls that IAP denies, because repeated probing counts toward its budgets. Whether a custom engine sees a request that an earlier policy has already denied is something I'd test first.

Limits

SwarmSentinel and SwarmSheriff only see traffic routed through them, and SwarmSentinel is a code boundary in one process, not a tamper-proof control. Google's gateway enforces at the network edge with attested identities, which my prototypes can't match. Model Armor's classifiers are far better at spotting injected text than my regex detectors. And SwarmSheriff's evaluation set is still too small to quote accuracy numbers.

I like a lot of Google's design, especially default-deny connectivity and identity on every hop, and I appreciate that the docs say plainly the LLM judge is probabilistic. My concern is narrower. In a swarm, the intent behind a tool call usually comes from another agent, and that agent may already be compromised. Tracking provenance and lineage gives you a reason to refuse that holds however the request is worded, and an LLM judge can't provide that.

Both repos SwarmSentinel (tool path) and SwarmSheriff (model path) are MIT licensed. If you're on Google Cloud and want to try the extension route, I'd like to hear the feedback.

What Amazon Bedrock AgentCore Gateway Can't See

Prasad MK — Tue, 13 Oct 2026 02:09:52 +0000

Amazon Bedrock AgentCore Gateway is a sensible place to put agent controls. Every tool call an agent makes through it can be checked against a Cedar policy before the tool runs, the decision is deterministic and lands in an audit log, and Lambda interceptors let you run your own code before and after each call. If your agents only ever needed rules like "the support agent may call refund for amounts under 100", you could stop there.

The incidents I keep finding in multi-agent traces don't look like that. They're sequences of calls that each pass on their own, and a per-request gateway leaves two gaps that let them through. I've been building two open-source projects to cover those gaps: SwarmSentinel on the tool path and SwarmSheriff on the model path. Both are research prototypes. The AgentCore wiring below is a design I'm proposing, not something I've shipped, and I'll be specific about which parts exist.

Gap 1: the gateway has no memory of the session

Here is a scenario from SwarmSentinel's synthetic test suite. An orchestrator delegates invoice processing to a parser agent. The parser reads a vendor invoice that contains instructions to move money. It can't pay anyone itself, so it messages the ledger agent, which does hold payment rights, and asks it to "settle the invoice as instructed".

Now look at that last call from the gateway's position. The principal is the ledger agent. The action is payments.transfer. The ledger agent is allowed to make transfers, the amount is within its limit, and the arguments look normal. A Cedar policy evaluated against principal, action, resource and request context says yes, which is the right answer for the request it was given. Nothing in that request says the instruction came from a document an attacker controlled, two hops back, through a different agent.

You can't fix this by writing a better Cedar rule, because the information isn't in the request. It lives in the history of the session: who read what, who talked to whom, who spawned whom. A content classifier on the request doesn't help much either. "Settle the invoice as instructed" is not a malicious string.

I see the same shape elsewhere. A child agent is spawned with its own identity and broader rights than its parent ever had. Five fresh agent IDs each try the same action once, so no per-agent rate limit fires. A shared wiki gets forty small edits in a minute, each one harmless. A parent agent is shut down for misbehaving while its children carry on with valid credentials.

Gap 2: the model API is a second exit

The gateway sees tool calls. It doesn't see what an agent sends to its model provider. An agent that has just read ~/.aws/credentials or a customer table through a perfectly authorised tool call can paste that into its next prompt, and the request leaves through a path no tool policy covers. For agents on AgentCore Runtime calling Bedrock that path stays inside AWS, but plenty of agents call Anthropic, OpenAI or OpenRouter directly, and coding agents on laptops do it all day.

SwarmSentinel: integrity on the tool path

SwarmSentinel implements ASP (Agent Security Policy), a policy model I proposed in a preprint. ASP is a proposal, not an adopted standard. The policy is a JSON file, and the gateway checks each proposed action in a fixed order: lineage and admission, explicit deny list, default-deny allowlist, argument constraints, network scope, content-trust contamination, then write caps and repeated intent.

For this post, the interesting parts are the ones that keep state.

Lineage comes first. An agent has to be admitted, either by a spawn the gateway allowed or by a registry entry. A child's scope must sit inside its parent's, and delegation depth is capped.

Taint is what stops the invoice relay. When an agent reads content marked untrusted, its context is contaminated. The policy can take specific capabilities away from contaminated contexts (payments and writes, say) and can pass the taint along with the messages that agent sends. The ledger agent got its instructions from a tainted context, so it loses payment rights for that context. Taint can also be tracked per (agent, task) pair, which means a long-lived agent isn't poisoned forever by one bad web page. Only the owner can open a new task. The agent can't.

The rest are counters and graph checks. There are write caps per shared resource per time window, a step budget per agent, and repeated-intent throttling that counts distinct agent IDs. Graph tripwires look for reciprocal message clusters, coordinated writes, fan-out, injection reaching several agents and delegation loops. When one fires it can revoke an agent, and the revocation covers everything that agent spawned.

Every decision goes to a flight recorder with the rule that fired and the taint origin, and a dashboard replays the session as a graph. There's also a report-only mode that answers "what would this policy have done?" without blocking anything. I've used it to replay records from the AI Village dataset (AI Digest, "AI Village dataset", 2026), where real agents from several labs work together in one environment. Those runs are counterfactual: they show what the policy would have flagged or blocked, not anything it stopped.

Overhead is small because the checks are dictionary lookups and set operations. On an Apple M2 with 20 agents, the guard measured p50 0.024 ms in-process and 0.39 ms over localhost HTTP.

SwarmSheriff: confidentiality on the model path

SwarmSheriff is a single Go binary. One of its jobs is a loopback reverse proxy for LLM APIs: you set ANTHROPIC_BASE_URL or OPENAI_BASE_URL to point at it, and it classifies each outbound request against data classes the company defines in a policy file (source code, customer data, credentials, legal material). Each request gets a verdict of allow, hold or block (a fourth verdict, translate, is still a stub). Report mode is the default, and enforcement is switched on per class.

I picked the classification order on purpose. Source labels come first, then fingerprints of known documents, then patterns and keywords, and only then an optional classifier for whatever the deterministic steps left ambiguous. Clever wording doesn't get past the first three steps, because none of them is interpreting the text. It never stores request bodies, and a hosted classifier sees only features of the request unless a class opts in to raw content.

How the two compare

	SwarmSentinel	SwarmSheriff
Path it watches	Tool calls, spawns, agent messages	Requests to model APIs
Question it answers	Should this agent be allowed to do this, given the session so far?	Is company data about to leave in this prompt?
State	Lineage, taint, labels, rate windows, revocations	Mostly per request, plus labels pulled from SwarmSentinel
Policy	ASP JSON	Data-class JSON
Default mode	Enforce (synthetic), report-only (replay)	Report
Runs as	Python engine (FastAPI), MCP proxy, in-process guard	Go binary on loopback
Security property	Integrity	Confidentiality

The link between them

The two share one view of each agent, and this part is built and tested. An ASP policy can declare confidentiality classes, for example "any call whose arguments touch ~/.ssh or ~/.aws labels the context credentials", or "any call to mcp:crm:* labels it customer-data". Labels only accumulate, are inherited by spawned children and can follow messages.

When the agent then calls its model, the request carries its engine-issued credential. SwarmSheriff resolves that credential with SwarmSentinel and gets back the agent, its task, its labels, and whether it is tainted or revoked. A data class in SwarmSheriff can block on the label asp:credentials whatever the prompt text says. A revoked agent's model requests are refused, so revocation reaches the model path as well. An unknown credential or an unreachable engine fails closed. SwarmSheriff posts its decision back, and SwarmSentinel records it as an out-of-band row, so one timeline shows both paths.

The trade-off is that labels describe what a context has touched, not what a given prompt contains. A labelled agent sending a harmless request gets held anyway. I think that's the right default for credentials and the wrong one for most other classes, which is why it's set per class.

Wiring it into AgentCore

I wouldn't replace the Gateway's Cedar policies. Cedar handles the stateless rules well, and AWS can run automated reasoning over a Cedar policy set, which my JSON validator can't do. Cedar should decide whatever one request is enough to decide, and SwarmSentinel should get the rest.

Option A: a request interceptor

AgentCore Gateway can run a Lambda before each tool call. That Lambda sends the proposed call to SwarmSentinel's evaluate endpoint and denies the call if the engine says no. A rough sketch:

import json
import os
import urllib.request

ENGINE = os.environ["SWARM_ENGINE_URL"]
SESSION = os.environ["SWARM_SESSION_ID"]
# Tools whose results are untrusted content (web fetches, inbound email, uploaded files).
UNTRUSTED = {"web.fetch", "email.read", "files.read_upload"}


def handler(event, context):
    # Field names below are placeholders: map them from the interceptor event schema in the AgentCore docs.
    tool = event["toolName"]
    call = {
        "agentId": event["agentId"],
        "action": "tool",
        "target": tool,
        "spanId": event["requestId"],
        "detail": json.dumps(event.get("arguments", {}))[:4000],
        "reads": "untrusted" if tool in UNTRUSTED else "",
    }
    req = urllib.request.Request(
        f"{ENGINE}/api/swarm/sessions/{SESSION}/evaluate",
        data=json.dumps(call).encode(),
        headers={"Content-Type": "application/json", "Authorization": f"Bearer {service_token()}"},
    )
    with urllib.request.urlopen(req, timeout=2) as resp:
        verdict = json.load(resp)
    if not verdict["allowed"]:
        return deny(f"{verdict['rule']}: {verdict['reason']}")
    return allow(event)

service_token, allow and deny depend on your identity setup and the interceptor's response contract, so I've left them out. If the engine times out, the Lambda should fail closed, the same way SwarmSheriff does. In AWS's examples the request interceptor runs before the Cedar evaluation, so SwarmSentinel also sees calls that Cedar will go on to deny. I want that, because repeated attempts and probing count toward step budgets and repeated-intent limits.

Option B: SwarmSentinel's MCP proxy as a Gateway target

SwarmSentinel already exposes an MCP endpoint that fronts downstream MCP servers, with credentials it mints per agent. Registering that endpoint as an MCP target puts ASP behind the Gateway's inbound auth with no Lambda. The catch is identity: lineage depends on each agent presenting its own credential. If the Gateway forwards every agent under one outbound identity, SwarmSentinel sees one agent and lineage stops working. Check this before choosing B.

The model path and hosting

For agents in AgentCore Runtime, SwarmSheriff would run as a second process in the same container, with the agent's base URL pointed at loopback. I haven't tested Bedrock's SigV4-signed traffic through it yet. A proxy that changes the host can break the signature, so for now this is the more natural fit for agents calling Anthropic, OpenAI or OpenRouter endpoints directly.

SwarmSentinel itself keeps one shared state per swarm, so I'd run it as a long-lived service (ECS or App Runner, say) rather than inside AgentCore Runtime, where each session gets its own microVM. Durable sessions currently use a SQLite journal, which would need EFS or a move to DynamoDB.

What this doesn't solve

SwarmSentinel is a code boundary in one process, not a tamper-proof control. It only sees calls routed through it, and so does SwarmSheriff, which is why the repo includes a confinement lab that measures how agents can get around the proxies. Taint is coarse: it costs you some legitimate actions in exchange for not having to recognise the attack. SwarmSheriff's evaluation corpus is still small, so I don't quote accuracy numbers for it. And the engine authenticates callers with Clerk JWTs today, so a service credential for the interceptor is real work, not a config change.

You need per-request authorization whatever else you run, and AgentCore's is solid. But the failures I keep seeing in multi-agent traces build up over several calls and agents, and some leave through the model API instead of a tool. Catching those means something has to remember the swarm. I'd rather gateways did that themselves. For now it has to run beside them.

Both repos SwarmSentinel (tool path) and SwarmSheriff (model path) are MIT licensed. If you try the interceptor pattern, I'd like to hear where it breaks.

What the GitHub Outage Taught Us About Authentication Retries

Prasad MK — Tue, 18 Aug 2026 17:25:03 +0000

On August 17, 2026, GitHub went through a rough day. For nearly eight hours, large parts of the platform struggled. API requests, Actions, pull requests, Issues, webhooks, and Copilot all felt the impact. Error rates hit around 20% on web and API traffic. Archive and raw content downloads failed about half the time.

The official status page eventually said engineers had identified a problematic component and applied fixes. Later updates mentioned something more interesting: they partially disabled authentication-token retries because those retries were making the problem worse.

That detail is worth sitting with.

How retries turn a problem into a cascade

Most systems retry failed requests. It is a reasonable default. When an authentication call times out or returns an error, the client tries again. Under normal conditions this improves reliability.

Under stress it can do the opposite.

Here is the basic loop:

Something degrades the authentication service.
Clients start receiving failures or slow responses.
Those clients retry.
The extra traffic increases load on the already struggling service.
More requests fail, which triggers more retries.

The system enters a feedback loop. The retries that were meant to help become the main source of pressure. Engineers at GitHub observed this happening and chose to limit the retries so the authentication layer could recover.

This pattern is sometimes called a retry storm or a metastable failure. Once the system is in that state, removing the original trigger is not always enough. The amplified traffic keeps the service saturated.

Different approaches to the same problem

Microsoft and Google have spent years designing authentication systems that try to avoid this exact trap.

Microsoft Entra ID includes a backup authentication service that runs on separate infrastructure. When the primary system is degraded, the backup can continue serving tokens for existing sessions. Their client libraries (MSAL) are also written to respect rate limits and avoid aggressive retry loops. Continuous Access Evaluation lets them issue longer-lived tokens while still supporting near real-time revocation.

Google tends to favor short-lived access tokens paired with disciplined client behavior. Official client libraries use exponential backoff with jitter and only retry certain classes of transient errors. Permanent failures are not retried endlessly. Token validation often relies on cached public keys, which reduces live dependency on the identity service.

These systems were built with planetary scale and known failure modes in mind. Authentication is treated as critical infrastructure that needs its own isolation and fallback paths.

GitHub's architecture has historically been more tightly coupled. When authentication slows down, many other services feel it quickly. The recent outage made that coupling visible.

Why GitHub does not simply switch to Microsoft's identity stack

GitHub is owned by Microsoft, so the question comes up often: why not just use Entra ID and MSAL?

There are practical reasons.

GitHub was kept relatively independent after the acquisition. That independence mattered to the developer community. Moving the entire identity system onto Entra ID would have changed the product in ways that go beyond technical convenience.

The user models are also different. GitHub supports personal accounts, open source projects, and enterprises. Entra ID is optimized for organizational identity. Enterprises can already choose tighter integration through SAML, SCIM, and Enterprise Managed Users. Making that the only path would break the model that most individual developers use.

MSAL itself is a client library. It helps applications acquire tokens from Entra ID. It does not replace the need for GitHub to issue, validate, and manage its own tokens for Git operations, GitHub Apps, Actions, and the API. Solving the retry cascade requires changes inside GitHub's own authentication services, not just a different client library.

What this leaves us with

The outage was disruptive. It also made a common systems problem concrete. Retries are useful until they are not. When the thing you are retrying is the authentication layer that everything else depends on, the cost of those retries can rise quickly.

Good authentication systems plan for this. They limit amplification, provide fallback paths, and treat token services as critical infrastructure rather than just another internal dependency.

GitHub will likely continue improving its own stack. The rest of us can take the simpler lesson: when designing retry logic, ask what happens when the dependency is already under pressure. The answer is often more important than the happy path.

The Pizza-Builder Pattern: Stop Guessing, Start Building a Menu

Prasad MK — Sun, 16 Aug 2026 23:08:10 +0000

Ask an AI to "write a novel" and you get one of two outcomes, and both annoy me.

It guesses. Genre, protagonist, tone, all invented on the spot. You read the output, realize it missed, and burn a full generation cycle starting over.

Or it interrogates you. One clarifying question at a time, four turns deep, until you've typed more than the request would've taken to write yourself.

I got tired of both, so I built a Claude skill called pizza-builder. One screen, one menu, done.

Why a pizza counter

Build-your-own works for a reason most people never stop to think about. Someone already figured out which decisions matter (base, sauce, cheese, toppings) and narrowed each one to a short list of good options.

You don't invent a pizza from nothing. You compose one from a menu somebody designed well.

I applied that same logic to AI requests. Turns out the fix isn't a better prompt. It's a better structure.

The rules that make the menu work

Find the real decision points, not generic categories.
Weak: "Genre: Fantasy / Sci-fi / Mystery / Other"
Sharp: "Cozy small-town mystery / Hard sci-fi with a ticking clock / Literary fiction about a family secret / Epic fantasy with a political core"

Specific options let you react instantly. Generic labels just restate the question back at you, and nobody's ever been helped by that.

Match the question to its actual shape. Not every decision is "pick one of four." I count seven shapes that actually show up:

Toggle for a plain yes/no fact, like whether this needs to survive a reboot
Radio for a short list where picking one rules out the rest
Dropdown for six or more options, so the screen stays scannable instead of a wall of buttons
Skip option for anything an expert might actively argue against, like not bothering with a cache
Slider for a spectrum, "70% tomato, 30% pesto," instead of a fake list pretending to be a continuum
Checkbox for anything genuinely additive
Free text for a number, name, or path, because forcing that into buttons just looks tidy and helps nobody

Give every option a real "you decide." Not a shrug. A stated pick, with a reason, the way a good server tells you "I'll bring you the fish, it's better tonight" instead of handing the decision right back to you.

Synthesize, don't summarize. Four specific picks need to produce something that couldn't exist without all four combined. If the output just mentions each keyword once, the synthesis failed, not the menu.

The failure that taught me the hard rule

A sunset-image picker I built once generated a prompt that said "no horizon element" in one line and "bands from horizon to zenith" two lines later. Code ran fine. Buttons worked. The image came out as flat color stripes, because the prompt itself was fighting with itself and nobody caught it.

Same picker, different bug: it generated this line and shipped it straight to the user. "Can you write me a detailed, ready-to-use prompt for this, and flag anything I should reconsider?"

Every automated check passed. Syntax clean, buttons functional, nothing throwing errors. And the output was a question thrown back at the person who'd already answered five questions to get there.

That one stung more than the horizon bug, honestly, because it revealed something worse: passing tests tells you the code runs, not that the content does anything. So now every generated prompt gets read against three failure shapes before it ships, not just executed:

Missing anchor: nothing stated that makes the subject actually recognizable
Self-contradiction: one line quietly undercutting another
Punt-back: a question aimed at the reader instead of an answer

The mechanical test for the last one is almost embarrassingly simple: does the output contain "can you," "write me," or a question mark pointed at the user? If so, it's asking for the work instead of doing it, and I'd rather catch that before someone else does.

When to use this, and when not to

Route on structure, never on topic. I learned this the hard way, too. Any list of "domains this applies to" is guaranteed to miss the next domain, so I stopped keeping one.

Use the menu when:

There's no single obvious interpretation
A wrong guess costs a real redo: a full draft, a rendered image, a deck
The request has three or more independent decisions, not just one missing detail

Skip it when:

There's a clear default (a headline: just write two or three and move on)
The task is cheap to redo (a tweet, a rename)
You want an actual opinion, not a menu dressed up as one ("Python or JavaScript?")

Get the skill

The full SKILL.md lives here: https://github.com/prasad-m-k/claude-skills/tree/main/pizza-builder

Claude: Add it as a custom skill. Download the file, upload it in Claude's skill settings (or drop it into your skills folder on Claude Code), and it triggers on its own when a request fits.

ChatGPT: No native skill system, but the pattern still works fine. Create a Custom GPT or Project and paste the SKILL.md content into its instructions field. It'll follow the same dimension-and-shape logic when it builds a clarification screen.

Gemini (Gems): Same move. Create a Gem, paste the file into its instructions, and it runs the same rules: find the real decision points, classify their shape, build the menu fresh every time.

None of these platforms need the exact file structure. The value sits in the rules, not the format, so paste it in wherever the model keeps its persistent instructions and it holds.

The real takeaway

Early versions of this skill shipped a fixed template with pre-built dimension renderers. It only covered three of the seven shapes. Every new request had to get checked against that file's limits, and that check kept getting skipped, because who remembers to check a limit they don't know exists.

I didn't fix it with a bigger template. I deleted the template. Every menu gets built fresh now, against whatever the current rules say, every single time. Nothing cached, so nothing drifts out of sync while I'm not looking.

If I take one thing away from building this, it's that a good clarification system isn't a form with extra fields bolted on. It's a habit of figuring out, fresh, every time, which decisions actually matter and how each one deserves to be asked.

Testing Kubernetes NetworkPolicies Before They Hit Production

Prasad MK — Thu, 06 Aug 2026 16:36:14 +0000

You write a NetworkPolicy, apply it to a namespace, and move on. Nobody tests it. Nobody validates it against the actual traffic the service depends on. It sits there until a deploy breaks something, and the first sign of trouble is a pod that can't reach its database.

Most teams treat NetworkPolicies like documentation instead of code. Someone writes one, a reviewer skims the YAML, it gets merged, and it goes straight onto the cluster. There's no equivalent of a unit test for "can pod A talk to pod B on port 5432." That gap is where the 2am pages come from.

Here are three tools that close it, plus a CI setup that catches a bad rule before merge instead of during an incident.

Why NetworkPolicy bugs are hard to catch by reading YAML

A NetworkPolicy looks simple. Selectors, ports, ingress and egress rules. But three things make them easy to get wrong:

Default-deny is implicit. The moment you add any NetworkPolicy that selects a pod, all traffic not explicitly allowed is blocked. Add a policy meant to restrict one path, and you silently cut off three others that had no policy governing them before.
Selectors don't validate against real labels. A typo in app: payment-svc versus the pod's actual app: payment-service produces a policy that matches nothing. kubectl apply succeeds. The policy does nothing. You find out when traffic that should be blocked isn't, or worse, traffic that should be allowed is.
Multiple policies combine additively, and nobody traces the union. If three teams each own a policy touching the same namespace, the effective ruleset is the union of all three. Reading one YAML file tells you nothing about the combined effect.

None of this shows up in a code review unless the reviewer mentally simulates traffic flow across every policy in the namespace. That doesn't scale past a handful of services.

Tool 1: kubectl-np-viewer

kubectl-np-viewer is a kubectl plugin that renders the effective policy for a pod or namespace, not just the raw YAML. Instead of asking you to trace intersections across five NetworkPolicy objects by hand, it computes the resolved ingress and egress rules and prints them as a readable table.

Install:

kubectl krew install np-viewer

Run it against a namespace:

kubectl np-viewer -n checkout

Output shows, per pod selector, which ingress sources and egress destinations are actually permitted after all policies in the namespace are combined. This is the tool you reach for when someone asks "can the checkout service reach Redis" and nobody wants to trace four YAML files to find out.

Use it locally before you open a PR. If the effective policy doesn't match what you intended, fix the policy before it's even committed.

Tool 2: netpol-analyzer

netpol-analyzer (from the np-guard project) goes further. It's built for static analysis and CI, not just interactive inspection. It can:

Diff two policy sets and show exactly what connectivity changed
Detect policies that have no effect (selector matches zero pods)
Detect redundant rules already covered by a broader rule
Generate a full connectivity map for a namespace or cluster snapshot

The diff mode is the one that matters most for CI. Given the policies currently in the cluster (or a manifest directory) and the policies in your PR branch, it tells you the delta in allowed connections:

netpol-analyzer diff \
  --dir1 manifests/main \
  --dir2 manifests/pr-branch \
  --output text

A sample result looks like this:

Changed connections between main and pr-branch:
Denied connection: checkout-service -> inventory-db:5432 (was allowed, now denied)
Added connection: checkout-service -> new-cache:6379 (newly allowed)

That first line is the bug that takes down a service on merge, every time. Someone tightens egress for one workload and doesn't notice it also strips access to the database three lines down. A reviewer scanning YAML won't catch it. netpol-analyzer diff puts it right at the top of the output, which is the whole point of running it.

Run this as a required check on any PR that touches networkpolicy manifests. Treat a "Denied connection" line touching a live dependency as a blocking failure, not a warning.

Tool 3: Cilium connectivity test

If your cluster runs Cilium as the CNI, cilium connectivity test gives you something the first two tools don't: live traffic verification against a real cluster, not just static analysis of YAML. It deploys a set of test pods, generates traffic between them under different policy conditions, and reports pass/fail based on actual packet delivery.

cilium connectivity test --test-namespace cilium-test

This catches things static analysis can't, like a CNI-level misconfiguration, an eBPF program that didn't reload correctly after a policy update, or a policy that's syntactically fine but doesn't behave as expected once the dataplane applies it. Static tools tell you what the policy should do. This tool tells you what actually happens on the wire.

You don't want to run the full Cilium test suite on every PR. It needs a live cluster and takes minutes, not seconds. Reserve it for a staging environment, run on merge to main or on a schedule, not on every commit.

Building this into a CI pipeline

The goal is a pipeline where a NetworkPolicy change gets validated automatically, and a rule that breaks an existing dependency fails the build before a team member has to notice. Here's a structure that works with GitHub Actions, adjust for GitLab CI or Jenkins as needed.

Stage 1: Static validation on every PR touching NetworkPolicy manifests

name: netpol-validation

on:
  pull_request:
    paths:
      - 'manifests/**/networkpolicy*.yaml'
      - 'manifests/**/networkpolicy*.yml'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install netpol-analyzer
        run: |
          curl -sLo netpol-analyzer https://github.com/np-guard/netpol-analyzer/releases/latest/download/netpol-analyzer-linux-amd64
          chmod +x netpol-analyzer
          sudo mv netpol-analyzer /usr/local/bin/

      - name: Checkout base branch manifests
        run: |
          git worktree add /tmp/base origin/main -- manifests

      - name: Diff connectivity
        run: |
          netpol-analyzer diff \
            --dir1 /tmp/base/manifests \
            --dir2 manifests \
            --output text | tee netpol-diff.txt

      - name: Fail on denied connections to known dependencies
        run: |
          if grep -q "Denied connection" netpol-diff.txt; then
            echo "::error::NetworkPolicy change removes existing connectivity. Review netpol-diff.txt."
            exit 1
          fi

      - name: Check for ineffective policies
        run: netpol-analyzer eval --dir manifests --output text --fail-on-empty-selectors

This job runs on every PR that touches a policy file. It fails the build if the diff shows a connection being removed. That's the check that would have caught the checkout-to-inventory-db example above, before merge instead of during an incident.

Stage 2: Live connectivity test on merge to main

name: netpol-live-validation

on:
  push:
    branches: [main]
    paths:
      - 'manifests/**/networkpolicy*.yaml'

jobs:
  live-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up cluster access
        run: |
          echo "${{ secrets.STAGING_KUBECONFIG }}" > kubeconfig
          echo "KUBECONFIG=$(pwd)/kubeconfig" >> $GITHUB_ENV

      - name: Apply policies to staging
        run: kubectl apply -f manifests/ -n staging

      - name: Install cilium CLI
        run: |
          curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
          tar xzf cilium-linux-amd64.tar.gz
          sudo mv cilium /usr/local/bin/

      - name: Run connectivity test
        run: cilium connectivity test --test-namespace cilium-test --timeout 10m

This second stage runs only on merge, against a real staging cluster, and catches dataplane-level problems that static diffing can't see. It's slower and it needs a live cluster, so it doesn't belong on every PR, but it's the last line of defense before the same policies get promoted to production.

Stage 3 (optional): scheduled drift check

Policies drift from what's in git if anyone applies changes directly to the cluster. A nightly job that diffs the live cluster state against the manifests in main catches that:

on:
  schedule:
    - cron: '0 6 * * *'

Run netpol-analyzer diff between a live export of the cluster's policies (kubectl get networkpolicy -A -o yaml) and the manifests in git. Any difference means someone bypassed the pipeline, and you want to know before it causes a support ticket six weeks later when nobody remembers making the change.

What this catches in practice

A rule of thumb for interpreting output from netpol-analyzer diff: a removed connection touching anything with active traffic is a merge blocker. A newly allowed connection is worth a second look but rarely urgent. An empty-selector warning means someone wrote a policy that does nothing, which is a silent no-op rather than a security control, and it should get fixed even though it won't break anything today.

None of this replaces code review. It replaces the part of code review that asks us to mentally trace traffic across a namespace, which people are genuinely bad at and a diff tool is genuinely good at. Static check on every PR, live test on merge, drift check on a schedule. Do that and the first time you hear about a bad NetworkPolicy rule stops being an incident channel at 2am.

Object-Oriented Abstraction Didn't Disappear. It Moved Into Prompts

Prasad MK — Thu, 06 Aug 2026 06:10:56 +0000

You call car.start() and you don't care if it's a V8 or an electric motor underneath. That's abstraction: a stable interface, a swappable implementation.

Prompt engineering usually skips that. Most prompts are one long string doing the job of a method signature, a method body, and a config file at once. Swap the model and the string that worked yesterday quietly stops working today. No error, no warning, just a slightly wrong answer.

DSPy, a framework out of Stanford, applies OOP-style structure to this problem. Here's how it works.

Separate the contract from the wording

In DSPy you don't write a prompt. You write a Signature, an input/output contract with zero wording attached:

class GenerateAnswer(dspy.Signature):
    """Answer questions based on the given context."""
    context = dspy.InputField()
    question = dspy.InputField()
    answer = dspy.OutputField()

That's the interface. Given context and a question, produce an answer. It reads like a method signature: answer(context, question) -> answer.

Then you attach a strategy that satisfies it:

class RAG(dspy.Module):
    def __init__(self):
        self.generate = dspy.ChainOfThought(GenerateAnswer)

    def forward(self, context, question):
        return self.generate(context=context, question=question)

ChainOfThought reasons before answering. Swap it for Predict and you get a direct answer, no visible reasoning, same signature. The caller never notices. This is the same move as swapping a PostgresRepository for a MySQLRepository behind one Repository interface.

Compiling a prompt instead of writing one

This is where DSPy earns the comparison to a compiler. Give it labeled examples and a metric (did the answer match what you expected?) and it searches over prompt phrasings and few-shot sets, scores each one, keeps the best. You don't hand-write the final prompt. DSPy compiles it.

Say you tuned a prompt against GPT-4 and it performs well. Move to Claude, and that tuned wording often performs worse, because different models respond to different phrasing. Normally you retune by hand, from scratch. With DSPy, the signature stays fixed. You recompile against the new model, and the optimizer rebuilds a prompt suited to it. The contract survives the swap. Only the implementation gets rebuilt.

A worked example

Say your signature is summarize(document) -> summary, and your metric checks whether the summary mentions every named entity in the source document.

Run the optimizer with twenty labeled documents. It might try a zero-shot instruction first, score it, then try adding two few-shot examples, score that, then try a chain-of-thought variant that lists entities before summarizing. Whichever version scores highest on entity coverage becomes the compiled prompt. You never touched the wording. You touched the metric and the examples.

Switch the underlying model from GPT-4 to Claude next month, and you rerun the same compile step against the same twenty examples and the same metric. The signature summarize(document) -> summary never changes. Only the recipe underneath does.

The full mapping

Object-oriented concept	DSPy equivalent
Interface / abstract method	Signature
Concrete class	Module (`ChainOfThought`, `ReAct`, `ProgramOfThought`)
Constructor arguments	InputFields
Return type	OutputField
Compiler	Optimizer (`BootstrapFewShot`, `MIPRO`)
Polymorphism	Swapping modules under one signature

If you've spent time in Spring Boot or any typed backend, this table should feel familiar on sight. A signature is a method interface. A module is one implementation of it. The optimizer is what used to be a person doing prompt engineering by trial and error, now automated and measurable.

What next

Start with one signature for a task you already do by hand, question answering over a document, classification, extraction. Write the input and output fields with no prompt text at all. Attach dspy.Predict first, then dspy.ChainOfThought, and compare. Add ten labeled examples and a simple metric, then run BootstrapFewShot and look at what it generates. That's the fastest way to see the interface-and-implementation split in practice rather than on paper.

DSPy does need a training set and a metric to compile against, so it fits tasks with a checkable answer better than fully open-ended generation. Worth knowing going in, not a reason to avoid it.

Prasad MK writes on distributed systems, API governance, and the architecture underneath modern AI tooling.

When Caches Lie: The Hidden Invalidation Traps in a Multi-Instance Spring Boot Microservice

Prasad MK — Sun, 02 Aug 2026 16:46:07 +0000

Your local dev environment lied to you. It ran one instance, one JVM, one cache. Everything invalidated cleanly because there was nowhere else for stale data to hide.

Then you shipped two instances behind a load balancer, and the cache started telling a different story to different users.

The greenfield illusion

Here's the setup almost everyone starts with: a Spring Boot app, @EnableCaching. In local dev environment, with one pod, it works perfectly. Every write is immediately visible on the next read, because there is exactly one place for that data to live.

Scale to two instances behind an ingress router and the illusion breaks. A PUT lands on Instance A. A GET five seconds later lands on Instance B, because that's how load balancing works, and Instance B never saw the write. It's still holding whatever it cached ten minutes ago.

diagram 1: Two JVM instances with isolated local caches pointing at one shared database

This is not a bug. It's the correct behavior of two independent JVMs with no shared memory, doing exactly what you told them to do. The DB is the only thing in this picture that's telling the truth, and neither cache is listening to it in real time.

Most tutorials stop here and tell you to swap in Redis. That's necessary. It's not sufficient. Redis fixes the "two separate memory spaces" problem. It does nothing for the four scenarios below, and those are the ones that actually take down production.

Scenario 1: the invalidation call that quietly fails

@CacheEvict looks like a guarantee. It is not. It's a method call, and method calls fail.

Picture the sequence: your service commits a database transaction, then calls out to evict the cache entry. If that eviction call hits a network blip, a Redis connection pool exhaustion, or a two-second timeout, the eviction silently doesn't happen. The transaction already committed. Nothing rolls back. Nothing throws past your controller. Your monitoring dashboard shows 200 OK on the write endpoint, because from the caller's perspective, the write succeeded. It did. The cache just never found out.

Now the cache holds a value that will never self-correct until its TTL expires, which might be six hours from now if you set a long TTL to reduce database load (a reasonable thing to want, for the wrong reason).

A few concrete patterns that catch this instead of hoping it doesn't happen:

Set a TTL backstop even on entries you explicitly evict. If your write frequency for a given entity is roughly once every ten minutes, a TTL of two to three times that (20 to 30 minutes) means a dropped eviction call self-heals within half an hour instead of persisting until the next deploy.

Move the evict call after commit, not before, and make it retry. Spring's default behavior with @CacheEvict(beforeInvocation=false) already fires after the method body runs, but "after the method body runs" is not the same as "after the transaction commits" if you're inside a @Transactional boundary with deferred commit. Use TransactionSynchronizationManager.registerSynchronization to fire the evict in afterCommit, and wrap it in a retry with backoff:

@Service
public class OrderStatusService {

    private final CacheManager cacheManager;
    private final OrderRepository orderRepository;

    @Transactional
    public void updateStatus(Long orderId, String newStatus) {
        orderRepository.updateStatus(orderId, newStatus);

        if (TransactionSynchronizationManager.isActualTransactionActive()) {
            TransactionSynchronizationManager.registerSynchronization(
                new TransactionSynchronization() {
                    @Override
                    public void afterCommit() {
                        evictWithRetry(orderId, 3);
                    }
                }
            );
        } else {
            // no transaction context to hook, evict right away
            evictWithRetry(orderId, 3);
        }
    }

    private void evictWithRetry(Long orderId, int attemptsLeft) {
        try {
            cacheManager.getCache("orderStatus").evict(orderId);
        } catch (Exception ex) {
            if (attemptsLeft > 0) {
                evictWithRetry(orderId, attemptsLeft - 1);
            } else {
                // log loudly here, this entry is now relying on TTL alone
                log.error("Cache evict failed after retries for order {}", orderId, ex);
            }
        }
    }
}

The retry is not elegant. It's honest about the fact that eviction is a network call, and network calls fail in ways your happy-path tests will never exercise.

Note the guard around registerSynchronization. It only works when a transaction is actually in flight. Call updateStatus from a path that isn't wrapped in @Transactional, a batch job, a test, a refactor that strips the annotation, and registerSynchronization throws IllegalStateException: Transaction synchronization is not active instead of quietly doing nothing. Checking isActualTransactionActive() first means the method still evicts correctly outside a transaction, it just does it immediately instead of waiting for a commit that will never come.

There's a second version of this failure that has nothing to do with the network. @Transactional and @CacheEvict are both AOP proxies, and Spring does not guarantee which one runs first unless you set the order explicitly. Stack them on the same method and let Spring pick, and you get eviction and commit racing each other inside a single call. If the eviction fires first and the transaction rolls back afterward, the outcome is harmless: the cache is empty, and the next read repopulates it correctly from the DB. The dangerous direction is the other one. A method that writes to two cache keys and only one of them sits behind the failed step keeps the untouched write in place, because a transaction rollback undoes the database, not the cache. The afterCommit pattern above closes both directions at once. The evict call cannot fire before the transaction is durable, and it never fires if the transaction dies. That is the actual reason to register a TransactionSynchronization instead of stacking annotations and trusting Spring to run them in the order you want.

Scenario 2: the pod that wakes up already lying

Kubernetes rolling deployments and autoscaling mean your fleet of instances is never actually static. New pods come up cold. Old ones go away mid-request. And there's a window during pod startup that most cache architecture diagrams pretend doesn't exist.

diagram 2: Timeline showing a two second gap between a pod passing its readiness probe and its pub/sub subscription becoming active

Any invalidation broadcast during that window between traffic starting and the subscription connecting is silently lost, and the pod won't know until its TTL backstop expires.

Here's the timeline. At T0, the container starts. Somewhere around T0 plus 2 seconds, the readiness probe passes and Kubernetes starts routing real traffic to it. But the pod's Redis pub/sub subscription, the one it needs in order to hear "hey, this key was just invalidated," might not finish connecting until T0 plus 4 seconds. That's a two-second window where the pod is serving live reads with an empty local cache and no way to hear about invalidations happening elsewhere in the cluster.

If a write happens on another pod during that window and broadcasts an invalidation event, this pod never receives it, because it wasn't listening yet. It'll cache whatever it reads from the DB at that moment (which might already be correct) but it has no mechanism to know that a subsequent invalidation was meant for a key it hadn't even cached. The gap compounds if the write pattern is bursty right around a deploy, which, deploys being deploys, it often is.

What actually mitigates this:

Don't mark the pod ready until the pub/sub client reports itself connected, not just until the app context loads. Wire this into your readiness probe explicitly.
Treat the local L1 cache (Caffeine, say) as advisory and short-lived. A 30 to 60 second TTL on L1, backed by Redis as L2, means a missed invalidation during startup costs you at most a minute of staleness instead of an indefinite one.
Log every pub/sub connection event with a timestamp relative to pod start. If you've never measured this window in your own cluster, you don't actually know if it's 200 milliseconds or 4 seconds, and the fix you need depends entirely on which one it is.

Scenario 3: the race Spring has known about since 2012

This one isn't new. It's filed as Spring Framework issue SPR-9304, declined rather than fixed. If you're relying on @Cacheable and @CacheEvict to coordinate with each other, they don't, not by default, and the framework maintainers have said as much for over a decade.

diagram 3: Timeline of a slow GET reading a stale value and overwriting a cache after a concurrent PUT already evicted it

DB says APPROVED. Cache says PENDING. Nothing threw an error. This is the exact race described in Spring's SPR-9304, declined rather than fixed.

Walk through it in order. Thread 1 starts a slow SELECT at t=0ms, maybe it's hitting an unindexed column or waiting on a lock. While that read is in flight, Thread 2 runs a PUT: the UPDATE commits at t=20ms, and the @CacheEvict fires at t=21ms, clearing the cache entry. Everything looks fine at this point. The cache is empty, which is a safe state.

Then, at t=120ms, Thread 1's slow read finally returns. It read the row before the update happened, so it's holding the old value. Spring's @Cacheable does exactly what it's designed to do: it writes that value into the now-empty cache, at t=121ms. There's no eviction after this point to clean it up, because as far as the system knows, nobody wrote anything new. The cache now holds a stale value indefinitely, right next to a database that has the correct one.

Nobody threw an exception. Nothing logged an error. The only symptom is a support ticket three days later asking why a customer's order still shows "pending" after they got a confirmation email.

The slow query is not the only trigger. Plenty of Spring setups route @Transactional(readOnly=true) reads to a database replica through an AbstractRoutingDataSource, a standard way to keep load off the primary. That routing turns an ordinary GET into a race against replication lag instead of query cost. The PUT commits on the primary at t=20ms. A read-only GET lands on a replica that has not replayed that write yet, reads the old row at t=25ms, and caches it. No slow query, no lock contention, just a replica a few milliseconds behind. This version fires far more often than the slow-query version, because replication lag is a constant background condition of the architecture, not an occasional unlucky query plan. readOnly=true also switches Hibernate to manual flush mode for that session, so a stray write inside a read-only method fails silently instead of throwing.

The fix that actually works in practice is a short-TTL safety net combined with versioned reads, not a synchronization primitive bolted onto @Cacheable (Spring's cache abstraction doesn't give you a clean hook for that without writing your own CacheInterceptor, which is more surface area than most teams want to maintain). A version column or updated_at timestamp compared at write time catches the case where the read result is older than the latest write, whether that staleness came from a slow query or a lagging replica, and lets you skip writing it into the cache at all:

public Optional<Order> cacheAwareRead(Long orderId) {
    Order fresh = orderRepository.findById(orderId).orElseThrow();
    Long lastKnownVersion = versionTracker.getLatestVersion(orderId);

    if (lastKnownVersion != null && fresh.getVersion() < lastKnownVersion) {
        // this read is older than a write we already know happened
        // do not let it land in the cache
        return Optional.of(fresh);
    }

    cacheManager.getCache("orders").put(orderId, fresh);
    return Optional.of(fresh);
}

It's a small check. It closes a gap that's been sitting open in a widely used framework since 2012, which tells you this isn't a corner case, it's a structural property of caching next to a mutable store.

Scenario 4: what your frontend doesn't know it doesn't know

Say you've handled all three scenarios above. Your backend cache is now consistent with your database within a reasonable window. There's still one more layer where staleness sneaks back in: the contract between your API and whatever's consuming it.

A user submits a form. Your frontend gets a 200 back. It then re-fetches the same resource to refresh its view, and that GET happens to hit a replica or a CDN edge that hasn't picked up the change yet. From the user's point of view, they just submitted something and it didn't take. That's not a backend cache bug. It's a read-your-own-writes problem, and it lives at the API contract layer.

The fix is to hand the client something it can use to demand freshness on its next call. An ETag or a version token, returned on the write response and echoed back on the next read, lets the backend decide: is the cached response older than what this client already knows about? If yes, skip the cache and hit the DB.

@PutMapping("/orders/{id}")
public ResponseEntity<OrderResponse> updateOrder(
        @PathVariable Long id, @RequestBody OrderUpdateRequest request) {

    Order updated = orderService.update(id, request);
    String versionToken = String.valueOf(updated.getVersion());

    return ResponseEntity.ok()
        .eTag(versionToken)
        .body(OrderResponse.from(updated));
}

@GetMapping("/orders/{id}")
public ResponseEntity<OrderResponse> getOrder(
        @PathVariable Long id,
        @RequestHeader(value = "If-None-Match", required = false) String clientVersion) {

    OrderCacheEntry cached = orderCacheService.get(id);
    String cleanClientVersion = clientVersion != null ? clientVersion.replace("\"", "") : null;

    if (cleanClientVersion != null && cached != null
            && Long.parseLong(cleanClientVersion) > cached.getVersion()) {
        // the client already knows about a write newer than what we have cached
        Order fresh = orderService.forceRead(id);
        return ResponseEntity.ok().eTag(String.valueOf(fresh.getVersion())).body(OrderResponse.from(fresh));
    }

    return ResponseEntity.ok().eTag(String.valueOf(cached.getVersion())).body(OrderResponse.from(cached));
}

It closes a gap that no amount of backend cache tuning touches, because the problem isn't in the backend. It's in the assumption that the client and server agree on what "current" means.

One line in that handler is doing quiet, necessary work. The HTTP spec wraps ETag values in quotes, so If-None-Match arrives as "12345", not 12345. Passing that straight into Long.parseLong throws NumberFormatException on every single request, which is the kind of bug that survives code review because a manual curl test with a bare number in the header works fine and the quotes only show up when a real browser or HTTP client sends them. Stripping the quotes before parsing is the difference between this shipping and this getting reverted after the first production 500.

A checklist worth pinning to the PR template

Does @CacheEvict call happen after transaction commit, with a retry on failure, or does a network blip leave it silently unevicted?
Do you have a TTL backstop on every cache entry, even the ones you explicitly evict, sized to two or three times your typical write frequency for that entity?
Have you measured the gap between the pod's readiness probe passing and its pub/sub client actually connecting? If you haven't measured it, you don't know if it's 200ms or 4 seconds.
Is L1 (local, in-process) cache TTL short enough that a missed invalidation during a pod's startup window costs you a minute, not an hour?
Does a slow read that started before a write, but finishes after it, have any way to detect that it's about to write stale data back into the cache?
Does the API return a version token or ETag on writes, and does the read path actually check it against the client's last known version, or does it just cache-and-serve regardless?

None of these are exotic. The expensive part isn't writing the fix. It's noticing the gap exists before a customer does.

Oracle Services & Port 1521: A DBA Refresher on Listeners, Services, and Failover

Prasad MK — Thu, 23 Jul 2026 21:19:59 +0000

Every Oracle DBA has typed sqlplus user/pass@host:1521/orclpdb1 a thousand times without stopping to ask what's actually happening on the other end of that colon. It's old material, and it's the kind of old material that quietly underpins every outage post-mortem involving "the app couldn't connect" or "failover didn't work like we expected." Worth a refresher.

This post covers six things:

What actually happens on port 1521
The design philosophy underneath all of it: decoupling identity from location
Why one database ends up with multiple services
How the listener tells that traffic apart
What really happens when a service fails over
Given a service name, how to find the host it's actually running on

1. Port 1521 is a door, not a database

Port 1521 is the default TCP port for the Oracle Net Listener, not for the database itself. The listener is a separate process (tnslsnr) that sits in front of one or more database instances and brokers new connections. It doesn't execute SQL, it doesn't hold data, and it doesn't even have to run on the same machine as the database.

Here's the flow for a typical connection:

figure 1: Listener and port 1521 architecture

A few details worth re-remembering:

The client never asks for "the database." It asks for a service name (SERVICE_NAME=orclpdb1), resolved either from tnsnames.ora, an LDAP/OID directory, or an EZConnect string like host:1521/orclpdb1.
Services register themselves with the listener. The listener doesn't discover them on its own. This happens two ways:
- Dynamic registration: the instance's PMON background process registers every ~60 seconds (or immediately on startup/shutdown) with any listener it knows about via the local_listener / remote_listener parameters.
- Static registration: an explicit SID_LIST_LISTENER entry in listener.ora, mostly needed for tools that connect before the instance is fully up (RMAN starting an instance, for example).
Once the listener matches the requested service to a running handler, it hands the socket off (dedicated server mode) or routes it through a dispatcher (shared server mode), and drops out of the picture for the rest of that session.

So "port 1521" really means the address where the broker for a set of services is listening. It's not a direct pipe into a specific database.

2. The design philosophy: decoupling identity from location

Before getting into why multiple services exist, it's worth naming the principle that makes all of it possible, because it's the same principle that explains multiple services, traffic identification, and failover in one shot.

A service answers "what workload is this?" An instance answers "what physical process is executing it right now?" Oracle deliberately keeps those two questions independent, so the answer to one can change without touching the other. The relationship between services and instances isn't 1:1. It's many-to-many, and it's meant to be reshaped at runtime.

figure 2:Services and instances map many-to-many

That many-to-many mapping delivers four things:

Location transparency. An application connects to sales_svc, never to "instance 2 on node B." The service name is a stable contract. Which physical instance actually answers it can change hourly, during maintenance, during rebalance, or during failover, and the client's connection string never has to know. It works a lot like a DNS name pointing at a rotating set of IPs, just one layer further up the stack.
Independent scaling on both axes. Add a new instance (scale out the cluster) and existing services can spread onto it without redefinition. Add a new service (a new workload) and it can be placed on existing instances without adding hardware. If services and instances were rigidly paired, a capacity change on either side would force a redesign of the other.
Policy attaches to the logical unit, not the physical one. Resource Manager plans, preferred/available instance lists, TAF settings, Data Guard role bindings: all of it is defined on the service. That's deliberate. Policy should travel with "what this workload is," not with "which machine happens to be running it right now." An instance is disposable infrastructure. A service is the thing that actually has business meaning.
Instances become interchangeable, fungible capacity. Once policy lives on the service side, an instance is just an anonymous unit of CPU and memory that services can be poured into or out of. That's what makes failover, rolling patching, and load rebalancing possible without the application ever noticing. You're relocating the logical unit onto different physical capacity, not rebuilding the workload's identity.

The one-line version: Oracle services turn "which server am I talking to" into a question the database answers for you, by treating instances as fungible capacity and workloads as the durable, addressable, policy-bearing thing. Everything below (dynamic registration, multiple services per database, traffic identification, service relocation) is just the plumbing that keeps that separation true at runtime.

3. Why does one database need multiple services?

Every Oracle database has always had at least one service (historically identified by the SID), but real systems almost never stop at one. A few concrete reasons multiple services show up:

Workload isolation. OLTP traffic (short, latency-sensitive transactions) and reporting/batch traffic (long, resource-hungry queries) compete for the same buffer cache and CPU if you let them share a service. Giving each its own service lets you attach different Resource Manager consumer groups, different degrees of parallelism, and different priorities to each.
RAC placement control. In a RAC cluster, a service can be defined with preferred and available instances. That lets you say "this reporting workload only ever runs on node 3, unless node 3 is down" without touching the app's connection string.
Multitenant (CDB/PDB). Every pluggable database gets its own service name automatically. orclpdb1, orclpdb2, and so on all live inside one container instance but are addressed, secured, and monitored as fully separate services.
Zero-downtime maintenance and versioning. You can spin up a new service for a new app version, let both old and new services point at the same schema during a rolling deploy, then retire the old service. No DNS or IP changes required.
Data Guard role routing. Active Data Guard setups commonly expose a read-write service (only active on the primary) and a read-only service (active on physical standbys), so applications can be pointed at the correct role without hardcoding "which machine is primary this week."

figure 3: Multiple services routing to different workload lanes

The underlying data files, buffer cache, and SGA are shared. Services are a software-defined routing and policy layer on top of the same physical database, not separate databases.

4. How is traffic told apart per service?

This is the part people gloss over: the listener and the database distinguish traffic almost entirely by the SERVICE_NAME string the client presents at connect time. Nothing deeper than that at the network layer.

The connection string (//host:1521/service_name) or tnsnames.ora entry carries the service name in the CONNECT_DATA section of the TNS descriptor.
The listener matches that string against its registered service table (visible via lsnrctl services) and routes the session to an instance/handler currently offering that service.
Once connected, the session is tagged with that service for its entire lifetime. You can see this live with:

  SELECT sid, serial#, service_name, username, module
  FROM v$session
  WHERE service_name NOT IN ('SYS$USERS','SYS$BACKGROUND');

That service tag is what Resource Manager uses to apply consumer-group mappings (DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING with SERVICE_NAME), what AWR/ASH uses to break down load by workload, and what Enterprise Manager uses to draw separate performance charts per service, all from the same instance.
Nothing about the SQL, the user, or the schema determines the service. It's purely a connect-time declaration. Two sessions from the same user, same schema, same host, can land in completely different Resource Manager buckets purely because one connected via oltp_svc and the other via batch_svc.

This is also why service naming discipline matters operationally: a mistyped or reused service name silently puts traffic into the wrong priority bucket, with no error raised anywhere.

5. What happens when a service fails over?

"Failover" means different things depending on the layer, but the RAC case is the classic one, so let's walk through it end to end.

figure 4: Service failover sequence in a RAC cluster

Step by step:

Detection. Clusterware (OCSSD/CRSD) detects a node or instance is unreachable via missed heartbeats, not by the listener itself.
Existing sessions break. Any session connected through the failed instance is gone. Mid-transaction work is lost unless it's covered by TAF (Transparent Application Failover), FAN-aware connection pools, or a JDBC/UCP replay driver that can transparently re-establish the session and, in some cases, replay in-flight calls.
Notification. FAN (Fast Application Notification) publishes a service-down event almost immediately (versus clients waiting on a TCP timeout), and it propagates via ONS or the RAC-aware drivers so connection pools can proactively drop dead sessions instead of discovering them on next use.
Relocation. Clusterware relocates the service definition itself via srvctl relocate service (automatically, per the service's preferred/available instance list), so the service is now offered from a surviving instance. This is a service moving, not data moving. The underlying database is still the same shared storage.
Reconnection. New connection requests resolve through the SCAN listener, which always points at currently-available instances, so the app doesn't need to know which physical node is now hosting the service. Well-configured connection pools reconnect automatically. Anything with uncommitted work at the time of the failure has to be resubmitted by the application. Oracle can preserve the session, not the unfinished transaction.

The practical takeaway: failover protects availability of the service, not continuity of in-flight work. If your application logic assumes a failover is invisible, that assumption only holds if you've actually configured TAF/FAN or a replay-capable driver. Plain JDBC thin connections with no retry logic will simply throw errors when their instance disappears.

Quick reference commands

# What is the listener actually offering right now?
lsnrctl services

# RAC: where is a service running, and what's its preferred/available config?
srvctl status service -d orclcdb -s sales_svc
srvctl config service -d orclcdb -s sales_svc

# From SQL: who is connected to which service, right now
SELECT inst_id, service_name, COUNT(*) 
FROM gv$session 
GROUP BY inst_id, service_name
ORDER BY inst_id;

6. Given a service name, how do you find the host?

Since services are decoupled from location by design, there's no single universal reverse lookup, but you can trace it depending on what access you have.

If you have DB access, query it directly:

SELECT a.inst_id, a.name AS service_name, i.host_name, i.instance_name
FROM gv$active_services a
JOIN gv$instance i ON a.inst_id = i.inst_id
WHERE a.name = 'sales_svc';

gv$active_services (or v$active_services on a single instance) shows which instance(s) currently offer the service. gv$instance maps inst_id to an actual host_name. This works for both single-instance and RAC.

If it's RAC, ask Clusterware directly:

srvctl status service -d orclcdb -s sales_svc
srvctl config service -d orclcdb -s sales_svc

status gives the current host. config gives the preferred/available instance list, so you also know where the service could fail over to.

If you only have listener access, not DB access:

lsnrctl services

This shows which services and instances that specific listener knows about, but only reflects that one listener's view, not necessarily the full picture in a SCAN setup.

If you're starting from just a connection string:

The HOST in tnsnames.ora or an EZConnect string is what the client resolves first, but in a RAC/SCAN setup this is usually the SCAN listener's hostname, not the actual instance host. You still need the SQL or srvctl approach above to find where it's really running.

The practical takeaway: gv$active_services joined to gv$instance is the fastest, most accurate reverse lookup when you have DB access. srvctl status service is the equivalent when you only have OS-level Clusterware access.

Quick reference commands

# What is the listener actually offering right now?
lsnrctl services

# RAC: where is a service running, and what's its preferred/available config?
srvctl status service -d orclcdb -s sales_svc
srvctl config service -d orclcdb -s sales_svc

# From SQL: who is connected to which service, right now
SELECT inst_id, service_name, COUNT(*) 
FROM gv$session 
GROUP BY inst_id, service_name
ORDER BY inst_id;

Closing thought

None of this is new. Services and port 1521 have worked this way since 8i/9i, and RAC service relocation since 10g. But it's exactly the kind of foundational detail that's easy to half-remember, which is where "the failover didn't behave like I expected" incidents come from. Worth keeping the mental model sharp: one port, many services, each one a routable, policy-carrying unit that can move independently of the data it sits on top of, because instances are fungible capacity and services are the durable identity layered on top of them.

The Counter-Press Architecture: How Distributed Systems Intercept Failures at the Edge

Prasad MK — Tue, 21 Jul 2026 15:53:46 +0000

Introduction

In football (soccer), a Gegenpress (counter-press) isn't merely a defensive tactic, it's an aggressive, high-speed offensive trap. The moment a team loses possession in the attacking third, they don't retreat into a low block to regroup. Instead, nearby players immediately swarm the ball carrier, exploit the opponent's brief moment of disorganization, and strike before the opposing team can structure their defense.

Spain lost the ball in Argentina's final third during the 2026 World Cup. Within two seconds, three Spanish players had closed down the ball carrier. Nobody dropped back to reorganize. Nobody waited for Argentina to build an attack. They swarmed the loss and had the ball back before Argentina's shape even reformed.

That's a counter-press, or gegenpressing if you want the German. Calling it a defensive tactic undersells it. The moment a team loses the ball in the attacking third, nearby players swarm the carrier, exploit the few seconds of disorganization, and strike before the opponent can settle into anything resembling a defense.

I watched that sequence against Argentina and kept thinking about a rule I use constantly in system design: don't fall back, intercept early. The cheapest place to fix a problem is where it starts, not three hops downstream after it's had time to spread.

In distributed systems and resilient software design, the exact same principle applies: intercepting an unexpected state transition or failure at the edge before it cascades downstream.

Here are three patterns from distributed systems that run on the exact same logic.

Circuit breakers and speculative fallbacks

Picture a low block defense: an API dependency fails or times out, and the system's answer is to let the error bubble all the way up the call stack. That triggers context switches, retry loops, and expensive error handling three layers away from where the problem actually happened. It's the software equivalent of retreating into your own box and hoping the other team doesn't find the gap.

A counter-pressing architecture skips that. A service mesh or local circuit breaker catches the timeout right at the gateway and serves pre-cached or speculative data on the spot. The request resolves before the rest of the system even knows something failed.

Soccer	System
Ball loss in the attacking third	API timeout or dependency failure
Retreat to a low block	Error bubbles up the call stack
Swarm the ball carrier	Circuit breaker trips at the edge
Win it back and counter	Serve cached data before the client notices

Recovering isn't the hard part. Recovering before anyone downstream notices is.

Optimistic concurrency with fast-path retries

Pessimistic locking is a low block for databases. Threads want the same resource, so you lock it early, put everyone else to sleep, and make them wait their turn. It works. It's also slow, and it gives up ground you didn't need to give up.

Optimistic concurrency control plays a higher line. Threads assume no conflict, because most of the time there isn't one. When a commit does hit a collision, the system doesn't sleep the thread or force a context switch. It fires an immediate spin-retry right at the L1/L2 cache boundary, while the data is still warm in memory.

Soccer	System
Ball loose in midfield	Lock or version conflict detected
Sit deep and defend	Thread sleeps, context switch triggered
Press instantly while the position favors you	In-cache spin-lock retry
Win it back before the opponent regroups	Commit completes before other threads catch up

That window closes fast. Wait even a beat too long and the cache goes cold, same as a press that arrives a half-second late against a defense that's already reset.

Backpressure as interception, not a wall

A fast producer overwhelming a slow consumer is a familiar problem in stream processing. Do nothing about it, and the buffer fills up, packets start dropping, and eventually something crashes. That's a defense getting run over because nobody pressed the ball early enough to matter.

A counter-pressing stream architecture applies backpressure at the producer itself, before the surge ever reaches the broker. Reactive operators throttle the source, route the overflow to a dead-letter queue, and keep the rest of the pipeline running without a full rebalance.

Soccer	System
Opponent building an attack	Producer throughput spikes
Wait for it to reach your box	Buffer fills, packets drop downstream
Press the source immediately	Backpressure applied at the edge
Force a turnover before it develops	Overflow routed to a dead-letter queue

Why this keeps showing up

None of these three patterns are related to each other technically. Circuit breakers, OCC, and backpressure solve different problems in different layers of a system. What they share is where they choose to act.

A retry loop three services downstream costs you latency and a confused client. A circuit breaker at the edge costs you milliseconds. A thread that sleeps and context-switches costs a full scheduler round trip. A spin-lock retry costs a handful of cache cycles. A buffer that overflows three hops down costs you dropped data and a painful rebalance. Backpressure at the source costs you one throttle signal.

Same failure, wildly different price depending on where you catch it.

Figure 1: The Gegenpress tactic mapped to resilient system design patterns.

Spain's press works for the same reason. Winning the ball back isn't really the goal. Winning it back while the opponent is still disorganized, before they've had a second to reset, is what turns a turnover into a goal. Catch failure, contention, or overload as close to the source as you can, while the state is still cheap to recover, and you get the software version of the same thing: a problem that never gets the chance to become a bigger problem.

Locking Down the Pipeline: Enforcing Contract Integrity Against Autonomous AI Agents

Prasad MK — Sat, 30 May 2026 18:01:22 +0000

Parts 1 through 3 assumed one thing: a human is in the loop. A developer runs the local gate, reads the failure, and makes a deliberate decision. Even in Part 3, the vibe coder is still present. They feed the spec to the AI, read the output, and decide whether to push.

Part 4 removes that assumption entirely.

Autonomous AI agents, tools like Devin, AutoGPT, or custom LangChain pipelines, can now write code, run tests, interpret failures, and open pull requests without a human reviewing each step. This is not a future scenario. Teams are already running these workflows today.

The drift problem does not disappear in this environment. It accelerates. And it gets a new capability: the ability to cover its own tracks.

Why Autonomous Agents Break the Framework

An AI agent tasked with a refactor will do whatever it takes to satisfy the local objective. If it changes the pagination logic and the REST Assured test fails, it does not stop and ask a developer for guidance. It looks at the failure, determines that the test is an obstacle, and rewrites the assertion to make the build green.

From the agent's perspective, the task is complete. The build passes. The PR opens.

From the system's perspective, the contract was just silently redefined by an automated process that had no awareness of downstream consumers, no knowledge of the versioning rules from Part 2, and no constraint preventing it from touching protected files.

The governance framework built in Parts 1 and 2 relied on human judgment at the decision point. CODEOWNERS works when a human reviewer looks at the PR. A verbal rule about not mutating tests works when a developer reads it and understands why it exists.

Neither of these holds when the contributor is an agent running at machine speed.

The Solution: Programmatic Rails, Not Prose Rules

You cannot solve an automated problem with a social solution. Telling an AI agent to follow the rules in its system prompt is not a governance strategy. Context windows drift. Model updates change behavior. Prompt instructions get deprioritized when the agent is focused on satisfying a local objective.

The framework needs to be deterministic. The rails need to be structural. The enforcement needs to happen at the system level, not the prompt level.

Three layers make this work.

Layer 1: The Constraint File as a Machine-Readable Contract

The first layer moves the governance rules out of prose and into a structured file that the agent is required to parse before acting.

Create a .ai-rules.json file at the root of the repository:

{
  "repository_constraints": {
    "api_versioning": "STRICT",
    "breaking_changes": "NEVER_MUTATE_EXISTING_TESTS",
    "known_domain_rules": {
      "/api/v1/users": {
        "pagination_base": 1,
        "enforced_by": "UserWorkflowVerificationTest.java"
      }
    }
  }
}

This file does two things. It tells the agent exactly which domain rules govern each endpoint, and it explicitly names the test file that enforces each rule. When the agent is tasked with modifying anything under /api/v1/users, it must parse known_domain_rules first and treat those constraints as non-negotiable inputs, not suggestions.

The critical shift here is the difference between a rule the agent reads and a rule the agent loads as structured data. Prose in a system prompt gets weighed against the agent's objective. A JSON constraint file that the orchestration script injects into the agent's context window before execution is a boundary condition, not a preference.

Commit this file to the repository and version it alongside the API. When the contract evolves, the constraint file evolves with it in the same PR. The rules are traceable, reviewable, and visible to every contributor, human or automated.

Layer 2: The Pre-Commit Hook as the Deterministic Bouncer

The constraint file sets expectations. The pre-commit hook enforces them at the moment the agent tries to commit.

This is the most important layer because it operates entirely outside the agent's control. No matter what the agent decided to do, no matter what it changed, the hook runs before the commit is allowed to proceed.

#!/bin/bash
# Contract integrity check - runs before every commit

# Step 1: Run the protected REST Assured contract tests
mvn test -Dtest=UserWorkflowVerificationTest

if [ $? -ne 0 ]; then
  echo "ERROR: Contract tests failed. The commit is blocked."
  echo "Fix the underlying code. Do not modify the test assertions."
  exit 1
fi

# Step 2: Verify the agent did not touch the protected test file
if git diff --name-only | grep -q "UserWorkflowVerificationTest.java"; then
  echo "ERROR: AI Agent attempted to modify a protected contract test."
  echo "Functional changes require a new API version or architectural sign-off."
  exit 1
fi

The hook does two things in sequence. It runs the contract tests and blocks the commit if they fail. Then it checks the git diff and blocks the commit if the agent touched the protected test file at all, regardless of whether the tests pass.

That second check is the one that matters most. An agent that rewrites the assertion to make a failing test pass will produce a green test result. Without the diff check, the first gate would not catch it. The combination of both checks closes that gap entirely.

If the agent cannot commit, it cannot open a PR. If it cannot open a PR, the drift never reaches the repository.

Layer 3: The Coder and Auditor Pattern

The two layers above are defensive. They block bad commits from landing. The third layer is diagnostic. It determines why a failure happened and instructs the agent on how to fix it correctly.

The pattern is a separation of roles. Agent A is the Coder. It writes and refactors code. Agent B is the Auditor. It reviews the delta when the gate fails. These are two distinct LLM instances with different objectives, and they must never be the same instance self-reviewing its own output.

The reason this separation matters is the same reason a developer should not approve their own PR. An agent asked to both write code and verify its correctness will optimize for satisfying its own objective. The Auditor needs to be a genuinely independent process with a different prompt, a different focus, and explicit authority to reject the Coder's output.

When the pre-commit hook fails, the orchestration layer triggers the Auditor with this prompt:

System: You are an automated architectural gatekeeper.
Your job is to determine if a code change introduced a breaking
regression or a valid feature expansion.

Input Artifacts:
1. Git diff of the change: [Insert diff]
2. Test failure log: [Insert REST Assured terminal output]
3. Enforced rules schema: [Insert .ai-rules.json contents]

Task: Analyze whether the code change violates any constraint
defined in the rules schema. If it does, generate a rejection
log that instructs the Coder agent to revert the specific change
and fix the underlying logic.

Constraint: Under no circumstances should the test suite assertions
be modified. The tests define the contract. The code must conform to them.

The Auditor does not fix the code. It produces a rejection log that describes exactly what the Coder did wrong and what it needs to do differently. The Coder then receives that rejection log as its next input and retries.

This loop continues until the pre-commit hook passes cleanly, meaning the contract tests are green and the protected test files are untouched.

How the Full Framework Holds Together

Stepping back across all four parts, the same contract flows through every layer.

The Spring Boot application exposes its live OpenAPI spec at /v3/api-docs. REST Assured derives its tests from that contract. Postman derives its collections from that contract. CODEOWNERS enforces that nobody modifies the core test files without cross-team review. API versioning ensures that behavioral changes ship as new endpoints, not as silent mutations to existing ones. The .ai-rules.json file encodes the domain rules as machine-readable constraints. The pre-commit hook enforces those constraints at commit time, regardless of whether the contributor is human or automated. The Auditor agent closes the diagnostic loop when something goes wrong.

At no point does the framework rely on trust, memory, or discipline. Every layer is structural. Every enforcement is deterministic. The contract is defined once, in the code, and every tool downstream, whether it is a developer, a vibe coder, or an autonomous agent, operates within the same boundaries.

Closing the Series

The zero-drift problem is not a tooling problem. The tools, REST Assured, Postman, Git, OpenAPI, were always capable of solving it. The missing piece was a coherent framework that connected them into a single chain of enforcement, from the individual developer's local machine all the way to an autonomous agent operating without human oversight.

That chain is now complete. Start with Part 1 today. The local loop costs less than an hour to set up and pays back immediately. Add the governance layer from Part 2 when the team grows. Introduce the AI prompt discipline from Part 3 when AI tools enter the workflow. Apply the programmatic rails from Part 4 when agents start opening PRs on their own.

The build should be green because the contract is intact. Every time. At every scale.

The Zero-Drift API Series

Intro: The Zero-Drift API Series: Stop Trusting a Green Build You Can't Explain
Part 1: A Guide to Stop Breaking Merges: Unifying Postman and REST Assured in Spring Boot
Part 2: Who Approved This Change? Managing API Contracts and Test Rot in Large Engineering Teams
Part 3: The AI Superpower: How Vibe Coders Use OpenAPI as a Semantic Anchor
Part 4: Locking Down the Pipeline: Enforcing Contract Integrity Against Autonomous AI Agents

The AI Superpower: How Vibe Coders Use OpenAPI as a Semantic Anchor

Prasad MK — Sat, 30 May 2026 17:53:16 +0000

Parts 1 and 2 solved the drift problem for developers who write code deliberately. A local REST Assured gate, Postman wired to the live spec, CODEOWNERS enforcing review on core contracts, and API versioning as the hard rule when behavior changes.

Now introduce a different kind of contributor. A vibe coder.

A vibe coder is not a junior developer making rookie mistakes. They are often highly productive engineers who use AI tools like Cursor, Copilot, or a plain LLM chat window to generate, refactor, and iterate on code rapidly. The focus is on intent and outcome, not on typing syntax. The speed is real. So is the risk.

The Problem AI Introduces

An AI model generating code has no awareness of your running system. It does not know that your pagination is 1-based, that a specific field was renamed three sprints ago, or that a downstream mobile client breaks if a response envelope changes shape.

It knows patterns. It knows what a Spring Boot controller generally looks like. It knows what a TypeScript fetch service should probably contain. When it does not have exact information, it fills the gap with a confident-looking guess.

This is called a hallucinated contract. The generated code compiles. The types look plausible. The field names are reasonable. And the first time it runs against your actual API, something quietly breaks because the AI invented a payload shape that does not match reality.

The drift problem from Part 1 gets faster and more confident with AI in the loop. Without a semantic anchor, the AI is just a very fluent source of test rot.

The Fix: Feed the Spec, Not Just the Prompt

The OpenAPI spec that has been doing all the work in Parts 1 and 2 is the answer here too. Instead of asking the AI to guess your contract, you hand it the exact live definition from your running application and treat it as a non-negotiable constraint.

When the AI works from /v3/api-docs, it is no longer guessing. It knows the exact endpoint paths, the required fields, the optional fields, the data types, the pagination parameters, and the response envelope shape. Every piece of code it generates is grounded in what the system actually does right now.

This is the semantic anchor. The same single source of truth that keeps Postman and REST Assured in sync becomes the context that keeps AI-generated code from drifting before it is even written.

Three Prompts That Close the Loop

The following prompts cover the three stages where AI intersects with the zero-drift workflow.

Prompt 1: Generate the REST Assured Test Suite from the Live Spec

Instead of writing boilerplate test classes by hand, boot the application, grab the OpenAPI JSON from http://localhost:8080/v3/api-docs, and hand it to the AI with this prompt.

System: You are an expert backend QA engineer.
I will provide my local application's raw OpenAPI/Swagger JSON specification.

Task: Generate a complete REST Assured integration test class in Java
using @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT).

Rules:
1. Use standard given().when().then() syntax.
2. Every endpoint in the specification must have at least one contract
   validation test covering the status code and core payload fields.
3. Implement random local port setup using @LocalServerPort.

Here is the OpenAPI specification JSON:
[Paste http://localhost:8080/v3/api-docs output here]

The AI now generates tests that match your actual endpoints, your actual field names, and your actual response structure. Not a plausible approximation of them.

This also means the tests are immediately runnable. There is no cleanup pass to fix field names the AI invented.

Prompt 2: Debug a Failing Test Without Reading the Stack Trace

When a local REST Assured test fails after an AI-assisted refactor, the natural instinct is to dig through the stack trace manually. There is a faster path. Feed the failure back to the AI with the relevant controller code and let it diagnose itself.

Context: My local REST Assured test suite just failed during a local check.

The Error:
[Paste the JUnit failure output here, e.g.:
 JSON path currentPage expected 1 but was 2]

The Relevant Controller/Service Code:
[Paste your Spring Boot Controller or Service class here]

Task: Analyze the code and the test failure. Identify where the contract
logic diverged. Fix the underlying Java code to restore backward
compatibility with the expected contract. Explain exactly why the
runtime behavior shifted.

Constraint: Do not modify the test assertion. The test reflects the
contract. The code must conform to it.

The constraint at the end is critical and worth stating explicitly every time. Without it, the AI's default instinct is to make the test pass by any means available, including rewriting the assertion. That is exactly the silent mutation problem from Part 2. The prompt structure prevents it.

Prompt 3: Generate Frontend Code with Zero Payload Drift

When a vibe coder moves to building a UI component or a mobile client that consumes the Spring Boot API, the same anchor applies. Do not let the AI guess the payload shape. Give it the spec.

Context: I am building a frontend component in React/TypeScript that
communicates with my Spring Boot backend.

Input: Here is the live contract definition from my backend:
[Paste http://localhost:8080/v3/api-docs output here]

Task: Generate a TypeScript Fetch/Axios service and the corresponding
interface types for the /api/v1/users endpoint.

Rules:
1. All property names must match the schema definitions exactly.
2. Respect optional and required field flags from the spec.
3. Pagination handling must be based strictly on the parameters
   defined in the specification. Do not assume defaults.

The result is a TypeScript service that matches the backend contract on the first generation. No runtime surprises when the frontend hits the actual API. No field name mismatches discovered in a browser console after a deploy.

Why This Matters Beyond Convenience

There is a deeper point here worth stating plainly.

The zero-drift framework in Parts 1 and 2 is built on one principle: the contract lives in the code, and everything else derives from it. REST Assured derives from it. Postman derives from it. CODEOWNERS enforces that nobody silently redefines it.

When AI enters the workflow without this anchor, it introduces a third source of truth, which is the model's best guess. That guess is invisible, confident, and wrong in ways that are hard to detect until something breaks downstream.

Feeding the spec into the AI does not slow down the vibe coder's workflow. It makes the output trustworthy on the first pass, which is actually faster than the debug cycle that follows a hallucinated contract.

The local REST Assured gate from Part 1 remains the final verification. If the AI introduced drift anywhere, the gate catches it in under five seconds. The developer does not need to know exactly what the AI got wrong. They feed the failure back with Prompt 2, fix the code, and run the gate again.

The loop is fast. The contract stays clean. The AI becomes an accelerator rather than a liability.

What Comes Next

Parts 1 through 3 handle the developer-in-the-loop case, whether they are writing code manually, working in a large team with governance constraints, or using AI tools to generate at speed.

Part 4 removes the developer from the loop entirely. When autonomous AI agents are opening pull requests without a human reviewing each change, the governance framework needs to be deterministic and programmatic. Pre-commit hooks, constraint files baked into the repository, and a multi-agent Coder/Auditor pattern that treats the AI like an untrusted contributor.

Part 4: Locking Down the Pipeline: Enforcing Contract Integrity Against Autonomous AI Agents

The Zero-Drift API Series

Intro: The Zero-Drift API Series: Stop Trusting a Green Build You Can't Explain
Part 1: A Guide to Stop Breaking Merges: Unifying Postman and REST Assured in Spring Boot
Part 2: Who Approved This Change? Managing API Contracts and Test Rot in Large Engineering Teams
Part 3: The AI Superpower: How Vibe Coders Use OpenAPI as a Semantic Anchor
Part 4: Locking Down the Pipeline: Enforcing Contract Integrity Against Autonomous AI Agents

Who Approved This Change? Managing API Contracts and Test Rot in Large Engineering Teams

Prasad MK — Sat, 30 May 2026 17:39:22 +0000

Part 1 established a clean local loop. REST Assured runs in under five seconds on the developer's machine. Postman pulls from the live OpenAPI spec. The gate is reliable, and the contract is grounded in actual code.

Now add 50 developers to that picture.

Suddenly the local loop is not enough. One developer's intentional optimization is another downstream team's production regression. And here is the uncomfortable truth: the test suite itself becomes the attack surface.

The Trap Nobody Talks About

When a REST Assured test fails because a developer changed API behavior, there are two ways to make it green again. Fix the code, or fix the test.

Fixing the test is faster. It feels harmless. The build goes green. The PR merges. And somewhere downstream, a team that depended on the original behavior starts seeing failures they did not cause and cannot explain.

This is not a discipline problem. It is a structural one. When a single developer can silently redefine an API contract by updating an assertion, the test suite stops being a gate and starts being a liability.

The Opposite Trap Is Just as Bad

The instinctive overcorrection is to declare all existing tests immutable. Nobody touches them. Ever. Only new tests get added.

This sounds safe. It is not.

When an API genuinely needs to evolve, the old test stays active and stays failing. Teams start using @Ignore just to get deployments through. The test suite becomes a graveyard of outdated assertions that nobody trusts and nobody deletes. This is test rot, and it is just as damaging as silent contract mutation, just slower.

The dilemma looks like this:

Approach	The Benefit	The Trap
Never change old tests	Prevents silent contract rewrites	Causes permanent build failures when changes are intentional. The suite becomes a historical graveyard.
Freely modify existing tests	Keeps the suite clean and passing	One developer can redefine the contract for the entire organization without downstream teams knowing.

Neither extreme works at scale. The answer is a governance layer that distinguishes between the two cases explicitly.

The Architectural Solution: Three Levers

You do not need to throw away REST Assured or install a complex contract broker to solve this. Three levers, used together, handle the governance problem with tooling teams already have.

Lever 1: API Versioning as a Hard Rule

When a functional change modifies payload structure, field behavior, or pagination logic, the existing endpoint is not touched. It stays exactly as it is, along with its REST Assured tests.

The change ships under a new version.

/api/v1/users and its test suite remain intact and passing. /api/v2/users is introduced with a brand new test class that reflects the new behavior. Downstream teams consuming v1 are never broken. They migrate to v2 on their own timeline.

This rule eliminates the core dilemma entirely. There is no decision to make about whether to update an old test, because the old endpoint and its tests are never touched.

Lever 2: CODEOWNERS to Enforce Review on Core Contracts

GitHub and GitLab both support a CODEOWNERS file that assigns mandatory reviewers to specific directories. This is the enforcement mechanism for the versioning rule.

Place the core REST Assured contract tests under a protected path:

src/test/java/com/yourorg/contracts/

Then add a CODEOWNERS entry:

src/test/java/com/yourorg/contracts/ @api-architecture-team

Now any PR that modifies a file inside that directory cannot merge without explicit approval from the API architecture group. A developer can freely add new test files for new features anywhere else. But touching an existing contract test requires a deliberate, cross-team sign-off.

The gate is no longer social. It is structural.

Lever 3: Deprecation Windows Instead of Deletion

When old functionality genuinely needs to retire, the process is not a silent test deletion. It is a staged, visible wind-down.

Mark the old endpoint as @Deprecated in the Spring Boot controller. Log a deprecation warning on every call so downstream teams see it in their monitoring. Schedule the removal for a future sprint with a communicated deadline. When that sprint arrives, the old endpoint and its REST Assured tests are deleted together, in the same PR, with full visibility.

Nobody is surprised. Nobody discovers a broken contract in production. The retirement is as explicit as the original contract.

When You Need More: Consumer-Driven Contract Testing

The three levers above handle the majority of real-world cases without additional infrastructure. But if your organization has many independent teams consuming the same APIs, and the coordination cost of manual review is becoming a bottleneck, the next step is Consumer-Driven Contract Testing using Pact or Spring Cloud Contract.

The model works like this. Each downstream team that consumes an API publishes a consumer contract that explicitly declares what it expects. When Team A's developer makes a change that affects /api/v1/users, the CI pipeline automatically pulls all active consumer contracts from a shared broker and runs them against the new code. If any consumer contract fails, the merge is blocked and the affected team is notified before anything ships.

The key difference from the lever approach is that enforcement becomes fully automated. No human reviewer needs to catch the conflict. The pipeline catches it.

This is the right investment when teams are large enough that CODEOWNERS review becomes a bottleneck, or when downstream consumers are external and cannot be coordinated manually.

What This Looks Like in Practice

A developer on Team A refactors the user pagination logic and opens a PR.

Without governance: the test fails, they update the assertion, CI goes green, Team B finds out when their service breaks in staging.

With governance: the test fails. The CODEOWNERS rule blocks the PR from merging without architecture review. The reviewer looks at the change, determines it is a functional behavioral shift, and asks the developer to version the endpoint. /api/v2/users is introduced. Team B is notified of the new version. Team A ships. Nobody breaks.

The difference is not more process. It is the right structure in the right place.

What Comes Next

At this point the framework handles individual developers and distributed teams. The contract is protected, the versioning rule is enforced, and the governance layer is structural rather than cultural.

Part 3 introduces a different kind of contributor: AI-assisted developers using tools like Cursor or Copilot, who are generating code faster than any review process was designed to handle. The same OpenAPI spec that anchors Postman and REST Assured becomes the semantic anchor that keeps AI-generated code from hallucinating payload shapes and silently drifting from the contract.

Part 3: The AI Superpower: How Vibe Coders Use OpenAPI as a Semantic Anchor