DEV Community: Ankit Verma

A question on Discord found two bugs in my RAG system. Fixing them found a third.

Ankit Verma — Sun, 13 Sep 2026 14:19:05 +0000

Every test passed. The tests had the same blind spots as the code.

A developer saw Ossian's MCP server in a community showcase and asked two questions:

How do you handle cases where retrieved docs and stored memories conflict? Or when multiple
pieces of context ultimately come from the same underlying source?

I had a confident answer to both. Documents and agent memory live in separate tables, are reached
through separate tools, and never meet in a prompt. Duplicates are caught at ingest by content hash.
Done.

Before replying I read the code to make sure the answer was true. It was half true, and the half
that wasn't is the interesting part.

Bug one: three passages looked like three sources

Retrieval returns chunks, not documents. Ossian took the top six and numbered them for the model,
so a prompt could look like this:

[1] engineering-handbook.txt  …
[2] engineering-handbook.txt  …
[3] engineering-handbook.txt  …
[4] platform-architecture.md  …

To the model — and to the person reading the citations — that is three sources agreeing and one
dissenting. It is one source said three times. The system prompt even tells the model "if the
context conflicts with itself, say which sources disagree", and there was no way for it to tell one
source from several.

Ingest-time deduplication does nothing here. These are distinct chunks of one legitimate document.

The fix groups chunks by document id before the prompt is built: one number per document, every
passage kept under it, the document ranked by its best chunk. Two files that merely share a filename
stay separate, because the grouping is on id. On a live question, six retrieved chunks now become
five citations, with one runbook contributing two passages under a single number.

Bug two: memory decayed from the wrong moment

Agent memory is ranked similarity × importance × 0.5^(age / 30 days). Recency matters for memory in
a way it never does for documents: a runbook from three years ago is as true as one from today, a
stated preference from three years ago is not.

The question is what "age" means. The query measured it from created_at.

An agent that restates a fact — "the user still prefers British English" — hits a deduplicating
upsert, which updates updated_at and nothing the ranking reads. A preference confirmed every day
for three months decayed exactly as if it had been said once, three months ago.

The obvious fix is wrong. Recall already records last_used_at, and a comment on it claimed that
recording use "keeps a live memory from decaying away." It didn't — and it shouldn't. Recall
returns everything that matches, so an old preference and the newer one contradicting it are
recalled together. Refresh both on read and they tie on recency, which is the one signal that
lets the newer one win.

So age now runs from the last time something was said, not the last time it was read.
Measured against the running system, after three recalls of both:

Memory	Score
"switched the editor to the light theme" (fresh)	0.765
"prefers the dark theme" (90 days old)	0.102
…then the dark-theme preference is restated	0.817

The test suite had a recency test, and it passed the whole time. It backdated created_at — the
same column the bug read. The test and the code shared an assumption, so the test could only ever
confirm it. The new test fails against the old query; I checked by putting the old line back.

Bug three: found by building the next thing

With those fixed, I built what I'd meant to build anyway: a Kafka Connect sink, so a table that
Debezium streams into Kafka becomes a corpus that follows the database.

Its first end-to-end run showed a batch of changes failing with HTTP 500, twice, then succeeding on
the third attempt. The sink's backoff did its job. The backend's log said:

insert or update on table "ingest_events" violates foreign key constraint
Key (document_id)=(…) is not present in table "documents".

A DELETE event removed the document, then recorded the event row pointing at the id it had just
deleted. The foreign key rejected it. Because the batch loop didn't catch it, every event in the
batch failed with it.

And it only happened once per document. The retry found nothing to delete, recorded a null id, and
succeeded. Any pipeline that retries — which is every pipeline worth running — hid it completely.
There were no tests for the event API at all, so nothing else was going to.

Building the sink

A few decisions that are easy to get wrong:

The event id comes from Kafka, not from Debezium. Ossian's event API is idempotent on a
caller-supplied id, so the sink needs one that is stable under redelivery. Debezium's source position
looks ideal and isn't: every row of an initial snapshot shares one LSN, so two different rows would
collapse into one id and the second would be silently discarded as a duplicate. The sink uses the
connector name, topic, partition and offset, plus the record timestamp to survive a topic being
recreated with offsets starting from zero.

A blanked row is removed, not skipped. If an update empties every text column, leaving the old
document in place means the corpus keeps answering from text the source no longer has. If none of
the configured columns exist on the record, though, it's rejected as a misconfiguration — otherwise a
typo in ossian.text.fields deletes the whole table from the corpus.

Placeholders are refused. With Postgres' default replica identity, an update that doesn't touch
a large column sends Debezium's __debezium_unavailable_value instead of the text. Indexing that
would replace a real article with a sentinel string.

Offsets never move ahead of delivery. put() is synchronous. Rate limits and 5xx back off using
Retry-After and retry; a record Ossian rejects goes to the dead-letter queue while the rest keep
flowing; a 401 stops the task, so a bad key can't drain an entire topic into the DLQ.

End to end against a Postgres table: a snapshot of three rows became three documents answerable with
citations; an update from 180 to 90 days changed the answer and left no old chunk still saying 180;
a delete removed the document and its chunks; a blanked row disappeared; and resetting the sink's
offsets replayed the whole topic — 18 events before, 18 after, no new documents.

What's still open

The original question isn't fully answered, and I said so in the reply.

Memories have no link back to the document they were learned from, so a memory that goes stale when
its source document changes is never flagged. And deduplication is exact: "prefers dark mode" and
"likes dark mode" both persist, and a contradicting memory doesn't supersede the older one — both come
back, and recency decides. Keeping documents and memory apart is staying. The rest is an
open design discussion.

The lesson I keep relearning

None of these three were exotic. Each survived because something around it agreed with it: a test
that backdated the same column the query read, a citation format that looked right in every
screenshot, a retry loop that turned a deterministic failure into a transient one.

The fastest way I know to find that kind of bug is to explain the system to someone who asks a
precise question — and check the code before you hit send.

Ossian: dockndevai.github.io/ossian-site ·
github.com/dockndevai/ossian ·
Kafka Connect sink: github.com/dockndevai/ossian-kafka-connect ·
Issues: #1,
#2,
#5

Originally published on Medium.

Written with Claude (Anthropic), working in the codebase it describes; every number and code sample
was verified against the running system. Reviewed and published by
@dockndevai.

Filter Ordering Is the Whole Game: Building an LLM Gateway on Spring Cloud Gateway

Ankit Verma — Fri, 28 Aug 2026 12:15:00 +0000

Every team that runs more than one self-hosted model eventually builds the same thing. Someone
stands up vLLM for a 70B model, someone else runs Ollama for the small stuff, and within a month
you need to answer questions nobody asked at the start: which team burned 40 million tokens last
week? Can we stop the analytics service from calling the expensive model? What happens when
vLLM falls over mid-sprint?

Spring Cloud Gateway already solves routing, retries and circuit breaking. What it doesn't know is
that the interesting part of an LLM request is inside the JSON body, and that the interesting
part of the response is a usage object that arrives last. So I built a starter that adds exactly
that - virtual keys, token quotas, usage metering and failover - as a library you drop into an
existing gateway.

The code is at dockndevai/spring-llm-gateway.
This post is about the parts that were harder than they looked.

Counting requests is the wrong unit

The stock RequestRateLimiter counts requests. For LLM traffic that number means almost nothing -
one request can be worth four tokens or forty thousand. A per-minute token budget is the only
limit that maps to load and spend.

That creates an awkward problem: you don't know what a request costs until it's finished. So the
quota filter reserves an estimate up front -
prompt_chars / chars_per_token + max_tokens - and reconciles afterwards against the real usage
scraped off the response. Over-reservations get refunded, under-reservations charged.

Reserving up front is the part that matters. If you only charged after the fact, a burst of
concurrent requests would all pass a check that each of them individually would fail. You can watch
the reconciliation work in the response headers:

req 1: HTTP 200  remaining=138
req 2: HTTP 200  remaining=98
req 3: HTTP 200  remaining=52

Each request reserves 62 and actually uses ~40, so ~22 comes back before the next one is measured.
The numbers only add up because the refund happened.

Reading the response without consuming it

Metering needs the usage object, which means reading the response body - without breaking
streaming for the client.

Two things make this tractable. First, DataBuffer.toString(Charset) does not advance the read
position, so you can observe a chunk and still pass it downstream untouched. Second, usage is
last in both response shapes: a top-level field on a non-streamed response, and the final data:
frame of an SSE stream. So you never buffer the whole body - just a bounded 8 KB rolling tail.

The parsing has one trap. The obvious regex is "usage"\s*:\s*\{[^{}]*\}, and it works right up
until it doesn't: current vLLM and OpenAI builds nest prompt_tokens_details and
completion_tokens_details inside the usage object, and a flat character class can't match
across nested braces. I ended up with a brace-balanced scan that walks backwards from the last
"usage" and skips anything truncated or null - which also handles the intermediate stream
frames, where "usage":null appears on every chunk.

There's one more thing you have to do, and it's easy to miss: vLLM never sends a usage block for a
stream unless you ask for it. If the request doesn't set stream_options.include_usage, every
streamed request meters as zero tokens. The gateway injects it when absent - and then has to fix
Content-Length on the request decorator, because the body just got longer.

Ordering is the actual design

This is where most of the real work went. Spring Cloud Gateway assembles the filter chain after
route selection, and the orders interact in ways that are invisible until something silently
returns zero.

Order	Filter
`MIN_VALUE`	`RemoveCachedBodyFilter`	stock
`MIN_VALUE + 1000`	`AdaptCachedBodyGlobalFilter`	stock
`MIN_VALUE + 1500`	body parsing	mine
`-2`	metering	mine
`-1`	`NettyWriteResponseFilter`	stock
`1, 2, 3...`	route filters - auth, quota, circuit breaker	by list position
`LOWEST`	`NettyRoutingFilter`	stock

Three constraints hold it together:

Body parsing must run after AdaptCachedBodyGlobalFilter. The route predicate that selects on
model has to read the body during route selection. That consumes it. AdaptCachedBodyGlobalFilter
is what swaps the cached bytes back onto the exchange - parse any earlier and you read an empty
body.

Metering must run before NettyWriteResponseFilter (order < -1). That filter captures its own
exchange.getResponse() reference before writing. Install a response decorator later in the chain
and it is simply never written into. No error, no warning - every request just meters as zero
tokens.

Route filters are ordered 1, 2, 3... by their position in the YAML list, which puts them after
both global filters. That's what lets auth and quota rely on the parsed body already being there.

I nearly got this wrong in an interesting way. I started to force explicit orders on the auth and
quota filters to guarantee auth ran first - then found that default filters and route filters are
index-ordered into a single stably sorted list. With LlmAuth/LlmQuota in default-filters
and a route-level CircuitBreaker, forcing orders interleaves them as auth -> breaker -> quota.
Keeping the natural index ordering and documenting it was the right call.

One more: the rewritten request's getBody() has to be re-subscribable. A DataBuffer can only be
read once, and both retries and the circuit breaker re-subscribe. Flux.defer wrapping a freshly
wrapped byte array on every subscription.

Two reactive traps that produce no error

These both cost me real time, and neither throws anything.

Mono<Void> and switchIfEmpty. The auth filter looked like this:

return principalResolver.resolve(exchange, token)
    .flatMap(principal -> authorize(exchange, chain, principal))
    .switchIfEmpty(Mono.defer(() -> unauthorized(exchange)));  // wrong

Reads fine. It's broken. chain.filter() returns Mono<Void>, which always completes empty - so
switchIfEmpty fired on every successful request. The upstream was called and then a 401 was
written over the top of it. The tell was a log line showing the stub upstream had received the
request while the client got a 401. Fold the Optional explicitly instead; don't put
switchIfEmpty downstream of anything returning Mono<Void>.

exchangeToMono releases the connection when its Mono terminates. The failover handler fetched
the secondary upstream's response and handed the body to a ServerResponse to be written later.
Status 200, correct headers, empty body - because the ServerResponse Mono completed immediately,
WebClient released the connection, and then something tried to read the body. For a streaming
proxy you want a plain WebHandler that writes straight to the exchange inside the callback, so the
connection stays alive exactly as long as the body is streaming.

What only showed up when I ran it

The build was green and the feature list was complete. Then I ran it against a real Ollama on a
laptop, and four things fell out immediately.

The worst: auth.mode=jwt wouldn't start at all. The Keycloak resolver bean was guarded by
@ConditionalOnBean(ReactiveJwtDecoder.class) - but my auto-configuration wasn't ordered after
ReactiveOAuth2ResourceServerAutoConfiguration, so the condition was evaluated before that decoder
existed and backed the bean off entirely. The application then died complaining about a missing
PrincipalResolver, which points nowhere near the actual cause.

My tests missed it because I'd unit-tested the resolver class directly and never booted the
application in that mode. That's the lesson worth keeping: @ConditionalOnBean on a bean from
another auto-configuration is an ordering bug waiting to happen, and testing a class is not
testing its wiring. The fix was afterName ordering plus dropping the condition so a genuinely
missing decoder names itself in the error. I wrote the integration test first and confirmed it
failed against the old wiring before fixing it.

The other three were smaller but all real: the README's own quick-start command didn't work
(-am on spring-boot:run drags the parent POM into the reactor - "Unable to find a suitable main
class"); ollama: {} in YAML binds to nothing, because Spring's binder needs at least one leaf
property to create a map entry; and my failover tests used localhost:1 as the "dead" port, which
on macOS burns six DNS queries and ~10 seconds before failing instead of refusing. With a properly
closed port it refuses in 50ms, and the test got 4x faster.

That last one produced a false alarm I had to walk back. I'd measured failover at 15 seconds and
started writing it up as a performance problem. It was entirely an artifact of the port I'd picked

real failover is 0.23 seconds end to end, including the model generating a reply.

Worth knowing if you build on SCG 4.3

A few things that cost me time and aren't well signposted:

The config prefix moved. It's spring.cloud.gateway.server.webflux.* now, and the starter is spring-cloud-starter-gateway-server-webflux. The old spring-cloud-starter-gateway is a deprecated alias.
Model ids contain dots, so config map keys need bracket notation: models."[llama-3.1-70b-instruct]". The dotted form binds as nested properties and silently never matches.
Multi-argument filters need the expanded args form. CircuitBreaker=name=x,fallbackUri=y binds only the first positional argument and drops fallbackUri without a word of complaint. I lost a while to a fallback that was never being invoked.
Raise resilience4j's TimeLimiter. The 1-second default will trip on healthy generation traffic.
Building on JDK 23+? Annotation processors on the classpath are off by default, so spring-boot-configuration-processor silently produces no metadata and your IDE completion quietly stops working. <proc>full</proc> brings it back.

The shape that worked

The thing I'd repeat on the next one: lean on the framework's machinery instead of reimplementing
it. Failover here is the stock CircuitBreaker filter with fallbackUri: forward:/__llm/fallback

I supply only the replay handler. The state machine, the sliding window, the half-open probing all stay Spring's. Same with the model predicate, which is modelled directly on Spring's own ReadBodyRoutePredicateFactory rather than inventing a body-caching scheme.

The whole thing is a dependency plus YAML. If you already run Spring Cloud Gateway, you add one
artifact and two filter names and you have virtual keys, token quotas, per-tenant metering and
failover.

Code, README and the full test suite: github.com/dockndevai/spring-llm-gateway

8 safe-by-default MCP servers for infra — and the governance model behind them

Ankit Verma — Sun, 23 Aug 2026 16:26:47 +0000

🚀 Update (Sep 8): the suite is now 11 servers (added Grafana) and is live on Product Hunt today → https://www.producthunt.com/products/dockndevai-mcp-server-suite — feedback + support very welcome!

Giving an AI agent access to production infrastructure is a great way to move fast — and a great way to have it drop a database because a prompt was ambiguous. I wanted the upside without the footguns, so I built a family of Model Context Protocol servers that share one safe-by-default governance model.

The problem

MCP lets an agent call tools. For infra, those tools can be delete_topic, DROP TABLE, delete_resource_group. The usual answer is "just don't expose the dangerous ones" — but then the server isn't useful when you do need them. I wanted graduated, explicit control instead.

The governance model

Every server shares the same layered policy engine:

Access modes — read-only → read-write → admin. Tools above the current mode are never even registered, so the model can't call what it can't see.
Allowlists — scope to specific realms / namespaces / topics / databases / projects / subscriptions.
Protected resources — system/prod resources (kube-system, the system DB, internal Kafka topics, the master realm) are readable but never mutable.
Destructive gating — deletes need an explicit ALLOW_DELETE flag on top of admin mode.
Typed confirmation — high-impact ops (delete a project / resource group) require a confirm value that echoes the exact target name. A boolean isn't enough.
Secret redaction — credentials are stripped before anything reaches the model.
Dry-run + audit — preview writes without executing; every guarded op emits a JSON audit line to stderr.

The servers

All MIT-licensed, TypeScript, published on npm as @dockndevai/mcp-*:

mcp-kubernetes — pods, logs, deployments, scale/restart, apply, exec
mcp-kafka — topics, consumer groups + lag, create/alter/reset
mcp-clickhouse — schema, queries, SQL-classified read/write/destructive gating
mcp-debezium — CDC connector status, config, lifecycle
mcp-oci — Oracle Cloud discovery + Terraform generation
mcp-azure — Azure Resource Manager inventory, tags, VM power, lifecycle
mcp-azure-devops — boards, repos, pipelines, projects
mcp-keycloak — realms, users, clients, roles, groups

Try one

npx -y @dockndevai/mcp-kubernetes

Or in Claude Code:

claude mcp add kubernetes -e KUBECONFIG_PATH=~/.kube/config -e K8S_MODE=read-only -- npx -y @dockndevai/mcp-kubernetes

Every repo has per-client setup for Claude, Cursor, Codex, VS Code, and Windsurf.

Repos: https://github.com/dockndevai

I'd love feedback — especially on whether the mode + typed-confirmation split is the right default for infrastructure MCP servers.

Update — v0.1.1: machine-readable safety

Every tool across all 8 servers now ships MCP tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint), derived automatically from each tool's access capability. The safe-by-default model is no longer just documented in the README — the host can read it and decide what to auto-approve versus prompt on. A test keeps every hint consistent with its tool's capability. Live now on npm and the official MCP registry.