DEV Community: Venkatesh S

From Java Monoliths to LangGraph Agents: What Transfers

Venkatesh S — Mon, 27 Jul 2026 08:40:38 +0000

I spent ten years building enterprise Java systems — a payments monolith broken into Spring Cloud microservices, banking applications, and currently a Cisco security platform. More recently I moved into building LLM systems seriously. The surprise wasn't how much I had to learn. It was how much transferred.

Agentic systems are distributed systems. The vocabulary is new; the failure modes are not.

Timeouts and retries — same problem, worse defaults

A Java service that calls a flaky dependency without timeouts is a junior-engineer mistake. An agent that calls an LLM without them is somehow normal. LLM calls are slow, occasionally hang, and fail in novel ways (malformed JSON, refusals, half-finished tool calls). Everything Resilience4j taught me applies directly: bounded retries with backoff, circuit breakers that fail fast with an honest error instead of silently substituting something expensive, and — this one is new — semantic failure detection, because an LLM can return HTTP 200 with garbage inside.

In Atlas, the gateway circuit-breaks to 503 + Retry-After when the model endpoint is down. That is deliberately boring. Boring is the point.

Idempotency — now with side effects an LLM chose

Backend rule: any handler that can be retried must be idempotent. Agent rule: same, except the retry might be the planner deciding to call the tool again because the first response "didn't look right." If your agent's tools mutate state, you need idempotency keys and an audit trail more than a chat UI needs streaming.

Atlas routes every write through an MCP tool server with audience-scoped tokens (RFC 8707) and an append-only, hash-chained audit log, behind a human-in-the-loop checkpoint for governed actions. That design is not AI innovation — it's what any bank-grade action API looks like. The innovation is refusing to drop the standard just because an LLM is the caller.

Cost budgets are the new capacity planning

In backend work you plan capacity: connection pools, thread pools, rate limits. In LLM systems the scarce resource is spend. The same machinery applies — token-bucket rate limiting, a per-user daily budget guard (in Atlas: Redis-backed, keyed per user per UTC day), alerting at a spend ceiling — plus one new lever: routing. Atlas routes simple queries to a small model and reserves larger models for multi-step reasoning, the way you'd route reads to a replica. Cost regression is gated in CI like a latency budget.

State machines beat vibes

LangGraph clicked for me precisely because it isn't magic: it's an explicit state graph with checkpoints — nodes, edges, persisted state, resumability. If you've designed a workflow engine or a saga with compensation steps, you already think this way. The planner–executor pattern with a durable human-approval checkpoint is a state machine an enterprise architect would recognize from a 2010 BPM diagram. That's a compliment.

What's genuinely new

Honesty requires the other list. Three things had no backend equivalent and took real work:

Evaluation is statistical, not binary. Tests pass or fail; eval metrics move in distributions. Learning to set floors and regression bands (and when to reach for paired bootstrap and McNemar) was new muscle.
The retrieval layer is a ranking problem. Hybrid search, reranking, and abstention thresholds are closer to information retrieval than to anything in Spring.
Prompt injection inverts the trust model. Backend security trusts your own data; RAG security cannot — retrieved documents are untrusted input to the model. Spotlighting and quarantine have no REST-era analogue.

The pitch, condensed

If you're a backend engineer eyeing AI engineering: your instincts about timeouts, idempotency, budgets, audit, and state are not legacy baggage — they're the exact discipline most LLM systems are missing. Learn retrieval, learn evals, keep the discipline.

Permission-Aware RAG: Enforcing Document ACLs at Retrieval Time

Venkatesh S — Mon, 27 Jul 2026 08:40:19 +0000

Most RAG demos have a security model of "none." Documents go into one shared index; anyone who can ask a question can surface content from any document. In a compliance or financial domain, that's not a rough edge — it's a disqualifier.

When I built Atlas, an enterprise copilot for a compliance domain, the first architectural decision was where access control lives. There are three options, and two of them are wrong.

Option 1 (wrong): filter after generation

Generate the answer, then check whether the user was allowed to see the sources. By then the LLM has already read the restricted content and may have leaked it through paraphrase, summary, or even its refusal ("I can't tell you about the Q3 restructuring memo…" is itself a leak). Post-hoc filtering treats a security boundary like a UX problem.

Option 2 (wrong): filter the final chunk list

Retrieve top-k from the shared index, then drop chunks the user can't access. Better, but it corrupts retrieval quality: your top-k gets silently thinned, and ranking scores were computed against a corpus the user shouldn't see. Worse, timing and scoring side-channels can reveal that restricted documents exist.

Option 3: make clearance part of the retrieval predicate

In Atlas, every chunk in pgvector carries a clearance label (public | analyst | compliance | restricted) as metadata, written at ingestion. Retrieval filters by the caller's verified clearance before ranking. This is the actual shape of the dense retrieval query (sparse tsvector search shares the same predicate builder):

SELECT id, document_id, content, clearance, metadata,
       (1 - (embedding <=> ?::vector)) AS score
FROM atlas_chunk
WHERE clearance = ANY(?)        -- caller's visible labels, enforced BEFORE ranking
ORDER BY embedding <=> ?::vector
LIMIT ?;

One detail worth copying: both retrieval paths — dense kNN and sparse full-text — build their WHERE clause from a single shared RBAC filter builder, so there is exactly one trust boundary and it cannot be bypassed. The predicate is pushed into SQL, never applied as an optional post-filter.

The property this buys you: cross-clearance leakage becomes structurally impossible, not "tested for." The restricted document isn't ranked lower or filtered out — from the query's perspective, it does not exist.

The part everyone skips: proving it stays true

A design is worth little without a regression gate. Atlas has a negative-access hard gate in CI: an eval suite where low-clearance users ask questions whose only correct sources are restricted documents. The passing behavior is a grounded refusal — and the gate fails the build on any leakage. It runs GPU-free against committed cassettes, so it gates every merge like a unit test.

Two implementation notes that mattered:

Clearance comes from the verified identity, not the request. The gateway asserts an independently-verified internal JWT; the RAG engine resolves clearance from that assertion and ignores any client-supplied clearance header when a valid assertion is present. (An attacker who can pass clearance=SECRET as a parameter owns your corpus.)
The semantic cache is clearance-partitioned. A cached answer computed for a high-clearance user must never be served to a low-clearance one. Cache keys include the clearance tier — a leak path that's easy to miss because caching is "just performance."

Takeaway

Ten years of backend work taught me that authorization belongs in the data path, not bolted on after. RAG doesn't change that rule; it just gives you a new data path to forget it in. Put the ACL in the retrieval predicate, then write the eval that fails your build when someone breaks it.

Eval-Gated AI Releases: Treating Retrieval Quality Like Unit Tests

Venkatesh S — Tue, 21 Jul 2026 22:42:49 +0000

No backend team would merge a PR that fails the test suite. Yet many AI teams ship prompt and model changes with no automated quality check at all — they eyeball a few responses and hope. The failure mode is silent: answers get slightly less faithful, citations drift, and nobody notices until a user does.

Atlas (my enterprise RAG copilot project) treats evals exactly like tests: a merge is blocked on a quality or cost regression. Here's the shape of that pipeline and what I learned building it.

What gates the build

Every PR runs an offline, deterministic, GPU-free eval gate against committed cassettes:

Metric	Baseline	Floor	Policy
RAGAS faithfulness	0.706	0.656	blocking, regression band 0.05
Answer relevancy	0.832	0.782	blocking
Context recall	0.738	0.668	blocking, band 0.07
Citation correctness	1.000	—	tracked
Adversarial suite (injection)	100% pass	100%	zero-tolerance, blocking
Cost per request	committed baseline	reduction target	blocking (separate cost gate)

Three design decisions made this workable:

1. Floors + regression bands, not perfection. A hard "faithfulness ≥ 0.9" gate would have blocked every merge forever. Instead: an absolute floor (never ship below this) plus a regression band relative to the committed baseline (never get meaningfully worse). This is the eval equivalent of ratcheting code coverage.

2. Deterministic and GPU-free in CI. Live-model evals in CI are slow, expensive, and flaky. Atlas replays committed cassettes — recorded model interactions — so the gate is fast and reproducible. Cassettes are re-recorded deliberately when behavior is supposed to change, which turns "the model changed" into a reviewed diff instead of an ambient surprise.

3. Cost is a gated metric, not a dashboard. A prompt change that doubles token usage is a regression even if quality holds. Atlas's cost gate asserts the measured cost-reduction against a committed baseline and fails the build when cost regresses below the band — the same way a p95-latency budget would gate a backend service.

The payoff: decisions you can defend

The gate earned its keep when I fine-tuned a QLoRA adapter for citation formatting. The benchmark (paired bootstrap, McNemar) showed: citation format validity 0.00 → 0.955 (p ≈ 0), serving cost −79% on the same GPU — and a faithfulness trade-off of −0.10, still above the floor. The promotion gate had the data to make that call explicitly, and the pipeline is proven to both promote and block a candidate in CI.

Without evals, that's a vibes decision. With them, it's an engineering decision with a paper trail.

Where to start (smaller than you think)

You don't need RAGAS on day one. Start with:

20–30 golden questions with expected source documents — your retrieval hit-rate suite.
A handful of adversarial prompts (injection attempts, out-of-scope questions) with expected refusals.
A CI job that fails when either regresses.

That's an afternoon of work, and it changes the culture immediately: prompts become code, model swaps become reviewed changes, and "did we get worse?" has an answer.

Backend engineering solved this problem twenty years ago — we just called it testing. AI systems don't need a new philosophy; they need the old one applied.