DEV Community: Feezan Khattak

You Might Not Need a Vector Database

Feezan Khattak — Wed, 30 Sep 2026 16:25:14 +0000

A RAG pipeline is a lot of parts: a chunker, an embedding model, a vector database, a retriever, usually a reranker, and an eval harness to tell you when retrieval quietly got worse.

Cache-augmented generation (CAG) deletes all of it. You put the entire knowledge base in the prompt, cache it at the provider, and ask your question. No retrieval step, so no retrieval mistakes.

That sounds like a stunt until you do the arithmetic. Then it turns into a straightforward rule.

TL;DR

CAG = the whole corpus in the prompt + prompt caching. With a hosted API, that's the whole implementation.
The rule: cache the corpus when it's less than ~10× what you'd retrieve. Cached input costs about 0.1× normal input, so 10× the tokens costs roughly the same.
The real constraint is the cache TTL, not the context window. Entries expire in minutes.
Sparse traffic makes CAG the most expensive option, not the cheapest.
Keep RAG for big corpora, fast-changing content, per-user permissions, and provenance.

What CAG actually is

The paper that named it — Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks — preloads the documents into the model's context, computes the KV cache once, and answers queries against that. No retriever at query time.

Running your own model, you hold that cache in GPU memory. On a hosted API you get the same effect from prompt caching: the provider stores the processed prefix and charges a fraction to reuse it.

So CAG isn't a framework you install. It's a prompt layout:

Put the whole corpus at the front, in a fixed byte-for-byte order.
Mark the end of it as a cache breakpoint.
Put the question after the breakpoint, where it changes freely.

Not the same as caching responses. Response caching stores the answer so an identical question skips the model. Prompt caching stores the processed input so a different question over the same documents skips re-reading them. They compose well — response cache in front, prompt cache behind.

The arithmetic

Cached input tokens cost roughly 0.1× normal input tokens — that holds for Anthropic and, on current models, OpenAI. Writing the cache costs about 1.25×, once.

That one number gives you the decision. Let C be your whole corpus in tokens and R the tokens you'd send per query with retrieval:

Caching the corpus costs about the same as retrieving when C ≈ 10 × R.

Below that, CAG is cheaper per query and has no vector database to run.

Worked through at $5 per million input tokens (so cached reads around $0.50 per million):

Approach	Tokens per query	Cost per query
No cache, whole 200K corpus every time	200,000	$1.00
CAG, 200K corpus cached	200,000 cached	$0.10
RAG, 6K of retrieved chunks	6,000	$0.03
CAG, 20K corpus cached	20,000 cached	$0.01

At 200K tokens, retrieval still wins on tokens. At 20K — a product manual, an API reference, a policy handbook — caching the lot is cheaper than retrieving part of it, and you delete the entire pipeline.

And the table leaves out everything RAG costs besides tokens: a vector database to run, an embedding model called on every write and every query, a chunking strategy to tune, a reranker, and an eval harness to catch silent regressions.

The TTL is the real constraint

Everyone asks whether the corpus fits. Current flagship models take a million tokens; most corpora that matter fit. The question that actually decides it is how often you ask.

Provider	Cache lifetime
Anthropic	5 minutes by default, 1-hour TTL at 2× the write price. Every read refreshes the timer.
OpenAI (GPT-5.6+)	A prefix stays eligible for about 30 minutes after its last use.
Gemini	Implicit caching on by default for 2.5 and newer.

So the break-even is about traffic shape:

Gap between queries sharing the corpus	What happens
Under 5 minutes	Every query refreshes the entry. Pay the write once, read cheaply forever. CAG shines.
5–60 minutes	Use a longer TTL or re-warm on a schedule. The doubled write price needs a few reads to pay off.
Hours apart	Every query is a cold write at 1.25–2× full price. CAG is the most expensive option available.

A trap worth knowing: every provider has a minimum cacheable prefix, and below it caching is skipped with no error — you just quietly pay full price forever. Anthropic's is model-dependent (512–4096 tokens), OpenAI's is 1,024 on GPT-5.6+, Gemini needs 2,048–4,096. Check the usage fields on the response, not the docs.

The layout in Spring Boot

Corpus in the system prompt with a cache breakpoint; question in the user message, after it.

@Service
public class HandbookAnswerService {

    private final AnthropicClient client = AnthropicOkHttpClient.fromEnv();

    // Built once at startup, in a fixed order. Any byte that changes here
    // invalidates the cache for every request that follows.
    private final String corpus;

    public String answer(String question) {
        MessageCreateParams params = MessageCreateParams.builder()
                .model("claude-opus-5")
                .maxTokens(4096L)
                .systemOfTextBlockParams(List.of(
                        TextBlockParam.builder()
                                .text(corpus)
                                .cacheControl(CacheControlEphemeral.builder()
                                        .ttl(CacheControlEphemeral.Ttl.TTL_1H)
                                        .build())
                                .build()))
                .addUserMessage(question)   // after the breakpoint - varies freely
                .build();

        Message response = client.messages().create(params);

        // The only proof that caching is working.
        log.info("cache write={} read={} uncached={}",
                response.usage().cacheCreationInputTokens(),
                response.usage().cacheReadInputTokens(),
                response.usage().inputTokens());

        return text(response);
    }
}

Two details do most of the work:

The corpus never varies. Caching is a prefix match on exact bytes. A timestamp, a reordered map, a "user: Feezan" line — any of those changes the prefix and every request after it misses. Sort deterministically, interpolate nothing.

The question sits after the cached block. Putting the breakpoint at the very end is the common mistake: each unique question writes its own entry that nothing ever reads, so you pay the write premium on every call.

Log those usage fields from day one. If the write count is large on every call, something upstream is rewriting your prefix — and this failure is invisible, because requests still succeed and only the bill changes. Assert on it in a test.

When RAG still wins

The corpus doesn't fit, or barely fits. Above a few hundred thousand tokens you're re-reading everything for every question, and long-context accuracy isn't free either — measure it.
It changes constantly. One edited document changes the prefix and rewrites the whole cache at the write premium. A handbook revised weekly is ideal; a ticket system is not.
Different users see different documents. Per-user filtering means a separate cached prefix per permission set.
You need provenance. RAG hands you the chunks, so you can cite them. With everything in context, you're trusting the model to cite honestly.
Volume is large and steady. At high volume, retrieval's smaller prompt wins again.

The hybrid is usually the answer at scale: cache the stable material (schemas, policies, API reference, instructions) as a fixed prefix, and retrieve only the volatile part after the breakpoint.

The takeaway

RAG is a solution to a size problem. If your knowledge base isn't actually that big, you can skip the pipeline, cache the whole thing, and spend the time you saved on answer quality instead of retriever tuning.

Count your corpus before you build. If it's under about ten times what your retriever would send per query, try the boring version first.

I write about backend engineering and payments at feezankhattak.com. The longer version of this post has the full gotchas list, and I build free in-browser developer tools — no sign-up, nothing uploaded.

If Your Spring AI Streams Die After 60 Seconds, It's Not Your Network

Feezan Khattak — Tue, 29 Sep 2026 03:24:33 +0000

If you're on Spring AI 2.0.1 and your streaming calls die after about a minute with OpenAIIoException: Stream failed, stop debugging your proxy.

Since 2.0.1, every request carried a hard-coded 60-second per-call timeout that overrode whatever you configured. spring.ai.openai.timeout and spring.ai.openai.chat.timeout were ignored. Any streaming turn longer than a minute was killed, no matter what you set.

Spring AI 2.1.0-M1, released 25 September 2026, fixes it. That alone is worth the ten minutes — but the release also lands the change that the next two versions are built on.

TL;DR

The 60-second timeout regression is fixed. Your configured timeout is honoured again.
Message parts: messages are now an ordered list of typed parts, so reasoning, tool calls and media round-trip in the order the model produced them.
OpenAI Responses endpoint: one property. Needed if you want tool calling and reasoning effort on a current OpenAI flagship.
VectorStore.upsert(): write embeddings you computed somewhere else, with stable ids that make re-runs safe.
It's a milestone. APIs can change, and it's built against Spring Boot 4.2.0-M2.

The fix, first

The generic lesson is one I keep relearning in payments work: the shortest timeout in the chain wins, and it's usually one you didn't set. A load balancer, a gateway, an SDK default — or, here, a framework regression.

When a call dies at a suspiciously round number of seconds, that's rarely a coincidence. 60 seconds, 29 seconds, 30 seconds: those are configured limits, not network weather.

Three other fixes in the same release:

OpenAI strict-mode tool schemas are backfilled with additionalProperties: false at every object level, which fixes strict mode for MCP or hand-written schemas that didn't come from Spring AI's own generator.
Bedrock messages carrying media but no text no longer send an empty text block, which the Converse API rejected with a 400.
TextReader closes its stream instead of leaking a file descriptor per document.

Message parts: the actual headline

Until now, a Spring AI message was text plus side lists for tool calls and media. That can't represent what current models return: reasoning interleaved with tool calls, text between two images, or provider-specific blocks that have to go back verbatim next turn.

Now AssistantMessage, UserMessage and ToolResponseMessage hold an ordered list of MessagePart:

Part	Holds
`TextPart`	plain text
`ReasoningPart`	the model's reasoning
`ToolCallPart`	a requested tool call
`ToolResultPart`	the result you sent back
`MediaPart`	an image or file
`UnknownPart`	raw JSON for a block the adapter doesn't model yet

UserMessage question = UserMessage.builder()
    .part(TextPart.of("What changed between these two charts?"))
    .part(MediaPart.of(firstChart))
    .part(MediaPart.of(secondChart))
    .build();

AssistantMessage answer = chatModel.call(new Prompt(question)).getResult().getOutput();
for (ReasoningPart reasoning : answer.getReasoning()) {
    System.out.println("Reasoning: " + reasoning.text());
}
System.out.println(answer.getText());

Two details matter more than they look:

OpaquePayload on reasoning and tool-call parts holds things that must be replayed unchanged — an Anthropic thinking signature, a Gemini thought signature. Without a defined home for those, a framework either drops them (and the model loses its train of thought mid tool loop) or grows a provider-specific hack.

UnknownPart keeps the raw JSON of blocks the adapter doesn't understand yet. Nothing is silently dropped when a provider ships a new block type faster than the framework models it.

Your existing code keeps working: getText(), getMedia() and getToolCalls() are now views over the parts, and the old constructors still produce the legacy order. Streaming subscribers reading getText() see the same deltas as before.

Two caveats the release is explicit about: only the new OpenAiResponsesChatModel produces and consumes parts natively in M1 (the rest get refactored in RC1), and chat memory repositories don't persist parts yet — so only InMemoryChatMemoryRepository preserves reasoning across turns. With a persistent repository the conversation works, but the model reasons from scratch each turn.

The OpenAI Responses endpoint

# chat-completions (default) | responses
spring.ai.openai.chat.api=responses

The rest of your spring.ai.openai config stays where it is.

The reason isn't novelty, it's correctness: from GPT-5.4 onward, Chat Completions doesn't support tool calling combined with a reasoning effort other than none. Responses does. Building an agent on a current OpenAI flagship? That's the endpoint.

It's built on message parts, so encrypted reasoning rides in a ReasoningPart and goes back verbatim across tool calls — the model keeps reasoning through the loop. It's deliberately stateless (every call sends the whole Prompt), so ChatMemory, advisors and RAG behave exactly as before. HostedTool switches on OpenAI's server-side tools: web search, file search, code interpreter, remote MCP, image generation.

Pre-computed embeddings

VectorStore.add() always computed embeddings with the store's own model. That's wrong whenever you already have vectors — a batch API ran overnight at a discount, another team owns the pipeline, a multimodal model embedded an image, or you're migrating from a system that exports text and vectors together.

float[] embedding = ...; // computed elsewhere
Document document = new Document("8f14e45f-ceea-467a-9a3e-5b1c2d6f7a90", "some text",
        Map.of("source", "manual"));

vectorStore.upsert(List.of(new EmbeddedDocument(document, embedding)));

The name is the feature. Writing the same id again replaces the row, so an ingestion job with stable ids can be re-run after a failure instead of duplicating everything — the difference between an idempotent pipeline and a nightly job nobody dares restart.

Supported here: pgvector, Redis, Elasticsearch, Qdrant. Other stores throw until they opt in.

Should you take it?

Take it now if	Wait for GA if
You're hitting the 60-second stream timeout	2.0.x is stable for you with no symptoms
You need tool calling and reasoning effort on a current OpenAI model	You need reasoning persisted across turns (RC1)
You want to write vectors computed elsewhere	Your vector store isn't one of the four
You can move to Spring Boot 4.2	You're pinned to an earlier Boot line

What RC1 is building on this

Message parts are the foundation for most of what's next: parts across all providers, session management moving into core (with repositories that persist the full part structure, so reasoning finally survives persistent storage), and MCP 2026-07-28 support. Spring AI Agents ships as a separate experimental project in November 2026, planned to merge into Spring AI 3.0 mid-2027.

The one thing to do after reading this: grep your logs for Stream failed. If it's there, you've just found out why.

I write about backend engineering and payments at feezankhattak.com — the longer version of this post has the full gotchas list, and I build free in-browser developer tools, no sign-up and nothing uploaded.

There's a New AI Model That Can't Write a Single Word

Feezan Khattak — Sat, 26 Sep 2026 15:42:59 +0000

Count the LLM calls in your backend that end in a parser.

Not the ones that write an email or summarise a document — the ones where you asked a chat model a question whose answer is a boolean, a label, or a number between 1 and 5. Then you wrote a prompt template, a JSON schema, a parser, and a retry for the time it replied "Sure! Here's my assessment:" instead of {"urgent": true}.

Jev, released by TypeSafe AI on 15 September 2026, deletes that whole layer — by refusing to generate text at all.

TL;DR

Jev answers typed questions: yes/no, pick-one, score-on-a-rubric. No prose, ever.
Every answer comes back with a probability, and most with a confidence as well.
No prompt template, no schema, no parser, no repair step. The shape is the request.
TypeSafe quotes $0.042 per million input tokens, free output, 70–500 ms. Cheap enough to check every expensive call instead of sampling.
It does not replace your chat model. It decides what to do with what the chat model produced.

What a "System One model" is

The name refers to fast, intuitive thinking as opposed to slow deliberation. TypeSafe describes the target as "a judgment a knowledgeable person makes in a second given the right context."

Three things matter to a backend engineer:

It isn't autoregressive. No token-by-token generation, which is why answers arrive in a few hundred milliseconds instead of seconds.
The output is type-constrained. You don't ask for JSON and hope. The type is part of the request.
Answers carry calibrated probabilities, not just values.

The trade is absolute: it gives up string generation entirely. It cannot write you a sentence.

The entire API surface

Primitive	You ask	You get back
Noul	a yes/no question	a truth value in `[0, 1]`
Choice	pick one label	the label, a probability per option, and a confidence
Score	place on an ordered rubric	a continuous value, the legend, per-level probabilities, and a confidence

That's it. Three shapes.

A Noul has no separate confidence because the value is the certainty — 0.5 means undecided. A Score is continuous, so 1.1 on a three-level rubric means "just past the middle level", which is what makes a threshold like 2.0 mean something.

From Spring Boot

There's a Spring AI Community integration, version 0.1.0 on Maven Central:

<dependency>
    <groupId>org.springaicommunity</groupId>
    <artifactId>spring-ai-starter-typesafe</artifactId>
    <version>0.1.0</version>
</dependency>

Set TYPESAFE_API_KEY, and auto-configuration gives you a TypeSafeClient. Here it is triaging a payment complaint — the kind of ticket I spend my working life near:

SystemOneResponse response = typeSafeClient.systemOne(
        "Customer says: my card was charged twice for order 4417 but only one order shows up.",
        Map.of(
            "duplicate_charge", Noul.builder()
                    .instructions("Is the customer reporting being charged more than once for one purchase?")
                    .whenTrue("The customer describes multiple charges for a single order")
                    .whenFalse("The customer describes a single charge, or a different problem")
                    .build(),
            "queue", Choice.builder()
                    .instructions("Which queue should handle this?")
                    .option("payments-ops", "Duplicate charges, stuck payments, reconciliation mismatches")
                    .option("disputes",     "Chargebacks, evidence submission, representments")
                    .option("billing",      "Invoices, refunds, subscription changes")
                    .build(),
            "urgency", Score.of("How urgent is this for the customer?",
                    "Can wait", "Needs attention today", "Money is missing right now")));

double duplicate = response.noulValue("duplicate_charge");
String queue     = response.choiceValue("queue");
double certainty = response.choice("queue").confidence();

Three questions, three typed answers, one call, no parsing.

Write real option descriptions. The Spring post measured the difference: the same ticket with bare labels instead of described options dropped confidence from 0.82 to 0.60. The descriptions are how the model learns what your labels mean.

The idea worth stealing: confidence is a routing decision

This is the part that changes how you write the calling code, even if you never use Jev.

The answer tells you what. The confidence tells you whether to act on it unattended. One number can't carry both — which is exactly why the classic "rate this 1–5" prompt fails: one catastrophic flaw gets averaged into a middling score.

So you get three branches, not two:

if (certainty < 0.5) {
    humanReview(ticket, response);          // undecided is not "wrong"
} else if (duplicate > 0.8) {
    payments.investigateDuplicate(ticket);  // confident and true
} else {
    route(queue, ticket);
}

"Not unattended" becomes a first-class outcome instead of an error case. Anyone who has built a fraud queue or a dispute workflow already recognises this shape.

Cost and latency — and who is claiming what

Source	Claim
TypeSafe	$0.042 / MTok input, output free ("too cheap to meter")
TypeSafe	70–500 ms end to end; 40–200× faster, 40–400× cheaper than frontier LLMs
TypeSafe's cookbook	a 14-question call at $0.000043 and 111 ms vs ~$0.0018 / 1.8 s for a small chat model
Spring blog author, measured on a laptop	median 275 ms for one question, 310 ms for three

The multipliers are vendor benchmarks on tasks the vendor chose. Treat them as an order of magnitude, not as a line in a business case — but note that two extra questions cost the Spring author 35 ms, because everything is answered against one read of the state.

The real consequence isn't saving money on calls you already make. It's that at thousandths of a cent, a check can run in front of every expensive call rather than on a sample: a gate on a model cascade, a relevance filter before a long-context generation, a verification pass on output you currently ship unchecked.

Things that will bite you

No streaming. Nothing is generated token by token, so there is nothing to stream.
State must be a string, object, array or null. A bare number or boolean returns a 422.
Per-document work is one call per document. Reranking a top-20 list is twenty calls — filter first, rank the survivors.
Choice always names a winner, because the probabilities sum to one. If "none of these" is a real answer, ask it as a separate Noul.
Early access, US West Coast. That's a real latency budget if you serve from elsewhere.

What a typed output does not fix

A typed answer can't be malformed. That genuinely kills a class of bug: the parse failure, the apology instead of an answer, the "yes, but..." where you expected a boolean.

It does not make the answer right. A calibrated probability is still a probability. In payments, where I work, that means:

Keep it out of the authorization path. A decision that moves money needs a deterministic rule and an audit trail, not a model's 0.91.
Anything you must justify to a regulator or card scheme needs a reason. "The model scored it 0.87" is not one.
Threshold choice is a product decision. Where you set the confidence floor decides how much work lands on a human.

Where this leaves us

Most backends have been using one tool for two jobs, because there was only one tool. Chat models write. System One models decide.

Go and look for the calls in your codebase that end in a parser. That's the list.

I write about payments and backend engineering at feezankhattak.com. The longer version of this post covers the guardrail and reranking integrations, and I build free in-browser tools for backend developers — no sign-up, nothing uploaded.

Your Payment Webhook Handler Probably Processes Duplicates

Feezan Khattak — Sat, 19 Sep 2026 05:53:22 +0000

The customer's browser reaches your success page and calls your API: "the payment worked, please ship my order."

Anyone can make that call. With curl. For free.

The fix everyone knows is "use webhooks." The part fewer people get right is what happens next: webhooks arrive more than once, out of order, and the most common duplicate check in Spring Boot doesn't catch duplicates at all.

TL;DR

Never fulfil an order because the browser said so. Only a verified webhook can tell you money moved.
Verify the signature over the raw request body, before parsing.
Deduplicate with INSERT ... ON CONFLICT DO NOTHING — not save() and catch the exception. In Spring Data JPA that version quietly does nothing.
One change can produce two events with different ids. Make applying a change twice harmless.
Store, then return 200. Do the slow work later.

The browser can't be trusted — even when it's honest

Two separate problems:

It's forgeable. POST /orders/123/confirm can be sent by anyone who reads your JavaScript.
It's unreliable. The customer closes the tab, loses signal on the redirect, or finishes 3-D Secure and the callback never lands. They've paid, and you never heard about it.

The browser can hint — show a spinner, an optimistic message. It must never be what triggers fulfilment.

Verify the signature over raw bytes

@PostMapping("/webhooks/stripe")
public ResponseEntity<Void> handle(
        @RequestBody String payload,                        // raw String, not a DTO
        @RequestHeader("Stripe-Signature") String signature) {

    Event event;
    try {
        event = Webhook.constructEvent(payload, signature, webhookSecret);
    } catch (SignatureVerificationException e) {
        log.warn("rejected webhook with bad signature");
        return ResponseEntity.badRequest().build();
    }
    ...
}

The signature covers the exact bytes the provider sent. Bind the body to a DTO first and you've lost them — re-serialising produces different JSON (key order, whitespace, number formatting), verification fails, and someone "temporarily" turns it off. Stripe's docs say it plainly: any change to the raw body makes verification fail.

The signature also covers a timestamp, which stops a captured request being replayed later. Stripe's libraries reject anything older than five minutes by default. Keep your server clock synced, and never set the tolerance to 0 — that disables the check.

The duplicate check that doesn't check

Delivery is at-least-once. You'll get the same event again after a timeout, after your 500, and occasionally for no visible reason. So you record processed event ids — and this is what most of us write first:

@Transactional
public void process(Event event) {
    try {
        processedEvents.saveAndFlush(new ProcessedEvent(event.getId(), Instant.now()));
    } catch (DataIntegrityViolationException duplicate) {
        return;                       // already processed
    }
    applyEffects(event);
}

It looks right. It fails in two different ways.

1. It never sees the duplicate. ProcessedEvent's id is the provider's event id — assigned by you, never null. With no @Version field, Spring Data can't tell it's new, so save() calls merge() instead of persist(). Merge loads the existing row and updates it. No exception, and the repeated event gets processed again.

2. When it does throw, you can't continue. Two deliveries racing each other can both reach the insert, and one does get the duplicate-key error. But that exception passes through the repository's own transactional proxy, which marks your transaction rollback-only — and PostgreSQL has already aborted the transaction anyway. You catch it and return normally, and Spring throws UnexpectedRollbackException at commit. That's a 500, which the provider retries — for up to three days, in Stripe's case.

What works: let the database answer

CREATE TABLE processed_webhook_events (
    event_id     text PRIMARY KEY,
    processed_at timestamptz NOT NULL
);

public interface ProcessedEventRepository extends Repository<ProcessedEvent, String> {

    // 1 = we claimed it, 0 = already processed. Never throws on a duplicate.
    @Modifying
    @Query(value = """
            INSERT INTO processed_webhook_events (event_id, processed_at)
            VALUES (:eventId, now())
            ON CONFLICT (event_id) DO NOTHING
            """, nativeQuery = true)
    int markProcessed(@Param("eventId") String eventId);
}

@Transactional
public void process(Event event) {
    if (processedEvents.markProcessed(event.getId()) == 0) {
        return;                       // already processed
    }
    applyEffects(event);              // same transaction: if this throws, the claim rolls back too
}

Three things this gets right:

Duplicates are a normal result, not an error. The transaction stays usable.
Concurrent deliveries are safe. The second insert waits for the first transaction to finish, then inserts nothing.
Half-processed events aren't marked done. The claim commits or rolls back with the effects, so a failure gets retried properly.

I checked the SQL behaviour for this post on PostgreSQL (via PGlite): the first delivery claims one row, a duplicate returns zero with no error and leaves the transaction usable, a plain INSERT duplicate aborts the transaction, and a rolled-back claim is processed again on retry.

Event ids aren't the whole story

Stripe notes that it sometimes generates two separate event objects for the same change — with different ids. Your id table won't catch those. Which leads to the more robust idea...

Handle events as target states, not steps

Events don't arrive in order either. Stripe's own example: creating a subscription can produce customer.subscription.created, invoice.created, invoice.paid and charge.created in any order. And don't sort by the event's created field — it's in whole seconds, so events can tie.

So a succeeded event should mean "this payment is now captured", not "advance one step". And guard against moving backwards:

public enum PaymentState {
    INITIATED(0),
    REQUIRES_ACTION(1),
    AUTHORIZED(2),
    CAPTURED(3),
    REFUNDED(4);

    final int rank;   // every constant must declare one - the compiler enforces it

    PaymentState(int rank) { this.rank = rank; }
}

private void applyState(Payment payment, PaymentState incoming) {
    if (incoming.rank <= payment.getState().rank) {
        return;       // stale, out of order, or a repeat - ignore it
    }
    payment.setState(incoming);
}

Two small choices matter here:

The rank lives on the enum, not in a Map. A map returns null for a state someone adds next year, and the unboxing throws inside your webhook handler.
Repeats become harmless. A second CAPTURED is ignored whatever its event id — which covers the two-events-one-change case.

Failure states don't fit on a line: a failed payment can still succeed when the customer tries another card. For those, fetch the object's current state from the provider's API. The API is the source of truth; events are notifications.

Return 200 fast — but only after storing

Providers treat a slow response as a failure and retry. Do fulfilment, emails and invoices inline, and you've built your own retry storm.

@PostMapping("/webhooks/stripe")
public ResponseEntity<Void> handle(@RequestBody String payload,
                                   @RequestHeader("Stripe-Signature") String sig) {
    Event event = verify(payload, sig);     // fast
    inbox.store(event.getId(), payload);    // INSERT ... ON CONFLICT (event_id) DO NOTHING
    return ResponseEntity.ok().build();     // ack - a duplicate still gets a 200
}

Then process the inbox asynchronously, using the markProcessed claim above.

The status code cuts both ways:

Return a 500 on a transient problem and you get a free retry.
Return a 200 on an event you didn't store and it's gone for good — the provider thinks it was delivered.

So: acknowledge once the event is durably stored, not once it's fully processed. And make the inbox insert conflict-safe too, or a duplicate delivery hits the primary key and gets a 500.

Checklist

[ ] Nothing fulfils on a browser-side success signal
[ ] Webhook body bound as a raw String, verified before parsing
[ ] Signature timestamp tolerance left on (not 0)
[ ] Event ids claimed with INSERT ... ON CONFLICT DO NOTHING, in the same transaction as the effects
[ ] No save()-and-catch for dedupe
[ ] Handlers set a target state and ignore regressions
[ ] Store, then 200; process asynchronously; duplicates still get a 200
[ ] livemode checked, so test events can't touch production
[ ] Amounts and order ids checked against your own records

Your provider's webhook is the only party that can tell you money moved. Verify it, expect it twice, expect it out of order — and let the database, not an exception handler, decide what's a duplicate.

Part 3 of a series on building payment systems as a backend engineer. The full version on my site also covers validating event amounts against your own records. If you're debugging signatures right now, there's a free webhook signature verifier that runs entirely in your browser — nothing is uploaded.

Your Payment Timed Out. Was the Customer Charged?

Feezan Khattak — Tue, 15 Sep 2026 17:28:23 +0000

Every other payment failure tells you something.

A decline is an answer. A validation error is your bug. A refused connection means nothing happened.

A timeout means you don't know. The charge may have succeeded, failed, or never arrived — and one of those outcomes has the customer's money.

Most integrations treat a timeout as a failure. That's a guess. It's wrong often enough to generate a steady stream of "I was charged but my order failed" support tickets.

TL;DR

A read timeout is ambiguous. Never mark the payment FAILED because of one.
A connect failure is different: nothing was sent, so it's safe to record as failed and retry.
You need an explicit UNKNOWN state, or your code is forced to guess.
Resolve unknowns in layers — and "not found" is not "failed".
The shortest timeout in your request chain wins, and it's usually one you forgot.

What can be true after a timeout

You can't tell these apart from your side:

What actually happened	Customer charged?	Naive code records
Request never left your network	No	failed ✅
Arrived, issuer declined	No	failed ✅
Arrived, issuer approved, response lost	Yes	failed ❌
Still in flight when you gave up	Possibly, a moment later	failed ❌

The third row is the expensive one: the customer sees the charge, your system has no record of it, and they contact support or dispute it. The fourth is worse — the charge completes after you've told them it failed, so they try again.

Don't manufacture your own timeouts

Before handling ambiguity better, stop creating it. Authorization has one genuinely slow step — the issuer's fraud check — and if your read timeout is shorter than that, you're turning successful payments into ambiguous ones yourself.

HttpClient client = HttpClient.newBuilder()
        .connectTimeout(Duration.ofSeconds(5))   // can't reach them: fail fast, safe to retry
        .build();

HttpRequest request = HttpRequest.newBuilder(uri)
        .timeout(Duration.ofSeconds(30))          // read: generous, above realistic p99
        .header("Idempotency-Key", key)
        .POST(body)
        .build();

Short connect timeout. If the connection never opens, nothing was sent. Unambiguous.
Long read timeout. Once the request is out, giving up early buys you nothing. The payment keeps processing at the other end whether you're listening or not.

The shortest timeout in the chain wins

Your client's 30 seconds is irrelevant if something in front of it gives up sooner. Check every hop:

Load balancer or ingress idle timeout
Reverse proxy (proxy_read_timeout in nginx)
Service mesh
API gateway
Your framework's own request timeout
The browser, for synchronous user-facing requests

A real example: your service waits 30 seconds for the payment provider, but it sits behind an AWS API Gateway REST API, whose integration timeout defaults to 29 seconds. A payment that takes 31 seconds becomes a 504 for your caller — while the charge completes behind it. If the caller retries, you're one step away from a double charge.

It's also a strong argument for making payment initiation asynchronous, so no user-facing timeout can interrupt it.

UNKNOWN is a required state

public enum PaymentState {
    INITIATED,
    AUTHORIZED,
    CAPTURED,
    DECLINED,   // a real answer from the issuer
    FAILED,     // we know it didn't happen
    UNKNOWN,    // we genuinely don't know - must be resolved
    REFUNDED
}

DECLINED and FAILED are knowledge. UNKNOWN is the honest absence of it — and it's what lets you defer the decision instead of guessing:

try {
    ChargeResult result = provider.charge(payment.storedRequest(), payment.getIdempotencyKey());
    payment.setState(result.approved() ? AUTHORIZED : DECLINED);

} catch (HttpConnectTimeoutException | ConnectException nothingSent) {
    // The connection was never established, so nothing was sent.
    // Not ambiguous: record it, and it's safe to retry.
    payment.setState(FAILED);

} catch (IOException ambiguous) {
    // Sent, then lost - including a read timeout. We do NOT know.
    payment.setState(UNKNOWN);
    log.warn("payment {} ambiguous, key={}", payment.getId(), payment.getIdempotencyKey());

} finally {
    payments.save(payment);
}

The order of those catch blocks matters: HttpConnectTimeoutException is a subclass of HttpTimeoutException, which is an IOException, so the specific case has to come first. This structure was compiled on Java 21 and checked against real sockets — a refused connection and a connect timeout land in FAILED, a server that accepts the request and never replies lands in UNKNOWN.

If you use a provider's SDK rather than a raw HttpClient, it wraps these in its own exception types, so check which ones it throws. The split is what matters: "couldn't connect" is an answer; "sent and didn't hear back" isn't.

And make the customer-facing message match what you actually know: "We're confirming your payment" — never "Payment failed, please try again," which invites the duplicate charge you're trying to avoid.

Resolving unknowns — and the mistake that fails real payments

The provider knows what happened. But how you ask matters, because the obvious way to ask can give you a confident wrong answer.

Resolve in layers:

Webhooks. Most unknowns resolve themselves when the provider sends the success or failure event. The sweep job below is for the ones that don't.
Replay the identical request with the same idempotency key while the provider still remembers it. You get the saved result of the original — or, if it never executed, it runs now, exactly once. Stripe keeps keys for at least 24 hours, returns the saved result even for a stored 500, and rejects a replay whose parameters differ.
A strongly consistent lookup — by the provider's own ID if you got one, or a list call filtered by customer and time window, matched on your reference in metadata.

The trap is treating "not found" as "never happened". Some providers don't offer a lookup by idempotency key at all, and search endpoints are often eventually consistent. Stripe's docs say not to use its Search API right after a payment: new data is normally searchable within a minute, and it can take longer during an outage. A resolver that marks a payment FAILED because a search came back empty will, sooner or later, fail a payment that succeeded.

@Scheduled(fixedDelay = 30_000)
public void resolveUnknownPayments() {
    Instant graceCutoff = Instant.now().minus(Duration.ofMinutes(2));

    for (Payment p : payments.findByStateAndUpdatedAtBefore(UNKNOWN, graceCutoff)) {
        try {
            Optional<ChargeResult> answer = resolve(p);
            if (answer.isPresent()) {
                adopt(p, answer.get());            // definitive: approved or declined
            } else if (p.isUnknownLongerThan(Duration.ofHours(1))) {
                escalate(p);                       // a human or reconciliation takes over
            }
        } catch (Exception e) {
            // A lookup error is not an answer. Leave it UNKNOWN.
            log.error("could not resolve payment {}", p.getId(), e);
        }
    }
}

private Optional<ChargeResult> resolve(Payment p) {
    // Replay only well inside the key's retention window, and only if the
    // customer's intent still stands (order not cancelled in the meantime).
    if (p.getFirstAttemptAt().isAfter(Instant.now().minus(Duration.ofHours(20)))
            && p.orderStillWantsPayment()) {
        ChargeResult replay = provider.charge(p.storedRequest(), p.getIdempotencyKey());
        if (replay.isDefinitive()) {
            return Optional.of(replay);
        }
    }
    // Strongly consistent lookup - never a search endpoint.
    return provider.findByReference(p.getCustomerRef(), p.getReference(), p.getFirstAttemptAt());
}

Four details that matter:

Grace period from the last attempt (updatedAt, not createdAt), so you don't resolve a payment underneath a retry that's still running.
Nothing becomes FAILED because it wasn't found. Only a definitive answer changes the state. Everything else stays UNKNOWN and escalates.
A lookup error stays UNKNOWN. Never let a resolver bug downgrade a payment.
Persist the key and the exact request before the first call. A replay needs both.

Alert on age, not just errors

A few UNKNOWN payments are normal. What matters is the shape:

Count rising — provider degradation, or your timeouts are too tight.
Age rising — your resolver is broken. This one is invisible on error dashboards, because nothing throws: payments just pile up in limbo.

Alert on the age of the oldest unresolved payment. If anything has been UNKNOWN for more than a few minutes, someone should know.

Retrying after a timeout

Safe under one condition: the same idempotency key. With the original key, a retry completes the original operation or returns its saved result. With a new key, you've authorized a second charge.

And the key only protects you while the provider remembers it. Retry after the retention window and it's a brand-new request — which is why you also need the database guard from part 1 of this series, where a payment stuck in UNKNOWN blocks a second attempt at the database level.

Cap retries at two or three. Hammering a struggling provider is how a slowdown becomes an outage.

Checklist

[ ] Read timeouts never write FAILED
[ ] Connect failures handled separately from read timeouts
[ ] An explicit UNKNOWN state
[ ] Read timeout above your realistic p99, connect timeout short
[ ] Every timeout in the chain audited — gateway, proxy, load balancer, client
[ ] Idempotency key and exact request persisted before the call
[ ] Resolution in layers: webhooks → same-key replay → strongly consistent lookup
[ ] "Not found" stays UNKNOWN; no search endpoints in the resolver
[ ] An alert on the age of the oldest UNKNOWN payment

A timeout is a question, not an answer. Code that treats it as an answer is wrong a meaningful fraction of the time — and always in the direction that costs the customer money.

Part 2 of a series on building payment systems as a backend engineer. The full version on my site goes deeper on the resolver. I also build free, in-browser tools for payment engineers — including a card decline code lookup and a webhook signature verifier.

Your Payment Code Can Still Charge Twice (Even With Idempotency Keys)

Feezan Khattak — Sun, 13 Sep 2026 05:55:46 +0000

A double charge is the payment bug customers actually notice. Two lines on a bank statement, an angry support ticket, and if it's handled badly, a dispute — which costs you a fee whether you win or not.

Most engineers reach for idempotency keys, and they should. But idempotency keys stop duplicate requests. They don't stop your own system making two legitimately different requests for the same order. That second category is where double charges that survive code review come from.

TL;DR

Idempotency keys stop duplicate requests, not duplicate intents.
Disabling the Pay button is a courtesy, not a control.
Most remaining causes are races, and read-then-write checks lose races.
The reliable fix is a partial unique index — and which states it covers decides whether it works at all.
Never hold a database transaction open across the call to your payment provider.

Eight ways a customer gets charged twice

#	Cause	Fix
1	Customer double-clicks Pay	UI guard + server idempotency
2	Retry sent with no idempotency key	Always send one
3	Retry sent with a new key	Generate before attempt 1, persist it
4	Two browser tabs / two devices	Server-side unique constraint
5	Mobile app retries on resume	Key survives app restart
6	Webhook handler charges again	Handlers record state, never charge
7	Two concurrent requests race a check	DB constraint, not `if (exists)`
8	Dunning job races a manual retry	Guard scoped to the billing period

Causes 1–3 and 5 are solved by doing idempotency properly. 4, 6, 7 and 8 are not — they're your system charging twice with two different, valid requests.

The race that careful-looking code loses

// WRONG: two concurrent requests can both pass this check
public Payment charge(String orderId, long amountMinor) {
    if (payments.existsByOrderIdAndStateIn(orderId, SUCCESSFUL_STATES)) {
        throw new AlreadyPaidException(orderId);
    }
    return doCharge(orderId, amountMinor);   // both threads arrive here
}

Both requests run existsBy... before either commits. Both see nothing. Both charge. You'll never see it in local testing; you will see it during a retry storm or a double-tap on a slow mobile connection.

synchronized doesn't save you either — two instances behind a load balancer don't share a lock. Only the database can win this race.

The index that looks right and isn't

The obvious fix is a partial unique index allowing one successful payment per order:

-- Looks right. Doesn't prevent the double charge.
CREATE UNIQUE INDEX ux_payments_order_successful
    ON payments (order_id)
    WHERE state IN ('AUTHORIZED', 'CAPTURED', 'SETTLED');

Walk through the race with it in place:

Request A inserts its payment as INITIATED. The index ignores that state.
Request B inserts its payment as INITIATED. Also ignored — both inserts succeed.
Both call the payment provider. Both charges go through.
Both try to mark themselves AUTHORIZED. Only now does the constraint fire — on the second one.

The index stopped you recording a double charge. It didn't stop you making one. The customer's card was already charged twice by step 3.

The index that actually works

Invert it. Cover every state except the ones where no money is held:

-- At most one payment per order that is in flight OR holds money.
CREATE UNIQUE INDEX ux_payments_order_active
    ON payments (order_id)
    WHERE state NOT IN ('DECLINED', 'FAILED', 'VOIDED', 'REFUNDED');

Now request B fails at the INSERT, before anything leaves your system. And you get two properties for free:

A payment stuck in UNKNOWN (a timeout — the money may or may not have moved) blocks a second attempt until it's resolved. That's exactly the situation behind most real-world double charges.
A genuinely declined payment doesn't block a retry, so the customer can try another card, and failed attempts stay in the table as history.

Both indexes were tested against PostgreSQL for this post: the first accepts both INITIATED rows, the second rejects the second one at insert, blocks on UNKNOWN, and still allows a retry after DECLINED.

Wiring it up in Spring

The losing request needs to fail inside your code, before the provider call, and the provider call must not run inside a transaction:

public Payment charge(String orderId, long amountMinor) {
    Payment payment;
    try {
        // Short transaction: the INSERT is where the losing request fails.
        // saveAndFlush makes the constraint fire here, not later at commit.
        payment = tx.execute(status ->
                payments.saveAndFlush(Payment.initiated(orderId, amountMinor)));
    } catch (DataIntegrityViolationException alreadyInFlight) {
        // Lost the race, and no provider call was made.
        // Return the payment that won instead of a 500.
        return payments.findActiveByOrderId(orderId)
                .orElseThrow(() -> alreadyInFlight);
    }
    // External call runs with no database transaction held open.
    return execute(payment);
}

Three details that matter:

saveAndFlush, not save. With a plain save, Hibernate may defer the insert until commit — so the violation surfaces somewhere your catch isn't.
TransactionTemplate (tx), not @Transactional on a private helper. Spring's transaction proxy doesn't intercept a class calling its own methods, so the annotation would silently do nothing.
No transaction around execute(). Holding one open across a multi-second HTTP call pins a connection and a row lock for the whole duration. Under load, that alone takes you down.

Never charge from a webhook handler

Webhooks are delivered at least once, so duplicates are guaranteed, not hypothetical. A handler that initiates a charge will eventually charge twice.

Handlers should only record what already happened. If an event genuinely needs to trigger a charge, enqueue a job that goes through the same guarded path as above.

Recurring billing needs a period-scoped guard

Cause 8 is subtle. A dunning job retries March's failed renewal. Meanwhile, support manually retries it. Both correctly use fresh idempotency keys — the provider's key retention expired days ago — so keys can't protect you. Two charges for one month.

The guard has to live in your data, scoped to the billing period, with the same index shape:

CREATE UNIQUE INDEX ux_sub_payments_period_active
    ON subscription_payments (subscription_id, period_start)
    WHERE state NOT IN ('DECLINED', 'FAILED', 'VOIDED', 'REFUNDED');

Idempotency keys protect a request. Your schema protects a business outcome.

When it happens anyway

Refund first, investigate second. A fast refund usually prevents a dispute, and a dispute costs you a fee regardless of outcome.

Then run this on a schedule — with the index in place it should always return zero rows, which makes it an excellent canary:

SELECT order_id, COUNT(*)
FROM payments
WHERE state IN ('AUTHORIZED', 'CAPTURED', 'SETTLED')
GROUP BY order_id
HAVING COUNT(*) > 1;

If it ever returns a row, either the index is missing in some environment or something is writing around it.

Checklist

[ ] Partial unique index per order covering in-flight states, not just successful ones
[ ] saveAndFlush in its own short transaction, violation caught there
[ ] No transaction held open across the provider call
[ ] Webhook handlers record state and never charge
[ ] Recurring payments guarded per billing period
[ ] Duplicate-charge canary query running on a schedule

This is part 1 of a series on building payment systems as a backend engineer. The full version on my site adds the attempt-log schema for diagnosing the double charges that do slip through. I also build free, in-browser payment engineering tools — including a card decline code lookup and a settlement reconciliation diff.