DEV Community: Vinicius Pereira

Your LLM's confidence score is lying to you

Vinicius Pereira — Mon, 20 Jul 2026 11:31:33 +0000

The most expensive bug in an LLM system is not the output that is obviously broken. That one you catch. It is the fluent, well formed, confidently wrong answer that reads exactly like a correct one and walks straight into your system of record because nothing was standing between the model and production.

"Reads correct" and "is correct" are two different properties, and the model only optimizes the first. So the answer that is almost right, the near miss, comes back just as polished as the answer that is exactly right. You cannot tell them apart by looking. Neither can the model.

Similarity is not confidence

When teams want to gate on that, they reach for a number. Usually one of two: the model's own reported confidence, or the similarity score of whatever chunk it retrieved. Both feel like confidence. Neither one is.

Ask a model how sure it is and you get more generated text, sampled from the same distribution that just produced the wrong answer. It is a guess about a guess. And a similarity score measures proximity, not truth. The near miss scores high precisely because it sits close to the right answer. That is the whole trap: a similarity cutoff hands its highest marks to the single most dangerous output in the system, the confident near miss, and waves it through.

Compute it from what you can check

The way out is to stop reading confidence off the model and start computing it from signals you can verify without trusting the model at all. Three of them carry most of the weight.

Grounding asks whether the output is actually entailed by the evidence you gave it, not whether it reads well. An answer nobody can trace back to a source is not trustworthy, however clean it looks. A lexical-overlap baseline gets you moving, but the real seam is a (claim, evidence) -> float function, so you can swap in an NLI model or a cross-encoder the moment overlap is too blunt. Entailment over proximity.

Agreement resamples the generator on the same input and measures whether the answer survives. A model that flaps between three answers when you ask three times is telling you something a single clean draw hides. A minority draw is low confidence even when it parses perfectly. This is the one signal that costs you extra generations, so you price it in on purpose.

Validation is the cheap one, and the one people skip. Does the output satisfy the schema you declared, every required field present and the right type? A structurally invalid output should never reach the auto-accept ceiling no matter how good the other signals look, so one missing field drags the score down instead of passing silently.

The gate, as code

I wrote this up as a small reference implementation, confidence-gate, so the pattern has something runnable behind it. The generator goes in as a plain callable, which keeps the whole thing deterministic and runs it with no API key. Wiring it looks like this:

from pydantic import BaseModel

from confidence_gate import (
    AgreementSignal, GroundingSignal, Policy,
    ReviewQueue, Router, Task, ValidationSignal,
)


class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    total: str
    date: str


def generator(prompt: str) -> str:
    # Swap in a real model call. The gate does not care where the text
    # comes from, only whether it survives the signals.
    return '{"invoice_number": "INV-4021", "vendor": "Northwind Supplies", "total": "1840.00", "date": "2026-02-11"}'


queue = ReviewQueue(":memory:")
signals = [ValidationSignal(Invoice), GroundingSignal(), AgreementSignal(generator, k=3)]
router = Router(generator, signals, Policy(), queue)

verdict = router.route(
    Task(
        task_id="inv-1",
        prompt="extract invoice fields",
        evidence="Invoice INV-4021 from Northwind Supplies. Date 2026-02-11. Total 1840.00 USD.",
    )
)
print(verdict.decision, verdict.aggregate_confidence)

Run the bundled demo over three documents, one clean, one ambiguous, one with no supporting evidence, and it prints the decision for each. Captured output:

TASK            DECISION        CONF  SIGNALS
----------------------------------------------------------------
northwind-clean AUTO_ACCEPT     1.00  validation=1.00 grounding=1.00 agreement=1.00
globex-partial  HUMAN_REVIEW    0.50  validation=0.75 grounding=0.50 agreement=0.67
unknown-source  ABSTAIN         0.00  validation=1.00 grounding=0.00 agreement=1.00

review queue: 1 pending
  #1 globex-partial  conf=0.50  :: min aggregate 0.500 (accept>=0.75, abstain<=0.35) -> human_review

The middle row is the whole argument. globex-partial validates against the schema except for one missing field, only half of it is supported by the evidence, and the generator does not fully agree with itself on resampling. No single number screams "wrong", so the gate refuses to auto-accept and refuses to abstain. It routes the item to a person and keeps the receipt.

The bottom row is the one that would have burned you. unknown-source is structurally perfect, validation 1.00, and the model reproduces it on every resample, agreement 1.00. By any signal you could read off the model, it is maximally confident. Grounding is 0.00 because nothing in the evidence supports it. A confident, self-consistent answer with no external support, and the gate abstains.

Tune on the cost of a false accept

Look at how the aggregate falls out. It is the minimum of the signals, not the average. That is deliberate, and it comes from the cost structure rather than a taste for being strict. A cache miss or a re-run costs seconds. A false accept ships a confident wrong answer into the record with no model left in the loop to hedge it. Those two outcomes are not symmetric, so you do not tune the threshold to hit some acceptance rate. You tune it on what a single false accept costs you. A gate is only as strong as its weakest verified signal, so one failing signal is enough to deny auto-accept. Abstaining is cheap. Being confidently wrong is the bill you do not see coming.

That leaves three outcomes instead of two. Auto-accept when the external signals agree. Human review, with the full audit trail, when they are mixed and a person should look. Abstain when there is no support at all, because refusing to answer beats guessing when you cannot check the guess. The model proposes, the code disposes.

What this does not do

It does not make a wrong answer right. It refuses to ship one silently, which is a smaller and far more achievable goal. If your evidence is itself wrong, grounding will happily confirm a wrong answer against it, so the quality of your source text just became part of your trust boundary. The default entailment is lexical overlap, which is blunt: it catches "the answer cites nothing in the source" but not "the answer inverts the one number that mattered", so anything real gets a stronger function through that seam. Agreement costs you K generations per item, which is real latency and real money traded for a convergence signal. And none of this is a product with uptime promises. It is the shape of the design, made runnable.

A confident wrong answer and a confident right one are the same string until something outside the model checks. Be that something.

Clear the Lineup: the passage my RAG retrieved and then threw away in silence

Vinicius Pereira — Tue, 14 Jul 2026 23:24:30 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

rag-quality is a small, self-contained RAG pipeline: it retrieves passages from a corpus (sparse, dense, and hybrid modes), then generates a grounded answer that cites the passages it used and abstains when the context does not support one. It ships with an eval harness that scores retrieval quality.

Recently I commented on a dev.to thread about non-deterministic retrieval, the ways a RAG can fetch the right passage and still answer wrong. In reply, the author added one more check to his list: verify the context you asked for is the context you actually got. That check named the exact hole I had in this repo, one stage past retrieval, in assembly, against a bug I had shipped and not noticed.

Bug Fix or Performance Improvement

The generation step assembles retrieved passages into a context block and sends it to the model. Here is what it did:

def build_prompt(question: str, hits: list[Hit]) -> str:
    blocks = [f"[{h.chunk.id}] {h.chunk.text}" for h in hits]
    context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS]
    return f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"

MAX_CONTEXT_CHARS is 6000. It joins every retrieved passage, then hard-slices the string to 6000 characters. If the joined context runs longer, the tail is gone. The tail is the lowest-ranked hits, and the lowest-ranked hit is often the one that actually holds the answer: retrieval surfaced it, ranking put it last, the slice deleted it. No exception, no log, no return value that hints anything was cut. The caller gets a shorter prompt and believes it is complete.

What makes this one sting is the asymmetry a few lines down in the same file:

if message.stop_reason == "max_tokens":
    text += "\n[warning: answer truncated at the token limit]"

The code already knows truncation is dangerous. It shouts when the model's output is cut. It stays silent when the model's input is cut, which is the more dangerous of the two, because a truncated input can quietly remove the one passage the answer depended on and the model will confidently answer from what is left.

And the tests stayed green because nothing exercised the failure. The sample corpus is 17 short documents; its joined top-k context fits comfortably under 6000 characters, so the slice never removed anything in the bundled runs. The eval harness measures retrieval quality (hit@k, MRR, recall@k), whether the right passage was retrieved, not whether it survived assembly into the prompt. The bug lives in the gap between those two stages, which is precisely the gap no metric was watching.

Code

The whole change, the fix, the tests, and the optional Sentry and Gemini tooling, is one pull request:

Fix silent context truncation in build_prompt #1

vinimabreu posted on Jul 14, 2026

build_prompt joined every retrieved passage and hard-sliced the result to MAX_CONTEXT_CHARS, dropping the tail with no exception, log, or return signal. The tail is the lowest-ranked hits, and the lowest-ranked hit can be the one that holds the answer. The same file already warns on output truncation but stayed silent on the input side.

This packs whole passages in rank order up to the budget, never cuts a passage mid-text, and returns dropped_ids so the caller announces what was left out, mirroring the existing output-truncation warning.

Also adds:

env-gated Sentry spans across the pipeline (no-op without sentry-sdk and SENTRY_DSN), so an over-budget request shows total_context_chars, dropped_ids, and a context.truncated_chars measurement on the rag.build_prompt span
scripts/root_cause.py, an optional Gemini helper that localizes the loss from a trace plus the source
five tests that run key-free and offline

View on GitHub

My Improvements

Stop slicing a string. Pack whole passages in rank order until the next one would blow the budget, never cut one mid-text, and return the ids you dropped so the caller can announce them.

-def build_prompt(question: str, hits: list[Hit]) -> str:
-    blocks = [f"[{h.chunk.id}] {h.chunk.text}" for h in hits]
-    context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS]
-    return f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"
+def build_prompt(question: str, hits: list[Hit]) -> tuple[str, list[str]]:
+    separator = "\n\n"
+    budget = config.MAX_CONTEXT_CHARS
+    blocks = [f"[{hit.chunk.id}] {hit.chunk.text}" for hit in hits]
+
+    kept, dropped_ids, used = [], [], 0
+    for hit, block in zip(hits, blocks):
+        extra = len(block) + (len(separator) if kept else 0)
+        if used + extra <= budget:
+            kept.append(block)
+            used += extra
+        else:
+            dropped_ids.append(hit.chunk.id)
+
+    if dropped_ids:
+        log.warning("context truncated: %d passage(s) dropped: %s",
+                    len(dropped_ids), ", ".join(dropped_ids))
+
+    context = separator.join(kept)
+    prompt = f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"
+    return prompt, dropped_ids

That diff is the essence of the change. The shipped version also opens the rag.build_prompt span from the Sentry section around this same code, which is how those numbers reach the trace.

Generator.answer now unpacks the tuple and appends a note that mirrors the existing output warning, so the drop is visible in the answer itself:

if dropped_ids:
    note = ", ".join(dropped_ids)
    text += f"\n[warning: {len(dropped_ids)} passage(s) dropped from context: {note}]"

The budget stays in characters here on purpose. The stronger version counts the real token budget of the target model; I noted that in the docstring rather than building it, because the point of this fix is the missing signal, not a better ruler.

Proof. Same input in both runs: four bulky decoy passages plus a short answer chunk that is retrieved but ranked last, so the joined context is 7343 characters against a 6000 budget. Real console output, captured with the repo's own interpreter, no API key.

Before, the answer chunk is gone and nothing says so:

MAX_CONTEXT_CHARS      = 6000
full joined context len= 7343
answer chunk id        = expenses::answer
answer text present in FULL joined context? True
answer text present in BUILT prompt?        False
answer sentence in built prompt?            False
------------------------------------------------------------
RESULT: answer chunk SILENTLY DROPPED. No exception, no warning, no log. Caller gets a prompt missing the answer.

After, packing skips the one passage that will not fit, keeps the small answer chunk behind it, and reports the id it dropped:

context truncated: 1 passage(s) dropped to fit 6000-char budget: expenses::3
MAX_CONTEXT_CHARS      = 6000
full joined context len= 7343
answer chunk id        = expenses::answer
dropped_ids reported   = ['expenses::3']
answer text present in BUILT prompt?        True
answer sentence in built prompt?            True
------------------------------------------------------------
  expenses::0        -> KEPT (verbatim)
  expenses::1        -> KEPT (verbatim)
  expenses::2        -> KEPT (verbatim)
  expenses::3        -> dropped whole
  expenses::answer   -> KEPT (verbatim)
------------------------------------------------------------
RESULT: answer PACKED IN, and the passage that did not fit (['expenses::3']) is reported to the caller. No silent loss.

Four behavioural tests lock the contract: an over-budget answer chunk is either packed in or named in dropped_ids and never silently gone, over-budget context reports its dropped ids, no passage is ever cut mid-text, and within-budget context drops nothing. A fifth pins the tuple return type so a caller cannot regress to the old string signature. All five run with the repo's own interpreter, no key and no network.

The rule I kept: any place that decides what the model sees has to announce what it dropped. A retriever that finds the right passage and an assembler that quietly deletes it produce the same wrong answer, but only one of them leaves a trace. The output truncation in this file already knew that. The input side just needed to learn the same manners.

Best Use of Sentry

The fix for "silent" is instrumentation. I added env-gated Sentry spans across the pipeline (off unless SENTRY_DSN is set and sentry-sdk is installed, so the repo never hard-depends on it). The span that matters here is rag.build_prompt:

total_context_chars and max_context_chars
n_dropped and dropped_ids
a context.truncated_chars measurement
a warning-level event whenever n_dropped > 0

Now a wrong answer on a long-context request is not a mystery. The pipeline is instrumented end to end: rag.answer to rag.retrieve (hit_ids) to rag.build_prompt (dropped_ids) to rag.generate (stop_reason). The trace below exercises the assembly step directly, so it shows rag.answer to rag.build_prompt, where the truncation happens and the dropped ids surface.

On the over-budget request, the rag.build_prompt span reads total_context_chars 7343 against a max_context_chars of 6000, a context.truncated_chars measurement of 1343, and the dropped_ids, sitting directly under the rag.answer transaction. The truncation that used to be invisible is now a value on a span.

Best Use of Google AI

To make the localization repeatable I added scripts/root_cause.py: it takes a Sentry trace plus the source files that ran, and asks Gemini to point at the line. The prompt is fixed and paste-ready:

You are debugging a RAG pipeline that returned a wrong answer. Here is a Sentry trace and the relevant source. The retrieval span shows the correct passage was retrieved, but the final answer is wrong. Localize the exact line where the retrieved context is lost, explain the mechanism, and note whether the loss is signalled to the caller. Then propose a minimal fix.

The script is env-gated on GOOGLE_API_KEY or GEMINI_API_KEY and fails with a clear message when no key is set, so it never runs blind.

Run against the pre-fix source with the real trace, on gemini-2.5-flash, it returned this (verbatim):

The Sentry trace clearly indicates that the rag.retrieve span successfully identified expenses::answer as a relevant passage. However, the subsequent rag.build_prompt span shows a total_context_chars of 7343 and a max_context_chars of 6000, leading to 1343 characters being truncated. The symptom confirms that expenses::answer was "absent from the assembled prompt," which directly led to the LLM's abstention (I could not find this in the provided documents.).

1. Exact Line Where Context is Lost. The context is lost in app/generate.py, specifically at this line within the build_prompt function: context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS].

2. Mechanism of Loss: naive string slicing after concatenation. The function joins all blocks into a single long string, then applies a hard character limit by slicing that entire concatenated string. If the total length exceeds config.MAX_CONTEXT_CHARS, the slice cuts characters from the end of the combined string. Passages later in the hits list are more likely to be truncated, and even a highly relevant passage like expenses::answer can be cut off if it falls past the character limit.

3. Signalling of Loss. "No exception, warning, or dropped-id was reported to the caller." The function simply returns the truncated string; there are no checks, logging.warning calls, or exceptions that would inform the calling Generator.answer method. Sentry captured the context.truncated_chars measurement, but that is instrumentation-level reporting, not a signal within the application's business logic.

4. Minimal Fix. Adjust build_prompt to iteratively add passages, respecting config.MAX_CONTEXT_CHARS without splitting individual passages, so higher-ranked complete passages are prioritized. [It then produced a pack-to-fit loop that stops adding blocks once the budget is reached.]

The model landed on the same line I did, read the mechanism straight off the trace measurements, and proposed the same pack-to-fit shape as the fix above. Sentry said which chunk vanished and by how much; Gemini said which line threw it away and why nothing noticed.

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

The night I leaked my own payment key, and what caught it

Vinicius Pereira — Tue, 14 Jul 2026 21:30:52 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I build a SaaS for small businesses. Restaurants, small shops, the kind of owner who needs to track sales, stock, cash flow and finances without wrestling a heavyweight ERP. For the first months I was heads-down on features, because that is what pays: the product has to solve a real problem the day you ship it.

Then one night taught me something no feature ever could.

The mistake was entirely mine

During a deploy, a configuration of my own doing exposed a payment API key in production. The exposure window was short, but the risk was not theoretical. Anyone who found that credential could try to use it. In production. Against real money.

Here is the part I got right, and it was as much luck as foresight: weeks earlier I had built an emergency mechanism for exactly the kind of unlikely event I did not believe would ever happen to me.

Every payment integration ran behind a monitor that watched for abnormal usage. A sudden jump in transaction volume. Calls from origins it did not recognize. Sharp changes in behavior. Anything outside the profile it expected. If any of those tripped, the integration did not wait for a human. It moved itself into a protection mode, blocked new operations, and paged me immediately.

That is exactly what happened.

The circuit breaker did its job

The monitor flagged the anomaly and locked the integration into protection mode before a single improper transaction went through. The alert woke me in the middle of the night. I revoked the compromised credential, generated a new key, rolled it across the whole infrastructure, and validated everything before the first customer opened the app in the morning.

No customer was affected. No improper transaction went through. No financial data was exposed.

And none of that made the mistake any smaller.

What I actually learned

The lesson was not "be more careful." Careful is not a strategy. I had been careful, and I still shipped the mistake at the worst possible layer.

The real lesson was that security cannot rest on not making mistakes. It has to assume mistakes will happen, and it has to cap the blast radius when they do. The thing that saved me that night was not my discipline. It was a layer I had built specifically to survive my own discipline failing.

So I rebuilt the security architecture around that idea:

Automated rotation of sensitive credentials, so a leaked key has a short life by default.
Hard segregation between development and production, so the two can never bleed into each other.
Detailed audit logging of critical events, so every sensitive action leaves a trail.
Extra validation during deploy to stop secrets from being exposed in the first place.
Continuous monitoring of every external integration.
Real-time alerts for anything that looks abnormal.
A mandatory security review before any release reaches production.

The question I ask before I ship anything now

Every new feature I build now starts from one question:

If I make a mistake tomorrow at 3 a.m., will the system protect my customers before they notice?

That night changed what engineering means to me. Building is no longer just shipping features. It is building systems that expect human failure and protect the people who trust them anyway.

The bug that night was mine. The thing that smashed it was a layer I had built for a version of me that would eventually slip. Build that layer before you need it. You will need it.

My weekend challenge entry: an AI announcer for the beach-sport duels my futevôlei crew plays every day. Gemini writes the call, ElevenLabs voices it. Ring the bell yourself, the demo is live.

Vinicius Pereira — Fri, 10 Jul 2026 12:10:16 +0000

DEV Weekend Challenge: Passion Edition Submission

Vinicius Pereira

Jul 10

I gave my futevôlei crew an AI announcer (DuelUp Live)

#devchallenge #weekendchallenge #ai #googleaichallenge

4 min read

I gave my futevôlei crew an AI announcer (DuelUp Live)

Vinicius Pereira — Fri, 10 Jul 2026 09:52:42 +0000

This is a submission for the Weekend Challenge: Passion Edition

What I Built

Every Sunday my futevôlei crew meets on the sand in Niterói, Brazil. The rivalries are real: there are rematches people talk about all week, comebacks nobody is allowed to forget, and at least one duel per month that ends with someone buying açaí for everybody.

I am so deep into this passion that I have been building a whole platform for it: DuelUp, a competitive system for amateur beach sports, with ELO ratings, rank tiers from Sand to Legend, and virtual-coin stakes on real matches. It is not launched yet, but the crew already lives by its rules.

This weekend I built the piece every real rivalry deserves and none of us had: an announcer.

DuelUp Live is a fight bill for beach-sport duels. You pick the sport (footvolley, beach tennis, tennis or beach volleyball), write in the two sides with their nicknames and ranks, set the moment ("17-16, match point, blazing sun") and the bad blood ("rematch after last Sunday's comeback"). Then you ring the bell, and an electrifying AI sports announcer calls the match live, in English or in Portuguese with a Brazilian announcer voice, the kind we grew up hearing on the radio.

Gemini writes the play-by-play. ElevenLabs gives it the voice. The rank tiers come straight from DuelUp's real ELO system, and the announcer uses them as drama: when Silver faces Gold, you get an underdog story.

Demo

live.duelup.app

No signup, nothing to install. Click ⚡ Example to load a ready duel (each sport has three, they cycle), then 🔔 RING THE BELL.

Two things worth knowing while you try it:

Every ring is a fresh call, written and voiced on the spot. Ring the bell twice on the same duel and you will never hear the same broadcast.
It takes about 20 seconds, because it genuinely is an LLM writing a script and a TTS model performing it, live.

Code

github.com/vinimabreu/duelup-live

Next.js 14, App Router, zero extra runtime dependencies. Two API routes, one for the writing, one for the voice, both plain REST calls with no SDKs.

How I Built It

The writing (Gemini). The announcer is gemini-flash-latest with a prompt that treats narration as radio, not prose: present tense, short explosive sentences, 90 to 120 words, written for the ear because a TTS voice will perform it. Two decisions mattered more than the prompt itself:

Thinking disabled (thinkingBudget: 0). An announcer should not overthink. Without this, the model's reasoning tokens were eating the output budget and truncating calls mid-sentence. With it, calls come back complete and fast.
A model cascade. If gemini-flash-latest returns a 503 under load, the route falls through to gemini-flash-lite-latest, then gemini-2.0-flash. The demo has already survived a real Google traffic spike this way. A demo that only works when the weather is nice is not a demo.

Temperature is 1.0, which is why no two calls are alike. The prompt is also sport-aware: tennis gets called on the court, everything else on the sand, and the rank tiers get woven in as favorite-versus-underdog tension.

The voice (ElevenLabs). eleven_multilingual_v2 with stability set low, because announcers are not stable people. Portuguese calls go to Eduardo, a Brazilian voice that sounds like he has narrated a thousand matches. English calls go to Charlie, who brings the energy. The waveform player on the bill is wired to the real audio element: real progress, real duration, and the bars dance while he shouts.

The bill. The design is a vintage fight poster: paper texture, ink borders, Anton for the headline type, a black promoter plate with the real DuelUp logo, "TALE OF THE TAPE" for the context fields, and the rank badges from the actual game next to each fighter. The whole UI switches between English and Portuguese, announcer included.

Prize Categories

Best use of Google AI and Best use of ElevenLabs. They are not decorations here, they are the two halves of the product: Gemini is the announcer's brain, ElevenLabs is his lungs.

One personal note. Futevôlei is not a theme I picked for a challenge. It is the sport I play every single day, and the crew in this post is my real crew. Getting to spend a weekend building for it, and then writing about it here, was the rare kind of work that does not feel like work.

The platform came first, but the announcer went live before the platform did. That feels right. Passion projects do not follow roadmaps.

Built by Vinicius Pereira · vinimabreu.dev · github.com/vinimabreu

Lead Quorum: a multi-agent lead qualifier that refuses to guess (ADK + A2A)

Vinicius Pereira — Wed, 08 Jul 2026 07:35:25 +0000

This post is my submission for DEV Education Track: Build Multi-Agent Systems with ADK.

What I Built

Most lead scoring demos share a failure mode: you paste messy notes, a model returns a confident number, and nobody can answer why it is a 60 and not a 40. I built the opposite. Lead Quorum is a distributed multi-agent qualifier where the number is set by deterministic code, the explanation provably adds up to the score, and when two independent readers disagree about the input, the system abstains instead of guessing.

Three production failures it kills by construction:

Opaque scores. Every point is granted and explained on the same line of code, and a test parses the explanation and asserts the named points sum to the score. The reason cannot drift from the number.
One model grading its own homework. Two readers running two different Gemini models extract the same lead independently, as separate services. Agreement between different models is real signal, not a model agreeing with itself.
Fake confidence on thin input. "They mentioned possibly renewing" should not score like "they renewed in March."

What a run looks like. Clear notes, both models agree:

CONFIRMED  60/100: +35 team of 30 seats >= 25; +15 reachable decision maker (vp);
           +10 renewed at least once

Ambiguous notes, the models read "possibly renewing" differently:

EXCLUDED   readings disagree on which signals fire (prior_relationship: False vs True);
           abstaining instead of scoring a contradiction

That EXCLUDED is the feature. The score would differ depending on which reading you believe, so the honest output is no score, plus exactly what disagreed. A defensible abstention beats a fake-precise number built on a contradiction.

Cloud Run Embed

Open it, paste your own messy lead notes, and watch the audit trail: the score, the reason that reconciles to it, the two independent readings side by side, and the EXCLUDED abstention when they disagree about which rules fire. The two readers are separate Cloud Run services the orchestrator reaches over A2A.

Your Agents

Five roles. Two are LLMs, three are deterministic code, and that split is the design.

Enrichment reader (LlmAgent, gemini-flash-latest): extracts structured fields from raw notes. Temperature 0, pinned output schema, instructed to leave absent signals at their defaults instead of guessing.
Rederive reader (LlmAgent, gemini-2.5-flash-lite): reads the same notes from scratch, independently, on a deliberately different model. Runs as its own microservice, exposed over the A2A protocol with ADK's to_a2a() and consumed through its agent card with RemoteA2aAgent.
Scoring agent (custom BaseAgent, no LLM): applies the rubric. Each rule grants its points and writes its reason in the same branch, and the agent refuses to emit a result whose reason does not reconcile to its score.
Corroboration agent (custom BaseAgent, no LLM): compares the two readings in score-space. Same rules fire, values close: CONFIRMED. Same rules fire, readings drift: REVIEW, score stands, flagged. A rule flips between readings: EXCLUDED, no score, verdict names the flipped rule and both values.
Orchestrator (SequentialAgent + ParallelAgent): the two readers run concurrently, so wall-clock is one LLM round-trip, then scoring and corroboration run as pure code. Exactly two LLM calls per lead.

The readers deploy as independent Cloud Run services and the orchestrator reaches them over A2A, so the second opinion could be swapped for a different vendor or framework tomorrow without touching the pipeline.

Key Learnings

Put the LLM only where judgment lives. Multi-agent systems get expensive when every step is a model call. Two parallel calls per lead, everything downstream deterministic and unit-tested, made the system cheaper per unit of trust, not pricier.
Score-space beats value-space for corroboration. Two readings 500 dollars apart on the same side of a threshold produce the same score; that is drift, not contradiction. Comparing which rules fire, instead of raw field equality, lets abstention trigger only when the disagreement actually changes the answer.
Reconciliation has to be enforced, not intended. "Keep the reason next to the points" is a discipline, and disciplines rot. A failing test does not.
A2A is what makes independence credible. Behind an agent card, the second reader is a black box that could be any model, any framework, anywhere. That is the difference between a second opinion and an echo.
ADK's workflow agents are underrated. SequentialAgent and ParallelAgent gave me a deterministic, testable topology with no LLM routing where none was needed. The surprise of the build: RemoteA2aAgent has no output_key, so remote responses land in the event log, and a small capture adapter was the missing piece to keep the distributed pipeline identical to the local one.

Code: github.com/vinimabreu/lead-quorum, MIT, 16 tests, including a script that runs the full distributed topology locally.

Your LLM bill has two sides. Build the ledger that shows both.

Vinicius Pereira — Tue, 07 Jul 2026 20:57:08 +0000

Every RAG cost estimate starts the same way: input tokens equal top_k times chunk size, plus some overhead. Most of them stop there too. Then the invoice arrives, it is three times the estimate, and the team spends a sprint tuning chunk sizes while the actual money leaks somewhere that formula never touches.

The bill for an LLM system is set by total input and output tokens, summed across every call a query triggers. Not per call. Not input only. Every call, both directions. If you only remember one sentence from this post, that is the one.

Why chunk math lies

Four leaks, in the order I usually find them in real pipelines:

Calls you forgot exist. A query condenser rewriting the user question. A silent SDK retry. A JSON repair call after a malformed response. An agent loop that took three turns instead of one. Each is a full-priced API call that no chunk formula predicts, and some frameworks make them without telling you.

Scaffolding re-sent on every call. System prompt, tool schemas, formatting instructions, few-shot examples. This fixed overhead rides along on every single call, and in multi-call pipelines it often outweighs the retrieved chunks you are carefully tuning.

The output side, billed at a premium. Output tokens cost several times input on every major provider's list price, five times on current Claude models. A dashboard that tracks input tokens is watching the cheap half of the transaction.

Multi-call synthesis re-billing its own draft. Refine-style chains regenerate the full running answer at every step. Only the final draft is delivered, but all N drafts are billed at output rates, and each discarded draft is re-sent as input to the next call. The answer component of your cost scales with N, and no chunk tuning touches it. Prompt caching does not rescue it either: the growing answer sits mid-prompt, which breaks the prefix match.

Stop counting tokens. Start reading them.

The fix is not a better estimate. It is refusing to estimate: every provider already returns exact usage on every response, so the whole job is logging what the API tells you and adding it up per query.

Where you hook matters more than it looks:

The SDK call site is the reliable choke point. A thin wrapper around each create() sees server-reported usage for every call you make, streaming or not.
Framework callbacks are leakier. Some read server-reported usage properly and propagate through nested calls (LangChain's usage metadata callback does). Others estimate with a local tokenizer instead of reading the wire, LlamaIndex's TokenCountingHandler being the documented example. And none of them see calls made outside the framework.
HTTP-layer hooks have a streaming blind spot. An httpx response hook fires when headers arrive, before the body. For non-streaming calls you can read the body inside the hook; for SSE streams the usage arrives at the end of a body the hook cannot safely consume. If you want wire-level capture for streams, you need a logging proxy or a transport wrapper that tees the stream.

For most codebases the call-site wrapper is one afternoon of work and covers everything that matters.

The ledger

Columns that earn their keep, learned from what actually gets queried later:

column	why it exists
query_id	ties every call in one user request together; propagate it with a contextvar
call_index	0, 1, 2... within the query; the column that exposes hidden calls
model	mixed-model pipelines bill at mixed rates
input_tokens, output_tokens	the two sides of the bill, server-reported
cache_read, cache_write	cached input bills at its own rates; on Anthropic these are two separate fields billed differently
stop_reason	a truncated response often triggers a retry you will want to find
latency_ms	free to record, and cost and latency investigations are usually the same investigation
stage	"condense", "retrieve-answer", "repair"; the label that turns rows into a story

A minimal implementation, SQLite plus a contextvar:

import sqlite3, time, uuid
from contextvars import ContextVar

query_id: ContextVar[str] = ContextVar("query_id", default="")

db = sqlite3.connect("tokens.db")
db.execute("""CREATE TABLE IF NOT EXISTS calls(
    query_id TEXT, call_index INT, model TEXT,
    input_tokens INT, output_tokens INT,
    cache_read INT, cache_write INT,
    stop_reason TEXT, latency_ms INT, stage TEXT)""")

def record(model, usage, stop_reason, latency_ms, stage=""):
    qid = query_id.get() or str(uuid.uuid4())
    idx = db.execute(
        "SELECT COUNT(*) FROM calls WHERE query_id=?", (qid,)
    ).fetchone()[0]
    db.execute("INSERT INTO calls VALUES (?,?,?,?,?,?,?,?,?,?)",
        (qid, idx, model,
         usage["input"], usage["output"],
         usage.get("cache_read", 0), usage.get("cache_write", 0),
         stop_reason, latency_ms, stage))
    db.commit()

Wrapping a non-streaming Anthropic call:

import anthropic, time

client = anthropic.Anthropic()

def chat(stage="", **kwargs):
    t0 = time.monotonic()
    resp = client.messages.create(**kwargs)
    u = resp.usage
    record(
        model=resp.model,
        usage={
            "input": u.input_tokens,
            "output": u.output_tokens,
            "cache_read": getattr(u, "cache_read_input_tokens", 0) or 0,
            "cache_write": getattr(u, "cache_creation_input_tokens", 0) or 0,
        },
        stop_reason=resp.stop_reason,
        latency_ms=int((time.monotonic() - t0) * 1000),
        stage=stage,
    )
    return resp

Set the contextvar once at the top of each user request (query_id.set(str(uuid.uuid4()))), pass a stage label at each call site, and the ledger fills itself.

The streaming gotchas

Streaming is where naive capture quietly loses data, and the two big providers fail in different directions:

OpenAI only emits usage on a streamed call if you ask for it: pass stream_options={"include_usage": True}. The usage then arrives on the final chunk, with every earlier chunk carrying usage=None. Forget the flag and the ledger records nothing while the meter runs.

Anthropic splits usage across the stream: message_start carries the input side, and the output count arrives via message_delta near the end. If you use the SDK's streaming helper, the accumulated final message carries complete usage, so stream.get_final_message().usage is the easy path.

Field names do not agree across providers. prompt_tokens versus input_tokens, cached tokens under prompt_tokens_details.cached_tokens on one API and cache_read_input_tokens on another. And some self-hosted OpenAI-compatible servers omit usage entirely. Normalize into your own schema at write time, not at query time.

What the first week shows you

Having turned this on in production pipelines, the surprises arrive in a reliable order.

First surprise: call count. A rewrite step someone added in March, a retry that fires on every truncated response, an agent loop budgeted for one turn that averages 2.4. GROUP BY query_id and the queries with eight rows instead of two are your bill.

Second: the scaffolding. The fixed prompt overhead, multiplied by the call count you just discovered, frequently outweighs the retrieved context. People tune the 2,000 variable tokens and ignore the 3,500 fixed ones riding on every call.

Third, and only third: chunk size and top_k, the things everyone tunes first.

The ledger also makes redundancy visible. If your corpus has near-duplicate documents, they show up as repeating, near-identical input deltas across queries: you are paying to send the model the same paragraphs again and again, and the fix (dedup before the context window) is a quality fix that happens to cut the bill.

The whole analysis, honestly, is one query:

SELECT query_id, COUNT(*) AS calls,
       SUM(input_tokens) AS input,
       SUM(output_tokens) AS output,
       SUM(input_tokens + 5 * output_tokens) AS cost_proxy
FROM calls
GROUP BY query_id
ORDER BY cost_proxy DESC LIMIT 20;

The 5 is a list-price output-to-input ratio; swap in your provider's. The top 20 rows of that query are worth more than any cost-optimization blog post, including this one.

The companion package is on GitHub and PyPI: token-ledger is exactly this ledger, packaged. pip install llm-token-ledger, zero dependencies including the dashboard, one afternoon.

Sid Probstein made the argument from the architecture side this week, comparing what RAG stacks actually send across frameworks, and his closing line is the right summary of the whole subject: that number, not top_k, is your bill.

Why is this a 60 and not a 40?

Vinicius Pereira — Sun, 05 Jul 2026 19:10:43 +0000

The worst thirty seconds of my week used to happen in review. Someone senior would put a finger on one row of my output, a row scored 60 out of 100, and ask why it was a 60 and not a 40. And I did not know. Not on the spot.

What I did instead was open the scoring code, re-read the branches, do the arithmetic in my head against that input row, and reconstruct the answer live while the room waited. Half the time I got it right. The other half I said "let me check and get back to you," which in front of people who make decisions on that number is the same as admitting the number is a guess.

It took me a couple of those meetings to see the real problem. A score you cannot defend the moment you are asked is not a feature. It is a liability. If the person consuming your output has to trust you instead of the number, you did not ship a scoring system. You shipped yourself as a dependency, and you will be that dependency at 6pm on a Friday.

Two code paths, one quiet lie

The failure is almost always the same shape. The scoring function adds points. Somewhere else, usually a different file, a second function assembles the human-readable reason. Both encode the same thresholds. They agree on the day you write them, the tests pass, everyone moves on.

Then a threshold changes. It changes for a good reason, in the scoring function, and it ships. Nobody touches the reason builder, because why would they. It is in another file, the tests are still green, the diff looked complete. Now the number moves and the story stays put. The reason string is not merely stale. It is confidently, specifically wrong.

Take one concrete row and hold it fixed for the rest of this post:

lead = {"monthly_spend": 6000, "seats": 30, "renewed": True}

Here is the code that scores it, split across the two paths:

# BEFORE: the number and its justification live in two functions
# that happen to share a set of magic thresholds.

def score_lead(lead):
    score = 0
    if lead["monthly_spend"] >= 8000:   # bumped from 5000 last quarter
        score += 40
    if lead["seats"] >= 25:
        score += 35
    if lead["renewed"]:
        score += 25
    return min(score, 100)

def explain(lead):
    reasons = []
    if lead["monthly_spend"] >= 5000:   # nobody bumped this one
        reasons.append("high spend")
    if lead["seats"] >= 25:
        reasons.append("large team")
    if lead["renewed"]:
        reasons.append("renewed before")
    return ", ".join(reasons)

Run our row through both:

>>> score_lead(lead)
60
>>> explain(lead)
'high spend, large team, renewed before'

The spend threshold moved to 8000, so 6000 no longer earns the 40 points and the lead lands at 60. But explain still tests against the old 5000 line, so it leads with "high spend" as the top reason. The number went down because spend stopped counting, and the explanation says it is high because of spend. You are now debugging your own explanation against your own arithmetic, out loud, in front of the person who asked.

Build the reason where you add the points

The fix is boring and it is the whole point. Build the reason on the same line where you add the points. Same if, same condition, same line of sight. The reason is not documentation written after the fact. It is a byproduct of the decision, emitted at the instant the decision is made.

# AFTER: one branch owns both the points and the words for them.
# You cannot change the number without seeing the sentence.

def score_lead(lead):
    score = 0
    parts = []
    if lead["monthly_spend"] >= 8000:
        score += 40
        parts.append(f"+40 monthly spend {lead['monthly_spend']} >= 8000")
    if lead["seats"] >= 25:
        score += 35
        parts.append(f"+35 team of {lead['seats']} seats >= 25")
    if lead["renewed"]:
        score += 25
        parts.append("+25 renewed at least once")
    final = min(score, 100)   # weights top out at 100, so the cap never bites
    reason = f"{final}/100: " + ("; ".join(parts) if parts else "no signals fired")
    return final, reason

Same row, one call now:

>>> score_lead(lead)
(60, '60/100: +35 team of 30 seats >= 25; +25 renewed at least once')

It is a 60 and not a 40 because team size and a prior renewal fired and spend did not, and the numbers add up in front of you. There is no second file to forget, because there is no second file. The if that grants 40 points is the only place that can claim the 40.

One caveat about that cap. Here 40 plus 35 plus 25 lands exactly on 100, so min never changes the total. If your raw points can overshoot the ceiling, either design the weights to land on the cap, or name the cap in the reason and reconcile against the pre-cap total. The mismatch has to stay visible, never quietly absorbed.

A test that catches the drift before the meeting does

"Just build it in the same place" is a discipline, and disciplines rot the first sprint you are in a hurry. So I pin it down as a property: the point values named in the reason must sum to the score. If they ever stop summing, the audit trail has started lying, and I want CI to fail instead of a stakeholder to notice.

import re

def test_reason_points_sum_to_score():
    for lead in SAMPLE_LEADS:
        score, reason = score_lead(lead)
        named = [int(p) for p in re.findall(r"\+(\d+)", reason)]
        assert sum(named) == score, reason

The test parses its own audit trail and checks the arithmetic against the score the function returned. Add a signal that bumps the score but forget to append its line, and the sum comes up short. Append a line but fat-finger the number, and it comes up wrong. Either way it goes red. Notice what the test does not do: it says nothing about whether the reason is well written. It checks that the reason is not lying. Those are different jobs, and in a review only the second one matters.

Sometimes the defensible score is zero

The pattern pushed me somewhere I did not expect. In crosswatch, the small harness I pulled this out of, I corroborate every number across two independent providers before scoring anything. Agree, and the region is CONFIRMED. Drift a little, REVIEW. Contradict each other hard, and the region is EXCLUDED and scores zero on purpose.

Not the average of the two numbers. Zero. When two independent sources disagree that badly, the honest answer to "what is the value here" is "we do not know," and the average of two contradictions looks exactly as confident as a good number while meaning nothing. A zero I can stand behind beats a 55 I cannot, and the EXCLUDED row carries the reason for it: the providers disagreed by this many points, so no score was more honest than one built on a contradiction.

The same welding buys something operational. Because every reason ships attached to its number, and the raw readings are stored with provenance (which provider, which run, which timestamp), I can rescore months of stored data under new thresholds without collecting anything again. Collection is expensive, judgment is free. Change a weight, rescore in milliseconds, and every reason moves in lockstep with every number, reconciliation test still standing guard.

So the next time someone points at a row and asks why it is a 60 and not a 40, the answer is already sitting in the cell next to the 60, and a test has already confirmed it adds up. Everything after that is just deciding where your if statements go.

crosswatch is on GitHub, MIT licensed, stdlib only, 63 tests: github.com/vinimabreu/crosswatch

I wrote a free 84-page handbook to take you from "what is a token" to working AI agents

Vinicius Pereira — Fri, 03 Jul 2026 20:29:37 +0000

For the past months, whenever someone asked me how to get into AI, I had no good single answer. The material out there is either academic theory, marketing hype, or fifty scattered tutorials that each assume you read the other forty-nine.

So I wrote the thing I wanted to hand over: Artificial Intelligence in Practice, an 84-page handbook that starts at zero and ends with you running agents with tools. It is free, and it just went up on GitHub:

https://github.com/vinimabreu/ai-in-practice

Who it is for

People starting out in AI, or curious about it, who want one coherent path instead of a pile of tabs. It assumes no AI background. If you can open a terminal, you can follow it. Developers already building with LLMs will find parts of it review, though the evaluation and cost chapters may still earn their time.

The path it walks

Foundations. What tokens, context windows and temperature actually are, and why a model "hallucinates". Setting up a real dev environment without pain.
Running models on your own machine. Ollama, picking a model family for your task, and honestly sizing what your hardware can run, including Apple Silicon.
RAG. Making an AI answer from YOUR documents instead of making things up, explained without hand-waving: embeddings, vector databases, chunking, and when RAG is the wrong answer.
Agents. The loop that turns a chatbot into something that does work: CrewAI, LangGraph, tools, browsing, MCP, and agents from different frameworks talking to each other.
Going deeper. Fine-tuning vs RAG, what the APIs really cost, security and privacy of autonomous agents, prompting as a method, and how to test AI systems so they do not embarrass you in front of a customer.
Three capstone projects, from a document Q&A assistant to a research agent crew, ending with what it takes to go from project to product.

Everything in it was tested by hand. Prices and version numbers were verified in mid-2026, and the book tells you to re-verify them, because this field does not sit still.

Why free

Because the version of me from a few years ago needed this and could not have paid for it. It is under CC BY-NC-SA: share it, translate it, use it to teach a course. Just keep it non-commercial and credit the source.

Two small asks

If you know someone who keeps saying "I want to learn AI but I do not know where to start", send them the link. That is exactly who it was written for.

And if it is useful to you, a star on the repo helps other people find it: https://github.com/vinimabreu/ai-in-practice

I will be serializing some of the deeper chapters here on dev.to over the next weeks, starting with the one I think is most neglected: how to actually test an LLM system. If there is a chapter you want first, tell me in the comments.

Vinicius Pereira

You can't debug a RAG you didn't instrument

Vinicius Pereira — Wed, 01 Jul 2026 18:25:40 +0000

Every few weeks someone opens a ticket that says some version of "I think the AI is getting worse?" The answers are still fluent, still confident, still cited. They're just subtly wrong, often enough that people notice and rarely enough that nothing obviously breaks. Then a few days quietly disappear into it.

The instinct is always to look at the model or the prompt. Almost every time I've chased one of these, the model did exactly what it was told. It read the top documents and answered from them. The problem was upstream, in what got retrieved and handed to it, and the reason it took days to find is that the retrieval step was a black box. We log the final answer. Sometimes we log the citations. We almost never log what the retriever actually saw and chose between.

You can't debug what you didn't instrument.

What to actually log

For every answer, I keep a small retrieval manifest next to it. Three things:

What was retrieved. The whole candidate set with scores, not just the ones that got cited. This is the part you'd expect.
What was excluded, and why. Each dropped candidate with a reason code: below the rank cutoff, filtered out by metadata, superseded or stale, out of license, deduplicated. This is the part nobody logs, and it's exactly where the blind spots live.
What was cited. What actually made it into the answer.

Here is roughly the shape of one entry:

{
  "query": "what is our refund window for enterprise?",
  "retrieved": [
    {"id": "policy-2024-11", "score": 0.86, "cited": true},
    {"id": "policy-2026-05", "score": 0.78, "cited": false}
  ],
  "excluded": [
    {"id": "policy-2026-05-draft", "reason": "status:superseded"},
    {"id": "sales-deck-q1", "reason": "below_rank_cutoff"}
  ]
}

Look at that for a second. The cited document is fourteen months old and scored higher than the current one, purely because it happened to be written more cleanly. In the answer, that is invisible. In the manifest, it is the first thing you see.

What it buys you

Two things that used to be guesswork become mechanical.

You can tell a reasoning problem from an evidence problem. When two runs disagree, or two deployments of the same model give different answers, diff the manifests first. Same evidence set and different answers means it is the model or nondeterminism. Different evidence sets means it is retrieval, and you were never going to fix that by tweaking the prompt. Right now most people debug this backwards, staring at the outputs, because the boundary was never captured.

The stale-document bug surfaces in minutes instead of days. The classic failure, where an outdated doc quietly outranks the current one, does not show up in the answer at all. It shows up immediately in the manifest as a top result with an old timestamp. You stop guessing and start reading.

The part people get wrong

The exclusion log is noisy. You are not going to read it on every query, and if you try you will drown. So log it always, surface it only when an answer gets flagged or when two results disagree. It is a black box recorder, not a dashboard.

The other trap is drift. The manifest only helps if the retrieval code emits it as it runs. The moment you rebuild it after the fact, or maintain it by hand, it becomes one more thing that can quietly disagree with reality, and now you are debugging your debugging.

The one-line version

Citations tell you what supported the answer. The exclusion log tells you what the answer was blind to. You need both to trust the thing, and almost everyone keeps only the first.

Most "the model is hallucinating" tickets are really "the retriever handed it the wrong evidence and it used it faithfully." Instrument the boundary and the model stops being the default suspect. That is the direction I have been building rag-quality around, the idea that the retrieval step should measure and report on itself instead of being trusted on faith.

So I am curious: what do you actually log from your retriever today? Just the citations, the full candidate set, or nothing until something breaks?

Sonnet 5 dropped today. Watch the other hand.

Vinicius Pereira — Tue, 30 Jun 2026 22:04:06 +0000

Sonnet 5 landed today and everyone's busy benchmarking it. Fair, it looks like a strong, cheap, very agentic model, close to Opus 4.8 for a fraction of the cost. But the most important model news this month isn't Sonnet 5. It's that Fable 5, Anthropic's most powerful model, has been switched off worldwide since June 12, and it's still down.

Not rate-limited. Not deprecated. Export-controlled. Anthropic got an emergency US Commerce Department directive citing national security after a way to jailbreak Fable 5's safeguards surfaced, and since they couldn't verify the nationality of every request in real time, they pulled it for everyone, Americans included. Mythos 5 got a partial reprieve for a handful of cyber-defense orgs. Fable 5 is still dark.

Sit with that. The most capable model on the market right now isn't gated by price, it's gated by law. "Available to everyone who can pay" quietly became "available to whoever the government allows." That's a different universe to build a company in.

Now look at what shipped in the same window: Sonnet 5, explicitly the cheaper, more agentic, good-enough model. The industry's answer to "the frontier just got pulled" is "you probably didn't need the frontier anyway." Which, honestly, has been true for most of us for a while. Most production AI fails on reliability, eval, and retrieval, not on raw model IQ. A bigger brain was never the thing standing between your demo and prod.

So the take: chasing the newest frontier was always a weak moat, and now it's a fragile one, it can be switched off by a government letter on a Tuesday, it literally just was. The teams that win the next year are the ones who (a) made their systems reliable enough to run a cheaper or open model without flinching, and (b) didn't bet the company on a single vendor in a single jurisdiction. Reliability and independence beat raw capability the moment capability becomes a policy decision.

Tell me I'm wrong. Is Sonnet 5 a real step change for your use case, or is the actual headline that the frontier now ships with an export-control letter attached?

What changed since the last scrape? A small change-detection layer (stdlib only)

Vinicius Pereira — Tue, 30 Jun 2026 20:23:53 +0000

Most of my scrapers answer one question: what's on the site right now. But that's almost never the question I actually have. What I care about is what changed since the last run. A new listing showed up, a price dropped, a product disappeared, a status flipped from open to closed. The current snapshot on its own doesn't tell me any of that.

For a while I rebuilt the same thing on every project: load last run's JSON, compare it to this run, work out what's new, what's gone, and what changed. It's never hard, but it's fiddly, and I kept getting the same details wrong (more on that below). So I pulled it into one small reusable piece and stopped rewriting it. It's called scrape-sentinel.

This post is about the design more than the tool, because the interesting part is the handful of decisions that make change detection annoying to get right.

The core idea

You give it the records from this run and a key, and it tells you what was added, removed, and changed since last time. For changed records, it tells you which fields moved and from what to what.

The diff itself is a plain function with no I/O:

from scrape_sentinel import diff

cs = diff(previous_records, current_records, key="sku", ignore_fields=["scraped_at"])

for r in cs.added:
    print("new:", r["sku"])
for changed in cs.changed:
    for d in changed.deltas:
        print(changed.key, d.field, d.old, "->", d.new)

Which prints something like:

new: W-104
W-101 price 39.0 -> 35.0
W-101 in_stock True -> False

The details that kept biting me

A few decisions are the whole reason this is worth extracting instead of rewriting inline every time:

Match by key, not by position. This is the big one. If you diff two lists positionally, a re-sorted page or a reordered API response looks like every row changed. Matching on a stable key (one field or a few) means a reordered run shows zero changes, which is correct.
The first run is a baseline. With no previous snapshot, everything looks new. The first run just records state and stays quiet instead of firing an alert for all 4,000 items.
Ignore the noisy fields. A scraped_at timestamp or a session token changes every single run. You drop those from the comparison, or restrict it to an allow-list of fields you actually care about.
Write snapshots atomically. The state file is written to a temp file and renamed, so a run that dies halfway can't leave you with a corrupted snapshot that breaks the next comparison.

Using it for real

In practice you want the diff plus the I/O around it: load the last snapshot, run your scraper, compare, alert, save the new snapshot.

from scrape_sentinel import (
    CallableSource, PipelineConfig, SnapshotStore,
    ConsoleAlerter, WebhookAlerter, run_once,
)

def scrape() -> list[dict]:
    # your requests / Playwright / API code, returns a list of dicts
    return fetch_products()

config = PipelineConfig(
    key="sku",
    ignore_fields=["scraped_at"],
    alerters=[
        ConsoleAlerter(title="catalog", key_fields=("sku",)),
        WebhookAlerter(SLACK_URL, title="catalog", key_fields=("sku",)),
    ],
)

changes = run_once(CallableSource(scrape), SnapshotStore("./.state"), config)
print(changes.summary())   # {'added': 1, 'removed': 1, 'changed': 1, 'unchanged': 2}

Alerts go to the console, a webhook (Slack or Telegram), or a JSON change log. There's also a CLI with a --fail-on-change exit code, so you can put it on a cron job or a CI step and have the next step run only when something actually moved:

scrape-sentinel run --source json:catalog.json --key sku --state ./.state --webhook "$SLACK_WEBHOOK"

What it is not

It's not a scraper. It doesn't crawl anything for you. You bring your own requests, Playwright, or API client and hand it a list of dicts, and it owns the diff, the alert, and the snapshot. It's also standard library only, no dependencies, so dropping it into an existing project doesn't pull in a tree. The diff being a pure function is what made it easy to test heavily, which is where most of the suite lives.

Repo

MIT licensed: https://github.com/vinimabreu/scrape-sentinel

Honestly curious how other people handle this. Do you diff inside the database, keep snapshots on disk like this, hash each record, or something cleaner? It feels like the kind of thing everyone quietly rebuilds, so I'd like to know what I missed.