Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
The bill was 50 cents a run.
Not a month. Not a thousand runs. One run.
Paste a design proposal in, get back a grounded review with precedent from our shelf of books and internal postmortems, and watch half a dollar evaporate.
Part 1 was the happy version of this story: a Go binary, Gemini's hosted File Search, no vector database, and a free tier that made the whole thing look like a free lunch.
Then the free tier ran out and the paid rates arrived, and the free lunch turned out to be a tasting menu.
Five people on the team, several runs a day each, and a number that scaled with how hard the tool was thinking.
This post is what we replaced it with, what we measured, and the one component we built, benchmarked, and then deleted.
The word "free" was doing a lot of work
Here is the thing the pricing page tells you about File Search, and it is all true.
Storage is free. Query-time embeddings are free. You pay once at indexing time.
Here is the thing it does not put in bold.
The search is performed by a model. It is a tool the model calls mid-answer. Which means the retrieved chunks land in that model's context as ordinary input tokens, and the model reasons its way through them before replying.
And on Gemini 3.5 Flash, reasoning costs $9.00 per million output tokens.
Look at that output row.
9,529 tokens out, of which 8,493 were the model thinking. For a call whose entire job is "find the relevant passages and hand them over."
We paid a reasoning model to reason about which paragraphs to copy.
The first fix was structural and it helped a lot. The draft step used to do its own search, which meant three drafts meant three searches. Splitting search out so one search feeds every draft took a run from about $0.70 to about $0.12.
Six times cheaper, and still the wrong shape.
Because the expensive part was never the searching. It was the thinking attached to it.
Retrieval does not need a brain.
It needs an index, a similarity function, and a tiebreaker. Every one of those is a thing you can run on a laptop while it charges.
What we swapped in
A local Chroma store, embedded with Qwen3-Embedding-0.6B, searched with dense vectors plus BM25, fused with Reciprocal Rank Fusion.
Every model call moved to DeepSeek V4 Flash on Atlas Cloud at $0.14 in and $0.28 out per million.
Roughly ten times cheaper per token than what we were on, with a 1M context window and JSON mode.
The retrieval service is Python, running as a child process of the Go binary, on a port nobody else talks to. It starts with make run and dies with it.
I did try to talk myself out of that boundary.
Go has no mature CUDA story. The honest route is exporting the model to ONNX and linking a Go runtime through cgo, matching driver and CUDA and cuDNN versions exactly. PyTorch is the path a million people have already debugged, including on WSL2, where CUDA passthrough breaks in creative ways.
Adding a process boundary was cheaper than adding a whole new class of "works on my machine."
Chunking is where the accuracy actually lives
Everyone talks about which embedding model to pick. Almost nobody talks about what you feed it, and that is where the wins were.
Our corpus is 71 markdown files, and most of them are books that used to be PDFs. PDFs converted to markdown are full of things that are not text.
The cleaner strips the converter's own footer, lines that are only a page number, picture-text blocks, and hyphenated line breaks that split a word across two lines.
The one that surprised me is running headers.
A converted book repeats the chapter title at the top of every single page, usually promoted to a markdown heading.
Left in, it shreds the text into confetti, and a passage about paper reactors comes back with "Ship Project and Civilian Power" wedged into the middle of a sentence.
The rule that fixed it is embarrassingly simple: a short line that appears five or more times in one file is page furniture, not prose.
Then the chunking. About 300 words, hard cap 450, split on sentence boundaries, with two sentences carried into the next chunk so a quote that straddles a boundary survives in at least one piece. A real section heading starts a new chunk, once the current one has enough in it to stand alone.
And then the part I would steal even if you take nothing else from this post.
Embed more than you store.
The chunk you store is the text the draft is allowed to quote. The string you embed has a header glued on top of it:
EMBED_MODEL = "Qwen/Qwen3-Embedding-0.6B"
# what goes into the embedding, per chunk
f"{title} ({kind}) › {heading_path}\n\n{text}"
# and the query side gets an instruction, because Qwen3-Embedding is
# instruction-tuned on queries only. documents are embedded as they are.
QUERY_PROMPT = (
"Instruct: Given an abstract pattern or principle, retrieve historical cases, "
"documented examples, and named principles from books and essays that show "
"the same pattern\nQuery: "
)
A chunk in the middle of chapter nine might only say "he decided otherwise."
With the header, that vector still knows it is Rickover, in a book about Rickover, in a section about paper reactors. Without it, it is a pronoun floating in space.
The query instruction is the Part 1 idea, "search with the pattern, not the post," pushed one layer down into the embedding itself. The corpus is cases. The queries are patterns. Saying so out loud to a model that was trained to listen costs nothing.
Dense finds the paraphrase, BM25 finds the name
Vector search is great at "these two paragraphs mean the same thing" and oddly bad at "this paragraph contains the word Rickover."
Keyword search is the reverse.
So we run both, take 40 candidates each, and fuse them.
flowchart TD
Q[one query from call 1] --> DN[dense top 40, Qwen3-Embedding]
Q --> BM[BM25 top 40, title + text]
DN --> RRF[Reciprocal Rank Fusion, k=60]
BM --> RRF
RRF --> SEL[keep the best 4 per query]
SEL --> CAP{2 chunks from this file already?}
CAP -- yes --> SKIP[skip, so no book dominates]
CAP -- no --> KEEP[keep it]
KEEP --> RR[round-robin merge, 3 queries]
SKIP --> RR
RR --> DUP{near-duplicate of one picked?}
DUP -- yes --> DROP[drop: a post synced twice]
DUP -- no --> OUT[12 passages go to the draft]
classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
classDef dense fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
classDef lex fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a
classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
classDef bad fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
class CAP,DUP decision
class Q start
class DN,RRF dense
class BM,SEL,RR lex
class KEEP,OUT good
class SKIP,DROP baddef hybrid(self, query: str, k: int = CANDIDATES) -> list[tuple[int, float]]:
score: dict[int, float] = {}
for ranked in (self.dense(query, k), self.lexical(query, k)):
for r, i in enumerate(ranked):
score[i] = score.get(i, 0.0) + 1.0 / (RRF_K + r + 1)
return sorted(score.items(), key=lambda x: -x[1])[:k]
That is the whole fusion. Six lines.
The reason it works is that it never compares the two scores. A cosine similarity of 0.82 and a BM25 score of 14.3 have nothing to say to each other. Ranks do.
A document at position 3 in both lists beats one that is first in a single list, and the constant k (60 is the number the literature settled on) keeps the top of each list from steamrolling everything else.
BM25 indexes the title and heading alongside the text, same as the embedding header does, so naming a book in your query actually finds pages from that book.
A few rules keep the final twelve honest. At most two chunks per file per query, so one 400-page book cannot fill every slot. Queries merge round robin, so each of the three contributes its best passage before any of them gets its fourth.
And anything with a word-overlap above 0.6 against something already picked gets dropped, because our blog corpus has a couple of posts that got synced twice and they were politely returning themselves as two independent sources.
The reranker that got fired
Standard advice says: retrieve broadly, then rerank with a cross-encoder. So we did that, with bge-reranker-v2-m3.
Then a run took six and a half minutes and I went looking.
The retrieval service's own log had it in one line: search: 3 queries -> 8 chunks in 117.5s.
Three queries times 40 candidates is 120 cross-encoder forward passes on a 4 GB GTX 1650 that is also drawing the desktop. It was not thrashing. It was just honest work on unfit hardware.
Before ripping it out, we measured. The golden set builds itself out of finished sessions: take Call 1's retrieval queries, pair them with the files the accepted draft actually cited, and you have a retrieval test built from real usage rather than from vibes.
The reranker won on MRR and hit@4. It genuinely put better passages nearer the top.
It also did not change recall@12 at all, and recall@12 is the only number with a consumer.
The draft call gets all twelve passages in its prompt. It reads all twelve. There is no top-4 cutoff downstream, no truncation, nothing that treats passage 1 differently from passage 9.
So the reranker was spending 108 seconds improving an ordering that nothing downstream reads. It was optimising a metric we had accidentally chosen because it appears in every retrieval paper, not because our pipeline consumed it.
Out it went, with the reasoning written into the top of retriever.py so the next person does not "fix" its absence.
And the honest caveat, which lives there too: this is one golden session. A harder query might genuinely need reranking to pull the right passage into the top twelve. The code is in git history, the eval is a make target, and when the golden set is fat enough to mean something we will run it again.
Measure before you delete. Also measure before you keep.
Two hours of embedding, four laptops
3,639 chunks at roughly 1.8 seconds each on a shared 4 GB GPU is about two hours, which is about one hour and fifty minutes more than anyone wants to wait.
But embedding is deterministic and the corpus splits cleanly by file. So it parallelises across people, not just across cores.
flowchart TD
C[71 files, 3,639 chunks] --> S[shard_files: disjoint quarters]
S --> M[four laptops, --shard i/4]
M --> R{sha256 AND chunk count match?}
R -- yes --> SK[already embedded, skip]
R -- no --> EM[embed, halve the batch on OOM]
EM --> G[commit the shard, hand it back]
SK --> G
G --> I[rag/integrate.py]
I --> V{same embedding model everywhere?}
V -- no --> X[refuse: mixed vectors lie quietly]
V -- yes --> O{every file in exactly one shard?}
O -- in two --> X
O -- in none --> W[warn, merge what arrived]
O -- yes --> F[copy vectors into db/chroma]
classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
classDef work fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
classDef box fill:#6ea8ff,stroke:#2f5fc4,color:#1a1a1a
classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
classDef bad fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
class R,V,O decision
class C start
class M,EM work
class S,G,I,SK box
class F good
class X,W bad# on four different machines, one quarter each
uv run rag/ingest.py --shard 1/4 --out db/chroma-shards/1
uv run rag/ingest.py --shard 2/4 --out db/chroma-shards/2
# ... then, once everyone commits their shard back
uv run rag/integrate.py db/chroma-shards/*
The merge does no embedding at all. It checks that every shard used the same embedding model, that no file landed in two shards, and that no file landed in none, then copies the vectors into one store.
Those checks are not paranoia. Mixed embedding models do not crash. They return confidently wrong neighbours forever, which is a far worse failure than a stack trace.
Building this also shook out a real bug in the incremental logic. Resume was deciding "already done" by comparing the file's sha256 against what was in the store.
A run killed halfway through a file leaves that file's sha perfectly correct and its chunk count short, so it would have been marked done forever, silently missing half a book.
The fix is one AND: sha256 and chunk count both have to match.
Then we committed the finished store. db/chroma is 69 MB in git, which is nothing, and it means nobody else on the team ever embeds anything. Clone, run, search.
Now that retrieval is free, spend it on drafts
The nice thing about killing your most expensive call is that the cheap calls get interesting.
A draft is now a few tenths of a cent. So instead of drafting once and retrying on failure, the pipeline fires three or four drafts at once at temperature 0.7 and lets them race.
Each reply gets resolved and checked in Go as it lands. Drafts that fail the checks are rejected on the spot. The first one that passes goes on to the review call. Only if all of them fail does the batch retry, with the closest draft's failures appended to the prompt.
Every draft is kept and shown, including the rejected ones with the checks they failed, because "here are four attempts and why three of them were bad" is more useful to the person reading than one draft and a shrug.
This did produce one genuinely dumb bug, which parallelism made much more likely.
In one run the first batch produced a draft that passed every check. The reviewer then asked for a revision. Eight redrafts later, none of them passing, the pipeline shipped the closest failing redraft.
It had a passing draft in hand and threw it away for a worse one. The fix is the obvious fallback: if no redraft passes, ship the draft that already did, with the reviewer's notes attached.
Retries are cheap. Losing work you already paid for is not.
The check that replaced trusting the grounding metadata
Handing the passages in ourselves unlocked the thing I actually care about.
When a hosted search tool returns grounding metadata, you know which chunks the model looked at. You do not know that the words it put in quotation marks are in any of them.
Now the pipeline can prove it.
Every history.sources[].passage in the output has to appear word for word in a chunk from the file it names. Same for an authority's exact_words when it cites one of the given files. Markdown, line wrapping and quote style are normalised away first, and a ... marks an omission, with the remaining pieces required to appear in order.
A failure is not a warning. It is a check failure, exactly like a length violation or an unresolved law citation, and it feeds the same retry loop that everything else does.
This is the difference between "the model had access to the right book" and "the model quoted the right book correctly," and only one of those is worth showing a reviewer.
What the bill looks like now
One real run, end to end, with 15 model calls including 12 drafts across three batches:
- 210,976 input tokens, 55,285 output tokens
- $0.045, about ₹4.3
- retrieval: 0.5 seconds of it, and none of the money
Against $0.70 for the same pipeline shape on hosted search. Fifteen times cheaper, and the expensive part is now the part that does the actual writing, which is how it should be.
Two honest footnotes on that number.
Atlas reports most of each draft's input as cached, and we charge every input token at the full rate, so $0.045 is a ceiling, not an estimate.
And latency went up, not down. Retrieval dropped from 117 seconds to half a second, but DeepSeek thinks hard before each draft, so a full run is still two to four minutes.
We bought cost, not speed. For "paste a design doc, come back with a coffee," that is the right trade. For a chat box it would be the wrong one.
What I would steal
If your retrieval is a model call, you are paying reasoning prices for a lookup.
Pull it onto your own machine, spend the effort on cleaning and chunking rather than on model selection, embed a contextual header you never show anyone, fuse dense and lexical on ranks instead of scores, and check the quotes rather than trusting them.
Then build the eval before you build the clever part. Ours told us to throw the clever part away, which saved 108 seconds a search and a permanent dependency we did not need.
The best component in this system is the one that is not in it.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
The exact math, not a black box
Visualize blast radius at a glance
Every factor that feeds the score
Here's the goal:
- A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
- A 300-line UI change in one file, fully covered by…
Click below to try LiveReview with your codebase:
Top comments (2)
The reranker measurement is where this article really earns its keep. Most retrieval guides treat cross-encoders as a mandatory default without verifying whether downstream stages truncate the candidate list. If the generation prompt receives all twelve passages anyway, spending two minutes of forward passes to optimize MRR inside a prompt the model attends over in parallel is dead weight.
Prefixing the ancestor heading path onto the embedding text is another practical save. Converted PDFs leave pronouns and dangling bullets completely unanchored in vector space unless the section hierarchy is attached directly to the embedded string. Clean, measured engineering.
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support