DEV Community: Ace-2504

The model had the right evidence. It still got the answer wrong. I ran a controlled 60-question RAG experiment to find out whether retrieval or the reader was the real bottleneck. The results surprised me. When Better Retrieval Doesn't Mean Better Answer

Ace-2504 — Sun, 23 Aug 2026 17:20:20 +0000

Ace-2504

Aug 23

When Better Retrieval Doesn't Mean Better Answers.

#rag #llm #machinelearning #ai

10 min read

When Better Retrieval Doesn't Mean Better Answers.

Ace-2504 — Sun, 23 Aug 2026 05:50:57 +0000

Fine-tuning teaches the shape. Retrieval supplies the facts.

Everyone building RAG right now is tuning their retriever — better reranking, deeper retrieval, smarter chunking. I ran six experiments to see how much that helps once retrieval is already good.

The short version: once the relevant passage was consistently reaching the context window, the retrieval-side changes I tried stopped moving answer quality — and a controlled test pointed at the reader, not the retriever, as the limiting factor. Here's how I got there, with numbers I can defend and the failures reported alongside the wins.

The setup: three systems, one model

I used Yu-Gi-Oh as the test domain on purpose: thousands of cards, intricate timing rules, dense proper nouns, and a base model that's weak on it out of the box (closed-book, the base model scores just 1.83/5 on correctness and 0.18/2 on groundedness — see below). That weakness is a feature — it lets the three systems separate instead of all scoring the same.

All three systems are the same google/gemma-2-2b-it. Only one lever changes at each step:

System A — base. The model as shipped, closed-book. Adapter off.
System B — fine-tune. A QLoRA adapter (rank 16, α 32, 4-bit NF4, all linear modules) trained on 2,683 teacher-distilled, judge-verified Yu-Gi-Oh Q&A pairs, early-stopped at the best validation checkpoint (~1 epoch; 3 epochs overfit). Still closed-book. (Validation perplexity was 3.87. That measures next-token fit on held-out text, not factual correctness or grounding — it's a training-health number, not evidence about answer quality, and I don't use it as such.)
System C — fine-tune + retrieval. System B, but a hybrid retriever prepends real passages before it answers.

Keeping it one model with an adapter toggle matters: any score difference comes from the lever I moved, not from a different model underneath.

A three-layer way to read the results

Before the numbers, a framing that makes the rest precise. "RAG" is usually drawn as retrieval → answer, but there are three distinct layers, and they can fail independently:

Retrieval — does the relevant evidence make it into the retrieved set at all? (Measured by recall@k.)
Evidence utility — does the retrieved context actually contain sufficient, usable evidence for this question — not buried among distractors, not split across passages, not one fact among several competing ones?
Reader — can the model extract the right clause, reason over it, and produce the correct answer?

High recall only speaks to Layer 1. It says nothing about whether Layer 2 is sufficient or whether Layer 3 can use it. Keeping these separate is what lets the results below say something defensible.

Result 1: fine-tuning shifted the shape; retrieval drove the larger correctness gain

All three systems answer the same 60 held-out questions, each scored 0–10 by a blind, reference-grounded judge (rubric details later). The headline, with paired statistics:

System	Mean ± SE	vs. previous	95% CI of difference	p (t / Wilcoxon)
A — base, closed	3.98 ± 0.39	—	—	—
B — fine-tune, closed	5.25 ± 0.54	+1.27	[0.42, 2.17]	0.007 / 0.012
C — fine-tune + retrieval	8.05 ± 0.42	+2.80	[1.72, 3.87]	<0.001 / <0.001

Both step-ups are statistically significant on paired tests over the same 60 questions (these were the two comparisons I set out to make; treat them as strong exploratory evidence rather than a confirmatory benchmark).

The more interesting signal is where each jump comes from. Breaking the score into its four rubric components:

System	Correctness /5	Completeness /2	Groundedness /2	Clarity /1
A	1.83	0.97	0.18	1.00
B	2.35	1.03	0.87	1.00
C	3.85	1.55	1.65	1.00

Fine-tuning (A → B) moved groundedness the most (0.18 → 0.87) — the model learned to sound like a rulings answer and stopped confidently inventing card text. It also nudged correctness up (1.83 → 2.35). So fine-tuning did encode some facts — I'm not claiming it can't. But the larger correctness gain came from retrieval (B → C: 2.35 → 3.85), where real passages entered the prompt.

That's the first finding, stated carefully: in this setup, fine-tuning's main contribution was answer shape and grounding behavior, while retrieval contributed the larger factual-correctness gain. "Fine-tuning teaches the shape, retrieval supplies the facts" is a useful mnemonic for that pattern — but it's a metaphor for a difference in magnitude here, not a literal claim that fine-tuning cannot store facts.

Result 2: once retrieval was good, the retrieval-side changes I tried didn't move answer quality

System C averages 8.05/10, not 10 — the study reports 40 of 60 questions already at a perfect 10, so the lost points sit in a short tail. The retriever feeding it is already strong: hybrid dense (all-MiniLM-L6-v2) + BM25 fused with Reciprocal Rank Fusion, top-5 over 42,412 chunks. recall@5 = 0.933; the gold passage is at rank 1 about 72% of the time (recall@1 = 0.72) and in the top-5 for 93%. Recall does vary by source — rulings 0.96, card facts 0.93, archetype/lore lower on a small sub-sample — so "high recall" is an average, not uniform.

So I tried to buy back the tail from the retrieval side. Six experiments:

Cross-encoder reranking (retrieve top-20 → rerank → top-5). Result: no help, and slightly worse — 8.05 → 7.55 on the main set; on a balanced 60-question split the drop (8.25 → 7.37) was statistically significant.
Was the reranker mis-ranking? Result: no — recall@5 was identical (0.933) and gold stayed at rank 1 either way. The drop tracked the small reader reacting to reshuffled context, not a ranking defect.
Deeper retrieval (rerank a wider candidate pool). Result: recall rose (0.93 → 0.95 → 0.97) but the mean score did not move.
Chunk-repair (a card's effect text was split across 1000-char chunks; I reconstructed it with adjacent-chunk expansion). Result: it fixed the retrieval defect — the stranded clause was back in context — but the answer stayed incomplete; the reader didn't use the recovered clause. (This defect was rare: ~0.2% of cards.)
Failure re-diagnosis by question type. Result: the remaining tail is dominated by yes/no reasoning inversions — the model flips the answer with the correct passage present — rather than by missing facts.
Six in-model reader fixes (self-consistency, self-verification, quote-then-answer, and more), tested on two hard cards. Result: the in-model tricks didn't fix it; the one change that did was swapping in a stronger reader on the same context (next section).

The pattern across experiments 1–4: each change I tested improved (or held) a retrieval metric while producing no measurable end-to-end gain on this evaluation — and reranking actually cost a little. That is evidence of diminishing end-to-end returns from retrieval optimization in this high-recall regime, which is a narrower and more defensible statement than "retrieval stops mattering."

Why would more recall not help? Because recall is a Layer-1 metric. The tail failures here look like Layer-2/Layer-3 problems: the evidence is present but the reader mis-reads it (the yes/no inversions), or a recovered clause is ignored. High retrieval recall does not guarantee successful evidence utilization — and that gap is where the remaining points went.

The reader test — what it does and doesn't establish

Experiment 6 is the most informative, because it holds the retrieved context fixed and changes only the reader:

same question → same retrieved context → different reader → different answer.

On the two hardest cards (Blackwing FAM and Endymion), a stronger reader given the identical passages produced the complete, correct answer that the fine-tuned 2.6B reader kept under-delivering, including the exact effect clause it dropped.

Here's the honest scope. This is a controlled, same-context comparison — stronger evidence for a reader limitation than simply observing that RAG helps. But it was run on two cards, and the reference-grounded judge is stochastic run to run, so the study reports the result qualitatively (complete vs under-answered) rather than as fixed per-question scores. So the defensible conclusion is:

This comparison provides evidence that reader capability is a limiting factor for at least some of the remaining failures — not proof that the reader is the single binding constraint across the whole tail.

The failure taxonomy in experiment 5 (a tail dominated by reasoning inversions with the passage present) is consistent with that reading, but it's a diagnosis, not a controlled manipulation. The clean way to settle it is an experiment I did not run — see "The strongest next test."

What this suggests if you're building RAG

Scoped to settings resembling this one (a small reader, a fact-dense domain, already-high recall):

If recall is already high, verify that added retrieval effort is actually changing answers before investing in it — in this study, reranking/deeper-retrieval/chunk-repair each moved retrieval metrics and not answer quality.
Recall is necessary, not sufficient. A passage in the context window is not a fact in the answer; check evidence utility and reader behavior, not just recall@k.
When retrieval is good and quality has plateaued, the higher-leverage move may be the reader — a stronger model, or RAG-aware training (the RAFT / preference-tuned line) — rather than a cleverer retriever.
This does not say retrieval is unimportant: 5.25 → 8.05 was retrieval. It says retrieval's marginal return fell off once recall was already high here.

And a diagnostic workflow that generalizes better than any single number:

1. Measure retrieval recall (Layer 1)
2. Inspect evidence sufficiency on failures (Layer 2)
3. Measure end-to-end answer quality
4. Test with oracle / gold context (isolates the retrieval ceiling)
5. Compare readers on identical context (isolates the reader ceiling)
6. Decide whether retrieval or reader is the dominant limitation — then spend effort there

How the evaluation was designed to reduce judge bias

Every answer was scored by an LLM judge (gemini-3.1-flash-lite) that is reference-grounded (handed the gold answer plus the verbatim evidence and told to grade against that, not its own knowledge — which is what lets it fairly score a domain it wasn't trained on), blind (never shown which system produced an answer), pointwise (one answer at a time), and run over answers in shuffled order for extra blindness. The rubric sums to 10 (correctness 0–5, completeness 0–2, groundedness 0–2, clarity 0–1) with two guardrails: inventing a card detail forces groundedness to 0, and a correct refusal must outscore a confident wrong answer. Systems are compared paired (same 60 questions) with bootstrap 95% CIs cross-checked by paired t-test and Wilcoxon.

What this design does not yet establish, and I'd add before calling the measurement airtight: the judge temperature wasn't fixed, each answer was judged once (no repeated-judge consistency measured), and judge scores were never calibrated against human ratings. The study already notes the judge is stochastic per question; that variance is real and unquantified. Treat the 60-question means as reliable and individual per-question scores as noisy.

The whole study cost about $3.26 in GPU and API spend.

Limitations

The boundaries of what this experiment tested, and where I would not extend the conclusions:

Single specialized domain (Yu-Gi-Oh). Retrieval behavior and reader difficulty may differ substantially in legal, medical, financial, scientific-literature, enterprise-KB, multi-hop, or open-domain settings.
60 held-out questions. Enough for the paired step-ups to reach significance; too small to characterize RAG in general or to pin a universal recall threshold.
LLM-as-judge, single pass, temperature unfixed, no human-agreement calibration.
Entity overlap not excluded. The held-out set is built from pages held out of training (a document-level split) and was decontaminated against training questions — so it tests generalization to unseen pages/questions. But the same cards/entities can appear across training and test pages, so this is not an entity-disjoint evaluation.
The reader finding rests on two cards in a controlled swap, reported qualitatively.
Recall ≠ evidence utility — I measured recall, and inferred utility failures from error analysis rather than measuring sufficiency directly.
One retriever/reranker configuration (MiniLM + BM25, RRF, one cross-encoder). Other configs may behave differently.
No oracle-context experiment was run, so the retrieval ceiling and reader ceiling aren't cleanly separated numerically.
Findings are strongest for small readers in a high-recall regime; they may not hold where recall is the actual bottleneck or where multi-hop reasoning dominates.
No human evaluation or cross-domain replication — both would be needed to call this more than a case study.

Takeaway

Fine-tuning changed how this model answered and improved grounding, but retrieval produced the larger factual-correctness gain. Once recall was already high, the retrieval-side changes I tested did not improve end-to-end answer quality, and a same-context reader swap on two hard cards pointed at reader capability as a limiting factor for at least some remaining failures.

These findings come from a 60-question evaluation in a single specialized domain with one model and one retriever, so they are best read as a controlled engineering case study — not a universal recall threshold and not proof that readers always dominate retrieval. The transferable lesson is diagnostic: once relevant evidence is consistently reaching the context window, measure whether the reader can actually use it before spending more effort optimizing retrieval.

The strongest next test

The cleanest way to turn the reader story from "suggested" into "demonstrated" is a 2×2 that separates the two ceilings — which I have not run:

	Actual retrieved context	Oracle / gold context
Weak reader (2.6B)	measured (System C)	isolates the retrieval ceiling
Strong reader	isolates residual reader gains	upper bound with perfect evidence

Comparing the columns shows how much quality is lost to imperfect evidence (retrieval ceiling); comparing the rows shows how much is lost to the reader (reader ceiling). Run across all 60 questions with fixed judge settings and repeated judging, it would replace the two-card qualitative result with a quantitative separation — and it's the experiment I'd prioritize before making any stronger claim.

Code, the 60-question eval set, the judge, and all six experiments: github.com/Ace-2504/short-answers-broken-rag. Live arena where all three systems answer and a live judge scores them: harman-ygo-slm.vercel.app.

The Bugs That Hide Behind "It Works": Debugging a Multithreaded C++ Proxy Server

Ace-2504 — Mon, 08 Jun 2026 08:37:27 +0000

As a student, one of the biggest lessons I've picked up is that a program which compiles and runs is not the same as a program that is correct. It really hit home while I was building a multithreaded C++ network proxy server — a project that authenticates users with SHA-256, enforces role-based website filtering, and caches HTTP responses using a custom LRU cache.

The happy path worked on pretty early. The real learning started when I went looking for the bugs that don't announce themselves — the ones that only show up under concurrency, fragmented packets, or heavy load. Here are four of the most interesting and challenging issues I ran into and how I fixed them.

1. The Thread Pool That Never Detached

The proxy spawns 20 persistent worker threads at startup. The idea was simple: create the workers, then detach them from the main thread so they run independently.

The bug was an ordering mistake. My detachment loop ran before the threads were created:

for (auto& t : workers) t.detach();   // workers is still EMPTY here
workers.emplace_back(...);            // threads created afterward

Because the vector was empty when the loop ran, the iteration did nothing — the threads were never actually detached. It's the kind of bug that compiles cleanly, passes a quick test, and then causes resource and lifecycle problems once the server runs for a while.

The fix: I changed the order of the code. First, I created all the worker threads. Then, I detached them. What I learned: when working with threads, the order of setup steps matters. It is part of making the program correct, not just a small coding detail.

2. A Data Race Hiding Inside the Logger

My logging system looked thread-safe. Every write to proxy.log was guarded by a std::lock_guard<std::mutex>, so 20 threads writing at once could never scramble the file.

But the timestamps came from ctime(&now). Under POSIX, ctime returns a pointer to a globally shared static buffer. My mutex protected the file stream — it did NOT protect that hidden global buffer. So two threads formatting timestamps at the same time could corrupt each other's strings, even though the file writes themselves were perfectly safe.

The fix: switched to the thread-safe version, ctime_r(), which writes into a buffer. The lesson was a subtle one: locking the obvious shared resource isn't enough. You also have to think about the shared state hiding inside the standard library functions you're calling.

3. The TCP Read That Assumed Too Much

My request handler made a single call:

recv(client_socket, buffer.data(), BUFFER_SIZE, 0);

This assumes that one recv() gives you one complete HTTP request. TCP makes no such promise. TCP is a byte stream, not a message protocol — a request can arrive split across several packets. If a header got cut in half, my request.find("Host:") parsing would just fail, and a valid request would get dropped for no obvious reason.

The fix: replaced the single read with a loop that keeps calling recv() until the \r\n\r\n header terminator has fully arrived. For my proxy, this handled the request headers; a fuller HTTP implementation would also need to handle bodies using Content-Length or chunked transfer encoding. This turned out to be one of the most common (and most underestimated) mistakes in network programming: treating a stream like a neat sequence of separate messages.

4. Defending Against Hanging Connections

Worker threads are a limited resource — I only have 20. A single unresponsive remote server that never closes its connection could tie up a worker forever, and 20 of those stalls would quietly take the whole proxy offline.

My defense was setting a receive timeout with setsockopt() using SO_RCVTIMEO, set to one second. If a remote server goes quiet mid-response, the thread frees itself instead of hanging forever. Combined with proper HTTP status codes sent back to the client — 502 Bad Gateway when DNS resolution fails, 504 Gateway Timeout when the upstream stalls — the proxy fails gracefully instead of dying silently.

What Debugging Concurrent Code Taught Me

The common thread across all four bugs is the same: concurrency and networking break the assumptions that work perfectly in single-threaded, single-packet test runs.

Setup order is part of correctness, not a detail.
A mutex protects what it wraps — and nothing else, including hidden global state.
The network gives you a byte stream, never a tidy message.
Limited resources need timeouts, or one bad peer takes everything down.

None of these bugs threw an error or crashed the compiler. They lived in the gap between "it runs" and "it is correct" — and for me, closing that gap has been the most rewarding part of learning systems programming.

If you've built low-level networked systems in C++, I'd love to hear which subtle concurrency bug cost you the most time to track down.

Turning Behaviour into Climate Accountability

Ace-2504 — Tue, 24 Feb 2026 18:57:22 +0000

One winter morning in Delhi, the AQI crossed 460 — officially hazardous. Schools shut. Hospitals filled. Transport slowed. Offices continued. Every day, millions of environmental decisions are made. Whether to drive or take public transport. Whether to allow remote work. Whether to burn waste or compost. These choices influence emissions — yet we rarely measure the pollution that did not happen because of them.

That is the missing piece.

Most environmental systems measure what exists: current emissions, fuel consumption, air quality levels. They do not formally measure avoided emissions. Counterfactual attribution addresses this gap by comparing two clearly defined scenarios:

Business-as-usual baseline — what would have happened
Verified alternative — what actually occurred

Impact = Baseline − Verified Alternative.

A Practical Example: Urban Commuting
Consider a professional living 14 km from work. A 28 km daily round trip in a petrol car (≈ 120 g CO₂/km) results in about 3.36 kg CO₂ per day. Across 240 working days, that equals approximately 806 kg CO₂ annually. That is the baseline.

Now introduce two behavioural shifts:

Work from home two days per week → avoids ~322 kg per year
Shift to electric metro on remaining days (≈ 15 g CO₂/km) → saves ~423 kg

Total avoided emissions ≈ 745 kg CO₂ Remaining footprint ≈ 61 kg

That is a reduction of over 90%, derived from a measurable behavioural delta — not offsets, not assumptions.

If the remaining 61 kg is neutralised through verified sequestration (e.g., monitored urban trees at ~10 kg per tree annually), the commuting footprint approaches net zero.

Baseline → Reduction → Net Impact. Each step is quantifiable.

Preventing Double Counting
A critical issue emerges: duplication.

If an employee reports work-from-home reductions, and their employer's HR department also reports the same reduction in ESG disclosures, total claimed impact exceeds actual impact.

To prevent this, the framework introduces a non-duplication constraint.

Let:

Δ_total = Verified avoided emissions
C_i = Claim attributed to entity i

The system enforces:

Σ Cᵢ ≤ Δ_total

No combination of claims may exceed the verified delta.

Each reduction event is assigned:

A unique registry ID
A defined ownership tag
A claim status flag

Work-from-home emissions may be attributed to the enabling institution, while individual dashboards reflect behavioural contribution — without generating duplicate claimable credits unless formally allocated.

This ensures:

Measurability
Additionality
Non-duplication
Audit defensibility

Without duplication control, avoided-emission accounting collapses under verification.

Enterprise Implications
Most sustainability reports rely on estimated participation and averaged emission factors. A verification-first counterfactual architecture instead:

Establishes defined baselines
Verifies behavioural change in a privacy-preserving manner
Models avoided emissions at corridor level
Applies attribution constraints
Produces audit-ready metrics

Under such a system, work-from-home is not merely HR flexibility. It becomes measurable climate infrastructure.

The Broader Shift
This logic applies anywhere a conservative baseline can be defined:

Agricultural burn avoidance
Household waste diversion
Fuel switching in buildings
Distributed renewable adoption
Industrial efficiency upgrades

For decades, climate systems focused on measuring what we emit.

The next frontier is measuring what we prevent — without inflation, without duplication, and without ambiguity.

Climate accountability will not be built on slogans.

It will be built on baselines, constraints, and verifiable delta.

Patent Filed, 2026. A technical paper detailing the verification-first MRV architecture, statistical bias bounds, and attribution constraints is under preparation.

How McKinsey Frameworks Fixed My Scattered API

Ace-2504 — Sun, 04 Jan 2026 15:51:33 +0000

While building the backend for Ace Rentals, I realized that my authorization logic, although functionally correct, felt increasingly fragile. Ownership checks were scattered across multiple routes, duplicated in several places, and easy to forget when adding new endpoints. Over time, this made the system harder to reason about and increased the risk of subtle security gaps.

Rather than continuing with incremental fixes, I stepped back and redesigned authorization as a system-level concern using centralized middleware. This article explains how I identified the issue, how I reasoned about the redesign, how it was implemented, and what improved as a result.

Previous Authorization Approach

Authorization checks were implemented directly inside individual route handlers. Each protected endpoint contained its own logic to verify whether the current user was allowed to perform the requested action.

While this approach worked initially, it introduced several challenges as the application grew:

The same ownership logic was copied across multiple routes
Route handlers mixed business logic with authorization concerns
Adding new endpoints required manual checks, increasing the chance of mistakes

The system did not fail outright, but correctness increasingly depended on careful and repetitive implementation.

Identifying the Design Issue

The problem was not a missing check or a faulty condition—it was a structural issue. Authorization was treated as an implementation detail instead of a rule enforced consistently by the architecture.

This meant security relied on developer memory rather than system guarantees. As the number of routes increased, so did the effort required to maintain consistency and confidence in the design.

Design Analysis and Decision-Making

To avoid patching individual routes, I analyzed the problem at a design level.

I used MECE-style analysis to verify whether authorization was applied consistently across all relevant routes. This exposed gaps and overlap in enforcement.
I compared alternative approaches and found that centralized middleware offered the best balance between reuse, clarity, and maintainability.
I set clear constraints for the refactor—no change in behavior, minimal surface area, and focused scope—to avoid unnecessary complexity.

This structured approach helped ensure the redesign addressed the root cause rather than its symptoms.

Authorization Redesign and Implementation

Authorization was elevated to a system-level concern and implemented through middleware:

Ownership checks were consolidated into a single, reusable middleware function
API endpoints were reorganized around resources rather than actions, improving clarity
Authorization was enforced early in the request pipeline, before any business logic executed

With this setup, routes automatically inherit authorization rules. Developers no longer need to remember to reapply checks when adding or modifying endpoints.

Observed Impact of the Redesign

The redesign produced clear and measurable improvements:

Authorization logic now exists in one place instead of being duplicated
All protected routes enforce authorization consistently by default
Maintenance effort and regression risk were significantly reduced
Resource relationships are clearer through resource-based routing
Route handlers are simpler and focus only on business logic

A qualitative review showed that core domain logic is stable. Remaining risk is isolated to shared authorization middleware, where it is explicit, visible, and easier to reason about.

Key Learning

Repeated logic is often a signal of a deeper design issue. Treating security as a structural concern—rather than a manual step—leads to systems that are easier to maintain, extend, and trust over time.

Design Decisions and Learnings

This article provides a high-level summary of the decisions and outcomes from redesigning authorization in Ace Rentals.

For a deeper look into my learning experience—including diagrams, structured reasoning, trade-offs, and implementation details—you can read the full write-up here:

👉 https://ace-2504.github.io/api-design-blog/

The full version documents the complete thought process and lessons learned while designing and refactoring the system.