Intro
Model providers keep shipping bigger context windows: 100k tokens, 200k, over a million in some cases. The marketing pitch is simp...
For further actions, you may consider blocking this person and/or reporting abuse
The line that matters is "being in context and being attended to are not the same thing", and the uncomfortable part is that you cannot tell the difference by reading the answer. A RAG answer can cite the right chunk and still come from the model's memory, with the citation as decoration.
The only way I have found to actually know is counterfactual: drop the evidence, corrupt it, swap it for a lookalike, re-run the same generator, and watch whether the answer moves the way a grounded one must. If the answer survives with the evidence gone, the context was upholstery. I ended up building a small probe that does exactly this (grounding-probe on my GitHub) after getting burned by a pipeline that silently truncated context for weeks while the bot kept answering fluently. Nobody noticed, because fluency is not grounding.
On the margin point, one addition from measuring it: the real bill is total input plus output across every call a query makes, not chunk math. Priced per call, "top 15 chunks to be safe" is the most expensive insurance nobody ever claims on.
Your counterfactual framing sent me to run it against our own system this weekend, so here is what came back, including the parts that went against me.
We inject a corpus of small curated notes into our agents, 122 of them, and we had only ever measured their cost, never their value. I built your four arms: real, dropped, corrupted so the rule is inverted, and a lookalike note of similar shape.
On four hand-picked notes: 0 of 4 upholstery, 4 of 4 moved the answer, 3 of 4 proved attention through the inverted arm. The inversion arm is the one I would push people toward hardest. A model that follows the WRONG rule when you corrupt the note is positive evidence it read the text, where an unchanged answer is only absence of evidence. Caveat that matters: I picked those four for having crisp checkable rules, which biases hard toward load-bearing, so 4 of 4 says "these four are", not "ours are".
Then I ran all 122, and three things are worth passing back to you.
One, the echo control. I added an irrelevant note of the same shape and the model parroted its identifiers back 46 times out of 49. Echo rate 94 percent, against a 6 percent baseline in the no-note arm. So "the evidence appears in the answer" was nearly free on this reader, and the only informative arm was the one with the evidence removed. That killed my original metric at row 27 and I had to relabel every verdict down to the narrower claim the data actually supports, which is "the model would not have said this without the note", not "the note was useful".
Two, a blind spot in the method that I have not solved. Only 49 of 122 were measurable at all. The other 48 carry no identifier to check for, because they are judgements and architecture decisions rather than facts. The method needs something anchor shaped in the output. So "unmeasurable by this probe" and "upholstery" look identical from the outside, and treating them as the same thing would have deleted exactly the notes that hold our reasoning. I think that is the sharpest edge on the tool.
Three, a confound I could not remove. Notes that transferred averaged 8.4 checkable anchors, notes that did not averaged 4.4. So part of my transfer rate is just how many anchors a note happens to carry, not how good it is.
All of that on a local qwen2.5-coder 7B at temperature zero, one question per note. Evidence about the mechanism, not about a frontier reader.
The last thing I found was not about grounding at all. A share of those notes had never been delivered to the model in the first place, and the channel that delivers them was not writing to its own log, so I could not tell how many. I fixed the logging this morning and I am not going to quote a number until it has collected real ones. Worth checking before trusting any grounding result, because I nearly published a finding that was actually about a delivery bug.
PARAMETRIC_LEAK is the same failure in different clothes and I would not have caught ours without building the drop arm first. Thanks for writing it up.
This is the best bug report the method has ever received, and finding two is the one I will be chewing on for a while. You are right that "unmeasurable by this probe" and "upholstery" look identical from the outside, and the honest move is exactly what you did: a separate verdict, never deletion on absence of evidence. The direction I want to try for judgement-shaped notes is manufacturing the anchor behaviorally instead of textually: a note that holds an architecture decision should flip a forced choice downstream, so the probe becomes corrupt the judgement, watch the decision. Your inversion arm generalized from strings to choices. No identifier needed, just a task whose output is discrete enough to move.
The 94 percent echo rate deserves to be published on its own. It puts a number on the thing everyone assumes silently: presence in the answer is free, and only removal is informative. And your relabel to "the model would not have said this without the note" is the claim counterfactuals actually license, so I already put it in the grounding-probe README, credited to you with a link to this thread. It says it your way now.
The delivery bug might be my favorite part. Half the practical value of grounding harnesses is that they force the delivery path to become observable, and "I nearly published a grounding finding that was actually a delivery bug" is a sentence worth framing. On the anchor-count confound: report transfer conditional on anchors per note, or as per-anchor rates, and the residual gets you closer to note quality. If you write this up as a post, link it back here. It deserves more eyes than a comment thread.
Built it the same day, because it was too good an idea to leave in a comment. Ten judgement-shaped notes from our own store, local 7B, temperature zero, four arms: real, dropped, inverted, and an irrelevant lookalike.
Inverted note flipped the forced choice 10 out of 10. Lookalike moved nothing, 10 out of 10. So judgement notes are attended to and they do govern, and your point stands: the inversion arm generalises from strings to choices, and the discrete output kills the echo problem completely, because there is nothing to parrot.
The uncomfortable half is the useful half. My first six cases were general good practice, things like use a service account rather than a human credential, fail closed when a safety classifier is unsure, do not benchmark a provider at saturating concurrency. All six came back already-known: with no note at all, the model picked our answer anyway. Read, obeyed, and redundant.
So I wrote four where our decision contradicts the obvious answer. Refuse an invoice the payment provider finalized at zero and marked PAID. Temperature zero is not deterministic. A green staging run does not validate a change to the low-cost model path. A non-zero exit from our watcher is normal. On all four the no-note baseline chose the default, and the note moved it to ours.
So the conclusion I did not expect: a judgement note earns its budget exactly where it contradicts the model's prior, and nowhere else. Which is the behavioural twin of what the text probe already told us, that the value is the repo-specific fact and not the general wisdom, because the model brings the wisdom for free. That gives us a writing rule rather than just a measurement: before injecting a decision, ask whether a competent model would already choose it, and if so do not spend the tokens.
Two instrument bugs, both caught by controls, and I mention them because your method is what made them visible. First, every case's correct answer was the same letter, so a position bias would have been indistinguishable from governance. Caught before running and rebalanced. Second, my lookalike check asked whether the irrelevant note avoided the wrong answer, but on a counter-default case the model's own baseline IS the wrong answer, so a correctly inert control scored as a hijack. It read 6 of 10 when it was really 10 of 10. The right question is whether an irrelevant note moves the answer away from the no-note baseline. It does not.
Limits, so nobody quotes this as more than it is: n is 10, I wrote the cases and picked ones with a crisp either/or, and a forced binary choice is a far easier target than real behaviour in a long agent run. Governing a multiple choice answer is not the same as governing an action taken under pressure. The honest next step is sampling the real 48 rather than hand-picking, and I am not pruning anything on this.
Thanks for the README credit, and for the anchors-per-note suggestion on the confound. Reporting transfer conditional on anchor count is obviously right and I had been treating it as a caveat instead of a fix.
"A judgement note earns its budget exactly where it contradicts the model's prior, and nowhere else" is a law worth naming, and it inverts how most teams write their agent rule files. The typical corpus is your first six cases: general good practice the model brings for free, each one paying rent in context and changing nothing. The four counter-default notes are the whole product. It also gives you a pruning tool for existing corpora: test every note against the no-note baseline and delete the ones the model already agrees with. Prior-agreement pruning, essentially free with the harness you just built.
Your second instrument bug is the subtle one and your fix is exactly right: hijack is movement away from the no-note baseline, not agreement with a wrong answer the model already held. A control that punishes the model for its own prior is measuring the wrong thing, and I suspect a lot of published injection evals get this wrong quietly.
One experiment for the pile when you sample the real 48: keep the same notes and vary where they sit in the context, near the top, buried mid-corpus, last. Same forced choice, measure flip rate by depth. That closes the loop with the article we are commenting under, because it turns "attention is the budget" from a thesis into a curve. And write this up as a post. Two people have now built the thing in a weekend; that is the strongest signal a method ever gets.
Ran it the same day. Curve first, then the part where my framing was wrong.
Same 10 counter-default notes, same wording, one governing note pinned at 0 / 25 / 50 / 75 / 100 percent of the block. Every case asked in both option orders and scored on the content chosen. Local qwen2.5-coder 7b at temperature 0. Two block sizes, so length is a variable rather than an assumption.
At 60 notes, about 2.4k tokens: 95, 85, 80, 85, 100. No context at all scores 55.
At 450 notes, about 19.8k tokens: 85, 80, 85, 85, 100.
I pre-specified ends versus middle as the test, since that is the lost-in-the-middle prediction. It gave +14.2 points at 2.4k, Fisher exact p 0.046, and +9.2 points at 20k, p 0.23. So my own contrast did not survive the larger condition, and I am not going to lead with the pooled p.
What did replicate is more interesting than what I went looking for. The middle is 83.3 percent at both sizes, identical. The note sitting immediately before the question was applied 40 out of 40 times, every case, both lengths. And the first position decayed from 95 to 85 as the context grew eight times longer.
So it is not a U that deepens with length. It is a strong recency effect with everything else flat, and the primacy half washes out as the context grows. Last versus every other depth, pooled, is 40/40 against 136/160, +15 points, p 0.005. I am flagging that as post-hoc because I chose the contrast after seeing the curve. It is the pre-registration for the next run, not a result.
The obvious rival is that last is not a depth at all, it is adjacency to the question. That is running now: same design, but N notes reserved at each edge that the target can never occupy, so the extreme positions sit inside the block instead of against a boundary. If the advantage survives the padding it is about lateness in the block. If it vanishes, the honest claim is recency and nothing about depth.
Prior-agreement pruning is built, with the validity gate your point about the lookalike control implies: a note is only a deletion candidate if the note and its INVERSION produce different behaviour. Otherwise the question is not governed by the note and a no-note agreement says nothing about it. My first version of that gate was wrong in the dangerous direction, so a note that governed nothing would have been nominated for deletion.
One instrument note, because it would have handed me your hypothesis for free and I nearly shipped it. ollama truncates a prompt that exceeds num_ctx, silently, and it discards the FRONT first. Unset, an 11,848 token prompt evaluated 2,050 tokens and a marker pinned to the first line became unrecoverable. Set to 32768 it evaluated all of it and recovered. Both are a normal 200, and prompt_eval_count is the only tell. Because truncation eats the front, it deletes the position-0 arm and leaves the last-position arm intact, which manufactures exactly the curve this experiment is looking for. Anyone reproducing this on a local runtime should pin the window and assert the largest observed prompt token count stayed clear of it.
On writing it up, not yet. The pre-specified contrast failed to replicate, the strongest number is post-hoc, and the control that could overturn the framing is still running. When it settles I would rather publish the version that survived those than the version I hoped for this morning.
The line I'd underline for anyone in a regulated domain: "the system doesn't fail loudly, it fails convincingly." Latency and cost you notice. A confidently-wrong answer assembled from loosely-related context you don't, until it's in front of a customer or attached to a number that matters. We feed agents the minimum task-scoped context instead of the full bucket, and the token savings are almost a side effect. The real reason is that a small, curated payload is the only version we can actually reason about when the output has consequences. The narrower the context, the less room a plausible-but-wrong answer has to hide.
That reframing is the one I wish more teams internalised early: curation isn't primarily a cost optimisation, it's a correctness strategy that happens to save tokens. In regulated or high-consequence domains especially, the review surface matters as much as the answer itself. If a human has to audit why the model said what it said, a 200-token task-scoped payload is legible and a 90k dump is not, even if both technically "contain" the right information. The "less room to hide" phrasing is sharp too, because it captures something the loud-failure crowd tends to miss: plausibility scales with the surface area of loosely related material you hand the model, not with how wrong the underlying reasoning is. Narrow context isn't just cheaper and faster, it's the version you can actually defend after the fact.
"Plausibility scales with the surface area of loosely related material you hand the model, not with how wrong the reasoning is" is the sentence I'll be quoting. That's the mechanism nobody names. And you closed the loop I was circling: narrow context is the version you can defend after the fact. A 90k dump technically contains the answer the same way a landfill contains your keys, true and useless the moment someone asks you to point to it. Great piece. It put words to something we'd been doing by instinct.
The landfill line is going straight into my slides, credit attached. It captures the thing that "unlimited context" marketing quietly elides: retrieval isn't just about whether the answer is present, it's about whether it's addressable under the constraints that matter, latency, attention, and after-the-fact review. Presence without addressability is the same failure mode dressed up as a feature.
The instinct point is the one I keep coming back to in these threads. A lot of teams doing this well arrived at narrow, task-scoped context by pain rather than principle: something failed convincingly in production, someone had to reconstruct why, and the 90k payload turned out to be unreviewable. The instinct is downstream of the incident. The teams that skip the incident are usually the ones who treated context as a data engineering problem from day one rather than a prompt problem, which is really the whole argument compressed: the discipline predates LLMs, we just keep rediscovering it with new vocabulary.
Steal away, credit optional. "Addressable under the constraints that matter" is the cleaner version of what I was circling. Presence is cheap. A model can have the answer sitting in context and still never surface it under latency and attention limits, which in a regulated flow is indistinguishable from not having it at all. And the review point is the one teams underrate: narrow context isn't just faster, it's what makes the decision defensible when someone audits it six months later.
"Indistinguishable from not having it at all" is the version that should be in the docs. That's the collapse the big-window pitch depends on you not making: it treats presence and retrieval as the same event, when in production they're separated by latency budgets, attention decay, and eventually an auditor. Under any of those three, unaddressable context is just latency and cost with extra steps.
The six-month audit point is the one I'd add to the original post if I were rewriting it. Most of the failure modes I listed are observable in the first week: latency shows up in dashboards, cost shows up in invoices, lost-in-the-middle shows up in evals if you're looking. Defensibility is the one that only bills you later, and by then the payload shape is baked into the system and expensive to unwind. Worth designing for on day one precisely because it's the failure mode you can't A/B test your way out of.
@cyclopt_dimitrisk The smallest possible version of this get to me in an email triage bot I shipped. Make.com watches Gmail, gpt-4o-mini classifies each message into four labels, reply, newsletter, spam, wait. The first version passed the full email body into the prompt because the window fit it easily, so why not.
Then the newsletters arrived. A single HTML newsletter can be tens of thousands of characters of tracking links and inline CSS, and the classifier started missing obvious calls because the actual signal- sender, subject, first few lines, was buried under markup. "Being in context and being attended to are not the same thing", exactly as you put it.
The fix was boring: strip the HTML, classify on the subject plus the first few hundred characters, pass nothing else. Accuracy went up, and the whole bot costs about £1 a month to run. Same lesson at toy scale: the model did not need more context, it needed less, chosen on purpose.
I now apply the same rule in every automation I build: each module gets the minimum payload the next step needs, nothing "just in case". It is cheaper, faster, and easier to debug when something mislabels.
The email triage example is a perfect miniature of the whole problem, and honestly the fact that it's £1/month makes the point sharper, not weaker. The failure mode doesn't need scale to appear, it just needs signal buried in noise, and an HTML newsletter is a very efficient noise generator. Tracking pixels, inline styles, and utm-laden links are exactly the kind of tokens that look like content to a tokenizer and mean nothing to a classifier.
The "minimum payload the next step needs" rule is the one I wish more automation tutorials led with. There's a default instinct to pass the whole object between steps because it's convenient and the window fits it, and the cost of that instinct only shows up later as either accuracy drift or a bill that grows faster than usage. Curating the payload at each hop is unglamorous compared to prompt tweaking, but it's usually where the actual quality gains hide. Nice fix.
I agree that larger context windows are useful, but they aren't a substitute for good system design. Smart retrieval, filtering, and summarization usually deliver better accuracy, lower costs, and faster responses than simply sending everything to the model. In production, context quality matters far more than context quantity.
Thanks, that's basically the thesis in one paragraph, and better compressed than my post managed.
The one thing I'd add: I think the "quality over quantity" framing is right but it can make the work sound softer than it is. In practice, "context quality" is a stack of concrete engineering decisions - chunk boundaries, embedding choice, hybrid retrieval vs. pure vector, reranking, dedup, recency weighting, summarization thresholds, task-specific routing. Each of those has measurable effects on latency, cost, and answer quality, and each is testable. So it's less a philosophy and more a pipeline you can profile and regress on, the same way you would a search system.
The trap I see teams fall into is treating retrieval as a one-time setup ("we picked a vector DB, we're done") rather than as an evolving component with its own metrics and its own on-call surface. Recall@k on a labeled eval set, cost per successful task, p95 latency by query type - once those are dashboards, the "just send more tokens" instinct dies pretty quickly, because you can see exactly what it costs and what it doesn't buy you.
Have you found particular metrics or eval setups that worked well for driving those decisions on your side?
The point about cost and latency scaling with everything you put in the window matches what the attention math implies, and retrieval quality starts dropping long before the limit does. Choosing what to include is engineering work that a bigger window lets you postpone rather than skip. Do you have a rule of thumb for when to summarize versus retrieve fresh?
Good question and I don't think I have a clean rule, but the heuristic I've landed on is roughly: summarize when the information is stable and referenced often, retrieve fresh when it's volatile or referenced rarely.
The reasoning is that summaries are a form of caching, and caches only pay off when the read-to-write ratio is high enough to justify the staleness risk. A running summary of a long conversation is worth it because you'll reference it on every turn and the underlying facts don't change. A summary of a policy document that gets updated quarterly is worth it because reads massively outnumber writes. But a summary of last week's ticket volume, when the numbers change hourly and only get asked about occasionally, is worse than fresh retrieval because you're paying the summarization cost to serve stale data.
The other axis is fidelity loss. Summarization is lossy by definition, so if the downstream task needs exact quotes, specific numbers, or legal precision, you retrieve fresh even if it's expensive. Summaries are for gist, retrieval is for ground truth. A lot of production bugs come from summarizing things that needed to stay verbatim, then debugging why the model confidently paraphrased a number wrong.
God, imagine a llm with unlimited context and how bad context drift is gonna be
haha right? unlimited context basically means unlimited ways for the model to mash turn 12's half-baked idea together with turn 380's actual answer and hand you back some Frankenstein reply with full confidence.
Honestly at this point I don't want more context I want better forgetting.