DEV Community: jidonglab

Why Great Engineers Bomb the Behavioral Interview Round

jidonglab — Thu, 27 Aug 2026 03:48:24 +0000

He nailed the system design. Distributed rate limiter, shard plan, back-of-envelope math, graceful degradation when Redis falls over. Best design hour I'd sat in on all quarter.

Then the last interviewer asked him to describe a project of his that failed, and he said, "Honestly, I can't think of one."

That sentence ended the loop. Not the design round. Not the coding round, which he also passed. The behavioral interview round killed the strongest candidate in the pipeline by every technical measure in that debrief.

I've sat on both sides of this. Here's what's actually happening in that room.

TL;DR

The behavioral interview round scores four things: ownership, conflict handling, impact awareness, and self-awareness. Likability is not on the rubric.
Interviewers write quotes, not vibes. In the debrief, the most specific quote wins the argument. Vague answers give your advocate nothing to fight with.
The single most common killer is the "we" trap: 90 seconds of "we migrated, we decided, we shipped" and the interviewer cannot write one bullet about you.
STAR is table stakes. The part that separates a hire from a no-hire is what you'd do differently now.
Target 90 seconds per answer, max 3 sentences of setup before you enter your own story, and at least one real number.

What does the behavioral interview round actually test?

It tests whether someone can hand you a problem and stop thinking about it.

Every rubric I've worked with is a rewording of that. Ownership: do you drive things or get assigned things? Conflict: can you disagree with a staff engineer without folding or starting a war? Impact: do you know whether your work mattered, in units? Self-awareness: can you name a thing you got wrong without a lawyer present?

The mechanism matters more than the rubric. During your answer I'm typing quotes into a doc. Days later, five people who each saw 45 minutes of you argue for an hour, and the person defending you has only my notes. If my notes say "solid communicator, seemed collaborative," you're dead. If they say "caught the double-write bug in staging, paged himself at 2am, wrote the runbook so it wouldn't happen twice," you're hired. Give people ammunition. That's the whole game.

Why do great engineers bomb the behavioral interview round?

Because they've spent ten years being rewarded for the exact opposite instincts.

Engineering culture trains modesty. Saying "I did that" feels like stolen valor when six people touched the service. So you say "we," you round your contribution down, and you read as a passenger.

It also trains precision. You won't say "cut latency by 60%" because it might have been 45%, and only on the write path. So you say "it got noticeably faster," which is worthless to me.

And your best work is often your least narratable work. Two years grinding a legacy billing system into something maintainable has no dramatic arc. The person who spent three weeks on a flashy demo interviews better than you, and that's genuinely unfair. The fix isn't to become that person. It's to build the narrative your work deserves before someone asks for it cold.

What is the "we" trap and how do I get out of it?

The "we" trap is telling a team story so thoroughly that you delete yourself from it. It's the number one reason a strong engineer gets a "no signal" score instead of a hire.

I once counted 14 uses of "we" and zero uses of "I" in one answer. The work sounded excellent. I could not tell you what the candidate did.

The fix is mechanical. Use we for context. Use I for decisions.

Bad:

"We were seeing timeouts on checkout, so we looked at the query patterns and we added a cache layer, and after that we got the p99 down."

Better:

"Checkout was timing out about 2% of the time. I pulled the slow query log and found one N+1 in the cart serializer that was firing 300 queries per request. I argued for fixing the serializer instead of adding a cache, because caching would have hidden it. Two of us pushed back on that for a day. We fixed the serializer, p99 went from about 1.9s to 240ms, and the cache work got cancelled."

Same story. Same team. One of them is a bullet in the debrief doc.

How do I answer "tell me about a time you disagreed with someone"?

The question is not asking whether you were right. It's asking whether you can hold a position with evidence and then commit to a decision that went against you.

Three ways people fail it:

The doormat. "I raised my concern, they disagreed, so I went with their call." No conviction, no data, no follow-up. Reads as someone who won't stop a bad launch.
The martyr. "I warned them the schema wouldn't scale, they ignored me, and six months later it fell over exactly like I said." You just told me you'd rather be right than effective. Also that you're still angry.
The ghost. "I don't really have conflicts, I get along with everyone." Nobody believes this and it burns 30 seconds.

What lands is the boring version: you disagreed, you brought a number or a prototype instead of an opinion, you set a decision deadline, you lost, you committed anyway, and you said what you'd watch for. Then add the sentence most people skip: what you learned about why the other person made that call. That's the self-awareness box, and almost nobody checks it.

Is STAR still the right format for behavioral answers?

STAR is fine scaffolding, but Situation-Task-Action-Result is the floor, not the ceiling. Everyone hits it. On a panel day I'll hear eight structurally identical STAR answers and they blur.

Add a fifth beat: what I'd do differently now. One or two sentences at the end.

"Looking back, I'd have shipped the read path behind a flag first instead of doing both at once. The rollback took four hours and it should have taken four minutes."

That line proves the experience actually updated you. It also quietly answers the failure question before anyone asks it.

How long should a behavioral answer be?

Ninety seconds to two minutes. Setup gets three sentences, maximum, before you appear in your own story.

I've heard four-minute answers where the candidate was still describing the org chart at minute two. By then I'm not scoring the story, I'm scoring whether you notice people have stopped listening. Worse question for you.

If you're not sure, stop early and ask, "Want me to go deeper on the technical side or on how the team handled it?" That question alone reads as senior.

How do you practice behavioral interviews without a friend to run them?

Out loud, timed, recorded. Reading notes silently doesn't work, because the failure mode is verbal. You'll find out that you say "we" constantly, that your setup runs 90 seconds, and that you have no number for the project you're proudest of.

I got obsessed enough with this problem that I built a tool for it. Preterview runs voice mock interviews with three different interviewer styles, reads your resume or portfolio first so the questions are about your actual work, and hands back a written report on what you said (full disclosure: I built it, it's at preterview.com). The reason I made it voice-based rather than text: typing your answers hides every problem that matters here.

Whatever you use, the artifact you want is a story bank: six written stories, each tagged for what it covers. A shipped thing, a failure, a conflict, an ambiguous problem, a time you mentored someone, a time you changed your mind. Six flex to cover about forty questions. Trying to have a unique story per question is how people freeze.

What kills candidates instantly in a behavioral round?

Naming and blaming a specific ex-teammate. Saying you can't think of a failure. Trashing your last codebase without saying what you did about it. Claiming scope that dies on one follow-up, which is the fastest trust collapse there is: say "I built the platform," fail to answer who owned deploys, and everything else you said gets re-audited.

And the quiet one: answering a people question with pure architecture. Ask how you handled a teammate who kept shipping without tests, get a description of CI config, and I've learned you route around humans instead of talking to them.

So why do great engineers bomb the behavioral interview round?

Because they treat it as small talk wrapped around the real interview, when it's the round with the widest scoring spread and the least preparation behind it. The technical rounds have a ceiling everyone's grinding toward. The behavioral round is scored on ownership, conflict, impact, and self-awareness, and most strong engineers walk in with no stories prepared, no numbers, and a lifetime of habit that says "we" when it should say "I." Write six stories down, put a real number in each, say what you'd do differently, keep it under two minutes, and rehearse it out loud. It's the cheapest interview points available and almost nobody picks them up.

Vibe Coding Is Fine. Vibe Debugging Is What Kills You

jidonglab — Wed, 26 Aug 2026 16:02:20 +0000

We shipped a job that sent order confirmation emails. It deduped by keeping message IDs in an in-memory Set. Clean code. An agent wrote it, I skimmed it, tests passed, staging was happy for two weeks.

Then traffic spiked, autoscaling added a second instance, and a few hundred customers got the same email twice. Two Sets. Zero shared state. Obvious in hindsight, invisible in review.

Here's the part that actually cost me: finding it took nine hours, and roughly seven of those were spent watching an AI agent confidently fix things that were not the bug. Vibe coding didn't hurt me. Vibe debugging did.

TL;DR

Vibe coding works because generation has a cheap oracle: you run it, you see if it does the thing. Wrong guesses die instantly.
Vibe debugging breaks because debugging is a search over runtime state, and your agent has never observed your runtime. It only sees text.
A wrong guess while coding costs you a rejected diff. A wrong guess while debugging costs you a plausible diff that changes behavior without fixing the cause. Now you have two bugs.
The fix is not a better prompt. It's a deterministic repro. A failing test is the best prompt you will ever write.
After ~3 failed attempts, the agent's own rejected theories are polluting its context. Reset the session instead of pushing harder.

What is vibe coding, and why does it usually work?

Vibe coding is letting a model write code you accept without fully reading it. I do it every day and I'm not going to pretend otherwise.

It works because the feedback loop is short and the correctness check lives outside the model. Does the endpoint return 200? Does the page render? Does the CLI print the right thing? You don't need to understand the diff to evaluate it, because reality evaluates it for you in about four seconds.

Generation is also the thing language models are genuinely best at. Millions of examples of "function that takes X and returns Y" exist in training data. Your CRUD handler is not a novel artifact.

Why is vibe debugging so much worse than vibe coding?

Because a bug is a claim about something that happened inside a process the model never watched. It gets a stack trace, a log line, and your grumpy one-sentence description. From that it does the only thing available: pattern-match your symptom against the most statistically common cause of that symptom.

That's right often enough to be dangerous. When it's wrong, it doesn't return an error. It returns a fix.

My duplicate-email bug looked exactly like a race condition, because it was a concurrency problem. So the agent proposed a mutex around the Set. Textbook answer for a single process. Completely useless across two of them. The tests still passed. The duplicates kept going out. And now the code had a lock in it that would live there forever, quietly making every future reader assume the concurrency question had been handled.

What does an AI agent actually do when it can't find the bug?

Four moves, in this order, every time. You'll recognize all of them:

The null guard. if (!x) return; The crash disappears. The bug relocates downstream, where it now looks like a different bug.
The try/except sponge. Wrap it, log it, call it "handled." Your error rate drops. Your correctness doesn't change.
The rewrite. Narrow fixes failed, so it widens the scope and reimplements the whole function. Now the diff is 200 lines and you cannot tell which line mattered, or whether anything got fixed versus reshuffled.
The confident close. "Fixed! The issue was that the Set wasn't thread-safe." Stated as fact. Nothing was run.

None of this is the model lying. It's what you get from a system optimized to produce a plausible patch, with no ground truth to check itself against. Give it no oracle and it will invent the feeling of one.

Why does the agent get worse the longer you debug with it?

Because its own wrong theories become context. Turn 1 it says the issue is the lock. Turn 4 it's reading "the issue is the lock" as evidence in its own transcript. You've built a machine that cites itself.

I've watched an agent spend twelve turns defending a diagnosis I had already disproven in turn three, because my disproof was one line and its theory was six paragraphs.

Three strikes and you reset. Open a clean session and paste only facts: the repro steps, the actual observed values, what you ruled out and how. Not the conversation. The evidence.

How do I stop the vibe debugging death spiral?

Five rules. They cost real time up front and they've saved me entire days.

1. No repro, no agent. Spend the first hour making it fail on command. This is not preparation for the work, it is the work. Once you have a deterministic failing test, the agent's search has a terminal condition it can check by itself, and its hit rate goes up enormously. Before that, you're asking it to guess.

2. Make it explain before it edits. Ask for the causal chain with file:line citations. Which line writes the bad value, which line reads it, what's between them. If it can't cite, it's guessing, and you just saved yourself a patch. One extra turn, kills most bad fixes.

3. Cap the diff. "Fix this in 10 lines or tell me you can't." Big diffs are where unfixed bugs hide. The constraint also forces a real diagnosis, because you can't shotgun a whole module in 10 lines.

4. Delete the fix and re-derive it. Once it's green, revert the patch and confirm the test goes red again. Embarrassing but true: a solid share of "fixes" I've accepted were placebo. The real change was a restart, a cleared cache, or an unrelated edit made three turns earlier in the same session.

5. Feed it observations, not adjectives. Don't say "it's flaky." Have the agent add instrumentation, run it yourself, and paste the actual values back in. It is an inference engine running on a bad prior until you hand it data. Accuracy is a function of what's in the context window, not how sternly you phrase the request.

What finally fixed the duplicate emails?

A unique constraint on (order_id, template) and an insert that catches the conflict. Three lines. The dedupe belonged in the database, which is the only thing in that system both instances agreed on.

The agent could have gotten there in one turn if I'd given it the right input. Instead of "emails are duplicating, here's the handler," the prompt that would have worked was: "this dedupe uses process-local memory; we run 2+ instances." That's not a prompting trick. That's me having done the diagnosis. The model was never going to discover the deploy topology by staring at the file.

Is vibe coding still worth it?

Yes, and I'd argue it gets more worth it once you stop vibe debugging. Let the agent generate freely in places where reality checks the work fast: UI, scaffolding, glue, one-off scripts, anything with a visible output. Then move your attention from reading diffs to owning the feedback loop. Tests, repros, instrumentation, and a clear picture of what your system does at runtime.

The skill that appreciates here isn't prompt writing. It's being the person who can make a bug happen on demand. That's the one thing an agent still can't do for you, and it's the input that makes everything else it does actually work.

So: is vibe coding dangerous? Vibe coding is fine. It fails cheaply and visibly, because running the code tells you immediately whether it worked. Vibe debugging is what kills you, because an AI agent can't observe your runtime state and will answer a bug report with the most statistically plausible fix instead of the correct one. That fix often passes your tests, changes behavior, and hides the real cause under a lock or a null guard. Fix the loop, not the prompt: build a deterministic repro first, demand a cited causal chain before any edit, cap the diff size, revert-and-re-verify every fix, and start a fresh session after three failed attempts.

Why Strong Engineers Fail Coding Interviews: A Scorecard Autopsy

jidonglab — Wed, 26 Aug 2026 03:43:04 +0000

The strongest candidate I ever voted no on solved the problem in eleven minutes. Clean. Optimal. Caught the edge case I normally have to hint at twice.

Then I opened my notes to write the scorecard and found one line: "Solved it. I have no idea how."

That is the short version of why strong engineers fail coding interviews. Not because they can't code. Because nothing they did survived the trip from the room to the scorecard. The interview is not the thing being graded. The document I write forty minutes later is the thing being graded, and you are not in the room when it gets read.

TL;DR

Strong engineers fail coding interviews mostly on signal density, not correctness. A silent correct answer scores lower than a narrated near-miss.
Interviewers score 3-4 rubric axes (problem solving, coding, communication, and for senior roles, judgment) and each axis needs quotable evidence, not vibes.
The decision happens in the debrief, where ambiguity defaults to no. "Lean hire" across the board is a rejection at most companies.
The most common senior failure is solving a senior problem like a junior: no scoping, no tradeoffs, no failure modes, no tests.
Fix it by talking in sentences your interviewer can transcribe verbatim: assumption, tradeoff, complexity, test.

What do interviewers actually score in a coding interview?

Not "did you get the answer." Almost every structured loop I've been part of scores a fixed rubric, and correctness is one box inside one axis. Here is roughly what the form looks like:

Axis	What it's really asking	What lands on the scorecard
Problem solving	Did you scope before you built?	"Asked whether input fits in memory before choosing an approach."
Coding	Would this survive code review?	"Named things well, extracted a helper, no off-by-one."
Communication	Could I follow you in real time?	"Told me the plan first, then coded the plan."
Judgment (senior+)	Do you know what breaks in prod?	"Unprompted, called out the retry storm risk."

Notice what every right-hand cell has in common: it's a sentence I could only write because you said something out loud. That's signal density. Your score is capped by how much evidence you handed me.

Here is the brutal asymmetry. If you go quiet and produce a perfect solution, I have four boxes and one piece of evidence. If you narrate your way to a solution that's 85% done when time runs out, I have four boxes and twelve pieces of evidence. The second person gets the offer. I have watched it happen more times than I can count.

Why do strong engineers fail coding interviews?

Because the habits that make you effective at work are invisible under a 45-minute clock. The six failure modes I see most:

1. Silent solving. You think in your head, which is exactly what a great engineer does at their desk. In an interview it reads as a black box. I can't score a black box, so I score the one line I could observe.

2. Coding before scoping. Senior engineers at real jobs have context. In an interview you have none, and you skip the part where you get it. Twenty seconds of "how big is n, can I mutate the input, are duplicates possible" is worth more rubric points than the entire back half of your solution.

3. Solving a senior problem like a junior. This kills more experienced candidates than anything. You are given a deliberately vague prompt, you pick the first reasonable approach, you implement it flawlessly, and you never say why that one. No tradeoff, no alternative rejected, no "this breaks if the file doesn't fit in memory." Flawless execution with zero judgment reads as mid-level. That's a downlevel or a no.

4. Treating a hint as an insult. Hints are not charity, they're a probe. I'm testing collaboration. Candidates who tense up, argue, or ignore the hint and keep typing generate the single worst scorecard sentence there is: "Hard to redirect." That one line has ended more loops than any wrong answer.

5. Gold-plating. You spend fourteen minutes on a generic class hierarchy for a problem that wanted a dictionary and a loop. Now you're out of time with nothing running. "Over-engineered the setup" is not a compliment on a rubric.

6. No tests, ever. Not "write a test framework." Just walk one concrete input through your code out loud. Empty list, single element, duplicate. Candidates who do this catch their own bugs, and catching your own bug in front of me is a higher score than never having one. I promise. Self-correction is the trait I'm actually hunting for.

What actually happens in a hiring debrief?

Four to six interviewers sit down, each with a written scorecard and a vote on a scale like strong no / no / lean no / lean hire / hire / strong hire. Then three mechanics take over, and none of them are about you.

Ambiguity defaults to no. If nobody in the room is willing to fight for you, you are not hired. "Lean hire" from everyone is a rejection, because a room full of shrugs means nobody's reputation is attached to you working out.

Evidence beats seniority. The interviewer who wrote 400 words of direct quotes wins the argument against the staff engineer who wrote "seemed fine, kind of quiet." Vibes lose to transcripts. This is your leverage: you are writing your own advocate's script during the interview.

Anchoring is real. Whoever speaks first sets the frame, and everyone else's language drifts toward it. A well-run debrief collects written scores before anyone talks, precisely to blunt this. Plenty of debriefs are not well run.

How do you get a good scorecard written about you?

Speak in transcribable sentences. Interviewers are typing while you talk, and the sentences that make it into the doc are short, declarative, and structured. Feed them.

Four templates that turn into rubric points almost automatically:

Assumption: "I'm assuming the input fits in memory. If it doesn't, I'd want to stream it. Which do you want?"
Tradeoff: "Hash map here costs O(n) memory and buys O(1) lookups. This is read-heavy, so I'll take that trade."
Complexity, unprompted: "This is O(n log n) because of the sort. The rest is linear."
Test: "Let me run [] through it. Line 3 would index into an empty list, so I need a guard."

Also: narrate your dead ends instead of hiding them. "I considered two pointers, but the array isn't sorted and sorting costs more than it saves" is pure judgment signal. Deleting that thought silently earns you nothing.

And when you get stuck, say the actual shape of the stuck: "I know I need the previous index but I'm not tracking it yet." That's a debuggable statement. "Um, sorry, one sec" is not.

Do AI interviews change any of this?

Less than you'd think. AI screeners and auto-graded async rounds still score against a rubric, and the newer ones weight your explanation heavily precisely because the code alone stopped being a reliable signal. The scarce thing shifted from producing an answer to demonstrating you understand the answer.

If anything, narration matters more now. When any candidate can produce plausible code, the tiebreaker is whether you can defend it under follow-up questions. Onsite loops have already adapted: expect more "why did you choose that" and more "now change this requirement and tell me what breaks."

So why do strong engineers fail coding interviews?

Strong engineers fail coding interviews because interviews score observable evidence, not ability, and the decision is made in a debrief from a written scorecard the candidate never sees. A silent correct solution produces one line of evidence across a four-axis rubric, while a narrated partial solution produces a dozen, so the narrator wins. The fix isn't more LeetCode. It's making your reasoning audible: state your assumptions before coding, name the tradeoff behind every data structure you pick, call out complexity unprompted, walk one edge case through your code out loud, and treat hints as collaboration rather than criticism. Interview like the transcript is the deliverable, because it is.

What's the most unfair rejection you've gotten? I'd bet the scorecard said something you'd never guess.

Why AI Coding Agents Never Delete Code (I Measured It)

jidonglab — Tue, 25 Aug 2026 15:41:07 +0000

My agent opened a PR: +214 / -3. The task was "remove the legacy retry path."

It did not remove the legacy retry path. It wrote a new retry path, put the old one behind a flag that defaults to on, added a compatibility shim between them, and deleted three lines of comments. Tests passed. The code was, honestly, fine code. There was just more of it than before, and I had asked for less.

That is the pattern I keep hitting, across models and across harnesses: AI coding agents never delete code. They add. Then they add a wrapper around what they added. Then, when you say "simplify this," they add a simplification layer.

I got curious and ran git log --numstat over six months of my main side project, splitting commits by author. My hand-written commits sat around 3 lines added per line removed. The agent-authored ones were north of 10:1. Same repo, same feature work, same me reviewing. The difference was not skill. It was direction: the agent only knows how to go one way.

TL;DR

AI coding agents never delete code because deletion is a whole-repo claim ("nothing else uses this") and the agent only sees the handful of files it happened to open.
Adding is locally verifiable: write code, run tests, green. Deleting is globally verifiable: prove no caller anywhere breaks. Agents optimize for the thing they can check.
The bloat has a signature: parallel helpers, old paths kept behind flags, defensive branches for impossible states, and commented-out tombstones.
Fix it by making deletion the task, not a side effect. Separate subtraction PRs, hand the agent the grep evidence, and require a negative net diff.
If you want it to stick, ratchet it in CI. A failing check works; a line in CLAUDE.md does not.

Why do AI coding agents never delete code?

Because deleting code is a claim about the entire repository, and the agent is working from a keyhole view of it.

Three things stack up:

1. Verification is asymmetric. When an agent adds a function, the proof it worked is right there: the test it just wrote goes green. When it removes a function, the proof is the absence of breakage everywhere, including the template file it never opened, the dynamic import, the string-keyed dispatch table, the cron job. The agent cannot run that proof. So it hedges.

2. Context is partial by construction. Your agent read 12 files. Your repo has 900. It has no idea that formatCurrency already exists in lib/format.ts, because it grepped utils/ and found nothing. Adding is safe under uncertainty. Deleting under uncertainty is how you page someone at 2am.

3. Doing more looks like helping. I can't see anyone's training data, but the behavior is consistent with a strong prior that a thorough answer is a long answer. Ask a model to clean up a file and reply with a 40-line deletion and nothing else, and it reads as lazy. Reply with a refactor, a docstring, a type guard, and a comment explaining the deletion you didn't do, and it reads as diligent. One of those is right and it is not the one that gets rewarded.

Put those together and you get an agent that treats your codebase like a whiteboard nobody is allowed to erase.

What does agent-written bloat actually look like?

It has a fingerprint. Once you can see it, you cannot unsee it in a review.

The parallel helper. A second slugify, a second retryWithBackoff, a second date formatter, each slightly different in the edge cases. Nothing is broken. Six months later a bug fix lands in one of the three and not the others.

The flag that defaults to old. "I've kept the previous implementation behind USE_LEGACY_SYNC for safety." Now you own two code paths forever, and one of them is untested in production and untested in CI because nobody flips it.

Defensive code for states that cannot happen. if (!user) return null inside a function that only runs behind auth middleware. Individually harmless. A hundred of them and your codebase turns to fog, because now every null check is ambiguous: is this a real case or agent lint? Worse, those branches swallow the bugs that would have told you something upstream is broken.

Comment tombstones. // Previously handled by processQueue(), see PR #412. Commented-out blocks "kept for reference." Your VCS already does this. The agent does not trust that you have one.

Isn't this just bad prompting?

Partly, yes. Prompting moves the needle and you should do it. "Delete the legacy retry path. Do not write a replacement. The new path in sync/retry.ts already covers every caller" gets you dramatically closer than "clean up the retry logic."

But it decays. Put "prefer minimal diffs" in your system prompt and watch it hold for the first two tool calls of a long agent loop, then evaporate somewhere around step 15 when the model is deep in its own context and optimizing for closing the loop. Instructions are weak forces. Structure is a strong one.

The honest version: prompting fixes maybe a third of this. The rest is on you for asking an agent with a keyhole view to make a whole-repo decision without giving it the whole-repo evidence.

How do you make an AI coding agent actually delete code?

Give it what it is missing: evidence, permission, and a hard constraint.

1. Split the subtraction pass from the addition pass. Never in the same PR. Feature PR ships the new path. A separate PR, a separate agent run, does one thing: remove the old one. When deletion is the only success criterion, the model stops hedging, because hedging is now visibly failing the task.

2. Hand it the call-graph evidence up front. Do not ask "is this used?" Run the search yourself and paste the result:

rg -n --stats 'legacyRetry|LEGACY_RETRY' -g '!node_modules' -g '!dist'
rg -n 'legacyRetry' -g '*.{html,ejs,yaml,yml,json,sql}'   # the ones it will forget

Dynamic dispatch, config files, and templates are exactly where an agent's confidence outruns its knowledge. I have had an agent declare a function dead because it only searched .ts files. It was called from a Handlebars template. Provide the evidence, or verify the claim yourself before you accept it.

3. Make the constraint numeric. "The net line count of this diff must be negative" is a constraint a model can actually evaluate against its own output. "Be concise" is not.

4. Tests are the oracle, and the agent must run them. Not "this should be safe to remove." Run the suite, paste the output. Claimed-green and actually-green are different states of the world.

5. Ratchet it in CI. This is the part that survives contact with a busy week.

# .github/workflows/diet.sh
ADD=$(git diff --numstat origin/main... | awk '{a+=$1} END {print a+0}')
DEL=$(git diff --numstat origin/main... | awk '{d+=$2} END {print d+0}')
if [ "$ADD" -gt $(( DEL * 4 + 200 )) ]; then
  echo "Diff is $ADD added / $DEL removed. Justify with the 'growth' label or cut it."
  exit 1
fi

Crude on purpose. It is not measuring quality, it is creating friction in the one direction that currently has none. Add a label escape hatch for genuinely new features and let it annoy you the rest of the time.

What does it cost you to ignore this?

It compounds, and it compounds against the agent itself.

Next week that agent reads the codebase it wrote. Three near-identical helpers means retrieval pulls the wrong one. Two flag-gated paths means it has to reason about both. Fog-level null checks mean it cannot tell which invariants are real. You pay in tokens, in latency, and in the agent making worse decisions because its own past output is the noise in its context.

The trap is that none of this shows up as a bug. It shows up as your agent getting slowly, unaccountably dumber in your repo while it stays sharp in a fresh one. That is not model drift. That is the landfill you let it build.

So, why do AI coding agents never delete code?

Because adding code is a local action an agent can verify on the spot, while deleting code is a global claim it has no way to prove from inside a partial view of your repository, and under uncertainty the safe move is always to add. The behavior is rational given what the agent can see. The fix is not a better model or a sterner prompt, it is structure: run deletion as its own task with its own PR, hand the agent the search results that prove a symbol is unused, require a negative net diff, make it run the tests instead of predicting them, and put a lines-added ratchet in CI so growth costs something. Do that and agents delete code just fine. They were never refusing. They were just never asked in a way they could answer.

Any Questions for Us? 7 Questions to Ask the Interviewer

jidonglab — Tue, 25 Aug 2026 03:38:54 +0000

The strongest candidate I ever debriefed lost the hire decision in the last four minutes.

He shredded the system design round. Then the hiring manager said "any questions for us?" and he said, "No, I think you covered everything." One line went into the scorecard: no questions, can't tell if he wants this job or a job. We hired someone with visibly rougher code who had asked us what our rollback story looked like, and then asked a follow-up when our answer got mushy.

That was the moment I understood that the questions to ask the interviewer are not a politeness ritual. They are a scored round with no rubric published, and they are also the only round where you get to interview them.

TL;DR

The "any questions?" segment is scored in most structured loops, usually as evidence of role-specific interest and seniority, not as trivia.
Bad questions ask for things the careers page already answers ("what's the culture like?"). Good questions ask for a specific past event the interviewer had to live through.
The single most useful trick: ask the same question to three different interviewers and compare the variance. The disagreement is the signal, not the answer.
Match the question to the role: recruiter gets process, peer gets daily reality, hiring manager gets decision rights, skip-level gets roadmap risk.
If an answer is vague, ask one calm follow-up. How they handle being probed tells you more than the answer itself.

Why do interviewers score the questions you ask?

Because it is the cheapest way to separate people who want this team from people who want an offer. Most scorecards have a "motivation" or "role fit" line, and for a lot of interviewers, your questions are the only evidence that line ever gets.

There is a second reason, and it is more brutal: your questions leak your seniority. A junior asks about the tech stack. A mid-level asks about the process. A senior asks about ownership, blast radius, and who decides what. Nobody plans it that way, but every interviewer I know pattern-matches on it instantly.

What questions should you ask the interviewer?

Ask about events, not adjectives. Adjectives ("collaborative", "fast-paced") are free to say and impossible to check. Events have to be remembered, and people are bad at inventing them under time pressure. Here are seven that consistently pull real information out of a room, roughly in order of how much they've paid off for me.

1. "What happened to the last person in this role?"
Backfill or growth? Promoted internally or quietly gone? If it's a backfill and nobody wants to say where the person went, you've learned something the offer letter will not tell you. Ask it flat, without apology. It's a normal question.

2. "Tell me about the last incident that woke someone up at 3am. What changed afterward?"
This one is my favorite because it's a three-for-one. You learn the on-call reality, whether postmortems produce actual changes, and whether the person can talk about failure without getting defensive. Teams with a healthy blameless culture answer this in cheerful detail. Teams without one get weirdly clipped.

3. "How does code get from my laptop to production, and how long does that take today?"
Note the word today. Everyone has a beautiful CI diagram. Ask how long it took last Tuesday. "About twenty minutes, we deploy a few times a day" and "well, we batch into a release train and QA signs off" are two completely different jobs.

4. "How much of a sprint goes to work nobody outside engineering asked for?"
That's tech debt, tooling, migrations, and reliability. If the honest answer is zero, you're joining a feature factory and your first year will be spent apologizing for a codebase you didn't write. A real number, even a small one, means someone is protecting engineering time.

5. "Who decides what I work on next quarter, and what happens when I disagree?"
Decision rights are the actual org chart. The first half is easy for them to answer. The second half is where you find out whether engineers here are partners or ticket-takers. Listen for a concrete instance of an engineer changing a plan.

6. "What's the most recent thing this team shipped that didn't work, and how did you find out?"
"How did you find out" is the payload. If the answer involves dashboards, an experiment, or a customer metric, they measure. If nobody remembers anything failing, either nothing ships or nobody checks, and both are bad.

7. "If I'm doing great six months in, what does that look like specifically? And who besides you has to agree?"
The second sentence separates managers who can actually promote you from managers who can only advocate. Ask it politely and you will occasionally watch someone realize, live, that they don't know.

What questions kill your rating?

Questions whose answers are on the careers page, and questions that can't be wrong. "What's the culture like?" cannot be answered badly. "Do you like working here?" is answered by a person whose employer is in the room. "What's a typical day?" gets you a job-description recital.

Also skip these until an offer exists: total comp, vacation policy, remote flexibility beyond what's already posted. Not because wanting them is shameful, but because the recruiter is the right person and post-offer is the right time, when your leverage is at maximum. Asking a peer interviewer about PTO burns your one high-signal slot on information you'll get for free later.

And one warning about the clever ones. "What's your biggest weakness as a manager?" reads as a gotcha you got from a listicle. If you want that information, ask "what's something this team is bad at that you're actively trying to fix?" Same information, no theater.

What's the trick that actually works?

Ask the same question in every round, then compare the answers. This is the highest-leverage thing in this entire post.

Pick one: "what's the biggest risk to this team hitting its goals this year?" Ask the recruiter, the peer, the manager, and the skip-level. Write down all four answers.

If they're broadly consistent, that team has a shared understanding of reality. That is rarer than you think, and it correlates with almost everything good.

If the manager says "hiring" and the peer says "the legacy billing service" and the skip-level says "we're waiting on a decision from another org," you've just mapped the dysfunction in ten minutes of small talk. The variance is the finding. I've turned down offers on variance alone and never regretted it.

How do you ask a hard question without sounding hostile?

Frame it as your own planning, not as an audit. "I want to understand what my first quarter would actually look like, so: how does code get to production today?" is the same question as an interrogation, minus the interrogation.

Then, if the answer is vague, ask exactly one follow-up. One. "Got it. What did that look like the last time it came up?" That's it. Two follow-ups on a sore spot and you're the person who cross-examined the team.

Watching how someone handles that single follow-up is worth more than their answer. Somebody who says "honestly, that's a mess right now and here's what we're doing about it" has just given you a better reason to take the job than any perk in the packet.

So what are the best questions to ask the interviewer?

The best questions to ask the interviewer are specific, event-based, and answerable only by someone who works there: what happened to the last person in this role, what the last incident changed, how long a deploy takes today, how much time goes to unglamorous engineering work, who decides your roadmap, what recently failed and how they found out, and what success at six months concretely looks like. Ask two or three of them per round, ask one of them in every round so you can compare answers across interviewers, and treat inconsistency between those answers as your most reliable signal. You will get a better read on the team than any Glassdoor page, and you will score higher on a rubric line most candidates never knew existed.

What's the one question you always ask? I'm collecting good ones.

Using AI in a Live Coding Interview: How Interviewers Know

jidonglab — Mon, 24 Aug 2026 03:37:06 +0000

The candidate was good. Too evenly good.

Every answer started with the same pause. Not a thinking pause, where someone squints and says "hmm, okay, so the naive thing would be..." A flat pause. Four seconds of nothing, then a paragraph of clean, confident, perfectly structured explanation. Then another flat pause. Then more fluency. Using AI in a live coding interview does not usually get caught by a dramatic gotcha moment. It gets caught by rhythm.

I've sat on both sides of this table. Here's what actually shows up from the interviewer's chair, and the part almost nobody tells you: getting spotted is not the thing that sinks you.

TL;DR

Interviewers rarely detect AI use through tooling. They detect it through rhythm: even-width pauses, code that arrives already refactored, and answers that are more polished than the person speaking them.
The killer test is not a question about your code, it's a change to the problem. Owners edit their solution. Transcribers restart from zero.
You don't get "caught" and confronted. You get quietly scored low on "can defend own work" and rejected with no feedback, which is why nobody learns this lesson.
In interviews that explicitly allow AI, the bar moves to how you use it: showing your prompt, distrusting the output, and rejecting part of it out loud is the highest-scoring move available.
Take-home tests aren't dead. They mutated into "here's the code, defend it" and "here's a broken PR, review it."

How do interviewers know you're using AI in a live coding interview?

They almost never know. They notice a cluster of small mismatches and quietly stop trusting the signal. Here are the seven that come up most.

1. The latency signature. Human thinking is uneven. People stall on the hard part and speed up on the easy part, backtrack, say "wait, no." AI-assisted answers invert that: silence, then fluency, at a constant interval. It's not the pause that's suspicious. It's that every pause is the same width.

2. Code that arrives already refactored. Real people write for i in range(len(arr)) and clean it up later, if ever. Under time pressure, in minute three of a thirty-minute problem, nobody writes a helper function with a perfect name, a typed signature, and a docstring. That's not skill. That's a different process.

3. It solves the general problem, not mine. Good interviewers take a known problem and bend one constraint: the array is nearly sorted, the API is rate-limited, you can't hold the input in memory. A pasted answer solves the textbook version and sails straight past the bend. That's the loudest tell of all, and it's completely deniable, which makes it useless as an accusation and perfect as a filter.

4. Edge cases handled, but not noticed. The code guards against empty input. Then I ask "what happens on empty input?" and they scroll up to find their own guard. Reading your own code for the first time in front of me is a distinct facial expression.

5. Typing cadence. Not paste detection, just rhythm. People type in bursts around the hard parts and slow down on variable names. Transcribing looks different: steady, even keystrokes with small stalls at line breaks, where the eyes jump back to the source.

6. Register mismatch. They speak like a person ("yeah so I'd just kinda loop through the thing") and their comments read like documentation ("Time complexity: O(n log n), dominated by the sort"). Two different writers sharing one window.

7. The follow-up ends it. This is the cheap, reliable one, and it needs no suspicion to work. Don't ask about the code. Change the world around it. "Now the input arrives as a stream and doesn't fit in memory." Someone who owns their solution edits it, muttering, deleting a line, keeping the shape. Someone who received it starts over from an empty screen.

Does using AI in an interview automatically disqualify you?

Not the way people imagine. Almost no interviewer runs a tribunal. There's no confrontation, no "were you using ChatGPT," no dramatic reveal, because accusing someone with no proof is a terrible idea and everyone knows it.

What actually happens is worse for you: the interviewer writes something like "clean solution, couldn't extend it, unclear ownership" and moves the rating down. You get a polite rejection with no reason attached. So you never find out which part broke, and you run the same play at the next company.

The disqualifier was never the tool. It was the eight seconds where you couldn't explain the thing on your own screen.

What should you do if the interview allows AI?

Use it visibly and disagree with it in front of them. Plenty of teams now run "bring your assistant" interviews, and the scorecard shifts from did you produce code to are you dangerous with a code generator. That's a real, gradeable skill and most candidates score badly on it because they go quiet the moment the tool takes over.

Concretely, the moves that read well:

Say what you're going to ask for before you ask for it. "I'll have it stub out the retry wrapper, I want to write the backoff math myself."
Read the output out loud and reject a piece of it. "This catches the generic exception, which will swallow the timeout I care about. I'm rewriting that block." One honest rejection is worth more than ten correct lines.
Name what you'd verify. "I don't trust this off-by-one. Give me a second, I'll run it on an empty list and a single element."
Never let generated code enter your file unread. The moment you ship a line you can't explain, you've handed the interviewer tell number four.

The uncomfortable truth is that all of this is a performance skill, and performance skills only survive contact under pressure if you've rehearsed them. Talking through your reasoning while someone silently watches is genuinely hard the first few times, and your first rep should not be the job you want. I got interested enough in this problem to build a tool for it: Preterview runs voice mock interviews with three different interviewer personalities, scores your resume and portfolio, and hands back a written report on where you drifted, rambled, or went silent. Full disclosure, I built it, so take the recommendation with the appropriate salt: preterview.com/en.

Are take-home tests dead now?

No, but the honest version of them changed shape. If a take-home can be finished by any assistant in four minutes, its grading value is zero, and hiring teams figured that out fast.

What replaced them:

Defense sessions. You submit the take-home, then spend twenty minutes explaining decisions and making one live change to it. Same take-home, entirely different filter.
Code review as the test. They hand you a PR, often deliberately generated, and score what you catch. This one is brutal and I love it, because "spot the plausible wrong thing" is now most of the job.
Debugging over building. A repo, a failing test, a clock. Assistants help here, which is the point: they want to see you drive.

The pattern behind all three: writing code was never the scarce skill they were measuring. It was just the cheapest proxy for it. AI didn't kill the proxy, it exposed it.

What do interviewers actually score?

Rarely the optimal solution. Read enough hiring debriefs and the reasons for a yes cluster into three things: would I want this person next to me at 6pm on a Friday during an incident, do they notice when they're wrong before I tell them, and do they make problems smaller instead of bigger.

Nobody has ever written "reached the O(n) solution" as the reason a hire went through. They write "explained the tradeoff clearly," "caught their own bug," "asked the right question about scale." Which is exactly why an unexplainable perfect answer scores lower than a defended imperfect one. The perfect answer contains no information about you.

So, can interviewers really tell?

Interviewers can rarely prove you're using AI in a live coding interview, and most won't try. What they detect is the gap between the quality of your code and the quality of your explanation of it: uniform pauses, solutions that arrive pre-refactored, answers that solve the textbook problem instead of the constrained one you were given, and the collapse that happens when they change a requirement and ask you to adapt. That gap is what gets scored, not the tool. Use whatever help the interview allows, but never put a line on the screen you can't defend, extend, and argue with out loud, because defending it is the actual test and always was.

Gradient Accumulation Loss Bug: Why accum=8 Isn't Batch Size 8

jidonglab — Sun, 23 Aug 2026 15:34:54 +0000

You drop per_device_train_batch_size from 8 to 1 and set gradient_accumulation_steps=8 to fit the model on one GPU. Same math, less memory — that is the whole selling point. Then the fine-tune comes out measurably worse than the large-batch run, and the loss curve sits visibly above it from step one.

That gap is the gradient accumulation loss bug: the standard accumulation loop computes a mean of per-micro-batch means, not the mean over all tokens in the optimizer step. When your micro-batches contain different numbers of supervised tokens — which they always do in instruction tuning — the two are not equal, and short examples end up dominating the gradient.

TL;DR

loss.backward() per micro-batch with reduction="mean", then dividing by G, gives each micro-batch weight 1/G instead of giving each token weight 1/N. That is only correct when every micro-batch has the same unmasked-token count.
Each token in micro-batch g gets its gradient scaled by n̄ / n_g (mean token count over that micro-batch's count). A 20-token answer next to an 800-token answer gets ~40x the per-token pull it deserves.
DDP repeats the same error one level up: all_reduce averages gradients across ranks, so uneven token counts between GPUs re-skew the batch.
Fix: compute num_items_in_batch = total unmasked label tokens across all micro-batches and all ranks, use reduction="sum", and divide by that one number (times world_size to undo DDP's mean).
Transformers ships the fix since v4.46 via num_items_in_batch plus average_tokens_across_devices. A custom compute_loss override that omits the kwarg silently reinstates the bug.

What is the gradient accumulation loss bug?

It is the mismatch between mean-of-means and the true token mean. The objective you think you are minimizing over an optimizer step is

L = (1 / N) * Σ_g Σ_t  ℓ(g, t)        N = Σ_g n_g

where n_g is the number of non-masked label tokens in micro-batch g and ℓ(g,t) is the per-token cross-entropy. The objective the naive loop actually minimizes is

L̂ = (1 / G) * Σ_g [ (1 / n_g) * Σ_t ℓ(g, t) ]

Pull out the weight on a single token in micro-batch g:

true weight: 1 / N
naive weight: 1 / (G · n_g)

Their ratio is N / (G · n_g) = n̄ / n_g. The gradient contribution of every token is inversely proportional to how many other supervised tokens happened to share its micro-batch. That is a data-ordering artifact, not a property of your loss.

How much gradient weight does a short example actually steal?

Concretely. Two micro-batches, G = 2, completion-only masking, n = [20, 800]. Then N = 820, n̄ = 410.

Tokens in the 20-token answer: scaled by 410 / 20 = 20.5x
Tokens in the 800-token answer: scaled by 410 / 800 = 0.51x
Relative skew between the two examples: 40x

Instruction datasets are exactly this shape. "Yes, that's correct." sits in the same shuffle as a 900-token code explanation. With per_device_train_batch_size=1, every micro-batch is one sample, so the naive loop hands each sample equal weight regardless of length — you have silently switched from token-level to sequence-level averaging, without choosing to.

Sequence-level averaging is a defensible objective. It is just not the one your config claims, it is not what your bs=8 baseline computed, and it makes short, terse, low-information targets the loudest gradient signal in the run. In practice you see it as a model that gets curt, drops formatting on long generations, and emits EOS early.

Why does DDP make it worse?

Because DistributedDataParallel all-reduces gradients with mean, not sum. So even with gradient_accumulation_steps=1, eight ranks holding n = [12, 15, 640, 30, 22, 700, 18, 25] unmasked tokens produce a mean of eight independently normalized gradients. The two ranks that drew long samples get their tokens crushed by the same n̄ / n_g factor.

This is why the bug survives so long in multi-node setups: people validate their loss on a single GPU with accum=1, where mean-of-means is trivially correct, and never re-check after scaling out.

How do you fix the gradient accumulation loss bug?

Count tokens for the entire optimizer step first, then use reduction="sum" and normalize once. The counting has to happen before any backward(), which means materializing the micro-batches for the step up front.

import torch
import torch.distributed as dist
from torch.nn.functional import cross_entropy

def optimizer_step(model, micro_batches, optimizer, world_size):
    # 1) Count real supervised tokens across the WHOLE step.
    #    labels are shifted, so the last position has no target -> slice [..., 1:]
    num_items = sum(
        (mb["labels"][..., 1:] != -100).sum() for mb in micro_batches
    ).to(torch.float32).cuda()

    # 2) Every rank must divide by the SAME global denominator.
    if world_size > 1:
        dist.all_reduce(num_items, op=dist.ReduceOp.SUM)

    for i, mb in enumerate(micro_batches):
        logits = model(input_ids=mb["input_ids"],
                       attention_mask=mb["attention_mask"]).logits

        shift_logits = logits[..., :-1, :].contiguous()
        shift_labels = mb["labels"][..., 1:].contiguous()

        # sum, not mean: normalization is global, not per micro-batch
        loss = cross_entropy(
            shift_logits.view(-1, shift_logits.size(-1)).float(),
            shift_labels.view(-1),
            ignore_index=-100,
            reduction="sum",
        )

        # DDP averages grads over ranks -> multiply by world_size to undo it
        loss = loss * world_size / num_items

        # only sync gradients on the final micro-batch
        if i < len(micro_batches) - 1 and world_size > 1:
            with model.no_sync():
                loss.backward()
        else:
            loss.backward()

    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    optimizer.zero_grad(set_to_none=True)

Three details that bite people writing this by hand:

The shift. labels[..., 1:] — if you count (labels != -100).sum() on the unshifted tensor you over-count by one per sequence that ends on a supervised token. Small, but it makes your bs-vs-accum equivalence test fail at 1e-3 and you will spend an afternoon on it.
.float() on the logits. In bf16 the summed loss over 8k tokens loses low bits fast. Upcast before the softmax, or accept noise larger than the effect you are measuring.
world_size multiply. Skip it and your effective learning rate is 1/W of what you set. This one hides beautifully — the run still trains, just slower, and you blame the LR schedule.

If you use Hugging Face Trainer or TRL's SFTTrainer on a recent version, this is already handled: the trainer computes num_items_in_batch for the accumulation window and threads it into the model's loss function, with average_tokens_across_devices gathering the count across ranks. The trap is subclassing:

# reintroduces the bug — kwarg dropped, model falls back to per-batch mean
class MyTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        ...

# correct — accept it and pass it through
class MyTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False,
                     num_items_in_batch=None):
        outputs = model(**inputs, num_items_in_batch=num_items_in_batch)
        ...

Custom model classes have the same failure: if your forward does not accept and honor num_items_in_batch, the kwarg is swallowed by **kwargs and you are back to mean-of-means with no warning.

Why do some people never see this?

Because packing hides it. If you train on packed sequences — concatenate documents to a fixed max_seq_length — then n_g is nearly constant across micro-batches, n̄ / n_g ≈ 1, and mean-of-means is approximately right. Pretraining pipelines are packed, which is why this bug is an SFT problem, not a pretraining problem.

It also disappears when every sample is a fixed-format short answer (classification-style tuning), and it is small when you compute loss over the full sequence including the prompt, since prompt length dominates and varies less than completion length. Completion-only masking, variable answer lengths, and per_device_train_batch_size=1 is the worst case — and it is the default recipe for QLoRA on a single 24 GB card.

How do you verify your trainer is correct?

One test, two minutes, and it is exact. Fix the seed, disable dropout, use fp32, take 8 samples with deliberately lopsided label counts, and compare:

# A: one batch of 8
loss_a = step_and_get_grad_norm(model, [batch_of_8])
# B: 8 micro-batches of 1, same samples, same order
loss_b = step_and_get_grad_norm(model, [b0, b1, b2, b3, b4, b5, b6, b7])

assert abs(loss_a - loss_b) / loss_a < 1e-5

Correct normalization gives agreement to float32 round-off. The buggy path will be off by percent-level or worse on skewed data — and the sign of the gap tells you which examples are being over-weighted. Run the same check with torchrun --nproc_per_node=2 against the single-GPU result to catch the DDP half.

Does this affect DPO and RL fine-tunes?

Yes, and it is more consequential there because the loss is a difference. In DPO, chosen and rejected completions rarely have the same length; a preference pair normalized per-sequence versus per-token changes what the implicit reward is measuring, which is why length-normalized DPO variants exist as a separate knob. Policy-gradient fine-tunes have the identical decision at the group-mean level: normalizing each rollout by its own length weights short rollouts more per token. The mechanism is the same n̄ / n_g factor.

The difference is that in DPO/RL the normalization is an explicit modeling choice you can defend. In SFT accumulation, nobody chose it — it fell out of a memory-saving config flag.

So why isn't accum=8 the same as batch size 8?

Because gradient accumulation reproduces a larger batch only if you normalize the loss over the total token count of the whole optimizer step. The default loop takes the mean inside each micro-batch and then averages those means, which weights each micro-batch equally instead of each token equally — so every token's gradient gets multiplied by n̄ / n_g, and short supervised targets can carry tens of times their fair share. The fix is mechanical: sum the per-token losses, divide once by the global unmasked-token count gathered across micro-batches and ranks, and multiply by world_size to undo DDP's mean all-reduce. Verify with a bs=N versus bs=1 × accum=N gradient-norm equality test in fp32; if they do not match to round-off, your accumulated run is optimizing a different objective than the one you benchmarked.

Filtered Vector Search: Why HNSW Recall Collapses at 1% Selectivity

jidonglab — Sun, 23 Aug 2026 03:32:43 +0000

You ship a RAG system. Retrieval looks great. Then a customer asks for per-tenant isolation, you add WHERE tenant_id = 'acme' to the vector query, and nothing visibly breaks — the API still returns 10 chunks, the LLM still answers, latency is fine. Six weeks later someone notices the answers got worse for small tenants.

Nothing broke loudly because filtered vector search doesn't fail with an exception. It fails by silently returning the 10 nearest neighbors that the graph traversal happened to reach, which is not the same set as the 10 nearest neighbors that match your filter. Below roughly 1-3% filter selectivity, those two sets barely overlap.

TL;DR

HNSW is a graph, and a filter deletes nodes from it. Search is greedy traversal over neighbor links; excluded nodes cut the edges the traversal needs to walk.
The cliff is a percolation threshold. With layer-0 degree M0 = 2M (default M=16 → 32), retaining a fraction p of nodes leaves mean retained degree ≈ 32p. Below p ≈ 1/32 ≈ 3% the matching subgraph shatters into disconnected components and greedy search can only see the component it started in.
Post-filtering under-returns. Fetch ef_search candidates, then filter, and you get fewer than k rows — pgvector before 0.8.0 did exactly this.
Fixes, in order of preference: partition/shard by the filter key; use a flat (brute-force) index for small filtered sets; enable iterative index scans (hnsw.iterative_scan in pgvector 0.8+); use filter-aware graphs (Qdrant's payload subgraph links, Weaviate's ACORN strategy).
Measure filtered recall@k against brute force per selectivity bucket, not global recall. Global recall@10 of 0.98 tells you nothing about the 0.5%-selectivity tenant.

Why does adding a metadata filter break HNSW recall?

Because HNSW recall depends on graph connectivity, and a filter is a node deletion on that graph.

HNSW (Hierarchical Navigable Small World) doesn't scan vectors. It builds a multi-layer proximity graph: every node links to M neighbors chosen to be a mix of near and long-range, layer 0 holds all vectors with up to M0 = 2M links, and upper layers hold exponentially thinning samples that act as an express lane. A query enters at a fixed entry point in the top layer, greedily hops to whichever neighbor is closer to the query vector, descends a layer when it hits a local minimum, and at layer 0 runs a best-first search maintaining a candidate heap of size ef_search.

The whole thing works because of the small-world property: any node is a few hops from any other. That property is a function of the edge set. When you filter, you're asking for nearest neighbors within the induced subgraph on matching nodes — and nobody built an index for that subgraph.

What selectivity is the actual cliff?

Take random (uncorrelated) filters retaining a fraction p of nodes. Each surviving node keeps each of its ≈ M0 neighbors with probability p, so mean retained degree is d̄ ≈ M0 · p. Random graph percolation (Molloy–Reed) says a giant connected component exists roughly when mean degree exceeds 1. So:

p_c ≈ 1 / M0 = 1 / (2M)

M = 16  → M0 = 32 → p_c ≈ 3.1%
M = 32  → M0 = 64 → p_c ≈ 1.6%
M = 64  → M0 = 128 → p_c ≈ 0.8%

Below p_c, the matching set is not one graph — it's confetti. Traversal reaches one fragment and reports whatever is in it as "nearest."

Two caveats make reality worse than this estimate:

Greedy search is not BFS. Even inside a connected component, best-first search with heap size ef_search terminates when no candidate improves the frontier. Getting from fragment to fragment often requires walking through non-matching nodes whose distance is worse — exactly the moves greedy search refuses to make. Degradation starts well above p_c, typically in the 5-20% band.

Real filters are correlated with the embedding space. lang = 'ja', doc_type = 'invoice', tenant_id = 'acme' — these select clusters, not random samples. That cuts both ways: the matching set is more internally connected (good), but it sits in one region of the space while the entry point sits somewhere else (bad). The search descends the hierarchy toward the globally nearest region, and if that region is filtered out, you land in a local minimum with no matching neighbors to escape through.

So you get two distinct failure regimes: shattering on uncorrelated low-selectivity filters, and unreachability on correlated ones. Different fixes.

Why does my filtered query return fewer than k rows?

That's post-filtering. The engine asks HNSW for ef_search candidates by pure vector distance, then discards the ones that fail the predicate. If 1% match, an ef_search=40 scan returns ~0 rows.

pgvector before 0.8.0 did this. The reproduction is embarrassingly simple:

-- pgvector < 0.8.0, or 0.8+ with iterative_scan = off
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

SET hnsw.ef_search = 40;  -- default

SELECT id FROM chunks
WHERE tenant_id = 'acme'          -- 0.4% of 5M rows
ORDER BY embedding <=> :q
LIMIT 10;
-- returns 0-2 rows. No error. No warning.

pgvector 0.8.0 added iterative index scans, which re-enter the index and keep scanning until LIMIT is satisfied or a budget is exhausted:

SET hnsw.iterative_scan = relaxed_order;  -- off | strict_order | relaxed_order
SET hnsw.max_scan_tuples = 20000;         -- default; raise for very low selectivity
SET hnsw.scan_mem_multiplier = 4;         -- work_mem multiple for the scan buffer
SET hnsw.ef_search = 100;

strict_order guarantees results come back in exact distance order; relaxed_order allows slight reordering and gets better recall for the same budget (re-sort in your app if order matters). max_scan_tuples is the honest knob — it's a hard ceiling on effort, and when you blow through it you get partial results again, just later. For a tenant holding 0.4% of 5M rows, 20k scanned tuples covers ~80 matching rows in expectation. That may be enough for k=10; it is not enough for a reranker that wants 200 candidates.

How do I fix filtered vector search?

In roughly this order:

1. Partition by the filter key when the key is low-cardinality and always present. Multi-tenancy is the obvious case. One index per tenant (or a tenant-aware payload index in Qdrant, a partition key in Milvus, a partitioned table with per-partition HNSW in Postgres) turns a 0.4%-selectivity filtered search into a 100%-selectivity unfiltered search. Selectivity becomes 1.0 and every problem in this post disappears. The cost is index-count overhead and worse behavior on cross-tenant queries.

2. Brute-force below a threshold. If the filtered set is a few thousand vectors, exact scan over 3k × 1024 dims is a handful of milliseconds and gives recall 1.0. Qdrant does this automatically via cardinality estimation and a full_scan_threshold; Weaviate offers a flat index and a dynamic index that switches from flat to HNSW as a collection grows. In Postgres, a partial index or simply letting the planner pick a sequential scan for a highly selective predicate achieves the same thing — check EXPLAIN ANALYZE and stop assuming the index is a win.

3. Filter-aware graph construction. Qdrant builds additional links restricted to payload-defined subgraphs (tuned with payload_m) so that filtered traversal has real edges to walk. ACORN — which Weaviate exposes as filterStrategy: acorn — takes the predicate-agnostic route: it builds a denser graph and, during traversal, expands through the two-hop neighborhood of filtered-out nodes, effectively bridging fragments at query time. Both trade index size and build time for filtered recall.

4. Raise ef_search last. It's the reflex fix and the weakest one. It widens the frontier but cannot cross a disconnected component boundary, so it buys you linear latency for sublinear recall in exactly the regime where you need it.

How do I measure filtered recall@k?

Bucket by selectivity, compare against exact brute force, and never report a single global number.

import numpy as np

def filtered_recall_at_k(index_search, embeddings, mask, queries, k=10):
    """index_search(q, k, mask) -> list[int] of row ids from the vector DB."""
    sub_ids = np.flatnonzero(mask)
    sub = embeddings[sub_ids]
    sub /= np.linalg.norm(sub, axis=1, keepdims=True)

    recalls = []
    for q in queries:
        q = q / np.linalg.norm(q)
        truth = sub_ids[np.argsort(-(sub @ q))[:k]]        # exact, filtered
        got = index_search(q, k, mask)                      # ANN, filtered
        recalls.append(len(set(truth) & set(got)) / k)
    return float(np.mean(recalls)), len(sub_ids) / len(embeddings)

for p in [0.5, 0.2, 0.05, 0.01, 0.004, 0.001]:
    mask = np.random.rand(len(embeddings)) < p
    r, actual_p = filtered_recall_at_k(search, embeddings, mask, queries)
    print(f"selectivity={actual_p:.4f}  recall@10={r:.3f}")

Run it twice: once with a random mask (models the shattering regime) and once with a mask derived from a real metadata field like tenant_id or lang (models the unreachability regime). The two curves look different, and the second one is the one your users experience.

Two things to watch for. First, len(got) < k — count truncation separately from misranking, because under-returning points at post-filtering while low overlap points at graph fragmentation. Second, log selectivity per production query so you know your actual distribution; the aggregate is usually dominated by a few large tenants while the complaints come from the long tail.

The direct answer

Filtered vector search collapses at low selectivity because HNSW retrieves by walking a proximity graph, and a metadata filter deletes nodes from that graph without repairing its edges. With the default M=16 (32 links at layer 0), retaining a fraction p of nodes leaves mean degree ≈ 32p, so around p ≈ 3% the matching subgraph drops below the percolation threshold and fragments — greedy search then returns the best vectors inside whichever fragment it landed in, which are not the true nearest matching neighbors. Post-filtering makes it visible by returning fewer than k rows; pre-filtering hides it by returning k wrong rows. Fix it by partitioning on the filter key when you can, brute-forcing small filtered sets, enabling iterative index scans (hnsw.iterative_scan in pgvector 0.8+) or filter-aware graphs (Qdrant payload links, Weaviate ACORN) when you can't — and validate with filtered recall@k bucketed by selectivity, because a global recall number will never show you the failure.

YaRN vs NTK RoPE Scaling: Why 4x Context Breaks Short Prompts

jidonglab — Sat, 22 Aug 2026 15:30:23 +0000

You bump rope_theta from 500000 to 4000000, the model suddenly survives a 128k needle-in-a-haystack test, you ship it — and two weeks later someone notices the 1.5k-token tool-calling eval lost a few points. Nobody touched the short-prompt path. RoPE scaling touched it for you.

This is the least-discussed trade in long-context work: every method that stretches a rotary model's context window pays for it in positional resolution at short distances. Position Interpolation pays a lot. NTK-aware base scaling pays less. YaRN pays least, because it explicitly refuses to touch the dimensions that carry local order. Knowing which dimensions get squeezed tells you exactly which capabilities will regress.

TL;DR

RoPE scaling (Position Interpolation, NTK-aware base scaling, YaRN) extends context by lowering rotary frequencies so out-of-range positions map back into phases the model saw in training.
Lowering frequencies compresses relative-position resolution at distance 1–16 tokens — the high-frequency dims that encode local word order. That is why short prompts regress: exact copying, argument boundaries, syntax, dedup.
Linear PI scales every frequency by 1/s and is the most destructive. NTK-aware raises the base to s^(D/(D-2)), which barely touches high frequencies but under-interpolates the lowest ones. YaRN ramps per dimension: extrapolate short wavelengths, fully interpolate long ones, blend the middle — plus an attention temperature t where sqrt(1/t) = 0.1·ln(s) + 1.
Llama 3.1's "rope_type": "llama3" config is a production-grade version of the same idea, gated on original_max_position_embeddings.
Always re-run your short-context eval suite after any RoPE change. Long-context wins are loud; short-context losses are silent.

What does RoPE scaling actually change?

RoPE encodes position by rotating query/key vectors in 2D planes. For head dimension D, dimension pair d ∈ [0, D/2) gets angular frequency:

θ_d = base^(-2d/D)

so its wavelength in tokens is λ_d = 2π · base^(2d/D). Attention scores depend only on the relative rotation (m − n)·θ_d, which is what makes RoPE relative-position-aware for free.

With base = 10000 and D = 128: λ_0 ≈ 6.3 tokens (a full rotation every ~6 tokens) and λ_63 ≈ 5.2 × 10^4 tokens (barely a fraction of a turn across the whole trained window). The low-index dims are a fine ruler for local order; the high-index dims are a coarse ruler for "roughly where in the document."

RoPE scaling changes θ_d. That is the entire mechanism, and every variant differs only in which θ_d it changes and by how much.

Why does raising rope_theta extend context at all?

Because extrapolation fails on phase, not on magnitude. During training at length L, dimension d only ever sees relative rotations in [0, L·θ_d]. Feed it position 100k when it trained to 8k and the low-frequency dims land in an angular region the model has literally never seen — attention logits go out of distribution and the output degenerates, usually into repetition or a total collapse of retrieval.

Raising base shrinks every θ_d, so position 100k now produces the same rotation angles that position 8k used to. Nothing is out of distribution anymore. That is why base grew from 10000 (Llama 2, 4k) to 500000 (Llama 3, 8k) to 1000000 (Mistral v0.3, 32k) as native windows grew.

The three classic scaling recipes, for scale factor s = L_new / L_train:

Linear PI: θ_d' = θ_d / s. Uniform. Every dimension loses resolution by s.
NTK-aware: base' = base · s^(D/(D-2)). For D = 128 the exponent is ~1.016, so the highest-frequency dim is scaled by roughly s^(2/126) — essentially untouched — while the lowest-frequency dim absorbs nearly the full s.
YaRN: a per-dimension blend of the two, plus a logit temperature correction.

Why does RoPE scaling hurt short prompts?

Because the tokens in a 1.5k-token prompt are still separated by 1–16 positions, and PI-style interpolation divides the angle those distances produce.

Concretely, under linear PI with s = 8, two adjacent tokens that used to differ by 1.0 radian in the fastest dimension now differ by 0.125 radians. The dot product between rotated queries and keys becomes nearly identical for m − n = 1 and m − n = 2. The model still knows roughly where things are; it loses the crisp local ordering signal it was trained to rely on.

What that breaks, in my experience, in rough order:

Verbatim copying from prompt to output (IDs, hashes, quoted strings).
Tool-call argument boundaries — where one JSON field ends and the next begins.
Code indentation and bracket matching in short files.
Anything relying on "the sentence immediately before this one."

There is a second, independent effect: when you stretch the position grid, the average attention logit distribution shifts and entropy rises. YaRN handles this with an attention temperature applied to the softmax, implemented for free by scaling q and k by sqrt(1/t):

sqrt(1/t) = 0.1 · ln(s) + 1

And a third: if you change base at inference on weights fine-tuned at the old base, every layer sees shifted phase simultaneously. NTK-aware scaling is tolerable zero-shot; PI and YaRN really want a short fine-tune (a few hundred steps at the target length) before they behave.

Why is YaRN better than NTK-aware base scaling?

YaRN's insight: interpolation is only necessary for dimensions whose wavelength exceeds the trained context. If a dimension completes 32+ full rotations inside the original window, the model has already seen every phase it can produce — extrapolating it costs nothing. If a dimension does not complete even one rotation in the original window, it must be interpolated fully or it goes out of distribution.

Define rotations-per-context r_d = L / λ_d, then with α = 1, β = 32:

import math

D, base, L, s = 128, 10000, 4096, 8.0
alpha, beta = 1.0, 32.0

for d in range(D // 2):
    lam = 2 * math.pi * base ** (2 * d / D)   # wavelength in tokens
    r = L / lam                               # rotations within trained context
    gamma = min(1.0, max(0.0, (beta - r) / (beta - alpha)))  # 0 = keep, 1 = interpolate
    theta = base ** (-2 * d / D)
    theta_new = (1 - gamma) * theta + gamma * (theta / s)
    if d in (0, 20, 21, 44, 45, 63):
        print(f"d={d:2d} lam={lam:10.1f} r={r:8.2f} gamma={gamma:.2f}")

For Llama-2 geometry (base=10000, L=4096, D=128) the boundaries land at λ = 128 tokens and λ = 4096 tokens, i.e.:

dims 0–20: r > 32 → γ = 0, frequencies untouched. Local order survives intact.
dims 21–44: ramped blend.
dims 45–63: r < 1 → γ = 1, fully interpolated by s. These carry global position, where losing resolution is cheap.

That is the whole difference. NTK-aware achieves something similar as a smooth side effect of the power law, but it leaves the lowest-frequency dims slightly under-interpolated, which is why pure NTK-aware tends to underperform at the very top of the extended range. YaRN's piecewise ramp is explicit about it.

What does Llama 3.1's rope_scaling config do differently?

It is YaRN's ramp expressed in wavelength ratios, and it is gated on the original window:

// config.json — Llama 3.1 style
"rope_theta": 500000.0,
"max_position_embeddings": 131072,
"rope_scaling": {
  "rope_type": "llama3",
  "factor": 8.0,
  "low_freq_factor": 1.0,
  "high_freq_factor": 4.0,
  "original_max_position_embeddings": 8192
}

original_max_position_embeddings: 8192 is the key field. Wavelengths shorter than 8192 / high_freq_factor are left alone; wavelengths longer than 8192 / low_freq_factor are divided by factor; the rest are smoothed between. Same three-zone structure, different parameterization.

If you are applying YaRN yourself at serve time, vLLM takes it as JSON:

vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
  --max-model-len 131072 \
  --rope-scaling '{"rope_type":"yarn","factor":4.0,
                   "original_max_position_embeddings":32768}' \
  --enable-chunked-prefill

Two things people get wrong here. First, factor must match what the weights were fine-tuned with — a YaRN-tuned checkpoint already has the scaling in its config, and passing your own on top double-applies it. Second, raising --max-model-len also multiplies KV cache demand per sequence, so your achievable batch size drops; the throughput hit often exceeds the quality hit.

How do you measure short-context regression from RoPE scaling?

Do not evaluate a RoPE change with a long-context benchmark alone. Needle-in-a-haystack is nearly free to pass — retrieval of a single distinctive string is the easiest long-context task there is, and it will look perfect while short-prompt reasoning quietly rots.

The minimum honest protocol:

Freeze a short-context suite (≤2k tokens): your real tool-calling traces, a code-completion set, exact-match extraction, plus raw perplexity on a held-out short corpus.
Run it on base and scaled configs with identical sampling and identical seeds. Compare deltas, not absolutes.
Add a distance-stratified probe: ask the model to copy the Nth token back for N ∈ {1, 2, 4, 8, 16, 64}. PI-style damage shows up as a clean gradient — worst at N = 1–4, recovering by N = 64. That gradient is the fingerprint of high-frequency compression, and it distinguishes RoPE damage from an unrelated regression.
Only then run long-context evals with multiple distractors and multi-hop retrieval.

If the short-context delta is unacceptable, the fix is usually not a different scaling formula — it is routing. Serve two endpoints from the same weights, one at native length and one scaled, and dispatch on input token count. The extra memory is one model copy; the alternative is paying the interpolation tax on every 800-token request you handle.

Should you scale RoPE at all?

Often, no. If your workload is 95% short prompts with an occasional long document, a scaled endpoint plus retrieval beats a globally stretched model. If you are consuming a frontier API — Claude Opus 4.5, Sonnet 4.5, GPT-5.x — you do not own rope_theta anyway; those windows come from native long-context training, not post-hoc interpolation, which is precisely why they do not exhibit the short-prompt cliff. The transferable lesson is the eval discipline: when you adopt any longer-context checkpoint, re-run the short-context suite before you assume it is a pure upgrade.

Direct answer: 4x RoPE scaling breaks short prompts because interpolation-based methods lower rotary frequencies uniformly, and the high-frequency dimensions they compress are exactly the ones encoding relative position at distances of 1–16 tokens. Linear Position Interpolation divides all frequencies by s and does the most damage; NTK-aware base scaling (base · s^(D/(D-2))) spares high frequencies but under-interpolates low ones; YaRN keeps dimensions completing 32+ rotations within the original window untouched, fully interpolates those completing fewer than one, ramps the middle, and applies an attention temperature of sqrt(1/t) = 0.1·ln(s) + 1. Use YaRN or Llama 3.1-style rope_scaling with a short fine-tune at target length, and always measure the regression with a distance-stratified short-context probe rather than a needle test.

Attention Sinks: Why Sliding-Window KV Eviction Breaks Your LLM

jidonglab — Sat, 22 Aug 2026 03:27:58 +0000

A chat server runs clean for three hours. Then, at the exact turn where the rolling KV cache first exceeds its 4096-token window and evicts the oldest block, the model stops producing English. Not degraded output — broken output. Repeated tokens, random punctuation, perplexity in the thousands. Nothing else changed: same weights, same sampler, same prompt template.

The cause is attention sinks. Your eviction policy threw away the first few tokens of the sequence, and those tokens were not carrying information — they were carrying the softmax normalizer that every other head depends on.

TL;DR

Attention sinks are the first few tokens of a sequence (usually token 0, often the BOS token) that absorb a large share of attention probability mass in most heads and layers, despite being semantically empty.
They exist because softmax must sum to 1. A head that wants to attend to nothing still has to put its mass somewhere, so training pushes it onto a token every query can see under causal masking — position 0.
Evict those tokens from a sliding-window KV cache and the mass gets redistributed onto real tokens, shifting every value vector. Perplexity explodes by orders of magnitude within a few tokens.
The fix is StreamingLLM's: keep the first 4 tokens pinned forever, roll the window over everything else, and assign RoPE positions by cache slot, not by original token index.
If you use a hosted API (Claude Opus 4.x, GPT-5.x), you never touch a KV cache — but the same logic bites you when you hand-roll conversation truncation and drop the prefix your prompt cache and your model both depend on.

What is an attention sink in a transformer?

An attention sink is a token position that consistently receives disproportionate attention weight across heads and layers while contributing almost nothing semantically. Instrument any decoder-only model — Llama, Mistral, Qwen — and print the attention distribution for a mid-depth layer. Past layer 2 or so, a large fraction of heads put most of their probability on position 0, regardless of the query. In many heads the first token takes more mass than the entire rest of the context.

This is not a bug in the checkpoint. It is a structural consequence of the architecture, and it shows up in every model trained with causal masking and standard softmax attention.

The companion finding is massive activations: the residual-stream hidden state at those sink positions contains a handful of feature dimensions with magnitudes hundreds to thousands of times larger than typical. Those dimensions are nearly constant across inputs. The model is using the sink token as a fixed bias vector — a place to park attention and a place to store a learned constant.

Why does softmax force the model to create attention sinks?

Because softmax has no "none of the above" option.

attn_weights = softmax(QK^T / sqrt(d) + causal_mask)   # rows sum to exactly 1
out = attn_weights @ V

Consider a head that has already found what it needs — say, an induction head with no matching prefix in this context. The correct output is "add nothing to the residual stream." There is no way to express that. The row must sum to 1, so something gets multiplied by its value vector and written into the residual.

The trick the model learns: designate a token whose value vector is near-zero (or whose contribution is a constant the rest of the network compensates for), give it a large key-query alignment, and dump the surplus mass there. Attending to it is a no-op.

Which token? It has to be visible to every query. Under a causal mask, exactly one position satisfies that for every sequence length: position 0. Positions 1–3 usually get recruited too, since early tokens are visible to nearly everything and carry little content.

This is also why the "softmax off by one" proposal (adding a +1 to the denominator so rows can sum to less than 1) exists, and why some recent open-weight models — OpenAI's gpt-oss family among them — ship a learned per-head sink logit appended to the attention logits. That gives every head a real null option, so it does not need to hijack a token to get one. Models trained that way are far more tolerant of window eviction, because the sink is a parameter rather than a cache entry.

What actually breaks when you evict token 0?

The moment position 0 leaves the KV cache, every head that was dumping 40–80% of its mass there has to renormalize over the remaining keys. That surplus does not vanish — softmax redistributes it proportionally onto the surviving tokens.

So a head that was effectively writing nothing now writes a large weighted average of whatever happens to be in the window. Every value vector in that layer shifts. The perturbation compounds through the remaining layers, and the residual stream leaves the distribution the LM head was trained on.

The failure is abrupt, not gradual. StreamingLLM (Xiao et al., 2023) demonstrated the clean version of this: dense sliding-window attention that evicts the oldest tokens shows stable perplexity right up to the eviction threshold, then jumps by orders of magnitude within a handful of tokens. Re-adding just four initial tokens restores stable perplexity over millions of tokens of streaming input.

Four is the number to remember. Not one — BOS alone is usually not enough, because positions 1–3 typically carry sink duty too.

How do you fix sliding-window KV eviction?

Pin the sinks, roll everything else. The subtlety is positional encoding: with RoPE, cached keys are already rotated by their original absolute position. If you evict a middle block and leave the survivors rotated at their original indices, you have punched a hole in the position sequence — the model sees positions [0,1,2,3, 5000,5001,...] with a 5000-slot gap it never saw in training.

RoPE is relative, so the correct fix is to encode by position within the cache, not position within the stream. Two ways to do it:

import torch

class SinkWindowCache:
    """Keep n_sink initial tokens forever; roll a window over the rest.
    Stores UNROTATED keys so RoPE can be applied by cache slot at read time.
    """
    def __init__(self, n_sink: int = 4, window: int = 1020):
        self.n_sink, self.window = n_sink, window
        self.k, self.v = None, None  # [B, H, T, D], keys pre-rotation

    def update(self, k_new, v_new):
        self.k = k_new if self.k is None else torch.cat([self.k, k_new], dim=2)
        self.v = v_new if self.v is None else torch.cat([self.v, v_new], dim=2)

        budget = self.n_sink + self.window
        if self.k.shape[2] > budget:
            s = self.n_sink
            self.k = torch.cat([self.k[:, :, :s], self.k[:, :, -self.window:]], dim=2)
            self.v = torch.cat([self.v[:, :, :s], self.v[:, :, -self.window:]], dim=2)

        # positions are cache slots, NOT original token indices
        pos = torch.arange(self.k.shape[2], device=self.k.device)
        return apply_rope(self.k, pos), self.v

The alternative, used by llama.cpp's context shift, is to keep keys rotated and apply a delta rotation to the survivors when you discard a block. Because RoPE rotation composes, multiplying cached keys by the rotation matrix for -Δ is exactly equivalent to having encoded them at the shifted position. That is what llama_kv_cache_seq_add does, and it is why --keep N exists on the CLI:

# keep BOS + first 4 tokens pinned, shift the rest down on overflow
./llama-server -m model.gguf -c 8192 --keep 4

What you must not do is the naive version — slice the rotated key tensor and carry on. It is two lines of code, it runs, it produces tokens, and it is wrong in a way that only shows up after the window fills.

Also note the ordering constraint: sinks must stay at the front of the cache. Some hand-rolled implementations concatenate [window, sinks] because it is easier to index. Under causal attention the relative offsets are then inverted, and you get a different flavor of the same corruption.

Why don't production inference servers hit this?

Because they refuse to evict. vLLM under memory pressure preempts a sequence and recomputes its prefix rather than dropping keys from a full-attention model. Sliding-window attention is only applied when the model config declares it — Mistral 7B's 4096-token window, Gemma's alternating local/global layers — because those models were trained with that mask and learned their sinks inside the window.

That distinction is the whole thing: window eviction is safe if and only if the model was trained under the same mask. Applying a sliding window at inference to a model trained with full attention is a train/test mismatch on the attention denominator.

Learned-eviction schemes (H2O, SnapKV, and their descendants) all rediscover this empirically. Their "heavy hitter" sets always include the first tokens, because those tokens have the highest accumulated attention score by a wide margin. If you implement a KV compression policy and don't hard-pin the first few slots, your policy will spend its budget rediscovering them anyway — or fail catastrophically the one time it doesn't.

Do attention sinks matter if I only call Claude or GPT-5?

Directly, no — you never manage a KV cache through the Anthropic or OpenAI API, and the serving stack handles this correctly. Indirectly, yes, in two places.

First, conversation truncation. When your agent loop trims history to fit a context budget, the safe move is the same shape as the sink fix: pin the prefix (system prompt, tool definitions, the first few turns), and drop from the middle or the tail. Sliding a window that eats the head of the conversation is the semantic analogue of evicting token 0, and it also invalidates every prompt-cache prefix downstream of the cut.

Second, quantization. Massive activations live at sink positions. Per-tensor activation quantization computes a scale over a tensor containing outliers hundreds of times the median, which crushes everything else to a handful of representable levels. Any calibration set that under-samples sink positions produces scales that break the model at exactly those tokens. Per-channel or per-token scales exist largely because of this.

How do you detect attention sink loss in production?

Log two things. First, at generation time, the fraction of attention mass on the first four cache slots for a mid-depth layer — run with output_attentions=True on a canary request and check it is not near zero. Second, rolling mean token logprob. Sink loss shows up as a step function in the logprob trace within 5–20 tokens of the eviction event, not a slow drift. If your degradation curve looks like a cliff, correlate its timestamp with the moment your cache crossed its budget.

The direct answer

Sliding-window KV eviction breaks your LLM because the first few tokens of a sequence are attention sinks: softmax rows must sum to 1, so heads that want to output nothing dump their surplus probability mass onto the one position every causal query can see. Evicting those tokens forces that mass onto real content, perturbing every value vector in every layer at once, and perplexity jumps by orders of magnitude the instant the window rolls past the start. Fix it by pinning the first four tokens permanently, rolling the window over the rest, and assigning RoPE positions by cache slot (or applying a delta rotation to survivors) so the model never sees a gap in its position sequence. If the model was trained with a sliding window or ships learned sink logits, this is already handled — otherwise, do not evict.

Claude Prompt Caching: Why Agent Loops Miss the 20-Block Lookback

jidonglab — Fri, 21 Aug 2026 15:25:29 +0000

Your agent starts a run with cache_read_input_tokens at 40K and climbing. Twelve tool calls later, reads drop to zero and cache_creation_input_tokens jumps to the full conversation length — on every single turn. Nothing in your prompt changed. No timestamp, no reordered tool, no model switch. The prefix is byte-identical.

You just hit the 20-block lookback window, and it is the single most expensive thing about Claude prompt caching that nobody puts in their retro.

TL;DR

A cache_control breakpoint searches backward through at most 20 content blocks to find an existing cache entry. One agentic turn with 11 parallel tool calls emits 22+ blocks and blows past that — the next request finds nothing and rewrites the whole prefix at 1.25x.
Fix it by placing rolling breakpoints every ~15 blocks, not one marker on the last block. You get 4 breakpoints per request total; spend 1 on tools+system and rotate the other 3 through the message list.
Invalidation is tiered, not all-or-nothing: tool_choice, images, and toggling thinking preserve the tools+system cache. Only tool-definition changes and model switches force a full rebuild.
Changing the system prompt mid-run nukes everything downstream — unless you append a {"role": "system", ...} message to messages[] instead (Claude Opus 5, Opus 4.8, Fable 5; not Sonnet 5).
input_tokens in the usage block is the uncached remainder only. Total prompt size is input_tokens + cache_creation + cache_read. Dashboards that graph input_tokens alone will show you a flat line while you burn cache writes.

Why does Claude prompt caching miss in the middle of an agent loop?

Because cache lookup is bounded. Prompt caching is a prefix match on exact bytes, but a breakpoint doesn't scan the entire history for a matching entry — it walks backward a limited number of content blocks. That limit is 20. If the previous request's cached block is more than 20 blocks behind your new breakpoint, the lookup fails, and the API treats your request as cold even though a perfectly valid entry exists.

Chat apps never see this. One user turn is one text block; one assistant turn is one text block. You'd need ten round trips to move 20 blocks, and you place a breakpoint on each turn anyway.

Agent loops are different. Count what a single "turn" actually appends:

Assistant message: 1 thinking block + 1 text block + N tool_use blocks
User message: N tool_result blocks

An agent that fires 8 parallel tool calls appends 8 tool_use + 8 tool_result + 2 = 18 blocks in one round trip. Two of those turns and your single trailing breakpoint is 36 blocks past the last cached point. Silent miss. No error, no warning field — just cache_read_input_tokens: 0 and a cache-creation charge for the full history.

The economics are brutal at scale. On Claude Opus 5 at $5/MTok input, a cache read is ~$0.50/MTok and a 5-minute cache write is ~$6.25/MTok. A 12x price swing per turn, triggered by a config detail you never set.

How do I place breakpoints so an agentic loop keeps hitting the cache?

Stop putting one marker on the last block. Rotate a small set of markers through the message list at a stride shorter than the lookback window. Every breakpoint is both a write point and a read point, so a trailing chain of them means each new request always finds a prior entry within 20 blocks.

The budget matters: 4 breakpoints per request, total, across tools + system + messages. Spend one on the last system block (it caches tools and system together, since render order is tools → system → messages) and rotate the remaining three.

CACHEABLE = {"text", "image", "tool_use", "tool_result", "document"}
STRIDE = 15          # < 20-block lookback, with headroom
MSG_BREAKPOINTS = 3  # 4 total minus the one on the system block

def place_breakpoints(messages):
    """Re-place cache_control so no two markers are >STRIDE blocks apart.
    Mutates plain-dict messages in place (round-trip SDK objects with
    .model_dump() first — you cannot set cache_control on a response object)."""
    flat = []
    for m in messages:
        if isinstance(m["content"], str):
            m["content"] = [{"type": "text", "text": m["content"]}]
        for b in m["content"]:
            b.pop("cache_control", None)   # clear last request's markers
            flat.append(b)

    marks, pos = [], len(flat) - 1
    while pos >= 0 and len(marks) < MSG_BREAKPOINTS:
        # thinking blocks can't carry cache_control — skip back to an eligible one
        while pos >= 0 and flat[pos].get("type") not in CACHEABLE:
            pos -= 1
        if pos < 0:
            break
        marks.append(pos)
        pos -= STRIDE

    for i in marks:
        flat[i]["cache_control"] = {"type": "ephemeral"}
    return messages

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=8192,
    system=[{"type": "text", "text": SYSTEM_PROMPT,
             "cache_control": {"type": "ephemeral"}}],   # caches tools + system
    tools=TOOLS,                                          # sorted, frozen for the run
    messages=place_breakpoints(history),
    thinking={"type": "adaptive"},
)

Two details that bite:

Clear the old markers. Breakpoints you set on turn 5 are still sitting in the history you resend on turn 6. If you don't strip them you'll exceed 4 and get a validation error, or worse, waste your budget on positions that no longer help.

Thinking blocks are not cacheable anchors. cache_control goes on text, image, tool_use, tool_result, and document blocks. Landing a stride on a thinking block and silently dropping the marker is exactly how you end up 20+ blocks apart again.

Which request changes actually invalidate the Claude prompt cache?

Not all of them, and this is where most teams over-engineer. There are three tiers — tools, system, messages — and a change only invalidates its own tier and everything after it.

Change	Tools	System	Messages
Tool definitions (add/remove/reorder)	❌	❌	❌
Model switch	❌	❌	❌
`speed`, web-search/citations toggle	✅	❌	❌
System prompt content	✅	❌	❌
`tool_choice`, images, thinking on/off	✅	✅	❌
Message content	✅	✅	❌

Read the practical consequence: you can force tool_choice on one turn, flip thinking off on the next, and keep the tools+system cache intact. Stop threading a "cache-safe mode" flag through your call sites for those.

What you cannot do casually is touch the tool array. Tools render at position 0, so adding one tool for one turn rebuilds everything. Serialize tool definitions deterministically — sort by name, sort_keys=True on any JSON schema you generate — because a dict-ordering flip in your schema builder is a full-prefix invalidation that looks like nothing in a diff.

Model switching has no escape hatch; caches are model-scoped. If you want a cheap Haiku 4.5 pass over a long context, that's a separate cache lineage, not a discount on the existing one.

How do I inject a new instruction mid-run without losing the cache?

Append it to messages[] as a system-role message instead of editing the top-level system field:

history.append({"role": "user", "content": "..."})
history.append({
    "role": "system",
    "content": "Terse mode enabled — keep responses under 40 words.",
})

Editing top-level system changes bytes ahead of the entire conversation, so every cached turn gets reprocessed at full price. A role: "system" message sits after the history and leaves the cached prefix intact.

It's supported on Claude Opus 5, Opus 4.8, Fable 5, and Mythos 5 with no beta header — not Sonnet 5, which returns a 400. Wrap it and fall back to putting the instruction in a user-turn block.

There's a security bonus that's easy to miss: this is a non-spoofable operator channel. Instructions embedded as text inside user or tool content can be forged by anything that writes into that content — a scraped page, a tool response, a file. A role: "system" message cannot.

Why do my parallel sub-agent requests all pay full price?

Because a cache entry only becomes readable once the first response begins streaming. Fire ten identical-prefix requests simultaneously and none of them can read what the other nine are still writing. You pay ten cache writes at 1.25x — worse than not caching at all.

The fix is a two-phase fan-out: send one request, await the first streamed token (not the full response), then release the remaining N−1. They read the entry the first one just wrote. On a 100K-token shared prefix across ten workers, that's the difference between ten writes and one write plus nine reads.

Same reasoning applies to sub-agents and compaction passes. If a fork rebuilds system or tools with any difference from the parent, it misses the parent's cache entirely. Copy model, system, and tools verbatim, then append fork-specific content at the end.

Why doesn't my prompt cache at all, even with the marker set?

Probably the minimum cacheable prefix — and it is not monotonic across model generations:

Model	Minimum
Claude Opus 5, Fable 5, Mythos 5	512 tokens
Opus 4.8, Sonnet 5, Sonnet 4.6, Sonnet 4.5	1024 tokens
Opus 4.7, Haiku 3.5	2048 tokens
Opus 4.6, Opus 4.5, Haiku 4.5	4096 tokens

Below the threshold, the marker is accepted and ignored: no error, cache_creation_input_tokens: 0. A 3K-token prefix caches on Opus 5 and Sonnet 4.5, and silently doesn't on Opus 4.6 or Haiku 4.5. Teams that route cheap sub-tasks to Haiku and expensive ones to Opus routinely find their Haiku path has never cached once.

The other silent killer is TTL. The default entry lives 5 minutes. If a tool call in your loop takes 6 minutes — a long build, a slow scrape, a human approval gate — the entry expires mid-turn and the next request rebuilds from scratch. Use {"type": "ephemeral", "ttl": "1h"} on those paths. The write costs 2x instead of 1.25x, so break-even moves from two requests to three, which is trivially cleared by any agent loop.

How do I actually verify this?

Log all three usage fields per request, not one:

u = resp.usage
total_prompt = u.input_tokens + u.cache_creation_input_tokens + u.cache_read_input_tokens
print(f"read={u.cache_read_input_tokens} write={u.cache_creation_input_tokens} "
      f"raw={u.input_tokens} total={total_prompt}")

The diagnostic pattern for the lookback bug is unmistakable: cache_read grows normally for the first few turns, then collapses to 0 while cache_creation equals the full history — and it flips exactly on the turn where a burst of parallel tool calls landed. Correlate the miss with the number of blocks that turn appended and you'll see the 20-block cliff in your own logs.

If you want the API to tell you directly, there's a cache diagnostics beta: client.beta.messages.* with the cache-diagnosis-2026-04-07 beta flag, passing diagnostics: {previous_message_id: <prior response id>}. It reports why a request missed instead of leaving you to diff prompt bytes by hand.

The short answer

Claude prompt caching misses inside agent loops because each cache_control breakpoint only searches backward 20 content blocks for a matching entry, and a single agentic turn with parallel tool calls easily appends more than 20 blocks — so a lone trailing breakpoint lands out of range and the API rewrites the entire prefix at 1.25x instead of reading it at 0.1x. Fix it by rotating breakpoints through the message list at a ~15-block stride within the 4-marker budget, keeping one marker on the last system block, stripping stale markers before each request, skipping thinking blocks as anchors, and switching to a 1-hour TTL wherever tool latency can exceed five minutes. Then confirm it with cache_read_input_tokens rather than the input_tokens field, which reports only the uncached remainder and will happily look healthy while your cache does nothing.

Why temperature=0 Isn't Deterministic: LLM Batch Invariance

jidonglab — Fri, 21 Aug 2026 03:23:10 +0000

Same weights. Same prompt. Same server. temperature=0. You run it twice and token 217 comes back different, and from there the two outputs have nothing in common. Nothing in your stack sampled anything. Your seed didn't matter because greedy decoding never drew a random number.

The culprit is not your code. It's that the floating-point reduction order inside your GEMM and attention kernels depends on how many other people's requests were in the batch alongside yours.

TL;DR

Why temperature=0 isn't deterministic: greedy decoding is deterministic given the logits, but the logits aren't deterministic given the server. Kernels pick reduction/split strategies based on runtime batch shape and occupancy, and floating-point addition is not associative, so the last bits move.
The perturbation is tiny (roughly 1e-6 to 1e-3 in logit units for a bf16 pipeline), but argmax is discontinuous. One near-tie flips, and after that the two trajectories are sampling from different contexts entirely.
Main sources: split-K GEMM, split-KV attention (FlashDecoding), MoE expert grouping, chunked-prefill boundaries, prefix-cache hits, and tensor-parallel all-reduce order.
Fix at the kernel level with batch-invariant kernels (fixed split counts, no atomics, fixed merge order) — real bitwise reproducibility, at a throughput cost. There is no fix through a hosted API.
If you're doing RL, this is not cosmetic: sampler and trainer disagree on logprobs for the same tokens, so your "on-policy" GRPO/PPO step is quietly off-policy at step 0.

Why temperature=0 isn't deterministic: the short version

Greedy decoding is a pure function of the logit vector. So the question is whether the forward pass is a pure function of (weights, tokens). It isn't — it's a function of (weights, tokens, batch shape, occupancy, kernel selection).

A GPU matmul doesn't sum the K dimension in one thread. It splits the work, produces partials, and combines them. How it splits depends on the shape: with M=1 (one decode row) there's no row parallelism, so the library reaches for split-K to fill the SMs; with M=256 it doesn't need to. Different split → different summation order → different rounding.

In bf16, an element carries 8 explicit mantissa bits. Accumulation is usually in fp32, so end-to-end logit agreement lands somewhere around 1e-6 to 1e-3 absolute. That is invisible in the text — until it isn't.

Worth killing a common misdiagnosis: this is usually not nondeterministic atomics in the forward pass. Most inference kernels are run-to-run deterministic for a fixed shape. That's exactly why the bug feels haunted: rerun it alone at 3am and it reproduces fine, then it drifts under production traffic, because your batch composition is a function of other users.

Where does the batch dependence actually come from?

Five places, in rough order of how often they bite:

Split-K GEMM. Decode-time M equals the number of sequences currently decoding. That number is traffic. Kernel heuristics switch on it.
Split-KV attention. FlashDecoding chunks the KV cache and merges partial attention outputs with a log-sum-exp rescale. The number of chunks is picked from sequence length and available parallelism, so a long sequence alone on the GPU gets a different split than the same sequence sharing with 31 others. The merge is a sum of rescaled partials — order matters.
Chunked prefill. Whether token 2048 is computed at the tail of a prefill chunk or in a mixed prefill+decode batch changes the kernel path it goes through.
Prefix caching. A cache hit reuses KV computed under some earlier batch shape. Cache hit and cache miss are numerically different runs of the same prompt. This one surprises people because prefix caching is supposed to be semantically transparent — it is, to within rounding.
MoE routing. Per-expert GEMMs have M = number of tokens routed to that expert in this batch. Co-tenants change your expert's tile shape.

And one config-level source that isn't load-dependent but breaks reproducibility across deployments: tensor parallelism degree. TP=2 and TP=4 build different all-reduce trees, so they sum partial hidden states in a different order. Same model, same prompt, different numbers. Pin TP if you compare across clusters.

Why does a 1e-6 logit change flip a whole answer?

Because argmax is discontinuous, and because divergence compounds.

The flip only needs the top-2 logit gap to be smaller than the perturbation. Most positions are safe — the model is confident and the gap is >1 nat. But near-ties are not uniformly distributed. They cluster exactly where you'd expect: ", " vs " ", list-marker choices, synonym pairs, the first token after a heading, whether to open a code fence now or after one more sentence.

Once one token flips, the two runs no longer share a context. Everything after is a legitimately different generation. So measure divergence-free rate over N runs and the index of first divergence, not average per-token difference — the latter saturates immediately and tells you nothing.

How do I measure this on my own stack?

Run the same request repeatedly and look at two things: whether outputs diverge, and how thin the top-2 margins are. Run the probe at temperature=1 so the reported logprob differences are interpretable in logit units (log p_i − log p_j = z_i − z_j, since softmax's shift cancels).

import asyncio, collections
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="x")
PROMPT = "Summarize the tradeoffs of split-K GEMM in three sentences."

async def probe(load=0):
    """One measured request, optionally with `load` junk requests co-scheduled."""
    async def junk(i):
        await client.chat.completions.create(
            model="m", messages=[{"role": "user", "content": f"count to 200, run {i}"}],
            max_tokens=256, temperature=0)

    task = client.chat.completions.create(
        model="m", messages=[{"role": "user", "content": PROMPT}],
        max_tokens=256, temperature=0, logprobs=True, top_logprobs=2, seed=0)
    # co-tenants change the batch shape our request is computed in
    r, *_ = await asyncio.gather(task, *[junk(i) for i in range(load)])
    toks = [c.token for c in r.choices[0].logprobs.content]
    gaps = [c.top_logprobs[0].logprob - c.top_logprobs[1].logprob
            for c in r.choices[0].logprobs.content if len(c.top_logprobs) > 1]
    return toks, gaps

async def main():
    runs = [await probe(load=l) for l in (0, 0, 8, 32, 32)]
    base = runs[0][0]
    for i, (toks, gaps) in enumerate(runs):
        first = next((k for k, (a, b) in enumerate(zip(base, toks)) if a != b), None)
        thin = sum(g < 1e-3 for g in gaps)
        print(f"run{i}: first_divergence={first}  thin_margins(<1e-3)={thin}/{len(gaps)}")

asyncio.run(main())

first_divergence=None on every row means you're batch-invariant for these shapes. What you normally see instead is None for the two idle runs and an integer for the loaded ones — that's the signature. The thin_margins count tells you how exposed the prompt is: a response with 40 sub-1e-3 margins is a coin flip waiting for traffic.

Can I actually make it deterministic?

Yes, on hardware you control, with batch-invariant kernels. The rule is that the reduction strategy must be a function of the shape you promise, not of runtime occupancy:

fixed split-K count (or none) regardless of M;
fixed-size KV chunks in attention, so the number of partials depends only on sequence length;
one block per row for RMSNorm instead of switching strategies at small batch;
no atomic accumulation anywhere in the path;
a fixed-order tree merge for partials.

This is what the batch-invariant kernel work that appeared in 2025 does, and inference stacks have started shipping toggles for it — vLLM has a batch-invariant mode in recent versions; check your version's flag rather than trusting a name from a blog post. Expect a throughput hit, concentrated at small batch and long context, because you're deliberately declining the occupancy-driven tuning that made those shapes fast.

Determinism also requires freezing everything else that selects a kernel:

# reproducibility profile — every one of these changes the numbers
vllm serve $MODEL \
  --tensor-parallel-size 4 \        # TP degree changes all-reduce order
  --dtype bfloat16 \                # not "auto"
  --max-num-seqs 64 \
  --no-enable-prefix-caching \      # cache hit != cache miss, numerically
  --no-enable-chunked-prefill       # chunk boundary changes the kernel path
# plus: pin the vLLM version, the attention backend, and the GPU generation

Turning off prefix caching and chunked prefill is a large performance sacrifice. Do it for an eval or repro rig, not for your serving fleet.

What about hosted models like Claude Opus 4.x or GPT-5.x?

You can't fix it, and you shouldn't build as if you could. temperature=0 on any hosted endpoint is best-effort, not a contract. Providers batch across tenants, route across hardware generations, and update kernels without telling you. OpenAI's seed is explicitly best-effort and pairs with a fingerprint field precisely so you can detect that the backend moved; Anthropic's API doesn't offer a seed at all. Treat model output as a distribution you sample, not a function you call.

Design accordingly:

Cache at the application layer, keyed by a hash of the exact rendered prompt. That is the only real determinism available to you.
Evaluate with n≥5 and report an interval. A single-run "regression" between two prompt versions is usually noise. Use paired sampling on the same items.
Never gate CI on exact string equality. Assert on parsed fields, schema validity, or a judge score.
Make agent tools idempotent and keyed by a request id, because a retried step can silently take a different branch.
Log tokens and top logprobs, not just the final text. Without margins you can't tell a real behavior change from a coin flip.

Why this breaks RL fine-tuning specifically

This is the case where "it's only the last bits" stops being true. Your rollout worker (paged attention, small decode batches) and your trainer (large fused prefill-style batches) compute different logprobs for the exact same token sequence. So the importance ratio π_train/π_sample is not 1 at step 0, even though the data is genuinely on-policy.

Symptoms: nonzero KL against the reference at the very first step, PPO ratio clipping firing before the policy has moved, and a gradient bias that no hyperparameter sweep will explain. Either record the behavior logprob from the sampler and use it as the denominator (with truncated importance sampling), or make the two paths bitwise identical with batch-invariant kernels so the ratio is exactly 1. Averaging the discrepancy away is not an option; it's systematic, not zero-mean.

The short answer

temperature=0 isn't deterministic because greedy decoding only removes the sampler's randomness, not the numerical variability of the forward pass. GPU kernels choose their reduction and split strategy from the runtime batch shape, floating-point addition isn't associative, so logits shift by roughly 1e-6 to 1e-3 depending on how many co-tenant requests shared your batch — and any position where the top-2 logit gap is thinner than that shift can flip, after which the two generations diverge completely. On your own hardware you can get bitwise reproducibility with batch-invariant kernels plus a pinned TP degree, dtype, backend, and version, paying throughput for it. On a hosted API you cannot, so cache prompts at the application layer, evaluate over multiple samples with intervals, and stop asserting exact strings.