Backend developer and indie maker. I build AI apps that teach you to think: Way of the Agent (learn to build AI agents as a game) and Argumentator (your debate coach).
The "favourite issue" failure is the part I'll remember from this. It reads like salience rather than missing information: five long results, and the most dramatic one wins regardless of the question. That also explains why the failures were identical in both arms on the three hard jobs: the trim removed noise, not signal.
Did you try a third arm: full text kept, but the five-field summary of every result placed first? It would separate the two effects you're seeing, whether it's the length of the results or their position in the window that pulls the model toward issue 412.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
The salience angle is the part that stuck with me too — five long results and the dramatic one wins no matter what was asked, that's not missing info. Haven't run the summary-first arm, and it's the clean split between length and position. Going to run that one, same tasks, three arms. Which would you bet on — the length of the results, or where they sit in the window?
Backend developer and indie maker. I build AI apps that teach you to think: Way of the Agent (learn to build AI agents as a game) and Argumentator (your debate coach).
My bet is position. In my agents the first long result tends to anchor everything after it, and the later ones get skimmed. So I'd expect summary-first to beat full text even at the same token count. If length turns out to matter more, that's the more interesting result. Please post the numbers when you run it.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Position's my bet too, honestly — the first long result sets the frame and everything after gets read through it. Same token count, summary-first vs full text, that's the clean split. I'll post the numbers when the run's done — if position wins, would that change how you order results in your own agents, or is it just something you live with?
Backend developer and indie maker. I build AI apps that teach you to think: Way of the Agent (learn to build AI agents as a game) and Argumentator (your debate coach).
Honestly, yes, it would 🙂 If position wins, I'd lead with a one-line summary of each result and put the long bodies after, so no single result gets to set the frame. If it's length, I'd just cap each result and move on. Really curious what your three arms show, ping me when the numbers are in!
This matches what we've seen with web extraction and search tools too: returning raw markdown/HTML or full JSON payloads makes the model anchor on whatever happens to be longest or loudest in the context window, rather than what actually answers the query. Trimming to a tight schema (or truncating bodies + returning only load-bearing fields) improves both accuracy and token cost at the same time—especially on multi-turn loops where old tool outputs compound fast.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Same shape, different tool — that's reassuring, actually. The multi-turn compounding is the bit that surprised me most, old outputs just sitting there getting heavier every turn. Did you measure the accuracy side too, or was it mostly the token bill that pushed you to trim?
Token bill was what forced the change first — once a loop hit 15–20 turns with full web/search payloads, input tokens were dominating the cost and pushing smaller models near their context ceiling. The accuracy improvement showed up right after as a side effect: fewer "lost in the middle" misses and far fewer retry loops where the agent re-called the same search because the previous result was buried under boilerplate. Haven't run a formal two-arm A/B like yours with scored pass rates, though — mostly observed it from production traces dropping in turn count once we capped returned fields.
The result that surprised me most is not the token saving but the 0.88 versus 0.92 second-look rate — the trim did not make the agent lazier about verifying, which was the objection I would have raised first. One detail I keep coming back to is that the five kept fields still include the first 300 characters of the body, so issue 412's "crash on launch" signal survives the trim in miniature. That makes me wonder whether the remaining salience effect is carried by that excerpt rather than by length itself, and whether a title-plus-labels-only arm would change the 17 of 24. If the excerpt is doing the work, the rule might be less "trim everything" and more "keep the shortest field that still lets the model rank issues". Did the trimmed arm's five remaining failures cluster on jobs where the decisive detail sat beyond those 300 characters?
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That's the sharpest read on it so far, but the failures don't actually back the excerpt theory. The three jobs the trimmed arm fails are the same three the full arm fails. The model just kept answering 412 no matter what it got, so the 300-char cutoff isn't what broke those. The title+labels-only arm is the one that settles your question though, and it's next on my list. If the excerpt is doing the work, where would you bet it lands, closer to 17 or back down near 9?
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
Your run answers the half my replay could not: mine called no model, so it measured how the window grows and nothing about the answers. One detail in your setup makes the token gap a lower bound. The rule that turns a repeated call into a one-line note was on in both arms, and before you added it the full arm ran past the window and died. So 1,421 against 673 tokens per turn is the full arm with its worst habit already trimmed. If you publish the per-job table for both runs together with the title-plus-labels arm, readers can see on which jobs the arms disagree, and with twelve jobs that is where the result lives.
We're a place where coders share, stay up-to-date and grow their careers.
The "favourite issue" failure is the part I'll remember from this. It reads like salience rather than missing information: five long results, and the most dramatic one wins regardless of the question. That also explains why the failures were identical in both arms on the three hard jobs: the trim removed noise, not signal.
Did you try a third arm: full text kept, but the five-field summary of every result placed first? It would separate the two effects you're seeing, whether it's the length of the results or their position in the window that pulls the model toward issue 412.
The salience angle is the part that stuck with me too — five long results and the dramatic one wins no matter what was asked, that's not missing info. Haven't run the summary-first arm, and it's the clean split between length and position. Going to run that one, same tasks, three arms. Which would you bet on — the length of the results, or where they sit in the window?
My bet is position. In my agents the first long result tends to anchor everything after it, and the later ones get skimmed. So I'd expect summary-first to beat full text even at the same token count. If length turns out to matter more, that's the more interesting result. Please post the numbers when you run it.
Position's my bet too, honestly — the first long result sets the frame and everything after gets read through it. Same token count, summary-first vs full text, that's the clean split. I'll post the numbers when the run's done — if position wins, would that change how you order results in your own agents, or is it just something you live with?
Honestly, yes, it would 🙂 If position wins, I'd lead with a one-line summary of each result and put the long bodies after, so no single result gets to set the frame. If it's length, I'd just cap each result and move on. Really curious what your three arms show, ping me when the numbers are in!
This matches what we've seen with web extraction and search tools too: returning raw markdown/HTML or full JSON payloads makes the model anchor on whatever happens to be longest or loudest in the context window, rather than what actually answers the query. Trimming to a tight schema (or truncating bodies + returning only load-bearing fields) improves both accuracy and token cost at the same time—especially on multi-turn loops where old tool outputs compound fast.
Same shape, different tool — that's reassuring, actually. The multi-turn compounding is the bit that surprised me most, old outputs just sitting there getting heavier every turn. Did you measure the accuracy side too, or was it mostly the token bill that pushed you to trim?
Token bill was what forced the change first — once a loop hit 15–20 turns with full web/search payloads, input tokens were dominating the cost and pushing smaller models near their context ceiling. The accuracy improvement showed up right after as a side effect: fewer "lost in the middle" misses and far fewer retry loops where the agent re-called the same search because the previous result was buried under boilerplate. Haven't run a formal two-arm A/B like yours with scored pass rates, though — mostly observed it from production traces dropping in turn count once we capped returned fields.
The result that surprised me most is not the token saving but the 0.88 versus 0.92 second-look rate — the trim did not make the agent lazier about verifying, which was the objection I would have raised first. One detail I keep coming back to is that the five kept fields still include the first 300 characters of the body, so issue 412's "crash on launch" signal survives the trim in miniature. That makes me wonder whether the remaining salience effect is carried by that excerpt rather than by length itself, and whether a title-plus-labels-only arm would change the 17 of 24. If the excerpt is doing the work, the rule might be less "trim everything" and more "keep the shortest field that still lets the model rank issues". Did the trimmed arm's five remaining failures cluster on jobs where the decisive detail sat beyond those 300 characters?
That's the sharpest read on it so far, but the failures don't actually back the excerpt theory. The three jobs the trimmed arm fails are the same three the full arm fails. The model just kept answering 412 no matter what it got, so the 300-char cutoff isn't what broke those. The title+labels-only arm is the one that settles your question though, and it's next on my list. If the excerpt is doing the work, where would you bet it lands, closer to 17 or back down near 9?
Your run answers the half my replay could not: mine called no model, so it measured how the window grows and nothing about the answers. One detail in your setup makes the token gap a lower bound. The rule that turns a repeated call into a one-line note was on in both arms, and before you added it the full arm ran past the window and died. So 1,421 against 673 tokens per turn is the full arm with its worst habit already trimmed. If you publish the per-job table for both runs together with the title-plus-labels arm, readers can see on which jobs the arms disagree, and with twelve jobs that is where the result lives.