The latest articles on DEV Community by B McGhee (@bmcghee). https://dev.to/bmcghee https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3872085%2F7dae6755-e98d-4a71-a3fd-ae53ea182dec.jpeg https://dev.to/bmcghee en B McGhee Fri, 10 Apr 2026 16:07:53 +0000 https://dev.to/bmcghee/my-ai-pipeline-had-a-1m-token-context-window-the-output-still-got-worse-333l https://dev.to/bmcghee/my-ai-pipeline-had-a-1m-token-context-window-the-output-still-got-worse-333l <h2> Fixing a context window problem in an AIOps investigation pipeline </h2> <p>The pipeline stitches context from three repos, calls Gemini with a chain-of-thought prompt, and posts root cause analysis to Slack and Jira. At some point output quality dropped.</p> <h2> Diagnosis </h2> <p>A character count diagnostic showed the actual repo sizes:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>frontend ~527k tokens backend ~311k tokens legacy ~7.9M tokens </code></pre> </div> <p>The fixed 50/35/15 budget split was loading the same proportion of irrelevant code regardless of ticket type. A scheduling bug got the same legacy allocation as an auth bug.</p> <p>Models don't attend uniformly across long contexts. Irrelevant content degrades output quality, it doesn't just take up space. The ceiling wasn't the constraint. Context selection was.</p> <h2> Constraints to consider </h2> <p><strong>Model rate limits and context window.</strong> Already hitting the API directly so context caching is available, but the 1M token ceiling is hard. The fix had to work within it, not around it.</p> <p><strong>Context quality vs. quantity.</strong> A smaller focused window consistently outperforms a larger noisy one for reasoning tasks. This ruled out "just get a bigger window" as a solution.</p> <p><strong>Latency.</strong> A secondary concern alongside quality: the time from bug filed to Slack/Jira report. Runner queue time and repo checkout compound. Addressed separately via infrastructure, not the script.</p> <p><strong>Multi-language repos.</strong> Three different primary languages (TypeScript, Go, legacy Node.js) with different directory conventions. The routing table had to account for each independently.</p> <h2> Fix </h2> <p><strong>Label-based routing.</strong> Extended the ticket fetch to include labels and components, then mapped those to repo-specific dirs and budget splits:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>scheduling → frontend: 45% / backend: 45% (scheduling handlers only) / legacy: 10% auth → frontend: 55% (providers, hooks) / backend: 35% / legacy: 10% default → 50/35/15 </code></pre> </div> <p><strong>Prompt restructure.</strong> Stable content (architecture context, codebase) at the top, ticket at the bottom. Better attention on the ticket, enables implicit caching across back-to-back runs.</p> <h2> The point </h2> <p>Use deterministic pre-filtering before the LLM sees any code. The model sees less. The output is better. Reach for context selection before a bigger window.</p> ai gemini llm promptengineering