Note: I've recently had the opportunity to present on this topic twice over the last two weeks—one talk diving deep into the loop engineering work and agentic iterations, and another covering the full 5-layer optimization stack. While the exact slides, internal metrics, and specific implementation details remain proprietary, I wanted to share the broader architectural framework and general best practices behind the core ideas.
We have all been there: you type a prompt, wait for an LLM to spin its wheels, and watch it confidently generate five paragraphs of polite narrative alongside a completely misaligned solution. Or worse, you spin up an autonomous agent loop to tackle a refactoring task, only to realize it has burned through a massive chunk of your token budget because it spent thirty iterations reading and rereading multi-megabyte log files.
The industry-wide knee-jerk reaction to these failures is usually to write longer prompts, dump more files into the window, or throw a larger, more expensive model at the problem. But if we look at AI integration through a systems engineering lens, these symptoms point to a structural flaw: we are treating prompt engineering as a silver bullet while ignoring the actual architecture of how we feed, house, and govern our models.
First, Respect the Token Mechanics
Before optimizing the stack, it helps to treat tokens as an engineering constraint rather than just an API billing line item. Token volume directly dictates inference latency, attention degradation, and hallucination rates.
More importantly, tokenization isn't spelling. LLMs don't read characters; they process numerical IDs mapped to statistical subword chunks. Because of how tokenizers parse text:
- Code is denser than prose: Uncommon class names, UUIDs, leading whitespace, and syntax punctuation split into far more chunks than standard English sentences.
- Logs are silent budget killers: Pasting identical multi-page stack traces repeatedly across turns burns context capacity exponentially.
-
Measure, don't guess: Every LLM interaction ultimately boils down to a fundamental REST contract. Inspecting the actual input/output token counts (
prompt_eval_countandeval_count) and execution duration against your target production tokenizer is the only way to baseline real performance.
To move from guesswork to predictable, production-ready AI workflows, we need to organize our approach around a 5-layer optimization stack: Prompt, Context, Harness, Loop, and Meta.
Layer 1: Prompt Engineering (The Assembly Layer)
Prompt engineering sits at the absolute bottom of the optimization stack—not because it is the most important, but because it is the closest to "assembly language".
Just as writing raw assembly code gets you closest to the bare-metal CPU instructions—giving you maximum direct control over every register and byte, but requiring immense manual effort for anything complex—writing prompts puts you right at the metal of the model's token weights. Treating prompt engineering as the bottom layer means recognizing that trying to solve systemic engineering failures by writing longer, more complex assembly prompts is like trying to write an entire modern application in raw assembly. Direct control over your instructions matters, but treating prompts like magic incantations leads to bloated text and erratic behavior.
- Drop the Persona Hype: Telling a model it is a "10x principal staff engineer" wastes tokens and adds zero deterministic value. Boring, explicit prompt contracts—defining the exact job, boundary fence, allowed modifications, and output shape—outperform elaborate persona prompts every single time.
- Adopt "Caveman" Efficiency: Strip away conversational padding. Compare a polite, 35-token request ("Could you please look at the code I selected and refactor it to remove duplication...") against a direct, 24-token contract:
Refactor selected code. Remove duplication. Preserve behavior.
Modify selected code only. Return code only. Minimal diff.
Saving 10 input tokens is nice, but preventing the model from generating 500 tokens of conversational narration around the code block is the real latency and cost win.
- Match Models to the Task: Avoid extremes. Don't route simple JSON reformatting or log classification to a massive reasoning model, and don't force a tiny 3B local model to untangle complex multi-file architecture if it takes four failed retries to get right. Optimize for the cheapest completed job, not the cheapest individual token.
Layer 2: Context Engineering (Controlling the View)
One does not simply dump an entire repository into a prompt. Passing raw, unconstrained codebases creates signal-to-noise degradation, increases latency, and causes models to lose track of critical constraints buried in the middle of the window.
- Understand What IDE Tools Actually Do: Modern editor integrations use workspace indexing, semantic vector search, inline grounding (open files and cursor selections), and diff awareness to surface relevant code. However, baked-in tools optimize context; they do not optimize intent. If your acceptance criteria are fuzzy, better semantic retrieval won't save the output.
-
Expand on Demand: Never start by feeding the model the whole project tree. Start tightly scoped: pass the selected code block, the exact failure log, the current
git diff, and explicit acceptance criteria. Only pull in adjacent interfaces or upstream callers if the first pass lacks sufficient type or contract awareness.
Layer 3: Harness Engineering (Environmental Guardrails)
Harnessing is engineering the environment around the model so it doesn't start from zero context on every single execution. A harness doesn't require a heavyweight agent framework; it can be as straightforward as a structured instruction file, a local CLI validation script, and a strict JSON schema.
A strong harness relies on four pillars:
- Prepopulate: Surface known facts—such as repository rules, architectural boundaries, and service connection maps—before the model has to ask or guess.
- Structure: Enforce deterministic input and output schemas so downstream scripts can parse model responses reliably without fragile regex gymnastics.
- Memory: Persist prior architectural decisions and known constraints so the model doesn't re-litigate settled patterns.
- Tools: Give the model scoped, executable commands to inspect state, run linters, and validate its own work.
Think about cross-service architecture discovery: instead of letting an LLM burn tokens spelunking through dozens of repositories every time someone needs to trace message queues, APIs, or database connections, you build a lightweight scanner that extracts that dependency metadata into a clean, structured schema once. You pay the discovery cost programmatically once, and every subsequent AI interaction benefits from deterministic context.
Layer 4: Loop Engineering (Managing Iteration Safely)
Loop engineering takes those harness capabilities and allows an LLM to iterate autonomously over a task—measuring state, acting, verifying results, and repeating.
However, the most dangerous word in loop engineering is another. Just because an agent can keep iterating doesn't mean it should. Without strict architectural constraints, agentic loops easily enter a death spiral: a test fails, an orchestrator spawns a sub-agent to fix it, the sub-agent runs the entire test suite, hits the exact same failure, and spawns more workers that all pay to re-read identical multi-megabyte log files. Optimization becomes amplification when you pay AI to reread noise.
If you are building automated test-generation or refactoring loops, enforce four architectural guardrails:
- Single Process Execution: A central orchestrator controller must own execution state. Sub-agents should generate code or patches, never trigger full environment runs independently.
- Smallest Possible Verification Scope: During the inner loop, execute only the specific unit test file or module that was just modified. Reserve full test-suite runs for the final verification gate.
- Useful Log Tails: Never pipe raw stdout/stderr streams directly back into the prompt. Write the full log to a flat file on disk, and pass only the exit status, coverage delta, and the last 10 relevant lines of the error stack to the model.
-
Deterministic Exit Code Routing: Use strict exit codes (
0= step passed, proceed;non-zero= parse error fingerprint and route to fix, retry, or abort) so control flow lives in deterministic code, not LLM guesswork.
Layer 5: Meta-Prompt Engineering (Governance and Control)
Meta-prompting sits at the top of the stack as a control and governance layer that optimizes how a task gets solved before heavy execution even begins.
- Pre-Optimize Intent: Before routing a complex request to an expensive frontier model, use a lightweight classifier step to inspect the prompt, assess task complexity (low, medium, high), enforce strict output constraints, and route to the appropriate model tier with a capped token budget. Asking "What is the cheapest, fastest, most reliable path to a verified result?" upfront routinely cuts downstream token waste in half.
- Enforce Repository-Wide Defaults: Relying on individual developers to remember prompt hygiene every time doesn't scale. Commit persistent rule files directly into your repositories to standardize task boundaries, coding conventions, and concise, filler-free output rules across the entire team automatically.
-
Govern Loop State: At the meta level, track failure fingerprints across iterations. If an agent produces the same error hash twice, kill the loop or escalate rather than letting it burn retries. Checkpoint valid states, reset stale context windows, and pass the
git diff—never the entire universe.
Takeaway
Tokens equal control, but fewer tokens is not the ultimate goal—less useless work is the goal.
When an AI workflow feels slow, expensive, or unreliable, resist the urge to write a longer prompt. Look up the stack instead, find where you are forcing the model to repeatedly parse noise or rediscover things your deterministic tooling already knows, and cut that work out of the loop entirely.
Top comments (0)