Why structured context outperforms prompt engineering — and what my cousin's backyard taught me about it.
Breaking ground
If you've spent any time building with AI — whether it's a side project, an enterprise workflow, or just trying to get consistent results from a chatbot — you've probably lived through a version of this cycle.
You write a prompt. It works beautifully. You try something similar the next day and it falls apart. You rephrase. You add more detail. You try chain of thought, persona prompting, few-shot examples. You read an article about a magic sentence someone adds to every prompt. You try that too. Sometimes it helps. Sometimes it doesn't. Sometimes the exact same prompt gives you a different result on a different day and you have no idea why.
So you dig deeper. You study prompt engineering best practices. You test variations. You regression-test prompts against different models. The results are real — some promising, some frustrating, some downright confusing. And underneath all of it, one question keeps surfacing: why is this so inconsistent?
I spent months in that cycle before I realized I was solving the wrong problem. I wasn't writing bad prompts. I was building on ground I hadn't tested — pouring effort into the entrance of a tunnel without understanding what was happening in the passage or what was waiting at the other end.
It turns out that when you type a prompt, your words enter the model and travel through dozens of computational layers. Each layer transforms what came before. Somewhere in the middle — not at the surface, not at the end — the actual reasoning happens. The model doesn't think where you can see it. It thinks deep in the passage, in a workspace that's far more constrained than most people realize.
How you structure what goes into that passage determines what reaches the place where thinking happens. And that realization changed everything about how I work.
I first learned the principle from a tunnel my cousins, my brother, and I tried to dig when I was eight years old.
To the library
We'd just watched The Great Escape. The fascinating part — obviously — was the tunnels.
We decided we could dig one too.
We had a site picked out — under the backyard fence at my cousins' house, between the yard and the side driveway. There was a slight bank that hid the hole, one of the selling points. And it would get us into the backyard rather than the neighbor's yard, which we all agreed probably wouldn't go over well. Especially given that we were still dealing with the fallout from when my cousins tossed vegetables they didn't want to eat over the neighbor's fence.
But we knew we couldn't just start digging. We needed a plan. So collectively, we looked at one another and said — "To the library!"
In a 1970s world before home computers, before the internet, before Google, we had one source of information. A fantastic one. The local library. The librarians knew everything, knew where everything was. And my grandfather was always willing to take us, because reading is important.
We found exactly what we were looking for — how to properly build, buttress, and vent a tunnel. We asked for paper and pencil, wrote it all down, and checked out a few books to verify our plans once they were drawn up.
The silence
If you've ever watched an AI model generate a response, you've experienced a strange kind of silence. Tokens stream across the screen — words appearing one after another — and you watch, waiting to see if what comes out is going to be useful or if it's going to miss entirely. You can see the output forming, but you can't see the thinking. The reasoning is happening somewhere you don't have access to, in a passage between your input and its response that you can't observe and can't control. All you can do is watch and hope the result reflects what you intended.
My dad knew that kind of silence. He just experienced it differently.
He always said the noise and rumble of five boys together — all close in age — was fine. He and my uncle would listen for screams that needed attention or tempers to flare. But the one thing both of them never, ever wanted was silence. Silence was terrifying because, in their words, "They're plotting! They're planning!"
They were right to worry. The silence meant we were deep in execution — past the point where the plan was just a plan.
In fairness to us, we'd acquired 2x6 boards and 3/4-inch plywood and 16-penny nails to buttress our tunnel as we dug. We had a plan to build a hand-driven air pump, just like in the movie. We'd even assigned my brother as the safety officer — because he's thorough and has an eye for detail.
We got about three feet down and four feet out — for the record, that produces a lot of dirt — when we needed light. My oldest cousin was dispatched to retrieve my uncle's shop light from the garage. And that's when things unraveled — not because the tunnel had failed, but because my uncle pulled up in the family van, saw my cousin walking toward the fence with a shop light and a long extension cord, and suddenly three of the five heads popped up from behind the sloped bank.
His first reaction: "This can't be good."
He pulled out his pipe, filled it, lit it, took a few puffs, and slowly walked over to investigate. Rather than say too much, he calmly told us — "I think you boys need a break. Why don't you come in for a drink of water."
That was his code for "Stop immediately!" — and time to fetch my dad, who'd also just pulled up and gone into the kitchen.
The silence — whether it's five boys underground or a model working through your prompt — is where the real work happens. It's also where things go wrong if the structure isn't in place.
The design review
My dad — ever the engineer — was extremely thorough in his investigation of our work. I remember hearing them discuss how impressed they were with our attention to detail and safety. They wanted to see our drawings and plans.
However, the tunnel had to go.
They framed it perfectly. "A big hole in the backyard will make it unsafe to play football and mow the grass." That made complete sense to us. So per their direction and oversight, we filled it back up.
Then came the design review. They sat us down at the kitchen table and told us how impressed they were — especially with the teamwork and our desire to dig deeper into research to ensure a positive outcome. But while we had a plan, it turned out to be a plan with gaps. At eight to ten years old, we didn't know how to account for the weight of dirt, the possibility of water, or how those forces would impact our magnificent tunnel.
They explained it calmly, in great detail. My dad drew pictures to help us see where our plan was flawed. After discussing it among ourselves, we determined no further tunnel efforts would be made anytime soon.
This came as great relief to my dad and uncle — because after their review, I remember my dad turning to my uncle and saying, "Balderdash — I think we just accidentally showed them how to do it right."
For the record, my uncle was a firefighter who'd responded to tunnel collapses. I'm truly sorry we terrified him. But he handled it with calm and levelheadedness, as did my dad. I'm fairly certain, however, that my cousins, my brother, and I were directly responsible for both of them going gray. Not gradually — in real time. You could practically watch it happen. Every summer they'd age a year for every week we were together.
And they were both right — the silence was dangerous.
The buttressing held
Here's the thing — we weren't wrong about the methodology. We went to authoritative sources. We extracted the relevant information. We built a structured plan with drawings and notes. We assigned roles, including a safety officer. We buttressed as we went. We adapted the plan as realities emerged during execution.
Five kids between eight and ten years old independently built a governed workflow with a design review, acceptance criteria, and role-based responsibility.
What we got wrong were the constants. We didn't know the load-bearing properties of soil. We didn't account for water saturation. We didn't understand how lateral pressure increases with depth. We had the methodology right and the discipline right — but we were missing the domain constraints that would have made the plan safe at scale.
Literally, at depth.
The narrow passage
Years later, my dad used the tunnel story to teach me something about business calculus. Not the mathematics — the thinking. He wanted me to understand what it means to hold your constants steady and isolate the variable you're actually testing. He used our tunnel to show me: we had the variable right (the digging), but the constants were undefined (the structural engineering). Without those constants, we couldn't know whether our effort would produce a result or a collapse.
It turns out that's a precise description of how AI models process your input.
When you send a prompt to a large language model, your words don't arrive at a single destination. They pass through dozens of computational layers — think of them as successive chambers in a long tunnel. Each layer transforms the information from the one before it. Early layers handle surface-level patterns — grammar, word relationships, basic structure. Deeper layers handle progressively more abstract reasoning — inference, analogy, multi-step logic.
Anthropic recently published research that revealed something remarkable about what happens in those deeper layers. Their researchers built a tool called the Jacobian lens and used it to look inside Claude — not at what it says, but at what it's working with before it says anything. What they found is a small internal workspace they call J-space.
Imagine a workbench at the far end of a long tunnel. Everything you send into the entrance has to travel through the full passage to reach that bench. The bench itself is small — it can hold a few dozen concepts at a time. That's it. And it accounts for less than a tenth of everything happening inside the model. But that small workbench is where virtually all the meaningful reasoning happens — the inference, the composition, the multi-step thinking that makes the difference between a generic response and a genuinely useful one.
When Anthropic's researchers cleared the workbench entirely, the model could still talk fluently. It could classify sentiment. It could answer multiple-choice questions. It could retrieve facts. All the surface-level capabilities survived. What collapsed was the deeper work — the kind of reasoning you actually need when the task is complex, nuanced, or requires connecting ideas across multiple steps.
The tunnel is narrow. The workbench is small. What reaches the other end matters enormously.
What the dirt weighs
Now think about what most people are doing when they "engineer" a prompt.
They're adjusting the entrance. Rearranging the words. Adding a persona. Trying "think step by step." Appending a magic sentence. These are boards laid across the opening — and they do help at shallow depth. But they don't change the structural properties of the passage. They don't control what reaches the workbench. And they don't account for what the dirt actually weighs.
Chroma Research tested 18 frontier models and documented what happens as you send more material through the passage. Performance doesn't degrade gradually — models maintain near-perfect accuracy up to a point and then fall off a cliff. The passage fills up. Worse, material that's semantically similar but irrelevant to the task actively misleads the model. It's not neutral noise. It's debris in the tunnel — and it pushes useful concepts off the workbench to make room for things that don't belong there.
This is why prompt engineering hits a ceiling. Every trick is a separate board laid in isolation. None of them reinforce each other. None of them compound across sessions. And critically, none of them address the actual bottleneck: the workbench is finite, and what fills it determines the quality of reasoning. Adding more tokens — even well-intentioned ones — can actively degrade the output if they crowd out the concepts that matter.
Research has shown that reordering examples in a prompt can shift accuracy by more than 40%. A technique that improves results on one model may degrade them on another. The same input, rearranged, produces a fundamentally different outcome — not because the information changed, but because the path through the passage shifted.
And the ground itself is moving. Prompts that worked before a model update stop working after — not because the prompt changed, but because the model's interpretation shifted. Practitioners call this "prompt rot." The boards worked last month. The ground underneath moved.
My dad's design review was about exactly this. We had boards. We had nails. We even had buttressing. What we didn't have was the structural engineering — the constants that would have told us whether our supports could bear the load at depth. We were building in the dark, and it was working at three feet. It would not have worked at six.
The load-bearing walls
I studied econometrics in college in the mid-nineties. Our textbook was Gujarati's Basic Econometrics — a book that, thirty years later, I couldn't use to run a regression if my life depended on it. But I could tell you why the regression matters.
The math was the vehicle. The reasoning was the cargo.
Econometrics teaches you to think about systems with multiple interacting variables. The first thing you learn — before any model, before any dataset — is the concept of ceteris paribus: all other things being equal. You hold your constants steady so you can isolate and measure the thing that's actually changing. Without that discipline, you can't attribute causality to anything.
The second thing you learn is correlation. Specifically, that correlation is not causation — and that when your model is missing variables, the correlations you do observe can mislead you completely. This is called omitted variable bias, and it's what happens when you see a relationship between two things and conclude that one causes the other, when in reality they're both being driven by something you didn't measure.
Back to the tunnel.
When someone adds a "magic sentence" to a prompt and gets a better result, they observe a correlation between the phrasing and the output. Naturally, they conclude the sentence caused the improvement. But what else changed? The conversation history was different. The task complexity was different. The model version may have been different. The rest of the context window was different. Without holding those things constant, you can't attribute the improvement to the sentence. Maybe it helped. Maybe the ground happened to be firmer at that particular spot, and the board would have held without nails.
This is the fundamental problem. Prompt tuners are running experiments with no control group. Every observation is confounded. They're changing multiple variables simultaneously and attributing the outcome to the one variable they think they changed. Gujarati would have circled that in red on the first problem set.
My dad taught me the same insight with a tunnel. My professors formalized it with regression models. It's the same lesson: define your constants before you test your variables.
Here's what that looks like in practice.
When I work with an AI on my side project, the model sees a set of knowledge files, skill definitions, rules, and conventions every time a session starts — before I type a single word. These are the constants. They define how the system interprets instructions, what standards it enforces, what patterns it follows, and what governance constraints apply. They don't change between sessions. They don't depend on how I phrase my prompt. They're the load-bearing walls of the tunnel.
And here's why that matters at the level of how the model actually works: those constants aren't just providing information. They're curating what reaches the workbench. Structured knowledge files, explicit conventions, typed artifacts — these are high-signal, low-noise inputs. They give the model's reasoning workspace the right concepts to work with, rather than forcing it to infer structure from vague instructions or guess at context that should have been stated.
The prompt — the thing I actually type — is the variable. And because the constants are set, the variable's job gets smaller. It doesn't have to carry context, establish conventions, remind the system what it already knows, or compensate for structural gaps.
It just has to say what's next.
When your constants are zero, it doesn't matter how clever your variables are. The function can't converge.
That's the prompt engineering problem in one sentence. People are spending enormous energy tuning variables when the constants aren't set. They get a good result, they can't reproduce it, and they blame the model. Sometimes the model did change. Often it didn't. But because their constants were never defined, they have no way to isolate what actually caused the difference — so what reaches the workbench is different every time, and the reasoning produces different results every time. They're observing a correlation between their prompt and the output without controlling for anything else. In econometric terms, their model isn't identified. Every coefficient is biased.
Anthropic's research illustrates this beautifully. When their researchers replaced a single concept on the workbench — swapping "France" for "China" — multiple downstream answers changed together. Capital, language, continent, currency — all shifted from one shared representation change. That's how constants work. One well-structured input propagates across every downstream output. One knowledge file change in my system propagates across every session, every task, every workflow. You're not fixing answers one at a time. You're setting the shared representation that all downstream reasoning draws from.
That's the leverage of a shared constant: change it once, and its influence propagates across every downstream variable it governs. And because the constant is explicit and controlled, you can observe that relationship instead of guessing at it.
Measuring the depth
I'm not theorizing. I built it and I measured it.
My dad used to call this kind of evidence a SWAG — a Serious Wild-Ass Guess. Not a peer-reviewed study. Not a controlled experiment with a thousand participants. But an educated, measured, real-world observation with enough data points to form a thesis worth testing. That's what I have. And I think the research explains why it works.
Here's what I actually built.
I work on a side project where I use an AI model as my primary collaborator — design, code generation, documentation, the whole development lifecycle. Early on, I was doing what everyone does: typing detailed prompts, providing context in the conversation, and hoping the model remembered what I'd told it three exchanges ago. I was digging without buttressing.
So I restructured. I created a layered context architecture — a set of files the model sees before I say a word:
Knowledge files — thirteen always-loaded reference documents covering coding conventions, workflow rules, library-specific gotchas, and architectural decisions. These are the equivalent of the books we checked out of the library. They're versioned, governed, and curated. Irrelevant material gets removed because I learned — before the Chroma research confirmed it — that extra tokens don't help. They're debris in the tunnel.
Skills — step-by-step procedures for recurring tasks, written in a format the model can execute consistently. Think of these as the buttressing instructions — not "figure out how to shore up the walls" but "place the 2x6 here, nail it here, check the load here." They remove ambiguity from the passage. And they improve with use — roughly 12% of my time goes toward editing the skills through their own workflow. The procedures get better because they process their own feedback. That's a compounding return that no prompt trick can match.
Rules and conventions — behavioral guardrails and coding standards the model enforces automatically. Import ordering, naming patterns, section headers, test structure. These are the soil constants — the properties of the ground that don't change between digs.
Typed artifacts — structured files that bridge design intent to execution. A design spec defines what to build. A handoff file tells the code agent how to build it. A context file tracks what actually happened during execution. Each one is a load-bearing wall that keeps the passage aligned from entrance to exit.
Governance gates — binary checkpoints between phases. Did the design spec pass validation? Did the code pass lint? Did the tests pass? Do the acceptance criteria match the delivered work? These are the safety inspections — the moments where my dad and uncle would have checked the tunnel depth against the buttressing strength.
None of this is a prompt. None of it changes based on how I phrase a question. All of it is in place before the conversation starts. The constants are set. The tunnel is buttressed. And only then do I type what's next.
Here's what the numbers show.
Before the restructuring: 56% waste rate across eight sessions. Three issues closed in fourteen hours. More than half my time lost to stale context, missed conventions, and back-and-forth that should have been unnecessary. Cost per issue: $1,158.
After the restructuring — same model, same human, same codebase — I closed 41 issues in ten days. Cost per issue: $134. That's an 8.6x improvement.
The sharpest data point: seven issues completed in roughly five hours with zero waste. Forty-two phase transitions, 100% compliance with every gate check, zero regressions.
I also measured the tunnel itself. My initial context architecture consumed roughly 46,000 tokens of the model's attention before I typed anything. Through two rounds of optimization — removing material that wasn't earning its space, restructuring files for higher signal-to-noise, moving reference material to on-demand loading — I reduced that to around 27,000 tokens. A 41% reduction in fixed overhead. Not by adding a magic sentence, but by removing debris from the passage so the workbench had room for the concepts that matter.
Nothing changed except the constants.
Is this proof? No. It's a SWAG. It's one person, one project, one workflow. But the direction is consistent, the magnitude is large, and the research converges from three directions: how models reason internally, how they behave under load, and where the industry is heading.
Stanford, SambaNova, and UC Berkeley published the ACE framework in late 2025, showing that editing and growing input context — rather than tuning prompts or fine-tuning the model — delivered a 10.6% improvement on agentic tasks and reduced latency by 86.9%. Context treated as a living, structured resource outperformed prompt optimization. Anthropic's J-space research tells us why — context architecture works because it curates what fills the model's finite reasoning workspace. The tunnel began as a metaphor. The research suggests it may also be a surprisingly accurate description of what's happening computationally.
When the ground shifts
Here's the part that should worry anyone relying on prompt engineering in production.
Prompt tuning is implicitly fitting to a specific model's behavior — to how this model, this version, interprets this phrasing. When the model updates — new training run, new architecture, new alignment tuning — the ground shifts. Every carefully crafted prompt is potentially invalidated. The boards were fitted to ground that no longer exists.
Structured context is far less exposed to this problem. Knowledge files, governance rules, typed artifacts, and acceptance criteria are not inherently tied to one model's particular phrasing behavior. It's a contract layer. Swap the model underneath and the contract still holds, because you're not relying on how one specific model interprets a clever sentence. You're providing the information any sufficiently capable model needs, in a form it can consume regardless of its internal architecture.
And the J-space research gives us reason to expect that sufficiently capable models have something analogous to a constrained reasoning workspace — an area where actual inference happens. The workspace may differ in size or connectivity across models. But the principle is the same: structured, high-signal context fills that workspace more reliably than unstructured prompts, regardless of which model you're working with.
It's the same principle as programming to an interface instead of an implementation. Prompt tuners are programming to the implementation — and every model update is a breaking change.
Andrej Karpathy put it plainly in June 2025: in every serious AI application, "context engineering" — not prompt engineering — is the core skill. Shopify's CEO said the same thing the same week. By mid-2025, Gartner had declared the shift. The question isn't whether this matters. The question is whether you're building for it or still rearranging words at the entrance.
What I'm not saying
A few things I want to be clear about — because what I'm not saying matters as much as what I am.
Prompt engineering isn't worthless. For a one-off question, a casual conversation, a narrow and predictable task — phrasing matters and a well-constructed prompt genuinely helps. At shallow depth, the dirt is light and a few boards hold just fine. The people writing articles about magic sentences aren't wrong. They're solving a real problem at a real scale — just not the scale where things start to collapse.
I'm also not claiming to have cracked open the black box. The J-space research is Anthropic's, not mine, and it's early. The researchers are careful to separate what they observed from deeper claims about what's really happening inside these systems. What I am saying is that their findings line up with what I observed on my own: structuring the context — not tuning the prompt — is what drives consistent, reproducible, high-quality results. The workbench is narrow. The passage is real. And what you send through it matters more than how you phrase the invitation to dig.
My evidence is a SWAG, not a proof. One person, one project, one workflow. But an 8.6x improvement doesn't happen by accident, and the research — ACE, Chroma, J-space, prompt brittleness — all converges on the same explanation. I can't tell you the exact correlation coefficient between structured context and output quality. But I can tell you the direction is consistent, the magnitude is large, and every time I've tightened the constants, the results have improved. Gujarati would want a larger sample size. My dad would say the tunnel held.
Start with the library
The moral of the tunnel story isn't that we were wrong to dig. We were industrious, organized, and genuinely creative. The moral is that we went to authoritative sources, built a structured plan, assigned roles, buttressed as we went, and adapted to realities as they emerged.
We had the methodology right. We were missing the constants — the structural engineering that would have told us whether the passage could hold under real weight.
Every prompt you write is the entrance to a tunnel. The question isn't whether the model can generate a response — it almost always can. The question is whether the passage between your input and its reasoning is buttressed well enough for the right concepts to reach the workbench where thinking actually happens.
The people tuning prompts are digging enthusiastically at three feet. It's working. The dirt is light and the walls are holding.
But the dirt gets heavier. And if your walls aren't buttressed, you won't know it until they aren't.
Start with the library. Set the constants. Then dig.
References
- ACE Framework — Xu, Z., et al. "Autonomous Context Engineering for LLM Agents." Stanford University, SambaNova Systems, UC Berkeley. October 2025. arXiv:2510.04618
- J-Space Research — Anthropic. "A Global Workspace in Language Models." July 6, 2026. anthropic.com/research/global-workspace. Full paper: transformer-circuits.pub/2026/workspace. Open-source implementation: github.com/anthropics/jacobian-lens. Interactive demo: neuronpedia.org/jlens.
- J-Space Coverage — Heaven, W.D. "Anthropic Found a Hidden Space Where Claude Puzzles Over Concepts." MIT Technology Review, July 9, 2026. technologyreview.com
- Context Window Performance Degradation — Chroma Research. "Studying 18 Frontier Models on Context Window Performance." July 2025. chroma.com/research
- Prompt Sensitivity & Ordering Effects — Lu, Y., et al. "Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity." ACL 2022. arXiv:2104.08786
- Context Engineering Shift — Karpathy, A. Post on X, June 25, 2025. x.com/karpathy/status/1937902205765607626. Lütke, T. Post on X, June 18, 2025.
- Gujarati, D. Basic Econometrics. McGraw-Hill, 1978 (5th edition, 2008). The econometrics textbook referenced in this article for its treatment of ceteris paribus, omitted variable bias, and model identification.
- Gartner — "Context Engineering Is In, Prompt Engineering Is Out." Analyst briefing, mid-2025.
This is the third post in a series on the economics of AI-assisted development. The first post covers the waste problem. The second post covers carrying costs. This one covers what to build instead.
The views and opinions expressed here are my own and do not represent those of my employer or any other organization, entity, or group I am affiliated with. This post reflects my personal experience on a personal project.
Top comments (0)