A few months ago, if someone on my team said an AI workflow was “bad,” the first question was usually: What prompt are we using?
Now I find myself asking a different question:
Where does the model get feedback, and what is allowed to change after it fails?
That shift sounds small, but it changes almost everything.
Lilian Weng’s July 2026 post, “Harness Engineering for Self-Improvement”, helped frame a pattern many agent builders have already been wrestling with in practice. The model is still important. Of course it is. But the system wrapped around the model — tools, memory, workflow, permissions, evaluation, file state, background jobs, feedback loops — is becoming just as important as the prompt itself.
Weng defines a harness as the system around a base model that orchestrates execution: how the model plans, calls tools, acts, manages context, stores artifacts, and evaluates results.
That is the part that caught my attention.
I work close enough to customer support workflows to know that “better AI” rarely means “one better answer.” In practice, it means you stop getting the same repeated failures, the system actually recovers instead of freezing when things get ambiguous, and human handoffs become a whole lot smoother. Plus, you finally get a system that doesn't completely forget why something broke last Tuesday.
That is not prompt engineering anymore.
That is harness design.
A Prompt Is an Instruction. A Harness Is a Working Environment.
Prompt engineering is not useless. I still care about clear goals, constraints, examples, and tone.
But a prompt is mostly instruction.
A harness is environment.
The difference matters because most agent failures are not just wording problems. They are system problems:
The model gets too much context and loses the relevant bit.
The tool result is noisy.
The agent retries the same broken action.
The workflow has no memory of previous failures.
The evaluation step is vague or missing.
The system cannot tell whether an answer is merely plausible or actually verified.
You can patch some of this with a better prompt. For a while.
But eventually the prompt becomes a crowded apartment: instructions, warnings, examples, tool notes, edge cases, memory snippets, formatting rules, and “please don’t do that weird thing again” all squeezed into one place.
A harness gives those responsibilities proper rooms.
Memory can live in files or structured stores.
Evaluation can run as a check, not a paragraph of advice.
Tools can have permissions.
Retries can have policy.
Logs can become evidence.
Failures can be mined instead of forgotten.
That is why I do not read “harness vs prompt engineering” as a replacement story. It is more like a promotion. Prompt engineering becomes one component inside a larger runtime system.
Forget Code Rewriting: The Real Reality of AI Self-Improvement
The hype around AI self-improvement loves to paint a dramatic picture: a model autonomously rewriting its own weights, training its next iteration, and scaling up recursively. But that’s the Hollywood version.
Maybe someday. But near-term self-improvement looks much more boring and much more useful.
It starts outside the model.
In Weng’s framing, the practical near-term path is not necessarily a model directly editing its own weights. It is the harness becoming an optimization target. The system learns better ways to select context, manage state, run workflows, evaluate outputs, and preserve useful artifacts.
That matches what I see in developer tooling.
If an agent fails a coding task, maybe the model is not smart enough. But maybe the harness failed first:
It did not inspect the right file.
It did not run the failing test.
It did not preserve the error trace.
It did not know when to ask for clarification.
It did not compare the final patch against the original requirement.
This is where AI agent loop design becomes interesting. A minimal loop might look like this:
- Plan the next action.
- Execute through a tool.
- Observe the result.
- Store useful evidence.
- Evaluate against the goal.
- Update the workflow or memory.
- Try again with better state.
That loop is not magic. It is just software engineering applied to agent behavior.
But the moment you can actually look under the hood and tweak that loop, the system finds a real path to self-improvement—without having to pretend the model magically became its own R&D lab overnight.
The New Unit of Optimization Is the Run
What I like about the harness framing is that it moves attention from the answer to the run.
A run has traces. Tool calls. Files touched. Tests executed. Errors ignored. Assumptions made. Human interventions. Final evidence.
That means the system can learn from more than success or failure.
The Self-Harness paper makes this explicit. It describes a loop where an LLM-based agent improves its own operating harness through weakness mining, harness proposal, and proposal validation. The important part is not “the agent rewrites itself” in some vague way. It is that failures are clustered, proposed harness changes are bounded, and accepted edits must pass regression checks.
That last part matters.
Self-improvement without regression testing is just automation with confidence issues.
The same pattern appears in newer harness research around coding agents. Agentic Harness Engineering frames harness evolution around observability: editable components, distilled experience, and decisions that can later be checked against outcomes.
That is the version of AI self-improvement I can actually imagine trusting in production.
Not an agent randomly rewriting its system prompt because it “reflected.”
A system that says:
“This failure pattern happened 14 times.”
“The root cause appears to be missing project-state recovery.”
“Here is a narrow harness change.”
“Here is the regression set.”
“Here is what improved, and here is what did not.”
That is less cinematic. It is also much closer to how reliable engineering works.
ECC Shows the Developer Version of This Shift
The GitHub repo affaan-m/ECC is a useful signal because it turns the harness idea into something developers can poke at.
The repo describes itself as a harness-native operator system for agentic work, spanning skills, memory optimization, security scanning, cross-harness workflows, and support for tools like Claude Code, Codex, Cursor, OpenCode, and others.
I would not treat any GitHub project as proof that a concept is mature. But I do think ECC shows why the harness idea is spreading.
Developers do not just want a smarter model.
They want agents that remember project conventions.
They want repeatable workflows.
They want security boundaries.
They want skills that can transfer across tools.
They want logs, hooks, rules, and recovery paths.
That is exactly the layer prompt engineering does not fully cover.
A prompt can tell an agent to be careful.
A harness can prevent it from touching production secrets, require tests before completion, preserve failure traces, and route uncertain cases to a human.
That difference is not cosmetic. It is the difference between advice and infrastructure.
So Is Prompt Engineering Over?
No.
But the era where prompt engineering was the main interface for improving AI systems is fading.
A good prompt still matters. It tells the model what the work is. It shapes behavior. It encodes constraints.
But for long-running agents, the harder problems now sit around the prompt:
What context should be loaded?
Which tool should be available?
What should be persisted?
What counts as done?
Who verifies the result?
What happens after failure?
What is allowed to change next time?
That is why “AI Harness” feels like a more useful keyword than another prompt trick.
For a small team, the practical move is not to build a grand self-improving agent on day one. It is to make your current loop visible.
Start with one workflow.
Log every tool call.
Save the important artifacts.
Define a verifier.
Separate memory from instructions.
Let failures produce structured notes.
Only then consider letting the system propose changes to the harness.
The scaling law bottleneck is not that models stopped improving. It is that better base models do not automatically give you better deployed systems. The gap between capability and reliability is increasingly filled by harnesses.
That is where I think Lilian Weng’s post lands hardest.
Prompt engineering asks: What should I say to the model?
Harness engineering is really about one big question: how do you build a system that lets a model screw up, learn from it, verify the fix, and try again—all without turning your entire workflow into a total black box of hidden states? Having watched countless shiny AI features blow up in actual user-facing apps, I know exactly which problem I’d rather spend my time solving.
Top comments (0)