Saanj Vij

Posted on Aug 24 Originally published at sanjvij.netlify.app

Beyond Prompt Engineering: Why Modern AI Workflows Require Orchestration Platforms on Top of Tools Like Claude Code

#ai #devops #llm #machinelearning

A team of forty engineers gives every one of them Claude Code. Individually, each session is impressive — specs get read, code gets written, tests get run. Six weeks later, the platform team is fielding a different complaint: nobody can predict what an agent will do twice in a row. The same spec, run on a Tuesday and then again on a Thursday, produces two different implementations, one of which quietly skips the security review step because the context window rotated it out of memory.

This isn't a prompting problem. It's an architecture problem, and no amount of CLAUDE.md tuning fixes it.

The Evolution of AI-Assisted Development

The trajectory so far has been a steady climb in autonomy. GitHub Copilot started as inline autocomplete — a probabilistic next-token suggestion inside an editor, with a human approving every line before it landed. Then came interactive CLI agents: Claude Code, Aider, and their peers, which can read a codebase, plan a multi-file change, execute shell commands, and run tests, all inside a single conversational loop. We covered the first leg of that trajectory in Prompt Engineering: The New Professional Literacy, back when writing a good prompt was itself the differentiating skill. This piece picks up several steps downstream, once prompting stopped being the bottleneck and execution architecture became the constraint.

That jump — from "suggests a line" to "can act inside your repository" — is what makes these tools genuinely useful. It's also exactly where the current generation of tooling hits a ceiling. An interactive CLI agent is built around a session: one human, one terminal, one context window, one thread of conversation. That model scales beautifully to an individual developer's productivity. It does not scale to "run this across forty repositories with auditable, repeatable behavior," because nothing in the design of a chat session was built for repeatability. Two runs of the same session, with the same spec, are two independent draws from a probability distribution — not two executions of a program.

Enterprises evaluating AI-assisted development are running into this ceiling now, and the instinct is almost always to reach for a better spec.

The Limits of Spec-Driven Development on Its Own

We've argued before that the traditional SDLC is cracking under AI-speed development, because two of its core assumptions — human-paced review and single-author commits — don't hold once an agent can generate a day's worth of diffs in an hour. Spec-Driven Development — formalized in tooling like GitHub's Spec Kit, which structures work into spec.md and plan.md artifacts before any code is written — is a real improvement over freeform prompting, and a real attempt to replace those broken assumptions. It forces intent to be written down before implementation starts, which catches a category of ambiguity that would otherwise get resolved by the model guessing.

But a spec handed to an interactive LLM session is still probabilistic guidance, not deterministic enforcement. The distinction matters more than it sounds:

Prompting with context means giving the model a document and trusting it to follow the document. Compliance is a function of how well the model attends to that document at generation time.
Deterministic enforcement means the constraint is checked and applied by code that runs outside the model's control, regardless of what the model "decided" to do.

A spec.md file only ever achieves the first kind of compliance. As a session runs longer, the context window fills with tool output, file contents, and intermediate reasoning — and the spec, read once at the start, competes for attention with everything that came after it. The model can hallucinate a function signature that doesn't exist, drift from an architectural constraint stated on line 4 of the spec by the time it's writing file 12, or simply skip a verification step because nothing forced it to run. None of this is a flaw in the model reasoning "badly" — it's the expected behavior of a probabilistic system operating without hard boundaries. A spec is instructions for a system that can choose not to follow them; a harness is a system that cannot proceed without following them.

The Anatomy of an Agent Harness and the Need for Determinism

The agent harness is the physical execution environment surrounding the model — the shell it runs commands in, the file system it can touch, the hooks that fire before and after each tool call, the permission boundaries that decide what it's even allowed to attempt. The model inside the harness should stay probabilistic; that's the source of its usefulness. But the harness around it has to be 100% deterministic, or every guarantee you think you have is actually just a suggestion the model happened to follow this time.

Concretely, this means enterprise-grade guardrails live in software, not in prose. A CLAUDE.md file that says "always run tests before committing" is a request. A pre-commit hook that refuses the commit if tests haven't run is a guarantee. The difference shows up the first time an agent under deadline pressure — or context-window pressure — decides the instruction in the markdown file is less urgent than finishing the task.

A minimal execution-hook configuration that enforces this at the harness level looks something like:

{
  "hooks": {
    "preToolUse": [
      {
        "matcher": "Bash",
        "command": "scripts/verify-branch-protection.sh",
        "blockOnFailure": true
      }
    ],
    "postToolUse": [
      {
        "matcher": "Edit|Write",
        "command": "npx eslint --max-warnings=0 $CLAUDE_FILE_PATH",
        "blockOnFailure": true
      },
      {
        "matcher": "Edit|Write",
        "command": "npm test -- --related $CLAUDE_FILE_PATH",
        "blockOnFailure": true
      }
    ]
  }
}

Nothing in that configuration asks the model nicely. If the linter fails, the edit is rejected before it ever reaches disk — the model doesn't get a vote. This is the practical form that "the harness must be deterministic" takes: AST-aware linters instead of style guidance, automated test runners instead of "please write tests," and permission proxies that hard-block dangerous shell commands instead of a prompt asking the model to be careful. A permissions proxy that says "don't touch the filesystem outside this directory" is a request; a hypervisor boundary that makes it physically impossible is a guarantee — the isolation angle covered below. Determinism at the harness layer is the same idea applied to what the agent is permitted to execute, not just where it's permitted to execute it.

The Shift to Orchestration Engineering

A hardened harness solves determinism for a single agent, running a single task, in a single session. It does not solve the problem of running many tasks, across many agents, with state that needs to persist and branch and retry. That's a different layer of the stack — orchestration — and it does four things a standalone agent session structurally cannot:

State management and graph logic. An orchestrator models the work as a graph, not a conversation: nodes for plan, implement, test, review; edges with conditions for retry-on-failure or escalate-to-human. A failed test run doesn't end the session — it routes back to an earlier node with the failure captured as new state. A single chat session has no native concept of "go back to step 2 with this new information without forgetting who I am."

Multi-agent topology. Instead of one session carrying planning, implementation, testing, and security review inside one increasingly bloated context window, an orchestrator assigns each concern to a specialized agent — a Planner, a Coder, a Test Writer, a Security Auditor — each with its own tightly scoped context. This isn't a cosmetic reorganization: an agent whose entire context is "review this diff for injection vulnerabilities" produces measurably more focused output than the same model asked to also remember the original spec, the file tree, and its own prior implementation reasoning. I mapped this exact topology in detail in Inside the ADLC Engine Room: How Multi-Agent Pipelines Actually Work — the plan/implement/verify/review phases described there are what an orchestrator's graph nodes actually execute. How those specialized agents hand context to each other without stepping on one another is its own architecture problem, covered in The Agentic Protocol Stack.

Workspace isolation. Running agents concurrently against the same working directory is a recipe for one agent's half-finished edit corrupting another's test run. Production orchestrators solve this by programmatically managing isolated workspaces — Git worktrees for lightweight local isolation, or ephemeral Docker Sandboxes / microVMs — the same hypervisor-backed isolation I detailed in MicroVM Sandboxing for Claude Code — for stronger boundaries when the agent needs to execute untrusted or --dangerously-skip-permissions-style commands. Each agent gets a disposable environment; nothing persists that wasn't explicitly merged back.

Token and context optimization. Left unmanaged, context windows only grow — every tool call's output, every file read, every intermediate thought accumulates until the model's attention is spread across thousands of tokens of stale information. An orchestrator actively fights this: isolating subtasks so their context doesn't leak into the parent session, routing low-complexity work (a rename, a lint fix) to smaller/cheaper models instead of burning a frontier model's context on it, and compressing or summarizing completed steps before passing state forward.

A simplified state-machine loop capturing the retry/escalate pattern:

def run_task(task, max_retries=2):
    state = {"task": task, "attempts": 0}
    while True:
        plan = planner_agent.run(state)
        result = coder_agent.run(plan, workspace=new_worktree())
        verdict = test_runner.run(result)

        if verdict.passed:
            return security_auditor.run(result)

        state["attempts"] += 1
        if state["attempts"] > max_retries:
            return escalate_to_human(state, verdict)

        state["last_failure"] = verdict.errors  # fed back, not discarded

Every branch in that loop is something a single chat session has no mechanism to express on its own — it either keeps going in one thread or the human starts over.

Architectural Patterns of Modern Orchestrators

In practice, orchestration platforms cluster into two deployment patterns, without getting into specific vendors here — that comparison deserves its own dedicated piece.

Local lightweight orchestrators run on a developer's machine and manage local CLI sessions cleanly: isolating each task on its own Git branch or worktree, capping context per session, and preventing one long-running agent conversation from ballooning into an unmanageable token bill. These are the right fit when a human is still driving and wants AI assistance parallelized across a few concurrent branches of work.

Production and background frameworks go further — automated engines that take an issue, run the full plan-implement-test-review graph in a sandboxed cloud environment, and open a pull request with no human in the loop until review time. This is the pattern that actually delivers on "AI writes the PR" at organizational scale, because the determinism guarantees described above are enforced by infrastructure the whole team shares, not by an individual's local hook configuration.

The Platform Engineering Blueprint

Put together, the architecture that actually holds up at scale has three distinct layers, and conflating any two of them is where most "why doesn't this work reliably" complaints come from:

Interactive LLM Usage (single session)          Orchestrated Multi-Agent System
─────────────────────────────────────           ──────────────────────────────────────────
                                                   ┌─────────────┐
  ┌───────────────────────────┐                    │ Orchestrator │  ← state graph, retries,
  │   Human ⇄ CLI Agent        │                    │ (Control &   │    escalation, routing
  │   (one context window)     │                    │  State Engine)│
  │                             │                    └──────┬───────┘
  │  spec.md read once ─────┐  │                            │
  │                          ▼  │                ┌───────────┼───────────┐
  │  plan → code → test →   │  │           ┌──────▼─────┐┌───▼────┐┌─────▼─────┐
  │  (context grows,        │  │           │  Planner   ││ Coder  ││  Security  │
  │   attention dilutes)    │  │           │   Agent    ││ Agent  ││  Auditor   │
  │                          │  │           └──────┬─────┘└───┬────┘└─────┬─────┘
  └──────────────────────────┘                     │  each in its own    │
                                                     ▼  isolated worktree /│
                                              ┌─────────────┐ sandbox      │
  No hard boundary enforcing                  │  Agent      │◄─────────────┘
  spec compliance — only the                  │  Harness    │  deterministic hooks:
  model's own attention decides.              │ (per agent) │  lint, test, permission
                                               └─────────────┘  proxy — non-negotiable

The blueprint is Spec-Driven Development (intent) + Agent Harness (deterministic execution runtime) + Orchestrator (control and state engine) — three layers, each solving a problem the others structurally can't. A spec without a harness is a suggestion the model may or may not follow. A harness without an orchestrator is a single reliable agent with no way to coordinate with others or recover from failure gracefully. An orchestrator without a hardened harness underneath it is just coordinating chaos faster.

A better prompt cannot substitute for a worse architecture. If your organization's AI strategy is currently a shared CLAUDE.md file and a hope that everyone's interactive sessions behave consistently, the honest read is that you don't have an AI platform yet — you have forty independent experiments that happen to share a system prompt.

This is the same pattern I described in AI Isn't Eliminating Software Engineering. It's Moving the Bottleneck. — clearing one constraint doesn't remove it, it relocates it. There, the bottleneck moved from writing code to reviewing it. Here, it's moved from generating a plan to enforcing that execution actually followed it. And at the organizational level, that's the exact gap I mapped in The ADLC Transition: Enterprise Strategy, Developer Identity, and the Review Gap — the technical architecture is solvable; the organizational one is where it quietly stalls.

If you're evaluating this for your own team: what's actually enforcing your AI agents' behavior today — a document they're supposed to read, or code that runs whether they read it or not?

DEV Community