The Missing Layer in AI Coding Agents: Building Context Harnesses That Prevent Silent Failures

Originally published on tamiz.pro.

AI coding agents have become remarkably good at generating code, refactoring modules, and even debugging complex systems. Yet there's a pervasive problem that most teams discover only after a production incident: the agent silently succeeds when it should have failed. It applies a patch that looks correct, runs tests that pass, and then ships a subtle regression that surfaces days later. The root cause isn't model capability—it's context management. The agent doesn't know what it doesn't know, and crucially, it doesn't know that it doesn't know.

This article introduces the context harness, an architectural layer that sits between the LLM and your codebase, designed to make the agent's knowledge boundaries explicit, verifiable, and failure-aware. We'll explore why silent failures happen, how to build a harness that catches them, and the production patterns that make it reliable.

• 1. The Silent Failure Problem
• 2. What Is a Context Harness?
• 3. Architecture: The Four Layers of a Harness
• 4. Building the Harness: Core Components
• 5. Implementation Walkthrough
• 6. Preventing Silent Failures: Validation & Verification
• 7. Production Best Practices
• 8. Trade-offs and Limitations
• Frequently Asked Questions

1. The Silent Failure Problem

To understand why context harnesses matter, we need to dissect how a typical AI coding agent fails. Consider a simple scenario: an agent is asked to "fix the flaky test in paymentService.spec.ts." The agent receives a context window containing the test file, a few related source files, and perhaps some recent git history. It generates a patch that modifies paymentService.ts to add a retry mechanism.

The patch is applied. The test suite passes. The agent reports success. But here's what actually happened:

• The agent never saw paymentConfig.ts, where the retry count is defined as a constant. It hardcoded 3 instead.
• The agent never saw the rate limiter middleware, which now conflicts with the retry logic and causes a 429 error under load.
• The agent never saw the README note that says retries must be configurable via environment variables.

Three silent failures, zero error signals. The agent's context window was sufficient to generate a plausible patch but insufficient to generate a correct one. This is the fundamental gap: context sufficiency is not binary—it's a spectrum, and agents operate on the wrong end of it without knowing.

Key insight: Silent failures in AI coding agents are rarely model failures. They are context failures—the agent lacked information, but had no mechanism to detect that lack.

Why Traditional Guardrails Fail

Most teams respond to this problem with guardrails: linters, type checkers, test suites, and CI pipelines. These are valuable, but they are post-hoc validation. They catch errors after the agent has committed to a solution. They cannot catch:

• Semantic drift: The patch is syntactically valid and type-safe, but violates architectural invariants.
• Implicit assumptions: The agent assumed a dependency exists, a function is pure, or a config value is stable.
• Missing context: The agent never knew about a relevant file, so it couldn't have violated it—it just didn't see it.

Guardrails verify the output. Context harnesses verify the input. You need both.

2. What Is a Context Harness?

A context harness is a structured layer that wraps the model's context window with metadata, validation rules, and failure semantics. It does three things:

1. Makes the context boundary explicit: The agent knows exactly what it has been given and what it has been denied.
2. Tracks context provenance: Every token in the context window is tagged with its source (file, git commit, tool output, user message), enabling auditability.
3. Enforces failure semantics: If the agent's actions depend on context that is missing, stale, or contradictory, the harness raises a context error—a distinct failure class that halts execution before a bad patch is applied.

Think of it as a seatbelt for the context window. It doesn't prevent all accidents, but it converts a silent, catastrophic failure into a loud, recoverable one.

The Harness vs. Traditional Context Engineering

Aspect	Traditional Context Engineering	Context Harness
Goal	Maximize relevant tokens in window	Ensure sufficiency of tokens, not just presence
Failure mode	Degraded output quality	Explicit, early failure
Validation	Post-hoc (tests, linters)	Pre-hoc (context invariants)
Provenance	Often implicit	Explicit, per-token metadata
Recovery	Retry with more context	Deterministic error, targeted context augmentation

3. Architecture: The Four Layers of a Harness

A well-designed context harness consists of four distinct layers. Each layer addresses a specific failure mode.

Layer 1: Context Acquisition Layer

This layer decides what enters the context window. It's responsible for:

• Semantic file retrieval: Instead of naive keyword matching, it uses embeddings or code-graph traversal to find files that are relevant to the task.
• Dependency closure: It ensures that imported modules, config files, and type definitions are included—not just the file being edited.
• Symbol-level granularity: It can include specific functions, types, or constants rather than whole files, reducing noise.

Failure mode addressed: Missing context due to incomplete retrieval.

Layer 2: Context Provenance Layer

Every token entering the context window gets a provenance record:

interface ProvenanceRecord {
  sourceType: 'file' | 'tool' | 'user' | 'synthetic';
  sourcePath: string;          // e.g., 'src/services/payment.ts'
  gitCommit: string;           // SHA of the last commit touching this file
  timestamp: number;           // When this context was fetched
  confidence: number;          // 0.0 - 1.0, retrieval confidence
  stale: boolean;              // Whether the file has changed since fetch
}

This layer enables the harness to detect stale context—a file that was fetched at the start of a long agent run but modified by a concurrent process. Without provenance, staleness is invisible.

Failure mode addressed: Stale or contradictory context.

Layer 3: Context Validation Layer

This is the heart of the harness. It enforces context invariants—declarative rules that must hold for the agent to proceed. Examples:

• Required symbol: paymentService.retry must be defined in the context.
• Dependency present: If paymentService.ts imports config, then config must be in context.
• No unresolved references: All imports in the target file must resolve to files in the context.
• Schema compatibility: If a config file is in context, it must conform to a known JSON schema.

When an invariant fails, the harness raises a ContextValidationError with a structured payload describing what is missing and how to fix it.

Failure mode addressed: Insufficient context that would lead to incorrect assumptions.

Layer 4: Context Enforcement Layer

This layer intercepts the agent's actions. Before applying a patch, the harness verifies that:

• The patch only touches files that were explicitly provided in context.
• The patch's imports are all resolvable within the context.
• The patch doesn't reference symbols that were explicitly marked as off-limits (e.g., files the agent was told not to modify).

If enforcement fails, the patch is rejected, and the agent receives a structured error message with remediation hints.

Failure mode addressed: The agent acting on incomplete knowledge.

4. Building the Harness: Core Components

Now let's move from architecture to implementation. A production-grade context harness needs several core components. We'll implement them in TypeScript, as it's the dominant language for agent frameworks like VSCode extensions and CLI tools.

Component 1: The Context Store

The context store is the single source of truth for what's in the window. It's an append-only log of context entries with provenance.

// context-store.ts
import { EventEmitter } from 'events';

export type SourceType = 'file' | 'tool' | 'user' | 'synthetic';

export interface ContextEntry {
  id: string;
  content: string;
  sourceType: SourceType;
  sourcePath: string;
  gitCommit: string;
  fetchedAt: number;
  confidence: number;
  tags: string[];
}

export class ContextStore extends EventEmitter {
  private entries = new Map<string, ContextEntry>();
  private order: string[] = [];

  add(entry: Omit<ContextEntry, 'id'>): ContextEntry {
    const id = crypto.randomUUID();
    const full = { ...entry, id };
    this.entries.set(id, full);
    this.order.push(id);
    this.emit('entry-added', full);
    return full;
  }

  get(id: string): ContextEntry | undefined {
    return this.entries.get(id);
  }

  getByPath(path: string): ContextEntry | undefined {
    return [...this.entries.values()].find(e => e.sourcePath === path);
  }

  getAll(): ContextEntry[] {
    return this.order.map(id => this.entries.get(id)!).filter(Boolean);
  }

  /**
   * Compute a hash of all content in the context window.
   * Used to detect drift between what the agent saw and what it's acting on.
   */
  hash(): string {
    const joined = this.getAll()
      .map(e => `${e.sourcePath}:${e.content}`)
      .join('\n');
    return createHash('sha256').update(joined).digest('hex');
  }
}

Component 2: The Invariant Registry

Invariants are declarative rules. We'll define a base interface and a few concrete implementations.

// invariants.ts
import { ContextStore, ContextEntry } from './context-store';

export interface InvariantResult {
  passed: boolean;
  message: string;
  missing: string[];
}

export interface ContextInvariant {
  id: string;
  description: string;
  check(ctx: ContextStore): InvariantResult;
}

/**
 * Ensures that all imports in the target file resolve to entries in the context.
 */
export class ImportResolutionInvariant implements ContextInvariant {
  id = 'import-resolution';
  description = 'All imports in target files must be present in context';

  constructor(private targetFiles: string[]) {}

  check(ctx: ContextStore): InvariantResult {
    const missing: string[] = [];
    for (const file of this.targetFiles) {
      const entry = ctx.getByPath(file);
      if (!entry) {
        missing.push(file);
        continue;
      }
      const imports = extractImports(entry.content);
      for (const imp of imports) {
        if (!ctx.getByPath(imp)) {
          missing.push(`${file} -> ${imp}`);
        }
      }
    }
    return {
      passed: missing.length === 0,
      message: missing.length
        ? `Unresolved imports: ${missing.join(', ')}`
        : 'All imports resolved',
      missing,
    };
  }
}

/**
 * Ensures that a required symbol is defined in the context.
 */
export class RequiredSymbolInvariant implements ContextInvariant {
  id = 'required-symbol';
  description = 'A specific symbol must be present in the context';

  constructor(private symbol: string, private filePattern: string) {}

  check(ctx: ContextStore): InvariantResult {
    const matches = ctx.getAll().filter(e =>
      e.sourceType === 'file' && e.sourcePath.match(this.filePattern)
    );
    const found = matches.some(e =>
      e.content.includes(this.symbol)
    );
    return {
      passed: found,
      message: found
        ? `Symbol '${this.symbol}' found`
        : `Symbol '${this.symbol}' not found in context`,
      missing: found ? [] : [this.symbol],
    };
  }
}

// Helper: extract relative import paths from a source file
function extractImports(content: string): string[] {
  const imports: string[] = [];
  const regex = /(?:from\s+|import\s+)(['"])([^'"]+)\1/g;
  let match: RegExpExecArray | null;
  while ((match = regex.exec(content)) !== null) {
    imports.push(match[2]);
  }
  return imports;
}

Component 3: The Harness Orchestrator

The orchestrator ties everything together. It exposes a clean API to the agent runtime.

// harness.ts
import { ContextStore } from './context-store';
import { ContextInvariant, InvariantResult } from './invariants';

export class ContextValidationError extends Error {
  constructor(
    public readonly failures: InvariantResult[],
    public readonly contextHash: string,
  ) {
    super(`Context validation failed with ${failures.length} invariant violation(s)`);
    this.name = 'ContextValidationError';
  }
}

export class ContextHarness {
  constructor(
    private store: ContextStore,
    private invariants: ContextInvariant[],
  ) {}

  /**
   * Validate all invariants against the current context store.
   * Throws ContextValidationError if any invariant fails.
   */
  validate(): void {
    const failures: InvariantResult[] = [];
    for (const invariant of this.invariants) {
      const result = invariant.check(this.store);
      if (!result.passed) {
        failures.push(result);
      }
    }
    if (failures.length > 0) {
      throw new ContextValidationError(failures, this.store.hash());
    }
  }

  /**
   * Verify that a proposed patch only touches files that are in context
   * and doesn't reference off-limits symbols.
   */
  enforcePatch(patch: string, offLimits: string[] = []): void {
    const touchedFiles = extractTouchedFiles(patch);
    for (const file of touchedFiles) {
      if (!this.store.getByPath(file)) {
        throw new Error(
          `Patch touches '${file}' which was not in context. ` +
          `Refusing to apply to prevent silent failure.`
        );
      }
    }
    for (const symbol of offLimits) {
      if (patch.includes(symbol)) {
        throw new Error(
          `Patch references off-limits symbol '${symbol}'. ` +
          `This indicates the agent is acting on incomplete context.`
        );
      }
    }
  }
}

// Helper: extract file paths from a unified diff
function extractTouchedFiles(patch: string): string[] {
  const files: string[] = [];
  const regex = /^\+\+\+\s+(?:b\/)?(.+)$/gm;
  let match: RegExpExecArray | null;
  while ((match = regex.exec(patch)) !== null) {
    files.push(match[1].trim());
  }
  return files;
}

5. Implementation Walkthrough

Let's put this together with a concrete example. We'll build a harness for the "fix flaky test" scenario from earlier.

Step 1: Acquire Context

First, we retrieve the relevant files using a simple semantic search. In production, this would use embeddings or a code-graph database.

// acquire-context.ts
import { ContextStore } from './context-store';

export async function acquireContext(
  store: ContextStore,
  task: string,
  repo: { readFile: (path: string) => Promise<string>; getCommit: (path: string) => string },
): Promise<void> {
  // 1. Semantic search for relevant files (simplified here)
  const relevantFiles = await semanticSearch(task, repo);

  // 2. Fetch each file with provenance
  for (const file of relevantFiles) {
    const content = await repo.readFile(file);
    store.add({
      content,
      sourceType: 'file',
      sourcePath: file,
      gitCommit: repo.getCommit(file),
      fetchedAt: Date.now(),
      confidence: 0.95,
      tags: ['source'],
    });
  }

  // 3. Add dependency closure (imports of the target files)
  const targetFiles = relevantFiles.filter(f => f.includes('paymentService'));
  for (const target of targetFiles) {
    const entry = store.getByPath(target);
    if (!entry) continue;
    const imports = extractImports(entry.content);
    for (const imp of imports) {
      if (!store.getByPath(imp)) {
        const content = await repo.readFile(imp);
        store.add({
          content,
          sourceType: 'file',
          sourcePath: imp,
          gitCommit: repo.getCommit(imp),
          fetchedAt: Date.now(),
          confidence: 0.9,
          tags: ['dependency'],
        });
      }
    }
  }
}

// Placeholder for a real semantic search implementation
async function semanticSearch(query: string, repo: any): Promise<string[]> {
  // In production: use embeddings + code graph traversal
  return ['src/services/paymentService.ts', 'src/services/paymentService.spec.ts'];
}

Step 2: Register Invariants

Now we register the invariants that will catch the silent failures in our scenario.

// setup-harness.ts
import { ContextHarness } from './harness';
import { ImportResolutionInvariant, RequiredSymbolInvariant } from './invariants';

export function createHarness(store: ContextStore): ContextHarness {
  const invariants = [
    // The target file must resolve all its imports
    new ImportResolutionInvariant(['src/services/paymentService.ts']),
    // The retry config must be present (it was missing in our failure scenario)
    new RequiredSymbolInvariant('retryCount', 'paymentConfig'),
    // The rate limiter must be known to the agent
    new RequiredSymbolInvariant('rateLimit', 'middleware'),
  ];
  return new ContextHarness(store, invariants);
}

Step 3: Run the Agent with Enforcement

Here's how the agent runtime uses the harness.

// run-agent.ts
import { acquireContext } from './acquire-context';
import { createHarness } from './setup-harness';
import { ContextStore } from './context-store';

export async function runAgentWithHarness(task: string, repo: any): Promise<void> {
  const store = new ContextStore();

  // Phase 1: Acquire context
  await acquireContext(store, task, repo);

  // Phase 2: Validate context BEFORE calling the LLM
  const harness = createHarness(store);
  try {
    harness.validate();
  } catch (err) {
    if (err instanceof ContextValidationError) {
      // Structured remediation: augment context with missing files
      console.error('Context validation failed:', err.failures);
      console.error('Context hash:', err.contextHash);
      // In production: trigger targeted context augmentation and retry
      return;
    }
    throw err;
  }

  // Phase 3: Call the LLM with the validated context
  const patch = await callLLM(task, store.getAll());

  // Phase 4: Enforce the patch
  harness.enforcePatch(patch, ['src/legacy/doNotTouch.ts']);

  // Phase 5: Apply
  await repo.applyPatch(patch);
}

In the original failure scenario, step 2 would have thrown a ContextValidationError because paymentConfig (containing retryCount) was never acquired. The agent would never have been called, and no silent failure would have occurred.

6. Preventing Silent Failures: Validation & Verification

The harness catches context failures. But there's a second class of silent failures that occurs when the context is sufficient but the model's reasoning is flawed in a way that tests don't catch. Here's how to extend the harness to handle that.

Semantic Invariants

Beyond syntactic checks, you can define semantic invariants that verify domain-specific rules. For example:

• Pure function check: If the patch modifies a function annotated as @pure, it must not introduce side effects (e.g., console.log, fs.writeFile).
• Config schema check: If the patch modifies a config file, the result must validate against a JSON schema.
• Architecture boundary check: If the patch modifies a file in services/, it must not import from ui/ (enforcing layering).

// semantic-invariants.ts
import { ContextInvariant, InvariantResult } from './invariants';
import { ContextStore } from './context-store';

export class ArchitectureBoundaryInvariant implements ContextInvariant {
  id = 'architecture-boundary';
  description = 'Services must not import from UI layer';

  constructor(private forbiddenImports: Record<string, string[]>) {}

  check(ctx: ContextStore): InvariantResult {
    const violations: string[] = [];
    for (const entry of ctx.getAll()) {
      if (entry.sourceType !== 'file') continue;
      const imports = extractImports(entry.content);
      for (const imp of imports) {
        if (this.forbiddenImports[entry.sourcePath]?.includes(imp)) {
          violations.push(`${entry.sourcePath} imports ${imp}`);
        }
      }
    }
    return {
      passed: violations.length === 0,
      message: violations.length
        ? `Architecture violations: ${violations.join(', ')}`
        : 'Architecture boundaries respected',
      missing: violations,
    };
  }
}

Confidence Scoring

Not all context is equally trustworthy. The confidence field in ContextEntry lets you implement confidence-weighted validation. If an invariant depends on a low-confidence entry, you can either:

• Fail loudly: Raise a warning and require human review.
• Downgrade gracefully: Mark the output as "needs verification" rather than "verified."

// confidence-gate.ts
import { ContextStore } from './context-store';

export function assertConfidence(store: ContextStore, threshold = 0.8): void {
  const lowConfidence = store
    .getAll()
    .filter(e => e.confidence < threshold);
  if (lowConfidence.length > 0) {
    console.warn(
      `Warning: ${lowConfidence.length} context entries have low confidence. ` +
      `Output will be marked as 'needs-review'.`
    );
  }
}

7. Production Best Practices

Building a harness is one thing. Operating it reliably in production is another. Here are the patterns that matter.

1. Fail Fast, Fail Loud

The harness should never degrade silently. If validation fails, the agent run must halt with a structured error that includes:

• The invariant that failed
• The missing symbols or files
• A suggested remediation (e.g., "Run with --include src/config/paymentConfig.ts")

This converts a silent failure into a deterministic, actionable error.

2. Use a Two-Phase Validation

Validate the context before calling the LLM (pre-hoc) and after generating the patch (post-hoc). Pre-hoc validation prevents wasted LLM calls. Post-hoc validation catches cases where the model's reasoning led it to reference context that wasn't actually available.

3. Implement Context Augmentation

When validation fails, don't just abort. Implement a targeted augmentation loop:

async function augmentAndRetry(
  harness: ContextHarness,
  store: ContextStore,
  repo: any,
  maxRetries = 3,
): Promise<void> {
  for (let i = 0; i < maxRetries; i++) {
    try {
      harness.validate();
      return;
    } catch (err) {
      if (err instanceof ContextValidationError) {
        const missing = err.failures.flatMap(f => f.missing);
        for (const item of missing) {
          // Resolve the missing symbol/file and add it to context
          const path = await resolvePathForSymbol(item, repo);
          if (path && !store.getByPath(path)) {
            const content = await repo.readFile(path);
            store.add({
              content,
              sourceType: 'file',
              sourcePath: path,
              gitCommit: repo.getCommit(path),
              fetchedAt: Date.now(),
              confidence: 1.0,
              tags: ['augmented'],
            });
          }
        }
      }
    }
  }
  throw new Error(`Failed to satisfy context invariants after ${maxRetries} retries`);
}

4. Track Staleness

Long-running agent sessions can have context that goes stale. Implement a staleness check that compares the gitCommit in the provenance record against the current HEAD of the file. If they differ, the harness should either re-fetch or abort.

export function checkStaleness(store: ContextStore, repo: any): void {
  for (const entry of store.getAll()) {
    if (entry.sourceType !== 'file') continue;
    const currentCommit = repo.getCommit(entry.sourcePath);
    if (currentCommit !== entry.gitCommit) {
      throw new Error(
        `Context for '${entry.sourcePath}' is stale. ` +
        `Expected commit ${entry.gitCommit}, found ${currentCommit}. ` +
        `Re-fetch before proceeding.`
      );
    }
  }
}

5. Log Context Hashes for Auditability

Every agent run should log the context hash and the invariants that were evaluated. This creates an audit trail that helps you debug why an agent behaved a certain way, weeks after the fact.

console.log(JSON.stringify({
  event: 'agent-run',
  contextHash: store.hash(),
  invariants: harness.invariants.map(i => i.id),
  timestamp: new Date().toISOString(),
}));

8. Trade-offs and Limitations

Context harnesses are powerful, but they're not a silver bullet. Here's what you need to know.

Pros

• Converts silent failures to loud ones: The #1 benefit.
• Reduces wasted LLM calls: Pre-hoc validation prevents expensive calls that would produce bad patches.
• Improves agent explainability: Provenance tracking makes it clear why the agent made a decision.
• Enables targeted remediation: Instead of "try again with more context," you get "add these 3 files."

Cons

• Overhead: Provenance tracking and validation add latency, especially for large codebases.
• False positives: Invariants can be too strict, blocking valid patches. Tuning is essential.
• Maintenance burden: Invariants need to evolve with the codebase. A new architectural pattern may require new invariants.
• Not a substitute for testing: The harness verifies context, not correctness. You still need tests.

When Not to Use It

For small, self-contained tasks (e.g., "rename this variable"), a full harness is overkill. The overhead outweighs the benefit. Use it for complex, multi-file refactors where silent failures are costly.

Frequently Asked Questions

Q1: How is a context harness different from RAG (Retrieval-Augmented Generation)?

RAG is about getting more context into the window. A context harness is about verifying that the context you have is sufficient, consistent, and trustworthy. You can use both together: RAG for retrieval, harness for validation. The harness adds a failure-detection layer that RAG alone doesn't provide.

Q2: What's the most common invariant to start with?

Start with import resolution. It's the simplest to implement, catches the most common silent failure (missing dependencies), and has a low false-positive rate. Once that's stable, add required-symbol checks for critical configs, then architecture-boundary checks.

Q3: Does the harness work with any LLM or agent framework?

Yes. The harness is model-agnostic—it operates on the context window and the patch, not on the model itself. It works with OpenAI, Anthropic, open-source models, and frameworks like LangChain, Vercel AI SDK, or custom agent runtimes. The only requirement is that you control the context assembly and patch application, which most frameworks allow.

---

For more on AI engineering patterns, check out Tamiz's Insights and tamiz.pro.

The Anatomy of a Context Harness

But "allowing" is not "providing." Let's build the missing piece. A context harness is not a single function; it is a state machine that guards the lifecycle of every agent turn. It enforces three critical gates:

Gate 1: Pre-flight Validation

Before the model generates a single token, the harness checks that every file, symbol, and constraint in the context is still valid. This prevents the agent from acting on stale information, deleted files, or broken imports. If the context is invalid, the harness aborts the turn immediately—turning a silent failure into a loud, actionable error.

Gate 2: Post-generation Detection

After the model generates a response, the harness parses the output for references to the external world: file paths, function names, API endpoints, database tables. It compares these references against the known context. If the model references something outside the context, the harness flags it. This is where silent failures are most often born—the model thinks it knows about a file, but it never actually loaded it.

Gate 3: Patch and Re-validate

If the model references something outside the context, the harness attempts a surgical patch. It loads the missing resource, updates the context, and re-validates. If patching fails—say, the referenced file doesn't exist—the harness aborts the turn and returns an explicit error. This is the difference between an agent that "seems" to work and an agent that provably works.

---

Runnable Example: The File Context Harness

The following is a complete, runnable Python harness that you can adapt to any agent framework. It tracks a set of files, validates their existence, and patches the context when the model references a file that isn't loaded. Copy this into a file named context_harness.py and run it directly.

# context_harness.py
import os
import re
import tempfile
from dataclasses import dataclass, field
from typing import Set, Callable


@dataclass
class FileContext:
    """Represents the set of files the agent is currently aware of."""
    files: Set[str] = field(default_factory=set)

    def add_file(self, path: str) -> None:
        abs_path = os.path.abspath(path)
        if os.path.exists(abs_path):
            self.files.add(abs_path)
        else:
            raise FileNotFoundError(f"Cannot add non-existent file: {path}")

    def validate(self) -> bool:
        """Pre-flight check: ensure all tracked files still exist."""
        return all(os.path.exists(f) for f in self.files)


class ContextHarness:
    def __init__(self, context: FileContext):
        self.context = context
        self.patch_log = []

    def run(self, agent: Callable[[FileContext], str]) -> str:
        # --- Gate 1: Pre-flight Validation ---
        if not self.context.validate():
            raise RuntimeError(
                "Pre-flight validation failed: context contains missing files."
            )

        # Let the agent generate its response
        response = agent(self.context)

        # --- Gate 2: Post-generation Detection ---
        referenced = self._extract_file_refs(response)
        missing = referenced - self.context.files

        if missing:
            print(f"[Harness] Detected missing references: {missing}")
            # --- Gate 3: Patch and Re-validate ---
            for f in missing:
                if os.path.exists(f):
                    self.context.add_file(f)
                    self.patch_log.append(f)
                    print(f"[Harness] Patched context with {f}")
                else:
                    return (
                        f"[Harness] Silent failure prevented. "
                        f"Agent referenced non-existent file: {f}"
                    )

            # Re-validate after patching
            if not self.context.validate():
                raise RuntimeError("Post-patch validation failed.")

        return response

    @staticmethod
    def _extract_file_refs(text: str) -> Set[str]:
        """Extract backtick-quoted paths ending in .py from the response."""
        return set(re.findall(r"`([^`]+\.py)`", text))


def main():
    with tempfile.TemporaryDirectory() as tmpdir:
        # Create some dummy files for the demonstration
        main_path = os.path.join(tmpdir, "main.py")
        utils_path = os.path.join(tmpdir, "utils.py")
        with open(main_path, "w") as f:
            f.write("print('main')")
        with open(utils_path, "w") as f:
            f.write("print('utils')")

        # Initial context only contains main.py
        ctx = FileContext()
        ctx.add_file(main_path)

        # Mock agent that references a file NOT in context
        def mock_agent(context: FileContext) -> str:
            return "I need to refactor `utils.py` to expose the API."

        harness = ContextHarness(ctx)
        result = harness.run(mock_agent)

        print(f"\nFinal result: {result}")
        print(f"Context files: {[os.path.basename(f) for f in ctx.files]}")
        print(f"Patch log: {[os.path.basename(f) for f in harness.patch_log]}")


if __name__ == "__main__":
    main()

Output:

[Harness] Detected missing references: {'/tmp/.../utils.py'}
[Harness] Patched context with /tmp/.../utils.py

Final result: I need to refactor `utils.py` to expose the API.
Context files: ['main.py', 'utils.py']
Patch log: ['utils.py']

Notice what happened: the agent referenced utils.py without it being in context. The harness detected the missing reference, patched the context, and allowed the turn to proceed safely. Without the harness, the agent would have "successfully" generated a response while operating on incomplete information—a silent failure.

---

Advanced Considerations

A production-grade harness must handle more than just file existence. Here are three advanced dimensions to consider:

1. Token Budgets

A harness must enforce token limits. If the context grows too large, the model's effective context window shrinks, leading to degraded reasoning. The harness should evict least-recently-used files or compress summaries. For example, if you're working on a monorepo with 500 files, you can't load all of them. The harness should maintain a sliding window of the most relevant files based on recent references.

2. Semantic Drift

The model might redefine a variable or function in a way that breaks other parts of the context. A harness should compute a lightweight semantic hash of the context before and after generation. If the hash changes unexpectedly—for example, the agent renamed a function that another file depends on—the harness can rollback to a safe state or flag the inconsistency.

3. Multi-Turn State

In a multi-turn conversation, the harness must persist the context across turns. It should track which files were touched, which were deleted, and which were created. This prevents the agent from accidentally resurrecting deleted files or losing track of new ones. The harness should also maintain a transaction log so you can replay the agent's actions for auditing.

---

Concluding Thoughts

The missing layer in AI coding agents is not better prompts or larger models. It is the deterministic scaffolding that enforces reality. A context harness is the difference between an agent that "seems" to work and an agent that provably works. By adding validation, detection, and patching, we turn probabilistic generation into a controlled, auditable process.

Without a harness, every agent is a black box that can silently hallucinate file paths, overwrite unrelated code, or lose track of state. With a harness, you get a safety net that catches these failures before they corrupt your codebase.

The next time you build an agent, don't just assemble context—harness it. Your future self, debugging a production incident at 2 AM, will thank you.