DEV Community

Abhishek Banerjee
Abhishek Banerjee

Posted on Originally published at deeper-in-tech.hashnode.dev on

Long-Horizon Agent Execution: How We Handle Non-Deterministic Failures and Token Burn

#architecture #devops #aitools #aicoding

When I consult for engineering teams building autonomous agent systems, the initial demo is always impressive. An agent reads a ticket, generates a file, runs a test, and creates a clean pull request in under two minutes.

Then comes the real world.

A client asks the agent to perform a long-horizon task: "Migrate 45 database models from TypeORM to Prisma, update all repository classes, fix breaking service calls, and ensure the test suite passes."

Suddenly, the agent isn't running for 90 seconds it's running for 6 hours. And around hour 3, things go terribly wrong:

  • The LLM context window fills up with thousands of lines of terminal output, causing context rot. The agent forgets its original objective and starts reverting its own changes.

  • A transient network timeout occurs during a package installation. The agent panics, gets trapped in an infinite retry loop, and burns $600 in API tokens while you sleep.

  • A non-deterministic failure occurs on step 32 out of 40. Because there's no state persistence, your only option is to wipe the workspace and restart from step 1.

If you are deploying autonomous agents for multi-hour or multi-day tasks, you cannot treat agent execution as a single continuous API session. You need deterministic checkpointing, snapshotable state management, and mathematical human-in-the-loop thresholds.

Here is how I architect long-horizon agent execution pipelines that recover from failures gracefully without burning through your client's budget.

The Core Architecture: The Snapshotable Agent State Machine

Instead of letting an LLM run an unbounded loop in a single process, we treat the agent as a stateless executor operating on an immutable state graph.

At every discrete action boundary, the orchestrator freezes the workspace and creates a Checkpoint Snapshot :

┌──────────────────────────────────────────────┐
               │ Task Orchestrator (Temporal) │
               └──────────────────────┬───────────────────────┘
                                      │
           ┌──────────────────────────┼──────────────────────────┐
           ▼ ▼ ▼
 ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
 │ Checkpoint 01 │ │ Checkpoint 02 │ │ Checkpoint 03 │
 │ ───────────────── │ │ ───────────────── │ │ ───────────────── │
 │ - Git Commit SHA │ ───► │ - Git Commit SHA │ ───► │ - Git Commit SHA │
 │ - Context Window │ │ - Context Window │ │ - Context Window │
 │ - Token Spend │ │ - Token Spend │ │ - Token Spend │
 └───────────────────┘ └───────────────────┘ └─────────┬─────────┘
                                                                 │
                                                    [Step 4 Fail / Divergence]
                                                                 │
                                                                 ▼
                                                       ┌───────────────────┐
                                                       │ Rollback to CP-03 │
                                                       │ (Zero Token/Code │
                                                       │ Waste) │
                                                       └───────────────────┘
Enter fullscreen mode Exit fullscreen mode

A complete checkpoint consists of three synchronized artifacts:

  1. The Workspace State: A clean git tree commit capturing exact filesystem mutations.

  2. The Context Memory State: A pruned, summarized array of messages and tool execution outputs.

  3. The Financial Budget Ledger: Cumulative API cost, execution duration, and confidence scores.

If Step 35 fails catastrophically, we don't start over. We roll back the filesystem and context window to Checkpoint 34 , tweak the prompt or tool input, and resume execution.

1. Concrete Checkpoint Schema & State Storage

Here is how we define a snapshotable agent state contract in TypeScript using Zod:

// src/orchestrator/types.ts
import { z } from "zod";

export const AgentStepStatusSchema = z.enum([
  "PENDING",
  "EXECUTING",
  "CHECKPOINTED",
  "FAILED_RETRYABLE",
  "REQUIRES_HUMAN_INTERVENTION"
]);

export const CheckpointSchema = z.object({
  id: z.string().uuid(),
  taskId: z.string(),
  stepIndex: z.number().int().nonnegative(),
  gitCommitHash: z.string().length(40),
  status: AgentStepStatusSchema,
  metrics: z.object({
    cumulativeTokensUsed: z.number().int(),
    cumulativeCostUSD: z.number().positive(),
    loopCountCurrentStep: z.number().int(),
  }),
  contextWindowSnapshot: z.array(
    z.object({
      role: z.enum(["system", "user", "assistant", "tool"]),
      content: z.string(),
      toolCallId: z.string().optional(),
    })
  ),
  timestamp: z.string().datetime(),
});

export type Checkpoint = z.infer<typeof CheckpointSchema>;
Enter fullscreen mode Exit fullscreen mode

2. Mathematical Human-in-the-Loop (HITL) Thresholds

How do you prevent an agent from burning money in an infinite loop without requiring a human to watch terminal logs continuously?

You implement a mathematical Intervention Score ($I$) calculated at every checkpoint execution.

If (I \ge 0.75), or if any hard boundary is hit, the orchestrator immediately freezes execution, creates a rollback checkpoint, and pings an engineer via Slack or Linear with an interactive resume button.

Here is the production implementation of our threshold engine (src/orchestrator/threshold-engine.ts):

// src/orchestrator/threshold-engine.ts
import { Checkpoint } from "./types";

export interface ThresholdConfig {
  maxCostLimitUSD: number;
  maxLoopPerStep: number;
  maxTotalSteps: number;
}

export interface InterventionVerdict {
  shouldPause: boolean;
  reason?: string;
  interventionScore: number;
}

export function evaluateHumanIntervention(
  checkpoint: Checkpoint,
  config: ThresholdConfig,
  latestStepConfidence: number // Value between 0.0 and 1.0 from LLM
): InterventionVerdict {
  const { metrics, stepIndex } = checkpoint;

  // 1. HARD BOUNDARY CHECKS (Immediate Circuit Breakers)
  if (metrics.cumulativeCostUSD >= config.maxCostLimitUSD) {
    return {
      shouldPause: true,
      reason: `🚨 Budget Cap Reached: Accumulated $${metrics.cumulativeCostUSD.toFixed(2)} (Limit: $${config.maxCostLimitUSD})`,
      interventionScore: 1.0,
    };
  }

  if (metrics.loopCountCurrentStep >= config.maxLoopPerStep) {
    return {
      shouldPause: true,
      reason: `🔄 Infinite Loop Risk: Step ${stepIndex} attempted ${metrics.loopCountCurrentStep} times without state resolution.`,
      interventionScore: 1.0,
    };
  }

  // 2. WEIGHTED MATHEMATICAL THRESHOLD EVALUATION
  const loopWeight = (metrics.loopCountCurrentStep / config.maxLoopPerStep) * 0.4;
  const costWeight = (metrics.cumulativeCostUSD / config.maxCostLimitUSD) * 0.4;
  const uncertaintyWeight = (1.0 - latestStepConfidence) * 0.2;

  const score = loopWeight + costWeight + uncertaintyWeight;

  if (score >= 0.75) {
    return {
      shouldPause: true,
      reason: `⚠️ High Risk Score (${score.toFixed(2)} >= 0.75): Execution divergence detected.`,
      interventionScore: score,
    };
  }

  return { shouldPause: false, interventionScore: score };
}
Enter fullscreen mode Exit fullscreen mode

3. Real-World Client Failure Modes

In client projects where agents run long-horizon tasks, these two failure patterns occur constantly if you don't build defensive state controls:

Failure Mode 1: Context Window Degradation (Context Rot)

  • What Happened: After 40 tool calls, the context window contained 80,000 tokens of raw build logs and error stack traces. The LLM started ignoring instructions at the top of the prompt, editing files in the wrong directory, and repeating commands it executed hours earlier.

  • How We Fixed It: Implemented Context Window Compaction at every checkpoint. Tool outputs are summarized into 3-line structural outcomes (STATUS: PASS, MUTATED_FILES: [a.ts, b.ts]), and historical raw logs are stripped from the context window and offloaded to an external database file.

Failure Mode 2: The "Ghost Git State" Crash

  • What Happened: An agent executed a database migration CLI command that generated untracked files on disk, but failed on a unit test. When rolling back using simple git checkout ., the untracked migration files remained on disk, corrupting all subsequent retry attempts.

  • How We Fixed It: Hardened checkpoint rollbacks to use clean filesystem resets (git clean -fd && git reset --hard <CHECKPOINT_HASH>), guaranteeing a 100% pristine environment state on every step recovery.

4. Non-Trivial Terminal Execution

Here is what executing a long-horizon migration task looks like when a step fails, triggers our mathematical threshold engine, and pauses for human approval in terminal logs:

# 1. Start a long-horizon migration task with budget and loop limits
$ npx agent-orchestrator run \
  --task "migrate-typeorm-to-prisma" \
  --max-budget 15.00 \
  --max-loops-per-step 3

[Orchestrator] Task Started: migrate-typeorm-to-prisma
[Orchestrator] Initializing Checkpoint 001 (Git SHA: a7f3b2d...)

[Step 01/12] Refactoring User Model...
  └─ Checkpoint 001 created. Cost: $0.42 | Status: PASS
[Step 02/12] Refactoring Order Model...
  └─ Checkpoint 002 created. Cost: $0.88 | Status: PASS

[Step 03/12] Updating Payment Repository Interfaces...
  ├─ Execution Attempt 1: Failed TypeScript Compilation.
  ├─ Execution Attempt 2: Failed TypeScript Compilation.
  ├─ Execution Attempt 3: Failed TypeScript Compilation.
  │
  └─ 🚨 THRESHOLD ENGINE INTERVENTION TRIGGERED
     Reason: Infinite Loop Risk: Step 3 attempted 3 times without state resolution.
     Intervention Score: 1.00 (LoopWeight: 0.40, CostWeight: 0.08, Uncertainty: 0.20)

[Orchestrator] Rolling back workspace to Checkpoint 002 (Git SHA: c8d1e9f)...
[Orchestrator] Workspace clean. State safely paused.
[Orchestrator] Pinging engineer on Slack with state payload and restore token...

# Slack Message Sent: "Task 'migrate-typeorm-to-prisma' paused at Step 3. Click here to inspect diff or inject guidance."
Enter fullscreen mode Exit fullscreen mode

The Verdict

Architectural Choice

|

Unbounded Agent Script

|

State-Machine Agent Orchestrator

|
|

Failure Recovery

|

Restart from scratch

|

Deterministic Rollback to Last Checkpoint

|
|

Cost Control

|

Unlimited API token burn

|

Hard Mathematical Circuit Breakers

|
|

Context Hygiene

|

Degrades over time (Context Rot)

|

Compacted & Pruned at Every Step

|
|

Production Readiness

|

Low (Demo Toy)

|

High (Enterprise Grade)

|

My Takeaway as a Consultant: Stop letting agents run in infinite execution loops. If you want agents to handle complex, multi-hour refactoring jobs safely, build a snapshotable state machine. Treat git trees and context windows as recoverable database transactions, enforce mathematical budget thresholds, and build clean human-in-the-loop pause states.

### 💡 Need High-Impact Technical Content for Your Engineering Team?

I partner with developer-tooling startups, SaaS platforms, and engineering teams to translate complex infrastructure, agentic systems, and backend architecture into publication-grade technical writing.

Whether you need deep-dive architecture essays, hands-on developer tutorials, or technical counter-narratives:

📩 Email: abhishekninja2018@gmail.com

💼 LinkedIn: linkedin.com/in/abhishekninja

🐦 X (Twitter): @AvishekBanzzov

✍️ Medium: medium.com/@abhishekninja2018

💻 Dev.to: dev.to/abhishekninja_writer

🛠️ Capabilities: Long-form Technical Essays | Hands-On Tutorials | Developer Tooling Deep-Dives | Technical Counter-Narratives
Enter fullscreen mode Exit fullscreen mode

Top comments (0)