The Hard Part of Long-Running Agents Is Not the Model

For the past few months, I have been building NoodleTomato, an agent system for AI video generation.

It does not generate a single six-second clip. It takes a brief and works for hours: planning a video, writing a script, delegating work, generating dozens of shots, producing narration, selecting music, creating captions, assembling a timeline, and rendering a finished master.

I expected model quality to be the bottleneck. It was not. The model was capable enough. The failures came from the system around it.

A long-running agent is a distributed system

A chat assistant makes one response, then a person checks it. The feedback loop closes in seconds.

A video-production agent can make hundreds of dependent decisions before anyone sees the result. Each decision may write persistent state, spend money, or launch a job that settles an hour later. The run can outlive a worker process, a deployment, the model's context window, and the user's attention.

This changes the meaning of failure. A bad chat answer is visible. A bad long-running agent can produce a coherent result based on a false premise introduced two hours earlier.

Imagine a 40-shot documentary. Nine video jobs have finished; 31 are still running. If the media inventory exposes only settled jobs, the agent sees nine clips and reasonably concludes that nine clips are all it has. It may distribute those clips across the full timeline. The remaining clips arrive later but are never linked.

The model did not hallucinate. It reasoned correctly from an incomplete snapshot.

State must describe absence

An agent's state should represent not only what exists, but also what is expected:

• completed items;
• pending items;
• failed items;
• expected totals;
• partially settled batches;
• fallbacks and truncations already applied.

If a batch is incomplete, the state must make that impossible to mistake for completion. Silent fallbacks are especially dangerous because every later step treats the fallback output as authoritative.

For long runs, the artifact store—not the conversation—should be the source of truth. Workers should be disposable. A replacement worker must be able to read versioned state and continue without reconstructing the project from chat history.

Separate intent from consequences

The most useful architectural rule has been simple: the model proposes; deterministic systems decide what becomes real.

The model can propose a shot list, a narration edit, or a render. Code owns the consequences:

• validating the operation;
• checking permissions;
• reserving budget;
• writing committed state;
• starting paid work;
• settling actual cost;
• recording an idempotency key.

The model should never grant itself a capability, approve its own output, or author the amount shown on a billing card.

This also means distinguishing three questions that are easy to conflate:

1. Capability: Can the system perform this operation?
2. Permission: Has the user allowed it?
3. Budget: Can this production still afford it?

A refusal must name the real category. Otherwise, a model will often turn “the budget is exhausted” into the more plausible-sounding “this feature is unavailable.”

Put checks on the delivery path

Any quality check the agent chooses to run is optional in practice. The checks that protect the user must run outside the agent's control, directly on the path to delivery.

It also helps to separate two concepts:

• Validation: Is the project structurally well-formed?
• Audit: Does the production make sense as a finished video?

A video project can compile while omitting half its narration. A valid timeline is not necessarily a good or complete timeline.

Budget across the production

Workers are temporary, but the production is the unit the user approved. Retry limits, tool grants, and budgets therefore need production-level accounting.

If each replacement worker receives a fresh retry allowance, a limit of three attempts can become 30 attempts across ten workers.

For paid operations, the robust pattern is:

1. Estimate the operation in code.
2. Reserve the maximum approved amount.
3. Perform the work.
4. Settle at actual cost.
5. Refund the unused balance.
6. Use an idempotency key so retries cannot double-spend.

Do not wait for long jobs

The coordinator should not remain alive while video generation or rendering runs. It should end its turn and be woken when the job settles.

This makes a turn last seconds rather than hours. Deployments and worker crashes no longer interrupt the conceptual run. A new worker reads the same state and continues.

Build replay before you need it

Standard chat traces are not enough for a multi-hour run with thousands of observations. The most valuable debugging tool we built was state replay: reconstruct the project's artifact store at a specific revision, then run selection, validation, audit, and assembly logic offline.

When a production costs real money, replay lets you verify a diagnosis without paying to reproduce the entire failure.

The real agent moat

Better models help, but they do not solve dishonest state, duplicated spending, skipped checks, or non-resumable jobs.

The value of an agent is proportional to how long it can be left alone. A tool that needs intervention every ten minutes is sophisticated autocomplete. A tool that can take a video brief in the evening and deliver a trustworthy result in the morning starts to feel like a colleague.

Crossing that gap is primarily a systems-engineering problem.

The model proposes. Code and humans dispose. Nothing the model says should become money spent, state committed, or a video delivered without passing through a boundary outside the model.