In 2026, the pitch is everywhere: deploy an AI agent, walk away, let it run. We tried this. By day three of our first continuous deployment, the API bill had tripled and the pipeline was stuck in a retry loop, calling the same endpoint 400 times per hour. Nobody had told us the agent had no exit condition. According to McKinsey's State of AI in 2024, organizations broadly struggle to achieve measurable ROI from AI initiatives, particularly in autonomous deployments. That finding matched our experience exactly.
This is a retrospective on what we set out to build, what broke, what we learned, and the specific design decisions that turned a runaway cost center into something that actually earns its keep.
What We Set Out to Build
The goal was straightforward: a continuous orchestration system that would monitor incoming data, classify it, route it to the right downstream process, and handle errors without human intervention. Think of it as a traffic controller that never sleeps. We wanted it running on n8n, triggering a reasoning model for classification decisions, and writing results to a database for downstream consumption.
The appeal was real. Eliminating the manual triage step alone would free up hours per week. The architecture looked clean on a whiteboard: webhook in, LLM classifies, branch routes, done.
We were wrong about almost every assumption baked into that diagram.
What Happened: Three Failure Modes in Sequence
The first failure was the loop problem. Our classification node had no circuit breaker. When the downstream API returned a 429 rate-limit error, the pipeline retried immediately, hit the same limit, and retried again. The n8n execution log filled up in under two hours. We had built a system that responded to failure by accelerating into it.
The fix was mechanical: exponential backoff with a hard cap on retry attempts, plus a dead-letter queue for executions that exhausted retries. Not glamorous. Completely necessary.
The second failure was more expensive. We had assumed that sending every incoming event to a reasoning model was the right call. It wasn't. A large fraction of events were trivially classifiable by a simple conditional check: if the payload contains field X with value Y, route to path A. We were paying LLM inference costs for decisions a basic IF node could make in milliseconds. Routing simple cases through a conditional filter before they ever reached the LLM cut our inference spend materially, without touching classification accuracy on the complex cases that actually needed reasoning.
The third failure was the one I'm most embarrassed about, because it came from a careless API assumption. During our first Stripe product creation inside the pipeline, the API call included a recurring parameter set to null. We thought omitting the value was the same as omitting the field. It wasn't. Stripe created two prices: one correct one-time payment at $297, and one spurious monthly subscription at $297. We caught it before a customer was charged monthly for a one-time product, but it took a manual archive in the Stripe Dashboard to fix. Now our factory pipeline never includes the recurring field at all: not null, not false, just absent. The lesson generalizes: in any API integration running without human review, the difference between a missing field and a null field can create real financial consequences.
The Real Cost Structure Nobody Talks About
When founders calculate the cost of a 24/7 agent, they typically count LLM inference. That's the smallest part of the actual bill.
The full cost stack looks like this:
- Compute and hosting: The n8n instance, the database, the queue infrastructure. These run whether the pipeline is busy or idle.
- LLM inference: Highly variable. Spikes during error loops, during high-volume windows, and whenever a prompt grows longer than intended because context wasn't trimmed.
- Monitoring and alerting: You need something watching the watcher. Execution logs, error rate dashboards, and cost anomaly alerts are not optional for a system running unattended.
- Error recovery labor: Every failure that isn't handled automatically becomes a manual intervention. At 3am. This is the cost that doesn't appear in any invoice but shows up in founder burnout.
The honest tradeoff here is worth naming directly: a 24/7 autonomous pipeline is not a "set it and forget it" system. It is a system that trades active human labor for active monitoring infrastructure. If you don't have the monitoring in place, you haven't reduced your operational burden; you've just delayed it and made it more chaotic when it arrives. For very small teams without dedicated ops capacity, a hybrid approach, combining scheduled batch jobs with targeted real-time triggers, often delivers better reliability than a fully continuous system.
We've written more about the cost dynamics of running LLMs in production in our post on LLM load testing and cost problems, which covers how inference costs behave under realistic traffic patterns.
Lessons Learned: What the Working Version Looks Like
After rebuilding the pipeline twice, here's what the stable version actually contains.
Explicit trigger conditions with defined exit states. Every execution path has a terminal condition. The system knows when a task is done, when it has failed definitively, and when it should escalate rather than retry. This sounds obvious. It is almost never implemented correctly on the first build.
A tiered decision architecture. Simple conditional logic runs first. Only events that fail the conditional filter reach the reasoning model. This keeps inference costs proportional to actual complexity, not to volume.
Outcome metrics defined before deployment. We now write down, before shipping any continuous pipeline, what success looks like numerically: throughput targets, acceptable error rates, cost-per-execution ceilings. Without these, you have no way to know whether the system is working or merely running.
Hybrid scheduling for persistence-sensitive tasks. Frontier reasoning models handle complex classification well. They handle long-running task persistence poorly. For tasks that require state across multiple hours or days, we use scheduled n8n workflows with database checkpoints rather than trying to keep a single agent execution alive. The reasoning layer handles decisions; the scheduler handles continuity. Separating these concerns is what ForgeWorkflows calls agentic logic: the model reasons, the orchestration layer persists.
API field hygiene as a first-class concern. After the Stripe incident, we added an explicit review step to every new API integration: confirm which fields must be absent (not null, absent) when not applicable. This applies to payment APIs, CRM writes, and any integration where a default value has a meaningful side effect.
For a broader look at how these design principles apply across team workflows, our post on AI transformation and rewiring team workflows covers the organizational side of the same problem.
What We'd Do Differently
Start with a scheduled batch job, not a continuous trigger. Every pipeline we've built that started as a scheduled job and later became real-time was easier to debug, cheaper to operate, and faster to ship than the ones we designed as continuous from day one. The real-time requirement is almost always less urgent than it feels at the design stage. Prove the logic works in batch first.
Build the cost anomaly alert before the pipeline goes live. Not after the first runaway bill. The alert should fire when per-hour inference spend exceeds a defined threshold, and it should pause the pipeline automatically, not just notify. We now treat this as a deployment prerequisite, not an afterthought.
Treat the monitoring layer as a separate build, not a feature. Our early pipelines had monitoring bolted on. The stable ones have monitoring designed in parallel with the core logic. The execution log, the error rate dashboard, and the cost tracker are not optional extras; they are the system's nervous system. A pipeline you can't observe is a pipeline you can't trust, and a pipeline you can't trust will eventually cost you more than it saves.
Top comments (0)