DEV Community: Lars Winstand

A Claude Max billing error is what finally made me stop running agents on chat subscriptions

Lars Winstand — Thu, 15 Oct 2026 06:10:48 +0000

I didn’t stop using Claude because Claude was bad.

I stopped putting agent workloads on consumer chat plans because I got tired of pretending a chat subscription was infrastructure.

That distinction matters a lot more than people admit.

For interactive work, Claude Pro and Claude Max feel great. You pay a monthly fee, stop obsessing over token counts, open Claude.ai, and get work done. For solo thinking, writing, debugging, and one-off coding sessions, that experience is genuinely good.

But once you move from "help me think" to "do this job while I sleep," the whole setup starts to feel shaky.

A billing edge case was the thing that finally made me admit it.

Not a dramatic outage. Not a benchmark. Not a model-quality debate.

A billing problem.

And that was enough.

The real problem: annoyances become operational risks when you automate them

A weird usage pause in a browser tab is annoying.

A weird usage pause in an unattended workflow is a production problem.

A vague limit message is survivable when you can click retry.

It’s not survivable when an n8n workflow is halfway through 600 records and nobody is watching.

Some account-policy weirdness is tolerable when you’re brainstorming.

It’s unacceptable when OpenClaw or a Python worker is chaining tasks overnight.

That’s the whole argument:

consumer chat plans are optimized for human tolerance; agent systems need machine-readable failure modes.

Once I looked at it that way, the rest got obvious fast.

Anthropic already tells you where serious workloads belong

This part is hiding in plain sight.

Anthropic splits its products into different buckets:

Claude.ai / Claude Pro / Claude Max under consumer-style usage
Anthropic API / Claude Console under commercial developer usage

That product split is the tell.

You can argue about the philosophy, but the implementation is clear: if you are powering software, automations, workflows, or user-facing systems, the API side is the real substrate.

The consumer side is for humans using a chat app.

A lot of builders blur that line because modern chat apps are very, very good. Claude.ai is polished. ChatGPT is polished. They make it easy to forget that convenience and operational fitness are not the same thing.

They are not interchangeable.

What Claude Max actually gives you

Claude Max is attractive because it feels flat-rate.

That’s the hook.

But the language matters. It’s sold as giving you something like 5x or 20x more usage than Pro, not a deterministic compute budget.

For an interactive human, that can be fine.

For an agent, that’s a red flag.

Anthropic’s own docs say availability can vary based on things like:

message length
file size
conversation length
tool use
model choice
effort level
artifact creation
multi-step tasks like browsing or code execution

That list is basically a description of what happens as agents become more useful.

The more agentic your system gets, the less predictable your effective capacity becomes.

That is backwards for operations.

Operators want boring systems:

explicit limits
known caps
documented retries
clear failure modes

A human can work around fuzzy capacity. A workflow cannot.

Why API errors are better than chat weirdness

This was the thing I resisted for too long.

Metered APIs feel scarier at first because the costs are explicit.

But for unattended systems, explicit beats ambiguous every time.

APIs fail in ways machines can understand.

That means you can build real control loops.

Typical API-side failures look like this:

402 for billing issues
429 for rate limits
5xx / overload for transient service problems

Those are not nuisances. They are features.

If n8n gets a 429, you back off and retry.

If your worker gets a 5xx, you queue and retry.

If you hit a spend cap, you alert, fail over, or intentionally pause.

That is actual operations.

Here’s a minimal Python example with retries around a chat-completions-style API:

import os
import time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ["OPENAI_BASE_URL"],
)

def run_with_retry(messages, model="gpt-4.1"):
    delay = 2
    for attempt in range(5):
        try:
            return client.chat.completions.create(
                model=model,
                messages=messages,
                timeout=120,
            )
        except Exception as e:
            if attempt == 4:
                raise
            time.sleep(delay)
            delay *= 2

response = run_with_retry([
    {"role": "system", "content": "You are a precise coding assistant."},
    {"role": "user", "content": "Summarize this deployment log and flag failures."}
])

print(response.choices[0].message.content)

That kind of code is boring.

Boring is good.

Billing systems are part of your production stack

This is the lesson I think most AI builders learn late.

People obsess over model quality and ignore billing behavior.

That’s a mistake.

If billing can stop requests, billing is part of your runtime.

If usage enforcement can pause your workload, usage enforcement is part of your runtime.

If account state can break automation, account state is part of your runtime.

Once you accept that, consumer chat plans look a lot less appealing for agent execution.

Because they’re not designed to be operated like infrastructure.

The cost story gets weird as soon as agents get busy

A lot of people move to chat subscriptions because they hate token anxiety.

Fair. I hate it too.

But hiding the meter is not the same as solving the economics.

Once your workflows get heavier, the complexity comes back somewhere else:

long context windows
tool calls
multi-step chains
output-heavy transforms
retries
fan-out jobs
overnight runs

At that point, “flat-feeling” usage doesn’t remove operational complexity. It just makes it harder to reason about.

Here’s the practical comparison:

Option	What you’re really buying
Claude Max	A premium consumer chat experience with variable effective capacity, great for interactive use
Anthropic API	Metered programmable access with documented limits, spend controls, and machine-readable errors
OpenAI-compatible flat-rate API layer	Predictable monthly pricing for agents and automations, with API semantics developers can actually build around

That third category is where this got interesting for me.

Because the real problem wasn’t "Claude vs OpenAI" or "which model is smartest."

The real problem was: how do I run agents without babysitting token spend and without trusting a browser-first product to behave like infrastructure?

The migration path is easier than people think

If you’re still doing most of your work in chat apps, moving to API-driven workflows is less painful than it sounds.

1. If you already use the OpenAI SDK

You can usually keep your client code shape and swap the base URL.

That’s the easiest off-ramp from browser habits.

Example:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["STANDARD_COMPUTE_API_KEY"],
    base_url="https://api.standardcompute.com/v1"
)

response = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "system", "content": "You are a senior SRE."},
        {"role": "user", "content": "Classify these incidents by severity and probable root cause."}
    ]
)

print(response.choices[0].message.content)

That’s the appeal of an OpenAI-compatible API layer: minimal rewiring.

If your stack already speaks OpenAI, you don’t want a migration project. You want a config change.

2. If you live in n8n

This is where chat subscriptions really fall apart.

n8n wants explicit request/response behavior. It wants credentials, retries, branching, and observability.

That maps naturally to APIs.

A practical pattern in n8n looks like this:

Trigger on new records
Batch items
Call an OpenAI-compatible chat endpoint
On 429/5xx, retry with backoff
On billing or auth failure, route to Slack/email alert
Continue processing the rest of the queue

That’s a workflow.

A browser subscription can’t give you that shape.

3. If you’re using CLI agents or harnesses like OpenClaw

Tools like OpenClaw make it very easy to build heavier autonomous workflows.

That’s great, but it also means you hit infrastructure limits faster.

Quick-start tooling makes experimentation easy:

curl -fsSL https://openclaw.ai/install.sh | bash
npm i -g openclaw
openclaw onboard

The trap is assuming the thing that felt fine in interactive testing will hold up under unattended execution.

Usually, it won’t.

What I use instead for automations

My current opinion is simple:

use Claude Pro or Claude Max for interactive work
use APIs for anything unattended
use an OpenAI-compatible layer if you want portability and less migration pain
use flat monthly pricing if your biggest operational headache is token anxiety

That last point is why products like Standard Compute make sense for a certain class of builder.

If you’re running agents in n8n, Make, Zapier, OpenClaw, or custom workers, the painful part is not just model access. It’s cost unpredictability.

Standard Compute’s pitch is basically the thing I wish more vendors admitted people want:

predictable monthly pricing
no per-token billing drama
OpenAI-compatible API calls
support for automations and agent workloads
routing across models without forcing you to rebuild your stack

That’s a much better fit for production-ish automations than pretending a chat subscription is your compute layer.

A concrete dev setup

If I were setting this up from scratch for a small automation stack, I’d do something like this:

Environment

export OPENAI_API_KEY="your_standard_compute_key"
export OPENAI_BASE_URL="https://api.standardcompute.com/v1"

App code

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ["OPENAI_BASE_URL"],
)

resp = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "system", "content": "Extract action items and owners."},
        {"role": "user", "content": "Meeting notes: Alice owns the migration. Bob will review the API logs."}
    ]
)

print(resp.choices[0].message.content)

Operational rules

retry 429 and 5xx
alert on 401/402-style failures
cap concurrency at the workflow layer
log request IDs and job IDs together
keep prompts versioned in git
test with small batches before overnight runs

That is dramatically more trustworthy than “hopefully the chat plan still likes me at 2:13 AM.”

My actual takeaway

The weird lesson here is that predictability matters more than generosity.

I would rather have:

a clear spend cap than a vague promise of more usage
a hard billing error than a silent stop
a documented rate limit than fuzzy throttling
an API I can swap underneath my workflows than a chat UI I have to work around

Chat subscriptions optimize for the feeling of abundance.

Operators need the reality of control.

So yeah: a Claude Max billing error is what finally got me to stop running agents on chat subscriptions.

Not because Claude is bad.

Because once an agent touches production data, runs overnight, or powers automations in n8n, Make, Zapier, OpenClaw, or your own workers, billing, limits, retries, and compatibility stop being side details.

They become the whole game.

I finally cut AI context bloat by 80% and the fix was embarrassingly simple

Lars Winstand — Wed, 14 Oct 2026 14:12:07 +0000

A few months ago I watched an agent do something deeply cursed.

It had already fetched the data.
It had already parsed the data.
It had already stored the useful fields.

And then it pasted the entire raw tool response into the next model call anyway.

Full JSON blobs. Logs. Metadata. Useless fields. Everything.

The next step got slower.
The step after that got more expensive.
Then the model started doing that annoying thing where it answers the easy part of the task and quietly ignores the important part.

That was the moment the bug became obvious:

Most “context window problems” are not model problems.
They are state management problems.

Once you see that, you start spotting it everywhere in n8n flows, LangGraph agents, MCP tool loops, and custom OpenAI-compatible orchestration code.

The actual fix

Stop replaying raw tool output into every prompt.

Keep full results in structured state outside the model.
Send only:

a short summary
the exact fields needed for the next step
or the exact evidence slice if verification matters

That one change usually does more for agent quality than switching from one frontier model to another.

The anti-pattern

Here’s the bad version:

result = weather_api.get_forecast("Austin")

messages.append({
    "role": "user",
    "content": f"Use this API response to decide whether to send an alert:\n{result}"
})

If result contains 200 lines of hourly forecast data, provider metadata, timestamps, debug fields, and trace IDs, you just turned your prompt into a junk drawer.

Now the model has to:

find the relevant fields
ignore the irrelevant ones
remember prior steps
plan the next action
not get distracted by noise

Models can do this.
They are just not especially efficient at it.

The better pattern

Store the full result in app state.
Pass only the fields the next step needs.

result = weather_api.get_forecast("Austin")
state["weather_raw"] = result

weather_summary = {
    "temperature_f": result["current"]["temp_f"],
    "conditions": result["current"]["condition"],
    "wind_alert": result["alerts"].get("wind", False)
}

messages.append({
    "role": "user",
    "content": f"Decide whether to send an alert using this data: {weather_summary}"
})

Same workflow.
Smaller prompt.
Less distraction.
Better odds the model focuses on the decision instead of re-parsing junk.

Prompt size is not the same as prompt quality

A lot of builders confuse these two ideas:

the model can fit a long context
the model should receive a long context

Those are not the same.

A giant prompt is often just a badly managed state store wearing a trench coat.

Even when GPT-5, Claude Opus, Grok, Qwen, or Llama can technically accept the full history, performance still gets mushier as stale content piles up. Latency rises. Cost rises. Attention gets diffused.

This is why long-running agents often feel smart for 5 steps and weirdly flaky by step 20.

The 3 patterns, ranked

Approach	What happens in practice
Raw tool output in prompt	Biggest prompt growth. Highest distraction. Latency and cost climb every step.
Summarized tool result in prompt	Usually the best balance. Smaller prompt footprint, but bad summaries can hide edge cases.
Structured state outside prompt	Best default for multi-step agents. Small recurring prompts, cleaner reasoning, better orchestration.

My opinionated take: structured state outside the prompt should be the default.

If you are building anything more serious than a toy bot, replaying raw tool output into every step is lazy architecture.

What should live where?

This is the mental model I wish more agent builders started with:

Short-term conversational state stays attached to the thread.
Long-term facts and tool results live outside the prompt in a database or memory store.
The next model call gets only the smallest useful slice of that state.

That’s it.

Not complicated. Just disciplined.

Real examples

LangGraph: checkpoint state, don’t replay transcripts

LangGraph gets this right.

Instead of stuffing previous tool output back into every prompt, persist graph state and resume by thread.

from langgraph.checkpoint.postgres import PostgresSaver

DB_URI = "postgresql://postgres:postgres@localhost:5432/postgres?sslmode=disable"

with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
    graph = builder.compile(checkpointer=checkpointer)

That is the grown-up version of memory.

The model gets what the next node needs.
Postgres keeps the rest.

n8n: trim memory on purpose

n8n is refreshingly practical here.

If you are building AI workflows in n8n, you already have the tools to avoid context bloat:

Simple Memory
Redis
Postgres
Xata
Zep
Motorhead
Chat Memory Manager

The existence of Chat Memory Manager should tell you everything.

If a workflow platform ships a node specifically to inspect and reduce memory, then “just append to chat history forever” is obviously not the best pattern.

MCP: tool output is app state first, prompt material second

MCP makes this cleaner because tools already have structure.

Example tool definition:

{
  "name": "get_weather",
  "description": "Get current weather information for a location",
  "inputSchema": {
    "type": "object",
    "properties": {
      "location": { "type": "string" }
    },
    "required": ["location"]
  },
  "outputSchema": {
    "type": "object"
  }
}

That should push you toward a simple pattern:

call the tool
store the full structured result
pass only the needed fields into the next prompt

For example:

tool_result = {
    "temperature": "72F",
    "conditions": "cloudy",
    "wind_alert": False,
    "hourly": [...],
    "provider_metadata": {...},
    "trace_id": "abc123"
}

state["weather_result"] = tool_result

prompt_payload = {
    "temperature": tool_result["temperature"],
    "conditions": tool_result["conditions"],
    "wind_alert": tool_result["wind_alert"]
}

The prompt becomes a working note, not a warehouse.

Prompt caching helps, but it does not fix bad context hygiene

This part gets misunderstood a lot.

Anthropic prompt caching is useful when you keep reusing the same stable prefix.

That means things like:

system instructions
tool definitions
fixed documents
repeated scaffolding

It does not magically make a sloppy prompt good.

Claude caching only helps when the prefix is identical, and cache entries last 5 minutes by default. That’s great for repeated prefixes. It does nothing to make a giant raw tool dump more relevant.

If anything, caching can hide bad architecture for a while because the bill hurts less before quality problems become obvious.

Example:

import anthropic

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},
    system="You are an AI assistant tasked with analyzing literary works.",
    messages=[
        {
            "role": "user",
            "content": "Analyze the major themes in Pride and Prejudice."
        }
    ],
)

Use caching for stable prefixes.
Do not use it as an excuse to keep shoving junk into prompts.

A quick before/after refactor

Here’s what this looks like in a normal agent loop.

Before

def run_step(state, user_request):
    search_results = search_docs(user_request)
    crm_data = fetch_customer_record(state["customer_id"])

    prompt = f"""
    User request: {user_request}

    Search results:
    {search_results}

    CRM data:
    {crm_data}

    Decide what to do next.
    """

    return llm(prompt)

This works at first.
Then search_results gets huge.
Then crm_data gets huge.
Then every step drags around more baggage.

After

def run_step(state, user_request):
    search_results = search_docs(user_request)
    crm_data = fetch_customer_record(state["customer_id"])

    state["search_results_raw"] = search_results
    state["crm_data_raw"] = crm_data

    prompt_payload = {
        "user_request": user_request,
        "top_matches": [doc["title"] for doc in search_results[:3]],
        "customer_tier": crm_data["tier"],
        "open_tickets": crm_data["open_tickets_count"],
        "billing_status": crm_data["billing_status"]
    }

    prompt = f"Decide what to do next using this state: {prompt_payload}"
    return llm(prompt)

This version is less fragile.

And when the model needs exact evidence later, you can pull the raw state back in selectively.

When should you keep raw output in the prompt?

There are real exceptions.

Sometimes the next step genuinely needs exact source material.

Keep raw output in-context when you need:

legal or compliance citations
exact stack traces for debugging
code review on full diffs
extraction from messy text
verification against original evidence

But even then, pass the exact slice that matters.
Not the entire upstream universe.

The rule I use now

This is the heuristic that cleaned up most of my agent workflows:

If information is needed for reasoning, summarize it.
If it is needed for retrieval, store it.
If it is needed for verification, pass the exact slice.

That rule has saved me more latency, more cost, and more agent quality than swapping models ever did.

Why this matters even more when you pay per token

This problem gets expensive fast in production.

If your agent runs 24/7 in n8n, Make, Zapier, OpenClaw, or a custom loop, every extra blob you keep replaying becomes recurring cost.

That’s one reason teams end up obsessing over token usage dashboards instead of fixing the architecture.

A cleaner state design reduces prompt bloat at the source.

And if you are using an OpenAI-compatible endpoint with flat monthly pricing, this gets even nicer because you can optimize for agent quality and throughput instead of constantly asking whether one more tool call is going to spike the bill.

That’s the part I like about Standard Compute: it works as a drop-in OpenAI API replacement, so you can keep your existing SDKs and agent code, but stop thinking in per-token panic mode while you clean up orchestration.

The architecture fix still matters.
You just get to apply it without token anxiety hanging over every experiment.

Practical checklist

If your agent feels slower and dumber over time, check these first:

Are you pasting raw tool output into every next prompt?
Are you storing canonical results outside the model?
Are you summarizing tool results before the next reasoning step?
Are you passing exact evidence only when verification is required?
Are you using prompt caching only for stable prefixes?
Are you trimming memory in n8n or your orchestration layer?
Are you treating the model like a reasoning engine instead of a database?

If you fix just those, you will probably get a bigger improvement than another model bake-off.

I went looking for a model fix.

What I found was an architecture fix.

Most “context window problems” are really just us using chat history as a storage layer.

Once you stop shoving raw tool output back into the next prompt, agents get faster, cheaper, and noticeably less confused.

Not because the models changed.

Because we finally stopped making them carry our state management mistakes.

I thought GPT-4o got 80% faster, but it was just prompt caching messing with my benchmark

Lars Winstand — Sun, 11 Oct 2026 06:11:15 +0000

I got fooled by a benchmark that looked incredible.

Same agent workflow. Same model. Same code path.

Second run was dramatically faster.

My first reaction was the same dumb little hit of engineer dopamine most of us get: nice, we optimized something.

We didn’t.

The prompt prefix matched, the cache kicked in, and my “model improvement” was mostly reuse.

That sounds obvious after the fact. But a lot of LLM benchmark posts still miss it. People run the same agent twice, latency drops, and suddenly GPT-4o or Claude or Gemini is supposedly faster, cheaper, or more stable.

Usually the benchmark just got warm.

If you build agents in n8n, Make, Zapier, OpenClaw, or your own Python workers, this gets even easier to miss because so much of the request stays identical between calls.

The benchmark looked great right up until I checked usage

Agent workloads are basically built to trigger prompt caching.

A typical request includes:

a large system prompt
tool schemas
examples
conversation history
planner / critic prompts
structured output schemas

The user’s latest message is often the smallest part of the request.

The expensive part is the scaffolding.

So when the second run gets much faster, that doesn’t automatically mean the model improved. It often means the provider reused the big stable prefix.

For production, that’s good.

For benchmarking, it can absolutely lie to you.

What to log if you want numbers that mean anything

If you’re not logging cache-related usage fields, you’re guessing.

Here are the ones that matter:

Provider	What to look for
OpenAI Prompt Caching	`usage.prompt_tokens_details.cached_tokens`
Anthropic Prompt Caching	cache usage fields in the response usage object
Google Gemini Implicit Caching	`usage.total_cached_tokens` or Vertex AI `cachedContentTokenCount`
OpenRouter	sticky routing behavior and shared `session_id`

OpenAI is the easiest example.

Prompt caching is enabled by default on supported models. It kicks in for prompts over 1,024 tokens, and the response tells you how many prompt tokens were served from cache.

Example response shape:

{
  "usage": {
    "prompt_tokens": 2006,
    "completion_tokens": 300,
    "prompt_tokens_details": {
      "cached_tokens": 1920,
      "audio_tokens": 0
    }
  }
}

If cached_tokens goes from 0 on run one to 1920 on run two, you did not discover a magical optimization.

You discovered that the prefix matched.

That’s the story.

Why the second run also looked cheaper

Because caching changes both latency and cost.

OpenAI has been explicit about this: cached input is cheaper than uncached input on supported models.

Google says implicit cache hits on Vertex AI can get major discounts too.

So when someone says:

Gemini 2.5 Pro got way cheaper in our pipeline

my first question is not:

what changed in the model?

It’s:

did you log usage.total_cached_tokens?

If the answer is no, the pricing story is incomplete.

Quick Python example: log the cache fields or don’t trust the benchmark

Here’s a minimal OpenAI-compatible example.

from openai import OpenAI
import time
import json

client = OpenAI()

messages = [
    {"role": "system", "content": "You are a careful code review assistant."},
    {"role": "user", "content": "Summarize this repository architecture and list the top risks."}
]

def run_once():
    start = time.time()
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        temperature=0
    )
    elapsed = time.time() - start

    usage = resp.usage
    cached = 0
    if hasattr(usage, "prompt_tokens_details") and usage.prompt_tokens_details:
        cached = getattr(usage.prompt_tokens_details, "cached_tokens", 0)

    print(json.dumps({
        "latency_s": round(elapsed, 3),
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "cached_tokens": cached
    }, indent=2))

run_once()
run_once()

If the second call is faster and cached_tokens jumps, that result is warm-cache behavior.

Not raw model speed.

If you use curl, same rule

You don’t need a full benchmark harness to catch this.

curl https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "system", "content": "You are a careful code review assistant."},
      {"role": "user", "content": "Summarize this repository architecture and list the top risks."}
    ]
  }' | jq '.usage'

Run it twice.

Check whether cached token fields appear or increase.

That one step will save you from a lot of fake benchmark confidence.

Anthropic and Gemini can fool you just as easily

This is not an OpenAI-only problem.

Anthropic

Anthropic supports prompt caching with cache_control. Cache lifetime is usually around 5 minutes.

from anthropic import Anthropic

client = Anthropic()

resp = client.messages.create(
    model="claude-opus-4-1",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},
    system="You are an AI assistant...",
    messages=[
        {"role": "user", "content": "Analyze the major themes in Pride and Prejudice."}
    ]
)

That means follow-up requests can look much faster if the cacheable prefix is unchanged.

There’s also an annoying timing edge case: if your streamed response takes several minutes, you can accidentally fall out of the cache window and think the benchmark is random.

It isn’t random. Your timing is just sloppy.

Gemini

Gemini 2.5+ models also have implicit caching behavior.

So yes, the second call may be faster.

No, that does not mean Gemini “learned” your workflow.

If you’re on Vertex AI, inspect cached token fields. If you’re not logging them, you’re missing the main reason the benchmark changed.

OpenRouter can make the benchmark look cleaner than it really is

This one matters if you benchmark through OpenRouter.

OpenRouter uses sticky routing after a cached request so later requests for the same model and conversation can keep hitting the same provider-side cache.

That’s useful in production.

It’s bad for sloppy benchmarking because now the router is helping your benchmark stay warm.

Example payload:

{
  "model": "openai/gpt-4o",
  "messages": [
    {"role": "user", "content": "Summarize this codebase"}
  ],
  "session_id": "agent-run-42"
}

If later calls improve, you need to ask whether the model got better or whether OpenRouter kept sending you back to the same warm provider.

Those are very different explanations.

n8n can fake the win before the request even reaches the model

This is the part people really hate hearing.

Sometimes the model didn’t get faster.

Sometimes your workflow got shorter.

In n8n, nodes like Remove Duplicates can reduce how many items actually reach the LLM across runs.

So the benchmark story becomes:

n8n filtered duplicate items
fewer requests hit GPT-4o or Claude
later runs looked faster and cheaper
everyone credited the model

That’s not model improvement.

That’s upstream dedupe.

The same thing can happen in Make, Zapier, or custom queues if retries, filters, or batching behavior change between runs.

My benchmark rule now: separate cold from warm or don’t publish it

This is the simplest fix.

If your benchmark doesn’t separate cold and warm runs, it’s unfinished.

Here’s the checklist I use now.

1. Log cache metrics on every request

At minimum:

OpenAI: usage.prompt_tokens_details.cached_tokens
Gemini: usage.total_cached_tokens or cachedContentTokenCount
Anthropic: cache usage fields in the response usage object
OpenRouter: routing/session behavior

2. Hash the fully rendered prompt

Do not just version the user message.

Hash the final request body after templating, including:

system prompt
developer prompt
tool definitions
tool order
examples
conversation history
response schema

If any of those change, cache behavior changes too.

3. Benchmark cold and warm separately

Run one set with unique prefixes.

Run another set with stable prefixes.

Label them honestly:

cold-start latency
warm-cache latency

Do not average them into one vanity number.

4. Control the timing window

Cache TTL matters.

If your benchmark drifts across cache expiration windows, your numbers will drift too.

5. Verify the workflow didn’t reduce actual model calls

Before celebrating a cost or latency win, confirm:

no dedupe node removed items
no retry logic changed request count
no batching layer altered call volume
no router stickiness hid the real path

A practical benchmark harness pattern

This is the shape I’d recommend for agent teams.

import hashlib
import json
import time
from openai import OpenAI

client = OpenAI()

def prompt_fingerprint(payload: dict) -> str:
    raw = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(raw.encode()).hexdigest()

def run(payload: dict):
    fp = prompt_fingerprint(payload)
    start = time.time()
    resp = client.chat.completions.create(**payload)
    elapsed = time.time() - start

    usage = resp.usage
    cached = 0
    if getattr(usage, "prompt_tokens_details", None):
        cached = getattr(usage.prompt_tokens_details, "cached_tokens", 0)

    return {
        "fingerprint": fp,
        "latency_s": round(elapsed, 3),
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "cached_tokens": cached,
    }

payload = {
    "model": "gpt-4o",
    "messages": [
        {"role": "system", "content": "You are a strict API design reviewer."},
        {"role": "user", "content": "Review this OpenAPI spec and list breaking changes."}
    ],
    "temperature": 0
}

print(run(payload))
print(run(payload))

This won’t solve every benchmarking problem.

But it will stop you from calling a warm cache hit a model breakthrough.

The annoying truth: prompt caching is good, fake benchmark stories are not

Prompt caching is not cheating.

It’s one of the best features for real agent workloads.

If you run long-lived automations, coding agents, support bots, or multi-step workflows, you absolutely want big stable prefixes to be reused.

That’s a real production win.

The mistake is claiming you measured model quality, raw speed, or provider pricing when what you really measured was cache friendliness.

Those are different things.

Once you notice this, a lot of benchmark posts start looking suspiciously magical.

the second run is always faster
the giant agent gets “smarter” after turn three
Claude suddenly becomes cheaper in a multi-turn evaluator
Gemini “settles in” after a few calls

Maybe.

But usually the boring answer wins.

The prefix matched.
The router stayed sticky.
The workflow dropped duplicates.

Why this matters even more for teams building agents

If you’re building serious automations, you need predictable behavior, not benchmark theater.

That’s one reason I like what Standard Compute is doing.

Standard Compute gives you unlimited AI compute for a flat monthly price through an OpenAI-compatible API. So if you’re running agents in n8n, Make, Zapier, OpenClaw, or custom workers, you can keep the same SDK patterns without living in per-token panic.

That doesn’t remove the need for honest benchmarking.

It just means you can optimize for actual workflow performance instead of obsessing over every token spike from repeated agent calls.

For teams running lots of long-context, multi-step automations, that tradeoff makes a lot of sense.

The takeaway

If you benchmark repeated LLM calls without checking cache fields, your numbers are probably lying.

Log the usage fields.
Separate cold from warm.
Hash the full prompt.
Check the workflow layer.

Then publish the numbers.

Anything else is just benchmark fan fiction.

Our chatbot not replying in groups turned out to be a Slack plumbing bug, not an AI bug

Lars Winstand — Sat, 10 Oct 2026 22:11:44 +0000

If your bot works in DMs but goes weirdly silent in Slack channels, start by blaming your event handling.

Not GPT-5.
Not Claude.
Not your prompt.

We lost a bunch of time learning this the dumb way.

Our bot was great in 1:1 chats. In Slack DMs it could hold context, call tools, summarize results, and generally look like a competent AI teammate.

Then we dropped the exact same bot into a busy channel.

It became a ghost.

Sometimes it replied.
Sometimes it replied twice.
Sometimes the backend finished successfully and Slack showed nothing.
Sometimes it answered in the wrong thread.

That kind of failure is extra annoying because it makes you debug the wrong layer first.

We blamed the model.
We swapped GPT-5 for Claude.
We trimmed prompts.
We argued about whether Llama or Qwen would be more reliable in channels.

None of that mattered.

The real problem was that we treated Slack group chats like DMs, and Slack absolutely does not work that way.

The first bug: DMs and mentions are different event types

This is the first thing I’d check in any Slack bot.

A direct message to your app comes in as a message event with channel_type="im".
A mention in a channel comes in as app_mention.

Those are different event streams with different behavior.

DM example:

{
  "event": {
    "type": "message",
    "channel_type": "im",
    "channel": "D024BE91L",
    "text": "Hello hello can you hear me?"
  }
}

Channel mention example:

{
  "event": {
    "type": "app_mention",
    "channel": "C123ABC456",
    "text": "<@U0LAN0Z89> is it everything a river should be?",
    "ts": "1515449522.000016"
  }
}

That one distinction explains a lot of “works in DMs, fails in channels” bugs.

In DMs, the shape is simple: user talks to bot.

In channels, everything gets more fragile:

mention parsing
channel membership
permissions
thread routing
rate limits
duplicate events

If your code has one generic handleIncomingMessage() path for everything, there’s a decent chance that’s your bug.

Why it only broke in busy channels

Quiet channels hide bad architecture.

If your bot mostly lives in DMs or low-traffic channels, a synchronous handler can look fine for weeks.

Then one incident channel gets busy.
Five people mention the bot.
One tool call takes 8 seconds.
One GitHub request stalls.
One retry comes in.
And suddenly your “AI reliability issue” is obviously just broken event lifecycle handling.

Slack expects a fast HTTP 200 acknowledgment.

The practical rule is simple:

you have about 3 seconds to ack

If you don’t ack quickly, Slack may retry the event, and now you’re in duplicate-processing land.

This is the dangerous flow:

Receive app_mention
Call GPT-5 or Claude
Query your vector DB
Call GitHub
Call Jira
Summarize everything
Finally reply to Slack

That feels natural when you first build it.

It’s also how you build a bot that disappears under load.

The boring fix that actually worked

We moved all slow work out of the request path.

That means:

ack immediately
enqueue work
process in a background worker
post the final answer later

Here’s the shape in Slack Bolt for Python:

from slack_bolt import App

app = App(process_before_response=True)

def ack_fast(ack):
    ack("Accepted")

def run_long_process(respond, body, logger):
    user_text = body["event"]["text"]

    # slow work goes here
    # call model
    # call tools
    # build final answer

    respond("Completed")

That’s the idea, but in production I’d usually push this into a queue instead of doing everything inside the listener.

Something more like this:

from fastapi import FastAPI, Request, BackgroundTasks
import os
import json

app = FastAPI()

@app.post("/slack/events")
async def slack_events(req: Request, background_tasks: BackgroundTasks):
    payload = await req.json()

    # Slack URL verification
    if payload.get("type") == "url_verification":
        return {"challenge": payload["challenge"]}

    event = payload.get("event", {})
    event_id = payload.get("event_id")

    # 1. dedupe by event_id
    # 2. enqueue background work
    background_tasks.add_task(process_event, event_id, event)

    # ack immediately
    return {"ok": True}


def process_event(event_id: str, event: dict):
    # idempotency check here
    # model call here
    # tool calls here
    # post reply here
    pass

The important part is not the framework.

The important part is the split:

webhook path does validation + dedupe + enqueue
worker does the expensive stuff

Reliable Slack bots are event systems first and AI apps second.

The second bug: Slack can drop messages even when your backend succeeded

This one was nastier.

We assumed that if our worker finished and called chat.postMessage, the message would show up.

That is not safe in busy channels.

Slack rate-limits chat.postMessage at roughly 1 message per second per channel.

Per channel.

That means if your bot posts:

“thinking...”
“checking GitHub...”
“checking Datadog...”
“found 3 issues...”
“here’s the answer”

and three people mention it in the same thread at once, you’ve basically built a rate-limit machine.

The ugly part is that your logs can still look fine while users see silence.

What we changed

We stopped treating outbound messages like free writes.

The new rules were:

Ack immediately
Queue all work
Dedupe retries
Serialize outbound posts per channel
Throttle to about 1 msg/sec/channel
Prefer one solid answer over five cute updates

That improved reliability more than any prompt tweak.

Not because the model got smarter.
Because the plumbing stopped fighting the app.

Thread state is where a lot of bots quietly fall apart

In DMs, state is easy.

In channels, state is thread-shaped.

If you reply without the right thread_ts, your answer lands in the wrong place or becomes useless noise in the channel.

A lot of teams do this:

receive a new message
call conversations.replies
rebuild the whole thread
send that to the model
repeat forever

That works at first.
It’s also expensive, slow, and increasingly brittle.

A better pattern is to keep your own compact thread state.

Store:

channel
root thread_ts
normalized user turns
tool outputs
model outputs
a compact rolling summary

Use Slack history as recovery, not as your primary memory system.

Here’s the tradeoff:

Approach	What actually happens
Rebuild thread from Slack every turn	Easy to start, but slower, noisier, and more fragile under load
Maintain your own thread state	Slightly more engineering, much more reliable for long-running agents

If you’re running workflows in n8n, Make, Zapier, OpenClaw, or your own worker stack, this matters even more.

Those tools make it easy to ship a bot quickly.
They also make it easy to hide state bugs until traffic spikes.

A practical event pipeline that doesn’t embarrass you

If I were rebuilding this from scratch, I’d do it like this.

1) Split ingress by conversation type

Handle DMs and mentions separately.

def route_event(event: dict):
    event_type = event.get("type")
    channel_type = event.get("channel_type")

    if event_type == "message" and channel_type == "im":
        return "dm"

    if event_type == "app_mention":
        return "channel_mention"

    return "ignore"

Different paths should have different logic for:

permissions
mention parsing
reply routing
thread handling

2) Ack first, think later

Anything slow goes to background work.

That includes:

GPT-5 calls
Claude calls
embeddings/search
GitHub lookups
Jira lookups
Notion reads
Google Drive scans
internal tool calls

Your webhook should be boring.
Boring is good.

3) Treat retries as normal

Slack retries are not edge cases.

Store event_id and make processing idempotent.

Pseudo-code:

def process_event(event_id: str, event: dict):
    if already_processed(event_id):
        return

    mark_processing(event_id)

    try:
        handle_event(event)
        mark_processed(event_id)
    except Exception:
        mark_failed(event_id)
        raise

If you skip this, duplicate replies are just a matter of time.

4) Rate-limit outbound messages per channel

One queue per channel is a sane default.

Pseudo-code:

from collections import defaultdict
from queue import Queue
import time

channel_queues = defaultdict(Queue)

def post_with_throttle(channel_id: str, message: dict):
    q = channel_queues[channel_id]
    q.put(message)

    while not q.empty():
        next_msg = q.get()
        slack_client.chat_postMessage(**next_msg)
        time.sleep(1.0)

In real code you’d use a proper worker, lock, or async queue, but the design point stands:

throttle per channel, not globally

5) Prefer edits over spam

If you want progress updates, update one message instead of posting five new ones.

That usually looks cleaner for users and is friendlier to rate limits.

6) Own your thread state

Slack is a transport layer.
It should not be your source of truth.

Minimal thread record example:

{
  "channel": "C123ABC456",
  "thread_ts": "1712345678.123456",
  "participants": ["U111", "U222"],
  "messages": [
    {"role": "user", "text": "check prod errors"},
    {"role": "assistant", "text": "Looking into GitHub and Datadog"}
  ],
  "summary": "Investigating production error spike after deploy",
  "last_updated": "2026-10-10T12:00:00Z"
}

That gives you a stable context source for long-running agents.

Slack and Discord have the same class of failure

The labels differ, but the architecture lesson is the same.

Slack has:

app_mention
threads
chat.postMessage
channel-specific behavior

Discord has:

interactions
deferred responses
followups
edit windows

Different APIs, same rule:

respond fast, queue slow work, dedupe retries, and control outbound writes

If you do long-running agent work inline, both platforms will punish you.

Quick local debugging checklist

When a bot works in DMs but not in channels, this is the checklist I’d run in order.

Verify the incoming event type

Log the event shape.

import json

def debug_event(payload):
    print(json.dumps(payload, indent=2))

Check whether you’re actually receiving app_mention and not assuming all traffic is message.

Verify channel membership and scopes

Make sure the app is actually in the channel and has the scopes it needs.

Verify fast ack timing

Log how long your webhook takes before returning 200.

import time

start = time.time()
# validate + enqueue
elapsed = time.time() - start
print(f"ack path took {elapsed:.3f}s")

If that number is drifting upward, you’re moving work back into the request path.

Verify dedupe

Log event_id and retry headers.

If duplicate replies exist, you probably don’t have idempotency under control.

Verify thread routing

Make sure replies use the correct thread_ts.

Verify outbound rate limiting

If the bot is chatty in one channel, assume rate limits are part of the problem until proven otherwise.

Where this connects to AI infra

This bug looked like a model problem because model calls were the most visible slow step.

That’s common in agent systems.

People blame GPT-5, Claude, Grok, or tool-calling reliability when the real issue is orchestration:

bad queue design
bad retries
bad event routing
bad rate limiting
bad state management

That’s also why predictable AI infrastructure matters.

When you’re building bots, automations, or long-running agents, you want the freedom to offload the slow work without obsessing over every token or every retry path.

That’s the appeal of Standard Compute: you can keep your app OpenAI-compatible, swap in a flat-rate endpoint, and let agents run in the background without per-token cost anxiety creeping into every architecture decision.

Especially if you’re wiring together Slack, GitHub, Jira, Notion, n8n, Make, Zapier, or custom workers, the expensive part is usually not one single prompt. It’s the whole loop.

The annoying truth

Our bot never needed a better personality.
It needed better manners.

The real fix was not prompt engineering.
It was:

separate handling for message.im and app_mention
ack within 3 seconds
background workers for slow tasks
dedupe on retries
thread-aware replies
per-channel outbound throttling
state outside Slack

If your chatbot stops replying in groups while behaving perfectly in DMs, start with the boring plumbing.

That’s the stuff that actually fixes production bots.

My support bot stopped forgetting everything when I changed AI memory write triggers

Lars Winstand — Sat, 03 Oct 2026 22:09:39 +0000

I had a support bot that looked great in demos and acted like it had amnesia in production.

The failure mode was embarrassingly simple:

customer already rebooted the router three times
customer already confirmed the firmware version
customer already asked for email follow-up instead of SMS
bot asked them to do all of that again

The stack was not exotic:

n8n for orchestration
pgvector for long-term memory
an OpenAI-compatible API for model calls
GPT-5.4 for most runs
Claude Opus 4.6 for harder replies

The model had the transcript. The model had a big context window. The workflow had retrieval.

And it still behaved like it had never seen the case before.

The fix was not a better model.

The fix was changing when memory gets written.

The rule that fixed my support agent: write memory only on 5 event types — confirmed facts, preference changes, state-changing troubleshooting steps, decisions, and outcomes.

Once I did that, retrieval got smaller, cleaner, and actually useful.

The real bug: I stored chat history instead of case history

My first version wrote almost every turn into pgvector.

That included junk like:

"yes"
"still broken"
"I already tried that"
"ok"
"can you explain that again"

At first this feels safe. More data should mean better recall, right?

I think that is exactly backwards for support workflows.

For support bots, full-transcript memory is usually worse than event memory because it:

pollutes retrieval
increases repeated mistakes
bloats prompts
burns tokens reloading low-value context

If your workflow runs continuously in n8n, Make, Zapier, OpenClaw, or a custom agent loop, this gets worse over time. Every ticket adds more clutter. Every retrieval step gets noisier. Every prompt carries more baggage.

The bad fix I tried first

Naturally, I tried the wrong fix before the right one.

I changed the memory writer to summarize every conversation chunk before storing it.

That sounded smarter. It was not.

Now I had compressed noise instead of raw noise.

GPT-5.4 and Grok 4.20 both did the same thing overloaded memory systems usually do: they latched onto plausible but low-value details, then repeated steps that were already resolved.

This is the part a lot of people miss when building support automations:

Bigger memory is not better memory.

If your memory layer is sloppy, the model does not become more informed. It becomes more distractible.

The 5 memory write triggers that actually worked

I stopped writing every turn and only wrote memory on 5 event types.

1. Confirmed facts

Store facts only when they are explicitly confirmed.

Good examples:

Customer is on Netgear Nighthawk R7000
Account is on the Pro plan
ISP is Comcast
Device is running firmware 1.0.11

Bad examples:

Customer probably has an old router
This sounds like a DNS issue
User may be on the free tier

If the model inferred it, it does not belong in long-term memory.

2. Preference changes

These are easy to ignore and they matter constantly.

Examples:

Customer prefers email over phone
Do not restart devices during business hours
Contact only after 5 PM local time
Wants concise replies, not step-by-step explanations

A support bot can be technically correct and still feel broken if it forgets preferences.

3. State-changing troubleshooting steps

This was the biggest one.

Store the step only if it changed system state.

Examples:

Factory reset completed
DNS changed from ISP default to Cloudflare
Firmware rolled back from 1.0.11 to 1.0.9
Router moved from bridge mode to router mode

Do not store suggestions.

Bad memory:

Suggested factory reset
Recommended checking DNS
Asked customer to reboot

That is not state. That is conversation.

4. Decisions

Examples:

Escalate to Tier 2
Pause automation and wait for screenshot
Schedule on-site technician visit
Close loop pending customer response

Decisions shape what the next run should do. They belong in memory.

5. Outcomes

Examples:

Issue resolved after firmware rollback
Ticket closed unresolved due to inactivity
Packet loss persisted after modem replacement
Escalation accepted by network team

Outcomes stop the agent from reopening dead paths.

The rule in table form

Event Type	Store It?
Confirmed hardware model	Yes
Customer says "ok"	No
DNS actually changed	Yes
Bot suggested DNS change	No
Customer changed contact preference	Yes
Conversation summary of small talk	No
Escalate to Tier 2	Yes
Issue resolved after rollback	Yes

What the memory payload started to look like

Instead of stuffing transcript chunks into pgvector, I started storing structured events.

Something like this:

{
  "ticket_id": "T-18422",
  "event_type": "state_change",
  "timestamp": "2026-02-14T16:22:11Z",
  "summary": "Factory reset completed on Netgear Nighthawk R7000",
  "metadata": {
    "device": "Netgear Nighthawk R7000",
    "action": "factory_reset",
    "status": "completed"
  }
}

Or this:

{
  "ticket_id": "T-18422",
  "event_type": "preference_change",
  "timestamp": "2026-02-14T16:31:02Z",
  "summary": "Customer prefers email follow-up instead of SMS",
  "metadata": {
    "channel": "email"
  }
}

That gave retrieval something useful to rank.

A practical write filter

This is the kind of logic I wish I had started with.

export function shouldWriteMemory(event: {
  type: string;
  confirmed?: boolean;
  changedState?: boolean;
}): boolean {
  switch (event.type) {
    case "confirmed_fact":
      return !!event.confirmed;
    case "preference_change":
      return true;
    case "state_change":
      return !!event.changedState;
    case "decision":
      return true;
    case "outcome":
      return true;
    default:
      return false;
  }
}

And the retrieval side stayed aggressive about keeping context small.

export function buildMemoryContext(memories: MemoryRecord[]): string {
  return memories
    .sort((a, b) => b.score - a.score)
    .slice(0, 5)
    .map((m) => `- [${m.event_type}] ${m.summary}`)
    .join("\n");
}

That slice(0, 5) mattered more than I expected.

Not 50 records. Not the whole ticket history. Just the highest-signal events.

How this changes the prompt

Before:

Here is the previous conversation history:
[huge blob of transcript chunks and summaries]

After:

Relevant case memory:
- [confirmed_fact] Customer is on Netgear Nighthawk R7000
- [state_change] Factory reset completed
- [state_change] DNS changed from ISP default to Cloudflare
- [decision] Escalation to Tier 2 pending screenshot
- [preference_change] Customer prefers email follow-up

Do not repeat completed troubleshooting steps.
Continue from the current case state.

That second prompt is shorter and much harder for the model to misunderstand.

What changed in production

The behavior shift was immediate.

The bot stopped:

asking customers to repeat resolved steps
re-suggesting fixes that already failed
losing handoff context between runs
wasting tokens on giant memory payloads

It got better at:

continuing open tickets correctly
handling asynchronous follow-ups
escalating with cleaner summaries
respecting customer preferences

This also improved reliability in long-running workflows. If your agent is running all day in n8n or Make, bad memory rules compound. Good memory rules compound too.

Why this matters even more if you pay per token

This is not just a quality problem. It is also a cost problem.

Bad retrieval design quietly becomes an expensive habit:

larger prompts
more retrieval overhead
more retries after wrong answers
more model calls because the workflow keeps looping on already-resolved issues

If you are paying per token, noisy memory punishes you twice: once in quality and again on the bill.

This is exactly why I like using an OpenAI-compatible layer that does not make me obsess over every token while I tune workflows.

With Standard Compute, I can keep the same OpenAI SDK patterns, route across models like GPT-5.4, Claude Opus 4.6, and Grok 4.20, and run agent-heavy automations without watching a per-token meter all day.

That does not remove the need for good memory design. Nothing removes that. But it does make iteration a lot less annoying when you are testing retrieval, prompts, and multi-step automations in n8n or custom agent loops.

If you are building a support bot, my advice is simple

Treat memory like a timeline of state changes, not a backup of the conversation.

Support is not a creative writing task. It is closer to a state machine with human language wrapped around it.

Store:

what is true
what changed
what was decided
what happened next

Do not store every sentence just because you can.

That one change fixed more of my support bot's "memory" than switching models ever did.

If your agent keeps forgetting obvious things, I would look at memory write triggers before I blame the model.

TIL the best automations reduce context window by using cheap models for boring steps and saving Claude Opus for the hard part

Lars Winstand — Sat, 03 Oct 2026 06:10:16 +0000

I learned this the annoying way: I inspected a support-ticket automation that was doing cleanup, classification, extraction, summarization, and escalation decisions with the same expensive model.

It worked.

That was the problem.

When a workflow works, nobody asks whether it’s wasting money and context on glorified janitor tasks.

And most automations absolutely are.

If you’re building with n8n, Make, Zapier, LangChain, OpenClaw, or plain API calls, the highest-leverage optimization usually isn’t a better prompt.

It’s this:

use cheap, fast models for boring steps
use expensive reasoning models only for ambiguous or high-risk steps
shrink context before it reaches your strongest model

That pattern saves money, reduces context bloat, and usually makes the workflow easier to reason about.

The real waste: treating every step like frontier reasoning

A lot of teams say they’re building agents with GPT-5, Claude Opus, or Gemini.

Then you inspect the actual pipeline and half the calls are doing this:

classify this ticket
extract these fields
rewrite this into valid JSON
tag this document
summarize this thread
check whether required keys exist

That is not the same job as:

reason through an ambiguous escalation with conflicting evidence

Yet people keep paying frontier-model prices for both.

That makes no sense.

Anthropic’s pricing makes the argument pretty clearly:

Claude Haiku 4.5: $1 input / $5 output per 1M tokens
Claude Opus 4.6: $5 input / $25 output per 1M tokens

That’s a 5x spread.

So if your workflow starts with extraction, tagging, cleanup, or first-pass summarization, that’s where I’d optimize first.

Not the final reasoning step.

Context bloat is usually self-inflicted

The second problem is even more common: people keep sending giant prompts to the wrong model.

Typical anti-pattern:

take a messy email thread
add CRM notes
add internal comments
add a giant system prompt
send all of it to the best model
repeat for every step

So classification sees the entire novel.
Extraction sees the entire novel.
Summarization sees the entire novel again.

That’s backwards.

The cheap model should do the trimming.

Use a fast model to:

normalize text
dedupe repeated content
extract fields
compress long threads
convert unstructured junk into structured JSON

Then pass the smaller result to Claude Opus 4.6 or GPT-5 only if the task is actually hard.

That’s how you reduce context window pressure in practice.

Not with heroic prompt surgery. With better pipeline design.

The vendors are basically telling us to do this

Google is pretty blunt here.

The Gemini API pricing page offers Batch API pricing at a 50% cost reduction for supported models.

That’s basically Google saying:

stop using premium synchronous calls for bulk background work

Example pricing called out in the original research:

Gemini 3.8 Flash standard: $0.75 input / $3.75 output per 1M tokens
Gemini 3.8 Flash Batch: $0.375 input / $1.875 output per 1M tokens

If you’re summarizing 10,000 support conversations overnight, that should be batch work.
If you’re evaluating a single churn-risk enterprise escalation, that’s where premium reasoning belongs.

Anthropic sends the same signal with prompt caching.

If your automation repeats long system instructions or reusable context, Claude Opus 4.6 prompt cache hits/refreshes at $0.50 per 1M tokens can change the economics a lot.

Same idea:

don’t recompute what you can cache
don’t reason deeply about what you can preprocess cheaply

Routing is now an engineering primitive

This gets more interesting once routing becomes part of the API layer.

OpenRouter’s provider routing lets you optimize for:

price
throughput
max latency
fallbacks

That means routing is no longer just “I like Claude for writing.”

It becomes operational.

Cheap first. Fast second. Strongest only when needed.

A support workflow can prefer low-cost providers for first-pass classification, then escalate if:

latency spikes
output fails validation
confidence is low
the case matches a risky category

That’s much better than blind loyalty to one model vendor.

Example request shape:

{
  "model": "openai/gpt-5.4",
  "messages": [
    {
      "role": "user",
      "content": "Classify this ticket"
    }
  ],
  "provider": {
    "sort": "price",
    "allow_fallbacks": true,
    "preferred_max_latency": 2
  }
}

That little provider block matters more than most people think.

It means your workflow can optimize for cost, speed, and reliability at the same time.

The pattern I keep coming back to: cheap model -> validate -> escalate

If I had to recommend one pattern to most teams, it would be this:

cheap model does extraction or classification
validate the output
escalate only failures or ambiguous cases

That’s it.

This is the cleanest way to stop paying premium-model prices for routine work.

Example: support ticket triage

First pass:

classify urgency
extract account ID
detect billing / outage / legal / enterprise keywords
produce structured JSON

If the JSON validates and confidence is high, continue.
If validation fails or the case is risky, escalate to Claude Opus 4.6 or GPT-5.

Example schema:

{
  "urgency": "low | medium | high | critical",
  "category": "billing | bug | feature_request | cancellation | legal | other",
  "account_id": "string | null",
  "needs_human": true,
  "confidence": 0.91,
  "reason": "Customer mentions production outage and enterprise SLA"
}

Example: document extraction pipeline

Cheap model:

parse invoice, PDF, email, or form
extract fields to JSON
normalize dates, totals, IDs

Validation step:

required keys exist
number formats are valid
dates parse correctly
totals aren’t contradictory

Expensive model:

only repair malformed outputs
only reason over ambiguous or conflicting data

That’s better engineering, not just cheaper inference.

Practical orchestration examples

n8n pattern

n8n is good at this because you can split the workflow into explicit stages.

A simple shape:

Webhook node receives support ticket
Cheap LLM call extracts structure
Code node validates schema
IF node checks confidence / category / validation result
Strong model handles exceptions
Ticket gets routed to human or auto-resolved

Pseudo-flow:

Webhook
  -> LLM (cheap model)
  -> Code (validate JSON)
  -> IF (valid && confidence > 0.85 && category != legal)
      -> Auto-route
      -> Else -> LLM (Claude Opus 4.6 / GPT-5)

LangChain example

If you’re using LangChain, swapping models by step is straightforward.

from langchain.chat_models import init_chat_model

classifier = init_chat_model("anthropic:claude-3-5-haiku-latest")
reasoner = init_chat_model("openai:gpt-5")

raw_ticket = "Customer says their production workspace is down and legal will review the SLA."

classification_prompt = f"""
Extract JSON with fields:
- urgency
- category
- needs_human
- confidence
- reason

Ticket:
{raw_ticket}
"""

first_pass = classifier.invoke(classification_prompt)
print(first_pass)

Then validate before escalating:

import json

def should_escalate(payload: dict) -> bool:
    if payload.get("confidence", 0) < 0.85:
        return True
    if payload.get("category") in {"legal", "billing"}:
        return True
    if payload.get("urgency") == "critical":
        return True
    return False

payload = json.loads(first_pass.content)

if should_escalate(payload):
    final = reasoner.invoke(
        f"Review this support case and decide next action:\n\n{raw_ticket}\n\nFirst pass:\n{json.dumps(payload)}"
    )
    print(final)

Batch the boring stuff

If the user does not need the result right now, batch it.

Examples:

nightly summarization of support threads
CRM note cleanup
bulk document compression
asynchronous tagging jobs
historical conversation labeling

That’s exactly the kind of workload where lower-cost batch processing wins.

A simple stack I’d actually use

If I were building this today, I’d reach for something like:

Claude Haiku 4.5 for extraction, tagging, classification, and structured cleanup
Gemini Flash Batch for bulk asynchronous summarization
Claude Opus 4.6 or GPT-5 for exception handling and edge-case reasoning
OpenRouter for routing, fallback, and price-aware provider selection
n8n or LangChain for orchestration

That gives you a clean separation:

cheap model for discipline
expensive model for judgment

That’s the split most automations need.

Yes, this adds complexity

It does.

Using one strong model everywhere is simpler.

Routing adds:

more prompts
more failure modes
more testing
more evaluation work between steps

For small workflows, one strong model is often fine.

If your automation is:

low volume
rarely triggered
expensive to debug
not context heavy

then simplicity may be the better tradeoff.

But once repeated boring steps show up thousands of times a day, routing starts paying for itself very quickly.

One warning: cheap isn’t always cheap

Raw token pricing can lie to you.

Tokenizer differences, retries, malformed outputs, and escalation rates all matter.

The original post also pointed out that Anthropic notes newer tokenizers can produce about 30% more tokens for the same text in some cases.

So don’t stop at pricing tables.

Measure end-to-end workflow economics:

how often first pass succeeds
how often you escalate
how much context you removed before reasoning
how many retries happen
how much latency each stage adds
how much human review you avoided

That’s the real unit of cost.

Routing cheat sheet

Option	What it’s actually good for
Claude Haiku 4.5	Cheap extraction, tagging, classification, and structured cleanup at $1 input / $5 output per 1M tokens
Claude Opus 4.6	Hard reasoning, ambiguous validation, and exception handling at $5 input / $25 output per 1M tokens
Claude Opus 4.6 prompt caching	Repeated system prompts and reusable context, with cache hits/refreshes at $0.50 per 1M tokens
Gemini 3.8 Flash standard	Fast synchronous summarization or lightweight user-facing steps at $0.75 input / $3.75 output
Gemini 3.8 Flash Batch	Background summarization and bulk processing with 50% lower cost at $0.375 input / $1.875 output
OpenRouter routing	Price sorting, latency controls, throughput-aware routing, and fallbacks across providers

What I’d optimize first

If you want the least painful, highest-impact change, start with one of these.

1. Support triage

Send first-pass classification and tagging to a low-cost model.
Escalate only:

billing disputes
legal threats
enterprise accounts
critical incidents
low-confidence outputs

2. Document extraction

Use a cheap model to turn PDFs, emails, and forms into JSON.
Validate the schema.
Send only broken outputs to a stronger model.

3. Background summarization

Move async summaries, note cleanup, and bulk compression onto a batch-friendly model.
Keep synchronous decision-making on your strongest model.

4. Context cleanup before reasoning

This one is underrated.

Before the expensive model sees anything, run a cheap pass that:

removes duplicates
extracts facts
compresses long threads
strips irrelevant metadata

That alone can dramatically reduce context usage and improve downstream consistency.

My blunt rule

Use the cheapest model that can reliably produce valid structure.

Use the strongest model only where mistakes are actually expensive.

Batch anything the user does not need immediately.

Cache repeated prompts whenever the vendor supports it.

And measure escalations, not just token price.

That last one matters most.

Because the real question isn’t:

which model is best?

It’s:

which step deserves intelligence, and which step just needs discipline?

Once you look at automations that way, sending everything to one expensive model starts to look less like sophistication and more like laziness.

One more thing: if you’re tired of building this kind of routing while also babysitting per-token costs, that’s basically the problem Standard Compute is trying to remove. It gives you an OpenAI-compatible API with flat monthly pricing, so you can run automations and agents without constantly watching token spend. Same SDK shape, less pricing anxiety.

That part is especially appealing if you’re running n8n, Make, Zapier, OpenClaw, or custom agent workflows all day.

Quick test plan

If you want to try this this week, do this:

# 1) pick one workflow with repeated boring steps
# 2) identify the expensive model calls
# 3) split first-pass extraction/classification from final reasoning
# 4) add schema validation
# 5) log escalation rate
# 6) compare cost + latency + quality after 3-7 days

Metrics to log:

first_pass_success_rate
schema_validation_failure_rate
escalation_rate
avg_tokens_before_cleanup
avg_tokens_after_cleanup
p50_latency
p95_latency
human_review_rate

That gives you a real answer fast.

And in most cases, the answer is the same one I learned the annoying way:

your best model should not be doing your boring work.

I stopped letting one agent do everything and my tool calls got way less cursed

Lars Winstand — Fri, 02 Oct 2026 14:10:57 +0000

I realized I was asking one poor model to do four jobs badly.

Plan the task
Pick the right tool
Recover from failures
Write a clean final answer like nothing exploded

That setup looks great in a demo.

In production, it’s how you get a Slack bot that says “done” after failing on step two.

I’ve seen the same failure pattern in n8n, LangChain, OpenAI function-calling loops, and custom Python agents:

one giant GPT node with access to everything
retries handled by vibes
broken API responses treated like “reasoning problems”
too much context dumped into one loop

The fix was not a better prompt.

The fix was splitting orchestration from generation.

Stop asking one model to be the planner, dispatcher, retry loop, validator, and summarizer.

Give it supervision.

The problem: one agent is doing workflow engine work

A lot of agent setups are really orchestration engines wearing chatbot costumes.

We tell the model:

think through the task
choose tools
inspect outputs
detect failures
retry if needed
decide when it’s done
summarize the result

That is not one job.

That is a control plane.

And once you treat it like one, a lot of weird agent behavior suddenly makes sense.

What actually makes tool-calling agents flaky?

Most tool-use failures are not because GPT-5, Claude Opus 4.6, or Grok 4.20 are “bad at reasoning.”

They’re usually workflow bugs.

Common ones:

Too many tools exposed to one agent
No explicit retry policy when an API fails
Retrieval is unbounded, so the model gets flooded with junk
State from one task leaks into another
The same model is doing reasoning, validation, exception handling, and cleanup

That last one is the killer.

When the agent breaks, people usually respond by making the prompt longer.

More instructions.
More examples.
More all-caps warnings.

Sometimes that helps.
Usually it just creates a more polite failure.

The better question is:

Which parts of this workflow should not be left to model improvisation?

That question leads straight to the supervisor pattern.

The pattern: supervisor + narrow workers

This is the version I keep coming back to:

A supervisor plans the work
Specialist workers handle narrow tasks
The runtime handles retries, validation, tracing, and state transitions
A final step summarizes the result

The supervisor is not there to be clever.

It’s there to keep the system honest.

It decides:

which worker gets the task
which tools are allowed
whether a failure should retry or stop
what counts as success

That one change removes a shocking amount of chaos.

A minimal example with the OpenAI Agents SDK

The OpenAI Agents SDK already nudges you in this direction.

Minimal shape:

from agents import Agent, Runner

planner = Agent(
    name="Planner",
    instructions="Break work into steps and delegate to the right worker."
)

result = await Runner.run(planner, "Find the latest API pricing changes and summarize them")
print(result.final_output)

That snippet is tiny, but the important part is architectural:

Agent defines behavior
Runner owns the loop

That is already better than raw “model, keep calling tools until you feel done.”

If you add handoffs, validation, and tracing, you get a much safer setup without building everything from scratch.

Why narrow workers beat one giant generalist

Because tool choice gets worse as the tool list grows.

If you give one agent 12 tools, it will eventually get creative in a bad way.

A web research worker should do web research.
A file retrieval worker should do file retrieval.
A reporting worker should format output.

Not every worker needs access to every tool.

That’s how you reduce blast radius.

Example: bounded retrieval matters

One of the easiest reliability wins is limiting retrieval.

Don’t do this:

fetch everything
dump it all into context
hope the model sorts it out

Do this instead:

from agents import Agent, FileSearchTool, WebSearchTool

agent = Agent(
    name="ResearchWorker",
    tools=[
        WebSearchTool(
            search_content_types=["image", "text"],
            image_settings={"max_results": 3, "caption": True}
        ),
        FileSearchTool(
            max_num_results=3,
            vector_store_ids=["VECTOR_STORE_ID"],
        ),
    ],
)

Even here, I’d usually split this into two workers.

one worker for web
one worker for internal files
supervisor decides which one to call

The point is not elegance.
The point is fewer weird failures.

What breaks real agents is boring operational stuff

Most agent demos don’t die on reasoning.

They die on:

HTTP timeouts
429s from Notion
malformed Jira payloads
partial failures halfway through a long run
state that gets corrupted after a retry

This is why long-running workflow tooling matters.

If your automation touches Gmail, HubSpot, PostgreSQL, Slack, and an internal API, the main problem is often not “smarter prompting.”

It’s resumability.
Checkpointing.
Explicit failure handling.

That’s why tools like LangGraph are interesting.

Not because they make prompts prettier.
Because they treat agents like workflows.

Install is simple:

pip install -U langgraph

The bigger shift is mental.

You have to admit your agent is not just a chat loop.

It is a workflow with failure modes.

State is not one thing

This part gets ignored a lot.

People say “context” like it’s one big blob.

It isn’t.

There’s session state, tool state, conversation history, sandbox state, intermediate outputs, retry metadata, and final outputs.

When teams blur all of that together, agents get haunted fast.

You start seeing things like:

today’s invoice parser using yesterday’s customer record
a failed code execution polluting the next task
retries reusing stale tool outputs

A supervisor layer helps because it can control state transitions explicitly.

Something like:

open a fresh job session
pass only required state to each worker
checkpoint outputs after each step
isolate execution environments
clean up resources at the end

That is much easier to debug than one giant conversational loop with mystery memory.

Which stack should you use?

My blunt version:

Option	Best use case
OpenAI Responses API	Short-lived workflows where you want direct control over tool dispatch, loop logic, and state
OpenAI Agents SDK	Multi-agent workflows where you want handoffs, guardrails, sessions, tracing, and a managed runner
LangGraph	Long-running, stateful automations where durable execution and resume-after-failure matter most

And the tradeoff view:

Option	What you’re really signing up for
OpenAI Responses API	More plumbing, more control, less hidden behavior
OpenAI Agents SDK	Good default if you want supervisor-style structure without building every runtime piece yourself
LangGraph	More workflow engineering up front, much better operational behavior for complex automations

If you’re already building in Python, the OpenAI Agents SDK is a pretty reasonable place to start.

If your automations run long and touch lots of systems, I’d look hard at LangGraph.

If the workflow is short and deterministic, the Responses API may be enough.

When this is overkill

Not every automation needs a mini control plane.

If your agent:

uses one or two deterministic tools
runs for a short time
has simple success/failure logic

then a single loop with strict schemas may be fine.

I would not build a tiny bureaucracy for a support-ticket draft assistant that reads one knowledge base and writes one reply.

The point is not “always build a supervisor.”

The point is:

stop pretending one model call should be the supervisor.

My rule now

If an agent:

has more than a couple tools
touches external APIs
runs long enough that restart is painful
needs retries that aren’t trivial

then I assume it needs supervision.

Not because the models are weak.

GPT-5, Claude Opus 4.6, and Grok 4.20 can all be excellent workers.

But production reliability usually comes from workflow design, not model heroics.

The smartest model in the world will still have a bad day if you make it do planning, routing, validation, retries, cleanup, and summarization inside one fuzzy loop.

Make the model good at the thing it should do.

Keep orchestration in the runtime.

That one change made my tool calls a lot less cursed.

Practical checklist

If you want to clean up a flaky tool-calling agent this week, I’d start here:

[ ] Split planner/supervisor from worker tasks
[ ] Reduce tool access per worker
[ ] Add explicit retry rules outside the prompt
[ ] Bound retrieval results (ex: max_num_results=3)
[ ] Validate tool outputs with code, not instructions alone
[ ] Separate session state from task state
[ ] Add tracing so failures are inspectable
[ ] Decide what should retry vs escalate vs fail fast

That gets you much further than another 300 tokens of prompt engineering.

One more thing if you run agents all day

Once you start breaking agents into supervisors, workers, retries, and long-running loops, usage goes up fast.

That’s when per-token pricing starts messing with architecture decisions.

Teams end up optimizing for cost anxiety instead of reliability.
They avoid retries.
They shrink context too aggressively.
They hesitate to let automations run continuously.

That’s exactly why tools like Standard Compute are interesting for agent-heavy workflows.

It gives you an OpenAI-compatible API with flat monthly pricing, so you can run automations, supervisors, and worker loops without staring at token burn all day.

If you’re building in n8n, Make, Zapier, OpenClaw, or your own Python stack, predictable pricing changes the way you design systems.

You can afford to build the safer architecture instead of the cheapest-looking demo.

And for agents, that trade is usually worth it.

We thought our GPT-5.4 agent got lazier in production — it was a 3-bug workflow teaching it to quit

Lars Winstand — Thu, 01 Oct 2026 22:10:28 +0000

We thought our GPT-5.4 agent got lazier in production — it was a 3-bug workflow teaching it to quit

We had an n8n agent that looked great in staging.

It would:

search
pull docs
compare sources
verify claims
write a grounded answer

Then we shipped it.

In production, the same task started ending after one shallow pass.

Symptoms were exactly what people usually call “model laziness”:

shorter outputs
fewer tool calls
more confident wrong answers
less evidence of multi-step reasoning

Our first instinct was to blame GPT-5.4.

That was the wrong diagnosis.

The real issue was boring and very fixable:

retry cap dropped from 6 to 2
one n8n branch treated a partial answer as success
our OpenAI-compatible API path was rewarding the first acceptable-looking response instead of the best one

Once we fixed the workflow, quality came back.

That changed how I think about “lazy” agents in production.

Most of the time, the model did not suddenly get worse. Your orchestration started ending runs early.

The production symptom that fooled us

In staging, the agent trace looked like this:

retrieve sources
inspect docs
compare claims
verify one weak point
synthesize answer
return final output

In production, it looked more like this:

retrieve one weak source
write answer anyway

Same task class. Same model family. Very different behavior.

And because the final answer still looked polished, it passed casual review more often than it should have.

That is the dangerous part.

A broken agent rarely looks broken in an obvious way. It often looks efficient.

The 3 bugs that made GPT-5.4 look lazy

1) Retry cap dropped from 6 to 2

This was the biggest quality hit.

For simple classification, 2 retries can be fine.

For research, debugging, document synthesis, or anything with tool use, 2 is often a trap. One bad retrieval result plus one tool hiccup and the agent is out of budget.

Example of the kind of config drift that causes this:

{
  "task_type": "research",
  "max_retries": 2,
  "timeout_seconds": 20
}

That looks harmless until your workflow depends on search + fetch + verify.

2) A branch treated partial output as success

In our n8n flow, one formatting branch said, effectively:

if output matches schema
and output length is above a minimum
mark run successful

That means the agent could skip retrieval depth and still win.

This is how you accidentally train an agent to stop early.

Pseudo-logic:

const passed =
  isValidJson(response) &&
  response.answer.length > 280;

if (passed) {
  return "success";
}

That is not quality control.

That is a shallow-answer reward function.

3) The API path rewarded “acceptable” over “best”

This one is common in OpenAI-compatible stacks.

If your app accepts the first plausible answer and never checks whether the expected tool path ran, the orchestration layer starts selecting for speed, not depth.

That can happen whether you are routing to GPT-5.4, Claude Opus 4.6, or Grok 4.20.

The model is not “choosing to be lazy” in some abstract sense.

Your workflow is telling it:

if you look done quickly enough, you pass

Why agents look worse in production than in staging

Because production has constraints that staging often hides.

In a clean test harness, a model usually gets:

one prompt
predictable context
generous timeout
no weird branch logic
no flaky tools

In production, the agent sits inside a box made of:

retry limits
timeout settings
parser requirements
tool wrappers
success conditions
fallback branches
queue pressure
rate limiting

That box matters more than people want to admit.

A strong model inside a bad loop will look worse than a decent model inside a clean loop.

The fastest way to tell if the model is actually the problem

Run the same task through the same scaffold and change one variable at a time.

Not “same prompt, different environment.”

Actually the same scaffold:

same system prompt
same tool definitions
same retry budget
same max output settings
same evaluator
same API path
same success criteria

If you compare production n8n against a clean notebook script, you are not isolating the model.

You are changing the entire experiment.

The debugging signal that mattered most: stop reason

Not vibes.

Not output length alone.

Not “this answer feels thinner.”

Stop reasons told us far more than final-answer scoring.

For Anthropic agents, useful stop reasons include values like:

end_turn
max_tokens
tool_use
pause_turn

For OpenAI-compatible workflows, inspect whether:

the expected tool calls actually fired
the reasoning path was used when expected
the run ended because the model finished
or because your orchestration layer decided it had enough

If you only evaluate the final answer, you are debugging blind.

What convinced us it was the workflow, not GPT-5.4

We ran the same broken production scaffold against multiple model families.

What we saw:

GPT-5.4 looked bad under the production n8n flow
GPT-5.4 looked fine under the staging flow
Claude Opus 4.6 also looked bad under the broken production flow
Grok 4.20 looked bad too

That pattern matters.

When three strong models all become “lazy” in the same way, the workflow is usually guilty.

Here is the mental model I use now:

If this changes	Suspect
One model regresses, others stay stable	model or provider issue
All models regress under one workflow	orchestration bug
Output gets shorter after retry/timeout changes	early stopping
JSON validity improves while answer quality drops	parser-first reward problem

The false leads we chased first

We spent too long blaming the model layer.

Our guesses were reasonable:

maybe GPT-5.4 regressed
maybe the OpenAI Responses API settings changed behavior
maybe reasoning effort was too low
maybe output token limits were clipping the answer

Those are all real failure modes.

They just were not the main problem here.

The actual issue was simpler:

staging rewarded grounded completion
production rewarded acceptable formatting

Agents optimize for whatever your workflow rewards.

If your automation says “close enough,” GPT-5.4, Claude Opus 4.6, and Grok 4.20 will all start looking suspiciously eager to be done.

Workflow patterns that create fake laziness

These are the ones I would audit first.

Parser-first design

If the main objective is valid JSON, many agents will satisfy the parser before they satisfy the task.

Example smell:

if (schema.safeParse(output).success) {
  return success;
}

That should almost never be the whole success condition for a research task.

Direct-answer escape hatches

If n8n, Make, Zapier, OpenClaw, LangGraph, or your custom loop allows a final answer before retrieval or verification, expect shallow completions.

For research-class tasks, tool use often should not be optional.

Tiny retry budgets on multi-step tasks

This one is everywhere.

People use one global retry budget for everything:

max_retries: 2

That might be fine for:

classification
extraction
light transformations

It is usually bad for:

web research
code debugging
document comparison
multi-source synthesis

Evaluating only the final message

This is the worst one.

If you only score the final text, you hide:

skipped tool calls
failed searches
parser shortcuts
premature exits
timeout-driven summaries pretending to be conclusions

My strong opinion: this single habit causes teams to think they are comparing models when they are actually comparing orchestration mistakes.

What we changed

We did not switch models.

We changed the workflow.

Fix 1: restore retry budget

We moved the retry cap back from 2 to 6 for research-class tasks.

agent_profiles:
  classification:
    max_retries: 2
  research:
    max_retries: 6
  debugging:
    max_retries: 6

Fix 2: require retrieval for research tasks

We added a hard gate.

If the task is research, at least one retrieval step must happen before a run can pass.

Pseudo-code:

function validateRun(run) {
  if (run.taskType === "research" && run.toolCalls.search < 1) {
    return { ok: false, reason: "missing_required_retrieval" };
  }

  if (!run.outputSchemaValid) {
    return { ok: false, reason: "invalid_schema" };
  }

  return { ok: true };
}

Fix 3: stop treating formatting as quality

We changed success conditions so a run could not pass on formatting alone.

That meant checking trajectory, not just output shape.

Fix 4: compare traces, not just answers

We reviewed traces side by side in our observability stack and checked stop reasons across both the OpenAI-compatible path and the Anthropic path.

That made the difference obvious fast.

A practical checklist for debugging “lazy” agents

If you think your production agent got worse, this is the order I would check things in.

1) Diff staging vs production config

diff staging-agent.yaml production-agent.yaml

Look for changes in:

retry caps
timeouts
token/output limits
tool availability
branch logic
fallback behavior
evaluator thresholds

2) Inspect stop reasons

Log them explicitly.

{
  "run_id": "abc123",
  "model": "gpt-5.4",
  "stop_reason": "end_turn",
  "tool_calls": 0,
  "task_type": "research"
}

If research tasks are ending with zero tool calls and still passing, that is your bug.

3) Count tool calls per task type

A simple metric catches a lot:

SELECT
  task_type,
  AVG(tool_call_count) AS avg_tool_calls,
  AVG(retry_count) AS avg_retries,
  AVG(output_chars) AS avg_output_chars
FROM agent_runs
WHERE created_at >= NOW() - INTERVAL '7 days'
GROUP BY task_type;

If tool-call counts collapse after a deploy, investigate the workflow before blaming the model.

4) Compare multiple model families under the same scaffold

This is where an OpenAI-compatible API setup helps.

If GPT-5.4, Claude Opus 4.6, and Grok 4.20 all fail the same way under one loop, the loop is probably broken.

5) Score trajectory, not just answer text

Track things like:

retrieval attempted
verification attempted
tool errors encountered
stop reason
retries used
output grounded in cited material

Why this matters more when you run lots of agents

This kind of bug gets expensive fast when you are running automations all day.

Not just in dollars. In bad outputs, hidden regressions, and wasted debugging time.

Teams running agents in n8n, Make, Zapier, OpenClaw, or custom OpenAI-compatible stacks usually hit the same wall:

they start by asking “which model is best?”

Then eventually they realize the more useful question is:

“what exactly is our workflow rewarding?”

That is also why predictable API infrastructure matters.

When you can swap models without rewriting your stack, compare traces cleanly, and run lots of evals without per-token anxiety, it gets much easier to find orchestration bugs instead of arguing about vibes.

That is a big part of why Standard Compute is interesting for agent teams: it is a drop-in OpenAI-compatible API, so you can keep your existing SDKs and workflows, route across GPT-5.4, Claude Opus 4.6, and Grok 4.20, and test agent behavior without every debugging session turning into a billing event.

For teams running automations 24/7, flat monthly pricing is not just a finance preference. It changes how aggressively you can evaluate, compare, and fix agent systems.

The rule I use now

Before blaming GPT-5.4 for getting lazy:

inspect the trajectory
check stop reasons
compare the exact same scaffold
verify required tool steps happened
make sure your success condition is rewarding quality, not just completion

Sometimes a model really does regress.

More often, production taught your agent that quitting early is the winning move.

I thought my n8n LLM bug was OpenAI and it turned out to be 5 boring workflow failures

Lars Winstand — Wed, 30 Sep 2026 14:11:15 +0000

I thought my n8n LLM bug was OpenAI and it turned out to be 5 boring workflow failures

I had one of those bugs that makes you immediately blame the model provider.

The workflow was still spinning.
The user had no response.
There was no obvious error.
And the AI step was the last interesting thing in the execution.

So I did the obvious bad move: I blamed OpenAI, retried the run, and piled more retries onto a workflow that was already jammed.

The thing that broke the illusion was simple:

The provider logs showed the model had already responded.

n8n was the thing still hanging.

That was the moment I had to admit the bug was in my orchestration layer, not in the model.

If you run AI agents in n8n, Make, Zapier, OpenClaw, or your own queue-based workflow stack, this matters a lot:

A lot of “the LLM stopped responding” bugs are not model outages. They’re worker backlog, Redis payload issues, missing timeouts, BullMQ stalls, or agent loops.

I lost hours to this once. You probably don’t need to.

The fast sanity check that changed how I debug this

Before touching prompts, retries, or provider dashboards, compare two timestamps:

When the provider says the request completed
When your workflow says the execution completed

If the provider finished at 12:01:04 and your client is still hanging at 12:01:45, the model is not your first suspect anymore.

That’s an orchestration problem until proven otherwise.

Why n8n makes this look like a model outage

When you’re tired and staring at a stuck execution, it feels like the model call is frozen.

But in queue mode, the path is longer than people remember:

the client stays connected to the main or webhook instance
the execution may run on a worker
the result has to come back through Redis
the workflow still has to serialize, pass data around, and finish cleanly

Any slowdown in that path can look exactly like “OpenAI stopped responding.”

That’s why this bug is so annoying. The model can be done, but the user still sees a spinner.

The 5 workflow bugs I now check before blaming the model

These are the five issues I found, in the order I’d check them now.

1) n8n was waiting on a worker, not on the LLM

What it looked like

The execution sat long enough that it felt like a dead model call.

Why it fooled me

From the outside, these look identical:

model is slow
worker is backed up

If all you have is a hanging HTTP request, your brain picks the AI step every time.

What I checked

I compared provider-side completion time with workflow completion time, then checked queue delay and worker throughput.

What proved it

The provider had already returned the response.
The worker path had not finished processing it.

What I’d do now

Check worker pressure first.

# Example things to inspect in your deployment
# Docker
docker ps
docker logs n8n-worker --tail 200

# Kubernetes
kubectl get pods
kubectl logs deployment/n8n-worker --tail=200

If you’re using queue mode, look for:

too few workers
workers stuck on other jobs
queue backlog spikes
long execution times unrelated to provider latency

If upstream finished and your workflow didn’t, the queue is guilty until proven innocent.

2) Redis was carrying a huge response through the queue

What it looked like

Small runs worked.
Large runs felt flaky or frozen.

Why it fooled me

Big outputs make people suspect token generation or streaming first.

That’s reasonable, but wrong surprisingly often.

What I checked

I compared small-output runs vs large-output runs and looked at where the delay appeared.

What proved it

The slowdown tracked payload size, not model latency.

That means the bottleneck was response handoff, serialization, queue transport, or downstream processing.

Practical fix

Stop shipping giant blobs through your workflow if you don’t need to.

Bad pattern:

{
  "messages": [...],
  "full_context": "very large text...",
  "tool_results": [...],
  "raw_model_output": "huge output..."
}

Better pattern:

{
  "response_id": "resp_123",
  "summary": "short result for next step",
  "storage_key": "s3://bucket/run-123/output.json"
}

If your agent produces large intermediate artifacts:

store them in object storage
pass references, not full payloads
trim chat history aggressively
avoid unnecessary JSON expansion between nodes

A giant JSON blob moving through Redis is one of the most boring ways to fake an LLM outage.

3) I had no real timeout, so the workflow could wait forever

What it looked like

Requests didn’t fail cleanly.
They just kept hanging.

Why it fooled me

No error means people assume “the model is still thinking.”

That’s a terrible default assumption.

What I checked

I audited timeout settings in three places:

the HTTP client
the n8n workflow or AI node
any proxy or webhook layer in front

What proved it

There was no meaningful upper bound forcing the run to fail fast and surface the actual bottleneck.

Practical fix

Set explicit timeouts everywhere.

Example Node.js request with a timeout:

const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 45000);

try {
  const res = await fetch("https://api.openai-compatible-endpoint.com/v1/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": `Bearer ${process.env.API_KEY}`,
    },
    body: JSON.stringify({
      model: "gpt-5.4",
      messages: [
        { role: "user", content: "Summarize this ticket" }
      ]
    }),
    signal: controller.signal,
  });

  const data = await res.json();
  console.log(data);
} finally {
  clearTimeout(timeout);
}

And in workflow systems, I’d strongly recommend:

per-step timeouts
execution-level timeouts
retry limits
dead-letter handling for repeated failures

A timeout is not just for protection. It’s a debugging tool.

4) BullMQ stalled because the worker blocked the event loop

What it looked like

Jobs were retried or marked stalled even though the code looked fine at first glance.

Why it fooled me

The agent appeared to stop mid-task, and the LLM call was the easiest thing to blame.

What I checked

I looked for CPU-heavy work inside the same Node.js worker process handling queue jobs.

Typical offenders:

giant JSON parsing
expensive regex work
PDF extraction
big Markdown transformations
synchronous loops over large payloads

What proved it

The worker blocked the event loop long enough for BullMQ to treat the job as stalled.

If you block the event loop for 30+ seconds, that’s not an OpenAI problem.
That’s your worker design.

Minimal example of the kind of thing that hurts

// Bad: CPU-heavy work in the queue worker process
worker.process(async (job) => {
  const huge = JSON.parse(job.data.bigPayload);

  // pretend this is expensive
  for (let i = 0; i < 500000000; i++) {
    // blocking the event loop
  }

  return huge;
});

Better options

move CPU-heavy work to a separate service
chunk large transforms
use worker threads for CPU-bound work
keep queue workers focused on orchestration, not heavy processing

Useful checks:

# Watch CPU and memory
htop

# If containerized
docker stats

If your queue worker is doing too much non-I/O work, your “LLM outage” may just be a blocked Node process.

5) LangGraph was looping until recursion limits kicked in

What it looked like

The agent kept “working” but never gave a useful final answer.

Why it fooled me

From the user’s perspective, a loop and a hang feel the same.

No answer is no answer.

What I checked

I traced graph transitions and counted repeated states.

What proved it

The graph was revisiting the same path until recursion limits stopped it.

What I’d do now

Instrument state transitions aggressively.

Pseudo-example:

visited = []

def log_state(state_name, state):
    visited.append(state_name)
    print({
        "state": state_name,
        "visited_count": len(visited),
        "recent_path": visited[-5:]
    })

And add hard guards:

max iterations
repeated-state detection
explicit terminal states
tool-call limits

If your agent can re-enter the same state forever, it’s not autonomous. It’s a polite infinite loop.

My current debugging order for “LLM stopped responding”

This is the checklist I wish I had used first.

1. Check provider status page and request logs
2. Confirm whether the provider actually completed the request
3. Compare provider completion timestamp with n8n/BullMQ/LangGraph timing
4. Check queue backlog and worker health
5. Inspect payload size and response handoff path
6. Verify timeouts at client, workflow, and infrastructure layers
7. Look for event loop blocking or CPU-heavy worker tasks
8. Trace agent loops and recursion behavior
9. Only then blame the model

That order is less dramatic than posting “OpenAI is down again,” but it gets to the truth faster.

A practical example: compare provider logs to workflow timing

If you’re using an OpenAI-compatible endpoint, your app code probably already gives you enough hooks to do this.

Example:

const startedAt = Date.now();

const response = await client.chat.completions.create({
  model: "gpt-5.4",
  messages: [{ role: "user", content: "Classify this support ticket" }],
});

console.log({
  providerReturnedAtMs: Date.now() - startedAt,
  responseId: response.id,
});

Then compare that with:

n8n execution duration
worker completion time
webhook response time
downstream node timing

If provider time is 4 seconds and user-visible completion is 45 seconds, the model is not your biggest problem.

Why this gets expensive fast in agent workflows

This is the part people underestimate.

False outage diagnosis is expensive.

Every blind retry can:

clog your queue more
duplicate downstream work
increase latency for unrelated jobs
burn more model usage
make the original bug harder to see

That last part matters a lot if you’re paying per token and debugging high-volume automations. Retries during incident response can turn into their own bill.

That’s one reason I like using OpenAI-compatible endpoints that are easy to swap in and out. If you can test the same workflow against another compatible provider without rewriting your app, it becomes much easier to separate provider issues from orchestration issues.

And if your automation stack runs constantly, flat-rate compute is honestly a better fit than per-token pricing. During debugging, retries and test runs stop feeling like financial damage.

That’s the appeal of something like Standard Compute:

OpenAI-compatible API
works with existing SDKs and HTTP clients
useful for n8n, Make, Zapier, OpenClaw, and custom agent workflows
flat monthly pricing instead of per-token billing

For teams running agents all day, that’s not just a pricing detail. It changes how safely you can test, retry, and debug.

The actual lesson

The lesson is not that OpenAI, Anthropic, or Grok never fail.

They do.

The lesson is that workflow bugs impersonate model outages extremely well.

So now my default assumption is:

If the provider already answered, the bug is probably mine.

That assumption has saved me more time than any prompt tweak ever has.

If you’re debugging hanging LLM workflows in n8n or another automation stack, start with the response path, not the model.

That’s usually where the real problem is hiding.

I thought my cheap AI workflow was fine until I counted 3,000 tiny calls

Lars Winstand — Wed, 30 Sep 2026 06:11:07 +0000

I didn’t get burned by one giant GPT-5 prompt.

I got burned by a workflow that looked cheap.

You know the kind:

fetch a batch of records
classify each one
extract a field or two
write a short summary
move on

Lead enrichment. Support triage. CRM cleanup. Scraped page normalization.

Nothing fancy. Nothing that looks like it should trigger budget panic.

And that’s exactly why it’s dangerous.

The trap: every step is cheap, but the workflow isn’t

The mistake is simple: people inspect one AI call at a time.

They say:

“It’s just a classifier.”
“It’s only extracting pain points.”
“The summary is two sentences.”

All true.

But nobody multiplies.

Here’s a totally normal n8n flow:

Fetch 1,000 leads
-> Loop Over Items (Batch Size: 1)
-> OpenAI node: classify company
-> OpenAI node: extract pain points
-> OpenAI node: draft outreach angle

That is already:

1,000 records * 3 AI steps = 3,000 model calls

And that’s before:

retries
validation passes
fallback prompts
confidence scoring
second-model QA

This is not an edge case. This is a standard automation pattern.

If you build with n8n, Zapier, Make, or custom worker queues, you’ve probably done this already.

Tiny prompts can be the expensive part

Not just in dollars.

Also in:

request throughput
rate limits
queueing behavior
retry storms
operational complexity

A lot of teams focus on token count because that’s what pricing pages train you to do.

But provider limits are usually not just about tokens.

OpenAI separates RPM and TPM for a reason. You can be nowhere near your token-per-minute cap and still hit request-per-minute limits. Their docs explicitly call out the idea that if your RPM is 20, then 20 requests of only 100 tokens each can still max you out.

That changes the architecture discussion.

If your workload is “analyze one giant contract,” token cost is the problem.

If your workload is “touch 8,000 CRM rows and make 3 tiny decisions on each,” request multiplication is usually the real problem.

Why this fools people in testing

Because staging lies.

A test run on 20 records looks cheap.

Then someone points the workflow at:

the whole HubSpot table
the full Zendesk queue
a scraped Apify dataset
a backlog of support tickets

And suddenly the cost shape changes.

Not because prompts got bigger.

Because you turned on a machine that makes tiny calls thousands of times.

Quick way to estimate whether your workflow is actually cheap

I now do this before I trust any AI automation:

items_per_day=1000
ai_steps_per_item=3
retry_rate=0.1
validation_calls_per_item=1

base_calls=$((items_per_day * ai_steps_per_item))
validation_calls=$((items_per_day * validation_calls_per_item))
retry_calls=$(python3 - <<'PY'
items=1000
steps=3
retry_rate=0.1
print(int(items * steps * retry_rate))
PY
)

echo "Base calls: $base_calls"
echo "Validation calls: $validation_calls"
echo "Retry calls: $retry_calls"

Even rough math is enough.

If the answer is “we’re making 4,000 to 10,000 model requests a day,” you do not have a tiny workflow.

You have a high-frequency AI system.

The official fixes are real, but they solve specific problems

I’m not anti-OpenAI Batch API or anti-Anthropic prompt caching.

Both are good.

But they are not universal fixes for “my automation explodes into thousands of micro-calls.”

1) OpenAI Batch API: great if you can wait

OpenAI’s Batch API is legitimately useful.

It offers a 50% discount versus synchronous calls, and OpenAI explicitly positions it for jobs like large-scale classification and embeddings.

That maps well to:

nightly enrichment
backfills
offline cleanup jobs
large async processing queues

Example request shape:

{"custom_id":"request-1","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-3.5-turbo-0125","messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"Hello world!"}],"max_tokens":1000}}

But there are tradeoffs:

asynchronous only
can take up to 24 hours
no streaming
not a drop-in answer for real-time routing

If your support workflow needs to classify a ticket now, Batch is not your answer.

2) Anthropic prompt caching: amazing when the prefix repeats

Anthropic prompt caching is also very real.

When you have a large repeated prompt prefix, it can cut both latency and cost dramatically.

That’s excellent for:

long system prompts
many-shot examples
repeated document context
“chat with a book” style workloads

Example:

import anthropic

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},
    system="You are an AI assistant tasked with analyzing literary works.",
    messages=[
        {"role": "user", "content": "Analyze the major themes in Pride and Prejudice."}
    ],
)

But caching helps when requests share stable context.

It does not magically fix workloads where every row is different:

every Salesforce lead is unique
every Zendesk ticket is unique
every scraped page is unique

You can’t cache uniqueness.

The real question: what cost problem do you actually have?

Most teams skip this and jump straight to model comparisons.

That’s backwards.

First figure out the shape of your workload.

Option	Best fit
OpenAI synchronous API	Real-time flows where latency matters, but you still deal with per-token billing and RPM/TPM limits
OpenAI Batch API	Large asynchronous jobs where lower cost matters more than immediate completion
Anthropic prompt caching	Repeated prompt prefixes, long shared context, and workloads that benefit from cache reuse
Per-item multi-step automation in n8n, Make, or Zapier	Easy to build, easy to underestimate, and very likely to multiply request count fast

If your workload is a few giant prompts, token pricing is the main issue.

If your workload is thousands of tiny calls, the issue is usually a mix of:

request count
retries
fan-out per item
provider rate limits
automation platform task limits
engineers getting conservative because every extra step feels billable

That last one matters more than people admit.

Per-token pricing changes how people design automations

This is the hidden tax.

Teams start making worse technical decisions because they’re trying not to trigger more model calls.

I’ve seen teams do all of these:

Remove QA steps because “it’s probably fine.”
Skip classifying every inbound support ticket, so humans still sort manually.
Avoid enriching lower-priority leads because the long tail feels too expensive.
Collapse separate steps into one messy prompt because another call feels too costly.

That is not clean engineering.

That is workflow design shaped by billing anxiety.

And the model bill is not the only meter.

You may also be managing:

Zapier task limits
n8n execution volume
OpenAI RPM and TPM
Anthropic spend caps
fallback routing logic across providers

At some point this stops being a prompt engineering problem and becomes systems design.

What a lot of automations really are: tiny AI committees

This is the mental model that finally made it click for me.

A lot of record-level automations do not make one AI decision.

They make a committee.

For one item, you might do:

classify intent
extract fields
normalize to JSON
score confidence
draft a response
run safety review

Every call is defensible.

Together, they behave like a swarm.

That’s why I’m increasingly opinionated about this:

For high-volume operational workflows, pricing model matters almost as much as model quality.

Not because GPT-5, Claude Opus, Grok, Qwen, or Llama are bad.

Because once your team stops fearing each micro-call, you build better automations:

cleaner step separation
better validation
more second-pass QA
longer-running agents
fewer hacks to save pennies

What I’d do in practice

If I’m building a real system, I’d break the problem down like this.

Use batching when:

the job is asynchronous
latency does not matter
the workload is large and predictable
you want the cheapest possible processing path

Use caching when:

the prompt prefix is large and repeated
many requests share the same context
latency and cost both benefit from reuse

Rethink pricing architecture when:

each item triggers multiple small calls
workflows run continuously
retries and validation are common
the team is hesitating to add useful AI steps because of cost anxiety

That third category is where a lot of agent and automation teams actually live.

A practical workflow audit you can run this week

If you have an n8n, Make, Zapier, or custom agent workflow, map it like this:

workflow_audit:
  items_per_day: 1000
  ai_steps_per_item: 3
  average_retries_per_100_calls: 12
  validation_calls_per_item: 1
  fallback_model_enabled: true
  real_time_steps:
    - classify_ticket
    - route_priority
  async_steps:
    - nightly_summary
    - enrichment_backfill
  repeated_prompt_prefixes:
    - support_policy_context
    - extraction_schema

Then answer these questions honestly:

How many items arrive per day?
How many AI nodes run per item?
How many retries happen in production?
How many validation or fallback calls happen after failures?
Which steps truly need real-time latency?
Which steps could be batched?
Which prompts actually share enough prefix to benefit from caching?

That exercise usually reveals one of two stories.

Story A: your problem is giant repeated context

If that’s true, prompt caching can help a lot.

Story B: your problem is death by 10,000 tiny calls

If that’s true, you need to stop evaluating cost one prompt at a time.

You need to think in workflow volume.

Where Standard Compute fits

This is exactly why products like Standard Compute are interesting for agent and automation workloads.

If you’re running lots of small calls across n8n, Make, Zapier, OpenClaw, or custom workers, flat-rate unlimited compute changes the design space.

Instead of asking:

“Can we afford one more validation step?”
“Should we skip QA on lower-value records?”
“Can this agent run all day?”

You can build the workflow you actually want.

Standard Compute is a drop-in OpenAI-compatible API, so you can usually swap it into existing SDKs or HTTP clients without rebuilding your stack. Under the hood it routes across models like GPT-5.4, Claude Opus 4.6, and Grok 4.20, with batching, prompt optimization, and throttling designed for exactly this kind of high-frequency automation work.

That matters if your bottleneck is not one giant prompt.

It matters if your bottleneck is thousands of tiny decisions.

My takeaway

Before you optimize prompts, count calls.

Before you compare GPT-5 vs Claude, count calls.

Before you celebrate a cheap staging run, count calls.

Most teams think their cost problem is “big prompts are expensive.”

A surprising number actually have a different problem:

small prompts, repeated constantly, inside workflows that looked harmless.

That was my mistake.

If you’re building AI automations, don’t price a single request.

Price the swarm.

My headless AI server setup stopped dying when I got boring about queues and 30-second watchdogs

Lars Winstand — Tue, 29 Sep 2026 14:13:11 +0000

I realized our setup was fake-stable the first time a background agent stayed “running” while doing absolutely nothing.

No crash.

No stack trace.

Docker said the container was up. The process existed. CPU was low. Memory looked fine.

Meanwhile, webhook jobs were stacking up like a sink full of dishes.

That was the week I stopped trusting "container is running" as a useful definition of healthy.

If you’re moving from a laptop-and-good-vibes setup to a real headless AI server for always-on agents, this failure mode shows up fast.

Usually the stack looks something like this:

n8n or a custom webhook service
Stripe / GitHub / Discord / Slack webhooks
cron jobs
long-running GPT-5 or Claude tasks
maybe OpenClaw or another gateway in front
a few Docker containers and a prayer

Everything looks fine until one worker hangs without technically dying.

The fix was not a new framework.

It was three boring things:

separate ingress from execution
make jobs replay-safe
put workers behind one OpenAI-compatible endpoint

That’s the whole post.

The architecture mistake that caused most of the pain

The biggest mistake was letting webhook intake, cron scheduling, and LLM-heavy execution all happen in the same process.

That works on a laptop.

It gets weird in production.

A slow Claude call delays a Stripe webhook.
A giant PDF parse blocks the event loop.
A CPU spike from OCR makes your Discord bot miss heartbeats.
One stuck worker makes the whole service look haunted.

The clean split is the same pattern n8n uses in queue mode:

one main instance handles timers and webhooks
jobs get pushed into Redis
separate workers pull jobs and execute them
shared state goes into a real database

That separation matters more than most teams think.

Ingress should stay lightweight.
Execution should be allowed to be slow, retryable, and disposable.

A minimal version of the pattern

[Stripe/GitHub/cron/webhooks]
            |
            v
    [main ingress service]
            |
            v
        [Redis queue]
            |
    -------------------
    |        |        |
| --- | --- |
    v        v        v
[worker] [worker] [worker]
    |        |        |
| --- | --- |
    ---------+--------
            |
            v
   [OpenAI-compatible endpoint]
            |
            v
 [GPT-5.4 / Claude Opus 4.6 / Grok 4.20]

If you’re using n8n, this is basically why queue mode exists.

Also: once you do this, local-only state starts breaking things.

That means:

SQLite stops being a serious option for shared execution state
local filesystem storage becomes a problem for binary data
PostgreSQL and S3-compatible storage start looking very reasonable

This is the boring infrastructure tax for not losing jobs.

A queue does not save you from bad job design

A queue gives you another chance.

That’s it.

Whether that second chance helps or hurts depends on idempotency.

If a worker does this:

charge a card
send an email
write to Salesforce
crash before ack

...then the queue may redeliver the job.

If the task is not replay-safe, you just built a duplicate side-effect machine.

This is not theoretical. It’s how these systems behave.

With Celery, for example, tasks can be redelivered if the worker dies before acknowledgment. Features like acks_late help, but they do not magically make your side effects safe.

The real fix is application-level:

store idempotency keys
record processing state in a database
make external side effects safe to retry

Stripe will absolutely teach you this lesson

Stripe’s webhook model is basically a giant sign that says: expect duplicates, expect retries, design accordingly.

If your endpoint is down, Stripe retries undelivered events for up to 3 days.

You can also recover manually by listing undelivered events.

Example:

curl -G https://api.stripe.com/v1/events \
  -u "sk_test_...:" \
  -d ending_before=evt_001 \
  -d "types[]=payment_intent.succeeded" \
  -d "types[]=payment_intent.payment_failed" \
  -d delivery_success=false

The important part is not the curl command.

The important part is that your system has to know whether it already processed evt_001.

A simple pattern is a table like this:

CREATE TABLE processed_events (
  event_id TEXT PRIMARY KEY,
  event_type TEXT NOT NULL,
  processed_at TIMESTAMP NOT NULL DEFAULT NOW()
);

Then your worker logic becomes:

async function handleStripeEvent(event) {
  const alreadyProcessed = await db.processed_events.findUnique({
    where: { event_id: event.id }
  });

  if (alreadyProcessed) {
    return { ok: true, duplicate: true };
  }

  await doSideEffects(event);

  await db.processed_events.create({
    data: {
      event_id: event.id,
      event_type: event.type,
    }
  });

  return { ok: true };
}

That pattern is not glamorous.

It is also the difference between “retries are safe” and “why did we send 4 receipts?”

Docker restart policies are useful, but they did not solve the real problem

Yes, use restart policies.

They help with crashes.

docker run -d --name worker --restart unless-stopped my-worker-image

But restart policies only help when the process exits.

My actual problem was worse:

the process was alive
the container was alive
the worker was not making progress

That’s a different class of failure.

Docker does not know your Node.js event loop is wedged.
Docker does not know your Python worker is stuck in a bad call.
Docker does not know your queue heartbeat stopped.

That’s why “container is up” is not enough.

BullMQ was the first tool that made this obvious

BullMQ has a much more useful idea of health than Docker does.

It marks jobs as stalled when an active worker stops renewing its lock.

By default, that check is around 30 seconds.

That is exactly the kind of signal I needed.

Because a worker that is still running but hasn’t renewed a lock in 30 seconds is not healthy in any meaningful sense.

A basic BullMQ worker looks like this:

import { Queue, Worker } from 'bullmq';
import IORedis from 'ioredis';

const connection = new IORedis(process.env.REDIS_URL);

const queue = new Queue('jobs', { connection });

const worker = new Worker(
  'jobs',
  async job => {
    console.log(`Processing job ${job.id}`);

    // your LLM / automation work here
    return await runTask(job.data);
  },
  { connection }
);

worker.on('completed', job => {
  console.log(`Job ${job.id} completed`);
});

worker.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed`, err);
});

And a retry config with jitter should be your default, not an afterthought:

await queue.add('test-retry', { foo: 'bar' }, {
  attempts: 8,
  backoff: {
    type: 'fixed',
    delay: 1000,
    jitter: 0.5,
  },
});

That jitter: 0.5 matters a lot.

Without jitter, a worker fleet tends to fail together and retry together.

That’s how a small outage turns into a retry storm.

CPU-heavy jobs can make a healthy-looking worker useless

This one bit me hard.

If you run CPU-heavy work in the same Node.js process that needs to keep queue heartbeats alive, you can create stalls without crashing anything.

Examples:

OCR
PDF parsing
image conversion
local embedding generation
giant JSON transforms

The worker process is technically alive.

But queue progress says otherwise.

That’s why sandboxed processors or separate worker processes are worth it.

If a task can block the event loop, isolate it.

What should supervise what?

The best answer I found was: use multiple layers, but give each layer one job.

Option	What it’s actually good at
systemd watchdog + service restart	Detects hung processes via heartbeat instead of only exit codes. Great for VM or bare-metal workers.
Docker restart policy	Restarts exited containers. Good default. Weak for hung-but-still-running workers.
BullMQ or Celery	Decouples ingress from execution, supports retries and redelivery, and tolerates worker death if jobs are replay-safe.

My ranking for real-world usefulness:

queue semantics
stall detection / watchdogs
container restart policies

Container restarts save you from crashes.

Queues save you from reality.

The other thing that cleaned this up: one OpenAI-compatible endpoint

This was a much bigger deal than I expected.

Before, different workers were calling different providers directly:

one service used OpenAI
another used Anthropic
another used Groq
somebody had an OpenRouter integration
retry logic was duplicated in multiple places

That gets ugly fast.

Especially when every layer retries differently.

OpenAI-style clients already need backoff for 429s and transient failures. If your queue retries jobs and your model client also retries aggressively, you can easily create a retry storm from both sides.

Putting all workers behind one OpenAI-compatible endpoint fixed a lot:

one auth pattern
one SDK shape
one timeout policy
one retry policy
one place to route requests between models
one place to observe failures and rate limits

That architecture is a great fit for AI agents and automations.

It also makes provider swaps much less painful.

Why this matters for teams running always-on agents

If you have:

n8n workers
Make or Zapier handoffs
custom Node/Python agents
webhook-driven automations
internal tools firing LLM jobs all day

...then the model layer should not be reimplemented in every service.

A single OpenAI-compatible endpoint gives you a clean boundary.

This is exactly why products like Standard Compute are interesting for this setup.

You point your workers at one endpoint, keep your existing OpenAI-compatible SDKs, and avoid wiring provider-specific behavior into every queue consumer.

The other practical benefit is cost predictability.

When agents run 24/7, per-token billing gets annoying fast. A flat monthly model is a lot easier to reason about when you’re scaling background jobs, retries, and automations across a worker fleet.

For this kind of architecture, that matters more than people admit.

The catch with a single endpoint

A single endpoint also centralizes failure.

That is real.

If your gateway is down or misconfigured, every worker feels it.

So if you do this, you still need:

health checks
request timeouts
circuit breakers where appropriate
good logs and metrics
fallback behavior for critical paths

Still, I would take one well-observed gateway over a zoo of half-maintained provider clients every time.

Especially for teams that want interchangeable workers.

When this is overkill

If your setup is tiny, don’t cargo-cult a distributed system.

If you have:

one internal script
no external webhooks
no concurrency
no long-running jobs

...then a single process with systemd, strict timeouts, and a dead-letter path may be enough.

You do not need Redis, BullMQ, PostgreSQL, S3, and six workers to rename files once an hour.

But once you add:

Stripe or GitHub webhooks
concurrent jobs
long-running GPT-5 / Claude tasks
multiple team members expecting reliability

...the “simple” setup stops being simple.

It becomes fragile.

And fragile always looks cheap right before it gets expensive.

The baseline I trust now

If I were rebuilding a headless AI server setup today, this is the baseline:

1) Separate ingress from execution

webhooks and cron hit a lightweight main service
execution happens in workers pulling from Redis via BullMQ or Celery

2) Make every job replay-safe

store idempotency keys
record processing state in a database
assume duplicates will happen

3) Use supervision that can detect hangs

Docker restart policies for crashes
BullMQ stalled-job detection for queue workers
systemd watchdog where OS-level supervision makes sense

4) Put model access behind one OpenAI-compatible endpoint

one client shape
one retry policy
one place for routing, auth, quotas, and observability

5) Move shared state off local disk

PostgreSQL instead of SQLite for multi-worker setups
S3-compatible storage instead of local files when workers share binaries

6) Tune retries once

queue retries and model-client retries should cooperate
add jitter so workers do not stampede together

A practical checklist

If you want something actionable, start here:

# 1. run Redis

docker run -d \
  --name redis \
  --restart unless-stopped \
  -p 6379:6379 \
  redis

# 2. split your app into:
#    - ingress service
#    - worker service
#    - shared database

# 3. add health checks that measure progress, not just process existence

# 4. make every external side effect idempotent

# 5. point all workers at one OpenAI-compatible endpoint

If you do only those five things, your setup will already be less fragile than a surprising number of “production” agent stacks.

The weirdly comforting part

The most reliable headless AI server setup I’ve used is boring on purpose.

Not the fanciest agent framework.
Not the prettiest architecture diagram.
Not the stack with the most logos in it.

Just:

queue in the middle
watchdog on the worker
one OpenAI-compatible endpoint in front

That setup assumes workers will freeze.
It assumes webhooks will retry.
It assumes jobs will replay.
It assumes somebody will eventually ship CPU-heavy nonsense into a process that was supposed to stay responsive.

That’s why it works.

And if you’re running always-on AI agents, “nothing broke at 3 a.m.” is a much better definition of success than “the container was still up.”

My Google AI API 500 errors stopped being scary when I stopped retrying the whole workflow

Lars Winstand — Mon, 28 Sep 2026 22:10:33 +0000

I knew something was wrong when one flaky Gemini call turned into:

3 duplicate HubSpot updates
1 duplicate Jira issue
4 Slack messages
and a very annoyed ops channel

At first it looked like a normal "Google AI API is throwing random 500s" problem.

It wasn’t.

The real bug was that we were retrying the entire automation instead of retrying the model call.

That distinction matters a lot once your workflow has side effects.

If you’re seeing intermittent 500, 502, 503, or 429 errors from Gemini or Vertex AI, the fix usually is not “add more retries everywhere.” The fix is to move the retry boundary.

The first thing that surprised me: Google was already retrying

If you use the Gemini Python SDK, Google already retries transient failures by default.

That includes transient 429 and 5xx responses, with exponential backoff.

So if your production automation is still blowing up after “we added retries,” one of these is probably true:

You’re using raw HTTP or an automation HTTP node, so you don’t actually have safe retry behavior.
You wrapped the entire workflow in retry logic, which replays every side effect.

That second one is the expensive mistake.

If your workflow does this:

fetch lead
build prompt
call Gemini
write to PostgreSQL
update HubSpot
send Slack message
retry whole thing on failure

...then one transient model failure becomes a duplicate generator.

The model error is annoying.

The replay damage is worse.

What a “random 500” usually means in practice

A lot of teams treat 500 as "Google is broken."

Sometimes that’s true.

But with Gemini and Vertex AI, 500 can also mean overload, dependency failures, shared-capacity pressure, or quota-related behavior that doesn’t show up as a clean 429.

That’s why these incidents feel spooky in production:

one run succeeds
the next one fails
the retry half-works
then downstream systems are in a weird state

If you only watch for explicit rate limits, you miss the actual pattern.

Your "random 500s" may really be burst traffic, project-wide contention, or spend throttling wearing a different mask.

My opinionated take: workflow-level retry is lazy engineering

If a workflow has side effects, retrying the whole thing is the wrong default.

The retry boundary should sit around the model call, not around everything before and after it.

This is the rule I trust now:

collect data once
call the model in an isolated step
write side effects once
replay only the failed step

If you don’t do that, you get the usual mess:

duplicate HubSpot or Salesforce writes
duplicate Gmail or SendGrid emails
duplicate Slack or Discord alerts
duplicate Zendesk or Jira tickets
half-finished database state

At that point it’s not really an LLM problem anymore.

It’s a workflow design problem.

The pattern that fixed it for us

The cleanest version of this is to isolate the LLM call into its own sub-workflow or worker.

Instead of this:

fetch records
build prompt
call Gemini
update CRM
send Slack alert
retry whole workflow if anything fails

Do this instead:

fetch records
persist a replay-safe payload
call a sub-workflow or worker that only handles Gemini
return structured output
perform downstream writes once
route failures to an error handler

That one change removes most of the blast radius.

If Gemini throws a transient 5xx, only the model step retries.

Your CRM write doesn’t happen twice.

Your Slack alert doesn’t fire twice.

Your upstream data fetch doesn’t get repeated for no reason.

A concrete n8n version

n8n is actually pretty good at this if you use the primitives it gives you.

Useful pieces:

Error Workflows
Error Trigger
sub-workflows
execution data
log streaming

A decent shape looks like this:

Main workflow

Fetch records
Normalize payload
Save payload + execution ID
Call sub-workflow for Gemini
Receive structured result
Write downstream side effects once

Error workflow

Start with Error Trigger
Capture failed execution ID
Log prompt + payload
Alert Slack
Queue replay of only the Gemini step
Escalate after max attempts

That turns a noisy crash into something debuggable.

If you use raw HTTP, you own the retry logic

This is where a lot of teams accidentally create their own outage.

If you call Gemini through direct REST, an n8n HTTP Request node, Make, Zapier, or a custom worker, you need to implement retry policy yourself.

Minimum bar:

retry_on = [408, 429, 500, 502, 503, 504]
max_attempts = 4
base_delay_seconds = 1
max_delay_seconds = 60
use_jitter = True

And the retry should wrap only the model request.

Not the whole business process.

Example: isolated retry wrapper in Python

import random
import time
import requests

RETRY_ON = {408, 429, 500, 502, 503, 504}
MAX_ATTEMPTS = 4
BASE_DELAY = 1
MAX_DELAY = 60


def call_gemini_with_retry(url, headers, payload):
    attempt = 0

    while attempt < MAX_ATTEMPTS:
        attempt += 1
        response = requests.post(url, headers=headers, json=payload, timeout=60)

        if response.status_code < 400:
            return response.json()

        if response.status_code not in RETRY_ON:
            response.raise_for_status()

        if attempt == MAX_ATTEMPTS:
            response.raise_for_status()

        delay = min(BASE_DELAY * (2 ** (attempt - 1)), MAX_DELAY)
        jitter = random.uniform(0, delay * 0.25)
        time.sleep(delay + jitter)

That’s still not enough by itself.

You also need idempotency around whatever happens after the model returns.

The retry boundary I trust now

This is the design I’d recommend to anyone running AI automations in production.

1. Make the model call idempotent

Give each model request a stable operation ID tied to the business event.

Not the execution attempt.

For example:

lead_enrichment:hubspot_contact_12345
support_triage:zendesk_ticket_98765
invoice_review:invoice_2026_00412

If the same job replays, your system should recognize it as the same operation.

2. Persist enough context to replay one step

Store:

prompt input
normalized source data
model name
temperature
candidate count
execution ID
business object ID
timestamp

If you don’t log the exact request shape, replay becomes guesswork.

3. Cap retries and fail deliberately

Do not let a worker spin forever because one model is having a bad hour.

After max attempts, route to one of these:

dead-letter queue
delayed retry
human review
fallback model path

Fallback routing is not cheating.

It’s production engineering.

4. Smooth traffic before blaming the model

A lot of “random instability” is really bursty traffic.

If your cron job wakes up and slams Gemini with a huge batch, shared-capacity systems can get weird fast.

Paced workers beat spiky workers.

Queues beat bursts.

Are you debugging one API key when the whole project is overloaded?

This is another easy trap.

Gemini and Vertex AI limits are not always about a single request or a single API key.

They can be project-wide.

So if you have:

an n8n workflow
a Make scenario
a Python worker
a Zapier automation
maybe a background agent

...all hitting the same Google project, failures can look random unless you correlate them with project-wide traffic.

That means you should track at least:

requests per minute
tokens per minute
requests per day
spend spikes
retry volume
error rate by status code

If you only inspect one failing execution, you’ll miss the real cause.

SDK vs raw HTTP vs managed routing

This is one of those boring implementation details that decides whether your week stays calm.

Option	What happens when Gemini gets flaky
Gemini API via official SDK	Safer defaults. Built-in transient retry behavior. Less custom work.
Gemini API via direct REST or n8n HTTP Request	You own retries, jitter, caps, and safe replay boundaries. Easier to get wrong.
Vertex AI pay-as-you-go	Shared-capacity behavior means burst shape matters a lot.
Vertex AI Provisioned Throughput	Better when you need more consistent service and retries alone aren’t enough.

My bias: if you’re doing direct HTTP in production, be honest that you’re taking on reliability work.

That’s fine.

Just don’t pretend it’s the same as using an SDK with sane defaults.

Terminal commands that are actually useful during incident response

If you’re testing Vertex AI auth manually:

gcloud auth print-access-token

If you want to inspect whether your worker is replaying too aggressively, log attempt counts explicitly:

grep "gemini_attempt" app.log | tail -100

And if you aren’t logging operation IDs yet, fix that first.

A lot of debugging pain disappears once you can answer this question quickly:

Did the model fail once, or did our workflow replay the same business event four times?

What changed for me after fixing the boundary

Once we stopped retrying the whole workflow, the incidents got much less dramatic.

We still saw transient model failures.

That part never fully goes away.

But the failures became contained:

one failed model step
one replayable unit
one error record
zero duplicate side effects

That’s a very different operational story.

Where Standard Compute fits if you’re tired of this game

A lot of teams end up here because per-token pricing makes them afraid to add the reliability layers they actually need.

They avoid extra retries.
They avoid fallback models.
They avoid always-on agents.
They avoid richer automation because every failure path has a billing consequence.

That’s exactly the problem Standard Compute is trying to remove.

Standard Compute gives you an OpenAI-compatible API with unlimited AI compute at a flat monthly price, so you can run agents, retries, batching, and automations without token anxiety.

If you’re building on n8n, Make, Zapier, OpenClaw, or custom workers, that matters more than people admit.

Predictable cost changes architecture decisions.

It’s a lot easier to build safe retry boundaries and fallback paths when every extra call doesn’t feel like a tiny financial penalty.

The takeaway I wish I had earlier

If your Google AI API 500 errors keep showing up in production, stop asking only:

how do we retry harder?

Ask better questions:

Is the model call isolated from side effects?
Can I replay only the Gemini step?
Am I using the official SDK or raw HTTP?
Am I tracking project-wide traffic instead of staring at one failed request?
Do I need better throughput guarantees instead of more retry code?

When an LLM stops responding, the winning move usually isn’t prompt magic.

It’s boring architecture:

idempotency keys
isolated model steps
capped retries
jitter
sub-workflows
dead-letter queues
clean error lanes

Less exciting than blaming Gemini.

Much more effective.