DEV Community: orbiresearch

The approval queue pattern: putting a human in the loop without putting them in the way

orbiresearch — Sat, 26 Sep 2026 14:00:56 +0000

§ 01 · The problem with "human in the loop"

Every serious agent deployment ends up with a human in the loop somewhere. The phrase has become a checkbox. It is treated as a safety property you either have or do not have. In production it is neither binary nor free.

A human in the loop has a cost, and the cost is latency plus attention. If the agent escalates everything, the human becomes the bottleneck the agent was supposed to remove. If the agent escalates nothing, the human is decoration. Most teams land in a worse place than either: the agent escalates an unpredictable mix of trivial and critical items, the human learns that most items are trivial, and they start approving without reading. Now you have a human in the loop who is functionally not in the loop. You have built a rubber stamp and called it governance.

The approval queue pattern exists to make the human expensive on purpose, and to spend that expense only where it changes the outcome.

§ 02 · The core idea

An approval queue is a durable, ordered list of proposed actions that the agent has decided it should not execute on its own authority. The agent does the reasoning and the drafting. It stops at the action. A human approves, edits, or rejects. The decision and the human who made it are recorded.

Three properties make this a pattern rather than a button:

◆ It is durable. An approval item survives a crash, a restart, and the human going home for the weekend. It lives in a table, not in memory.

◆ It is ordered and bounded. Items have priority and an age. An item that sits unapproved past its SLA is itself an event that triggers escalation, not silence.

◆ It is reversible at the boundary, not after. The human decides before the irreversible action, not after a notification that it already happened.

The unit of the queue is the proposed action, never the conversation. You are not asking a human to review a transcript. You are asking them to approve one specific, executable thing.

── The four fields ──

§ 03 · The four fields every approval item must carry

An approval item that contains only "the agent wants to do X, approve?" forces the human to reconstruct context they do not have. They will either over-investigate (slow) or approve blind (dangerous). Every item must carry four fields.

◆ The action. The exact, executable operation, in plain language and in its concrete form. Not "send outreach." Instead: "Send this email, shown in full, to this address." The human approves the artifact, not a description of it.

◆ The justification. Why the agent proposes this now. The trigger, the rule, the data that led here. One or two sentences. If the agent cannot state why, that is itself a reason to reject.

◆ The blast radius. What this action touches and how hard it is to undo. "One outbound email, not recallable" is a different decision than "one row update, fully reversible." The human is pricing risk. Give them the price.

◆ The confidence and the alternative. What the agent would do if this were rejected, and how sure it is. A low-confidence proposal with an obvious fallback is a fast approve or fast reject. A high-confidence proposal with no fallback deserves the human's full attention.

These four fields turn a vague ask into a decision a busy person can make in seconds without becoming a rubber stamp.

── What belongs in the queue ──

§ 04 · What goes in the queue, and what does not

The discipline is in what you do not route. An approval queue that contains everything is a denial-of-service attack on your own operators.

Route to the queue when at least one is true: the action is irreversible or expensive to undo, the action is externally visible (a customer sees it, money moves, a record leaves your system), or the agent's confidence is below a threshold you set per action class.

Do not route when the action is internal, reversible, and inside the agent's defined scope. Reading data, drafting, summarizing, updating a record the agent owns and can roll back: these are inside the boundary. If you route them, you train your operators to stop reading, and the one item that mattered slips through behind forty that did not.

The threshold is a dial, not a default. Set it per action class. Outbound customer communication might require approval at any confidence. An internal tag update might require approval only below 70 percent. Write the thresholds down. They are part of the system, not a runtime guess.

── The failure modes ──

§ 05 · The failure modes

Rubber stamp. The queue fills with low-stakes items, the operator approves in bulk, and the queue stops being a control. Fix: aggressively reduce what enters the queue, and measure approval time per item. If median approval time drops below the time it takes to read the action, your operators are not reading.

Queue as graveyard. Items pile up unapproved because no one owns the queue. The agent stalls or, worse, starts taking the unapproved actions because a timeout was wired to "proceed." Fix: every queue has an owner and an SLA, and the SLA breach escalates to a person, never to auto-approve.

Context starvation. Operators approve blind because the four fields are missing or thin. Fix: treat a proposal with a weak justification as a defect in the agent, not a judgment call for the human.

Silent scope creep. The set of actions that bypass the queue grows over time, each addition reasonable, until the agent is doing things no one is reviewing. Fix: the routing rules live in version control and are reviewed on the same cadence as the trust boundary.

── The file that holds the queue ──

§ 06 · The contract behind the queue

Create an APPROVALS.md in the agent repository. It lists every action class the agent can propose, the routing rule for each (always, never, or below a confidence threshold), the SLA for each priority level, the escalation target when an SLA is breached, and the owner of the queue. It records the last review date.

This is the document your operations lead reads on day one and your auditor reads on the worst day. An approval queue without a written contract is not a control. It is a habit, and habits drift.

── End of pattern ──

◆ A human in the loop is a cost. Spend it only where it changes the outcome.

◆ The unit of the queue is one executable action with four fields, never a transcript.

If your median approval time is shorter than the time to read the action, you do not have a human in the loop. You have a rubber stamp.

ORBIRESEARCH

Originally published on the OrbiResearch Lab. We build production AI agents at orbiresearch.com.

Retry, Backoff, and Circuit Breakers for LLM API Calls

orbiresearch — Sat, 26 Sep 2026 14:00:20 +0000

Retry LLM API calls on 429, 5xx, and timeouts with exponential backoff plus jitter, honor the Retry-After header when the provider sends one, cap total retry time with a deadline, and wrap the whole thing in a circuit breaker so a degraded provider cannot take your system down with it. Never retry non-idempotent actions blindly, and never retry validation errors at all. That is the summary. The rest of this article is why each rule exists, what the naive version costs, and the code shape we ship.

§ 01 · LLM APIs fail differently

Every API fails. LLM APIs fail with personality. Four modes dominate:

Rate limits (429). The most common and the most misunderstood. A 429 is not an error in your code, it is the provider telling you to slow down. We once treated it as fatal and it cost us a night of production downtime, the full story is in our 429 postmortem.

Overload (529 and friends). Provider-side congestion. Distinct from 429 because slowing down helps your standing but the recovery timeline is not yours to control.

Timeouts. Long generations on loaded infrastructure can exceed any sane client timeout. The dangerous part: a timeout does not mean the request failed. The provider may have completed it after you hung up, which matters enormously if the call had side effects downstream.

Degraded output. The sneakiest mode: the API returns 200 and the content is wrong, truncated, or malformed. No retry logic sees this unless your validation layer feeds it. Silent regressions are their own topic, and we wrote one.

── The retry ladder ──

§ 02 · The retry ladder

The first decision is not how to retry, it is whether.

Retry freely: 429, 5xx, network errors, timeouts on read-only calls. These are transient by definition.

Retry carefully: timeouts on calls whose results trigger side effects. The request may have succeeded invisibly. Before retrying, the system needs an idempotency story, a way to guarantee the side effect happens once even if the call happens twice. We learned that one expensively.

Never retry: 400s, validation errors, authentication failures, content policy refusals. The request is wrong, and resending a wrong request is paying twice for the same no. Retrying a 401 in a loop is how you turn an expired key into a locked account.

── Backoff, jitter, and the Retry-After header ──

§ 03 · Backoff, jitter, and the Retry-After header

Immediate retries against a rate-limited endpoint are a self-inflicted denial of service: every client that failed at second zero retries at second one, together, and the stampede keeps the endpoint saturated. Two mechanisms break the stampede.

Exponential backoff spreads retries over time: wait 1s, then 2s, then 4s, then 8s. Jitter spreads them across clients: randomize each wait so a thousand failed requests do not become a thousand synchronized retries. And when the provider sends a Retry-After header, that number wins over your formula, it is the provider telling you exactly when to come back.

async function callWithRetry(request, opts = {}) {
  const maxAttempts = opts.maxAttempts ?? 5;
  const deadline = Date.now() + (opts.maxTotalMs ?? 60_000);

  for (let attempt = 1; ; attempt++) {
    try {
      return await llm.call(request);
    } catch (err) {
      if (!isRetryable(err) || attempt >= maxAttempts) throw err;

      const retryAfter = err.headers?.["retry-after"];
      const base = retryAfter
        ? Number(retryAfter) * 1000
        : Math.min(1000 * 2 ** (attempt - 1), 30_000);
      const wait = base / 2 + Math.random() * (base / 2); // jitter

      if (Date.now() + wait > deadline) throw err; // budget exhausted
      await sleep(wait);
    }
  }
}

Two details in that snippet carry most of the value. The deadline: retries consume time your user or your queue is waiting through, and unbounded patience is not resilience, it is a hung system with good intentions. And the cap on backoff: waiting 512 seconds because the formula says so helps nobody.

── Circuit breakers, or knowing when to stop knocking ──

§ 04 · Circuit breakers, or knowing when to stop knocking

Retries handle a request that failed. Circuit breakers handle a provider that is failing. The difference matters: when an endpoint is down for ten minutes, ten thousand well-behaved retrying requests are still ten thousand requests achieving nothing, burning your rate limits, your latency budgets, and your queue depth.

A circuit breaker watches the failure rate. When it crosses a threshold, the circuit opens: calls fail fast without touching the API. After a cooldown, a few probe requests test the water. Success closes the circuit, failure keeps it open. Three states, one job: stop paying for calls that cannot succeed.

In agent systems the breaker earns its keep twice over, because agents retry at the task level too. An agent that cannot reach its model will often rephrase, replan, and try again, multiplying the underlying API calls. Without a breaker, one degraded provider turns a patient agent into a very expensive metronome. Breaker state changes belong on your dashboard, and if they surprise you there, your instrumentation has gaps.

── The queue is your shock absorber ──

§ 05 · The queue is your shock absorber

Everything above assumes a request in flight. The system-level question is what happens to work that arrives while the provider is down. The answer that survives production: a persistent queue. Requests land in the queue, workers drain it through the retry and breaker machinery, and a provider outage becomes a growing queue instead of lost work. When the provider recovers, the queue drains, and nothing that mattered disappeared during the gap. Every system we ship is built on this shape, most visibly the nine-agent logistics fleet, where a restart or an outage resumes from the last confirmed message.

── The fallback model is uptime engineering ──

§ 06 · The fallback model is uptime engineering

This month a frontier model vanished from the market for twenty days on a government decision. No retry policy fixes that. The last layer of this stack is a fallback model: a second provider or a smaller model, pre-tested against your workload, behind the same interface. Not because the fallback is as good, but because degraded service beats no service, and because the switch has to be a config change, not an engineering sprint that starts the morning of the outage.

── The checklist ──

§ 07 · The checklist

[ ] 429 and 5xx retried with exponential backoff and jitter

[ ] Retry-After header honored over the local formula

[ ] Total retry time capped by a deadline, per request

[ ] Non-idempotent calls have an idempotency key before any retry

[ ] 400s, auth failures, and refusals are never retried

[ ] Circuit breaker per provider, state visible on the dashboard

[ ] Persistent queue in front of the model, drains after outages

[ ] Fallback model tested against the real workload, switchable by config

── Closing ──

§ 08 · Closing

None of this is exotic. Backoff, breakers, queues, and fallbacks predate language models by decades, and the teams that treat an LLM endpoint like any other unreliable dependency get boring, reliable systems. The teams that treat it as magic get to write postmortems. We know, we wrote ours, and building so the next one never happens is most of what production-grademeans.

── End of pattern ──

ORBIRESEARCH

Originally published on the OrbiResearch Lab. We build production AI agents at orbiresearch.com.