An agent that runs on events, not chat — webhooks + a queue, idempotent execution, and a dead-letter queue

#ai #agents #llm #python

Project 8 of my "Agentic AI from Zero" series flips the usual model: instead of you typing at the agent, the agent wakes up on an event — a webhook POST, a message on a queue — does its job, and goes back to sleep. No chat loop. And it never processes the same event twice.

Built on a free 8B model (Llama 3.1 via NVIDIA NIM), with all the plumbing as deterministic Python and only the decision left to the model.

Listen to webhooks + a queue

Two real event sources, no external infra:

# webhook.py — stdlib http.server, no framework
class WebhookReceiver(BaseHTTPRequestHandler):
    def do_POST(self):
        body = self.rfile.read(int(self.headers["Content-Length"]))
        ev = Event.from_json(body)          # id = idempotency key
        QUEUE.append(ev)                    # enqueue, don't block on the model
        self._respond(202, {"queued": ev.id})

The receiver only enqueues — it returns 202 Accepted immediately and never waits on the LLM. A file-backed append-only queue (queue.jsonl) is the durable buffer; a worker drains it.

Execute on triggers

A dispatcher maps event type → handler; each handler is where the model actually reasons (triage an order, classify+draft an email reply, summarize+route a file):

def worker_tick():
    ev = QUEUE.claim()                      # FIFO cursor
    if LEDGER.seen(ev.id):                  # idempotency
        return "skip: already processed"
    handler = DISPATCH[ev.type]
    for attempt in range(1, MAX_TRIES + 1):
        try:
            out = handler(ev)               # <- the one place a model call happens
            LEDGER.mark(ev.id); return out
        except TransientError:
            time.sleep(backoff_delay(attempt))   # bounded exponential
        except BusinessReject:
            LEDGER.mark(ev.id); return "rejected" # NOT an error — never retry
    DLQ.add(ev, last_error)                  # exhausted → dead-letter

Idempotent execution

The event id is the idempotency key. A processed-ledger records every handled id, so a re-delivery is a no-op — no duplicate charge, no duplicate email. In the recorded run, ord-1001 was delivered twice (once via a real webhook POST, once re-queued); the second delivery hit the ledger and spent zero model calls.

Dead-letter + retry

Transient failures (a downstream 503) retry with bounded exponential backoff; business rejects (an order with a $0 total) are not errors and never retry. After N attempts a still-failing event goes to a dead-letter queue with its payload and last error, so the worker moves on instead of jamming.

The real recorded run

Against NVIDIA NIM (llama-3.1-8b), captured live:

ord-1001 via a real POST http://127.0.0.1:8808/events → 202, handled.
ord-1001 re-delivered → idempotency no-op (7 deliveries → 6 distinct ledger rows).
eml-2002 downstream 503'd twice → backoff → handled on try 3.
ord-1002 downstream stayed down → 3 tries exhausted → dead-lettered.
ord-1003 ($0 total) → business-rejected, not retried, no model call spent.

Only the downstream success/failure is scripted (so the demo is reproducible); every LLM call is live.

The pattern: the model decides, but the queue, dedup ledger, retry policy, and DLQ are plain deterministic code — which is what makes an event-driven agent safe to leave running unattended.

Real transcript + full code in the repo (comments):