DEV Community: Luyu Fang

Your FastAPI Webhook Returned 200. Why Did the Order Run Twice?

Luyu Fang — Thu, 23 Jul 2026 06:30:40 +0000

Your webhook endpoint returned 200 OK.

The provider still delivered the event again.

That is not necessarily a provider bug. Most webhook systems promise at-least-once delivery, not exactly-once delivery. A timeout, a lost response, a process restart, or a manual replay can send the same logical event more than once.

The dangerous implementation looks harmless:

@app.post("/webhooks/provider")
async def receive(event: dict):
    create_order(event)
    return {"ok": True}

If the same event arrives twice, create_order() runs twice. Replace "order" with invoice, email, provisioning job, CRM record, or AI-agent action and the failure becomes expensive quickly.

Here is the small pattern I use to make the HTTP boundary safer.

1. Verify the signature against the raw body

Do not parse and re-serialize the payload before signature verification. Whitespace, key order, and encoding can change the byte sequence.

import hashlib
import hmac

def valid_signature(body: bytes, supplied: str, secret: bytes) -> bool:
    expected = hmac.new(secret, body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, supplied)

Use hmac.compare_digest() rather than a normal string comparison.

2. Treat the provider event ID as a database key

An in-memory set is not enough. It disappears on restart and does not coordinate multiple workers.

Create a durable inbox table with a unique event ID:

CREATE TABLE webhook_inbox (
  event_id TEXT PRIMARY KEY,
  event_type TEXT NOT NULL,
  payload TEXT NOT NULL,
  state TEXT NOT NULL DEFAULT 'accepted',
  attempts INTEGER NOT NULL DEFAULT 0,
  last_error TEXT,
  accepted_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

The uniqueness constraint is the important part. Two concurrent requests can both pass a separate "does this ID exist?" query. Let the database perform the reservation atomically.

For a minimal SQLite example:

cursor = db.execute(
    """
    INSERT OR IGNORE INTO webhook_inbox(event_id, event_type, payload)
    VALUES (?, ?, ?)
    """,
    (event_id, event_type, body.decode()),
)
db.commit()

duplicate = cursor.rowcount == 0

In PostgreSQL, use INSERT ... ON CONFLICT DO NOTHING.

3. Acknowledge durable acceptance, not completed business work

Keep the HTTP path short:

read the raw body;
verify the signature;
validate the minimum envelope;
persist the inbox record;
return 202 Accepted.

Do not call three downstream APIs before responding. Slow processing increases the chance that the provider times out and retries.

The endpoint can look like this:

from fastapi import FastAPI, Header, HTTPException, Request, status

app = FastAPI()

@app.post("/webhooks/provider", status_code=status.HTTP_202_ACCEPTED)
async def receive_webhook(
    request: Request,
    x_signature: str = Header(default=""),
):
    body = await request.body()
    if not valid_signature(body, x_signature, secret):
        raise HTTPException(status_code=401, detail="Invalid signature")

    event = json.loads(body)
    cursor = db.execute(
        "INSERT OR IGNORE INTO webhook_inbox(event_id,event_type,payload) VALUES(?,?,?)",
        (str(event["id"]), str(event["type"]), body.decode()),
    )
    db.commit()

    return {
        "accepted": True,
        "duplicate": cursor.rowcount == 0,
    }

The worker processes accepted rows separately. A crash after the database commit does not lose the event.

4. Test the duplicate directly

Do not wait for production retries to discover whether idempotency works.

def test_duplicate_delivery_has_one_durable_acceptance():
    event = {"id": "evt_duplicate_1", "type": "order.paid"}

    first = signed_post(event)
    second = signed_post(event)

    assert first.status_code == 202
    assert first.json()["duplicate"] is False
    assert second.status_code == 202
    assert second.json()["duplicate"] is True

Also test:

an invalid signature;
two concurrent deliveries of the same ID;
a worker crash after inbox persistence;
a downstream timeout;
events delivered in reverse order;
a stale event trying to overwrite newer state.

5. Duplicate delivery is not the only problem

Events can arrive out of order. A delayed subscription.updated event may arrive after subscription.cancelled.

Store the provider timestamp or sequence number and define allowed state transitions. An older event should become a no-op rather than silently restoring stale state.

Retries also need a terminal state. Record attempt count, last error, and the next retry time. After the configured limit, move the event to a dead-letter state that can be inspected and replayed intentionally.

The practical rule

Your webhook should create one durable inbox record, not perform one irreversible business action inside the HTTP request.

That single design choice makes duplicates, crashes, retries, and debugging much easier to reason about.

I published the complete runnable FastAPI receiver, tests, production checklist, and reusable review prompt here:

https://codez.win/guides/fastapi-webhook-idempotency?utm_source=devto&utm_medium=article&utm_campaign=webhook_20260723

What is the most expensive duplicate webhook failure you have seen: double billing, duplicate emails, or repeated provisioning?

A Verifiable AI Agent Workflow: 7 Checks Before You Add Autonomy

Luyu Fang — Mon, 20 Jul 2026 06:22:53 +0000

Most agent projects fail before the model does.

The failure usually starts with an unclear outcome, an overpowered tool, or a demo that has no repeatable pass/fail test. The agent looks impressive once, then becomes impossible to trust in production.

Here is the checklist I now use before adding more autonomy.

1. Define one verifiable outcome

Avoid goals such as "help the user with support."

Use an outcome a human can inspect:

Given a support ticket and the account history, produce a draft reply with cited evidence and route it to a human reviewer.

A good outcome has a clear input, output, owner, and acceptance test.

2. Separate three kinds of work

Every step belongs in one of these buckets:

Deterministic work: API calls, schema validation, database reads, file transforms.
Model judgment: classification, summarization, ranking, drafting.
Human approval: sending messages, spending money, changing permissions, or deleting data.

This separation makes failures easier to locate. It also prevents a model from being asked to do work that ordinary code can do more reliably.

3. Give every tool a typed contract

An agent tool should look like an API, not a vague capability.

{
  "name": "get_order",
  "input": {
    "order_id": "string"
  },
  "output": {
    "status": "enum",
    "items": "array",
    "refund_eligible": "boolean"
  },
  "errors": [
    "ORDER_NOT_FOUND",
    "UPSTREAM_TIMEOUT"
  ]
}

Define timeouts, retry rules, idempotency, and the exact errors the model may receive. Do not return an unbounded blob and hope the model interprets it correctly.

4. Make approval gates explicit

A human approval gate is not a weakness. It is a product feature when the consequence is expensive or irreversible.

Useful approval gates include:

sending an external email
issuing a refund
publishing content
changing account access
running a destructive command

The agent can prepare the action and explain its evidence. The human approves the final side effect.

5. Design failure cases before the happy path

Write at least five realistic failures before implementation:

required data is missing
two sources contradict each other
the tool times out
the model returns invalid structured output
the requested action exceeds the user's permission

For every failure, define whether the system retries, asks a question, escalates, or stops.

6. Build pass/fail evaluations

"Looks good" is not an evaluation.

Create a small suite with cases such as:

known-good input produces the expected action
missing evidence prevents the action
an injected instruction inside retrieved content is ignored
a repeated request does not create a duplicate side effect
a timeout produces a recoverable state

Start with 10 carefully chosen cases. Run them after every prompt, tool, or model change.

7. Ship the smallest observable workflow

The first milestone should fit into one day:

one input type
one model decision
one read-only tool
one structured output
one approval screen
one event log

Do not begin with a swarm of agents. Add another agent only when it removes a measured bottleneck.

A reusable planning prompt

Turn the AI agent idea below into a verifiable MVP.

Return:
1. one measurable user outcome
2. inputs and outputs
3. deterministic steps, model-judgment steps, and human approval gates
4. a typed contract for every tool
5. five realistic failure cases
6. pass/fail evaluation cases
7. the smallest milestone testable in one day

Do not add autonomy unless it is required for the outcome.

Agent idea:
[Describe the user, task, data, tools, and constraints]

The goal is not maximum autonomy. The goal is a workflow that can earn trust one observable step at a time.

I keep the copyable prompt and related workflow examples in this AI agent workflow guide.

Your AI Agent MVP Does Not Need More Autonomy

Luyu Fang — Sat, 18 Jul 2026 04:22:24 +0000

Most AI agent MVPs start with the wrong question:

How much can we make autonomous?

A better first question is:

What is the smallest useful outcome a human can verify?

The difference matters. An autonomous demo can look impressive while hiding unreliable decisions, unclear permissions, and failure states that nobody has tested. A narrow, reviewable workflow is less dramatic, but it can become a real product.

1. Draw the boundary before choosing tools

Split the workflow into three kinds of work:

Deterministic steps: validation, parsing, database reads, calculations, and format conversion.
Model judgment: classification, summarization, ranking, and drafting where uncertainty is expected.
Human approval: sending messages, changing production data, spending money, or publishing externally.

This boundary tells you where an LLM is useful and where ordinary code is safer. It also prevents the agent from quietly gaining permissions just because a demo needs to look seamless.

2. Give every tool a typed contract

An agent tool should not be described as “search the system” or “update the record.” Define its inputs, outputs, timeouts, permission checks, and failure responses.

For example, a lead-research tool can return:

the public source URL;
extracted facts;
confidence for each fact;
missing fields;
a structured error when the page cannot be read.

The model can then reason over evidence instead of inventing a successful result. Typed contracts also make tool calls testable without invoking the full agent.

3. Test failures before adding autonomy

Five evaluation cases are usually more valuable than five more tools:

a normal request with complete data;
missing or contradictory input;
a tool timeout;
a low-confidence model response;
a request that needs permission the agent does not have.

Each case needs an observable pass/fail rule. “The answer looks good” is not a rule. “The agent cites the source, marks the missing field, and does not call the write tool” is.

4. Keep the first external action reviewable

For an early release, prefer read-only tools. Let the agent prepare a draft, proposed database change, or command plan, then require a human to approve the final external action.

This is not a permanent limitation. It is how you collect evidence about where the system is reliable enough to automate next.

A practical MVP sequence

Choose one narrow outcome.
Write the expected input and output.
Separate deterministic code from model judgment.
Define typed tools and permission boundaries.
Add five realistic evaluation cases.
Keep the final high-impact action behind approval.
Record failures and only automate the stable parts.

The goal of an agent MVP is not to imitate a fully autonomous employee. It is to prove that one workflow can produce repeatable value without hiding uncertainty.

I turned this sequence into a reusable, editor-verified workflow on Codez Win:

https://codez.win/guides/ai-agent-workflows?utm_source=devto&utm_medium=article&utm_campaign=guide_launch_20260718

What is the smallest agent workflow you have seen deliver repeatable value in production?