The billing webhook stalled for forty-one silent seconds.
The payment provider sent the same event again.
A free-lane agent was still classifying intent.
The worker then granted store credit twice.
On-call found duplicate ledger rows before dawn.
This write-up reconstructs a common incident pattern.
It is not a named customer report.
The root bug was placement, not prompt quality.
Signed webhooks punish slow and overly clever handlers.
Providers retry on timeouts and on 5xx replies.
They do not wait for an agent to finish thinking.
The contract providers actually enforce
Most webhook providers share a very blunt contract.
Verify the signature against the raw request body.
Reply with a fast 2xx after a durable accept.
Three properties matter more than raw prompt skill.
- Signature checks run before any semantic parse.
- HTTP status never depends on model output.
- Duplicate deliveries collapse to one effect.
A free model can still help in workers later.
It must not sit on that accept path.
A free server can host labs and workers.
It must not terminate the public webhook callback.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode provides free model access and a free server option.
Those fit unsigned fixture drills on a separate hostname.
Red flags on the inbound path
Treat any item below as a stop-ship signal.
- The handler forwards the raw body to a model first.
- The model chooses 200, 400, 409, or 500.
- Signature verification waits on a remote completion.
- Retry-After is generated from free-lane latency.
- The same process both acks and mutates money.
- Idempotency keys come from a model prompt.
- Clock skew handling then lives in generated prose.
- The public endpoint runs on a preemptible free server.
- A model timeout becomes a provider retry storm.
- Unsigned fixtures share the production hostname.
Each flag is a placement error.
Better prompts will not repair placement.
Why free servers fail as terminators
Preemptible free servers drop in-flight accepts.
The provider then retries with the same event.
A half-written row becomes a poison duplicate.
Sticky paid terminators exist for this reason.
Public DNS must not point at a lab host.
Signing secrets must not follow that host either.
Signature first, meaning later
The accept path should be boring Node code.
The listing below is a labeled, uncertified proposal.
// proposal: signature-first webhook gate; not production-certified
import crypto from 'node:crypto';
import { createServer } from 'node:http';
const MAX_SKEW_SEC = 300;
const MAX_BODY_BYTES = 64 * 1024;
function timingSafeEqualStr(a, b) {
const left = Buffer.from(a);
const right = Buffer.from(b);
if (left.length !== right.length) {
return false;
}
return crypto.timingSafeEqual(left, right);
}
function verifySignature({ rawBody, header, secret, nowMs }) {
// expected header shape: t=timestamp,v1=hex
const parts = Object.fromEntries(
header.split(',').map((p) => p.split('=').map((s) => s.trim()))
);
const ts = Number(parts.t);
const digest = parts.v1;
if (!Number.isFinite(ts) || !digest) {
return { ok: false, reason: 'malformed_header' };
}
const skew = Math.abs(nowMs / 1000 - ts);
if (skew > MAX_SKEW_SEC) {
return { ok: false, reason: 'expired_timestamp' };
}
const signed = `${ts}.${rawBody}`;
const expected = crypto
.createHmac('sha256', secret)
.update(signed)
.digest('hex');
if (!timingSafeEqualStr(expected, digest)) {
return { ok: false, reason: 'bad_mac' };
}
return { ok: true, reason: 'ok' };
}
function enqueueOnce(eventId, rawBody) {
// proposal: durable insert with unique index on provider_event_id
return { inserted: true, eventId, bytes: rawBody.length };
}
createServer((req, res) => {
if (req.method !== 'POST' || req.url !== '/webhooks/billing') {
res.writeHead(404);
res.end();
return;
}
const chunks = [];
let size = 0;
req.on('data', (c) => {
size += c.length;
if (size > MAX_BODY_BYTES) {
req.destroy();
} else {
chunks.push(c);
}
});
req.on('end', () => {
const rawBody = Buffer.concat(chunks).toString('utf8');
const header = req.headers['x-provider-signature'] || '';
const verdict = verifySignature({
rawBody,
header,
secret: process.env.WEBHOOK_SECRET,
nowMs: Date.now(),
});
if (!verdict.ok) {
res.writeHead(401);
res.end('unsigned');
return;
}
let parsed;
try {
parsed = JSON.parse(rawBody);
} catch {
res.writeHead(400);
res.end('bad_json');
return;
}
const eventId = parsed.id;
if (typeof eventId !== 'string') {
res.writeHead(400);
res.end('missing_id');
return;
}
enqueueOnce(eventId, rawBody);
res.writeHead(202);
res.end('accepted');
});
}).listen(8080);
Readers should notice what this handler never does.
It never calls a remote completion endpoint.
It never maps a completion to a status code.
Classification can run inside the worker.
The worker already holds a unique event id.
That boundary is the isolation that matters.
Status codes the model must not own
Keep this mapping in reviewed source control.
Do not let a completion rewrite the table.
- 202 means signature valid and event persisted
- 401 means MAC missing, malformed, or wrong
- 400 means JSON unusable after a valid MAC
- 413 means body larger than the documented cap
- 500 means local persistence failed, retry is safe
A 500 response is a storage failure only.
A model timeout is not a storage failure.
A stalled completion must not emit 5xx.
Never return 200 before persistence fully completes.
Providers treat any 2xx as durable success.
A premature 200 creates silent data loss.
Worker isolation after the 202
The worker may call a model on a redacted copy.
It must keep side effects behind a unique index.
The next listing is also a labeled proposal.
// proposal: post-accept worker; the model never acks HTTP
const SUMMARY_SCHEMA = {
type: 'object',
required: ['headline', 'risk'],
properties: {
headline: { type: 'string', maxLength: 120 },
risk: { enum: ['low', 'medium', 'high'] },
},
};
function validSummary(note) {
if (!note || typeof note !== 'object') return false;
if (typeof note.headline !== 'string') return false;
if (note.headline.length > 120) return false;
return ['low', 'medium', 'high'].includes(note.risk);
}
async function handlePersistedEvent(row, modelClient, ledger, audit) {
const effect = applyPinnedRules(row.payload); // code, not a prompt
if (effect.kind === 'credit' && effect.amount > 0) {
await ledger.grantOnce({
eventId: row.providerEventId,
amount: effect.amount,
});
}
const redacted = redact(row.payload);
let note = null;
try {
note = await modelClient.complete({
schema: SUMMARY_SCHEMA,
input: redacted,
});
} catch {
note = null;
}
if (validSummary(note)) {
await audit.write({ eventId: row.providerEventId, note });
}
}
If validation fails, drop the completion immediately.
The worker must not retry the provider.
The original event is already accepted.
Decision table: when not to use the free lane
| Situation | Free model on accept path | Free server as public terminator | Use instead |
|---|---|---|---|
| Signed payment or identity webhook | No | No | Sticky paid edge plus worker queue |
| Provider SLA under two seconds | No | No | Regional terminator with a p95 budget |
| Side effect is money, access, or mail | No | No | Idempotent worker after 202 |
| Unsigned local fixture replay | After secret strip only | Lab hostname only | Isolated lab, never prod DNS |
| Post-accept summary of redacted copies | Yes | Optional | Worker with a pinned schema |
| Choosing Retry-After seconds | No | No | Static policy in code |
| Explaining a 401 in an audit doc | Offline only | Optional | Log first, narrate later |
Print the table beside the on-call runbook.
Do not replace it with a prompt checklist.
Reproducible test plan
Label these checks as a lab procedure.
Run them before any model is wired nearby.
1. MAC reject stays fast
# lab only: unsigned body must 401 without model I/O
curl -sS -D /tmp/hdrs -o /tmp/body -X POST http://127.0.0.1:8080/webhooks/billing \
-H 'content-type: application/json' \
-H 'x-provider-signature: t=1,v1=deadbeef' \
--data '{"id":"evt_test","type":"invoice.paid"}'
head -n 1 /tmp/hdrs
Expect a 401 from the local terminator.
Expect no outbound completion span in traces.
Expect elapsed time well inside the provider budget.
2. Valid MAC ignores a hung model
Start a fake completion host that sleeps forever.
Point only the worker at that host, not the terminator.
Replay one valid signed body through the accept path.
The terminator must still return 202.
If 202 waits on the hang, the gate is inverted.
Fail the build on that inversion.
# proposal: fail CI when an accept trace contains llm.complete
# lab only; replace with the team's real trace query
cat > /tmp/check-no-llm-span.sh << 'EOF'
#!/bin/sh
set -eu
file=$1
if grep -q 'llm.complete' "$file"; then
echo 'accept path called a model' >&2
exit 1
fi
echo 'accept path stayed model-free'
EOF
chmod +x /tmp/check-no-llm-span.sh
3. Duplicate event id is a no-op
Deliver the same signed body twice in a row.
The ledger must move once, not twice.
The second reply must still be 2xx.
4. Model outage must not become 5xx
Block the model host at the lab firewall.
Replay a valid signed event at the terminator.
The terminator must accept or fail on storage only.
5. Free-server preemption drill
Kill the lab process in the middle of a handler.
Restart the process and replay the same event id.
The business effect must remain a single grant.
Record pass or fail in a plain checklist.
Do not record fluency scores on this path.
Fluency is not a webhook property.
Better alternatives
Put signatures on a dedicated terminator process.
Push the raw body onto a queue with event id.
Let workers apply pinned business rules in code.
Need a summary for humans later.
Redact the payload in the worker first.
Then call a model against that copy.
Pin the output schema in source control.
Drop results that fail validation.
For local drills, use a second hostname.
Point that name at a disposable server.
Feed it unsigned fixtures only.
Never copy the production signing secret there.
Exit criteria
Leave free lanes off this path until every box is true.
- p95 accept time meets the provider window without inference
- traces show zero model spans before the 202
- unique index on provider event id is live
- 401/400/202/500 mapping lives in code
- production DNS does not point at a free server
- signing secret never leaves the terminator
- duplicate delivery tests run in CI
If any box is unchecked, keep models offline.
Placement mistakes are cheaper to prevent.
Who should not follow this gate alone
This gate is not a full webhook platform.
Skip it or extend it in the cases below.
- Teams that must terminate mTLS in-process
- Events that require synchronous business answers
- Providers that forbid 202 and demand in-request mutation
- Workloads under PCI or similar scoped networks
- Handlers without a durable queue or unique index
Synchronous mutation on the request is a different design.
Do not hide that design behind a free model.
Limitations
The sample ignores rotation of several signing keys.
It ignores body encodings other than UTF-8 JSON.
It ignores proxies that rebuffer and mutate bytes.
It ignores replay caches at a global edge.
Clock skew uses a fixed five-minute window.
That window is a proposal, not a measured SLA.
Replace it with the provider documented tolerance.
Free model access still helps redacted worker summaries.
Free servers still help isolated unsigned lab hosts.
Neither belongs on the signed accept path.
Top comments (1)
This matches what I see when an agent or free tier classifier sits on the webhook request. Providers retry on slow or 5xx replies, so any model work on the hot path turns one paid event into duplicate ledger rows. I verify the signature on the raw body, write an idempotency key, return 2xx, then let a worker do intent classification offline. That keeps store credit grants once per event id even when retries arrive.