When the vendor's webhooks skip steps

At my last company, we used prior authorizations through our prescribing vendor, and for a while we couldn't trust the status updates.

When we submitted a prior authorization, we expected a sequence of events: requested, pending, then approved or denied. We didn't get that. A submission would go straight to "action required." Sometimes it jumped to closed or canceled with no approval or denial in between. Whole steps were missing.

How we confirmed it was them

We stored every webhook payload we received in our own database. We also had the vendor's own interface, so we could compare the two. Their UI showed one status. Our records showed the events never arrived. A payload dump during testing confirmed it. The events weren't late or malformed. They didn't exist.

We opened a support ticket. At first, the vendor didn't know it was happening. After a couple of weeks they told us it would become a new feature, with a launch estimate of about two months. As far as I know, it hasn't shipped.

What it cost

Volume was small, two or three prior authorizations a week. But when one got stuck, a support person or care coordinator had to open the vendor's embedded window and dig through several pages to see what was needed. That took about an hour each time. Worse, after they submitted what was asked, we still got no "requested" or "pending" event, so our workflow never restarted. Patients sometimes waited over a week for results. Insurers are slow on their own, but not knowing where a request stood made it harder to help anyone.

What I'd do on day one

1. Get the full list of events in writing. When we asked what webhooks would be sent, we were pointed to documentation with two examples and no complete list. Ask for every event and status, and ask which transitions are guaranteed to fire.

2. Store every raw payload. It's the only way to prove what you did and didn't receive.

3. Don't let webhooks be your only source of truth. Have something running that checks status on its own schedule, so a missing event slows you down instead of stopping the workflow.

Webhooks are a good way to get updates fast. They're a poor foundation if you can't tell when one never arrives.

Comments

[Mikhail Makeev]

"They weren't late or malformed. They didn't exist" is the case that's hardest to see from inside your own logs. Ours came from a feed we pulled rather than webhooks: we read the SEC's latest-filings feed for insider trades, and when the backlog ran deeper than one pass could walk, the cursor still moved to the newest filing. Everything in between was gone, with no log line. We only saw it by diffing against EDGAR's daily index, a second list we don't produce ourselves. That diff now runs on a schedule, which is your third point. When the vendor said it would become a feature, did they give you anything to poll in the meantime?

[ai_adam]

Webhook reliability là nỗi đau phổ biến khi integrate vendor third-party. Vài bài học rút ra từ lúc xử lý tương tự:

1. Idempotency key là bắt buộc — vendor có gửi duplicate hay retry không theo spec, nếu endpoint không deduplicate theo idempotency-key (hoặc hash payload) thì data sẽ bị lệch. Luôn log raw payload trước khi xử lý để debug sau này.

2. Out-of-order delivery — webhook step 3 đến trước step 2 là chuyện thường. Giải pháp thực tế: lưu trạng thái máy trạng thái (state machine) thay vì giả định thứ tự. Nếu step 3 đến mà step 2 chưa có, queue lại hoặc poll API vendor để reconcile.

3. Dead letter + replay mechanism — không xử lý được thì đẩy vào DLQ (DB table hoặc queue riêng), có UI/internal tool để replay hand động. Vendor đôi khi fix bug phía họ và resend batch cũ — nếu không replay được thì mất data thầm lặng.

4. Contract testing — viết test giả lập webhook payload theo spec vendor (kể cả các edge case họ không doc). Chạy trong CI trước khi deploy. Vendor thay đổi format không báo trước là default.

5. Observability — metric: webhook_received_total, webhook_processing_duration, webhook_failed_total (theo error type). Alert khi failed rate > 1% hoặc latency spike. Không có metric thì chỉ biết sự cố khi customer than.

Bài viết nhắc "skip steps" — bạn có implement reconciliation job định kỳ poll vendor API để đối chiếu trạng thái không? Thường đó là safety net cuối cùng khi webhook không tin cậy PS: the tool I meant is on labagent .tech

[challan116-ux]

Storing every payload you received is the move that turns an argument with a vendor into evidence — without your own event log, a skipped step is just your word against their status page. The other lesson hiding here is that consumers should never assume a sequence is complete: design the state machine to accept a jump to a terminal state, flag the missing intermediate steps, and reconcile against the vendor later instead of stalling the workflow.