DEV Community: Yuriy

Why Webhooks get lost and why they sometimes arrive twice

Yuriy — Mon, 29 Jun 2026 07:00:00 +0000

Webhooks often look deceptively simple.

One service sends an HTTP request. Another service receives it. The receiver returns 200 OK, processes the payload, and everyone moves on.

That model works until the webhook becomes part of a real business process.

A missed event can mean a customer message that never reaches a chatbot, an order that does not appear in a CRM, a subscription that is not updated, or a payment that succeeds but does not unlock access for the customer.

But lost webhooks are only half of the problem. The other half is duplicate delivery. A reliable webhook integration must be prepared for both.

What does it mean to lose a webhook?

A webhook is lost when an event is sent but never successfully processed by the receiving system.

There are many possible causes:

the receiving server is temporarily unavailable;
a reverse proxy is misconfigured;
the application crashes after accepting the request;
the database is overloaded;
a TLS certificate expires;
a deployment introduces an unexpected error;
a DNS or network change routes traffic to infrastructure that is no longer active.

For a low-risk notification, this may be inconvenient but manageable. For a production integration, it can become a business problem.

Imagine a payment provider sends an event confirming that a customer has paid. If your application never processes that event, the customer may be charged but never receive access to the product.

That is not just a technical failure. It creates support work, refunds, manual reconciliation, and a loss of trust.

DNS migrations can create both loss and duplication

A common example is moving an application to a new server.

The usual approach looks simple:

Deploy the application to a new server.
Update the DNS record.
Wait for traffic to move.
Shut down the old server.

The problem is that DNS does not update everywhere at the same moment.

Resolvers, ISPs, corporate networks, and individual machines may cache the old address for different amounts of time. Even when the DNS TTL is low, some traffic can continue reaching the old server after the record has been changed.

During the transition, different webhook senders may reach different servers:

Webhook provider
        |
        +--> old server
        |
        +--> new server

If the old server is already offline, webhooks sent there may fail or be lost. If both servers are active, the same event can potentially be processed twice. That creates a different class of failure.

A chatbot might send two replies to the same message. A CRM integration might create duplicate records. A payment workflow might credit a balance twice or issue the same product twice.

Why retries are necessary

Webhook delivery should normally use an at-least-once model.

That means a sender retries delivery when it cannot confirm that the receiving system accepted the event. This is much safer than silently dropping an event after one failed attempt.

However, at-least-once delivery has an important consequence:

The receiving system must assume that the same event can arrive more than once.

A sender may retry because:

the destination returned a 5xx response;
the connection timed out;
the destination was temporarily unavailable;
the sender did not receive the response, even though the receiver processed the request;
the provider intentionally retries to protect against transient network failures.

This is normal behaviour. It is not necessarily a bug in the provider. The receiving side must be designed accordingly.

Idempotency is the protection against duplicates

The key concept is idempotency.

An idempotent operation produces the same final result whether it runs once or several times.

For webhook processing, this usually means storing the provider’s event identifier before running the business logic.

A simplified flow might look like this:

Receive webhook
        |
        v
Validate signature
        |
        v
Check event ID
        |
        +--> already processed --> return 200 OK
        |
        v
Store event ID
        |
        v
Run business logic
        |
        v
Return 200 OK

For example, if a payment provider sends an event with the ID evt_123, your application should record that ID after receiving it.

If the same event arrives again, the application should recognise it as a duplicate and avoid repeating the business operation.

Returning 200 OK is still important: it tells the sender that the event has been safely handled, even when no additional action was needed.

Acknowledge quickly, process safely

Another useful pattern is to separate acceptance from processing.

Instead of performing all business logic directly inside the webhook request handler, the application can:

Validate the request.
Store the event durably.
Return a successful response quickly.
Process the event asynchronously.

This reduces the risk of timeouts and makes failures easier to recover from. It also gives you a durable record of what happened, which is essential when investigating an incident.

Of course, this only works if the storage step itself is reliable. Returning 200 OK before the event has been saved somewhere durable can still cause data loss.

A practical webhook reliability checklist

A production webhook integration should answer these questions clearly:

What happens when the destination is temporarily unavailable?
Are failed deliveries retried?
How long are retries performed?
Can the same event be delivered more than once?
Does the application use event IDs or idempotency keys?
Are incoming requests stored before expensive processing starts?
Can the delivery history be inspected later?
Can a failed event be replayed safely?
During a migration, how long will the old endpoint remain available?
Can the team distinguish between an event that was never received and one that was received but failed during processing?

If these questions do not have explicit answers, reliability depends too heavily on everything going right.

And in production, everything eventually does not.

Reliability is not just an HTTP status code

Returning 200 OK is not the same as handling an event reliably.

Reliable webhook processing requires a clear delivery model, retry behaviour, idempotent business logic, durable event storage, and enough visibility to understand what happened when an incident occurs.

That is why webhook infrastructure matters.

Adal Cloud is designed to make webhook delivery predictable and observable: incoming requests are retained, delivery attempts can be inspected, failed deliveries can be retried, and stored requests can be replayed when needed.

When webhooks carry events that matter to customers, payments, or business operations, the delivery path should not be the least reliable part of the system.

Why I made Adal CLI Open Source

Yuriy — Wed, 24 Jun 2026 07:00:00 +0000

When you build an app that runs on someone else’s machine, trust becomes a very practical issue.

It is not just about branding, positioning, or saying the right things on a landing page. The user is allowing your software to run in their local environment. Depending on what the app does, that environment may include files, terminal commands, configuration, credentials, or work projects.

So the question becomes simple: how can a user know that the app is doing only what it is supposed to do?

For Adal CLI, I decided that the answer should not be “just trust me.”

It should be verifiable.

Local apps need a different level of trust

A web app usually runs on infrastructure controlled by the product team. A local app is different. It runs directly on the user’s device.

That changes the trust model.

If a CLI tool reads files, sends requests, modifies configuration, or interacts with local projects, users have every right to ask:

What exactly does it read?
What does it send over the network?
Does it collect anything unnecessary?
Can I build it myself instead of downloading a binary?
What happens if the project stops being maintained?

These are fair questions. And for developer tools, they are especially important.

Open Source makes trust verifiable

This is one of the main reasons I made Adal CLI open source.

Open source does not magically make a project secure or trustworthy. But it changes the relationship between the developer and the user.

Instead of asking people to believe a claim, it gives them a way to verify it.

Anyone can inspect the code and see what the app actually does. If someone does not want to use a prebuilt binary, they can build the project from source. If a developer wants to review how data is handled, they can do that. If someone finds a problem, they can open an issue or submit a pull request.

That possibility matters, even if most users never read the source code themselves.

It also makes development more transparent

Another benefit of open source is that people can see how the project evolves.

They can look at the changelog, commits, issues, and pull requests. They can see which bugs were fixed, which features were added, and how decisions were made over time.

For me, that transparency is important. A local tool should not feel like a black box.

This is especially true for CLI tools, because they often operate close to the developer’s actual work: repositories, config files, scripts, and terminal workflows.

Open Source is not a shortcut

There is one important caveat: publishing the code is not enough.

Open source is not a substitute for responsibility. A project can be open and still be confusing, insecure, poorly documented, or hard to build.

So if open source is meant to support trust, it should come with the basics:

clear documentation;
reproducible builds;
a changelog;
a license;
instructions for building from source;
an explanation of what data the app uses and why.

Without those things, “the code is open” can become more of a slogan than a real advantage.

Reducing dependency on one maintainer

There is another reason I care about this: sustainability.

If a tool becomes useful to people, they should not be completely dependent on one person maintaining it forever.

With an open source project, the community can fork it, fix critical issues, adapt it to specific needs, or continue development if the original maintainer slows down or moves on.

That does not guarantee anything, of course. But it gives the project a better chance to survive beyond a single author.

Final thought

Not every app has to be open source.

But if you are building something that runs locally, touches files, works with terminal commands, reads configuration, or handles sensitive data, openness can become one of the strongest arguments for trust.

Trust is hard to earn through promises.

It is much easier to build through transparency, verifiability, and respect for the user.

That is why Adal CLI is open source.

When you just need to inspect a webhook before writing code

Yuriy — Sat, 13 Jun 2026 04:16:51 +0000

Webhooks are usually discussed as something your application should process automatically.

A payment provider sends an event. A Git hosting service notifies you about a push. A SaaS product sends a status update. Your backend receives the request, verifies it, stores it, and runs the required logic.

But before all of that, there is often a much simpler step: you just need to see what the webhook actually looks like. Not the example from the documentation. Not the expected payload in your head. The real request.

What method does it use? Which headers are included? How is the body structured? Where is the event type stored? Are there signatures, timestamps, nested fields, IDs, or provider-specific details that are not obvious from the docs?

For this first step, setting up a local server can be unnecessary work.

You can use a tunneling tool, expose your local application, write a temporary handler, log the request, format the output, and then inspect the result. That works, but sometimes it is too much for a simple question: what exactly does this service send?

This is one of the scenarios where Adal can help.

Adal gives you a permanent webhook endpoint URL. You can paste that URL into the webhook settings of a service like Stripe, GitHub, Slack, Dodo Payments, or any other provider that sends webhook events.

When the webhook arrives, Adal stores the request and shows it in the interface.

You can inspect:

HTTP method
path
headers
body
payload structure
request size
delivery details

This is useful before you write the real backend handler.

You can send a test webhook, inspect the actual request, compare it with the documentation, and only then decide how your application should process it.

This is also useful when debugging. Sometimes the problem is not in your handler at all. The webhook may not be reaching the expected URL. The payload may not match the event you selected. A header may be missing. A signature may be different from what you expected.

Having the raw request visible makes this much easier to understand.

This does not replace proper webhook processing. You still need your backend, validation, signature verification, retries, idempotency, and business logic for production workflows.

But before that, there is a very practical first step: receive the webhook, look at it and understand what was actually sent.

That is the small use case Adal is designed to make simple.

No temporary server just to inspect a payload. No manual log formatting. No guessing based only on documentation. Just a real incoming webhook, shown in a way that is easy to inspect before you write the code that handles it.

Building Adal: predictable webhook delivery without black boxes

Yuriy — Thu, 11 Jun 2026 16:29:50 +0000

Hi DEV community,

I'm Yuriy, a fullstack developer and DevOps engineer with 10+ years of experience. I'm currently building Adal — a webhook delivery and observability platform for developers.

The idea came from a simple frustration: webhook delivery often feels like a black box.

A provider sends a request. Something fails. Maybe the endpoint was down, maybe the payload was different than expected, maybe the retry happened later, maybe the logs are incomplete. As developers, we often end up guessing instead of debugging from clear facts.

Adal is my attempt to make webhook delivery more predictable.

The core principles are simple:

permanent endpoint URLs;
full request visibility;
delivery logs that explain what happened;
replay without waiting for the provider;
retries that are visible and controlled;
no hidden transformations;
infrastructure that can be inspected and reasoned about.

I'm building Adal as a practical developer tool, but also as an engineering experiment: how much of webhook infrastructure can be made explicit, observable, and boring in a good way?

The stack is mostly Go, PostgreSQL, Redis, React, Docker, Cloudflare R2, and a regional architecture for receiving and delivering webhook requests. The CLI is open source and written in Go.

On this DEV organization page, I plan to write about the technical side of building Adal: webhook infrastructure, backend reliability, regional delivery, observability, product decisions, failures, trade-offs, and lessons from building a SaaS from scratch.

This is still an early-stage project, but it is already public and usable.

If you work with webhooks, build integrations, or care about predictable backend systems, I'd be happy to hear your feedback.