The first version of our webhook system was about twenty lines of code. On OrderCreated, we grabbed the customer's endpoint URL, built a JSON payload, and called HttpClient.PostAsync right there in the request pipeline. It worked in the demo. It worked for the first few customers. Then it broke, three separate times, in three different ways — and each time taught us something the "just POST it" mental model doesn't prepare you for.
This is the writeup of what we ended up building, why each piece exists, and the mistakes that got us there. It’s ASP.NET Core and EF Core, but the ideas apply to any stack.
Incident #1: the silent drop
A customer emailed support asking why their payment-captured webhook never arrived. Nothing in our logs showed a failure — the HttpClient.PostAsync call had thrown an exception (their endpoint was mid-deploy and refused the connection), and our code caught it, logged a warning, and moved on. The event was gone. No retry, no record, no way to know it had ever been attempted.
That’s the core problem with firing webhooks inline: the delivery only exists in memory, for the duration of one HTTP call. If it fails, or if your process dies between “handled the event” and “sent the webhook,” the event disappears. There’s no debugging trail, because there’s nothing durable to look at.
The fix isn’t “add a retry loop.” It’s a change of model: don’t send webhooks, record deliveries. A background worker sends them later, and can keep trying until it succeeds or gives up in a way you can observe.
Incident #2: the double-send
Once we had a background worker polling a WebhookDeliveries table and sending pending rows, we scaled it to two instances for redundancy. Within a day, a customer got the same order.created webhook twice, four seconds apart. Two workers had picked up the same row in the same polling cycle, both sent it, both marked it delivered.
This is where “at-least-once delivery” stops being a slogan and starts being something you actually have to design for — both on the sending side (don’t claim the same row twice) and on the receiving side (idempotency keys aren’t optional, they’re the contract).
Incident #3: the vanishing transaction
The last one was subtler. We inserted the delivery row in a separate SaveChangesAsync call, right after the one that created the order. Most of the time this was fine. Then a deploy went out mid-request, the process was killed between the two calls, and the order existed with no corresponding webhook ever queued. No error, no retry — the event simply never entered the system.
This is the “dual write” problem: two separate writes (business state + webhook intent) that need to succeed or fail together, but aren’t wrapped in anything that guarantees that.
With those three lessons behind us, here’s the system we ended up with.
Architecture
We don’t send webhooks inside the request. We write a delivery intent to the database, in the same transaction as the business change that caused it. A background worker later claims pending deliveries, with real row-level locking so two instances can’t grab the same one, and sends them with retries, backoff, and a circuit breaker per subscriber.
Request → DB write (order + delivery row, same transaction)
→ WebhookDeliveries table (Pending)
→ Worker claims row (SELECT ... FOR UPDATE SKIP LOCKED)
→ HTTP POST with HMAC signature
→ Success: Delivered
→ Failure: backoff, retry, eventually Failed (dead letter)
The delivery row being written in the same transaction as the order is the fix for Incident #3 — this is the transactional outbox pattern applied narrowly to webhooks. If the order commit fails, there’s no orphaned delivery. If it succeeds, the delivery is guaranteed to exist.
Domain model
public enum WebhookEventType
{
OrderCreated = 0,
PaymentCaptured = 1,
SubscriptionCancelled = 2
}
public enum WebhookDeliveryStatus
{
Pending = 0,
Processing = 1,
Delivered = 2,
Failed = 3,
Cancelled = 4
}
Subscriptions carry not just one secret, but room for a previous one — we’ll get to why in the rotation section.
public sealed class WebhookSubscription
{
public Guid Id { get; set; }
public string TenantId { get; set; } = string.Empty;
public Uri EndpointUrl { get; set; } = default!;
public string Secret { get; set; } = string.Empty;
public string? PreviousSecret { get; set; }
public DateTimeOffset? PreviousSecretExpiresOnUtc { get; set; }
public bool IsActive { get; set; }
public DateTimeOffset CreatedOnUtc { get; set; }
}
The delivery row is the thing everything else revolves around:
public sealed class WebhookDelivery
{
public Guid Id { get; set; }
public Guid SubscriptionId { get; set; }
public WebhookEventType EventType { get; set; }
public WebhookDeliveryStatus Status { get; set; }
public string Payload { get; set; } = string.Empty;
public int AttemptCount { get; set; }
public int MaxAttempts { get; set; } = 10;
public DateTimeOffset AvailableOnUtc { get; set; }
public DateTimeOffset CreatedOnUtc { get; set; }
public DateTimeOffset? DeliveredOnUtc { get; set; }
public int? LastStatusCode { get; set; }
public string? LastError { get; set; }
public WebhookSubscription Subscription { get; set; } = default!;
}
Unlike our first version, we don’t overwrite LastError and lose history. A separate table keeps every attempt:
public sealed class WebhookDeliveryAttempt
{
public Guid Id { get; set; }
public Guid WebhookDeliveryId { get; set; }
public int AttemptNumber { get; set; }
public int? StatusCode { get; set; }
public string? Error { get; set; }
public DateTimeOffset AttemptedOnUtc { get; set; }
}
This one table is what turned support tickets from “let me check the logs” into “let me query this delivery’s attempt history” — worth the extra table on its own.
Writing the delivery in the same transaction as the business change
This is the piece that fixes Incident #3. The enqueuer doesn’t call SaveChangesAsync itself — it adds to the same DbContext the caller is already using, so the commit is atomic with whatever business operation triggered the event.
public sealed class WebhookDeliveryEnqueuer(WebhooksDbContext dbContext)
{
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);
public async Task EnqueueAsync<TData>(
string tenantId,
WebhookEventType eventType,
string eventName,
TData data,
CancellationToken cancellationToken)
{
var subscriptions = await dbContext.WebhookSubscriptions
.Where(s => s.TenantId == tenantId && s.IsActive)
.ToListAsync(cancellationToken);
var payload = new WebhookPayload<TData>(
Guid.NewGuid(), eventName, DateTimeOffset.UtcNow, data);
var json = JsonSerializer.Serialize(payload, JsonOptions);
foreach (var subscription in subscriptions)
{
dbContext.WebhookDeliveries.Add(new WebhookDelivery
{
Id = Guid.NewGuid(),
SubscriptionId = subscription.Id,
EventType = eventType,
Status = WebhookDeliveryStatus.Pending,
Payload = json,
AvailableOnUtc = DateTimeOffset.UtcNow,
CreatedOnUtc = DateTimeOffset.UtcNow
});
}
// Deliberately no SaveChangesAsync here - the caller commits
// this together with the business write that triggered the event.
}
}
public sealed record WebhookPayload<T>(
Guid EventId,
string EventType,
DateTimeOffset OccurredOnUtc,
T Data);
If your write volume is high enough that adding rows to the same transaction becomes a bottleneck, the standard next step is a proper outbox table plus a separate relay process — worth its own article, and it’s on the list at the end. For most systems, the same-transaction approach above is enough.
Claiming deliveries without double-sending
This is the fix for Incident #2. LockedUntilUtc with a plain WHERE clause is optimistic — it doesn't stop two workers from reading the same candidate rows in the same instant, before either has written its lock back. What actually stops that is a database-level row lock. In Postgres:
public sealed class WebhookDeliveryClaimer(WebhooksDbContext dbContext)
{
public async Task<IReadOnlyList<WebhookDelivery>> ClaimAsync(
int batchSize,
CancellationToken cancellationToken)
{
var now = DateTimeOffset.UtcNow;
var ids = await dbContext.Database
.SqlQuery<Guid>($"""
SELECT "Id" FROM "WebhookDeliveries"
WHERE "Status" = {(int)WebhookDeliveryStatus.Pending}
AND "AvailableOnUtc" <= {now}
ORDER BY "CreatedOnUtc"
LIMIT {batchSize}
FOR UPDATE SKIP LOCKED
""")
.ToListAsync(cancellationToken);
if (ids.Count == 0)
{
return [];
}
var deliveries = await dbContext.WebhookDeliveries
.Include(d => d.Subscription)
.Where(d => ids.Contains(d.Id))
.ToListAsync(cancellationToken);
foreach (var delivery in deliveries)
{
delivery.Status = WebhookDeliveryStatus.Processing;
}
await dbContext.SaveChangesAsync(cancellationToken);
return deliveries;
}
}
FOR UPDATE SKIP LOCKED tells Postgres: lock the rows you select, and if another transaction already has some of the candidate rows locked, skip them instead of waiting. Two workers running this at the same instant get disjoint sets of rows — no coordination needed beyond what the database already gives you for free. This one query is the difference between "usually doesn't double-send" and "cannot double-send."
Signing and sending
We sign the timestamp together with the payload, so a captured request can’t be replayed indefinitely:
public static class WebhookSignature
{
public static string Create(string secret, DateTimeOffset timestamp, string payload)
{
var signedPayload = $"{timestamp.ToUnixTimeSeconds()}.{payload}";
using var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(secret));
var hash = hmac.ComputeHash(Encoding.UTF8.GetBytes(signedPayload));
return Convert.ToHexString(hash).ToLowerInvariant();
}
}
One thing we changed after launch: our header names used to be entirely made up (X-MyApp-Signature and friends). If you're building this for external integrators, it's worth aligning with the emerging Standard Webhooks convention instead of inventing your own — webhook-id, webhook-timestamp, webhook-signature, with the signature as v1,. Integrators increasingly have libraries that already expect this shape, and you save every future customer a support ticket.
public sealed class WebhookHttpSender(HttpClient httpClient)
{
public async Task<HttpResponseMessage> SendAsync(
WebhookDelivery delivery, CancellationToken cancellationToken)
{
var timestamp = DateTimeOffset.UtcNow;
var signature = WebhookSignature.Create(
delivery.Subscription.Secret, timestamp, delivery.Payload);
using var request = new HttpRequestMessage(HttpMethod.Post, delivery.Subscription.EndpointUrl);
request.Content = new StringContent(delivery.Payload, Encoding.UTF8, "application/json");
request.Headers.Add("webhook-id", delivery.Id.ToString("N"));
request.Headers.Add("webhook-timestamp", timestamp.ToUnixTimeSeconds().ToString(CultureInfo.InvariantCulture));
request.Headers.Add("webhook-signature", $"v1,{signature}");
return await httpClient.SendAsync(request, cancellationToken);
}
}
Not hammering a dead endpoint
Our original retry loop kept hitting a failing endpoint every couple of seconds regardless of how consistently it was failing. Once a customer’s server started returning 500 for everything for an hour, we were sending it thousands of pointless requests. Polly’s circuit breaker fixes this at the HttpClient level, scoped per subscription:
builder.Services.AddHttpClient<WebhookHttpSender>(client =>
{
client.Timeout = TimeSpan.FromSeconds(10);
})
.AddResilienceHandler("webhook-pipeline", pipeline =>
{
pipeline.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 0 // retries are handled by our own backoff scheduling, not Polly
});
pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
FailureRatio = 0.5,
SamplingDuration = TimeSpan.FromSeconds(30),
MinimumThroughput = 5,
BreakDuration = TimeSpan.FromSeconds(60)
});
});
The circuit breaker doesn’t replace our own retry/backoff bookkeeping in the database — it protects us from wasting a connection attempt on an endpoint that’s currently failing consistently, and fails fast so the worker can move on to other deliveries instead of waiting out a timeout.
The worker
public sealed class WebhookDeliveryWorker(
IServiceScopeFactory scopeFactory,
ILogger<WebhookDeliveryWorker> logger) : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using var timer = new PeriodicTimer(TimeSpan.FromSeconds(2));
while (await timer.WaitForNextTickAsync(stoppingToken))
{
await ProcessBatchAsync(stoppingToken);
}
}
private async Task ProcessBatchAsync(CancellationToken cancellationToken)
{
await using var scope = scopeFactory.CreateAsyncScope();
var dbContext = scope.ServiceProvider.GetRequiredService<WebhooksDbContext>();
var claimer = scope.ServiceProvider.GetRequiredService<WebhookDeliveryClaimer>();
var sender = scope.ServiceProvider.GetRequiredService<WebhookHttpSender>();
var deliveries = await claimer.ClaimAsync(batchSize: 20, cancellationToken);
foreach (var delivery in deliveries)
{
await SendDeliveryAsync(dbContext, sender, delivery, cancellationToken);
}
}
private async Task SendDeliveryAsync(
WebhooksDbContext dbContext,
WebhookHttpSender sender,
WebhookDelivery delivery,
CancellationToken cancellationToken)
{
delivery.AttemptCount++;
var attempt = new WebhookDeliveryAttempt
{
Id = Guid.NewGuid(),
WebhookDeliveryId = delivery.Id,
AttemptNumber = delivery.AttemptCount,
AttemptedOnUtc = DateTimeOffset.UtcNow
};
try
{
using var response = await sender.SendAsync(delivery, cancellationToken);
attempt.StatusCode = (int)response.StatusCode;
delivery.LastStatusCode = attempt.StatusCode;
if (response.IsSuccessStatusCode)
{
delivery.Status = WebhookDeliveryStatus.Delivered;
delivery.DeliveredOnUtc = DateTimeOffset.UtcNow;
delivery.LastError = null;
}
else if (IsPermanentFailure(response.StatusCode))
{
delivery.Status = WebhookDeliveryStatus.Failed;
delivery.LastError = $"Endpoint returned {(int)response.StatusCode} (not retried).";
}
else
{
ScheduleRetry(delivery, $"Endpoint returned {(int)response.StatusCode}.");
}
}
catch (Exception ex)
{
logger.LogWarning(ex, "Webhook delivery {DeliveryId} failed", delivery.Id);
attempt.Error = ex.Message;
ScheduleRetry(delivery, ex.Message);
}
dbContext.Set<WebhookDeliveryAttempt>().Add(attempt);
await dbContext.SaveChangesAsync(cancellationToken);
}
// Most 4xx responses mean "this will never succeed, stop retrying."
// 408/409/425/429 are the exceptions - they're explicitly transient.
private static bool IsPermanentFailure(HttpStatusCode statusCode) =>
(int)statusCode is >= 400 and < 500
&& statusCode is not (HttpStatusCode.RequestTimeout
or HttpStatusCode.Conflict
or (HttpStatusCode)425
or HttpStatusCode.TooManyRequests);
private static void ScheduleRetry(WebhookDelivery delivery, string error)
{
delivery.LastError = error;
if (delivery.AttemptCount >= delivery.MaxAttempts)
{
delivery.Status = WebhookDeliveryStatus.Failed;
return;
}
delivery.Status = WebhookDeliveryStatus.Pending;
delivery.AvailableOnUtc = DateTimeOffset.UtcNow.Add(CalculateBackoff(delivery.AttemptCount));
}
private static TimeSpan CalculateBackoff(int attempt)
{
var seconds = Math.Min(3600, Math.Pow(2, attempt) * 10);
var jitter = Random.Shared.NextDouble() * 0.3 * seconds; // avoid thundering herd on shared outages
return TimeSpan.FromSeconds(seconds + jitter);
}
}
The 4xx-vs-retry distinction is one of those details that’s easy to skip and expensive to skip: without it, a customer who fat-fingers their endpoint URL (a permanent 404) gets ten identical retries spread over an hour instead of an immediate, actionable failure.
Not letting one noisy tenant starve everyone else
Once we had a few large customers with dozens of subscriptions each, a batch import from one tenant could fill the claim queue and delay delivery for every other tenant. We cap concurrent in-flight sends per tenant using .NET’s built-in rate limiter:
builder.Services.AddSingleton<PartitionedRateLimiter<string>>(_ =>
PartitionedRateLimiter.Create<string, string>(tenantId =>
RateLimitPartition.GetConcurrencyLimiter(tenantId, _ => new ConcurrencyLimiterOptions
{
PermitLimit = 10,
QueueLimit = 100,
QueueProcessingOrder = QueueProcessingOrder.OldestFirst
})));
The worker acquires a lease for the delivery’s TenantId before sending and releases it afterward — a five-line addition that turned "one customer's import degrades everyone's webhooks" into a non-issue.
Dead letters and replay
Deliveries that exhaust MaxAttempts land in Failed. That's not the end of the story — sometimes the customer's endpoint was down for an hour and is fine now. We keep a small admin endpoint to requeue:
app.MapPost("/admin/webhook-deliveries/{id:guid}/replay",
async (Guid id, WebhooksDbContext db, CancellationToken ct) =>
{
var delivery = await db.WebhookDeliveries.FindAsync([id], ct);
if (delivery is null || delivery.Status != WebhookDeliveryStatus.Failed)
{
return Results.NotFound();
}
delivery.Status = WebhookDeliveryStatus.Pending;
delivery.AttemptCount = 0;
delivery.AvailableOnUtc = DateTimeOffset.UtcNow;
await db.SaveChangesAsync(ct);
return Results.Ok();
});
Because WebhookDeliveryAttempt kept full history instead of just the last error, replay decisions are actually informed — you can see whether all ten attempts got connection refused (endpoint was down) or a mix of 401s (probably a rotated secret the customer hasn't updated).
Rotating secrets without breaking anyone
The first time we needed to rotate a leaked secret, our only option was “change it and hope the customer updates their config before the next event.” That’s a bad conversation to have. Now rotation supports a grace period — sign with both secrets, let the receiver accept either:
public static class WebhookSignatureVerifier
{
public static bool IsValid(string secret, string? previousSecret,
string timestampHeader, string signatureHeader, string payload)
{
if (!long.TryParse(timestampHeader, out var unixSeconds))
{
return false;
}
var timestamp = DateTimeOffset.FromUnixTimeSeconds(unixSeconds);
if (DateTimeOffset.UtcNow - timestamp > TimeSpan.FromMinutes(5))
{
return false; // reject stale requests - mitigates replay
}
var candidates = new[] { secret, previousSecret }
.Where(s => !string.IsNullOrEmpty(s));
return candidates.Any(candidate =>
{
var expected = WebhookSignature.Create(candidate!, timestamp, payload);
return CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(expected),
Encoding.UTF8.GetBytes(signatureHeader.Replace("v1,", "")));
});
}
}
PreviousSecretExpiresOnUtc gets checked by a cleanup job that nulls out the old secret once the grace period passes — rotation stops being a coordinated event and becomes something you can do any time.
Idempotent receivers (the other half of the contract)
Everything above guarantees at-least-once delivery, never exactly-once. If you’re on the receiving end of somebody else’s webhooks — or writing example code for your own integrators — this is the piece that’s usually missing from writeups:
app.MapPost("/webhooks/incoming", async (
HttpRequest request,
IDistributedCache cache,
CancellationToken ct) =>
{
var webhookId = request.Headers["webhook-id"].ToString();
if (string.IsNullOrEmpty(webhookId))
{
return Results.BadRequest();
}
var cacheKey = $"webhook-processed:{webhookId}";
var alreadyProcessed = await cache.GetStringAsync(cacheKey, ct);
if (alreadyProcessed is not null)
{
return Results.Ok(); // already handled - respond success, do nothing
}
using var reader = new StreamReader(request.Body);
var payload = await reader.ReadToEndAsync(ct);
// ... verify signature, then process payload ...
await cache.SetStringAsync(cacheKey, "1",
new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromDays(7) }, ct);
return Results.Ok();
});
Seven days of dedup window comfortably outlives any retry schedule you’d reasonably configure on the sending side.
Seeing it happen: observability
The thing we didn’t have during Incident #1 was any way to see the problem before a customer reported it. Now every attempt is a traced span:
private static readonly ActivitySource ActivitySource = new("Webhooks.Delivery");
using var activity = ActivitySource.StartActivity("webhook.delivery.attempt");
activity?.SetTag("webhook.subscription.id", delivery.SubscriptionId);
activity?.SetTag("webhook.delivery.id", delivery.Id);
activity?.SetTag("webhook.attempt", delivery.AttemptCount);
And a couple of counters feed a dashboard that answers “is delivery healthy right now” at a glance:
private static readonly Meter Meter = new("Webhooks.Delivery");
private static readonly Counter<long> Delivered = Meter.CreateCounter<long>("webhook.delivered");
private static readonly Counter<long> FailedPermanently = Meter.CreateCounter<long>("webhook.failed");
private static readonly ObservableGauge<int> BacklogDepth = Meter.CreateObservableGauge(
"webhook.backlog.depth", () => GetPendingCount());
Backlog depth is the one that would have caught Incident #1 immediately instead of via a support ticket — a pending count that only grows tells you something is wrong well before any individual customer notices.
Testing the concurrency claim, not just the happy path
The bug from Incident #2 would have shown up in a test that actually exercises two workers against a real database — an in-memory provider hides row-locking behavior entirely.
[Fact]
public async Task Two_concurrent_claimers_never_claim_the_same_row()
{
await using var postgres = new PostgreSqlBuilder().Build();
await postgres.StartAsync();
// seed 50 pending deliveries...
var options = new DbContextOptionsBuilder<WebhooksDbContext>()
.UseNpgsql(postgres.GetConnectionString()).Options;
async Task<IReadOnlyList<WebhookDelivery>> ClaimBatch()
{
await using var db = new WebhooksDbContext(options);
return await new WebhookDeliveryClaimer(db).ClaimAsync(25, default);
}
var results = await Task.WhenAll(ClaimBatch(), ClaimBatch());
var claimedIds = results.SelectMany(r => r.Select(d => d.Id)).ToList();
Assert.Equal(claimedIds.Count, claimedIds.Distinct().Count());
}
Testcontainers spinning up a real Postgres for this test is worth the extra CI time — it’s the only way to actually prove FOR UPDATE SKIP LOCKED is doing what you think it's doing, rather than trusting that it is.
For the sending side, WireMock.Net stands in for the customer’s endpoint and lets you script exactly the failure sequence you want to verify against (timeout, then 500, then 200 — does the delivery end up Delivered with the right attempt count?).
Try it locally without a real customer endpoint
You don’t need a live third-party server to exercise any of this — a docker-compose.yml with Postgres and a throwaway "echo" receiver gets you an end-to-end loop on your own machine:
version: "3.8"
services:
postgres:
image: postgres:16
environment:
POSTGRES_DB: webhooks
POSTGRES_PASSWORD: postgres
ports:
- "5432:5432"
webhook-worker:
build: ./src/WebhookWorker
depends_on:
- postgres
environment:
ConnectionStrings__Webhooks: "Host=postgres;Database=webhooks;Username=postgres;Password=postgres"
echo-receiver:
build: ./src/EchoReceiver
ports:
- "5099:8080"
EchoReceiver is a five-line minimal API that logs whatever it receives and returns 200 (or a status code you configure via query string, for testing retries deliberately). Point a WebhookSubscription.EndpointUrl at http://echo-receiver:8080/webhooks and you can watch retries, backoff, and eventual delivery happen in your own logs — no ngrok, no webhook.site, nothing leaving your machine.
If you’d rather not build and maintain any of this yourself, Svix is open source and self-hostable, and implements most of what’s above (plus a lot more) out of the box. Worth evaluating before committing engineering time to a homegrown version — build vs. buy applies here just like anywhere else.
What can still go wrong
+---------------------------------------------+-----------------------------------------------+
| Failure | What we do about it |
+---------------------------------------------+-----------------------------------------------+
| Endpoint is down / unreachable | Delivery stays Pending, retried with backoff |
| Endpoint returns 5xx | Retried with backoff, circuit breaker per sub |
| Endpoint returns 4xx (permanent) | Marked Failed immediately, no retry |
| Endpoint returns 408/409/425/429 | Treated as transient, retried |
| Worker crashes mid-send | Row unlocked once claim window elapses |
| Two workers claim at once | Prevented by FOR UPDATE SKIP LOCKED |
| Process dies between order write and enqueue | Prevented by same-transaction write |
| Secret leaked | Rotate with grace period via PreviousSecret |
| Clock drift between sender and receiver | 5-minute timestamp tolerance on verification |
| Subscription deleted with deliveries queued | Claimer filters by IsActive before sending |
+---------------------------------------------+-----------------------------------------------+
DB polling vs. a message queue
We get asked, reasonably, why the worker polls a table every two seconds instead of consuming from RabbitMQ or Azure Service Bus.
+------------------------+---------------------------------+------------------------------------+
| Concern | DB polling (this article) | Message queue |
+------------------------+---------------------------------+------------------------------------+
| Infra to operate | None beyond the DB you already | A broker to run, monitor, upgrade |
| | have | |
| Delivery latency | Bounded by poll interval (~2s) | Near-instant |
| Replay / dead letter | Just a SQL query | Broker-specific tooling |
| Horizontal scaling | Needs row-level locking | Built into the broker's semantics |
| Operational familiarity | Whoever can write SQL | Requires broker-specific expertise |
+------------------------+---------------------------------+------------------------------------+
For us, at our volume, a two-second worst-case latency to start a delivery attempt was an easy tradeoff against not running another piece of infrastructure. If you’re sending tens of thousands of webhooks a minute, or need sub-second delivery, that math flips — a queue is the better call.
When not to build this at all
If webhooks aren’t core to your product — you have a handful of internal integrations, or losing an occasional notification is genuinely fine — this entire system is over-engineering. A fire-and-forget POST with a single retry is a legitimate choice for low-stakes notifications.
But if customers build automations on top of your events — payments, orders, subscription state, identity — a dropped webhook is a support ticket at best and a broken customer integration at worst. That’s the line where the complexity above earns its keep.
Where we landed
A webhook isn’t just an HTTP request — it’s an integration contract, and contracts need versioning, rotation, and a way to recover when something on either end goes wrong. Every piece in this article exists because we skipped it once and paid for that in a support queue. Durable delivery, real row locking, dead-letter replay, and idempotent receivers turned “we hope this arrived” into something we can actually verify.
What’s next
- A full transactional outbox with a dedicated relay process (we touched on this, deserves its own deep dive)
- Full compliance with the Standard Webhooks spec
- Multi-region webhook delivery and failover
- A self-service webhook delivery dashboard for customers
Tags: Software Development, Dotnet, Csharp, ASP.NET Core, Webhooks, Distributed Systems, Backend Development, System Design
Top comments (0)