Before the 2 AM Page: 5 Questions I Ask Every .NET Service Before It Ships

Read enough postmortems and you notice something: the monsters are never new. A message arrives twice and a customer is charged twice. A partner endpoint goes down and takes your whole delivery queue with it. An AI agent calls a write tool in a loop. One missing Where shows tenant A's invoices to tenant B. A 4,000-line class makes every fix break something two screens away.

None of these need clever engineering to prevent. They need someone to ask the boring question before launch instead of after the page. So I turned them into a short pre-mortem I run on every .NET service before it ships. Five questions, one small safeguard each. The code is the shape, not the full thing, and the trimmed parts are marked.

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. Complete, tested .NET 10 + AI projects with full source code and in-depth PDF guides, with new ones added regularly. Join Tech Skill Builder

Question 1: "What happens if this message arrives twice?"

Not "if". When. Stripe says outright that an endpoint can receive the same event more than once. Message brokers redeliver after a crash. A mobile client retries on a flaky connection. Any handler that changes money or state will eventually see a duplicate.

The answer I want to hear is "nothing": the second copy is noticed and acknowledged, and no work runs twice. The cheapest reliable way is to let the database decide, with a unique index on the message id:

// Model configuration: the unique index is the real lock, not an if-check in C#
b.Entity<InboxMessage>().HasIndex(m => m.MessageId).IsUnique();

public async Task HandleAsync(OrderPaid msg, CancellationToken ct)
{
    db.Inbox.Add(new InboxMessage(msg.MessageId, DateTimeOffset.UtcNow));
    try { await db.SaveChangesAsync(ct); }
    catch (DbUpdateException) { return; }   // seen before: ack and do nothing
                                            // (real code checks it really was the unique violation)

    // record the follow-up work in the same transaction, process it in a background worker
    // (the transaction and worker are omitted here)
}

Two follow-ups I always ask: is the id the sender's id (not one you generate on arrival), and does the endpoint answer fast? A handler that takes 12 seconds invites the sender to time out and retry, which is how one payment turns into three receipts.

Question 2: "What happens when the thing I call is down for an hour?"

This one is about the calls you make, for example webhooks you send to customers. One customer's endpoint dies. The sender retries in a tight loop, threads pile up waiting on timeouts, and every healthy customer's deliveries stall behind the dead one.

What I want to see: deliveries live in a table or queue, not in a request thread. Each one has its own next-attempt time, the delay grows with every failure, it has jitter so retries don't arrive in waves, and there's a cap after which the delivery is parked for a human:

static DateTimeOffset? NextAttemptAt(int attempt, DateTimeOffset now)
{
    if (attempt >= MaxAttempts) return null;                  // park it in the dead-letter store
    var ceiling = Math.Min(3600, Math.Pow(2, attempt));       // seconds, capped at one hour
    return now + TimeSpan.FromSeconds(Random.Shared.NextDouble() * ceiling);  // "full jitter"
}

// The worker only picks rows where NextAttemptAt <= now, with a short HttpClient timeout,
// so one dead endpoint can't block anyone else. (Worker loop and replay endpoint omitted.)

The dead-letter store matters as much as the backoff. "We gave up" must be something you can see and replay, not a log line from three days ago.

Question 3: "What can the AI agent do without a human?"

The story goes like this: the agent has a refund_order tool, a confused prompt makes it refund the wrong order, then it "double-checks" by calling the tool again. Nobody approved anything, and the token bill for the conversation beats the refund.

So I list every tool and sort them into two piles: reads (look up an order, search the docs) and writes (refund, cancel, email the customer). Reads can run on their own. Writes wait for a person. Then I ask what stops a loop. In Microsoft.Extensions.AI both answers fit in a few lines:

IChatClient agent = new ChatClientBuilder(innerClient)
    .UseFunctionInvocation(configure: f => f.MaximumIterationsPerRequest = 5)  // no endless tool loops
    .Build();

var tools = new List<AITool>
{
    AIFunctionFactory.Create(LookupOrder, name: "lookup_order"),                                 // read: runs
    new ApprovalRequiredAIFunction(AIFunctionFactory.Create(RefundOrder, name: "refund_order")), // write: waits
};
// The host still has to show the approval request to a human and send the answer back.
// That loop, the audit trail and the per-run cost budget are left out here.

One warning: ApprovalRequiredAIFunction only marks the tool. If your host never surfaces the approval request, nothing is gated. And a system prompt that says "please ask before refunding" is not a control at all.

The third part is a budget per run, checked in code, plus evals in CI so a prompt change that makes the agent trigger-happy fails a build instead of a customer.

Question 4: "What stops tenant A from seeing tenant B's data?"

If the honest answer is "every developer remembers to add the filter", that's the horror story waiting to happen. Put the filter in the model once, with an EF Core global query filter on every tenant-owned entity (b.Entity<Invoice>().HasQueryFilter(i => i.TenantId == _tenant.Id) in OnModelCreating). Then make it provable with a test that would have caught the bug:

[Fact]
public async Task Tenant_B_never_sees_tenant_A_invoices()
{
    await using (var asA = TestDb.For(TenantA))      // helper that builds a DbContext for one tenant (omitted)
    {
        asA.Invoices.Add(new Invoice(/* ... */));
        await asA.SaveChangesAsync();
    }

    await using var asB = TestDb.For(TenantB);
    Assert.Empty(await asB.Invoices.ToListAsync());
}

Two places I check by hand in code review: every IgnoreQueryFilters() call (each one needs a reason), and anything that doesn't go through EF at all. If you added a RAG chatbot, the vector search is exactly that. It needs its own tenant filter, or the model will happily quote another customer's documents. And meter AI usage per tenant, so one customer can't burn everyone's credits.

Question 5: "Can someone change this file without fear?"

The scariest item isn't an outage. It's the controller where the database calls, the email sender and the business rules all live together, so nobody touches it, so it never gets better.

I don't ask "is it Clean Architecture?". I ask: is there one obvious home for a business rule, and can I test that rule without a database? Then I keep the answer from rotting with an architecture test that fails the build when the core starts depending on infrastructure:

[Fact]
public void Domain_does_not_depend_on_infrastructure()
{
    var result = Types.InAssembly(typeof(Order).Assembly)          // NetArchTest.Rules
        .ShouldNot().HaveDependencyOnAny("Microsoft.EntityFrameworkCore", "Microsoft.AspNetCore")
        .GetResult();

    Assert.True(result.IsSuccessful);
}

It won't fix a team. It does stop the next shortcut from quietly undoing the last refactor.

The pre-mortem checklist

• [ ] Every handler that changes state survives a duplicate (unique index on the sender's id)
• [ ] Endpoints that receive webhooks acknowledge fast and do the slow work in the background
• [ ] Outgoing calls retry with capped, jittered backoff and end in a dead-letter store you can replay
• [ ] Agent write tools need a human, tool loops have a cap, every run has a budget
• [ ] Tenant filters live in the model, a test proves isolation, vector search filters by tenant too
• [ ] An architecture test keeps the core free of infrastructure

None of this is exciting, and that's the point. The teams that sleep through the night aren't luckier. They answered these five before the 2 AM page did it for them.

Go deeper with complete projects

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. There is no "horror stories" project in it. What it has is complete, working, tested .NET 10 + AI projects with full source code and in-depth PDF guides, and several of them are the long answer to the questions above: human approval for agent tools with ApprovalRequiredAIFunction, permission-aware RAG that keeps one user's documents out of another user's answers, CI evals for AI agents, and fail-closed guardrails. New projects are added regularly.

Join Tech Skill Builder for $5/month

Which of these five questions would your current service fail?