DEV Community

Cover image for The .NET Production Checklist Is Really a List of Places the Framework Lies to You
Kazem
Kazem

Posted on

The .NET Production Checklist Is Really a List of Places the Framework Lies to You

Most of this checklist isn't about .NET at all. It's a list of every place the framework's defaults will quietly hurt you in production, because the happy path was built to make dotnet run work on your laptop, not to survive three replicas behind a load balancer with a database that has a connection limit.

None of these are exotic problems. They're the kind of thing you only learn by getting paged for them once.

Migrations: the happy path runs them on every replica

The default story is simple: your app starts up, EF Core applies pending migrations, you're live. It works great with one instance.

It falls apart the moment you scale past one. Three replicas start at roughly the same time, all three try to apply the same migration, and now you're debugging a race condition in your schema instead of your code. Sometimes it's harmless: the migration is idempotent, one replica wins, the others fail quietly and move on. Sometimes it isn't, and you get a half-applied schema or a deadlock on the migrations history table.

The fix is boring: migrations run as a single deploy step, not as part of application startup. One job, one shot, before any replica of the new version starts serving traffic. It's less convenient than "it just happens," but "it just happens" is exactly the kind of behavior that's fine until it isn't.

Health checks: "alive" and "ready" are not the same question

/health as a single endpoint answers one question badly. What you actually need are two different questions with two different consequences:

  • Is the process alive? (Should Kubernetes restart the pod?)
  • Is the process ready to take traffic? (Should the load balancer route to it?)

A service can be alive and not ready — it's up, but its database connection is down, or it's still warming a cache. If your only health check conflates the two, either you route traffic to a pod that can't serve it, or you restart a pod that's actually fine and just waiting on a dependency to come back.

Split them: /health/live checks that the process itself is up. /health/ready checks that its dependencies are reachable. Only /health/ready gates traffic. This is a small amount of extra wiring for a failure mode that otherwise shows up as "why did we get 500s during a routine dependency blip."

Shutdown: three timeouts, each one has to be bigger than the last

Graceful shutdown looks like it should be free — the framework gives you IHostApplicationLifetime, you hook ApplicationStopping, done. But shutdown in a container orchestrator is actually a chain of three separate timeouts, and if you only configure the first one, the other two will cut you off anyway:

  1. IHostApplicationLifetime: your code's chance to stop accepting new work and finish what's in flight.
  2. ShutdownTimeout: how long the host waits for that to happen before it forces termination. This has to be set above your longest in-flight request, or the host kills requests that were about to finish cleanly.
  3. terminationGracePeriodSeconds: how long Kubernetes waits before sending SIGKILL. This has to be set above ShutdownTimeout, or Kubernetes kills the process before your own shutdown timeout even gets a chance to fire.

Get the ordering wrong and you'll see intermittent failed requests during every deploy, and they'll look like application bugs. They're not. They're a shutdown grace period that's smaller than the request it was supposed to protect.

Connection pools: the number that has to be true across every replica

This one is a simple formula that's easy to forget applies at all once you're running more than one instance:

Max Pool Size × replicas < DB max_connections
Enter fullscreen mode Exit fullscreen mode

A pool size of 100 is fine for one replica against a database that allows 200 connections. Scale to three replicas and you're asking for 300 connections against a limit of 200. That shows up as intermittent connection exhaustion under load, usually during a traffic spike, which is exactly when you have the least patience for debugging it.

Pair this with actual resiliency: EnableRetryOnFailure() for transient database errors, and a standard resilience handler on your outbound HTTP clients. Networks fail. The question is whether your service treats that as routine or as an unhandled exception.

The runtime image: no SDK, no root

FROM mcr.microsoft.com/dotnet/aspnet:10.0-noble-chiseled AS runtime
WORKDIR /app
COPY out .
USER $APP_UID
ENTRYPOINT ["./Sample.Api"]
Enter fullscreen mode Exit fullscreen mode

Two things happen here that are easy to skip if you just want a working Dockerfile. This is a chiseled/distroless image: no shell, no package manager, no SDK, nothing beyond what the app needs to run. If someone gets code execution inside the container, there isn't much there to pivot with. On top of that, USER $APP_UID runs the process as a non-root user instead of the container default.

Neither of these makes the app faster or fixes a bug you'll notice in staging. They're the difference between a compromised container being a dead end and being a foothold. The publish step that gets you here is also worth keeping:

dotnet publish src/Sample.Api -c Release -o out /p:PublishReadyToRun=true
Enter fullscreen mode Exit fullscreen mode

PublishReadyToRun=true precompiles a chunk of the IL to native code ahead of time, which matters for the next item.

Warm-up: the first requests are the slowest ones

.NET JITs your code as it runs: tier-0 first, optimizing later tiers as methods get hot. That's a reasonable tradeoff for a long-running process. It's a bad one for the first few requests a brand-new pod receives, because those requests get compiled cold, right when they're also the ones deciding whether your rollout looks healthy.

The fix is to send a warm-up request after deploy, before the pod joins the load balancer. It costs a few seconds. What it buys you is not showing your actual users the slowest version of every code path, right as a new deploy goes live.

Everything else on the list is the same pattern

Structured logging to stdout, so your log aggregator doesn't have to parse whatever format someone chose two years ago. OpenTelemetry traces and metrics with a correlation ID that survives a hop across services, so a slow request can actually be traced instead of guessed at. Response compression and output caching where they pay off, not everywhere, just where the numbers say so. Resource limits and DOTNET_GCHeapHardLimitPercent set explicitly, because an unbounded GC heap in a container with a memory limit is just a slower way of getting OOMKilled. Backups verified by an actual restore, because a backup nobody has restored is a hope, not a backup.

None of these are interesting individually. What they have in common is that they're all invisible until the day they aren't — and by then it's an incident, not a checklist item.

The CI gate that keeps this from being a one-time exercise

The checklist part is only half of it. None of it stays true unless CI enforces it on every change:

dotnet format --verify-no-changes → dotnet build -warnaserror → dotnet test → vulnerability scan
Enter fullscreen mode Exit fullscreen mode

Format check first, so style drift doesn't show up as noise in review. Build with warnings as errors, so a warning today doesn't become a bug next quarter because everyone got used to ignoring the build output. Tests. Then a vulnerability scan against your dependencies, because dotnet list package --vulnerable catching something in CI is a much better day than catching it after it's in production.

The tooling underneath is less important than the discipline of running it on every PR instead of remembering to run it manually before a release: CSharpier or dotnet format for formatting, Roslyn analyzers and SonarAnalyzer for static analysis, NetArchTest or ArchUnitNET if you want architecture rules enforced by a test rather than a code review comment, and Husky.NET for pre-commit hooks.

Why this checklist reads the way it does

Every item here comes from the same root cause: the defaults optimize for one instance on your machine, not for several instances that all have to agree on the same schema, the same connection budget, and the same shutdown sequence at once. Add an orchestrator with its own lifecycle rules on top, and that gap becomes the whole checklist.

None of this is specific to .NET's quirks. It's what "production" means once more than one thing has to agree on the state of the world at the same time.

Top comments (0)