Originally published at prepstack.co.in
Generating one PDF report takes 2–4 seconds in a headless Chromium. Easy. Generating 1.2 million PDF reports in 48 hours during end-of-month is a different problem entirely.
Inline generation blocks request threads. A single Chromium OOMs after ~200 pages. The same partner re-downloads the same report 4 times. Workers crash mid-render. By Monday the API is a smoking crater. This is the architecture — built at Mattrx (multi-tenant marketing analytics, .NET 9, Azure SQL, 110k MAU) — with real code and production metrics.
The mental model
accept fast → enqueue → render in a pool → store → notify → cache at the edge. Every box scales independently. Failures retry without blocking the next request. The user never waits on a thread for the render.
1. Accept fast — the API never renders
// POST /api/reports/request returns 202 + jobId immediately (CQRS via MediatR)
app.MapPost("/api/reports/request", async (ReportSpec spec, ISender sender, CancellationToken ct) =>
{
var jobId = await sender.Send(new RequestReport(spec), ct); // validate, persist, enqueue
return Results.Accepted($"/api/reports/{jobId}", new { jobId });
});
The hardline rule: render happens on a worker, a different process, ideally a different App Service Plan. The API never launches Chromium.
2. Bound the workers with a queue
Hangfire drains the queue — retries, idempotency, a visible dashboard. Switch to Azure Service Bus + KEDA past ~5M/day. Workers autoscale 0 → 40 on queue depth and scale-to-zero the other 720 hours/month (Mattrx only peaks 48h/month).
3. Browser pooling — the heart of the optimization
// A pool of long-lived browsers; a fresh PAGE per render. Recycle every ~200 pages.
await using var page = await pool.RentPageAsync(ct);
await page.SetContentAsync(html, new NavigationOptions { WaitUntil = new[] { WaitUntilNavigation.Networkidle0 } });
await page.WaitForSelectorAsync("body[data-charts-ready='1']"); // wait for charts, not a guess
var pdf = await page.PdfDataAsync(new PdfOptions { Format = PaperFormat.A4 });
The optimizations that matter, ranked: browser pooling (~600ms/PDF vs launching per render), Networkidle0 + an explicit charts-ready signal (otherwise PDFs render mid-animation), recycle every ~200 pages (prevents OOM), SetContent instead of file:// navigation, pre-render charts as static SVG. Launching Chromium per PDF is the single most common mistake.
4. Store in blob, cache at the edge — not SQL
PDFs are big binary blobs. Azure Blob (private, hot tier, 7-day lifecycle TTL) with SAS-signed URLs, and Azure Front Door in front. A 38% CDN cache-hit rate (same report re-downloaded) at zero engineering effort is the cheapest performance win there is. SQL is the wrong tier for binaries.
5. Status feedback
SignalR push for the happy path (partner UI feels instant), plus a polling fallback on /api/reports/{jobId} every 3s so it always works.
The failure modes that almost killed us
- Chromium memory leak under load → recycle the browser every N pages.
- Same job rendered twice → idempotency key on the command; the second enqueue is a no-op.
- CDN cached an expired SAS → cache-key must not include the SAS token; set correct TTLs.
- Chart lib used setTimeout for animations → pre-render charts static; wait on an explicit ready flag, not a timer.
- Long job exceeded the Hangfire timeout → chunk the work / raise the invisibility window.
Production metrics (8-month run)
p95 time-to-PDF 9s · avg render 2.4s · failure rate 0.04% (was 4.2%) · per-PDF cost $0.0004 · CDN cache hit 38% · workers scale 0→40 · ~$1,100/mo total infra · ~$1,300/mo saved by sizing the worker pool right.
Three habits prevent 90% of the pain: the API never renders, browser pooling on day one, storage + CDN, not DB.
Full guide with all the code (MediatR command, browser pool, Hangfire job, autoscale rules, Front Door config), the diagrams, and every failure mode in depth is on PrepStack.
Top comments (0)