DEV Community: Pankaj Sharma

When “Works on Lambda” Breaks on ECS: A Concurrency Bug from Misusing Singletons in .NET

Pankaj Sharma — Fri, 01 May 2026 08:07:22 +0000

We recently migrated a .NET Core application from .NET 8 to .NET 10 and moved its hosting model from AWS Lambda to ECS. The service acts as a wrapper around a downstream API that reads and updates client “fact find” data.

Post-migration, we started seeing critical issues:

Users were seeing other users’ data
Updates were being applied to the wrong clients
Users could access data they shouldn’t have permission to see

This was a data isolation failure. The root cause turned out to be a misuse of a singleton combined with differences in execution models between Lambda and ECS.

Architecture Context

The service:

Accepts user requests
Adds headers (auth/user context)
Calls a downstream API
Returns processed responses

A shared configuration object (ConnectionOptions) was introduced to manage headers for outbound calls.

This object:

Was registered as a singleton
Was mutated per request to inject headers

Why It “Worked” on Lambda

AWS Lambda has a different execution model:

Each invocation typically runs in isolation
Even with warm starts, concurrency per instance is limited
No true parallel execution within a single instance (by default)

So:

Mutating a singleton per request appears safe
Concurrency issues are masked

The flawed design went unnoticed.

What Changed in ECS

ECS runs your app as a long-lived service:

Multiple requests are processed concurrently
Threads share the same memory space
Singleton services are shared across all requests

Example

public class ConnectionOptions
{
    public Dictionary<string, string> Headers { get; set; }
}

services.AddSingleton<ConnectionOptions>();

connectionOptions.Headers["Authorization"] = userToken;

The Problem

Two requests arrive at the same time:

Request A	Request B
Sets header to User A	Sets header to User B
Calls downstream API	Calls downstream API

Because both share the same singleton:

Headers overwrite each other
Requests leak state
Downstream API gets incorrect user context

This leads to:

Cross-user data exposure
Incorrect updates
Non-deterministic failures

Root Cause

Mutable shared state in a singleton in a concurrent environment.

More precisely:

ConnectionOptions held request-specific data
It was registered as a singleton
It was mutated per request

This violates a core rule:

Singletons must be stateless or immutable.

Why This Is Hard to Catch

This issue didn’t show up in:

Local development (low concurrency)
Lambda (isolated execution)
Unit tests (typically single-threaded)

It only surfaced under:

Concurrent load
Multi-threaded execution (ECS)

Fixes

Option 1: Use Scoped Lifetime

services.AddScoped<ConnectionOptions>();

Each request gets its own instance.

Option 2: Make It Immutable

public class ConnectionOptions
{
    public IReadOnlyDictionary<string, string> Headers { get; }

    public ConnectionOptions(Dictionary<string, string> headers)
    {
        Headers = headers;
    }
}

Create a new instance per request instead of mutating.

Option 3: Avoid Shared State (Preferred)

Pass headers explicitly:

await client.CallAsync(headers);

Or:

var request = new HttpRequestMessage();
request.Headers.Add("Authorization", token);

This removes shared mutable state entirely.

Key Takeaways

1. Execution Model Matters

Lambda and ECS behave differently:

Lambda hides concurrency issues
ECS exposes them

2. Singleton ≠ Safe

Singletons are:

Shared globally
Unsafe if mutable

Use only for:

Stateless services
Immutable configuration

3. Don’t Store Request Data in Singletons

If data changes per request:

It should not live in a singleton

4. Concurrency Bugs Are Non-Deterministic

They:

Are hard to reproduce
Appear random
Often show up only in production

Final Thought

This wasn’t a .NET 10 issue or an ECS issue.

It was a design flaw that only became visible when the runtime stopped masking it.

When migrating from Lambda to ECS (or any long-running service model), review:

Singleton usage
Shared mutable state
Request-scoped data handling

That’s where subtle, high-impact bugs tend to hide.

Chromium + Puppeteer Broke After ASP.NET 8 10 Upgrade — The Hidden Snap Trap in Docker

Pankaj Sharma — Fri, 01 May 2026 08:05:26 +0000

We upgraded a service from ASP.NET 8 to 10. No application code changed—but our Puppeteer-based document pipeline completely broke.

The root cause wasn’t .NET. It was a subtle base image change:

Debian → Ubuntu, which replaced a working Chromium install with a non-functional Snap wrapper inside Docker.

The Setup

Our document generation pipeline works like this:

Render HTML using Puppeteer
Use Chromium as the headless browser
Post-process assets with Sharp
Run everything inside a Docker container

This setup worked reliably on ASP.NET 8.

What Changed?

As part of the upgrade:

From: mcr.microsoft.com/dotnet/aspnet:8.0 (Debian-based)
To: mcr.microsoft.com/dotnet/aspnet:10.0 (Ubuntu-based)

At first glance, nothing looked risky. But this OS-level change had a critical side effect.

The Real Problem (Root Cause)

On Debian:

apt install chromium installs a real Chromium binary

On Ubuntu:

apt install chromium installs a Snap wrapper, not the browser itself

That distinction is the entire problem.

Why This Breaks in Docker

Snap packages are not just alternative installers—they rely on a runtime model that assumes:

a full init system (systemd)
background services (snapd)
elevated privileges

Docker containers deliberately avoid all of these.

So while apt install chromium appears successful, it installs a deferred execution wrapper, not an actual browser.

When Puppeteer tries to launch Chromium, there is nothing usable to execute.

Symptoms (and Why They’re Misleading)

Installation appears successful:

apt install chromium

But Puppeteer fails with:

Failed to launch the browser process!

On inspection:

which chromium
# /usr/bin/chromium-browser

file /usr/bin/chromium-browser
# POSIX shell script, not a binary

This is the key signal:

Chromium is not actually installed—only a wrapper script exists.

What We Tried (That Didn’t Work)

These approaches all failed:

Installing additional libraries (libx11, libnspr4, fonts, etc.)
Switching Puppeteer executable paths
Using snap install chromium
Installing from default Ubuntu repositories
Using Puppeteer’s bundled Chromium (in our case)

All of these fail for the same reason:

Snap is fundamentally incompatible with standard Docker containers.

The Fix

The solution is to avoid Ubuntu’s Snap-based Chromium entirely and install Chromium from a non-Snap APT source that provides a real binary.

Once we did this:

Chromium installed correctly
/usr/bin/chromium existed as a real executable
Puppeteer launched successfully

Example Dockerfile

# Install Chromium from a non-Snap source
RUN add-apt-repository ppa:xtradeb/apps -y \
    && apt update \
    && apt install -y \
       chromium \
       fonts-liberation \
       libx11-xcb1 \
       libxcomposite1 \
       libxdamage1 \
       libxrandr2 \
       libgbm1 \
       libasound2 \
       libnspr4 \
       libnss3 \
       ca-certificates \
    && rm -rf /var/lib/apt/lists/*

Puppeteer configuration:

ExecutablePath = "/usr/bin/chromium",
Args = new[]
{
    "--no-sandbox",
    "--disable-setuid-sandbox"
};

Alternative Approaches (Tradeoffs)

There are a few ways to solve this problem:

Use a non-Snap APT source (what we did)

*   Keeps Ubuntu base image

*   Requires managing external repo

Switch back to Debian-based images

*   Simpler dependency model

*   Diverge from official runtime images

Use Puppeteer’s bundled Chromium

*   Less setup

*   Larger image size, less control

We chose option 1 to keep compatibility while maintaining control over dependencies.

Key Insight

This issue is not specific to Puppeteer or ASP.NET.

Any headless browser setup (Puppeteer, Playwright, Selenium) running inside Ubuntu-based Docker containers can hit this failure mode.

The underlying problem is a mismatch between:

Snap’s system-level assumptions
Docker’s minimal runtime model

Lessons Learned

Base Images Are Part of Your Runtime Contract

Framework upgrades can silently change your OS layer. Treat base images as a first-class dependency.
“Installed” Does Not Mean “Runnable”

Package managers can install indirections instead of binaries. Always verify the actual executable:
```
file $(which chromium)
```
Snap and Containers Don’t Mix

Snap packages are not designed for containerized environments. Avoid them in production Docker setups.
Validate Critical Dependencies Early

If your system depends on external binaries (like browsers), validate them explicitly after upgrades—not after deployment failures.

Final Thoughts

This issue took longer to diagnose than expected because:

Installation appeared successful
Errors were misleading
The OS-level change wasn’t obvious

If you’re upgrading to ASP.NET 10 and rely on headless browsers, validate your Chromium setup early.

The failure mode is silent—and easy to miss.

Hello World — Building and Learning in Public

Pankaj Sharma — Fri, 01 May 2026 08:04:18 +0000

Hello World — Building and Learning in Public

I work on backend systems, mostly in the .NET ecosystem, trying to understand how systems behave under scale, failures, and real-world constraints.

This blog is where I document that process in public.

What I’ll Write About

Real production issues and debugging Challenges we face while building systems (and how we deal with them) System design trade-offs .NET internals and performance

Why This Exists

I’m learning in public.

Most posts will come from real problems, mistakes, and things that didn’t work the first time. Writing helps me think clearly and refine my understanding.

Final Thought

If you’re trying to move beyond just shipping code to actually understanding systems, this might be useful.