"I can handle the connections," it said. "I am one. I am shared. I am efficient," it said.
And for a while — it could.
A Home in settings.py
The Singleton lived in settings.py.
Not a glamorous address. Nobody visits settings.py to admire the architecture. You go there to set a database password, flip a flag, move on. It's the utility drawer of a Django application — full of things that need to exist somewhere, and settings.py is where they ended up.
But the Singleton didn't mind. It had a job, and the job was simple: connect to Consul once, at startup, before the first request ever arrived. Then wait. When someone needed a service address, a config value, a secret — it was already there. Already connected. Already warm.
# settings.py
CONSUL_CLIENT = consul.connect()
One line. That's all it took to exist.
And then the app would boot, and the workers would load settings.py, and they'd find it sitting there like a receptionist who'd come in early and made the coffee. I'm already here. I've already done the hard part. Just ask me.
# consul.py
class ConsulClient():
def __init__(self):
self.client = settings.CONSUL_CLIENT
No drama. No overhead. No redundant connections eating resources on a system that didn't need them. One request came in, the server handled it, sent a response, moved on. One worker. One world. One Singleton, perfectly matched to the life it had been given.
It was good at its job because its job made sense.
Then Came Streaming
But one night — no warning, no context, no ceremony — a requirement came in.
Streaming.
Server-Sent Events. The browser needed live data — pushed from the server continuously, without asking. A stock ticker. A status feed. Something alive.
SSE is async by nature. Our app was not. So we brought in Django Channels and swapped WSGI for ASGI. The app could now handle multiple connections simultaneously. Multiple async workers, all spinning, all alive at the same time, all serving requests in parallel.
The Singleton's world had changed.
Nobody told the Singleton.
The Ghost in vCluster
Locally, everything worked fine.
The Singleton was still humming. Still proud. Still delivering. We ran the test suite. Green. We hit the endpoints manually. Responsive. We spun up the dev server and watched the SSE stream come through clean and steady. We looked at each other and said the thing you should never say out loud in a codebase.
Looks good to me.
Then, before anyone could reach for the staging deploy button, one voice from the back of the call.
"Hey — didn't we try ASGI last time and run into something?"
The room went quiet in that particular way it does when nobody wants to be the first to say yes, actually.
Someone had. Six months ago, a different feature, a different branch. ASGI had been the plan then too — and it had hit something nobody could fully explain at the time. The branch got shelved. The decision got quietly filed under not yet and life moved on. Nobody wrote it down. Nobody had to. It just lived in the room, in the people who'd been there.
So we didn't push to staging. Not yet.
We pushed to vCluster instead — the environment that exists specifically so nothing surprises you later, the place you send code when you're confident but not that confident. When you need to be sure before you let it touch anything that matters.
It broke.
Badline errors. Consul interaction failing under any real load. Requests timing out. Workers dying quietly, without drama, without useful stack traces. Just — gone. And then the challenging part: we couldn't reproduce it locally no matter what we tried.
We added logging. We reran the tests. We hit the endpoints harder. Nothing. Clean every time. The local environment held firm, serene, unbothered, completely useless as evidence.
The bug lived on vCluster and nowhere else.
Someone suggested it was a networking issue. We checked the networking. Fine. Someone suggested the Consul instance itself was flaky. We tested the Consul instance directly. Responsive. Someone else said maybe it was a memory leak, a timeout misconfiguration, a version mismatch between environments. We chased each theory to its end and each one closed without resolution.
We were debugging a ghost. It knew we were looking. It kept not being there.
Nobody trusted their fix enough to push it forward. Nobody wanted to be the person who turned a vCluster problem into a staging problem, and a staging problem into a production incident. That question was still in the room. It hadn't left. So the branch sat. And the bug sat with it. And every morning someone would open the logs again, scroll through the same errors, form a new theory, and by afternoon that theory would be dead too.
We stared at the code until the code told us something.
The Race Condition Has No Manners
Load was the key.
It wasn't in the logs we'd been reading — it was in the number we'd stopped noticing. vCluster ran with real concurrency. Local dev did not. We'd been so focused on what was failing that we hadn't sat long enough with when. Low traffic hid it completely. One or two requests and the Singleton held. But the moment traffic climbed — multiple async workers spinning up simultaneously, all of them loading settings.py, all of them reaching for the same Consul connection, all of them absolutely certain they were the only one in the room — it collapsed.
They were not the only one in the room.
The Singleton had spent its entire life in an orderly queue. One request. Then the next. Then the next. It had been built for a world with manners — a world where nobody cut in line, nobody reached across the table, nobody grabbed the door handle while you were still holding it. It had never been taught what to do when twelve versions of itself showed up at the same moment, all with the same idea, all with the same credentials, all reaching for the same single connection it had been holding patiently since boot.
A race condition has no manners.
Multiple workers touch the same shared state at the same time. No coordination. No turn-taking. No signal. They all arrive at the same door simultaneously and none of them knows the others are there. In our case, the shared state was the Consul client — that single connection initialized back in settings.py, back when the world was synchronous, back when shared meant efficient instead of contested.
Twelve workers. One connection. No locks between them.
When they all reached for it at once, the Singleton had no protocol for that. It had never needed one. It had been given a world where it would never need one, and it had thrived there, and then the world had been quietly replaced while it wasn't looking.
# settings.py — The problem
CONSUL_CLIENT = consul.connect()
# consul.py — The problem, continued
class ConsulClient():
def __init__(self):
self.client = settings.CONSUL_CLIENT
One door. Twelve hands reaching for the handle at the same moment.
The Singleton didn't fail because it was careless. It failed because it was trusting — trusting that the world it had been built for was still the world it lived in. And the hardest part of sitting with that, once you finally see it, is the recognition that you made the same assumption. You trusted the same world. You pushed the same code into a changed environment and called it a deployment.
The Singleton wasn't the only one who hadn't been told.
One Line Removed, One Line Rewritten
The fix wasn't dramatic. That's not how these things go.
# consul.py — The fix
class ConsulClient():
def __init__(self):
self.client = consul.client()
Remove the Singleton from settings.py. Initialize the connection on the class itself. Each worker gets its own client when it needs one. No shared state. No contested door. No assumption that order is guaranteed.
One line removed, one line rewritten. The kind of diff that looks too small for how long it took to find. But what it represents is larger than the change — it's the moment you stop trusting that the world will stay the world you designed for, and start building things that can survive the world changing without warning.
The Track Matters as Much as the Will
The Singleton was not wrong.
It was built for a synchronous world, and it thrived there. Everything it promised, it delivered — as long as requests arrived one at a time, as long as the workers waited their turn, as long as the app's soul was orderly.
Async servers are not orderly. They are parallel by design. Twelve workers alive simultaneously is not a failure state — it's the goal. And shared state that was invisible as a problem with one worker becomes a liability with twelve. The thing that made the Singleton efficient in one world made it brittle in the next.
I think I can was enough when there was one hill and one engine. When there are twelve engines and one door, I think I can isn't a strategy. It's an assumption.
And assumptions live in settings.py, quiet and warm, right up until the load increases and the workers spin up and the world you trusted turns out to have changed shape while you weren't watching.
The track matters as much as the will.
The Singleton knew the track.
It just didn't know the track could change.
Top comments (0)