DEV Community: C W

The Ghost in the Machine: Hunting a Memory Leak I Couldn't See the Source Of.

C W — Wed, 15 Jul 2026 08:09:42 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Most bug stories start with a stack trace in your own code. This one starts with a stack trace in code I'd never seen, couldn't read, and wasn't allowed to touch — a closed-source, decades-old ERP that a critical overnight process depended on. No source, no debugger, and no PRs to link at the end of this post, because there was never any of my own code to change.

What I did have was a process that fell over in the small hours, a vendor whose only advice was to "call support," and a stubborn refusal to accept "it just does that sometimes" as an answer.

Here's how I smashed a bug I could only ever observe from the outside.

The symptom

Every night, an integration job that shuttled financial settlement data between two systems would throw this at me:

System.OutOfMemoryException
   at System.Text.Encoding.GetChars(Byte[] bytes, Int32 index, Int32 count)
   at System.Text.Encoding.Convert(...)
   at <VendorLib>.SQLSupport.Extensions.ToByteArray(String s)
   at <VendorLib>.SQLSupport.Helper.Serializer.ToDataBuffer(IEntity entity)
   at <VendorLib>.SQLSupport.ServiceProxy.ActivityServiceProxy.GetNextRecord(...)
   ...

OutOfMemoryException. On a server with tens of gigabytes of RAM. The recommended fix from the vendor? Just a phone number.

The errors clustered in a roughly 20-minute window in the early hours, and I'd learn later that interactive users were hitting the same wall during the day without anyone connecting the dots. The problem had apparently been shrugged at for a long time.

False trail #1: "It's corrupt data"

My first theory was a poisoned record. The stack trace dies while serialising a string into a byte buffer — ToByteArray reading some field's length and trying to allocate. A garbage length value would make it try to allocate gigabytes in one shot and throw instantly. That fit the "fails on a specific record" idea perfectly.

Except it didn't hold up. The errors pointed at multiple different datasets, on different internal methods, not one record in one place. When even a security/session-setup call throws OutOfMemory, you're not looking at one bad row — you're looking at a process that's already on its knees before it even gets to your data.

False trail #2: "The server's out of memory"

The obvious next thought: But here's the thing about a .NET OutOfMemoryException: it very often has nothing to do with physical RAM. It means the process couldn't get the memory it asked for — and a 32-bit process can only ever address about 2GB, no matter how much RAM the machine has. That ceiling is baked into the architecture, not the hardware.

So I opened Task Manager, found the long-lived process behind the failing component, and watched it.

140 MB.

The process throwing "out of memory" was sitting at 140 megabytes. The server had oceans of free RAM. This was never a RAM problem. It was an address space problem — or it would be, if something were filling that 2GB up. But at 140MB? Something didn't add up. Unless I was looking at the wrong moment in the cycle.

That's when I stopped guessing and started measuring.

The stakeout: teaching myself what the process did all day

I couldn't attach a profiler to vendor software in production. So I did the crude, reliable thing: a scheduled script that logged the process's committed memory, working set, and handle count once an hour, to a CSV. Keyed off the service account rather than the process ID, so it would re-find the right process automatically each day even across daily server restarts.

Then I waited a day and looked at the curve.

Time    CommitMB   Handles
10:00      293       2889
11:00      387       3779
12:00      485       4650
13:00      578       5596
...        ...        ...
21:00     1361      13429
22:00     1451      14361   <- flatlines here (users log off)
02:00     1456      14492
03:00      178       1781   <- reboot resets it

There it was. A dead-straight climb of ~96 MB every single hour, from a low baseline right up toward the ceiling — and then a hard reset overnight. A textbook memory leak, drawn in a perfectly clean line.

Two details made it click:

Handles rose in lockstep with memory — about 970 per hour, tracking the megabytes almost proportionally. Memory and handles leaking together is the fingerprint of unreleased references: every leaked object clings to both a chunk of memory and an OS handle. In an old reference-counted component, that's a missed release on a hot path.
The leak flatlined when people logged off at night. It was usage-driven — every operation leaked a little, so when the work stopped, the growth stopped. Not a background timer; a per-operation leak.

And the "why 02h00?" finally made sense. The leak built all day and by the wee hours the process was near its ceiling and its free space was fragmented. That's exactly when the overnight job attempted its single largest allocation — a big data pull. An already-exhausted, fragmented 2GB address space couldn't hand over one contiguous block that size, and bang — OutOfMemoryException. The daily reboot then wiped the slate, and the cycle began again, which is precisely why nobody had ever caught it in daylight.

The 140MB I'd seen earlier? I'd happened to look just after a reboot. The stakeout caught the whole story the snapshot couldn't.

Confirming the culprit

The process was a COM+ surrogate host. Its executable path gave away the last piece:

C:\WINDOWS\SysWOW64\dllhost.exe /Processid:{GUID}

SysWOW64 = 32-bit on 64-bit Windows (yes, the naming is backwards). Matching that GUID against the registered COM+ applications pinned it to one specific component — the vendor's data-access layer that every operation, interactive and batch, funnelled through. One leaking process, every code path. That's why the overnight job and the daytime users failed for the same reason.

Diagnosis complete:

A 32-bit COM+ component leaks memory and handles at ~96 MB/hr under load (unreleased references). Over a day it approaches the 2GB address-space ceiling; fragmentation plus the overnight job's large allocation tips it into OutOfMemory. The nightly reboot masks it.

So not RAM, or corrupt record. Just a reference leak meeting a 32-bit wall.

Smashing it (without a compiler)

I couldn't fix the leak since it's not my code. So the goal shifted from fix the bug to make the system resilient despite an unfixable dependency.

The component ran under COM+, and COM+ has a feature made for exactly this: application recycling. You can tell it to tear down and respawn the process when it crosses a memory threshold — automatically, mid-life, without a server reboot.

I set the memory limit to 1 GB (1048576 KB — a clean 1024×1024, comfortably below the 2GB wall with a full gigabyte of headroom so any large allocation still finds contiguous space):

COM+ application → Properties → Pooling and Recycling → Memory limit (KB) = 1048576

There was one nervous moment. The component's own logs kept saying hasrecycled=[False] while messages were in process — it refused to recycle while busy, and it was always busy. I braced for the fix to do nothing. But that flag was a per-instant status, not a permanent refusal: COM+ waits for a gap between operations and takes it. Under real load there were enough sub-second gaps, and it started recycling right at the 1GB mark.

Before and after

The same homemade logger that caught the bug proved the fix. Same CSV, different shape.

Before: memory climbs to ~1.45GB, address space fragments, overnight job fails almost nightly, users hit errors as the day wears on.

After: the process now cycles up and down — climbs to 1GB, recycles to baseline, climbs again — never approaching the ceiling. The tell-tale sign it was genuinely recycling and not just rebooting: the process ID changed mid-day, without a server restart, with memory dropping to baseline at that exact moment.

The overnight and daytime errors stopped. Days of clean logs, including busy days that used to be the worst offenders. Harmony restored — not by changing a line of the offending code, but by boxing it in so its worst habit could never reach the failure point.

The twist: the bug has a sibling

Here's the part that keeps it from being a tidy ending. A second environment runs its own independent copy of the same software — same component, same leak, its own separate install. Fixing one did nothing for the other. The same recycling mitigation now has to be replicated there. A leak in shared-but-copied software isn't one bug; it's one bug per copy.

What I'm proud of, and what I learned

Proud of: solving it with nothing but observation. No source, no profiler, no vendor cooperation — just a stack trace read carefully, a snapshot that lied, and a homemade logger that told the truth. The whole diagnosis came from treating a black box as knowable if you measure it patiently enough.

Learned:

OutOfMemoryException is a liar's name. It rarely means "the machine is out of RAM." Nine times out of ten it means a single process hit its own ceiling — and for 32-bit processes that ceiling is ~2GB no matter what the server has.
A single snapshot can send you the wrong way. 140MB "proved" it wasn't memory. Only the trend over time told the real story. Measure the curve, not the point.
Memory and handles rising together is a gift of a clue — it points straight at unreleased references rather than a runaway cache.
You don't always get to fix the bug but sometimes you fix the blast radius. Recycling didn't cure the leak; it made the leak unable to hurt anyone. For a dependency you can't change, resilience around it is a legitimate win.
A reboot that "fixes" something every night is a bug wearing a disguise. If a scheduled restart is load-bearing, there's an unsmashed bug hiding behind it.

The ghost is still in the machine — I never exorcised the leak itself. But it's in a box now, recycled back to harmless before it can ever reach the wall. Sometimes smashing a bug means making sure it can never land the hit.

Dōjō 道場 — karate made me do it.

C W — Sat, 11 Jul 2026 07:46:00 +0000

This is a submission for Weekend Challenge: Passion Edition

What I Built

Dōjō (道場) is a Shito-Ryu based karate kata training companion, built around the Passion Edition prompt taken literally, to say that passion is the fire that drives us.

The home screen is a living flame that grows with your training, fed by both your streak and the effort you put in each day. If you show up consistently, it builds and feeds the flame; if you put in a hard session, it flares up that same day. It's a real canvas particle fire and not a static graphic.

The core is a count-along kata drill with a live sensei voice. Pick a kata, set the tempo, and a sensei counts you through it in Japanese — ichi, ni, san… — on the beat, with a kiai shout on the traditional counts and an optional metronome. Pause/resume mid-drill, and every kata's counts are editable, since those conventions vary by branch. Finishing a drill logs the session and feeds the flame.

Rounding it out: a kata hall grouped by their respective lineage (Pinan, Shuri-te, Naha-te), with a belt-and-history view, and a one-tap Kiai button to test the chosen voice.

I have been practicing karate for 30 years and my goal was to build something that feels like stepping into a dōjō, not opening a fitness dashboard — and to make the voice feel like a real sensei counting beside you.

Demo

Code

chantleyw / dojo

Dōjō · 道場

A Shito-Ryu (糸東流) training companion with a live sensei voice.

Dōjō is a karate practice app built around a single emotional idea: passion is the fire that drives us. The home screen shows a flame that grows with your training streak — a cold ember at zero, a roaring fire past 30 days. Pick a kata and a sensei voice counts you through it in Japanese, on the beat — ichi, ni, san… — with a kiai shout on the traditional counts and an optional metronome tick.

Built for the DEV.to Weekend Challenge (Passion Edition).

✨ Highlights

A flame that grows with your streak. Layered SVG plume with a Framer Motion flicker, driven by fire = min(1, streak / 30). Reduced-motion holds it still.
A live count-along drill. Choose a kata; the sensei counts in Japanese on the beat, kiai-ing on the traditional counts. Tempo…

View on GitHub

How I Built It

The stack is Vite + React + TypeScript + Tailwind, with Framer Motion, Zustand for state, the Web Audio API for the metronome, and Netlify Functions for the backend. State persists per-device in localStorage — so no accounts are required.

The sensei voice (ElevenLabs)
This is the core of the app, and getting it to feel live drove most of the interesting decisions.

A custom voice, built in Voice Design. The sensei is a purpose-built deep, gravelly grand-master voice created in ElevenLabs Voice Design — the kind of voice that could bark a battle command. The app pulls every voice on the account into a picker, but defaults to the sensei.

Authentic Japanese counting. Numerals are generated programmatically as hiragana (いち, に, さん…, composing じゅういち for 11, and so on) and spoken with Turbo v2.5 using language_code: "ja". That language flag was the unlock because without it, an English-leaning accented voice pronounces さん like the English word "san"; with it, you get a proper sahn pronunciation.

Making it land on the beat. A count drill can't afford a network round-trip per number. So on "Begin," the app pre-generates one clip per unique word, caches them as object URLs keyed by text, then plays the cached audio on each metronome beat. The count and the kiai are also given separate expressiveness profiles — the counts stay crisp and consistent, while the kiai runs at low stability / high style for a raw, shouted delivery.

Keeping the key safe. The link is public, so the ElevenLabs key can never touches the browser. A tiny Netlify Function proxies /api/tts and /api/voices, holding the key in an env var and forwarding requests server-side. The client only ever talks to my own domain.

Never hard-failing. If the proxy returns a non-200 — a quota cap, a rate limit — the app falls back to the browser's built-in speech synthesis (preferring a Japanese voice) and keeps counting. I actually hit this live during testing when a capped key ran dry, and the drill kept running on the fallback voice without a hiccup. Generation is also concurrency-limited with retries so the free tier's request limits don't drop individual clips.

The flame is a canvas particle system — rising flame particles, floating embers, and an additive glow, all scaling with a single fire value from 0 to 1. Rather than tie that purely to the streak, I compute it from both consistency and effort: fire = 0.7 × (streak / 30) + 0.4 × (today's minutes / 45), capped at 1. A long streak carries a strong baseline, a hard session flares it up the same day, and a single huge day can't max it out alone. Reduced-motion users get a still flame instead.

Prize Categories

Best Use of ElevenLabs. The sensei voice is the core of the app: a custom generated Voice Design grand-master voice, spoken with Turbo v2.5 and language_code: "ja" for authentic Japanese counting, pre-generated and cached per word so it lands on the beat, with separate expressiveness profiles for the counts and the kiai which are all served through a serverless proxy that keeps the API key off of the client, with a seamless fallback to the browser voice so a public link never hard-fails.