The Ghost in the Machine: Hunting a Memory Leak I Couldn't See the Source Of.

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Most bug stories start with a stack trace in your own code. This one starts with a stack trace in code I'd never seen, couldn't read, and wasn't allowed to touch — a closed-source, decades-old ERP that a critical overnight process depended on. No source, no debugger, and no PRs to link at the end of this post, because there was never any of my own code to change.

What I did have was a process that fell over in the small hours, a vendor whose only advice was to "call support," and a stubborn refusal to accept "it just does that sometimes" as an answer.

Here's how I smashed a bug I could only ever observe from the outside.

The symptom

Every night, an integration job that shuttled financial settlement data between two systems would throw this at me:

System.OutOfMemoryException
   at System.Text.Encoding.GetChars(Byte[] bytes, Int32 index, Int32 count)
   at System.Text.Encoding.Convert(...)
   at <VendorLib>.SQLSupport.Extensions.ToByteArray(String s)
   at <VendorLib>.SQLSupport.Helper.Serializer.ToDataBuffer(IEntity entity)
   at <VendorLib>.SQLSupport.ServiceProxy.ActivityServiceProxy.GetNextRecord(...)
   ...

OutOfMemoryException. On a server with tens of gigabytes of RAM. The recommended fix from the vendor? Just a phone number.

The errors clustered in a roughly 20-minute window in the early hours, and I'd learn later that interactive users were hitting the same wall during the day without anyone connecting the dots. The problem had apparently been shrugged at for a long time.

False trail #1: "It's corrupt data"

My first theory was a poisoned record. The stack trace dies while serialising a string into a byte buffer — ToByteArray reading some field's length and trying to allocate. A garbage length value would make it try to allocate gigabytes in one shot and throw instantly. That fit the "fails on a specific record" idea perfectly.

Except it didn't hold up. The errors pointed at multiple different datasets, on different internal methods, not one record in one place. When even a security/session-setup call throws OutOfMemory, you're not looking at one bad row — you're looking at a process that's already on its knees before it even gets to your data.

False trail #2: "The server's out of memory"

The obvious next thought: But here's the thing about a .NET OutOfMemoryException: it very often has nothing to do with physical RAM. It means the process couldn't get the memory it asked for — and a 32-bit process can only ever address about 2GB, no matter how much RAM the machine has. That ceiling is baked into the architecture, not the hardware.

So I opened Task Manager, found the long-lived process behind the failing component, and watched it.

140 MB.

The process throwing "out of memory" was sitting at 140 megabytes. The server had oceans of free RAM. This was never a RAM problem. It was an address space problem — or it would be, if something were filling that 2GB up. But at 140MB? Something didn't add up. Unless I was looking at the wrong moment in the cycle.

That's when I stopped guessing and started measuring.

The stakeout: teaching myself what the process did all day

I couldn't attach a profiler to vendor software in production. So I did the crude, reliable thing: a scheduled script that logged the process's committed memory, working set, and handle count once an hour, to a CSV. Keyed off the service account rather than the process ID, so it would re-find the right process automatically each day even across daily server restarts.

Then I waited a day and looked at the curve.

Time    CommitMB   Handles
10:00      293       2889
11:00      387       3779
12:00      485       4650
13:00      578       5596
...        ...        ...
21:00     1361      13429
22:00     1451      14361   <- flatlines here (users log off)
02:00     1456      14492
03:00      178       1781   <- reboot resets it

There it was. A dead-straight climb of ~96 MB every single hour, from a low baseline right up toward the ceiling — and then a hard reset overnight. A textbook memory leak, drawn in a perfectly clean line.

Two details made it click:

1. Handles rose in lockstep with memory — about 970 per hour, tracking the megabytes almost proportionally. Memory and handles leaking together is the fingerprint of unreleased references: every leaked object clings to both a chunk of memory and an OS handle. In an old reference-counted component, that's a missed release on a hot path.
2. The leak flatlined when people logged off at night. It was usage-driven — every operation leaked a little, so when the work stopped, the growth stopped. Not a background timer; a per-operation leak.

And the "why 02h00?" finally made sense. The leak built all day and by the wee hours the process was near its ceiling and its free space was fragmented. That's exactly when the overnight job attempted its single largest allocation — a big data pull. An already-exhausted, fragmented 2GB address space couldn't hand over one contiguous block that size, and bang — OutOfMemoryException. The daily reboot then wiped the slate, and the cycle began again, which is precisely why nobody had ever caught it in daylight.

The 140MB I'd seen earlier? I'd happened to look just after a reboot. The stakeout caught the whole story the snapshot couldn't.

Confirming the culprit

The process was a COM+ surrogate host. Its executable path gave away the last piece:

C:\WINDOWS\SysWOW64\dllhost.exe /Processid:{GUID}

SysWOW64 = 32-bit on 64-bit Windows (yes, the naming is backwards). Matching that GUID against the registered COM+ applications pinned it to one specific component — the vendor's data-access layer that every operation, interactive and batch, funnelled through. One leaking process, every code path. That's why the overnight job and the daytime users failed for the same reason.

Diagnosis complete:

A 32-bit COM+ component leaks memory and handles at ~96 MB/hr under load (unreleased references). Over a day it approaches the 2GB address-space ceiling; fragmentation plus the overnight job's large allocation tips it into OutOfMemory. The nightly reboot masks it.

So not RAM, or corrupt record. Just a reference leak meeting a 32-bit wall.

Smashing it (without a compiler)

I couldn't fix the leak since it's not my code. So the goal shifted from fix the bug to make the system resilient despite an unfixable dependency.

The component ran under COM+, and COM+ has a feature made for exactly this: application recycling. You can tell it to tear down and respawn the process when it crosses a memory threshold — automatically, mid-life, without a server reboot.

I set the memory limit to 1 GB (1048576 KB — a clean 1024×1024, comfortably below the 2GB wall with a full gigabyte of headroom so any large allocation still finds contiguous space):

COM+ application → Properties → Pooling and Recycling → Memory limit (KB) = 1048576

There was one nervous moment. The component's own logs kept saying hasrecycled=[False] while messages were in process — it refused to recycle while busy, and it was always busy. I braced for the fix to do nothing. But that flag was a per-instant status, not a permanent refusal: COM+ waits for a gap between operations and takes it. Under real load there were enough sub-second gaps, and it started recycling right at the 1GB mark.

Before and after

The same homemade logger that caught the bug proved the fix. Same CSV, different shape.

Before: memory climbs to ~1.45GB, address space fragments, overnight job fails almost nightly, users hit errors as the day wears on.

After: the process now cycles up and down — climbs to 1GB, recycles to baseline, climbs again — never approaching the ceiling. The tell-tale sign it was genuinely recycling and not just rebooting: the process ID changed mid-day, without a server restart, with memory dropping to baseline at that exact moment.

The overnight and daytime errors stopped. Days of clean logs, including busy days that used to be the worst offenders. Harmony restored — not by changing a line of the offending code, but by boxing it in so its worst habit could never reach the failure point.

The twist: the bug has a sibling

Here's the part that keeps it from being a tidy ending. A second environment runs its own independent copy of the same software — same component, same leak, its own separate install. Fixing one did nothing for the other. The same recycling mitigation now has to be replicated there. A leak in shared-but-copied software isn't one bug; it's one bug per copy.

What I'm proud of, and what I learned

Proud of: solving it with nothing but observation. No source, no profiler, no vendor cooperation — just a stack trace read carefully, a snapshot that lied, and a homemade logger that told the truth. The whole diagnosis came from treating a black box as knowable if you measure it patiently enough.

Learned:

• OutOfMemoryException is a liar's name. It rarely means "the machine is out of RAM." Nine times out of ten it means a single process hit its own ceiling — and for 32-bit processes that ceiling is ~2GB no matter what the server has.
• A single snapshot can send you the wrong way. 140MB "proved" it wasn't memory. Only the trend over time told the real story. Measure the curve, not the point.
• Memory and handles rising together is a gift of a clue — it points straight at unreleased references rather than a runaway cache.
• You don't always get to fix the bug but sometimes you fix the blast radius. Recycling didn't cure the leak; it made the leak unable to hurt anyone. For a dependency you can't change, resilience around it is a legitimate win.
• A reboot that "fixes" something every night is a bug wearing a disguise. If a scheduled restart is load-bearing, there's an unsmashed bug hiding behind it.

The ghost is still in the machine — I never exorcised the leak itself. But it's in a box now, recycled back to harmless before it can ever reach the wall. Sometimes smashing a bug means making sure it can never land the hit.