S2 — Heap Corruption Crashes: How to Diagnose and Fix Them

In the Crash Pattern series, we classify crashes by their shape — the way they present themselves in backtraces, logs, and runtime behavior. This helps us reason about failures systematically instead of chasing symptoms.

Where S1 crashes are clean, local, and deterministic, S2 crashes are the opposite. They are delayed, misleading, and often nondeterministic. They frequently appear far away from the real defect, and they often disappear when we add logging, change optimization levels, or run under a debugger. These properties make S2 one of the most frustrating categories in real‑world C++ systems.

In this article, we examine what heap corruption crashes are, how they behave, how we diagnose them, and how we fix them. Our goal is to build a reliable mental model so we can recognize S2 quickly and avoid wasting time on misleading crash locations.

---

What Is a "Heap Corruption Crash"?

A heap corruption crash occurs when the heap allocator discovers that its internal state has been damaged. The corruption itself happened earlier, but the allocator only detects it later when it tries to allocate, free, or manage memory.

Identity 1 — The crash location is rarely the bug location
The allocator discovers the corruption long after the real defect occurred, often in unrelated code.

Identity 2 — Allocator‑Detected Failures
Heap corruption crashes are typically detected by the memory allocator, not by our application. The allocator (e.g., glibc) prints an error message to stderr and aborts the program when it encounters inconsistent heap state. This is why the crash location appears inside malloc, free, new, or delete, even though the real bug happened earlier.

---

What Heap Corruption Crashes Look Like

Heap corruption has a distinctive “shape”. The symptoms are delayed, misleading, and often nondeterministic.

1. Delayed Symptoms
The distance from heap corruption to crash is varied depends on the memory layout. So this deduces the deployed and variant symptoms:

• Crash happens far away from the corruption
• Crash appears random
• Crash moves between runs
• Crash disappears under debugger
• Crash disappears with logging
• Crash disappears with optimization changes

2. Allocator‑Level Signals
The memory allocator gives the error message before abort. So the messages show up in stderr, core dump, log file, etc. Typical glibc messages are:

• “double free or corruption”
• “invalid pointer”
• “corrupted size vs. prev_size”
• “pointer being freed was not allocated”
• “malloc(): memory corruption”

These are strong S2 indicators.

3. Backtrace Characteristics

• Top frame inside malloc/free/new/delete
• Or inside memcpy/memmove
• Or inside unrelated code
• Or completely nonsensical

4. Nondeterminism

• Crash location changes
• Crash timing changes
• Crash frequency changes

If the crash moves around, it is almost never S1. It is almost always S2 or S3.

---

Likely Patterns — Root Causes Behind Heap Corruption Crashes

Heap corruption crashes originate from mechanisms that corrupt user memory or allocator metadata. The typical patterns include:

1. Buffer Overflows / Underflows

• Writing past end of vector/array
• Off‑by‑one errors
• Overwriting allocator metadata

2. Use‑After‑Free

• Dangling pointers
• Returning references to freed memory
• Async callbacks firing after destruction
• Stale iterators

3. Double Free / Invalid Free

• Freeing twice
• Freeing stack memory
• Freeing memory from a different allocator

4. Mismatched Allocation / Deallocation

• new[] / delete
• new / delete[]
• malloc / delete
• new / free

5. Wild Writes

• Writing through corrupted pointers
• Writing through uninitialized pointers
• Writing through stale iterators

---

Diagnostic Techniques for Heap Corruption Crashes

Heap corruption crashes require us to catch the corruption at the moment it happens, not at the moment the allocator aborts. The crash location cannot be trusted, so tools and instrumentation play a central role in diagnosing S2.

1. Reproduce Under the Right Conditions

Heap corruption often disappears in debug builds or when the memory layout changes. Reproducing the issue may require:

• the same optimization level
• the same allocator
• the same timing
• the same memory layout

Reproducibility is often the hardest part of S2.

2. Address Sanitizer (ASan)

ASan is the most effective tool for diagnosing heap corruption. It detects:

• buffer overflows and underflows
• use‑after‑free
• double free
• wild writes

ASan reports the corruption at the point where it occurs, not where the allocator later crashes. ASan stack traces are trustworthy and usually point directly to the defect.

Keep in mind: ASan requires the program to be recompiled with -fsanitize=address.
Recompilation changes code generation and timing, so some crashes may disappear or change shape under ASan.

3. Valgrind / Memcheck

Valgrind is slower but valuable when ASan cannot be used. It:

• detects many classes of heap corruption
• works in production‑like environments
• does not require compiler instrumentation

4. Guard Allocators

Debug allocators such as Electric Fence(efence), jemalloc debug mode, tcmalloc debug mode, or glibc’s MALLOC_CHECK_ help detect:

• invalid frees
• double frees
• metadata corruption

They work by adding guard pages, canaries, or stricter validation around heap operations.

5. Heap Poisoning

Heap poisoning fills memory with known byte patterns so that invalid accesses fail early and deterministically.

For S2, poisoning must be applied in both directions:

• Poison on allocation — exposes uninitialized reads, overflows, and underflows.

• Poison on free — exposes use‑after‑free and stale pointers.

We cannot rely on knowing the corruption pattern in advance.
Using both poisoning modes ensures that the corruption becomes visible regardless of whether it happens before or after the free.

6. Binary Search for the Corruption

When the corruption window is large, we can narrow it using a binary‑search approach:

• instrument half of the suspected code for example, insert canaries or basic invariant checks in that half
• run the workload and observe whether corruption is detected
• repeat on the remaining half until the corruption point is isolated

This divide‑and‑conquer method is extremely effective for difficult S2 cases.

7. Inspect Allocation and Deallocation Sites

This technique is only effective after the corruption window has been narrowed by tools or binary search.
Once we know which object or subsystem is involved, we examine:

• where the object is allocated
• where it is freed
• who owns it (unique_ptr, move)
• who stores references to it (shared_ptr, raw pointer)

This often reveals lifetime mismatches, stale pointers, or unexpected ownership flows.
It is not a full code review — it is a focused inspection of a small, relevant region identified by earlier steps.

---

Remediation Steps for Heap Corruption Crashes

Fixing S2 is about fixing the corruption source, not the crash.

1. Fix the Corruption Source

• correct the overflow
• fix the use‑after‑free
• fix mismatched allocation
• fix double free

2. Add Invariants

• validate sizes
• validate indices
• validate pointer lifetimes

3. Strengthen Ownership

• use unique_ptr
• use shared_ptr carefully
• avoid raw new/delete

4. Add Defensive Allocator Settings

• enable debug malloc in production replicas
• add guard pages
• add canaries

---

Examples

These three examples illustrate the three major diagnostic paths: ASan, Guard Allocator and Manual Poisoning.

Example 1 — Using ASan to Diagnose a Heap Buffer Overflow

Buggy Code

#include <vector>

void process(std::size_t n)
{
    std::vector<int> v(n);
    for (std::size_t i = 0; i <= n; ++i) { // BUG: off-by-one
        v[i] = 42;
    }
}

int main()
{
    process(4);
}

ASan Output (Simplified)

ERROR: AddressSanitizer: heap-buffer-overflow
WRITE of size 4 at 0x602000000014
    #0 process(...) overflow.cpp:7
    #1 main overflow.cpp:14

0x602000000010 is located 0 bytes to the right of 16-byte region allocated here:
    #0 operator new[](…)
    #1 std::vector<int>::vector(...)

Diagnosis

ASan reports a heap-buffer-overflow and shows:

• the exact write location (file name, line number)
• the allocation site
• the fact that the write is “0 bytes to the right” of the buffer

This points directly to the off‑by‑one loop condition.

Fix

for (std::size_t i = 0; i < n; ++i) { // FIX
    v[i] = 42;
}

---

Example 2 — Using a Guard Allocator (jemalloc debug mode) to Detect Use‑After‑Free

Buggy Code

#include <cstdio>
#include <cstdlib>

struct Node {
    int value;
};

int main()
{
    Node* p = static_cast<Node*>(std::malloc(sizeof(Node)));
    p->value = 123;

    std::free(p);          // freed here

    std::printf("%d\n", p->value); // BUG: use-after-free
}

jemalloc Debug Output (Simplified)

jemalloc: error: pointer to freed memory
Abort (core dumped)

Diagnosis

The guard allocator aborts immediately when the freed pointer is accessed.

This confirms a use‑after‑free.

The backtrace shows the invalid access at p->value.

Fix

std::printf("%d\n", p->value);
std::free(p);
p = nullptr; // optional: clear stale pointer

---

Example 3 — Using Manual Heap Poisoning to Expose a Stale Pointer

Buggy Code

#include <cstdlib>
#include <iostream>

struct Entry {
    int value;
};

Entry* g_cache = nullptr;

Entry* allocate_entry()
{
    Entry* e = static_cast<Entry*>(std::malloc(sizeof(Entry)));
    e->value = 42;
    return e;
}

void free_entry(Entry* e)
{
    std::free(e); // BUG: caller still holds g_cache
}

int main()
{
    g_cache = allocate_entry();
    free_entry(g_cache);

    std::cout << g_cache->value << "\n"; // stale pointer
}

This may run “fine” or corrupt unrelated memory.

Add Manual Poisoning

static constexpr unsigned char POISON = 0xDD;

void free_entry(Entry* e)
{
    std::memset(e, POISON, sizeof(Entry)); // poison on free
    std::free(e);
}

Observed Behavior

Instead of silent corruption, the program prints a poisoned value:

-572662307

(0xDDDDDDDD interpreted as an integer)

This confirms a stale pointer.

Fix

free_entry(g_cache);
g_cache = nullptr; // FIX

Or redesign the cache to avoid raw pointers.

---

When It’s Not Heap Corruption Crash

A crash may look like heap corruption at first glance but still belong to a different category.

We treat a crash as S2 only when the allocator detects corrupted heap metadata and the stack is still trustworthy.

Red flags that indicate misclassification:

• The crash does not occur inside malloc/free/new/delete (allocator is not involved → not S2)
• The stack trace is broken or unwinds into nonsense (stack itself is corrupted → S3)
• The crash happens immediately after a function returns (return address overwritten → S3)
• The crash is fully deterministic (S2 is usually timing‑dependent → likely S1/S3/S4)
• The failure disappears when serialized (thread‑interleaving dependent → S4)
• The crash only appears on specific machines or builds (environment‑dependent UB → S5)

If any of these appear, the crash is not S2.
It likely belongs to another category, and heap‑corruption techniques will not help.

---

Summary

Heap corruption failures behave differently from ordinary crashes.
They are nondeterministic, often delayed, and the crash location rarely matches the bug location. The allocator is only the final victim; the corruption usually happens much earlier.

Different techniques provide different levels of visibility.

Technique Comparison

Technique	Strengths	Requires Recompile	Runtime Overhead	When to Use
ASan	Precise detection of overflows, UAF, OOB; excellent stack traces	Yes	High	When you can recompile and timing changes are acceptable
Valgrind (Memcheck)	Detects overflows, UAF, invalid reads/writes without recompiling	No	Very High (20–50×)	When reproducibility is stable and performance is not a concern
Guard Allocators (jemalloc/tcmalloc debug)	Detect UAF, double free, invalid free under realistic timing	No (allocator switch only)	Moderate	When ASan changes behavior or cannot be used
Manual Poisoning	Exposes stale pointers and lifetime bugs in isolated subsystems	No (local instrumentation only)	Low	When you cannot change the global allocator but can instrument locally
Binary Search Instrumentation	Narrows large corruption windows; works when tools fail	No	Low	When corruption is nondeterministic or not reproducible under tools

Choosing the right one depends on constraints such as reproducibility, timing sensitivity, and whether you can recompile or change the allocator.
The goal is always the same:
find and fix the corruption source, not the crash site.

---

Key Takeaways

1. Heap corruption is nondeterministic
Symptoms vary run‑to‑run. The allocator crash is not the root cause.

2. Crash location ≠ bug location
The allocator reports the moment of detection, not the moment of corruption.

3. Use the right visibility tool
Each technique exposes a different class of corruption:

• ASan → precise overflow/UAF detection
• Valgrind → deep checking without recompiling
• Guard allocators → realistic‑timing lifetime errors
• Poisoning → stale pointers in local subsystems
• Binary search instrumentation → narrowing large or nondeterministic windows

4. Ownership and lifetime discipline matter more than the allocator
Most S2 failures are ownership problems, not allocator problems.

5. Fix the corruption source
Do not patch the crash. Remove the bug that corrupted memory.

---