Sanitizer Stacks Look Identical Until You Normalize Them: A C++ Crash Triage Harness

The build had been green for days. That was the problem.

A C++ service ran AddressSanitizer and UndefinedBehaviorSanitizer in a nightly job. The logs were long. Many entries looked identical except for addresses and line numbers. The team spent more time grouping reports than fixing bugs. A model could help, but only if it saw a normalized signal.

This article walks through a small C++ harness that turns noisy sanitizer output into stable bug signatures. It then uses a free model endpoint to label unknown signatures and a free server option to keep the triage view visible. The result is a workflow, not a benchmark.

Why raw stack text is a bad key

Sanitizer output changes every build. Addresses shift with ASLR. Line numbers move when someone edits a file. Thread ids are noise. If the key is the whole text, the same null dereference becomes dozens of new reports.

The fix is to keep what matters. Function names matter. Sanitizer type matters. File basename matters after redaction. Addresses do not matter. Line numbers help only after a report has been confirmed.

A stable signature lets a cache remember a previous verdict. The model is not asked twice for the same bug class. That matters when the free endpoint is rate-limited or slow.

Step 1: Build with sanitizers

Use a debug build, not production optimization.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CXX_FLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
cmake --build build -j
ctest --test-dir build --output-on-failure 2>&1 | tee sanitizer.log

The -fno-omit-frame-pointer flag keeps stack frames readable. Without it, the normalizer has less to work with.

Step 2: Normalize before sending anything

The harness reads sanitizer blocks and emits one tab-separated line per report: a hash and a normalized stack. Compile it as a standalone tool.

#include <iostream>
#include <string>
#include <regex>
#include <vector>
#include <cstdint>
#include <sstream>

static uint64_t fnv1a(const std::string& s) {
    uint64_t h = 1469598103934665603ULL;
    for (unsigned char c : s) {
        h ^= c;
        h *= 1099511628211ULL;
    }
    return h;
}

static std::string normalize_frame(const std::string& line) {
    std::string out = line;
    out = std::regex_replace(out, std::regex("0x[0-9a-fA-F]+"), "<addr>");
    out = std::regex_replace(out, std::regex("(/[^ ]+[.]cpp):([0-9]+)"), "<file>:<line>");
    out = std::regex_replace(out, std::regex("[(][^)]*[)]"), "(...)");
    return out;
}

static void emit(const std::vector<std::string>& frames) {
    std::ostringstream joined;
    for (size_t i = 0; i < frames.size(); ++i) {
        if (i) joined << "|";
        joined << frames[i];
    }
    const std::string text = joined.str();
    std::cout << fnv1a(text) << '\t' << text << '\n';
}

int main() {
    std::string line;
    std::vector<std::string> frames;
    bool active = false;

    while (std::getline(std::cin, line)) {
        bool marker = line.find("ERROR: AddressSanitizer:") != std::string::npos ||
                      line.find("runtime error:") != std::string::npos;

        if (marker) {
            if (active && !frames.empty()) emit(frames);
            active = true;
            frames.clear();
            frames.push_back(normalize_frame(line));
            continue;
        }

        if (!active) continue;

        if (line.empty() || line.find("SUMMARY: AddressSanitizer:") != std::string::npos) {
            if (!frames.empty()) emit(frames);
            active = false;
            frames.clear();
            continue;
        }

        if (line.size() >= 5 && line.compare(0, 5, "    #") == 0) {
            frames.push_back(normalize_frame(line));
        }
    }

    if (active && !frames.empty()) emit(frames);
}

The program does not classify. It only makes duplicates visible. That separation is deliberate. A hash collision or a bad regex would be easier to audit than a model hallucination.

Compile and run it:

c++ -std=c++17 -O2 normalizer.cpp -o normalizer
./normalizer < sanitizer.log > reports.tsv

Step 3: Label only unknown signatures

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For the classification layer, the walkthrough uses MonkeyCode's free model access and free server option. The exact request shape is not part of this article; the shell below is intentionally generic and should be adapted to the actual endpoint.

#!/usr/bin/env bash
set -euo pipefail
DB="triage.sqlite"
sqlite3 "$DB" "CREATE TABLE IF NOT EXISTS reports (
  sig TEXT PRIMARY KEY,
  tag TEXT NOT NULL,
  reviewed INTEGER NOT NULL DEFAULT 0
);"

while IFS=$'\t' read -r sig text; do
  existing=$(sqlite3 "$DB" "SELECT tag FROM reports WHERE sig='$sig';")
  if [[ -n "$existing" ]]; then
    echo "cache hit $sig -> $existing"
    continue
  fi

  prompt="Classify this sanitizer report. Choose exactly one: null-deref, use-after-free, buffer-overflow, stack-overflow, signed-overflow, noise. Report: $text"
  tag=$(curl -sS -X POST "$MODEL_ENDPOINT" \
    -H "Content-Type: application/json" \
    -d "$(jq -nc --arg p "$prompt" '{prompt:$p}')" | jq -r '.tag')

  case "$tag" in
    null-deref|use-after-free|buffer-overflow|stack-overflow|signed-overflow|noise) ;;
    *) tag="noise" ;;
  esac

  sqlite3 "$DB" "INSERT OR IGNORE INTO reports(sig, tag) VALUES('$sig', '$tag');"
  echo "$sig -> $tag"
done < reports.tsv

The allowlist is not cosmetic. If the model returns an unexpected string, the script stores noise instead of a raw value. Never trust a free model to return a clean enum.

Step 4: Keep the triage view on a free server

A local SQLite file is useful. A short Python server makes it visible to other maintainers. Deploy this single file to the free server option, keep it behind authentication, and point it at the same database.

import html
import http.server
import socketserver
import sqlite3

DB = "triage.sqlite"

def rows():
    con = sqlite3.connect(DB)
    cur = con.execute(
        "select sig, tag, reviewed from reports order by reviewed, sig desc limit 200"
    )
    data = cur.fetchall()
    con.close()
    return data

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        body = "<h1>Crash triage</h1><table border=1>"
        body += "<tr><th>signature</th><th>tag</th><th>reviewed</th></tr>"
        for sig, tag, reviewed in rows():
            body += (
                "<tr><td>"
                + html.escape(str(sig))
                + "</td><td>"
                + html.escape(tag)
                + "</td><td>"
                + str(reviewed)
                + "</td></tr>"
            )
        body += "</table>"
        self.send_response(200)
        self.send_header("Content-Type", "text/html")
        self.end_headers()
        self.wfile.write(body.encode())

if __name__ == "__main__":
    with socketserver.TCPServer(("0.0.0.0", 8080), Handler) as httpd:
        httpd.serve_forever()

The page is read-only. Review happens in a merge request, not in the dashboard.

What this harness does and does not do

The normalizer groups reports by function sequence and sanitizer type. It removes addresses, threads, and exact line numbers. That is the core artifact. The model adds a suggested class. The server adds shared visibility. The cache prevents duplicate model calls.

A decision table for what to keep:

Signal	Keep or drop	Reason
function names	keep	bug class usually lives in the function sequence
sanitizer type	keep	different failure modes
file basename	keep	distinct components can share function names
addresses	drop	ASLR noise
thread ids	drop	not stable across runs
exact line numbers	drop first pass	edits create false duplicates
binary path	redact	reduces source leakage

Limitations

The hash is not a proof of duplicate bugs. Two signed overflows in the same function can collapse into one signature. A developer still has to open the source and confirm. The model tag is a suggestion, not a root cause.

The free endpoint may be rate-limited or unavailable. The cache helps only after a signature has been seen. A fresh flood of new signatures will still hit the endpoint. Redact proprietary names before sending stack text. Do not send full source, environment variables, or hostnames.

Who should not use this approach:

• safety-critical or regulated code that needs deterministic, auditable triage;
• teams that cannot review every model-generated tag;
• codebases where stack names are sensitive and redaction is not implemented;
• projects without sanitizer coverage in CI in the first place.

What to try first

Run the normalizer on an existing sanitizer log. Count how many unique signatures remain after addresses and line numbers are removed. If the count drops sharply, the cache and model step may be worth adding. If it does not drop, the team has many distinct bugs, not a deduplication problem. That result is useful too.

The model is the last step, not the first. The normalization is what makes the workflow repeatable.