The Crash Router Trusted a 0.97 Confidence Score. It Needed a Vote.

A C++ service processed thousands of crash reports a week. The reports landed as JSONL files. Each record had a backtrace, a sanitizer note, a platform tag, and a last_frame string. A maintainer wired a free model endpoint into the triage stage. The model read the top eight frames and returned one bucket plus a confidence score. One Tuesday morning it sent an ASan heap-use-after-free report to the stack_overflow bucket. The score was 0.97. The bucket was wrong. The report stayed out of the heap-corruption queue for two days. The fix was delayed because the routing system obeyed a single high-confidence number.

The endpoint and the free server option came from MonkeyCode's free tier. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free endpoint was used only as a ranking layer. It was never the source of truth.

The failure was not a prompt problem. It was a decision design problem. A single model, even a useful one, should not be allowed to route alone when a local deterministic rule disagrees. The team replaced the one-answer design with a vote. The local rule engine kept the final veto. The model supplied ranked candidates. The router only auto-routed when the two sources agreed.

What to build

The reference harness is not production code. It is a dependency-light C++17 program that reads two flattened files. crash_reports.tsv holds one report per line. model_rankings.tsv holds the free model's top candidates for the same report IDs.

The crash report fields are:

• id
• sanitizer
• signal
• frame0

The model ranking fields are:

• id
• bucket1|bucket2|bucket3
• score1|score2|score3

The free server generated the ranking file on a separate host. The local machine only ran the build and the voter.

The local rule engine

The local rules were intentionally tiny. High precision mattered more than recall. Four matchers worked over a combined string of the sanitizer, signal, and frame0:

std::string localRule(const CrashReport& r) {
    std::string text = r.sanitizer + " " + r.signal + " " + r.frame0;
    if (text.find("heap-use-after-free") != std::string::npos ||
        text.find("double-free") != std::string::npos) return "heap_corruption";
    if (text.find("stack-overflow") != std::string::npos) return "stack_overflow";
    if (text.find("SEGV") != std::string::npos ||
        text.find("null") != std::string::npos) return "null_deref";
    if (text.find("pthread") != std::string::npos ||
        text.find("deadlock") != std::string::npos) return "lock_order";
    return "unknown";
}

These rules are not a classifier. They are a veto. When they return unknown, the model may still route the report, but the decision is labeled model_only. When they return a bucket, the model must either agree or face manual review.

The vote

The decision function asks three questions.

1. Does a local rule match?
2. Where does the local bucket appear in the model ranking?
3. How strong is the model's top score?

The table used by the harness:

Local rule	Model rank of local bucket	Top model score	Decision
unknown	no requirement	>= 0.65	auto_route:model_only
unknown	no requirement	< 0.65	manual_review
matched	rank 1	>= 0.60	auto_route:agreed
matched	rank 1	< 0.60	manual_review
matched	rank 2 or later	any	manual_review
matched	not present	>= 0.85	manual_review
matched	not present	< 0.85	quarantine

The important rows are the last two. A high-confidence model answer does not override a clear local rule. It only escalates the report for a human. A lower-confidence disagreement is quarantined.

Reproducible C++ harness

The full program reads both TSV files, applies the local rule, scores the model ranking, and writes one decision per report.

#include <algorithm>
#include <fstream>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>

struct CrashReport {
    std::string id;
    std::string sanitizer;
    std::string signal;
    std::string frame0;
};

struct RankedCandidate {
    std::string bucket;
    double score;
};

struct ModelRank {
    std::string id;
    std::vector<RankedCandidate> candidates;
};

std::vector<std::string> split(const std::string& s, char delim) {
    std::vector<std::string> parts;
    std::stringstream ss(s);
    std::string item;
    while (std::getline(ss, item, delim)) {
        parts.push_back(item);
    }
    return parts;
}

std::vector<double> splitScores(const std::string& s, char delim) {
    std::vector<double> parts;
    std::stringstream ss(s);
    std::string item;
    while (std::getline(ss, item, delim)) {
        try { parts.push_back(std::stod(item)); } catch (...) { parts.push_back(0.0); }
    }
    return parts;
}

std::string localRule(const CrashReport& r) {
    std::string text = r.sanitizer + " " + r.signal + " " + r.frame0;
    if (text.find("heap-use-after-free") != std::string::npos ||
        text.find("double-free") != std::string::npos) return "heap_corruption";
    if (text.find("stack-overflow") != std::string::npos) return "stack_overflow";
    if (text.find("SEGV") != std::string::npos ||
        text.find("null") != std::string::npos) return "null_deref";
    if (text.find("pthread") != std::string::npos ||
        text.find("deadlock") != std::string::npos) return "lock_order";
    return "unknown";
}

std::string decide(const std::string& local, const ModelRank& mr) {
    if (local == "unknown") {
        if (mr.candidates.empty()) return "quarantine";
        if (mr.candidates[0].score >= 0.65) return "auto_route:model_only";
        return "manual_review";
    }

    int pos = -1;
    for (std::size_t i = 0; i < mr.candidates.size(); ++i) {
        if (mr.candidates[i].bucket == local) {
            pos = static_cast<int>(i);
            break;
        }
    }

    if (pos == 0) {
        if (mr.candidates[0].score >= 0.60) return "auto_route:agreed";
        return "manual_review";
    }

    if (pos > 0) return "manual_review";

    if (!mr.candidates.empty() && mr.candidates[0].score >= 0.85) {
        return "manual_review:strong_model_disagreement";
    }
    return "quarantine";
}

int main(int argc, char** argv) {
    if (argc != 4) {
        std::cerr << "usage: crash_voter reports.tsv rankings.tsv decisions.tsv" << std::endl;
        return 1;
    }

    const char tab = 9;
    const char nl = 10;

    std::ifstream reports(argv[1]);
    std::ifstream rankings(argv[2]);
    std::ofstream decisions(argv[3]);

    std::vector<CrashReport> reps;
    std::string line;
    while (std::getline(reports, line)) {
        if (line.empty()) continue;
        std::vector<std::string> f = split(line, tab);
        if (f.size() < 4) continue;
        reps.push_back({f[0], f[1], f[2], f[3]});
    }

    std::vector<ModelRank> ranks;
    while (std::getline(rankings, line)) {
        if (line.empty()) continue;
        std::vector<std::string> f = split(line, tab);
        if (f.size() < 2) continue;
        ModelRank mr;
        mr.id = f[0];
        std::vector<std::string> buckets = split(f[1], '|');
        std::vector<double> scores = splitScores(f.size() > 2 ? f[2] : "", '|');
        for (std::size_t i = 0; i < buckets.size() && i < scores.size(); ++i) {
            mr.candidates.push_back({buckets[i], scores[i]});
        }
        ranks.push_back(mr);
    }

    for (const auto& rep : reps) {
        std::string local = localRule(rep);
        std::string decision = "missing_rank";
        for (const auto& mr : ranks) {
            if (mr.id == rep.id) {
                decision = decide(local, mr);
                break;
            }
        }
        decisions << rep.id << tab << local << tab << decision << nl;
    }
    return 0;
}

Compile and run:

g++ -std=c++17 -O2 crash_voter.cpp -o crash_voter
./crash_voter crash_reports.tsv model_rankings.tsv decisions.tsv

A minimal crash_reports.tsv:

crash-01    heap-use-after-free SIGSEGV __asan_report_load8
crash-02    stack-overflow  SIGSEGV __asan_stack_malloc_0
crash-03    heap-use-after-free SIGSEGV __asan_report_load8

A matching model_rankings.tsv:

crash-01    heap_corruption|stack_overflow|unknown  0.93|0.04|0.03
crash-02    stack_overflow|heap_corruption  0.98|0.02
crash-03    stack_overflow|heap_corruption  0.91|0.07

The first two rows auto-route. The third row becomes manual_review because the model and the local rule disagree, even at 0.91.

Why this beat prompt tuning

Prompt tuning changed what the model said. It did not change what the router was allowed to do with a wrong answer. The vote added a second, independent source. The local rules were cheap to audit. They did not require embeddings, a training set, or a retrain job. The model remained useful for reports the rules could not name. The harness recorded the disagreement path instead of silently believing the score.

Limitations

The rule set is static. New sanitizer messages or architectures will produce unknown and fall through to model-only routing. That path still needs monitoring. The harness assumes the model ranking file has already been validated. Real endpoints can return empty bodies, malformed lists, or repeated candidates. The example also uses raw score thresholds. Teams with labeled crash data should tune those thresholds with a small evaluation set instead of adopting the defaults.

Who should not use this

Do not use this if no human owns the manual_review and quarantine queues. The system can hide wrong model output in a queue that nobody reads. Do not use it for autonomous response, such as paging, rollback, or patch execution. Do not use the model route if the crash bucket IDs feed a downstream tool that mutates state. This is a triage aid, not a safety control.

If a free model tier is the cheapest way to generate ranked candidates on a separate free server, this harness turns that output into an auditable vote. The local rule is what keeps the 0.97 from making someone miss a heap bug.