I built a C++ optimizer that trusts neither me nor the LLM

Tracing JITs have a trick that AOT-compiled C++ never gets to use: they watch the values flowing through hot code, specialize for them, and guard the assumption with a cheap check. V8 and LuaJIT do this at runtime. I wanted it for a C++ trading system, ahead of time.

So I built pgspec — a small pipeline that profiles a binary on recorded workload, finds the values that dominate at runtime, and regenerates the build so those values become compile-time constants on the hot path. Guarded, with a generic fallback, and with an LLM in the loop proposing specializations.

The most important design decision turned out to be this: nothing in the pipeline is trusted — not the rule-based heuristics, not the LLM, not even my own benchmark. Every proposal has to survive a bit-exact differential test and beat the baseline on a benchmark before it ships. That gate ended up rejecting my code, my benchmark methodology (twice), and one of the LLM's ideas. Here's the tour.

The pipeline

profile (value histograms)  →  proposer (rules | Claude)  →  codegen (specialized.hpp)
        →  bit-exact differential test  →  benchmark gate  →  miss-rate monitor

The hot path is a policy template. The generic spec reads everything from the order at runtime:

struct GenericSpec {
    OrderType type(const Order& o) const      { return o.type; }
    uint32_t  lot_size(const Order& o) const  { return o.lot_size; }
    double    tick_size(const Order& o) const { return o.tick_size; }
};

template <class Spec>
inline Result process_order(const Order& o, Spec spec) {
    const uint32_t lot = spec.lot_size(o);
    const uint32_t lots = o.qty / lot;     // real division with GenericSpec…
    const uint32_t odd  = o.qty % lot;
    // … fee switch on spec.type(o), notional on spec.tick_size(o) …
}

A profiling build records per-site value histograms — crucially, the joint distribution of value tuples, not per-field marginals. Two fields can each look bimodal while only two of their four combinations ever co-occur, and a guard is a conjunction, so coverage is a property of the joint distribution. The analyzer turns the top tuples into generated specs:

// AUTO-GENERATED
struct Spec0 {
    [[gnu::always_inline]] static inline bool guard(const Order& o) {
        const unsigned g0 = (unsigned)(o.type == OrderType::Ioc);
        const unsigned g1 = (unsigned)(o.lot_size == 100u);
        const unsigned g2 = (unsigned)(o.tick_size ==
                                       std::bit_cast<double>(0x3f847ae147ae147bULL));
        return (g0 & g1 & g2) != 0u;
    }
    constexpr OrderType type(const Order&) const      { return OrderType::Ioc; }
    constexpr uint32_t  lot_size(const Order&) const  { return 100u; }
    constexpr double    tick_size(const Order&) const { return std::bit_cast<double>(0x3f847ae147ae147bULL); }
};

After inlining, qty / 100 becomes a magic-number multiply, the fee switch folds away, and the tick multiply is a constant. The guard is deliberately branchless — each compare materializes as a flag, combined with bitwise &, so the whole thing costs one heavily-biased branch. (The naive && chain compiled to three data-dependent branches and cost 14% on workloads that drift from the profile.)

The correctness contract is brutal and simple: the specialized binary must produce bit-identical results to the generic one on any input. Guards select which code path, never what it computes. The pipeline enforces this with checksums over both the profiled-like workload and an adversarial one, built with -ffp-contract=off so FP folding stays IEEE-identical.

The LLM in the loop

The fun part. The analyzer has two proposers behind one contract: a rule-based one (top-k joint tuples) and Claude. The model receives the hot-path source plus the profile — including exact bit patterns of observed doubles — and returns specs as structured JSON, under a contract that says, roughly: whenever all your guard terms hold, every accessor must equal the field exactly, or the differential test will reject you.

The trust boundary is explicit:

• Every expression the model returns must tokenize against a whitelist (fields of o, enum values, numeric literals, std::bit_cast<double>(0x…ULL), comparison/arithmetic operators). I tested it with system("rm -rf /") in a guard term — rejected before codegen.
• Whatever passes is compiled, differential-tested bit-for-bit, and benchmark-gated like any other spec.
• Any failure — no key, invalid JSON, rejected expression, model refusal — degrades to the rule proposer with a warning.

The live run exceeded expectations. The model returned the two correct hot tuples with exact bit patterns, deliberately stopped at two specs ("the remaining ~5% is spread over 12+ tuples at ~0.42% each; each would cost a dispatch branch for well under 1% gain"), warned that upstream-computed tick sizes might differ by 1 ulp from the profiled bits so the guard would safely miss, and suggested checking that via the hit counters — it had noticed the monitoring macros in the source. Its "what I'd try next" list included hoisting dispatch per burst. More on that below, because that idea got rejected — by measurement, not by me.

This is the same conclusion the community keeps arriving at from the code-review side — see "From Vibe-Coding to Verification" by Tamiz Uddin and "Why AI-Generated Code Needs Independent Verification" by pangxueyuan2creator: LLM output is a proposal, and deterministic verification is what makes it shippable. pgspec just applies that to performance work, where "correct" isn't enough — it also has to be faster, and that's an empirical question.

What measurement actually taught me

Every one of these surprised me, and every one came from the pipeline disagreeing with my expectations.

1. My first benchmark lied. The checksum was a serial FNV chain (xor → mul → xor), whose ~5-cycle latency was the critical path. It hid the hot-path cost entirely and "proved" specialization made things slower. Independent accumulators fixed it. The benchmark is part of the product.

2. macOS core scheduling produced a 1.9x phantom. Identical binaries differed by 1.9x between pipeline runs — P-cores vs E-cores. The fix: interleave the binaries' runs and take min-of-N, so scheduling noise hits everyone alike.

3. Throughput-bound loops punish this optimization; latency-bound loops reward it. With independent iterations, Apple Silicon's out-of-order core hides udiv latency in an otherwise-idle divider unit, while the magic-multiply replacement competes for shared ALU ports: 0.74x. Put the division on a serial dependency chain (position feedback into the next order's size, like real inventory/risk logic) and the same transform gives 1.17–1.19x. Same code, opposite verdicts.

4. Generated code pushes surrounding code past inliner thresholds. With two specs, the hot-loop lambda silently stopped being inlined, which forced the running position through a store-load on the serial chain. It looked exactly like "multi-spec doesn't pay" (0.84x) until the disassembly showed the outlined call. One always_inline later: 1.17x, with two specs beating one.

5. Partial specialization is a branch-predictor hack. A spec that pins only type == Ioc (99% of orders) and keeps lot/tick as runtime reads made the adversarial workload 1.4–1.6x faster: the guard acts as a predictable pre-classifier, splitting one unpredictable 4-way switch into two heavily-biased paths. The rule proposer can't express this; the LLM contract can.

6. The LLM's per-burst idea lost — and the gate caught it. Hoisting the guard cascade to the head of each instrument burst sounds great, but run membership still has to be verified per order, so you save almost nothing and pay run bookkeeping: 1.13x vs 1.19x for the plain cascade. The idea implicitly assumes a feed-provided dispatch key. Which led to…

7. The dispatch-strategy crossover is a function of spec count. With a real instrument id, a generated switch jumps straight to the instrument's spec and the guard only verifies. At 2 specs the cascade wins (the switch adds nothing); at 6 Zipf-weighted instruments the keyed dispatch wins (1.10x vs 1.06x, and 11% faster than the cascade in throughput) because cascade cost grows with N while the switch is O(1). Nobody chooses the strategy: the pipeline builds all three binaries and the gate picks per profile.

The part I'd defend in a design review

The pipeline ends with an accept/reject gate and a production monitor:

• Gate: a specialization (or dispatch strategy) that doesn't beat the alternatives on the primary workload is reverted. It has, so far, rejected a throughput regression, a mis-inlined build, and an LLM-suggested structural change. The proposer suggests; measurement decides.
• Monitor: the generated header embeds the profiled generic-path share; a -DPGSPEC_MONITOR build (~2% overhead, zero when compiled out) counts live guard hits and flags drift when the live miss-rate exceeds twice the expectation. On the profiled workload: 4.98% live vs 5.00% expected. On a shifted one: 95.8%, flagged. A shipped specialization is a bet that the profiled distribution holds — and the day it breaks is exactly the volatile day you care about, so the fallback path has to stay fast, not merely correct.

What's next

Instrument-aware LLM proposals (the model already sees the instrument column; it should emit the keyed dispatch table too), Intel PT/LBR sampling instead of instrumented profiling builds, and — once compilers catch up — C++26 reflection (P2996) to generate the probes and Spec structs from annotated types instead of an external Python codegen.

Repo: github.com/nfedotenko/pgspec. The whole thing is ~1200 lines including the pipeline, and every number above is reproducible with ./pipeline.sh.

If you're into this corner of the world, TickForge by Tapesh Chandradas is a nice hand-written counterpart — the manual discipline pgspec tries to automate.

Comments

[Odinaka Robert Nnamani]

The joint-distribution point is the one that stuck with me. Two fields can each look bimodal while only two of their four combinations ever show up.