Dropped Exports, Green Asserts: Grading AI C++ Patches With nm

A plugin vendor filed a crash report that was not a crash. Their process started, then dlopen failed with undefined symbol: _ZN4mesh6Loader9fromFileERKSs. The library under test had shipped a green CI run two days earlier. An AI-authored patch had “simplified” a loader by making fromFile an inline header helper and leaving a new fromPath overload for the unit tests to call.

Asserts passed. The .so no longer exported the symbol the plugin had linked against. Functional goldens do not describe linkage. Coverage on a changed hunk does not describe linkage either. The missing check was an export list.

What a symbol golden actually grades

A symbol golden is a sorted list of mangled names the shared object must still provide after the patch. The grader builds the library, dumps dynamic symbols, and diffs that set against the list. Extra exports can be warnings. Missing exports are failures.

Renames show up as one deletion and one addition. The report prints them as a pair so a reviewer does not treat a rename as two unrelated events. Mangled names are the source of truth. Demangled text is for humans only.

c++filt output changes with the filt implementation. The committed golden therefore stores nm names, not prose. This article treats the export set as part of the eval spec for AI C++ patches, next to compile and test. It does not replace libabigail. It catches a class of silent regressions that unit tests reward.

A compact fixture

The fixture is a two-file shared library plus one plugin-shaped client. The client is not run as a unit test. It exists to make the failure mode obvious when the export disappears.

include/mesh/loader.hpp

#pragma once
#include <string>
#include <vector>

namespace mesh {

#if defined(MESH_BUILDING)
#define MESH_API __attribute__((visibility("default")))
#else
#define MESH_API
#endif

class MESH_API Loader {
public:
  static std::vector<unsigned char> fromFile(const std::string& path);
  static std::vector<unsigned char> fromPath(const std::string& path);
};

} // namespace mesh

src/loader.cpp

#define MESH_BUILDING
#include "mesh/loader.hpp"
#include <fstream>
#include <iterator>
#include <stdexcept>

namespace mesh {

std::vector<unsigned char> Loader::fromFile(const std::string& path) {
  std::ifstream in(path, std::ios::binary);
  if (!in) throw std::runtime_error("open failed");
  return std::vector<unsigned char>(
      std::istreambuf_iterator<char>(in),
      std::istreambuf_iterator<char>());
}

std::vector<unsigned char> Loader::fromPath(const std::string& path) {
  return fromFile(path);
}

} // namespace mesh

test/test_loader.cpp — the naive eval that stays green after an ABI break

#include "mesh/loader.hpp"
#include <cassert>
#include <filesystem>
#include <fstream>

int main() {
  auto p = std::filesystem::temp_directory_path() / "mesh_eval.bin";
  {
    std::ofstream out(p, std::ios::binary);
    out.put('A');
  }
  auto bytes = mesh::Loader::fromPath(p.string());
  assert(bytes.size() == 1);
  assert(bytes[0] == 'A');
  return 0;
}

A model that deletes fromFile from the .cpp and marks it inline in the header still satisfies this main. Downstream dlopen does not. That split is the whole reason to grade the .so.

Numbered eval workflow

1. Build the unpatched tree as a shared object with hidden visibility by default and explicit default on the API.
2. Dump dynamic defined symbols with nm -D --defined-only --format=posix.
3. Keep only names that start with _Z, plus any documented C wrapper prefix.
4. Sort uniquely and store the list as goldens/libmesh.so.syms.
5. Apply the candidate patch, rebuild with the same flags, and dump again.
6. Fail the eval on any missing golden symbol. Record extras separately so a human can promote them.

Hidden visibility matters. Without -fvisibility=hidden, every template instantiation becomes noise. The golden then rots on the first STL bump, and the harness starts accusing every model of ABI breakage.

Build flags the grader must pin

The compile line is part of the spec. Changing -fvisibility or C++ ABI flags makes two symbol dumps incomparable. Pin the compiler, the language dialect, and the soname in the same manifest that pins the eval prompt.

CXX ?= g++
CXXFLAGS += -std=c++17 -O2 -fPIC -fvisibility=hidden -fvisibility-inlines-hidden
LDFLAGS_SO += -shared -Wl,-soname,libmesh.so.1

libmesh.so.1: src/loader.cpp include/mesh/loader.hpp
    $(CXX) $(CXXFLAGS) -I include -o $@ src/loader.cpp $(LDFLAGS_SO)

test_loader: test/test_loader.cpp libmesh.so.1
    $(CXX) $(CXXFLAGS) -I include -o $@ test/test_loader.cpp -L. -lmesh \
      -Wl,-rpath,'$$ORIGIN'

A later flag drift is a harness bug. It is not a model regression. Rebuild HEAD against the golden before scoring any patch.

Dump and grade

tools/dump_syms.sh

#!/usr/bin/env bash
set -euo pipefail
so="${1:?shared object}"
# posix format: name type value size
nm -D --defined-only --format=posix "$so" \
  | awk '$2 ~ /^[TDBW]$/ { print $1 }' \
  | grep -E '^_Z' \
  | sort -u

T, D, B, and W cover text, initialized data, BSS, and weak defaults that plugins actually bind to. Undefined symbols (U) stay out. Absolute debug noise stays out. The awk column test is the filter. Do not pipe nm through c++filt before the golden diff.

tools/grade_syms.py

#!/usr/bin/env python3
"""Compare dumped dynamic symbols against a committed golden list."""
from __future__ import annotations

import argparse
import pathlib
import sys


def load_list(path: pathlib.Path) -> list[str]:
    lines = [
        ln.strip()
        for ln in path.read_text(encoding="utf-8").splitlines()
        if ln.strip() and not ln.lstrip().startswith("#")
    ]
    return sorted(set(lines))


def main() -> int:
    p = argparse.ArgumentParser(description="Symbol-set grader for C++ patch evals")
    p.add_argument("--golden", type=pathlib.Path, required=True)
    p.add_argument("--actual", type=pathlib.Path, required=True)
    p.add_argument("--allow-extra", action="store_true")
    args = p.parse_args()

    golden = load_list(args.golden)
    actual = load_list(args.actual)
    missing = [s for s in golden if s not in actual]
    extra = [s for s in actual if s not in golden]

    if missing:
        print("FAIL missing exports:")
        for s in missing:
            print(f"  - {s}")
    if extra:
        label = "NOTE extra exports:" if args.allow_extra else "FAIL extra exports:"
        print(label)
        for s in extra:
            print(f"  + {s}")

    if missing:
        return 2
    if extra and not args.allow_extra:
        return 3
    print("PASS symbol golden")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run order after a candidate patch:

chmod +x tools/dump_syms.sh tools/grade_syms.py
make clean && make libmesh.so.1
./tools/dump_syms.sh ./libmesh.so.1 > /tmp/actual.syms
python3 tools/grade_syms.py \
  --golden goldens/libmesh.so.syms \
  --actual /tmp/actual.syms

Exit 2 means the patch deleted or renamed a public entry. Exit 3 means the surface grew. Growth is not always wrong. It is a different decision from “tests passed.”

Score the triple, not a single boolean

Folding compile, tests, and symbols into one pass/fail bit hides the failure mode. AI patch evals need the mode. The table below is a decision matrix, not a measured leaderboard.

compile	tests	symbols	interpretation
fail	n/a	n/a	does not build; stop
pass	fail	pass	functional miss
pass	pass	fail	silent ABI break
pass	fail	fail	mixed miss; inspect both diffs
pass	pass	pass	candidate, not a ship decision

Labeled example, unexecuted on a public corpus: a patch that inlines fromFile and keeps fromPath for tests yields compile=pass, tests=pass, symbols=fail. That triple is the point of the harness.

Wiring the grader into an AI patch eval

The eval loop stays boring on purpose.

1. Check out the base commit and confirm the golden still matches HEAD. If it does not, the harness is broken, not the model.
2. Apply the model patch as a single diff. Reject patches that touch goldens/libmesh.so.syms unless the task prompt explicitly asks for an API change.
3. Rebuild. If the link of libmesh.so.1 fails, record a compile/link miss, not an ABI miss.
4. Run test_loader. A test fail is a functional miss.
5. Dump and grade symbols. A symbol miss is an ABI miss even when step 4 passed.
6. Store the triple (compile, tests, symbols) as the score. Do not OR the bits into one trophy.

Step 1 is the sanity check for this envelope. A golden that already mismatches HEAD will accuse every model of ABI breakage. Step 2 stops the model from “fixing” the eval by rewriting the spec. Both steps are cheaper than another round of prompt tweaking.

Projects that already freeze ABI with a linker version script can dump that file instead of raw nm output. The grader then diffs the global: names. The idea is the same. The noise floor is lower because internal instantiations never enter the list.

LIBMESH_1.0 {
  global:
    _ZN4mesh6Loader8fromFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE;
    _ZN4mesh6Loader8fromPathERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE;
  local:
    *;
};

Commit the map. Pass --version-script at link time. Teach dump_syms.sh to fail if a golden name is present in the map but absent from nm. Two sources of truth that disagree are a harness incident.

Where a free codegen host fits

Patch generation still needs a model endpoint. Local grading still needs a Linux box with g++ and nm.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode advertises free model access and a free server option. Those two claims are the only product facts used here. This workflow does not depend on a particular model name, quota, or GPU SKU. A free server is useful only when the same Makefile, the same nm flags, and the same golden file run there and on the maintainer laptop.

If the remote image ships a different libstdc++ or a different default visibility, the symbol set is a different spec. The eval then measures the image, not the patch. Teams that already self-host models can keep generation local and still use the grader unchanged. The artifact is the symbol list and the exit codes, not the vendor.

Limitations

The method assumes ELF shared objects and GNU or LLVM nm posix output. Static libraries, Apple Mach-O, and Windows PE need different dumpers. LTO and --gc-sections can drop a symbol that a non-LTO golden still lists. Inline namespaces, version scripts, and the std::string dual ABI (__cxx11) make goldens toolchain-specific.

Template-heavy headers export whatever got instantiated in the .so. That set is not a stable API. Projects in that shape should golden a version script instead of the raw dump. The grader in this article does not check sizeof, field layout, or exception specifications. abidiff covers that layer. This envelope is the cheap filter that unit tests skip.

Do not treat a passing symbol golden as proof that plugins will load. dlopen still cares about RPATH, soname, and dependent .so files. Those belong in a separate loader smoke test. Mixing them into grade_syms.py blurs the failure mode the table above exists to keep distinct.

Who should skip this

Header-only libraries have no .so to dump. Applications that never export a plugin ABI gain little. Eval suites that already run a full abidiff gate do not need a second, weaker list. Windows-only shops should not pretend nm -D is portable.

Do not use this as a reason to freeze every internal helper. Visibility defaults exist so the golden can stay small. If the dump contains thousands of std:: instantiations, fix the flags before blaming the model. A noisy golden is not stricter. It is less reproducible.

Closing

AI C++ evals that only score asserts will keep accepting patches that move the public surface. A sorted symbol file is a dull artifact. It is also a reproducible one: same compiler, same visibility, same nm format, same exit codes.

Keep the golden next to the tests. Refuse patches that edit it without an explicit API task. Promote extras only when a human intends a new export. Clone the scripts, break fromFile on purpose, and confirm exit 2 before scoring any model output. A hosted generator is optional, and only honest if it repeats that same dump.