We catalogued 55+ AI-agent failures in low-level code — and shipped 124 verified skills to fix them

✨ Disclosure: this article was drafted with AI assistance. Every technical claim in it is source-traced in the linked repository (registry/claims.yaml, 177 primary sources). The failure classes below come from real, documented incidents — not vibes.

AI coding agents are excellent at boilerplate and unreliable at low-level code — and the failures are not random. They cluster into predictable classes with a recognizable signature: the code looks correct, compiles, and still does the wrong thing.

We spent three research passes collecting 55+ documented failures (source-traced, not anecdotes) and turned them into 124 verified engineering skills. Here's what we learned.

---

TL;DR — the 5 failure classes

1. Assembly hallucinations — invented mnemonics, inverted operand order, silently truncated immediates.
2. Fake parallelism — thread-safe-looking code that runs on one thread.
3. Rust API drift & crate hallucination — behavioral API changes and nonexistent crates that resemble real ones.
4. Misleading verification — "passing" harnesses that never test the target.
5. Systems-level blind spots — VM leaks invisible to heap tools, timing side channels.

The fix isn't "be more careful". It's mechanical gates: assemble → disassemble → compare bytes; measure real parallelism; check the API actually exists; make your harness able to fail.

---

1. 🧠 Assembly hallucinations

Agents invent instructions that do not exist. One generated CDC COMPASS pseudo-ops (JOB, SST, OCT) for a program that "looked like assembly". Another produced movqad — no such instruction.

The dangerous failures don't error — they silently corrupt:

; What the agent wrote            ; What actually assembled
imul eax, eax, 38                  ; 69 c0 00 00 00 00  (the 38 is DROPPED)
mov (%rax), %eax                   ; 8b 00              (fine — but one byte vs. eax?)

imul eax, eax, 38 assembles to 69 c0 00 00 00 00 — the immediate is silently discarded by the parser. This is exactly the bug class behind BBoeOS PR#584. Add AT&T/Intel operand inversion, missing size hints (inc [counter] vs inc qword [counter]), and "AX is 8-bit" claims, and you have a reliable failure generator.

The gate:

gcc -c sample.s && objdump -d sample.o       # AT&T
gcc -c -masm=intel sample.s && objdump -d -M intel sample.o   # Intel

If the disassembly doesn't match what you wrote — same mnemonic, same operands, same size — you didn't write that instruction. Calibration: LLM disassembly gets exact matches right ~14% of the time; decompiler "fixes" are correct ~37%.

2. 🔁 Fake parallelism

Models produce ConcurrentHashMap/atomics that look thread-safe but execute everything on one thread. The CONCUR benchmark (arXiv:2603.03683) catches deadlocks and races that linear benchmarks cannot — because the code isn't actually concurrent.

Our gate is mechanical, not syntactic:

Program	Live threads	Wall-clock	Verdict
"thread-safe" demo	1	1.206 s	fake parallelism
real pthread split	4	0.304 s	real parallelism

Count live threads and measure wall-clock scaling. Thread-safe syntax is not parallelism.

3. 🦀 Rust API drift & crate hallucination

RustEvo² (arXiv:2503.16922) is the clearest dataset: models nail stabilized APIs at 65.8%, but behavioral changes (same signature, different semantics) at only 38%. Performance collapses from 56.1% to 32.5% for APIs added after the training cutoff. RAG helps (+13.5%) — but you can't RAG what doesn't exist yet.

And the supply chain angle is worse: agents hallucinate crates that do not exist but resemble real ones — a typosquatting risk (serde-json vs serde_json). Studies report 5.2% (commercial) to 21.7% (open-source) package hallucination rates.

cargo info serde-json    # exit 101 — does not exist
cargo info serde_json    # the real crate

The gate: verify existence (cargo search/crates.io API), pin the toolchain, and treat behavioral changes as the most dangerous class. In crypto specifically, only 23.3% of generated Rust compiles and 57% of that is vulnerable — with nonce reuse the leading cause (arXiv:2604.27001).

4. 🎭 Misleading verification

The worst class of all: a "passing" test that doesn't test the target.

• A fixed-shape allclose oracle certifies buggy GPU kernels as correct — fuzz + fp64 reference catches 9/9 (arXiv:2606.20128).
• Kernels "pass review, then segfault under load" (arXiv:2602.19594).
• Ghostty's 37–130 GB VM leak: the page pool reused an mmap without ever calling munmap — an agent was the trigger, not the cause (mitchellh.com/writing/ghostty-memory-leak-fix).

The Iron Law: a harness that cannot fail is not evidence. The ablation test is: break the target — does your test catch it? If it still passes, your test is decoration.

5. 🧊 Systems-level blind spots

• Timing side channels: early-exit string comparison leaks the matching prefix (CWE-1254; ~40% of Copilot's crypto code). Measured on our host: gcc -O2, 500k × 256-byte compares — early-exit memcmp 0.054 s vs constant-time ~0 s.
• UB assumptions: "works at -O0, breaks at -O2" — the crash site is a symptom; the corruptor is elsewhere.

---

What we built: 124 verified skills

Each skill is a compact SKILL.md that answers five questions before a line is written:

Question	Why it matters
When to use / when not to	the agent loads the right tool, not everything
What the agent often gets wrong	the named failure classes above
How to reason correctly	the positive process, not just "don't"
What to verify / how	executable gates, not vibes
Where the knowledge comes from	every claim → primary source

Verified, not asserted: 65 of 124 skills were validated by actually running examples on real toolchains (GCC 16.1, rustc 1.97, GDB, objdump, CMake/Ninja). The remaining 59 are honestly marked researched with the exact command that would verify them.

git clone https://github.com/TrothByte/low-level-skills-trothbyte
python tools/validate.py     # 124 skills + registry + 177 sources, all gate in seconds

Also installable via npx skills add TrothByte/low-level-skills-trothbyte or as a Claude Code plugin marketplace.

The takeaway

The agents aren't broken — our expectations are. "It compiles" was never the bar for low-level code. The bar is: assemble → disassemble → compare bytes; measure real parallelism; check the API exists; make the test able to fail. The 55+ catalogued failures become 124 skills that encode exactly those gates.

If you write, review, or debug C, C++, Rust, assembly, kernels, or firmware with an AI — you'll recognize these failures. The library is free and MIT-licensed: github.com/TrothByte/low-level-skills-trothbyte.

Surveys with full source traces are in the repo's research/ folder. Found a failure we haven't catalogued? Open an issue — new skills must be source-traced and differentiated from the existing 124.