I read a CUDA inference engine's symbols with Ghidra, then made it 12% faster on my 3090

I run one job at home that never sleeps, a coder model quantized to IQ1_M on an RTX 3090, and my agent sessions hit it all day. Before I changed a single flag I wanted to know what the binary was actually doing, so I read it. The engine is Strata 0.1.44, behind a systemd service on :8090, with llama-swap on :8080 proxying it so a whole GGUF collection shares one entry point.

Reading the binary

Ghidra 12.1.4, headless, on the Linux box. 40 segments, 8.88 MiB of host code in .text, 50.3 MiB of CUDA fatbin images and every one of them built for sm_86, which is the card I own, so there's no JIT surprise hiding at startup. The symbol table survives, not stripped, 12,446 procedures and 54,524 strings. It's an NDEBUG build with zero assert strings anywhere in the binary, and the -O3 markers are present.

The demangled names confirmed the module split was real and running, core::Verifier::run and Verifier::batch_launch with an AdaptGate gating verify windows on expert swaps, RemoteExpertOpt::adapt and PeerExperts::adapt for the adaptive expert tier, detect_cpu_topology_impl plus PoolAffinity behind the 7-worker pinning, prefill::detail::mmq_resident_sort_rows as the prefill path. The serve loop is Python, the engine talks over stdio. Three independent angles agreed on that architecture, the docs, the symbols, and live stage timing from STRATA_DECODE_TIMING.

Startup facts from the log. Expert cache 7325 slots eating 13.94 GiB of a 24.5 GiB card, KV 32768 rows resident plus 6.19 GiB pinned in RAM, hit rate steady at 94-96.7%, MTP draft head 212.9 MiB, --spec-min-p auto-refined from 0.55 down to 0.35, --pcie-frac pinned at 0.20. RAM side, 62G total with 23G available before I touched anything, GPU at 23.8 of 24.5 GiB. With the card that full, every idea I tried had to fit in the remainder.

The failure that mattered most

My sessions don't hold one conversation. They alternate, a cron run, a group chat, a call, back to the long one. So the probe was exactly that, two conversations, alternating turns, measure the return to the first one.

Baseline, returning to conversation one cost a full 10.4 to 10.8s reread. Every switch threw away whatever that session had already read. Inside a single session the prefix cache already works, a 30k prompt repeated in the same session came back in 0.07s and appended turns ran 0.07 to 0.32s. Across sessions nothing parked, so every switch pays a full reread.

The guard is three args, --prompt-cache 3 --conversation-cache-mib 4096 --conversation-cache-slots 4. Parked conversations survive alternation, the same test went from 10.8s to 0.4s. The side effect I didn't expect, shared prefixes across sessions hit the same system prompt root checkpoint, so a cold cron start comes back nearly free, and a 30k read after the parktest came back in 5.3s instead of 10.3s.

Then the tax. At 4096 MiB and 4 slots my in-session small reads went from a clean 0.07s to +0.15 to 0.35s per turn, and an agent loop is constant small reads. I dropped to 3072 and 3, occasional tail 0.07 to 0.35s, near stock. The final config is smaller than the first one that worked, and the tax is why.

The greedy trap

The second failure came from the request shape. My agent stack sends no sampling parameters to local providers, and with none in the request the engine sits at its default, greedy. Greedy on an IQ1_M coder model produces loops, the same token run repeating.

Fix is a sampling block in the service JSON, temperature 0.6, top_p 0.95, top_k 20. runconfig.py lines 20-24 confirm those keys land. Decode stayed inside the probe band, so the win there is correctness.

The swap window

One decode window measured 18.76ms, router plus plan plus bell 2.36ms, VRAM hits 2.29, waits about 1.1ms total, head 0.93. The expert tier swaps its slots between windows, so the swap is dead time in every window.

STRATA_ADAPT_OVERLAP=1 lands the swaps during decode instead. Log after the change read "130 windows took swaps still landing, waited 0.11ms in all", window 18.76 to 17.73ms, median decode 95.6 against 91.9-92.6. The other arm, --adapt-async 1, medians 92.6, and the two conflict, so overlap won and async got dropped the same day.

Two traps in the plumbing

serve/server.py spawns the engine with cached args, so editing the JSON does nothing until the service restarts, the file on disk looks right, the running engine disagrees. A restart is 18-19s, then a poll of /v1/models and a real chat probe before I trust it.

calibrate spawns its own engine, VRAM exclusive, so the service has to be stopped first. My first calibration run had the workers and adapt-tier arms all fail to start because the just-stopped engine was still holding the card. Every number from that run was junk.

What I tested and threw away

• KV prefetch, it engaged and logged layer prefixes, 66k prefill stayed 18.0s before and after. Zero on PCIe 4.
• Fused int8 prefill kernels, engaged, 30k prefill went 10.44 to 10.53s. Flat on IQ1_M.
• --prefill-head-alias, needs short reads off, every prompt read pays 0.3-0.6s of head reload. An agent loop is constant small reads, net negative, skipped by analysis.
• Hugepages, the engine wants 11992 mappings of 2 MiB, the kernel handed back 11170, fragmented, MAP_HUGETLB stayed unavailable, and the reservation would have eaten 22G of RAM for nothing. Reverted to zero.
• The refusal-steering speed projection, 0.2-0.4% per token cost, it changes behavior. No.
• Upstream 0.1.44.1, eleven commits of fixes, nothing perf. Skipped, avoid the risk.

Final numbers

Same probes, interleaved, same session. Short decode 108.7 stock to 121.5 final, +12%. Prefill 2998 to 3004 tok/s, flat, the wins are all decode-side. Session alternation 10.4-10.8s to 0.2-0.5s, about 25x. Persistence mode on with nvidia-smi -pm 1, it was off, and that setting doesn't survive a reboot.

What I'd do differently

Measure the access pattern before the flags. The 25x came from how my sessions alternate, nothing in the engine, and a probe that holds one conversation forever would never have found it.

The dead ends are the deliverable. Six knobs that looked plausible and did nothing, each one with its numbers, so the next person doesn't spend a day re-testing them.

Hardware, Linux gaming box, Ryzen 7 5800X, 64G DDR4, RTX 3090 24G, PCIe generation 4 probed at 25.9 GB/s host to device. October 2026, IQ1_M coder quant, cold-cache probes.