How I Found an IEEE 754 Violation in an AI Chip Company's Math Kernel

Görkem TUNA — Tue, 25 Aug 2026 14:01:36 +0000

A field report from auditing Tenstorrent's tt-metal SFPU kernels — with the exact method, so you can do it too.

The bug

Tenstorrent — Jim Keller's chip company — ships tt-metal,
the software stack for their RISC-V-based AI accelerators. Deep in its SFPU (Special Function
Processing Unit) kernels, ttnn.atan2 had this behavior:

torch.atan2(float("inf"), 0.0)   # → 1.5708  (π/2, correct per IEEE 754)
ttnn.atan2(float("inf"), 0.0)    # → 0.0     (wrong)
ttnn.atan2(float("inf"), -0.0)   # → 3.14159 (π, also wrong — should be +π/2)

Not a rounding nit. A hard violation of the IEEE 754 special-case table that every
C library, every GPU vendor, and PyTorch implements identically — in an op that
gradient-based navigation, robotics, and phase computation rely on.

Why nobody caught it

The existing test suite generates random inputs in a finite range. Random floats never
produce inf, and essentially never produce exact ±0 in the denominator. The special-value
table is precisely the set of inputs your fuzzer will never sample.

This is the meta-lesson of every kernel bug I have found:

Random testing explores the measure-one set. Bugs live in the measure-zero set.

The hunt

I audited the kernel against a table I wrote before reading the code — the oracle comes
first, or you will unconsciously excuse whatever the code does. For atan2(y, x) the
interesting cells are y ∈ {±0, ±∞} crossed with x ∈ {±0, ±∞, finite}.

Then read the kernel as an algorithm, not as code. The SFPU implementation:

computes min = min(|x|,|y|), max = max(|x|,|y|),
reduces to atan(a) on [0,1] where a = min/max,
applies quadrant corrections,
and keeps a rescue branch: "if both inputs are zero, return ±0".

The rescue was guarded by min == 0. Now trace y = +inf, x = +0:

min = 0, max = inf
The |y| ≥ |x| branch correctly computes r = π/2 − 0 = π/2 ✅
The rescue checks min == 0 → true → r = 0 ❌

The rescue was written for "both zero" but its condition only tests one of the two.
Since 0 ≤ min ≤ max, the correct both-zero condition is max == 0 — which also implies
min == 0, and never fires for the infinite case.

The fix

One comparison, three architecture copies (Wormhole, Blackhole, Quasar — math kernels are
duplicated per hardware target, and a fix that misses a copy is incomplete):

-v_if(min == 0.0f) {
+v_if(max == 0.0f) {

Plus a deterministic regression test pinning the four inf/zero cells against the torch
golden — the random-range test that would have caught this costs ten lines.

The fix was reviewed by three Tenstorrent engineers and
merged into main.

The general method

Write the oracle first. The special-value table, from the standard — before reading a line of kernel code.
Read the kernel as an algorithm. Strip the intrinsics: it is range reduction → polynomial → exponent reconstruction → sign fix-up. Reconstruct it in plain fp32.
Hand-trace every special value through the model: ±0, ±inf, NaN, ±1, ±INT_MIN, boundary−ε, boundary+ε, boundary-exact. One divergent cell is a finding.
Fix minimally, everywhere. All architecture copies. Mirror the codebase's idioms.
Deterministic regression test with the exact edge values.

I have since catalogued four more recurring patterns from auditing 15+ kernels — clamping
the range-reduction driver instead of guarding the output, boundary strictness
mismatches vs the reference, order-of-operations overflow, and INT_MIN sign-magnitude
conversion edges. The full playbook:

→ github.com/gundi61/kernel-audit-playbook

The meta-lesson

The highest-value bugs in 2026 are not behind random fuzzing — the agents already run
fuzzers. They are behind specification-first reading: pick the table the hardware
implicitly promises (IEEE 754, a reference implementation's docs), trace the implementation
against it by hand, and test exactly the cells a sampler cannot reach.

The measure-zero set is where the money is.

Found, fixed and merged as
issue #54241 +
PR #54240 in tenstorrent/tt-metal.