DEV Community

Cover image for I pointed my memory SDK at physics simulations instead of sensors — it worked once, and failed informatively twice
VAAS-X
VAAS-X

Posted on

I pointed my memory SDK at physics simulations instead of sensors — it worked once, and failed informatively twice

Every other test I'd run on vaas-x so far was telemetry: industrial sensors, wearable IMUs, agent session logs. That left an open question — does outcome-grounded retrieval transfer to a domain with no sensor stream at all, where the "state" is a scientific simulation's own parameters? I ran three independent physics/computing systems through it to find out, held to the same bar as everything else: a ground truth computed independently of the SDK for every test point, and the honest result reported whether or not it flattered the product.

Same three calls, applied to simulation output

from vaasx import Bootstrap

brain = Bootstrap(api_key="YOUR_API_KEY", device_id="physics_qec")

brain.ingest([{
    "state": {"text": "Syndrome: 1000000100010001010010011011000000..."},
    "action": {"text": "correction=000000000011111100000000..."},
    "outcome": {"text": "learned recurring burst fault pattern", "success": True},
}])

hits = brain.query("Syndrome: 1000000100010001010010011011000000...", k=1)
print(hits[0]["score"], hits[0]["text"])
Enter fullscreen mode Exit fullscreen mode

Win #1: a quantum error-correction decoder learning one fault, cleanly

A 64-bit random linear code (n=64, k=32) mathematically cannot correct an arbitrary weight-6 error — it's past the code's guaranteed correction distance. But real hardware often produces the same high-weight fault repeatedly: a stuck line, a recurring burst on the same physical bits. I queried a specific weight-6 burst syndrome before vaas-x had ever seen it (no match, exactly as expected — the code genuinely can't solve this algebraically), ingested exactly one episode describing that syndrome and its correction, then queried the identical syndrome again.

The correct correction came back — and I didn't just check that a hit came back, I ran the recalled bits through the code's own check_correction() to confirm it actually fixes the error. To be clear about what this does and doesn't show: it's a recall test, not a generalization test. Memory remembers a specific pattern it's been shown once; it doesn't predict corrections for syndromes it's never seen.

The harder question: two continuous-parameter systems

The QEC case is categorical — a syndrome either matches something seen before or it doesn't. The harder test is continuous interpolation: given a physics simulation's continuous input parameter (a coupling constant, a field ratio), can a text-embedding query predict the correct phase for a value memory's never been shown? I tested this against two independent ground truths: a real Metropolis Monte Carlo lattice simulation (13 kappa values ingested, averaged across 3 independent MC chains per point to smooth out genuine sampling noise near the transition) for a lattice φ⁴ scalar field theory, and real exact diagonalization (20 h/J ratios ingested) for a transverse-field Ising model's quantum phase transition. Both ground truths were recomputed fresh for every held-out query.

Lattice φ⁴: 5 of 8 held-out kappa values correctly classified. Transverse-field Ising: 4 of 10 held-out h/J ratios correctly classified.

The fix I tried that didn't work — and why that's the useful part

In both systems, every miss had the same shape: memory defaulted toward whichever phase had more representation in what was ingested (7 of 13 φ4 episodes were "ordered"; 7 of 20 TFIM episodes were "critical"), instead of discriminating finely near the actual phase boundary. Before publishing this, I tried a legitimate fix: k=7 retrieval with a score-weighted majority vote instead of trusting a single top-1 hit — standard practice, not a way of engineering the answer.

It changed nothing. The TFIM run predicted "critical" for every single query, identical before and after the fix, down to the query. That rules out top-1 noise as the explanation — the retrieved neighbourhood itself is dominated by the majority class regardless of the query value.

My read: this is a real, useful limitation, not a bug. A short, numerically-dense query string like "h/J=0.350" differs from a training example by only a few characters in an otherwise near-identical string — a general-purpose sentence embedding model has little reason to weight that span heavily, and it shows up exactly like this. Retrieval does coarse region classification correctly (every miss landed on the wrong side of a real, close decision boundary, not a wildly wrong one), but it's not a substitute for a numerical model when the question is "exactly where is the line."

What this does and doesn't show

Categorical or recurring-pattern memory — QEC's syndrome recall, or industrial/wearable channel classification, where the signal dominates the description — is where this SDK is validated to work well. Fine interpolation across a continuous parameter from a bare numeric label in the query text is not, at least not without more deliberate encoding work than a raw f-string. If you're evaluating memory-augmented retrieval for a scientific-computing use case, match the shape of your problem to what's actually proven here.

Full runnable code for all three systems, plus the exact numbers: reproduction guide. Technical writeup: whitepaper PDF.

Top comments (0)