DEV Community

Cover image for 98 Calls, 98 Errors, 0 Detections: The Bug My Test Suite Called Green
Debashish Ghosal
Debashish Ghosal

Posted on AI-assisted

98 Calls, 98 Errors, 0 Detections: The Bug My Test Suite Called Green

#ai #programming #testing #healthydebate

A detector returned None on every call for two full corpus runs, and the test suite reported green the whole time.

Not "it ran and found nothing." It never executed a successful call. Every signal I trusted — green tests, firing detectors, clean telemetry — said fine. The only reason I found it is one line in a summary file I'd been skimming past.

This is the kind of bug except Exception: return None makes possible, and most of us have written that line.

The run that looked great

I was piloting 100 synthetic agent traces through the LLM-augmented detectors in agentwatch — one tool in agentic-suite, a suite of open-source, harness-agnostic security tools for AI agents. The summary came back and it looked genuinely good:

traces_processed: 100
chat_calls: 363
embed_calls: 98
errors: 98
total_tokens: 52,578
json_parse_success: 363
json_parse_fail: 0
Enter fullscreen mode Exit fullscreen mode

363 chat calls. 100% JSON parse success. 52,578 tokens. Detectors firing: hallucination: 98, confusion_pattern: 2, quality_degradation: 1. All sensible numbers.

Then one line: embed_calls: 98, errors: 98.

Every single embedding call failed. And I didn't notice — because the corpus run counts fires, not non-fires. A detector that never fires doesn't show up as "0." It's simply absent. And absent is not zero. Absent means this detector never ran.

This wasn't the first time, either. An earlier run, FT-11b, had 24 embed calls with 24 errors — same 100% failure rate, same invisibility. The bug had been there across two full corpus runs (FT-11b at 25 traces, FT-11d at 100 traces), and between them 122 embed calls had all failed. The error rate was in the telemetry the whole time; it just wasn't in the assertions.

The error was sitting in the logs the entire time:

LLM embed call failed: Error code: 404 - {'error': {'message':
"Model 'all-MiniLM-L6-v2' not found. Available models:
Llama-3.2-3B-Instruct-4bit, Qwen3-4B-Instruct-2507-4bit, Qwen3-8B-4bit,
Qwen3.5-4B-4bit, Qwen3.5-9B-MLX-4bit"}}}
Enter fullscreen mode Exit fullscreen mode

Logged 98 times. The tests passed, so nobody went looking for it.

The code path that made it invisible

The client was written to be resilient — a failed embedding shouldn't crash the pipeline:

async def embed(self, text):
    try:
        resp = await client.embeddings.create(model=self.embed_model, input=text)
        return list(resp.data[0].embedding)
    except Exception as exc:
        self._record_error(start)
        logger.warning("LLM embed call failed: %s", exc)
        return None  # ← silent failure
Enter fullscreen mode Exit fullscreen mode

The detector treats None as "no vector available," which is reasonable:

current_vector = await self._client.embed(output_text)
if current_vector is None:
    return None  # "can't compare vectors → no anomaly"
Enter fullscreen mode Exit fullscreen mode

And the test reads None as "didn't fire," which is correct for a negative case:

fired = anomaly is not None
fire_ok = fired == scenario.expect_fire   # False == False → pass
Enter fullscreen mode Exit fullscreen mode

Three different meanings collapsed into one None:

A None from "I checked and found no anomaly" == a None from "I couldn't check because the API failed" == a None from "I never called the API at all."

Without call telemetry, those are indistinguishable. All three look like a passing negative case.

The reason this stayed hidden in the corpus runs is structural. Those runs count fires. output_drift wasn't "0" in the anomaly counts — it was absent. And absent reads exactly like "the detector ran and found nothing" when you're scanning a summary.

Making the detector actually work

The endpoint had no embedding model at all. OMLX only serves chat completions — all-MiniLM-L6-v2 isn't on the list. So when embed() comes back empty, I now ask the same model to rate how similar two texts are, on a 0-to-1 scale:

if current_vector is not None and baseline is not None:
    sim = cosine_similarity(current_vector, baseline)
else:
    # Embedding unavailable — fall back to chat-based similarity
    sim = await self._chat_similarity(baseline_text, output_text)
distance = 1.0 - sim
if distance >= self._threshold:
    return Anomaly(..., evidence={"method": "chat-fallback", ...})
Enter fullscreen mode Exit fullscreen mode

The prompt is small and strict: "Compare two texts and rate their semantic similarity on a scale of 0.0 to 1.0. Return only JSON: {"similarity": <float>}." The distance is 1.0 - similarity. It's coarser than vector cosine — the model gives you a judgment, not a continuous metric — but it gets the fire-or-don't-fire decision right.

tip: When a primary API is unavailable, a chat-based fallback can produce correct verdicts with the same model. The tradeoff is precision, not correctness. Record the method in the evidence dict so operators know which path produced a given verdict.

There's a subtlety I almost missed. The first call has to set the baseline, so it returns None regardless — same as the vector path. The fallback only kicks in on the second call forward, once there's a baseline text to compare against. Get that ordering wrong and you fire on the very first trace, which is noise.

The check I should have written first

This is the part most suites skip. Before each scenario, snapshot the client's call counters. After, measure the delta for that scenario:

stats_before = dict(client._stats)
anomaly = await det.detect_async(summary, spans, pool=None)
stats_after = dict(client._stats)

llm_calls = stats_after["chat_calls"] - stats_before["chat_calls"]
embed_calls = stats_after["embed_calls"] - stats_before["embed_calls"]
llm_called_ok = (llm_calls + embed_calls) > 0

ok = fire_ok and sev_ok and type_ok and expl_ok and evid_ok and llm_called_ok
Enter fullscreen mode Exit fullscreen mode

If the LLM was never called, the scenario fails — even if the detector returned the "right" answer. A silent None with zero calls is a hidden skip, not a pass.

That check is the difference between a real pass and a hidden skip, and it's the one most suites don't have. It's also cheap: the client already tracks its own call counts for telemetry, so the harness just reads them.

After the fix, both scenarios run with real calls and real evidence:

✅ ED1  output_drift  fire=True   sev=critical  llm_calls=1  llm_errors=2
✅ ED2  output_drift  fire=False  sev=None      llm_calls=1  llm_errors=2
Enter fullscreen mode Exit fullscreen mode

The two llm_errors are the failed embed calls — still 404, still logged, but now they don't define the outcome. The one llm_calls is the chat fallback doing the actual work. The model's actual verdicts:

ED1 (completely different texts):  {"similarity": 0.1}   → distance 0.9 → fires critical
ED2 (nearly identical texts):      {"similarity": 0.95}  → distance 0.05 → silent (correct)
Enter fullscreen mode Exit fullscreen mode

Both were the right call. And this time I can prove the model actually made them.

This isn't about embeddings

The shape repeats anywhere a dependency returns a sentinel on failure:

Dependency Sentinel Looks like
LLM embed() None "no anomaly"
DB fetchrow() None "no baseline"
HTTP GET "" "no data"
Cache get() None "not in cache"
File read() "" "file is empty"

In every case, the failure wears the costume of legitimate emptiness. The test passes. The system is broken. Nobody knows — until someone counts the calls or checks the status, which is the thing the test didn't do.

The fix is always the same shape: prove the dependency was exercised, not just check the output. Count the calls. Check the response status. Assert the network was reached. A None return is only trustworthy if you can prove the call happened.

What I changed

A missing anomaly type is not "zero anomalies." It's "this detector never ran." Check for presence, not just count.

Log warnings are not alerts. The 404 was logged 98 times in one run, 24 in an earlier one, and nobody saw it. If a failure matters, make it an assertion, not a log line.

None is a terrible sentinel because it doesn't tell you which of its three meanings it has. "I checked," "I couldn't check," and "I never tried" all collapse into the same value and the same passing test.

The deeper habit change for me was the assertion. Before this, my tests asked "did the detector return the right answer?" After this, they also ask "did it actually do the work that produces an answer?" The first question can be satisfied by accident. The second one can't.

All of it is fixed in agentwatch v0.1.0, with the llm_called_ok check enforced on every LLM scenario.

The question worth asking of your own repo: which green test would still pass if the service behind it were down? The answer is usually the one that asserts the output but never asserts the dependency ran.

References

Top comments (2)

Collapse
 
vitayao profile image
Ziwei Yao •

Does llm_called_ok distinguish attempted calls from successful, parsed responses? In the snippet it only checks that the call-count delta is positive. If the embedding request and the chat fallback both fail, a negative scenario could still return None with a positive call count. A case where both paths are unavailable seems worth adding alongside ED1/ED2; it should report an evaluation error rather than a clean negative.

Collapse
 
innokentyb profile image
Kent Bodrov •

Ziwei's availability test would close one gap. I'd also keep results split by method: a cosine distance and a model's 0–1 similarity judgment need separate calibration. Passing ED1/ED2 establishes those two extremes, not the behavior near your alert threshold. A held-out set concentrated around that boundary could show whether fallback changes false alarms or missed drift.