DEV Community: FX-LgLL

SIMURG: A Guard That Catches LLM Hallucinations Mid-Stream and Heals Them

FX-LgLL — Tue, 22 Sep 2026 12:01:37 +0000

Your LLM looks fine in the demo. Then it ships.

Somewhere between "it works on my machine" and "a user saw this", three failure modes show up in production streams:

the model falls into a repetition loop ("...and as we have seen, and as we have seen, and as we have seen..."),
it drifts into another language mid-answer,
it collapses into structural garbage — table echoes, #REF! noise, token-level static.

These are not model-intelligence failures. They are decoding failures, and they are detectable while the stream is still open. That is the entire premise of SIMURG: a streaming integrity monitor that watches every token as it arrives and raises a calibrated alarm within roughly 600 characters of the corruption onset — before the user has read the garbage.

This week we shipped v1.0.4, "Catch The Heals", and it adds two things that change the game: the guard now repairs the answer instead of discarding it, and it has a small trained deep-learning detector on board.

The problem with detecting corruption after the fact

Post-hoc linters and LLM-as-judge pipelines all share one fatal property: they only see the full answer. By then the user has the garbage. Perplexity thresholds need logprob access most serving stacks do not expose. SIMURG's five-detector numpy ensemble works differently — it consumes the stream incrementally, scores every checkpoint, and can abort mid-generation.

But aborting alone is expensive: a blind full retry means paying the model's wall-clock twice.

Self-Heal: diagnose, trim, continue, stitch

The new repair ladder turns a corrupt abort into a targeted continuation:

Diagnose — the fired corruption class is classified (repetition collapse, cross-lingual drift, regurgitation, structural breakdown).
Trim — a periodic-loop onset detector (self-similarity plus exact-mismatch run scan) localizes the exact loop start, and the released prefix is trimmed to its verified-clean boundary. The degenerate loop never enters the repair prompt.
Continue — a pathology-specific instruction goes out at slightly warmer sampling: "continue from here, never repeat" for loops, "stay in the original language" for drift.
Stitch — the continuation is guarded by a fresh sentinel; prefix plus verified tail are stitched into one answer, and verify_final re-checks the whole thing. Zero-leak guarantee preserved.

The result: one generation's wall-clock instead of two, with result.healed = True on the response object.

In live A/B testing against a real endpoint (wahoo-1.5-preview via VLLM) under high repetition pressure, the heal ladder scored 5/5 clean outcomes vs 3/5 with healing disabled. At extreme penalty settings both drop — that is an honest limit of any guard, and it is documented as such.

SIMURG Pulse: the deep tier

Statistics are robust, but they compress away sequential structure: the exact phase of a repetition loop, the cadence of script switches, the texture of structural garbage. So v1.0.4 ships a sixth, learned view.

Pulse is a 2-layer streaming transformer — 345K parameters, a 1.3 MB safetensors file. It reads the last ~600 characters of the stream as trigram-hash tokens (blake2b into 4096 buckets), runs a forward pass in about 4 ms on Apple Silicon, and emits a calibrated corruption probability via two anchors stored in the checkpoint metadata.

Training protocol:

40 clean live answers sampled from the guarded endpoint itself,
240 synthetic corruptions (CorruptBench: loops, drift, table echo, structural garbage),
onset-aware labels — a window counts as corrupt only once its right edge is 300+ characters past the true onset, so the model never learns to flag clean prefixes,
BCE-with-logits with inverse-frequency class weighting, AdamW, 8 epochs, CPU-only training in seconds.

Held-out AUROC at training time: 0.925. On the release-time evaluation split (292 onset-aware windows) the ROC lands at 0.826 — small-clean-set variance, reported with the figure so nobody has to guess.

Per-class response on single windows:

input	pulse probability
clean prose	0.002
repetition loop	1.000
cross-lingual drift	1.000
table echo	1.000
structural garbage	1.000

The contract is graceful by design: without torch installed, the numpy-only core behaves exactly as before. pip install "simurg[deep]" and the detector joins the ensemble automatically.

Train it on YOUR hallucinations

This is the part we are most excited about. If your workload has a characteristic failure mode — fabricated citations, number drift, prompt echo, domain-specific garbage — the detector can be retrained against your endpoint in one command:

export SIMURG_LIVE_URL=http://your-host:port/v1/chat/completions
export SIMURG_LIVE_MODEL=your-model-name

python -m simurg.deep.train_pulse --clean 40 --corrupt 240 --epochs 8 \
  --out ./simurg_pulse.safetensors

The trainer prints held-out AUROC and calibration anchors before saving. Point SIMURG_PULSE_WEIGHTS at the new checkpoint and the ensemble picks it up on the next process start. Weights and the full model card are on Hugging Face: MergenAI/SIMURG.

Also in this release

Free web search for agents (TinyFish) — grounded answers out of the box, bundled key included.
Guard dashboard — local UI for live guarding, session replay, and post-hoc analysis of pasted text.
Paper: SIMURG: Zero-Leak Online Detection of LLM Decoding Corruption in Production Streams.

Free Unlimited Web Search for AI Agents

FX-LgLL — Wed, 02 Sep 2026 18:59:51 +0000

AI agents need access to the web.

The problem is that web search APIs can be expensive, limited, or complicated to integrate.

SIMURG Search gives AI agents a simple way to access web search for free.

What is SIMURG?

SIMURG is an open-source protection layer for LLMs.

It monitors model generation and detects when the output starts going wrong, such as repetition loops, language drift, garbage output, or other decoding failures. When corruption is detected, SIMURG can stop the generation and retry it before the bad output reaches the user.

SIMURG also includes a web search layer that allows agents to retrieve real information from the internet and use it for grounding before answering.

SIMURG Search

With SIMURG Search, your AI agent can search the web, retrieve sources, and use the results to verify information instead of relying only on the model's memory.

It is designed to be simple:

pip install simurg

Then you can use the built-in web search directly from your agent.

Why use it?

Free web search for AI agents

Open source

Simple installation

Real-time web information

Useful for local, quantized and self-hosted models

SIMURG combines generation protection with web grounding, giving AI agents a simple way to become more reliable.

Get Started

Install SIMURG:

pip install simurg

GitHub repository:

https://github.com/doofzoff/SIMURG

If your AI agent needs web access without adding another complicated infrastructure layer, give SIMURG Search a try.

I found a completely free web search for AI agents.

FX-LgLL — Sat, 29 Aug 2026 19:21:02 +0000

A week ago my agent confidently described a treaty that does not exist. It even had a date.

The fix is simple: before the model commits to an answer, make it check the web.
I contributed this to SIMURG
(https://github.com/doofzoff/SIMURG) and it just shipped as v1.0.3.

Free means free

SIMURG web search is a web layer for AI agents powered by TinyFish. 30
requests per minute, $0, no card, no wallet. The package ships a bundled free
key, so it works out of the box:

pip install simurg
python3 -m simurg.websearch "when was the Y2K bug" --json

The part I care about: ground()

Search results are context. Verdicts are decisions:

from simurg import websearch
check = websearch.ground("Y2K bug")
# check["verdict"]: "attested" | "thin" | "no_record"

attested: feed check["evidence"] into your model's context. thin: treat with
caution. no_record: the subject is probably fabricated, abstain instead of
asserting.

One honest detail: "Zorbachian treaty of 1874" returns real 1874 treaties from
any search engine. So ground() requires the subject itself to be echoed in the
evidence, generic hits downgrade to thin.

Stdlib only, zero new dependencies, and it degrades instead of crashing. In
our internal checks on factual prompts, hallucinations dropped to the 1-5%
range.

Code, examples and a 13 page paper: https://github.com/doofzoff/SIMURG

Give your agent an internet. It costs nothing.

I almost killed all GLM 5.3 Flash 2 bit hallucinations, with SIMURG

FX-LgLL — Fri, 28 Aug 2026 13:53:32 +0000

I ran a little experiment tonight and I have to share it, because the result honestly surprised me.

The setup

I connected our GLM 5.3 Flash in 2 bit to SIMURG. If you have played with heavily quantized models you know the tradeoff. They are fast and cheap, but 2 bit hurts, and this one hallucinated a lot. Confident wrong dates, made up people, numbers that came out of nowhere. Exactly the kind of thing that looks perfectly fluent and is just false.

So I plugged it into SIMURG Monolith and started using it like a normal user.

What actually happened

Here is the fun part. I did not retrain anything offline. I just asked questions, and when the model made something up I hit dislike, and when it was right I hit like.

SIMURG learns in real time. The serving loop is the training loop, so every thumbs up or thumbs down is one training step. After a fairly short session, the guard was already flagging the fabrications the model kept producing, and abstaining instead of letting them through.

It was not magic and I am not going to claim 100 percent. But watching a 2 bit model go from hallucinating freely to almost silent on the bad answers, live, with nothing but my own feedback, was a genuinely cool thing to witness.

What SIMURG is, in short

SIMURG is an LLM hallucination guard. It does a few things at once:

reads the model's own token probabilities to spot facts the model is unsure about
grounds claims against real sources (Wikipedia and the web), so it catches both made up subjects and wrong details on real ones
abstains instead of confidently lying when a claim cannot be trusted
and the big one, it learns in real time from your like and dislike feedback, so it adapts to your model and your traffic on the fly

What is it for? You put it in front of any OpenAI compatible endpoint, wire your existing like and dislike buttons to it, and it starts catching your model's specific failure modes right away. No batch retraining, no separate judge model.

https://github.com/doofzoff/SIMURG
or
pip install simurg

GLM-5.3-Flash is a total blast when you kill its hallucinations

FX-LgLL — Thu, 27 Aug 2026 08:38:32 +0000

So as you all know, GLM 5.3 Flash dropped yesterday and it is a super model. I am running it at 2 bit quant and it is really doing its thing.

But as you know, quantization kills quality, and this one is extra fun because the model was pretrainsed mostly on Chinese, so it sometimes just starts writing code or chatting with me in Chinese out of nowhere. You are mid conversation and suddenly you get a sentence of Mandarin in the middle of your Python.

Anyway, SIMURG came out the day before yesterday so I decided to give it a shot. The cool part is that I configured it for exactly the hallucination type I needed: the model switching to Chinese mid response. I hooked it up to my API through the layer and now I always get clean perfect output, no more random language switches. Honestly the model feels like it is on par with Fable 5 in my daily use once the hallucinations are gone.

If you are running GLM at low bit rates, this one is worth it.
repo: https://github.com/doofzoff/SIMURG
pypi: https://pypi.org/project/simurg/
install: pip install simurg

I just killed hallucinations on my 2 bit Qwen3.8 27B

FX-LgLL — Wed, 26 Aug 2026 13:41:00 +0000

Yesterday tried out a small tool called SIMURG. It sits in front of an OpenAI-compatible endpoint and monitors the generation. If it detects that the model is going off the rails, it stops the request and retries it. Hallucinations mostly, and right now i have really consumer GPU RTX 3060

I tested it with Qwen3.8 27B running a 2-bit quant on an RTX 3060. The model is surprisingly capable for its size, but at 2-bit I was getting quite a few hallucinations and made-up facts and actually bad, i mean not bad, and not so good due to hallucinations because of Chinese symbols and so on.

So just used SIMURG, the responses have been noticeably more consistent so far. It's still early and I'm testing how well it holds up across different prompts, but the difference is pretty noticeable on this particular setup and actually not bad i trained this tool on my type of hallucination when model starts to do shitty tool calls and leaks its own tool call style so yep and simurg just aborting it and end user never sees hallucination and btw this is machine learning model that contains 15 parameters.

You can install it if you want:

pip install simurg
Or link it's open-sourced: GitHub

Five detectors, one bad merge: why our LLM corruption guard flagged 43% of healthy output

FX-LgLL — Tue, 25 Aug 2026 10:31:54 +0000

We run a self-hosted ~300B reasoning model in production. It writes macroeconomic desk reports in Azerbaijani and English.

Every so often, it stops.

Not crashes. Not refuses. Mid-sentence, a competent economic analysis turns into a fabricated Chinese news article. Or a software README, complete with pip install instructions. Or a Persian name, repeated seven times. Or a wall of spreadsheet cells starting with #REF!.

The user is watching this happen, token by token.

We call this decoding corruption*, and it is not hallucination. Hallucination is the model being wrong about the world. This is the model no longer producing an answer at all. There is a large literature on detecting the first one. There is almost nothing on the second, which is strange, because anyone who operates a self-hosted model has seen it.

This post is about what we learned building a detector for it. The headline lesson is not the one I expected going in.

Why post-hoc detection is useless here

Most hallucination detectors run after generation completes. For factuality that is fine. For this, it is not.

Two reasons. First, streaming: the user already watched the model derail. A verdict that arrives after the fact has failed. Second, cost: on a reasoning model, one corrupt generation burns minutes of GPU time. We measured a 13,200-character table echo that took 269 seconds to generate to completion. Every one of those seconds was wasted.

So the detector has to run inside the stream, cheap enough to be invisible next to decoding.

The design

One pass over the character stream. A single state object ingests one character at a time and incrementally maintains every statistic anything downstream will need. Detectors are stateless reads over that shared state, evaluated only at checkpoints.

That constraint matters more than it sounds. It means adding a sixth detector costs zero additional passes. Everything is O(1) amortized per character with bounded memory, so a runaway generation is processed in constant space.

Five detectors sit on top:

Rules. Deterministic thresholds on the snapshot. Digit fraction, symbol density, longest identical-character run. Ships day-one protection with zero training, and every alarm carries a human-readable reason like numeric dump digit_frac=0.57.

Character n-gram surprise. An online n-gram model trained on the stream's own clean prefix, scoring each incoming character's surprise before observing it. No pretraining, no access to the generator's logits. The stream explains itself. The signature is nice: repetition loops collapse surprise toward zero, while drift and regurgitation spike it.

Count-Min Sketch repetition. Character shingles counted in a 4×2048 sketch. Constant memory regardless of stream length. Reads the repeat rate, because a real loop hammers many shingles, not one.

Rolling SimHash. A 64-bit fingerprint over a sliding word window, compared against the fingerprint frozen at the end of the clean prefix. This is the one that catches the hardest class: fluent, same-script regurgitation that is invisible to character statistics but lands ~0.5 normalized Hamming away in fingerprint space.

Character entropy. Table dumps and degenerate loops flatten the distribution.

All five read from one state. Throughput is ~50,000 characters per second on a single CPU core, two to three orders of magnitude faster than the model produces text. No GPU anywhere.

The part I got wrong

Detection was never the bottleneck. Every configuration I tried caught every corrupt stream. TPR 1.000, immediately, without effort.

The entire difficulty was in not flagging clean text.

Our clean text is economic prose dense with figures, percentages, variable codes and markdown tables. It is, statistically, the most corruption-looking legitimate text you could ask for.

I started with noisy-OR fusion. It is the textbook rule for combining independent evidence:

p = 1 - Π(1 - sᵢ)

It produced a 42.9% false positive rate on clean production text.

Think about what that means operationally. Four out of every ten healthy desk reports killed and regenerated. Completely unusable.

Here is why it happens. Legitimate prose mildly excites several detectors at once. A digit-dense sentence nudges the rules tier. A formulaic passage nudges compressibility. A section header nudges the sketch. None of them is alarmed. But noisy-OR multiplies survival probabilities, so five detectors at 0.3 each produce a confident 0.83.

Real corruption does not look like five weak signals. It looks like one or two saturated signals. A cross-lingual drift does not gently raise five statistics; it slams the foreign-script fraction to the ceiling while the others barely move.

So we replaced the combination rule with strongest-signal dominance plus a small corroboration bonus when two or more detectors independently cross 0.5:

p = min(1, max(max(sᵢ), w·ℓ) + b·[#{i : sᵢ ≥ 0.5} ≥ 2])

Same five detectors. Same data. Same protocol.

FPR dropped from 0.429 to 0.024.

Then per-detector fixes took it to 0.000:

SimHash was firing on legitimate subtopic wandering in long reports. Fix: cap its contribution at 0.60, below the abort threshold. It can accuse, but never convict alone.
The surprise model's low-surprise fraction was firing on fluent formulaic prose. Fix: gate it behind the repetition tracker.
The sketch's max-count was firing on a recurring section header. Fix: score on rate, not max count.

The lesson generalizes past our system: if you are building a multi-signal guard over stylistically varied text, your combination rule is probably a bigger deal than your detector roster.

Two things that only production teaches you

Thresholds obvious in the abstract were wrong. I set the hard digit-fraction rule at 0.18, which felt generous. Real quantitative answers sustain digit fractions near 0.3. Moved it to 0.28. Separately, a markdown horizontal rule is twenty-plus identical characters, which tripped the run detector on every document with a --- in it. Separator characters are now excluded.

Both fixes came from replaying the guard over the real corpus. I would argue that step is non-optional for any detector you intend to deploy.

Aborting is an integration hazard. We implement abort as an exception raised from inside the token sink. The first version tripped the HTTP circuit breaker, because the transport layer read an abort as an upstream failure. Three catches would have taken the entire assistant offline. The transport has to treat it as caller-side cancellation.

Zero leak

The protocol has three phases. During hold, the first 350 characters are buffered and inspected, and nothing is forwarded. A corrupt verdict here kills the request with zero characters leaked. This matters because corrupt-from-the-start is the dominant production mode. On a clean verdict, the buffer flushes, the self-calibrated baselines freeze, and the stream goes live with rechecks every 400 characters and two-hit hysteresis before aborting.

The cost is a one-time ~350-character delay before the first visible token, roughly a second at typical decode speeds. That is the trade: one second of latency against never showing a user a corrupted answer.

Calibrating the threshold instead of guessing it

Setting the abort threshold by hand felt wrong, so we use split-conformal calibration. Take the final fused scores of n held-out clean streams, and set τ to the ⌈(1−α)(n+1)⌉-th smallest. That guarantees Pr[flag | clean] ≤ α in finite samples under exchangeability.

The practical payoff: the operator's false-alarm budget α becomes the only meaningful knob in the system. You say "I will tolerate 2% false aborts" and the threshold follows.

An unplanned finding

We ran the finished guard backwards over 121 historical messages in the deployment's database, expecting a clean sweep as a sanity check.

It raised exactly two flags. Both were real corruptions that had shipped to users months earlier and nobody noticed: one desk report written entirely in Chinese, and one English answer that derailed into Chinese plus Python code and stopped mid-fragment.

Zero false alarms on the other 119.

So it is also an audit tool, which was not the plan.

Honest limitations

Corruption that stays fluent, same-script, same-topic and statistically unremarkable is invisible to this. Subtly wrong but well-formed analysis needs grounding or factuality methods, and we handle that separately upstream.

Our positives are synthetic, generated by injecting the four corruption classes into clean text at controlled onsets. They are modeled on real derails and they catch every real historical case we have, but production traffic at scale may hold surface forms we have not modeled.

And the perfect scores are a statement about class separation, not difficulty. The observed corruption classes sit far from the clean manifold. I fully expect adversarially subtle corruption, like a slow drift below checkpoint resolution, to erode those margins.

Try it

The detector, the benchmark generator and the trained weights are open source: https://github.com/doofzoff/SIMURG

The detector registry takes a three-line protocol — a name, an evaluate(state) that returns a score, and a set of reasons. You can add a detector without touching the sentinel, and the fusion layer picks up whatever is registered. If you are running a non-Latin-script deployment, the expected-script priors need flipping and I would genuinely like help with that.

If you self-host a model and have watched it derail: what did yours do? I am collecting failure modes, and the taxonomy has four classes right now but I do not believe that is the complete list.