I ran a little experiment tonight and I have to share it, because the result honestly surprised me.
The setup
I connected our GLM 5.3 Flash in 2 bit to SIMURG. If you have played with heavily quantized models you know the tradeoff. They are fast and cheap, but 2 bit hurts, and this one hallucinated a lot. Confident wrong dates, made up people, numbers that came out of nowhere. Exactly the kind of thing that looks perfectly fluent and is just false.
So I plugged it into SIMURG Monolith and started using it like a normal user.
What actually happened
Here is the fun part. I did not retrain anything offline. I just asked questions, and when the model made something up I hit dislike, and when it was right I hit like.
SIMURG learns in real time. The serving loop is the training loop, so every thumbs up or thumbs down is one training step. After a fairly short session, the guard was already flagging the fabrications the model kept producing, and abstaining instead of letting them through.
It was not magic and I am not going to claim 100 percent. But watching a 2 bit model go from hallucinating freely to almost silent on the bad answers, live, with nothing but my own feedback, was a genuinely cool thing to witness.
What SIMURG is, in short
SIMURG is an LLM hallucination guard. It does a few things at once:
- reads the model's own token probabilities to spot facts the model is unsure about
- grounds claims against real sources (Wikipedia and the web), so it catches both made up subjects and wrong details on real ones
- abstains instead of confidently lying when a claim cannot be trusted
- and the big one, it learns in real time from your like and dislike feedback, so it adapts to your model and your traffic on the fly
What is it for? You put it in front of any OpenAI compatible endpoint, wire your existing like and dislike buttons to it, and it starts catching your model's specific failure modes right away. No batch retraining, no separate judge model.
https://github.com/doofzoff/SIMURG
or
pip install simurg
Top comments (0)