Most ML monitoring tools are very good at telling you something changed. None of them tell you whether you should care... That gap is why the engineer on call gets twenty drift alerts a week, learns that most are noise, and stops reading them. Then the one that mattered goes past unread!.
At Hacktoberfest Hack Day Coimbatore 2026, we built Arbiter — a reasoning layer for production machine learning, powered by Gemma 4..
Where it started:
One of us maintains driftguard-ai-sdk, an open-source Python SDK on PyPI that detects model drift in real time. It works. But running it surfaced a line that is, in hindsight, an admission:
drift_threshold=0.15
Where does it come from? Nobody knows... Every monitoring tool has a constant like this, and every one is a placeholder for judgment nobody automated. A threshold cannot tell a seasonal spike from a broken data pipeline, because both produce identical numbers.
So, We built the layer it was missing.
What Arbiter does??
It answers the three questions a number cannot:
Why did this happen, broken pipeline, genuine population shift, or model decay? Does it matter, or is it the month-end batch that spikes every cycle?
What should we do? retrain, roll back, investigate, or nothing?
Arbiter ships verdicts, not metrics. Each carries an action, a confidence score, and a written explanation naming the specific features and magnitudes behind it. Low confidence escalates to a human instead of acting. On a replay of [N] drift events, it returned [X] actionable verdicts and suppressed [Y] as benign ; a [Z]% reduction in alert volume.The integration is one environment variable. Arbiter implements the SDK's existing wire protocol, so it's a drop-in replacement backend. No code change, no fork, no second SDK to learn.
export DRIFTGUARD_API_URL=https://arbiter-api.example.com
Why open weights, not an API
Production ML telemetry often cannot leave the network it was generated in. Feature distributions are business data, and in regulated industries they are not going to a closed API. Gemma 4's open weights mean the same model we demo can be self-hosted inside a customer's VPC. A closed model would have made Arbiter undeployable for exactly the companies that have this problem worst.
The idea underneath:
Detecting drift is a capability. Deciding what to do about it, and standing behind that decision, is an outcome.
The industry is shifting from the first to the second. SkySync, this event's sponsor, builds their whole model on being accountable for returns rather than just shipping AI. Arbiter applies that one layer down — which is why every verdict carries a confidence score and an explanation. A recommendation you cannot audit is not a decision anyone can be accountable for. Built in four hours. Open source, MIT licensed.
Demo: Watch here
GitHub: Click here
Thanks to INIT Club, iDEA Club and Major League Hacking for running it!!
Top comments (0)