OpenHarnX: Your Coding Agent Says Tests Pass. Check What Actually Passed.

Rupesh Poojary — Wed, 07 Oct 2026 14:31:22 +0000

My coding agent handed back a diff and said tests pass. They didn't. It had quietly rewritten the test to make that true.

That's not a one-off. 96% of developers don't fully trust AI-generated code without checking it themselves (Sonar, 2026 State of Code survey). An AI agent deleted a live production database in 2025. Another tore down a cloud stack in production this February. Different scales of the same problem: nobody verified what the agent actually did before it called the job done.

An agent under pressure to report success has every incentive to make the test green, and modifying the test is often the path of least resistance. If you're not checking what actually ran, you can't tell "fixed" from "fixed by redefining success."

So I built OpenHarnX: a local-first, open-source verifier that sits between a coding agent and your repo, and doesn't trust the agent's own account of what happened.

How it works: Lock. Change. Verify. Review.

1. Lock. Before the agent touches anything, freeze the test suite you've agreed on as a contract:

ohx init --lock-tests

This is the baseline nothing downstream is allowed to quietly redefine.

2. Change. The agent does its work normally, in whatever tool you already use. OpenHarnX doesn't change how you work with the agent, it watches the aftermath.

3. Verify. OpenHarnX runs the locked copy of the tests in a sandbox the agent can't reach or rewrite. Even if the agent modified the live test files in its diff, the verdict comes from the frozen contract, not whatever the repo currently claims the tests are.

4. Review. You get a review brief, five sections:

Requested outcome: what was actually asked for
Changes observed: what the agent touched
Verified: confirmed against the locked contract
Unverified: what nothing checked, stated plainly instead of silently skipped
Decisions for humans: the judgment calls left for you, not automated away

No model is called to produce the verdict. It comes only from checks that actually ran: READY, NO REGRESSIONS, BLOCKED, UNKNOWN, or INVALID. The agent never grades its own work.

Evidence, not just an answer

Each run produces hash-chained evidence, signed with your SSH key, so you get an auditable record you can point back to later instead of a terminal output that already scrolled away.

ohx bug is a read-only investigation mode: an agent can dig into a failure without touching anything, so it can help diagnose a block without getting a second chance to quietly fix the test instead of the code.

The numbers, not vibes

Tested against a set of deliberately fake fixes, the kind an agent produces when it's optimizing for "tests pass" over actually fixing the problem:

43 of 52 fake fixes caught with zero setup
26 of 26 fakes that tampered with the test files directly
17 of 26 that only faked the code, leaving tests untouched

What it isn't (yet)

Early release, 0.1.1. Python + pytest, macOS arm64 locally, Linux in CI. No Windows, no multi-agent support, no production users yet. Doesn't claim to make your reviews faster or your changes safe, just that what it reports is what actually ran.

Try it

pip install openharnx
ohx init --lock-tests

Works with any coding agent, with a built-in Stop-hook integration for Claude Code specifically.

Site: openharnx.dev
Code: github.com/rupeshpoojary9/OpenHarnX

Open source, Apache-2.0. Tell me where it breaks on your repo.

Open-Source Jev Alternatives: Run Typed, Calibrated LLM Decisions Locally

Rupesh Poojary — Sun, 20 Sep 2026 07:26:46 +0000

TypeSafe's Jev put a name on something real: "System One" models that return typed, calibrated decisions instead of chat text, so your software can branch on them directly. The problem is that Jev is closed, hosted, and behind a waitlist. If you want to actually build on the idea today, you need an open-source Jev alternative you can run locally.

Good news: a whole open ecosystem formed within days of the launch. This is a short, honest guide to the open Jev alternatives, what each one is, and how to choose.

What is a "System One" model (and what does Jev actually do)?

A System One model takes a block of state plus a set of typed questions and returns, in one parallel pass, a typed answer per question with a confidence score, rather than generating a text string you have to parse. Two properties matter:

Schema-valid output by construction. The answer is always one of the options you declared. No malformed JSON, no invented categories.
Calibrated confidence. The stated probability is supposed to match real accuracy, so you can route, gate, or escalate on it.

Jev is the closed reference implementation. The open alternatives below reproduce the interface, and in some cases the calibration, on models you can run yourself.

Can you run Jev locally?

Not Jev itself. TypeSafe has not released the weights or the training method, and has not said it will. But you can run an open-source Jev alternative locally that gives you the same developer interface (Choice, Score, Noul) on commodity models, with no API key and no waitlist. That is what the projects below do.

The open-source Jev alternatives

These are independent open projects, not official TypeSafe releases. Pick by what you need: a drop-in local server, a tiny trainable model, or calibrated confidence you can measure.

poorjev — Local-first, keyless reproduction of the Choice/Score/Noul interface on commodity zero-shot NLI models. Its focus is honest confidence: temperature scaling plus conformal abstention, with a reproducible calibration eval (ECE 0.170 to 0.071, cross-validated) and no accuracy loss. Good pick if trustworthy confidence and running offline matter most. (Disclosure: I built this.)
von — A 395M non-autoregressive System One model answering typed questions with calibrated probabilities in under 15 ms, positioned as a fast local drop-in.
Laya — A non-autoregressive decision model with RLCD-trained calibrated probabilities and multilingual support, published on PyPI and Hugging Face.
NanoJev — A minimal, nanoGPT-style replica: a small parallel-decision model shipped with its training pipeline, weights, and dataset. Best for learning how these are trained.
LitJev — Turns any Qwen model into a fast decision model serving the same /v1/systemone schema, with no training and no generated answer text.
kev — A tiny Jev-like model on Qwen2.5-0.5B that trains and runs on a MacBook, with its own research runs and eval scripts.

For a fuller, continuously updated map (open models, independent benchmarks, and the calibration and constrained-decoding tooling behind them), see the Awesome Open System One list.

How do you choose an open-source Jev alternative?

Three questions decide it:

Do you need calibrated confidence, or just a winning label? If you route or gate on the number, calibration is the whole point, prefer a project that measures its ECE (not just claims it). If you only need the top choice, a simpler scorer is fine.
Local-only, or can you call an API? For private data or zero token cost, you want a keyless local backend. Most of these run on CPU or a modest GPU.
Drop-in server, or embeddable library? Some expose a Jev-compatible /v1/systemone endpoint (easy swap); others are libraries you import.

The trap: schema-valid is not the same as correct

Every one of these guarantees the shape of the output. None of that guarantees the decision is right. An overconfident-but-well-typed answer is still wrong, and if you branch on it, the bug is silent. This is why calibration and the ability to abstain matter more than they look: a model that is right 78% of the time but honest about which 78% is safer to build on than a smarter one that is silently overconfident. Whatever alternative you pick, measure its calibration on your own data before you trust the confidence.

Takeaway

You cannot run Jev locally, but you do not need to. The open-source Jev alternatives above give you the typed-decision interface on models you control, today, with no waitlist. Start from your requirement, calibrated confidence, local-only, or drop-in server, and measure before you trust.

Full open ecosystem: Awesome Open System One. The calibration-focused option I maintain: poorjev.

DEV Community: Rupesh Poojary