We built an AI system to verify AI-generated code.
Then we discovered that our own AI reviewers weren't as independent as we thought.
That turned out to be one of the most valuable lessons from building Katse, our project for Tieto's Earned Trust in AI Code challenge at Junction 2026.
The question we were trying to answer sounds simple:
If AI writes the code, who makes sure it's correct?
And more importantly, how do we trust the answer?
What I Built
AI coding tools can generate hundreds or thousands of lines in minutes. But generating code and verifying it are two very different problems.
As AI-generated changes grow, reviewing every line becomes increasingly expensive. Asking another language model whether the code looks correct doesn't necessarily solve anything. It's still one AI asking another AI for an opinion.
We wanted something better.
Our solution, Katse, takes inspiration from scientific reproducibility.
Instead of simply reading an AI-generated pull request, Katse sends independent AI agents back to the original requirements and asks them to reconstruct the solution without seeing the submitted implementation.
It then compares the results, executes tests, checks for risks, and produces an evidence-based report highlighting what still requires human judgment.
The principle behind it is straightforward:
Don't trust the answer. Reproduce the reasoning and test the result.
What did we achieve?
We tested Katse against four real sample changes provided for Tieto's challenge.
One particularly interesting case involved 1,392 changed lines across 237 files.
Katse narrowed the change to 32 lines requiring targeted human attention, along with an unresolved question.
That isn't a claim that the remaining code is automatically safe. It means the pipeline produced evidence that helped focus the review.
On another change, Katse reproduced 174 passing regression tests, caught three of four seeded defects, and identified behavior that the original requirement hadn't explicitly requested.
And when the evidence was insufficient, Katse could recommend sending the change back instead of approving it.
Because the goal was never to replace the engineer.
It was to help engineers know exactly where their attention matters.
Demo
We built a working prototype that visualizes code dependencies, tracks verification evidence, and produces a review report before a human decides whether to accept the change.
The system distinguishes between what was checked, what remains uncertain, and what wasn't checked at all.
Katse itself cannot approve or merge a pull request. That decision belongs to a named human reviewer.
Watch our 2-minute, 44-second demo.
The complete project, documentation, tests, and evidence reports are available on GitHub.
Partner Technologies
Our architecture combines deterministic analysis with independent AI verification.
We used Gemma 4 locally to anonymize sensitive information before external model calls.
Gemini and NVIDIA Nemotron powered the independent reconstruction process, allowing us to compare implementations generated by different model families.
A deterministic analysis layer uses call graphs, PageRank, static checks, and executable tests to assess changes without relying entirely on model judgments.
We also integrated Entire to recover the original AI development context from Git checkpoints when available. This is important because the original prompt can tell us much more about the developer's intent than the final diff alone.
For Tieto's sample changes, the original prompts weren't available, so Katse explicitly fell back to the issue requirements rather than inventing missing context.
The most important design decision was keeping model confidence separate from actual evidence.
An AI claiming something works is not the same as an executed test demonstrating it.
Hackathon Experience
I worked alongside Meulen, Fabian, and Lucia, contributing mainly to the core concept and the design of Katse's verification pipeline.
One of the biggest challenges was ensuring the verification system itself could be trusted.
During our first full run, agents that were supposed to independently reconstruct the code were operating inside the original repository.
They could potentially access the very implementation they were meant to reproduce independently.
The problem came down to process isolation: the agents' working directories were configured, but the PWD environment variable still pointed elsewhere.
We addressed it by enforcing sandbox isolation, correcting the execution environment, and checking that the original repository remained unchanged.
Another issue appeared when agents exhausted their usage quotas halfway through a reconstruction. Initially, incomplete attempts were treated as valid results. We added checks so that unfinished or failed rebuilds could no longer be mistaken for meaningful evidence.
These weren't theoretical failure modes. They were problems we encountered while building the tool.
And they reinforced the central idea behind Katse:
A verification system must be able to question its own evidence.
Our prototype still has limitations. Four sample changes aren't enough to establish production reliability, and independent reconstruction needs more benchmarking against conventional code review.
But that's exactly why we wanted the system to expose uncertainty rather than hide it.
Finishing 2nd in Tieto's Earned Trust in AI Code challenge at Junction 2026 was a great recognition of our team's work.
Huge thanks to Meulen, Fabian, Lucia, Tieto, and Junction for the experience.
What's next?
AI will keep getting faster at writing software.
The difficult part is building the tools, processes, and evidence needed to trust what it produces.
The future of AI-assisted development isn't just generating more code. It's making that code accountable.
Project: Katse on GitHub
Demo: Watch on YouTube
Tags: #mlhacks #devchallenge #hackathon #ai
Top comments (0)