This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

ProofLoop: An AI Quality Checker Built for Data Annotators

What I Built

ProofLoop is an AI-powered, guideline-first quality checker built for data annotators, caption writers, and QA reviewers.

In annotation work, even a small mistake can make an answer invalid: inventing an action, missing an important step, adding unsupported details, or giving feedback that doesn't follow the task guidelines. Manually checking every response takes time, especially when working with strict annotation policies.

I built ProofLoop to make that process easier.

Users paste their actual task guidelines and an answer. ProofLoop reviews the answer against those rules, identifies specific mistakes, explains what needs to be corrected, and generates a suggested final answer that can be copied directly into an annotation workflow.

It also supports comparison and feedback reviews involving two outputs.

The goal isn't to replace human reviewers. It's to give annotators a practical second pair of eyes before they submit their work.

A little personality goes a long way

I also built Loop, an interactive space-themed robot mascot. It floats, reacts, and moves around the workspace, giving a quality-assurance tool a more playful identity without getting in the way of the actual work.

ProofLoop is a working prototype. I've had a few friends try it, and their initial feedback was encouraging. I haven't acquired paying customers yet, so validating it with real annotation workers is my next step.

Who could pay for it?

Independent annotators, freelance caption writers, QA reviewers, and small data-labeling teams could benefit from catching guideline violations earlier.

A potential business model is a free individual tier with paid options for higher usage or team workflows. Whether users would actually pay—and how much time the tool saves—still needs to be validated.

Demo

Try the live application: Open ProofLoop

I'll include screenshots of the working interface and an example review, showing the input workflow, feedback, suggested correction, and Loop mascot.

ProofLoop's interface featuring Loop, the interactive space-themed mascot, alongside the annotation review workspace.

ProofLoop evaluates an annotator's caption against supplied evidence and guidelines, identifies omissions, and suggests a more complete answer.

ProofLoop turns guideline-based feedback into a reusable correction, with a one-click copy action and downloadable output.

Code

GitHub repository: View ProofLoop on GitHub

ProofLoop is organized into separate modules for the interface, prompt construction, AI client, evaluation logic, and mascot. It includes an offline test suite for input validation and response handling.

How I Built It

I used Python and Streamlit to build the application, with custom HTML, CSS, JavaScript, and SVG for the interface and interactive mascot.

The AI integration uses the Backboard SDK with OpenRouter model routing. The deployment is configured to use Cohere's North Mini Code model (cohere/north-mini-code:free), an open-weight model. The hosted
application runs on Render and the code is maintained on GitHub.

The evaluation workflow is designed to compare answers with user-supplied guidelines and return structured feedback. The interface includes theme switching, a copy-ready correction, downloadable results, and Loop's custom animations.

This is a hosted prototype, not a fully local or offline AI system. The current implementation and model routing still need continued testing as the project evolves.

Why Does Open Innovation Matter?

Annotation quality assurance is a focused problem, and developers should have the flexibility to experiment with different models and approaches instead of being locked into one provider.

Using an open-weight model gives this project room to explore model choices, compare performance, and potentially move inference to infrastructure under my own control in the future.

For now, ProofLoop uses hosted inference. I haven't claimed local processing or complete data privacy, and the model's free endpoint may have rate limits or availability constraints.

The open-weight approach gives me a starting point for building a tool that can evolve with user feedback, different annotation workflows, and future deployment options.

My First User

I shared the prototype with a few friends and asked them to try it. Their initial reaction was positive, which encouraged me to keep improving the experience.

These were informal early tests, not paid customers or a formal usability study. I haven't validated willingness to pay yet. My next goal is to put ProofLoop in front of working annotators, learn where its feedback helps, and identify what needs improvement.

Prize Categories

-Best Use of Render — ProofLoop is deployed as a live web application on Render.
-Best Use of Backboard — The application uses the Backboard SDK in its AI review workflow.

I'm also submitting ProofLoop for the overall challenge.

#devchallenge #hf26challenge #hacktoberfest