This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 2 — Side Hustle.
"Great work on the website so far! Could we also add a reserved area where our customers can log in?"
If you freelance, you know that message. It is friendly, it is reasonable, and it is often work you never quoted for. The polite thing is to say yes. The expensive thing is also to say yes.
A freelance web developer I tested this with put it better than I can: out of five projects, three have at least one request outside the agreement. The big ones you notice. The small ones — "can you also write the descriptions?", "can you move that button?" — "add up, and in the end they're hours."
What I Built
ScopeLeak answers one question, before you reply to a client: is this covered by what we agreed?
You paste (or upload) your agreement and the client's new message. ScopeLeak splits the message into separate requests and labels each one:
- Included — covered by your agreement.
- Extra — explicitly excluded, or beyond a limit you set (pages, revision rounds, hours, support period).
- Needs clarification — your agreement doesn't settle it.
Every verdict comes with the exact clauses it relied on and the exact words of the client's request. Then it drafts a reply: you type in your hours, it picks up the hourly rate when a cited clause states one, and you edit the text before sending. It never sends anything, and it never invents hours.
It runs entirely on my laptop, with an open-weight model (Gemma 4 E4B via Ollama). The client's contract never leaves the machine — the page doesn't even load a web font.
Know before you say yes.
Demo
The run in the video is real, recorded end to end by a browser script; only the wait is cut — 7 minutes on my laptop's CPU while the same CPU was also running a headless browser and recording the screen (4–5 minutes without the recording). The agreement and message are synthetic and live in the repo's demo/ folder, so you can reproduce it.
Code
ScopeLeak
Know before you say yes.
A client sends a new request. Is it covered by what you agreed, or is it extra
work you are about to do for free? ScopeLeak reads your agreement and the
client's message and tells you — included, extra or ambiguous — citing the
agreement lines and the words of the request that justify the verdict.
It runs on open-weight models (Gemma) on your own machine: client contracts never leave your laptop.
Work in progress for the DEV Hacktoberfest Open-Source AI Challenge, Week 2 (Side Hustle). See SCOPE.md for what is being built this week and BACKLOG.md for what is deliberately not.
Demo (61 s): https://youtu.be/K8b9WwqRlU8 — reproduce it with the files in demo/.
How it works
- The agreement is split into numbered lines.
- The model identifies each request in the message, cites the relevant lines and an exact quote from the…
How I Built It
The stack is deliberately small: FastAPI, one HTML page with vanilla JS, PyMuPDF for PDFs, and any OpenAI-compatible model server (Ollama by default). No database, no Docker, no agents. The interesting part was not the stack. It was making a small model's answers trustworthy.
The rule: no verdict without evidence
A model that says "this is extra" is useless if it can't say why, and dangerous if it makes the why up. So the model has to cite its sources, and plain code checks every citation:
agreement ──► numbered clauses (L1, L2, …)
message ──► model: requests + reasoning + cited lines + verdict
│
▼
reference integrity validation (deterministic)
• every cited line must exist
• the request quote must appear in the message
• "extra" needs a line that EXCLUDES or LIMITS the request
• "included" needs a line that COVERS it, and no excluding line
• otherwise → "needs clarification"
That validator can prove a reference is real. It cannot prove the reference is relevant — that part stays probabilistic, which is why you always see the clauses next to the verdict.
When the guardrail made the AI worse
I built an evaluation set first: two synthetic agreements (a website, a branding job) and 19 client requests, each annotated included, extra or ambiguous, plus a keyword baseline — the regex a weekend script would use (also|add|another|extra|more…).
Then I ran Gemma 3 4B through four versions of the pipeline. Here is what happened, on the same 19 requests:
| Version | What changed | Model's own accuracy | After validation | Downgraded |
|---|---|---|---|---|
| keyword baseline | — | 58% | 58% | — |
| v1 | model copies quotes from the agreement | 74% | 53% | 7 |
| v2 | model cites numbered lines instead | 79% | 26% | 14 |
| v2.2 | (after fixing v2) reasoning before citations | 74% | 42% | 8 |
| v3 | separate "covering" and "excluding" citations | 68% | 63% | 2 |
The model's reasoning was mostly right from the start. My guardrails kept throwing it away, and each failure taught me something different:
- v1 — the model can't copy. It understood the contract but left quote lists empty, or glued a section heading onto a sentence, so exact matching rejected it.
-
v2 — my schema fought my prompt. The prompt showed clauses as
L18; the JSON schema said integers only. Under constrained decoding the model could not write the "L", and the shortest valid answer was an empty list — in all 19 cases. Its reasoning, meanwhile, was busy citing L18. -
v2.1/v2.2 — the model cites what covers, never what excludes. Asked for "lines that justify the verdict", a 4B model lists lines that include a request. An extra request has none, so it cited nothing — in v2.2, every single downgrade was an
extrawith an empty list. -
v3 — ask for the evidence you mean. Two lists,
covering_linesandexcluding_lines, and the validator requires the kind that matches the verdict. That also made a key rule enforceable by code instead of by prompt: something your agreement simply doesn't mention can never be "extra". Downgrades fell from 8 to 2, and the result was the first one that beat the baseline. Every remaining error was "needs clarification" — the model failing safe.
(Somewhere in there I also learned to run at temperature 0 with a fixed seed: an 11-point drop I was busy explaining turned out to be noise. The v1 and v2 rows above were measured at temperature 0.1; everything from v2.2 on, and everything below, at 0.)
A test set I wasn't allowed to tune on
Nineteen requests I had stared at for a day prove very little. So I froze v3, wrote two new scenarios (an IT support retainer and a social media contract, 20 requests) without labels, and had them labelled independently by a human and by ChatGPT before any model saw them. They agreed on 20 of 20 — which also tells you these cases have clean boundaries; real messages are messier.
| On the held-out set | Accuracy | Extra found (recall) | Extra precision | Called "included" but extra | Time per request |
|---|---|---|---|---|---|
| keyword baseline | 30% | 17% | 50% | 10 | 0 s |
| Gemma 3 4B | 55% | 42% | 83% | 3 | 31 s |
| Gemma 4 E4B | 85% | 92% | 85% | 1 | 170 s |
The keyword baseline collapsed outside the data it happened to fit. Gemma 3 4B missed every quantitative limit — six support hours already used, twenty posts instead of twelve, an ad budget of 500 instead of 300 — and missed them the expensive way, calling them included. Gemma 4 E4B caught all three, with the same prompt and the same code: I only changed the model name.
Its errors are worth reading too. It called a newsletter "extra" because the contract excludes "platforms other than Instagram" — exactly the doubt the second annotator raised while labelling. And it called a Saturday-night emergency "included" while citing the very clause that bills emergencies separately. That one came back to bite me.
Then I tried contracts I didn't write
A threat-intelligence SaaS agreement: the right clause in five of five tickets. The two bugs were mine, in the reply template — it quoted €50/hour right next to a cited clause saying "billed at EUR 120/hour", and offered "an additional quote" in answer to an SLA-credit claim. Now a rate stated in a cited clause wins, and with no hours entered the reply just points to the clause.
A web design agreement pasted from a PDF arrived as a single 2,000-character line, so the whole contract became "L1" and the model had nothing precise to cite. Long lines are now split into clauses. On the rerun, four of five tickets were right with the exact clause.
The fifth was the expensive kind of wrong. "Can we add a Careers page?" came back included, citing "add new pages… that won't be a problem" as covering — and "we will provide a separate estimate… billed at £60 per hour" as excluding. Its own reasoning said "requiring a separate estimate". The prompt already defined included as "a covering line applies and no excluding line does"; now the code enforces it (v3.1). A model that cites an exclusion and still says included gets "needs clarification", never "do it for free". Re-scoring the stored runs, it changes nothing on the development set and exactly one held-out case — the Saturday emergency.
Why Open Innovation Matters
Privacy is the product. An agreement holds a client's name, your rates, deadlines and penalties. My first user put it simply: contracts "have client data, rates, commercial information — I don't want to send them to external services." With an open-weight model on your own machine, that isn't a policy promise; there is no external service to send it to.
Swapping the model was a config change. ScopeLeak talks to any OpenAI-compatible endpoint. Moving from Gemma 3 4B to Gemma 4 E4B took one line in config.json and raised held-out accuracy from 55% to 85%. Next year's open model is the same one-line change — on my laptop, on a workstation on the LAN, or on a GPU server I control.
The cost per message is zero. That matters for the price a freelancer will accept. My first user would pay 5–10 € a month or 50–80 € once — "not 30 € a month". A tool that pays an API bill on every client email can't live at that price; one that runs on the user's hardware can.
Where closed would have won, honestly. A frontier API model would very likely be more accurate and answer in seconds rather than three minutes. For this job I'd still pick the open route: the trust problem was never the model's IQ. It was evidence, validation and privacy — and those I can only fully control when the whole pipeline is mine.
My First User
A freelance web developer — eight years freelancing, five full-time — gave me his agreement (his adaptation of the Contract Killer template, with clauses he says he really uses) and five tickets modelled on requests he really gets. I ran them through ScopeLeak and he reviewed every verdict and the draft reply. For transparency: he wrote the ticket texts with an AI's help, and polished his answers with an AI assistant; the opinions are his. He asked to stay anonymous. His answers were in Italian; these are my translations.
- He agreed with four of five verdicts. The fifth — a new Careers page — ScopeLeak marked "needs clarification", while he'd call it billable extra: "what needs clarifying is the price, not whether it's included." That's v3.1 being conservative on purpose.
- It caught two things he'd have given away: taking down a work-in-progress sketch his contract lets him show in his portfolio, and fixing Internet Explorer 11 "for free, to avoid an argument". Also the word "once" in the upload clause, "easy to forget".
- Would he pay? "Yes, if it's cheap": 5–10 €/month, 2–3 € per analysis, or 50–80 € one-off.
- His criticism: "'potentially extra' is too vague — I want 'extra, ~X hours, £Y'." The draft needed "no flat no's". And: "be careful not to sound like a lawyer."
I changed the first three that afternoon: a direct Extra badge, a warmer reply draft, a "not legal advice" line. The fourth — hour estimates generated by the AI — I deliberately left in the backlog. Invented estimates are the easiest thing for a model to get wrong, and the one my user would trust least once burned. It's the first thing on the list for v0.2.
The Part Where ChatGPT Scope-Crept My Project
I used Claude Code to build ScopeLeak and ChatGPT as a second reviewer. Within a few messages, the reviewer had proposed a model manager UI, an in-app benchmark, a scope timeline, LM Studio and vLLM adapters, a multi-client "Studio" edition and PDF export. Good ideas, every one.
So the repo has a SCOPE.md that was frozen on day one, and a BACKLOG.md where every later idea went — politely, and later. That's the whole point of ScopeLeak: you can say yes to a good idea without doing it for free, today.
Limits
- 39 annotated requests is a small evaluation, and the held-out set is a challenge set written after I'd seen v3's errors, not a random sample of real messages.
- The validator proves citations exist, not that they are relevant: a real but loosely related clause passes.
- About three minutes per request on a laptop CPU, so several minutes for a message with a few requests. Fine before replying to a client; too slow for a chat.
- It reads agreements; it is not legal advice.
Prize Categories
- Best Use of Gemma — Gemma 3 4B and Gemma 4 E4B run locally through Ollama, compared on the same frozen pipeline; Gemma 4 E4B is the default.
Top comments (0)