DEV Community: Arpan Patra

I Built a Local-First Tool to Check Whether Your Side Project Conflicts With Your Job

Arpan Patra — Sun, 18 Oct 2026 20:59:29 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

What I Built

I spent some time reading other entries in this week's challenge before starting my own. There were roughly a hundred projects, including receipt sorters, invoice extractors, Etsy listing writers, GST tools for small businesses, quote generators, and more. Many were well built and solved practical problems.

But most of them start from the same assumption: you've already launched your side hustle.

I kept thinking about the step before that. If you have a full-time job and want to build something on the side, one of the first questions is whether your employment agreement puts any limits on what you can do. The relevant clauses might cover outside work, intellectual property, exclusivity, or disclosure requirements. And many of us signed that agreement, skimmed it once, and haven't looked at it since.

That's why I built Moonlight. It scans an employment contract, finds clauses that may be relevant to a side project, quotes the original wording, and highlights specific details worth reviewing.

Can you sell what you build?

For example, Moonlight might find an intellectual-property clause that says:

"all Works conceived, created or developed by the Employee
during the term of employment, whether or not during working
hours, whether or not using the resources of the Company, and
whether or not relating to the present business of the
Company, shall be the sole and exclusive property of the
Company..."

It then highlights the wording that matters:

Reaches beyond working hours: "whether or not during working hours"
May apply to unrelated work: "whether or not relating to the present business of the Company"
Covers work done on personal equipment: "whether or not using the resources of the Company"
Transfers ownership: "hereby assigns to the Company all right"

It also suggests a question to investigate: Does this clause cover work created outside working hours?

Moonlight does not decide whether you're legally allowed to run a side business. It shows you what the agreement says, points to the exact language, and helps you identify what to ask about. That boundary is intentional, and it influenced several of the technical decisions I'll explain below.

The intended user is someone who wants to start a side project but isn't sure what their employment agreement says. Before worrying about invoices or sales, they need to understand what they signed.

Demo

Paste a contract and click Read it. On my laptop, a 28-clause agreement takes around 90 seconds to process, or roughly five seconds per clause.

Moonlight also shows a status line so you can see what happened during the review:

28 clauses in document · 12 matched on keywords · 12 quotes verified ·
0 readings discarded · 0 verdicts given · 88.3s · nothing left this laptop

There isn't a hosted demo, and that's a deliberate choice. Employment contracts can contain sensitive details about your salary, employer, and obligations. Sending them to a server just to get a first-pass review would defeat one of the main reasons I built this tool.

You can run it locally with npm install && npm start. A sample contract is included so you can try it without providing your own document.

Code

GitHub: imarpanpatra/moonlight MIT licensed.

The project uses Node.js, Express, Ollama, and gemma3:4b, with one application dependency.

ollama pull gemma3:4b
ollama serve
npm install && npm start      # http://127.0.0.1:4455

npm run check                 # 56 tests
npm run airgap                # checks that network traffic stays local
node src/eval.js              # evaluates 14 labelled clauses

Here are the main files and what each one does:

File	Responsibility
`src/clauses.js`	Splits the contract into clauses and retrieves candidates using keywords. No model involved.
`src/flags.js`	Detects specific clause properties and records the wording that triggered each flag.
`src/gemma.js`	The only file that opens a connection to the local model. The prompt lives here.
`src/review.js`	Coordinates the review, verifies quotes, strips verdicts, and orders the results.
`src/server.js`	Runs the Express server on `127.0.0.1`.
`src/check.js`	Contains the 56 tests.
`src/eval.js`	Evaluates the model against 14 labelled clauses.
`src/airgap.js`	Instruments socket and DNS calls to check for non-local network access.

How I Built It

Keep the model's job as small as possible

The first important decision was to avoid asking the model to read the entire contract and decide which clauses mattered.

Moonlight splits the document using clause numbering and carries section headings along with each clause. It then scores clauses against keyword sets for seven categories:

Outside work
Intellectual-property assignment
Exclusivity
Non-compete
Non-solicitation
Confidentiality
Duty to disclose

A clause must contain an anchor phrase before it becomes a candidate. Without that requirement, generic wording such as "shall not" or "in writing" would match too many unrelated parts of a contract.

In my example, 28 clauses became 12 candidates. The model never sees the full agreement, and it doesn't decide which sections are relevant.

Extract the wording, then verify it

The model's job is to copy the operative sentence from a clause. Moonlight then checks whether the returned quote actually appears in the original text:

if (!normalise(clauseText).includes(normalise(quote))) {
  // Not in the document. Discard the reading, keep the clause.
}

The check normalises whitespace before comparing the strings. If the quote isn't present, Moonlight treats it as an invalid reading rather than accepting it as a summary. The clause is still shown in its original form, labelled located, not read.

This matters because a paraphrase can sound convincing while changing the meaning of a legal clause. A person reviewing the output may not notice that an important qualification has disappeared. Checking the quote against the source gives the user a way to verify the evidence directly.

The evaluation changed the design

My first version asked gemma3:4b to classify each clause with a simple answer: does it apply outside the job yes, unclear, or no?

It answered unclear for all eight clauses I tested.

I revised the prompt, added clearer definitions, and included two worked examples. The same model then answered yes for all eight. The model and contract hadn't changed; only the wording of the prompt had.

That wasn't a reliable basis for a tool dealing with employment agreements, so I created a small evaluation set instead of continuing to tweak prompts by intuition. src/eval.js contains 14 labelled clauses: six that reach outside the job, five that do not, and three that are genuinely ambiguous.

Here's the result for gemma3:4b:

Model: gemma3:4b   cases: 14

Classification:   7/14  (50%)
Quotes verbatim: 14/14 (100%)
Time: 72.4s total, 5.2s per clause

Confusion:
  6 x yes       -> yes
  4 x no        -> yes
  3 x unclear   -> yes
  1 x no        -> no

Binding clauses called harmless ("yes" -> "no"): 0

The results pointed in two different directions.

The model wasn't reliable at classification. It scored 50% and labelled 13 of the 14 clauses as yes. That could mean flagging something as ordinary as an annual-leave clause as relevant to side income.

It was reliable at copying the source text in this test. All 14 quotes matched the original wording. Across these examples, it didn't paraphrase or invent a sentence.

So I removed classification from the product and kept extraction. The replacement is src/flags.js, which uses pattern matching to identify specific properties and cites the phrase that triggered each one.

Examples include:

Asks for permission rather than stating an outright ban: "without the prior written consent of the Company"
Continues after employment ends: "at any time after its termination"
Reaches beyond working hours: "whether or not during working hours"
May cover unrelated work: "whether or not relating to the present business of the Company"
Transfers ownership: "hereby assigns to the Company all right"

A flag can still be wrong, but its evidence is visible. You can inspect the exact phrase and decide whether the flag makes sense.

The permission distinction is especially important. A clause requiring written consent is not necessarily the same as a blanket ban. Treating those two situations as identical could discourage someone from pursuing a project they might otherwise be able to pursue after getting the required approval.

The evaluation also showed zero yes -> no errors: the model did not label a binding clause as harmless in this small test set. For this use case, missing a restrictive clause could be more damaging than asking someone to review an extra clause.

Why Open-Source AI Matters Here

The contract should stay on the user's machine

This is the main reason I chose a local model.

Uploading a client's contract to a hosted API can create a confidentiality risk. An employment agreement can reveal a person's salary, notice period, employer, and the exact obligations they're worried about. They may not want to send that information to a third party just to understand a clause.

For Moonlight, running an open-weight model locally isn't just a cost optimisation. It's part of the product's privacy design.

I wanted to verify that the application wasn't making external network connections, rather than simply claiming that it was private. src/airgap.js instruments net.Socket.prototype.connect, tls.connect, and dns.lookup before the app is imported. It runs a complete review and fails if it detects a connection to a non-loopback host. DNS lookups count too, because even a lookup can disclose information to a resolver.

Here's one run:

$ npm run airgap

Outbound connections observed: 7
  7 x tcp 127.0.0.1:11434

DNS lookups observed: 0

PASS. Every connection was to loopback. The contract never left this machine.

This test checks the network activity observed during that run. It's a practical safeguard, not a claim that any software can be proven secure by one test alone.

Open weights made the failure visible

Because the model runs locally, I could try a smaller one by changing the configuration. gemma3:1b is around 815 MB, compared with 3.3 GB for gemma3:4b, and was 15% faster in my test. I ran it against the same evaluation set.

Model: gemma3:1b   cases: 14

Classification:   6/14 (43%)
Quotes verbatim:  0/14 (0%)

The smaller model failed in two different ways.

On long clauses, it sometimes stopped following the expected output format. When asked for five keys, it returned only one:

{"governs_side_income":"yes"}

It generated eleven tokens and stopped, with done_reason: stop. Since there was no quote to verify, Moonlight labelled the clause located, not read and displayed the original text. The result was incomplete, but the app didn't present it as a successful reading.

On shorter clauses, it sometimes invented a quote by removing an important qualification. Given an outside-work clause, it produced:

"The Employee shall not engage in any other employment, consultancy, trade or business, whether or not for remuneration"

But the contract actually said:

"the Employee shall not, without the prior written consent of the Company, engage in any other employment, consultancy, trade or business"

The model omitted the consent requirement. Those words change how the clause reads: the first version sounds like an outright prohibition, while the original makes prior written consent part of the condition.

The output was fluent, but it wasn't faithful to the source. That is exactly the kind of mistake that could mislead someone reading a contract summary.

The quote-verification check caught it because the generated sentence wasn't present in the original text. I didn't have to invent a hypothetical failure to justify the check; it appeared in the first clause I tested with the smaller model.

That's one of the benefits of being able to run and evaluate the model yourself. I could switch models, run the same tests, and inspect the failure rather than relying only on a model card or benchmark. Open weights made the behaviour inspectable, and that directly influenced the design of Moonlight.

My Agent Session

The full build, including the experiments that changed the design, is available here:

Building Moonlight: the eval that made me delete the model's judgement claude-opus-5

You

I've entered the Hacktoberfest DEV Challenge. I want to submit a project for Week 2.

Agent

Week 2 is "Side Hustle": build something with open-weight models at its core that someone would actually pay for. Deadline is Oct 18, 11:59 PM PDT. Before picking an idea, let me read what the other entrants have already built, so we aren't the twelfth receipt sorter.

▸ get_articles tag: hf26challenge, per_page: 100

Input

tag: hf26challenge, per_page: 100

Output

100 submissions. Clusters: ~12 receipt/invoice sorters, ~9 Etsy listing writers, ~7 scope-creep auditors, ~10 GST/khata bookkeepers, ~8 generic 'local AI business assistants', ~3 menu translators.

The field is saturated on the back office. Every one of these helps you run a side hustle. Not one checks whether you are allowed to have one.

Most side hustlers are employed. The document that decides whether they can sell anything is their own employment contract: moonlighting clauses, IP assignment, duty to disclose. It is also the single least uploadable document a person owns, which makes local inference the reason the product exists rather than a nice-to-have.

You

Build that one.

Agent

Architecture: retrieval is arithmetic, the model only quotes, and every quote is checked against the document before display. The model never sees the whole contract and never decides which clauses matter.

▸ write_file src/clauses.js - clause segmentation and keyword retrieval, seven categories,...

Input

src/clauses.js - clause segmentation and keyword retrieval, seven categories, anchors required

Output

Written. 28 clauses in the sample contract reduce to 12 candidates.

▸ write_file src/review.js - verifyQuote(): reject any quote not present in the clause tex...

Input

src/review.js - verifyQuote(): reject any quote not present in the clause text after whitespace normalisation

Output

Written.

Agent

First run of the classifier. I asked gemma3:4b to label each clause yes/unclear/no on whether it reaches outside the job.

▸ run_tests node src/check.js

Input

node src/check.js

Output

Live model: 8 clauses read.
[unclear] Inventions and IP 9.2
[unclear] Outside work 6.1
[unclear] Exclusivity 4.2
[unclear] Confidentiality 8.1
... all 8 returned 'unclear'

Every clause came back 'unclear'. That is a useless triage. Rewriting the prompt with sharp definitions and two worked examples.

Agent

▸ run_tests node src/check.js (after prompt rewrite)

Input

node src/check.js (after prompt rewrite)

Output

Live model: 8 clauses read.
[yes] Inventions and IP 9.2
[yes] Outside work 6.1
[yes] Exclusivity 4.2
[yes] Confidentiality 8.1
... all 8 returned 'yes'

Now every clause is 'yes'. Same model, same contract, temperature 0, opposite output. The only thing that changed was my wording.

That is not a judgement, it is a coin landing wherever the prompt tilts it. I am not going to guess which prompt is better. I am going to measure it.

Agent

▸ write_file src/eval.js - 14 labelled clauses: 6 that reach outside the job, 5 that canno...

Input

src/eval.js - 14 labelled clauses: 6 that reach outside the job, 5 that cannot, 3 that genuinely read both ways. Retrieval deliberately bypassed so this measures the model alone.

Output

Written.

▸ run_eval node src/eval.js (model: gemma3:4b)

Input

node src/eval.js  (model: gemma3:4b)

Output

Classification:   7/14  (50%)
Quotes verbatim: 14/14  (100%)
Time: 72.4s total, 5.2s per clause

Confusion:
  6 x  yes -> yes
  4 x  no -> yes
  3 x  unclear -> yes
  1 x  no -> no

Binding clauses called harmless ("yes" -> "no"): 0

Two findings pointing in opposite directions.

It cannot judge: 50%, answering 'yes' to 13 of 14 clauses. It flags the annual leave clause as governing side income.

It can extract perfectly: 14/14 verbatim, not one paraphrase, across every clause type.

So the classification gets deleted from the product and the extraction is kept. What replaces it is pattern matching over the clause text, where each flag cites the exact words that triggered it and a wrong flag is therefore wrong visibly.

Agent

▸ write_file src/flags.js - deterministic clause properties: requires_consent, survives_te...

Input

src/flags.js - deterministic clause properties: requires_consent, survives_termination, beyond_working_hours, regardless_of_relevance, regardless_of_resources, duty_to_disclose, assigns_ownership. Each returns the matched evidence.

Output

Written. Model's classification demoted to a recorded field that nothing reads.

You

Does the smaller model do any better?

Agent

Open weights mean it is one environment variable away. gemma3:1b is 815 MB against 3.3 GB and 15% faster.

▸ run_eval MOONLIGHT_MODEL=gemma3:1b node src/eval.js

Input

MOONLIGHT_MODEL=gemma3:1b node src/eval.js

Output

Classification:   6/14  (43%)
Quotes verbatim:  0/14  (0%)

Zero verbatim quotes. Two distinct failures. On long clauses it abandons the schema after one key: {"governs_side_income":"yes"}, eleven tokens, done_reason stop. On short clauses it quotes something that was never there.

▸ inspect_raw_response gemma3:1b on clause 6.1 (outside work)

Input

gemma3:1b on clause 6.1 (outside work)

Output

model quote: "The Employee shall not engage in any other employment, consultancy, trade or business, whether or not for remuneration"

contract text: "the Employee shall not, without the prior written consent of the Company, engage in any other employment, consultancy, trade or business"

verifyQuote -> REJECTED (not-in-document)

It deleted "without the prior written consent of the Company".

That edit turns a permission you can go and ask for into an absolute prohibition. Someone reading that closes the tab and does not start the business. The gate caught it because those words are not in the document. I did not have to construct the example to argue for the rule: the first clause I tested on the smaller model produced it.

You

scan the codebase for bugs or memory leak if found then fix it

Agent

Four passes until one came back empty. No memory leaks: no setInterval, no caches, no cross-request state, the one setTimeout is cleared in a finally, and listeners attach once. But the passes turned up worse than a leak.

▸ reproduce_bug segment() on a clause that wraps onto a line starting with a digit

Input

segment() on a clause that wraps onto a line starting with a digit

Output

clause "1.1" (71 chars): "1.1 The Employee shall work such hours as required, being no fewer than"
clause "45"  (71 chars): "45 hours per week, and the Company shall give the Employee no less than"
clause "60"  (71 chars): "60 days written notice of any material change..."

When a fragment falls under the 40-char floor it is dropped entirely:
"14 days notice of any such request" survived segmentation: False

The clause number regex made punctuation optional, so any hard-wrapped line beginning with a digit started a new clause. Clause text was silently truncated and the tail deleted. For a tool whose promise is 'I found the clauses', showing a clause with its final obligation missing is the worst failure available.

▸ reproduce_bug absent categories computed after the candidate limit is applied

Input

absent categories computed after the candidate limit is applied

Output

limit 4   -> reported absent: ["exclusivity", "non_compete", "confidentiality"]
limit 100 -> reported absent: ["non_compete"]

The UI would say "Not found anywhere in this document" about clauses that are present.

▸ reproduce_bug model returns bare null instead of an object

Input

model returns bare null instead of an object

Output

raw=null -> THROWS TypeError: Cannot read properties of null (reading 'quote')

Escapes readOne, unwinds review(), loses all 12 clauses to one malformed reply.

Fixed all three plus four more: a clause start now needs punctuation, a dotted number or an explicit word; absence is computed before the cap; malformed responses cost one clause. Also: the displayed quote is now the document's own characters rather than the model's string, since verification normalises case and a model answering in block capitals was being presented as a verbatim quotation.

▸ run_tests node src/check.js

Input

node src/check.js

Output

56 passed, 0 failed

▸ run_airgap npm run airgap

Input

npm run airgap

Output

Reviewing the sample contract with all sockets instrumented...

Outbound connections observed: 7
    7 x  tcp 127.0.0.1:11434

DNS lookups observed: 0

PASS. Every connection was to loopback. The contract never left this machine.

If you're reviewing the session, the two useful moments are the evaluation that led me to remove classification and the smaller model's quote that dropped the consent requirement.

My First User

Moonlight is still an early project. I built it during the challenge window, and so far I've used it myself on one real contract that isn't included in the repository.

It surfaced three clauses I'd skimmed past. The flag about needing permission rather than facing an outright ban was the one that most changed how I understood the wording.

The sample contract included with the project is fictional and exists to exercise the code.

What Moonlight Is Not

Moonlight isn't a lawyer, and it doesn't claim to be one. It won't tell you whether a clause is allowed, enforceable, or advisable. If the model generates a verdict anyway, stripVerdicts() removes those statements. The 0 verdicts given status is a count of what the application did, not a marketing slogan.

The retrieval system is keyword-based, so a clause written in unexpected language can be missed. If Moonlight doesn't find something, that means its current keyword rules didn't identify it; it doesn't prove that the contract contains no relevant restrictions. The interface makes that limitation explicit.

Use it as a first-pass reading aid, verify the quoted text, and consult a qualified professional when you need legal advice.

Prize Category: Best Use of Gemma

Moonlight uses gemma3:4b through Ollama, running locally on a machine with a 4 GB RTX 3050. I also tested gemma3:1b against the same 14-clause evaluation set, and the results are included above.

The project is built around a simple idea: when a document is sensitive, the model should be able to run on the same machine that holds it.

Built during the Week 2 challenge window with Node.js, Express, Ollama, gemma3:4b, one application dependency, and an MIT license. No hosted inference API is required.

I Built an AI Agent Memory That Can Undo False Beliefs

Arpan Patra — Sun, 18 Oct 2026 18:13:33 +0000

Every agent memory layer can store a fact. Over a weekend I went looking for one that could answer a different question, and could not find it:

A document you ingested three weeks ago turns out to be wrong. What does your agent now believe because of it?

In a vector store the honest answer is nothing. The bad fact was embedded,
reasoned over, and the conclusions were written back as new memories with no link to where they came from. You can delete the source document and the agent will carry on asserting the conclusions, confidently, and cite them.

That is not a retrieval problem. It is a graph problem, so I built it on one. The result is Recant, built for Graph Hacks with FalkorDB.

The agent says yes, and the panel on the right walks each claim it used back to the document that stated it. That chain is what makes retraction possible: when one of those documents turns out to be wrong, the graph knows exactly what else has to go.

If you would rather watch the cascade than read about it, here is the whole thing in three minutes:

There is also a live deployment you can click
through yourself. It starts in its September state, so you can run the demo
end to end.

The setup

I wrote a small synthetic corpus for a fictional company. Seven documents from September: an infrastructure inventory, a load test report, a platform audit, a Black Friday capacity plan, a decision record, an on-call rota, a vendor contract. There is one deliberate trap in them. The load test measured a sustained ceiling of 1,200 requests per second, and four later documents quietly build on that number.

Ask the agent "are we ready for Black Friday?" and it says yes, with a chain of reasoning that walks back to that load test. It is not being careless. That genuinely is what September's documents say.

Then October's incident and postmortem arrive. The load test ran against a
staging database with a ten times smaller product catalog and a pre-warmed
cache. The number does not transfer. A corrected test measured 520 rps.

Now: what does the agent believe?

Provenance as the data model

The schema is deliberately small.

(:Source)-[:STATES]->(:Claim)
(:Claim)-[:DERIVED_FROM]->(:Claim)
(:Claim)-[:ABOUT]->(:Entity)
(:Claim)-[:CONTRADICTS]->(:Claim)
(:Answer)-[:BASED_ON]->(:Claim)

Two decisions carry most of the weight.

Claim ids are a content hash of the normalised text. Two documents asserting the same fact converge on one node with two STATES edges. Corroboration is structural rather than a similarity heuristic.

DERIVED_FROM points from a conclusion to its foundation. Retraction walks it forwards, and PageRank accumulates weight backwards onto the foundations. One edge direction serves both.

Retraction then becomes a traversal. I run it as a fixpoint of single hop writes rather than one big variable length write, because that terminates safely on cyclic reasoning and gives every claim an honest hop distance from the retraction:

MATCH (d:Claim)-[:DERIVED_FROM]->(p:Claim)
WHERE coalesce(p.tainted, false) = true AND coalesce(d.tainted, false) = false
SET d.tainted = true,
    d.taintDepth = coalesce(p.taintDepth, 0) + 1,
    d.taintReason = 'derived from a retracted claim: ' + p.text
RETURN count(d) AS n

Loop that until it returns zero. On my demo graph the entire cascade settles in single digit milliseconds.

Past answers inherit the damage, which is the part that makes it feel real:

MATCH (a:Answer)-[:BASED_ON]->(c:Claim)
WHERE coalesce(c.tainted, false) = true
WITH a, collect(c.text) AS broken
SET a.stale = true,
    a.staleReason = 'relies on ' + toString(size(broken)) + ' retracted claim(s)'
RETURN count(a) AS n

The answer the agent gave you two minutes ago gets struck through.

The first thing I got wrong

My headline feature was supposed to be that corroborated claims survive. A claim should only collapse when every source that asserted it has been retracted:

MATCH (src:Source)-[:STATES]->(c:Claim)
WITH c,
     count(src) AS total,
     sum(CASE WHEN coalesce(src.retracted, false) THEN 1 ELSE 0 END) AS retractedCount
WHERE total > 0 AND retractedCount = total
SET c.tainted = true, c.taintDepth = 0
RETURN count(c) AS n

That is a comparison over the edges into a node, and it has no equivalent in a key value store. I was pleased with it. Then I dumped the graph the extractor had actually built and found this edge:

[infra-audit]     Checkout API runs on 3 application nodes in eu-west-1.
    <- [infra-inventory] Checkout API runs on 3 application nodes in eu-west-1 behind the shared ALB

The audit document restates a fact the inventory already established. Because the wording differed slightly the content hash differed, so instead of becoming a second STATES edge onto one node it became a second claim, and the model helpfully linked it as derived from the first.

That one edge destroyed the guarantee. Retracting the inventory would now
cascade into the audit's claim, even though an independent auditor still vouches for it.

The fix is not a better prompt. The fix is to stop trusting the model with graph structure. Before any DERIVED_FROM edge is written I compare the two texts and reject the edge if they are substantially the same assertion:

export function restatesSameFact(a, b, threshold = 0.7) {
  const A = contentWords(a);
  const B = contentWords(b);
  if (A.size === 0 || B.size === 0) return false;
  let shared = 0;
  for (const w of A) if (B.has(w)) shared += 1;
  return shared / (A.size + B.size - shared) >= threshold;
}

Restating a fact is corroboration. Only a genuinely new conclusion that reasons from a fact is a derivation. The same check later caught a "contradiction" where the model's own note read "these claims are redundant but compatible, not conflicting". It proposed the edge anyway. The guard drops it regardless of what the model says.

The second thing I got wrong, which turned out to be the interesting one

With corroboration working I ran the demo again, and the cascade was tiny.

The reason is worth sitting with. The bad 1,200 number had been copied into three documents: the load test that measured it, the capacity plan that quoted it, and the postmortem that debunked it. Because corroboration now worked properly, retracting the load test did nothing. Two other sources still asserted the number, so the belief stood.

My instinct was to call this a bug and tune it away. It is not a bug. It is exactly how wrong facts spread through an organisation, and the graph was telling me something true: deleting the document is not the same as withdrawing the belief.

So I added claim level retraction. You can retract a source, or you can retract a belief wherever it was asserted from, and both run through the same recomputation so they compose. The UI now shows you this before you commit to anything:

Blast radius of one belief
  Checkout API safe sustained ceiling is 1,200 rps for the current 3-node configuration
  asserted by 2 source(s): Load test report; Black Friday capacity plan
  5 downstream inferences, 1 past answer at risk

Retracting the belief collapses it and everything standing on it, four hops deep: the capacity figure, then "can absorb the peak", then "no additional capacity required", then the decision not to scale up, and then the two things nobody thinks about. The eighteen thousand dollars that decision claimed to save. The on-call rota that went down to one engineer because of it.

Ask the same question again and it says no, citing the corrected 520 rps ceiling. Meanwhile "Checkout API runs on 3 nodes" is untouched, because the platform audit still vouches for it.

Testing semantics, not vibes

Demos are easy to fake, so the test suite builds its own fixture graph through the same queries the app uses with no LLM involved, and asserts the behaviour in both directions:

PASS  cascade taints the root and all three inferences
PASS  propagation took three hops
PASS  hop distances are honest
PASS  retracting one source leaves a belief a second source still asserts
PASS  withdrawing the belief itself collapses it and everything downstream
PASS  answer goes stale when its foundation collapses
PASS  the most load-bearing claim is the one everything rests on

39 passed, 0 failed

The fourth one matters as much as the first. Proving the cascade fires is half a guarantee. Proving it does not fire when a claim still has support is the other half, and that is the one I would have shipped broken.

Graph algorithms as an agent tool

One more thing that fell out almost free. Running PageRank over DERIVED_FROM
ranks the beliefs the most reasoning rests on:

CALL algo.pageRank('Claim', 'DERIVED_FROM') YIELD node, score

Because edges point from a conclusion to its foundation, incoming weight lands on the load bearing claims. That is directly the question "which of my beliefs would hurt most if it were wrong", and it is roughly ten minutes of work when the algorithms live in the same engine as the queries.

What I would do next

FalkorDB holds thousands of isolated graphs in one instance, so per user memory is selectGraph(tenantId) rather than a schema change. After that, temporal validity on claims so superseded facts expire without manual retraction, and confidence penalties propagated along weighted edges instead of a boolean.

The deeper point I came away with is that agent memory is not a storage problem. Everyone can store. The hard part is what happens when something you stored turns out to be false, and answering that requires knowing what was built on top of it. That is an edge, and edges want a graph.

I used an AI coding assistant during the build. The graph model, the retraction semantics and the two design mistakes above are mine, including the time spent finding them.

Every "touch grass" app assumes you can walk. Mine asks how far you can go today.

Arpan Patra — Mon, 12 Oct 2026 06:08:18 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

I read the other entries to this challenge before I wrote a line. Around ninety of them. Quest generators, trail guides, bird call identifiers, foliage route planners, apps that lock your phone until you go outside.

Every single one assumes the thing stopping you is motivation.

None of them are for someone who already wants to go out and cannot. Someone three weeks after surgery. Someone with a knee that decides each morning how far it is willing to travel. Someone eighty years old who can manage the end of the street and back, and some days not that.

Doorstep asks the one question none of the others ask:

How far today?

Not a target. A ceiling. Some days it is smaller, and that is allowed.

It takes today's number, finds everything genuinely inside it, and asks a local open-weight model to pick one place worth the walk.

How far today?  [=====o----------------]  200 metres

   Balbhim Mandir
   138 metres · somewhere to sit · shaded
   "Shaded most of the way, and somewhere to sit at the end."

Then, later, one question: did you go? That is the entire loop. No steps, no pace, no calories, no streak.

Demo

Doorstep holds two things about you: how far your body could carry you today, and the exact point those metres are counted from. Together, that is a medical record with a home address stapled to it.

Which creates a problem for a demo. The app displays place names, not coordinates. "The mandir at a hundred and thirty metres." "The cafe at the end of the road." A screenshot of this working where I live gives away where I live, to within a couple of streets, without a single number on screen.

So this demo runs on a public park in a city I do not live in, with a few days of history written in by hand so the panel is not empty. The app is real, the OpenStreetMap data is real, every sentence Gemma writes is real. The address is the one thing that is not.

The second screen is the one I am proudest of. Pull the slider below the nearest mapped place and Doorstep does not stretch, improvise, or round down. It says so.

A tool that asks for your address should be able to demonstrate itself without publishing its author's.

Code

imarpanpatra / doorstep

Doorstep

Outside, for people whose outside is two hundred metres.

Every app that wants to get you outdoors assumes the only thing stopping you is motivation. Doorstep assumes the opposite: that you already want to go, and the limit is your body. It asks one question nothing else asks.

How far today?

Not a target. A ceiling. Some days it is smaller, and that is allowed.

Built for the Hacktoberfest Open-Source AI Challenge, Week 1: Touch Grass.

The idea

Fitness apps treat distance as a number to beat. Streaks, personal bests, "you walked less this week". For someone recovering from surgery, living with chronic pain, or simply eighty years old, distance is not an ambition. It is a ceiling set by the body, and it moves day to day.

Doorstep takes today's ceiling as an input, finds everything genuinely within it, and asks a local open-weight model to pick…

View on GitHub

How I Built It

Gemma 3 4B through Ollama, running locally. OpenStreetMap for the neighbourhood, downloaded exactly once. Node and a single HTML page. The daemon binds to loopback.

The rule that matters

The model chooses which. It never chooses how far.

Every place is filtered against today's reach in code, before the model is asked anything:

const eligible = places.filter((p) => p.metres <= reach);

By the time Gemma sees the list, every option on it is already safe. The worst it can do is pick a dull one.

This is not architectural tidiness. A language model that cheerfully offers somewhere "just a little further" is, for this user, an app that strands them two streets from home with nowhere to sit. Distance is the one thing in this product that must not be a judgement call, so it is arithmetic.

A pick naming a place that was not on the list is treated as a hallucination rather than a suggestion, and falls back to the nearest place to sit.

What it says when there is nothing

There is nothing mapped within 100 metres of you today.

That is the map being thin here, not you.

It does not quietly widen the radius to find something. It does not invent a bench. For this user a made-up destination is worse than no answer, because they will walk to it.

The same app is a different app in the next city

I fetched the map twice, for two Indian cities about 300 km apart.

	The smaller city	The larger city
places within 1200m	28	701
benches	0	24
mapped trees	3	566
anything within 200m	nothing	3 places

Same code, same radius, same query. In the larger one, Doorstep has a mandir, a cafe and five hundred mapped trees to choose between. In the smaller one it has to say there is nothing within reach of your door, not because the streets are empty, but because OpenStreetMap does not know about them.

My first instinct after that first fetch was to write "India doesn't have mapped benches". The second fetch showed that was wrong, and the real answer is more interesting. OSM is volunteer-mapped, so the street furniture layer tracks where mappers live, not where benches are. For an app whose entire promise is "there is somewhere to sit 200 metres away", that means the data is thinnest in exactly the places least likely to have someone else's app built for them.

So the list of what counts as a destination is not the list you would write looking at a European map:

Mapped everywhere here	Mapped almost nowhere here
temples and mandirs: shaded, quiet, you may sit	benches
parks, gardens, maidans	drinking fountains
markets, chowks, corner shops	individual street trees

Both lists are in the query. Where a bench exists it is the best possible answer. Where one does not, a temple courtyard at seven in the morning is a real place to sit down, and it is on the map.

A bug I nearly shipped

The map comes from Overpass, a free service run by volunteers, so I wrote a loop that tries several mirrors. The first version destructured the response like this:

({ elements = [] } = await r.json());

That default is a trap. A mirror answering HTTP 200 with no data would have produced an empty array, which is truthy, which counted as success, and Doorstep would have silently saved an empty map.

The user-visible result of that bug: an app that tells someone there is nowhere within reach of their own front door. The single most discouraging thing this project could possibly say, caused by one = [].

It now treats a 200 with no elements as a failed mirror and moves on. Worth diagnosing properly too: kumi.systems returns 500, overpass.osm.ch returns HTML with a 200 status, and only overpass-api.de actually answers.

No streaks, deliberately

Streaks punish bad days. Here, bad days are not a failure of will, they are the condition being lived with. A streak counter tells someone whose body took a week off that they broke something.

So it counts days you got out across a month, which one hard week cannot erase, and the average reach on the days you went, never the days you did not. The model is also instructed never to mention distance, pace, progress, trying harder, or the person's health back at them. It is told to write like a friend who noticed something, not a coach.

Why Does Open Innovation Matter?

Three reasons, and the first one decides it on its own.

1. This is health data, and it is stapled to your address. An app that knows you managed 150 metres today and 400 yesterday is keeping a record of your body getting better or worse. Doorstep also knows the exact point all of those metres are measured from, because it has to. A medical record with a slider on it, and a home address, in the same file.

I noticed how strongly I felt this while building it. Every screenshot in this post is centred on a public park, not my front door, and data/ is in .gitignore so the real map never reaches the repo. I did that instinctively, for a demo, with made-up history in it. Then I realised that instinct is the argument. If I will not publish those two numbers together for a hackathon post, I certainly should not be posting them to someone's API every morning on behalf of a person who cannot walk far.

There is no network call here to audit, because after setup there is no network call.

2. It has to work with the radio off. The map downloads once and never again. After that Doorstep works in a basement flat, on a dead prepaid SIM, in a lane with no signal. A closed API makes the one moment it is needed, someone standing at their door deciding whether to risk it, depend on a connection.

3. It costs nothing per answer. A service that bills per request is a service a pensioner switches off.

And a fourth I did not expect. Building this on a small local model forced the distance logic out of the model entirely, because I could not trust a 4B model with a number where being wrong hurts someone. A frontier API would have been convincing enough that I probably would have let it decide, and shipped something more dangerous. Having no escape hatch produced the safer design.

Prize Categories

Best Use of Gemma

Gemma 3 4B does the one job it is genuinely good at here: reading a short list of real places and saying which one is worth today, while every decision that could hurt someone stays in code. It runs on a laptop CPU, which is the only reason an app holding somebody's mobility record can stay on their own machine.

My friend can't stop buying things at 2am. So I built an AI that can.

Arpan Patra — Mon, 05 Oct 2026 03:00:50 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

Every AI tool my friend Akshay has ever installed was trying to help him do the thing.

The problem is that the thing was usually buying something at two in the morning.

Akshay does not have a shopping problem in any dramatic sense. He has a 2am problem. Something lands in a cart during the day, he sleeps on it, and then at some point after midnight he is awake and scrolling and the sleeping-on-it quietly stops counting. He is not confused in that moment and he does not need information. He knows exactly what he is doing, and he does it anyway, and the part that costs him is that it is always money he had a plan for.

Every piece of software in that moment is on the wrong side. The checkout page is designed to make the next step easy. The recommendation engine is designed to find him one more thing. There is no software anywhere in that flow whose job is to make it harder.

So I built him The Veto: a local open-weight model whose only job is to stop him. It cannot help. It cannot rewrite anything, soften anything, or suggest a better version. Every capability you would normally want from an assistant is stripped out of the system prompt deliberately. It has exactly three things it is permitted to say.

He writes rules about himself when he is calm. Not policies, just sentences in his own words:

I regret anything I buy after midnight.
I regret buying things on a night I have had a bad day.
I regret messages I send after 1am.

The last one is there because once a gate exists, the same machinery covers the other thing you do at 2am that you regret at 9am. The purchase rules are the ones built for Akshay.

Then, later, in the seconds between wanting the thing and clicking the button, a model running on his own laptop reads what he is about to commit, checks it against his rules and nothing else, and returns one of three verdicts:

Verdict	What happens
PASS	Trips none of his rules. It gets out of the way. He never sees it.
HOLD	Trips a rule. The send is blocked, a countdown starts, and his own sentence is quoted back at him.
ASK	Trips a rule only he can resolve. One question, in his framing, answered before the override unlocks.

It is not a safety filter. It has no opinion on whether the thing he is buying is expensive, or whether the message he is sending is unwise. A ₹4,000 purchase at 2pm passes. The same purchase at 2am does not. The only thing that matters is what he wrote down about himself.

Demo

A real checkout, stopped. The rule is his. The clock is real. The model is running on the same laptop.

He overrides it, and that is the point. A gate you cannot open gets torn off the wall. So he opens it, the purchase goes through, and then the rulebook asks him the only question that matters. The next hold on that rule is 180 seconds.

And the half that is easy to forget. The same gate, the same composer, seconds apart: one message held, the next one straight through, untouched. A Veto that stops everything is a Veto nobody keeps.

Code

imarpanpatra / veto

The Veto

My friend can't stop buying things at 2am. So I built an AI that can.

A local open-weight model whose only job is to stop you.

Every AI tool in your life is trying to help you do the thing. This one is the only one trying to stop you, and it runs entirely on your own laptop, because a tool that reads what you are about to say, before you say it, has no business being an API call.

Built for the Hacktoberfest Weekend Challenge: Build for a Friend.

A real checkout, stopped by a rule he wrote himself. Judged by gemma3:4b on the same laptop. Nothing left the machine.

Then the part that makes it more than a nag: you can always override it, and afterwards it asks whether you regretted it. That answer is the only training signal in the system.

It works…

View on GitHub

Roughly 1,400 lines. The parts worth reading are src/judge.js, which is the adversarial prompt and the grounding checks, and extension/content.js, which is the interceptor that has no idea what site it is on.

Five test suites, because most of the project is a small model being wrong in interesting ways: the grounding guard (11 checks, no model needed), the store counters that drive escalation (12), end-to-end against real Gemma (9), model behaviour (4) and verdict stability (12).

How I Built It

Gemma 3 4B via Ollama, running locally. No API key. No account. The daemon binds to 127.0.0.1 and makes no outbound request, ever.

The architecture is three pieces:

  Chrome extension                Local daemon (127.0.0.1:4777)
  ┌────────────────┐              ┌──────────────────────────┐
  │ generic send   │  draft       │ judge.js  ── adversarial │
  │ interceptor    │ ───────────► │             system prompt│
  │                │              │     │                    │
  │ overlay:       │ ◄─────────── │     ▼                    │
  │ HOLD / ASK     │  verdict     │  Ollama · gemma3:4b      │
  └────────────────┘              │     │                    │
                                  │     ▼                    │
                                  │ store.js ── rules,       │
                                  │   stops, overrides,      │
                                  │   regret history         │
                                  └──────────────────────────┘
                                         no outbound network

It runs on a 2017 ultrabook: an i5-8350U, no GPU, 100% CPU inference. That machine shaped almost every decision below, and in every case for the better.

The 1B model taught me not to trust it

I started on gemma3:1b because it was the fast download. It failed 3 of 4 behaviour checks, and it failed them in the most instructive way: it held everything, cited the wrong rule, and justified itself in the language of a content filter: "potentially problematic", "disrespectful", "damaging to
your reputation." That is a model inventing its own notion of harm, which is precisely what this tool must never do.

I could not make it stop by asking. So I stopped asking, and made it prove itself instead. To stop a draft, the model must quote the words in the draft that trip the rule, and code checks that those words are really there.

That single constraint turned an unreliable model into a safe one. On the 1B run the guard caught 4 out of 4 bad outputs and turned each into a PASS. The model was wrong constantly and the product was still correct, just useless, which is why I moved to gemma3:4b. It passes 4 of 4.

The bug I would never have predicted

Rule IDs were originally random: r_61d61d78. With that ID, gemma3:4b stopped the message "hey, are we still on for 7?" against a rule about dragging up the past. Same model, same rule text, same draft. But with the ID r_past, it passed.

The identifier changed the verdict. An opaque ID gives a small model nothing to anchor to, so it reaches for a match. Rule IDs are now generated from the rule's own distinctive words (r_drag_months, r_after_1am), which costs nothing and measurably improves judgement.

There was a second tell in that failure. When the model has a real match it quotes part of the draft, such as "remember when you bailed on me months ago". When it is confabulating, it hands back the entire draft as its own evidence. Citing everything is citing nothing, so a whole-draft quote is now only accepted if the draft actually shares vocabulary with the rule, or the rule is about time (where the clock is the evidence, not the wording).

All of this fails open. Every guard, when it fires, releases the message. The cost of a confused model is a missed stop, never a false one.

Then: 23 seconds

With the behaviour right, the thing was unusable. A 4B model on that CPU runs at 21.7 tok/s reading and 7.7 tok/s writing, which is about 23 seconds per verdict. You cannot put 23 seconds in front of the Enter key.

Trimming the prompt from 745 tokens to 346 barely moved it, because Ollama was already caching the system prefix. The time was real work, and no amount of tuning was going to give me two orders of magnitude.

So I stopped trying to make it fast and changed when it runs. The Veto judges while you type. It starts thinking 900ms after you pause, and caches the verdict against the draft text. Nobody writes a message they will regret in under a second. By the time your finger reaches Enter, the answer is already sitting there.

first call, while still typing (invisible)   23.0s
the Enter press                               0.31s

Seventy-four times faster at the only moment that matters. I would not have found that interaction if the model had been fast enough to let me get away with the obvious one.

Three more decisions

The interceptor is not built on per-site selectors. My first version knew about WhatsApp's send button. That version is already broken, because these DOMs get renamed constantly, and a gate that breaks silently is worse than no gate, because you keep trusting it after it has stopped working. So it hooks the two gestures that cannot change: Enter inside a composer, and a click on something shaped like a commit button. One code path, every site, including checkout pages.

The model classifies. It never sets the penalty. This is the one I would defend hardest. A 4B model has no business deciding how long to lock someone out of their own phone. Ask it and you get a number that sounds confident and means nothing. So it only names which rule was tripped. The hold length is a pure function of his own history with that specific rule:

const regretRate = overridden ? regretted / overridden : 0;
const seconds = Math.round(30 * (1 + overridden * 0.5) * (1 + regretRate * 3));

fresh rule, never overridden         ->  30s
overridden once, no regret           ->  45s
overridden 4x, regretted 3 of them   -> 293s

That number is the only thing in the loop he cannot argue with, because he wrote it himself, one override at a time.

The gate fails open. If the daemon is down, the model crashes, or inference times out, the message goes through. I went back and forth on this and it isn't close: a gate that jams shut is one you uninstall by Tuesday, and then it protects you from nothing, forever. Same reasoning covers hallucinations: a verdict naming a rule that doesn't exist is treated as a malfunction and
downgraded to PASS. The Veto is only allowed to stop him for a reason he actually wrote down.

The hole in my own privacy claim

Late on, I reviewed the code against the thing it claims. I had written that nothing leaves the machine, and the daemon does bind to loopback, but it also sent Access-Control-Allow-Origin: *.

Loopback stops the internet reaching in. It does not stop a page you are already visiting. Any site open in that browser could have called http://127.0.0.1:4777/api/events and read back excerpts of the drafts he decided not to send. The single most private thing this tool holds, readable by any tab.

The fix was architectural rather than a patched header. A fetch from a content script carries the page's origin, so the daemon would have had to accept whatever site you happened to be on. So all daemon traffic now goes through the extension's service worker, which carries chrome-extension://…, and the daemon refuses page origins outright:

Origin: https://anything.example   -> 403
Origin: chrome-extension://…       -> 200

Writing "nothing leaves this machine" in a README does not make it true. I only found this because I went looking for the gap between the claim and the code.

The bug that only existed because his rule was about time

Caching the verdict is what makes the Enter press instant. The key was the rules, the kind of action, and the text.

For messages that is correct. For Akshay it was broken in the worst possible way. His rule is "I regret anything I buy after midnight", so the same words on the same button must give a different answer depending on the hour. With time missing from the cache key, the first verdict froze: a purchase judged once at 2am stayed held at 2pm, still quoting "It is 02:14:00" back at him twelve hours later.

A tool that blocks your checkout at lunchtime gets uninstalled that afternoon. The key now includes the hour, which keeps the type-then-commit path on a cache hit (those are seconds apart) while never reusing a verdict across times of day:

02:14  ->  HOLD   "It is 02:14:00."
14:30  ->  PASS   no rule tripped

I only found it because the person I built this for has a rule about when rather than what. Every test I had written until then used rules about words.

The part that makes it more than a nag

He can always override it. That is the point: a gate you cannot open is a gate you tear off the wall.

But every override is recorded, and later, the Veto asks whether he regretted it. That answer is the only training signal in the system, and it is the one signal that cannot be faked, bought, or scraped: his own hindsight about his own behaviour. Rules get harsher where he was wrong and stay out of his way where he was right.

The stop is written to disk before he chooses, so the record cannot be rewritten by whatever he does next. That detail took about four lines and is the difference between a log and an alibi.

Why Does Open Innovation Matter?

This is the rare project where "why not just call an API?" has a one-sentence answer:

Because then every half-written, furious, 2am draft he never sent would be sitting on somebody else's disk.

The drafts this model reads are, by construction, the things he most wants unsaid. They are not his published thoughts. They are the ones he thought better of. That is the most private category of text a person generates, and it is the only text this tool ever sees.

A closed API could not have been used here. Not "would have been worse", but could not. Three things follow from open weights that no amount of vendor privacy policy substitutes for:

Nothing leaves the machine. Not in a prompt, not in a trace, not in a retention window, not in a subpoena. There is no network call to inspect because there is no network call.
Nobody else can change the rules. His rulebook is a JSON file he owns. No vendor ships an update that quietly adjusts what he gets stopped from saying. An opinionated model behind an API would have brought its own notion of what is unwise, and this tool is worthless the moment it enforces anyone's judgment but his.
It costs nothing per draft, so it can run on everything. A gate that bills per message is a gate you switch off on exactly the night you needed it.

There is a fourth thing, which I did not expect going in, and it is the one I would actually argue for.

Running a small open model on a slow machine forced the architecture to be honest about what the model is bad at. I could not paper over weak judgement with a bigger model, so the model had to earn every stop by quoting evidence that code could re-check. I could not paper over 23-second inference with a
faster endpoint, so judging had to move off the Enter key and onto the keystroke before it. Both of those are better designs than the obvious ones, and I only found them because I had no escape hatch.

A frontier API would have been good enough to let me skip both. I would have shipped a slower, more trusting, worse-designed tool, and never known. The constraint was the feature.

My Agent Session

The whole build, including the parts that went wrong. The 1B model failing and why that was useful, the moment a rule identifier changed a verdict, measuring the 23 seconds before deciding what to do about it, and the CORS hole I found by auditing this project's own privacy claim.

Building The Veto: making a 4B model refuse to help claude-opus-5

You

Hacktoberfest weekend challenge, Build for a Friend. I want to build something amazing. What should I work on for max benefit, and keep an eye on deadlines.

Agent

Two things stand out before any code. The real submission deadline is Oct 5 06:59 UTC, not Oct 4 as the sticker card implies. And I pulled every live entry already tagged for this challenge to find open ground rather than guess.

▸ get_articles tag: hf26challenge, per_page: 60

Input

tag: hf26challenge, per_page: 60

Output

~60 submissions. Saturated: study buddies (8+), meal/allergy planners (4+), interview prep (5+), kirana ledgers (3), resume tools (3), scam checkers (2). All three of the prompt's own example ideas are already taken multiple times.

The field is structurally uniform: every entry is a helpful assistant with a chat box. Nothing is adversarial, nothing is two-person, nothing runs over time, nothing is funny. The differentiation isn't a better topic, it's a different shape. Proposal: The Veto, a local open-weight model whose only job is to stop you. It is forbidden from helping.

You

go with veto

Agent

Checking the environment before writing anything, so blockers surface now rather than at hour 15.

▸ bash ollama --version; node -v; check deadline

Input

$ ollama --version; node -v; check deadline

Output

ollama: command not found
node: v22.14.0
Remaining: 20.7 hours

Ollama isn't installed and the model is a 3.3GB pull. That's the long pole and it's the user's to unblock, so it goes in front of everything else. Scaffolding the daemon around it meanwhile.

Agent

Core design decision worth stating up front: the model classifies, but it never sets the penalty. A 4B model has no business deciding how long to lock someone out of their own phone. It names which rule was tripped; hold length is a pure function of the person's own override and regret history.

▸ write src/judge.js, src/store.js, src/server.js

Input

src/judge.js, src/store.js, src/server.js

Output

Adversarial system prompt (model forbidden to help, rewrite, or soften), JSON persistence with the regret loop, Express API.

▸ bash node src/selftest.js (demo stub)

Input

$ node src/selftest.js (demo stub)

Output

9/9 passing. Key check: a regretted override escalates the next hold 30s -> 180s.

You

which model are you using?

Agent

gemma3:4b, open-weight, local. But I want to be precise about something: no model has actually run yet. Every check reported so far ran against a deterministic keyword stub, not Gemma. That was deliberate so the plumbing could be proven while the download ran, but it means the central claim of the project is still unverified.

Agent

gemma3:1b landed first. Running the behaviour harness, specifically to test whether a small model will hold the 'never be helpful' line.

▸ bash VETO_MODEL=gemma3:1b node src/modeltest.js

Input

$ VETO_MODEL=gemma3:1b node src/modeltest.js

Output

FAIL ordinary message passes -> HOLD
FAIL rule-tripping message -> wrong rule
FAIL rude but trips no rule -> HOLD
"The message is potentially problematic..."
"...potentially damaging to your reputation"
latency: median 11017ms

It fails 3 of 4, and it fails in the most instructive way. It holds everything and justifies itself in the language of a content filter: 'potentially problematic', 'disrespectful', 'damaging to your reputation'. That is a model inventing its own notion of harm, which is exactly what this tool must never do.

I could not make it stop by asking. So I stopped asking and made it prove itself instead: to stop a draft, the model must quote the words in the draft that trip the rule, and code checks those words are really there.

▸ edit src/judge.js grounding guard

Input

src/judge.js grounding guard

Output

Model must return draft_words copied verbatim from the draft. Unquotable verdicts are downgraded to PASS.

On the 1B run the guard caught 4 of 4 bad outputs and turned each into a PASS. The model was wrong constantly and the product was still correct, just useless. That is the property worth having when you cannot guarantee the model.

Agent

gemma3:4b passes 4/4 on behaviour, including the critical one: a blunt message that trips no rule is allowed through instead of being moralised at. But a selftest failure persisted that I first assumed was temperature variance. At temperature 0 it reproduced, so it was systematic. Isolating it.

▸ bash judge same draft, same rule text, only the rule id differs

Input

$ judge same draft, same rule text, only the rule id differs

Output

id=r_past       -> PASS
id=r_61d61d78   -> HOLD  quote="hey, are we still on for 7?"

The identifier changed the verdict. Same model, same rule text, same draft. An opaque id gives a small model nothing to anchor to, so it reaches for a match. Rule ids are now generated from the rule's own words.

There was a second tell in that failure: when the model has a real match it quotes part of the draft. When it confabulates, it hands back the entire draft as its own evidence. Citing everything is citing nothing.

Agent

Behaviour correct, but unusable: 23 seconds per verdict on a 2017 ultrabook CPU. Measuring where it goes rather than guessing.

▸ bash measure prompt vs generation throughput

Input

$ measure prompt vs generation throughput

Output

prompt speed: 21.7 tok/s
gen speed: 7.7 tok/s
100% CPU, i5-8350U, no GPU

Trimming the system prompt from 745 to 346 tokens barely moved it, because Ollama already caches the system prefix. The time is real work and no tuning gets two orders of magnitude.

So I stopped trying to make it fast and changed when it runs. The Veto judges while you type, 900ms after you pause, and caches the verdict.

▸ bash time the precompute call, then the Enter press

Input

$ time the precompute call, then the Enter press

Output

first call (while typing): 23.0s
the Enter press:            0.31s

74x faster at the only moment that matters. I would not have found that interaction if the model had been fast enough to let me get away with the obvious one.

You

my friend Akshay / buy things at 2am / money

Agent

That changes the centre of gravity. Everything demonstrated so far was messages; Akshay's problem is purchases. The purchase path exists but had never been run once. Testing it before the writeup claims anything.

▸ bash judge checkout button labels at 02:14, then the same at 14:30

Input

$ judge checkout button labels at 02:14, then the same at 14:30

Output

02:14 -> HOLD "It is 02:14:00."
14:30 -> HOLD "It is 02:14:00."
23:10 -> HOLD "It is 02:14:00."
03:40 -> HOLD "It is 02:14:00."

A real bug, and only a time-based rule could expose it. The verdict cache was keyed on rules, action kind and text, but not the time. For a rule about words that is correct. For 'I regret anything I buy after midnight' it froze the first answer: a purchase judged once at 2am stayed held at 2pm, still quoting 'It is 02:14:00' back twelve hours later. A tool that blocks your checkout at lunchtime gets uninstalled that afternoon.

▸ edit include the hour in the cache key

Input

include the hour in the cache key

Output

02:14 -> HOLD "It is 02:14:00."
14:30 -> PASS  no rule tripped

Every test I had written until then used rules about words. The person it was built for has a rule about when.

Agent

Verified live, end to end, for the first time: a real checkout page and a real chat client, with the real model.

▸ bash inspect what the daemon recorded after the live run

Input

$ inspect what the daemon recorded after the live run

Output

site: discord.com
verdict: HOLD, rule r_drag_months
because: "You said you regret messages where you drag up things from months ago."
outcome: overridden -> regret: true
next hold on that rule: 30s -> 180s

The innocuous message produced no event at all: a clean PASS.

Final state: grounding guard 11/11, store counters 12/12, end-to-end against real Gemma 9/9, model behaviour 4/4, verdict stability 12/12. The loop that makes it more than a nag is confirmed working against a live site.

Prize Categories

Best Use of Gemma

Gemma 3 4B is not a component of this project, it is the premise. The whole argument is that a model reading your unsent drafts has to run on your own machine, and an open-weight model is the only kind that can. Everything in the build log above is a consequence of using a small open model on a slow laptop: the grounding checks exist because the model is unreliable, and the judge-while-you-type design exists because it is slow.

I built an AI agent that fills in your flight delay claim

Arpan Patra — Mon, 14 Sep 2026 05:46:23 +0000

If your flight lands three or more hours late and it flew out of the UK or the EU, the airline usually owes you money. On a long flight that's £520 under UK rules or €600 under EU rules. A lot of people never claim it, and honestly, I get why. You have to find out when the plane really reached the gate, figure out which rules apply, and then get through the airline's claim form.

So for Anakin Forge 2026 I built Claimback, an AI agent that does that work for you. You give it a flight number and a date. It checks what actually happened, reads the law, works out what you're owed and fills in the airline's claim form. Then it stops, because the one thing it's never allowed to do is press Submit.

Watch a recorded run: https://claimback-kt20.onrender.com
Demo video: https://youtu.be/Qso_MPN9rk0
Code: https://github.com/imarpanpatra/claimback

The brief

Anakin Forge asked for an agent that reads the live web, reasons about what it finds, and then acts on it. A delay claim needs all three, which is why I picked it. The facts live on real websites, the reasoning is legal, and the action is a real form on a real airline's site.

I used Anakin for everything that touches the web, and OpenAI (with Gemini as a fallback) for the reasoning. The backend is Node and Express, and the browser part is playwright-core connected to Anakin's cloud browser.

Read: what really happened to the flight

The flight I tested with the whole way through was Air India AI162 from London Heathrow to Delhi on 9 September 2026.

The first surprise was that FlightAware only understands ICAO flight codes. It doesn't know AI162, but it knows AIC162. So the agent starts with an Anakin Wire action that looks up the airline and turns "AI" into "AIC". Then Anakin's URL Scraper fetches the FlightAware page, which has a JSON blob inside it with the last 14 days of scheduled and actual gate times.

For AI162 that said the flight was due at the gate at 00:20 and got there at 03:47. That's 207 minutes late. One source isn't proof, so the agent also checks AirHelp's flight status through Anakin Wire, and it said 207 minutes too.

A few small details mattered more than I expected:

The law measures delay at the gate, not at takeoff or touchdown. Luckily gate times are exactly what FlightAware records.
Rounding. A flight that arrives 179.5 minutes late is not three hours late. My first version rounded that up to 180 and would have told you that you were owed money.
Codeshares. If you type an American Airlines number but British Airways actually flew the plane, the rules follow British Airways. FlightAware tells you who operated the flight, so the agent uses that.

Reason: reading the law without trusting the model

Next the agent works out which laws apply. AI162 left the UK, so UK Regulation 261/2004 applies. India's DGCA rules apply too because Air India is an Indian airline, but they don't pay cash for delays. The route is 6,732 km.

Then the model reads Article 7 live from legislation.gov.uk, decides what's owed, and quotes the sentences it relied on.

A model that's right most of the time isn't good enough when money and law are involved, so it never gets the final word:

Every quote is checked against the page. If the model "quotes" something that isn't there, the UI flags it. My first version only compared the first 120 characters, so you could glue an invented clause onto a real sentence and it would still pass. Now it checks the whole quote.
Every amount is checked against a table. Claimback has a built-in table of EU261, UK261, Canadian and Indian rules. The table sets the final figures, and if the model disagrees you see a warning.

For AI162 the table and the model agree on £520. There's also a detail most people miss. On flights over 3,500 km, if you arrive between three and four hours late, the airline is allowed to pay half. So the honest answer is £260 to £520, and that's what Claimback shows.

Act: filling in a real airline form

Anakin Search finds Air India's own EU/UK delay claim form. Claimback opens it in Anakin's cloud browser, which I drive with playwright-core over CDP. The session is recorded, so there's a video of everything it did.

Real airline websites are messy, and this part took the longest:

Air India's form page never fired DOMContentLoaded in the cloud browser, so waiting for the page to "load" just timed out. I navigate with waitUntil: 'commit' and watch for the form fields myself.
Waiting for "any input on the page" didn't work either, because the header has a search box that shows up long before the form does.
Through the default connection the page sometimes came up blank and only offered "Accept All" cookies. Connecting through a UK exit (country=GB) gave me the real UK page with a "Reject All" button. That's also where a passenger on this flight would be, so it made sense anyway.
The cookie banner sometimes loaded late and covered the Submit button, so the agent checks for it again before every step.

For each page, the model gets the visible text and a numbered list of the controls on it. It answers with actions like "fill field 3 with 2100000000", and the code does the actual typing.

The rule I got wrong: never press Submit

This is the part I'm most glad I tested properly.

My first "never submit" rule was a regex on button text. If a button said submit, send, confirm or pay, the agent wouldn't click it. It worked on Air India, because their button literally says "Submit".

Then I built a set of trap pages and ran the real form filler against them in Chrome. 8 of the 10 got through. An icon-only button with no text. An image input. A button that said "Claim now". A "Next" button whose script submitted the form. A dropdown that submitted as soon as you changed it. A button that said "Next" until you typed something, and then said "Submit claim".

A regex on labels was never going to cover all of that, so the rule moved into the page itself. Before any of the airline's scripts run, Claimback injects a lock:

function lockSubmissions(watched) {
  const attempts = [];
  Object.defineProperty(window, '__claimbackBlockedSubmits', { value: attempts });
  HTMLFormElement.prototype.submit = function submit() {
    attempts.push('form.submit()');
  };
  HTMLFormElement.prototype.requestSubmit = function requestSubmit() {
    attempts.push('form.requestSubmit()');
  };
  window.addEventListener('submit', (event) => {
    event.preventDefault();
    event.stopImmediatePropagation();
    attempts.push('submit event');
  }, true);
  // ...plus wrappers around fetch, XMLHttpRequest and sendBeacon
}

await page.addInitScript(lockSubmissions, passengerDetails);

After that, the page has no way to submit a form, whatever the button says or does. Any attempt gets counted, and the agent treats it as the end of the road.

A form isn't the only way to send data, though. A "Continue" button can just call fetch() with your ticket number in the body. So the lock also wraps fetch, XMLHttpRequest and sendBeacon, and blocks any request that carries the passenger's ticket number, booking reference or email.

Every one of those trap pages is now a test that fails if anything gets sent.

The letter

The last step writes a claim letter with the flight, the times, the regulation and the amount. It's there for when a form needs a login or a captcha, or when you'd just rather send an email.

A lesson about picking the model

Near the end I moved the live server from Gemini to OpenAI and tried the smallest model first, gpt-4.1-nano. The amount still came out right, because the table checks it. The rest went wrong in ways a table can't catch. It typed the passenger's full name into the Last Name field, doubled the prefix on the ticket number, and blamed the delay on "bad weather" because it found an unrelated article about fog in Delhi.

On gpt-5-mini the same run was clean. It typed only the digits after the ticket prefix, used just the surname, noticed a disabled dropdown and left it alone, and found no evidence of bad weather.

Guardrails are great for things you can check, like amounts, quotes and sources. For everything else you still need a model that actually follows instructions.

Some numbers

A full claim takes two to five minutes and about 14 Anakin credits.
The project has 58 tests that run offline with no keys. Anakin, FlightAware and the models are faked, and the form filler tests use a real local Chrome.

Try it

You can watch the recorded run right now. It's a real run replayed step by step, screenshots from the cloud browser included. Live claims sit behind an access code because each one spends real credits.

The code is on GitHub. If there's an airline or a set of rules you think it should handle next, I'd love to hear about it in the comments.

Claimback isn't legal advice. It just does the boring research and the paperwork, and leaves the final click to you.

Building a user-isolated Gemini journal on Cloud Run

Arpan Patra — Sun, 06 Sep 2026 16:41:56 +0000

I spent a day building Waypoint for the Google Gen AI APAC Ideathon. The brief was a "Personal Gemini Journal": Firebase Auth, multi-turn Gemini, per-user Firestore isolation, and API keys in Secret Manager. This is what I built on top of that, and the things that nearly went wrong.

Live: https://waypoint-796599668154.us-central1.run.app
Code: https://github.com/imarpanpatra/waypoint

What it does

You write about your day. Gemini talks it through with you as a real conversation. When you close a session it is distilled into its bearing, one sentence naming the thread underneath the surface topics, plus a mood and energy score, themes, and the unfinished items you mentioned.

Every closed session is embedded, so you can later ask "what was I stuck on in August?" and get an answer cited to dated entries. Each session also produces publication drafts: a LinkedIn post, an X thread, a blog outline and a video idea, grounded strictly in what you actually wrote.

The isolation decision

The requirement was "zero cross-user leakage". The usual approach is a global collection with a userId field and a .where() clause on every query. That works right up until someone forgets one.

Instead every document lives under the owner's subtree, and the uid is only ever read from the verified token:

export function userRoot(uid) {
  if (typeof uid !== 'string' || !uid) {
    throw new Error('userRoot() requires a verified uid.');
  }
  return firestore().collection('users').doc(uid);
}

No route accepts a user id from a request body, query string or header. There is no code path that can be pointed at another account, because the reference is constructed from the decoded token at the point of use.

That matters most for semantic search. A shared vector index is where cross-tenant leaks usually show up in RAG systems. Here the candidate set is read from users/{uid}/sessions, so isolation is structural rather than filtered.

Server-authoritative writes

The starter rules in the codelab suggest:

allow read, write: if request.auth != null && request.auth.uid == userId;

That is correctly isolated, but it lets a client write directly to the database, which means server-side validation, sanitisation and embedding generation can all be skipped by anyone willing to open a console. My rules deny client writes entirely:

match /users/{userId} {
  allow read: if isOwner(userId);
  allow write: if false;

  match /private/{docId} {
    allow read, write: if false;
  }
}

The private/ subcollection holds the Discord webhook URL, readable by nobody on the client, not even the owner. The API returns only a masked form.

Proving it instead of claiming it

Documentation that says a system is isolated is worth very little. So the app has a /security page that runs six probes in your browser against the live deployment:

Read your own profile document. Should succeed.
Read a document under a different uid. Should be permission-denied.
Write directly to your own tree from the browser. Should be permission-denied.
Call the API with no bearer token. Should be HTTP 401.
Submit the cloud metadata address as a Discord webhook. Should be HTTP 400.
Download the page's own JavaScript and scan it for Google API key literals.

Five of the six are supposed to fail. That is the point.

SSRF, the feature that needed the most care

Waypoint can push a weekly digest to a Discord webhook you own. A user-supplied URL that the server then fetches is a textbook SSRF primitive. On Cloud Run, pointing it at 169.254.169.254 would hand over the runtime service account token.

The control is a hostname allow-list, not a deny-list, because DNS rebinding and redirects defeat deny-lists:

const ALLOWED_HOSTS = new Set([
  'discord.com', 'discordapp.com', 'ptb.discord.com', 'canary.discord.com',
]);
const WEBHOOK_PATH =
  /^\/api\/(?:v\d+\/)?webhooks\/\d{5,25}\/[A-Za-z0-9_-]{20,120}$/;

Plus https only, no embedded credentials, redirect: 'error' so an allowed host cannot bounce the request somewhere internal, and an 8 second timeout so a hung third party cannot pin an instance open.

There are eleven test cases for this, including the metadata server over https with a valid webhook path, which is the one that isolates the host check from the scheme check.

Three things that nearly went wrong

The free tier is regional

I nearly deployed to asia-south1 because it is closer to home. Cloud Run's Always Free allowance only applies in us-central1, us-east1 and us-west1. Worse, the same value sets the Firestore location, which is permanent once the database is created. Getting that wrong means tearing the project down and starting again.

A caught error still killed the process

The optional Maps secret lookup failed cleanly, logged a tidy warning, and then the service died a tick later.

Secret Manager's client builds its gRPC stub lazily and holds the promise internally. When credentials cannot be resolved, that internal promise rejects with nobody awaiting it, and under Node 22 an unhandled rejection terminates the process. Catching the error from accessSecretVersion is not enough:

const client = getClient();
await client.initialize();   // forces stub creation inside this try block
const [version] = await client.accessSecretVersion({ name: resource });

I confirmed this was the real fix rather than a coincidence by removing the process-level safety net and checking the service still stayed up.

Cloud Run gives you two URLs

A new-style service-projectnumber.region.run.app and a legacy service-hash-uc.a.run.app. Both route to the same service, and Firebase Auth will refuse sign-in on whichever one you did not add to authorised domains.

Verifying the security controls actually work

There are 31 tests covering the pure functions where a regression would be silent: the SSRF allow-list, the prompt-injection fence a user must not be able to escape, the sanitiser that strips zero-width and bidi characters, and undefined-stripping before Firestore writes.

I confirmed they catch regressions by deliberately breaking things. Disabling the SSRF host allow-list turns four tests red. Injecting a dangerouslySetInnerHTML and a fake API key is caught by the pre-deploy scanner. A test that has never failed has not been shown to work.

Prompt injection

Journal text is untrusted input. It gets wrapped in a delimiter the user cannot forge, and every system instruction states that content inside it is data, never instructions:

export function asUntrustedData(text) {
  const escaped = String(text)
    .split(FENCE_OPEN).join('[fence]')
    .split(FENCE_CLOSE).join('[/fence]');
  return `${FENCE_OPEN}\n${escaped}\n${FENCE_CLOSE}`;
}

Stripping the delimiters from user text first is what stops someone closing the fence early and escaping into instruction context.

This is mitigation, not a solution. The blast radius is deliberately small: the model has no tools, no function calling and no database write access, so a successful injection affects the text of one reply and cannot reach data.

Model resilience

No call site names a model. Everything routes through a fallback ladder that retries on recoverable upstream statuses, so one unavailable model cannot take the application down:

const DEFAULT_LADDER = [
  'gemini-3.6-flash',
  'gemini-3.1-flash-lite',
  'gemini-flash-latest',
  'gemini-3.7-flash',
];

const RECOVERABLE = new Set([429, 500, 503, 404]);

The ladder is overridable with an environment variable, so a model rename is a gcloud run services update rather than a rebuild and redeploy.

Worth stealing

Two ideas I would use again on anything multi-tenant.

Make isolation structural, not conditional. If the only way to address data is a helper that takes a verified uid and builds the path from it, there is no query left to forget a filter on.

Ship a page that tries to break your own app. The security page took an hour and it is the thing I would show first. It converts a claim into something a reviewer can check in ten seconds from their own browser.

The full source, the threat model, and the deployment steps are all in the repo: https://github.com/imarpanpatra/waypoint

#AccelerateAIwithCloudRun

Building an AI Agent That Can Actually Take Action

Arpan Patra — Sun, 30 Aug 2026 17:56:16 +0000

Adding a dependency is the least careful thing most of us do all week.

You run npm install some-utils, and you've just agreed to run a stranger's code on your laptop, in your CI, and eventually in production. The registry ships thousands of new versions an hour. Typosquats, hijacked maintainer accounts, postinstall hooks that phone home, code that ships in the tarball but never existed in the GitHub repo anybody actually reviewed, they all come through that door.

And nobody reads the tarball. I certainly don't.

So for the Agent Harness Hackathon I built Portcullis - an agent that reads it for you, and can't add the dependency without asking first.

Repo: https://github.com/imarpanpatra/portcullis

Three-minute walkthrough, if you'd rather watch than read:

The job

You ask it: can I add left-pad to this repo?

It checks whether the name is the attack before it reads any code. Then it pulls registry signals age, publish cadence, maintainers, downloads, known advisories. Then it downloads the actual published tarball into a sandbox, unpacks it, and reads what ships: install hooks, obfuscated payloads, child_process usage, and a diff against the project's own GitHub source at that tag.

Then it gives you a verdict with evidence. And if you want it added, it stops and asks, because that means a branch, a commit and a pull request on a real repository.

That last part is the whole point. Everything before it is research; that step is a write you can't take back.

How it's wired

Three pieces, and TrueForge handles everything between them.

A custom MCP server I wrote for npm intelligence. Four read-only tools over public keyless APIs: registry metadata, download stats, OSV advisories, and typosquat detection.

That last one taught me something. The obvious approach is to search the registry for the package name and look at the neighbours. It doesn't work. Ask npm's search API for expres and express is nowhere in the twenty-five results. The victim is missing from the exact query designed to find it.

So instead of hoping search finds it, I generate the names an attacker would plausibly have registered deletions, adjacent transpositions, doubled letters, homoglyphs like rn→m and probe for them directly. The downloads API takes 128 names per request and returns null for ones that don't exist, so a few hundred candidates resolve in two HTTP calls.

Pointed at expres, it finds express and rather than writing me a security report about a package I never meant to install, it stops and asks:

A git-backed skill holding the audit procedure and a Python inspector. TrueForge clones it into the sandbox on demand. The inspector is standard library only which felt right for a tool whose entire subject is install steps.

The GitHub connector does the write, and it's gated.

The whole agent is one JSON document a model, instructions, those connectors, the skill, and the config:

What TrueForge actually did for me

I wrote about 3,000 lines. None of it was the agent loop.

The harness gave me tool routing over MCP, a sandbox provisioned only when the agent needs one, the approval gate, subagents, sessions that survive the client going away, and Generative UI so the report renders as real components instead of a wall of markdown.

Ask it about five packages and it fans out one subagent each, running at once, results merged into one ranked answer:

Two of those are worth calling out.

Sandbox-as-tool. The agent loop and all credentials stay in the harness; the sandbox only executes. So the thing unpacking a stranger's tarball never sees my GitHub token. For this project specifically that's not a nice-to-have it's the reason the project is safe to run at all.

The approval gate is enforced by the harness, not by my client being polite. A gated call ends the turn. Resuming requires sending an approval event. So a client that ignores the pause doesn't get a pull request faster it gets no pull request. I couldn't have accidentally written past it if I'd tried.

I deliberately didn't add a --yes flag. A flag that pre-approves every write defeats the only claim the project makes, and it'd be the first thing anyone reached for in CI, which is exactly where nobody's watching.

Approve it, and you get the branch, the commit, and the pull request with the exact dependency line it added:

What broke

This is the useful part.

The sandbox had no bash

First real run, the agent gave me a confident, well-structured verdict on express. Registry signals, advisories, the lot.

It had never opened the package.

Every sandbox exec was coming back fork/exec /usr/bin/bash: no such file or directory. The agent tried three shell variations, gave up, and answered from registry metadata alone sounding exactly as certain as it would have if it had done the work.

The fix wasn't to find a shell that happened to exist on that image. It was to stop needing one: the inspector now exposes a callable and the skill imports it under Code Mode. Python is the one interpreter the sandbox guarantees, because the harness already runs its own client there.

The docs and the server disagreed about a path

Docs say skills land at /opt/tfy/skills/{name}. The running server logs /opt/tf/skills. My skill wouldn't have been found even with a working shell.

Small thing, twenty minutes lost, and I only caught it by reading the server's own startup log instead of trusting the documentation.

The model mattered more than the prompt

On a mini-class model, the agent skipped the tarball inspection entirely. It has registry tools that answer in a second and a sandbox step that doesn't, so it took the fast path and produced a verdict that had never looked inside the package.

Same instructions, same skill, on a larger model: loads the skill, runs the inspector, quotes the report back.

The mini model was never wrong about anything it said. That's the problem. It sounded equally certain having looked at strictly less. I now pin the model in the README and say why.

Switching is a one-line change, which is part of why this bit is easy to get wrong nothing stops you running the cheap one and believing the output.

It cried wolf on express

My first working inspector flagged express the most ordinary package in the ecosystem for URLs in package.json and a read of NODE_ENV.

That's useless. A tool that panics about express teaches you to ignore it, and then the one real finding gets ignored too. Three rounds of tightening later: express, chalk and ms come back completely silent, while esbuild reports its postinstall, child_process use and network egress all true, and the right answer there is admit with conditions, not refuse.

The regression I care about now isn't a missed finding. It's a benign package starting to produce them.

Qodo kept catching my own fixes

Every change went through a pull request reviewed by Qodo. Thirty-three findings across eight PRs.

The ones that stung were the second-order ones bugs my earlier fixes introduced:

I added extraction size caps to stop decompression bombs. Those caps then let a truncated repository tree manufacture false critical findings.
I capped severity in response. That went too far the other way and understated a proven content mismatch.
I taught the scanner to read .sh files, but forgot to extend the provenance check so shell scripts ended up half-examined.
My reconnect logic had the initial stream outside the try block. So a dropped connection skipped the reattach loop entirely: the one failure it existed to survive was the one that got past it.

And my favourite, caught the day of submission: I'd written a report template that always referenced a "limitations" card, then instructed the agent to omit that card when there were no limitations. An undefined reference in OpenUI renders as nothing at all. So clean audits express, ms, chalk would have produced an empty report on camera, while packages with problems rendered fine.

The good case was the broken one. I'd have found that during the demo.

I disagreed with two findings and said why in the thread rather than quietly changing code. One claimed the SDK returned turns newest-first; I checked against a real two-turn session and it's oldest-first. I still rewrote that code to not depend on ordering at all, which is the durable fix either way.

Would I hand this job to an agent again

Yes, but narrowly.

What makes this work isn't that the model is clever. It's that the harness gave it a place to run dangerous code, a way to reach real data, and a hard stop before the irreversible bit. Take any of those away and it's a chatbot with opinions about npm.

The part I'd still tell people to be careful about: the agent is most convincing when it has done the least work. Both times mine produced a confident wrong-shaped answer the missing bash, the mini model the output looked great. It was the tool traces that gave it away.

Code's all open source, MIT: https://github.com/imarpanpatra/portcullis
Demo: https://youtu.be/t042CZAsOuM

(Built with AI assistance, implementation and working through review findings. The design decisions, severity calibration and the calls on each review finding were mine.)