What I Built
Every spec I have ever read answers the questions its author thought of. The expensive bugs live in the ones they didn't.
Here is a real sentence from a small checkout spec I wrote for this project:
If the gateway responds with a 500 error, the service retries the charge up to 3 times with exponential backoff.
Reasonable. Probably even tested. Now picture week three of a sprint, when someone asks: what if the gateway charges the card, and the response never arrives? The retry fires. Does the customer pay twice? Nothing in the docs says, because nobody asked.
That missing answer is what I went hunting for. I call it dark matter: you can't see it directly, but you can find the hole by looking at everything around it. The Dark Matter Engine is an offline scan that reads your docs, asks "what could go wrong here?", and reports the questions your documentation never settled. It runs at 2 AM. A local Gemma does the reading. Plain Python checks its work.
TL;DR. The Dark Matter Engine is an offline scan that reads your docs and lists the failure questions they never answer. A local Gemma 3 (4B) model proposes and judges scenarios, and plain Python checks every quote it relies on. A Task Scheduler job runs it at 2 AM. On my nine planted test pairs the default check flagged 6 of 9 omissions and wrongly flagged 2 of 45 documented behaviors; a stricter mode flagged 9 of 9 but wrongly flagged 23. On a real design document it found one genuine unanswered question and produced eight findings I had to throw out. It's a prototype that produces candidates for a human to review, not verdicts. MIT licensed, Python standard library only.
Who gets to be lazier: anyone who reviews specs or design docs and would rather not brainstorm every failure case by hand. Point it at a folder of Markdown or text. For each behavior the docs describe, it proposes a failure scenario, searches the docs for an answer, and reports whether the text actually establishes one.
My demo is two versions of the checkout spec. Version 1 documents retries on a gateway error and says nothing about a lost response. Version 2 adds one section on idempotency keys. For the scenario "the gateway charges the card but the response is lost and the client retries":
- Version 1: Partially answered. Gemma cites only the sentence about the normal payment call and asks the right question: how does the system ensure the order is charged exactly once when retries are involved? The spec never says, and the engine flags it as unresolved.
- Version 2: Answered, with the exact sentence and its line number: the service returns the stored result and never charges the order a second time. The verdicts are generated from the specification text, not hard-coded for these two versions.
A warning before the screenshots: this is a prototype that produces candidates for a human to review, not verdicts. On a real design document from another project it found one genuine unanswered question and also produced eight findings I had to throw out. The details are below, because they're the most interesting part.
Demo
To reproduce it yourself:
ollama pull gemma3:4b
python dark_matter.py demo --quick
That scans the checkout pair and runs the benchmark on it (about 5 minutes on my laptop), then writes one results page you can open in a browser. It's quicker than the 8.5-minute scan I mention below because the demo uses pre-written scenarios, while a plain scan first has the model propose scenarios from every section. I also left the full result files from my runs in the repo.
Code
Emmimal
/
dark-matter-engine
Offline scan that finds failure scenarios your docs never answered, using local Gemma via Ollama
Dark Matter Engine
An offline scan that finds the failure questions your documentation never answered.
Point it at a folder of Markdown or text docs. A local Gemma model (through Ollama) proposes failure scenarios from the behavior the docs describe, local search looks for evidence that each one is handled and you get one HTML report. Run it on a schedule and the report is waiting in the morning.
A potential gap means the inspected files did not establish the behavior. It is a question for an engineer to check, not proof that the behavior is undefined.
Built for the Hacktoberfest Open-Source AI Challenge, Week 3 (Slow and Steady). Python standard library only, no pip install. Ollama is the one external runtime.
Run it
ollama pull gemma3:4b # check the exact tag with `ollama list`
python dark_matter.py demo --quick
demo --quick scans the checkout spec pair and runs the benchmark…
MIT licensed. Python standard library only; Ollama is the one external runtime. The logs, result files and the offline recording are in the repo's evidence/ folder. Model: gemma3:4b, digest 68ceab7e468a.
How I Built It
I used Google's open-weight Gemma 3 (4B), run locally through Ollama, with temperature 0 and a fixed seed. Everything else is standard-library Python.
The steady part: how it avoids inventing gaps
Ask a language model "what's missing?" and it will cheerfully make things up. So the work is split:
- Gemma proposes scenarios from sections the docs actually contain.
- Plain lexical search fetches evidence before the model judges anything.
- Gemma classifies: Answered, Partially answered or Not established. (A Contradicted status exists in the code, but my corpus has no contradiction case, so I haven't tested it.)
- Python checks that every quote exists in the retrieved text. A quote it can't find makes the finding Unverified, never a gap.
- An Answered claim gets a second check. A Not established result gets a second search with a hand-written synonym list, then is classified again. Gemma decides what the text establishes. Python decides whether Gemma is allowed to say so.
The slow part: does it work?
I wrote nine small spec pairs, each with one planted omission. That's 54 scenarios: 9 planted gaps and 45 documented behaviors. The expected answers live in a separate file the scanner never reads. The first two pairs were used to tune the prompts. Three more were run once after tuning, with expected answers written beforehand but not committed before the run. For the last four, I committed the expected answers before running any model on them (commit d11c484). I wrote those four after the worker-crash failure below, so they deliberately include similar traps. I changed no prompt or engine logic after seeing results on them; I only added per-scenario reporting.
| Method | Planted gaps flagged | Documented behaviors wrongly flagged |
|---|---|---|
| Gemma, default check | 6 of 9 | 2 of 45 |
| Gemma, stricter two-step check | 9 of 9 | 23 of 45 |
| No-model word-coverage baseline (threshold 0.5) | 9 of 9 | 32 of 45 |
| Same baseline (threshold 0.4) | 6 of 9 | 19 of 45 |
"Flagged" means Not established or Partially answered. The baseline is a simple rule: how many of the scenario's words appear in the top three retrieved passages. I'm showing its whole sweep, not one hand-picked setting: thresholds 0.6, 0.7 and 0.8 also catch 9 of 9, but flag 37, 42 and 45 of the 45 documented behaviors.
At the same 6 of 9, the baseline wrongly flags 19 of 45 and the default check flags 2. These are two different methods, so this compares outcomes on my test set. It doesn't show that Gemma is better in general, and it says nothing about stronger retrieval than my simple one. What it does show is what the model contributes here: far fewer false alarms.
These are single runs on a small corpus that I partly built while tuning, so read the counts, not percentages. The last four pairs are the stronger evidence, but even they are not an independent evaluation. Two runs on the same machine gave identical results, which shows determinism, not robustness.
What a run costs, on one Windows laptop (I didn't record the hardware):
| Run | Time |
|---|---|
| One real scan, one spec, the model writes its own scenarios | about 8.5 minutes |
demo --quick, checkout pair, end to end |
about 5 minutes |
| 24 benchmark scenarios, default check | about 20 minutes |
| The same 24 scenarios, stricter two-step check | about 35 minutes |
| API fees | $0 |
Where it breaks
My favorite failure. For "a worker process crashes while running a job", the default check let Gemma quote the sentence about retrying a job that raises an error, and call the scenario Answered. The quote was real. The inference wasn't. Retrying after an error says nothing about what happens when the whole process dies.
My citation check can't catch that, because the sentence exists. A yes/no relevance check I added didn't catch it either. The stricter two-step check does, along with two other missed gaps, but it also flags about half of the documented behaviors. So I left the default alone instead of crowning whichever mode looked best on my own data.
What I learned: when a small model checks its own work, how strict you make the check is a product decision. It's a dial between missing gaps and crying wolf, and no setting of it is free.
A real spec I didn't write
To see whether that failure was just a quirk of my own test pairs, I ran the default settings once, with nothing tuned, on a design document from another project: ADR-22 from NATS, "JetStream Publish Retries on No Responders" (Apache-2.0; I did not copy it into my repo). The engine reported 12 scenarios and 0 potential gaps. (Its report counts only "Not established" as a potential gap; the one "Partially answered" finding sat in a separate section.) Then I read every finding against the document. The labels are one person's judgment, and they're in the repo.
- 3 were correct and properly supported.
- 1 was a genuine unanswered question: the ADR says what a publish returns after a timeout or after a last "no responders" error, but not what happens if the last attempt fails with some other error. The engine labeled it Partially answered, not a gap.
- 5 were marked Answered, but the quote they cited didn't support the claim.
- 3 were scenarios that contradict the spec or aren't about it. One said the client "does not attempt to send the message again", the opposite of the ADR's whole point. Another was about a Go timer failing to start. So one real finding and three correct confirmations, buried among eight that don't hold up. It's the same "real quote, unsupported inference" failure, on a document nobody tuned for, plus the scenario generator inventing hazards. This is one document labeled by one person, not a benchmark.
Other caveats
- My first run on the first two pairs went badly: Gemma labeled one planted omission "Partially answered" and called the other "Answered". Strictly, it found neither. I then added the relevance check, partial-answer reporting and a second metric ("unresolved", meaning Not established or Partially answered), using only those two pairs. That metric was defined after the first run.
- On a plain scan, where the model writes its own scenarios, it invents requirements the spec never states. In my scheduled run, one flagged "gap" was a false alarm. Treat generated scenarios as prompts for review, not findings.
- One export scenario in my own corpus is arguably ambiguous and got flagged in both versions. I left it as written.
-
Markdown and text only. No comparison with closed models.
Bugs I hit along the way
Gemma copied my prompt's placeholder. In an early real run, the report's "question to resolve" was literally the example text from my own prompt. The prompt no longer contains placeholder text, and any question that shares no words with its scenario is replaced by a plain fallback.
My relevance check was too literal. It rejected the right answer to the crash scenario in version 2 because the spec says "lease expires", not "crash". I loosened it, and the looser check then accepted the wrong quote in version 1. That is how I ended up with the dial described above.
-
Windows Task Scheduler caps the command at 261 characters. My first scheduled command was far longer, so the engine now writes a short wrapper script and schedules that. Task Scheduler also refuses to start on battery by default, so I had to change that setting for a 2 AM run to work on a laptop.
Why Does Open Innovation Matter?
The docs stay on my machine. Specs for systems that don't exist yet are exactly what I wouldn't paste into a hosted API. The engine refuses any model host that isn't localhost. It also patches Python's socket calls so a non-loopback connection or DNS lookup inside its own process raises an error, and a test checks that. That covers the Python process only, not what Ollama does.
It works offline. I turned Wi-Fi off, used a failing
ping google.comto show it was off, and ran the demo. It finished with the same result. The ping only shows the network was down; the completed run is the evidence.No API fees.
The weights are pinned. I recorded the model digest, so the benchmark can be reproduced instead of drifting under me. The model tag is a command-line flag, though I only benchmarked
gemma3:4b.
I never compared it with a closed model, so I'm not claiming this beats one on accuracy. My argument is privacy, offline use, cost and control.
The Lazy Payoff
I do not run it. After one setup command, Windows Task Scheduler does.
python dark_matter.py schedule ./docs --at 02:00
On Oct 22 the task fired by itself at 2:00 AM and had written the report and a log line two seconds later. That was fast because the docs hadn't changed, so it reused the saved result. When they do change, a real scan takes its time: about 8.5 minutes for one spec on my laptop. Nobody is waiting, so slow is fine. A sloth would approve.
After setup, the number of things I do each day is zero. As for the payoff: I ran it on my own spec corpus and, once, on a real design document. That run surfaced one genuine unanswered question, and also eight findings I had to read and throw out. So I can't honestly tell you how many hours it saves a team.
What happens when the gateway charges the card and the response disappears? Version 1 of my spec had no answer. Version 2 did, after I added the idempotency behavior.
That is the gap Dark Matter Engine is built to surface. It can't promise to find every missing answer, and it still produces false alarms. But it can read documentation overnight and leave me questions worth reviewing.
The sloth moves slowly. At 2 AM, nobody has to remind it to get started.
Prize Categories
Best Use of Gemma: a local Gemma decides whether each quote establishes the required behavior. Python validates every citation and every output format. I kept those jobs separate on purpose.
Top comments (0)