This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 4 - Ghost in the Machine.
What I Built
On Windows, a reStructuredText file came back from a loader as one single section. On Linux, the same file split at every heading.
The file used \n line endings. The loader split on PHP_EOL, which is \r\n on Windows. It was looking for a separator that wasn't there.
That is my kind of ghost: something you cannot see in the editor, but that changes what the program does.
An AI coding agent helped produce five fixes for these kinds of bugs in symfony/ai from my Windows machine. Four involved PHP_EOL being used to parse text from somewhere else. The fifth involved a probe that recognized php but not php.exe.
Those bugs inspired poltergeist, a local tool with two parts.
The scanner finds hidden characters. Carriage returns, byte order marks, zero-width spaces, non-breaking spaces, bidirectional overrides, mixed-script words and Unicode tag characters. Some tag characters render invisibly while encoding text, which makes them particularly interesting in files an agent reads.
poltergeist scan exposes those characters. poltergeist exorcise previews changes as a dry-run diff, making the hidden characters visible before anything is removed.
The model investigates platform assumptions. A hard-coded executable name, an implicit encoding or a filename with the wrong capitalization can look perfectly ordinary. The model reads the code and the reported symptom, suggests a category and proposes a small fix.
The scanner works without AI. The diagnostic part uses small, local open-weight models.
Demo
The page shows captured command output and benchmark results. It does not run inference in your browser.
From a checkout of the repository, you can generate examples and inspect them:
python scripts/make_examples.py haunted/
python -m poltergeist scan haunted/
python -m poltergeist exorcise haunted/ --eol lf
The last command previews the changes.
Code
The Python tool uses the standard library. Local model inference runs separately through llama-server from llama.cpp.
The project includes 68 tests, including a check for unexpected hidden characters in its own repository.
How I Built It
The model comparison used:
- Gemma 3 1B
- Gemma 3 4B
- Qwen2.5 3B
All three ran as 4-bit GGUF models on a CPU-only desktop with an i7-8700 and 64 GB of RAM.
Each model received code and a symptom. It had to choose one of five categories:
- Line endings
- Invisible characters
- Paths and the file system
- Executable naming
- Encoding and locale
It also had to explain the problem and suggest a fix.
There were two prompt versions. The plain prompt contained the code and symptom. The grounded prompt also exposed hidden characters, included scanner findings and supplied up to three entries from a short, hand-written collection of platform facts.
The inference client accepts only loopback addresses and ignores proxy settings. That behavior has a test. I did not disconnect the computer from the network, so I am not claiming a physically isolated setup.
The benchmark that looked too good
The first benchmark contained five cases based on the Symfony fixes and ten synthetic cases.
The platform-facts lookup classified all fifteen correctly without using a model.
That looked impressive until you considered how it was built: the facts had been written with those cases already available. It was measuring how well the lookup matched examples that had shaped it.
The lookup was then frozen in a commit. Ten additional cases were written afterward, using other languages and different wording.
On those ten new cases, the lookup scored 3 out of 10.
That became the more useful comparison.
| Approach | Correct category on 10 new cases | Fix passed keyword rubric | Median seconds per answer |
|---|---|---|---|
| Platform-facts lookup, no model | 30% | Not scored | Not measured |
| Gemma 3 1B, plain / grounded | 30% / 30% | 10% / 30% | 6.1 / 6.7 |
| Qwen2.5 3B, plain / grounded | 80% / 90% | 30% / 50% | 11.3 / 10.2 |
| Gemma 3 4B, plain / grounded | 40% / 40% | 30% / 30% | 17.5 / 20.1 |
One case is ten percentage points here. These are small exploratory results, not a general ranking of the models.
The proposed fixes were checked with a keyword rubric. They were not executed to prove that they fixed the bug.
What the results actually showed
The largest model did not perform best on these cases.
Qwen2.5 3B classified eight of the ten new cases correctly with the plain prompt and nine with grounding. Gemma 3 4B classified four correctly with either prompt.
That is an observation about this benchmark, not evidence that one model is better at software debugging in general.
Grounding helped some proposed fixes without consistently improving classification.
Gemma 3 1B went from one to three fixes passing the keyword rubric. Qwen went from three to five. Gemma 3 4B stayed at three.
On the original fifteen cases, grounded Qwen and Gemma 4B both reached 100% classification accuracy. But the lookup already scored 100% there. Those results do not establish that the models independently diagnosed the bugs.
A correct category could hide a wrong explanation.
One new case involved two filenames differing only in letter case. The smallest model chose the correct category, but explained it as a path-separator problem.
The other two models blamed line endings and incorrectly claimed that macOS uses CRLF.
The category score did not capture that failure. This is why the raw answers matter alongside the table.
The cases are small and come from one project. The benchmark is a smoke test that helps expose limitations, not proof that these models can safely repair arbitrary code.
The ghosts inside the project itself
The tool encountered the same problems it was supposed to diagnose.
Hidden characters in its own source.
The first self-scan reported 32 findings. Some intended textual escape sequences had been written as actual invisible characters. The code still ran.
Those characters were replaced with visible escapes, and a repository check was added.
An encoding failure in the test tooling.
A mutation-test script crashed when Python on Windows decoded a child process's output as cp1252. The tool checking the tool had its own platform assumption.
A benchmark process-management bug.
PowerShell variable names are case-insensitive. A $server variable overwrote a -Server parameter, disrupting cleanup and leaving a model server running.
That could have caused a later model's requests to reach the earlier server. The script now refuses to start if the port is already occupied.
A scanner that quietly skipped a file.
Windows PowerShell 5.1 saved a results table as UTF-16 with a byte order mark. The repository check caught it, but the scanner had silently skipped the file because it could not decode it.
The scanner now reports text files it could not read.
That last change matters to me: a quiet scan should not be confused with a complete scan.
Why Does Open Innovation Matter?
The material needed to diagnose these bugs can be sensitive: source code, configuration files and internal logs.
Running an open-weight model locally lets the diagnostic workflow use that material without requiring a hosted inference service. The client code enforcing the loopback restriction is available to inspect.
Open models also made comparison practical. The same cases could be run through three different models, with the answers retained for inspection. There was no need to assume the biggest model would be the best fit.
The runs used temperature zero, a fixed seed and a JSON output schema. The 1B model was run twice across all 25 cases and both prompts. All 50 answers matched, including their explanations, fixes and token counts. Timings differed.
Only the smallest model received that repeatability check. It does not establish reproducibility across different machines or llama.cpp builds.
The useful outcome was being able to inspect where the approach worked and where it failed.
My Agent Session
An AI coding agent, Claude Code, wrote most of the code and tests for poltergeist. I ran the benchmark in my own terminal on Windows. The agent also produced the five Symfony AI fixes that inspired the project.
Agent assistance did not prevent mistakes. The hidden characters, PowerShell variable collision and skipped-file behavior all appeared during development.
The project takes untrusted text as input. Its scanner can expose some suspicious characters before an agent reads them, but it is not a complete prompt-injection defense. An instruction does not need invisible characters to be malicious.
The Spirit of It
The carriage return comes from a physical machine moving a print head back across a page. Decades later, that old behavior can still break a modern loader because two systems disagree about what ends a line.
That connection to the past became the project's Halloween theme: invisible remnants and assumptions haunting otherwise ordinary files.
The scanner makes some of them visible. The local model suggests where else to look.
Sometimes it points at the wrong wall. Keeping those failures in the write-up is part of showing what the tool actually does.
Prize Categories
Best Use of Gemma
The project runs Gemma 3 1B and 4B locally through llama.cpp and compares their diagnostic outputs with another open-weight model and a non-model lookup baseline. The results include both the improvements and the cases where Gemma struggled.
Top comments (0)