DEV Community: Fabio Nonato

A Fruit Fly Brain Plays Doom

Fabio Nonato — Tue, 22 Sep 2026 16:18:52 +0000

Imagine I opened this post by telling you I got a fly to play Doom. Would you believe me?

You should not. The thing playing up there is not a fly, but its wiring diagram comes from one. It is a recurrent network built on a real fruit fly connectome: 49,393 neurons and 9 million synapses, all frozen, with one learned gain per synapse. An ordinary convolutional network feeds it the game frames and an ordinary MLP turns its activity into moves, and those two hold more than half of the parameters. So read every result in this post as "a network constrained to the fly connectome" and never as "a fly". Yes, the title breaks that rule. That is the hook, and the rest of the post is the correction.

Now, here is the thing: I did not write the code, the infrastructure or the first drafts of the docs. All of it came out of the Strands harness, an open-source coding agent from the Strands Agents team at AWS that you run from the terminal. It keeps its own state on disk between sessions and runs tools, tests and cloud deployments on its own. Here it wrote the pipeline that turns the fly's wiring into a network and the five ordinary Doom agents that served as teachers. It wrote the trainer that taught the fly-wired network to copy them, and the second training stage that tried to push it further. It also wrote the cloud setup that ran a GPU fleet in three AWS regions, and it recorded the footage. Five days, 28 commits, about 3,300 lines of Python and shell, twelve fine-tuning runs and three control runs.

What I actually typed

The harness keeps its own session state on disk, so I went back and read my side of the transcript. It is short. It started with one prompt, typos and all:

I just saw this: https://huggingface.co/mlabonne/chessfly

I want to make the same thing but to play doom! Look at the skills and memory and have a set up ready for us to train it using my fnp3 aws accountl. Also create a infographic style tutorial on how it works, both the conectome model as well as the training. Use this current directory as a infra/cdk for training and training source and data ETL pipeline source. Once we complete all, we should have this is a private github repo. If you find a way to start training while I'm away, go for it!

Across the seven sessions that followed, this is most of what I said:

Go for it!

Can you check how are we doing on the training of doomfly?

Do you have suggestions to speed up the training process?

how do I use tb to see all the runs so far? (tb is TensorBoard)

Let's stop and revert all this we should not pursue this anymore

Sorry, wrong session!

"Let's stop and revert all this" was me killing an easter egg I had asked for an hour earlier: a Doom mod with a fly paw for the player's hand and Stan the frog, the Strands mascot, as the enemies. The harness reverted the whole thing, previews and tests included, and went back to the training run. "Sorry, wrong session!" came right after I pasted a request meant for a different agent, about a different project.

I ran all of it from the terminal with the strands command, and I changed the model underneath it as I went. The first sessions ran on Claude Fable 5.1 on Amazon Bedrock. By the last day the same sessions were running on GPT-6 Astra at the highest reasoning effort. The transcript does not care which model is behind it, and neither does the repo.

Between my sentences

Everything else in the transcript is the harness at work, and reading it back showed me how much the Strands harness worked for me.

Each session persisted to disk, and the next one opened with a summary of the last. Seven sessions over five days read like one conversation, and I could close the laptop at night and pick up in the morning where it left off. Long tool outputs were paged out of the conversation as it grew and fetched back when the agent asked for them. About two hundred of those are still sitting on disk, and without that mechanism the logs from day one alone would have ended the project.

Background tasks paid off on the last day. While a two-hour evaluation ran on my Mac, the same agent widened the model's output from 22 Doom actions to 23 for the next experiment and wrote nine tests for it. It also confirmed that the widened model played the old scenarios bit-for-bit the same as before. The repo has an AGENTS.md that says never push, launch, terminate or delete without being told, and the harness asked every time. "Go for it!" was me answering. It also loaded my own writing and verification skills and the MCP tools I already use, which is why the docs it produced read like mine.

None of that is exotic on its own. What I had not seen before is all of it in one command, with nothing to wire up. The same agent is two lines of Python if you want it inside a script, and after this week I would not think twice about starting a project that way.

The part I trust

An agent that returns a plausible positive result is hard to trust. This one returned two negative results, with the run prefixes and standard deviations attached, because its rules say that negative results go in docs/ and not in the bin.

A short map of what the harness did, so the two results make sense. It first trained five ordinary Doom agents with PPO, a standard reinforcement learning method, one per scenario. Those are the teachers. It then distilled the teachers into one fly-wired network, the student, by having the student imitate their moves. The student ended up playing about as well as its teachers. The question the project was built for was whether a second stage of reinforcement learning, GRPO, could push it past that.

First, it could not. Twelve GRPO runs, one setting changed per run, and not one beat the student by more than the noise in the evaluation. GRPO keeps the new policy close to the student with a penalty. The penalty strength that stopped the policy from wandering into worse play was the same strength that stopped it from improving at all.

Second, and this one stings, the connectome does nothing measurable. The harness suggested and built two controls. One keeps the same graph but shuffles its edges, so every neuron keeps its degree and its sign and the biology is gone. The other removes the neuron layer entirely.

Backbone	Parameters	Training steps per second	Doom scores
Real connectome	48.2 M	646	baseline
Shuffled edges, two seeds	48.2 M	1,454	same, within noise
No connectome at all	27.7 M	14,686	same, within noise

One scenario as an example, ten episodes each: on defend_the_center the real connectome scored 18.6 ± 1.8, the two shuffled graphs 19.0 and 18.4, and the model with no connectome 18.3. The other four scenarios look the same. The fly wiring makes training 22 times slower and buys nothing on these five scenarios. That is why the caption says "wired like a fruit fly's brain" and not "a fruit fly's brain". The harness wrote that vocabulary rule into AGENTS.md on day two, before either result existed. It would have been a hard rule to write after a result I disliked.

What's next

Scores on the five scenarios stopped moving, and three of them sit at their ceilings, so the next test is a real level. The student already plays Freedoom II MAP01 zero-shot. With its aiming prior it survives twice as long as random and picks up items. It still dies in eight of ten episodes and never leaves the first two rooms. The wider output and a slot for a sixth scenario are done and tested. Next comes GRPO on that map with all three backbones: real, shuffled and none. If the wiring is ever going to matter, it will be where the policy has to learn rather than imitate. If the network learns anything there, the plan is fly versus fly in a duel.

The code is public at nonatofabio/doomfly-rl, and the trained checkpoints are on Hugging Face with the connectome file they need. If you want to watch a network shaped like a fly play Doom, start there.

Original post: Fabio's Blog

AI Generated pixel art needs a build system, not better prompts

Fabio Nonato — Mon, 07 Sep 2026 14:37:48 +0000

I rebuilt my tiny homage to Warhammer 40k this week. The original, Hive City Rampage, was a Python/Pygame (monstrocity), top-down grimdark shooter that ran but was a prototype. Now we have a sequel, Ashgate Siege, is Godot 4: gothic isometric, two missions, built for Mac and Android.

I have a confession though, the 95k lines of GDScript, 8 commits, roughly five hours was written by an AI agent under my direction. The sprites are generated too. I'm disclaiming that up front because the interesting part is what came out of it.

The easy part was the code

This is the simple finding. An engine port is exactly the shape of coding my agents were very good at: the target is well documented, the semantics known, and correctness is checkable by running the thing. Godot 4 plus GDScript is heavily represented in training data for any model. Zero agent struggle.

The hard part was four pictures of the same orc

Here's the hill I'm whiling to die on: for AI-assisted games, 2D sprite games are harder than 3D.

That sounds backwards, so: in 3D, the engine guarantees coherence. One mesh, one material, one light rig, and every frame of animation is consistent because it's the same object being transformed. The renderer is doing the work.

In sprite based games there is no shared object. Each sprite is an independent generated map of pixels. So:

Frame 2 of a walk cycle can be a different character than frame 1.
The light source can move between sprites in the same scene.
Proportions drift. Your massive size boss shrinks.
Limbs get cropped at cell edges.

None of that is caught by anything automatically. It just ships, and the game looks odd, like a badly executed collage.

Prompts as interface specs

The fix was to stop treating generation as commissioning art and start treating it as calling an API with a strict schema. Every prompt pins the geometry:

Production game sprite sheet: exactly 4 columns x 3 rows, 1536x1024 canvas, equal cells, solid pure magenta #FF00FF background for color-key import. NO text, shadows, smoke, or border. Every figure entirely within its own cell with ample margin; feet centered at same baseline in each row. All figures face screen RIGHT in three-quarter isometric view [...] Four columns are four coherent walking-cycle poses: left foot forward, passing, right foot forward, passing.

Each sentence has to do some work:

Magenta #FF00FF, not transparency. Alpha comes back unreliable. A color key is deterministic to strip.
Explicit grid and canvas. The importer crops on fixed coordinates. If the grid drifts, every sprite is off.
"feet centered at same baseline in each row." Without it the character bobs while walking.
Named poses. "Walking animation" returns four unrelated drawings. Naming the four phases is what makes them a cycle.
Negative constraints. "No text, shadows, smoke, or border" because models add decoration that breaks the silhouette.

The importer is a compiler, and tests are CI

If the prompt is a spec, something has to enforce it. Generated atlases go into art/, an importer crops and scales them into runtime assets, and a native test suite checks the result: 2,972 reference and combat assertions, 17,112 seam combinations, plus a 600-frame combat simulation. make test runs it.

The seam check is the one most important. It takes the belt pixels from the leg sprite, offsets them by the waist socket, and counts how many land on opaque torso pixels. Under 32 and the pose fails. Every view, every clip, every frame, every facing, every gait: 17,112 combinations.

That catches the failure mode that makes sprites look like a collage, which is a regenerated part that no longer meets the part next to it.

Then stress testing the mechanism

It is cheap to claim your tests work. So, I reimplemented the audit standalone, outside the engine, then fed it deliberately corrupted atlases to find out what it actually catches.

Baseline reproduces: 17,112 combinations, zero failures.

Then the damage:

Character 4% smaller → FAIL, 352 bad combinations
Character 8% smaller → FAIL, 6,728
Alpha edge eroded 2px → FAIL, 3,036
Torso shifted 6px down in the cell → PASS

Sweeping it: scale drift gets caught at 4% and up, and slips through at 3%. Vertical translation is never caught. I pushed it to 14px and it still passed, because shifting the torso down actually raises the minimum overlap from 59 to 249. The check gets happier while the sprite gets worse.

The seam audit measures whether two parts still meet. It says nothing about whether the pair sits correctly in the cell, because the waist socket moves with the torso. Catching that needs a different check, anchored to the cell rather than to the neighbouring sprite.

I would rather know the shape of the hole in the tests than believe the suite covers everything. The standalone auditor is tools/seam_audit.py in the repo. Plain Python, PIL and numpy, no Godot required, so you can run the numbers above yourself.

What I'd tell you to steal from my repo

Color-key over alpha. Deterministic beats convenient.
Pin the grid numerically in the prompt and crop on those exact numbers.
Name every animation phase. Never say "walk cycle" and hope.
Baseline your silhouettes and diff them. This is the whole safety net.
Keep source atlases and prompts in the repo. Regeneration is a build step, so its inputs are source code.

Try it

Mac and Android builds are on GitHub Releases. They're previews: macOS is ad-hoc signed and not notarized, Android uses a debug key, so both will warn you. Source builds with make run if you have Godot 4.3.

https://github.com/nonatofabio/hive-city-rampage-ii

How to give Claude "Long Term Memory" of your local files (No Docker required)

Fabio Nonato — Wed, 10 Dec 2025 07:29:48 +0000

The Problem: RAG is too hard

I have a folder of 50+ PDFs—technical specs, old logs, and some... interesting declassified government documents.

I want to open Claude and ask: "What happened in the 1952 sightings?" and have it answer using my files.

The standard advice for this is a nightmare:

Spin up a Docker container (Chroma/Qdrant).
Write a Python script to parse PDFs (LangChain/LlamaIndex).
Set up an API server to bridge it to the LLM.

People turned "Ctrl+F with semantics" into a distributed systems architecture.

The Solution: `local-faiss-mcp`

I got tired of the bloat, so I built local-faiss-mcp. It’s a CLI tool that runs 100% locally.

No Docker: Just a Python script.
No API Keys: Uses local embeddings (all-MiniLM-L6-v2) and FAISS.
MCP Native: Connects directly to Claude Desktop.

Here is how to set it up in 5 minutes.

Step 1: Get some "Cool" Data

To test a RAG system, you need real-world data. Let's use the ODNI Preliminary Assessment on Unidentified Aerial Phenomena. It’s declassified, text-heavy, and full of weird military jargon—perfect for testing semantic search.

Create a folder: mkdir ~/ufo_docs
Download this PDF: ODNI Preliminary Assessment 2021 (PDF)
(Optional) Throw in the CIA Simple Sabotage Manual just for chaos.

Step 2: Install the Tool

It’s a standard Python package.

pip install local-faiss-mcp

Step 3: Index the Docs (The Magic Part)

We used to have to write ingestion scripts. Now, we just use the CLI.

Run this command to recursively index your new UFO folder:

local-faiss index "~/ufo_docs/**/*.pdf"

You’ll see a progress bar as it:

Reads the PDFs (using pypdf).
Chunks the text.
Calculates embeddings on your CPU.
Saves a .index file to your disk.

Done. You now have a vector database.

Step 4: Connect to Claude

Open your Claude Desktop config file:

Mac: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Add this block:

{
  "mcpServers": {
    "local-faiss": {
      "command": "local-faiss-mcp",
      "args": ["--index-dir", "~/ufo_docs"]
    }
  }
}

Restart Claude. You should see a little "plug" icon indicate two tools are loaded: ingest_document and query_rag_store.

Or use claude in the terminal directly.

Step 5: Ask Questions

Now for the fun part. Open Claude and ask:

"Using your local faiss knowledge: Does the Pentagon's refusal to answer questions in the UAP Hearing count as 'General Interference with Organizations'?"

Claude will:

Call query_rag_store with your question.
The CLI will search your FAISS index and return the top N chunks.
Claude will synthesize the answer with citations.

Advanced: Enabling "God Mode" (Re-ranking)

If your dataset is huge, standard vector search can sometimes be "fuzzy." You might ask about "Project Mogul" and get results about "Project Grudge."

In v0.2.0, I added Re-ranking. This uses a second, smarter model (CrossEncoder) to double-check the results before giving them to Claude.

Update your config to enable it:

"args": ["--index-dir", "~/ufo_docs", "--rerank"]

Now the accuracy is much better, running entirely on your laptop.

Summary

We don't need Kubernetes to chat with a PDF. We just need a good CLI.

If you try this out with your own docs (or the UFO files!), let me know how the re-ranking performs for you.

🔗 Repo: github.com/nonatofabio/local_faiss_mcp