DEV Community: Lượng Lê

I taught my game generator to make cozy farms, drawing every tile in Python. Here's what broke along the way

Lượng Lê — Wed, 21 Oct 2026 15:16:17 +0000

Spawn Point turns one sentence into a playable Godot 4 game. A small fine-tuned model writes the game design as JSON; plain Python and tested Godot components do everything else. This month I pointed it at one genre and tried to make it good: the cozy farm.

Type spawnpoint create "nông trại mùa đông, trồng bắp cải, có cáo" (a winter farm, cabbages, a fox) and you get this kind of thing:

There are no image files anywhere in that picture. Every tile, tree, chicken and cabbage is drawn by Python at build time. Here is how, and the parts that went wrong.

1. The ground is one picture, not tiles

My first instinct was a tileset with autotile rules. Instead, the whole map's ground is painted as one image:

The level is a grid of characters (. grass, : path, ~ water).
Each terrain type becomes a pixel mask, upscaled 16x and blurred, then thresholded with a little noise. That one trick gives rounded corners and bumpy, hand-drawn-looking edges for free.
Outlines are just "pixels outside the mask that touch it", in a darker colour.
A cliff is "water pixels within 6 px below land". Islands get a brown cliff on their south side automatically.

Tufts, flowers, lily pads and reeds are sprinkled per cell with a seeded RNG, so the same farm always looks the same.

The same code paints four seasons by swapping a colour table and the tree list:

And in motion, winter and the desert:

What broke: I used scipy.ndimage for the blur and the distance transform. It worked on my machine and failed in CI, because scipy was never a dependency. I rewrote it with numpy (a separable blur and erosion), and the numpy erosion gave the map a thin light frame, because it treated "outside the picture" as land. The fix was padding with mode="edge".

2. Walking behind trees needs two small engine features

A top-down farm looks flat until the farmer can walk behind a tree. That needed:

Y-sorting: entities lower on screen draw in front.
Footprints: each object stands on the bottom of its drawing and only a small box at its feet collides. The collision used to be traced from the whole sprite, so a tree's crown blocked you.

Both became one-line options in the game graph ("y_sort": true, "base": [22, 14]).

3. A season, not a level

The farm became a season of a few days: 6:00 to 22:00, gold light in the evening, dark at night. You sleep in the farmhouse bed (or drop off in the fields), and overnight the crops you watered grow. Some days it rains and every crop stays watered; a winter farm gets snow instead. Seeds cost money, animals must be fed before they give eggs or milk, and a workbench sells tool upgrades.

The farm also saves itself to user:// (IndexedDB in the browser): the day, the money, every plot, the animals, which trees you chopped. Close the tab, come back, carry on. Two details that took a bug each:

The GIF recorder and the playtest bots must never touch the save, or the next recording starts from wherever the last bot left off.
R (restart) has to clear the save, or "restart" just reloads the same day.

4. The bug that made my farmer walk backwards

After recording the demo GIFs, someone pointed out the farmer was moonwalking: walking left while facing right.

The sprite flipping was fine. The GIF autopilot set every player's aim to "the nearest enemy", which is right for a shooter. On a farm, the nearest enemy was a crow, so the farmer kept facing the crow while walking away from it. A farmer with a sword had the same bug in real play, because sword aim followed the mouse. The fix was two lines: a farmer always faces where it walks.

Facing the right way now, and fishing from the pond's shore:

5. The model kept ignoring the fox

The model got "a farm with crows that eat the crops" right. It missed pests mentioned in passing: "có cáo" ("there's a fox"), "with a raccoon around". I wrote a sealed set of 16 such prompts (before any training) and the released model got 12.

The first training patch fixed the fox, then put pests on peaceful farms and started ignoring named farmers ("a cat farmer" came back as "Farmer"). The cause was in my synthetic data: every new example used the default farmer and almost all of them had pests. The model learned exactly that.

The second patch used half named farmers, more explicit "no pests / peaceful" farms, and more replay of the previous patch. Results, fully-right prompts:

set	v5.1	v5.2
sealed: farm pests named in passing (16)	75%	100%
30 real prompts (4-bit build you download)	80%	83%
sealed shooters (4-bit build)	88%	94%

v5.2 is a prompt or two behind on two dev sets, which is within my run-to-run noise. After release, "nuôi gà và câu cá" ("raise chickens and fish") came back once with Fish as a pest, so the normalizer now drops farm animals and fish from a farm's pest list. Some fixes belong in code, not in the weights.

6. Testing on a phone without a phone

I checked the touch controls with headless Chrome at phone size. My first script dispatched synthetic PointerEvents at the on-screen buttons, and nothing happened. The game was fine: the touch code calls setPointerCapture(), which throws for a pointer id the browser never saw. Real input through the DevTools protocol worked: the stick walked the farmer to the door and ACT went inside. The phone test did find one real problem: the seed buttons covered the middle of the screen. Seeds are now picked by tapping the hotbar.

Try it

pip install -U spawnpoint-studio
spawnpoint llm up      # serves the model (12 GB NVIDIA GPU + Docker, or Ollama)
spawnpoint create "a cozy autumn farm with pumpkins, watch out for crows"

Or play the demo farm in the browser, phone included: leluong141996-dev.github.io/Spawn-Point.

If you've generated art procedurally, what was the trick that made it stop looking generated? For me it was blurring a mask and thresholding it with noise.

My first fine-tuning patch made my model worse. The second one, with fewer examples and half the learning rate, fixed it

Lượng Lê — Tue, 20 Oct 2026 18:40:10 +0000

Spawn Point is an open-source game generator: a fine-tuned 7B model turns a sentence ("a platformer where a ninja rescues his mother") into a JSON game design, and plain Python draws every sprite and assembles a playable Godot 4 game. I recently shipped model v5.1 and Spawn Point 0.3.3. This post is about what I learned fixing a regression, and about a decision to make the project pixel art only.

The regression I shipped on purpose

v5 added side-scrolling shooters and a per-genre output format. It was clearly better on the new prompts, but a bit worse than v4 on my older hand-written sets (30 real prompts: 87% to 80%). The cause was my last training patch. It taught the model about enemies that are implied rather than named ("a tank shooting monsters"), and the model over-corrected: when a prompt named the player only indirectly, it answered with the default "Hero".

I released v5 anyway, wrote the weakness into the model card, and planned a patch.

Patch 1: more data, same learning rate. Mixed results.

My first v5.1 attempt added about 730 new examples (half of them naming the hero indirectly) on top of 800 replayed old rows, at the learning rate I'd used before. Result: the 30-prompt set and two dev sets improved a little, but my sealed shooter set, the cleanest benchmark I have, fell from 88% to 69%, and a second sealed set slipped too. Three prompts broke, and not because of the Hero problem: a wrong archetype, a genre confusion, a renamed rescue target. It was not a win, so I didn't ship it.

Patch 2: less of everything

The second attempt used about 370 new examples, 1,080 replayed rows, and half the learning rate (3e-5). Fully-right prompts, hand-written English and Vietnamese, whole pipeline:

set	v4	v5	v5.1
30 real-world prompts	87%	80%	87%
older sealed set (12)	67%	58%	75%
set of 16 (farms, tower defense...)	75%	75%	94%
sealed shooter set (16)	56%	88%	75%
newest set (12)	92%	92%	83%

Two honest caveats. Run-to-run noise at temperature 0.4 is one to three prompts per set, so a single row proves little; look at the pattern. And the shooter and newest sets still dip at full precision. The 4-bit AWQ build you actually download scored 88% on the sealed shooter set and 80% on the 30 prompts, which says as much about noise as about the model.

What I'd take from it:

A patch can fix one behavior and break another. Re-run every evaluation set after every patch, not just the one you were targeting.
Gentler beat bigger. Fewer new rows and a lower learning rate recovered the old sets without losing the new ones.
Keep a sealed set you don't look at while tuning. It is the only reason I caught that patch 1 looked fine on the dev sets.

Pixel art only

The other decision: Spawn Point used to offer two looks, pixel art and a smooth "cartoon" style. I'm dropping the second as a goal and focusing on pixel art. A prompt that asks for cartoon art now still gets pixel art, and the cartoon engine stays only for hand-written graphs. The practical reason is time: every new monster or item had to be drawn twice. The better reason is that a small project looks more coherent with one strong style.

So 0.3.3 is a pixel art release:

Six new monsters: a biting plant, a mushroom, a reaper, a flame spirit, a snail and an alien, each with idle, run and hurt animations. Attack and death are derived from existing frames, so a new creature gets them for free.
Names pick the art: PiranhaPlant, Mushroom, Reaper, FireSpirit, GiantSnail and Alien route to the new sprites.
spawnpoint sprites parade renders an animated sheet of heroes, monsters or items at a large scale.
Closer, finer demos: games use the finer pixel mode and zoom in, and every demo GIF is re-recorded.

I also checked that adding the new nouns to the model's prompt didn't hurt it: the sealed shooter set held at 94% with the served model, and the 30 real prompts moved from 80% to 77%, which is one prompt of noise.

Try it

pip install -U spawnpoint-studio
spawnpoint llm up        # downloads the v5.1 model (12 GB NVIDIA GPU + Docker, or CPU via Ollama)
spawnpoint create "a platformer where a ninja rescues his mother, with mushrooms and snails"

You can also play nine generated games in the browser, with touch controls on a phone: leluong141996-dev.github.io/Spawn-Point.

Repo: https://github.com/leluong141996-dev/Spawn-Point
Model (AWQ): https://huggingface.co/smoky1496/SpawnPoint-Coder-7B-AWQ
Model (GGUF for Ollama): https://huggingface.co/smoky1496/SpawnPoint-Coder-7B-GGUF

A small ask

The model files have been downloaded a few hundred times and the repo has no stars yet. I suspect most people run it, see what it does, and move on, which is fair. If you tried it and it did something useful or interesting, a star on GitHub or a like on the Hugging Face page helps other people find it. And if it did something wrong, an issue with the prompt you used helps me more than a star: my evaluation sets only grow from prompts that broke.

What would you want a model like this to draw next?

How splitting one giant JSON schema into per-genre schemas shrank my prompts 3x (and stopped a 7B model from planting wheat in platformers)

Lượng Lê — Sat, 17 Oct 2026 15:58:39 +0000

I'm building Spawn Point, an open-source 2D game generator. Instead of asking an LLM to write buggy GDScript or draw pixels, I fine-tuned a 7B model to output a game design as structured JSON. Plain Python and pre-tested Godot components then assemble that JSON into a playable game.

This is the story of the v5 release: one schema change that paid off more than any amount of training data, a brand-new genre, and a regression I knowingly shipped anyway.

The Problem: One giant schema to rule them all

Up until v4, the model filled out a single monolithic schema called GameBrief with around 20 fields: player, enemies, crops, towers, goal, hazards, length, feel, juice...

For a platformer, half of those fields are meaningless. For a cozy farming game, hazards and jump_tiles make zero sense.

Here's the catch with guided decoding: it guarantees the output is valid JSON matching your schema, but it doesn't guarantee the output makes sense. My 7B model would happily design a hardcore platformer with crops: ["Wheat"], or a tower defense game where the player has a double_jump ability. Both are 100% valid JSON. Both are completely broken game designs.

On top of that, the system prompt was massive because it had to explain every single field across every genre, plus include a full few-shot example turn.

The Fix: A Discriminated Union

In v5, the model outputs {"genre": "...", ...} first, and the schema immediately branches so only that genre's fields exist:

class _Look(BaseModel):
    genre: str          # Declared first so it leads every JSON object
    title: str
    palette: Palette = "night"

class PlatformerW(_Look):
    genre: Literal["platformer"]
    player: PlatformerPlayer
    enemies: list[PlatformerEnemy]
    ...

class FarmW(_Look):
    genre: Literal["farm"]
    crops: list[str]
    enemies: list[FarmPest]   # A pest can only be a walker or chaser
    ...

_UNION = TypeAdapter(
    Annotated[Union[PlatformerW, FarmW, ...], Field(discriminator="genre")]
)

Getting this to work smoothly with guided decoding took three specific tricks:

genre must come first in the token stream. With guided decoding, once the model commits to the "genre" token, the grammar mask locks in and only allows fields valid for that specific class. Since Pydantic orders inherited fields first, I declared genre: str at the top of the base _Look class and let subclasses narrow it down with Literal[...].
Close every object and strip OpenAPI keywords. Pydantic's .model_json_schema() emits an OpenAPI-specific discriminator keyword and leaves additionalProperties open by default. I added a post-processing step to strip discriminator, inject additionalProperties: false everywhere, and wrap the root in {"type": "object"} (since several inference servers require an object at the root).
Treat the LLM output as a wire format. My game assembler, session manager, and test suite still use the monolithic GameBrief internally. The new per-genre classes act strictly as a wire protocol, converting back and forth via from_wire() and to_wire(). This lets me iterate on what the LLM sees without refactoring the entire game engine.

The Results

3x shorter prompts: The system prompt is roughly a third of its old size: no more explaining a zoo of unrelated fields, and no more giant few-shot example turn.
50% smaller edit prompts: The entity-edit prompt dropped from 4,182 to 2,075 characters once I gave it a compact view of the entities.
No more mixed-up genres inside one design: Platformers with wheat crops are gone, simply because they are no longer expressible in the grammar. (The model can still pick the wrong genre for an ambiguous prompt; it just can't mix fields from two of them.)
Native Vietnamese support: Previously, I ran a translation step in front of the model. v5 drops it and reads Vietnamese prompts directly, trained alongside English.

Bonus: Adding a new genre is now dirt cheap

The best part of per-genre schemas is how easy it is to expand the engine. To add a side-scrolling space shooter in v5, all I needed was one small Pydantic model, a few synthetic data builders, and the Godot components.

The new shooter genre includes:

Four enemy archetypes (drifter, weaver, diver, gunship)
Power-ups that add guns
A three-layer parallax starfield
Auto-firing weapons (making it super mobile-friendly)

My favorite piece of component reuse is the boss fight: it's just a standard wave spawner with autostart = false that triggers when the regular level spawner emits its cleared signal. No new component required.

👉 You can play the generated games right in your browser here: Spawn Point Web Demo

The Honest Part: What got worse in v5

I evaluate the model by running hand-written prompts through the entire pipeline and grading the final game. A test only passes if the playable game matches everything requested in the prompt. (At temperature 0.4, run-to-run variance is about 1–3 prompts per set, so I only care about swings larger than that.)

Evaluation set	v4	v5 (full precision)	v5 (4-bit AWQ)
New capabilities (16 sealed prompts: shooters + v4 fixes)	56%	88%	75%
Real-world prompts (30 EN & VI human-written)	87%	80%	77%
Older sealed set (12 prompts, first nine genres)	67%	58%	n/a

On held-out synthetic examples with exactly one right answer, v5 gets 73% of briefs, 98% of tweaks and 100% of entity edits (v5 format only, so there is no v4 number to compare).

v4 couldn't be expected to know shooters, so the first row isn't a fair fight. The other two rows are real regressions.

Why did the older sets dip?

In my final training patch for v5, I added examples to fix a bug where the model missed implied enemies ("a tank shooting monsters"). I over-corrected: when the prompt names the player only indirectly, v5 now often answers with the default "Hero". Some old prompts are also genuinely ambiguous now that shooters and farms exist ("a space shooter" can be side-scrolling).

I decided to ship v5 anyway, document the exact weakness in the model card, and fix it properly in a v5.1 patch. I'm a firm believer that a model card that honestly lists regressions is way more useful to developers than one that only cherry-picks wins.

5 Takeaways for Structured LLM Outputs

If you're building with small local models and constrained decoding, here's what I learned the hard way:

Shrink the output space before adding data. I spent a lot of time curating training data for v3 and v4. Making invalid state combinations unrepresentable in the schema achieved more in a single afternoon.
Put your discriminator token first. Verify that your JSON schema actually forces the model to pick the branch before generating payload fields.
Decouple your LLM wire format from your app's internal state. Let the LLM read and write compact, specialized schemas while your backend works with a stable canonical model.
Patches skew the model, even with replay. My last fine-tune mixed old rows back in and still over-corrected on one behavior. Re-run all your eval sets after every patch, not just the one you were fixing.
Track and publish your regressions. Future you (and your users) will thank you for the honesty.

Try it out locally

If you have Python installed, you can spin up the local LLM and generate a playable Godot game in three commands:

pip install spawnpoint-studio

# Downloads and serves the model:
# Uses the 4-bit AWQ build on a 12GB+ NVIDIA GPU with Docker,
# or falls back automatically to the GGUF build via Ollama.
spawnpoint llm up

spawnpoint create "a side-scrolling space shooter with a boss"

Links

💻 GitHub Repo: leluong141996-dev/Spawn-Point
🤗 Model (AWQ): smoky1496/SpawnPoint-Coder-7B-AWQ
🤗 Model (GGUF for Ollama): smoky1496/SpawnPoint-Coder-7B-GGUF

Question for the community: If you've used guided decoding / structured outputs in production, did you go with one giant schema or break it down into smaller ones? What edge cases broke your setup? Let me know in the comments! 👇

I stopped letting the LLM write GDScript. Now one sentence becomes a playable Godot game in ~25 seconds

Lượng Lê — Sat, 10 Oct 2026 04:06:28 +0000

I'm an AI engineer, not a game developer. A few weeks ago I wanted to know: how fast can an AI agent go from "a ninja platformer where you collect stars" to something I'd actually want to play?

The usual answer is "let the LLM write the code". I tried that. It's slow, it burns tokens, and Godot 3/4 API mix-ups break the build all the time. Even when it compiles, the game feels flat: no coyote time, no screen shake, and hitboxes that don't match the sprites.

So I built Spawn Point around a different idea:

The LLM shouldn't write game code. It should design the game, and a compiler should build it.

What it looks like

pip install spawnpoint-studio
spawnpoint llm up      # local model, ~11 GB VRAM (or any OpenAI-compatible API)
spawnpoint create "a ninja platformer where you collect stars"

Design:
  "Ninja Star Dash": platformer, palette candy, goal collect_all, short, difficulty 2
  player: double_jump, stomp; jump 4.0 tiles; speed fast; feel floaty
  enemies: 4x Goblin (walker), 2x Spider (hopper); pickups: 10x Star; hazards: spikes
  [L0 core loop] ok in 5.5s
  ▶ playing at http://localhost:8060/
  [L1 goals & enemies] ok in 5.1s
  [L2 feel & polish] ok in 5.1s
Done in 25.7s (LLM 2 call(s), 977 prompt + 256 completion tokens).

The browser opens as soon as the first layer is playable and reloads as the next layers land. The whole game takes two LLM calls and about 1.2k tokens.

Then you iterate in plain language:

spawnpoint tweak out/ninja-star-dash "jump 20% higher and add a dash, night colors"
#   palette: "candy" -> "night"
#   player.abilities: [...] -> ["double_jump", "stomp", "dash"]
#   player.jump_tiles: 4.0 -> 4.8

Only what you asked for changes. The level layout stays the same.

How it works: LEGO, not prose

Instead of generating hundreds of lines of GDScript, the pipeline looks like this:

Design. The LLM fills a small, enumerated GameBrief (genre, abilities, enemy archetypes, pickups, goal, feel) using guided JSON decoding, so even a 7B model can't produce malformed output.
Assemble. Deterministic code turns the brief into a game graph: pre-tested Godot components, an ASCII level map and declarative rules. Levels are re-seeded until the compiler can prove that everything is reachable.
Compile. A Python compiler validates the graph in about 1 ms, with error messages written for an LLM to fix. Then Godot runs headless and builds the scenes itself, and a smoke test plays the game for a second before you ever see it.

	Typical "LLM writes code" agents	Spawn Point
How code is produced	GDScript from scratch	Pre-tested components + JSON graph
Time to a playable build	minutes to hours	seconds
Game feel	none by default	coyote time, jump buffer, squash & stretch, hit-stop, screen shake, particles, SFX
Hitboxes	drift from the visuals	traced from the sprite's pixels
Failure mode	crashes, syntax errors	structured diagnostics before Godot runs

Things I didn't expect to build

Bots that playtest your game and fix it. Three bot skill levels (novice, average, expert) press real keys and navigate with a jump-aware path planner. Then a bounded tuner adjusts the game:

before:  novice  won 1/3  deaths 5  (Spikes 4, Frog 1)
round 1: Player.Hurtbox.size_scale = 0.75   (Spikes caused 6/7 deaths)
         Player.Health.max_health = 4       (novice bots won only 1/3)
after:   average won 2/3

Right-click to edit. spawnpoint studio opens a local web cockpit with the game running in the middle. Right-click an enemy inside the game and type "double its health and make it shoot fireballs". The model returns a patch scoped to that entity, the compiler checks it, and the game reloads.

Pixel art with matching collisions. Sprites are generated procedurally, imported from your own sheets, or drawn by an image model through a plug-in (A1111 / any OpenAI-compatible images API). Collision polygons are traced from the pixels, so hitboxes always match what you see.

The model: SpawnPoint-Coder-7B

I wanted the whole studio to run offline, for free, on a consumer GPU. So I fine-tuned a 7B model (QLoRA, then AWQ 4-bit) just for these tasks: designing briefs, applying tweaks and editing entities. The compiler doubles as the data filter and the judge: every training example has to compile.

On 30 game ideas written by hand (English and Vietnamese, 153 checks):

	v1	v2
Prompts fully right	73%	87%
Checks passed	94%	97%
Held-out briefs (exact match)	36%	88%
Vietnamese → English	52%	97%

It's on Hugging Face: smoky1496/SpawnPoint-Coder-7B-AWQ. Claude, OpenAI or any OpenAI-compatible API work too.

Honest limits: it's tuned for 2D platformers and 2D top-down games. Ask for a 3D RPG and you'll get nothing special. The GDScript comes from the tested component library, not from the model.

Try it

⭐ Repo: https://github.com/leluong141996-dev/Spawn-Point
🤗 Model: https://huggingface.co/smoky1496/SpawnPoint-Coder-7B-AWQ
📦 pip install spawnpoint-studio (needs Godot 4.6)

No GPU? spawnpoint examples copy pixel_hopper && spawnpoint build pixel_hopper.json --run builds an example game without any LLM.

I'd love feedback, especially from people who actually make games: what's the first game you'd type in? Drop it in the comments and I'll post what comes out.

Three releases of my AI memory layer. The best bugs were found by other people.

Lượng Lê — Fri, 09 Oct 2026 16:49:53 +0000

I'm building Agent Brain Hub, an open-source shared memory layer for multi-agent systems.

Imagine a travel agent, a repair agent, and a finance agent all talking to the same user. Each should know what the others learned, without leaking sensitive data (like health records or income) between them. The architecture borrows loosely from cognitive psychology: a short-term working memory per session, a long-term store for facts and episodes, and an asynchronous "sleep cycle" that consolidates one into the other.

Over the last few days, I pushed three releases. Here's what broke, what the benchmarks showed, and why the best edge cases didn't come from my test suite. They came from people reading my posts.

v0.5: Point-in-time correctness (time-traveling memory)

Until v0.5, a replaced or expired fact was deleted at the next sleep cycle. Now facts are invalidated with validity intervals: "I moved to Saigon" closes "I live in Hanoi" with a timestamp and a pointer to the superseding record.

You can query the brain's state at any point in history:

await brain.recall({
  customerId,
  text: 'Book a taxi from my home',
  asOf: '2026-10-01T09:00:00Z',
});
// Returns the address known at that timestamp, not today's address.

When debugging multi-agent loops, this is exactly the question you need to answer: "What did the travel agent actually believe when it made that call?"

The bug the benchmark caught: Adding temporal queries exposed a privacy leak before release. Querying past states skipped the privacy filter on recent conversation turns, so one agent's private conversation could reach another agent. The new temporal benchmark scenarios caught it before it ever shipped.

v0.6: An entity graph, and an honest benchmark

Facts are now grouped into per-customer entities ("my car" = "xe" = "車" = "Honda Civic"), and retrieval follows the links between them:

Mia (personal agent):  "My car is a Honda Civic"
Kai (repair agent):    "The car broke down, it'll be in the shop for 3 days"

Penny (finance agent): "Should I set money aside for my Civic this week?"
→ availability: no car for 3 days (from Kai, 2 hours ago, expires in 4 days, via: Honda Civic)

Notice that Penny's question never mentions "car" or "shop". The graph links "Civic" to the car, and the car to its repair. Traversal only runs over memories the asking agent is already allowed to see, so it can't carry a private fact across domains (a medical allergy never reaches the booking agent).

I ran the benchmark with --no-graph to see if the graph actually pulls its weight:

Offline mode (hashing vectors)	With graph	Without graph
Multi-hop questions	100%	40%
All 37 scenarios	97.3%	81.1%
Leak rate	0%	0%
Mean prompt tokens	427	421

The part most benchmark posts leave out: if you switch from offline hashing vectors to an embedding model like bge-m3, the multi-hop scenarios pass even without the graph. The model already associates "the XPS" with "the laptop".

Right now, the graph earns its keep in:

Zero-config offline mode, with no embedding model.
Explainability: a deterministic trace of why a fact was pulled in.

My current scenarios aren't hard enough to show more than that against dense embeddings. Harder, adversarial scenarios are now in the backlog.

The edge cases I completely missed

I posted an earlier devlog on daily.dev and opened the repo to contributions. Three real insights came straight from readers.

1. Stepping on a contributor's toes

A contributor claimed an open issue and raised a PR. A few hours later, unaware of it, I shipped my own fix for the same issue.

Their PR turned out to catch a case mine missed: in Vietnamese, "không thích … nữa" ("don't like … anymore") was still treated as a conflict instead of an update to the old preference. I rebased their PR onto main, kept their authorship, added a small follow-up, merged it and tagged v0.6.1, the project's first outside contribution.

Takeaway: if your repo is public, check claimed issues and open PRs before you open your IDE.

2. "Headroom rather than a guarantee" (the race condition)

Working memory holds the last 40 turns. Once 24 unconsolidated turns pile up, the scheduler triggers a consolidation "sleep cycle".

A commenter, Ahmet Özel, pointed out that the 16 turns between 24 and 40 are "headroom rather than a guarantee": a slow summarizer or a queue backlog can still let turns fall off the end.

Following their suggestion, I wrote a burst test with a deliberately slow summarizer. Result: 10 out of 26 turns vanished silently. Turns arriving during the summarization call weren't part of the current batch, and the buffer trim after consolidation dropped them anyway.

Fixed in v0.6.2:

A turn is only evicted once it's inside a stored episode.
Hitting the 40-turn cap triggers a consolidation immediately instead of evicting turns.
The burst test is now part of the regression suite.

3. "What happens to a screenshot?"

Another reader asked how consolidation handles a turn that carries a screenshot.

Honest answer: it doesn't. The hub is text-only. And sending images to a remote multimodal model just to summarize them would expose private data and rack up API costs. Their suggestion, a local OCR pass before the consolidation loop, is now tracked in issue #25.

Key takeaways

Synthetic benchmarks catch the edge cases you anticipated when writing the code. Readers catch the assumptions you took for granted. Neither helps if you only share the numbers that make your project look good.

What's next for v0.7

Open-schema extraction: moving past the 19 hardcoded relation types.
Conflict resolution between agents: rules for when agent A and agent B assert contradictory facts about the same entity.

Question for anyone running multi-agent stacks:
How do you resolve conflicts when two independent agents write contradictory facts about the same entity? Last write wins? Confidence scores? CRDTs? Escalating to a human? I'd love to hear how you handle it.

Agent Brain Hub is MIT-licensed, has no telemetry, and runs fully local with docker compose up, or with any LLM provider (OpenAI-compatible, Anthropic, Gemini, Ollama…): github.com/leluong141996-dev/Agent-Brain-Hub.

My embedding model couldn't find "order 48207". Here's the one-line fix and what LongMemEval said next.

Lượng Lê — Thu, 08 Oct 2026 23:54:41 +0000

I maintain Agent Brain Hub, an open-source memory layer that several AI agents share. Last release I added real embedding models, and recall on my benchmark went from 88.5% to 100%. Then I wrote a harder test, and the model fell over on something any keyword search gets right.

The failure

A customer has twelve shopping conversations, one per order:

My order ORD-48200 for a blue backpack hasn't arrived yet
My order ORD-48201 for running shoes hasn't arrived yet
…
My order ORD-48211 for a water bottle hasn't arrived yet

Then another agent asks the memory: "what is going on with order 48207".

Retrieval	Finds ORD-48207?
Local feature hashing (lexical)	✅
`bge-m3` embeddings	❌

To the embedding model all twelve conversations mean the same thing ("a late order"), so they scored almost identically, and recency decided which four made it into the context. The one distinguishing token, 48207, is exactly what dense embeddings blur. Hashing vectors are crude, but they match exact tokens.

What didn't work: taking the max

My first fix took whichever signal was stronger: max(model, lexical). It still failed. The model's score for every order (~0.6) was above the lexical score of the right one, so the max was the model score everywhere and nothing changed.

What worked: adding them

// Hybrid (CombSUM): meaning + wording
const lexical = Math.max(0, cosine(qHash, item.hashVec));
return modelVecAvailable ? Math.max(0, dot(qVec, item.vec)) + lexical : lexical;

An item that matches in meaning and wording now beats items that are merely on topic. The sum is never below either signal, so paraphrase matches ("Can I drive to the airport?" → "no car for 3 days") keep the score that made them work. I added an exact_match category to the benchmark, with order numbers that differ by one digit, with and without the ORD- prefix, plus device model names. A unit test proves the fix: it fails if you revert it.

With bge-m3 on all 25 scenarios: 96% pass, 100% recall, 0% leaks.

Then I asked a public dataset

My own scenarios only compare versions of my own project. So v0.4.0 also runs LongMemEval-S (MIT): 470 questions, each hidden among ~50 past chat sessions. Is the session that holds the answer near the top?

Local hashing, no network	top 1	top 4	top 10
All questions	30.0%	52.8%	71.3%
"The assistant told me…"	66.1%	87.5%	92.9%
Preferences	3.3%	20.0%	43.3%

Honest takeaway: about half the time the right session is in the top 4, and preference questions are bad. That number is the reason to have a public benchmark: it tells me where to work next instead of telling me I'm done. (The embedding-model run needs a GPU; on CPU it would take days, so it's next.)

And it got fast

Retrieval used to scan every customer's memories, then filter. A per-customer index changed the tail:

100,000 episodes	before	after
recall p50	4.3 ms	0.6 ms
recall p95	40.1 ms	1.9 ms

Try it

git clone https://github.com/leluong141996-dev/Agent-Brain-Hub && cd Agent-Brain-Hub
docker compose up -d                                   # http://localhost:4317
npm run bench -- --url http://localhost:4317           # benchmark your own running hub

If your agents' memory ever failed in a way like this (a code, a name, a date the retriever couldn't see), I'd love it as a benchmark scenario. It's one JSON file, and every future release has to pass it.

👉 https://github.com/leluong141996-dev/Agent-Brain-Hub

My AI agents' shared memory now sleeps on its own, and stopped losing conversations

Lượng Lê — Wed, 07 Oct 2026 15:29:16 +0000

I'm building Agent Brain Hub, an open-source memory that several AI agents share. It's modeled on the human brain, and like a brain it needs sleep: that's when conversations get consolidated into long-term memory, expired facts get forgotten, and the brain reflects on what it learned.

Until this week, sleep only happened when someone clicked Run sleep cycle.

The bug that pushed me

While adding automatic sleep, I wrote a test for a long conversation and found a real problem. Working memory keeps only the last 40 turns. After 30 back-and-forth messages with nobody pressing the button, the first 10 questions were simply gone. They were never consolidated, so no agent could ever recall them.

Three ways the brain falls asleep now

Trigger	When	Default
Idle	The customer has gone quiet, so the session is over	30 min
Pressure	Too many turns are waiting to be consolidated	24 turns
Nightly	Once a day, for customers active since last night	03:00

The core check is small:

const trigger =
  turns >= cfg.maxPendingTurns ? 'pressure'
  : idleMinutes >= cfg.idleMinutes ? 'idle'
  : null;

The 30-minute idle default isn't arbitrary: it's the same gap the brain already uses to decide that a new session has started. The pressure threshold stays below the 40-turn cap, so nothing is dropped.

A few details I cared about:

Manual and automatic sleeps share one queue, so two runs never touch memory at the same time.
No surprise bills. On the first start, the server doesn't sleep every customer at once. With a paid LLM, that would mean a burst of summarization calls.
You can watch it happen. Automatic runs show up live in the brain view, with a note saying why the brain fell asleep.
Docker runs in UTC, so set TZ=Asia/Ho_Chi_Minh (or your zone), or "03:00" will be someone else's night.

Settings live in the UI (Settings → Sleep cycle) or in BRAIN_SLEEP_* environment variables. There are 9 new tests, including one that reproduces the lost-turns bug.

Try it with docker compose up. Feedback is very welcome, especially if you've solved memory consolidation differently in your own agents.

👉 https://github.com/leluong141996-dev/Agent-Brain-Hub

ai, #opensource, #javascript, #llm, #agent, #memory

Why multi-agent systems struggle with memory—and how I tried to fix it with Agent-Brain-Hub

Lượng Lê — Tue, 06 Oct 2026 19:18:51 +0000

Hey everyone! 👋

Over the past few months building and experimenting with LLM agents, I kept running into the same bottleneck: context fragmentation and state synchronization.

When you move beyond a single prompt-and-response loop to multi-agent workflows, managing shared state, dynamic context retrieval, and decision orchestration quickly gets messy. Most frameworks either bundle too much abstraction or leave memory management as an afterthought.

To tackle this, I started building Agent-Brain-Hub—a lightweight, centralized core designed to act as the cognitive layer for autonomous agents.

What is Agent-Brain-Hub?

Think of it as the persistent coordination and memory center for your agent systems. Instead of each agent keeping isolated context or bloating the prompt window with raw history, the hub coordinates:

Centralized Context & Memory: Provides structured retrieval and persistent memory across execution turns.
Decision Coordination: Helps decouple reasoning logic from agent-specific tooling.
Developer-Friendly Integration: Kept minimal so you can plug it into existing pipelines without rewriting your architecture.

Why Open Source?

Agent architecture is evolving rapidly, and no single setup fits every use case. I built this to solve my own pain points, but I want to turn it into a solid tool the community can genuinely rely on.

The codebase is completely open source:
👉 GitHub Repo: https://github.com/leluong141996-dev/Agent-Brain-Hub

How You Can Help

I’d love to get feedback from developers working with autonomous agents:

Does this fit into your current memory/orchestration stack?
What features or integrations (e.g., specific vector stores, frameworks) would make this indispensable for your workflows?

Check out the code, feel free to open issues or PRs, and if you find the concept interesting or useful, a ⭐️ on GitHub would mean the world to an indie open-source builder!

Looking forward to your thoughts in the comments.

I modeled AI agent memory on the human brain. Here's what worked and what didn't

Lượng Lê — Mon, 05 Oct 2026 17:38:46 +0000

Every AI agent I deployed had the same problem: it started each conversation from zero. And as soon as there was more than one agent (one for support, one for bookings, one for billing), each of them learned things about the same customer and kept it to itself.

So I built Agent Brain Hub: one memory that many agents share. This post is about the design, the brain metaphor behind it, and the mistakes I made along the way.

Repo: https://github.com/leluong141996-dev/Agent-Brain-Hub (MIT, docker compose up)

The goal in one example

The customer tells Kai, the repair agent: "My car is a Honda Civic, it broke down and will be in the shop for 3 days."
Later they ask Atlas, the travel agent: "I need a flight to Da Nang next week, window seat please."
Atlas answers: "I've got the context from Kai. I remember you have no car for 3 days, no need to repeat it. I'll look for a flight to Da Nang, window seat preferred. Also: a self-drive rental at the destination, would you like that?"

Atlas never talked to the customer about the car. The fact "car unavailable for 3 days" was written by Kai into shared memory, while the repair details stayed private to Kai.

Why a brain?

I needed a way to split a memory system into parts with clear jobs. The brain turned out to be a good map, and it made the live visualization explain itself. This is a metaphor, not neuroscience.

Region	Job in the system
Thalamus	Routes the input: language, intent, which agent
Brainstem	Redacts PII, catches crisis messages before anything is stored
Amygdala	Salience: urgency and emotion raise priority
Prefrontal cortex	Working memory for the current conversation
RAS	Retrieval: scope, then TTL, hybrid scoring, re-rank, token budget
Neocortex	Long-term facts and episodes, hot/warm/cold tiers, contradiction resolution
Hippocampus	Encodes new memories from each turn
Cerebellum	Skills: a playbook that succeeds 3 times in 7 days is promoted, versioned, can be rolled back
Basal ganglia	Picks the next best action with Thompson sampling
Corpus callosum	Handoff between agents
Forgetting, DMN	The sleep loop: decay, consolidation, reflection

Every request is traced through these regions over Server-Sent Events, so in the UI you can watch which region did what and which memories the answer used.

Three ways an agent plugs in

Native: create it in the UI and the hub answers with its own LLM.
REST or SDK: your agent keeps its own LLM and calls the brain around each turn:

const ctx = await brain.recall({ customerId, text, lang: 'en' });
const reply = await myLLM({ system: mySystemPrompt + ctx.promptBlock, user: text });
await brain.remember({ traceId: ctx.traceId, reply, lang: 'en' });

MCP: brain_recall, brain_remember, brain_profile and brain_feedback tools for Claude Desktop, Claude Code, Cursor and agent frameworks.

What worked

Scopes and an audit log, from day one. Every memory is private, shared or global, and every agent has read and write permissions. Every read, write and blocked access goes to an append-only log. Without this, "shared memory" is just a leak with extra steps.

One writer per kind of fact. Only the repair domain writes asset_issue, only travel writes trip_destination. This removed a whole class of agents overwriting each other.

Retrieval as a pipeline you can see. Breaking recall into scope, TTL, scoring, re-rank and budget stages, and showing each stage in the trace, made ranking bugs obvious.

Skills that earn promotion. A playbook only becomes a reusable skill after it succeeds 3 times within 7 days, and every version is kept. Bad promotions can be rolled back.

What I got wrong

Private data leaked through side doors. Retrieval respected scopes, but private details still reached other agents through the recent conversation turns, the handoff summary and the sleep-time reflection. Every path that builds a prompt needs the same visibility filter, not just the main retrieval.

Keyword matching across languages is a trap. Vietnamese "cửa sổ" (window) matched the keyword "sợ" (afraid) once accents were stripped, so a seat preference looked like anxiety. Matching now keeps diacritics when the input has them and uses word boundaries; Japanese uses substring matching because it has no spaces.

The bandit was too eager. Thompson sampling sometimes let an irrelevant action win on a lucky draw. Narrowing the exploration range around relevance fixed it.

Rules and the LLM contradicted each other. When both extracted a fact from the same message, single-valued facts could end up with two values. Now they're merged and deduplicated before writing.

Honest limitations

Embeddings are local feature hashing, so the hub runs with no external API. A real embedding model would be better.
Fact extraction is rules plus an optional LLM pass.
It's a single process on SQLite. PostgreSQL with pgvector is the next step for scale.

Try it

git clone https://github.com/leluong141996-dev/Agent-Brain-Hub
cd Agent-Brain-Hub
docker compose up -d   # → http://localhost:4317

It runs offline with no configuration. Pick an LLM later in Settings: Claude, GPT, Gemini, DeepSeek, or local models through Ollama or vLLM. The UI speaks English, Vietnamese and Japanese.

If you run several agents in production, I'd love to hear what you need from shared memory. Issues labelled good first issue are open, and a ⭐ on the repo helps other people find it.