I Built a 2,015-Entity Mythology Archive Using Autonomous Agents (Here's the Pipeline)

#ai #opensource #discuss #programming

I Built a 2,015-Entity Mythology Archive Using Autonomous Agents

A few weeks ago I started a project that sounded simple: build an archive of mythological entities — gods, monsters, heroes, relics — with a dossier page for each one. Two thousand entries. Rich, accurate, cross-referenced.

The naive way: hire writers, or sit in front of a spreadsheet for months.

The way I actually did it: I pointed an autonomous agent loop at the problem and let it grind while I slept. Here's the pipeline that made 2,015 entities possible, what broke along the way, and what I'd do differently.

The architecture: a stateful loop, not a script

The first mistake everyone makes with "automation" is writing a stateless script: fetch next item -> process -> exit. That's a robot, not an agent. It forgets everything between runs, can't recover from a mid-batch crash, and dies the moment reality stops matching its assumptions.

Instead, I used the pattern that AutoGPT and BabyAGI popularized: a stateful loop.

READ state.json  ->  DECIDE next step  ->  DO one action  ->  SAVE state.json
        ^                                                     |
        +-----------------------------------------------------+

The state file is the brain. It holds:

Which entity is in flight and what phase it's in (researching, writing, reviewing, published)
The task queue — a prioritized pool of work to pull from
Progress metrics — how many dossiers are done, the rolling quality-gate pass rate
Blockers — which entities are stuck and why

Because state lives on disk, a crashed tick doesn't lose work. The next tick reads the file, sees "entity #1,204 was in phase writing, 60% done", and continues. That single design decision — persistence — is the difference between an agent and a cron job.

The three-phase entity pipeline

Each entity went through three phases, and each phase was a separate concern:

Phase 1 — Research. A sub-agent gathered source material from public APIs and web sources: the entity's name, culture of origin, domain, physical description, myths it appears in, key relationships. The goal wasn't to write — it was to collect evidence.

Phase 2 — Drafting. A second pass turned the raw research into a structured dossier: summary, attributes, mythology, artifacts, legacy. Strictly templated so output stays consistent across 2,000 entries.

Phase 3 — Quality gate. Every dossier was validated against the same rubric before it was allowed into the archive: Does it have all required sections? Is the citation count above the floor? Does it parse as valid data? Failed entries went back to the queue with the failure reason attached — not deleted, retried with context.

This three-phase split matters more than it looks. It means a failure in drafting doesn't waste the research, and a failure in the quality gate doesn't corrupt the archive. It also means I can scale the write phase with parallel agents while keeping the gate sequential and strict.

Parallelism: the 40-agent swarm moment

The bottleneck was never the model — it was serial execution. One entity at a time, a minute each, that's 33 hours of pure grind.

The unlock was running many workers in parallel against a shared queue. The architecture is embarrassingly simple: N workers poll the queue, grab the next unclaimed entity, process it, write the result, mark it done. No worker talks to another worker; they only talk to the queue.

That's where the real throughput came from. Once the queue was the single source of truth, I could throw 10, 20, even 40 workers at it and the system just... worked. The queue is the lock, the workers are stateless consumers, and the quality gate is the bouncer.

What broke (and what that taught me)

Nothing survives contact with reality. Three failures stand out:

1. The class-name mismatch. Early on, I had one agent building the HTML pages and another writing the JavaScript that renders the entity cards. The CSS classes didn't match, and the whole page fell apart silently. Lesson: define the contract before you parallelize. Naming conventions are an API.

2. The dead-source problem. Research sources die or change. An API that worked yesterday 403s today. The pipeline needed a fallback chain: primary API -> backup API -> web search -> manual flag. An entity with zero sources is a blocked entity, not a failed entity — it goes to a quarantine queue, not the trash.

3. The scroll-dump trap. My first page designs were flat walls of text — a thousand lines of scrolling per entity. Terrible UX. The fix was forcing pagination and tabs: every entity page got structured sections, 6-7 tabs max, never a scroll dump. This was a design lesson learned the hard way: content volume is worthless if the presentation buries the reader.

The economics

Here's the part nobody talks about: this cost almost nothing in API spend.

Research and drafting ran on a mix of free-tier models and a cheap paid model, routed by task difficulty
The queue-based architecture means costs scale with work done, not with wall-clock time
The whole 2,015-entity run cost less than a coffee per hundred dossiers

The expensive resource was never tokens — it was my attention. The agent loop converts my attention from a blocking resource into an occasional review checkpoint. I check quality gates, unblock quarantine items, and adjust the rubric. The grinding, I don't do.

What I'd do differently

Three things, in order of regret:

Build the quality gate first. I wrote it second, after the pipeline was already producing. A gate designed from day one would have caught the consistency issues earlier.
Version the rubric. The gate's criteria drifted as I learned what "good" meant. The rubric should be a versioned artifact, not a string in a config file.
Instrument everything. I added counters late. If you're building a pipeline like this, log phase durations and failure reasons from tick one — that data is how you find the bottleneck instead of guessing.

Why this matters beyond mythology

The archive itself is a niche project. But the pipeline is not niche. Any task with these three properties is a candidate:

Lots of similar items (entities, products, articles, tickets)
Each item needs research + structured output
Quality matters more than speed (which is what the gate is for)

Product catalogs, documentation sites, knowledge bases, review aggregators — same shape. The pattern is: persistent state, a shared queue, parallel stateless workers, and a strict quality gate between production and publication.

That's the whole trick. Not one big model doing everything — a loop, a queue, and a gate. The agents do the volume; the architecture does the quality.

This is part of my ongoing series about building real things with autonomous agents. If you're doing similar work — archives, pipelines, agent loops — I'd love to hear what broke for you. The failure stories are always the best part.