Building a 5-Step AI Pipeline Without Letting Duplicate Requests Corrupt It

I built a tool that turns a book's raw text into a full set of AI-generated illustrations, as a take-home technical assessment. The pipeline runs five sequential steps — style, characters, portraits, chapters, illustrations — using the Gemini API for both text generation and image generation. The interesting engineering problem wasn't calling an AI API. It was making sure two overlapping requests couldn't run the same step twice.

The pipeline, step by step

Each project moves through five stages in order:

1. Style — establish a consistent visual style for the whole book.
2. Characters — extract and describe the characters that appear.
3. Portraits — generate a reference image for each character (capped at 2 characters, to keep runs bounded).
4. Chapters — break the book into illustratable chapter units (capped at 1 chapter per run, same reasoning).
5. Illustrations — generate the actual illustrations, matching each chapter's named characters back to their portrait images for visual consistency.

The caps aren't arbitrary laziness — they're a deliberate bound on how much a single pipeline run can do, which matters a lot when every step is a paid, rate-limited external API call.

The bug you don't see until two requests race

Here's the failure mode that shaped the whole backend architecture: what happens if the frontend fires the same "run this step" request twice — a double-click, a retry after a slow response, a network blip that makes the client think the first request failed when it didn't?

Without protection, both requests would see the step in the same "not yet started" state, both would proceed to call Gemini, and you'd either burn double the API budget or end up with two conflicting results racing to write the same step's output.

The fix lives in services/pipeline.js, which acts as a small state machine sitting in front of everything else. When a step-run request comes in, it atomically claims the step — meaning the claim-and-check happens as one indivisible operation, not a check followed by a separate write with a gap in between where a second request could sneak in. If a second request arrives while the step is already claimed, it gets rejected with a 409 Conflict instead of being allowed to proceed. The route layer (steps.js) is the only place that knows about HTTP semantics; pipeline.js itself has no Gemini or filesystem knowledge beyond delegating to a storage lock — it just tracks state transitions.

There's a second piece to this: staleness detection. If a step gets claimed and then the process crashes, or the request times out without ever marking the step complete or failed, that step would otherwise stay claimed forever — permanently stuck, un-retryable. The pipeline service detects steps that have been claimed for too long without resolving and treats them as stale, making them claimable again. Without this, a single dropped connection would permanently soft-lock a project.

Splitting "what to send Gemini" from "how to talk to Gemini"

The service layer is split in a way that made testing much easier than it would've otherwise been. services/gemini.js is a thin REST client — it only knows the wire protocol for the Gemini Interactions API and the Files API's resumable upload flow. It has zero opinions about prompts, character caps, or app logic.

services/steps.js is where the actual pipeline logic lives: building prompts, applying the 2-character and 1-chapter caps, and matching chapter-named characters back to their corresponding portrait images so the illustration step can reference the right visual for each character. Critically, the pure logic in this file — the capping rules, the name-matching — is factored out and unit-tested completely separately from the parts that actually call Gemini. That split meant I could verify the tricky logic (does character-name matching handle a chapter that mentions a character not in the portrait set? does the cap actually stop at 2?) without needing a live API key or hitting rate limits during every test run.

No database, on purpose

Projects and their state are stored as per-project, per-user JSON files on disk, using file locking (proper-lockfile) so concurrent requests can't race on writes to the same project file. No Postgres, no SQLite, no ORM. For a take-home assessment scoped to a single pipeline tool without multi-server deployment requirements, a database would have been solving a problem I didn't have — the file lock gives the same "no concurrent write corruption" guarantee a database transaction would, at a fraction of the setup cost.

What the frontend does — and deliberately doesn't do

The ProjectDetailPage polls the backend while a step is in a RUNNING state, and renders per-step status directly from what the server reports via status/stepState. It never tries to estimate or guess progress client-side — no fake progress bars ticking up based on elapsed time. If the server says a step is running, the UI shows running; if the server says complete or failed, that's what renders. It's a small restraint, but it avoids the specific flavor of bug where the UI tells the user something finished when the backend actually hasn't confirmed it yet.

State management is plain useState/useEffect per page — no Redux, no global store — because each page only ever needs its own project's data, never shared state across unrelated pages. Reaching for a state management library here would have added indirection without solving any problem the app actually has.

What I'd carry into the next project

The general pattern worth stealing: any time you have a multi-step process where each step is expensive, external, and possibly slow — API calls, long-running jobs, anything with real-world cost per attempt — build the "can this step run right now" check as an atomic operation from day one, and pair it with staleness detection for whatever happens when a step gets claimed but never finishes. It's tempting to skip this for a first version and add it "if it becomes a problem." In practice, double-submission is one of the first things that happens the moment a real user touches a slow UI, not an edge case you'll have time to bolt on later.