DEV Community: uttrai262005

Building a 5-Step AI Pipeline Without Letting Duplicate Requests Corrupt It

uttrai262005 — Sun, 13 Sep 2026 16:06:47 +0000

Building a 5-Step AI Pipeline Without Letting Duplicate Requests Corrupt It

I built a tool that turns a book's raw text into a full set of AI-generated illustrations, as a take-home technical assessment. The pipeline runs five sequential steps — style, characters, portraits, chapters, illustrations — using the Gemini API for both text generation and image generation. The interesting engineering problem wasn't calling an AI API. It was making sure two overlapping requests couldn't run the same step twice.

The pipeline, step by step

Each project moves through five stages in order:

Style — establish a consistent visual style for the whole book.
Characters — extract and describe the characters that appear.
Portraits — generate a reference image for each character (capped at 2 characters, to keep runs bounded).
Chapters — break the book into illustratable chapter units (capped at 1 chapter per run, same reasoning).
Illustrations — generate the actual illustrations, matching each chapter's named characters back to their portrait images for visual consistency.

The caps aren't arbitrary laziness — they're a deliberate bound on how much a single pipeline run can do, which matters a lot when every step is a paid, rate-limited external API call.

The bug you don't see until two requests race

Here's the failure mode that shaped the whole backend architecture: what happens if the frontend fires the same "run this step" request twice — a double-click, a retry after a slow response, a network blip that makes the client think the first request failed when it didn't?

Without protection, both requests would see the step in the same "not yet started" state, both would proceed to call Gemini, and you'd either burn double the API budget or end up with two conflicting results racing to write the same step's output.

The fix lives in services/pipeline.js, which acts as a small state machine sitting in front of everything else. When a step-run request comes in, it atomically claims the step — meaning the claim-and-check happens as one indivisible operation, not a check followed by a separate write with a gap in between where a second request could sneak in. If a second request arrives while the step is already claimed, it gets rejected with a 409 Conflict instead of being allowed to proceed. The route layer (steps.js) is the only place that knows about HTTP semantics; pipeline.js itself has no Gemini or filesystem knowledge beyond delegating to a storage lock — it just tracks state transitions.

There's a second piece to this: staleness detection. If a step gets claimed and then the process crashes, or the request times out without ever marking the step complete or failed, that step would otherwise stay claimed forever — permanently stuck, un-retryable. The pipeline service detects steps that have been claimed for too long without resolving and treats them as stale, making them claimable again. Without this, a single dropped connection would permanently soft-lock a project.

Splitting "what to send Gemini" from "how to talk to Gemini"

The service layer is split in a way that made testing much easier than it would've otherwise been. services/gemini.js is a thin REST client — it only knows the wire protocol for the Gemini Interactions API and the Files API's resumable upload flow. It has zero opinions about prompts, character caps, or app logic.

services/steps.js is where the actual pipeline logic lives: building prompts, applying the 2-character and 1-chapter caps, and matching chapter-named characters back to their corresponding portrait images so the illustration step can reference the right visual for each character. Critically, the pure logic in this file — the capping rules, the name-matching — is factored out and unit-tested completely separately from the parts that actually call Gemini. That split meant I could verify the tricky logic (does character-name matching handle a chapter that mentions a character not in the portrait set? does the cap actually stop at 2?) without needing a live API key or hitting rate limits during every test run.

No database, on purpose

Projects and their state are stored as per-project, per-user JSON files on disk, using file locking (proper-lockfile) so concurrent requests can't race on writes to the same project file. No Postgres, no SQLite, no ORM. For a take-home assessment scoped to a single pipeline tool without multi-server deployment requirements, a database would have been solving a problem I didn't have — the file lock gives the same "no concurrent write corruption" guarantee a database transaction would, at a fraction of the setup cost.

What the frontend does — and deliberately doesn't do

The ProjectDetailPage polls the backend while a step is in a RUNNING state, and renders per-step status directly from what the server reports via status/stepState. It never tries to estimate or guess progress client-side — no fake progress bars ticking up based on elapsed time. If the server says a step is running, the UI shows running; if the server says complete or failed, that's what renders. It's a small restraint, but it avoids the specific flavor of bug where the UI tells the user something finished when the backend actually hasn't confirmed it yet.

State management is plain useState/useEffect per page — no Redux, no global store — because each page only ever needs its own project's data, never shared state across unrelated pages. Reaching for a state management library here would have added indirection without solving any problem the app actually has.

What I'd carry into the next project

The general pattern worth stealing: any time you have a multi-step process where each step is expensive, external, and possibly slow — API calls, long-running jobs, anything with real-world cost per attempt — build the "can this step run right now" check as an atomic operation from day one, and pair it with staleness detection for whatever happens when a step gets claimed but never finishes. It's tempting to skip this for a first version and add it "if it becomes a problem." In practice, double-submission is one of the first things that happens the moment a real user touches a slow UI, not an edge case you'll have time to bolt on later.

Modeling Invoice Corrections as New Documents, Not Edits

uttrai262005 — Sun, 13 Sep 2026 16:01:41 +0000

Modeling Invoice Corrections as New Documents, Not Edits

I built a TypeScript/Express REST API for managing invoices through a controlled lifecycle — drafts can be edited, finalized invoices can be issued and exported as PDFs, and corrections are handled as replacement invoices rather than by mutating the original. Here's how the lifecycle works, and two real bugs I only caught by looking at the actual output.

Why an issued invoice can't just be "edited"

The core design decision in this project: once an invoice is issued, it becomes immutable. You can't PATCH its line items, its customer name, or its due date anymore. This mirrors how real invoicing and e-invoicing systems work — once an invoice is sent, its amounts are a fixed accounting record. If a mistake is found afterward, you don't rewrite history. You issue a correction.

The lifecycle has four states:

draft ──issue──────► issued
draft ──cancel─────► canceled
issued ──cancel────► canceled
issued ──replace───► replaced   (and a new invoice is created as `issued`,
                                  pointing back at this one)

A draft is editable and deletable, and can even have zero line items while it's being prepared. An issued invoice requires a customer name, at least one valid line item, and a due date — and from there it's a one-way door to either canceled or replaced.

The replace pattern, and why it needs a transaction

When a correction is needed, the API creates a brand-new, independently numbered invoice — issued immediately — linked back to the original via replacesInvoiceId. The original invoice is simultaneously marked replaced and linked forward via replacedByInvoiceId.

This needs two distinct Prisma relation names, because both foreign keys point back to the same Invoice model. It's an easy detail to get backwards the first time — I did, briefly — because Prisma doesn't complain loudly about ambiguous self-relations until you try to actually query both directions.

Both writes — creating the new invoice and updating the original — plus their status-history log rows, are committed inside a single Prisma $transaction. That's not a nice-to-have. Without it, a crash between step one and step two would leave the system in a state where invoice A claims it was replaced by invoice B, but invoice B doesn't exist yet — or worse, exists but doesn't know it's the replacement. A transaction makes that inconsistent state structurally impossible rather than just unlikely.

Two bugs that automated tests didn't catch

The test suite covers the domain state machine, validation, draft-only CRUD rules, lifecycle transitions, the transactional replace orchestration, and PDF generation — 56 tests, 97.95% domain coverage, 98.27% service coverage. And it still missed two real bugs.

Bug one: dividing by 100. The first PDF implementation divided every stored amount by 100 before rendering it — which is the correct move if you're storing USD in cents. This project defaults to VND, which has no practical minor subdivision. A 150000-cent value was rendering as VND 1,500.00 on the PDF instead of the correct VND 150,000. The unit test for this didn't catch it, because the test's expected value had been written with the exact same wrong assumption baked in. The test and the code were wrong in the same direction, so they agreed with each other perfectly. I only found it by generating a real invoice and reading the number with my own eyes.

That's the actual lesson here, more than the specific bug: automated tests validate that your code is consistent with what you expect — not that your expectations are correct. If the bug is in your assumptions rather than your logic, the test you write to catch it will often just encode the same mistake.

Bug two: text that wraps when you're not looking. The right-aligned totals lines — Subtotal, Tax, Total — were wrapping onto two lines for larger amounts, because they weren't given an explicit text width in PDFKit. The test suite used pdf-parse to extract text and check its content, which is exactly why it didn't catch this: extracted text reflects what the words say, not how they're laid out on the page. A wrapped line and an unwrapped line extract to the same string. Fixing it just meant giving the totals block a fixed position and width — but finding it required actually opening the generated PDF and looking at it.

Design choices that held up

A few decisions I'd make again without hesitation:

Integer money, always. Amounts are stored and calculated as integers, never floats, specifically to avoid floating-point rounding drift on currency math. The presentation layer is where currency-specific formatting belongs — not the storage layer.
A database sequence for invoice numbers, formatted as INV-YYYY-NNNN, assigned at creation time (even for drafts). A sequence is atomic under concurrent requests in a way that a naive count() + 1 query never is — two requests hitting that at the same millisecond will not collide.
PDFKit over Puppeteer. No headless browser dependency, no Chromium binary that might fail to launch in a constrained CI environment, faster tests. The tradeoff is a lower-level layout API — which is exactly what caused bug two above — but for a document with a fixed, simple layout, that tradeoff was worth it.
A layered architecture where the domain rules have zero dependency on Express or Prisma, and are fully unit-testable without touching a database at all. Repositories are the only layer that talk to Prisma; services combine domain rules with repository calls; routes stay thin.

The takeaway

If you're building anything that touches accounting data — invoices, ledgers, anything where "the record changed" needs its own record — model corrections as new, linked documents instead of in-place edits. It costs you a bit more schema complexity upfront (two relation names instead of one, a transaction instead of a single write) but it buys you an audit trail that can't silently drift, and it matches how the real-world process actually works. And whatever your test coverage percentage says, generate the real output and look at it — some bugs only exist in the gap between what your code returns and what a human sees.

Why I Stopped "Cleaning" Bad Data and Started Flagging It Instead

uttrai262005 — Sun, 13 Sep 2026 16:01:04 +0000

Why I Stopped "Cleaning" Bad Data and Started Flagging It Instead

I built a small internal tool for a support/fraud team as a take-home assignment: a dispute outcome tracker that closes the gap between disputed transactions and their final outcomes. Go backend, SQLite, React frontend, no auth, no Docker — deliberately scoped down to match what the assignment actually asked for.

The interesting part wasn't the CRUD. It was the seed dataset.

The dataset had seven distinct problems

I was handed 220 rows of "real-looking" dispute data to import. Buried in it were seven anomalies that don't show up until you actually look:

A duplicate case_id — two completely different rows sharing the same identifier.
A row missing user_id entirely.
A case marked status: open that somehow already had an outcome.
An outcome value of "maybe" — which isn't a valid enum value in any outcome-tracking system I've seen.
A negative amount (-42.5, on a transaction that should never be negative).
A created_at timestamp dated in 2027 — from the future, relative to the rest of the dataset.
A case with a captured outcome but an empty outcome_note, which shouldn't be possible if the capture flow is followed correctly.

My first instinct, honestly, was to write an importer that quietly fixes what it can and drops what it can't. That instinct was wrong, and it took me a minute to see why.

Fixing bad data hides the thing you built the tool to find

The entire point of this tool is to give a fraud/support team visibility into disputes that need attention. If the import script silently "fixes" a negative amount by flipping its sign, or drops the duplicate case_id, or quietly reclassifies "maybe" as "pending" — the team loses the one signal that actually matters: something upstream is broken, and a human needs to know.

So I flipped the approach. Every anomaly gets imported as-is, alongside a new ingest_warning column that flags exactly what's wrong with it. An analyst can pull up GET /api/cases?has_warning=true and see every flagged row in one place, with the specific issue attached. Nothing is silently dropped. Nothing is silently repaired.

There was one exception, and it's worth explaining because it's not actually a contradiction. The "maybe" outcome value never gets written into the outcome column — because doing so would corrupt every trend and count query downstream that assumes outcome only ever holds valid enum values. But the original "maybe" string isn't discarded either — it's preserved in the warning text, so nothing about the anomaly is lost. The rule isn't "never touch invalid data." It's "never let invalid data quietly become valid-looking data."

Auditability mattered more than mutability

The other design decision that shaped everything: outcomes aren't stored as a single mutable field that gets overwritten every time someone corrects it. Instead, there are two tables — cases, which holds the current denormalized state, and outcome_events, an append-only ledger where every capture and every correction is a new row.

This meant a correction to an outcome required a reason — the first capture doesn't, but any correction after that does, and the API rejects a reasonless correction without mutating the row at all. That single rule is enforced and tested directly against a real (in-memory) SQLite database, not mocked out, because it's the kind of rule that's easy to get subtly wrong at the boundary.

I also masked user_email and device_id by default in the list view — showing something like da***@inbox.test instead of the full address — with a ?reveal=true param and the single-case detail view as the only ways to see the unmasked value. It's not bulletproof PII protection, and I said so directly in the design doc rather than pretending it was more than it is. But defaulting to masked, rather than defaulting to exposed, felt like the right failure mode for a tool that fraud analysts would be using daily.

Testing the anomalies, not just the happy path

Every one of the seven seed-data anomalies got its own named unit test — things like TestNormalizeSeedRow_DuplicateCaseID_CASE00213 — so if a future change to the import logic breaks handling for one specific case, the test failure names the actual case_id that broke. That's a small thing, but it turns "some validation test failed" into "CASE-00213's duplicate handling regressed," which is a much faster thing to debug at 11pm.

What I'd tell someone building something similar

If you're building an ingestion pipeline for messy real-world data — disputes, transactions, support tickets, anything with a human downstream who needs to act on it — resist the urge to make your importer "smart" about cleaning things up. Smart importers that silently correct bad input are optimizing for a clean-looking database at the cost of the one thing your users actually need: knowing where the data is untrustworthy.

Flag it, log it, make it queryable. Let a human decide what "fixed" means. Your job is to make the mess visible, not to make it disappear.