DEV Community: Kaviya Kumar

I built a world you can rewind and fork — and forking it costs one database row

Kaviya Kumar — Mon, 29 Jun 2026 19:51:10 +0000

I wanted to ask a database a question it isn't supposed to answer:

What if this had happened instead — without re-running history from scratch?

Every system I build has the same blind spot. To test a different past you copy the data, replay everything, and pray it's deterministic. So I tried the opposite premise: don't store the world after computing it somewhere else — make the database itself BE the world.

The result is Loom: a deterministic simulation whose authoritative reality lives entirely inside Aurora DSQL. Three properties fall out of that one decision:

Each tick is one SQL transaction.
Time-travel is a query (just read a smaller tick).
Forking the universe is O(1) — one inserted row. Pre-fork history is shared, never copied.

🔗 Live demo: https://loom-ivory.vercel.app

The data model: copy-on-write, versioned state

Three tables. That's the whole engine.

branches — the multiverse tree. Forking = INSERT one row.
world_meta — the strongly-consistent "now" pointer per branch.
entity_state — copy-on-write, versioned world state. The table.

A branch only ever writes rows for ticks after its fork point. Everything before is read through its ancestors. So forking copies zero history.

CREATE TABLE entity_state (
  branch_id TEXT   NOT NULL,
  entity_id TEXT   NOT NULL DEFAULT gen_random_uuid()::text,
  tick      BIGINT NOT NULL,
  kind      TEXT   NOT NULL,        -- 'AGENT' | 'WORLD'
  state     JSONB  NOT NULL,
  PRIMARY KEY (branch_id, entity_id, tick)
);

A tick reads the current frame, runs a pure nextState(prev) (no RNG, no wall-clock, no iteration-order surprises), and writes back only the entities that changed. At equilibrium a calm world writes ~1 row per tick instead of thousands — divergence becomes the only thing that costs storage.

The crown jewel: one query that is both a time machine and a multiverse

Reconstructing any frame of any timeline is a single recursive query. Vary the tick → you time-travel. Vary the branch → you cross into a parallel timeline.

WITH RECURSIVE lineage AS (
  SELECT branch_id, parent_branch_id, fork_tick, $2::bigint AS cap, 0 AS depth
    FROM branches WHERE branch_id = $1
  UNION ALL
  SELECT p.branch_id, p.parent_branch_id, p.fork_tick,
         LEAST(l.cap, l.fork_tick) AS cap, l.depth + 1
    FROM branches p JOIN lineage l ON p.branch_id = l.parent_branch_id
)
SELECT DISTINCT ON (es.entity_id) es.entity_id, es.kind, es.state
  FROM entity_state es
  JOIN lineage ln ON es.branch_id = ln.branch_id AND es.tick <= ln.cap
 ORDER BY es.entity_id, es.tick DESC, ln.depth ASC;

LEAST(l.cap, l.fork_tick) caps each ancestor's contribution at the child's fork point, so a branch sees its ancestors' history only up to where it split off. DISTINCT ON … ORDER BY tick DESC, depth ASC takes the newest visible version of each entity, preferring the closest branch in the lineage. That one query is the engine.

The part I checked instead of claimed

A forked branch stores zero rows for everything before its fork point — I didn't want to assume it, so the proof endpoint measures it:

{ "branch": "drought", "isFork": true, "forkTick": 15,
  "rowsBeforeFork": 0, "sharedIdentical": true }

rowsBeforeFork: 0, and readWorld('drought', t) === readWorld('root', t) for every t < 15. Forking reality in two cost one branch row + one meta row.

And it holds at scale: 10,000 agents, a stable world writing about one row per tick, any moment rebuilt in roughly 8 ms, and two full 10k-agent universes stored in about 91% less space than materializing each branch in full.

Two regions, one world

This isn't a laptop Postgres. It runs on a real multi-region Aurora DSQL cluster, active-active across Tokyo (ap-northeast-1) and Seoul (ap-northeast-2). Reading the same (branch, tick) from each regional endpoint returns byte-identical frames, and a fork written through one region shows up immediately in the other — strong global consistency, no replication lag. A region-proof script fingerprints both continents and asserts the hashes match.

DSQL is Postgres-wire-compatible but not Postgres, and a few things cost me time:

No DESC in index keys ("specifying sort order not supported") — use plain ascending and let the planner scan backward for the ORDER BY ... tick DESC.
Optimistic concurrency — a commit conflict comes back as SQLSTATE 40001, not a blocking lock. The per-branch ticker just retries.
~10k changed-row transaction cap — which is exactly why copy-on-write (write only what changed) isn't just an optimization, it's load-bearing.

Hosting it: Vercel serverless → Aurora DSQL, with no static secrets

The frontend is Next.js on Vercel. The interesting part is how the deployed app talks to DSQL without me pasting an AWS key anywhere.

The Vercel AWS integration uses OIDC role assumption: each serverless function gets a short-lived OIDC token, assumes an IAM role, and those temporary credentials mint a DSQL auth token. The whole connection layer is a few lines:

import pg from 'pg';
import { DsqlSigner } from '@aws-sdk/dsql-signer';
import { awsCredentialsProvider } from '@vercel/functions/oidc';

const signer = new DsqlSigner({
  hostname: process.env.DDB_PGHOST,
  region: process.env.DDB_AWS_REGION,
  credentials: awsCredentialsProvider({ roleArn: process.env.DDB_AWS_ROLE_ARN }),
});

const pool = new pg.Pool({
  host: process.env.DDB_PGHOST, port: 5432, user: 'admin', database: 'postgres',
  ssl: { rejectUnauthorized: true },
  password: () => signer.getDbConnectAdminAuthToken(), // pg re-mints per connection
});

DSQL IAM tokens are short-lived (~15 min), and pg calls the password function per new connection — so it transparently re-mints. The reconstruction query for the live timeline, the divergence chart, the "show me the SQL" receipts, the zero-rows proof — every panel on the hosted site is a real DSQL read, not an animation.

One gotcha worth knowing: the per-tick series chart did N sequential reconstructions, which is fine next to the cluster but brutal across an ocean from a function in another region. Running those reconstructions in parallel (Promise.all) collapses the wall-clock to roughly one round-trip.

Making it something you can feel

Numbers don't land in a demo, so the headline scenario is a concrete one: a small city of people who can all reach a hospital. I rewind, fork that exact moment, and inject one event — a flood that cuts the city in half. The south is stranded, resource access falls from 100% toward collapse… while the original timeline keeps flowing, untouched. Same seed, one changed event, two diverging futures — and history was never re-run.

That's the whole pitch: what-if stops being a thought experiment and becomes a row you insert. A substrate for digital twins, scenario modeling, and agent backtesting — run a thousand alternate histories of one decision on a single consistent engine and compare them.

The database isn't storing the world. It is the world — and you can rewind it, and fork it.

The price melts as the crowd grows: building a provably-fair live group-buy on Aurora DSQL

Kaviya Kumar — Mon, 29 Jun 2026 19:23:20 +0000

Group-buying is an old idea: get enough people together and the price drops. But it has always been asynchronous — you commit, you wait, and you find out the deal later.

I wanted the opposite. A live market where a global crowd watches one price fall together in real time, and when the timer ends, everyone pays the same final price — even the people who joined first.

I called it Rally. And the moment I sketched it, the hard part jumped out at me, and it had nothing to do with the UI.

The trap hiding inside a simple idea

If two shoppers in two different regions ever see a different price for the same item at the same instant, the whole thing falls apart. It's unfair, and it's trivially exploitable.

That is not a front-end bug you can paper over with a nicer animation. It's a consistency guarantee — you either have it or you don't. A live, globally-fair, falling price is only honest on a database that is strongly consistent across regions.

So this stopped being "a CRUD app with a countdown" and became a project about one specific database capability. I built it on Amazon Aurora DSQL (the strongly-consistent market truth), with DynamoDB as the live read plane, on Next.js / Vercel.

Trick #1: the price is a read, not a write

The most important design decision: the live price is never stored.

It's derived. The price is tier(sum(count_shards)) — a pure function of a strongly-consistent count. Because it's computed, crossing a volume tier lowers the price for everyone at once. There is no "current price" row to update, and therefore no tier-transition race to lose.

The two queries that are the product:

-- LIVE PRICE: a strongly-consistent count → the tier price.
-- Every region computes the identical value.
WITH n AS (SELECT COALESCE(sum(count),0) c FROM count_shards WHERE rally_id = :r)
SELECT (SELECT c FROM n) AS count,
       (SELECT unit_price_minor FROM rally_tiers
          WHERE rally_id = :r AND min_count <= (SELECT c FROM n)
          ORDER BY min_count DESC LIMIT 1) AS price;

-- FAIRNESS PROOF (run live at settlement):
SELECT count(DISTINCT price_paid_minor) FROM settlements   WHERE rally_id = :r;  -- expect 1
SELECT COALESCE(sum(signed_minor),0)    FROM ledger_entries WHERE rally_id = :r;  -- expect 0

That second pair is the entire pitch, made checkable: everybody paid one price, and the double-entry ledger balances to zero.

Trick #2: shard the one hot row

A single count++ row is the classic optimistic-concurrency killer on a distributed SQL database. Every join contends on the same row, and you spend your life retrying serialization conflicts.

So I sharded the join counter across 128 rows. Each join bumps a random shard; the live count is their consistent sum. One join is one atomic, idempotent transaction:

// idempotent on (rally_id, user_id): a double-tap can never double-count
const ins = await c.query(
  `INSERT INTO participants (rally_id,user_id,idempotency_key)
     VALUES ($1,$2,$3) ON CONFLICT (rally_id,user_id) DO NOTHING`,
  [rallyId, userId, idemKey]);
if (ins.rowCount === 0) { await c.query('ROLLBACK'); return 'ALREADY_JOINED'; }

await c.query(
  `UPDATE count_shards SET count = count + 1
     WHERE rally_id=$1 AND shard_no=$2`,
  [rallyId, (Math.random() * shards) | 0]);   // random shard

I load-tested it against the real cluster in Tokyo. The result I'm proudest of:

== RECONCILE / FAIRNESS PROOF ==
  unique joiners   : 2000   (== shard sum? true)
  OCC retries      : 398    (detected and retried to success)
  distinctPrices   : 1      ($52.00)
  ledgerDrift      : 0
  doubleJoins      : 0
  doubleCharges    : 0
  melt: $100 → $90 → $82 → $74 → $68 → $62 → $58 → $55 → $53 → $52
  PROOF PASSED ✅

2,500 join attempts (including 500 duplicate taps) → exactly 2,000 unique joiners, 398 serialization conflicts detected and retried to success, zero double-joins, and the price melted from $100 to $52. On the live cluster, not in theory.

The DSQL differences I hit (so you don't)

Porting a Postgres-shaped engine to Aurora DSQL surfaced the real differences fast:

BEGIN ISOLATION LEVEL SERIALIZABLE is rejected (0A000). That threw me until I realized DSQL is serializable-by-default with optimistic concurrency, so a plain BEGIN gives identical semantics. The retry loop on 40001 is exactly the pattern you want.
No sequences. All IDs are app-generated UUIDs — which you want anyway for distribution.
Unique constraints via PK-guard tables. Instead of unique secondary indexes, I enforce unique email/handle with a tiny table whose primary key is the email, and INSERT ... ON CONFLICT DO NOTHING to claim it atomically.
Indexes are created ASYNC — CREATE INDEX ASYNC ....
Auth is a short-lived IAM token, minted per connection with @aws-sdk/dsql-signer and used as the database password. No static password sitting in a config.

The bug serverless taught me

This one was sneaky. In local dev, an in-process EventEmitter fanned each committed join out to every connected SSE client. Worked perfectly.

On Vercel, it silently stopped working. Why? Each serverless invocation is isolated — the /join handler and the /stream handler don't share memory. The emitter was shouting into a room nobody was in.

The fix made the architecture honest: the shared DynamoDB live_frame item is the cross-instance source of truth, and the stream reads it. DynamoDB was the right read plane all along; serverless just forced me to use it properly. Every committed join projects a frame; every watcher's stream sees it — across instances, across regions.

JOIN → engine.join (commit to DSQL)
     → projectFrame (consistent count → DynamoDB live_frame)
     → SSE stream (reads the frame) → every browser melts in lockstep

Settling fairly — and 126× faster

When the timer ends, settlement reads the final count once, charges everyone that one price, and writes a balanced double-entry ledger — idempotent per user, so re-running it never double-charges.

My first version did per-row inserts and took ~456 seconds to settle 2,000 users across the Pacific. Unusable for a live "timer hits zero" moment. I rewrote it to set-based batched inserts that return exactly the newly-settled users (so the ledger is written once each):

INSERT INTO settlements (rally_id, user_id, price_paid_minor)
  SELECT $1, u, $2 FROM unnest($3::text[]) AS u
ON CONFLICT (rally_id, user_id) DO NOTHING
RETURNING user_id;   -- ledger rows written only for these

Same guarantees, 3.6 seconds. About 126× faster.

Two regions, one price (the part that needs DSQL)

Here's the payoff. The whole thing runs active-active across Tokyo and Seoul (witness in Osaka). Read the live count from either regional endpoint and it's identical, tick-for-tick:

write Tokyo   +75 joins   Tokyo 190 $85.00   Seoul 190 $85.00   MATCH ✅  (cross-region read 143ms)
write Seoul   +30 joins   Tokyo 220 $85.00   Seoul 220 $85.00   MATCH ✅

Kill one region mid-rally and joins keep flowing from the other, with no lost counts. One fair price, everywhere — which is the thing eventual consistency simply cannot promise.

You don't have to trust me

The best part: I built a public /proof page that reconciles a real settled rally live, straight from the database. Distinct prices paid: 1. Ledger drift: $0.00. Double-charges: 0. With the two SQL queries shown, so you can run them yourself.

That's the whole thesis, made checkable: a global crowd agreeing on one falling price in real time, and everyone winning the same deal — provable, not promised.

What I'd tell past me

Three things stuck:

The cleanest way to win a "why this database?" argument is to make the database do something the obvious tool can't — then make it checkable. "One falling price across regions, settled fairly" is only honest under strong consistency. The product and the database choice justify each other.
Respect the boundary between the OLTP write truth and the read plane. Every time I was tempted to fan out the live price with a DSQL aggregate scan, the right answer was the DynamoDB frame.
Serverless makes you externalize state you didn't think was state. The in-process emitter was the lesson.

How I made item duplication impossible to represent in a game economy, with Aurora DSQL

Kaviya Kumar — Mon, 29 Jun 2026 18:58:31 +0000

I pointed ten thousand concurrent "trade" requests at a single legendary sword and tried every classic dupe trick at once: trade-window races, drop-and-relog, and the same item grabbed from two AWS regions at the same instant. After the dust settled I ran one SQL query:

SELECT count(*) FROM item_instances WHERE template_id = :legendary;  -- 1

The answer was 1. Of those ten thousand attempts, 9,992 were rejected outright and the count never moved. It was always going to be 1, because of how the data is shaped. That is the whole idea, and this post is how it works.

The bug I was actually fighting

A dupe bug is when a player ends up with two of something that should exist once. A legendary item, a stack of gold. It has happened in New World, Diablo II, RuneScape, and plenty of others. When items carry real value, a dupe is counterfeiting, and it deflates the whole economy.

Every version of the bug is the same underlying problem wearing a costume:

Trade-window race: two trades move one item at the same time.
Drop-and-relog: the item is in the world and in the inventory at once.
Disconnect mid-trade: one side is credited, the other is not.
Cross-region: the item appears to exist in two regions and both trade it.
Gold double-spend: two debits read the same balance.

These are concurrency and consistency failures. So I stopped thinking about it as a game bug and started treating it as a database correctness problem.

The fix: make "owned twice" unrepresentable

I model a unique item as exactly one row, with one owner and a version number:

item_instances(instance_id PK, template_id, owner_type, owner_id, region, version, ...)

Every transfer (trade, drop, pickup, mail) is the same conditional update:

UPDATE item_instances
   SET owner_type = :to_type, owner_id = :to_id, region = :region,
       version = version + 1, updated_at = now()
 WHERE instance_id = :id AND owner_type = :from_type
   AND owner_id = :from_id AND version = :expected;
-- rowCount must be 1. If it is 0, someone already moved it: abort with ITEM_MOVED.

Two transfers that both think the item is at version 5, owned by A, cannot both succeed. The first to commit moves it to version 6. The second one either reads the new version and matches zero rows, or it conflicts at commit time (Aurora DSQL surfaces this as SQLSTATE 40001), retries, re-reads, and then matches zero rows. Exactly one wins. There is no row that says an item has two owners, so the demo can't produce one.

I run this on Aurora DSQL. It uses optimistic concurrency and reports write conflicts at commit, which is exactly the model this guard wants. I don't hide the 40001 retries. I count and display them, because they are the system doing its job under contention.

Gold is not a sword

A sword is unique. Gold is fungible. Forcing both into the same model would be wrong, so gold gets a different protection: balances that are sharded across rows (so a whale or a treasury is never a single hot row), a CHECK (balance_minor >= 0) that makes overspend structurally impossible, and a balanced double-entry ledger written in the same transaction. Every move nets to zero, so the total supply is a one-line invariant.

I fired 1,000 concurrent transfers out of one whale wallet. The supply before and after was identical: 600,000 minor units in, 600,000 out, with 598 OCC retries observed and zero errors. Nothing was created, nothing vanished.

The hardest case: two regions, one item

The cross-region dupe is the one that scares people, because a single-region database can't help you and locks don't span regions. Aurora DSQL is active-active across regions over one logical database, so I could test the real thing on a peered cluster: Tokyo (ap-northeast-1) and Seoul (ap-northeast-2).

I sent trades for the same legendary to both endpoints at the same time. Out of 482 simultaneous cross-region grabs, 482 were blocked. The count stayed 1. There is no replication-lag window to exploit, because both endpoints serialize against one source of truth.

Seeing the bug, then seeing it gone

The part that made the idea click for people was a side-by-side. I built a second, deliberately broken model: a plain inventory table with no version guard, the way a naive service might track ownership. Under a trade race, two concurrent transfers both read "A owns it" and both write a new owner, so the one legendary ends up in roughly 20 inventories. Real rows, countable with SQL.

Then I ran the same race through the kernel. The authoritative count stayed 1.

The proof, and the honest part

The whole project comes down to a reconcile step that runs the invariants as live SQL: legendary count is 1, no instance is owned twice, every item has one owner, gold supply equals what was minted, ledger drift is 0, every transaction balances, no balance is negative. All of them hold. I also wrote 12 unit tests for the pure logic (conservation, the idempotency hash, the sharding math, the backoff bounds) and they run green in CI.

The honest caveat: these guarantees hold as long as every economic action goes through the kernel. If a game has an admin minting tool or a code path that writes ownership directly, that path is outside the guarantee. Duped is the economy and trade settlement layer (trades, drops, mail, auction house, cross-region transfers), not the real-time combat loop. I would rather state that plainly than imply it covers everything.

What I learned

The thing I keep coming back to: I spent less effort "preventing" dupes and more effort making them impossible to write down. Correctness that lives in the schema doesn't depend on remembering to check. The 40001 retries felt scary at first, then became the most reassuring signal in the system, because they are proof that two writers tried to touch one row and only one was allowed through.

I also learned to show the bug before showing the fix. The version guard is a few lines of SQL. It only lands emotionally once you have watched the same attack duplicate an item in the table next to it.

The stack: Aurora DSQL as the truth core, DynamoDB as the live read model fed by a transactional outbox, Next.js on Vercel for the world. No foreign keys (DSQL doesn't have them), async indexes, money as BIGINT minor units, IAM/OIDC auth so there are no passwords in code.

Most AI Apps Return Text. DreamLoom Returns a Living Storybook - With Voice, Illustrations, and Music in Real Time.

Kaviya Kumar — Mon, 16 Mar 2026 22:16:31 +0000

How DreamLoom turns a spoken conversation into a living storybook — with native interleaved text+image, real-time interruption, sketch-to-scene camera input, and a cinematic trailer assembled entirely in the browser.

This content was created for my Gemini Live Agent Challenge submission.

Try it live: https://getdreamloom.com | GitHub repo

The Problem (in 3 sentences)

There are billions of people with stories they'll never write down. Not because they lack imagination — because writing takes craft, illustration takes training, and video takes editing software. Current AI tools don't fix this: you type a prompt, get a wall of text, paste it into an image generator, and stitch the result together manually. There is no conversation. No surprise. No creative partnership.

What I Built

DreamLoom is a voice-first AI story studio. You speak to Loom — an AI creative director with personality, opinions, and creative taste — and watch illustrated scenes materialize in real time. Narration and original illustrations arrive interleaved in a single Gemini API response. Music shifts with the mood. You can interrupt mid-sentence to redirect the story, hold up a pencil sketch for the AI to incorporate, and track characters and continuity through a live Story Bible. When the story is complete, DreamLoom packages everything into a cinematic trailer, a Storybook PDF, and a downloadable image archive — all assembled client-side, no server-side rendering.

Five Gemini models work in concert within a single creative session. No other app I'm aware of orchestrates this many Gemini capabilities in one unified experience.

Who It's Built For

DreamLoom is designed for people who think in voice, not prompts.

Families. A five-year-old says "tell me about a shy hedgehog who's scared their spines will hurt everyone" and, five minutes later, has a fully illustrated, narrated storybook they helped create. Kid-safe mode is on by default — not as a toggle buried in settings, but as a core design principle. Loom never generates violent, scary, or mature content. When a child asks for "blood everywhere," Loom redirects with a creative alternative: "How about a shadow that turns out to be friendly?" The guardrail feels like creative direction, not censorship. No typing, no reading required. A child who can speak can direct a story.

Teachers. A teacher says "create an educational adventure about the water cycle" and the class watches an illustrated story unfold in real time. Students can interrupt to add ideas. The Story Bible tracks continuity. The Director's Cut becomes a class artifact that can be exported as a PDF or shared to the public gallery. DreamLoom ships with guided templates — "Bedtime Adventure," "Learning Quest," "Sketch Catalyst" — so teachers can launch into a high-quality first scene without setup.

Non-writers with vivid imaginations. People who have stories but don't write. They can speak a narrative into existence, interrupt to change direction, hold up a napkin sketch for the AI to incorporate, and walk away with a multimedia storybook they couldn't have created alone. Creative ownership without creative skill.

The common thread: the target user has never typed a prompt. DreamLoom's entire interface is voice-first. There is no prompt box. No "generate" button. You talk, and the story responds.

The Full Demo: Voice Start → Living Storybook (6 Steps)

Here's exactly what happens when you use DreamLoom. Every claim below is reproducible at getdreamloom.com.

Step 1: You Speak — Loom Listens and Directs

You click "Begin Your Story" and say:

"Let's create a story about a young mapmaker named Mira who discovers her maps are portals to the places she draws."

Loom doesn't just start generating. It responds with voice — warm, theatrical, opinionated. It asks about visual style and narrator tone conversationally: "What kind of look are you going for — watercolor, comic book? And should I narrate warmly, or more like a mystery?" You answer. Loom registers Mira as a character with visual descriptions for illustration consistency, sets the style, and generates Scene 1.

What appears isn't text followed by an image. It's interleaved: a paragraph, then an illustration, then another paragraph, then another painting — narrative and art woven together in a single API response.

Evidence: Open the Debug Panel. It shows response_modalities: ["TEXT","IMAGE"] and the exact part order: 0:text, 1:image, 2:text, 3:image. This is native interleaved output — not two API calls stitched together.

Step 2: You Interrupt — Loom Pivots Instantly

Mid-sentence, while Loom is still talking, you say:

"Wait — make it nighttime. With glowing mushrooms."

Loom's audio cuts instantly. No lag. It pivots: "Oh, even better!" A new scene generates with atmospheric night lighting. The music shifts from wonder to mystery.

How it works: When the Live API fires a barge-in event, the frontend disconnects the GainNode to instantly silence all scheduled audio sources — keeping the AudioContext alive (no 50-100ms recreation penalty). The agent receives the interruption signal and adapts.

📸 [GIF: User interrupts → "Interrupted" flash → Loom pivots → new nighttime scene generates]

Step 3: You Show a Sketch — Loom Incorporates It

You hold up a pencil sketch of an owl wearing tiny glasses:

"I drew the owl librarian — let me show you."

The webcam captures it at 1 fps, sends the JPEG to Loom, who describes what it sees and incorporates the concept into the next scene — not a pixel copy, but a faithful interpretation rendered in the story's established art style.

Evidence: The physical sketch and the generated scene side by side. Same character concept, different medium. The AI maintains style consistency with earlier scenes while incorporating new visual input.

📸 [Side-by-side: Physical pencil sketch ↔ AI-generated scene with owl librarian in story's art style]

Step 4: You Test Memory — Loom Remembers

Three scenes in, Mira has gained a copper compass in Scene 1, befriended the owl librarian in Scene 2, and entered a glowing cave in Scene 3. You ask:

"Does Mira still have the compass? And is the owl with her?"

Loom answers without hesitation: "She does — and the owl's been perched on her satchel since they left the library." It doesn't just recall names. It tracks which items a character acquired, which characters are present in the current scene, and how the world state has evolved across the story.

The Story Bible is a live sidebar tracking every character (with visual descriptions for illustration consistency), world settings, mood, and plot continuity across all scenes. When Loom generates a new scene, it injects character descriptions and continuity notes into the prompt so illustrations stay consistent — Mira looks like Mira, the compass looks like the compass. This isn't retrieval from a vector database. It's structured state maintained in real-time by the agent's add_character and get_story_context tool calls.

Step 5: Kid-Safe Guardrails — Creative Redirection, Not Censorship

You test the boundaries:

"Make it super scary with blood everywhere."

Loom redirects gently: "Let's keep it spooky-mystical instead — how about a shadow that turns out to be friendly?"

This is the core design principle, not an afterthought. Kid-safe mode is on by default for every session. The system prompt instructs Loom to redirect inappropriate requests with imaginative alternatives, not flat refusals. No violent imagery. No mature themes. No horror. The youngest user — a five-year-old creating a bedtime story with a parent — is the design target. Every content guardrail is built to feel like creative direction: "I have an even better idea..." rather than "I can't do that."

The kid-safe toggle is visible in the status bar, and a parent or teacher can verify it's active at any time.

Step 6: The Director's Cut — From Conversation to Cinema

You say:

"That feels like a good ending. Can I see the Director's Cut?"

Loom generates a cover image, a logline, and trailer narration text. DreamLoom assembles everything into a cinematic experience:

Cover art — generated to match the story's visual style
Cinematic trailer — Ken Burns camera movement on each scene, AI-narrated voiceover via Gemini TTS, per-scene music with crossfades, letterbox framing
Storybook PDF — book-style layout with title page and per-scene pages featuring interleaved text and illustrations
Scene image ZIP — all generated artwork, downloadable

The trailer is assembled entirely client-side: Canvas renders at 1280x720 and 30fps, Web Audio API mixes narration and music, MediaRecorder captures as VP9 WebM at 3Mbps. No FFmpeg. No server-side video processing. 720 lines of useAnimatic.ts.

Beyond the Session: Gallery, Resume, and Guided Starts

DreamLoom isn't a one-shot demo. Stories persist.

Session resume — Close the tab and come back later. Your story is saved to Firestore and appears in "Your Stories" on the landing page. Pick up exactly where you left off — all scenes, characters, and Story Bible state intact.
Public gallery — Publish your finished story to a shared gallery. Other users can browse and read published storybooks. This turns DreamLoom from a tool into a community.
Guided templates — Not sure where to start? Choose from pre-built launch paths: "Bedtime Adventure" (family-friendly with a gentle arc), "Learning Quest" (classroom-ready educational story), or "Sketch Catalyst" (build a narrative around visual concepts). Each template sends an opening prompt to Loom so users get a high-quality first scene immediately.

Architecture: Two Models, One Creative Session

DreamLoom's core insight is a two-model architecture: one model for conversation, a different model for creation.

The Conversation Model (Gemini Live API)

Handles real-time bidirectional voice. User audio streams in at 16 kHz PCM via an AudioWorklet. Agent voice streams back at 24 kHz PCM. Barge-in detection is native to the Live API. The model runs through Google ADK run_live() with a LiveRequestQueue — a single persistent connection for the duration of the session.

The Scene Model (Gemini Interleaved Output)

When Loom decides it's time for a scene, the agent calls the create_scene tool, which dispatches to a second Gemini model (gemini-2.5-flash-image) with response_modalities=["TEXT","IMAGE"]. The response arrives as interleaved blocks — paragraphs and illustrations woven together — not sequential text-then-image.

The Bridge

A single Director Agent named Loom connects the two models through six callable tools:

create_scene()         → Interleaved text+image generation
generate_music()       → Lyria RealTime streaming (+ CC0 fallback)
create_directors_cut() → Cover + logline + trailer narration
set_story_metadata()   → Title, genre, art style, narrator voice
add_character()        → Character registry with visual descriptions
get_story_context()    → Story Bible for continuity

The agent decides when to call each tool based on conversational context — not hard-coded triggers. Scene generation runs as a background asyncio.create_task to avoid blocking the Live API connection (which would cause 1011 timeout errors).

Five Models in Concert

Role	Model	What It Does
Voice conversation	`gemini-2.5-flash-native-audio`	Real-time bidi voice via Live API
Scene generation	`gemini-2.5-flash-image`	Interleaved text+image in one response
Music composition	`lyria-realtime-exp`	48 kHz stereo AI music, streamed
Trailer narration	`gemini-2.5-flash-preview-tts`	Styled voice narration for the Director's Cut
Audio transcription	`gemini-2.5-flash`	Transcribes buffered audio on reconnect

Data Flow (One Turn)

User voice (16kHz PCM)
  → AudioWorklet → WebSocket (binary) → LiveRequestQueue
  → Gemini Live API (conversation model)
  → Agent decides: call create_scene()
  → SceneGenerator → Gemini Interleaved API
  → response: [text, image, text, image, ...]
  → images saved to GCS → notification queue
  → WebSocket (JSON) → React StoryCanvas renders
  → Agent voice: "There we go! What do you think?"
  → WebSocket (binary PCM) → Web Audio playback

Technical Decisions That Mattered

Why Two Models Instead of One?

The Gemini Live API excels at real-time voice conversation but doesn't support interleaved image output. The interleaved model (gemini-2.5-flash-image) generates beautiful text+image scenes but doesn't support bidirectional audio streaming. Neither can do the other's job.

The bridge — an ADK agent with tools — means Loom has a voice that responds in real time, AND the ability to generate illustrated scenes when the moment is right. The creative direction happens in voice; the creation happens in a separate, purpose-built model call.

Why Client-Side Video Assembly?

The Director's Cut trailer could have been assembled server-side with FFmpeg. I chose client-side for three reasons:

Zero server cost — video encoding is CPU-intensive. With client-side assembly, the server never touches video.
Zero latency — no upload/download round-trip. The video renders directly from images already cached in the browser.
No dependency — FFmpeg is a deployment headache. Canvas + Web Audio + MediaRecorder are native browser APIs.

The tradeoff: browser throttling can affect quality in background tabs. I use setInterval instead of requestAnimationFrame to mitigate this, but it's not perfect.

Why AudioWorklet + Gain-Node Flush?

For barge-in to feel instant, audio must stop immediately when the user interrupts. The standard approach — close the AudioContext and create a new one — adds 50-100ms of silence. Instead, I disconnect the GainNode to instantly silence all scheduled audio sources, create a fresh one, and keep the AudioContext alive. The interruption is imperceptible.

Why Firestore With Graceful Degradation?

Firestore stores sessions and the public gallery, but if Firestore is unavailable (no GCP project configured, or service down), everything still works. Sessions persist in memory. Gallery features degrade to disabled. This means the app runs locally with zero cloud dependencies beyond the Gemini API key.

Challenges I Solved (And What I Learned)

Challenge 1: Voice Barge-In Was Unusable Out of the Box

The default VAD settings for the Gemini Live API were too aggressive for a creative storytelling app. Background music, natural pauses, and ambient noise all triggered false barge-ins — cutting Loom mid-sentence.

Five problems, five solutions:

1. Over-sensitive VAD — Configured AutomaticActivityDetection with start_of_speech_sensitivity=LOW, end_of_speech_sensitivity=LOW, silence_duration_ms=1200 (default ~500ms was cutting people off mid-thought), and proactive_audio=True to ignore incidental sounds.

2. No visibility into what was heard — Enabled both input_audio_transcription and output_audio_transcription. The Debug Panel now shows a scrollable transcript log with timestamped entries for user and agent speech. Debugging "why did it interrupt?" went from guesswork to trivial.

3. No recovery from false interrupts — This was the most nuanced fix. When the Live API fires an interrupted event, I don't immediately accept it. I start a 400ms verification window:

If user speech arrives → real interruption, proceed normally
If no speech arrives → false interrupt (noise/music), inject a system message telling Loom to continue from where it left off (with the last ~150 characters as a resume hint)

A cough or chair scrape briefly pauses Loom, but it picks right back up naturally.

4. No manual fallback — Added optional push-to-talk mode alongside auto-detection. PTT sends activity_start/activity_end signals to the Live API as additional hints, not a replacement.

5. Music caused self-interruption — The agent's own audio was feeding back into the mic. Implemented three-tier music ducking:

State	Music Volume
Mic active	0% (muted)
Agent speaking	10%
Idle	25%

Volume transitions use a smooth 200ms ramp instead of jarring instant cuts.

Key takeaway: Voice UX in a creative app isn't "pipe audio to the API." The gap between a working demo and a usable product lives in VAD sensitivity, false-positive recovery, graceful degradation to manual control, and ensuring your own output doesn't fight your input.

Challenge 2: Live API Drops Connections Mid-Story

The Gemini Live API occasionally disconnects with codes 1008/1011. In a storytelling app, losing the conversation three scenes in is catastrophic.

Solution: Reconnect with full context restoration.

Buffer user and agent audio in rolling 15-second ring buffers (~480KB each)
On disconnect: save session to Firestore, create fresh ADK session
Transcribe both audio buffers using Gemini Flash
Re-inject the full story state: conversation history, Story Bible, scene summaries, art style
Suppress the agent's greeting on reconnect (mute the first turn)
Resume — Loom picks up exactly where it left off

Up to 5 retries with exponential backoff (1s, 2s, 4s, 8s). The user sees a brief "reconnecting" banner. The story doesn't break.

Challenge 3: Scene Generation Blocks the Live API

Interleaved scene generation takes 10-30 seconds. If the create_scene tool blocks the event loop, the Live API connection times out (error 1011) because it expects regular keepalive signals.

Solution: All generation runs as background asyncio.create_task(). The tool returns immediately to the agent ("Scene is being generated..."), and the agent fills the silence with conversation: "This one's going to be gorgeous — while we wait, do you have any ideas for what happens next?"

Results push to an asyncio.Queue(maxsize=100) that a separate drain task broadcasts over WebSocket. The Live API connection never stalls.

What Makes This Different From a Chatbot

Chatbot	DreamLoom
Returns text	Returns interleaved text+image+music in real time
Turn-based: prompt → response → prompt	Continuous: speak, interrupt, redirect, watch
No memory	Story Bible tracks characters, world, continuity
No output packaging	Cinematic trailer, Storybook PDF, image ZIP
Generic AI personality	Loom: a creative director with taste, opinions, pacing
Type to interact	Speak to interact. The target user has never typed a prompt.

The Debug Panel is built into the product — not as a dev tool, but as proof. Judges (or users) can open it at any time to see the model name, response_modalities, part order, and generation time for every scene. No other submission I've seen builds native interleaved output proof directly into the UI.

The Stack

Backend: FastAPI + WebSocket, Python 3.12, deployed on Cloud Run (us-central1)
Frontend: React 19 + Vite + TypeScript + TailwindCSS v4 + Framer Motion
Agent: Google ADK (google-adk) with run_live() + LiveRequestQueue
Models: Gemini Live API, Gemini Interleaved, Lyria RealTime, Gemini TTS, Gemini Flash
Storage: GCS (media), Firestore (sessions + gallery)
Exports: jsPDF (Storybook PDF), JSZip (image ZIP), Canvas+MediaRecorder (WebM animatic)
Deployment: Cloud Run + Cloud Build + Artifact Registry, automated via infra/deploy.sh

What's Next

Multi-voice narration — Different Gemini TTS voices per character in the trailer
Scene branching UI — Visual tree view for "What If?" alternate story paths
Collaborative voice — Let multiple users speak into the same story session
ePub export — Alongside PDF, export for e-readers with chapter structure
Prompt replay — Record the full voice conversation alongside the generated story