DEV Community: Ahmed

Deploy succeeded but production is broken: chunk 404s, stale builds

Ahmed — Mon, 12 Oct 2026 23:59:06 +0000

We read 4,704 public posts from builders whose apps broke at or after launch and verified 538 recent cases. Deploy and build is the largest category at 23%. This episode covers one deploy pattern we have 22 verified reports of: the pipeline says success and production is wrong.

From the builder's side it looks like this. The deploy is green and the app works on their machine. In production a stylesheet is missing, or the console shows 404s for JavaScript chunks, or an API route that answers instantly in local development ends in a 504. Sometimes an alias keeps serving an older build, or pushes to the production branch stop starting deploys at all. The host's log ends in success in each of these cases, so that is not where anyone looks.

The causes differ. This article reproduces the one you can trigger in five minutes, a tab that was open during a deploy, and then gives you two tests to run after every deploy.

Reproduce it

A Vite project with one lazily loaded module. Add "type": "module" to package.json.

<!-- index.html -->
<button id="go">Settings</button>
<pre id="out"></pre>
<script type="module" src="/src/main.js"></script>

// src/main.js
document.querySelector('#go').addEventListener('click', async () => {
  const { render } = await import('./settings.js')
  render(document.querySelector('#out'))
})

// src/settings.js
export const render = (el) => { el.textContent = 'settings v1' }

Treat live/ as the host. A deploy removes the previous assets and copies the new build in.

npm i -D vite
mkdir live
npx vite build && cp -r dist/. live/            # deploy v1
python3 -m http.server 4173 -d live &           # the host
# open http://localhost:4173 and leave that tab alone
echo "export const render = (el) => { el.textContent = 'settings v2' }" > src/settings.js
npx vite build && rm -rf live/assets && cp -r dist/. live/   # deploy v2
# now click the button in the old tab

The click does nothing. The network panel shows a 404 for /assets/settings-<hash>.js, and Chrome logs Failed to fetch dynamically imported module. A fresh tab works. That asymmetry is why this one is so hard to reproduce from a bug report.

Why it happens

Vite puts a content hash in each chunk's file name. The tab loaded HTML and an entry script that name settings-<hash>.js as it was at load time. The import() does not run until the click, so the tab asks for that exact file later. The new deploy has new hashes and the old files are gone, so the request returns 404 and the import rejects. The new chunks exist on the server. The old ones, which the open tab wants, do not.

The Vite docs describe this case and tell you to serve the HTML with Cache-Control: no-cache, so browsers do not keep referencing old assets, and to listen for the vite:preloadError event, which carries the original import error as event.payload.

The other shapes of green-but-wrong in our reports have different mechanisms:

Production is its own build. The host runs a clean install in its own container, with its own Node.js version, package manager and cache. Without a committed lockfile, ^4.0.0 can resolve to a newer release than the one on your machine. A framework upgrade can also move the build output, and the host uploads whatever is in the configured output directory.
A stale build cache. On Vercel the cache key includes the project, framework preset, root directory, Node.js version, package manager and Git branch. Change one of those and the output can differ from a clean local build.
Functions waiting on something unreachable. A missing environment variable, a database that refuses the host's network, and a code path that never sends a response all look like a hang from outside, then a 504.
Deploy automation switched off. After a Vercel Instant Rollback, auto-assignment of production domains is turned off, so new pushes to the production branch do not replace the rolled-back deployment. A missing repository webhook, an Ignored Build Step script that exits 0 and git.deploymentEnabled set to false stop deploys in the same quiet way.

Catch it before your users do

The first test loads a page, deploys again underneath it, and then uses the page. It runs in the reproduction project above. Replace deploy() with your own build and upload steps, or point it at a staging environment where a second deploy can be triggered from the test.

// skew.spec.js
import { test, expect } from '@playwright/test'
import { execSync } from 'node:child_process'
import { writeFileSync } from 'node:fs'

const deploy = (v) => {
  writeFileSync('src/settings.js', `export const render = (el) => { el.textContent = 'settings ${v}' }\n`)
  execSync('npx vite build && rm -rf live/assets && cp -r dist/. live/')
}

test('a tab open during a deploy still works', async ({ page }) => {
  deploy('v1')
  await page.goto('http://localhost:4173')
  deploy('v2')
  await expect(async () => {
    await page.click('#go')
    await expect(page.locator('#out')).toContainText('settings', { timeout: 1000 })
  }).toPass({ timeout: 10000 })
})

// playwright.config.js
export default {
  webServer: {
    command: 'mkdir -p live && touch live/index.html && python3 -m http.server 4173 -d live',
    url: 'http://localhost:4173',
  },
}

npm i -D @playwright/test && npx playwright install chromium
npx playwright test

As written it fails after 10 seconds, because every click retries a chunk that no longer exists. It passes when the page recovers, either by reloading once or by still being served the old chunk. Both are acceptable outcomes. A dead button is not.

The second test runs after every deploy, against the production domain and not only the deployment's own URL. A rolled-back alias or a disconnected Git hook only shows up on the domain. Run it from your CI on the push, not from the host's build: a host that has stopped deploying also stops running your checks. Have /api/health return the commit the build was made from, for example by writing git rev-parse HEAD into the build.

set -e
url=https://your-domain.example
curl -fsS "$url/" -o /tmp/index.html
asset=$(grep -oE '/assets/[^"]+\.js' /tmp/index.html | head -1)
curl -fsS "$url$asset" -o /dev/null
curl -fsS --max-time 10 "$url/api/health" | grep -q "$(git rev-parse HEAD)"

curl -f fails on any status of 400 or above, so a missing stylesheet or chunk fails the script. --max-time 10 turns a function that hangs into a failure instead of a long wait. The last line fails when the live commit is not the one you just pushed, which covers a stale alias and pushes that stopped deploying. Adjust the asset pattern to your framework's output path.

The fix

Reload once when a chunk fails, with a guard so a chunk that is really missing cannot cause a reload loop.

window.addEventListener('vite:preloadError', (event) => {
  const last = Number(sessionStorage.getItem('chunk-reload') || 0)
  if (Date.now() - last < 10000) return
  event.preventDefault()
  sessionStorage.setItem('chunk-reload', String(Date.now()))
  window.location.reload()
})

Then remove the cause where your host lets you:

Serve the HTML with Cache-Control: no-cache in your host's headers config.
On Vercel, enable Skew Protection. With a supported framework it pins framework-managed requests (static assets, client-side navigations, prefetches) to the deployment that served the page. It supports Next.js, SvelteKit, Qwik, Astro and Nuxt, on Pro and Enterprise plans, and the default maximum age is one day. Full-page navigations are not pinned by default: a hard refresh gets the latest production deployment, and a client that detects the mismatch reloads.
Pin the toolchain and install from the lockfile. npm ci exits with an error if package-lock.json and package.json disagree, and never rewrites either file. Netlify reads .nvmrc or .node-version before falling back to its default Node.js version.

node --version > .nvmrc
git add package-lock.json .nvmrc
npm ci

If a clean local build is right and production is not, rebuild once without the cache. On Vercel that is vercel --force.
Deploy the backend first and keep changes additive. Leave old fields and endpoints working until no deployed client uses them. Make every function return a response on every path, errors included, so a failure is a status code and not a timeout.
After an Instant Rollback, use Undo Rollback on the production deployment tile, or promote a deployment. Both turn auto-assignment back on.

vercel promote <deployment-url>

Checklist:

The smoke test passes against the production domain straight after a deploy.
A page left open before a deploy can still navigate, or reloads once and then works, with no chunk 404s left in the console.
A commit pushed to the production branch appears in the host's deployment list and goes live without manual promotion.
A clean local build (npm ci, then the host's build command) produces the same file list as the deployed output.

What does your post-deploy check assert beyond a 200 on the home page?

Migration says applied, but production says column does not exist

Ahmed — Mon, 05 Oct 2026 21:57:41 +0000

We read 3,426 public posts from builders whose apps broke at or after launch, and verified 418 recent cases. Database and data loss is 6% of them. Six verified reports in that data describe the same failure: the deploy succeeded, the migration tool says everything is applied, and production disagrees.

From the builder's chair it looks like this. API routes return 500 straight after a deploy, and the logs say column "plan_id" does not exist. You run prisma migrate deploy or supabase db push and it reports nothing to apply. The feature stays broken for days, because every tool you ask says the schema is current. The quieter version: a data migration finishes without an error, and weeks later someone notices records missing. The missing ones are nearly always rows without a parent, such as events with no tenant or orders with no account.

Both versions come from one habit: trusting a success message instead of checking the database.

Reproduce it

Silent data loss in a data migration

You need Docker and psql. Save this as repro.sql:

-- old schema: events.tenant_id was never a foreign key
CREATE TABLE tenants (id int PRIMARY KEY, name text NOT NULL);
CREATE TABLE events  (id int PRIMARY KEY, tenant_id int, payload text);

INSERT INTO tenants VALUES (1, 'acme');
INSERT INTO events VALUES
  (1, 1,    'signup'),
  (2, 1,    'upgrade'),
  (3, NULL, 'webhook, before tenants existed'),
  (4, 99,   'import from a deleted tenant');

-- the migration: move events into a tenant-scoped table
BEGIN;
CREATE TABLE tenant_events (
  id int PRIMARY KEY,
  tenant_id int NOT NULL REFERENCES tenants(id),
  tenant_name text NOT NULL,
  payload text
);
INSERT INTO tenant_events
SELECT e.id, e.tenant_id, t.name, e.payload
FROM events e
JOIN tenants t ON t.id = e.tenant_id;
DROP TABLE events;
COMMIT;

SELECT count(*) AS moved FROM tenant_events;

docker run --rm -d --name pgrepro -e POSTGRES_PASSWORD=pg -p 54329:5432 postgres:17
until pg_isready -q -h localhost -p 54329; do sleep 1; done
psql postgresql://postgres:pg@localhost:54329/postgres -v ON_ERROR_STOP=1 -f repro.sql

psql reports INSERT 0 2 and COMMIT, then moved is 2. Four events went in and two came out. The old table is gone, and ON_ERROR_STOP never had an error to stop on.

A history table that says "applied"

This one needs a Prisma project, so here is the sequence instead of a fixture. A migration that adds subscriptions.plan_id fails in production, and prisma migrate deploy now refuses to apply anything after it. Most people find this way to unblock it first:

npx prisma migrate resolve --applied 20261001120000_add_plan_id
npx prisma migrate deploy   # reports nothing to apply
npx prisma migrate status   # reports the schema as up to date, exits 0

SELECT column_name FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = 'subscriptions' AND column_name = 'plan_id';
-- (0 rows)

Every tool is green. The column doesn't exist, and every request that reads it returns 500.

Why it happens

Migration tools trust their history table, not your schema

prisma migrate deploy applies the migrations in your folder that _prisma_migrations doesn't list as applied. The Prisma CLI reference says it does not check for drift. migrate status compares the same two things, the migrations folder and the table. Prisma's baselining guide says prisma migrate resolve --applied "adds the target migration to the _prisma_migrations table and marks it as applied". It writes a row and runs no SQL. Any statement that hadn't run when the migration failed will now never run.

Supabase behaves the same way. supabase db push records each migration in supabase_migrations.schema_migrations, and its reference says "subsequent pushes will skip migrations that have already been applied". supabase migration repair --status applied inserts that record directly.

The history table is a claim about the schema, and nothing checks the claim again at deploy time.

The right command ran against the wrong database

If DATABASE_URL in the pipeline, or on the laptop that ran the command, points at development or a preview branch, the migration runs there and that database's history looks perfect. On Vercel, each variable is scoped to Production, Preview or Development, and the docs say changes "are not applied to previous deployments, they only apply to new deployments". Correcting the URL changes nothing until you redeploy.

Nobody ran it

Build and deploy are automated, and schema changes are not. The new code goes live expecting a column whose creation depends on someone remembering a command. If development and production share one database, it gets worse: every migration runs first against real data, so there is no stage where it can fail safely.

Joins skip rows, and SQL doesn't call that an error

An inner join returns only matching rows. NULL = 1 is not true, so event 3 never matches. No tenant has id 99, so event 4 doesn't match either. The INSERT ... SELECT inserted every row the SELECT returned, which is all it was asked to do.

UPDATE ... FROM works the same way, and the PostgreSQL docs spell it out: "if there is no match for a particular accounts.sales_person entry", the join form "will not update that row at all". The command tag reports how many rows changed, and a count of 0 "is not considered an error". With an UPDATE, skipped rows stay in their old state. With a DROP after the copy, as in the repro, they are deleted.

The orphans usually exist because the old column had no foreign key, and the missing constraint is often the reason for the migration in the first place.

Catch it before your users do

Test 1: diff the live schema against the repo

This test ignores the history table and inspects the database itself. That way it catches all three cases: a lying history table, the wrong database and a migration that never ran. Save as scripts/check-schema.sh:

#!/usr/bin/env bash
# Fails if the live schema differs from prisma/schema.prisma.
# Prisma 7: the URL comes from prisma.config.ts.
set -euo pipefail

# exit 1: unapplied or failed migrations, or diverged history
npx prisma migrate status

# exit 2: the live schema differs from schema.prisma
npx prisma migrate diff --exit-code \
  --from-config-datasource \
  --to-schema=prisma/schema.prisma

If your prisma.config.ts reads DATABASE_URL, run it with the production URL from your CI secrets:

DATABASE_URL="$PROD_DATABASE_URL" bash scripts/check-schema.sh

After the resolve --applied sequence above, migrate status passes and migrate diff exits 2. The diff is the check that sees the gap. To print the missing SQL, swap --exit-code for --script.

On Supabase, use supabase db diff --linked, which "diffs local migration files against the linked project" using a shadow database built in a separate container. The reference doesn't document an exit code for a non-empty diff, so check the output and fail closed:

supabase migration list
out=$(supabase db diff --linked 2>&1)
grep -q "No schema changes found" <<<"$out" || { echo "$out"; exit 1; }

If the CLI ever changes that message, the check fails loudly instead of passing quietly, which is the safer way to break.

Test 2: a data migration moves every row or refuses

Seed a scratch database with the rows that break joins and run the migration. The test fails on one outcome only: the migration succeeds with rows missing. Save as scripts/test-data-migration.sh:

#!/usr/bin/env bash
# Usage: bash scripts/test-data-migration.sh path/to/migration.sql
set -uo pipefail
DB=postgresql://postgres:pg@localhost:54329/postgres

psql "$DB" -q -v ON_ERROR_STOP=1 <<'SQL' || exit 1
DROP SCHEMA public CASCADE; CREATE SCHEMA public;
CREATE TABLE tenants (id int PRIMARY KEY, name text NOT NULL);
CREATE TABLE events  (id int PRIMARY KEY, tenant_id int, payload text);
INSERT INTO tenants VALUES (1, 'acme');
INSERT INTO events VALUES (1, 1, 'ok'), (2, NULL, 'no tenant'), (3, 99, 'dangling');
SQL

if psql "$DB" -q -v ON_ERROR_STOP=1 -f "$1"; then
  moved=$(psql "$DB" -tAc "SELECT count(*) FROM tenant_events")
  [ "$moved" = 3 ] || { echo "FAIL: migration succeeded but moved $moved of 3 events"; exit 1; }
fi
echo "PASS: every row moved, or the migration refused"

Copy the migration part of repro.sql (from BEGIN to COMMIT) into move_events.sql and run:

bash scripts/test-data-migration.sh move_events.sql
# FAIL: migration succeeded but moved 1 of 3 events

A migration that raises an error passes this test. It fails the deploy and loses nothing. In a real project, build the scratch schema by applying your earlier migrations, with prisma migrate deploy against the scratch URL or supabase db reset on the local stack. Then insert the orphan rows.

The fix

Make migrations a deploy step that can fail the deploy. On Render, the pre-deploy command runs after the build and before it goes live. "If any command fails or times out, the entire deploy fails", and the last successful deploy keeps serving. On Railway, a failed pre-deploy command "will not be retried and the deployment will not proceed". On other hosts, run migrations in a CI job that the production deploy depends on. Run the schema check in the same step, so a lying history table also fails the deploy:

npx prisma migrate deploy && bash scripts/check-schema.sh

Give production its own database and credentials. Use separate databases for development, preview and production, with each DATABASE_URL scoped to its environment in your host's settings. Keep the production URL out of local .env files, where a local command or a coding agent can pick it up.

Repair history only after checking the schema. Run the information_schema query first. If the change exists, resolve --applied or repair --status applied is correct. If it doesn't, the fix depends on the tool. On Supabase, supabase migration repair --status reverted <version> deletes the record, so the next db push applies the migration. With Prisma, create a new migration that contains the missing change and deploy it.

Decide in the migration what happens to orphans. You can refuse them, keep them with a LEFT JOIN and a nullable column, or move them to a quarantine table. Any of those is a decision. Dropping them by accident is not. The cheapest guard counts unmatched rows and raises an error. That rolls back the transaction and fails the deploy step:

DO $$
DECLARE orphans int;
BEGIN
  SELECT count(*) INTO orphans
  FROM events e
  LEFT JOIN tenants t ON t.id = e.tenant_id
  WHERE t.id IS NULL;
  IF orphans > 0 THEN
    RAISE EXCEPTION '% events have no tenant', orphans;
  END IF;
END $$;

Put it straight after BEGIN in move_events.sql and Test 2 passes. The migration stops with 2 events have no tenant, and nothing is dropped.

Read agent-written migrations before production runs them. If an AI coding tool wrote the SQL, read it and run it against a staging copy first. Keep production credentials out of the environment the agent runs in, so it can't apply a migration directly.

Checklist

[ ] Production deploys run migrations in a step that fails the deploy on error.
[ ] After every deploy, prisma migrate diff --exit-code or supabase db diff --linked runs against production and reports no changes.
[ ] prisma migrate status exits 0, or supabase migration list shows the same migrations locally and on the remote.
[ ] Development, preview and production use separate databases, and the production URL lives only in the production environment's settings.
[ ] Every data migration with a join has a guard for unmatched rows, and a test fixture with a NULL parent and a dangling parent.
[ ] Nobody runs resolve --applied or repair --status applied until the information_schema query confirms the change.
[ ] A deliberately broken migration on staging fails the deploy, and the previous version keeps serving.

The symptoms we see in the reports, and the fixes for each cause, are on the troubleshooting page: https://research.gemmein.com/guides/database-migration-not-applied-in-production

After your last migration went out, what told you it had worked: the history table or the database?

AI API call hangs in production, then 504: timeouts and budgets

Ahmed — Mon, 28 Sep 2026 23:04:35 +0000

We read 1,662 public posts from builders whose apps broke at or after launch and verified 215 recent cases. AI model and API failures make up 4% of them. Six of the verified cases share one pattern: an outbound or AI call with no timeout, no budget and no fallback. In most of the six, nobody noticed for a while, because the app didn't log model calls or their outcomes.

From the builder's side it shows up as one of these:

A spinner that runs for minutes, then a 504 from Vercel or a 524 from Cloudflare. On localhost it was fine.
Users read polite fallback text while the spend dashboard looks normal, because failed calls cost almost nothing.
An agent or background job burns a week's budget in a day. Then the provider's spend limit trips and every user is cut off at once.

All three have the same cause. The call has no limits of its own, so the only limits that apply are set by other layers.

Reproduce it

Here is a Next.js App Router route using the OpenAI Node SDK with its defaults. Most generated code ships in this shape.

// app/api/summary/route.ts
import OpenAI from "openai";

const openai = new OpenAI(); // defaults: 10 min per attempt, 2 retries

export async function POST(req: Request) {
  const { text } = await req.json();
  try {
    const r = await openai.chat.completions.create({
      model: "gpt-5-mini",
      messages: [{ role: "user", content: `Summarise this:\n${text}` }],
    });
    return Response.json({ summary: r.choices[0].message.content });
  } catch {
    // the user sees this with a 200, and nothing is logged
    return Response.json({ summary: "Summary unavailable right now." });
  }
}

You don't need a provider outage to see it fail. The SDK reads OPENAI_BASE_URL, so you can point it at a server that accepts connections and never replies:

# terminal 1: a provider that never answers
node -e "require('http').createServer(() => {}).listen(4010)"

# terminal 2
OPENAI_BASE_URL=http://127.0.0.1:4010/v1 OPENAI_API_KEY=test npm run dev

# terminal 3
curl -s -X POST localhost:3000/api/summary \
  -H 'content-type: application/json' -d '{"text":"hello"}' -w '\n%{time_total}s\n'

curl waits for at least 15 minutes. On Vercel, the same request is killed at 300 seconds with a 504 FUNCTION_INVOCATION_TIMEOUT before the catch runs, so nothing is logged. Behind Cloudflare the user gets a 524 even sooner.

Now make the stub answer 500 instead: node -e "require('http').createServer((q, s) => { s.statusCode = 500; s.end('{}') }).listen(4010)". After a few seconds of retries the route returns 200 with the fallback sentence. Every monitor you have counts that as a success.

Why it happens

Every layer between the browser and the model has its own clock. In the default setup the clock in your own code is the longest, so an outer layer always runs out first.

Layer	Default limit	What happens
Cloudflare proxy	125 s with no response from the origin	524 to the user
Vercel function (fluid compute)	`maxDuration` 300 s	function terminated, 504
Node's `fetch` (undici)	300 s waiting for response headers	connection error
OpenAI and Anthropic Node SDKs	10 minutes per attempt	`APIConnectionTimeoutError`

Here is what the official docs say about each part:

The SDK timeout is per attempt, and retries multiply it. By default, both official Node SDKs retry connection errors, 408, 409, 429 and 5xx twice with a short backoff, and they retry timed-out requests too. That is where the 15 minutes comes from: three attempts, each waiting at least 300 seconds. For non-streaming requests with a large max_tokens, Anthropic's SDK raises its default timeout above 10 minutes, up to 60.
A non-streamed response sends nothing until the model finishes. Every proxy in front of it sees an idle connection. Anthropic's SDK docs recommend streaming for long requests for this reason. The SDK also throws an error for a non-streaming request expected to run past roughly 10 minutes, unless you set a timeout.
The platform ends the process, not the promise. Vercel terminates a function that runs past its maxDuration. Your catch never runs, and neither does the logging inside it.
The fallback hides the errors you did catch. A catch that returns a 200 turns every provider error into a success in your metrics. Failed calls use few tokens, so the cost chart stays flat.
Nothing caps spend per user or per run. Provider spend limits are monthly and cover a whole project or workspace. When OpenAI's hard limit is reached, requests return 429 with the code project_spend_limit_exceeded. Because the SDKs retry every 429, each user's request retries an error that won't clear until the limit resets. Anthropic returns 400 invalid_request_error when a workspace spend limit is reached, and retry logic that only looks for 429 won't recognise it.

Catch it before your users do

This suite starts a fake provider on localhost with three behaviours: it never answers, it plays an agent that never finishes, or it returns a spend-limit 429. It imports the wrapper from the next section, and its assertions define what that wrapper has to do. It needs no API key and costs nothing to run.

// lib/ai-guards.test.ts
import http from "node:http";
import type { AddressInfo } from "node:net";
import { afterAll, beforeAll, expect, test } from "vitest";
import { callModel, makeClient, newRun, runAgent } from "./ai";

process.env.OPENAI_API_KEY ??= "test";
let hits = 0;

// One fake provider; the URL prefix picks its behaviour.
const server = http.createServer((req, res) => {
  hits++;
  if (req.url?.startsWith("/stall/")) return; // accept, never answer
  res.setHeader("content-type", "application/json");
  if (req.url?.startsWith("/spent/")) {
    res.statusCode = 429;
    return res.end(JSON.stringify({ error: { code: "project_spend_limit_exceeded", message: "limit reached" } }));
  }
  res.end(JSON.stringify({ // "/loop/": a model that never says it is done
    id: "c1", object: "chat.completion", created: 0, model: "stub",
    choices: [{ index: 0, finish_reason: "stop", message: { role: "assistant", content: "keep going" } }],
    usage: { prompt_tokens: 500, completion_tokens: 500, total_tokens: 1000 },
  }));
});
const client = (mode: string) =>
  makeClient(`http://127.0.0.1:${(server.address() as AddressInfo).port}/${mode}/v1`);
const hi = [{ role: "user" as const, content: "hi" }];

beforeAll(() => new Promise<void>((done) => { server.listen(0, "127.0.0.1", done); }));
afterAll(() => { server.closeAllConnections(); server.close(); });

test("a provider that never answers fails inside the deadline", async () => {
  const t0 = Date.now();
  await expect(callModel(client("stall"), newRun(), hi, 2_000)).rejects.toThrow();
  expect(Date.now() - t0).toBeLessThan(3_000);
});

test("an agent that never finishes stops at its step cap", async () => {
  const run = newRun(5, 1);
  await expect(runAgent(client("loop"), run, "plan my week")).rejects.toThrow("run_budget_exceeded");
  expect(run.steps).toBe(5);
});

test("a spend-limit 429 surfaces its code after at most one retry", async () => {
  hits = 0;
  await expect(callModel(client("spent"), newRun(), hi)).rejects.toMatchObject({
    status: 429, code: "project_spend_limit_exceeded",
  });
  expect(hits).toBeLessThanOrEqual(2);
});

npm i -D vitest
npx vitest run lib/ai-guards

Each test takes a few seconds. Run the suite in CI next to your other tests.

The fix

Give every call its own limits, shorter than every layer outside it, and make failures visible.

// lib/ai.ts
import OpenAI from "openai";

export type Run = { steps: number; maxSteps: number; spentUsd: number; capUsd: number };
export const newRun = (maxSteps = 1, capUsd = 0.05): Run => ({ steps: 0, maxSteps, spentUsd: 0, capUsd });

// 20 s per attempt, one retry. The deadline in callModel caps the total.
export const makeClient = (baseURL?: string) => new OpenAI({ baseURL, timeout: 20_000, maxRetries: 1 });

// Set these from your model's price page; at 0 the spend cap never trips.
const USD_PER_TOKEN = { input: 0, output: 0 };

export async function callModel(
  client: OpenAI,
  run: Run,
  messages: OpenAI.Chat.ChatCompletionMessageParam[],
  deadlineMs = 45_000,
) {
  if (run.steps >= run.maxSteps || run.spentUsd >= run.capUsd) throw new Error("run_budget_exceeded");
  run.steps++;
  const res = await client.chat.completions.create(
    { model: "gpt-5-mini", messages, max_completion_tokens: 800 },
    { signal: AbortSignal.timeout(deadlineMs) }, // one clock for the whole call, retries included
  );
  const u = res.usage;
  run.spentUsd += (u?.prompt_tokens ?? 0) * USD_PER_TOKEN.input + (u?.completion_tokens ?? 0) * USD_PER_TOKEN.output;
  return res;
}

export async function runAgent(client: OpenAI, run: Run, task: string) {
  const messages: OpenAI.Chat.ChatCompletionMessageParam[] = [{ role: "user", content: task }];
  for (;;) {
    const res = await callModel(client, run, messages);
    const text = res.choices[0].message.content ?? "";
    if (text.includes("[done]")) return text;
    messages.push({ role: "assistant", content: text }, { role: "user", content: "Continue." });
  }
}

The route now fails loudly and leaves a record:

// app/api/summary/route.ts
import { callModel, makeClient, newRun } from "@/lib/ai";

const client = makeClient();

export async function POST(req: Request) {
  const { text } = await req.json();
  const t0 = Date.now();
  try {
    const r = await callModel(client, newRun(), [{ role: "user", content: `Summarise this:\n${text}` }], 25_000);
    console.info(JSON.stringify({ tool: "summary", status: "ok", ms: Date.now() - t0, usage: r.usage }));
    return Response.json({ summary: r.choices[0].message.content });
  } catch (err: any) {
    console.error(JSON.stringify({ tool: "summary", status: "error", code: err?.code ?? err?.name, http: err?.status, ms: Date.now() - t0 }));
    return Response.json({ error: "summary_failed" }, { status: 503 });
  }
}

Pick deadlineMs below both your proxy's timeout and your function's maxDuration, with time left over to log and respond. If an answer really needs longer, stream it so bytes reach the proxy early, or return a job ID and have the client poll. With streaming, a 200 only means the stream started. Log the call as failed if the stream ends without a finish_reason.

For a per-user daily cap, add up the user's spend for today from the same log rows before creating the run, and pass what's left as capUsd. Once those rows are in a table, this query shows the failures you already have. It sorts by failure rate rather than spend, because failed calls barely move a cost chart:

select tool, count(*) as calls,
       avg((status <> 'ok')::int) as failure_rate
from ai_calls
where created_at > now() - interval '7 days'
group by tool
order by failure_rate desc;

Checklist

[ ] Every provider client sets timeout and maxRetries explicitly.
[ ] Every call has one overall deadline (AbortSignal.timeout), shorter than maxDuration and your proxy's timeout.
[ ] Long answers stream, or run as a background job that the client polls.
[ ] Every agent run has a step cap and a spend cap. Every user has a daily cap, checked before the call.
[ ] A failed call returns a non-2xx status and writes one log line with tool, status, error code and duration.
[ ] OpenAI's 429 project_spend_limit_exceeded and Anthropic's spend-limit 400 show the user a clear paused state instead of starting a retry loop.
[ ] A provider hard spend limit is set as a backstop, not used as the budget.
[ ] The three tests above run in CI.

Gemmein (in beta) handles the per-user half of this for you: its AI tools spend or reserve a person's credits before the provider is called, and every call is recorded with who, tool, tokens, credits and outcome.

Which layer in your stack has the shortest clock today, and did you set it yourself?

What breaks when AI apps go live

Ahmed — Fri, 25 Sep 2026 22:03:13 +0000

This is a living report. The current numbers and method are at research.gemmein.com, updated daily.

We grouped 59 recent, verified reports from builders whose apps broke near or after launch. Most failures fall in a few places: sign-in, hand-written access rules, file storage and payment webhooks.

We read 4,469 public builder posts, judged 916 of them against a fixed definition of "stuck going live", and kept only cases backed by a quote checked against the original post. 59 verified cases fall in the last 12 months.

The patterns

1. Sign-in and sessions that don't hold in production

What it looks like: Users can't finish signing in, or sign-up stalls. Reset or email links fail or never arrive. The browser shows a signed-in user as logged out, a paying customer loses access to their account, or a token refresh signs real users out mid-session.

Why it happens: Sign-in has several parts, and different layers own them. An email has to arrive, and its link has to redirect to the right origin. A cookie needs the right domain, SameSite and Secure attributes. Refresh-token rotation can race across tabs or parallel requests, and the server has to write and read session storage. Each part works in local dev on one domain. It breaks on the production domain, in a second tab, or when a mail provider's link scanner opens a one-time link before the user does.

The fix: Test the full flow on the production domain with a fresh inbox and a second browser. Check cookie attributes and the redirect allow-list. Set up SPF and DKIM on the sending domain. Make token refresh single-flight so only one request rotates the token. Log every auth failure with a reason, not just a 401.

2. Access rules that let one customer reach another's data

What it looks like: It passes every test with one account. Then someone finds they can read, change or delete other customers' rows through the public API. Anonymous visitors can write into paying customers' records, a read the client can call returns credentials, or a new account makes itself an admin.

Why it happens: Once the database key ships to the browser, row-level security is the only boundary, and someone has to write it by hand. With RLS off on a table, anyone holding the public key has full access. A policy of using (true), or one that only checks that someone is signed in, passes every single-user test. If roles come from metadata the user can edit at sign-up, or from claims the client sets, users can grant themselves admin. A security-definer function the client can call skips policies entirely.

The fix: Turn on RLS for every table in exposed schemas and deny by default. Write separate select, insert, update and delete policies tied to ownership (auth.uid()), and add with check on writes. Keep roles in a table users cannot write to, or in server-only metadata. Then test with two real accounts and an anonymous client: user B must fail to read, change or delete user A's rows.

3. Access rules that lock out the rightful user, or that nobody can verify

What it looks like: Sign-up fails with a permission error. Signed-in users get empty results for their own data, or shared screens show the wrong records. Builders close to launch are stuck on their access rules and can't tell whether the rules are right.

Why it happens: RLS denies by default, and a policy is missing or doesn't match. An insert policy may exist with no select policy, so the read after the insert fails. A profile row may be inserted during sign-up before a session exists, so auth.uid() is null. A policy may compare the owner against the wrong column or id type. Claims set on the server never reach the rule that checks them. Nothing in the stack says whether the rules are correct, and most apps are only tested by one developer signed in as themselves.

The fix: Create per-user rows from a server-side trigger or backend call, not from the client during sign-up. Write policies per operation, and read the real error instead of trusting an empty array. Add a two-account test to CI that checks both what must succeed and what must fail, and run it before every deploy.

4. Payment webhooks that never grant access

What it looks like: The customer pays and the provider shows the charge, but the app still shows the paywall. Live checkout completes and nothing happens, paid invoices never reach the app, or the webhook route returns 503.

Why it happens: Access depends on a webhook that the app has to receive, verify and act on. The signature check fails when the framework parses the body before verification (it needs the raw bytes), or when the test-mode signing secret is still set in live. The live endpoint may never have been registered or subscribed to checkout.session.completed. Missing production env vars make the handler fail. Retries arrive twice or out of order, and the handler isn't idempotent.

The fix: Verify against the raw request body using the live endpoint's own secret. Register the live endpoint and subscribe it to the events you use. Make the handler idempotent by event id and return 2xx quickly. Reconcile against the provider's subscription state on sign-in or on a schedule, so one lost event doesn't cost a customer their access.

5. The whole auth and data backend goes down at once

What it looks like: Production sign-in, data reads and file loads all fail at the same moment, and every customer is locked out. On hosted app builders, there may be nobody to escalate to.

Why it happens: Auth, database and storage sit in one project on one dependency path. An exhausted quota, a paused project, a rotated or expired key, an env var lost in a deploy, or an upstream incident takes all of them down together. Small apps rarely have health checks or alerts, so customers report the outage first.

The fix: Add an external uptime check that covers sign-in and one data read. Keep keys and env config in one place and check them after every deploy. Learn your provider's plan limits and pause rules. Show a clear error state in the app instead of a blank screen.

6. File storage behind hand-written policies

What it looks like: Either every upload fails for every user, or anyone can open other customers' files without signing in. Builders who ask for owner-only uploads can't get them working.

Why it happens: Storage has its own policy layer, separate from table rules and keyed on bucket and object path. A public bucket, or a policy that doesn't tie the path prefix to the user's id, exposes every file. A missing insert policy, or a path convention that differs between client and policy, blocks every upload. Once a public URL is shared, it bypasses the rules entirely.

The fix: Keep buckets private. Store objects under a path that starts with the owner's id, and write insert, select and delete policies that compare that folder to the signed-in user. Serve files through short-lived signed URLs. Test upload and download as two different users and as an anonymous visitor.

7. Subscription and pricing logic rebuilt by hand

What it looks like: A subscription charge is never created, an upgrade charges the wrong amount, or the app and the payment provider disagree about who is subscribed. Some apps trust a price sent from the client.

Why it happens: A subscription is a state machine (trial, active, past due, upgrade, proration, cancel) split between the provider and the app's own table. Hand-rolled code copies that state instead of deriving it, computes proration itself, or accepts an amount or price id from the browser.

The fix: Make the provider the source of truth for subscription state, and keep only a cache keyed by customer id. Never accept a price or amount from the client; map a plan key to a price on the server. Let the provider compute proration, and preview the upgrade invoice before confirming.

Method, briefly

Posts come from Stack Overflow, GitHub issues and discussions, Hacker News, community forums and public Discord support channels. We publish aggregates and paraphrased patterns only: no names, handles or quotes. The open data is at research.gemmein.com/data.json.