DEV Community: Emmanuel

Our deploy timed out: one static worker, 60 seconds per page

Emmanuel — Thu, 05 Nov 2026 19:05:37 +0000

I run a site with practice exams for Dutch certification exams. Next to the exams there are guides: how buoys work for the small boating licence, which sound signals you need to know, how to book the exam. On 24 October a deploy failed, and the cause was a function that every one of those pages called without anyone thinking about it.

Here's what the build log said, what was actually slow, and the rule we wrote down afterwards.

What the build log said

At the time we were on Vercel's Hobby plan. The log listed the build machine as "2 cores, 8 GB", and Next.js 16.4.2 generated the static pages with a single worker:

Generating static pages using 1 worker (0/187) ...
Failed to build /(nl)/(site)/klein-vaarbewijs-1/betonning/page: /klein-vaarbewijs-1/betonning (attempt 1 of 3) because it took more than 60 seconds. Retrying again shortly.
...
Failed to build /(nl)/(site)/klein-vaarbewijs-1/bpr-en-rpr/page: /klein-vaarbewijs-1/bpr-en-rpr after 3 attempts.
Export encountered an error on /(nl)/(site)/klein-vaarbewijs-1/bpr-en-rpr/page: /klein-vaarbewijs-1/bpr-en-rpr, exiting the build.

Next.js gives each page 60 seconds to render during static generation. That's the default of staticPageGenerationTimeout. A page that takes longer is retried, up to three attempts, and if it still doesn't finish, the whole build fails.

Eight guides for the Klein Vaarbewijs 1 (the small boating licence) ran past the limit. One of them, the guide on BPR and RPR (the inland shipping rules), failed all three times. The deploy stopped after about 5 minutes 40 seconds, and production kept serving the previous build.

These are not heavy pages. The guide on buoys is text, some images and links to related guides. None of that should take a minute.

It had been close for a while

When I compared the logs of the deploys before it, the problem wasn't new:

An earlier deploy already hit "took more than 60 seconds" on four reference pages about navigation lights, such as the lights of a large motor vessel. They passed on a retry, and that build took about 18 minutes.
A redeploy of the same code built 184 pages with no timeouts, in 13 minutes.
The failing deploy had 187 pages, and this time the retries didn't save it.

So the build had been sitting right at the limit, passing or failing by luck. A plain redeploy might have passed, and that would have hidden the problem until the next batch of pages.

The cause: every page re-read the whole content tree

Every question on the site comes with JSON files: a draft, a check by a second model and a verification by a third. They ship with the deployment, in content/checks/<exam>/ and content/verifications/<exam>/. A guide only goes live once enough of its exam's questions have passed both checks, so deciding which guides exist means reading those files.

The function that decides is publishedTopics(). It's shared by the sitemap and the navigation, and every page renders the navigation. For each of about 100 topics, it evaluated the topic's state like this:

export function getTopicState(slug: string) {
  const topic = getTopic(slug);
  return evaluateTopic(topic, loadCheckFiles(topic.examId), loadVerificationFiles(topic.examId));
}

And this was the loader underneath:

function readDir<T>(dir: string, schema: { parse(data: unknown): T }): T[] {
  if (!existsSync(dir)) return [];
  return readdirSync(dir)
    .filter((f) => f.endsWith(".json"))
    .sort()
    .map((f) => schema.parse(JSON.parse(readFileSync(join(dir, f), "utf8"))));
}

Nothing is wrong with either function on its own. The problem is how often they ran. An exam's check and verification files were read, parsed and validated once for every guide of that exam, just to render one navigation. Then again for the next page, and the next. With one worker, every page waits for the one before it.

Turbopack had even pointed at the file, in the same log:

./lib/content/load.ts:19:18
Warning: The file pattern (<dynamic> '/content/checks/' <dynamic> | …) matches … files
Overly broad patterns can lead to build performance issues and over bundling.

That warning is about bundling (it can't narrow down a dynamic fs path), not about render time. But it was the right file.

The fix: parse once per process

The content files can't change inside a deployment, so there's no reason to parse them more than once. lib/content/load.ts now keeps the parsed directories in a Map for the life of the process, in production only:

const parsed = new Map<string, unknown[]>();

function readDir<T>(dir: string, schema: { parse(data: unknown): T }): T[] {
  const cache = process.env.NODE_ENV === "production";
  const hit = cache ? parsed.get(dir) : undefined;
  if (hit) return hit as T[];
  const files = existsSync(dir)
    ? readdirSync(dir)
        .filter((f) => f.endsWith(".json"))
        .sort()
        .map((f) => schema.parse(JSON.parse(readFileSync(join(dir, f), "utf8"))))
    : [];
  if (cache) parsed.set(dir, files);
  return files;
}

Why this works in Next.js:

next build runs with NODE_ENV=production, so the cache is on during static generation, not only on the live site.
Module state lives as long as the worker process. With one static worker, the first page pays for parsing and every page after it reuses the result.
In development and tests the cache is off, because there the files do change. A test pins both behaviours (trimmed):

vi.stubEnv("NODE_ENV", "production");
const first = loadCheckFiles("nl-kvb1");
expect(loadCheckFiles("nl-kvb1")).toBe(first);
// after vi.unstubAllEnvs():
expect(loadCheckFiles("nl-kvb1")).not.toBe(loadCheckFiles("nl-kvb1"));

The numbers: ten calls to publishedTopics() went from 3,445 ms to 187 ms. Our local build rehearsal dropped from about 10 minutes to under 4.

We also raised the timeout, with a comment that says what it is:

// The real fix is the parsed-content cache in lib/content/load.ts; this is headroom.
staticPageGenerationTimeout: 180,

Raising the timeout alone would have made the build pass and stay slow. Every new guide would have pushed it a bit closer to 180 seconds instead of 60. I wanted the timeout as a seatbelt, not as the fix.

The same pattern at runtime

The build was just where it showed first. Later that day we found the same pattern on the dynamic pages: the exam start, the result page, checkout and the account page evaluated topic states on every request, about 0.7 seconds of work for all topics. A second cache, one level up, now keeps the evaluated state per topic in production:

export function getTopicState(slug: string) {
  const cached = topicStateCache.get(slug);
  if (cached) return cached;
  const topic = getTopic(slug);
  const state = evaluateTopic(topic, loadCheckFiles(topic.examId), loadVerificationFiles(topic.examId));
  if (process.env.NODE_ENV === "production") topicStateCache.set(slug, state);
  return state;
}

The rule we wrote down

Our decision log now says: nothing a page renders may re-read the whole content tree per call. Load once, then reuse.

That isn't specific to JSON files. Anything in a shared layout, header or navigation runs once for every page in the build. If it does I/O or parsing that grows with your content, build time grows with pages times content. The site has grown since that day (the sitemap lists 193 URLs now), and with the old loader every one of those pages would have paid for it again.

What I'd tell someone else on Next.js 16

Read the "Generating static pages using N worker" line. On a small build machine N can be 1, and then every slow page adds straight to the build time.
Treat "took more than 60 seconds … Retrying" as a failure, even when the retry passes. Ours passed on retry before the day it didn't.
Look for work in shared components that grows with your content. Navigation and sitemaps are the usual suspects, because every page or every build touches them.
Cache what can't change inside a deployment at module level, in production only, and test that development still reads fresh.
Raise staticPageGenerationTimeout as headroom, not as the fix, and say so in the comment next to it.

We've since moved to Vercel Pro, but the rule stays. More build power would have hidden the problem, not fixed it.

I wrote this post with help from an AI assistant. The log lines, code and numbers come from our own repository.

Six practice exams from a thin question bank

Emmanuel — Thu, 05 Nov 2026 19:04:35 +0000

I build practice exams for Dutch certification exams. Each exam has a fixed set of practice exams, "Oefenexamen 1" to "6", and each one follows the official format: one question per slot, where a slot is a topic with a set number of points.

For most exams that's easy. The hospitality licence (SVH) has 40 slots with 6 to 8 verified questions in each, so six forms that never share a question come out on their own.

The Klein Vaarbewijs 2, the Dutch boating licence for open and tidal water, was different. Its 27 slots hold between 0 and 8 verified questions. The GPS slot has 2. The tide calculation slot has 2. One slot has none. Here's how we still got six forms out of it, and why they're a JSON file in the repo instead of a random draw.

What the engine did before

The original form builder is simple. Per slot, shuffle the questions with a fixed seed and deal them out: form 1 gets the first, form 2 the second, and so on. First it checks that every slot has enough:

/**
 * Builds the fixed practice exams ("Oefenexamen 1..n"). Forms never share a question,
 * and the same pool + seed always gives the same forms, so a form number stays stable.
 */
export function assembleForms(exam: ExamDefinition, pool: PoolQuestion[], seed: number): ExamItem[][] {
  const count = exam.forms.fixedCount;
  const short = coverage(exam, pool).filter((c) => c.have < c.needForForms);
  if (short.length > 0) {
    throw new Error(`not enough questions for ${count} forms: ...`);
  }
  // ...seeded shuffle per slot, then deal
}

With the KVB2 bank that throws, and every form shows as "binnenkort" (coming soon). Waiting for 6 verified questions per slot would take weeks, and for some slots it wasn't going to happen at all. The GPS slot failed four drafting rounds in our checks: the new questions had cues that gave the answer away, or relied on inference.

The obvious fix, and why I didn't use it

Let questions repeat. Keep the runtime draw, add a counter, and allow each question in at most N forms. That's a small change to the function above.

The problem is what the runtime knows about a question:

export interface PoolQuestion {
  id: string;
  version: number;
  topicId: string;
  slot?: number;
  code?: string;
  key: AnswerKey;
}

No sources. And two of our rules need them:

Source clashes. Two questions that rest on the same sentence of the same source don't go in one form. One tends to answer the other.
Exclusive pairs. Some questions give each other away without sharing a sentence. KVB2 lists 8 such pairs. One example: a few slot 17 questions ask what ALAT or NAP is (two reference levels for water height), and a slot 18 question rests on knowing both. Put them in the same form and the second one is free.

A random draw with a max-uses counter keeps the counting rule and loses both of these.

Assembling the forms offline

So KVB2's forms are assembled from the content files, where every draft still has its sources, and written to content/forms/nl-kvb2.json as question codes. The exam definition reads that file, and the engine serves exactly those forms.

Each draft becomes a candidate:

export interface Candidate {
  id: string;
  slot: number;
  /** Answer letter; "" for a question without one (a sleepvraag, ...). */
  key: string;
  units: string[];
  sourceIds: string[];
}

units are the source sentences the draft rests on, found by matching its verbatim quote against the source text split into sentences and lines. Two candidates that share a unit clash.

The assembler fills one form at a time, thinnest slot first. For each slot it ranks the candidates: least used so far, then the answer letter this form has the fewest of, then a seeded random tie-break. Then it filters:

let fits = ranked.filter((c) => !clashes(c, [...chosen.values()], opts.caps, opts.exclusive));
let usable = fits.filter(mayUse);

mayUse is the use cap, with a per-slot override:

const cap = opts.maxUsesBySlot?.[c.slot] ?? opts.maxUsesPerQuestion;
return cap === undefined || (used.get(c.id) ?? 0) < cap;

After a form is filled, a repair pass swaps questions from the most common answer letter to a rarer one, as long as the swap breaks no rule. The whole thing runs 200 times with different seeds and keeps the best result: fewest unfilled slots first, then the best key balance, then the least reuse. It's deterministic, so the same bank always gives the same forms.

For KVB2 the call looks like this:

const assembly = assembleFixedForms(slots, candidates, {
  forms,
  keys,
  caps: kvb2Retirements.caps,
  exclusive: kvb2Retirements.exclusive.map((e) => e.ids),
  seed: KVB2_FORM_SEED,
  maxUsesPerQuestion: KVB2_MAX_USES, // 2
  maxUsesBySlot: KVB2_USES_BY_SLOT,  // { 18: 3, 23: 3 }
  omitted: (nlKvb2.forms.omittedSlots ?? []).map((o) => ({ slot: o.slot, reason: o.reason })),
});

Where the limits came from

Each limit is a decision we wrote down, with a reason.

At most 2 forms per question means a slot needs 3 verified questions to fill all six forms. That became the floor: below 3, the slot goes back to drafting.

At most 3 forms for slots 18 and 23, which got stuck at 2. Further drafting kept producing questions that failed the checks. Someone who does all six forms sees those questions three times. We decided that's better than leaving the slot empty or forcing in weaker questions.

One slot left out. Slot 21 is a course calculation with wind drift, worth 3 points. No public source states the rule, and three drafting rounds failed. So it's omitted by decision. Each form has 26 questions instead of 27, scoring is on 47 points instead of 50, and the pass mark becomes 33 (70 %, rounded up). The Klein Vaarbewijs 2 page says so before anyone buys. The assembler takes omitted as an option, so the slot isn't reported as a gap.

The result from the current bank:

6 forms × 26 questions = 156 places.
139 different questions fill them: 126 used once, 9 twice, 4 three times.
Forms 1 and 2 share no questions. Form 6 shares 8 with earlier forms.
Every form has 7 or 8 each of a, b and c as the right answer.

Here's the part that surprised me. The bank holds 156 eligible questions, exactly as many as there are places. It still can't make six forms without repeats, because slots with 7 or 8 questions have leftovers while slots 18 and 23 run dry. The total doesn't matter. The thinnest slot does.

For SVH the same assembler runs only as a report. It confirms the bank supports six forms without sharing a question, and it writes a drafting brief: which slots are short, how many new drafts they need, and which answer letters would let every form balance.

Forms must not drift

Fixed forms have a second job: staying put. "Oefenexamen 3" has to be the same exam tomorrow as today, or a retake tells you nothing. Two tests guard that.

The first pins the committed file to the assembly of the current bank:

const { assembly } = assembleKvb2Forms(kvb2Bank().candidates);
expect(nlKvb2.forms.fixed).toEqual(kvb2FixedForms(assembly));

If a question is retired or a new one passes the checks, the assembly can change, and then the test fails until someone regenerates the file. So the forms change only in a commit that says so, never as a side effect.

The second came with the engine change. To serve KVB2, the engine learned fixed assignments and omitted slots. Every other exam had to come out exactly as before. The test copies the old engine verbatim from the commit before the change and compares:

for (const seed of [20261020, 1, 7])
  expect(JSON.stringify(assembleForms(exam, p, seed))).toBe(JSON.stringify(legacyAssembleForms(exam, p, seed)));

Two more details help. An attempt stores its own list of questions when it starts, so an exam in progress keeps its questions even if a form changes. And if a fixed form names a question that isn't in the database, that form isn't offered at all. No gap, no substitute.

What I'd take from this

If a rule needs data the runtime doesn't have, do the draw where the data is, and ship the result as a file.
Count the thinnest slot, not the bank. A bank with enough questions in total can still fail.
Make caps per slot, and write down why each one exists.
Leaving a slot out is better than padding it. Score on the points that are there and tell people before they pay.
Pin generated output with a test, so it changes only on purpose.

The tide slots are the thinnest part of this exam. If you're curious what they cover, there's a guide to tides for the Klein Vaarbewijs 2 (in Dutch). The roomier bank is Sociale Hygiëne.

I wrote this post with help from an AI assistant. The code, the numbers and the decisions come from our own repository.

Our exam questions store buoys as data, and a broken spec shows text instead of a guessed picture

Emmanuel — Thu, 05 Nov 2026 19:03:42 +0000

I build practice exams for Dutch certification exams. One of them is the Dutch boating licence, and many of its questions are pictures: a buoy, a traffic sign on a bridge, the lights of a boat at night seen from one side. The questions are written by a language model.

That's a risk you don't have with text. If a buoy is drawn with the wrong topmark, the question doesn't just look odd. It teaches the wrong rule, and the candidate learns it from us.

So no question in our content contains an image. Each one contains a small typed spec, and our own SVG components draw the picture from it. Here's how that works, and the one rule I care most about: a spec that doesn't parse renders as plain text, never as a picture we guessed.

Why not just use images

There were two obvious options, and our decision log lists both as rejected: free-form specs, and the official PNGs from the regulation.

Images from an image model were never an option. Our imagery rules say why: generated pictures invent details, like a wrong colour on a buoy or a wrong number of lights, and you can't cite them as a source.

The official PNGs would have been accurate. But a PNG is a fact you can't test. The rule ("a north cardinal mark has two black cones, both pointing up") lives in the pixels of each image, and in every question that uses one. With a spec, the rule lives in one place in code, and a test can check it. Copyright was never the issue: Dutch copyright law leaves laws and their annexes free (article 11 of the Auteurswet), so the drawings could have been used. It was a testing decision, not a legal one.

What a question stores

The specs are zod schemas. This is the whole spec for a buoy:

export const BuoySpec = z.object({
  mark: BuoyMark,
  form: z.enum(["default", "spar"]).default("default"),
  topmark: z.boolean().default(true),
  /** Kenteken painted on the buoy, e.g. "HD 4". */
  label: z.string().max(10).optional(),
});

BuoyMark is an enum of 12 marks (lateral-right, cardinal-n, safe-water and so on). Notice what's missing: colour, shape and what the topmark looks like. The question says which mark it is, not what it looks like. All visual kinds sit in one discriminated union, and every kind requires alt text:

export const VisualSpecStrict = z.discriminatedUnion("kind", [
  z.object({ kind: z.literal("vessel-lights"), spec: VesselLightsSpec, alt: z.string().min(1) }),
  z.object({ kind: z.literal("buoy"), spec: BuoySpec, alt: z.string().min(1) }),
  // ...signs, day signals, sound signals, safety signs, chart tasks, chart symbols
]);

In a question file, a buoy is just this:

"visual": {
  "kind": "buoy",
  "spec": { "mark": "cardinal-e", "form": "default" },
  "alt": "Een zwart-geel-zwarte boei met twee zwarte kegels, de bases naar elkaar toe."
}

The rules live in the renderer

The buoy component has one table, with one row per mark, and each row comes from the annex of the inland shipping regulations:

const RULES: Record<BuoyMark, Rule> = {
  "lateral-right": { body: "can", bands: [{ color: PAINT.red, from: 0, to: 1 }],
                     topmark: { type: "cylinder", color: PAINT.red } },
  "cardinal-e": { body: "pillar", bands: [/* black, yellow, black */],
                  topmark: { type: "two-cones", dir: "bases-together" } },
  "safe-water": { body: "sphere", bands: "vertical-red-white", topmark: null },
  // ...
};

Record<BuoyMark, Rule> means TypeScript won't compile if a mark is added to the enum without a rule. And some rules can't be overridden by the spec at all. In Dutch inland waters, a safe-water mark has no topmark, whatever the question asks for:

// Safe water never shows a topmark in BPR waters (bijlage 8, 6.3), whatever the spec says.
const topmark = spec.topmark ? rule.topmark : null;

Navigation lights go one step further. A light only shines over a fixed arc: a masthead light forward, a red sidelight on the port side, a stern light aft. Which lights you see depends on where you're standing. So the spec lists all the lights a vessel carries, plus the view, and the code decides what's visible:

export function visibleFrom(role: LightRole, view: View) {
  switch (role) {
    case "masthead": case "masthead-aft": return view !== "stern";
    case "sidelight-port": return view === "bow" || view === "port";
    case "sidelight-starboard": return view === "bow" || view === "starboard";
    case "stern": return view === "stern";
    // ...
  }
}

One of our questions describes a small motor boat with a masthead light, two sidelights and a stern light, seen from straight behind, and asks which lights you see. The spec lists all four lights with "view": "stern". The model that wrote the question never decides which lights are visible. The renderer draws one white light, because that's what the arcs allow. The drawing and the answer key come from different places, so a wrong key can't come with a matching wrong picture.

Seen head-on, the red light is on your right, because the boat's port side faces you. That's the kind of detail that's easy to get wrong once and copy forever. Here it's one line in the renderer and one test.

A spec that doesn't parse is text

This is the component that every question visual goes through:

export function Visual({ visual }: { visual: unknown }) {
  const parsed = VisualSpecStrict.safeParse(visual);
  if (!parsed.success) {
    const alt = typeof visual === "object" && visual && "alt" in visual
      ? String((visual as { alt: unknown }).alt) : "Afbeelding";
    return <div role="img" aria-label={alt} data-visual="invalid">{alt}</div>;
  }
  // switch on parsed.data.kind and render the right SVG
}

(Trimmed: the real one also has classes for a dashed box.)

It takes unknown on purpose: it trusts nothing it's given. If the spec is wrong, there's no fallback picture, no closest match, no default buoy. You get the alt text in a box. As the comment in the code puts it: a missing image is recoverable, a misleading one is not.

In practice that box should never appear, because there's a second layer. The draft files are loaded with the same schema (visual: VisualSpecStrict.nullable()), and a test parses every exam's drafts. A model that writes { "teken": "A.11" } instead of { "code": "A.11" } breaks CI before anything is imported. The fallback is for what slips past that.

Testing drawings without screenshots

These tests don't use screenshots. The renderers put small data attributes on the parts that carry meaning, and the tests read the markup:

it("cardinal topmarks point the right way", () => {
  const cones = (mark: string) => [...draw(mark).matchAll(/data-cone="(up|down)"/g)].map((m) => m[1]);
  expect(cones("cardinal-n")).toEqual(["up", "up"]);
  expect(cones("cardinal-e")).toEqual(["up", "down"]); // bases together
  expect(cones("cardinal-w")).toEqual(["down", "up"]); // points together
});

tests/visuals.test.tsx has 149 of these: light arcs per view, every buoy mark as buoy and spar with and without topmark, every traffic sign code, and one that hands the dispatcher a broken sign spec and expects the alt text back. A separate file with 176 tests checks the opposite: that the markup does not contain the answer. A question that asks "what does this sign mean?" shouldn't have the sign code in an attribute where anyone can read it in the page source.

For looking rather than testing, there's a dev-only gallery at /dev/visuals that draws every kind and variant, each set of lights from four sides, at night and as a sketch. That's where a new drawing is compared with the official image before it goes live.

The numbers

The renderers cover 12 buoy marks, 8 light roles, 107 traffic sign codes and 49 workplace safety signs and hazard pictograms. In our Dutch exams' draft files, 170 of 3,532 question drafts carry a visual (including redrafts, so it's not the live count). All 170 parse.

What I'd tell someone else

Store what something is, not what it looks like. "cardinal-e" can be tested. A colour list in each question can't.
Put the rules that can't change in code the content can't override. The safe-water topmark is a good example.
When the data is bad, show less, not something. A box with text is a visible bug. A plausible wrong picture is an invisible one.
Test the meaning in the markup. A few data attributes make SVG easy to assert on.

The same components draw the reference pages on the site, like the safe water mark without its topmark, the east cardinal mark, and the lights of a small motor boat from every side (all in Dutch).

I wrote this post with help from an AI assistant. The code, tests and numbers come from our own repository.

Three models, no human reviewer: how we gate AI-written exam questions

Emmanuel — Thu, 05 Nov 2026 18:53:27 +0000

I build practice exams for Dutch certification exams: workplace safety (VCA), the hospitality licence, two boating licences, marine radio and the taxi theory exam. The questions are written by a language model. Nobody reads every question before it goes live. I don't have the time, and I'm not a qualified skipper or safety instructor anyway.

That only works if the checking is better than "ask the model if it's sure". Here's the pipeline we ended up with, and what roughly 10,000 check results taught us.

The four steps

A. Draft from a source, with a quote. Every question starts from an official text: an article of the inland shipping regulations, a law, or the published exam requirements. The drafting model (mostly Claude Opus) writes the question, the options, an explanation, and a verbatim quote from the source that proves the answer.

B. A blind check by a second model. A fresh session on a different model (Claude Sonnet) gets the stem, the options and the source text, but not the key. It answers first and writes down why. Only then does it see our key and explanation, and it fills in six yes/no criteria: key correct, exactly one defensible answer, explanation correct, natural Dutch, unambiguous, at the level of the real exam. Any "no" is a fail.

C. One redraft. A failed question gets one redraft that addresses the findings, and the redraft goes through the checks again. It doesn't get a third round.

D. Independent verification by a third model. Same procedure again, blind first, by a model that neither wrote the question nor did step B (Claude Haiku or Fable). The instructions tell it to be stricter: if you'd hesitate to put this question in a paid exam, fail it.

A question is imported as verified only if both checks pass and the two checker models differ. That rule is in code, not in a prompt:

// Only counts when an independent model did it: the verifier must differ from the first checker.
const verified = !!v && v.model !== check.model && passesCheck(draft, v.result);

And the quote is checked by a test, not by a model:

expect(text, `${q.draftId}: quote not found verbatim in ${src.sourceId}`)
  .toContain(normalize(src.quote));

If the drafter "remembers" a rule that isn't in the cited text, CI fails before any model looks at it.

What the numbers say

Across our exams, the checks have produced 5,148 step B results and 4,608 step D results. Those include re-checks of redrafts, so they count results, not unique questions.

Step B failed 13.4 %.
Step D failed 14.3 % of what step B had already passed.

I expected the third model to mostly agree with the second. It didn't. And look at why step D failed questions:

Criterion that failed at step D	Times
Not at the level of the real exam	216
Explanation not correct	153
Not unambiguous	140
More than one defensible answer	109
Not natural Dutch	25
Key wrong	12

A wrong key, the thing everyone worries about, was the rarest reason. Most failures were about a question that's technically right but not good: too easy, an explanation that cites the wrong reason, or a second option you could argue for. Those are exactly the problems a candidate would complain about, and the first checker had let them through.

Three things that went wrong, and the rule each one left behind

1. A script broke 62 questions after they were written. An early batch balanced which letter is correct by moving keys around after the explanations were written. The explanations still said "b is correct" while b was now a distractor. Step B caught it. The rule now: write options in display order, never re-letter afterwards, and re-read each draft to check that the option the explanation calls correct says what the explanation claims.

2. A "blind" check that wasn't blind. One step B batch opened a file that showed the keys before writing its own answers. Its results looked great and were worthless. The batch had to rely on step D for its blind pass, and the runbook now says it outright: don't run a script or open a file that shows the keys until every blind answer of your batch is written. A check that saw the key first is not blind.

3. All three models missed a pattern. In many questions the right answer was the one option worded differently from the others, so you could find it without reading the question. None of the models flagged it, because each one checks the content, not the shape. That one needed a mechanical check in code, which I wrote about in an earlier post.

What this doesn't guarantee

Three models agreeing is not the same as an expert agreeing. They share training data and can share a blind spot, as the odd-one-out pattern showed. The quote check proves the source says something, not that the question tests it fairly. So the site says plainly that no person reads every question, every question shows its source, and every question in the answer review has a button to report an error. The reports are the human review we do have.

If you're generating questions, quizzes or anything else that has a single right answer:

Make the checker answer before it sees your answer. A model that reads your key first will usually agree with it.
Use a different model for each check, and enforce that in code.
Check the source mechanically. A verbatim quote plus a string match catches invented rules for free.
Count why things fail, not just how many. The rare failure (a wrong key) wasn't where most of the work was.

The process is described for candidates on our about page (in Dutch), and the exams themselves are at examenzaal.nl.

I wrote this post with help from an AI assistant. The numbers come from our own check files and code.

Our 404 page answers 200, and we decided to keep it that way

Emmanuel — Thu, 05 Nov 2026 18:52:36 +0000

I run a site with practice exams for Dutch certification exams. Each exam has a few fixed practice forms, at URLs like /examen/b-vca/1/, /examen/b-vca/2/ and so on. Last month I typed /examen/b-vca/999/ to check the error page.

The page looked right: "Deze pagina bestaat niet" ("this page doesn't exist"), the normal layout, a link back. Then I looked at the status code. 200.

A 404 page served with a 200 is what Google calls a soft 404. Here's why it happens in Next.js 16 with partial prerendering, what I tried, and why I ended up leaving it.

Why the status is already gone

The exam page is built the way Next.js 16 wants you to build it with cacheComponents: true. The page itself is a static shell, and everything that needs the request goes inside a Suspense boundary:

export const metadata: Metadata = {
  title: "Oefenexamen",
  robots: { index: false, follow: false },
};

export default function ExamPage({ params, searchParams }: PageProps<"/examen/[exam]/[form]">) {
  return (
    <Suspense fallback={<Notice title="Oefenexamen laden…" />}>
      <ExamForm params={params} searchParams={searchParams} />
    </Suspense>
  );
}

Inside, the component looks up the exam and calls notFound() when the form number doesn't exist:

if (!exam || !/^[1-9][0-9]*$/.test(form) || n > exam.forms.fixedCount) notFound();

With partial prerendering, the server sends the prerendered shell first and streams the dynamic part after it. By the time notFound() runs inside the Suspense boundary, the response headers, including 200 OK, have already gone out. Next.js still renders the not-found UI into the stream, so a person sees the right page. A crawler that only reads the status sees a normal page.

URLs that don't match any route are fine. They go to global-not-found.tsx and get a real 404. Only an unknown value inside a dynamic route that streams is affected.

What I tried

generateStaticParams. List the real forms, and anything else should 404. It doesn't work that way here. The listed params get prerendered, but an unlisted one still gets the shell and renders on demand, so we're back at the same stream.

dynamicParams = false. This is the classic switch for "only the params I listed". With cacheComponents enabled, Next.js doesn't allow it on the route.

Making the page fully static. Then notFound() would run at build time and the status would be right. But this page reads the visitor's session cookie and the database (who is this, and do they have access). It can't be static.

A proxy in front of the route. This is the one that actually works. A proxy.ts (what used to be middleware.ts) checks the exam and form number before the page runs and returns a 404 for unknown ones. The cost is that it runs on every request under /examen/, including the server actions that save answers and hand in the exam. It also needs a list of valid exam and form pairs at the edge, generated from the content and kept in sync. That's a new moving part on the hottest path in the app, to fix a URL nobody links to.

Why we kept the soft 404

What matters about a soft 404 is that a search engine might index an error page or waste crawl budget on it. So I checked what a search engine can actually do with these URLs:

Every exam page is noindex, nofollow, including the real ones. A practice exam isn't a page anyone should land on from search. The guides that link to it are.
/examen/ is disallowed in robots.txt. Well-behaved crawlers don't fetch the page at all.
Nothing links to a form that doesn't exist. The links to exam forms are generated from the same data that decides which forms exist.

So no crawler should ever see the 200, and if one does, the page tells it not to index anything. The fix would add risk to the part of the app where a bug costs a user their exam attempt. The problem it fixes has no visible effect.

We wrote it down as a decision with a condition: revisit it if a crawler or an uptime monitor ever needs the real status. Then the proxy plus a generated exam/form map is the way to do it.

What I'd tell someone else on Next.js 16

Check your error status codes with curl, not the browser. curl -s -o /dev/null -w "%{http_code}" <url> takes a second. The page looking right tells you nothing about the header.
notFound() inside Suspense can't change the status once the shell has streamed. If the status matters, the check has to happen before streaming starts: at build time, or in a proxy.
Decide per route whether the status matters. On public, indexable routes it does, so make those static or check them early. On routes that are noindex and disallowed anyway, a soft 404 can be the cheaper, safer trade.
Write the decision down with the condition that would change it. Otherwise someone (probably you) rediscovers the 200 in three months and spends an evening on it again.

The site is examenzaal.nl, practice exams for Dutch certificates. The real exam forms start from pages like the B-VCA practice page.

I wrote this post with help from an AI assistant. The code and the decision come from our own repository.

The right answer was the odd one out 67 % of the time

Emmanuel — Thu, 05 Nov 2026 17:58:20 +0000

I build practice exams for Dutch certification exams: the VCA safety certificate, the hospitality licence (Sociale Hygiëne), the small-boat licence, marine radio and a few more. One model drafts the questions. A second model answers them blind, without seeing the drafter's notes. A third model, which neither wrote nor checked the question, verifies it independently. Only then does a question go live.

All three models were happy with the questions. Then I counted something none of them had flagged: in many questions you could find the right answer without reading the question.

The cue

Here's a made-up question in the same style:

What do you do first when you find a colleague unconscious next to an electrical cabinet?

A. Call the company's emergency response officer
B. Call the emergency number
C. Make sure the power is switched off

Two options start with "Call". One doesn't. If you've taken a lot of multiple-choice tests, your eye goes straight to C, and here it's right.

That would be fine once. It isn't fine when it's a pattern.

What I measured

I took every verified single-choice question where exactly one option starts with a different first word than the others, and counted how often that odd option was the correct answer:

Exam	Key = the odd option
Sociale Hygiëne (hospitality licence)	67 %
VOL-VCA (safety, supervisors)	63 %
B-VCA (safety, basic)	51 %
Marifonie (marine radio)	42 %
Klein Vaarbewijs 1 (small-boat licence)	39 %

Most of these exams have three options, so by chance the key would be the odd option about a third of the time. The boating exam has four options, so chance is a quarter. Every exam was above chance, and the hospitality exam was at two thirds.

A second cue was even stronger: absolute words. Where every wrong option contained a word like "only", "always", "never" or "but not", and exactly one option didn't, that option was the key in 82 % (B-VCA), 89 % (VOL-VCA), 81 % (Sociale Hygiëne), 63 % (boating) and 60 % (marine radio) of the cases. "Absolute answers are usually wrong" is one of the first tricks test-wise people learn. Our questions rewarded it.

Why it happens

My best explanation: a language model writes the correct answer carefully, as a specific action or a qualified statement. Then it writes the distractors as a set, and a set comes out parallel: same opening verb, same structure, and often an absolute word to make sure each one is clearly wrong. The key ends up as the one option that doesn't look like the others.

A model reviewing the question checks whether the key is correct and the distractors are wrong. It does that well. It doesn't ask whether the key looks different, because that isn't a question about content.

Why shuffling doesn't help

The first idea people have is to shuffle the options. That fixes a cue in the position (for example "the answer is often B"). This cue is in the wording. C is still the odd one out when it's shown as A.

What we changed

1. A mechanical check on every draft. It's a few lines of TypeScript and runs before any model looks at the question:

const firstWord = (t: string) =>
  t.trim().split(/\s+/)[0].toLowerCase().replace(/[^\p{L}\p{N}]/gu, "");

/** True when the key is the only option with a different first word. */
function keyIsOddOneOut(options: { id: string; text: string }[], keyId: string): boolean {
  const key = options.find((o) => o.id === keyId)!;
  const others = new Set(options.filter((o) => o.id !== keyId).map((o) => firstWord(o.text)));
  return others.size === 1 && !others.has(firstWord(key.text));
}

The real check has two more rules: every distractor is restricted (by an absolute word) and the key isn't, and the key is the only "yes" among "no"s (or the reverse). Each language has its own word lists. A conditional like "if" or "unless" doesn't count as a restriction.

2. A rewrite pass that only touches the options. Every live question that failed the check got new wording for its options. The correct answer, the fact behind it and the cited source stayed exactly the same. The rewrite went through the same blind check and independent verification as a new question, and the original was only retired when the replacement passed.

3. The verifying model reads for it too. The third model now treats a form cue as a reason to fail a question. On a later batch of 56 new questions, it passed 51 and failed 5, all five on form cues. Those were redrafted.

What it doesn't solve

This catches the cheapest cues, not all of them. A longer, more careful key is another classic, and so is grammar that only fits one option. The check is a floor: it makes sure that a question can't be answered from its shape alone. It doesn't make a question good.

If you generate quizzes with an LLM

Count it. For your own bank, measure how often the key is the option with a different first word, and how often it's the only option without an absolute word. Compare with chance. It takes an afternoon.
Write distractors like the key. Same structure, same length, same kind of qualifier. If the key is an action, the distractors are actions.
Use absolute words in the key too, when they're true. Otherwise "never" becomes a giveaway.
Check form separately from content. A reviewer who checks whether an answer is right won't notice that it also looks different.

The practice exams this came from are at examenzaal.nl (Dutch, with the VCA exams also in English).

I wrote this post with help from an AI assistant. The numbers come from our own question bank and code.