DEV Community: Chan Meng

Reproducible is not correct: defect classes a green test suite cannot see

Chan Meng — Fri, 06 Nov 2026 12:25:44 +0000

Reproducible is not correct

Here is a measurement you can reproduce in eight lines, plus a dozen-line helper. It is from the example plans that ship with my floor-plan compiler — a U-shaped house built around a courtyard:

import { readFileSync } from "node:fs";
import { describe } from "@chanmeng666/archlang";

// Ray casting: count crossings of a ray from `point` towards +x. Points on an edge count as inside.
function pointInPolygon({ x, y }, polygon) {
  let inside = false;
  for (let i = 0, j = polygon.length - 1; i < polygon.length; j = i++) {
    const a = polygon[i], b = polygon[j];
    const cross = (b.x - a.x) * (y - a.y) - (b.y - a.y) * (x - a.x);
    const onEdge = cross === 0 &&
      Math.min(a.x, b.x) <= x && x <= Math.max(a.x, b.x) &&
      Math.min(a.y, b.y) <= y && y <= Math.max(a.y, b.y);
    if (onEdge) return true;
    if ((a.y > y) !== (b.y > y) && x < a.x + ((y - a.y) * (b.x - a.x)) / (b.y - a.y)) inside = !inside;
  }
  return inside;
}

const d = describe(readFileSync("examples/courtyard-house.arch", "utf8"));

for (const r of d.rooms) {
  const c = { x: r.bbox.x + r.bbox.w / 2, y: r.bbox.y + r.bbox.h / 2 };
  console.log(r.label, r.floor_polygon.length, "pts, bbox centre inside floor?", pointInPolygon(c, r.floor_polygon));
}
// Living 4 pts, bbox centre inside floor? true
// Entry 4 pts, bbox centre inside floor? true
// Dining 4 pts, bbox centre inside floor? true
// Gallery 8 pts, bbox centre inside floor? false   <-- here
// Bedroom 1 4 pts, bbox centre inside floor? true
// …

The Gallery is an eight-point room wrapped around a courtyard. Its bounding box has a centre, that centre is a perfectly good coordinate, and it is outside the room. Anything that reached for that point — to place a label, to start a route, to decide which side of a wall is "outside" — got a confident, silent, wrong answer.

That is not one bug. It is a defect class, and my project shipped six instances of it before anybody noticed. This post is about the three classes I have actually shipped, and what each one taught me about the difference between a gate that is green and a system that is right.

Class 1: a position derived from a shape's box instead of its shape

The six instances, all fixed in one release, all silent while they lived:

the point a room's name and area are drawn at;
the anchor a circulation route starts from — one plan reported a 10.9 m walk as 5.6 m;
the witness lines of automatic dimensions, which floated metres off a sloped facade;
swing into <room>, which gave a door a swing into a wall the room does not border;
furniture placed against wall, which backed a piece onto a wall outside its own room;
and the compass direction a window faces, which on a courtyard plan was reported backwards for every window, because the code found "outward" by comparing against the midpoint of the building's bounding box, and on a courtyard plan that midpoint sits in the courtyard.

Every one of them passed the whole test suite. Every one of them produced plausible output. The linter — the part of the system whose entire job is to say when a plan is unsound — reported none of them, because a wrong label position is not an unsound building, it is a correct building described wrongly.

What actually finds this class: not a test, a grep. Search for consumers of room.size or r.size.w with no nearby branch on the room's polygon. That pattern is a shape consumer reading the box when the ring is the honest datum. The fix is local and closed-form every time — probe one wall thickness off each face and ask which side has floor — and the general rule is the one sentence I wish I had written three releases earlier: a derived position comes from the shape, never from its bounding box or centroid.

Why a rectangle hides it. On a rectangular room the box centre is inside the room, and every one of those six code paths is correct. The entire class is invisible until a room is L-shaped, U-shaped, polygonal or round. If your fixtures are all rectangles, your suite is measuring the easy half of your input space.

Class 2: a generator that reproduces the same wrong text forever

I have a CI gate that regenerates every generated artifact and fails if the committed file differs. Grammars, error-code documentation, CLI reference, JSON schemas, a one-page language spec. It is a good gate. It has been green for a very long time.

It is also, on its own, vacuous — and this is the part that took me longest to see.

The gate compares a generator's output against the committed file. That proves the artifact is reproducible. It says nothing whatsoever about whether the generator is right. A generator that hardcodes a language fact instead of deriving it will reproduce the same wrong text on every run, for every release, with a green check beside it the whole way.

Both halves of that happened here, and both are in the public changelog:

the spec generator shipped a CLI reference three versions out of date, and omitted a keyword entirely, for three releases;
the grammar generator hardcoded a number regex that did not include the unit suffixes the lexer had accepted for months. The published grammar and the actual parser disagreed about the language, and nothing could see it, because each was internally consistent.

The rule that came out of it: a generator must derive from the source of truth — the same tables the compiler itself reads — and never retype it. And each generator needs a guard that fails when a source-of-truth entry has no rendering at all, because a missing row is exactly what a reflexive gate cannot notice.

The general form is worth stating on its own, because it is not specific to compilers:

A reflexive check is sound for a derived artifact and vacuous for a retyped one, and it cannot tell you which kind it is looking at.

Class 3: a surface no test ever executed

The third class is the cheapest to describe and the most embarrassing to find.

One release found five of them at once. A watch command that had not actually watched for files since it was written, across roughly two dozen releases. A PDF export that drew none of the wall hatching the SVG export drew. A formatter that rewrote pocket doors as hinged ones — silently changing the plan it was asked to tidy.

All five were found by running the surface. None of them by reading the code, and none of them by the test suite, which exercised the functions underneath each surface perfectly well and never once invoked the surface itself.

If you have a CLI, a formatter, a watcher or an exporter that is tested only through its internals, you have this class. The test that would have caught all five is unglamorous: run the actual command, on an actual file, and assert on what comes out.

What I do differently now

Three habits, one per class:

Every new language form ships with a byte-identity law, pinned by a test. A plan that does not use the new feature must render, describe and lint exactly as it did before — proved with a hash sweep across every shipped example, not by looking at two drawings. When a golden moves, that is a finding to explain before it is a diff to bless.
A green gate is asked what it proves. "Reproducible" and "correct" are different words and the gate only ever said the first one. Where a check is reflexive, the second check is a human deriving the artifact from the source of truth by hand, once, and comparing.
Non-rectangular fixtures exist in the suite now. The courtyard plan at the top of this post is in the repository precisely because it is the shape that exposes class 1. A test corpus made only of easy inputs is a measurement of the easy inputs.

None of this makes the system correct. It makes a specific family of silent wrong answers loud, which is a smaller claim and the only one I can support.

Reproduce it

Save the snippet at the top of this post as check.mjs, then:

mkdir bbox-check && cd bbox-check
npm init -y && npm install @chanmeng666/archlang@1.57.0
cp -r node_modules/@chanmeng666/archlang/examples .   # the example plans ship in the package
cp ../check.mjs . && node check.mjs

which prints, against version 1.57.0:

Living 4 pts, bbox centre inside floor? true
Entry 4 pts, bbox centre inside floor? true
Dining 4 pts, bbox centre inside floor? true
Gallery 8 pts, bbox centre inside floor? false
Bedroom 1 4 pts, bbox centre inside floor? true
Bedroom 2 4 pts, bbox centre inside floor? true
Bedroom 3 4 pts, bbox centre inside floor? true
Study 4 pts, bbox centre inside floor? true
Kitchen 4 pts, bbox centre inside floor? true
Bathroom 4 pts, bbox centre inside floor? true
Utility 4 pts, bbox centre inside floor? true

The compiler is MIT, has zero runtime dependencies, and compiles floor plans written as text to SVG, DXF and PDF: https://archlang.uk. There is a playground if you would rather not install anything: https://playground.archlang.uk.

If you have shipped one of these three, I would genuinely like to read about it. The first class in particular seems like something everyone with geometry in their system hits exactly once.

Floor plans that compile: byte-identical output, errors as data, and an access graph

Chan Meng — Fri, 06 Nov 2026 12:24:42 +0000

Floor plans that compile

I compiled the same floor plan twice and diffed the output:

import { readFileSync } from "node:fs";
import { createHash } from "node:crypto";
import { compile } from "@chanmeng666/archlang";

const src = readFileSync("laneway-house.arch", "utf8");
const a = compile(src).svg;
const b = compile(src).svg;

const sha = s => createHash("sha256").update(s).digest("hex");
console.log(Buffer.byteLength(a), sha(a) === sha(b), sha(a).slice(0, 16));
// 40104 true a44e7ed5d5bb1448

Identical. Not "visually identical" — the same 40,104 bytes, the same hash. Across the 37 example plans that ship with the project, 35 recompile byte-identically; the other two import from other files and produced no bytes at all in this in-memory harness, which is a missing file loader rather than a difference in output. Every output quoted in this post is from version 1.57.0 of the package.

That property is not a party trick. It is the thing that makes everything else in this post possible, and it is the thing an image generator cannot give you at any price.

The problem with a picture

Ask a model for a floor plan today and you get an image. An image has one interface: look at it. You cannot diff it, you cannot ask it whether the bathroom is reachable, and when you say "make the bedroom a metre wider" you get a new image where the kitchen also moved, for reasons nobody can name.

ArchLang takes the other road. You write source (trimmed here from the shipped laneway-house.arch example, which adds a bath partition, two more doors, two more windows and the furniture):

plan "Laneway House" {
  units mm
  grid 50
  north up
  dims auto all
  site { street north }

  wall id=w_lane   exterior  thickness 200 { (0,0) (7500,0) }
  wall id=w_east   exterior  thickness 200 { (7500,0) (7500,6500) }
  wall id=w_garden exterior  thickness 200 { (0,6500) (7500,6500) }
  wall id=w_west   exterior  thickness 200 { (0,0) (0,6500) }
  wall id=w_hall   partition thickness 100 { (4200,0) (4200,6500) }

  room id=r_live at (0,0) size 4200x6500 label "Living / Kitchen" uses living kitchen
  strip down at (4200,0) gap 0 width 3300 {
    room id=r_bath size 2500 label "Bath"    uses bath
    room id=r_bed  size 4000 label "Bedroom" uses bedroom
  }

  door id=d_front on w_lane at 2100 width 900 hinge near start swing into r_live
  door id=d_bath  pocket on w_hall at 900 width 800 slide right

  window on w_west   at 2400 width 1200
  window on w_garden at 700  width 1000
}

Coordinates are integer millimetres. Nothing places a door by eye: an opening is pinned to a distance along a named wall, so moving the wall moves the door. "Make the bedroom a metre wider" is one number, and the diff says so.

That compiles to SVG. Also DXF, PDF, PNG, and an ASCII plan for a terminal. Zero runtime dependencies; the whole compiler is TypeScript and runs in a browser as happily as in Node, which is why the playground has no server to send your plan to.

Reading a plan back, without looking at it

Here is the part I care about most. describe() turns the compiled plan into facts:

import { describe } from "@chanmeng666/archlang";

const d = describe(src);

console.log(d.caption);
// "Laneway House" — a 3-room floor plan, 48.75 m² total: Living / Kitchen (27.3 m²),
// Bath (8.25 m²), Bedroom (13.2 m²); 4 doors, 4 windows, entrances via d_front, d_garden.

console.log(d.totals);
// {
//   rooms: 3,
//   doors: 4,
//   windows: 4,
//   floor_area_m2: 48.75,
//   gross_internal_area_m2: 45.99,
//   gross_internal_area_all_levels_m2: 45.99,
//   net_floor_area_m2: 45.05
// }

console.log(d.access.entrances);       // [ 'd_front', 'd_garden' ]
console.log(d.rooms[0].adjacent);      // [ 'r_bath', 'r_bed' ]

Those areas are shoelace arithmetic over the room polygon, not an estimate off a raster. The access graph is a real modeled graph of which door connects which pair of spaces, so "is the bathroom reachable from the front door" is a query rather than a squint.

A text-only model can call this. It never has to render anything to find out whether it built what it was asked for — which is the whole reason this language exists.

Errors are data, and some of them carry the fix

compile() never throws. A broken plan comes back as a result with diagnostics, each carrying a byte span:

const broken = `plan "Demo" {
  units mm
  wall id=w1 exterior thickness 200 { (0,0) (5000,0) (5000,4000) (0,4000) close }
  room id=r1 at (0,0) size 5000x4000 label "Room"
  door id=d1 on w1 at 30000 width 900
}`;

compile(broken).diagnostics[0];

{
  "code": "E_ATTACH_POS_RANGE",
  "severity": "error",
  "message": "Door \"d1\" attachment position 30000 mm is outside the wall run (0…18000)",
  "span": { "start": 170, "end": 184 },
  "fixes": [{
    "title": "clamp the attachment position to 18000",
    "applicability": "machine-applicable",
    "fixId": "attach-pos-range",
    "edits": [{ "span": { "start": 170, "end": 184 }, "newText": "on w1 at 18000" }]
  }]
}

applicability: "machine-applicable" is a promise: the edit can be applied without asking anyone. arch fix walks the diagnostics, applies those edits to a fixpoint, and prints the unified diff it would write. An agent's correction loop is three commands and no image:

npx @chanmeng666/archlang compile plan.arch --json   # { ok, diagnostics, summary }
npx @chanmeng666/archlang fix plan.arch --dry-run    # the exact diff it would apply
npx @chanmeng666/archlang describe plan.arch --json  # did I build what was asked?

Every command takes --json and exits 0 ok, 2 user-source error, 1 IO, 3 bad usage. A typo earns the 3: --jsn gets did you mean --json? rather than being read as a filename.

Soundness is a separate layer

Compiling is not the same as being a sensible building. lint() is the layer that says so:

import { lint } from "@chanmeng666/archlang";

const twoRoomsNoDoors = `plan "Two rooms" {
  units mm
  wall id=w1 exterior thickness 200 { (0,0) (6000,0) (6000,4000) (0,4000) close }
  wall id=w2 partition thickness 100 { (3500,0) (3500,4000) }
  room id=r_k at (0,0) size 3500x4000 label "Kitchen" uses kitchen
  room id=r_b at (3500,0) size 2500x4000 label "Bath" uses bath
}`;

lint(twoRoomsNoDoors).map(d => `${d.code}: ${d.message}`);
// [
//   "W_ROOM_DISCONNECTED: Room \"Kitchen\" has no door or opening — it can't be entered.",
//   "W_ROOM_NO_FIXTURE: Kitchen \"Kitchen\" has no fixtures (sink, counter, stove…).",
//   "W_ROOM_DISCONNECTED: Room \"Bath\" has no door or opening — it can't be entered.",
//   "W_ROOM_NO_FIXTURE: Bathroom \"Bath\" has no fixtures (WC, basin, shower…).",
//   "W_NO_ENTRANCE: The plan has no exterior door or opening — there is no way into the building.",
//   "W_HABITABLE_NO_WINDOW: Kitchen \"Kitchen\" (14 m²) has no window on any of its walls; a room people live in needs daylight."
// ]

That plan compiles perfectly. It is also a building with two rooms you cannot enter. Separating "is it valid" from "is it sound" is a deliberate split, and it is why lint has its own exit behaviour and its own catalogue of codes.

What I am not claiming

Three things in this system are measurements, and I will defend them: reachability (a modeled access graph), exact areas (shoelace arithmetic, or πR² on a round room), and byte-identical determinism (the hash at the top of this post).

Everything else is advisory and the code says so. Clear widths are a nominal width with a constant allowance subtracted. Walk distances come off a coarse quantised grid. The project's own design records call those numbers deliberately coarse and refuse to dress them up, and I would rather tell you which numbers to trust than let you assume all of them.

Some other limits, stated before you find them:

It draws a plan, and only a plan. Heights exist in the language but draw nothing, because a floor plan is a horizontal cut. No sections, no elevations.
No BIM, no IFC, no round-trip. DXF export is one-way, for a draftsperson to open.
No structural, no MEP, no code compliance — the last one is a refusal rather than a gap. Asserting standards compliance is a liability claim this project will not make.
A concept tool, not a permit set. Dimensioned, consistent, exportable to real CAD. Not stamped construction documents.
Solo maintainer, MIT. Bus factor of one. The licence is chosen so the format outlives me if it is worth keeping.

Try it

Playground, compiles in your browser, no signup: https://playground.archlang.uk
Docs: https://archlang.uk
Source (MIT): https://github.com/ChanMeng666/archlang
npx @chanmeng666/archlang new -o plan.arch
MCP server, if your agent speaks it: npx -y @chanmeng666/archlang-mcp

If you build something with it, or break it, I would like to know either way.