Screenshotting every screen in the app found four bugs that no test was ever going to catch

I recently captured a card screenshot of every practice test in CogniPrep. The
useful side effect was not the images. It was that producing them meant opening
372 screens and looking at each one, which is the single most effective bug
finding technique I know and the one I do least often.

Four of the things I found, and what each one turned into.

1. "4200subscribers"

Chart labels were built by gluing a value to a unit the question bank supplies:

{hasValue ? `${value}${unit}` : '-'}

Which is correct for % and wrong for everything else. Units come in three
shapes and only one of them concatenates cleanly:

• %, k, m: attach directly. 45%, 24k.
• subscribers, MWh, thousand households: need a space. 1840 MWh.
• £, $, €, £m: the symbol has to lead and any remainder has to trail. £12m, not 12£m.

So there is now a twenty line function:

export function formatWithUnit(value: number | string, unit = ''): string {
  if (!unit) return String(value);

  if (/^[£$€]/.test(unit)) {
    const symbol = unit[0];
    const rest = unit.slice(1);
    const n = typeof value === 'number' ? value : Number(value);
    // "-£5", not "£-5".
    return Number.isFinite(n) && n < 0
      ? `-${symbol}${Math.abs(n)}${rest}`
      : `${symbol}${value}${rest}`;
  }

  if (unit.startsWith('%') || unit.startsWith(' ') || unit.length === 1) {
    return `${value}${unit}`;
  }

  return `${value} ${unit}`;
}

Three rules worth pointing at.

unit.length === 1 is the whole abbreviation rule. Any single character unit
attaches, any multi-character one takes a space, which gets 24k and
12 km right without a list of known abbreviations to maintain. The one
exception is % prefixed units like % per year, which is why that case is
tested separately.

unit.startsWith(' ') is an escape hatch. If the bank already wrote a leading
space, the function does not add a second one, so content can override the rule
without a code change.

And negatives. ${symbol}${value} produces £-5, which is wrong in every
locale I know of. The branch is three lines to get -£5, and it is the kind of
case that only ever shows up in a screenshot, because a test written from the
same mental model as the code would have used a positive number.

The test file pins the real units the banks use rather than units I imagined:
subscribers, MWh, km, EUR per kWh, %, % per year, k, £, £m,
$, and ' units' with its leading space.

2. Ink that was invisible on dark cards

const SHAPE_COLORS = {
  red: '#ef4444',
  blue: '#3b82f6',
  green: '#22c55e',
  yellow: '#eab308',
  black: '#1f2937',
};

Four of those are colours. The fifth is not a colour, it is "the default", and
#1f2937 is a very dark grey that disappeared into the dark theme's card.

// "Black" is the ink colour, so it follows the theme's foreground: a fixed
// #1f2937 vanished into the dark card the frames sit on.
black: 'currentColor',

With currentColor and a text-foreground class on the SVG, the glyph is dark
on light and light on dark, and the data still says "black" because that is what
the content means. The lesson generalises: in a palette that includes a theme
dependent value, the odd one out is usually the one named after the default
rather than after a hue.

3. A square board capped only in one dimension

One test's board is square: a grid of N by N cells. It was centred in a
container with a width cap and nothing else:

<div className="mx-auto w-full max-w-md lg:max-w-xl xl:max-w-2xl">

Width is also height when the thing is square. On a 1280x800 laptop a larger
board grew tall enough to push the Submit button below the fold. Nothing was
broken, nothing overflowed, and no test could see it, because the failure was
entirely a question of what fits in a viewport.

The fix caps the board against the viewport height as well as the width:

className="max-w-[max(16rem,min(28rem,calc(100dvh_-_19rem)))]
           sm:max-w-[max(16rem,min(28rem,calc(100dvh_-_14rem)))]
           lg:max-w-[max(16rem,min(36rem,calc(100dvh_-_15rem)))]
           xl:max-w-[max(16rem,min(42rem,calc(100dvh_-_15rem)))]"

Read from the inside out: min(28rem, 100dvh - 19rem) is "as wide as the design
allows, or as much height as is left over, whichever is smaller", and the outer
max(16rem, ...) is a floor so a landscape phone gets a usable board and
scrolls instead of a postage stamp. The subtracted figure is everything else on
screen, and it differs per breakpoint because the buttons stack on a phone. A
comment records what those numbers are made of, because 19rem is otherwise a
magic number that nobody will dare to change.

100dvh rather than 100vh, naturally. That is the other bug this would have
had on a phone.

4. A grid with a hardcoded column count

The answer tiles in another test were grid-cols-3 sm:grid-cols-6. Each tile
holds a row of glyphs that must not wrap, so at some widths a wide tile spilled
out of its track. Three columns is a guess about content that the CSS has no
way to check.

Replacing the guess with a measured minimum:

const widestOptionFrame = Math.max(1, ...(optionFrames ?? []).map((f) => f.shapes.length));
const optionTrackMin = Math.max(52, widestOptionFrame * (OPTION_GLYPH + 4) - 4 + 18) + 16;

style={{
  gridTemplateColumns: `repeat(auto-fit, minmax(max(${optionTrackMin}px,
    calc((100% - (var(--option-tracks) - 1) * 1rem) / var(--option-tracks))), 1fr))`,
}}

auto-fit plus minmax means the grid fits as many tracks as will hold the
widest tile, and the calc inside the max() is an upper bound expressed as
"the width of one track if there were N", so the layout never exceeds three
columns even when it could. --option-tracks carries N, which lets a Tailwind
breakpoint class raise it to six for short content without rewriting the
template. Six options then split evenly instead of four and two.

The arithmetic in optionTrackMin is glyph widths, their gaps, the padding and
border of the frame and the padding of the tile. It is ugly and it is a
duplicate of numbers that also live in CSS, which is the real cost of this
approach. The alternative is a ResizeObserver and a measurement pass, which is
a lot of machinery for a tile.

The method, since that is the actual point

None of these four is a logic bug. Every one of them is a thing that only
exists at a particular viewport, in a particular theme, with particular content,
which is exactly the intersection unit tests do not cover and exactly what you
see when you open the page and look at it. Three of the four were found while
doing something else entirely.

If you have a catalogue of screens and no reason to open all of them, give
yourself one. Generating marketing images turned out to be a perfectly good
excuse for a full visual sweep.

See it

• cogniprep.app/tests/numerical-reasoning lists every test that renders charts and tables, which is where the unit formatter runs. 66 cards on that page, each one a screenshot taken during the sweep.
• cogniprep.app/games/kenexa is the provider whose charts produced "4200subscribers", and cogniprep.app/games/aon is the other place the same formatter runs. The card images on both are the screenshots, so you are looking at the fixed rendering.

Comments

[Svg/icons]

The invisible SVG glyph says a lot. currentColor fixes the theme mismatch without changing what the data means.