DEV Community

Cover image for 15 green checks and a stale demo
Paragon Framework
Paragon Framework

Posted on

15 green checks and a stale demo

#programming #flutter #devops #testing

I have a release script with fifteen automated checks. It passed all fifteen. Then I opened my own live demo and the About screen showed the previous version.

Here is what it checks, on the release in question:

PASS  version 1.8.0 consistent (pubspec = CHANGELOG = app About screen)
PASS  public changelog announces v1.8 and marks it Latest
PASS  working tree clean (what you see is what packages)
PASS  no merge-conflict markers in tracked files
PASS  every local link on the sales site resolves
PASS  archive contains no internal-only files
PASS  archive contents are built from the current version (1.8.0)
PASS  the site is up (/ returns 200 and is not a Pages error page)
PASS  all 8 landing-page images load from the live site
PASS  analytics beacon present on every public page
Enter fullscreen mode Exit fullscreen mode

Not one of those was wrong. The version really was consistent. The changelog really was updated. The site really was up.

The demo is a separately compiled build, and my deploy script publishes the marketing site. Nothing built the demo, and nothing looked at what the demo was running.

Every check I own is scar tissue

The "is the site up" check exists because I once took the site offline. The "do the images load" check exists because a deploy deleted every image while the page still returned 200 — a status check alone said "fine" the whole time the logo was a broken icon.

They are all reactions to something that already went wrong. Which makes a fully green run a precise statement about my past mistakes, and silent about everything else:

green  ==  "none of my previous mistakes recurred"
green  !=  "nothing is wrong"
Enter fullscreen mode Exit fullscreen mode

Those feel identical at 1am when you want to ship.

The fix is embarrassingly cheap

Flutter writes a version.json next to the web build, populated from pubspec:

{"app_name":"my_app","version":"1.8.0","build_number":"7"}
Enter fullscreen mode Exit fullscreen mode

It's live, it's public, and it's a couple of hundred bytes. So you can just ask the deployed app what it is:

curl -s https://example.com/app/version.json
Enter fullscreen mode Exit fullscreen mode

The check compares that to the version you think you're shipping:

$live = (New-Object Net.WebClient).DownloadString(
    'https://example.com/app/version.json')
if ($live -match '"version"\s*:\s*"([0-9.]+)"' -and $Matches[1] -ne $pubVer) {
    Fail "live demo runs $($Matches[1]) but you are shipping $pubVer"
}
Enter fullscreen mode Exit fullscreen mode

The message matters as much as the check. Mine spells out the recovery, because forgetting the steps is the bug:

FAIL  live demo at /app/ still runs 1.7.0 but you are shipping 1.8.0.
      Rebuild and redeploy it: flutter build web --base-href /app/,
      inject_beacon.ps1 build/web, then copy into the Pages clone.
Enter fullscreen mode Exit fullscreen mode

Ten minutes of work for a hole that had been open for who knows how many releases.

Then I fixed it in the wrong place

This is the part worth your time, because the first failure is ordinary and this one isn't.

I rebuilt the demo, copied it into the GitHub Pages repo, pushed, watched the live version flip to the new number, and moved on.

Then I ran my deploy script for an unrelated change, and it put the old build back.

The layout is the whole story:

site/            <- source of truth, synced wholesale by the deploy script
  index.html
  app/           <- the published copy of the demo  ** still 1.7.0 **
  admin/

paragon-demo/    <- the GitHub Pages repo
  app/           <- what I fixed by hand          ** now 1.8.0 **
Enter fullscreen mode Exit fullscreen mode

I had fixed the destination while the source still held the previous build. The next deploy faithfully republished the stale one, exactly as designed.

For a few minutes I had a "verified fixed" demo on a countdown to breaking again, and the thing that would break it was my own tooling working correctly.

Four things I took from it

Verify the artifact, not the action. "I deployed the fix" and "the fix is live" are different claims. I had confirmed the first and assumed the second.

Poll, don't peek. My first "it's live now" check passed against a CDN copy that hadn't rebuilt yet. Checking once and believing it is how you confirm a lie:

until curl -s "$URL/version.json" | grep -q '"1.8.0"'; do sleep 5; done
Enter fullscreen mode Exit fullscreen mode

Fix the source, not the copy. If a pipeline generates a destination from a source, editing the destination is an illusion with an expiry date.

Check what the user touches. My pipeline was thorough about inputs: source files, version strings, archive contents, links. It had nothing to say about the compiled thing a stranger clicks, which is the only artifact that actually matters.

I found this the ordinary way, by looking at my own product like a user would, hours after telling myself it was done. Every check on that list was earned the same way. This one just cost less than the others, because I got there before a customer did.

Top comments (0)