A passing smoke test can still miss your broken page

Danilo Lapegna — Mon, 05 Oct 2026 13:00:54 +0000

A shop's home page opens. The About page opens. The smoke test passes. Meanwhile, /orders shows an error screen, and the test never went there.

That's the small example we'll build here: the same site, the same broken page, two test runs with different route lists. You'll see why a passing result needs a second question: did the test visit the pages that matter? Start by writing down one critical URL and what should appear there.

I wrote deep-smoke to open web-app pages in a browser and check for visible failures, including empty screens, error screens and broken assets. A smoke test is a quick check that the basics still work. It saves time, but the pages it finds depend on the links and starting URLs you give it. That limit matters just as much as the things it detects.

You can follow this example if you use Git, npm and a terminal. Use Node.js 20 or later for the current Playwright release. We'll use a tiny local site with no customer data, accounts or AI service. Playwright, the library that drives the test browser, needs an installed Chromium browser; the installation below downloads it. The software is free, though the download takes disk space and network access.

Start with the page you would hate to lose

Before installing anything, write down one critical URL and what someone should see there. For an orders page, that might be the order list for a signed-in customer and a sign-in page for someone without a session. Those are two different checks.

For a small, one-off change, opening those URLs manually may be enough for your immediate review. If you keep making changes to shared navigation or page layout, a repeatable browser check becomes useful. Either way, you need to decide which pages and user states count. A crawler cannot infer that an unlinked screen is essential to your business.

Our demonstration leaves out sign-in deliberately. It isolates one problem: a broken page that has no link from the home page.

Build a site with an easy-to-miss failure

For a reproducible run, use this specific version of the public repository. As of 5 October 2026, the package name deep-smoke is not available on the public npm registry, so these instructions run it from source.

git clone https://github.com/danilolapegna/deep-smoke.git
cd deep-smoke
git checkout a73bf08efbf8c561670cb58db8eecf47ddd36e10
npm install --no-save --ignore-scripts playwright
npx playwright install chromium

Inside that directory, create demo-server.mjs with this content:

import http from 'node:http';

const pages = {
  '/': '<h1>Demo shop</h1><a href="/about">About</a>',
  '/about': '<h1>About</h1><p>A small test shop.</p>',
  '/orders': '<main data-error-boundary><h1>Orders unavailable</h1></main>',
};

http.createServer((request, response) => {
  const page = pages[new URL(request.url, 'http://localhost').pathname];
  response.writeHead(page ? 200 : 404, {'content-type': 'text/html'});
  response.end(`<!doctype html><html><body>${page ?? '<h1>Not found</h1>'}</body></html>`);
}).listen(4173, '127.0.0.1', () => console.log('Demo: http://127.0.0.1:4173'));

The home page links to /about, but neither page links to /orders. The data-error-boundary attribute marks an error fallback: the screen shown when the intended page cannot render. deep-smoke treats that marker as a failure even though the server returns HTTP 200, which only says the request succeeded.

Start the site:

node demo-server.mjs

Leave that terminal running. Open http://127.0.0.1:4173/orders in your browser to see the failure we planted. If the server reports EADDRINUSE, port 4173 is already occupied. Stop here and use a free local port in both the server and the configurations below; don't stop an unrelated process to make room.

Watch the incomplete test pass

In the same directory, create demo.config.json:

{
  "baseUrl": "http://127.0.0.1:4173",
  "routes": ["/"]
}

In a second terminal, also inside the repository directory, run:

node bin/deep-smoke.js --level=3 --config=demo.config.json --evidence=demo-before.json

Level 3 follows links from the starting routes. In this site, it visits / and /about. The tested run reports:

routes      2 visited, 0 failed

The process exits with code 0, meaning this run passed. It also writes its evidence to demo-before.json: the routes visited, the configured checks and each route's result.

Open that JSON file and look at results. /orders is absent. A page missing from the report has not passed; it has not been tested. The crawler has done what we asked, and what we asked was incomplete.

Add the missing route, without fixing the site

Now change only routes in demo.config.json:

{
  "baseUrl": "http://127.0.0.1:4173",
  "routes": ["/", "/orders"]
}

Run the same check and save a separate report:

node bin/deep-smoke.js --level=3 --config=demo.config.json --evidence=demo-after.json

This time the tested run visits three routes and fails on one. The relevant output is:

routes      3 visited, 1 failed

Failing routes (1):
  x [anonymous] /orders
      error-boundary: error fallback rendered instead of the page

It exits with code 1. That's the expected result: the test finally saw the failure. The site's code stayed identical between the two runs. We changed the coverage, not the defect.

If you still see a passing result, check the config file named in the command, the URL and the route list in the new evidence file. Don't disable the error-boundary check to make this example pass. Seeing the deliberate failure is the point.

When you're finished, press Ctrl+C in the server terminal. The demo files and reports stay in your local clone; the original repository is unchanged.

What I want a green result to tell me

For a real release, I want a passing result attached to an explicit list of important pages and user states. The route list is part of the test. Reviewing it is engineering work, particularly when an assistant has added a screen that never made it into the navigation.

There are limits to that evidence. This demo tests only a visitor without a session. It doesn't check whether submitting an order works, whether the totals are correct or whether the backend enforces permissions. A page can render perfectly and still get all of those wrong. Those behaviors need their own tests.

deep-smoke supports configured signed-in test users (personas) and checks for expected redirects, but a redirect check is not an authorization audit. Likewise, a crawl stopped by its route or time budget cannot support a claim that it visited the whole intended surface. Keep those distinctions visible when reading the report.

Your next move can be small: take one critical URL from your own application, add it explicitly, run against a test environment using the production build, then confirm that the report contains the expected page and user state. If either is missing, repair the test coverage before using its result to approve a release. Use synthetic data; keep real customer sessions out of a shared example.

The source and assertions in deep-smoke are public and MIT-licensed. If you find a healthy page it rejects, a reproducible example is useful: it gives us something concrete to test and fix.