On 22 August my AI agent told me that a public lookup tool worked. It had called three different paths on the site and all three answered with status 200.
All three paths were invented. The agent had guessed them. Each one returned the same few kilobytes: the empty shell of a JavaScript application, the page a browser downloads before the real content arrives. When I opened the tool in a real browser, it showed "Unexpected technical error".
Status 200 means the server answered. It does not mean it answered the question you asked.
Three ways a page lies
The first way is the shell. A modern web application often answers every path with the same small HTML file and lets JavaScript fetch the content afterwards. A request that does not run JavaScript gets the shell, every time, for real pages and for nonsense ones. Status, size and a keyword search all look healthy. None of them saw the page.
The second way is the stale page. On 6 September I pushed new prices for my company's site. The push succeeded and the repository was correct, so I reported "pushed". The build that turns the repository into the website had failed. For 70 minutes the site served the old prices, with a 200 on every page. Nothing was broken from the outside. It was just yesterday's page, answering confidently.
The third way is my own. On 28 September I checked three pages with a small loop that downloaded each one into a file before reading it. One of the three was a control: a deliberately wrong address, which should come back as "not found". The loop wrote all three downloads into the same file name, so each overwrote the one before. When I read "the course page", I was reading the control's "not found" page. Every request had answered correctly. The mix-up was in my own plumbing.
The control: one address that must fail
The habit that catches all three is to give every check a control. Before you trust what a page says, ask the same site for a page that cannot exist, and compare.
import uuid
def read_page(fetch, url):
"""fetch(url) -> (status, text). Returns the text only if the page is
really the page, else None: could not check."""
host = url.split("/")[2]
control = f"https://{host}/no-such-page-{uuid.uuid4().hex[:8]}"
s_real, real = fetch(url)
s_ctrl, ctrl = fetch(control)
if s_real != 200:
return None
if s_ctrl == 200 and len(ctrl) == len(real):
return None # the site answers everything the same way: a shell
return real
If a nonsense path answers exactly like your real page, you have learned nothing about the real page. That is the shell case, caught before any conclusion is drawn.
The content check: what must be on this page
A control tells you that the site distinguishes pages. It does not tell you that you got the right one. For that, decide before reading what the page must contain if it is the page you meant: the product's title, the invoice number, the price you are checking, the version stamp of the latest build.
def is_the_page(text, must_contain):
return text is not None and all(m in text for m in must_contain)
For the stale price case, the "must contain" was the new price. For a build, it is the version stamp the build writes into the page. My sites now publish the commit they were built from, and a guard compares it with the repository every two hours. When they differ, the page is answering, and it is the wrong page.
For a JavaScript application, the content check runs in a real headless browser, after the page has rendered. Reading the raw HTML of a shell and searching it for a title is the same mistake as the status code, one step later.
One file per address
The third case needs no clever code, only a rule: every address gets its own output, named after the address, and the result line names the address it came from. Then a mix-up shows up as a mismatch you can see, not a quiet substitution. My fetch helper refuses to write two addresses into one file, and prints each page's title next to its URL, so a "not found" title under the course page's URL is visible at a glance.
What the agent may conclude
From a status code alone: nothing about the content. From a page that differs from the control and contains what it must contain: that this page, at this moment, says this. From anything else: "could not check", with the reason.
That rule cost my agents some speed. They now render pages, fetch a control, and compare, where they used to read a status and move on. It has also stopped every report of the form "the site works" that turned out to mean "the server answered".
Before you trust a page, ask it a question it can get wrong: a path that must not exist, and a word that must be there.
This is a shortened version of chapter 5 of my book in progress, Agents You Can Leave Alone, which I am writing in public on Leanpub: https://leanpub.com/agents-you-can-leave-alone
Originally published on devaland.com: A Page That Answers Is Not the Page You Asked For
Top comments (0)