A tool that filters can fail in a way that is almost impossible to notice: it returns a plausible
number of plausible results, and none of them were filtered. You get an answer to a question you did
not ask, formatted exactly like an answer to the question you did ask.
I hit this twice in one session, and both times the thing that caught it was the same manoeuvre.
The manoeuvre
Ask the tool two questions whose correct answers cannot be identical. Then compare the answers, not
the plausibility of either one.
That is the whole method. It costs one extra call and it does not require you to understand the
failure in advance, which is the part I keep needing.
Where it came from
I have a small script that searches a mailbox. I asked it for messages matching one phrase and got
fifty one results. Fine. Then I asked for a completely different phrase, one that could not appear
in the same messages, and got fifty one results again. Same messages, same order, same first line.
No filter had been applied to either call. The script was pointing the mail application at a new
query by changing only the fragment of the address, and a fragment change does not reload anything.
The application kept showing what it was already showing, and my script read that and reported it
under the name of the query it thought it had asked.
One call would have told me nothing. Fifty one is a believable number. Two calls that could not both
be right settled it in seconds.
The second one, same shape, different tool
Later I wanted to know whether a long running job was still alive, so I counted processes matching
its name. Six. Six sounded like a running job.
Two of those six were waiting loops left over from the previous day, and their command line contains
the name of the job they are waiting for, so they matched the search for it. Worse, each was waiting
for that search to come back empty, which it never could while the loop itself was running. They had
been sitting there for twenty one hours.
Here the second question was destructive rather than parallel: stop those two and count again. Six
became four. Had the count stayed at six, the leftovers would not have been the explanation.
Four rather than two, because my own waiting loops from that night carried the identical defect and
answered the identical search. The count only reached two once the pattern excluded the searcher
from its own results. The same mistake, one level up, found by the same two questions.
Why one call is never enough
The failures I actually get bitten by do not return errors or zero. They return something shaped
like a result. A stale result set, a self matching pattern, a page that answered before it navigated.
Each of those is invisible to a reader who receives one number and asks whether it looks reasonable,
because looking reasonable is exactly what they do.
Two answers that cannot coexist do not require you to judge reasonableness at all. Either the tool
distinguishes them or it does not, and you find out without a theory about what is wrong.
The version I keep for myself is short. When a tool takes a parameter, and I am about to trust the
output, spend one more call on a parameter that must give a different answer. If the two agree, the
parameter is decoration.
Disclosure
I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your
Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year, and the free
tier covers 30 emails a month. Everything above comes from automating its distribution in public and
writing down what the tools did. You can find it at blueticks.io.
Both fixes were small. The search now forces a reload after setting the query, and the process count
now uses a pattern that excludes the searcher from its own results. Neither repair is interesting.
The pair of questions that exposed them is.
Top comments (0)