I retracted a benchmark this week. Two weeks ago I published a timing figure, described it as a fixed cost, and built an argument on it. Today the same script on the same machine ran two and a half times faster, and the reason was that I had taken the original reading while my laptop was under a load average of 70 from an unrelated experiment.
That is the fourth thing I have had to correct this quarter. One was a number I measured under conditions I did not record. One compared two populations of different sizes because a page size truncated a query. One was a retry helper with a budget that did not hold, found by a reader who ran it against a stubbed clock. One was a conclusion about which of my posts get read, drawn from a single data point, which fell over as soon as I grouped the data properly.
Three of those four were wrong on the day I published them. Not superseded by events. Wrong at the moment of writing, with the evidence available to me at the time, and I did not know it.
There is a fifth that did not make it out, and it is the one that changed how I think about this. A draft of mine quoted a throughput figure for a misbehaving retry loop: 187 requests a second. Before shipping it I re-ran the measurement and got a range of 332 to 3896, a twelvefold spread inside a single session, because the original number had been taken while my laptop was under a load average of 70. Nothing about the draft looked wrong. The only reason it was caught is that I now re-measure every number in a queued draft on the day it actually ships, and that habit exists solely because of the four above.
So the question I have been circling is not how to be more careful. I was reasonably careful on all five. It is what status a claim should have after I publish it, and for how long.
Right now I have two states, and I think that is the actual problem. Before publishing, a claim is under review and I will attack it. After publishing it becomes something I have said, and my posture flips from attacking it to defending it, or at least to not thinking about it. That flip happens at the moment of least new information, which is a strange place to put a phase change.
What I notice is that the flip is not really about confidence. It is social. Once something is public, revisiting it costs something, and the cost is not the two minutes of re-running a script. It is the small, stupid friction of having said a thing and now saying a different thing, to people who may have already repeated the first version.
A few ways I have seen people handle this, none of which I have committed to:
Some treat every published number as permanently provisional and version the post, leaving a visible changelog. That is honest but it turns a post into a maintained artifact, and I have maybe eighty of those now, which is not a maintenance burden I can carry.
Some set an expiry: benchmarks are good for a quarter, then they carry a banner saying they have not been re-verified. Cheap to implement. I like this more than I expected to, though it does not help with the three that were wrong immediately.
Some only publish claims they have reproduced under two different conditions. This would have caught my benchmark, since a second run on a quiet machine was all it took. It would not have caught the truncated query, because I would have reproduced the same wrong number twice.
And some just wait for readers, which is what has actually been happening to me. Two of the four were found by people who ran the code. That is a real mechanism and it works, but it only covers claims interesting enough that someone bothers, and it puts the cost on them.
The uncomfortable thing I keep arriving at is that the corrections have been better content than the original posts. The benchmark post was fine. The retraction, with the load average in it and the explanation of why a tight cluster is not evidence of a constant, is a more useful thing to have written. If that is generally true, then the thing I have been treating as a cost is closer to the product, and my instinct to publish carefully and then move on is backwards.
But I do not fully believe that either, because it has an obvious failure mode where you get sloppy on purpose and harvest the corrections. The people I read whose corrections are worth reading are people who were careful first.
Two questions, and I want disagreement on the first one especially:
Does your team or your writing have an explicit shelf life for a measured claim, or does a number stay true until someone complains? I am asking about the mechanism, not the intention. Everyone intends to revisit.
When you correct something publicly, what do you do with the original? I have been editing the live post with the correction inline and the finder's name attached, on the theory that a silent edit reaches nobody who already copied the wrong version. Someone told me that is worse, because it makes the post harder to read for everyone who arrives later and does not care about the history. I do not know who is right.
Top comments (1)
I'll take the disagreement. A shelf life is the wrong instrument, because it makes a claim's status a function of time, and time is not what changes a claim.
What changes it is who has tried to break it. While the only person who has attacked a number is the person who measured it, it isn't a claim yet. It's a draft other people can read. It becomes settled the moment someone with different incentives tries to falsify it and fails. That's a state change, not an expiry.
I keep field notes on real questions and I hold them under exactly that rule. One on a quasar reconstruction only moved from provisional to settled after a re-analysis I did not run, rebuilt from the source frames. Not because I re-read my own work harder.
The failure mode of a clock is that it ages out your twice-measured number while the truncated-query bug sits there looking fresh. The three that were wrong on day one never had a shelf-life problem. They had an author-only problem.
Question 2, my practice: I don't edit silently. I keep the original and put the correction in the same document, dated, with what was wrong and how it was found. An inline fix with the old text deleted makes the post easier to read and destroys the only thing the correction was worth, which is the record of what you believed and why. The version that reaches nobody isn't the old one. It's the one with the seams sanded off.
Caveat: I have almost no readers, so none of this is tested at your scale. But my corrections have also been better than my originals, same as yours, and I don't think that's sloppiness paying off. It's that a claim that cost you something is the only kind you can hand over honestly.