Originally published at tothenoc.substack.com
The site was up for twelve days. It was also broken for twelve days.
The stylesheet of one of our sites went from 18,496 bytes to zero. The server kept serving it. Status 200, every request, for twelve days.
Nothing was down. The pages answered, the uptime check was happy, the logs were clean. The site just had no design anymore.
How it happened: a small working script, left in the web root since May. Called with no parameters, it used its defaults: write, to that stylesheet, an empty string. It wrote nothing over the file and printed the new length. In the access log the request looks harmless: 200, one byte. That one byte was the digit 0.
The worse part came after. I wanted the file back and there was nowhere to take it from.
- it was not in version control, the theme lived only on the server,
- the daily backup kept fewer days than the file had been broken,
- every backup archive we had was made after it was emptied.
I rebuilt it from pieces, an old copy from May and another stylesheet, and checked the result against the compressed size the server used to send, from the access log: 4,224 bytes against 4,419. About 96%. Close enough to use, not the same file.
Three problems, one question nobody asked in time: if this file disappears now, where do I get it from?
What we do now:
- every piece of web code that reaches production is also in version control, the server is the destination, not the archive,
- a check flags any file of zero bytes on production, an empty file is a defect, not content,
- backup retention is longer than the time we need to notice a problem,
- no working scripts left in the web root,
- a check that reads the size and the content, not only the status code.
A status code says the server answered. It does not say what it answered.
If you are deciding who watches this for you, this is what to ask: what an outsourced NOC actually covers, and what it costs.
-- Alex
Top comments (0)