At 14:07 on a Tuesday, four thousand three hundred products went out of stock on our storefront. Nothing had happened in the warehouse. Our hourly inventory import had read a file that was still being uploaded.
Our logistics partner drops a full inventory snapshot into an SFTP directory every hour, about seven megabytes, written straight to its final name. Our poller lists the directory every sixty seconds, takes anything matching the pattern, imports it, deletes it. For two years the upload finished in under ten seconds and the race never came up. That afternoon their link was slow, the write took over a minute, and our poller walked in halfway through.
The import did not fail. It read a perfectly valid CSV that happened to stop after sixty-one percent of the rows, and because the file is a snapshot rather than a set of changes, every product missing from it was treated as having no stock. We applied it in a single transaction exactly as designed. It was correct according to the only rule anyone had given it.
The proper fix belongs to the sender. Write to a temporary name and rename, or drop a marker file when the upload completes. A rename within a directory is atomic and the whole problem evaporates. Our partner agreed and shipped it a month later, which is an ordinary timescale for a change on somebody else's roadmap and is why you need the other half.
The other half is ours. The poller now requires a file's size and modification time to be unchanged across two polls thirty seconds apart before it will open it. And a snapshot is validated before it is applied: if the row count differs from the previous snapshot by more than ten percent, or if more than five hundred products would drop to zero, the import stops, keeps the previous state, and raises an alert with both files attached. That rule has fired twice since, once for a slow upload and once for a genuine catalogue change we were glad to hear about.
The presence of a file is not the completion of a transfer. Until the sender tells you otherwise, what you are reading is a work in progress with a convincing name.
– Sergey Shinder
Top comments (0)