Every month our credit control system produces a PDF statement for each of our six thousand trade account customers, and a script emails them out. The system writes the PDFs into one folder. A separate export gives the script the list of customers and their email addresses. For four years the script sent the first file to the first customer, the second to the second, and so on down both lists.
That worked because both lists came out of the system in the same order, by account number. In September a supplier patch changed how the PDFs were named, adding the branch code to the front of each file name. The folder now sorted by branch first. The customer list did not change.
The run started at nine. The first complaint arrived at twenty past. A roofing firm in Derby had received the statement of a plumber in Leicester, with his address, every invoice from the past three months and what he owed us. By the time credit control rang me at ten, the script had sent two hundred and twelve statements, and all but four had gone to the wrong customer. It stopped there only because our mail server limits it to two hundred an hour, which is the only reason it was not six thousand.
We stopped the job, wrote to every customer on both sides, and asked those who had received someone else's statement to delete it. Many of our trade customers are sole traders, so their statements are personal data, and we reported the incident to the Information Commissioner's Office within seventy two hours. The supplier's patch notes had mentioned the new file names. Nobody reading them knew that anything depended on the old ones.
The script now reads the account number printed inside each PDF and looks up the customer by that number. If the number in the file and the address it is going to do not belong to the same account, that statement is held back and the run carries on without it. Before each monthly run, credit control opens ten statements at random and checks them against the customer list.
Never pair two lists by position. Pair them by a value both records share, and check that it matches before anything leaves the building.
– Serguey Shinder
Top comments (1)
position-paired lists are the quietest way to leak financial data, and four years is exactly how long they run silently. the scary part is that a re-sort on either side doesn't fail loudly, it just reshuffles who gets whose statement, so nothing in the logs ever shows a defect. joining on account number plus a hard row-count check before the send step is the pattern I'd defend, and the mismatch check has to run before attachments exist, not after. worked through the same class of problem on payment statement pipelines, and the fix always ends up being a shared key rather than trust in ordering.