DEV Community

Serguey Shinder
Serguey Shinder

Posted on

Our Security Software Quarantined Our Own Despatch App on Three Hundred Depot PCs the Morning After We Updated It

Our despatch app is the one piece of software we write ourselves that every depot depends on. It runs on about three hundred PCs in transport offices and goods in, and it prints the load sheets that tell the yard what goes on which lorry.

In February a contractor shipped a new version that fixed a slow screen. Our device management pushed it out at five in the morning, as planned. At ten past six the first transport office opened it, and our endpoint security software removed it as malware. By half past six it had done the same at every depot. The yards loaded from paper until nine.

Nothing malicious had happened. The contractor had changed the tool that packaged the app, and the new build was unsigned. Our security software judges programs partly by reputation, and a brand new unsigned program that almost no other computer in the world has ever run looks exactly like what an attacker would deploy. The previous version had built up a reputation over months. The new one had none.

The first fix suggested on the call was to exclude the app's folder from scanning. We refused, because that would let anything placed in that folder run unchecked, on every depot PC. Instead we allowed that one file by its fingerprint, restored it from quarantine across the estate and submitted it to the vendor as a false positive.

We made three permanent changes. Every build of the despatch app is now signed with the certificate we already owned for our quote workbook, and our security policy trusts that publisher, so a genuine new version is recognised by who made it rather than by whether it has been seen before. New versions now go to ten PCs a day before everyone else, and those ten run the same protection as the rest. And the release checklist now ends with opening the app on a protected machine, not on the contractor's laptop.

Software you write yourself has to earn trust from your own security tools like anything else. Test a release on a machine that defends itself the way your real ones do.

– Serguey Shinder

Top comments (0)