The programme delivered on time, which is worth saying, because a lot of what follows sounds like criticism of people who did a hard thing well. The platform was better than what it replaced in every respect that had been specified. Six months later the finance team were still keeping their own records, and the reason was the data.
Cleansing had been in the original scope. It came out in the second quarter, when the date started to look uncertain, and it came out for a defensible reason: the new system was the priority, the records could be tidied afterwards, and nobody wanted the go-live held hostage to an exercise with no visible end. What arrived on the other side was twenty-three years of accumulated history exactly as it stood. Around nine thousand customers who were duplicates of each other under slightly different names. Supplier records with fourteen spellings of the same company. A status code retired in 2011 still attached to live accounts. Addresses in a free text field because the old system had one, and a delivery instruction habit where operations typed notes into a name field.
Then came the decision that did the lasting damage. The new platform validated its inputs properly, which was one of the reasons we bought it, and a meaningful share of the legacy records failed that validation. Rather than delay, we disabled validation for migrated data. From that day there were two classes of record in one system, and every report, every rule and every integration had to cope with both, forever.
The lesson is not that cleansing should never be descoped. Sometimes it must be. It is that the choice has to be made record by record and stated out loud: fix it, archive it and leave it behind with a documented way to read the old system, or bring it in dirty and accept the permanent cost. Ours was made once, implicitly, for everything.
We also learned that cleansing is not an IT task and will fail as one. Only the departments that created the records can say which of two customers is real.
A migration is the one moment when the whole organisation is looking at its data. Spending it on the move alone is a waste of the only attention that data ever gets.
– Serguey Shinder
Top comments (0)