On the second of June two hundred and twelve of our business customers received their May invoice twice. Nine of them received it three times.
Invoicing runs on eight workers that take customer accounts from a list. To stop two workers invoicing the same customer, each takes a lock in Redis first: a SET with NX and an expiry of sixty seconds, keyed on the customer. A minute was generous, because an invoice usually took about four seconds. When a worker finished, it deleted the key.
That month our largest customers had far more usage lines, and the tax service we call for every line was slow. The biggest invoices took between two and five minutes. After sixty seconds each lock simply expired. Redis has no idea whether the holder of a key is still working. Another worker found no lock on that customer, took one, and started the same invoice from the beginning. The first worker finished, saved the invoice, emailed it and deleted the key, which by then belonged to the second worker. So a third worker could start as well. That is where the triples came from.
A lock with an expiry is really a lease. It ends at a time, whether the work is finished or not, and the holder cannot tell that it has ended. A slow call or a long garbage collection pause looks the same from inside: the worker carries on, holding something it lost minutes ago.
We sent credit notes that afternoon and moved the guarantee out of the lock. Invoices have a unique constraint on customer and billing period, and the email goes out from an outbox only after that insert succeeds, so a second worker fails at the database instead of at the customer's inbox. The lock remains, to save wasted work, and each acquisition carries a fencing token, a number that only goes up, which the database checks on every write. A worker releases a lock with a script that deletes the key only if it still holds its own value. Workers renew their lease every twenty seconds while they work, and abandon the job if a renewal fails. And we alert on any lock that expires while its holder is still running.
Every expiry on a lock is a guess about how long the slowest job can take. Ours was a good guess for four years, and then the work grew.
– Sergey Shinder
Top comments (0)