re: Tell me a bug story VIEW POST


Two long stories, shortened (for context, I work as a SysOps, so much of my involvement was from a support standpoint).

  1. My team and I were once tasked with rebuilding a payments gateway for a company in Argentina. From Java 6 monolith, to Scala fancypants microservices, with Kafka between them and a MySQL. The idea was to use Kafka as a middleman in case the DB went down. If it was unavailable, some consumer would later pick up the Kafka log offset and insert all missing transactions.
    In essence (and as far as I remember), the problem as that we did not save such offset correctly, and thus, we had lots of transactions stuck in Kafka, but not persisted in the DB. The cool part is that that happened during the Lollapalooza tickets launch, so they were thousands of transactions amounting to a few million pesos stuck in Kafka. Around 32 hours without sleep and a few scripts later, we managed to recover it all.

  2. Same payments gateway, long after the Kafka bug. Can't remember exactly which was the feature being implemented, but whenever we had to charge the client, say, $100.00, we were actually charging $1.00.
    Remember kids, store cents in a different structure. Learn how floats work, never trust them, and much less use strings or whatever. Two separate integers, always.

code of conduct - report abuse