If you have to design a system that can serve 1B+ ad requests/day or ~200 GB/hour, then most likely your choice is going to be some fancy big data technology. And of course that makes sense too. These numbers would seem like a puny human against the hulk of a tool like Kafka, Spark, Flink, Hadoop and what not. It's easy to design that way.
If you have to design a system that can serve 1B+ ad requests/day or ~200 GB/hour, then most likely your choice is going to be some fancy big data technology. And of course that makes sense too. These numbers would seem like a puny human against the hulk of a tool like Kafka, Spark, Flink, Hadoop and what not. It's easy to design that way.
However, this obvious decision will soon convert into a dilemma if I say that we have this traffic not today, not in the next 6 months, but in future. And we don't want to spend much money right now. And we are already running late because we wanted this system yesterday. And it will be a team of two people, the other guy is a novice.
So, now it's tricky. We have to quickly design and implement a future scale proof system that is affordable and easy to maintain.
At #AdPushup we faced a similar kind of situation. And we came up with a system that recently scaled 10x. I was emotional and hence this article :)
https://medium.com/@udit.jindal/frugal-approach-to-scaling-c4d3b100a608
Do give it a read and let me know your thoughts.
PS-1: That other novice guy was me (Luckily!).
PS-2: If you find this interesting, let me know in the comments, I'll share how we developed an in-house ad auctioning system(based on OpenRTB 2.5) under 21 days.
Top comments (0)