Context
Some time ago my team in a company that I work on had really serious issue. I fixed one feature that started working as expected, but it also started sending a lot more notification emails to our clients - nearly 150k / a day.
As you probably guess we ran out of quota. What's worse it happened for all applications we develop.
How could we prevent it?
My advice probably is not ideal and won't save you in similar case, but it will:
- help you survive longer
- give more time to prepare a fix
- won't crash all applications or modules that are sending emails, fail gradually in other words
Solution
Solution is very simple. When developing such thing as sending emails that are very important from business perspective. You should divide quota per every application or module that would be using it.
Thanks to that:
- only one app or module would be affected or ran out of quota (the rest would be working as intended)
- it will give you a space to shift quota a little if needed
- it will give you more time to prepare a fix, because not all parts of your app would be affected
Additional steps
Of course this is not only thing you could do. You could:
- implement sending emails or notifications in bulk as my very smart colleague did :)
- implement metrics and alerting in order to spot your problem before your app would be crashed
Summary
So as you see following very simple rule of applications and modules separation on application or infrastructure level let's say gives you one very important thing. Your system is more resilient and your customers won't be affected by bugs so often.
Top comments (0)