Budgtr Downtime incident report.

Dayisi Tobi Iyanu — Sun, 09 Jun 2024 21:50:36 +0000

We would like to apologize to all our users for the downtime experienced last week. We understand the discomfort this would have caused, and we have done the necessary checks and fixes to ensure that this does not happen again.

We have provided an incident report of the downtime that occurred on the 5th of June, 2024. Also outlined is our response to the issue.

Issue Summary

The issue started from 6:03 AM to 8:52 AM WAT, requests to the website resulted in a 500 error as users couldn't access the service for this period. The cause of this outage was an untested change that was pushed into production, which resulted in a bug.

Timeline (All West African Time)

6:03 AM - New Change was pushed.
6:55 AM - The first Outage occurrence was logged in.
6:56 AM - Our monitoring system alerted us.
7:20 AM - failed change was rolled back.
7:30 AM - Successful change rolled back
8:00 AM - New change was tested and pushed into production.
8:30 AM - Server was restarted.
8:52 AM - System was restored at 100% functionality.

Root Cause

At 6:03 AM a new feature that was discussed and approved for development within the team was pushed to production without being tested. The new feature is a payment infrastructure that Budgetr would be using, but the APIs to be consumed were not consumed properly which resulted in breaking the whole code base which then caused the 500 error.

Resoluion and Recovery

At 6:56 AM our monitoring system alerted our engineers of the error, which escalated immediately. At 7:20 AM our engineers tried to roll back the change to fix it locally, but it failed due to some authorization constraints.

At 7:30 AM the proper authorization access was granted and our engineers were able to successfully roll back the changes. Our engineers went to work straight and fixed the error, after pushing it to the test environment, the test process took place and the results came out as positive.

At 8:00 AM our engineers pushed to production. To ensure a smooth sail of service, we restarted the servers at 8:30 AM and our service was confirmed to be 100% stable at 8:52 AM.

Corrective and Preventative Measures

In the last 4 days, we have conducted an internal review and analysis of the outage. The following are the actions that will be taken to ensure this issue doesn't occur again:

All new features would be pushed to the test environment by default.
Only authorized personnel can push tested and approved changes to production.
Detailed information concerning the features or changes being pushed to test should be provided in the commit messages.

Budgetr is committed to ensuring a seamless service for all our customers and as a result we constantly improve our technology and operational processes to prevent these issues. We sincerely apologize for the discomfort this issue would have caused you or your businesses and we appreciate your patience and understanding.

Sincerely,
The Budgetr Team.

What happens when you type google.com in your browser and press Enter

Dayisi Tobi Iyanu — Sat, 11 May 2024 23:04:57 +0000

Have you ever wondered what happens when you type in "www.google.com" on your browser? Why does the Google website appear on your browser instead of another website?
There is this wonderful thing that happens in the web universe. Let's take a walk.

John needs to research his coursework and then decides to check Google on the topic. He enters "www.google.com" on his Chrome browser, like all processes there is the first step. The first step is the DNS Requests.

DNS REQUESTS

There is something called an IP address. This is a unique dot-separated number combination that identifies a device connected to a computer network. IP addresses are boring to memorize and that is where the DNS comes in. Just like your phonebook stores phone numbers with a name to identify the numbers. Also, DNS stores IP addresses with hostnames (www.google.com) to identify the addresses. Therefore, the DNS is the phonebook of the internet. Hence, the DNS gets the hostname and searches for the IP address that is attached to it.

The IP address has been identified, what next? No, you didn't guess it, the TCP/IP (Transmission Control Protocol/Internet Protocol) comes next.

TCP/IP

These guys are responsible for the transmission of data from the server the IP address identifies with to John's computer. The TCP ensures that there is a reliable connection with John's computer and that the data are delivered in order. On the other hand, the IP ensures that the data is routed to the right destination (John's computer).

That's all, right? No! what if the data is malicious? We have a new guy called the Firewall.

Firewall

This guy loves his job, he is the security who scrutinizes the data that has been transferred into the computer and makes sure that they are safe based on predetermined security rules (He doesn't smile).

This must be it, well not yet. We take security serious
here and we don't want anyone minding John's business, so we got the perfect guys for the job, HTTPS/SSL.

HTTPS/SSL

These guys (Hypertext Transfer Protocol Secure/Secure Sockets Layer) make sure the connection between John's computer and the Google server is secure. It encrypts the data transferred between John and the Google server.

That is all about John, but let me tell you about what happened in the Google server when John made that request. Google has a lot of users, therefore, one server for all its users will be catastrophic. So to ensure optimum service Google makes use of a lot of servers. To ensure all servers are receiving the right amount of requests we employed a manager called the Load Balancer, this guy is terrific.

LOAD BALANCER

This guy ensures the distribution of network traffic across all the servers so they don't overwork and crash. Yeah yeah, it balances the load (I know you thought about it).

But what do these servers do actually? Sip more coffee and relax.

WEB SERVERS

These guys interpreted John's requests. They knew what action to carry out based on the HTTP request John sent, and communicated with the application server to process the request.

Yeah, I mentioned the application server, just relax this story will end.

APPLICATION SERVER

Google is an application and all the contents John saw were processed by this guy. He processed the request and sent the response John asked for.

Yeah, that must be all, well one thing left, it is the database. (Just one more cup of coffee)

DATABASE

This guy is Google's recordkeeper, to show John what he asked for, the application had to ask the database for these records.

Finally, the response is sent to John, and he can view the Google web page.

Now, I have come to the end of my story, I hope you enjoyed it but most importantly you now know what happens when you request that web page.

Odabo.

DEV Community: Dayisi Tobi Iyanu