The latest articles on DEV Community by Tony (@tonybnya). https://dev.to/tonybnya https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F763289%2F3d3c4506-8cf1-44ef-ac37-45b646f1f777.png https://dev.to/tonybnya en Tony Sun, 14 May 2023 09:45:47 +0000 https://dev.to/tonybnya/postmortem-2gjc https://dev.to/tonybnya/postmortem-2gjc <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--5HNrF84c--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://i.postimg.cc/dtrMWD8M/david-pupaza-he-Nw-Um-Et-Zzo-unsplash.jpg" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--5HNrF84c--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://i.postimg.cc/dtrMWD8M/david-pupaza-he-Nw-Um-Et-Zzo-unsplash.jpg" alt="error" width="800" height="533"></a></p> <h2> Issue Summary: </h2> <ul> <li>Duration of the outage: May 7, 2023, at 8:15 AM EDT - May 7, 2023, at 9:00 AM EDT</li> <li>Impact: Users trying to access the second web server were unable to do so during the outage.</li> <li>The service was completely down for 100% of the users trying to access it.</li> <li>The root cause was a misconfigured firewall rule that was blocking incoming traffic.</li> </ul> <h2> Timeline: </h2> <ul> <li>May 7, 2023, at 8:15 AM EDT: The issue was detected by a monitoring alert that showed that the second server was not responding to requests.</li> <li>Actions taken: The team investigated the second server logs and found no issues with the server.</li> <li>They then checked the load balancer logs and saw that requests were not reaching the second server.</li> <li>Misleading investigation/debugging paths that were taken: The team initially suspected a problem with the second server and spent time checking the server logs.</li> <li>The investigation was a misleading path because the issue was not with the server itself, but with incoming traffic.</li> <li>The incident was escalated to the network engineering team to investigate the network configuration.</li> <li>The incident was resolved by the network engineering team identifying the misconfigured firewall rule that was blocking incoming traffic to the second server and fixing it.</li> </ul> <h2> Root cause and resolution: </h2> <ul> <li>The root cause was a misconfigured firewall rule that was blocking incoming traffic to this server.</li> <li>The issue was fixed by the network engineering team identifying the misconfigured firewall rule and fixing it.</li> </ul> <h2> Corrective and preventative measures: </h2> <ul> <li>To prevent similar issues from happening in the future, the team will implement regular network configuration reviews to ensure that firewall rules are correctly configured.</li> <li>Tasks to address the issue include: <ul> <li>Reviewing firewall rules for all servers to ensure that they are correctly configured</li> <li>Implementing additional monitoring for incoming traffic to identify any further misconfigurations or anomalies</li> <li>Reviewing incident response procedures to ensure that misconfigurations are quickly identified and resolved.</li> </ul> </li> </ul>