How To Troubleshoot a Silent Webpage (A Docker Misadventure)

Introduction:

Ever faced a webpage that refuses to say a word? Join me in unraveling a recent hiccup in our Docker adventures, where an innocuous port-curling expedition turned into an unexpected outage. In this blog post, we'll navigate through the timelines, actions taken, and the crucial lessons learned from this episode.

The Incident: A Webpage Goes Mute

It all started on a regular Monday morning at 10:47 AM (WAT). Eager to check on the latest Docker container, I initiated a simple curl command on port 8080 mapped to the container's port 80. To my surprise, instead of a cheerful page, I was greeted with the ominous message: "empty reply from server."

The Impact: Silent Users and Empty Pages

The repercussions were swift. Our webpage turned into a mute spectator, rendering nothing but an empty response. Quick estimates suggested about 5% of our users were caught in this silent storm.

Root Cause: The Vanishing Server Start

The detective work began. The culprit? A server that mysteriously forgot to start after a recent operation. A simple oversight, yet one that sent ripples through our digital landscape.

Timeline: The Race Against Time

At 10:47 AM, the silent alarm was raised, not by a user complaint, but by our vigilant monitoring alert. The race against time had begun.

Actions Taken: Unveiling the Silent Server

The first move was to inspect the server's whereabouts. What port was it listening to? Was it even awake? A thorough check revealed the dormant state of our crucial server.

Resolution: Reviving the Silent Server

With the issue laid bare, the solution was straightforward: start the server! However, to prevent such muteness in the future, a Puppet script was introduced. Now, every server update triggers an automatic start, silencing any potential disruptions.

Initially using a bash script, this was used:

#!/usr/bin/env bash
# get Apache to run on the container and to return a page

service apache2 start

With Puppet for automation, let's create a Puppet manifest file. Let's call it apache_service.pp:

# File: apache_service.pp

# Define a class for managing the Apache service
class apache_service {
  service { 'apache2':
    ensure => 'running',
    enable => true,
    subscribe => File['/etc/apache2/httpd.conf'], # Trigger the service restart when the Apache configuration file is updated
  }
}

# Apply the class to ensure it is included when Puppet runs
include apache_service

Root Cause and Resolution: Learning from the Silence

The root cause was a temporarily faulty server, fixed but left in silence. The resolution? A proactive decision to script server starts, ensuring they speak up after every update.

Corrective and Preventative Measures: Scripts and Vigilance

To fortify our defenses, every engineer is now armed with mandatory pre and post-server operation scripts. Additionally, the Quality Assurance team is on high alert, (re)educated on the nuances of our vigilant monitoring tools.

Conclusion: Lessons from Silence

In the ever-evolving landscape of Docker and servers, even a momentary silence can be deafening. This incident taught us the importance of vigilance, automation, and proactive measures. Let this be a reminder to all: in the realm of webpages, silence is not always golden.