Amit chakraborty

Posted on Sep 10 Originally published at amitchakraborty.dev

The DevOps bootcamp · 1. What DevOps is, once the marketing is removed

#devops #cicd #idempotency #observability

The DevOps bootcamp · Chapter 1 of 14 · DevOps · new chapter every Thursday morning

By the end of this chapter: Get an accurate map of the territory before learning any tool.

Why this matters

If you learn the tools of DevOps without understanding the system they are trying to build, you will end up doing manual operations with more complicated software. You will write a Terraform script, then execute it manually from your laptop. You will build a Docker container, then manually SSH into a server to pull and run it. You will adopt the syntax of modern engineering while keeping the exact same bottlenecks that plagued the industry twenty years ago.

Before you learn how to write a pipeline, you must understand what a pipeline is replacing. Without a map of the territory, every tool—from Git to Kubernetes—looks like an isolated administrative chore. With the map, you can see how each tool hands off to the next, forming a continuous, automated system that takes a raw text file and safely turns it into a running, monitored service.

Before you start

To execute the worked example in this chapter, you need:

A computer running Linux or macOS, or a Windows machine with Windows Subsystem for Linux (WSL) installed.
A terminal application open.
Python 3 installed (it is used solely to provide a dummy web server for the example). You can verify this by running python3 --version in your terminal.

The wall of confusion

Historically, software engineering was divided into two distinct disciplines: Development and Operations.

Developers wrote the code. Their primary metric for success was shipping new features. They were incentivized to introduce change into the system. Once the code was written, they packaged it up and handed it off to the Operations team.

Operations managed the servers, the network, and the databases. Their primary metric for success was uptime. They were incentivized to prevent change, because change is the primary cause of outages.

This created a structural conflict known as the "wall of confusion." Developers would throw code over the wall. Operations would catch it, attempt to run it on servers that were configured differently than the developers' laptops, and watch it crash. Developers would claim "it works on my machine." Operations would refuse to deploy new versions because the software was deemed unstable. The business suffered because shipping anything took months of manual negotiation, ticketing, and scheduled downtime.

DevOps is not a job title, a specific team, or a toolset. It is the systemic resolution of this conflict. It operates on a single premise: the people who write the application must also be responsible for deploying and operating it. When the person writing the code is also the person who gets paged at 2:00 AM when it breaks, the incentives align. Software is written to be deployable, observable, and resilient from day one.

Operations as a software problem

If a single team is responsible for both writing code and keeping it running, traditional manual operations become impossible. A developer cannot manually log into fifty servers to update a configuration file while also writing the next feature.

To survive, engineers had to stop treating operations as a manual administrative task and start treating it as a software engineering problem. This shift requires replacing human actions with code.

Instead of a human reading a Word document to configure a server, we write code that defines the server's state (Infrastructure as Code). Instead of a human manually running a test suite before a release, we write code that listens for changes and runs the tests automatically (Continuous Integration). Instead of a human copying files to a production environment, we write code that moves the artifact through environments based on passing health checks (Continuous Delivery).

When operations become code, they gain all the benefits of software engineering. The infrastructure can be version-controlled, peer-reviewed, tested, and rolled back. If a server dies, you do not need to remember how you configured it three years ago; you simply execute the code to provision a new one.

The map of the territory

To move an application from a laptop to a production environment, it must pass through a specific sequence of stages. This sequence is the territory we will cover in this course. Every tool you learn will map to one of these stages.

1. Source Control
Every piece of the system—application code, infrastructure definitions, deployment scripts, and configuration—is stored in a version control system. This is the single source of truth. If it is not in version control, it does not exist.

2. Continuous Integration (CI)
When an engineer proposes a change, an automated system intercepts it. This system builds the software, runs the unit tests, and checks for security vulnerabilities. The goal of CI is to prove that the proposed change is technically sound and does not break existing functionality.

3. Artifact Creation
Once the code passes CI, it is packaged into an immutable artifact. In modern systems, this is almost always a container image. Immutable means that this exact package will not be changed as it moves forward. The exact same artifact that is tested in a staging environment is the one that goes to production.

4. Infrastructure Provisioning
Before the artifact can run, it needs an environment. Infrastructure as Code (IaC) tools read configuration files and communicate with cloud providers to ensure the necessary servers, load balancers, and databases exist and are configured correctly.

5. Continuous Delivery (CD)
The CD system takes the immutable artifact and schedules it to run on the provisioned infrastructure. It does this safely, often routing a small percentage of traffic to the new version first, monitoring for errors, and automatically rolling back to the previous version if the error rate spikes.

6. Observability
Once the code is running in production, the system must emit data about its internal state. This includes structured logs, metrics (like CPU usage or request latency), and traces (following a single request across multiple services). Observability allows the team to detect and resolve incidents before customers notice them.

A worked example

To understand the difference between manual operations and DevOps, we will look at a primitive deployment.

If you were deploying a static website manually, you would create a directory, write the HTML file, find the process ID of the old web server, kill it, and start a new one. If you do this manually, you will eventually make a typo, kill the wrong process, or forget a step.

Here is that exact process, translated into a script that treats the deployment as a software problem. It is idempotent, meaning you can run it once or a hundred times, and the result is the same: the correct version of the site is running.

Create a file named deploy.sh and paste the following code into it:

#!/usr/bin/env bash
# Exit immediately if a command fails, if a variable is unset, or if a pipe fails.
set -euo pipefail

SITE_DIR="/tmp/my-production-site"
PID_FILE="/tmp/my-production-site.pid"
PORT=8080

echo "--> 1. Provisioning environment"
# -p ensures it doesn't fail if the directory already exists (idempotency)
mkdir -p "$SITE_DIR"

echo "--> 2. Deploying code"
# Writing the application code to the infrastructure
cat << 'EOF' > "$SITE_DIR/index.html"
<!DOCTYPE html>
<html>
  <body>
    <h1>Automated Deployment Active</h1>
    <p>This was deployed by a script, not by hand.</p>
  </body>
</html>
EOF

echo "--> 3. Managing service state"
# Check if we have a record of a running server
if [ -f "$PID_FILE" ]; then
    OLD_PID=$(cat "$PID_FILE")
    # Check if the process is actually running
    if ps -p "$OLD_PID" > /dev/null; then
        echo "Stopping old service (PID: $OLD_PID)..."
        kill "$OLD_PID"
        sleep 1
    fi
fi

echo "Starting new service on port $PORT..."
cd "$SITE_DIR"
# Start the server in the background and discard its output
python3 -m http.server $PORT > /dev/null 2>&1 &
NEW_PID=$!

# Record the new process ID so we can manage it next time
echo "$NEW_PID" > "$PID_FILE"

echo "--> Deployment complete. Visit http://localhost:$PORT in your browser."

Make the script executable and run it:

chmod +x deploy.sh
./deploy.sh

Open your web browser and navigate to http://localhost:8080. You will see the deployed page.

Now, run ./deploy.sh again. Notice that it safely tears down the old state and brings up the new state without human intervention. This script is a microcosm of a CI/CD pipeline and Infrastructure as Code.

Where people get stuck

Symptom: You adopt a tool like Kubernetes or Terraform, but deployments still take weeks and require approval boards.
The fix: You have confused the tooling with the practice. If you use automation tools but wrap them in manual, human-gated processes, you are not doing DevOps. You must map your value stream—every step from code commit to production—and aggressively remove manual approval gates, replacing them with automated tests.

Symptom: You try to automate a process, but the automation script is incredibly complex and constantly breaks.
The fix: You are trying to automate a broken process. Automation accelerates whatever you are currently doing. If your manual deployment process requires copying files to seven different undocumented directories and restarting services in a highly specific, arbitrary order, automating it will create a brittle script. Simplify and standardize the manual process first, then automate the simplified version.

Symptom: The team builds a CI/CD pipeline, but developers bypass it to make "quick fixes" directly on the production servers.
The fix: The pipeline is either too slow or too difficult to use. If a pipeline takes 45 minutes to run, engineers will route around it during an emergency. The fix is to optimize the pipeline's speed and lock down SSH access to production servers. The pipeline must be the only way to change production.

Your tasks

Run and modify the automated deployment: Execute the deploy.sh script provided in the worked example. Open the script in a text editor, change the text inside the <h1> tags, save it, and run the script again. Refresh your browser to verify the change was deployed without you having to manually restart the Python web server.
Map a manual process: Think of a technical task you currently do manually (e.g., setting up a new laptop, configuring a local database, or releasing a side project). Write down every single discrete step required. Identify which steps rely on human judgment and which are purely mechanical.
Identify the wall: In your current or previous organization, identify who is responsible for introducing change (new features) and who is responsible for stability (uptime). Are they the same people? If not, how do they communicate, and where does friction occur? Write down one specific instance where these conflicting incentives caused a delay.

Your tasks this week

Do the exercises above before the next chapter. Reading a tutorial and doing
one are different activities and only one of them changes what you can build.

Stuck on any of them? Say so — describe what you tried and what happened:
tell me where you got stuck. I read every one, and the questions
that come back more than twice get answered in the next chapter.