Week 1. "Today you'll shadow me during a deployment."
What You'll Learn
By the end of this series, you'll be able to explain confidently:
- What actually happens after
git push - How GitHub Actions, Jenkins, GitLab CI, and other CI systems work conceptually
- Why runners exist and how they execute your code
- How production deployments differ from local deployments
- Why companies use artifacts, Docker images, registries, and deployment strategies
- How secrets, SSH keys, and environments are managed safely
- Why pipelines fail, how engineers debug them, and how rollbacks work
- The tradeoffs startups and large companies make in their CI/CD workflows
This series isn't about memorizing YAML syntax. It's about living through the production systems behind modern software delivery — one incident, one question, one term at a time.
Senior Engineer: Today you'll shadow me. Nothing to do yet — just watch.
Junior Engineer: I merged my first PR yesterday. It got approved, checks passed. How does it actually reach production from here?
Senior Engineer: Let me answer that with a story instead of a diagram.
Three years ago, at a company I worked at — Friday evening, 6:45 PM. Everyone's packed up. One developer, still at his desk, says, "just one small fix, two minutes." SSHs straight into the production server.
Junior Engineer: Wait — he SSHs into production? I thought developers usually weren't allowed to touch that directly.
Senior Engineer: Good catch. Depends entirely on where you work.
🏙️ Startup vs Big Tech — Who Gets to Touch Production
Small Startup — Founder or one of the first few engineers deploys manually. SSH key lives on someone's laptop. Single VM. No process, because nobody's had time to build one yet.
Growing Startup — GitHub Actions shows up. A Docker container or two. One staging server before production. SSH access still exists, but it's shrinking.
Product Company — Self-hosted runners, multiple environments (dev, staging, prod), an artifact registry, blue-green deployments, and someone has to approve before anything reaches real users.
Bank / Fintech — No SSH to production, full stop. Every deployment is logged and audited. Change approvals required. Secrets are tightly restricted. Compliance isn't optional — it's the job.
Senior Engineer: This story is from the first category. That matters — it's part of why what happens next was even possible.
📝 Production Note
Most companies never run production deployments directly from a developer's laptop — even when someone technically still has SSH access. Mature teams prefer deployments initiated by a pipeline, because every action a pipeline takes becomes logged, repeatable, and auditable. A human typing commands from memory is none of those things.
Senior Engineer: He runs three commands.
git pull
npm install
pm2 restart app
Junior Engineer: Why PM2? Why not just node app.js?
Senior Engineer: Because node app.js dies the moment the process crashes, or the terminal closes, or the server reboots. PM2 is a process manager — it keeps your app running, restarts it automatically if it crashes, and lets you do things like pm2 restart app without killing and re-launching by hand. Full conversation on process managers, later. For now — it's the thing making sure your app doesn't quietly die at 3 AM with nobody watching.
Junior Engineer: And npm install — I've also seen npm ci. What's the difference?
Senior Engineer: npm install can update your lockfile and pull slightly different versions if something upstream changed. npm ci installs exactly what the lockfile says, nothing more, nothing less — and it's faster, since it skips the resolution step. In production, you almost always want npm ci. This developer didn't know that yet either.
He runs it. Checks the homepage. Loads fine. Shuts his laptop. Leaves for the weekend.
Junior Engineer: That's... basically what I assumed deployment was.
Senior Engineer: That's how thousands of companies started. It's also, almost word for word, why thousands of companies eventually stopped doing it that way.
7:02 PM. PagerDuty goes off.
Junior Engineer: Sorry — what's PagerDuty?
Senior Engineer: Fair question, you've probably never had to use one.
Imagine production breaks at 2 AM. Who wakes up? Does AWS just know to call someone? Does Slack ring your phone?
No. Companies use an incident management platform — PagerDuty is one of the most common, there's also Opsgenie, VictorOps. When monitoring detects something serious — payment failures, API errors, CPU spikes, a service going unreachable — it doesn't send an email. It calls whoever's on-call. Loudly, on purpose. A sleeping engineer doesn't read email at 2 AM, but they'll pick up a phone that won't stop ringing.
Someone opens the incident Slack channel — already moving. Support is typing "getting calls about failed payments, is something down??"
🪞 If I asked you this in an interview
"Why do companies use incident platforms like PagerDuty instead of just emailing the on-call engineer?"
Because production failures need immediate attention. Email is asynchronous and easy to miss for hours — an on-call system escalates repeatedly, through calls and pages, until someone actually acknowledges it.
Someone opens GitHub.
Junior Engineer: Why GitHub? The problem's happening in production, not in the repo.
Senior Engineer: Good question — and it's the first question every experienced engineer asks in an incident.
Production was fine ten minutes ago. It isn't now. So before anyone blames the database, or the cloud provider, or Kubernetes, the first thing to check is always: what changed? Did somebody deploy? Which PRs merged? Which commit actually reached production? Who approved it?
📒 Senior Engineer's Notebook
The first question during an outage is rarely "who wrote the bug?" It's almost always "what changed recently?"
Junior Engineer: What happens while they're checking that?
Senior Engineer: In parallel — someone starts tailing logs.
Junior Engineer: What does "tailing logs" actually mean? I've heard the phrase, never really understood it.
Senior Engineer: Worth stopping on, you'll do this constantly once you're on-call.
Simplest version — on a single server, there's a literal command:
tail -f app.log
tail shows the end of a file. -f means "follow" — keep showing new lines live, instead of a one-time snapshot. So instead of scrolling a huge file, you watch it happen in real time.
That works with one server. It falls apart the moment you have twenty, each writing its own log, and the failed request could've hit any one of them.
Junior Engineer: So what do bigger companies use?
Senior Engineer: Centralized logging — the app ships every log line somewhere central instead of leaving it on the machine that wrote it. Common names: CloudWatch (AWS's native option), the ELK stack — Elasticsearch, Logstash, Kibana — Datadog, and Loki, Grafana's logging system. Different products, same idea: collect logs from everywhere, put them somewhere searchable.
Two things matter once you're searching. First — structured logs. Instead of a line that just says payment failed for user, a structured log looks more like:
{ "event": "payment_failed", "userId": "8213", "orderId": "A991", "timestamp": "..." }
Now you filter by field instead of grepping sentences and hoping the wording matches.
Second — correlation IDs. One request might touch five services on its way through — gateway, payment service, inventory service, and so on. A correlation ID is one unique value attached at the very start, logged by every service it passes through. Search that one ID, and you see the entire journey of that request across every system — instead of five unrelated log streams.
Junior Engineer: That's a lot more than "check the logs" sounded like.
Senior Engineer: It always is, once you actually need it under pressure. Back to Friday — someone's tailing logs, someone else pulls up metrics, and the room is still just asking: did anyone deploy recently?
Nobody knows yet. That's the part people never picture. For the first several minutes of a real incident, nobody knows anything. Just noise, and a clock running.
Junior Engineer: But he only changed one line.
Senior Engineer: I've sat through incidents shaped almost exactly like this more than once. Different companies, different bugs, same shape. Someone saves five minutes during a deploy, spends the next hour trying to earn it back.
Junior Engineer: Couldn't they just undo it? Roll back?
Senior Engineer: Depends. Can you always?
What if that deploy already ran a database migration — already changed a column, already wrote data in a new shape? Rolling back code is one git command. Rolling back everything the code already did to your data is a different problem, and it doesn't have a one-line answer. Deserves its own full conversation — we'll get there.
🚨 Beginner Mistakes
"I'll just SSH and fix production directly." Even if it's faster tonight, it creates configuration drift — the running server no longer matches what's in the repo, and nobody else knows what actually changed.
"I passed tests locally." That tells you your machine is fine. It says nothing about production's environment, data, or scale.
"I'll use npm install in CI." Now your build isn't reproducible — two runs of the same commit could quietly pull different dependency versions.
Senior Engineer: That Friday, they found the line, reverted it, restarted the service.
Junior Engineer: So — they just fixed the bug?
Senior Engineer: Fixing it took eight minutes.
Finding it took fifty-two.
Junior Engineer: That's a huge gap.
Senior Engineer: The code is rarely the expensive part. The investigation is. Five people, staring at dashboards and logs, figuring out which of that day's changes actually mattered.
🏢 Office Reality
You'll hear phrases like these constantly once you're in these conversations for real — worth decoding early, even before you've met all of them properly:
- "The deployment is green." Means the deploy process finished without errors. Does not mean production is healthy — those are separate claims, and this Friday is the perfect proof.
- "The pipeline is red." A step failed — could be a test, a build, a lint check. Doesn't yet tell you which.
- "The runner is stuck." The machine executing your pipeline steps hasn't responded — we haven't met runners properly yet, but we will.
- "Promote staging to production." Take a build that's already been running safely in a test-like environment, and push that exact same build to real users.
- "The artifact is corrupted." The packaged, built version of your app — the thing the pipeline produces and hands off to deployment — got damaged or built incorrectly. More on artifacts soon.
Whiteboard moment
Production
↓
Deployment
↓
Monitoring
↓
Incident
↓
Rollback
↓
Postmortem
Everything in this series lives somewhere on this line. Today we've walked from Deployment into Incident. Rollback and Postmortem are still ahead of us.
Why Not Just Hire More People?
Senior Engineer: Why do you think companies eventually stopped deploying by hand like this?
Junior Engineer: Because... it's slow?
Senior Engineer: Partly. What else?
Junior Engineer: People make mistakes doing repetitive stuff manually?
Senior Engineer: Exactly — now let's go one layer deeper than "mistakes happen," because that's not quite the real reason either.
Think about what it'd take to never have that fifty-two-minute scramble again. Suppose the company says: fine, before anything reaches production, someone runs through this, no exceptions.
□ Pull latest code
□ Install dependencies (npm ci, not npm install)
□ Build
□ Run unit tests
□ Run integration tests
□ Run database migration
□ Restart services
□ Verify health endpoint
□ Check metrics
□ Notify support
Ten steps. None hard. A junior engineer could run this correctly on day one.
Junior Engineer: So just... follow the list.
Senior Engineer: I used to think I could hold something like this in my head — careful, attentive, no checklist needed. Then one Friday, I forgot to run a database migration before restarting the service. I never trusted memory again after that. Mine or anyone else's.
Now imagine running that list correctly. Thirty times a day. Every day. For years. Never tired, never rushing because it's 6:45 on a Friday.
Junior Engineer: Nobody does that forever without slipping.
Senior Engineer: Right. The problem was never intelligence — it's repetition. Some companies tried hiring someone whose whole job is running this checklist.
Junior Engineer: Did that work?
Senior Engineer: For a while. Then someone asked — if the entire job is "ten fixed steps, same order, zero judgment required" — is that really work for a person?
Junior Engineer: ...that sounds like something a machine should do.
Senior Engineer: So that's what got built — something that runs the exact same checklist every time. No memory to trust. No "it worked yesterday."
Junior Engineer: Does it have a name?
Senior Engineer: It does. We'll live inside it for several conversations. Most engineers call it a pipeline — GitHub Actions, GitLab CI, Jenkins, depending where you work. Name changes, idea doesn't.
🪞 If I asked you this in an interview
"Why don't manual production deployments scale, even with careful, experienced engineers?"
Because the failure mode isn't a lack of skill — it's repetition. The same fixed checklist run correctly a thousand times eventually gets run incorrectly once, and at 30 deploys a day, "eventually" arrives fast. Automating the checklist removes reliance on memory and attention, not intelligence.
🎯 Interview Perspective
Interviewer: Why do companies use CI/CD?
Weak answer: Because it automates deployment.
Strong answer: CI/CD standardizes repetitive deployment steps so every deployment follows the same validated process. It reduces human error, guarantees consistent environments between runs, creates an audit trail of exactly what shipped and when, and lets teams deploy frequently because they trust the process — not despite it.
"My pipeline passed. Can I deploy?"
Junior Engineer: If the checklist runs every time, doesn't that slow people down for tiny fixes?
Senior Engineer: Check the math on Friday instead of guessing. Skipping the checklist saved maybe four minutes. The fix plus the investigation together cost sixty. So the real question was never "was the fix small" — it's what four minutes of skipped checking actually cost, once you count what came after.
Junior Engineer: Fifty-six extra minutes. Plus every payment that failed in that window.
Senior Engineer: That trade never looks expensive in the moment. Only in the postmortem, once someone adds it up.
Here's a version of this question I get from almost every junior engineer eventually — "my pipeline passed, can I deploy?"
Junior Engineer: I mean... that was going to be my next question.
Senior Engineer: You're asking the wrong question, gently. A passing pipeline doesn't mean production is safe. It means the tests you chose to write happened to pass. Different sentences entirely.
📒 Senior Engineer's Notebook
A deployment isn't finished when the pipeline turns green. It's finished when users are successfully using the feature.
Three Changes, One Broken Night
Senior Engineer: Here's a harder one — the kind with no obvious villain.
Same day, same repo, three developers. A updates the payment library. B rotates an environment variable. C upgrades the base Docker image. Each opens a PR. Each pipeline, individually, goes green. All three merge that afternoon. That evening, production breaks. Which one caused it?
Junior Engineer: The payment library. A.
Senior Engineer: Wrong.
Junior Engineer: The env variable, then — B. Feels like the sneaky one.
Senior Engineer: Wrong.
Junior Engineer: ...the Docker image, C?
Senior Engineer: Also wrong, on its own.
Junior Engineer: So none of them?
Senior Engineer: All three.
Junior Engineer: How, if each one passed alone?
Senior Engineer: Because none were ever tested together. The new Docker image changes how environment variables load — harmless alone. B renamed a variable the same day — also harmless alone, since B's pipeline never touched the new image. A's payment code was still reading that variable under its old name.
Junior Engineer: So once all three landed the same evening, the variable A's code needed just... wasn't there anymore.
Senior Engineer: Exactly. Three individually reasonable decisions that never met each other until they were already live. That's most real outages — not one bad call, three good ones that collided.
🪞 If I asked you this in an interview
"How can three separate pull requests each pass their own tests, yet still break production together?"
Because a passing pipeline only proves a change is safe in isolation. It says nothing about how that change interacts with other changes landing the same day. Integration risk lives between PRs, not inside any single one of them.
What You Should Be Able to Explain Now
(Without looking at Google)
Can you explain:
- Why manual deployments don't scale?
- Why investigation usually costs more than fixing?
- Why companies automate deployment checklists instead of hiring people to run them?
- Why "deployment succeeded" isn't the same as "production is healthy"?
- Why CI/CD exists as more than just "automation" for its own sake?
If yes — you're thinking like an engineer, not someone memorizing tool names.
Junior Engineer: So when I push code — who actually presses the button that kicks all this off?
Senior Engineer: Nobody. Not directly.
🔧 What Happens Behind the Scenes
You type
git push
Does GitHub immediately start running your tests?
No. First, GitHub receives a webhook — a notification that something happened in the repo. Then it checks whether .github/workflows/ contains any workflow file matching this event.
If no matching workflow exists — nothing happens. No pipeline. No runner. Nothing. Silence.
If one does match — only then does anything wake up.
Junior Engineer: Wake up where, though? Whose computer is actually running my tests?
Senior Engineer: That's the real question. And why can't GitHub just run the code itself, on its own servers, without any of this ceremony?
That's exactly where we're going next.
So — where did that machine come from?
Top comments (0)