DEV Community

Vinothsingh Elumalai
Vinothsingh Elumalai

Posted on

💀 The Most Dangerous Deletions Don't Break the Build. They Break the Deploy. An AI-Assisted Recovery Story 🤖

#devops #git #sre #webdev

Table of Contents

The PR That Looked Harmless

It started with a well-intentioned pull request.

We were migrating our public developer documentation site from a legacy PHP CMS to a modern static site generator. The migration PR deleted the old PHP application from the main branch and replaced it with the new stack.

Clean, right? CI passed. Reviews approved. Merge.

Five Minutes to 503

What nobody had documented: the production servers deploy directly from main.

They run a cron job every two minutes that does a git pull. The moment that PR merged, both servers pulled down the change, the PHP app disappeared, and the load balancer's health check endpoint vanished with it.

Within minutes:

  • Both backends failed health checks
  • The load balancer had nowhere to send traffic
  • 503 for everyone

The migration itself was fine. The code was correct. The new static site worked beautifully in staging. What broke wasn't the application — it was the deployment plumbing that nobody thought to preserve.

The Revert That Couldn't Land

The team caught it fast. A revert PR was raised and merged within thirteen minutes — the PHP app was back on main in GitHub.

Problem solved, right?

Wrong. The servers couldn't pull.

The deploy key — the SSH key that gave the servers read access to the repo — had been removed during the migration work. It was treated as cleanup. Something belonging to the old system.

But it was infrastructure. Without it, git pull returned:

ERROR: Repository not found.
fatal: Could not read from remote repository.
Enter fullscreen mode Exit fullscreen mode

The revert existed on GitHub. The servers couldn't reach it. The site stayed down.

Down the SSH Rabbit Hole

No problem, we thought. Generate a new key, add it to the repo, pull. Simple.

Except it wasn't.

Problem 1: Wrong Identity

The server had multiple SSH keys in ~/.ssh/. SSH kept offering the wrong one first. One key authenticated as a puppet automation user that had no access to this repo.

We tried:

  • IdentitiesOnly yes in the command
  • Unsetting SSH_AUTH_SOCK
  • Explicit -i flags
  • Testing with ssh -T git@github.com

Every time, GitHub saw the wrong identity.

Why This Happens

SSH offers keys in order. GitHub accepts the first valid key at the authentication layer, then checks repo access separately. If the wrong key authenticates first, you get "Repository not found" — which looks like a permissions error but is actually an identity error.

The old deploy key had worked because it was configured under a custom SSH host alias in the config:

Host docs-github
  HostName github.com
  IdentityFile ~/.ssh/old_deploy_key
  IdentitiesOnly yes
Enter fullscreen mode Exit fullscreen mode

That alias was gone too — the SSH config just had Host * with StrictHostKeyChecking no. Another piece of "cleanup" that was actually load-bearing infrastructure.

Problem 2: "Key Already In Use"

We generated a fresh ed25519 key and tried to add it as a deploy key on GitHub.

"Key is already in use"
Enter fullscreen mode Exit fullscreen mode

Deploy keys must be globally unique across all of GitHub. The key we generated on one server was somehow already registered elsewhere — likely on another repo from a previous automation setup.

Rounds of SSH debugging. Still no git pull.

The Workaround

We stopped fighting SSH and took a different path:

# 1. Download repo as tarball via GitHub API
curl -L -H "Authorization: token $TOKEN" \
  "https://api.github.com/repos/org/docs/tarball/main" \
  -o /tmp/site.tar.gz

# 2. Upload to S3
aws s3 cp /tmp/site.tar.gz s3://tmp-bucket/restore.tar.gz

# 3. Generate presigned URL (short expiry)
URL=$(aws s3 presign s3://tmp-bucket/restore.tar.gz \
  --expires-in 300)

# 4. Pull onto server and extract to web root
curl -o /tmp/site.tar.gz "$URL"
tar xzf /tmp/site.tar.gz -C /var/www/site/ --strip-components=1
Enter fullscreen mode Exit fullscreen mode

Health check started passing. Did the same on the second server. Site was back.

But this was a band-aid. The cron job would still fail every two minutes silently in the background.

The Proper Fix

We generated brand new keys on each server — unique keys, never used anywhere else. Added them to the repo's deploy keys via the GitHub API (faster and more reliable than navigating the UI while firefighting):

curl -X POST \
  -H "Authorization: token $TOKEN" \
  "https://api.github.com/repos/org/docs/keys" \
  -d '{"title":"server-1-deploy","key":"ssh-ed25519 AAAA...","read_only":true}'
Enter fullscreen mode Exit fullscreen mode

Created a dedicated SSH host alias in the config:

Host github-docs
  HostName github.com
  IdentityFile ~/.ssh/docs_deploy_key
  IdentitiesOnly yes
Enter fullscreen mode Exit fullscreen mode

IdentitiesOnly yes is the critical piece. It tells SSH: "use ONLY this key, don't try anything else in my keyring." Without it, SSH offers keys in order and GitHub accepts the first one that authenticates — even if that identity has no access to the repo you're trying to reach.

Updated the git remote to use the new alias:

git remote set-url origin git@github-docs:org/docs.git
Enter fullscreen mode Exit fullscreen mode

Ran git pull. "Already up to date." The cron job would now work every two minutes like it always had.

How the AI Agent Accelerated the Recovery

Here's what I haven't mentioned yet: most of this recovery was driven through an AI agent session.

When the site went down, I didn't start by SSH-ing into servers and running commands from memory. I opened a terminal session connected to our operational stack (AWS, GitHub, PagerDuty) and started asking:

"The docs site is down. Both backends are failing health checks. Check the load balancer target health and tell me what's happening."

Within seconds: both targets unhealthy, health check path returning 404. The PHP app was gone.

"Check the GitHub repo — what was the last merge to main? Show me the diff."

The migration PR. Deleted the entire PHP directory. Now I knew the cause without logging into GitHub.

"Generate a revert PR for that merge."

Done. But then the servers couldn't pull. So:

"SSH to the docs server and check why git pull is failing. Test the SSH connection to GitHub and tell me which identity it's using."

The agent diagnosed the identity mismatch — the puppet key authenticating first, the deploy key missing. It suggested the host alias fix with IdentitiesOnly yes before I even thought of it.

When we hit the "key already in use" dead end, the agent proposed the tarball workaround:

"Download the repo as a tarball via GitHub API, upload to S3, generate a presigned URL, and pull it onto the server with curl."

Four commands. Site back. Then it generated the new deploy keys and added them via the GitHub API — no UI fumbling, no "key already in use" errors because it checked existing keys first.


Without the agent: Extended downtime — SSH debugging alone would burn 20+ minutes of human context-switching between terminals, GitHub UI, and Stack Overflow.

With the agent: Most of the debugging was conversational. Ask → answer → next step. No tab switching. No copy-pasting SSH keys between windows. No Googling "GitHub deploy key already in use."

The agent dramatically compressed the recovery time by eliminating the cognitive overhead of troubleshooting under pressure. Every command was generated, every API call was formatted, every decision was informed by live data rather than memory. What would have been an hour-plus outage with manual debugging became a focused, conversational recovery.

What We Learned

1. Your deploy mechanism is infrastructure, not cleanup

The deploy key. The SSH config. The cron job. The git remote URL. These are all load-bearing. They look like leftover configuration but they're what keeps the site alive between merges.


Rule: If deleting something would prevent your site from receiving future updates, it's infrastructure — regardless of how old or "legacy" it looks.

2. main is production until it isn't

If your servers deploy from main, you cannot delete the application from main until DNS points somewhere else. The migration and the cutover are two separate steps that must happen in order:

  1. Deploy new site to new infrastructure
  2. Verify new site works
  3. Switch DNS / load balancer to new infrastructure
  4. Then — and only then — clean up the old app from main

We did step 1, skipped to step 4, and wondered why production broke.

3. SSH key management on servers is deceptively complex

A server with multiple keys in ~/.ssh/ will offer them in an unpredictable order. GitHub accepts the first valid key at the authentication layer, then checks repo access separately. If the wrong key authenticates first, you get "Repository not found" — which looks like a permissions error but is actually an identity error.

The fix is always: dedicated host alias + IdentitiesOnly yes.

4. Have a non-git restore path documented

When your deploy mechanism breaks, you need a way to get files onto the server that doesn't depend on the thing that's broken. For us it was:

GitHub API tarball → S3 → presigned URL → curl on server
Enter fullscreen mode Exit fullscreen mode

It's ugly. It works in three minutes. Document it before you need it.

5. API calls beat UI clicking during incidents

We wasted time trying to add deploy keys through the GitHub web UI (hitting "key already in use" with no helpful error context). Switching to the API was one curl command and done.

During an outage, every minute of fumbling in a UI is a minute your users see a 503.

The Irony

The migration itself was fine. The code was correct. The new static site worked beautifully in staging.

It was the deployment plumbing — the invisible infrastructure that makes git push turn into a live website — that nobody thought to preserve.

The most dangerous deletions aren't the ones that break the build. They're the ones that break the deploy.

I'm Vinothsingh Elumalai, a Platform Engineering leader managing infrastructure for a global SaaS platform. I write about the unglamorous operational work that keeps systems alive — especially the parts where things go wrong in ways nobody anticipated.

This is part of my AI-Native SRE series.

Follow for more war stories

Top comments (0)