DEV Community

orca_forge
orca_forge

Posted on Originally published at forge.workstyle.tech

Overwriting Each Other’s Work with Every Deployment — 3 Rollback Incidents in 2 Sessions

#kubernetes #cicd #git

📝 Originally published (in Japanese) at forge.workstyle.tech.

Two teams were working on the same set of microservices in parallel: one on the audio pipeline and the other on conversation quality. Their responsibilities were separate, and the files they worked on barely overlapped.

Yet, three times, they overwrote each other's work in production.

1st Incident: 5 Frontend Commits Disappeared

To apply a frontend fix, I created a branch from main, built, and deployed it.

Then, a user reported:

The record is gone, and a screen I thought was removed has reappeared.

A screen that was supposed to be removed was back, and a feature I added was missing. Upon investigation, the running image wasn't built from main.

$ grep -l "virtualpv:1.0.407" ~/kaniko-builds/*.yaml | xargs grep -- --branch
  --branch feat/character-list-preview
Enter fullscreen mode Exit fullscreen mode

That branch was 5 commits ahead of main. By building from main, those 5 changes were removed from production.

In this repository, main wasn't the source of truth for production. Unmerged branches piled up: 54, 12, 6, and so on. The common assumption that "main is the latest" didn't hold here.

I restored the previous tag using kubectl set image, cherry-picked the changes to the correct base, and rebuilt.

2nd Incident: 17 Backend Versions Disappeared

After the first incident, we established this procedure:

# Before building, check if you've missed any commits from the production base branch
git log --oneline HEAD..<production-base>
Enter fullscreen mode Exit fullscreen mode

Despite following this procedure, I caused another incident during the next build.

Before building the backend, I checked the running Pod to ensure my changes were included.

>>> inspect.signature(judge_transcript).parameters['max_inserted'].default
1     # My value. The running version included my changes.
Enter fullscreen mode Exit fullscreen mode

Feeling confident, I built from main. As a result, the other team's 17 versions disappeared, halting conversation logging and losing 60 minutes of data.

I checked if the running image included my changes, but I should have checked if my build included the running image's changes. These are two different things, yet I treated them as the same.

A simple diff would have revealed the issue:

$ git log --oneline <my-HEAD>..<production-base> | wc -l
20
Enter fullscreen mode Exit fullscreen mode

Running this once before building would have prevented the incident. Although we had a procedure, I didn't follow it when switching repositories. For the frontend, I even checked the manifest's --branch, but for the backend, I only verified a single value inside the Pod.

3rd Incident: The Team Following the Procedure Was Affected

Later, the opposite happened.

I fixed the backend and deployed it. Then, I switched to monitoring the rebuild but forgot to push to main.

The other team followed the correct procedure, comparing their branch to the production base and ensuring no commits were missed. Since my changes weren't in main or their branch at that time, their check passed. As a result, my changes were overwritten.

No matter how correctly one team checks, if the other doesn't push, incidents can't be prevented.

Branch Checks Weren't Enough

After the third incident, the other team nearly caused a fourth incident in another repository, which they caught just in time. They compared the SHA256 hashes of files in the running Pod with their branch:

pipeline.py     Match
brain_talk.py   ⚠️ Mismatch  Running=1a02…  Local=341e…
app.py          ⚠️ Mismatch  Running=8a41…  Local=4110…
Enter fullscreen mode Exit fullscreen mode

The mismatches were three delivery recovery fixes I had added. If they had done a full build, all would have been lost.

At that time, they had mistaken the base branch (they thought it was a feature branch, but it was actually built from main). The git log HEAD..<base> procedure assumes the base is known, so it fails if there's a misunderstanding. SHA256 comparison works without knowing the branch name.

Another lesson was that checking for file existence isn't enough. After the second incident, I started checking if the other team's conversation_log.py existed, but in this case, the files existed but had different contents. Existence checks wouldn't catch this.

Conclusion: Two Steps Are Enough

From the three incidents, two essential steps emerged:

1. Push to main immediately after deployment.

This is the core. If both teams follow this, main will always match production, and step 2 becomes automatic. The third incident occurred because I failed to do this.

2. Before building, compare the running Pod and your branch using SHA256.

This is the safety net. Even if the other team doesn't follow step 1, this will catch issues.

kubectl exec -n <ns> deploy/<dep> -- sh -c \
  'cd /app && find apps/services -name "*.py" | sort | xargs sha256sum' > /tmp/pod.txt
(cd <repo> && find apps/services -name "*.py" | sort | xargs sha256sum) > /tmp/br.txt
diff /tmp/pod.txt /tmp/br.txt   # Ensure differences are only your changes
Enter fullscreen mode Exit fullscreen mode

Step 2 alone couldn't prevent the third incident, and step 1 alone couldn't prevent the first. Both are necessary.

Additional Pitfalls Encountered

kubectl rollout status completion can't be trusted. "Successfully rolled out" appeared immediately, but two Pods were running simultaneously:

exista-voicepipe-<rs-old>-lpqtx   Running  ...:0.7.14
exista-voicepipe-<rs-new>-cv92x  Running  ...:0.7.15
Enter fullscreen mode Exit fullscreen mode

They converged after a few seconds, but during that time, testing would have resulted in inconsistent behavior. After deployment, count Pods directly.

A similar incident occurred in a batch job generating 200 clips. The Pod was replaced mid-generation, resulting in half the clips using different settings. I only noticed when logs showed the new gate hadn't triggered once.

Don't reuse build definitions. I once modified an old manifest with sed but forgot to update the --branch, causing the first incident. It's better to keep separate definitions for each purpose. The other team now includes a note to use the running tag as the base.

Beware of tag overwrites. Once, a chain of && broke, skipping a tag bump and pushing new content to an old tag. Rolling back to that tag no longer restores the original behavior. Without knowing this history, troubleshooting becomes impossible.

Generalizable Lessons

These incidents seem specific to parallel work, but they can happen even solo.

The first incident's root cause was the common assumption that main is the source of truth didn't apply here. The deployment branch was the source of truth, and main was outdated. This is common in environments with unstructured release flows.

Even solo, if past-you built from a different branch, the same issue arises. Your past self is as unpredictable as another team member.

The key is to check if your build includes what's currently running, not what you added. You remember what you added but not what's running. That's why mechanical comparison is essential.

Series: Mass-Producing Practical Voices from Diffusion TTS

This series documents designing voices from a single caption, creating training corpora, and mass-producing role-specific practical voices. This article is Part 4: Operations.

← Previous: Four Registration Paths, Zero Management Screens

→ Next: Failing When Chasing Unmeasured Metrics with Thresholds

All 18 Articles in the Series

  1. The High-Quality TTS Was Too Slow for Conversations
  2. Voice Gacha: Designing Voices Deterministically
  3. Selecting "Narrator-Like" Voices from 24 Candidates Using Metrics, Not Ears
  4. Stricter Quality Gates Let More Monotone Voices Through
  5. Speaking Speed Can't Be Changed After Training
  6. TTS That Changes "Recording Room" Every Generation
  7. One Rough Clip Ruins the Entire Style
  8. Where Did the AI's Habit of Stretching "Hello" Come From?
  9. The Character That Broke the TTS Input
  10. Hallucination Guard That Never Fired
  11. Three Characters Became a Verbal Tic
  12. Measuring Factory Defects as Product Traits
  13. Defects Invisible to Transcription
  14. 70 Minutes Lost to a Network Blink
  15. "ja" vs "JP": The Babbling Model
  16. Four Registration Paths, Zero Management Screens 17. Overwriting Each Other's Work with Every Deployment ← You are here
  17. Failing When Chasing Unmeasured Metrics with Thresholds

The insights are compiled in the Practical Voice Mass Production Pipeline from Diffusion TTS.

Top comments (0)