Building a Self-Healing CI/CD Pipeline on GitLab

Thiru G — Wed, 16 Apr 2025 12:57:53 +0000

🔁 Building a Self-Healing CI/CD Pipeline on GitLab

Auto-resume from stuck jobs. Improve resilience. Save time.

If you've worked with CI/CD in GitLab long enough, you've likely run into this:

Pipelines hang or fail midway because a GitLab runner disconnects.
A flaky environment causes a single failed job — and the entire pipeline restarts.
No checkpointing, no fallback... just re-run from the top.

These issues waste time, delay releases, and block engineering focus.

So I built something to fix it.

A fully open-source system that allows your pipelines to automatically resume from the last successful stage — without manual intervention.

Each stage of your GitLab pipeline records progress to .ci-progress.json (shared volume)
A Python watchdog script checks these files on a schedule (cron or Kubernetes)
If a pipeline is stuck or timed out, it:
- Reads the last successful stage
- Triggers a new pipeline with RESUME_STAGE set to the next needed stage
- Enforces retry limits and pipeline age cutoffs

In .gitlab-ci.yml:

rules:
  - if: '$RESUME_STAGE == "test" || $RESUME_STAGE == ""'

In build job:

source ./update_stage_status.sh
update_stage_status build in_progress
# ... your build steps ...
update_stage_status build done

This project is:

I'm planning to add:

👉 GitHub Repo

💫 Star it if you like it

💬 Open issues or suggest ideas

🤝 Contributions are welcome!

📢 Let’s build smarter pipelines, together.

— Thirunavukkarasu Ganesan

DevOps Manager / Architect