Benoit COUETIL 💫 for Zenika

Posted on May 24

🌱 Keep Feeding Your CI/CD — Or Watch It Die

#devops #productivity #programming #cicd

Initial thoughts
A pipeline has a metabolism
Anatomy of a starving pipeline
The maturity ladder we're stuck on
A proactive CI/CD engineer pays for themselves — eightfold
What "feeding" looks like in practice
The conversation cheat sheet
The AI era: the organism needs to digest faster than ever
Wrapping up
Further reading

The day our client said "great, the CI/CD is done!" — that was its first missed meal. A pipeline is not a deliverable with a due date. It's a living organism. And right now, across thousands of companies, it's slowly starving while everyone wonders why deployments keep getting harder.

Initial thoughts

We've all been there. A team spends weeks — sometimes months — setting up a solid CI/CD pipeline. Linting, unit tests, integration tests, automated deployments, maybe even some security scanning. The client nods approvingly. The budget line item gets checked off. And then... nothing. The pipeline enters maintenance mode, which in practice means no maintenance at all.

This "set it and forget it" mentality is one of the most expensive misconceptions in modern software engineering. As Jez Humble puts it: we should never treat transformation as a project to be embarked on and then completed so we can return to business as usual. CI/CD is not a destination — it's a discipline. The Toyota Production System understood this decades ago with the concept of Kaizen: continuous improvement is not a phase, it's the default state. (Spoiler: Toyota did okay with this approach.)

The core problem is economic misframing. Clients see CI/CD as an infrastructure cost — something you build once, like a road. But a pipeline is not a road. It's more like a garden. Or, if we're being precise, a living organism that needs regular feeding to survive.

This article is for every developer who's watched a pipeline slowly rot, and for every decision-maker who genuinely doesn't understand why stopping CI/CD investment is the most expensive decision they'll make this year.

A pipeline has a metabolism

A CI/CD pipeline consumes. Every day, it processes new code, new dependencies, new framework versions, new security advisories, new infrastructure configurations. It has inputs (commits, merge requests, schedules) and outputs (artifacts, deployments, reports). It has a digestive system (build stages), an immune system (tests and security scans), and a nervous system (notifications, dashboards, DORA metrics).

When we feed it properly — updating test suites, optimizing build times, adding new quality gates as the codebase evolves — it stays healthy. Fast builds, reliable tests, confident deployments. The team trusts it, leans on it, accelerates through it.

When we stop feeding it, the symptoms appear gradually, like any organism under stress:

Flaky tests multiply. Nobody curates the test suite, so intermittent failures become background noise. Developers start re-running pipelines "just in case." (The CI/CD equivalent of percussive maintenance on a vending machine.)
Build times creep up. Dependencies bloat, caches expire, parallelization configs become stale. What used to take 5 minutes now takes 20. Developers context-switch. Flow state dies.
Security scans get disabled. They're slow, they flag too many false positives, nobody has time to triage. "We'll re-enable them later." We won't.
Pipeline code becomes tribal knowledge. The engineer who set it up left six months ago. Nobody dares touch the YAML. It is now sacred, mysterious, and feared — the Ark of the Covenant of your repository.

The DORA research program at Google documents these anti-patterns extensively. A pipeline where failing tests go unaddressed is a pipeline in immune collapse. A pipeline where builds take longer than 10 minutes is a pipeline with a metabolic disorder. The organism isn't dead yet — but it's on life support, and nobody's checking the monitors.

Anatomy of a starving pipeline

Here's what pipeline starvation looks like month by month. If this timeline feels uncomfortably familiar, that's the point.

Month 1 — The pipeline is fast, green, and trusted. Developers push code with confidence. Tests catch real bugs. Deployments happen multiple times a week. Life is good. 💚
Month 3 — A few flaky tests appear. They get muted with a retry: 2 or an allow_failure: true. "We'll fix them when we have time." The test suite is now slightly porous — like a net with a few holes. Small fish start slipping through.
Month 6 — Build times have quietly doubled. Dependencies are two minor versions behind, three for the frameworks nobody wants to touch. Security scans get disabled "temporarily" because they add 8 minutes and flag 47 medium-severity CVEs that nobody has bandwidth to triage. The pipeline still runs. It just doesn't protect much anymore.
Month 9 — Developers stop trusting the pipeline. Red builds are normal. "It's always red, just merge anyway." The team develops tribal workarounds: "ignore this stage," "that test only works on Tuesdays," "the deploy job needs a manual retry." New developers can't tell real failures from noise. The pipeline has become Schrödinger's quality gate: simultaneously passing and failing, useful only when observed by someone who remembers the workarounds.
Month 12 — Manual QA is back. Hotfix cycles dominate sprint planning. The client asks why delivery is slower than last year. A consultant is brought in to assess the situation. Their first question: "When was the last time someone invested in the pipeline?" Silence. 💀

As Martin Fowler puts it with his kitchen metaphor: we can't cook without making a mess, but if we don't clean as we go, the muck dries up, gets harder to remove, and eventually prevents us from cooking at all. The pipeline is the kitchen. And right now, there's dried spaghetti sauce on the ceiling and nobody can find a clean pan.

The maturity ladder we're stuck on

CI/CD is not a binary state. It's a spectrum of maturity with five clearly defined levels:

Base — Version control exists, but builds are manual or semi-automated. Testing is an afterthought. Deployments involve a checklist, a prayer, and occasionally a sacrifice to the demo gods.
Beginner — Basic CI is in place. Unit tests run automatically. The team has a pipeline, but it's fragile and partially manual. Most organizations reach this level and declare victory.
Intermediate — This is where the industry average sits. The pipeline covers the full lifecycle: build, test, deploy. But it's maintained reactively — fixed when it breaks, ignored otherwise.
Advanced — A dedicated tools or platform team exists. Pipeline improvements are planned and prioritized alongside product features. Test suites are actively curated. DORA metrics are tracked and acted upon.
Expert — Cross-functional teams own their full delivery pipeline. Deployments are a non-event. The pipeline itself is tested, versioned, and continuously improved. Roll-forward-only strategies. Feature flags instead of release branches.

Here's the uncomfortable truth: most organizations sit at Level 2 and think they're at Level 4. They built a pipeline once, it worked, and they moved on. It's the engineering equivalent of going to the gym once and wondering why the abs aren't showing. But the maturity model is not a staircase we climb once — it's an escalator moving downward. Stop walking, and we slide back to where we started. Every month without active investment erodes our position on the ladder.

The jump from Level 2 to Level 3 is where most of the immediate ROI lives: automated quality gates that catch bugs before they reach production, build optimizations that save developer hours every week, security scanning that finds vulnerabilities before attackers do. But this jump requires sustained investment — not a one-time sprint, but a recurring budget line that funds pipeline health the same way we fund product features.

A proactive CI/CD engineer pays for themselves — eightfold

Here's where we switch from metaphors to spreadsheets. Because the economic case for continuous CI/CD investment isn't just compelling — it's overwhelming. A dedicated CI/CD engineer doesn't cost the organization money. They print it. (Legally, even.)

The HP FutureSmart transformation is perhaps the most documented case study in CI/CD history. A division of 400 engineers across the US, Brazil, and India invested continuously for three years in test automation and continuous integration. The results:

Overall development costs reduced by ~40%
Number of programs under development increased by ~140%
Development cost per program dropped 78%
Resources driving innovation increased eightfold

Read that last number again. Eightfold. The team didn't just get faster — they freed up so much capacity that they could pursue eight times more innovation. And the critical insight from this case study is that these savings were only possible on the basis of a large and ongoing investment in pipeline infrastructure. Not a one-shot project. Three years of continuous feeding.

The Suncorp Group (Australian financial services) reduced 15 complex insurance systems to 2, decommissioned 12 legacy systems, and reported savings of $225 million in 2015 and $265 million in 2016. Again: sustained, multi-year investment in delivery infrastructure.

The DORA research (spanning 2014–2024, tens of thousands of respondents) consistently shows that organizations with high-performing delivery pipelines are twice as likely to exceed their profitability, market share, and productivity goals. Teams that combine version control with continuous delivery are 2.5x more likely to have high software delivery performance.

The math is brutally simple. One FTE dedicated to pipeline maintenance — curating tests, optimizing builds, updating security gates, tracking metrics — can easily save three to five FTEs worth of manual QA, hotfix cycles, integration delays, and context-switching costs. HP proved the extreme case: their CI/CD investment freed up eight times the resources they put in. The pipeline engineer is not a cost center. They are the single highest-leverage hire on the team — a force multiplier on every developer-hour in the organization. (Or, in biological terms: the veterinarian costs far less than replacing the whole herd.)

As David Farley puts it: the real trade-off, over long periods of time, is between better software faster and worse software slower. There is no third option where we get good software cheap by not investing in delivery.

What "feeding" looks like in practice

Enough theory. Here's the concrete menu for a healthy pipeline — the regular meals that keep the organism thriving.

Curate the test suite

Tests are the pipeline's immune system. But an immune system that triggers on everything (false positives) or misses real threats (gaps in coverage) is worse than useless — it breeds mistrust. We need to regularly:

Remove or fix flaky tests (quarantine, don't mute)
Add tests for recent production bugs (the feedback loop)
Keep the full suite under 10 minutes — DORA's threshold for a healthy build
Review coverage reports not for percentage, but for which critical paths are covered

Optimize build times

Speed is not a luxury. When builds are fast, developers stay in flow. When builds are slow, they context-switch, and the cost of that context-switch compounds silently. Practical levers:

Cache aggressively (dependencies, Docker layers, build artifacts)
Parallelize independent stages
Fail fast — put lint and compile before heavy integration tests
Measure. We detailed practical techniques in GitLab CI Optimization: 15+ Tips for Faster Pipelines and pushed them even further in GitLab CI: Achieving 3-Second Jobs on Million-Line Codebases.

Add and maintain security gates

Security scanning is the pipeline's early warning system. But it only works if someone triages the results. An unreviewed security report is the equivalent of a fire alarm that everyone has learned to ignore — technically present, functionally absent.

Track DORA metrics and act on them

The four key metrics — deployment frequency, lead time for changes, change failure rate, and mean time to recovery — are the pipeline's vital signs. Tracking them without acting is like wearing a fitness tracker while eating pizza on the couch. Useful data, zero impact. We explored how to compute these in GitLab: A Python Script Calculating DORA Metrics.

Automate dependency updates

Outdated dependencies are cholesterol in the pipeline's arteries. They accumulate silently, create invisible security exposure, and eventually cause a painful emergency upgrade that blocks everything for a week. The dreaded "update all the things" sprint — every team's least favorite holiday. Tools like Renovate or Dependabot turn this into a steady, manageable trickle instead of a catastrophic flood.

Invest in pipeline UX

A pipeline that developers can't understand is a pipeline they'll work around. Self-documenting job logs, clear error messages, collapsible sections with context — these aren't cosmetic improvements. They're the difference between "I can fix this myself" and "I need to find the DevOps engineer." Every ticket avoided is capacity recovered.

Dedicate platform engineering time

Even a fractional allocation — 20% of one engineer's time — creates compound returns. Someone who actively watches pipeline health, proposes improvements, and treats the pipeline as a product (not a tool) transforms the team's delivery capability over months. At maturity Level 4, this becomes a dedicated tools team. But it starts with simply deciding that pipeline health is someone's explicit responsibility, not everyone's implicit afterthought.

The conversation cheat sheet

We know the pipeline needs feeding. But the budget holder doesn't. Here's how we frame the conversation in a language that decision-makers understand.

Speak in FTEs, not features

"We need to update our test suite" means nothing to a CFO. "We can save 2 FTEs of manual QA effort per quarter by investing 0.5 FTE in pipeline maintenance" is a sentence that changes budgets. Translate every pipeline improvement into the people-hours it saves or the risk it mitigates.

Show the trend, not the snapshot

A single DORA metric reading means little. A six-month trend showing lead time increasing from 2 days to 2 weeks while change failure rate climbs from 5% to 25% is a story that writes itself. Collect the data. Plot the graph. Let the numbers do the arguing.

Run the "what if we stopped" experiment

Ask the team: "What would happen if we removed all automated tests tomorrow?" Watch the room go pale. Then ask: "What's the difference between removing them and letting them slowly become unreliable?" The answer is: only the speed at which things break. The destination is the same.

Use the kitchen metaphor

For non-technical stakeholders, Fowler's kitchen analogy lands perfectly: "Imagine a restaurant that decides cleaning the kitchen is a one-time project. The first week is fine. By month three, health inspectors shut them down. Our pipeline is the kitchen where we prepare every release."

Frame as insurance, then as investment

Start with risk mitigation (security vulnerabilities, production incidents, compliance exposure). Once that resonates, pivot to growth: "With a healthy pipeline, we can ship twice as fast with half the defects. HP proved this with 400 engineers over 3 years."

The AI era: the organism needs to digest faster than ever

Think about what a CI/CD engineer really does: they automate the automator. Developers write code that automates business processes; CI/CD engineers write systems that automate developers. They are the meta-layer — the force multiplier on top of the force multiplier.

Now AI is doing the exact same thing. Tools like GitHub Copilot, Cursor, and Claude Code are accelerating how fast developers produce code. More commits, more features, more merge requests — faster than ever before. And here's the uncomfortable implication: if the pipeline was already starving at human speed, what happens when AI-assisted developers start shipping at 3x the pace?

Boris Cherny, creator and head of Claude Code at Anthropic, made this point bluntly during an AMA at Station F. When asked "What prevents you from building faster today?", his answer wasn't code reviews, wasn't hiring, wasn't technical debt. It was: "The new major issue is CI… especially GitHub Actions." The person building one of the most advanced AI coding tools on the planet is bottlenecked by CI/CD. If even Anthropic's pipeline can't keep up with AI-assisted development velocity, imagine what's happening in teams that stopped investing in theirs six months ago.

We are entering a world where writing code is becoming dramatically faster — but running and validating it is not. The organism isn't just hungry anymore. It's being asked to digest twice the food with half the stomach. The CI/CD engineer was already the highest-leverage hire on the team. In the AI era, they might be the most critical.

Wrapping up

Our pipeline is not a tool we installed and forgot. It's the heartbeat of our team's velocity — the living system that transforms code into value, commits into deployments, ideas into shipped features.

Every week we feed it — curating tests, optimizing builds, updating security gates, tracking metrics — it gets a little faster, a little more reliable, a little more trusted. The compound returns are extraordinary: HP saw 8x innovation capacity; DORA shows 2x profitability for high performers.

Every week we starve it, it degrades a little further. Tests rot. Builds slow. Trust erodes. And eventually, we're back to manual QA, hotfix sprints, and the question nobody wants to answer: "Why is everything taking so long?"

Feed your pipeline. Track its vital signs. Treat it like the living, breathing infrastructure it is. The cost of feeding is measured in hours per week. The cost of starvation is measured in months of lost velocity.

And if someone tells us "the CI/CD is done" — we now have the data, the metaphors, and the case studies to explain why that sentence is the most expensive thing they'll say all year.

Illustrations generated locally by Draw Things using Flux.1 [Schnell] model

DEV Community