Afante

Posted on Sep 5 Originally published at qaon.net

Is CI/CD Your Savior? Think Again.

A pragmatic look at the hidden costs, the small-batch myth, and why user trust matters more than deploy frequency.

I. Introduction

"CI/CD will save you money. It will make your team faster. It will eliminate bugs. It is the only way to build modern software."

We've all heard this. The industry preaches CI/CD as a universal solution. Every conference talk, every DevOps blog post, every vendor pitch tells you the same thing: adopt CI/CD or be left behind.

But is it really that simple?

After years of watching teams adopt (and struggle with) CI/CD, I've come to a different conclusion. CI/CD is a powerful tool—but it is not a magic bullet. When implemented poorly, it can cost more money than it saves, introduce more bugs than it prevents, and damage user trust faster than any traditional release process ever could.

This post is not an anti-CI/CD rant. It is a call for thoughtful, context-aware implementation. I want to walk you through the real costs, the hidden traps, and the practical framework I've arrived at after questioning the orthodoxy.

Let's start with the most uncomfortable question of all.

II. The Cost Analysis: Does CI/CD Actually Save Money?

The Common Sales Pitch

The CI/CD sales pitch is seductive:

"Automate everything and save developer hours."
"Deploy faster and reduce operational costs."
"Catch bugs earlier and reduce downtime."

It sounds logical. Who wouldn't want faster deployments, fewer bugs, and lower costs?

The Reality

Here's what the sales pitch doesn't tell you:

CI/CD requires infrastructure: runners, cloud compute, artifact storage.
CI/CD requires expertise: SREs, DevOps engineers, platform teams.
CI/CD requires maintenance: fixing broken pipelines, updating dependencies, managing flaky tests.
CI/CD requires context-switching: developers losing focus while waiting for pipelines to finish.

These costs are real. And for many teams, they outweigh the benefits.

The Math (For a Team Deploying Weekly)

Let's run the numbers for a typical team of 5 developers deploying once per week, running linters and unit tests 50 times per developer per day (a realistic number during active development).

Activity	Frequency	Manual Time	CI/CD Time	Difference
Linting	50x/week	25 min	25 min	Tie
Unit Tests	50x/week	100 min	100 min	Tie
Deploy to Staging	1x/week	5 min	1 min	CI/CD saves 4 min
Smoke Tests	1x/week	3 min	2 min	CI/CD saves 1 min
Deploy to Production	1x/week	15 min	1 min	CI/CD saves 14 min
Total		148 min	129 min	CI/CD saves 19 min/week

Monthly savings: ~76 minutes of human time. Less than an hour and a half.

Monthly cost of CI/CD:

SRE/DevOps salary: ~$5,000/month
Cloud compute costs: ~$500/month
Developer context-switching from flaky tests: ~$500/month of lost productivity

Total monthly cost: ~$6,000.

Cost per minute saved: $6,000 / 76 minutes = ~$79 per minute saved.

That is a terrible ROI. For a team deploying once a week, CI/CD is a net loss.

When CI/CD Actually Saves Money

CI/CD only becomes financially viable when three conditions are met:

High deployment frequency. If you're deploying multiple times per day, the time savings compound rapidly. A team deploying 10 times per day saves ~3,800 minutes per month—a completely different equation.
Large engineering teams. The cost of SREs and platform maintenance is diluted across more developers.
Complex infrastructure. When manual deployment is risky (multiple services, complex dependencies, compliance requirements), the safety net of automation is worth the cost.

The Startup Tax

For startups, the equation is even worse. A CI/CD platform can triple infrastructure costs and add maintenance overhead before product-market fit is even found.

Here's what a startup typically does:

"We set up GitLab CI with 4 different stages, parallel testing, and Kubernetes deployments before we even had our first paying customer."

Here's what they actually needed:

A simple deploy script (rsync or scp to a single server).
A few well-written unit tests.
Time spent talking to users, not debugging pipelines.

The result of the enterprise approach:

Cloud bill triples before they have revenue.
Developer velocity drops by 30-50% due to pipeline wait times.
The team feels "professional" because they have a complex pipeline—but they're still building the wrong product.

The "You Are Not Google" Rule

This is the single most important rule for startups:

"You are not Google. You do not have Google's problems. Do not use Google's solutions."

Google needs 10,000 deploys a day because they have billions of users and thousands of microservices. You have 10 users and a monolith. Your problems are fundamentally different.

The Simple Alternative

For a pre-product-market-fit startup, here's all you need:

#!/bin/bash
# deploy.sh — The entire "CI/CD" pipeline for a startup

echo "Building the app..."
npm run build

echo "Deploying to server..."
rsync -avz --delete ./dist/ user@your-server.com:/var/www/app/

echo "Restarting the server..."
ssh user@your-server.com "pm2 restart app"

echo "Done. Deployed in under 2 minutes."

Total complexity: One file. 10 lines.

Total maintenance burden: Zero.

Total cost: $0 for CI/CD infrastructure.

Total time spent on "DevOps": Zero.

The Rule: If your deploy process can be written in 10 lines of bash, do not use a CI/CD platform. The moment you introduce a YAML file, you introduce a new configuration language to learn, a new platform to debug, and a new tax on your engineering time. None of that is worth it before you have product-market fit.

III. The "Small Batches = Low Risk" Myth

The DevOps Mantra

"Deploy small, frequent changes to reduce risk."

This is one of the most widely repeated claims in DevOps. It sounds logical: smaller changes are easier to test, easier to review, and easier to roll back. Therefore, small batches must be safer.

The Reality

Small batches do not eliminate risk. They shift risk from a single, obvious failure to a slow accumulation of complexity and emergent problems.

Emergent Behavior: The Blind Men and the Elephant

Imagine 10 developers each making one tiny, perfectly safe change to a car:

Developer 1 adjusts the fuel injector (small, safe).
Developer 2 tweaks the ignition timing (small, safe).
Developer 3 recalibrates the brake pressure (small, safe).
Developer 4 updates the tire pressure sensor (small, safe).

Each change is independently reviewed, tested, and deployed. No bugs. Low risk.

Now, you turn the car on.

The engine knocks, the brakes lock up, and the tire sensor reads "flat" because the fuel injector, ignition timing, and brake pressure all interact in ways none of the individual developers anticipated.

This is emergent behavior—problems that only arise when independent, "safe" parts are combined. You cannot test for this with unit tests or even integration tests that run in isolation. You only discover it in the integrated, running system.

Real-World Examples of Accumulated Failure

Database Schema Drift
10 small deployments add 10 new columns to the database. Each is fine alone. But performance degrades slowly over 2 months. One day, a query that used to take 200ms now takes 12 seconds. Production crashes. No single deploy caused it. The accumulation did.

Cascading Dependencies
Service A changes a response field (deploy 1). Service B starts using it (deploy 2). Service C relies on Service B's old response format (deploy 3). All three deploys were tested independently. None of them broke. But when all three are live, Service C crashes because it's reading data that no longer exists.

Hidden Technical Debt
Each small deploy is "good enough" and "low risk." But 200 small deploys later, the codebase is a spaghetti mess of hacky fixes. The "big unforeseen problem" is that the team now takes 3x longer to add any new feature because the code is unmaintainable.

What Actually Reduces Risk?

If small batches alone don't reduce risk, what does?

Observability, Not Just Testing. You cannot test for emergent behavior. You can only observe it. Elite teams invest heavily in monitoring, logging, and distributed tracing to detect emergent issues before they affect all users.
Canary Deployments. Roll out changes to 1% of users first. Watch metrics. If anything looks wrong, roll back instantly.
Feature Flags with Kill Switches. The ability to instantly turn off a feature, without rolling back the entire deployment.
Chaos Engineering. Proactively injecting failures into your system to see how it behaves under stress. This is how you discover that your 10 small deploys have made your system fragile.
Regular "Hardening" Sprints. Even elite teams don't just deploy constantly without reflection. They have periods where they step back, consolidate, refactor, and address the accumulated technical debt that the 200 small deploys created.

The Takeaway

Small batches are a tool, not a silver bullet. They allow you to move faster and recover quicker. But they do not eliminate the need for deep architectural thinking, robust observability, and a healthy respect for the fact that the whole is always greater and more dangerous than the sum of its parts.

IV. The User Trust Problem: Deployment ≠ Release

The Critical Distinction

This is one of the most misunderstood concepts in modern software delivery:

Deployment: Moving code to production servers. A behind-the-scenes technical event.
Release: Making a feature available to users. A user-facing event.

Many teams conflate these two concepts. They assume that because they can deploy, they should release to everyone.

The result: users see half-baked, buggy, or confusing features and lose trust.

The "Feature Fatigue" Trap

When a team adopts CI/CD and trunk-based development, they often fall into a dangerous mindset:

"We can deploy anytime, so we should deploy everything as soon as it's 'code complete.'"

This creates a user experience that feels:

Unstable: The UI keeps changing. Buttons move. Workflows shift. The user can't build muscle memory.
Unpolished: Features are released with rough edges, missing error states, or poor performance because the team rushed to "get it out."
Untrustworthy: Users learn to dread updates because they associate them with broken functionality.

The cost: Users don't complain. They just leave. And they don't come back.

How Elite Teams Protect User Trust

The best teams use a set of tools and strategies to ensure that users never see a feature until it is polished, tested, and ready.

1. Feature Flags (Progressive Rollout)

This is the primary tool. Here's how it protects user trust:

Deploy the feature to production, but keep it flagged "off" for everyone except internal testers.
Test it internally for days or weeks.
Turn the flag on for 1% of real users. Monitor metrics aggressively.
Slowly increase to 5%, then 20%, then 50%, then 100%.
At any sign of trouble, turn it off instantly.

Result: By the time a feature reaches 100% of users, it has been running in production for days or weeks, under real traffic, and has been proven stable. The user sees a polished, working feature.

2. Canary Releases

Gradually roll out changes to a small percentage of users to monitor for issues before a full launch.

3. Beta Programs

Explicitly label new features as "Beta" or "Early Access." Users who opt in know they might be rough. They are volunteers, not victims.

4. Dark Launches

Deploy code but keep it completely hidden from users. Use it to test performance and gather metrics without user impact.

The Cost of Showing Too Early

Scenario	User Perception	Business Impact
Feature shown early, buggy, then fixed in 2 days	"This app is unreliable. They keep breaking things."	10% of users churn. Support tickets spike. Brand damage.
Feature shown early, polished, and works perfectly	"Nice! This app keeps getting better."	User retention improves. Word-of-mouth referrals increase.

The difference between these outcomes is not the code. It is the release strategy.

The Takeaway

Every time you show a user something new, you are asking them to trust you. If that thing is broken, confusing, or unpolished, you are spending that trust. If it happens too often, you go bankrupt.

Deploy constantly. Release carefully. Never let a user see a feature until it is polished and proven.

V. The Dirty Secret: The Code Graveyard

Unreleased, Abandoned Code

Fast deploys + feature flags + aggressive experimentation = a massive amount of code that never reaches users.

How Unreleased Code Accumulates

Abandoned Features
A developer builds a new feature, wraps it in a flag, deploys it, and tests it with 1% of users. The data shows users don't like it. Product decides to pivot. The flag stays "off" forever. The code sits in production, unused, for years.

Perpetual Beta
A feature is built, flagged as "Beta," and released to 5% of users. Feedback is mixed. The product team keeps tweaking it. Months pass. The feature never reaches 100%. It lingers in a permanent "Beta" state until it's eventually deprecated.

A/B Test Losers
Two versions of a feature are built (Variant A and Variant B). Both are deployed behind separate flags. Variant A wins the A/B test. Variant B is turned off and abandoned. 50% of the development effort is completely wasted.

Strategic Pivots
The company changes strategic direction. An entire initiative—maybe 6 months of work—is no longer relevant. The code is deployed, flagged off, and eventually deleted. Hundreds of thousands of dollars of engineering time, vaporized.

The Numbers

This is not theoretical. The industry has studied this extensively.

Microsoft Research found that 40-60% of features built by software teams are never used or are used so little that they don't justify the development cost.
Standish Group's Chaos Report found that 45% of features in a typical software project are never used.
In the world of A/B testing and feature flags, it's widely accepted that 50-80% of new features fail to improve the key metric they were designed to improve.

For Companies Without Abundant Budgets, This Is Catastrophic

The industry tells you: "Fail fast! Experiment! Deploy everything!"

What they don't tell you:

Experiments cost money. Real money.
The data is often noisy, incomplete, or misleading.
A/B tests and feature flags are cheap to set up but expensive to build the code for.
Dead code accumulates silently and is rarely cleaned up.

For a well-funded unicorn, 40% waste is an acceptable cost of experimentation. For a startup or a mid-sized company, that 40% waste can be the difference between profitability and bankruptcy.

A Real-World Budget Scenario

Let's imagine a 10-person engineering team with an annual burn rate of $1.5 million.

Scenario	Approach	Outcome
Traditional (No CI/CD)	Build 10 features. Plan carefully. Release 8. 2 fail.	20% waste. $300,000 wasted.
Fast Deploy (Unthoughtful)	Build 10 features. Deploy them all with flags. 6 fail. 4 succeed.	60% waste. $900,000 wasted.

The difference: $600,000 down the drain. For a startup, that's runway. For a mid-sized company, that's a team laid off. For an enterprise, that's a budget overrun that kills other initiatives.

The True Cost of Dead Code

Cost Type	Description	Financial Impact
Development Cost	Salaries paid to build features that never ship.	Direct, measurable. $10,000–$100,000+ wasted per abandoned feature.
Maintenance Cost	Dead code still needs to be compiled, tested, and deployed. It slows down pipelines.	Indirect. Longer pipelines = developer wait-time = lower productivity.
Cognitive Load	Developers must navigate through dead code, abandoned flags, and unused codepaths.	Hard to measure but significant. 10-20% productivity drag.
Technical Debt	Dead code interacts with live code in unexpected ways. It creates bugs.	Emergency fixes, outages, support tickets. All cost money.
Opportunity Cost	Engineering time spent on abandoned features could have been spent on features that actually deliver value.	The biggest cost of all.

How to Avoid the Code Graveyard (On a Budget)

1. Validate Before You Build
Don't write a single line of code until you've validated the need with users. Use prototypes, mockups, and user interviews. Cost: A few hours. Risk: Zero.

2. Kill Features Early
The earlier you kill a feature, the less it costs.

Kill it on a whiteboard: $0 wasted.
Kill it in a prototype: $1,000 wasted.
Kill it in an MVP: $10,000 wasted.
Kill it after full development: $100,000+ wasted.

3. Set a "Feature Expiration Date"
Every feature flag gets a deadline. If the feature hasn't reached 100% rollout by that date, it gets deleted. This prevents features from languishing in "perpetual Beta" purgatory.

4. Track the "Experiment Cost" Metric
Measure how much engineering time is spent on features that are ultimately abandoned. If the ratio exceeds 20-30%, you're experimenting too aggressively. Adjust your planning process accordingly.

5. Dedicate Regular Cleanup Sprints
Every quarter, spend 1-2 days cleaning up dead code. Delete permanently-off feature flags. Remove unused code paths. Refactor where possible.

The Bottom Line

For companies without abundant budgets, unthoughtful fast deployment is a financial trap.

The industry tells you: "Deploy everything. The data will guide you."
The reality is: "Every abandoned feature is thousands of dollars of wasted engineering time."

The smart approach:

Experiment cheaply (prototypes, user interviews, mockups).
Build expensively (only after validation).
Clean up aggressively (regular code graveyard maintenance).

VI. The Contradiction at the Heart of CI/CD

The Developer's Dilemma: Time Is a Zero-Sum Game

Every developer has a fixed number of hours in a day. Every hour spent writing tests is an hour not spent building features. Every hour spent debugging a flaky pipeline is an hour not spent delivering value.

When you ask a developer to:

Write the code
Write the unit tests
Write the integration tests
Write the E2E tests
Maintain the test suite
Debug the pipeline when tests fail

...you are asking them to do two jobs in the time they used to do one.

The result: Developers cut corners. They write the minimum viable tests. They stop expanding prematurely. They do exactly what you observed—they "spare" operations because they are measured on shipping features, not on test coverage.

This is not laziness. This is a rational response to an impossible workload.

The Tester's Contradiction: Automation vs. Human Judgment

Now consider the other side. If you add human testers to catch what developers miss, you introduce a fundamental tension:

The CI/CD pipeline is automated. Human testers are manual. The two do not naturally fit together.

Here is the contradiction:

The Automation Ideal	The Reality with Human Testers
"Code is merged. Tests run automatically. Deployment happens instantly."	"Code is merged. Automated tests run. But wait—we need to wait for QA to do manual testing. The pipeline pauses."
"Every change is deployed within minutes."	"Every change is deployed within minutes—after QA gives the green light, which takes hours or days."
"No human intervention. Everything is scripted."	"Human intervention is required. The pipeline must stop and wait."

The moment you add human testers, the "continuous" in CI/CD breaks. The pipeline is no longer fully automated. It now has a manual gate. And manual gates are slow, inconsistent, and expensive.

The Awkward Compromise

Most teams try to resolve this tension with an awkward hybrid:

Automated tests run first. They catch the low-hanging fruit.
If they pass, the pipeline stops and waits. A human tester is notified.
The human tester does exploratory testing. They find edge cases.
If they approve, the deployment proceeds. If they find bugs, the pipeline fails and the developer gets notified.

This "works" in the sense that it functions. But it has severe drawbacks:

The pipeline is no longer continuous. It is now "continuous until a human says yes."
The human tester becomes a bottleneck. Every deployment must wait for them.
The team is stuck in the worst of both worlds. They have the complexity of CI/CD and the slowness of manual QA.

The Real Cost: Adding Testers Doesn't Fix the Problem—It Just Moves It

Configuration	Developer Testing	Human Testers	Pipeline Speed	Bug Rate	Cost
No CI/CD, Manual Testing	Developers write minimal tests	Human testers catch bugs	Very slow (deployments take days)	Low	Moderate (QA salaries)
CI/CD, No Testers	Developers write minimal tests	No one catches what developers miss	Very fast	High	Moderate (infrastructure)
CI/CD + Human Testers	Developers write minimal tests	Human testers catch bugs	Stops and waits	Moderate	High (infrastructure + QA salaries)

The worst of all worlds: CI/CD + Human Testers. You pay for the infrastructure, you pay for the testers, and you still have a slow release process.

The Takeaway

"CI/CD does not eliminate the need for testers. It just makes the trade-off between speed and quality more visible—and more painful."

The industry has created an impossible standard: "We will deploy 10 times a day, with zero bugs, and no testers." That standard is not realistic. It is aspirational marketing masquerading as engineering best practice.

VII. The Infinite Regress of Test Automation

The "Test Engineer" Solution

When I point out that developers are "spare-ers" and testers are "cover-ers," someone inevitably says:

"The solution is obvious. Hire test engineers. They write automated tests. The tests run in CI/CD. Problem solved."

It sounds logical. Test engineers are trained to think like cover-ers. They write test code. The test code runs automatically. The pipeline stays fast and automated. No manual gate. No human tester bottleneck. Perfect.

But this solution contains a fatal flaw.

The Role Inversion Problem

Test engineers write test programs. The test programs are now the "product." The original application—the thing the testers used to test—is now just the "system under test."

Here is the problem: Test engineers face exactly the same constraints as developers.

Role	Product	Constraint	Result
Developer	Application code	Limited hours, pressure to ship features	Spares operations. Writes minimal tests.
Test Engineer	Test code	Limited hours, pressure to ship test suites	Spares operations. Writes minimal test coverage.

The role is inverted, but the problem is identical.

The test engineer has a fixed number of hours.
They are measured on how much test coverage they produce.
They must prioritize which test cases to automate.
They face pressure to ship the test suite, just like developers face pressure to ship features.
They are human. They make mistakes. Their test code has bugs.

Test engineers are not cover-ers anymore. They are spare-ers who happen to write test code.

The Infinite Regress

Now consider the implications:

Test engineers write test programs. They think like cover-ers, but they operate under spare-er constraints. They cut corners. They miss edge cases. Their test suites are incomplete.
Who tests the test programs? Test programs are code. They can have bugs. They can be incomplete. They can fail to catch the edge cases they were designed to catch.
Option A: Developers test the test programs. But developers are spare-ers. They will write minimal tests for the test programs. The same problem recurs.
Option B: Test engineers test each other's test programs. But now you have an infinite loop—test engineers testing test engineers' tests. At some point, someone must use human judgment to decide that the tests are "good enough."
Option C: The test program tests itself. But that is a circular dependency. A test program cannot validate its own completeness without a separate mechanism.

The result: You have created a new category of code that must be maintained, debugged, and updated. You have added complexity, not eliminated it. You have not solved the "cover-er vs. spare-er" problem. You have just pushed it up one level.

The Cost of the Test Engineer Solution

Cost	Description	Business Impact
Development Cost	Test engineers must be hired. They write and maintain test code.	Adds headcount. Adds salary expenses.
Maintenance Cost	Test code must be updated when the application changes. The suite grows over time.	Test suites become brittle. Flaky tests appear. Pipeline slows down.
Debugging Cost	When a test fails, someone must debug it. Is it a real bug? A flaky test? A test bug?	Developer time wasted on false positives.
Coverage Cost	Test engineers inevitably prioritize. Some edge cases are never automated.	The coverage gap remains. The most obscure bugs still escape.
Opportunity Cost	Time spent writing and maintaining test code is time not spent building the product.	Slower feature delivery. Less time for innovation.

Why the Test Engineer Solution Fails in Practice

Let me illustrate with a concrete example.

Scenario: You have a web application. You hire a test engineer. They write automated E2E tests using Selenium.

What happens:

The test engineer writes test cases for the critical paths: login, search, checkout.
They run in CI/CD. The pipeline passes. Everyone celebrates.
Two months later, the UI changes. A button moves. The test breaks.
The test engineer spends hours fixing the test. They are frustrated. This is not fun work.
Over time, the test suite becomes brittle. Tests fail randomly. Developers ignore the failures.
The test engineer leaves. No one knows how to maintain the tests. The suite is abandoned.
The team is back where they started, but now they have a broken test suite and no budget to fix it.

The lesson: Test engineers are not a magic bullet. They face the same constraints as developers. They are subject to the same pressures. Their test code is just as flawed as any other code.

The "Testing the Test Program" Problem

Now consider an even more subtle issue.

The test engineer writes a test to verify that the application's login works correctly. The test passes. How do we know the test is correct?

Maybe the test only checks that the HTTP status is 200, but the login actually failed silently.
Maybe the test checks the wrong element, so it passes even when the UI is broken.
Maybe the test passes locally but fails in CI due to environment differences.

How do you test the test program? The answer is: You don't. You rely on human judgment. You look at the test and say, "That seems right."

This is the "cover-er" mindset again. But the test engineer—who is now a spare-er under pressure—will naturally stop expanding prematurely. They will assume the test is correct. They will not probe for edge cases in their own test code.

The problem is not a lack of technical skill. The problem is the economic and cognitive constraints that apply to anyone who writes code, regardless of their job title.

The Real Solution: Testers, Not Test Engineers

The solution is not to replace testers with test engineers. The solution is to keep testers as cover-ers, use automation to amplify them, and accept that some manual testing will always be necessary.

Role	Job Description	Tools
Tester (Cover-er)	Find bugs through exploratory testing, edge case discovery, and user-centric testing.	Manual testing, exploratory testing, usability testing.
Automation Engineer	Write automation to handle repetitive tasks, freeing up testers.	CI/CD pipelines, automated regression tests, performance tests.
Developer (Spare-er)	Build the system, write unit tests, and maintain the codebase.	Unit tests, integration tests, code reviews.

The key: The tester stays a cover-er. They are not required to write code. They are required to think like a user, probe the system, and find bugs.

The automation engineer writes the code that handles the boring stuff. But they do not replace the tester. They complement them.

The Takeaway

"If you make testers write code, you turn cover-ers into spare-ers. You haven't solved the problem. You've just moved it."

The infinite regress is real. At some point, you must rely on human judgment. There is no way to automate judgment itself.

VIII. The Real Solution: Milestones + Small Batches

Why Milestones Alone Fail

The traditional milestone-only approach has well-known problems:

Too slow. Too risky. No feedback until the end.
Assumes you know everything upfront. You don't.
Integration hell at the end. Crunch time. Burnout.
You only discover that users hate the feature after 3 months of work.

Why Small Batches Alone Fail

As we've discussed, the small-batch-only approach also has critical flaws:

No strategic direction. You're moving fast, but possibly in circles.
Emergent complexity accumulates. Technical debt builds.
No time for big refactors because you're always shipping.
The codebase rots slowly. Eventually, velocity grinds to a halt.

The Winning Combination

The most effective teams use a dual-track approach:

Milestones define the "What" and "Why" (Strategy): Quarterly outcomes. Business goals. User-facing value.
Small batches define the "How" and "When" (Execution): Small, deployable units that move toward the milestone.

The Process

Step 1: Talk to Users Before You Code
Validate the need. Understand the problem. Don't write a single line of code until you've confirmed that someone actually wants what you're building.

Step 2: Prototype Before You Build
Show users low-fidelity mockups. Iterate on design. Cost: A few days. Risk: Zero.

Step 3: Build a "Concierge" or "Wizard of Oz" MVP
Don't build the full system. Do the work manually behind the scenes. If users love it, then build the automation.

Step 4: Build Small and Release Carefully
Use feature flags, canaries, and beta programs. Never let a user see a feature until it is polished and proven.

Step 5: Consolidate Regularly
Hardening sprints to refactor, improve tests, and address technical debt. This is not optional. It is essential to prevent the accumulation of technical debt and dead code.

The Cost-Savings Math

Practice	Cost	Benefit
User Validation (Pre-Development)	A few hours of interviews	Saves 40-60% of development budget
Prototyping (Pre-Development)	A few days of design work	Saves months of rework
MVP (Small Build)	2-3 weeks of development	Tests the market cheaply
Feature Flag Cleanup (Quarterly)	1-2 days of engineering time	Prevents long-term productivity drag
Hardening Sprints (Quarterly)	1-2 weeks of engineering time	Prevents technical debt accumulation

Total cost of disciplined approach: ~10-15% of engineering budget.

Total cost of unthoughtful fast deployment: 40-60% of engineering budget wasted on dead code.

Savings: 25-45% of your engineering budget.

A Real-World Example

Timeframe	Milestone (Strategic)	Small Batches (Execution)
Month 1	"Support Apple Pay."	Deploy 1: Add SDK. Deploy 2: Update API. Deploy 3: Add UI. Deploy 4: Enable flag for internal testers. Deploy 5: Fix bug. Deploy 6: Canary to 5% of users. Deploy 7: Roll out to 100%.
Month 2	"Improve checkout speed by 30%."	Deploy 8: Optimize database query. Deploy 9: Add caching layer. Deploy 10: Compress assets. Deploy 11: Profile and tune.
Month 3	Hardening Sprint	No new features. Only refactoring, test improvements, documentation, and dependency updates.

IX. The Startup Playbook

The Pre-Product-Market-Fit Phase

Before you have product-market fit, your only job is to find something people want. Everything else is a distraction.

Phase	Recommended Approach	Why
Pre-Product-Market-Fit	A simple deploy script (rsync, scp, or a basic PaaS like Heroku/Railway). Minimal CI (just linting and unit tests).	You need to move fast and change direction quickly. Complex infrastructure locks you into decisions you haven't validated yet.
Post-Product-Market-Fit	Gradual introduction of CI/CD. Start small (build, test, deploy). Add complexity only when the pain of manual processes exceeds the cost of automation.	You now have paying customers. Stability matters. But still, be conservative.
Scaling	Full CI/CD pipeline with feature flags, canary releases, and multiple environments.	You have the revenue and team to support it.

When to Actually Introduce CI/CD

Not when you think you need it. Not when a blog post tells you to.

Only when all of these conditions are met:

You have product-market fit. Users are paying, retention is good, growth is happening.
Manual deployments are painful. You're deploying multiple times a day, or the process is error-prone.
You have a dedicated engineer or team. Someone owns the pipeline and maintains it.
You have budget to spare. The cost of the platform + the maintenance time is negligible compared to revenue.

Until then, rsync and scp are your best friends.

X. Conclusion: The Pragmatic Path

The Key Takeaways

1. CI/CD is not a universal cost-saver.
It only pays off at scale—high deployment frequency, large teams, complex infrastructure. For a team deploying once a week, it's a net loss.

2. Small batches do not equal low risk.
They shift risk to emergent complexity and accumulated technical debt. The whole is always greater and more dangerous than the sum of its parts.

3. Deployment ≠ Release.
Deploy constantly. Release carefully. Protect user trust above all else. Never let a user see a feature until it is polished and proven.

4. The code graveyard is a luxury only the rich can afford.
For companies without abundant budgets, unthoughtful fast deployment is a financial trap. 40-60% waste is catastrophic, not strategic.

5. CI/CD creates a fundamental contradiction.
Automation and human judgment do not naturally fit together. Adding testers breaks the "continuous" in CI/CD. Removing testers allows bugs to reach users.

6. Test engineers are not the solution.
If you make testers write code, you turn cover-ers into spare-ers. The infinite regress is real. Test code has bugs. Test code requires maintenance. Test engineers face the same constraints as developers.

7. The real solution is milestones + small batches + human testers.
Strategy guides execution. Execution enables strategy. And human judgment—the "cover-er" mindset—cannot be automated away.

8. For startups, simple is better.
A 10-line bash script beats a 200-line YAML pipeline every time. Don't adopt enterprise tooling before you have product-market fit.

The Final Verdict

CI/CD is a tool, not a religion. It is not your savior. It will not fix poor planning, insufficient customer communication, or a lack of product discipline.

More importantly, CI/CD does not eliminate the need for human testers. It does not eliminate the need for the "cover-er" mindset. It does not automate judgment, curiosity, or the ability to probe for edge cases.

If you have no testers, you will ship more bugs. Developers are spare-ers. They will always prioritize features over tests.
If you force testers to write code, you turn them into spare-ers. The infinite regress is real. Test code has bugs. Test code requires maintenance. Test engineers face the same constraints as developers.
If you keep testers as cover-ers and use automation to amplify them, you get the best of both worlds. But it costs money and requires a culture shift.

The pragmatic path:

Start simple. Use a deploy script or a basic PaaS.
Validate your product. Talk to users. Find product-market fit.
Keep human testers as cover-ers. Their judgment is irreplaceable.
Use automation for the repetitive, predictable tasks.
Add CI/CD incrementally. Only when the pain of manual work exceeds the cost of automation.
Accept that perfection is impossible. Some bugs will ship. Monitor, rollback, and learn.

Think before you deploy. Plan before you code. Listen before you build. And never forget that judgment cannot be automated.

XI. Call to Action

What has your experience been with CI/CD? Has it saved you money or cost you more than you expected? Have you fallen into the code graveyard trap? Have you struggled with the tester contradiction?

Share your story in the comments. Let's learn from each other's mistakes and successes.

If you found this post valuable, subscribe for more pragmatic takes on software development.

This post was inspired by a long and thoughtful discussion with a skeptical engineer who asked all the right questions. Thank you for challenging the orthodoxy.