DEV Community: MeteorOps

Datadog Pricing: What You Pay and How to Cut the Bill

Michael Zion — Sat, 14 Nov 2026 21:30:02 +0000

On a Datadog bill for 50 hosts, the $15 per host is 17% of the total:

Bottom line

Datadog pricing is a price list of separate products, and each product has its own unit.

Infrastructure Monitoring costs $15 per host per month on the Pro plan (billed annually), or $18 on-demand.

APM adds $31 per host.

Logs cost $0.10 per GB to ingest plus $1.70 per million events to index for 15 days.

Custom metrics cost $5 per 100 above what your hosts include.

So the host price tells you very little. The bill follows 4 things your team controls:

How many hosts and containers run at the busiest hours of the month
How many tag combinations your custom metrics have
How many log events you index, and for how many days
How much of it you committed to in advance

In the worked example below, 50 hosts cost $4,360 a month at list price, and the $15 host line is 17% of it.

3 changes bring that bill to $2,953.

All prices here are Datadog's public list prices for the US site, in USD, checked on November 14, 2026 against the Datadog price list and the billing docs. Your contract can differ.

Datadog pricing is a price list, one unit per product

The Datadog price list has a row for every product, and you pay for each one you turn on.

These are the lines a typical startup setup uses:

Product	Unit	Billed annually	Month-to-month	On-demand
Infrastructure Pro	Host, per month	$15	$18	$18
Infrastructure Enterprise	Host, per month	$23	$27	$27
APM	APM host, per month	$31	$36	$36
APM Pro	APM host, per month	$35	$42	$42
APM Enterprise	APM host, per month	$40	$48	$48
Database Monitoring	Database host, per month	$70	$84	$84
Containers above the included ones	Container	$1 per month	$1 per month	$0.002 per hour
Custom metrics above the included ones	100 metrics, per month	$5	$5	$5
Log ingestion	GB	$0.10	$0.10	$0.10
Indexed logs, 15-day retention	1 million events	$1.70	$2.04	$2.55
Indexed spans, 15-day retention	1 million spans	$1.70	$2.04	$2.55
Synthetic API tests	10,000 test runs	$5	$6	$7.20
Synthetic browser tests	1,000 test runs	$12	$15	$18
RUM Measure	1,000 sessions	$0.15	$0.18	$0.22

Two things in this table decide most bills.

First: a host with Infrastructure Pro and APM costs $46 a month, 3 times the number on the pricing page.

Second: the last 8 rows are usage.

Nobody signs off on them. They grow with every deploy.

Host = a predictable floor. Usage = the part that surprises you.

A host is billed on the busiest hours of the month

Datadog counts your hosts every hour.

On the default plan (Datadog calls it the high watermark plan) you pay for the highest count of the month after the top 1% of hours is dropped.

A 30 day month has 720 hourly readings. The top 1% is 7.2 of them, and Datadog rounds that up to 8.

The APM billing page spells the rule out for APM hosts: Datadog charges on the ninth highest reading of the month (the eighth in February). Infrastructure hosts follow the same 99% rule.

Here is what that does to an autoscaling cluster:

You run 40 nodes all month
A load test keeps 100 nodes up during 8 hourly readings → you pay for 40 hosts
It shows up in a ninth reading → you pay for 100 hosts, for the whole month

Rule: 9 high readings set the price of the other 711.

The 8 readings are a budget for the whole month, and a test that starts mid-hour uses an extra one.

Datadog drops the top 1% of the 720 hourly readings. A ninth high reading sets the host count for the month.

What counts as a host: a server, a VM, a Kubernetes node, an Azure App Service Plan instance or a Heroku dyno. A VM seen by both the Agent and the cloud integration counts once.

If your fleet scales up and down every day, ask for the hybrid monthly/hourly plan. You commit to a monthly minimum and pay an hourly rate for host hours above it.

The same hourly counting applies to APM hosts and database hosts. (Yes, one long load test can raise the host line, the APM line and the container line.)

If you autoscale on Kubernetes, the settings that decide how long extra nodes stay up are in my guide on Kubernetes autoscaling without cost surprises.

Containers and custom metrics come with the host, up to a limit

Each Infrastructure host includes an allotment of other products:

Included with each host	Pro	Enterprise	Above the allotment
Containers	5	10	$1 per container per month prepaid, $0.002 per hour on-demand
Custom metrics	100	200	$5 per 100 per month at list price

The allotments are pooled across the account. 50 Pro hosts include 250 containers and 5,000 custom metrics, wherever they run.

A Kubernetes node that runs 20 containers uses 4 hosts' worth of the container allotment.

The on-demand container rate looks small: $0.002 an hour is $1.46 over a 730 hour month, 46% more than the prepaid $1.

Kubernetes pause containers are excluded from the count.

One mistake to avoid: installing the Agent inside every container. Datadog then bills each container as a host.

Run 1 containerized Agent per node.

Custom metrics multiply with every tag

This is the line that is hardest to predict.

A custom metric is any metric that does not come from a Datadog integration: everything you send through DogStatsD, a custom check or a log-based metric.

Datadog counts tag combinations, and 1 metric name can hold thousands of them:

1 custom metric = 1 metric name + 1 unique combination of tag values (the host tag included).

Take a metric named checkout.latency with 3 tags:

endpoint: 20 values
status: 3 values
pod: 50 values

If every combination reports, that is 20 x 3 x 50 = 3,000 custom metrics.

The metric type multiplies it again:

Metric type	Custom metrics per tag combination	checkout.latency	Cost above the allotment
Count, rate or gauge	1	3,000	$150 a month
Histogram (default aggregations)	5	15,000	$750 a month
Distribution	5	15,000	$750 a month
Distribution with percentiles	10	30,000	$1,500 a month

One metric name, one line of code, $1,500 a month.

1 metric name with 3 tags becomes 3,000 custom metrics, and 30,000 as a distribution with percentiles.

Now add a customer_id tag with 200 values and multiply the table by up to 200.

Two details work in your favor:

Only combinations that actually report are counted, and the count follows your most detailed tag
Billing uses the monthly average of the hourly count, so a 1 hour burst of new tag values costs 1/720 of a full month

How do you get the tags back under control without changing code? Metrics without Limits.

It lets you pick, per metric, which tags stay queryable. Datadog then bills two volumes:

Indexed custom metrics: the combinations of the tags you kept, at the $5 per 100 list price
Ingested custom metrics: everything your code still sends for the metrics you configured, at $0.10 per 100 above the allotment

Keeping only endpoint and status on checkout.latency takes it from 3,000 indexed combinations to 60.

Logs are billed twice: once to ingest, once to index

Every log you send pays $0.10 per GB at ingestion.

Every log you keep searchable pays again, per million events, by retention:

Index retention	Billed annually	Month-to-month	On-demand	300 million events a month (annual)
3 days	$1.06	$1.27	$1.59	$318
7 days	$1.27	$1.52	$1.91	$381
15 days	$1.70	$2.04	$2.55	$510
30 days	$2.50	$3.00	$3.75	$750

Ingestion is cheap. Indexing is the bill.

500 GB of logs costs $50 to ingest. The same logs, as 300 million events indexed for 15 days, cost $510.

That split is also the way out: you can ingest everything and index a part of it.

An exclusion filter on an index drops matching logs before they are indexed. The excluded logs still show in Live Tail, can still feed log-based metrics and can still go to your archive.

Every log pays for ingestion. Only the logs that pass the exclusion filters pay for indexing.

The two filters I would add first:

status:DEBUG                                        exclude 100%
source:nginx AND http.status_code:[200 TO 299]      exclude 95%

The first one can be switched off from the UI or the API during an incident, when you want the debug logs back.

The second keeps every 3xx, 4xx and 5xx request and 5% of the successful ones, which is enough to see the trend.

Two more log facts for your budget:

Indexed logs above your commitment are billed on-demand, at a 50% premium ($2.55 against $1.70)
Each index can have a daily quota. When a service starts logging at debug level in production, indexing stops at the quota. The logs still reach Live Tail and your archive, and you still pay the $0.10 per GB

For logs you must keep for months but rarely read, Flex Logs stores them at $0.05 per million events per month, with query capacity bought separately.

APM adds a second host price, with spans included

An APM host is any host that sends traces. It costs $31 a month on top of the Infrastructure price.

Each APM host includes:

150 GB of ingested spans a month
1 million indexed spans a month, kept for 15 days

Above that: $0.10 per ingested GB, and $1.70 per million indexed spans at 15-day retention.

Both allotments are pooled across your APM hosts.

50 APM hosts include 7,500 GB of ingested spans and 50 million indexed spans a month. The host count is the easier place to overpay: tracing left on for hosts that run batch jobs and internal tools nobody traces.

Only hosts that send traces are APM hosts. A node pool for untraced workloads is $31 a month saved per node.

If you do pass the allotment, set sampling per service in the tracing library:

export DD_TRACE_SAMPLING_RULES='[
  {"service": "checkout", "resource": "GET /checkout", "sample_rate": 1},
  {"service": "checkout", "sample_rate": 0.2}
]'

This keeps every trace that starts at GET /checkout and 20% of the other traces that start in the service.

Resource rules need a recent Java, Python, Go or PHP tracer. Other languages take the per-service rule only.

Retention filters then decide which ingested spans are indexed.

The Intelligent Retention Filter keeps a sample of errors and slow traces at no charge, so start by trimming your custom filters and the rate of the Error Default one.

A worked Datadog bill for 50 hosts

The setup, with my assumptions stated:

50 Kubernetes nodes, Infrastructure Pro, billed annually
APM on all 50 nodes, spans inside the allotment
1,000 containers on average (20 per node)
20,000 custom metrics
500 GB of logs a month, 300 million events, all indexed for 15 days

Line	Calculation	Per month
Infrastructure Pro	50 x $15	$750
APM	50 x $31	$1,550
Containers	(1,000 - 250 included) x $1	$750
Custom metrics	(20,000 - 5,000 included) / 100 x $5	$750
Log ingestion	500 GB x $0.10	$50
Indexed logs	300 x $1.70	$510
Total		$4,360

$4,360 a month is $52,320 a year.

The line everyone quotes ($15 per host) is $750 of it: 17%.

The 50 host bill at list price. The $15 per host line is 17% of it.

The same usage with no commitment, at on-demand rates, comes to $5,360 a month. That is 23% more for the same usage.

Now the same company after 3 changes:

Change	Before	After
Tracing only on the 35 nodes that run traced services	$1,550	$1,085
Metrics without Limits: 20,000 indexed custom metrics down to 8,000	$750	$150 + up to $15 for ingested
Exclusion filters: 30% of log events indexed	$510	$153
Everything else unchanged	$1,550	$1,550
Total	$4,360	$2,953

$1,407 a month less: 32%.

These are list prices. Datadog lists volume discounts from 500 hosts a month, and a sales conversation changes the numbers long before that.

Commit to your baseline, pay on-demand for the rest

Datadog has 3 price columns: billed annually, month-to-month and on-demand.

Commitment = a quantity per product that you pay for whether you use it or not.

On-demand = everything above it, at the highest price.

The billing docs give the formula:

allotments + committed usage = included usage. Billable usage minus included usage = on-demand usage.

The premium depends on the product:

Hosts: $18 against $15, 20% more
Indexed logs and spans: $2.55 against $1.70, 50% more
Containers: $1.46 against $1 over a full month, 46% more

So my rule is simple: commit to what you used in your quietest month, and let growth run on-demand until the next renewal.

Commit too low and you pay up to 50% extra on the difference (16% on APM hosts, 20% on hosts, 50% on indexed logs).

Commit too high and you pay 100% for something you never used.

Cut the bill in this order

Start where the money is. In the example above that is APM hosts, custom metrics and indexed logs.

Read the bill by product: Plan and Usage shows month-to-date and projected cost per product, and the top custom metric names by share of cost
Exclude logs you never search: debug logs, health checks and successful requests first
Shorten retention: 300 million events at 7 days cost $381 against $510 at 15 days. Send the rest to an archive in your own bucket
Set a daily quota on every index: a bad deploy then indexes 1 day's quota at most. Logs past the quota are not searchable, so add the warning threshold
Drop tags with Metrics without Limits: start with the 10 metric names at the top of the usage page. Datadog marks the tags queried in the last 30 days
Stop percentiles you do not graph: they double the count of a distribution metric
Turn tracing off where nobody reads traces: every host that sends traces is an APM host
Keep all spikes of the month inside 8 hourly readings, or move to the hourly plan: the high-water mark decides the month
Alert on your own usage: Datadog publishes it as metrics

The usage metrics for the last point:

datadog.estimated_usage.hosts
datadog.estimated_usage.containers
datadog.estimated_usage.metrics.custom
datadog.estimated_usage.logs.ingested_bytes
datadog.estimated_usage.logs.ingested_events
datadog.estimated_usage.apm_hosts
datadog.estimated_usage.apm.ingested_bytes
datadog.estimated_usage.apm.indexed_spans

Put an anomaly monitor on the log and custom metric ones. You then hear about a cost spike the same day it starts.

Filter logs.ingested_events by datadog_is_excluded:false to watch what gets indexed.

The same approach works on the cloud bill itself. I wrote it down in how to cut cloud costs without slowing teams.

Pay for Datadog when it saves an engineer

Yes, I help companies lower this bill. My first advice is to cut it before you leave it.

The alternative to Datadog is usually Prometheus, Grafana, Loki and an OpenTelemetry Collector that somebody on your team runs.

The license is free. The engineer is the cost.

My position:

Up to about 20 engineers: buy it. $4,000 a month is cheaper than the part of an engineer that a self-hosted stack takes
When the bill passes the cost of an engineer: cut it with the list above before you think about leaving
When it is still too high after that: move the most expensive signal first (often logs) and keep the rest

The exceptions: if you already run Prometheus well, or you are bound to keep data in your own account, start self-hosted.

Instrument with OpenTelemetry either way. Datadog accepts it, and it keeps the exit open.

Check how your OTLP metrics are billed first: metrics outside a Datadog integration count as custom metrics.

I compare the two paths in how to build a startup observability stack.

The Datadog cost checklist

Always:

1 Agent per node
A commitment equal to your quietest month
Exclusion filters on debug logs and successful requests
A daily quota on every log index
The shortest index retention your on-call needs, and an archive for the rest
No user, customer or request ID as a metric tag
Percentiles only on metrics that have a graph or a monitor
Tracing only on hosts where somebody reads traces
Monitors on the datadog.estimated_usage metrics
A look at the projected cost on the 10th of every month

FAQ

How does Datadog pricing work?

Each Datadog product has its own unit and price. Infrastructure Monitoring and APM are priced per host per month, logs per GB ingested and per million events indexed, custom metrics per 100, and Synthetics and RUM per test run and per session. You commit to a quantity per product and pay a higher on-demand rate above it.

How much does Datadog cost per host?

Infrastructure Pro is $15 per host per month billed annually and $18 on-demand. Enterprise is $23 and $27. With APM at $31, a traced host costs $46 a month before containers, custom metrics and logs.

Why is Datadog so expensive?

The host price is a small part of the bill. Custom metrics multiply with every tag, logs are charged at ingestion and again at indexing, and hosts are billed on the busiest hours of the month. In the 50 host example above, the host line is 17% of $4,360.

Is Datadog free?

Partly. Datadog has a free trial, and when it ends you can move to the Free tier: up to 5 Basic hosts with 1-day metric retention, at no cost. Container monitoring is listed from the Pro plan, at $15 per host.

How are Datadog custom metrics billed?

By the number of unique combinations of metric name and tag values, averaged per hour over the month. Pro includes 100 per host and Enterprise 200. Above that the list price is $5 per 100 a month, and a histogram or distribution counts 5 times per combination.

How much do Datadog logs cost?

$0.10 per GB ingested, plus $1.06 to $2.50 per million indexed events depending on retention (3 to 30 days, billed annually). Indexed logs above your commitment cost 50% more. Exclusion filters lower the indexing part, which is the larger one.

Does Datadog charge per user?

Infrastructure Monitoring, APM and Log Management have no per-user fee. The price list has per-seat prices for On-Call and Incident Management, and per-committer prices for CI and code security products.

What is the high-water mark in Datadog billing?

Datadog counts hosts every hour, drops the top 1% of hours and bills the highest remaining count. In a 30 day month that is the ninth highest hourly reading. Scale-ups that touch 8 hourly readings or fewer in the month do not raise the host count. A ninth sets it for the month.

How do I reduce my Datadog bill?

Exclude logs you never search, shorten index retention, drop unused metric tags with Metrics without Limits and turn tracing off on hosts nobody traces. Then commit to your baseline usage. In the example above these steps cut the bill by 32%.

A recap

If you take away 1 thing from this, let it be this:

Your tags, your log filters and your busiest 9 hours set the Datadog bill.

You don't NEED a consultant to fix it. The checklist above is the whole method, and a senior engineer can run it.

If you'd rather have someone who has done it before look at your setup, book a free production setup consultation.

P.S. Before you delete a metric or a log, check that no SLO depends on it. I covered those in SLO vs SLA vs SLI.

SLO vs SLA vs SLI: Set Targets Before You Sign a Contract

Michael Zion — Fri, 13 Nov 2026 15:30:04 +0000

Enterprise contracts ask for 1 number, your uptime SLA. Signing 99.9% takes 1 minute. Measuring it takes 1 quarter:

Bottom line

SLI (Service Level Indicator) = the measurement. The share of requests that went well.

SLO (Service Level Objective) = the target your team sets on that measurement.

SLA (Service Level Agreement) = the promise you sign with a customer, with a price for missing it.

They are built in that order: measure, target, promise.

	SLI	SLO	SLA
What it is	A measurement	An internal target	A contract clause
Who owns it	Engineering	Engineering and product	Sales, legal and the CTO
Who sees it	The team	The company	The customer
When it is missed	Nothing, it is a fact	The team shifts to reliability work	You pay service credits
Example	99.97% of API requests succeeded in the last 30 days	99.95% of API requests succeed over 30 days	99.9% monthly uptime, or 10% of the monthly fee back

The rule of this article: measured > SLO > SLA.

Your SLA is looser than your SLO, and your SLO is looser than what you measured. Each gap is your margin.

The targets I set for a B2B SaaS product API: an SLO of 99.95% over a rolling 30 days and an SLA of 99.9% per calendar month, after 1 quarter of measuring.

Below: how to define the SLI, how to pick the SLO, what an error budget buys you, what goes into the SLA clause, and the alerts that tell you before the customer does.

SLI, SLO and SLA are 3 layers of the same number

Start from 1 user request.

request → good or bad →

SLI: the share of good ones →

SLO: the share you aim for →

error budget: the bad ones you can afford →

SLA: the share you promise, with margin →

credits: what a miss costs you

The same number in 3 layers: the SLI you measured, the SLO you aim for, the SLA you promise. Each gap is margin.

Google's SRE book gives a 1 question test for telling an SLO from an SLA: "what happens if the SLOs aren't met?"

If there is no explicit consequence, it is an SLO.

If a contract names a consequence, usually money back, it is an SLA.

So an SLA always contains an SLO. The reverse is optional, and for most of your services it should stay that way.

SLO = a decision tool for you.

SLA = a commitment to someone else.

The same book defines an SLI as "a carefully defined quantitative measure of some aspect of the level of service". Carefully defined is the part that takes work, so it comes first.

An SLI is good events divided by all events

SLI = good events / total events.

That is the form Google's SRE workbook recommends, and it works for every kind of service:

Service type	SLI	Good event
API or web app	Availability	A request answered without a server error
API or web app	Latency	A request answered in under 300 ms
Data pipeline	Freshness	A record processed within 10 minutes
Data pipeline	Correctness	A record with the right output
Storage	Durability	An object that can still be read

3 choices decide whether the number means anything:

Measure where the user is. The workbook uses load balancer metrics because they are "closer to the user's experience" than the application server's logs. A crashed pod reports no errors, and the load balancer in front of it does
Measure a real user flow. Login, the core API call, checkout. A health check on /health can be green while checkout fails
Use a threshold for latency. "Requests under 300 ms" is a ratio you can target. An average hides the slow 1% that your biggest customer lives in

An availability SLI in Prometheus, from the request counter most HTTP frameworks already export:

sum(rate(http_requests_total{job="api",code!~"5.."}[30d]))
/
sum(rate(http_requests_total{job="api"}[30d]))

The query needs 30 days of metrics, and Prometheus keeps 15 days by default, so raise the retention first. If your ingress or load balancer exports its own request counter, count from that one.

If you have no metrics yet, the load balancer's access logs give you the same 2 counts. The startup observability stack article covers the tooling.

Rule: 2 SLIs per user-facing service, availability and latency. Add a third when a customer complains about something the first 2 missed.

An SLO is a target, a window, and an error budget

SLO = SLI target + time window.

"99.9% of API requests succeed, over a rolling 30 days."

The workbook's general-purpose window is 4 weeks, rolling. I use 30 days because the cloud SLAs you depend on count in months.

The useful half of an SLO is what it leaves over.

Error budget = 1 minus the SLO.

At 99.9% you may fail 0.1% of requests. With 10 million requests a month, that is 10,000 failed requests you are allowed.

In time, over 30 days:

SLO	Error budget	Full downtime it allows in 30 days
99%	1%	7 hours 12 minutes
99.5%	0.5%	3 hours 36 minutes
99.9%	0.1%	43 minutes
99.95%	0.05%	21 minutes 36 seconds
99.99%	0.01%	4 minutes 19 seconds
99.999%	0.001%	26 seconds

Each extra 9 cuts the error budget by 10: from 7 hours 12 minutes at 99% to 4 minutes 19 seconds at 99.99%.

A request-based SLO spends the same budget in failed requests, so 10 minutes down at peak costs more than 10 minutes down at 3 AM.

Read the table as a staffing plan.

43 minutes covers 1 incident where a person is paged, wakes up, finds the cause and rolls back.

4 minutes covers none. At 99.99% every recovery has to be automatic, because the budget is gone before a human opens the laptop.

5 nines leaves 26 seconds a month. (Enough time to read the alert.)

How to pick the number

Google's SRE book lists 5 rules for choosing targets:

"Don't pick a target based on current performance"
"Keep it simple"
"Avoid absolutes"
"Have as few SLOs as possible"
"Perfection can wait"

The workbook is blunter about the absolute: "100% reliability is the wrong target". A 100% target leaves no room for change, and change is the leading cause of outages.

How I apply the 5 rules at a startup:

Measure the SLI for 4 weeks before choosing anything
Ask what the user needs. An internal admin tool at 99% is fine. A payments API at 99% loses customers
Start the product API at 99.9% while no SLA exists, and move to 99.95% before you sign a 99.9% SLA
Promise 99.99% only when failover is automatic and a drill has proven it

The opposite mistake is real too. Google's SREs found that a service running far above its SLO teaches users to depend on the extra.

Their fix for Chubby, an internal lock service: in a quarter with no real failure, they take the system down on purpose.

At a startup the cheaper version is to spend the budget: ship faster, run the risky migration, do the cluster upgrade in daylight.

An unspent error budget = features you didn't ship.

Check your SLA against the SLAs you run on

Your service sits on top of other people's promises. Dependencies in a row multiply.

The AWS numbers, from its SLA pages:

AWS service	Setup	Monthly uptime in the SLA
EC2	Instances in 2 or more availability zones	99.99%
EC2	A single instance	99.5%
RDS	Multi-AZ	99.95%
RDS	Single instance	99.5%
Elastic Load Balancing	Across 2 or more availability zones	99.99%

A request that needs the load balancer, the compute and the database:

99.99% × 99.99% × 99.95% = 99.93%

Dependencies in a row multiply: 3 AWS SLAs of 99.99%, 99.99% and 99.95% leave 99.93%.

That is what the cloud commits to before your own code fails once: the level AWS pays credits below.

An SLA of 99.95% on top of it is a promise your suppliers never made to you.

Run the database on a single instance and the chain drops to about 99.48%. Signing 99.9% on that setup is a bet.

The provider's SLA is the floor it pays for missing. Real availability is usually higher, and your measurement is how you find out by how much.

Redundancy is the lever that raises the ceiling, and it has a price. The RTO vs RPO article lays out what each level costs.

An SLA is an SLO with a price

An SLA clause has 6 parts. Check each one before you sign, on either side of the table:

The metric: what counts as "available", and who measures it
The window: usually a calendar month
The target: the percentage
The exclusions: scheduled maintenance, the customer's own outages, beta features, force majeure
The remedy: a service credit schedule
The claim process: who has to notice, and by when

AWS's EC2 SLA is a good model to read, because it is short:

Monthly uptime of the region	Service credit
Below 99.99%, down to 99.0%	10%
Below 99.0%, down to 95.0%	30%
Below 95.0%	100%

2 details in it are worth copying.

The remedy is a credit against future bills, 100% of that month's EC2 charge at most.

The customer has to claim it: a support case, received by the end of the second billing cycle after the incident. A credit nobody claims is never paid.

Which also means: if your cloud bill is large and your region had a bad month, someone on your team should be opening that case.

A clause to hand your lawyer

Provider will make the Service available with a Monthly Uptime Percentage of at least 99.9%. Monthly Uptime Percentage is the share of minutes in a calendar month in which the production API, measured at Provider's load balancer, answers at least 99% of requests with a response other than a 5xx error. Minutes in which the Service is unreachable count as unavailable. Scheduled maintenance announced 72 hours ahead, failures caused by Customer's systems, beta features and force majeure events are excluded. If the Monthly Uptime Percentage falls below 99.9%, Customer may request a service credit of 10% of that month's fee, or 25% if it falls below 99.0%. Requests must be made within 30 days of the end of the month. Service credits are Customer's sole and exclusive remedy for a failure to meet this commitment.

It is a draft for your lawyer, with the engineering parts already in: a metric defined per minute, a named measuring point, a window, the exclusions and a remedy with a ceiling.

Track it on its own dashboard. The SLA counts calendar months and the SLO counts a rolling 30 days, so the 2 numbers differ on most days.

The 3 rules I hold when sales brings a contract:

SLA = SLO minus margin. Google Cloud's SRE fundamentals post gives an SLA of 99.9% backed by an internal SLO of 99.95%. You can spend the whole SLO budget and still have half the SLA's budget left
Promise what a quarter of measurement has shown. A number copied from a competitor's pricing page is a guess
Price the extra 9. A customer who needs 99.99% is asking for a second availability zone everywhere, automatic failover and 24/7 on-call. That is a different tier with a different price

The SLA glossary entry has the shorter definition if you need to send one to a colleague.

Alert on how fast the error budget burns

An SLO gives you the 1 alert worth waking up for: the budget is burning too fast.

The workbook's definition: "Burn rate is how fast, relative to the SLO, the service consumes the error budget."

Burn rate 1 = the budget lasts exactly the 30 days.

Burn rate 10 = it is gone in 3 days.

Burn rate 1,000 = everything is failing, and a 99.9% budget is gone in 43 minutes.

Burn rate is the slope of the error budget: 30 days at 1, 3 days at 10, 43 minutes at 1,000.

The alert setup Google recommends for a 99.9% SLO:

Action	Burn rate	Long window	Short window	Budget consumed
Page	14.4	1 hour	5 minutes	2%
Page	6	6 hours	30 minutes	5%
Ticket	1	3 days	6 hours	10%

The long window proves the problem is real. The short window proves it is still happening, so the page stops soon after the fix.

The paging rule in Prometheus, in the workbook's shorthand (0.001 is the error budget of a 99.9% SLO; for 99.95% use 0.0005):

expr: (
        job:slo_errors_per_request:ratio_rate1h{job="myjob"} > (14.4*0.001)
      and
        job:slo_errors_per_request:ratio_rate5m{job="myjob"} > (14.4*0.001)
      )
    or
      (
        job:slo_errors_per_request:ratio_rate6h{job="myjob"} > (6*0.001)
      and
        job:slo_errors_per_request:ratio_rate30m{job="myjob"} > (6*0.001)
      )
severity: page

Each ratio_rate series is a recording rule: errors divided by requests over that window.

2 pages and 1 ticket replace the CPU and memory pages that wake people for things users never felt. Disk space and certificate expiry stay as tickets.

That swap is most of the cure for alert fatigue.

One warning for small products, with the workbook's own numbers: at 10 requests an hour, 1 failed request is a burn rate of 1,000.

Its options: synthetic traffic, combining small services into 1 SLO, or lowering the SLO or lengthening its window. I start with the longer window.

Decide what happens when the budget runs out

An SLO with no consequence inside the company is a dashboard.

The consequence is an error budget policy, agreed while everyone is calm. Google publishes its example, and 3 lines carry it:

Budget spent: "we will halt all changes and releases other than P0 issues or security fixes until the service is back within its SLO"
1 incident eats more than 20% of the budget: the team must conduct a postmortem
The teams disagree: "the issue should be escalated to the CTO to make a decision"

The startup version I use fits in 4 lines:

Budget left: ship whenever you want
Half the budget gone in the first week: the on-call engineer reviews every deploy until the window recovers
Budget spent: the next sprint's first items are reliability work
1 incident over 20% of the budget: a written postmortem with 1 fix that gets done

Why does a CTO want this? Because "should we slow down and fix reliability" stops being an argument between product and engineering.

The number answers it every Monday.

The error budget is the only reliability metric I know that tells a team when to go faster.

The SLO document, on 1 page

1 page per user-facing service, under 8 headings:

Service and user flow: what the user is doing ("call the public API")
SLI: the good event, the total event, and the query that counts both
SLO: the target and the window
Error budget: in requests and in minutes
Alerts: the burn rates that page and the one that opens a ticket
Policy: what happens at half spent and at fully spent
External SLA: the number in contracts, and the margin to the SLO
Owner and review date: a name, and a quarterly check that the target still matches what users need

Heading 7 is the one sales should read before the next enterprise call.

Mistakes I keep seeing

An SLA signed before any SLI exists: the first measurement arrives with the customer's complaint
SLA equal to the SLO: every internal miss is a contract breach
The SLO copied from last month's graph: current performance is where you start measuring, and the user's need is what sets the target
A 100% target: no deploy is safe under it
Uptime measured by a ping on /health: the health check passes while the checkout fails
Average latency as the SLI: the slowest 1% is invisible in it
15 SLOs per service: nobody can say which one matters during an incident
99.99% promised on a single database instance: the supplier's own SLA for it is 99.5%
Cloud credits nobody claims: the provider waits for your support case

Checklist

Always:

Define 2 SLIs per user-facing service: availability and latency, as good events over total events
Measure at the load balancer, on a real user flow
Measure for 4 weeks before setting a target
Set 1 SLO per SLI with a 30 day window, and write the error budget in minutes and in requests
Multiply your dependencies' SLAs and check the SLA you sign sits under the result
Page on burn rate 14.4 and 6, ticket on burn rate 1
Write the error budget policy before the first incident
Sign an SLA only with a quarter of measurements behind it, looser than the SLO
Put exclusions, a credit schedule, a claim deadline and a sole remedy line in every SLA
Review each SLO every quarter with product in the room

FAQ

What's the difference between SLOs and SLAs?

An SLO is an internal target your team sets for a service. An SLA is a contract with a customer that includes a consequence, usually service credits, for missing a target. Set the SLO tighter than the SLA, so you find out about a problem before it costs you money.

Is an SLO part of an SLA?

Yes. Every SLA contains at least 1 objective, the number being promised, plus the consequence for missing it. Most SLOs have no SLA around them: they exist so the team can decide between shipping and fixing.

What is the difference between SLI, SLO and SLA?

The SLI is the measurement (99.97% of requests succeeded). The SLO is the target on it (99.95% over 30 days). The SLA is the promise to a customer (99.9% monthly, or credits). Measure, target, promise, in that order.

What is an example of an SLO?

"99.9% of requests to the public API return a successful response, measured at the load balancer over a rolling 30 days." A latency SLO for the same service: "95% of requests complete in under 300 ms over a rolling 30 days."

What is an error budget?

1 minus the SLO. A 99.9% SLO leaves a budget of 0.1%: about 43 minutes of full downtime in 30 days, or 10,000 failed requests out of 10 million. While budget remains the team ships; when it is spent the team works on reliability.

Which comes first, the SLO or the SLA?

The SLO, with the SLI before it. Run the SLO for a quarter, see what you achieve, then offer an SLA with margin below it. An SLA signed first means the customer's contract is setting your engineering priorities.

Does every service need an SLA?

No. Internal services and free tiers run on SLOs alone. Sign an SLA when a paying customer asks for one, and only for the service that customer uses.

Is 99.9% uptime good?

For most B2B SaaS products, yes. It allows 43 minutes of downtime in 30 days, which a small on-call team can meet with 1 region and 2 availability zones. 99.99% allows 4 minutes and needs automatic failover everywhere.

Who owns the SLO?

The team that runs the service owns the measurement and the alerts. Product agrees to the target, because the target decides how much time goes to features. The CTO owns the tie-break, and signs off on any SLA built on top.

A recap

If you take away 1 thing, let it be this: measured > SLO > SLA.

Measure the SLI where the user is. Set the SLO from what the user needs.

Spend the error budget on shipping. Page on the burn rate.

Sign the SLA last, with margin.

You don't NEED a DevOps expert for the first SLO. A load balancer log, a spreadsheet and 4 weeks get you the first honest number.

If you would rather set it up with a team that has worked with 200+ companies, book a free production setup consultation.

P.S. The on-call rotation that answers those 2 pages is its own design problem. I wrote about it in how to design on-call before you hire an SRE.

DevOps as a Service: What It Is, Costs, and When to Use It

Michael Zion — Thu, 12 Nov 2026 07:30:12 +0000

This guide was written by Arthur Azrieli, CTO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

DevOps as a Service (DaaS) means an outside team of DevOps engineers builds and runs your delivery and cloud operations for a recurring fee: CI/CD pipelines, infrastructure as code, Kubernetes or other runtimes, monitoring, security baselines and cost control. You get a team's skills for less than a team's payroll. In exchange, you have to manage the ownership, access and exit terms carefully.

It beats hiring when your DevOps workload is real but not big enough for two or three full-time specialists. It also fits when you need senior skills faster than you can recruit them, or when one engineer is currently the only person who understands production. It loses to hiring when infrastructure is your product, when the work is constant and deep enough to keep a team busy, or when nobody inside the company can own decisions about the platform.

This guide is for founders, CTOs and engineering leaders deciding how to cover DevOps. It covers what DaaS actually includes, the delivery models providers sell under that name, how pricing works and how to compare it with the true cost of hiring, a decision checklist, contract safeguards, and how to run the first 90 days.

What DevOps as a Service actually includes

"DevOps as a Service" gets used for three different things, which is why the search results feel contradictory:

A managed engineering service. A provider's engineers work on your cloud accounts, repositories and pipelines on an ongoing basis. This is what most buyers mean, and what this guide is about.
A hosted toolchain. GitHub, GitLab, Azure DevOps or AWS developer tools sold as SaaS. Useful, but that is buying tools, not getting DevOps done.
A platform product. An internal developer platform or "platform as a service" that hides infrastructure behind a portal. It reduces DevOps work for product teams, but someone still has to run it.

A serious DaaS engagement usually covers these workstreams. Most companies need four or five of them, not all eight:

Workstream	What the provider does	What you should get to keep
Cloud foundation	Account structure, networking, identity and access, landing zone, environments	Infrastructure as code in your repositories, an access model diagram
Infrastructure as code	Terraform or OpenTofu modules, state management, review workflow	Modules your engineers can read and change through pull requests
CI/CD	Build, test and deploy pipelines, environment promotion, rollback	Pipeline definitions in your repositories, a documented release process
Runtime	Kubernetes, ECS, Cloud Run or VM operations, upgrades, autoscaling	Upgrade runbooks, a version and support calendar
Observability	Metrics, logs, traces, dashboards, alert rules	Alerts that page the right owner, dashboards per service
Reliability and on-call	Incident response, postmortems, backup and restore testing	Runbooks, tested restore procedures, an escalation policy
Security and compliance	Secrets management, least privilege, vulnerability scanning, audit evidence	Evidence that maps to your SOC 2 or ISO 27001 controls
Cost control	Tagging, rightsizing, commitments, budgets and alerts	A cost report your finance team can read, owners per cost line

Look at the last column. It is the real test of a DaaS provider. The fee buys work, but the value you keep is the code, documentation and operating habits that stay when the engagement ends.

The four ways to get DevOps covered

DaaS is one of four models. The difference that matters most is who owns the outcome and who directs the work day to day.

	In-house DevOps team	DevOps as a Service	Staff augmentation	Project consulting
Who directs the work	You	Shared: you set priorities, the provider plans and executes	You	The provider, against a defined goal
Who owns production outcomes	You	Shared, defined in the contract	You	You, after handover
Typical commitment	Permanent headcount	Monthly, ongoing	Monthly per engineer	Fixed scope and timeline
Time to first useful work	Months (hiring plus ramp-up)	Days to weeks	Weeks	Weeks
Breadth of skills	Whatever you hired	A bench of specialists behind your main engineer	One person's skills	Whatever the project needs
Main risk	Slow hiring, single points of failure	Knowledge staying with the provider	You still need to manage and review the work	Handover to nobody
Best for	Large, constant platform work	Startups and scale-ups without a full DevOps team	A known backlog and a capacity gap	A migration or a one-off build

DevOps as a Service is the only model where direction and ownership are shared, and it starts in days to weeks.

Many companies combine them: DaaS to stabilize and build the foundation, then one internal hire who takes over day-to-day ownership while the provider stays on for depth and coverage. If the staff augmentation column looks closer to what you need, read how to choose a staff augmentation company. For a project with a clear end, read how to choose a DevOps consulting service.

When DevOps as a Service beats hiring

DaaS is usually the better choice when most of these are true:

The workload is real but uneven. Some months bring a migration, a compliance audit or a Kubernetes upgrade. Other months are mostly maintenance. A full-time hire is either overloaded or underused.
You need breadth one person cannot have. Networking, Kubernetes internals, Terraform design, database operations, security and cost optimization rarely sit in one head.
Production depends on one person. If your only infrastructure engineer goes on holiday and deploys stop, you have a bus factor of one. A provider gives you coverage and a second pair of eyes.
Hiring is slow or risky. Senior DevOps engineers are hard to assess if nobody on your team is senior in DevOps. A bad hire with production access is expensive to undo.
You are early. Teams of roughly 5 to 40 engineers usually do not need a dedicated platform team yet, but they do need production-grade foundations.

When hiring wins

Infrastructure is part of what you sell (a data platform, a developer tool, a hosting product).
The work is constant and deep enough to keep two or more specialists busy all year.
Regulation or customer contracts prevent outside parties from accessing production.
You are building a platform team that will serve many product teams, and you want it to grow with the company.

Even then, a provider often helps for the first months: setting up the foundations, helping you interview candidates and handing over to the new team. If you are unsure how much DevOps work you actually have, estimate your required DevOps capacity before deciding. The guide to building a DevOps team covers the hiring path.

How much DevOps as a Service costs

Providers rarely publish prices, and the ones that do are not comparable, because "a plan" can mean ten hours or a full-time engineer. What you can compare is the pricing model and what it includes.

Pricing model	How it works	Good for	Watch for
Monthly hours plan	A fixed block of engineering hours per month, often with tiers	Ongoing work with uneven demand	Unused-hour expiry, response times outside the plan
Dedicated engineer	One or more named engineers at a monthly rate	Steady workload, deep context	Whether a senior backs up the engineer, cover during leave
Managed service with service levels	A fee per environment or service, with uptime and response commitments	Operations you never want to think about	What counts as "in scope", penalties that are only credits
Fixed-price project plus retainer	A defined build, then a smaller monthly plan to run it	Foundations, migrations	Change requests, who fixes defects after go-live
Pay per hour	Billed monthly for hours worked	Advisory work, small or unpredictable needs	Availability when you need someone urgently

Compare it with the real cost of hiring

The fair comparison is not a provider's monthly fee against a salary. It is against the full cost of covering the same work in-house:

Loaded salary. The US Bureau of Labor Statistics puts the median software developer wage at $135,980 (May 2025). Experienced DevOps and platform engineers are often above that median. Wages are about 70% of total employer compensation cost in US private industry (BLS, June 2026), so a $136,000 salary is roughly $194,000 in total cost before recruiting, equipment and tooling.
Recruiting and ramp-up. Recruiter fees, interview time from your senior engineers, and the months before a new hire is fully productive.
Coverage. One engineer is not on-call coverage. Real coverage for production needs at least two people who can respond, or an escalation path to someone outside.
Breadth gaps. The specialist work your hire cannot do still needs a consultant, so add a buffer for it.
Cost of mistakes. Outages, security incidents and cloud waste from missing experience are real costs, even if they never appear in a budget line.

Compare a provider's quote with the full stack of hiring costs, not with the salary alone.

Put a one-year number on each line, then ask providers to quote the same scope. If the provider costs more but removes the coverage gap and the breadth gap, it can still be the cheaper option. If you have constant work for two engineers, hiring usually wins on cost within a year or two. To see how MeteorOps structures engagements, see the plans and pricing page.

Decision checklist: DaaS, hire, or both

Answer these honestly. Each "yes" in the left group points toward DevOps as a Service, and each "yes" in the right group points toward hiring.

Points to DevOps as a Service	Points to hiring
Fewer than about 40 engineers, no platform team yet	Several product teams that need a shared platform
Workload swings with launches, audits and migrations	Constant workload for two or more specialists
Only one person understands production today	You already have DevOps people who need reinforcement
You need results this quarter	You can wait three to six months for a hire to ramp up
You need several specialties occasionally	You need one specialty all the time
Nobody internal can assess DevOps candidates	A senior engineer can run the hiring process

A mixed result is common and usually means "both": a provider now, and a plan to hire one owner who works alongside them.

Count the yes answers on each side; a mixed result usually means using both.

What good DevOps as a Service looks like in practice

These are the signals that a provider runs DaaS as engineering, not as a ticket desk:

Everything lands in your repositories. Terraform, pipeline files, Helm charts and runbooks live in your Git organization and go through your pull request reviews.
Access is personal and least privilege. Each engineer has their own identity through your identity provider or AWS IAM Identity Center. Nobody shares root credentials or long-lived access keys.
A named lead engineer, with seniors behind them. You know who works on your systems, and design reviews involve someone more senior than the person doing the work.
Visible work. A shared backlog, short written updates, and decisions recorded where your team can find them.
Measured outcomes. Progress is tracked with delivery metrics such as the five DORA metrics (change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate), plus cloud cost and incident trends. Hours logged and tickets closed do not count as outcomes.
Your team gets more capable. Your developers can deploy, read dashboards and handle common incidents without asking the provider.

Contract safeguards before you sign

DaaS contracts fail at the edges: who owns what, how fast someone responds, and how you leave. Check these clauses:

Clause	What to look for
Intellectual property	All code, configuration and documentation created for you belongs to you, with no license fees to keep using it
Access and credentials	Your identity provider, individual accounts, removal on exit, no provider-owned accounts in your production path
Scope and response times	What is included, what is extra, and response times for production incidents in writing
Named people	Who your lead engineer is, notice before they change, and how a replacement gets onboarded
Security and confidentiality	Handling of secrets and customer data, background checks, and support for your audits
Exit and handover	A handover period, final documentation, access removal and a reasonable notice period
Commitment	Minimum term, how to scale hours up or down, and what happens to unused hours

A provider that resists the exit clause is telling you how the relationship will end. A good one plans for its own replacement from the first week.

The first 90 days of a DevOps as a Service engagement

Days 1 to 14: access and assessment. Set up individual access, review cloud accounts, repositories, pipelines, monitoring, incident history and the cloud bill. The output should be a written list of risks ranked by impact, not a new architecture.
Days 15 to 45: reduce the biggest risks. Typical first fixes: backups that are actually restorable, removing shared admin access, moving unmanaged production resources into infrastructure as code, alerts on customer-facing symptoms, and a rollback path for deployments.
Days 46 to 75: speed up delivery. Faster, more reliable pipelines, preview or staging environments that match production, and fewer manual release steps.
Days 76 to 90: review and plan. Compare the delivery metrics, incident count and cloud spend against day 1. Agree on the next quarter's priorities and adjust the plan size up or down.

A good engagement starts with a ranked risk list and earns the right to speed things up.

If the first two weeks produce a sales proposal for a large rebuild instead of a ranked risk list, slow down. A DevOps maturity assessment is a cheap way to get a baseline before you commit to anything larger.

Common mistakes with DevOps as a Service

Outsourcing ownership along with the work. Someone inside the company still has to make product and risk decisions about production.
Treating the provider as a ticket queue. If every change waits for a ticket to be picked up, delivery slows down. Ask for self-service paths for your developers.
Letting infrastructure live in the provider's accounts. Cloud accounts, domains, repositories and secrets stores should be yours from the start.
Buying tools before defining problems. New platforms do not fix unclear ownership or missing runbooks.
No exit plan. If you could not switch providers in a month, you are locked in, whatever the contract says.

FAQ

What is DevOps as a Service?

DevOps as a Service is an outsourced model where an external team builds and operates your delivery pipelines, cloud infrastructure, monitoring and related practices for an ongoing fee. Unlike buying DevOps tools, you are buying engineers' time and expertise, delivered inside your own accounts and repositories.

Is DevOps as a Service the same as managed DevOps services?

Mostly, yes. "Managed DevOps services" usually puts more emphasis on running existing systems against service levels, while "DevOps as a Service" also covers building new foundations. In practice, check the scope and response terms rather than the label.

How much does DevOps as a Service cost?

It depends on the pricing model (hours plan, dedicated engineer, managed service or project) and the scope. Compare quotes against the full one-year cost of hiring: loaded salary, recruiting, ramp-up, on-call coverage and the specialist help a single hire would still need.

Is DevOps as a Service worth it for a small team?

For teams without a dedicated DevOps engineer, it usually is, because one hire cannot cover all the specialties or provide on-call coverage alone. It is less attractive if your workload is small enough for a few hours of advice a month, or large and constant enough for two full-time hires.

Can a DevOps as a Service provider work alongside our in-house engineers?

Yes, and that is often the best setup. The provider brings depth and coverage, while your engineers keep product context and ownership. Make sure work goes through the same repositories, reviews and backlog.

What are the risks of DevOps as a Service?

The main risks are knowledge staying with the provider, broad access to production, and lock-in through provider-owned accounts or tools. Clear IP ownership, individual least-privilege access, documentation in your repositories and a written exit plan reduce all three.

How fast can a DevOps as a Service team start?

A good provider can start within days of signing, once access is granted. Expect a short assessment first. Visible improvements usually come within the first month, and bigger changes such as migrations take longer.

The short version

DevOps as a Service gives a startup senior DevOps coverage without building a team: foundations, pipelines, monitoring, security and cost control, delivered in your own accounts. Choose it when the work is real but uneven, when you need breadth and coverage, or when production depends on one person. Hire when infrastructure is your product or the work is constant. Either way, keep ownership, access and the exit plan on your side.

If you want senior engineers to take on your DevOps work this way, see how MeteorOps DevOps outsourcing works.

Observability Stack for Startups: What to Run at 5, 20 and 100 Engineers

Michael Zion — Wed, 11 Nov 2026 07:30:03 +0000

This guide was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

A five-person startup should start with error tracking, structured logs, the four golden signals (latency, traffic, errors, saturation) for each user-facing service, uptime checks, and three or four symptom-based alerts, all on one managed platform. Add distributed tracing with OpenTelemetry once requests cross several services. Self-hosting becomes worth it only when the bill outgrows a platform engineer's time.

A startup observability stack should answer three questions quickly: what is broken, who is affected, and what changed recently. It does not mean buying the biggest platform or instrumenting every function on day one. It means collecting enough signals to debug production, often Kubernetes, serverless jobs, managed databases and queues, without tribal knowledge or emergency SSH sessions.

What to run at 5, 20 and 100 engineers

Stage	Signals to have	Typical tooling	Who owns it
About 5 engineers, one or two services	Error tracking, structured logs, golden-signal dashboards, uptime checks, 3 to 5 alerts routed to one channel or pager	One managed platform on a free or entry tier (Grafana Cloud, Datadog, or your cloud's native monitoring) plus an error tracker such as Sentry	Whoever is on call, with one engineer setting conventions
About 20 engineers, several services and workers	Add OpenTelemetry tracing at service boundaries, deploy markers, queue and database dashboards, an on-call rotation, alert runbooks	Same platform on a paid tier, OpenTelemetry Collector in the cluster, log retention and sampling rules	A named owner for standards, usually in a platform or DevOps role
About 100 engineers, many teams	SLOs per service, per-team dashboards and alert routing, cost allocation, trace sampling strategy	Either a commercial platform under negotiated pricing or a self-hosted stack (Prometheus or Mimir, Loki, Tempo, Grafana) run by a platform team	A platform team that treats observability as an internal product

What it costs: Grafana Cloud vs Datadog vs self-hosted

List prices from the vendors' pricing pages as of October 2026. Real bills depend on hosts, active series, log volume and retention, so use these to compare shapes, not as quotes.

	Grafana Cloud	Datadog	Self-hosted LGTM (Loki, Grafana, Tempo, Mimir or Prometheus)
Free tier	10k metric series, 50 GB logs, 50 GB traces, 14-day retention, 3 users	Infrastructure only: up to 5 hosts, 1-day metric retention	Software is free
Paid entry	Pro: $19 per month platform fee, same included usage, then pay as you go	Infrastructure Pro: $15 per host per month billed annually ($18 on-demand)	Compute and storage for the stack, often a few hundred dollars a month at startup scale
Metrics	From $6.50 per 1k active series	Included per host; custom metrics billed separately	Storage and memory grow with series count
Logs	From about $0.55 per GB (process, write and retain combined)	$0.10 per GB ingested, plus $1.70 per million events indexed for 15 days	Object storage, cheap to keep
Traces / APM	Same per-GB pricing as logs	APM from $31 per host per month billed annually	Object storage
Hidden cost	High-cardinality labels inflate series	Per-host and indexing costs grow with every node and noisy log line	Engineer time: upgrades, scaling, on-call for the monitoring itself

Sources: Grafana Cloud pricing and Datadog pricing.

Start with the failures you need to diagnose

Tool selection gets messy when teams begin with vendor feature matrices. Start with the incidents you expect to handle instead. A payments API returning intermittent 500s, a background worker falling behind its queue, a database connection pool saturating after a traffic spike, or a canary release increasing latency in one region all need different signals. Metrics are best for spotting trends and triggering alerts, logs explain discrete events, traces connect work across services, and deploy or configuration events show what changed. Mapping those failure modes first helps you decide where deep instrumentation is worth it and where a simple log line, counter, or managed cloud metric is enough.

Map the failures you expect first, then pick the signal that answers each one.

For most startups, the first observability goals should be practical, boring, and tied to production outcomes. Prioritize the checks that shorten incidents and protect user experience before you chase perfect coverage:

Find customer-impacting issues fast. The team should know when users are seeing elevated error rates, slow requests, failed jobs, or broken workflows. Alerts should include enough context to point to a likely service, dependency, tenant, region, or recent deploy instead of merely announcing that “something is down.”
Connect symptoms to recent changes. Deployments, configuration changes, feature flags, and infrastructure updates should be visible during incident review.
Debug without direct server access. Engineers should not need to SSH into machines or manually inspect containers to understand production behavior.
Support on-call without constant noise. Alerts should point to real action, not every small fluctuation.
Keep costs predictable. Observability data can grow quickly, especially logs and high-cardinality metrics.

This keeps the stack tied to operational needs instead of tool sprawl. If you are still deciding how your engineering ownership model should work, it can help to define responsibilities early. A clear team model, like the one outlined in how to build a DevOps team, makes observability easier to operate because someone owns the standards, not just the dashboard links.

Build around the four core signal types

A useful observability stack usually combines logs, metrics, traces, and alerts. Each signal answers a different kind of question. Treating one as a replacement for all the others usually creates blind spots.

Logs

Logs explain what happened inside an application or service. They are useful for debugging specific failures, understanding request context, and reviewing edge cases after an incident.

Good startup logging practices include:

Use structured logs, such as JSON, instead of free-form text when possible.
Include request IDs, user or tenant identifiers where appropriate, service name, environment, and error details.
Avoid logging secrets, tokens, payment data, or unnecessary personal data.
Set retention based on debugging needs and cost, not default settings.

The common failure mode is logging everything. That feels safe until costs rise or engineers cannot find the few lines that matter. Start with application errors, key business workflows, background jobs, authentication events, and integration failures.

Metrics

Metrics show trends and current health. They are best for questions like: error rate increased, memory usage is climbing, queue depth is growing, or request latency changed after deployment.

Most startups should track a small set of service-level metrics first:

Request rate
Error rate
Latency, especially p95 or p99 where available
CPU and memory usage
Database connection usage
Queue depth and job failure rate

Be careful with high-cardinality labels, such as user ID, request path with raw IDs, or tenant ID on every metric. They can make queries expensive and dashboards hard to use. Use detailed labels where they help diagnosis, but do not attach every possible field to every metric.

Tracing

Distributed tracing helps you follow a request across services, queues, databases, and external APIs. It becomes important when one user action touches several components and no single log line explains the delay.

Tracing is especially useful when:

A monolith is being split into services.
Requests pass through an API gateway, service layer, and background workers.
External providers affect latency or reliability.
Engineers cannot tell which service owns a failure.

You do not need to trace every internal detail at first. Start with entry points, service boundaries, database calls, and external API calls. Make sure trace IDs appear in logs, so engineers can move between traces and log events during debugging.

Trace the boundaries first and put the trace ID in every log line.

Alerts

Alerts turn signals into action. Bad alerts wake people up for conditions that do not matter. Good alerts point to user impact, clear ownership, and a response path.

Early alerting should focus on symptoms rather than every possible cause:

High error rate on a user-facing service
Sustained latency above an agreed threshold
Critical background jobs failing repeatedly
Queue backlog growing beyond normal recovery
Database or infrastructure limits close to exhaustion

Page on what users feel, not on causes like a CPU spike.

If alerts are already noisy, fix that before adding more. A practical guide to handling alert fatigue can help you separate useful pages from low-value notifications.

Choose tools that match your stage

The right observability tools depend on your architecture, cloud provider, team size, budget, and operational maturity. A two-person engineering team should not run a complex observability platform unless there is a strong reason. A larger team with multiple services may need stronger standards and deeper query options.

Use these criteria when comparing options:

Setup effort: How long does it take to instrument one service and get useful dashboards?
Operational burden: Who patches, scales, backs up, and maintains the observability components?
Data model: Can the tool connect logs, metrics, and traces through shared fields like service name, environment, and request ID?
Cost controls: Can you manage retention, sampling, indexing, and ingestion limits?
Developer workflow: Can engineers find what they need without learning a complex query language on day one?
Integration fit: Does it work cleanly with your cloud, deployment pipeline, container platform, and incident process?

Managed tools are often a good fit early because they reduce maintenance work. Open source tools can be a good fit when you have the skills and time to operate them well. The main risk is choosing based on what looks powerful in a demo, then discovering the team cannot keep it clean in production.

Tool choice should follow the same discipline as the rest of your platform decisions. If you need a broader framework, use the same thinking you would apply when you choose DevOps tools for your team: match the tool to the team’s current workload, not an imagined future state.

Instrument the application, not just the infrastructure

Cloud dashboards can tell you that a container restarted, a database used more CPU, or a load balancer returned more 5xx responses. That is useful, but it rarely tells the whole story. Most startup incidents need application context.

For example, a latency spike may come from a slow database query, a third-party API timeout, a feature flag path, a cache miss pattern, or a single tenant sending unusual traffic. Infrastructure metrics alone will not explain those differences.

Good application instrumentation should include:

Clear service names and environment names, such as production, staging, and development.
Correlation IDs passed through requests, logs, and traces.
Consistent error handling with useful messages and stack traces where appropriate.
Custom metrics for important workflows, such as signup completion, checkout failure, import job duration, or notification delivery failure.
Deployment markers so teams can compare behavior before and after a release.

Do this incrementally. Pick one important user flow, such as login, checkout, data import, or report generation. Add logs, metrics, and traces around that path. Then repeat for the next critical flow. This produces value faster than trying to instrument every service perfectly before anyone can use the data.

Set sensible defaults before the stack grows

Observability gets harder to clean up later. Naming, retention, labels, and alert conventions should be simple enough for every engineer to follow.

Define these defaults early:

Service naming: Use consistent names across logs, metrics, traces, dashboards, and alerts.
Environment naming: Avoid mixing production and non-production data in the same views unless the distinction is clear.
Log levels: Decide what counts as debug, info, warning, error, and critical.
Retention: Keep high-value data long enough for debugging and reviews, but avoid keeping noisy data by default.
Sampling: Sample traces where volume is high, but keep enough detail for failed and slow requests.
Alert routing: Send pages to people who can act. Send lower-priority notifications somewhere that does not interrupt sleep.
Dashboard ownership: Every dashboard should have a purpose and an owner, or it will become stale.

If your startup uses Azure DevOps or is still setting up delivery foundations, observability should sit beside source control, pipelines, and release practices. The setup advice in Azure DevOps for startups can help keep those foundations organized.

Avoid the common startup observability traps

Most observability problems come from over-collecting, under-owning, or alerting on the wrong things.

Too many dashboards: A dashboard is useful only if someone uses it during incidents or reviews. Keep the main views focused on service health, user impact, and recent changes.
Logs without structure: Plain text logs are harder to query and correlate. Structured logs make production debugging faster.
Metrics with uncontrolled labels: High-cardinality metrics can increase cost and slow down queries. Review labels before they spread across services.
Alerts tied to causes instead of symptoms: CPU spikes may or may not affect users. Error rates, failed jobs, and latency usually make better primary alerts.
No incident feedback loop: After each incident, ask which signal was missing, which alert was noisy, and which dashboard helped. Then adjust the stack.
No cost review: Observability bills can grow quietly. Review ingestion, retention, and indexing on a regular schedule.

A simple review after each incident is often the best way to improve. If engineers spent 30 minutes looking for the right log field, add the field. If an alert fired three times with no action taken, remove it, lower its priority, or rewrite it. If a trace made the root cause obvious, expand tracing to the next important workflow.

Each incident review turns one gap or noisy alert into a concrete fix.

FAQ

What observability should a five-person startup set up first?

Error tracking, structured JSON logs with a request ID, the four golden signals for each user-facing service, an uptime check on the main user flow, and a handful of alerts on user-facing symptoms (error rate, latency, failed critical jobs). Use one managed platform so nobody has to run monitoring infrastructure.

Should a startup self-host Prometheus and Grafana?

Usually not at first. Self-hosting is cheaper in software but costs engineer time for upgrades, storage, scaling and on-call for the monitoring itself. It starts to pay off when your managed bill is larger than a meaningful share of a platform engineer's time, or when data residency requires it.

Do we need distributed tracing from day one?

No. Tracing pays off once a request crosses several services, queues or external APIs. Start with OpenTelemetry SDKs at service entry points and outbound calls, and put trace IDs in your logs so you can jump between them.

How do we build one observability platform that combines alerts, logs, metrics and traces?

Standardize on OpenTelemetry for instrumentation, send everything through an OpenTelemetry Collector, and use shared resource attributes (service.name, deployment.environment) so the backend can link logs, metrics and traces. Grafana Cloud and Datadog both accept OTLP, which keeps you portable.

How do we keep observability costs predictable?

Set log retention and indexing rules per environment, drop debug logs in production, keep high-cardinality values (user IDs, raw URLs) out of metric labels, sample traces while keeping errors and slow requests, and review the bill monthly with the engineers who generate the data.

Takeaway

Build your startup observability stack around the questions your team needs to answer in production. Start with structured logs, essential metrics, targeted tracing, and alerts tied to real user impact. Keep the setup small enough to operate, but consistent enough to scale.

The goal is not to collect every signal. The goal is to shorten the path between “something is wrong” and “we know what changed, who is affected, and what to fix next.”

Cluster Autoscaler vs Karpenter: Node Autoscaling Without Cost Surprises

Michael Zion — Tue, 10 Nov 2026 07:30:07 +0000

This guide was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

Cluster Autoscaler adds and removes nodes in node groups you define in advance. Karpenter skips node groups and launches whichever instance type best fits the pending pods, then consolidates underused nodes. On AWS, Karpenter usually packs workloads tighter and handles Spot better. Cluster Autoscaler is the portable choice that works on every major cloud. Either way, cap total capacity explicitly.

Node autoscaling is where Kubernetes cost surprises usually come from. A sensible pod autoscaler plus an unbounded node autoscaler can turn one traffic spike into a much larger bill that never scales back down. This guide compares the two node autoscalers, shows the configuration that bounds each one, and covers the workload-level guardrails that matter with either.

Cluster Autoscaler vs Karpenter at a glance

	Cluster Autoscaler	Karpenter
How it adds capacity	Raises the desired size of an existing node group (an Auto Scaling group, managed node group or VM scale set)	Creates instances directly from pending pod requirements, choosing from many instance types
Instance choice	Fixed per node group; you pre-create groups for each shape	Any type allowed by NodePool requirements (family, size, architecture, zone, Spot or On-Demand)
Scale-down	Removes nodes below a utilization threshold (default 50%) after they stay unneeded for a while (default 10 minutes)	Consolidation deletes empty nodes and replaces underused ones with cheaper, smaller nodes
Spot handling	Mixed-instance node groups plus a separate interruption handler	Built-in Spot and On-Demand selection, with interruption handling through an SQS queue
Disruption controls	PDBs and the `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` annotation	PDBs, NodePool disruption budgets, and the `karpenter.sh/do-not-disrupt: "true"` annotation
Cost ceiling	Node group max sizes and `--max-nodes-total`	`spec.limits` on each NodePool (CPU and memory)
Clouds	AWS, GCP, Azure and many others	AWS (v1 stable since 2024); on Azure through AKS Node Auto Provisioning, which is built on Karpenter
Main risk	Many node groups to maintain; slow, coarse scaling	Unbounded NodePools; churn from aggressive consolidation

Both tools react to pending pods, but only the ceiling you set stops the growth.

Bound Karpenter with NodePool limits and disruption budgets

A Karpenter NodePool without limits can keep launching nodes as long as pods are pending. Always set a CPU and memory ceiling, choose the consolidation policy, and add a disruption budget so consolidation never drains too much of the cluster at once.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
      expireAfter: 720h          # replace nodes every 30 days
  limits:
    cpu: "200"                   # hard ceiling for this NodePool
    memory: 400Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets:
      - nodes: "10%"             # never disrupt more than 10% of nodes at once
      - nodes: "0"               # no voluntary disruption during business peak
        schedule: "0 9 * * mon-fri"
        duration: 8h

Consolidation keeps saving money at night but leaves the cluster alone during the business peak.

When both Spot and On-Demand are allowed, Karpenter prefers Spot and falls back to On-Demand when Spot capacity is unavailable. Allowing both architectures lets it pick cheaper Graviton instances when your images are multi-arch. Mark pods that must not be moved by consolidation (long batch jobs, singletons during a migration) with karpenter.sh/do-not-disrupt: "true".

Bound Cluster Autoscaler with node group sizes and scale-down settings

Cluster Autoscaler can only grow node groups up to their configured maximum, so the max size of each group is your cost ceiling. The flags below make scale-down and expansion choices explicit:

command:
  - ./cluster-autoscaler
  - --cloud-provider=aws
  - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster
  - --expander=least-waste            # pick the group that wastes the least CPU and memory
  - --balance-similar-node-groups     # spread across zones evenly
  - --max-nodes-total=60              # cluster-wide ceiling
  - --scale-down-utilization-threshold=0.5
  - --scale-down-unneeded-time=10m
  - --skip-nodes-with-system-pods=true

Run one node group per zone for workloads with zonal volumes, and keep the number of node shapes small. If you need many instance types to chase Spot capacity, that is usually the point where Karpenter becomes simpler to operate.

Which one should you use?

Use Karpenter on EKS when workloads vary in shape, you want Spot with automatic fallback, or you are paying for half-empty nodes that Cluster Autoscaler will not consolidate.
Use Cluster Autoscaler when you run on several clouds, on GKE (which has its own node auto-provisioning), when a few uniform node groups already fit your workloads, or when your team prefers predictable, fixed node shapes.
On AKS, use the built-in cluster autoscaler or Node Auto Provisioning rather than installing either tool yourself.

Do not run both against the same nodes. When migrating from Cluster Autoscaler to Karpenter, create the NodePool, scale Cluster Autoscaler to zero replicas, then cordon and drain the old node groups gradually while PDBs protect critical workloads.

Understand Which Kubernetes Layer Is Scaling

Kubernetes autoscaling usually involves several independent control loops. They react to different signals and can amplify one another when configured carelessly.

Horizontal Pod Autoscaler (HPA): Changes the number of replicas for a workload, usually from CPU, memory, or a custom metric.
Vertical Pod Autoscaler (VPA): Recommends or changes CPU and memory requests for pods. Its operating mode needs careful review because changing requests can restart or reschedule workloads.
Cluster Autoscaler: Adjusts the number of nodes when pods cannot be scheduled or nodes are underused, depending on its configuration.
Karpenter: Provisions and removes nodes based on pending pod requirements and configured constraints.
Managed node pool autoscaling: Changes the size of a cloud provider’s node group according to its own minimum and maximum settings.

These layers do not share a single cost budget. An HPA can increase replicas, which creates unschedulable pods, which causes a node autoscaler to add nodes. If the HPA never scales down, or if pod requests are much larger than actual usage, the node count can remain high after traffic falls.

Each loop does its own job correctly, and the bill grows anyway.

Start by writing down the scaling path for each important workload:

What metric increases the replica count?
What is the minimum and maximum replica count?
What CPU and memory requests does each replica reserve?
Which node pool can run the workload?
What causes a new node to be created?
What prevents the workload from consuming unlimited capacity?

If you cannot answer these questions, changing a maximum replica count is not a cost-control strategy. It is a guess.

Set Resource Requests Before Setting Replica Limits

Autoscaling decisions depend on resource requests. The HPA commonly evaluates CPU utilization as a percentage of the CPU request, not as a percentage of the node's total capacity. A container with a request of 500 millicores running at 400 millicores is at 80% requested CPU, even if the node still has substantial unused capacity.

Inaccurate requests skew every scaling decision the HPA makes.

That makes inaccurate requests expensive in both directions:

Requests that are too low: The scheduler packs too many pods onto a node, and the workload may experience throttling or memory pressure before the HPA reacts.
Requests that are too high: Each replica reserves more node capacity than it needs, so scaling out creates nodes sooner and keeps them larger than necessary.
Missing requests: CPU-based utilization may be unsuitable or unavailable for the workload, and scheduling becomes less predictable.

Use measured usage from representative periods, including deployments, batch activity, and peak traffic. Do not set requests from a single quiet hour. For memory, leave enough room for normal working-set variation because memory pressure can terminate a pod before a CPU-based HPA responds.

A minimal workload definition should make the resource assumptions explicit:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: example/api:1.0.0
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "512Mi"

The image name in this example is illustrative. Replace it with the image and version used by your workload. The important part is that requests and limits reflect a tested operating range rather than arbitrary defaults.

Bound the pod autoscaler too

Node autoscalers react to pending pods, so an unbounded Horizontal Pod Autoscaler drives node cost directly. Give every HPA a deliberate maxReplicas, slow scale-down (a stabilization window of about 5 minutes), and a metric that reflects real demand. A slow downstream dependency can raise CPU, add replicas, and add nodes without helping users at all. Our Kubernetes HPA guide covers the configuration in detail.

Put hard limits around namespaces and node growth

Pod limits do not automatically create a cloud cost limit. If an HPA can create ten times as many replicas, a node autoscaler may add enough capacity to schedule them. You need limits at both the workload and cluster levels.

For each node pool or provisioner, define:

A minimum size that supports normal operation and planned disruption.
A maximum size that reflects the approved capacity and cost envelope.
The instance types or node sizes that the workload may use.
Scheduling rules that keep specialized workloads away from general-purpose capacity.
A scale-down policy that removes genuinely unused nodes without disrupting critical workloads.

Use a ResourceQuota in each namespace to prevent one team or environment from consuming all available capacity. Pair it with a LimitRange when you need default requests and limits for containers that omit them.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: application-quota
  namespace: production
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"

A quota is a scheduling guardrail, not a guarantee that the cloud bill will stay below a specific amount. Node overhead, system workloads, daemon sets, storage, load balancers, and other services still consume resources. Leave capacity for those components when choosing quota values.

Review node autoscaling alongside workload placement. A pod with a restrictive node selector, taint requirement, or topology rule may force the provisioner to create a new node even when another node has spare capacity. This can be the correct availability decision, but it should be visible in the cost review.

Teams operating several clusters should document their node-pool limits and ownership clearly. For example, importing or consolidating clusters can expose inconsistent autoscaling rules that were previously hidden by separate cloud accounts or environments. A documented migration plan, such as the approach described in this Kubernetes cluster import case study, can help keep infrastructure policy explicit during that work.

Test Failure Modes Before Production Traffic Finds Them

Autoscaling should be tested as a system rather than as an isolated HPA object. A useful test starts with a known workload and observes every step: metric change, replica change, pending pods, node creation, application recovery, and scale-down.

Run these checks in a non-production environment or during an approved production exercise:

Increase request load until the HPA reaches several replica levels.
Confirm that new pods become ready and that readiness probes prevent traffic from reaching unready instances.
Observe whether pending pods trigger node growth as expected.
Stop the load and measure how long replicas and nodes take to fall.
Check whether scale-down causes evictions, queue loss, connection failures, or latency spikes.
Repeat the test while a deployment, node drain, or dependency slowdown is occurring.

Watch these signals during the test:

HPA desired replicas and current replicas.
Pending pods and their scheduling events.
Node count, node utilization, and unschedulable nodes.
CPU throttling and memory pressure.
Request latency, error rate, queue depth, and saturation.
Replica and node count after demand returns to normal.

Do not judge success by node utilization alone. A cluster can look efficient while users experience high latency because the application is waiting on a database or another service. Conversely, a low-utilization node may be necessary because the workload has strict placement rules or requires spare capacity for disruption.

Create a Cost Review That Matches Scaling Behavior

Cost control fails when engineers review the bill after the fact and cannot connect it to a scaling event. Record the decisions that can increase capacity and assign an owner to each one.

At minimum, review:

Changes to HPA minimum and maximum replicas.
Changes to CPU and memory requests.
Changes to node-pool or provisioner limits.
New taints, tolerations, selectors, or topology constraints.
Changes to custom metrics and their collection intervals.
Namespaces that approach their resource quotas.

Use separate limits for development, staging, and production. A staging environment that inherits production replica limits can create unnecessary spend during load tests or a stuck job. Development clusters often need a firm upper bound because an accidental deployment can otherwise remain active until someone notices.

When a workload needs more capacity, ask which constraint is actually failing:

If requests are too high, recalibrate them from representative usage.
If the HPA reacts to the wrong signal, change the metric rather than raising the maximum.
If pods cannot fit efficiently, review placement rules and node shapes.
If the workload has a legitimate sustained increase, update the capacity plan and budget deliberately.
If several clusters use different policies, standardize the policy before copying another configuration.

Raising a limit is the right answer only for a legitimate sustained increase.

For teams managing AWS and Kubernetes together, infrastructure changes should be reviewed as one capacity decision rather than as separate application and cloud changes. The practices in this AWS and Kubernetes management case study are relevant to that type of review. Managed Kubernetes services such as Azure Kubernetes Service can reduce control-plane administration, but they do not remove the need to bound workload and node scaling.

When the Configuration Needs a Larger Review

Get a second review when autoscaling has become difficult to explain, when several teams share a cluster, or when a single traffic event can materially affect infrastructure spend. The review should examine manifests, metrics, scheduling events, quotas, node limits, and recent cost changes together.

Useful deliverables include:

A diagram showing the HPA, scheduler, node autoscaler, node pools, and external dependencies.
A table listing minimum and maximum replicas for each production workload.
A resource-request review based on observed usage.
A namespace quota and ownership map.
A test record showing scale-up and scale-down behavior.
An escalation rule for reaching a capacity or spending threshold.

Infrastructure teams can also compare patterns used in production Kubernetes environments, including the operational approach described in this scalable Kubernetes and cloud infrastructure case study. Treat those patterns as material for review, not as a substitute for testing your own workload.

FAQ

Is Karpenter better than Cluster Autoscaler?

On AWS it is usually more cost-efficient, because it picks right-sized instances per workload, uses Spot with fallback, and consolidates underused nodes. Cluster Autoscaler is simpler to reason about, supports every major cloud, and is fine when a few uniform node groups fit your workloads.

Does Karpenter work on Azure or GCP?

On Azure, yes: AKS Node Auto Provisioning is built on Karpenter and is managed for you. There is no supported Karpenter provider for GKE; use GKE's cluster autoscaler and node auto-provisioning there.

How do I stop Karpenter from overspending?

Set spec.limits for CPU and memory on every NodePool, restrict instance categories and sizes in requirements, enable consolidation, keep accurate pod requests, and alert on node count and spend. Without limits, any burst of pending pods can launch new nodes.

Why is my Cluster Autoscaler not scaling down?

Common causes: pods with local storage or the safe-to-evict: "false" annotation, PDBs that allow 0 disruptions, kube-system pods without PDBs, nodes above the utilization threshold, or pods that cannot fit on any other node because of affinity, taints or zonal volumes. Check the autoscaler logs and the cluster-autoscaler-status ConfigMap.

Can I run Karpenter and Cluster Autoscaler together?

Only on separate sets of nodes, and only temporarily during a migration. If both manage the same capacity they make conflicting decisions. Scale Cluster Autoscaler to zero once Karpenter NodePools are serving the workloads.

Takeaway

Pick the node autoscaler that fits your cloud and workload mix, then set autoscaling limits in layers. Give workloads accurate resource requests, bound HPA replicas, cap node-pool growth, use namespace quotas, and test the complete path from demand to cloud capacity. Then review scaling changes as capacity decisions with clear ownership. This approach keeps autoscaling useful during real demand without allowing an HPA or node provisioner to define your cloud bill by accident.

Liveness vs Readiness Probe: Kubernetes Config That Avoids Restart Loops

Michael Zion — Mon, 09 Nov 2026 07:30:11 +0000

This guide was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

A Kubernetes liveness probe tells the kubelet when to restart a container; a readiness probe only removes the pod from Service traffic and never restarts it. To avoid restart loops, keep liveness checks local to the process, move dependency checks to readiness, use a startupProbe for slow boots, and set timeouts of 2 to 3 seconds.

The goal is not to make liveness probes pass more often. It is to use them only for conditions a restart can actually fix, and to give the application enough time to prove it is truly wedged before Kubernetes kills it.

Liveness vs readiness vs startup probes

Probe	Question it answers	What happens on failure	What it should check
`livenessProbe`	Is the process wedged beyond recovery?	The kubelet kills and restarts the container (per `restartPolicy`).	Local process health only: event loop responsive, no fatal internal state.
`readinessProbe`	Can this pod take traffic right now?	The pod is removed from Service endpoints. No restart.	Dependencies, warm caches, draining state.
`startupProbe`	Has the app finished booting?	Liveness and readiness are held back until it passes. If it never passes within `failureThreshold x periodSeconds`, the container is restarted.	Initialization complete: migrations, cache load, model load.

Only liveness and startup failures restart the container; readiness only controls traffic.

Understand what a liveness probe is allowed to do

A liveness probe answers one narrow question: “Should Kubernetes restart this container?” If the probe fails enough times, the kubelet restarts the container according to the pod’s restart policy. That is powerful, but it is also blunt.

A liveness probe should detect problems such as:

A process deadlock where the application stops serving all useful work.
A stuck event loop that no longer responds locally.
A broken internal state that only a process restart can clear.
A worker process that is alive at the operating system level but cannot make progress.

A liveness probe should usually avoid checking:

Database reachability.
Message broker availability.
Third-party API status.
Availability of another internal service.
Long-running migrations or warm-up tasks.

If the database is unavailable and every pod fails its liveness probe because of that database check, Kubernetes will restart every pod. The restart will not fix the database. It will add load, increase cold starts, clear in-memory caches, and make recovery harder.

Put dependency checks in readiness so an outage removes pods from traffic instead of restarting them all.

Use the right probe for the right job:

startupProbe: gives slow-starting containers time to initialize before liveness checks begin.
readinessProbe: controls whether the pod receives traffic.
livenessProbe: restarts the container when it is truly unhealthy.

If you manage Kubernetes manifests through infrastructure as code, keep this distinction explicit in your modules or templates. The same principle applies whether you apply raw YAML, use Helm, or deploy Kubernetes resources using Terraform.

Start with a safe baseline configuration

Begin with conservative thresholds, then tighten them after observing real startup time and failure behavior. Do not copy probe values across services without checking how each service starts, warms up, and handles load.

Here is a risky liveness probe:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: example/api:1.0.0
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 1
            failureThreshold: 3

This can restart the container after roughly 20 seconds: 5 seconds of initial delay, then 3 failed probes spaced 5 seconds apart. If the service sometimes needs 45 seconds to load configuration, run migrations, compile templates, or warm caches, this pod can restart forever.

A safer pattern separates startup, readiness, and liveness:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: example/api:1.0.0
          ports:
            - name: http
              containerPort: 8080

          startupProbe:
            httpGet:
              path: /startupz
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 24

          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
            successThreshold: 1

          livenessProbe:
            httpGet:
              path: /livez
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3

In this example, the startup probe allows up to about 120 seconds for startup: 24 failures multiplied by a 5-second period. While the startup probe is still failing, Kubernetes does not run the liveness probe. After startup succeeds, liveness begins.

A startupProbe window longer than slow startup lets the app finish booting before liveness can restart it.

This matters for applications with variable boot times, such as Java services, applications that load large models, services that hydrate caches, or workers that recover state during startup.

Use practical threshold math before shipping

You should be able to explain every probe value in a deployment review. If the answer is “we copied it,” the probe is not tuned.

Use this rough calculation:

restart window ≈ initialDelaySeconds + (failureThreshold × periodSeconds)

For a liveness probe like this:

livenessProbe:
  httpGet:
    path: /livez
    port: http
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Kubernetes may restart the container after about 60 seconds of failing liveness checks. That may be fine for a stateless API with a 10-second normal startup time. It may be too aggressive for a batch worker that can pause during garbage collection, checkpoint recovery, or CPU pressure.

Use these starting points as a practical baseline, then adjust with real data:

startupProbe: set failureThreshold × periodSeconds higher than your slow but acceptable startup time. If p95 startup is 70 seconds, start around 120 seconds.
readinessProbe: keep it responsive. A period of 5 to 10 seconds is common because it controls traffic routing.
livenessProbe: keep it less aggressive than readiness. A period of 10 to 30 seconds with a failure threshold of 3 is a safer default for many services.
timeoutSeconds: avoid the default of 1 second for applications that can pause briefly under load. Start with 2 or 3 seconds if you have no better data.
initialDelaySeconds: prefer startupProbe for startup handling. Use initialDelaySeconds only when startup behavior is simple and predictable.

Probe tuning should live next to deployment configuration, not in someone’s notes. If your platform team provisions services and dependencies through Kubernetes-native control planes, document probe defaults in the same place you manage patterns such as deploying AWS resources using Crossplane on Kubernetes.

Design health endpoints that match probe intent

The endpoint behind the probe matters more than the YAML. A clean probe configuration still fails if /health does too much work.

A good pattern is to expose separate endpoints:

/livez: checks that the process can respond and has not entered a fatal internal state.
/readyz: checks whether the application can serve traffic right now.
/startupz: checks whether initialization has completed.

For example, an HTTP service might implement behavior like this:

GET /livez
200 OK if the process event loop is responsive
500 only if the process is internally unrecoverable

GET /readyz
200 OK if the service can accept traffic
503 if required dependencies are unavailable or the app is draining

GET /startupz
200 OK after bootstrapping is complete
503 while migrations, cache loading, or state recovery are still running

Here is a minimal Node.js example that keeps liveness local and pushes dependency checks into readiness:

import express from "express";

const app = express();

let started = false;
let shuttingDown = false;

async function checkDatabase() {
  // Replace with a cheap ping or connection-pool status check.
  // Do not run expensive queries here.
  return true;
}

app.get("/startupz", (req, res) => {
  if (started) {
    return res.status(200).send("ok");
  }

  return res.status(503).send("starting");
});

app.get("/livez", (req, res) => {
  // Keep this local. Do not check the database, cache, or third-party APIs.
  return res.status(200).send("ok");
});

app.get("/readyz", async (req, res) => {
  if (shuttingDown || !started) {
    return res.status(503).send("not ready");
  }

  const databaseOk = await checkDatabase();

  if (!databaseOk) {
    return res.status(503).send("database unavailable");
  }

  return res.status(200).send("ok");
});

process.on("SIGTERM", () => {
  shuttingDown = true;
  setTimeout(() => process.exit(0), 10_000);
});

app.listen(8080, async () => {
  // Run startup work here.
  started = true;
});

For worker services without HTTP servers, an exec probe can work, but use it carefully. Every probe spawns a process inside the container. At scale, expensive exec probes can add measurable overhead.

livenessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - test -f /tmp/worker-alive
  periodSeconds: 15
  timeoutSeconds: 2
  failureThreshold: 4

If you use an exec probe, keep the command fast and deterministic. Avoid commands that call external systems, perform file tree scans, invoke package managers, or depend on shells that may not exist in minimal images.

Roll out probe changes with evidence

Treat probe changes like production behavior changes. They can cause restarts, remove pods from service, and change rollout timing.

Use this rollout process:

Measure current startup time. Check container logs, application metrics, and Kubernetes events. Record typical and slow startup durations.
Add or tune startupProbe. Make sure the allowed startup window exceeds slow but valid startup.
Split liveness and readiness endpoints. Keep restart logic separate from traffic-routing logic.
Deploy to a small scope first. Use one environment, one namespace, or one replica set before changing every workload.
Watch events during rollout. Confirm that restarts are not increasing unexpectedly.
Load test or replay traffic if possible. Validate that probes still pass during CPU pressure, garbage collection pauses, and dependency latency.

Useful commands:

kubectl get pods -n app

kubectl describe pod -n app api-7c9f6d8f7f-x2abc

kubectl logs -n app api-7c9f6d8f7f-x2abc --previous

kubectl get events -n app --sort-by=.lastTimestamp

Look for messages like:

Liveness probe failed: HTTP probe failed with statuscode: 503
Back-off restarting failed container
Readiness probe failed: Get "http://10.0.1.25:8080/readyz": context deadline exceeded

If --previous logs show that the application was still starting when it was killed, your startup window is too short or your liveness probe is starting too early. If logs show request latency spikes before liveness failures, your timeout may be too low or the endpoint may share overloaded application threads.

When Kubernetes is part of a larger platform rollout, probe settings should be reviewed with resource requests, limits, rollout strategy, and dependency provisioning. For example, an application deployed on Amazon Elastic Kubernetes Service (EKS) can still fail because a probe ignores slow boot behavior, even if the cluster itself is healthy. The same operational checks apply when you deploy Apache Airflow on AWS EKS or run a custom API service.

Avoid the common liveness probe traps

Most restart loops come from a few repeatable mistakes.

Trap 1: Dependency checks in liveness

If /livez fails when PostgreSQL, Redis, Kafka, or an external API is unavailable, you are asking Kubernetes to restart your application because another system has a problem.

Move dependency checks to readiness:

readinessProbe:
  httpGet:
    path: /readyz
    port: http
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: /livez
    port: http
  periodSeconds: 15
  timeoutSeconds: 2
  failureThreshold: 4

Trap 2: No startup probe for slow applications

initialDelaySeconds can work for simple services, but it is a fixed delay. It does not adapt to real startup progress. A startupProbe lets Kubernetes wait until startup succeeds, then begins liveness checks.

Use startup probes for services with:

Large dependency injection graphs.
Database migrations or schema checks.
Cache warm-up.
Model loading.
State recovery after shutdown.

Trap 3: Timeout too low under CPU pressure

The default timeoutSeconds is 1 second. That can be too low for applications under CPU throttling, garbage collection, or temporary I/O pressure.

If probes fail during load but the application recovers without a restart, increase timeoutSeconds, increase failureThreshold, or make the probe endpoint cheaper. Also check CPU limits. A pod with a tight CPU limit can fail probes because the process cannot get scheduled quickly enough.

Trap 4: Redirects, auth, or middleware on health routes

Health endpoints should avoid authentication middleware, redirects, rate limits, and expensive request logging. The kubelet counts any status from 200 to 399 as success, so 401, 403 and 5xx responses fail the probe. Redirects to the same host are followed; a redirect to a different host is treated as success with a warning event, which can hide a broken endpoint.

Make health routes boring:

Return directly from the application.
Do not require tokens.
Do not redirect HTTP to HTTPS inside the pod unless the probe is configured for HTTPS.
Do not call external services from liveness.

Trap 5: Probe port does not match the container

Named ports reduce mistakes when container ports change:

ports:
  - name: http
    containerPort: 8080

livenessProbe:
  httpGet:
    path: /livez
    port: http

If you use service meshes or sidecars, confirm whether the kubelet probes the application container directly or whether probe rewriting is active. Misconfigured sidecar behavior can make a healthy application look unhealthy.

Fix the common probe error messages

These are the exact strings you will see in kubectl describe pod and kubectl get events, with what they usually mean.

Each common probe error points to a different fix, so read the event message before changing thresholds.

Liveness probe failed: HTTP probe failed with statuscode: 500

The endpoint answered, but with an error. Usually /livez runs dependency checks (database, cache, another service) that are down. Make liveness local and move those checks to readiness. If the app really returns 500 because it is broken, check kubectl logs POD --previous for the cause of the crash before tuning anything.

Liveness probe failed: Get "http://10.0.1.25:8080/livez": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The endpoint did not answer within timeoutSeconds (default 1 second). Common causes are CPU throttling from a tight CPU limit, garbage collection pauses, or the health route sharing a saturated thread pool. Raise timeoutSeconds to 2 or 3, raise failureThreshold, and check CPU throttling metrics.

Liveness probe failed: dial tcp 10.0.1.25:8080: connect: connection refused

Nothing is listening yet, or the probe points at the wrong port. During startup this means the liveness probe started too early: add a startupProbe. Otherwise, check that the probe port matches the container port, ideally by using a named port.

Back-off restarting failed container

The container restarted several times and the kubelet is now waiting with an exponential back-off (up to 5 minutes) before the next attempt. The pod shows CrashLoopBackOff. Find whether the restarts come from probe failures (events say "Container api failed liveness probe, will be restarted") or from the process exiting on its own (exit code in kubectl describe pod).

Readiness probe failed but the container is not restarting

That is expected. Readiness failures only remove the pod from Service endpoints. If the pod never becomes ready, fix the dependency or the readiness logic; restarts will not help.

Use this final checklist before merging

Before you merge a liveness probe change, verify these points:

/livez checks only local process health.
/readyz handles dependency checks and traffic eligibility.
/startupz exists for slow or variable startup.
The startup window is longer than slow but valid startup time.
The liveness restart window is long enough to survive short pauses.
timeoutSeconds is realistic for your runtime and CPU limits.
Health endpoints bypass auth, redirects, and expensive middleware.
Probe failures are visible in logs, metrics, or Kubernetes events.
The rollout plan limits blast radius.

FAQ

Does a readiness probe failure restart the container?

No. Only liveness and startup probe failures restart a container. A failing readiness probe removes the pod from Service endpoints until it passes again.

Does a startup probe failure restart the container?

Yes. If the startup probe keeps failing past failureThreshold x periodSeconds, the kubelet kills the container and restarts it, just like a liveness failure.

What are good default liveness probe values?

A practical starting point is periodSeconds: 10, timeoutSeconds: 2 and failureThreshold: 3, with a startupProbe covering your slowest normal boot. Tune from observed startup time and pause behavior, not copied defaults. The official reference is the Kubernetes guide to configuring liveness, readiness and startup probes.

Should a liveness probe check the database?

No. If the database goes down, every pod fails liveness and restarts at once, which adds load and slows recovery without fixing anything. Check dependencies in readiness.

Can I use the same endpoint for liveness and readiness?

You can, but it is a common source of restart loops. Separate /livez and /readyz endpoints let readiness react to dependencies while liveness stays local.

Liveness probes are useful when they restart containers that cannot recover on their own. They become dangerous when they replace readiness checks, dependency monitoring, or startup handling. Start conservative, separate probe responsibilities, watch real failure events, and tune based on observed behavior rather than copied defaults.

How to Build a DevOps Team

Michael Zion — Sun, 08 Nov 2026 07:30:09 +0000

This guide was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

If you're considering how to build a DevOps team the best way possible, this one's for you.

This post sums up the advice I give people building DevOps teams: CTOs, VPs of R&D, Team Leaders, and more.

You'll get a step-by-step path from philosophy and mission to roadmap, team shape by stage, sizing, and hiring (or bringing in outside help), plus a short FAQ.

TL;DR

I'll give you the bottom line.

These are the steps for building your DevOps team:

Polish your DevOps Philosophy
Understand the DevOps Lead responsibilities: Product, Project, People, Service, Architecture
Define your Team's Mission Statement
Set useful DevOps Goals & Practices
Set Guiding DevOps Principles
Define the DevOps team's roadmap & strategy
Set a definition of done
Common DevOps teams pitfalls
Pick a team shape that fits your stage
Calculate your team's required capacity
Hire the right DevOps Engineers

You can also apply these to existing DevOps teams and level-up your organization.

Let's get started!

Polish your DevOps Philosophy

You must first understand why organizations need DevOps, and how it can work in practice.

DevOps is an enabler role.

It's meant to enable the developers to build, improve, and take ownership over the system.

Let's break it down:

Build = Create something new. Could be a new microservice, a new database, or a new monitoring dashboard.
Improve = Introduce a change into something that exists. Could be fixing a bug, changing a database schema, or changing an alert threshold.
Take ownership = Take charge of problems that arise with what you built and improved. This means when something needs improvement, the owner is the one who does it.

The above are what the developers should do.

Ownership includes on-call: the team that builds a service should be the one paged for it. DevOps owns the platform underneath it, plus the tooling that keeps on-call manageable (actionable alerts, dashboards, runbooks).

Now, DevOps should enable it.

Note: enabling is not giving full permissions and letting them have fun.

To enable is to give developers what they need to build, improve, and own the system.

But do it safely, and in a way that points the developers' energy in the right direction.

Understand the DevOps Lead Responsibilities

You have 5 hats as a DevOps Lead:

Product (Platform)

Your clients are your company's developers.

Provide them with tools, knowledge, and automations.

Tools = Polished automations.

Automations = Automated knowledge.

Knowledge = Hard-earned information & insights.

Understand their requirements, and use Tools, Automations, and Knowledge to fulfill them.

Understand why they want what they want: Is it because of an underlying issue with the system? If yes, solve it before building anything.

If a tool or automation will save time, consider building or implementing it.
Service

Don't let your developers wait until you do something "the right way".

Sometimes they need immediate help to complete something.

Help them first, and invest time later in automation and tooling.
Project

Managing your DevOps team requires managing the work it does.

Turn the team's philosophy into a mission.

Turn the team's mission into goals.

Turn the team's goals into a roadmap.

Turn the team's roadmap into tasks.

Prioritize the tasks.

Set simple roles and responsibilities.

Hold your team members accountable to progression.

Make it easy to inform relevant team members on updates.

Make it easy to consult with a teammate on a subject of their expertise.
People

Each DevOps team member enjoys some types of work more than others.

One enjoys sharing knowledge, another enjoys being of service, and some enjoy building tools.

Different teammates are also interested in different subjects: Monitoring, Infrastructure, CI/CD, etc.

A team member that's working on what they enjoy is more fulfilled and productive.

Strive to overlap each teammate's goals with the company's goals.
Architecture

You have 2 architectures to worry about.

1) The company's product architecture (built by the developers).

2) The DevOps platform's architecture (built by the DevOps team).

Help the developers understand and control the application's effect on the infrastructure.

Enable the developers to make informed decisions by providing context.

Finally, build the platform to support the developers requirements.

At every step, only limit decisions that damage the company.

Examples: High cost, Stability impairment, Restricts observability, etc.

Define Your DevOps Team's Mission Statement

Here's one for you:

"Enable the developers to build, improve, and own the system".

It's pretty minimal, so it's going to help your team stay focused.

A healthy sign it catches on: When the team debates a decision regarding a task, they ask "Does it help us enable the developers to build, improve, or own the system?".

That's when you win.

Set Useful DevOps Goals & Practices

Your company's success will be determined by its speed and its product's quality.

To make it happen, remember that programming is science, not maths. I'll explain.

People used to think programming would be a mathematical discipline.

They thought programmers would write functions and prove them mathematically.

That's not what happened. Programming is a scientific discipline.

You write code, you test it, and you assume it's good until a test fails.

In essence, you're experimenting.

You might ask: "wtf? how's this related to setting useful goals?"

The answer is that the first thing you want to enable is running experiments easily.

So here are some useful goals:

Developers can easily test their code in a consistent manner
Production and testing environments are identical (Production will benefit from the quality of the tests)
Developers can easily collaborate
Developers can understand the state of the system and the impact of changes

Let's translate those goals into smaller goals or practices that will achieve the goals:

Developers can create a testing/production environment with "One-Click"
A continuous integration process lets developers collaborate by agreeing on the current, up-to-date version of the system
Auto-create dashboards and alerts for new services

Set Guiding DevOps Principles

Some useful DevOps principles that help save your team time, improve the speed of delivery, and keep the system healthy:

PoC before doing things "the right way"
Make it work, then make it better
The entire system should be fully recoverable from Git
Use tools with a big community and well-documented interface
Equip key-developers with DevOps knowledge to be the first point-of-contact for their team members (super users)

For more principles with examples, see The CTO DevOps Handbook.

Define the DevOps Team's Roadmap & Strategy

Roadmap = Goals * Strategy.

Once you set the goals (as mentioned above), you can start prioritizing.

#1 - The DevOps Categories

Enabling developers requires a DevOps Engineer to handle and enable the following:

Provision infrastructure
Deploy workloads
Monitor the system
Recover from issues
Scale up and down
Track & test changes (Codebase Management)
Secure the system
Store & retrieve data
Configure the system

#2 - Examine each goal through each category

Every DevOps goal you set should be examined through the lens of each category.

Together, the categories cover every aspect of building, improving, and owning a software operation.

Let's do an example:

Goal: Create a "One-Click Environment Automation"
Categories to address:

How should its infrastructure be provisioned?
How should its workloads be deployed?
How should its metrics and logs be sent, stored, and queried?
...

#3 - Strategic Principles

Spend at least 50% of the team's capacity on DevOps goals as soon as you responsibly can:

Support the developers and teach them how to self-support to get there
Easy to modify > Perfect:

When time forces you to do something imperfectly, do it in a way that's easy to improve later
Prerequisites first:

Codebase Management -> Infrastructure -> Deployment -> Configuration -> Data Management.
At least moderate foundations quickly, reinforce later, but never weak foundations:

If something is a foundation (like infrastructure) and early on you lack boilerplates or experience with it, don't settle for weak foundations. But don't over-invest in strong ones if that takes too long, either.

Instead, moderately invest in the foundation, and revisit it later.

Set a Definition of Done

Think of it as the checklist a component must pass before you call it production-ready.

Ask the following questions for every single component in your system:

Monitoring: Are there metrics, logs, traces, and alerts setup in an actionable way?
Availability: Is there a mechanism to keep it alive during incidents?
Resiliency: Can it recover from an error quickly?
Recovery: Can it be fully restored to a previous state?
Testability: Is it possible to test changes to it?
Deliverability: Is there a process to release changes to it?
Persistency: Will its data persist if the system is hindered?
Integrable: Does it have a consistent and predictable interface allowing integration with it?
Security: Is it accessible only by the parts of the system that absolutely need it?
Dependencies: Are its dependencies fully tracked and managed?

Common DevOps Team Pitfalls

Pitfall: DevOps work blocks developers work

Indicator: Developers need to wait for DevOps team changes to complete before continuing work

Cause: The DevOps team doesn't utilize its own practices (AKA shoemaker's son always goes barefoot)
Pitfall: Only support developers and maintain the system

Indicator: No progress on any DevOps goal or task

Cause: Either there are no clear DevOps goals or repeating developers requests haven't been automated
Pitfall: Adopting 'Best' practices instead of 'Suitable' practices

Indicator: Introducing methodologies and adhering to principles that go against the company's goals

Cause: Prioritizing methodology over company goals, usually because of a disconnect from the company goals or due to lack of DevOps experience

Pick a Team Shape That Fits Your Stage

There's no single right structure, but these rules of thumb fit most startups:

Fewer than ~10 developers: usually no dedicated DevOps hire yet. A developer who enjoys infrastructure, backed by a consultancy or a fractional DevOps engineer for the foundations, is often enough.
Around 10 to 20 developers: time for the first DevOps engineer. Hire someone senior enough to set direction alone.
Beyond that: a small platform team that treats the internal platform as a product, with the developers as its clients.

As you grow, you'll choose between a central platform team and DevOps engineers embedded in product teams. Embedded engineers move fast but build one-off solutions; a platform team builds shared paths but can drift from developers' needs. A common middle ground: a platform team owns the shared platform, and each product team has a super user.

Outside help makes sense when you need foundations fast, can't justify a full-time hire yet, or are still hiring. Insist that everything lives in Git so your team can own it. Here's how to choose a DevOps consulting service.

Calculate Your DevOps Team's Required Capacity

Required DevOps Capacity = (Scale * Complexity) / Leverage.

Leverage = Level of DevOps Engineers * Company Resources * Team Focus
Scale = Number of instances of each component * Number of people in the organization
Complexity = Number of components * Number of teams

Hire the Right DevOps Engineers

The types of DevOps Engineers:

Barrels vs. Ammo

~ Ammo = People who can complete tasks but won't initiate them.

~ Barrel = People who understand what tasks are needed next but won't complete them.

Interviewer Tip: During the technical interview with a candidate ask about past DevOps accomplishments, and ask how the tasks were created and who did them.
Aspiration-Oriented vs. Prevention-Oriented

~ Aspiration-Oriented = Has goals, positive feedback encourages and focuses them while negative feedback discourages them and kicks them off track.

~ Prevention-Oriented = Avoids problems, positive feedback makes them lay back and reduce capacity while negative feedback focuses them and keeps them on track.

Interviewer tip: See if there's a common theme for projects the candidate did in the past. People focused on security tasks and deep attention to specific details are more likely to be prevention-oriented, while people who initiated many projects spanning multiple (DevOps) categories are more likely to be aspiration-oriented.

Working with service providers:

Set clear desired results

And let the DevOps service provider assist in exploring the goal and provide perspective from other companies.
Expect transparency on progress

And judge it against the latest plan.
Expect clear planning

And make sure it has clear goals, takes into consideration the risks, and has a strategy that adheres to your DevOps principles.

Frequently asked questions

How many DevOps engineers do I need?

It depends on scale, complexity, and leverage. As a rule of thumb, one DevOps engineer can support roughly 10 to 20 developers when developers own their services and most work is automated. Less automation or more teams means more capacity; check it with the formula above.

What roles make up a DevOps team?

Early on, one or two generalist DevOps engineers, with someone wearing the five DevOps Lead hats. As the team grows, people specialize by category (infrastructure, CI/CD, observability, security), and some teams add a reliability (SRE) focus.

Should DevOps be a separate team?

Only as a platform team that enables developers to own their services. It shouldn't become a team developers hand work over to, because that rebuilds the wall DevOps was meant to remove. If developers wait on DevOps tickets to ship, you've hit the first pitfall above.

What should be in a DevOps team charter?

The mission statement, the goals and principles that follow from it, and a definition of done. Add clear ownership boundaries: what the DevOps team owns, what developers own, and who is on call for what.

Should I hire DevOps engineers or outsource?

Hire when there's steady, full-time DevOps work and you can attract someone senior enough to work without guidance. Use a consultancy or fractional engineer when you need foundations quickly, lack the expertise to judge candidates, or need to bridge the gap until a hire lands.

Summary

We covered a lot in this one-pager, and still left plenty out.

If you take away 1 thing from it, let it be this: A simple DevOps team mission statement is the most significant thing you need.

It sounds simplistic and abstract, but without it there's no guiding principle for what DevOps in your organization should look like.

Hope you enjoyed, and send me an email at michael@meteorops.com if there's anything else you'd like to see in here.

Calculating your Company's Required DevOps Capacity

Michael Zion — Thu, 05 Nov 2026 07:30:08 +0000

This article was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

The capacity conversation usually starts after the workaround breaks

A client recently asked the platform-engineering question that shows up when a company has grown past the “one senior engineer keeps delivery moving through heroics” stage. A handful of deployment scripts had turned into the operating layer for engineering: CI/CD workflows, cloud account and project structure, Kubernetes and serverless deployment paths, infrastructure-as-code modules, runtime standards, incident support, security guardrails, cost controls, secrets and access patterns, and the enablement work that stops teams from losing days to broken environments, unclear ownership, or knowledge trapped in chat threads.

The budget question was direct: why does this environment need three DevOps engineers instead of one? The useful answer is not “because DevOps is important.” It is capacity math. Start with committed roadmap work, then add the operational load that rarely fits neatly into sprint planning: production escalations, pull request reviews, Terraform drift, failed pipelines, certificate and dependency updates, audit evidence, cost investigations, access requests, policy exceptions, flaky test infrastructure, base image patching, and the steady stream of “quick” developer support. In 2026, that background load is often heavier than the original platform plan assumed: more AI-assisted code reaching review, more short-lived environments, more compliance evidence, more cloud spend scrutiny, and more security checks pushed into the delivery path. If one engineer is expected to review infrastructure changes, unblock releases, tune cluster policies, handle on-call escalations, prepare audit artifacts, investigate cloud spend, and coach product teams through platform changes, staffing is no longer a preference debate. It is a business decision about how much delivery delay, burnout, single-point-of-failure risk, and platform instability the company is willing to carry in the system that ships and runs its software.

Until that moment, the budget conversation had been relatively easy.

If storage utilization is climbing, you can show the graph and justify another few hundred dollars a month for disks.

If CI queues are delaying merges, deployments still require hand-holding, cloud accounts have weak guardrails, incidents are pulling senior engineers out of roadmap work, and product teams are waiting days for platform fixes, the case is harder to express. You are not buying “1.5 DevOps engineers”; you are buying release throughput, operational safety, faster recovery, cost control, compliance readiness, and fewer blocked engineers across the company.

That is the gap these capacity formulas are meant to close. They take symptoms that usually arrive as scattered complaints — slow deployments, rising incident load, a platform backlog that never shrinks, fragile environments, security and compliance requests, cloud cost surprises, and constant support interruptions — and turn them into planning inputs that engineering, product, and finance can discuss without relying on the loudest anecdote in the room.

🛠️ Practical enough for planning, honest about uncertainty

Use the formulas below as planning heuristics, not as a precise model of human work and not as a staffing ratio to copy from another company. A five-team SaaS business, a regulated enterprise, and a small product group built mostly on managed cloud services will make different assumptions about ownership, risk, automation, on-call coverage, compliance effort, lead time, and how much platform complexity they are willing to operate themselves.

The point is not to produce a magic number. The point is to make the constraints visible: where operational pressure is coming from, which teams are absorbing it, which roadmap work is being delayed, and what tradeoff the business is making if it chooses not to hire, reduce scope, standardize tooling, improve self-service, retire complexity, or simplify the platform. A useful capacity model should make the next decision clearer, even if the number is still an estimate.

I call these formulas because they create a repeatable way to reason about capacity, not because engineering work can be measured with mathematical precision.

DevOps and platform work is shaped by interruptions, context switching, unclear ownership, production risk, and organizational friction. A useful model has to be simple enough for engineering managers, platform leads, and executives to use together, while still being honest enough to show where the pressure is really coming from.

Decisions this capacity model should help you make

Help you understand how many DevOps Engineers you need
Provide you with tools to reason with your CFO
Encourage you to learn martial arts to successfully defeat your CFO in combat if all else fails

🤷 Caveats

This model is not written in stone; I’m open to learning from your experience and improving it based on what you see in real engineering organizations.
Calibrate the formulas before using the result
Most of the formulas use inputs you can count: developers, services, environments, pipelines, incidents, and support channels. Those numbers are useful, but they do not explain the whole capacity picture. Qualitative factors such as ownership clarity, automation coverage, documentation quality, supportability, and maintenance debt can change the real capacity requirement as much as raw volume does. Ten services with consistent deployment patterns may be easier to support than three services with bespoke pipelines, unclear runbooks, and no obvious owner.
DevOps capacity is not the same as one full-time DevOps engineer

Use the formulas without fooling yourself

Start by identifying what your current DevOps capacity can reliably support, then treat that as the baseline unit. For example:

160 hours of DevOps engineering per month = 20 microservices, 3 development teams, a defined release cadence, a known incident load, and a specific amount of platform support.

The variables will differ by company, and the numbers only matter if they reflect real work. The point is to stop treating “one DevOps engineer” as a generic planning unit and instead measure the operating load that a given amount of capacity is carrying.
As the company grows, look for indicators the existing capacity isn’t able to handle more than it does and hire more DevOps capacity

🚨 The indicators that more DevOps capacity is required

There are 2 main indicators that more DevOps capacity is required:

Support work is useful until it becomes the platform interface.

Some support load is healthy. When the company is growing, the queue often shows where developers are getting stuck: unclear deployment paths, missing documentation, brittle environments, confusing access flows, or platform defaults that do not match real use.

The failure mode is different. If the same questions, access requests, deployment failures, and environment fixes keep coming back, support has stopped being discovery. It has become capacity leakage. At that point, DevOps-to-developer efficiency improves only when recurring support is converted into self-service, safer defaults, clearer ownership, or automation that removes the ticket altogether.
Different engineers and teams are solving the same problems in different ways for themselves.

This often starts when DevOps or platform hiring lags behind demand and developers cannot wait for every environment, pipeline, permission, or deployment issue to be handled centrally. Some local problem-solving is healthy, but it becomes expensive when every team builds its own scripts, Terraform patterns, CI templates, dashboards, or access workflows in isolation.

It is acceptable for product teams to solve DevOps use cases when those solutions become contributions to a shared platform or a documented standard. It is a warning sign when the same problem is being solved five different ways and none of those solutions are reusable, supportable, or owned.

Example client indicators:

Company #1 example:

A company with roughly 300 developers had about 40 DevOps engineers spread across stateless infrastructure, data infrastructure, monitoring, CI/CD, and data streaming. Each group owned a Slack support channel. Over time, those channels became the default front door for pipeline failures, infrastructure questions, access requests, environment issues, and production follow-ups.

Capacity signal: Several DevOps teams needed 1–3 engineers available mainly for support rotation. That was not just an on-call coverage issue. It showed that platform engineers had become a manual intake layer for work that should have been automated, documented, delegated, or exposed through self-service. The visible symptom was “too many questions in Slack”; the underlying capacity problem was that routine delivery work still required a specialist to interpret, approve, or execute it.

What changed: The teams reviewed recurring requests every week, grouped the highest-volume patterns, automated repeatable tasks, and created self-service paths for common developer needs. They also paired with key developers to move platform knowledge into product teams. The capacity gain did not come from asking DevOps engineers to work faster. It came from removing avoidable handoffs and making routine work executable without waiting for an answer in Slack.
Company #2 example:

A company with 30 developers had one DevOps engineer supporting delivery, infrastructure, and developer requests.

Capacity signal: The DevOps engineer spent most of their time on support tickets opened by developers. As common deployment errors kept recurring, some developers began bypassing the standard deployment process to keep work moving.

What changed: The company added monthly capacity through a part-time DevOps consultant and trained key developers to provide first-line support inside their teams. That combination reduced the single-person bottleneck without immediately turning every platform problem into a full-time hiring decision.

With those caveats in place, here is the planning model for estimating DevOps capacity.

A practical way to estimate DevOps capacity

A useful way to estimate required DevOps capacity is to start with the work the platform team must absorb, then adjust for risk, complexity, and the level of self-service already in place:

Why this works as a planning model

The bigger the stuff to build and maintain - the more working hands you need
The more variations you need to support - the more working hands you need
The more self-service you have, the fewer hands-on interruptions you should need

Let’s dive into the ingredients of scale, complexity, and leverage.

The baseline: what “DevOps capacity” actually includes

The guiding principle here is simple: a DevOps engineer should enable delivery, not become a permanent checkpoint for every delivery task.

That means enabling teams to provision infrastructure, deploy workloads, store and process data, observe services, and recover from incidents with as little unnecessary handoff as possible. When every change still requires a DevOps engineer to approve, debug, click, or explain the same thing again, capacity is being consumed by friction rather than leverage.

DevOps = Enable ownership of the relevant stakeholders over the infrastructure, monitoring, security, architecture, data, configuration, deployment, orchestration, testing, and development.

‘DevOps Capacity’ is the time & effort invested in achieving “DevOps” as defined above.

⛰️ Scale - Size Matters

Start by defining the scale you are asking the platform to support. In capacity planning, “scale” is not only traffic volume. It also includes the number of services, environments, deployment paths, cloud accounts or projects, runtime patterns, data systems, compliance boundaries, and teams that depend on the platform.

“Michael, did you just create a recursive definition?”

Why yes of course! this is the rationale:

Larger systems require more time and effort to understand, build, secure, operate, and maintain.
Larger teams and client bases create more internal and external use cases to support, which increases the load on platform, infrastructure, and release processes.

To make the two types of scale above easier to discuss, I use two simple capacity models.

They are not scientific laws. They are structured ways to compare the demand created by your systems, teams, and operating model with the DevOps capacity your organization actually has:

Map the system scale and operational surface area

Resources Used could mean many things, some of them are: Storage volume, Number of CPU cores, Number of nodes, Memory used, etc.
System throughput is the work your systems process over time: user requests, scheduled jobs, batch runs, queue events, streaming workloads, and any other flow that creates operational load. Throughput matters because each flow tends to bring its own deployment risk, scaling pattern, observability requirement, failure mode, ownership boundary, and support expectation. A quiet batch job that fails every morning can consume more DevOps attention than a high-traffic service with mature dashboards, alerts, and rollback paths.
Instances per Service refers to the number of replicas of the services developed within your company, for example: Kubernetes Pods, serverless functions executions, etc.

A client's system scale example: 1-year-old startup

Scale: 3 AWS ECS clusters, 6 microservices, 3 main environments, 3–150 ephemeral development environments, and 6–40 container instances per environment

Approach: Infrastructure and deployments are managed with infrastructure as code. The company hired a part-time DevOps consultant to implement the initial system and train the development team to own routine changes, instead of creating a permanent dependency on the consultant for every deployment or environment issue.

Organization Scale

Number of Developers refers to the number of developers, and shame on you for bothering yourself with reading this line
Total Users refers to the number of active clients_

Noteworthy:

B2C will usually have more clients, with more standardized interfaces - DevOps Engineers most likely will be less involved in direct ways with client facing features.

B2B on the other hand will usually have less clients, with more customized interfaces per client - DevOps Engineers are more involved with client-facing features through the developers._

A Client's Organization Scale Example - Healthcare Company:

Scale: 200 developers divided into teams of 4–12 people, 10 DevOps engineers and SREs, high autonomy per team, and no single standardized process for all developers

Approach: Infrastructure and deployments are managed with infrastructure as code to support “one-click environments.” The company also used a part-time DevOps consultant to implement an initial system and train the development team to take ownership of it.

Bonus: Check our availability for a call about scaling DevOps in your organization in a more gradual way using MeteorOps.

🥴 Complexity - More Options, Less Clarity

I know, I know… It looks similar to the chapter about Scale.

In this context, complexity is not only the number of systems you run. It is the number of choices engineers must make without strong defaults, and the uncertainty around which path is considered safe, supported, and maintainable. That uncertainty creates decision latency and makes it more likely that two teams solve the same problem in different, incompatible ways.

You are operating in a complex environment when an engineer knows what outcome they want, but still has to ask, “Is this how we do it here?”

So, what does complexity have to do with both the system and the organization?

More complex system - More time and effort are required to map the available options, and customize (copy, paste, modify) or generalize (refactor, parameterize) solutions
More complex organization - More time and effort are required to communicate, provide support, cascade initiatives through the company, and lower likelihood of initiatives getting accepted

This time I’ll present you with the formulas for complexity without drumrolls:

System Complexity

Number of Tools refers to any tool, either existing or custom-built, that helps build and maintain the system. More tools mean more capabilities, more potential integrations between tools, and more potential ways to achieve the same goals. (Tools have an upside as well!)
Number of Platforms means any platform on which you run workloads or consume resources (e.g. - AWS, Kubernetes, Linux, Jenkins). Using more platforms requires supporting more ways of running each workload or consuming each resource.
Number of Codebases refers to the repositories where your systems are developed and maintained. The more separate codebases you operate, the harder it becomes to keep development workflows consistent: build conventions drift, CI/CD pipelines fork, dependency upgrades happen unevenly, and release practices become harder to support at scale. A larger number of repositories does not automatically mean you need more DevOps capacity, but it does increase the value of shared templates, standards, and automation that prevent every team from reinventing the same delivery path.
Number of Services is the number of services developed by the developers: more services, more different units with different logic to them, which in turn creates more unique operational requirements.

To sum up, ‘System Complexity’ increases when there are more services, running on more platforms, managed by more tools, across more codebases.

A Client's System Complexity Example - Low Scale, High Complexity:

Complexity: The entire system was 1 service deployed on just 1 server, but was deployed using multiple deployment tools (chef, ansible, bash scripts), from multiple different repositories (about 10), with a specific set of steps required to fully set it up
Approach: Containerize everything, Consolidate the deployment to one docker-compose file, Consolidate the configuration and automation to one tool (ansible), Consolidate the repositories into one

Organization Complexity

Number of Teams - More teams mean supporting and maintaining more ways of work
Number of Hierarchy Levels - More middle-management layers in the organization make it more likely decisions and communication will cascade through it, and so increasing the time & effort for decision-making and communication
Level of Hierarchy-Reliance for Communication - Exactly the same effect, but focused on the culture of the company

🪄 Is there anything that makes DevOps capacity more efficient?

Calculating DevOps capacity is not just a headcount exercise. Done well, it exposes where delivery is actually getting stuck: repeated support requests, fragile release paths, manual environment provisioning, unclear ownership, noisy incident handoffs, and the morale cost of constant interruption.

The better planning question is not “how many DevOps engineers do we need?” It is “which constraint should the next unit of capacity remove?” In many organizations, adding another person to absorb tickets only makes the bottleneck quieter for a while. The higher-leverage move may be self-service environment creation, paved-road CI/CD templates, standardized cloud and Kubernetes defaults, documented escalation paths, workable access flows, or clearer ownership boundaries that show when a product team can act independently and when platform help is required. If the discussion is stuck in anecdotes, a structured DevOps maturity assessment can give teams a shared language for deciding what to automate, what to standardize, what to delegate, and where specialist DevOps or platform engineering capacity still matters. The goal is not to keep the platform team busy. It is to make the same capacity support more teams, safer releases, and faster recovery without turning that team into the permanent approval queue for every deployment, pipeline change, cloud request, access issue, or post-incident follow-up.

Valid question! And this is the formula for DevOps ‘Leverage’:

Documentation Coverage - Out of the internal company’s tools and processes, what percentage has documentation?
Automated Processes Coverage - Out of the manual processes used by the developers, what percentage was automated?
Engineering Talent Concentration Score - The seniority (relevant experience) of your engineering team, divided by the acceptance rate of engineers to your company.

More experience and lower acceptance rates improve the score.
Tools Community Size - How widely adopted are the tools being used by the company? And how active is the open community working on and with it?

Example:

Another client of ours had all of its infrastructure managed manually.

To increase the documentation-coverage and the automated-processes-coverage, it decided to adopt an infrastructure-as-code tool.

They considered Terraform and Pulumi, and ended up choosing Pulumi, in order to avoid hiring more full-time DevOps Engineers, and delegate responsibilities for infrastructure management to the developers.

It increased the documentation-coverage by the sheer fact that all of the infrastructure is now described in a Git repository.

It increased the automated-processes-coverage because every piece of new infrastructure was introduced into Git, would exist in every new environment, and would get automatically reconciled if its state drifts.

And the formula for the score (One last formula, I promise):

Note that this variable does NOT say “DevOps Engineering Talent Concentration Score”, because the assumptions are that:

Every engineer on the team can take part in filling the DevOps capacity - it doesn’t have to be a specific team or person
More experienced engineers require less training and support, and so consume less time for the DevOps Capacity

Example:

MeteorOps hires 7 engineers out of every 1,000 vetted.

We only vet DevOps Engineers with extensive production experience in at least two previous companies.

Our median seniority is 8 years of experience.

It means the ETCS (Engineering Talent Concentration Score) is 114.

🧠 Engineering Talent Concentration Score - The biggest leverage you can get

If you can take only one thing from this article, make it the engineering talent score.

Having a DevOps candidates pool of engineers with relevant experience, and hiring only the top percentile of that pool, is an insanely high-leverage action.

You get engineers with perspective on how to do things right, and perhaps more importantly, how to do things wrong.

Enjoying someone’s learning from successes, and learning from failures, saves your company lots of time learning the hard way.

External DevOps help can be useful, but it should be framed as a capacity and capability decision, not a way to postpone operating-model choices. A good partner reduces bottlenecks by building reusable platform patterns, documenting how the system is supposed to run, tightening delivery and runtime practices, and transferring knowledge back to the teams that will own the work later. If the engagement simply adds more hands to the same ticket queue, you may buy short-term relief while the underlying capacity problem keeps compounding. Use outside support when it is tied to a concrete outcome: stabilizing delivery, creating self-service infrastructure, improving cloud or Kubernetes operations, lowering incident load, or covering a specialist gap while you hire, train, or reorganize internally.

I’m not going to do that.

(P.S. - I’m the Marketing Director)

To Summarize

Instead of writing more words, here are all of the formulas:

I hope you find it useful!

I’m also interested in learning about how you used he formulas, and what insights you had as a result.

If you came up with new useful variables and formulas, please send it to me so I can take credit over inventing them.

If your CFO isn’t convinced

Terraform Project Structure: 3 Layouts That Scale

Michael Zion — Wed, 04 Nov 2026 07:30:09 +0000

This guide was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

A good Terraform project structure keeps reusable modules separate from the root modules that deploy them, gives every environment and major component its own state file, and pins provider and module versions. Start with one directory per environment, split state by blast radius as you grow, and add Terragrunt only when duplication actually hurts.

Most guides stop at "put variables in variables.tf". That is the easy part. The decisions that matter are how you split state, how environments differ without copy-paste drift, how changes are reviewed and applied, and when to change the layout as the team grows. This guide covers all of them with three reference layouts you can copy, sized for teams of roughly 1 to 5, 5 to 20, and 20 to 50 engineers.

Terraform project structure basics: files and modules

Inside any single Terraform directory, follow HashiCorp's Terraform style guide file names. They are boring on purpose: anyone opening the directory knows where to look.

File	What goes in it
`terraform.tf`	The `terraform` block with `required_version` and `required_providers`
`providers.tf`	All `provider` blocks and their configuration
`backend.tf`	The backend configuration (where state lives)
`main.tf`	Resource and data source blocks (split into topic files such as `network.tf` or `iam.tf` when it gets long)
`variables.tf`	Variable blocks, in alphabetical order
`outputs.tf`	Output blocks, in alphabetical order
`locals.tf`	Local values
`.terraform.lock.hcl`	Provider dependency lock file; commit it

The more important distinction is between two kinds of directories:

Child modules are reusable building blocks (a VPC, a database, a service). They take variables, create resources and return outputs. They have no backend and no provider configuration of their own.
Root modules are the directories you actually run terraform plan and apply in. Each root module has its own backend and therefore its own state file. A root module is a unit of deployment, and its state is its blast radius.

Everything else in this guide follows from that: your directory layout is really a decision about how many root modules you have and what each one owns.

Each root module owns one state file, so your layout decides how much one plan can break.

Three Terraform project layouts compared

Layout	Best for	State split	Main risk
1. Directory per environment	1 to 5 engineers, one cloud account per environment	One state per environment	Every plan touches everything in that environment
2. Modules plus live, split by component	5 to 20 engineers, several services and accounts	One state per environment per component	Copy-paste of backend and provider config across directories
3. Terragrunt hierarchy	20 to 50 engineers, many accounts and regions	One state per unit, generated backends	An extra tool and more indirection to learn

Layout 1: one directory per environment

This is the layout HashiCorp recommends when you are not using HCP Terraform workspaces: each environment gets its own directory and state, and shared code lives in local modules.

infra/
  modules/
    network/
      main.tf
      variables.tf
      outputs.tf
    service/
      main.tf
      variables.tf
      outputs.tf
  environments/
    staging/
      backend.tf
      providers.tf
      terraform.tf
      main.tf          # calls ../../modules/network and ../../modules/service
      terraform.tfvars
    production/
      backend.tf
      providers.tf
      terraform.tf
      main.tf
      terraform.tfvars

It is easy to read, easy to review, and each environment can only break itself. The cost shows up as the environment grows: a single production state with a few hundred resources means slow plans, lock contention when two people need to change unrelated things, and a scary diff every time.

Layout 2: modules plus live, split by component

When plans get slow or different people own different parts of the infrastructure, split each environment into components with their own state. The common split is by layer, ordered by how often things change.

infra/
  modules/                 # versioned building blocks
    vpc/
    eks-cluster/
    rds-postgres/
    service/
  live/
    staging/
      network/             # VPC, subnets, NAT, DNS zones
      data/                # databases, caches, buckets
      cluster/             # Kubernetes cluster and add-ons
      services/
        api/
        worker/
    production/
      network/
      data/
      cluster/
      services/
        api/
        worker/
  global/
    iam/                   # org-wide roles, SSO, CI identities
    dns/                   # registrar-level and shared zones

Each leaf directory is a root module with its own backend key, such as production/data/terraform.tfstate. Changing a service no longer locks or plans the network. Losing or corrupting one state file affects one component. Ownership maps cleanly to CODEOWNERS rules on directories.

Layout 3: Terragrunt with an account, region and environment hierarchy

With many accounts and regions, layout 2 turns into dozens of nearly identical backend.tf and providers.tf files. Terragrunt removes that duplication by generating them from a hierarchy of configuration files.

live/
  root.hcl                  # remote state and provider generation, shared by all units
  prod-account/
    account.hcl             # account id, account name
    eu-west-1/
      region.hcl
      production/
        env.hcl
        network/terragrunt.hcl
        data/terragrunt.hcl
        cluster/terragrunt.hcl
  staging-account/
    account.hcl
    eu-west-1/
      region.hcl
      staging/
        env.hcl
        network/terragrunt.hcl
        data/terragrunt.hcl
        cluster/terragrunt.hcl

A unit's terragrunt.hcl points at a versioned module and supplies only its inputs and dependencies. Terragrunt can also run many units in dependency order. We published a ready-to-use example in our Terraform starter boilerplate for GCP using Terragrunt.

Adopt Terragrunt when the duplication is real, not on day one. A team of three with two environments gets more confusion than value from it.

What about workspaces and Terraform Stacks?

Two features often come up as alternatives to directories:

CLI workspaces (terraform workspace new staging) keep several states for one configuration in one backend. HashiCorp's workspace documentation says they are not a suitable isolation mechanism when deployments need different credentials and access controls, and are not a tool for system decomposition. Use them for short-lived copies (a feature branch test), not for staging versus production.
HCP Terraform workspaces are a different thing: each is a separate state with its own variables, permissions and runs. If you use HCP Terraform, one workspace per environment and component is the recommended pattern.
Terraform Stacks in HCP Terraform let you define components once and deploy them to many environments, regions or accounts. HashiCorp announced general availability at HashiConf 2025. Consider them if you are already on HCP Terraform and repeat the same infrastructure across many deployments.

How to split Terraform state by blast radius

The right number of state files is the smallest number that keeps each one safe to change. These are the boundaries that matter, in roughly the order teams need them:

Environment. Production and non-production never share a state, and ideally never share a cloud account.
Account or project. One state per cloud account or GCP project keeps credentials simple.
Change frequency. Networking changes a few times a year; services change weekly. Do not make a service deploy plan the VPC.
Data versus stateless resources. Databases and buckets deserve their own state with prevent_destroy on critical resources, so a refactor in a service directory cannot touch them.
Ownership. If two teams change different resources, give them different states so they do not block each other's locks or reviews.

Add state boundaries in this order as the team and infrastructure grow.

Signals that a state is too big: plans take more than a couple of minutes, unrelated changes queue on the lock, or reviewers stop reading the plan output. Signals that you split too far: every change needs coordinated applies across five directories, or you spend more time wiring outputs between states than writing resources.

Remote state, locking and backend configuration

Never keep state on a laptop or in Git. Use a remote backend with locking, encryption and versioning turned on.

Cloud	Backend	Locking
AWS	`s3` with bucket versioning and encryption	S3 lock file with `use_lockfile = true` (Terraform 1.10 and later); DynamoDB locking is deprecated
Google Cloud	`gcs` with object versioning	Built in
Azure	`azurerm` storage account	Built in, using blob leases
Any	HCP Terraform, Spacelift, env0, Scalr	Managed by the platform

An S3 backend for the layout 2 example looks like this:

terraform {
  backend "s3" {
    bucket       = "acme-terraform-state-prod"
    key          = "production/data/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true
  }
}

The S3 backend documentation notes that locking is opt in and that DynamoDB-based locking is deprecated. If you are migrating, you can set both during the transition and run terraform init -reconfigure.

Keep one state bucket per account (or at least separate production), restrict who can read it (state contains secrets such as generated passwords), and turn on versioning so you can recover from a bad write.

Sharing values between states

Once state is split, components need each other's values: the cluster needs subnet IDs, services need the database endpoint. You have three options:

Data sources that look up real resources by tag or name (for example data "aws_vpc"). Loosely coupled and our default.
Published parameters: one component writes outputs to AWS Systems Manager Parameter Store or a similar store, and others read them. Explicit and permission-controlled.
terraform_remote_state: reads another state's outputs directly. Simple, but it requires read access to the whole state file, which can expose secrets.

Prefer lookups and published parameters; remote state reads expose the whole state file.

Module versioning and pinning

Pin everything that can change underneath you:

terraform {
  required_version = ">= 1.10"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"
  # ...
}

module "api" {
  source = "git::https://github.com/acme/terraform-modules.git//service?ref=v2.4.1"
  # ...
}

Registry modules take a version argument; Git modules use ?ref= with a tag.
Commit .terraform.lock.hcl so every run uses the same provider builds.
Version your own modules with tags and a changelog once more than one root module uses them. Upgrade staging first, then production.
Local modules (source = "../../modules/vpc") cannot be versioned. That is fine in layout 1, but it means every environment gets every module change at once.

Naming and conventions that keep a codebase readable

Use snake_case for resource names and do not repeat the resource type (aws_s3_bucket.assets, not aws_s3_bucket.assets_bucket).
Put environment, service and owner in default tags or labels at the provider level, so every resource is attributable for cost and incidents.
Give variables types, descriptions and validation; avoid huge untyped map(any) inputs.
Keep environment differences in .tfvars or inputs (instance sizes, replica counts), not in conditional logic scattered through modules.
Run terraform fmt and terraform validate in CI, plus a linter such as TFLint and a policy scanner such as Checkov or Trivy.

CI/CD for Terraform: plan on pull request, apply on merge

Your structure only works if changes flow through review. The pattern that holds up at every size:

A pull request triggers terraform plan for every root module whose files (or whose modules) changed.
The plan output is posted to the pull request for review.
Merging to main applies the reviewed plan, with a manual approval for production.
CI authenticates with short-lived OpenID Connect (OIDC) credentials, not stored cloud keys. Plan jobs get read-only roles; apply jobs get write roles limited to their account.

Reviewers approve the exact plan that gets applied, and CI never holds long-lived keys.

A minimal GitHub Actions plan job for one root module:

name: terraform-plan
on:
  pull_request:
    paths:
      - "infra/live/production/data/**"
      - "infra/modules/rds-postgres/**"

permissions:
  id-token: write
  contents: read
  pull-requests: write

jobs:
  plan:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: infra/live/production/data
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-plan
          aws-region: eu-west-1
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init -input=false
      - run: terraform plan -input=false -no-color -out=tfplan

Hand-written path filters break down past a few dozen root modules. At that point use a tool built for it: Atlantis or Digger (self-hosted, pull request driven), Terragrunt's run commands, or a managed platform such as HCP Terraform, Spacelift or env0.

Refactoring an existing Terraform codebase safely

Most teams reading this already have a structure they want to change. Terraform has declarative tools for this, so you rarely need to edit state by hand:

moved blocks rename resources or move them into modules within the same state, and show the move in the plan.
import blocks (Terraform 1.5 and later) bring existing resources under management, which is how you move a resource into a new state: import it in the new root module, then remove it from the old one.
removed blocks (Terraform 1.7 and later) drop resources from state without destroying them, using lifecycle { destroy = false }.

Split one component at a time, and confirm that both the old and new root modules plan with no changes before moving on. Back up the state files first; with bucket versioning you also get a rollback path.

Import first, remove without destroying, and only move on when both sides plan clean.

Common Terraform structure mistakes

One giant state for everything. Slow plans, lock queues, and a production database in the same blast radius as a DNS record.
Copying environments instead of calling modules. Staging and production drift apart within weeks.
Using CLI workspaces for production isolation. Same backend, same credentials, one wrong terraform workspace select away from an incident.
Unpinned providers and modules. A routine plan suddenly shows changes nobody made.
Applying from laptops. No review, no audit trail, and state drift when two people apply at once.
Deeply nested modules. Modules that call modules that call modules make plans unreadable. Keep it to one or two levels.
Click-ops fixes during incidents that never get imported. Reconcile drift weekly with scheduled plans.

Terraform project structure checklist

Child modules and root modules live in separate directories (or repositories).
Production has its own state, credentials and ideally its own cloud account.
Each state covers one environment and one component with a clear owner.
Remote backend with locking, encryption and versioning.
Terraform, provider and module versions pinned; lock file committed.
Plans run on pull requests; applies happen from CI after review.
CI uses OIDC roles, read-only for plan.
Default tags or labels on every resource.
Scheduled drift detection.

Terraform project structure FAQ

What is the best folder structure for Terraform?

Separate reusable modules from root modules, and give each environment its own root module and state. Small teams can use one directory per environment; growing teams split each environment into components such as network, data, cluster and services. Add Terragrunt when backend and provider configuration is duplicated across many directories.

Should I use a monorepo or multiple repositories for Terraform?

A monorepo is simpler for small teams because modules and environments change together in one pull request. HashiCorp's style guide recommends separate repositories for modules once they are shared widely, because tags and access control work per repository. Many teams keep a modules repository and a live infrastructure repository.

Should I use Terraform workspaces for environments?

Not CLI workspaces. They share one backend and credentials, which HashiCorp's own documentation says is unsuitable when deployments need separate access controls. HCP Terraform workspaces are different and are a good way to separate environments if you use that platform.

How many resources should one Terraform state hold?

There is no fixed number. Split a state when plans become slow, when unrelated changes wait on the same lock, or when critical data resources share a state with frequently changed services. Many healthy states hold tens to a few hundred resources.

When should I use Terragrunt?

Use Terragrunt when you manage many environments, accounts or regions and keep copying the same backend, provider and input configuration. For a small team with two environments, plain Terraform with a directory per environment is easier to understand.

How do I share outputs between Terraform states?

Prefer data sources that look up resources by name or tag, or publish values to a parameter store. The terraform_remote_state data source also works but requires read access to the other state file, which may contain secrets.

Does this structure work with OpenTofu?

Yes. OpenTofu uses the same configuration language, module and backend model, so the layouts, state splitting and CI pattern apply unchanged. Check feature-level differences such as backend options against the OpenTofu documentation.

For the bigger picture of where Terraform fits next to CI/CD, GitOps and observability tools, see our guide to DevOps tools by category. If you want senior engineers to restructure an existing Terraform codebase with your team, book a free DevOps setup consultation.

On-Call Rotation: How to Design a Schedule for Small Teams

Michael Zion — Tue, 03 Nov 2026 07:30:10 +0000

This guide was written by Arthur Azrieli, CTO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

An on-call rotation is a schedule that names one engineer (the primary) to answer production pages for a fixed period, usually one week, with a secondary as backup and a clear escalation path above them. For a small team, the right rotation is weekly primary and secondary, paging only on customer impact, with written handoffs and runbooks.

You do not need a Site Reliability Engineering (SRE) team to run on-call well. You need named owners, a schedule people can actually sustain, alerts that only fire when a human must act, and an escalation policy that gets the right person involved without a debate at 3 a.m. This guide covers each of those for teams of 3 to 15 engineers: rotation types, team-size math, escalation rules, paging hygiene, handoffs, compensation and tooling, plus a schedule template you can copy.

On-call rotation basics: primary, secondary and escalation

Every working rotation has the same three layers, whatever tool you use:

Primary on-call: receives every page first, acknowledges it, and owns the first response: triage, mitigation, and calling for help.
Secondary on-call: is paged automatically if the primary does not acknowledge in time, and joins any incident that needs a second pair of hands.
Escalation contact: usually an engineering lead or CTO, who joins for customer-facing outages, security incidents, and decisions with business risk such as rolling back a migration or putting the product into read-only mode.

A shift is the period one person holds the primary pager. The rotation is the order in which people take shifts. The schedule is the rotation laid out on a calendar with overrides for holidays and sick days. When people search for "on call schedule" they usually want the last one, so there is a template further down.

Rotation types compared

Pick the rotation type from your team size and where your engineers live, not from what a large company does. These are the patterns that work in practice:

Rotation type	How it works	Good for	Watch out for
Weekly primary only	One person holds the pager for 7 days, then hands over	Teams of 3 to 4 with low page volume	No backup when the primary is unreachable; burnout if pages are frequent
Weekly primary and secondary	This week's secondary becomes next week's primary	Most teams of 4 to 10	Each person holds a pager 2 weeks out of every N
Business hours plus best effort	Formal paging only during working hours; nights are best effort or limited to SEV1	Very small teams, B2B products with daytime usage	Must be agreed with leadership and reflected in customer commitments
Weekday and weekend split	One person covers Monday to Friday, another covers the weekend	Teams where weekend load is low but non-zero	Weekend shifts feel heavier; rotate them fairly
Follow the sun	Two or three regions each cover their own daytime, 8 to 12 hours each	Teams spread across time zones with at least 2 to 3 people per region	Handoff quality between regions becomes the main failure point
Service-based rotations	Each team carries its own pager for the services it owns	Organizations with several product teams and clear ownership	Needs a cross-team escalation path for shared infrastructure

Default recommendation for a startup: weekly primary and secondary, handover on a weekday morning (never Friday evening or Monday 9 a.m.), with paging limited to customer impact. Move to follow the sun only when you genuinely have people awake in another region.

How many engineers you need for a sustainable rotation

Rotation math is simple, and it is the most common thing teams get wrong. With a weekly rotation of N people, each person is primary one week in N. Add a secondary and each person holds a pager 2 weeks in N.

Engineers in rotation	Primary weeks per year (each)	Pager weeks per year with secondary (each)	Verdict
2	26	52	Not sustainable for 24/7. Use business hours only.
3	about 17	about 35	Fragile. One vacation breaks it.
4	13	26	Workable if pages are rare.
6	about 9	about 17	Comfortable for most startups.
8	6.5	13	Healthy for 24/7 from one site.

Google's SRE book gives two reference points that hold up well outside Google: on-call should take no more than 25% of an engineer's time, and a 24/7 rotation from a single site needs at least eight engineers (six per site for a two-site follow-the-sun setup). It also suggests a ceiling of about two incidents per 12-hour shift, because each real incident takes hours once you include the follow-up. See Being On-Call in the Google SRE book.

If you are below those numbers, that is normal for a startup. Compensate by narrowing what pages (customer impact only), shortening the paging window, and widening the rotation: include every engineer who ships to production, not just the "infra person".

Severity levels and paging rules

Paging rules decide whether on-call is tolerable. Agree on severity before the first bad night, then map each level to a response:

Severity	Example	Response
SEV1	Checkout, login or the core API is down for most users; data loss risk; security incident	Page primary immediately, 24/7. Escalate to secondary after 5 minutes without acknowledgement.
SEV2	Partial degradation, one region or a subset of customers affected, a critical background job stopped	Page during the agreed coverage window; otherwise notify the on-call channel.
SEV3	Elevated errors below customer impact thresholds, a flaky job, a degraded internal tool	Ticket for the next working day. Never pages at night.
SEV4	Noisy alert, cleanup, known bug	Backlog.

The 5 minute acknowledgement for user-facing services and around 30 minutes for less time-critical systems match the response targets in the Google SRE book. Pick numbers your team can meet consistently; a promise of 5 minutes that is missed every second weekend is worse than an honest 15.

Every page should pass three tests: it reflects customer impact or imminent risk of it, the on-call engineer can take a concrete action, and the alert carries enough context to start (service, environment, severity, dashboard link, runbook link, recent deploys). "CPU above 80% for 5 minutes" usually fails all three. "Login error rate above 2% for 5 minutes in production" passes. If your team already ignores alerts, fix that before building the rotation; our guide to handling alert fatigue covers how.

An alert that fails any of the three tests belongs in a ticket or the backlog, not on someone's phone at night.

Escalation policy example

An escalation policy is the ordered list of who gets paged, and when, if nobody acknowledges. Write it down in your paging tool and in the runbook. A good default for a small team:

Escalation policy: production-critical
Level 1: current primary on-call
         page by push + SMS + phone call
         escalate if not acknowledged in 5 minutes
Level 2: current secondary on-call
         escalate if not acknowledged in 10 minutes
Level 3: engineering lead (or CTO)
Repeat:  restart at level 1 up to 2 times

Escalation policy: business-hours
Level 1: current primary on-call, push notification only
         Mon to Fri, 09:00 to 18:00 local time
Outside hours: post to #oncall, create a ticket

Also write down when the primary should escalate by choice, not just by timeout: the incident affects paying customers, the failing component is not clear after 15 minutes, a rollback risks data inconsistency, the fix needs access the primary does not have, or the same alert keeps firing after mitigation. Escalating early is a skill, not a failure.

Each level only gets paged when the one before it stays silent, so a missed page never ends with nobody responding.

On-call schedule template

Copy this for a six-person team running weekly primary and secondary, with handover on Wednesday at 10:00. Wednesday handovers keep both people at work for the handoff and avoid starting a shift on a weekend.

Week starting (Wed 10:00)	Primary	Secondary	Notes
Week 1	Engineer A	Engineer B
Week 2	Engineer B	Engineer C
Week 3	Engineer C	Engineer D	Release freeze, extra care
Week 4	Engineer D	Engineer E
Week 5	Engineer E	Engineer F	Public holiday Thursday: override to Engineer A
Week 6	Engineer F	Engineer A

Rules that keep it fair: swaps are allowed if both people agree and the tool is updated, holidays are distributed across the year, nobody is primary two weeks in a row, and new joiners shadow one full shift as an extra secondary before taking primary.

Each engineer spends a week as backup before taking the pager, so they start their primary week already aware of what is going on.

Handoff checklist

A five-minute handoff prevents most "nobody told me" incidents. The outgoing primary posts this in the on-call channel:

Incidents and pages from the past shift, and anything still open.
Deploys, migrations or config changes that changed production behavior.
Known degraded dependencies, temporary alerts, or silenced monitors (with expiry).
Upcoming risky work: launches, load tests, database maintenance.
Alerts that fired but were not actionable, so they get fixed instead of ignored.

Runbooks and access for the person holding the pager

The on-call engineer should never need to ask "am I allowed to roll this back?" Write runbooks for the incidents you can predict, starting with the top five by frequency: roll back a bad deploy, restart or scale a stuck worker, handle database connection exhaustion, investigate a spike in API errors, rotate a leaked secret. Each runbook needs symptoms, first checks, safe mitigations, risky actions to avoid, and who to call.

Access is the part teams forget. The primary needs enough access to diagnose and mitigate, but standing admin access for everyone is how incidents get worse. The pattern that works: read access to logs, metrics and cluster state for everyone in the rotation, plus time-limited elevated access (just-in-time roles, or an audited break-glass role) that the on-call engineer can grant themselves during an incident. This also answers a common question: you can remove standing production access from the team without breaking on-call, as long as the elevation path is tested before you need it.

Read access lets anyone diagnose, and elevation is granted only for the incident that needs it.

On-call compensation and burnout

On-call is work outside normal hours, and teams that treat it as free eventually lose their most reliable engineers. The common compensation models are:

Flat stipend per shift: a fixed amount for each week on primary, regardless of pages. Simple and predictable.
Per-incident or hourly pay: paid for time actually spent responding outside hours. Fairer when load is uneven, more admin.
Time off in lieu: a late-morning start or a day off after a rough night. Cheap and often the most appreciated.
Combination: a small stipend plus time off after any night page.

Labor rules on standby time vary by country, so check local requirements before you set a policy. Beyond money, the strongest burnout controls are operational: track pages per shift, review every night page in a weekly on-call review, and give the on-call engineer explicit permission to spend quiet shift time fixing the alerts that woke them up.

On-call tools: what each is good at

The tool matters less than the policy, but the tool should make schedules, overrides and escalations easy to see and change.

Tool	Strength	Note
PagerDuty	Mature scheduling, escalation policies and integrations	The reference product; cost grows with team size
incident.io	On-call plus incident workflow inside Slack	Strong if incidents already run in Slack
Grafana Cloud IRM	On-call and incident response next to Grafana dashboards and alerts	Natural fit if you already use Grafana
Rootly, FireHydrant	Incident management with on-call scheduling	Compare their paging reliability and pricing for your size
Opsgenie	Long-time PagerDuty alternative	Atlassian ended new sales on June 4, 2025 and shuts it down on April 5, 2027; plan a migration (Atlassian notice)

Whatever you choose, test it: trigger a test page to each person's phone during onboarding, confirm phone-call escalation works with Do Not Disturb on, and review the schedule for gaps every month.

Common on-call mistakes in small teams

One hero on permanent call. The person who built the infrastructure answers every page. It feels efficient until they leave or burn out.
Paging on symptoms nobody can act on. Resource alerts without user impact train people to ignore the pager.
No secondary. A dead phone battery becomes a full outage.
Handover on Monday morning. The outgoing person is already in planning meetings and the handoff gets skipped.
Runbooks that live in someone's head. The first incident handled by a new engineer becomes a guessing game.
No review of pages. The same noisy alert wakes someone every week and nothing changes.

A 30-day plan to launch on-call before you hire SRE

Week 1: list your production services and name an owner for each. Agree severity levels and the paging window with leadership.
Week 2: audit alerts. Delete or downgrade anything that does not pass the three tests. Add the missing customer-impact alerts (error rate, latency, failed critical jobs).
Week 3: set up the schedule and escalation policy in your tool, write the top five runbooks, and set up just-in-time elevated access.
Week 4: run a game day: trigger a realistic failure in staging and have the primary handle it end to end. Fix what was confusing, then go live.

Alerts get fixed before the schedule goes live, so the first real shift starts with pages worth answering.

Plan to revisit after the first month: pages per shift, time to acknowledge, and which alerts woke people for nothing. When the rotation itself becomes the bottleneck, that is the signal to formalize the role; our guide on how to build a DevOps team covers that decision.

On-call rotation FAQ

What is an on-call rotation?

An on-call rotation is a repeating schedule that assigns one engineer at a time to respond to production alerts, usually for a week. A secondary backs up the primary, and an escalation policy defines who gets paged if nobody acknowledges.

How do you set up on-call rotations for a small engineering team?

Start with a weekly primary and secondary rotation that includes every engineer who ships to production, page only on customer impact, and hand over midweek with a short written checklist. With fewer than four people, limit formal paging to business hours plus SEV1 at night.

How does an on-call schedule work?

The schedule assigns shifts on a calendar, the paging tool routes alerts to whoever is on shift, and overrides cover holidays and swaps. If the primary does not acknowledge within the agreed time, the escalation policy pages the secondary, then a lead.

How long should an on-call shift be?

One week is the most common and works well for small teams because it limits handovers. Follow-the-sun teams use 8 to 12 hour shifts per region. Avoid shifts longer than a week unless pages are very rare.

Should engineers be paid for on-call?

Most companies compensate on-call through a stipend per shift, pay for time spent on incidents, time off in lieu, or a mix. Unpaid on-call with frequent night pages is one of the fastest ways to lose senior engineers, and local labor rules may require compensation.

Do I need an SRE team to run on-call?

No. Product engineers can run a healthy rotation if ownership, alerting, runbooks and escalation are in place. An SRE function becomes worth it when reliability work consistently competes with product work and the rotation is large enough to need dedicated ownership.

How many people do you need for 24/7 on-call?

Google's SRE guidance puts the minimum at eight engineers for a single-site 24/7 rotation, or six per site for two sites. Smaller teams can still cover 24/7 by keeping night paging to true emergencies and keeping page volume low.

If you need production covered before your team is big enough to carry a pager sustainably, MeteorOps provides SRE on-call support that plugs into your existing rotation and tooling.

Kubernetes HPA: How It Works and How to Stop Flapping

Michael Zion — Mon, 02 Nov 2026 07:30:09 +0000

This guide was written by Arthur Azrieli, CTO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

The Kubernetes Horizontal Pod Autoscaler (HPA) adds or removes pod replicas for a Deployment, StatefulSet or other scalable workload based on a metric such as CPU utilization. Every 15 seconds it compares the current metric to your target, computes ceil(currentReplicas × current / target), and scales within your minReplicas and maxReplicas.

That is the whole idea. Most of the trouble comes later, when replica counts start bouncing up and down, targets show <unknown>, or a GitOps tool keeps resetting the replica count. This guide covers how the algorithm really behaves (including the parts most tutorials skip), copy-paste manifests for the common cases, the behavior settings that stop scaling flaps, how HPA fits with VPA, KEDA, Karpenter and Argo Rollouts, and a troubleshooting table built from the actual controller error messages.

Versions referenced: Kubernetes 1.37 (the current release in October 2026), metrics-server v0.9.0 and KEDA v2.21. The autoscaling/v2 API used throughout has been stable since Kubernetes 1.23, so everything here works on any supported cluster unless a section says otherwise.

Kubernetes autoscalers compared: HPA, VPA, KEDA and node autoscalers

"Kubernetes autoscaling" means four different things, and mixing them up causes most bad designs. The HPA only changes the number of pods. It never changes pod size and never adds nodes.

Autoscaler	What it changes	Typical signal	Use it when
Horizontal Pod Autoscaler (HPA)	Replica count of a workload	CPU or memory utilization, custom or external metrics	Stateless services where more pods means more capacity
Vertical Pod Autoscaler (VPA)	CPU and memory requests of each pod	Historical usage	Right-sizing requests, or workloads that cannot scale out
KEDA	Replica count, including to and from zero (it creates and manages an HPA for you)	Event sources: queue length, Kafka lag, cron, Prometheus queries, cloud services	Queue consumers and event-driven workers
Cluster Autoscaler or Karpenter	Number and type of nodes	Pending pods that cannot be scheduled	Always, if the HPA can create more pods than your nodes can hold

The usual production setup is HPA (or KEDA) for pods plus Cluster Autoscaler or Karpenter for nodes. If the node layer is missing, the HPA happily creates pods that sit in Pending, and you get the cost of a scale-up decision without the capacity. Our guide on setting up Kubernetes autoscaling without cost surprises covers the node side.

How the Horizontal Pod Autoscaler algorithm works

The HPA is a control loop inside kube-controller-manager. It does not react to events. It wakes up on an interval (--horizontal-pod-autoscaler-sync-period, 15 seconds by default), reads metrics, and computes a desired replica count:

desiredReplicas = ceil( currentReplicas × currentMetricValue / desiredMetricValue )

Example: 4 replicas averaging 90% CPU utilization against a 60% target gives ceil(4 × 90 / 60) = ceil(6.0) = 6 replicas. At 45% utilization it gives ceil(4 × 45 / 60) = 3.

For resource metrics, "utilization" is usage divided by the container's request, not its limit and not node capacity. A pod requesting 250m CPU and using 200m is at 80%. This single fact explains a large share of HPA surprises: change the request and you change the scaling math, even though nothing about the traffic changed.

The HPA recomputes desired replicas on a fixed interval and skips any change that falls inside the 10% tolerance.

The details that shape real behavior

The formula is the easy part. These rules, all from the Kubernetes documentation, decide what actually happens:

Tolerance. If the ratio current / target is within 10% of 1.0, the HPA does nothing. With a 60% CPU target, anything between 54% and 66% is a no-op. This dead band is your first protection against jitter.
Unready pods are set aside. When scaling on CPU, pods that are not yet ready, or whose latest metric sample predates readiness, are excluded from the average. If the result would be a scale up, those pods are then assumed to use 0% of the target, which dampens the scale up.
Missing metrics are treated conservatively. Pods with no metric are assumed to use 100% of the target when the HPA is considering a scale down, and 0% when considering a scale up. If that reverses the direction or lands inside the tolerance, the HPA skips the action.
Startup windows. --horizontal-pod-autoscaler-initial-readiness-delay (default 30 seconds) and --horizontal-pod-autoscaler-cpu-initialization-period (default 5 minutes) control how the controller treats CPU samples from freshly started pods. These are control plane flags, so on EKS, GKE or AKS you cannot change them. Design the workload around the defaults instead.
Multiple metrics take the maximum. With CPU and a requests-per-second metric, the HPA computes a desired count for each and uses the highest. If one metric cannot be fetched and the others suggest a scale down, the scale down is skipped. Scale up still works.
Status shows the raw value. kubectl get hpa reports the original average, not the adjusted one used for the decision. That is why the TARGETS column can look like it should trigger a scale and nothing happens.
Scale down is smoothed by default. The HPA keeps the highest recommendation from the last 5 minutes before scaling down. That default stabilization window is why replicas linger after a spike.

Prerequisites: metrics-server and resource requests

Resource metrics (CPU and memory) come from the metrics.k8s.io API, which metrics-server provides. Many managed clusters ship it, many do not. Check first:

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl top pods -n production

If kubectl top fails, install metrics-server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Then make sure every container in the target pods has a CPU request (and a memory request if you scale on memory). The HPA computes utilization per pod across all its containers, so a single sidecar without a request breaks CPU utilization scaling for the whole workload. If your requests are guesses, fix them before tuning the HPA. Our guide to Kubernetes requests and limits shows how to size them from real usage.

Create your first HPA

The fastest way is kubectl autoscale. Recent kubectl versions use --cpu with a percentage (the older --cpu-percent flag is deprecated):

kubectl autoscale deployment api -n production --min=3 --max=20 --cpu=60%

For anything you run in production, keep the HPA in version control as YAML instead:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Watch it work:

kubectl get hpa api -n production --watch
kubectl describe hpa api -n production

describe is where the useful information lives: current metrics, the three status conditions (AbleToScale, ScalingActive, ScalingLimited) and recent scaling events with reasons.

Remove spec.replicas when the HPA owns the count

The Kubernetes docs recommend removing spec.replicas from the Deployment once an HPA manages it. Otherwise every kubectl apply, Helm upgrade or Argo CD sync resets the replica count to the value in Git, the HPA scales it back, and you get a sawtooth that looks exactly like flapping. In Helm charts, the common pattern is to render replicas only when autoscaling is disabled. In Argo CD, you can also add /spec/replicas to ignoreDifferences for the Deployment.

HPA metric types: resource, container, pods, object and external

The autoscaling/v2 API supports five metric source types. Pick the one that measures the real bottleneck, not the one that is easiest to wire up.

Type	What it measures	Needs	Good for
`Resource`	CPU or memory across the whole pod, as utilization or average value	metrics-server	CPU-bound web services
`ContainerResource`	CPU or memory of one named container	metrics-server, Kubernetes 1.30 or newer (stable)	Pods with sidecars (service mesh proxies, log shippers) that distort the pod average
`Pods`	A custom per-pod metric averaged across pods	A custom metrics adapter such as Prometheus Adapter	Requests per second or in-flight requests per pod
`Object`	A metric describing another Kubernetes object	A custom metrics adapter	Requests per second on an Ingress
`External`	A metric from outside the cluster	An external metrics adapter (KEDA provides one)	Queue depth, Pub/Sub backlog, Kafka lag

Scaling only the application container when a sidecar is present:

metrics:
  - type: ContainerResource
    containerResource:
      name: cpu
      container: app
      target:
        type: Utilization
        averageUtilization: 60

Scaling on a per-pod custom metric (for example, exposed by Prometheus Adapter as http_requests_per_second):

metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "50"

A few rules of thumb from running these in production:

CPU is a good signal when traffic turns directly into CPU work: serialization, compression, TLS, rendering.
Memory is a poor scaling signal for most runtimes. The JVM, Go, Node.js and Python rarely give memory back quickly after load drops, so a memory-based HPA scales up easily and then refuses to scale down. Use it only when memory per pod clearly falls as you add pods.
Requests per second or in-flight requests beat CPU for I/O-bound APIs that spend their time waiting on databases or other services.
Queue depth or lag is the right signal for workers. A worker blocked on a slow dependency shows low CPU while the queue grows, so CPU scaling never fires. Use KEDA here.

Why HPAs flap, and how to stop it

Flapping means the replica count oscillates: up on a spike, down a minute later, up again. Users feel it as latency spikes during scale ups, connection churn during scale downs, and a cluster that never settles. The causes are almost always one of these:

Requests set too low. A pod that normally uses 300m with a 100m request sits at 300% utilization at idle. Small traffic changes swing the ratio wildly. Fix the request first.
A noisy metric. Per-second request rates, garbage collection pauses and batchy traffic produce sharp short peaks. The HPA sees them every 15 seconds and reacts.
Startup CPU spikes. JIT compilation, cache warm-up and connection pool creation make new pods look overloaded, which triggers more scale up, which starts more pods.
Scale down that is too fast. Removing many pods at once pushes the survivors over the target, and the HPA adds them back.
A target with no headroom. A 90% CPU target leaves nothing for the minute or two it takes a new pod to become ready.
Something else is changing replicas. A GitOps sync resetting spec.replicas, two HPAs pointing at the same Deployment, or KEDA and a hand-written HPA on the same target.

Causes 1, 5 and 6 are configuration errors you fix in the workload. Causes 2, 3 and 4 are what the behavior field exists for.

Half of flapping is a configuration error in the workload; the other half is what scaling behavior settings are for.

HPA behavior: stabilization windows and scaling policies

The behavior field lets you control scale up and scale down separately. Each direction has a stabilization window, one or more rate-limiting policies, and a selectPolicy that decides which policy wins. Fields you do not set are merged with these defaults:

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
      - type: Percent
        value: 100
        periodSeconds: 15
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15
    selectPolicy: Max

In words: by default the HPA can double the replica count (or add 4 pods, whichever is more) every 15 seconds with no delay, and can remove every pod above minReplicas in one step once the 5-minute scale-down window has passed. That is aggressive in both directions.

What each field does

stabilizationWindowSeconds: the HPA looks at all recommendations made during this window and uses the most conservative one. For scale down that is the highest recommendation, so a brief dip does not remove pods. For scale up it is the lowest, so a brief spike does not add pods.
policies: rate limits. type: Pods, value: 4, periodSeconds: 60 means at most 4 pods per minute. type: Percent, value: 10, periodSeconds: 60 means at most 10% of current replicas per minute (fractions are rounded up, so 10% of 72 replicas allows 8).
selectPolicy: Max picks the policy allowing the biggest change, Min the smallest, and Disabled turns off scaling in that direction entirely.
tolerance (per direction): overrides the cluster-wide 10% dead band for this HPA. It is beta in Kubernetes 1.35 and 1.36 and generally available from 1.37, so check your cluster version before relying on it.

A stabilization window ignores short dips and spikes by acting on the most conservative recent recommendation.

Anti-flap profiles you can copy

Starting points, not final answers. Tune them with the load test in the next section.

Workload	Scale up	Scale down	Why
Latency-sensitive API, fast startup (under 30 seconds)	No window, max of 100% or 4 pods per 60 seconds	300 second window, 20% per 60 seconds	React fast to real load, shed capacity slowly
Slow-starting service (JVM, large caches)	60 second window, 50% per 60 seconds	600 second window, 10% per 60 seconds	Stop startup spikes from compounding; keep warm pods
Spiky, bursty traffic	No window, 100% per 30 seconds	900 second window, 1 pod per 120 seconds	Bursts return; keeping capacity is cheaper than churn
Batch or queue worker on CPU	No window, 4 pods per 60 seconds	300 second window, 25% per 60 seconds	Usually better on KEDA with queue depth

The latency-sensitive API profile as YAML:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      selectPolicy: Max
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
        - type: Pods
          value: 4
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      selectPolicy: Min
      policies:
        - type: Percent
          value: 20
          periodSeconds: 60
        - type: Pods
          value: 2
          periodSeconds: 60

With selectPolicy: Min on scale down, the HPA removes whichever is smaller, 20% or 2 pods, per minute. At 30 replicas that is 2 pods a minute, so once the 5-minute window has passed it takes about 13 more minutes to return to 3. That is deliberate: slow scale down is the cheapest insurance against flapping.

Holding capacity for five minutes and then shedding 2 pods a minute is cheap insurance against flapping.

How to pick the target utilization

The target is really a headroom setting. Ask how much traffic can grow in the time it takes a new pod to become ready and serve. If traffic can rise 50% in the two minutes it takes to start a pod, existing pods need room to absorb that, so a target near 100 / 1.5, about 65% of the point where latency degrades, is the ceiling. If your service degrades at 80% CPU, that gives roughly 50% to 55%.

Most CPU-bound services land between 50% and 70%. Below 50% you pay for a lot of idle capacity. Above 75% you are betting that scale ups always finish before the next spike.

Test HPA behavior before production traffic does

Do not trust an HPA you have not watched under load. In a staging environment with realistic requests:

# terminal 1: decisions and events
kubectl get hpa api -n staging --watch
kubectl get events -n staging --field-selector involvedObject.name=api --watch

# terminal 2: pods coming and going
kubectl get pods -n staging -l app=api --watch

# terminal 3: load from outside the cluster for 10 minutes, then stop
hey -z 10m -c 100 https://staging.example.com/api/health-heavy

Record five things: time to first scale up, replicas added per minute, whether new pods were ready before the next decision, whether latency recovered, and how replicas stepped down after the load stopped. If replicas climb past what the load needs, your scale up is outrunning readiness. If they drop and come straight back, slow the scale down.

HPA with VPA, KEDA, Karpenter and Argo Rollouts

HPA and VPA

Do not let both act on the same resource metric. If the HPA scales on CPU utilization and VPA raises CPU requests, utilization drops, the HPA removes pods, utilization rises, and the two loops chase each other. Safe combinations are VPA in recommendation-only mode (updateMode: "Off") to inform your requests, or HPA on a custom metric such as requests per second while VPA manages CPU and memory.

Give the HPA and VPA different signals, or keep VPA in recommendation mode, so they never fight over one metric.

HPA and KEDA

KEDA is not a competitor to the HPA. A KEDA ScaledObject creates and manages an HPA for you and feeds it external metrics from its 70+ built-in scalers, plus it handles activation from zero replicas. Never point your own HPA and a KEDA ScaledObject at the same Deployment. Choose KEDA when the signal lives outside the pod (queues, streams, schedules) or when you need scale to zero.

HPA and node autoscaling

When the HPA adds pods that do not fit, they stay Pending until Cluster Autoscaler or Karpenter adds a node, which commonly takes one to several minutes. Your scale-up speed is therefore the slower of pod startup and node provisioning. If that is too slow for your traffic pattern, raise minReplicas, lower the target, or keep a little spare node capacity with low-priority placeholder pods that real workloads can preempt (see our guide to Kubernetes PriorityClasses).

HPA during rollouts

During a rolling update the HPA scales the Deployment, and the Deployment controller spreads replicas across old and new ReplicaSets. Startup CPU from new pods can trigger scale ups mid-rollout, so the slow-starting profile above helps here too. If you use progressive delivery, point scaleTargetRef at the Rollout instead of a Deployment (apiVersion: argoproj.io/v1alpha1, kind: Rollout). Our Argo Rollouts guide covers how canary and blue-green steps interact with the HPA.

Can the HPA scale to zero?

Only for object and external metrics, and only with the HPAScaleToZero feature gate. That gate was alpha (off by default) from Kubernetes 1.16 to 1.36 and became beta, on by default, in 1.37. CPU and memory cannot scale to zero because they can only be measured on running pods. On older clusters, or for anything event driven, use KEDA, which has supported scale to zero for years.

HPA troubleshooting: error messages and fixes

These are the messages the HPA controller actually emits, as seen in kubectl describe hpa.

Symptom or message	Cause	Fix
TARGETS shows `<unknown>/60%`	No metrics yet, metrics-server missing, or a container without a request	Wait one or two minutes after creation; run `kubectl top pods`; add requests to every container
`failed to get cpu utilization: missing request for cpu in container X of Pod Y`	A container (often a sidecar or init-injected proxy) has no CPU request	Add a CPU request to every container, or switch to `ContainerResource`
`unable to get metrics for resource cpu: ...` with reason `FailedGetResourceMetric`	The `metrics.k8s.io` API is unavailable	Check `kubectl get apiservice v1beta1.metrics.k8s.io` and the metrics-server pod logs
`did not receive metrics for targeted pods (pods might be unready)`	Pods are crash looping or never become ready	Fix readiness first; see our guide to liveness probes without restart loops
`ScalingLimited True TooManyReplicas`	The desired count hit `maxReplicas`	Raise `maxReplicas` only if downstream systems (database, cache, APIs) can take it
`ScaleDownStabilized` while load is low	The scale-down window is holding the highest recent recommendation	Expected. Shorten the window only if idle cost matters more than churn
Replicas jump back to a fixed number after every deploy	`spec.replicas` in the manifest overrides the HPA on each apply or sync	Remove `spec.replicas`, or ignore it in Argo CD
Memory HPA never scales down	The runtime keeps memory after load drops	Scale on CPU or a work metric instead
More pods, same latency	The bottleneck is a database, lock or external API	Cap `maxReplicas` at downstream capacity, add pooling and backpressure

Production HPA checklist

Every container in the target has a CPU request based on measured usage.
spec.replicas is removed from the Deployment manifest, or ignored by your GitOps tool.
minReplicas is at least 2 (3 for multi-zone services) and covers normal traffic.
maxReplicas is set from downstream capacity and node budget, not from a round number.
The metric matches the bottleneck: CPU for compute, requests per pod for I/O-bound APIs, queue depth (via KEDA) for workers.
Scale up is faster than scale down, and scale down has a window of at least 300 seconds.
Readiness probes reflect real readiness, and startup time is known.
A node autoscaler is in place and its provisioning time is part of your scale-up budget.
VPA is not changing the same resource the HPA scales on.
You have watched the HPA through a load test and a scale down.

Kubernetes HPA FAQ

Is HPA useful for Kubernetes?

Yes, for stateless workloads where more pods means more capacity. It lets you run fewer replicas at quiet times and absorb peaks without manual intervention. It is the wrong tool for stateful systems that cannot simply add replicas, and for workloads whose bottleneck is outside the pod.

What is a good HPA CPU threshold?

Most CPU-bound services work well between 50% and 70% of the CPU request. Use the headroom calculation above: the slower your pods and nodes start, and the faster your traffic grows, the lower the target should be.

What is the difference between HPA and VPA?

The HPA changes how many pods run. VPA changes how much CPU and memory each pod requests. They can work together if they act on different signals, for example HPA on requests per second and VPA on resources, or VPA in recommendation mode only.

What is the difference between KEDA and HPA?

KEDA builds on the HPA. It creates an HPA for you, feeds it metrics from event sources like queues and streams, and adds scale to zero. Use plain HPA for CPU or custom metrics on services; use KEDA for event-driven workers.

How often does the HPA check metrics?

Every 15 seconds by default. Metrics-server itself scrapes kubelets on its own interval, so the data the HPA sees can be somewhat older than that.

Why does my HPA show unknown targets?

Either metrics are not available yet (normal for a minute or two after creation), metrics-server is missing or broken, or at least one container in the pods has no request for the resource you scale on. kubectl describe hpa shows which one.

Why is my HPA not scaling down?

The 5-minute default scale-down stabilization window, a metric still above the target minus the 10% tolerance, a memory metric that never drops, or a second metric that keeps the maximum recommendation high. Check the conditions and events in kubectl describe hpa.

Can I use HPA and a fixed replica count in Helm?

Not on the same object. Render replicas in the Deployment only when autoscaling is disabled, so a Helm upgrade does not reset the HPA's decision.

If your autoscaling still behaves unpredictably after this, the cause is usually in the workload or the node layer rather than the HPA itself. MeteorOps engineers tune this for startups as part of Kubernetes consulting.

SRE vs DevOps: Key Differences and Which Your Startup Needs

Michael Zion — Sun, 01 Nov 2026 07:30:04 +0000

This guide was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

SRE vs DevOps in one paragraph: DevOps is a culture and set of practices for shipping software faster by removing the wall between development and operations. Site reliability engineering (SRE) is a specific, opinionated way to run production that came out of Google: it sets reliability targets (SLOs), spends an error budget against them, and treats operations as a software problem. Google sums it up as "class SRE implements interface DevOps". DevOps optimizes the path to production. SRE optimizes what happens once you are there.

Below is how the two differ in goals, daily work, metrics, skills and pay, how they fit together, and which one a startup actually needs at each stage.

SRE vs DevOps at a glance

	DevOps	SRE
What it is	A culture and set of practices across the whole delivery lifecycle	A job role and an engineering discipline for running production
Origin	A movement among practitioners at many companies, around 2009	Google, from 2003, written up in the Site Reliability Engineering book (2016)
Main question	How do we ship changes faster and safer?	How reliable must this service be, and are we meeting that?
Core practices	CI/CD, infrastructure as code, automation, shared ownership, fast feedback	SLIs and SLOs, error budgets, toil reduction, incident response, blameless postmortems, capacity planning
Typical metrics	The five DORA metrics: change lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate	SLO attainment, error budget burn rate, toil percentage, incident count and duration, pages per shift
Prescriptiveness	Principles; little detail on how to run services	Concrete rules, such as caps on toil and on-call time
Where the work sits	Pipelines, environments, platforms, developer workflow	Production systems, observability, on-call, reliability of user-facing services

DevOps owns how changes reach production; SRE owns how production behaves once they land.

What DevOps actually means

DevOps started as a reaction to a familiar split: developers were measured on shipping features, operations on keeping things stable, and every release became a fight. DevOps says one team should own the whole path from commit to production, with automation doing the repetitive work.

A common way to describe it is CALMS: Culture, Automation, Lean, Measurement and Sharing. In practice that means CI/CD pipelines, infrastructure as code, small and frequent changes, environments anyone can recreate, and developers who can see and act on how their code behaves in production.

"DevOps engineer" is the job title that grew out of this. In most companies a DevOps engineer builds and runs the delivery platform: cloud infrastructure, CI/CD, containers and Kubernetes, secrets, and the tooling that lets developers ship on their own.

What SRE actually means

SRE is what you get when you ask software engineers to run operations. Its central idea is that 100% reliability is the wrong target. Users cannot tell 99.99% from 100%, and chasing the last fraction slows every release. So SRE makes reliability an explicit, measured number:

SLI (service level indicator): what you measure, such as the share of requests that succeed in under 300 ms.
SLO (service level objective): the target for that indicator over a window, such as 99.9% over 30 days.
Error budget: the failure you are allowed. A 99.9% SLO over 30 days allows about 43 minutes of full downtime, or the equivalent in failed requests.

While budget remains, teams ship freely. When it runs out, the priority shifts to reliability work until the service is healthy again. That one rule turns the old dev vs ops argument into a shared, data-driven decision.

The error budget turns a reliability target into a clear rule for when to ship and when to fix.

SRE also sets hard limits on operational load. Google's SRE teams aim to keep toil, the manual, repetitive, automatable work that "scales linearly as a service grows", below 50% of each engineer's time. Of the remaining operational time, no more than 25% should be on-call.

How much downtime does each SLO allow?

SLO	Allowed downtime per 30 days	Per year
99%	7 hours 12 minutes	about 3.65 days
99.5%	3 hours 36 minutes	about 1.8 days
99.9%	43.2 minutes	about 8.8 hours
99.95%	21.6 minutes	about 4.4 hours
99.99%	4.3 minutes	about 53 minutes

"class SRE implements interface DevOps"

Google's own SRE workbook frames the relationship this way. DevOps is the interface: a set of principles. SRE is one concrete implementation of those principles, with specific practices for each of them:

Reduce organizational silos: SREs share ownership of production with developers and use the same tools.
Accept failure as normal: error budgets and blameless postmortems.
Implement gradual change: small changes, canary releases, fast rollback.
Tooling and culture are interrelated: SREs use the same tools as developers, and the toil cap forces automation.
Measure everything: SLOs and error budgets drive decisions.

Each DevOps principle has a concrete SRE practice behind it.

The workbook also names the differences: DevOps is broader and targets silos across the whole organization, while SRE is narrower, more prescriptive, and focused on services and their users.

Responsibilities side by side

Area	DevOps engineer	Site reliability engineer
CI/CD	Builds and owns pipelines, build caching, release automation	Adds safety: canaries, automated rollback, release gates tied to SLOs
Infrastructure	Provisions cloud, networks, clusters with Terraform or similar	Capacity planning, load testing, failure-mode design, disaster recovery drills
Observability	Installs and maintains the monitoring and logging stack	Defines SLIs and SLOs, alerting on symptoms and burn rate, dashboards for incidents
Incidents	Often on the rotation, fixes infrastructure causes	Leads incident response, runs postmortems, tracks follow-ups to completion
Developer experience	Self-service environments, templates, internal platform	Production readiness reviews, reliability guidance for service owners
Code	Infrastructure code, scripts, pipeline config	Automation and tooling, sometimes changes to application code for reliability

In small companies these roles overlap heavily, and one senior engineer often does both. The split becomes real once you have enough services and traffic that reliability needs a dedicated owner.

Skills: where they overlap and where they don't

Shared core: Linux, networking, a major cloud provider, containers and Kubernetes, infrastructure as code, CI/CD, a scripting or programming language, and monitoring.

Leans DevOps: pipeline design, build systems, Terraform module design, developer tooling, platform engineering, cost-aware infrastructure.

Leans SRE: distributed systems failure modes, SLO design, statistics for latency and error rates, performance and capacity analysis, incident command, writing production-grade software in Go, Python or Java.

SRE vs DevOps salary

The two pay similarly, with SRE slightly ahead in most surveys. In the 2024 Stack Overflow Developer Survey, median yearly salaries were:

Role	United States median	Global median
SRE	$166,500	$99,099
DevOps specialist	$145,000	$72,611
Cloud infrastructure engineer	$165,000	$96,666

The 2025 survey no longer breaks out SRE, but reports a US median of $165,000 for "DevOps engineer or professional". Seniority, location and company size move pay far more than the title does. Treat the gap between the two roles as small.

Which does your startup need?

Stage	What you need	Why
Pre-product or MVP, 1 to 10 engineers	DevOps practices, owned by developers, with outside help for the foundation	You need CI/CD, infrastructure as code and basic monitoring. Reliability targets would be guesses.
Early customers, 10 to 30 engineers	A first DevOps or platform engineer, plus simple SLOs on your one or two critical user journeys	Delivery speed is the bottleneck. A light SLO keeps reliability from being argued by gut feel.
Growth, 30 to 100 engineers	A platform team and your first SRE, or SRE practices embedded in the platform team	Incidents, on-call load and enterprise customers' uptime expectations make reliability a full-time job.
Scale, 100+ engineers or strict SLAs	A dedicated SRE function alongside platform engineering	Error budgets, capacity planning and incident management across many teams need clear ownership.

Two practical rules:

Hire for the bottleneck. If releases are slow and painful, you need DevOps capability. If releases are fine but outages, pages and customer complaints are the problem, you need SRE capability.
Don't build a 24/7 rotation too early. Google's guidance puts the minimum for a single-site 24/7 rotation at eight engineers. Most startups get further with business-hours paging, good runbooks and clear severity rules. Our guide to designing an on-call rotation for a small team covers the math.

Let the problem you feel most decide which capability to hire first.

How DevOps and SRE work together

In a healthy setup, the platform or DevOps team builds the paved road (pipelines, environments, infrastructure modules), and SRE makes sure what runs on that road meets its reliability targets. Concretely:

Product teams own their services and their SLOs.
The platform team makes the right thing the easy thing: standard deploys, built-in metrics, safe rollbacks.
SREs set the SLO framework, review production readiness, lead major incidents and push systemic fixes back into the platform.
Error budget policy decides, in advance, what happens when a service runs out of budget.

A practical first step for any team is cleaning up alerting so that pages map to user impact. Our article on handling alert fatigue shows how, and the observability stack guide covers what to run at each team size.

Common mistakes

Renaming the ops team "SRE". Without SLOs, error budgets and a toil cap, it is the same ops team with a new title.
Hiring a "DevOps engineer" to be the only person who can deploy. That rebuilds the silo DevOps was meant to remove.
Setting SLOs at 99.99% by default. Each extra nine costs a lot. Start from what users notice and what your dependencies can deliver.
Measuring only uptime. Latency and correctness matter to users as much as availability.
Treating it as either/or. Most teams need DevOps practices first and SRE practices as they grow, not one instead of the other.

FAQ

Is SRE better than DevOps?

Neither is better; they solve different problems. DevOps improves how you build and ship software. SRE improves how reliably it runs. SRE is often described as one specific implementation of DevOps principles, so a mature organization usually uses both.

Is DevOps a part of SRE, or SRE a part of DevOps?

SRE is generally seen as an implementation of DevOps. Google's phrase is "class SRE implements interface DevOps": DevOps defines the principles, and SRE is a concrete, prescriptive way to apply them to running production services.

Who earns more, SRE or DevOps?

SREs usually earn slightly more. In the 2024 Stack Overflow Developer Survey, the US median was $166,500 for SREs and $145,000 for DevOps specialists. Seniority, location and employer matter more than the title.

Can a DevOps engineer become an SRE?

Yes, it is one of the most common paths. Add stronger programming skills, learn SLO and error budget design, get comfortable leading incidents and postmortems, and go deeper on distributed systems failure modes and capacity planning.

Will AI replace SRE jobs?

AI tools are taking over parts of the work, such as summarizing incidents, correlating alerts and drafting runbooks and postmortems. The judgment parts remain: choosing reliability targets, deciding trade-offs during incidents and designing systems that fail safely. The role is shifting toward that work, not disappearing.

What is the difference between SRE and platform engineering?

Platform engineering builds the internal products developers use to ship: templates, pipelines, environments, an internal developer platform. SRE focuses on the reliability of the services running in production. Many startups combine them in one team until they grow.

Does a startup need an SRE?

Usually not as a first infrastructure hire. Most startups need DevOps capability first, with simple SLOs on critical user journeys. A dedicated SRE starts to pay off when incidents, on-call load or customer uptime commitments take a meaningful share of engineering time.

The short answer

DevOps gets your changes to production quickly and safely. SRE keeps production as reliable as your users need, and no more. Start with DevOps practices, add SLOs early and cheaply, and bring in dedicated SRE when reliability becomes a job of its own.

Need senior help with either? MeteorOps provides DevOps consulting and SRE and on-call support for startups, from first pipelines to SLOs and incident response.