DEV Community: Samson Tanimawo

Building a Career in SRE: From Junior to Staff

Samson Tanimawo — Wed, 12 Aug 2026 01:27:37 +0000

I've been in SRE for about 10 years. Started as a junior, made my way through mid, senior, and now staff levels. Here's what I learned about growing in this career path.

Junior → Mid

The main shift: learn to operate without constant guidance.

As a junior, you'll be handed tickets and shown how to handle them. Your job is to do them well and ask questions. At mid-level, you should be able to take an ambiguous problem and produce a reasonable first attempt without help.

Signs you're ready to move up:

You can handle a routine on-call shift without senior help
You can write a post-mortem that doesn't need major revisions
You're improving your team's processes without being asked

What I wish I'd focused on: fundamentals. Linux, networking, the tools your team actually uses. Not the fancy stuff. The boring deep knowledge pays off forever.

Mid → Senior

The main shift: from executing well to choosing well.

Mids know how to solve problems. Seniors know which problems to solve. You stop asking 'how do I implement this' and start asking 'is this the right thing to implement.'

Signs you're ready:

You're turning down work that doesn't make sense, with good reasons
Other engineers come to you for design input
You're anticipating failures, not just responding to them
You own an area of the system without anyone telling you to

What I wish I'd focused on: writing and communication. The best senior SREs are also the best writers. Not fancy writing. Clear, boring, structured writing. It's the highest-leverage skill at this level.

Senior → Staff

The main shift: from individual contribution to organizational leverage.

Seniors ship features. Staff engineers ship teams. At staff level, your biggest impact is making other engineers more effective — through design leadership, process improvements, mentorship, and systems thinking.

Signs you're ready:

You spend more time in strategic work than tactical work
Your ideas show up in other teams' roadmaps
You're the person leadership asks to sanity-check big decisions
You own organizational problems, not just technical ones

What I wish I'd focused on: saying no gracefully. Staff engineers who can't say no get pulled into everything. Staff engineers who can say no with good reasons become force multipliers.

The whole journey

Here's the pattern: every level up is about expanding scope and reducing supervision. Junior: small scope, lots of help. Mid: medium scope, less help. Senior: large scope, peer-level. Staff: org-wide scope, shaping others.

The biggest mistake is trying to be promoted by working harder at the current level. You get promoted by demonstrating the next level's behavior before you're asked to.

The quiet truth

Most SRE career growth doesn't come from learning new technology. It comes from learning people, communication, and judgment. The technical floor gets you to mid-level. Everything above that is about humans.

If you're stuck at senior and wondering why you're not hitting staff, it's almost certainly not a technical gap. It's a leverage or communication gap. Work on that. It changes everything.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Incident Automation: What to Automate, What to Leave to Humans

Samson Tanimawo — Tue, 11 Aug 2026 13:19:18 +0000

Incident response automation is a trap. Some things should be automated. Some things absolutely should not be. Getting the line wrong is worse than automating nothing.

What to automate

1. Alert enrichment. Before a human sees an alert, automate pulling in related data: recent deploys, dependent service health, historical correlation. Save the human 10 minutes of context-gathering.

2. Known-good remediations. If an alert always has the same fix (restart service X, clear cache Y), automate the fix. But: require a human confirmation for the first 30 days before full auto.

3. Communication scaffolding. When an incident starts, auto-create the Slack channel, invite the on-call, post an initial status template, update the status page with a placeholder. Humans fill in the details.

4. Post-incident paperwork. Auto-generate a post-mortem template with timeline pulled from chat and monitoring. Humans edit and refine.

5. Routine handoff. When on-call rotates, auto-summarize what's been happening and who's been paged. Saves the incoming engineer 15 minutes of catching up.

What not to automate

1. Root cause analysis. AI or rules-based systems can suggest causes, but the final call has to be human. The 'wrong' root cause ends up in the post-mortem and misleads future responders.

2. Impact assessment. 'How many users are affected' needs context only humans have. Automation will miss business-critical customer segments.

3. Executive communication. Your VP of customer success doesn't want a templated bot message. They want 'here's what's happening, here's what we're doing, here's when I'll update you next' from a human.

4. Deciding severity. Yes, automate the initial guess. But a human has to confirm. Severity drives organizational response, and 'a bot marked this as sev-3' will be questioned the moment it matters.

5. The critical decision during an outage. Should we roll back? Should we fail over? Should we scale up? These are judgment calls with consequences. Don't hand them to a bot.

The principle

Automate the mechanical, human the judgmental. If the task has a clear right answer and no downside if it's wrong, automate. If it requires context, accountability, or judgment, leave it to humans.

The measurement

After adding automation, ask two questions:

Are humans faster to resolution?
Are humans feeling more in control, or less?

If resolution is faster but humans feel like they've lost visibility, you over-automated. Pull back.

The emotional piece

Humans run incidents because humans are accountable. Automation that undermines that accountability creates downstream problems. Build tools that make humans faster and more confident, not tools that replace them. There is a big difference.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Infrastructure Drift: Detecting and Preventing It

Samson Tanimawo — Tue, 11 Aug 2026 01:45:12 +0000

Your Terraform says the firewall rule is A. The cloud console says it's B. Someone changed it manually and didn't tell anyone. This is infrastructure drift, and it's a reliability killer.

How drift happens

The emergency edit. At 2 AM, someone clicks a button in the console to fix an outage. They promise to 'put it in Terraform later.' They don't.

The helpful colleague. Someone from a different team edits a shared resource because it was blocking them. They didn't know it was Terraform-managed.

The console-first culture. Some teams never adopt IaC. They edit everything manually, and then one day someone runs Terraform plan and sees 200 unexpected changes.

The partial migration. You migrated some resources to Terraform but not others. Nobody documented which is which.

Why drift is dangerous

The next Terraform apply will either revert the manual change (surprise!) or fail because of conflicts.
The manual change often has no audit trail. 'Why is this firewall rule open?' No answer.
Disaster recovery becomes a guessing game. You can redeploy from Terraform, but you'll lose the manual edits.
Security audits become impossible because your declared state doesn't match real state.

Detection

Run terraform plan on a schedule. Daily or more. Alert on any unexpected diffs. The first time I set this up, we had 40 drift items in our main AWS account. Now we have zero because the alert forces cleanup within a day.

Cloud-native drift detection. AWS Config, GCP Asset Inventory. These track changes at the cloud level and can flag manual edits independently of IaC tools.

Git as source of truth. If your IaC is in Git and the cloud state differs, Git wins — either manually reconcile or explicitly accept the drift by updating the code.

Prevention

Remove manual edit permissions. This is the nuclear option and the most effective. If engineers can't edit resources manually, drift stops happening. Instead, they have to write Terraform (or equivalent) to make changes.

Yes, this slows down emergency responses. That's why you build fast-path deployment pipelines for IaC — commit, auto-approve, auto-apply within 5 minutes.

Write-only service accounts. Grant humans read-only in prod. Only CI/CD can write. Breakglass accounts exist for emergencies but require audit trail and review.

The cultural shift

The hardest part isn't the tools. It's the mindset shift from 'the console is the source of truth' to 'Git is the source of truth.' Teams used to clicking in the console will resist. Give them better tooling (fast apply pipelines, good Terraform examples) and the resistance fades.

You'll know you've made it when 'put it in Terraform first' becomes the default response to any infrastructure change request. Until then, drift will bite you.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The Engineer Who Owns Nothing: A Cautionary Tale

Samson Tanimawo — Mon, 10 Aug 2026 13:21:02 +0000

I'm going to tell you about an engineer I worked with. Call him Mark. Mark was talented, well-liked, and utterly ineffective. Here's what I learned from watching him.

What Mark did

Mark's technical skills were real. He wrote good code. He gave thoughtful design review comments. He spoke well in meetings.

Mark's problem: he didn't own anything.

He worked on whatever was in front of him. He fixed bugs in code he didn't write. He helped other engineers with their services. He never said 'this is mine.' Everything was 'we should figure out who handles that.'

Why this was a problem

When something broke in production, Mark would help debug — for a few minutes. Then he'd disengage because it 'wasn't really his area.' Nobody ever held him accountable because the code wasn't explicitly assigned to him.

When planning happened, Mark didn't propose projects. He waited for projects to be assigned to him. Assigned projects rarely came because leaders couldn't predict what he'd actually commit to.

When reliability work needed doing — alerts to tune, dashboards to fix, runbooks to write — Mark agreed it was important and waited for someone else to do it.

Mark got good performance reviews for the first two years. He was technically capable, pleasant, and unobjectionable.

What happened

At year three, the company hit hard times. Leadership started asking: 'what has this person done? what do they own?' Mark had no answer.

He got laid off. It wasn't a performance issue in the normal sense — he hadn't done anything wrong. He just hadn't owned anything.

The lesson for SREs

SRE is especially vulnerable to this trap. Everything is shared infrastructure. It's tempting to be the person who 'helps out everywhere' without ever claiming a specific service.

Don't. Claim something.

Own an SLO
Own a runbook library
Own the post-mortem process
Own the alerting hygiene for a service
Own the capacity planning model

It doesn't have to be big. But it has to be explicitly yours, with consequences if it fails and credit if it succeeds.

The meta-lesson

Ownership isn't the same as being busy. You can be in a million meetings and own nothing. You can write 10,000 lines of code and own nothing. Ownership means: when something in your area breaks, you are the first person called, and you are the person who gets credit when it works.

Pick something this week. Ask your manager: 'can I explicitly own X?' Write it down. Defend it.

Don't be Mark.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Error Budget Policies That Hold Leadership Accountable

Samson Tanimawo — Sun, 09 Aug 2026 23:39:50 +0000

Error budgets are useless without a policy. 'We're out of error budget' should trigger consequences. If it doesn't, you don't have an error budget — you have a vanity metric.

Here's a policy that actually works.

The four states

Healthy (< 70% of budget used). Business as usual. Feature development proceeds at full speed.

Watch (70-90% used). Feature velocity continues but new risky changes require explicit sign-off from an SRE. No gate, just attention.

Constrained (90-100% used). Feature freezes. Only reliability work and critical bug fixes until we're back below 90%.

Breached (> 100% used). Incident-level response. Leadership informed. Post-mortem for why we blew through. Feature work stays frozen until we recover and identify systemic causes.

The part most policies miss

The feature freeze in 'constrained' state is the part that actually changes behavior. Everything else is documentation. Without consequences, teams ignore the budget.

The freeze has to be real. Leadership can't override it for a 'really important feature' — that's exactly the time the freeze matters. The only exception is a legitimate emergency fix, and those should be rare.

Selling this to leadership

Executives hate feature freezes. They see it as slowing the business. Counter-argument: feature freezes during budget exhaustion protect the business. Shipping features onto broken infrastructure creates more breakage, which burns more budget, which is a doom loop.

Frame it as: 'the feature freeze is a safety valve. When it triggers, it's because something's wrong and we need to fix it before making it worse.'

Also: a good policy lets you spend the budget aggressively when you have it. Feature teams should be encouraged to experiment, deploy fast, and take risks when you're at 30% budget used. The freeze is only for when the safety margin is gone.

The review cadence

Weekly error budget review, 15 minutes max. Who attended: SRE lead, engineering manager, maybe a PM. Decisions: are we in healthy/watch/constrained? Any actions for the coming week?

Monthly broader review with leadership. Trends over time. Investment decisions.

The escalation

If a team enters 'constrained' state three times in a quarter, that's a systemic issue. Escalate to engineering leadership with a proposal: either invest in reliability or accept a lower SLO formally.

The endgame

A mature organization uses error budget policy to balance feature velocity against reliability automatically. Nobody is negotiating individual decisions. The framework does the work.

Getting there takes 6-12 months of discipline. The first few freezes will feel painful. After that, they become routine, and something surprising happens: you stop having them as often. The policy is working.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Dependency Injection for Observability

Samson Tanimawo — Sun, 09 Aug 2026 13:42:41 +0000

Want your code to be easy to observe? Use dependency injection for observability concerns. Sounds dry. Hear me out.

The problem

Your code calls log.info(...) directly. In tests, you can't verify what was logged. In prod, if you want to change the logger, you're grepping the codebase. If you want to add tracing, you're editing every call site.

Same for metrics. Same for tracing. Same for error reporting.

The fix

Pass the observer in. Functions take a logger, metrics, or tracer as an argument (or constructor dependency). The function doesn't know what's behind it.

func HandleOrder(order Order, deps Deps) error {
    deps.Metrics.Inc("order.received")
    deps.Logger.Info("processing order", "id", order.ID)
    // ...
}

Now:

In tests, pass in a mock that captures all calls
In prod, pass in the real logger
Changing backends (Datadog → Prometheus) is one wire-up change
Adding tracing is one new field in Deps

Why most codebases don't do this

It feels verbose. 'Why do I have to thread a logger through every function?' Engineers hate boilerplate.

The alternative is a global singleton. Easy to use, impossible to test cleanly, nightmare to refactor.

The boilerplate is worth it. Especially for observability, where you will want to swap implementations later.

The subtle win

Dependency-injected observability forces you to think about what you're observing. When you have to explicitly pass the logger, you notice that a function is calling it 8 times. Is that too much? Is the logging doing real work? Would one structured log at the end of the function be better?

Functions with injected dependencies tend to have better observability because the developer had to look at it.

The starting point

You don't need to refactor everything at once. Pick one critical path — the checkout flow, the auth path. Refactor just that path to use dependency injection for observability. See if tests and debugging get easier.

If yes, expand. If no, you found something else is the bottleneck.

The bigger idea

Observability is code. Treat it with the same architectural discipline you treat the rest of your codebase. Dependency injection is one tool. There are others (context objects, middleware, decorators). Pick one, apply it consistently, and your future-you will be able to observe your code in ways that are impossible with ad-hoc log.info calls everywhere.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Load Balancer Tuning: Lessons from Production

Samson Tanimawo — Sat, 08 Aug 2026 23:34:11 +0000

Load balancers are the silent infrastructure. You don't think about them until they start dropping connections at 2 AM. Here are the settings that have bitten me in production.

Connection timeouts

Default connection idle timeout on AWS ALB is 60 seconds. Your app might have requests that legitimately take 90 seconds. Result: the ALB drops the connection mid-request, your user sees a 502, and your logs show nothing because the app was still processing.

Rule: set the LB idle timeout higher than your longest legitimate request. For most APIs, 120-300 seconds. Verify it matches your app's own timeout.

Health check intervals

Default health check interval is usually 30 seconds, with 2 failures before marking unhealthy. That's up to 60 seconds of traffic sent to a dying instance before it's removed.

Rule: 10-second intervals, 2 failures. 20 seconds to remove a bad instance is much better. Yes, slightly more load on your service. Worth it.

Unhealthy threshold

Marking an instance unhealthy after 2 consecutive failures is usually right. Marking it healthy again after 2 consecutive successes is usually wrong — it lets half-broken instances come back prematurely.

Rule: require 3-5 consecutive successes to mark healthy. Slower recovery, more stable routing.

Slow start / connection draining

When a new instance comes online, it's often cold — empty caches, no warmed connections. If the LB sends it full traffic immediately, it performs badly and might get marked unhealthy.

Rule: enable slow start (AWS calls it 'slow start mode' on ALB). Ramp traffic over 30-60 seconds. Worth it for any service that needs warming.

Sticky sessions

Sticky sessions feel like a solution. They're usually a problem.

Rule: avoid them. If you need them, your app has shared state that should be in a database or cache, not in-process. The one exception: WebSocket connections, where stickiness is unavoidable.

Cross-zone load balancing

Disabled by default on some LBs. With cross-zone disabled, an instance in AZ-A only serves traffic from AZ-A. If AZ-A has fewer instances, that AZ's traffic gets uneven distribution.

Rule: enable it unless you have a specific reason not to. Costs a small amount of cross-AZ traffic but gives you even load distribution.

The meta-lesson

Load balancer defaults are designed for 'will work for everybody, badly.' Any given workload needs tuning.

Spend one afternoon reviewing your LB configs. Ask: 'is this default right for my app?' At least half the defaults will be wrong. Tune them. Future you will thank you during the next incident.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Capacity Planning for Startups

Samson Tanimawo — Sat, 08 Aug 2026 13:32:21 +0000

Capacity planning sounds like enterprise spreadsheet work. For a startup, it's 'don't get embarrassed when traffic spikes, don't go broke overprovisioning.' Here's the pragmatic version.

The 3 questions

1. What does normal look like right now? Peak RPS, p99 latency, CPU/memory utilization at peak. If you don't know these, stop and measure. You cannot plan without baseline.

2. What's the next expected spike? A launch. A press mention. A marketing campaign. The Black Friday of your industry. Put these on a calendar.

3. How long does it take to add capacity? Minutes (autoscaling)? Hours (VM provisioning)? Weeks (vendor contracts)?

Your capacity plan is the gap between expected spike and response time.

The startup hack: overprovision early

At startup scale, overprovisioning is cheap. An extra $5k/month of slack is trivial compared to the embarrassment of 'we went down during the launch.'

Run at 30-40% peak utilization. Yes, that's wasteful. It's also a 3x buffer for unexpected spikes. Worth it until you're big enough to care about the efficiency.

The autoscaling reality

Autoscaling is great but has limits:

Cold start times mean it can't handle traffic that doubles in 30 seconds
Provisioning limits mean you can only add X instances per minute
Downstream dependencies (databases, queues) usually don't autoscale

Test your autoscaling before you need it. The first time I depended on autoscaling in production, it worked. The second time, it didn't, because our database connection pool was capped. That was the real bottleneck.

The 3 bottlenecks to check

For every scaling test, verify:

Stateless service capacity. Usually easy — just add more instances.
Database capacity. Connection counts, query latency, replication lag. Usually the real bottleneck.
Third-party dependencies. Rate limits on external APIs, email providers, payment processors. A sudden 10x spike usually hits someone's rate limit.

The launch checklist

Before any planned spike:

Overprovision by 3x what you think you need
Pre-warm caches and connection pools
Confirm your paging rotation is ready
Prepare a rollback plan for the feature being launched
Schedule the launch during your team's awake hours, not off-hours

The real lesson

For startups, the goal of capacity planning is not efficiency. It's confidence. If you have to spend a little more to avoid panic during growth, spend it. Optimize for efficiency later, when you have a year of traffic history to work from.

Right now, your job is to stay standing. Do that first.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

How We Handled Our First Major Outage (And Survived)

Samson Tanimawo — Sat, 08 Aug 2026 01:27:13 +0000

Three years ago we had our first real outage. Six hours of downtime. Thousands of angry users. Multiple executives on the call. Here's what we did right, what we did wrong, and what we'd do differently.

What we did right

1. Communicated immediately. The moment we knew we had a problem, we updated the status page and emailed our biggest customers personally. Not when we had answers. When we had a question.

2. Had a single incident commander. One person making calls. Not a committee. When the CEO tried to direct technical work, the IC politely rerouted and told her where her help was actually needed (talking to customers).

3. Took care of our people. During hour 4, I ordered food. During hour 5, I forced the primary engineer off the call for 20 minutes to walk outside. Long incidents destroy people. You have to feed them and force them to rest.

4. Wrote it down as we went. We had a shared doc with a live timeline. When the post-mortem came, we had every decision captured.

What we did wrong

1. Tried to fix the root cause during the incident. For the first 2 hours, we were digging into why the database was struggling. We should have been mitigating (rolling back) first.

2. Let too many people 'help.' By hour 3, we had 12 engineers in the call. Half of them were useless. The IC should have kicked people out sooner.

3. Gave optimistic estimates. 'We'll be back in 30 minutes.' We were not back in 30 minutes. That miscommunication was worse than saying 'unknown.'

4. Didn't prepare the executive communication. The CEO had to answer customer questions in real time with no script. We should have drafted talking points for her after hour 1.

What we'd do differently

Mitigate first, investigate second. Always.
Cap the number of active engineers at 4 during an incident. Others go on standby.
Default to 'unknown' for estimates. Only give a number when we're sure.
Assign someone explicitly to 'executive liaison.' Their job is to keep the C-suite informed without interrupting the technical team.

The aftermath

The post-mortem was brutal and cathartic. We identified 14 action items. We actually did 11 of them over the next quarter.

The outage was the best thing that happened to our reliability culture. It turned reliability from 'a thing SRE owns' into 'a thing everyone takes seriously.' I wouldn't wish a 6-hour outage on anyone, but I also wouldn't trade the lessons.

The final lesson

Your first major outage will happen. Prepare for it by running game days. The game days will feel silly until the real thing happens, at which point every muscle you trained will kick in.

Incident response is a skill. Skills need practice. Practice now.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The Economics of Reliability: When to Invest, When to Accept Risk

Samson Tanimawo — Fri, 07 Aug 2026 13:19:02 +0000

Reliability is not a virtue. It's an investment. Too little and you lose customers. Too much and you can't afford to ship. The question is: where's the right balance?

The error budget framing

The SRE book covers this well. Pick an SLO (say 99.9% uptime). That's 43 minutes of budget per month. If you're at 99.95%, you have budget to spend on risky things. If you're at 99.85%, you need to stop shipping risk.

This works. But it doesn't answer 'what SLO should I pick?' Let me give you a framework.

The three questions

1. What do users expect? A consumer banking app needs 4 nines or more. A developer tool can get away with 3. A beta product can live with 99%. Ask users (or watch churn numbers).

2. What does an outage cost? Dollars of lost revenue + dollars of customer churn + hours of engineering time. For a checkout-heavy product, an hour of downtime might cost $500k. For a B2B internal tool, $5k.

3. What does the next 9 cost? Going from 99% to 99.9% might cost $50k of engineering work. Going from 99.9% to 99.99% often costs $500k or more. Each 9 is 10x harder.

The math

Invest in reliability up to the point where the next $1 invested saves less than $1 of outage cost over the amortization period.

If moving from 99% to 99.9% costs $50k and would save $200k over a year in reduced outage damage, invest. Easy call.

If moving from 99.9% to 99.99% costs $500k and saves $100k, don't invest. Accept the risk, and spend the engineering time on something with better ROI.

The hidden cost

Over-investing in reliability has a hidden cost: team velocity. Teams that chase 99.999% uptime spend so much on tests, canaries, staging environments, and approval gates that they ship slowly. Competitors with 99.5% reliability but 5x your velocity will win the market.

Reliability that kills velocity is bad reliability. Measure both.

The political reality

The hardest part isn't the math. It's defending 'we're not going to fix this' when a VP demands reliability improvements. You need explicit agreement, in writing, on the target SLOs — so 'we're not fixing this' is 'we agreed on 99.9% and we're at 99.92%, which is within budget.'

The pragmatic answer

Most teams I've worked with should pick:

99.9% for production-critical services
99% for internal tools
Best-effort for dev environments

Then measure, spend the error budget, and stop arguing about it. The math is usually clearer than the politics.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Why Your Status Page Should Be Boring

Samson Tanimawo — Fri, 07 Aug 2026 01:28:56 +0000

A good status page is boring. Calm design, minimal copy, clear current state. If your status page feels exciting, something is wrong.

Here's what I've learned from running status pages for three different products.

What users actually want from a status page

1. Is it me or is it you? The #1 question. Answer it in the first 3 seconds of landing.

2. If it's you, what exactly is broken? Not 'we're experiencing issues.' Specifically: 'API endpoint /v2/checkout is returning 500 errors for ~15% of requests.'

3. How long until it's fixed? Even 'unknown' is better than no estimate. 'Investigating' with a last-updated timestamp beats silence.

4. Should I keep retrying? If you're broken and expect to stay broken for a while, tell users to back off. Your support queue will thank you.

What users don't want

Corporate-speak ('we're aware of a potential service degradation')
Vague promises ('working to resolve as quickly as possible')
Technical jargon they can't parse
Delayed acknowledgments (updating the page 20 minutes after an outage)

The update cadence rules

Acknowledge within 5 minutes of detection
Update every 15-30 minutes during active investigation
Mark as monitoring as soon as mitigation is in place, even if cause is unknown
Mark as resolved when you're confident it's fixed — not before
Always do a final post-incident summary

The hard part: being honest

The temptation is to minimize language. 'A small number of users' when it's actually 20%. 'Minor issue' when it's a real outage.

Don't. Users trust a status page that's honest with them. The first time you get caught minimizing, you've lost credibility that takes years to earn back.

The automation trap

Automated status pages that say 'all systems operational' while your product is clearly broken are worse than no page. Users lose trust in the entire signal.

If you automate, automate the detection to trigger human review. Don't automate the reassurance.

What a boring status page looks like

Uptime over 30 days. Current state of each service (green/yellow/red). A list of recent incidents with their post-mortems linked.

That's it. No marketing copy. No animated elements. Boring.

Boring is trustworthy. Make your status page boring.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Building Trust with Product Teams as an SRE

Samson Tanimawo — Thu, 06 Aug 2026 13:09:42 +0000

SRE teams that fight with product teams don't get things done. SRE teams that get along with product teams get surprising amounts of reliability work done by product engineers themselves. Here's how to build that trust.

Start by making them faster, not slower

The common SRE pattern: introduce yourself by adding gates. 'You need to do X before you can deploy.' 'Your service needs Y before production.' Product teams immediately see you as friction.

Better pattern: introduce yourself by removing friction. 'I noticed your CI takes 20 minutes. I can get it to 6. Interested?' Now you're useful. The gates come later, and you have trust to spend.

Speak their language

Product engineers care about: shipping, feature quality, user complaints. They don't care about: error budgets, SLIs, observability stacks.

Translate. Instead of 'we're over our SLO,' say 'users are seeing errors on checkout — here's what's hitting them and what it's costing.' The facts are the same. The reception is totally different.

Share ownership of incidents

When a product team's code breaks prod, resist the urge to fix it yourself. Instead: be in the room, coach them through it, let them own the fix.

Yes, it's slower. Yes, sometimes they'll ask awkward questions. That's exactly the point. They're learning. After 3 incidents, they'll be better engineers and more grateful to your team.

Give credit publicly

When a product team does reliability work well, say so. Publicly. In eng all-hands. On the CEO's Slack thread.

This sounds performative. It's not. It's you saying 'reliability is valued here, and we notice when people invest in it.' Other teams see it and start wanting credit too. You're using recognition as a reliability multiplier.

Absorb the hit sometimes

Sometimes a product team will say 'we don't have time for the runbook right now, can you just do it?' Say yes. Write the runbook. Don't make a thing of it.

Do this too often and you're a service team. Do this never and you're hostile. Do this sometimes, strategically, and you're building long-term trust. Read the room.

The end goal

After a year of this, product teams start coming to you before launches. 'We're about to ship X — any reliability concerns?' That's when you know you've won. They see you as a partner, not a checkpoint.

SRE culture work is slow. It compounds. Invest in it from day one.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com