DEV Community: Samson Tanimawo

The Engineer Who Owns Nothing: A Cautionary Tale

Samson Tanimawo — Fri, 12 Jun 2026 20:15:33 +0000

I'm going to tell you about an engineer I worked with. Call him Mark. Mark was talented, well-liked, and utterly ineffective. Here's what I learned from watching him.

What Mark did

Mark's technical skills were real. He wrote good code. He gave thoughtful design review comments. He spoke well in meetings.

Mark's problem: he didn't own anything.

He worked on whatever was in front of him. He fixed bugs in code he didn't write. He helped other engineers with their services. He never said 'this is mine.' Everything was 'we should figure out who handles that.'

Why this was a problem

When something broke in production, Mark would help debug — for a few minutes. Then he'd disengage because it 'wasn't really his area.' Nobody ever held him accountable because the code wasn't explicitly assigned to him.

When planning happened, Mark didn't propose projects. He waited for projects to be assigned to him. Assigned projects rarely came because leaders couldn't predict what he'd actually commit to.

When reliability work needed doing — alerts to tune, dashboards to fix, runbooks to write — Mark agreed it was important and waited for someone else to do it.

Mark got good performance reviews for the first two years. He was technically capable, pleasant, and unobjectionable.

What happened

At year three, the company hit hard times. Leadership started asking: 'what has this person done? what do they own?' Mark had no answer.

He got laid off. It wasn't a performance issue in the normal sense — he hadn't done anything wrong. He just hadn't owned anything.

The lesson for SREs

SRE is especially vulnerable to this trap. Everything is shared infrastructure. It's tempting to be the person who 'helps out everywhere' without ever claiming a specific service.

Don't. Claim something.

Own an SLO
Own a runbook library
Own the post-mortem process
Own the alerting hygiene for a service
Own the capacity planning model

It doesn't have to be big. But it has to be explicitly yours, with consequences if it fails and credit if it succeeds.

The meta-lesson

Ownership isn't the same as being busy. You can be in a million meetings and own nothing. You can write 10,000 lines of code and own nothing. Ownership means: when something in your area breaks, you are the first person called, and you are the person who gets credit when it works.

Pick something this week. Ask your manager: 'can I explicitly own X?' Write it down. Defend it.

Don't be Mark.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Error Budget Policies That Hold Leadership Accountable

Samson Tanimawo — Thu, 11 Jun 2026 21:23:20 +0000

Error budgets are useless without a policy. 'We're out of error budget' should trigger consequences. If it doesn't, you don't have an error budget — you have a vanity metric.

Here's a policy that actually works.

The four states

Healthy (< 70% of budget used). Business as usual. Feature development proceeds at full speed.

Watch (70-90% used). Feature velocity continues but new risky changes require explicit sign-off from an SRE. No gate, just attention.

Constrained (90-100% used). Feature freezes. Only reliability work and critical bug fixes until we're back below 90%.

Breached (> 100% used). Incident-level response. Leadership informed. Post-mortem for why we blew through. Feature work stays frozen until we recover and identify systemic causes.

The part most policies miss

The feature freeze in 'constrained' state is the part that actually changes behavior. Everything else is documentation. Without consequences, teams ignore the budget.

The freeze has to be real. Leadership can't override it for a 'really important feature' — that's exactly the time the freeze matters. The only exception is a legitimate emergency fix, and those should be rare.

Selling this to leadership

Executives hate feature freezes. They see it as slowing the business. Counter-argument: feature freezes during budget exhaustion protect the business. Shipping features onto broken infrastructure creates more breakage, which burns more budget, which is a doom loop.

Frame it as: 'the feature freeze is a safety valve. When it triggers, it's because something's wrong and we need to fix it before making it worse.'

Also: a good policy lets you spend the budget aggressively when you have it. Feature teams should be encouraged to experiment, deploy fast, and take risks when you're at 30% budget used. The freeze is only for when the safety margin is gone.

The review cadence

Weekly error budget review, 15 minutes max. Who attended: SRE lead, engineering manager, maybe a PM. Decisions: are we in healthy/watch/constrained? Any actions for the coming week?

Monthly broader review with leadership. Trends over time. Investment decisions.

The escalation

If a team enters 'constrained' state three times in a quarter, that's a systemic issue. Escalate to engineering leadership with a proposal: either invest in reliability or accept a lower SLO formally.

The endgame

A mature organization uses error budget policy to balance feature velocity against reliability automatically. Nobody is negotiating individual decisions. The framework does the work.

Getting there takes 6-12 months of discipline. The first few freezes will feel painful. After that, they become routine, and something surprising happens: you stop having them as often. The policy is working.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Dependency Injection for Observability

Samson Tanimawo — Wed, 10 Jun 2026 20:47:02 +0000

Want your code to be easy to observe? Use dependency injection for observability concerns. Sounds dry. Hear me out.

The problem

Your code calls log.info(...) directly. In tests, you can't verify what was logged. In prod, if you want to change the logger, you're grepping the codebase. If you want to add tracing, you're editing every call site.

Same for metrics. Same for tracing. Same for error reporting.

The fix

Pass the observer in. Functions take a logger, metrics, or tracer as an argument (or constructor dependency). The function doesn't know what's behind it.

func HandleOrder(order Order, deps Deps) error {
    deps.Metrics.Inc("order.received")
    deps.Logger.Info("processing order", "id", order.ID)
    // ...
}

Now:

In tests, pass in a mock that captures all calls
In prod, pass in the real logger
Changing backends (Datadog → Prometheus) is one wire-up change
Adding tracing is one new field in Deps

Why most codebases don't do this

It feels verbose. 'Why do I have to thread a logger through every function?' Engineers hate boilerplate.

The alternative is a global singleton. Easy to use, impossible to test cleanly, nightmare to refactor.

The boilerplate is worth it. Especially for observability, where you will want to swap implementations later.

The subtle win

Dependency-injected observability forces you to think about what you're observing. When you have to explicitly pass the logger, you notice that a function is calling it 8 times. Is that too much? Is the logging doing real work? Would one structured log at the end of the function be better?

Functions with injected dependencies tend to have better observability because the developer had to look at it.

The starting point

You don't need to refactor everything at once. Pick one critical path — the checkout flow, the auth path. Refactor just that path to use dependency injection for observability. See if tests and debugging get easier.

If yes, expand. If no, you found something else is the bottleneck.

The bigger idea

Observability is code. Treat it with the same architectural discipline you treat the rest of your codebase. Dependency injection is one tool. There are others (context objects, middleware, decorators). Pick one, apply it consistently, and your future-you will be able to observe your code in ways that are impossible with ad-hoc log.info calls everywhere.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Load Balancer Tuning: Lessons from Production

Samson Tanimawo — Tue, 09 Jun 2026 20:36:49 +0000

Load balancers are the silent infrastructure. You don't think about them until they start dropping connections at 2 AM. Here are the settings that have bitten me in production.

Connection timeouts

Default connection idle timeout on AWS ALB is 60 seconds. Your app might have requests that legitimately take 90 seconds. Result: the ALB drops the connection mid-request, your user sees a 502, and your logs show nothing because the app was still processing.

Rule: set the LB idle timeout higher than your longest legitimate request. For most APIs, 120-300 seconds. Verify it matches your app's own timeout.

Health check intervals

Default health check interval is usually 30 seconds, with 2 failures before marking unhealthy. That's up to 60 seconds of traffic sent to a dying instance before it's removed.

Rule: 10-second intervals, 2 failures. 20 seconds to remove a bad instance is much better. Yes, slightly more load on your service. Worth it.

Unhealthy threshold

Marking an instance unhealthy after 2 consecutive failures is usually right. Marking it healthy again after 2 consecutive successes is usually wrong — it lets half-broken instances come back prematurely.

Rule: require 3-5 consecutive successes to mark healthy. Slower recovery, more stable routing.

Slow start / connection draining

When a new instance comes online, it's often cold — empty caches, no warmed connections. If the LB sends it full traffic immediately, it performs badly and might get marked unhealthy.

Rule: enable slow start (AWS calls it 'slow start mode' on ALB). Ramp traffic over 30-60 seconds. Worth it for any service that needs warming.

Sticky sessions

Sticky sessions feel like a solution. They're usually a problem.

Rule: avoid them. If you need them, your app has shared state that should be in a database or cache, not in-process. The one exception: WebSocket connections, where stickiness is unavoidable.

Cross-zone load balancing

Disabled by default on some LBs. With cross-zone disabled, an instance in AZ-A only serves traffic from AZ-A. If AZ-A has fewer instances, that AZ's traffic gets uneven distribution.

Rule: enable it unless you have a specific reason not to. Costs a small amount of cross-AZ traffic but gives you even load distribution.

The meta-lesson

Load balancer defaults are designed for 'will work for everybody, badly.' Any given workload needs tuning.

Spend one afternoon reviewing your LB configs. Ask: 'is this default right for my app?' At least half the defaults will be wrong. Tune them. Future you will thank you during the next incident.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Capacity Planning for Startups

Samson Tanimawo — Mon, 08 Jun 2026 20:14:20 +0000

Capacity planning sounds like enterprise spreadsheet work. For a startup, it's 'don't get embarrassed when traffic spikes, don't go broke overprovisioning.' Here's the pragmatic version.

The 3 questions

1. What does normal look like right now? Peak RPS, p99 latency, CPU/memory utilization at peak. If you don't know these, stop and measure. You cannot plan without baseline.

2. What's the next expected spike? A launch. A press mention. A marketing campaign. The Black Friday of your industry. Put these on a calendar.

3. How long does it take to add capacity? Minutes (autoscaling)? Hours (VM provisioning)? Weeks (vendor contracts)?

Your capacity plan is the gap between expected spike and response time.

The startup hack: overprovision early

At startup scale, overprovisioning is cheap. An extra $5k/month of slack is trivial compared to the embarrassment of 'we went down during the launch.'

Run at 30-40% peak utilization. Yes, that's wasteful. It's also a 3x buffer for unexpected spikes. Worth it until you're big enough to care about the efficiency.

The autoscaling reality

Autoscaling is great but has limits:

Cold start times mean it can't handle traffic that doubles in 30 seconds
Provisioning limits mean you can only add X instances per minute
Downstream dependencies (databases, queues) usually don't autoscale

Test your autoscaling before you need it. The first time I depended on autoscaling in production, it worked. The second time, it didn't, because our database connection pool was capped. That was the real bottleneck.

The 3 bottlenecks to check

For every scaling test, verify:

Stateless service capacity. Usually easy — just add more instances.
Database capacity. Connection counts, query latency, replication lag. Usually the real bottleneck.
Third-party dependencies. Rate limits on external APIs, email providers, payment processors. A sudden 10x spike usually hits someone's rate limit.

The launch checklist

Before any planned spike:

Overprovision by 3x what you think you need
Pre-warm caches and connection pools
Confirm your paging rotation is ready
Prepare a rollback plan for the feature being launched
Schedule the launch during your team's awake hours, not off-hours

The real lesson

For startups, the goal of capacity planning is not efficiency. It's confidence. If you have to spend a little more to avoid panic during growth, spend it. Optimize for efficiency later, when you have a year of traffic history to work from.

Right now, your job is to stay standing. Do that first.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

How We Handled Our First Major Outage (And Survived)

Samson Tanimawo — Sun, 07 Jun 2026 21:13:56 +0000

Three years ago we had our first real outage. Six hours of downtime. Thousands of angry users. Multiple executives on the call. Here's what we did right, what we did wrong, and what we'd do differently.

What we did right

1. Communicated immediately. The moment we knew we had a problem, we updated the status page and emailed our biggest customers personally. Not when we had answers. When we had a question.

2. Had a single incident commander. One person making calls. Not a committee. When the CEO tried to direct technical work, the IC politely rerouted and told her where her help was actually needed (talking to customers).

3. Took care of our people. During hour 4, I ordered food. During hour 5, I forced the primary engineer off the call for 20 minutes to walk outside. Long incidents destroy people. You have to feed them and force them to rest.

4. Wrote it down as we went. We had a shared doc with a live timeline. When the post-mortem came, we had every decision captured.

What we did wrong

1. Tried to fix the root cause during the incident. For the first 2 hours, we were digging into why the database was struggling. We should have been mitigating (rolling back) first.

2. Let too many people 'help.' By hour 3, we had 12 engineers in the call. Half of them were useless. The IC should have kicked people out sooner.

3. Gave optimistic estimates. 'We'll be back in 30 minutes.' We were not back in 30 minutes. That miscommunication was worse than saying 'unknown.'

4. Didn't prepare the executive communication. The CEO had to answer customer questions in real time with no script. We should have drafted talking points for her after hour 1.

What we'd do differently

Mitigate first, investigate second. Always.
Cap the number of active engineers at 4 during an incident. Others go on standby.
Default to 'unknown' for estimates. Only give a number when we're sure.
Assign someone explicitly to 'executive liaison.' Their job is to keep the C-suite informed without interrupting the technical team.

The aftermath

The post-mortem was brutal and cathartic. We identified 14 action items. We actually did 11 of them over the next quarter.

The outage was the best thing that happened to our reliability culture. It turned reliability from 'a thing SRE owns' into 'a thing everyone takes seriously.' I wouldn't wish a 6-hour outage on anyone, but I also wouldn't trade the lessons.

The final lesson

Your first major outage will happen. Prepare for it by running game days. The game days will feel silly until the real thing happens, at which point every muscle you trained will kick in.

Incident response is a skill. Skills need practice. Practice now.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

The Economics of Reliability: When to Invest, When to Accept Risk

Samson Tanimawo — Sat, 06 Jun 2026 20:16:40 +0000

Reliability is not a virtue. It's an investment. Too little and you lose customers. Too much and you can't afford to ship. The question is: where's the right balance?

The error budget framing

The SRE book covers this well. Pick an SLO (say 99.9% uptime). That's 43 minutes of budget per month. If you're at 99.95%, you have budget to spend on risky things. If you're at 99.85%, you need to stop shipping risk.

This works. But it doesn't answer 'what SLO should I pick?' Let me give you a framework.

The three questions

1. What do users expect? A consumer banking app needs 4 nines or more. A developer tool can get away with 3. A beta product can live with 99%. Ask users (or watch churn numbers).

2. What does an outage cost? Dollars of lost revenue + dollars of customer churn + hours of engineering time. For a checkout-heavy product, an hour of downtime might cost $500k. For a B2B internal tool, $5k.

3. What does the next 9 cost? Going from 99% to 99.9% might cost $50k of engineering work. Going from 99.9% to 99.99% often costs $500k or more. Each 9 is 10x harder.

The math

Invest in reliability up to the point where the next $1 invested saves less than $1 of outage cost over the amortization period.

If moving from 99% to 99.9% costs $50k and would save $200k over a year in reduced outage damage, invest. Easy call.

If moving from 99.9% to 99.99% costs $500k and saves $100k, don't invest. Accept the risk, and spend the engineering time on something with better ROI.

The hidden cost

Over-investing in reliability has a hidden cost: team velocity. Teams that chase 99.999% uptime spend so much on tests, canaries, staging environments, and approval gates that they ship slowly. Competitors with 99.5% reliability but 5x your velocity will win the market.

Reliability that kills velocity is bad reliability. Measure both.

The political reality

The hardest part isn't the math. It's defending 'we're not going to fix this' when a VP demands reliability improvements. You need explicit agreement, in writing, on the target SLOs — so 'we're not fixing this' is 'we agreed on 99.9% and we're at 99.92%, which is within budget.'

The pragmatic answer

Most teams I've worked with should pick:

99.9% for production-critical services
99% for internal tools
Best-effort for dev environments

Then measure, spend the error budget, and stop arguing about it. The math is usually clearer than the politics.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Why Your Status Page Should Be Boring

Samson Tanimawo — Fri, 05 Jun 2026 20:41:09 +0000

A good status page is boring. Calm design, minimal copy, clear current state. If your status page feels exciting, something is wrong.

Here's what I've learned from running status pages for three different products.

What users actually want from a status page

1. Is it me or is it you? The #1 question. Answer it in the first 3 seconds of landing.

2. If it's you, what exactly is broken? Not 'we're experiencing issues.' Specifically: 'API endpoint /v2/checkout is returning 500 errors for ~15% of requests.'

3. How long until it's fixed? Even 'unknown' is better than no estimate. 'Investigating' with a last-updated timestamp beats silence.

4. Should I keep retrying? If you're broken and expect to stay broken for a while, tell users to back off. Your support queue will thank you.

What users don't want

Corporate-speak ('we're aware of a potential service degradation')
Vague promises ('working to resolve as quickly as possible')
Technical jargon they can't parse
Delayed acknowledgments (updating the page 20 minutes after an outage)

The update cadence rules

Acknowledge within 5 minutes of detection
Update every 15-30 minutes during active investigation
Mark as monitoring as soon as mitigation is in place, even if cause is unknown
Mark as resolved when you're confident it's fixed — not before
Always do a final post-incident summary

The hard part: being honest

The temptation is to minimize language. 'A small number of users' when it's actually 20%. 'Minor issue' when it's a real outage.

Don't. Users trust a status page that's honest with them. The first time you get caught minimizing, you've lost credibility that takes years to earn back.

The automation trap

Automated status pages that say 'all systems operational' while your product is clearly broken are worse than no page. Users lose trust in the entire signal.

If you automate, automate the detection to trigger human review. Don't automate the reassurance.

What a boring status page looks like

Uptime over 30 days. Current state of each service (green/yellow/red). A list of recent incidents with their post-mortems linked.

That's it. No marketing copy. No animated elements. Boring.

Boring is trustworthy. Make your status page boring.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Building Trust with Product Teams as an SRE

Samson Tanimawo — Thu, 04 Jun 2026 20:16:58 +0000

SRE teams that fight with product teams don't get things done. SRE teams that get along with product teams get surprising amounts of reliability work done by product engineers themselves. Here's how to build that trust.

Start by making them faster, not slower

The common SRE pattern: introduce yourself by adding gates. 'You need to do X before you can deploy.' 'Your service needs Y before production.' Product teams immediately see you as friction.

Better pattern: introduce yourself by removing friction. 'I noticed your CI takes 20 minutes. I can get it to 6. Interested?' Now you're useful. The gates come later, and you have trust to spend.

Speak their language

Product engineers care about: shipping, feature quality, user complaints. They don't care about: error budgets, SLIs, observability stacks.

Translate. Instead of 'we're over our SLO,' say 'users are seeing errors on checkout — here's what's hitting them and what it's costing.' The facts are the same. The reception is totally different.

Share ownership of incidents

When a product team's code breaks prod, resist the urge to fix it yourself. Instead: be in the room, coach them through it, let them own the fix.

Yes, it's slower. Yes, sometimes they'll ask awkward questions. That's exactly the point. They're learning. After 3 incidents, they'll be better engineers and more grateful to your team.

Give credit publicly

When a product team does reliability work well, say so. Publicly. In eng all-hands. On the CEO's Slack thread.

This sounds performative. It's not. It's you saying 'reliability is valued here, and we notice when people invest in it.' Other teams see it and start wanting credit too. You're using recognition as a reliability multiplier.

Absorb the hit sometimes

Sometimes a product team will say 'we don't have time for the runbook right now, can you just do it?' Say yes. Write the runbook. Don't make a thing of it.

Do this too often and you're a service team. Do this never and you're hostile. Do this sometimes, strategically, and you're building long-term trust. Read the room.

The end goal

After a year of this, product teams start coming to you before launches. 'We're about to ship X — any reliability concerns?' That's when you know you've won. They see you as a partner, not a checkpoint.

SRE culture work is slow. It compounds. Invest in it from day one.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Incident Command: The Skills They Don't Teach You

Samson Tanimawo — Wed, 03 Jun 2026 20:24:48 +0000

Running a production incident is a skill. Most of the skill isn't technical. Here's what nobody told me when I started running incidents.

Skill 1: Calling the cadence

During an incident, time warps. Everyone is heads-down in logs. Nobody remembers when they last updated the status channel.

The incident commander's job is to force a cadence: 'Update in 5 minutes. What do we know? What do we need?' Without this, the incident drags on because no one is aggregating context.

Skill 2: Saying 'I don't know, and here's what we're doing to find out'

Stakeholders want certainty. You can't give it. The temptation is to guess.

Don't guess. Say 'We don't know the cause yet. We're investigating X and Y. I'll update in 10 minutes.' Trust builds on honesty, not performance.

Skill 3: Interrupting your engineers

Your engineers are investigating. They don't want to stop and explain. But if you don't interrupt, you can't make decisions.

Do it anyway. Say 'I know you're busy. 30 seconds — what have you learned?' Most engineers will appreciate the structure, even if they complain.

Skill 4: Knowing when to stop investigating and start mitigating

The temptation in an incident is to find the root cause. The right action is usually to mitigate first and investigate second.

'We don't know why, but rolling back stops it' is a win. Don't feel bad. The post-mortem can figure out why.

Skill 5: Managing morale

Long incidents grind people down. Notice when your team is flagging. Bring in a relief shift. Say 'good job, let's take 10.' Acknowledge that it's hard.

The worst incidents I've been in were the ones where the team ran out of emotional energy before the problem was fixed. That's on the commander.

Skill 6: Declaring the incident over

Incidents often drag on past actual resolution because nobody wants to declare victory. Declare it. 'Issue resolved at 15:47. We'll keep monitoring for 30 minutes but the incident is closed.' People need permission to exhale.

The real job

Incident command is emotional labor disguised as technical work. The best commanders I know are calm, honest, and generous with credit. The worst are the ones who try to be the smartest technical person in the room — that's not the job.

You can't learn this from a book. You learn it by running incidents badly and asking for feedback afterward. The feedback is usually uncomfortable. It's also how you get good.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

How AI Is Changing SRE Workflows (Without Replacing SREs)

Samson Tanimawo — Tue, 02 Jun 2026 20:46:44 +0000

I get asked this question a lot: 'Is AI going to replace SREs?' Short answer: no. Long answer: AI is changing what SREs spend their time on, and the SREs who adapt will have a huge edge.

What AI is actually good at in SRE workflows

1. First-pass triage. AI can look at 50 alerts and tell you the 5 most likely to be related to an ongoing incident. Beats manual correlation every time.

2. Log summarization. 100,000 log lines into a 20-line summary highlighting anomalies. The summary isn't always right, but it's a starting point.

3. Runbook generation. Given an alert type and historical incidents, AI can draft a runbook. You edit it; you don't write from scratch.

4. Post-mortem first drafts. Pull from chat logs, ticket history, monitoring data. Generate a structured timeline. Human polishes it.

5. Routine query generation. 'Show me the error rate for service X grouped by endpoint for the last 24 hours.' AI writes the query, you run it.

What AI is bad at

1. Judgment calls under pressure. When multiple things could be wrong, a good SRE uses instincts built from years of experience. AI guesses.

2. Novel failures. AI is pattern-matching on history. A truly new failure mode looks like noise to it.

3. Organizational politics. 'Who do I wake up at 3 AM' is not a technical question. AI doesn't help here.

4. Accountability. When something breaks, someone needs to own the decisions that got made. AI can't own anything.

The new SRE workflow

AI does the first 30%. Human does the crucial middle 40% (judgment, decision-making, stakeholder communication). AI does the last 30% (writing up, following up, documenting).

This lets SREs handle 2-3x more incidents with the same quality. Not by working harder — by delegating the mechanical parts.

What to learn

If you're an SRE and you're not using AI tools in your workflow, you're leaving 30% of your productivity on the table. Not because AI is magic, but because the boring parts of your job don't need a human.

Start small: feed an alert into ChatGPT or Claude and ask for possible causes. See what you get. Then add it to your actual on-call tooling.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

Security Monitoring for SRE Teams

Samson Tanimawo — Mon, 01 Jun 2026 20:54:02 +0000

Security used to be a separate team. Increasingly, SRE teams are being asked to own the monitoring side of it. Here's a practical framework that doesn't turn you into a SOC analyst.

What you already have

Your existing observability stack is half of a security monitoring solution. You already collect logs, metrics, and traces. You already have alerts. The missing piece is usually what to look at.

What to add

1. Authentication anomalies. Alert on impossible logins (user from Tokyo, then from Paris 10 minutes later), brute force patterns, and unusual session durations.

2. Privilege escalation patterns. New admin role granted. Service account added to a sensitive group. Kubernetes role binding changed.

3. Unusual data access. A user or service reading 10x the normal volume of records. Downloads from sensitive S3 buckets. Queries against PII tables by accounts that don't normally touch them.

4. Outbound traffic anomalies. A process that has never called an external IP suddenly connecting to one. Large egress volumes during off-hours.

5. Failed auth spikes. Not just the login endpoint. Internal auth, API keys, mTLS — anywhere auth happens.

What not to do

Don't try to build a SIEM yourself. You'll burn out chasing alerts. Either use a managed SIEM or stick to a small set of high-signal alerts.

Don't treat security alerts like reliability alerts. Security alerts often need investigation, not immediate fix. The triage workflow is different.

The handoff

If you're on the SRE side owning security monitoring, you need a clear escalation path to someone who does security full-time. Your job is detection; theirs is response.

The worst pattern: SRE team gets a suspicious alert, doesn't know what to do with it, and tables it. Weeks later, a real incident traces back to that alert. Define the handoff up front.

The quick win

If you do nothing else, implement #1 and #5. Failed auth spikes and login anomalies catch 70% of opportunistic attacks. The rest is shoring up for the 30%.

Security monitoring is just reliability monitoring with a different threat model. Treat it the same way: start with high-signal basics, prune noise aggressively, and escalate when you're unsure.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com