Mr. November and the Myth of the Green Dashboard

#devops #sre #metrics #architecture

I was wondering how often we let a big green dashboard fool us into thinking our systems are actually healthy, I mean, it was not actually just wondering but I used this topic in a job interview last week. It turns out, sports analytics solved this exact same riddle years ago and IT specialists can -and should- learn a lot from how scouts separate the box score, the huge amount of saber metrics and even the Money Ball ideas from the game tape.

Take Derek Jeter -even if you are not a big fan of the Yankees-. If you evaluate his career strictly through modern defensive metrics like Ultimate Zone Rating (UZR) or Defensive Runs Saved (DRS), the data is brutal¹. Statistically, he grades out as one of the worst fielding shortstops in baseball history. If Jeter were a microservice, an automated auditing tool would flag him for immediate deprecation.

Yet, nobody in their right mind would cut him from a roster. Why? Because metrics struggle to quantify high-leverage execution. When the playoffs arrived and the pressure maxed out, Jeter became “Mr. November.” The box score said his range was a liability, per se, but the game tape showed the guy who hit a walk-off home run in Game 4 of the 2001 World Series and anchored five championship teams. Now, look at how we run IT operations. We have the exact same blind spot, and it cuts both ways.

1. The Legacy “Mr. November” in Your Stack

Every engineering org has a Jeter in their architecture. It is that clunky, monolithic service written eight years ago. It fails every modern DORA metric you throw at it: deployment frequency is low, automated test coverage is sketchy, and its p99 latency isn’t winning any awards and might be eating a big piece of your budget error pie.

On paper, it looks like a technical debt that needs to be rewritten immediately. But when peak day traffic hits and your shiny, stateless, Kubernetes pods² start thrashing under unexpected connection pooling issues, that ugly legacy monolith just sits there, quietly processing 100% of your checkout transactions without dropping a single packet. The metrics say it’s a liability; the game tape says it’s keeping the company in business.

2. The 2010 Chargers and the “All-Green” Outage

What happens when you have the exact opposite problem, I mean, when you optimize purely for the box score and ignore the tape or the old school scout six sense?

Look at the 2010 San Diego Chargers. They finished the regular season ranked #1 in the NFL in Total Offense Yards and #1 in Total Defense Yards -of course this one means the least amount because they just allowed 4,345-. By the two biggest macro-metrics in football, they were an unstoppable juggernaut. Yet, they didn’t even make the playoffs. Their special teams unit -the joint between offense and defense- was catastrophic, giving up blocked punts and kickoff returns - they gave up 528 yards which is almost 200 above the league average³- that cost them game after game. We see ships capsizing -pun intended- like this every week in IT as SRE.

Your infrastructure metrics are great: CPU and memory utilization across your nodes are sitting at a comfortable 40%. Your CI/CD pipeline metrics are awesome: 100% of builds are passing and deployed in under four minutes, no disruptions. Your Grafana dashboards are a sea of calming green.

Meanwhile, your users can’t log in because an IAM policy update silently broke a cross-region API gateway, or a third-party DNS provider is blackholing traffic from half the country. You won the infrastructure box score, but you lost the customer game tape.

Watching the Tape in IT

Collecting telemetry is easy or is supposed to be easy, you got the point. We can log every container restart, trace every database query, and alert on every minor spike. But measuring everything is not the same as understanding anything.

When we rely strictly on raw metrics without context, we make bad engineering decisions. We spend months refactoring stable legacy systems just to make a code-quality dashboard look prettier, or we celebrate 99.99% server uptime while the actual user point of view is very bad and our systems looks completely broken.

Metrics aren’t useless per se, they tell you where to look at. But they will never replace the operational context of watching how your systems and your engineers, actually behave when the game is on the line.

And that’s the whole point of this, you can even have someone who codes as the gods: superhuman speed and reliability, flawless scripts, elegant lines of code but, does this uber-coder behaves with the same performance during a red sea of alerts? Not everyone is capable to just perform on average during an outage -of any kind- and of course we also have metrics for that⁴, I know, the whole point of this post it’s to complain about them and now here we are… Jokes aside, even those metrics will not measure exactly what we want, because the metrics are just the mathematical representation of something we defined and even that they mean something inherently, we need to learn how to interpret every piece of data and WHEN to interpret in a certain way.

Data are not flat 2D figures that look nice in a dashboard (I mean, they do look really nice) data are vectors that moves across time and direction, which means that location is important, understanding that and make your team able to do the same are the most unseen tasks as SRE⁵ .

Thanks for reading. Leave a comment and subscribe!

DEV Community

Mr. November and the Myth of the Green Dashboard

Top comments (0)