DEV Community: Raleigh Schickel

The Finding Nobody Implemented

Raleigh Schickel — Thu, 25 Jun 2026 17:23:21 +0000

The DORA research said culture predicts engineering performance. Nicole Forsgren's most important finding never made it into a single commercial product.

As a computer scientist, I love data. Things feel good, things feel bad, but our biases shape those feelings, and data is what pulls the signal out. I've believed that since my first software engineering job.

At that first job, I was fortunate to work with people who fully believed in measuring things. This was before the age of observability, but the organization was advanced in terms of how it measured the product. Not from a data analytics standpoint. I mean that as engineering teams, as we built features, we made hypotheses about behavior and made sure we measured it. We had anomaly detection in place well before the analytics team did.

For all that, the only things we tracked about the engineering team itself were deployment frequency (we were an early continuous deployment shop) and whether we hit our delivery dates. That was the extent of it.

In 2017, I attended DevOps Days. That was the year they did the Monsters of DevOps tour: Gene Kim, Jez Humble, Kelsey Hightower, Nicole Forsgren, among others. Hightower threw out his prepared talk and just told his story. An unexpected diversion into human struggle and vulnerability in a conference about automation. My favorite talk was still Forsgren's. She presented the data on correlations between deployment frequency and high-performing organizations, and then spent the last fifteen or twenty minutes on something I hadn't heard anyone talk about seriously before: how to measure culture using Westrum typology and the research behind it.

Hightower got there from a completely different direction and landed in the same place. The people who gave a damn were the ones producing the generative cultures Forsgren was measuring.

Of everything I heard that week, it was the ability to measure culture that landed hardest. I spent the next couple of years trying to figure out how to actually do it, because she didn't show the algorithm. The internet had almost nothing on operationalizing Westrum at that point. Over time, as her work gained traction, more information surfaced, and I was eventually able to piece together a measurement approach. I've run it at several organizations since, usually when I first join. If you want to move culture in a direction, you need to know where it is.

That was 2017. We are now most of the way through 2026, and the industry still hasn't figured out what to do with that finding.

DORA came out of that same research community, with Forsgren as one of its architects. If you read the DORA reports carefully, culture is there. Westrum shows up. The acknowledgment that organizational culture type is one of the strongest predictors of software delivery performance made it into the text. And then every commercial implementation of DORA metrics that I've ever seen quietly dropped it. Deploy frequency, lead time for changes, change failure rate, mean time to restore. Those made the dashboards. Culture didn't.

That's not an accident, and it's not just that culture is harder to dashboard. It's that a Westrum score is inconvenient in a way that a deployment frequency metric never is. Deployment frequency can't indict you. A culture assessment that shows your organization is pathological or bureaucratic absolutely can. Nobody building a commercial product wanted to be the tool that told a VP their engineering culture was broken, because that VP is also the buyer. So the finding that most directly implicates leadership quietly became an appendix, and every tool in the market followed the same logic.

Last week Cortex released DRIVE, a new framework for measuring engineering organizational health. It covers Delivery, Reliability, Initiatives, Vigilance, and Efficiency. The metrics are deploy frequency, lead time for changes, incident counts, CVE status, cloud spend, token costs. They are, in other words, DORA metrics with some security and infrastructure measurements added.

Cortex is upfront about the philosophy behind it. Their website opens with this: "Software engineering is having its industrial revolution. We've gone from writing code by hand to building software factories." They go further in their OpEx review section, which describes a practice with "roots in manufacturing, where Operational Excellence emerged as a discipline for treating an entire factory as one observable, continuously improving unit." Amazon and Google are cited as exemplars.

Culture does not appear in the framework.

Cortex isn't wrong about the factory framing. AI agents are allowing some engineering work to be done in a more factory-like sense. It's fair to introduce new concepts in which to measure these changes and their impacts. But until the engineering organization is composed of nothing but agents, this is just a layer on top of the most important part.

Engineering teams are people. The work is creative, non-deterministic, dependent on psychological safety and trust in ways that deployment pipelines are not. The industry has been trying to apply factory floor logic to that work for decades, and the frameworks keep reflecting that aspiration back at us while leaving Forsgren's most important finding on the floor.

AI is accelerating this. The workforce cuts in favor of agent spend, the framing of engineers as replaceable capacity to be optimized rather than people worth developing and leading. These aren't new instincts, they're old instincts with better cover. I use AI every day. It absolutely increases capacity. But the question of whether your engineers feel psychologically safe, whether they feel ownership over their work, whether the organization is generative or pathological. None of that changes because you gave everyone a coding agent. If anything it gets harder to see, because the output metrics look better while the culture quietly degrades.

Forsgren's research didn't just find that generative culture correlates with good delivery metrics. It found that culture is a precondition for the practices that produce those metrics in the first place. The causality runs culture first, practices second, metrics third. Organizations running DORA dashboards without addressing culture are missing the foundational preconditions for the metrics to mean anything. And when the numbers don't move, the response is to mandate the number. Deploy more frequently. Reduce your lead time. The metric becomes the goal, which is exactly backwards. Deployment frequency goes up when people who give a damn go fix their deployment pipelines. The dashboard didn't do that. The culture did.

Hire the right people, they'll build the culture, which implements the practices, which enables the metrics. The industry keeps trying to shortcut to the last step.

Meta is the clearest current example and also the most complicated one, in both directions. For most of their existence they were revered as a top-tier engineering organization. Not because they stumbled into good culture but because Zuckerberg built it deliberately. The psychological safety, the bottom-up technical decision-making, the model of engineers choosing projects they believed in. That was a product of his leadership, and he understood exactly what it produced.

He's now dismantling it. Keystroke loggers. Forced reconfigurations into data labeling work. Engineers who once had near-full autonomy over what they worked on are being redirected by executive edict. The Westrum survey would show you exactly where that culture score is heading. It wouldn't change anything, because the person who built the culture has decided the frontier AI race is worth the cost, and he owns the company. No board is going to stop him.

That's the real limit of measurement, and it's worth being clear about it. The data tells you where you are. It can't govern what a decision-maker chooses to do next. What it can do is make the cost visible to leaders who are willing to look and to engineers deciding where they want to work.

The finding Forsgren put in that room in 2017 was real. The research was solid. The industry picked up everything around it and left the most important part behind, and the frameworks keep arriving to confirm that choice. DRIVE is just the latest one.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

Debugging Predictability at the Team Level

Raleigh Schickel — Mon, 22 Jun 2026 14:28:06 +0000

What the numbers are trying to tell you, and how to get them to say it out loud.

In the last post I wrote about what predictability is and what it actually measures. This one is about what to do when the number looks wrong.

When I first started using this metric, diagnosing an anomaly took real time. I'd read through everything, cross-reference the supporting metrics, form a hypothesis, test it against the data. It was slow. But practice creates patterns, and over time I've learned that there are only a handful of situations that tend to produce the readings you're going to see in most organizations. The root causes fall into recognizable categories. And once you know what to look for, the chain of evidence is usually pretty short.

The patterns I see most often:

The undercommitting team. The overcommitting team. The team that never fully understood the scope. The product manager who cannot say no. The organization whose leadership cannot stay focused long enough to let a team finish anything. And the team that's being eaten alive by quality issues because nobody ever decided that quality was a cultural priority.

There are probably others. But those cover most of what I've seen across a dozen or so organizations over the years.

I'm going to walk through two of them in detail, because the point isn't to give you a taxonomy. The point is to show you how to read the chain of metrics so you can do the diagnostic yourself, whatever pattern you're looking at.

The product manager who cannot say no

I walked into an organization and ran my standard first-week metrics pull: the last six months of sprint data across all the teams. Before I even opened the spreadsheet I already knew one thing: this particular team had a hard time meeting project delivery dates. That was the organizational understanding. Nobody had been able to explain why.

I looked at predictability first. Sprint over sprint, it was declining, and had been for months. It was sitting in the 30% to 40% range. That's not a rough patch. That's a pattern.

The next thing I looked at was velocity. Velocity was flat, maybe slightly increasing. The amount of work this team was completing every sprint was actually quite stable. So the team wasn't falling apart. They were getting work done at a consistent pace. Which meant the predictability problem wasn't about execution capacity.

So I looked at sprint commitments. And that's where it fell apart.

Every sprint, the team was completing less than the original commitment. Originally just consistently less, but over time it grew to be dramatically less. And yet, the commitment on the next sprint was always larger than the commitment on the previous sprint. Sprint after sprint after sprint. The team was behind, and the plan kept getting bigger.

The mechanism was two things happening simultaneously. Anything not completed in a sprint rolled over into the next one. And the product manager responsible for prioritization kept adding new work on top of the rollover. Nobody was removing anything. The pile just kept growing.

This wasn't an engineering failure. The engineers were doing what they said they'd do, roughly speaking, and doing it at a consistent pace. The problem was that nobody was willing to make hard choices about what should be done in a given period of time. The product manager wasn't able to look at the rollover, look at the new requests coming in, and say: these can wait. As a result, the commitment was always unrealistic, the team was always behind, and the project was always going to miss its date because the work that mattered for the project was competing with everything else on the list.

The data made that visible. It took about twenty minutes to read.

The team that never fully understood the scope

Different organization. One of my teams was consistently hitting around 150% predictability. That means every sprint they were completing significantly more work than they originally committed to. You might read that as a good thing. I read it as a question.

Are they just underestimating what they can do? Or is something else happening?

Velocity looked fine: flat to increasing. That ruled out the team falling apart. Then I looked at the percentage of the original plan completed. And here's where it got strange. They were doing more total volume than planned, but they were completing less of the actual planned work than they said they would. More work done, less of the right work done. That's a specific shape.

The next metric: work added to sprints. Every sprint, twenty to thirty points of new tickets were being added. And nothing was being removed. Good sprint hygiene says that when you add work mid-sprint, you should pull something else out to protect the commitment. That wasn't happening here.

At that point I started looking at the actual tickets being added. What I found was that as engineers worked on planned tickets, they kept discovering work that hadn't been captured anywhere. Not tangential things, not scope creep in the traditional sense. Work that was required to complete the project, that nobody had identified before the sprint started. They were adding it because it had to be done. And this was happening sprint over sprint over sprint.

The accountability here isn't really about individuals dropping the ball. This was an early-stage company that had been operating Kanban, just-in-time, figure-it-out-as-you-go. The shift to sprints happened for real reasons: the company had grown, other parts of the organization needed to plan around Engineering, and the API dev tools space runs on external release cadences that don't wait for you. Sprints were the right call. But adopting sprint ceremonies without installing sprint discipline just moves the chaos inside the timebox. The engineers had never been in an environment where deep pre-sprint scoping was expected, and nobody was giving them the time to do it even if they'd known to ask.

The team had developed an intuition about this. They knew from experience that there was always more work hiding in the project than had been identified, so they undercommitted on their initial estimates to give themselves room. The undercommit wasn't pessimism. It was the scar tissue of a team that had never broken the cycle.

Which means the learning wasn't primarily theirs. If the team had never been taught to plan at that level of depth, that's not a performance problem. That's a coaching gap. And the coaching gap belongs to me.

What happens after you have the diagnosis

The metrics get you to the root cause. What happens next is a different problem entirely, and I want to be honest about the distinction.

How I bring the diagnosis into a conversation depends on what hat I'm wearing. When I'm acting as a Scrum Master and running retros directly, I can bring this into the room myself. It's the seed for the entire retro. I'll put the historical charts up, not just the most recent sprint, and I'll ask the team: How did we do? What do you think is happening here? This is a little outside what we'd expect. What's behind it?

The goal is to point them toward the topic and let them find the answer. Usually they will. Their observations might not go straight to the root cause, but they're not wrong. They're seeing real things. You listen, you validate what's real, and then you keep asking why until you get to the thing underneath the thing. It's the classic five whys, except you're doing it in a room full of people who know things you don't.

If they can't get there on their own, I'll shift tactics: here's the chain I followed. I started with predictability, looked at velocity, then looked at what was being added to sprints. And when I looked at the tickets that got added, here's what I noticed. What's the story behind that?

When I'm not in the Scrum Master role, when I'm functioning as Head of Engineering with team managers between me and the teams, the channel is different but the approach is the same. Somewhere in my regular one-on-ones with my managers, I'll shift to: how's the team doing? That's the door. The metrics are fair game from there. I might be steering the conversation toward something I've already seen in the data, but the manager doesn't need to know that. What I'm trying to find out is whether they're seeing what I'm seeing.

If they are, we work through it together. If they aren't, that tells me something about the manager. It also tells me something about myself: either I haven't made the expectation clear enough, or I haven't done enough to show them how.

Here's the part I want to be clear about: I've never hit a situation where following this chain didn't get me to a root cause. The methodology is reliable for the diagnostic. What happens after is not. You can identify exactly what's broken and still watch the team keep doing the thing. People are cats. They're going to do what they're going to do. The best you can do is create the conditions where doing the right thing is easier than doing the wrong thing, and have a different kind of conversation when that doesn't work.

The metrics don't solve the problem. They just make it impossible to pretend you don't know what it is.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

What Predictability Actually Measures (and Why Velocity Is the Wrong Question)

Raleigh Schickel — Sun, 21 Jun 2026 14:52:21 +0000

Why I invented a metric I'd never heard of, and why the industry eventually caught up.

I started using the term "predictability" in 2019, and I was pretty sure I was making it up.

The situation that produced it was not subtle. I had just taken over an engineering organization where several members of the remote team had been running mouse jigglers to fake activity while collecting paychecks. Not occasionally. For weeks. We turned the team over, brought in new people, and I came in as the person running Engineering for the first time. My CTO wanted to tear the spyware off everyone's laptops immediately, which was the right instinct: you can't build a culture of trust while surveilling the people you're supposed to trust. But the founder-CEO was not exactly in a trusting mood, and reasonably so. He needed to be convinced, on a regular basis, that the new team was performing.

So I started collecting sprint metrics. Standard stuff: velocity, cycle time, points added and removed. Nothing revelatory. What I was actually trying to do was build a picture I could show to someone who was skeptical, something that said "here is evidence that the team is working and working well." And as I kept collecting and analyzing, I started adding to it. Percentage of plan complete (of the tasks the team committed to at the start of a sprint, how many actually got done). I added that one because the other metrics looked fine on paper and we were still late on things. I needed to answer why.

At some point I added a predictability metric and started tracking it without knowing quite what I was looking at. Over time it became clear that the health of that number was a leading indicator into the health of everything else. I've been running it across organizations ever since, and it keeps proving itself out.

So what is it?

At the sprint level, it's a ratio. The team commits to a certain number of story points at the start of a sprint, based on historical velocity, based on what they think they can get done. At the end of the sprint, you look at what they completed. Completed points divided by committed points, expressed as a percentage. That's it. I've always asked my teams to land between 85% and 115%. A little under usually means the work was close: someone was out sick, a ticket carried over but was nearly done. A little over means the team finished what they planned and pulled in more work, which is healthy behavior. I've validated that range across seven organizations now. It holds.

Outside that range is where it gets interesting. Significantly under means the team's read on their own capacity was wrong in a way that's going to hurt you when you try to set a delivery date. Significantly over means they radically underestimated, which sounds like a good problem until you realize the rest of the company may have planned around the original date and now everyone has to scramble.

One thing I want to be precise about: predictability measures capacity, not content. It tracks how much the team completed versus how much they committed to, deliberately ignoring what the work actually was. That's intentional. Teams rarely control their own prioritization. A PM can swap work mid-sprint. A P0 can land and demand immediate attention. A deal-critical task can materialize from nowhere. All of those are valid reasons to change what's in a sprint. The question predictability answers is narrower: given whatever was asked of this team, did they accurately forecast how much they could handle? If yes, that's a healthy team, even if the playing field kept shifting under them.

The sprint-level ratio is also a self-correcting tool when a team is using it well. A healthy team comes out of a bad sprint asking "why didn't we hit it and what do we do differently next time?" They don't need a manager to tell them something is wrong. The metric gives them the signal and the retrospective is where they diagnose it. When a team is consistently off, not a bad sprint here and there but off across multiple sprints, that's when coaching needs to start. Something structural is happening that the team isn't fixing on its own.

Why the business actually cares

I've spent a lot of time with engineering teams that resist the idea of delivery dates. The Agile community helped build that resistance, and story points exist partly because reasoning about time is genuinely hard. I can't accurately predict how long it will take me to make dinner on any given night, and I've been cooking for decades. Software is harder. The instinct to move away from date commitments is understandable.

But the rest of the company runs on dates. Marketing needs to know when to have launch materials ready. Support needs to understand the feature before it reaches customers. Finance builds revenue models around when things go live. When Engineering can't give a credible answer to "when will this be done," it doesn't make the rest of the company stop planning. It just means they plan around guesses instead of data. That's worse.

Predictability gives you a way to set dates that are grounded in something real. Say you have a quarter-long project. You groom it, break it down, end up with a pile of threes and fives. You have historical velocity, so you can divide total points by average velocity to get a sprint count. But if you know your team runs at 80% predictability consistently, you know you need to adjust that estimate. You bake in the buffer. You set a date that's realistic and as aggressive as possible given what you actually know about how this team operates.

And then you run the thing, and you watch it.

The part nobody talks about

At Stoplight, this is where the early warning system mattered more than the estimate. We had long-running projects, some running a full quarter. Before we had any of this in place, Engineering couldn't be trusted to deliver on time, and the rest of the organization had internalized that. After we implemented these systems, we started hitting dates more consistently. But not always. You do the research, you groom the work carefully, and you still find dragons. Something you didn't see coming adds scope. Someone gets sick at the wrong moment. A dependency slips.

What changed was how we handled it. When we could see early that we were going to need another sprint and a half, we said so. Here's what happened, here's why, here's the revised date. That transparency never eroded the trust we were building. If anything, it built more of it. The rest of the company could adjust their own work, reschedule launches, reset expectations with customers. They were no longer surprised.

The sin was never being late. The sin was not communicating. Letting people plan around a date you knew wasn't going to hold without telling them. That's what damages trust. Lateness is a fact of software development. Silence is a choice.

Why not DORA?

DORA metrics have been popular for the better part of a decade, and I understand the appeal: they came out of serious research and the underlying intent was right. But the four metrics they landed on measure your deployment pipeline, not your Engineering team. Deployment frequency tracks how often you push to production. Lead time for changes measures how long a commit takes to reach production. Change failure rate counts releases that required a hotfix or rollback. Mean time to recovery tracks how fast you get back up after an outage.

The first two are gameable in ways that should be obvious to anyone who has run a team. One-line changes are still changes. You can increase deployment frequency without ever shipping meaningful value to a customer. Lead time for changes goes down when you skip code review and run fewer tests, which is fine right up until the moment your change failure rate goes up and you've spent your stability gains buying back your speed. DORA's implicit argument is that these metrics are self-correcting against each other, and in theory that's true. The 2024 report defines elite performers as teams that deploy on demand: genuinely continuous deployment, sub-day lead times, 5% failure rate. That's a real bar. The next tier down, high performers, deploy somewhere between daily and weekly and carry a 20% change failure rate. One in five deployments causing a production problem, on a team that might be shipping every day. That qualifies as high performance. Make of that what you will.

Mean time to recovery is mostly a measure of your monitoring. Change failure rate is a quality metric, which matters, but it's defined in terms that made more sense before continuous deployment was the norm. Hotfixes and rollbacks are things you do when you're not rolling forward.

None of those metrics tell you whether your team is operating in a healthy way. They don't tell you whether the team can accurately forecast its own capacity, or whether it's burning out trying to hit a date that was never realistic, or whether a steady drip of unplanned work is quietly consuming the capacity that was supposed to go somewhere else. Predictability doesn't answer all of those questions either, but it's the leading indicator that tells you when to start asking them.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

I Can't Tell If the Model Matters

Raleigh Schickel — Fri, 19 Jun 2026 14:58:51 +0000

What I actually found when I set out to test heterogeneous AI code review.

For the last couple of months, I've been running a two-agent code review workflow in my terminal. Left window: Claude Code doing implementation. Right window: a second Claude Code instance prompted to be adversarial, specifically tasked with finding problems in whatever the left window produced. It worked surprisingly well.

A few weeks ago, someone sent me a Lenny's Podcast episode featuring Dan Shipper. One of the things he talked about was using competing frontier models for coding and reviewing, the idea being that different model lineages have different blind spots, and a reviewer trained differently than the author catches things the author misses. That felt like an important gap in the workflow. I set out to fill it.

I went in expecting to write a post about model diversity as a reliability strategy. That's not what happened.

The plan that didn't survive contact with the code

A search for a Gemini code review GitHub Action surfaced one with a reasonable README. The first test ran it against a deliberately bad changeset. Gemini 2.5 Pro flagged some things, missed an obvious multi-tenancy violation entirely, and tagged everything as medium severity regardless of actual severity. Not because Gemini is bad at code review, as I'd eventually learn, but because the action was feeding it diff hunks through a four-line generic prompt with no repo context. The technical notes from this session describe it well: "a frontier model reviewing through a straw."

The terminal Claude reviewer, running with the same adversarial framing but full repo access, found everything planted in the bad changeset plus a couple of things that weren't planned for.

That gap prompted a hypothesis: was the difference in findings about model lineage, or about context and prompt quality? The Gemini action had neither. The terminal reviewer had both. The lineage variable was confounding from the start.

To isolate it, the Claude GitHub Action went in next. Default configuration, Sonnet 4.6: similar shortcomings to the Gemini action. Bumped to Opus 4.8 with full repo checkout and an adversarial prompt encoding the codebase's cardinal rules: same results as the terminal reviewer. The hypothesis held. Context and prompt quality were doing the work. Model lineage was a secondary variable at best.

Why didn't we just use the official Gemini action?

Good question. One exists. A search earlier in the session never found it. The answer, when Claude was asked about it directly: a poorly-worded query on its part. Major miss. What had been running was a 2-star fork of an abandoned project, last updated fourteen months prior, on a deprecated Node runtime, pinned to a mutable version tag with write access to pull requests. The Marketplace rewards "exists and has a README," not "is good." Read the source of anything you hand a write token and an API key.

So the first-party action went in next. This is where the afternoon got complicated.

The first-party Gemini action with 2.5 Pro returned an empty response with a hidden API error. Flash returned a 400 on every call. Bypassing the MCP server entirely and feeding the PR diff directly to the API via a text prompt failed at the command line with exit 1 and zero diagnostic output. Several more configurations. None produced a review. The failure was diagnosed as a model-layer issue, specifically that 2.5 Pro returns empty responses in this context, and the effort was abandoned.

A new hypothesis entered: Does Gemini have a better understanding of how to implement Gemini tooling and flows than Claude does?

I installed the Gemini CLI locally and handed it the history of what had been tried, the PR, and an explanation of what a working implementation needed to do. Twelve minutes later, no questions asked, it surfaced a notification that it had fixed everything. It also consumed 60% of the free tier quota in the process.

The fix: GEMINI_CLI_TRUST_WORKSPACE: true. A flag that had been set in an earlier version of the integration and dropped during a rewrite. The empty response failures confidently attributed to a model-layer issue were actually a workspace trust regression introduced during that rewrite, and then misdiagnosed. Gemini caught the bug. The session had been abandoned one step short of the solution, on a problem an agent caused, with a diagnosis another agent got wrong.

What the repo-aware Gemini actually found

Once it had full repo context and an adversarial prompt encoding the codebase's cardinal rules, the Gemini CLI running gemini-2.5-pro reviewed the same bad fixture the diff-only action had seen earlier. Four critical findings, including the cross-tenant data leak the diff-only version had completely missed, plus a no-tests policy violation, SQL injection, and a hardcoded secret logged in plaintext. Real differentiated severities. Same model family, completely different harness, completely different results.

That's the thesis, confirmed twice. The diff-only action missed the multi-tenancy violation because nothing told it that tenant-scoping is the repo's cardinal rule and it couldn't go look. The repo-aware CLI caught it immediately. The only variable was context.

What I'm running now, and what I still don't know

The current setup: Claude Code action as the primary automatic reviewer, full repo checkout, CLAUDE.md loaded, adversarial prompt. Gemini CLI as a secondary on-demand reviewer, same adversarial framing, same repo access. I'll run both for a few more weeks before drawing any conclusions about whether lineage diversity adds signal or just noise.

A few things this session settled:

First-party tooling over third-party, every time. The abandoned fork was the only Gemini integration that reliably posted reviews during this entire session. It was also the worst option by every other measure. That shouldn't be the tradeoff, and it won't be.

Context and prompt quality dominate model choice. A reviewer without repo context is reviewing through a straw regardless of which model sits behind it. The harness is the variable that matters most.

The models have different working styles, and that turned out to matter more than expected. Claude asked before acting. The Gemini CLI fixed things autonomously and surfaced a notification when done. Neither approach is wrong. But knowing which mode you're working with changes how you supervise it.

The thing that caught bugs most reliably across this entire session wasn't any reviewer. It was running the code. Every misdiagnosed failure, every hallucinated fix, every confident wrong answer got caught when something actually executed and returned an error. Agents reviewed. Agents triaged. Agents misdiagnosed. Execution caught it. The human shepherd directing all of this wasn't in the error chain. The next agent run was.

The lineage-diversity hypothesis is still open. What this session established is that harness quality dominates model capability as a variable, and that a same-lineage reviewer with adversarial framing and full context is already heterogeneous in the way that actually matters. Whether adding a genuinely different lineage on top of that adds anything is a question for more data.

I'll report back.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

Okay, But I'm Still Using It

Raleigh Schickel — Thu, 18 Jun 2026 14:19:27 +0000

Where I'm actually deploying AI coding tools at my next organization, and why.

If you read my last post, you know I'm not a believer in the uncritical sense. Context rot is real. The confidence-to-accuracy gap is real. The autonomous decision-making that nobody asked for is real. I've got the debugging hours to prove it.

So why am I walking into my next organization with a plan to deploy Claude Code on day one?

Because "not ready to replace your engineering team" and "genuinely useful in specific, bounded ways" are not mutually exclusive. The mistake I see people making is treating this as binary: either AI is going to 10x everything or it's overhyped garbage. That's not what I found. What I found is a tool with a real ceiling that happens to be above the floor for a lot of valuable work. The question worth asking isn't "is it good?" It's "what is it actually good at, and how do you keep it from screwing up the rest?"

Here's where I've landed.

Getting oriented in a new codebase

Every time I join a new organization, I spend the first few weeks trying to get oriented in a codebase I've never seen. Some of it is reading. Some of it is asking questions. Historically, that means interrupting a new teammate I barely know, who has their own work to do and may or may not actually remember how that part of the codebase works.

Claude Code changes that equation. The ability to ask questions in plain language: "where does this feature actually live?", "what's touching this module?", "why does this exist?" Getting back a reasonably accurate answer on demand is genuinely valuable. Not because it's always right, but because it's always available and it gives you a starting point that you can verify.

I want to be clear: this is less valuable for engineers who've been living in a codebase for two years. They already have that map in their heads. But for someone walking in fresh, having an agent available on demand to answer questions is a real accelerant, and it removes a recurring tax on the people who would otherwise be answering them.

It also changes the complexity of what I can hand a new hire on day one. Before, I'd rely on simple, low-stakes tasks to get someone oriented. Now I can hand a new employee something meaningfully complex, tell them to ask Claude where to start, and let them engage with the real work earlier. That's good for the new hire and good for the team.

Code review, but ask it to be critical

Claude is a legitimately effective code reviewer. With one caveat that I think gets chronically glossed over: you have to explicitly ask it to be critical.

Left to its own devices, it defaults to cheerleading. It wants to approve things. Ask it for "a code review" and you'll get something along the lines of "This is awesome, ship it!" but not much in the way of critical feedback. Ask it to be highly critical and something different happens. It starts finding the things you actually want: violations of established patterns, scaling concerns, places where the logic doesn't hold.

I've been experimenting with running two separate Claude instances, one as the coding agent and one as the review agent. Watching them interact is genuinely entertaining. The review agent has no loyalty to the coding agent's decisions and will say so plainly: this isn't going to scale, this violates the practices we established, this was a poor decision. It's more candid than most human code reviews, because there's no social friction involved.

The failure mode to watch for: the review agent can inherit the same context rot as the coding agent. I had a situation where I'd established linting rules early in the process. The coding agent stopped running the linter before creating PRs. The review agent didn't catch it either. I found out when it hit the CI pipeline and failed. I had to go scold both of them.

Use it as a first pass, not a final gate. Have engineers run Claude's review before sending code to a human reviewer. It'll catch a meaningful percentage of issues. It does not replace the human reviewer, and you shouldn't position it that way.

Test generation as a diagnostic tool

This one doesn't get enough attention.

Engineers chronically underinvest in testing, not because they don't value it, but because it's tedious and it competes with shipping. The result is codebases with inadequate coverage that make everything else harder.

Claude is good at test generation, but the most useful thing about it isn't the tests it produces. It's what happens when it struggles. Hard-to-test code is almost always a symptom: a god class, too many responsibilities in one place, hidden dependencies. When you ask Claude to write tests for something and it can't do it cleanly, that's information. The struggle surfaces design problems that were already there.

I'd have engineers use it as a two-part workflow. Ask Claude to write tests for the thing you just built. If it can't do it cleanly, treat that as a signal that the code needs to be reconsidered before it ships. You're not just generating tests. You're using the test generation process as a lightweight design review.

Internal tooling

Every engineering organization has a graveyard of internal tools that nobody built because they weren't customer-facing, weren't on the roadmap, and never made it past "someone should probably do that someday." Scripts to automate reporting. Dashboards for operational visibility. Small utilities that would save hours every week if they existed.

The metrics dashboard I described in my last post was exactly this. Thirty minutes of manual weekly work, automated in eight hours. Not customer-facing, and not something I could have justified pulling an engineer away from a project with actual stakeholders waiting on it. But real value, delivered.

This is where the current limitations matter least. Internal tooling can tolerate roughness. The users are technical. The stakes of a runtime error are lower. If the agent makes a weird autonomous decision about a script that three people use internally, you catch it, you fix it, you move on.

Use internal tooling as a proving ground. Let engineers develop their instincts for how to work with these tools: where to trust it, where to verify, how to prompt it effectively. Do it on things where the cost of being wrong is low before you do it on things where it isn't.

Production code, for the right class of problems

This is where the conversation usually goes sideways in both directions. People either say "never let AI touch production" or "just ship whatever it generates." Both are wrong.

There's a class of production work where the risk profile is manageable, and the common thread is this: the problem is well-defined, the contract is explicit, and failure is predictable and debuggable.

Third-party integrations are the clearest example. When you're integrating with an external service, you're coding to a specification someone else wrote. The inputs and outputs are defined. The edge cases are documented. There aren't a lot of hidden performance traps or security landmines if you stay within the bounds of the API contract. This is work that can consume a senior engineer's time without really requiring senior judgment. That's exactly what you want to offload.

Dependency upgrades are another one. Rote work that someone has to do but nobody wants to do. Well-understood expected behavior. The test suite tells you whether it worked. I'd let Claude handle those, with guardrails. My rule of thumb: if updating a dependency requires touching more than three files beyond the version bump itself, stop. Something more complex is happening and it needs a human making the call.

The opportunity cost argument matters here. Every hour a senior engineer spends on an integration spec or a dependency upgrade is an hour they're not spending on genuinely novel problems: the architecture decisions, the hard bugs, the performance investigations that actually require experience and judgment. Offloading the well-defined work isn't laziness. It's resource allocation.

The non-negotiables are proper monitoring, scrutiny before anything hits production, and clear criteria for when to escalate. AI-generated production code isn't inherently reckless. Unreviewed, unmonitored AI-generated production code is.

What I'm actually expecting

I want to be honest about the fact that I'm walking into this with a plan, not a guarantee. Everything I've described is based on experiments I ran on my own projects, not on deploying this across an organization at scale. Some of it will work the way I expect. Some of it won't. I'll probably find new failure modes I haven't encountered yet.

What I'm not going to do is either throw the tools at my team without a framework for using them, or keep them locked up because they're imperfect. The engineers I'm going to be working with are going to be using these tools with or without my guidance. My job is to give them a smarter way to engage with them than "just try stuff and see what happens."

If my thinking on this changes once I've actually run it at scale, I'll write about that too.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

I Can't Tell If You're Selling Me Something

Raleigh Schickel — Wed, 17 Jun 2026 15:49:30 +0000

What I actually found when I stopped reading about AI and started running my own experiments.

Everywhere you turn right now, someone is telling you how AI is going to transform your workflow, your team, your organization, your life. The content is relentless, and it is almost universally positive. Glowing. Evangelical, even.

I'm not here to tell you that's all a lie. I genuinely don't know. That's kind of the problem.

We live in a media environment where the line between advertising and information has been blurring for years, and AI is accelerating that blur in ways I don't think we've fully reckoned with. When I read a breathless LinkedIn post about how some engineering leader 10x'd their team's output with AI coding agents, I find myself asking: is this a real person sharing a real experience? Is it a paid placement? Is it content generated by the very tools being promoted?

I have no way to tell. Neither do you.

And it's getting worse, not better. The most qualified people to evaluate these tools honestly, the ones with enough experience to have real judgment, are also the busiest. They don't have time to write takes. Which leaves a lot of space for everyone else: the shiny-object adopters who are genuinely excited, the vendors with obvious incentives, and an increasingly murky middle ground of content that looks like an opinion but might be something else entirely. The financial relationship between a writer and the tools they're praising is almost never disclosed. And now the tools themselves can generate content praising the tools. Think about that for a second.

I'm not making accusations. I'm describing a problem that I think we have a collective responsibility to sit with rather than just nodding along. The appropriate response to an information environment you can't fully trust isn't paralysis. It's going and finding out for yourself.

So that's what I did.

Why I finally got off the fence

I've been watching this space with skepticism for a while. Being a cynic about new technology doesn't mean I think it's worthless. It means I've been around long enough to know that being an early adopter has bitten me in the ass more times than I care to count. I've watched programming languages rise to prominence and quietly fade. I've seen frameworks become religious movements and then become legacy problems. I hold new things loosely until I have my own data.

What eventually pushed me off the fence was something specific: Anthropic releasing Claude Code as a proper agentic coding environment. Not a chat window you paste code into. Not a side panel in your editor with access to a single file. Something with access to the entire codebase, something that could chain connections across files on its own. That felt meaningfully different from what had come before.

I was also surrounded by colleagues who were already deep in it, which gave me both the social pressure and the budget to stop waiting. I sat down, installed the tools, and started running experiments. Not to validate the hype. To find out where it actually held up and where it fell apart.

Here's what I found.

The thing I needed built

I've kept a metrics tracking spreadsheet for years, a way to stay close to the health of my engineering organization. Sprint data, defect trends, the signals that tell you whether a team is healthy or just busy. It was held together with Python scripts and manual entry, and it took me about 30 minutes a week to update. I liked that 30 minutes, honestly. The manual process kept me close to the data in a way that was valuable. But I was in a time crunch, and I decided to find out whether I could get that time back.

I sat down with Claude Code and started what people are calling "vibe coding," essentially describing what I wanted in plain language and letting the agent build it.

Within eight hours, I had something usable. I want to sit with that for a second, because it is genuinely remarkable. Eight hours to a working application that automated a workflow I'd been doing manually for years. If that's all I had to report, I'd be writing a very different article.

But I kept going. I iterated. I added features. And that's when things got interesting.

Context rot is real. The further we got from the beginning of the session, the more the agent seemed to forget guidance I'd already given it. Preferences I'd established. Constraints I'd set. It would drift back toward doing whatever it wanted to do, as if the earlier instructions had simply evaporated. For a personal project, annoying. For an enterprise codebase with established standards, that's a serious problem.

It deleted features without telling me. As we were iterating, the agent would occasionally decide that something we'd already built wasn't necessary anymore and quietly remove it. I only caught it because I was doing my own QA pass through the session output. When I asked about it, there was no good explanation. It just made a call. On its own. Without asking.

It made bizarre judgment calls. There were moments where it felt like the agent was trying to impress me, pushing ahead of the requirements, making decisions that weren't asked for, showing off what it could do. I've landed on an analogy for this: it behaves like a young child trying to get a parent's approval. Eager to demonstrate capability. Acting out of bounds not out of malice but out of a kind of nascent, unformed ambition. That's not an insult. It's an observation about where the technology is in its development. Young. Early. Still figuring out the edges of what it should and shouldn't do.

For a personal project, I can absorb that. For production software at any meaningful scale, I can't.

The thing I built for fun

My wife and I have a recurring problem: we can never decide what to watch. So I built an app. A simple Android application that pulls TV show data from an API, stores a list in a database, uses Google Auth for login, and randomly selects what we're watching tonight.

Claude built it in about an hour.

It took another eight hours to get it to actually work.

That distinction matters more than it might seem. "Built" and "works" turned out to be very different things. After that first hour, Claude was confident. "We're done. Here's how to get it running." I'd follow the steps. I'd hit an error. I'd report back. And to Claude's credit, it never got defensive. Every time I said "this doesn't actually run," the response was some version of "you're right, there's a problem, let me fix it." No gaslighting me about whether the error was real.

But we did this over and over. Configuration problems. Runtime errors that didn't surface until I was actually trying to install the thing on a phone. It felt less like collaboration and more like QA-ing a junior developer who is genuinely trying but keeps missing things they should have caught.

The app works. We use it. But the gap between "Claude thinks it's done" and "it is actually done" was significant, and I don't think that gap should be hand-waved away.

The litmus test I've been running for years

Before any of this, before the current hype cycle, I've had my own informal benchmark for evaluating AI coding capability. Every three to six months, I grab a complex chunk of code from whatever I'm working on. Hundreds of lines. A god method. The kind of function that has grown over years into something nobody wants to touch, but that you can look at and see, clearly, how it could be broken apart.

I hand it to the AI and say: refactor this.

No specific instructions. No hints. Just: here's a mess, make it better.

Refactoring is actually a well-understood problem. There are established patterns, canonical books, clear inputs and outputs. It's mechanical enough that you'd think a sufficiently capable AI should be able to at least produce something that compiles. For years, what I got back was a complete rewrite, and the more complex the code I gave it, the less likely the output was to actually run. Which is a striking failure mode for something that is supposedly fluent in code.

Within the last six or seven months, something changed. Claude stopped trying to rewrite everything and started making selective edits. Targeted changes. It began to look less like a student who didn't read the assignment and more like someone who actually understood what refactoring means.

It's not there yet. I still haven't gotten back something I'd call production-ready from this test. But the direction of travel is real, and that matters.

Where this leaves me

I want to be honest about something before I close.

I've described real limitations. I've been as critical as I know how to be. But I also built two applications I use, and I'm planning to bring these tools into my next organization. That's not a fully negative take, and I'm aware of the irony: I started this piece by asking whether the positive AI content you're reading is trustworthy, and I'm ending it with some positive observations of my own.

I don't have a clean resolution to offer. What I have is this: I ran my own experiments instead of taking someone else's word for it, and I'm telling you exactly what I found, including the parts that didn't work. Whether that's enough to trust is a judgment call you'll have to make yourself.

The discourse isn't going to get more honest on its own. The incentives all point the other way. So run your own experiments. Stay skeptical of the output, including this one.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

Priority Inversion: The Bug Triage Failure Nobody Talks About

Raleigh Schickel — Tue, 16 Jun 2026 14:28:14 +0000

Why your team is probably working on the wrong bugs, and why they don't know it.

One of the first things I do when I walk into a new engineering organization is look at the bug queue. Not because I'm a bug-metrics person (I'm not) but because bug queues tell you things about organizational health that sprint data doesn't. The queue is where pressure from the outside world meets a team's internal capacity to respond, and how that collision gets managed says a lot.

There's a reason I care about this beyond Engineering hygiene. As the top-level Engineering leader in most organizations, I own quality. Where there's a Quality team, it typically answers to me. That creates a real tension: I'm trying to care for the people building the product while simultaneously reasoning about what the product's current health is costing the business. How much friction is the defect backlog creating? What level of investment do we need to make to get ahead of it? Those questions require data, and data requires visibility. Without visibility into what's actually broken, I'm flying blind on both sides of that equation.

In a recent role, I pulled up the Jira board and immediately noticed something odd. Every single bug in the system had been created by an engineer. Not a single ticket sourced from Customer Support. Not a product manager, not a customer, not a support request. Just engineers, all the way down.

That's unusual enough to raise a question: is this all the bugs? Or are these the bugs that engineers happened to decide were worth writing down?

The answer, as I started poking around, was the second one.

The bugs weren't in Jira. They were in Slack.

There was a channel where Engineering and Customer Support were collaborating on customer-reported issues. In real time, constantly. When a customer hit something, a Customer Support rep would drop it in Slack. An engineer who happened to be paying attention might pick it up. If they chose to work it, they might create a Jira ticket. If they didn't, the issue would just sort of evaporate. Sit there unanswered or get resolved via a one-off Slack message, no record anywhere.

The result was a defect system that was, at best, a partial view of reality. The real defect queue lived in a Slack channel that nobody had formal ownership of, nobody was triaging, and nobody was tracking. The Jira board was a collection of bugs that engineers had self-selected for visibility. Everything else was in the shadows.

This is the part that usually gets skipped when people talk about bug triage failures. Everyone talks about priority inversion, P2s getting worked while P0s sit. The more fundamental problem is work that isn't captured at all. You can't invert priorities you don't know exist. And you can't be accountable for doing the right things when everything is hidden.

Recency bias is not a triage system

Once I understood that the real queue was in Slack, the pattern started making sense. What passed for prioritization in this org was almost entirely driven by recency. The most recent thing got worked on. Anything that was a day or two old was out of sight, and without active visibility, out of mind. Engineers were constantly firefighting whatever fire had started most recently, not the biggest or most important one.

Customer Support had noticed this from their side. Their experience was blunt: if something didn't get picked up immediately, it never got picked up. An issue that didn't get a response in 24 hours was effectively dead. They'd learned to re-surface things just to get attention, which added noise to the channel and trained engineers to respond to volume rather than urgency.

For their part, engineers weren't being malicious or even particularly negligent. They were underwater. Full-time fire mode. When you're just trying to keep up with the incoming rate of issues, you work what's in front of you. The recency bias wasn't a choice, it was the default behavior of a system with no prioritization structure and a constant stream of new inputs.

The inversion was real, but it was a symptom. The root cause was invisible work and the absence of any mechanism to assign priority before picking things up.

The fix nobody asked for: don't blow up their workflow

My first instinct in situations like this is usually wrong. Or rather: the instinct is right, but the implementation is off. The right answer involves a unified system, priority labels that mean something, and a real triage process, but all of that is correct in the abstract and completely useless if nobody uses it because it's too much overhead on top of a job that's already overwhelming. If your solution requires people to fundamentally change how they work every day, you're going to get justified resistance.

I don't have a playbook for these situations. I have a toolbox. And the first thing I reach for is: where do people actually work, and how do I fit the solution around that instead of expecting them to fit themselves around the solution?

Customer Support wasn't going to use Jira. They had their own tools, their own workflow, and asking them to learn another system to report bugs was adding friction to people who were already stressed. Engineering wasn't going to abandon their board. The Jira setup was messy, but it was theirs.

The solution that actually worked started with capturing the work without changing how it got reported. I built a Jira form linked directly in Slack. Customer Support fills out a form in the channel they're already in, a ticket gets created automatically, and a Zapier integration posts the ticket back into the thread. The Slack conversation continues, but now it's anchored to a Jira ticket, and every comment in that thread gets synced back to the ticket. No new tool, no context switch, no asking underwater people to adopt overhead they don't have time for.

The second piece was giving everyone a single view into Customer Support-sourced work without disturbing Engineering's board. I built a separate dashboard that surfaced everything Customer Support had reported, regardless of where it sat in Jira, sorted top to bottom by priority. Customer Support had a single place to check status. I had a single place to triage and set expectations. Engineering could see what was actually out there and pick work that mattered rather than work that was recent.

None of it worked until we dealt with the vocabulary problem. The severity labels in Jira existed, but nobody was using a consistent definition. "P0" meant different things to different people. Before any of the tooling mattered, I needed Customer Support and Engineering aligned on what each level meant and what the associated expectation was. How fast should a P0 be acknowledged? How fast should it be resolved? That shared vocabulary is the foundation everything else sits on, and it's almost always the thing that gets skipped.

Data as a forcing function for accountability

Once the workflow was running and there was a few weeks of actual data, something important became possible that hadn't been before: accountability.

This is what I was actually after. Getting everything into the system and wrapping data around it wasn't an end in itself. It was the prerequisite for being able to look at the team, and have the team look at itself, and ask whether we were doing the right things. You cannot hold anyone accountable, including yourself, for something that's invisible.

I didn't have to tell anyone they were doing it wrong. I just showed them the numbers. We're creating this many P0s per week. We're resolving this many. The backlog is growing. Here's a P0 that's been open for eleven days.

That last one gets attention. An eleven-day-old P0 creates two useful conversations. The first is with Customer Support: was this actually a P0, or did we mislabel it? Maybe it was really a P2 and the label is inflating the severity. The second is with Engineering: if this really is a P0, why hasn't it been touched? That conversation is the triage mechanism working the way it's supposed to, as an organizational sanity check, not as a performance review.

The goal in these conversations was never to assign blame. It was to get to a shared understanding of what we owed our customers and whether we were delivering on it.

Managing up and across

There's a third thing visibility buys you that doesn't get talked about enough: the ability to protect your team from the rest of the organization.

At this company, there was a weekly operations meeting. It was a small company, so the room included the CEO, the CFO, a couple of people from Customer Support, and a couple from Engineering and Product. We talked about running the business. And without fail, quality came up. Every week. People would drum on specific things that weren't getting fixed, express frustration about the pace of resolution, and I had essentially nothing to say back. The team was working, but I had no information. I couldn't tell anyone what was getting fixed, how much labor we were spending on defects, or what it would actually cost to prioritize the things they were complaining about. I was walking into that meeting empty-handed every week.

Once the system was in place, that changed. I now had data showing how much of the team's capacity was going toward defect repair. I had a ranked list of what was being worked and what wasn't. And when someone in that meeting pointed at a specific bug and asked why it wasn't fixed yet, I could answer the question: it didn't rise to the level of the things we were already working on. Here's what's above it. What would you like us to deprioritize to address it?

That reframe matters. It shifts the conversation from "Engineering isn't fixing things" to "Here are the tradeoffs, and you're now part of making them." It builds trust with the executive team because they can see the work and the reasoning. And it protects the Engineering team from being blamed for prioritization decisions that were never really theirs to begin with. Without a structured system, those decisions were just defaulting to recency, and nobody had agreed to that.

The retention argument

There's a version of this story where I make the customer relationship case: unresolved high-severity bugs damage trust, damaged trust puts revenue at risk, and so on. That's all true. But the argument I find myself making more often is about your people.

Engineers who spend their careers in full-time fire mode turn over. Not always immediately, and sometimes not loudly, but they do. The engineers who have options will eventually go somewhere where the work feels less futile. A mark of a healthy Engineering organization is limited turnover, and high turnover is frequently a signal that the environment is too chaotic to feel sustainable.

The path out of fire mode runs through prioritization. And prioritization requires visibility. The accountability structures I put in place weren't only about product health or customer satisfaction, they were about creating an environment where the team could do their best work instead of constantly reacting to whatever came in last.

Priority inversion is the thing people name. But the thing underneath it is almost always an information problem. The team isn't making bad choices. They're making default choices in the absence of structure, and default choices in an interrupt-driven environment trend toward whatever was most recently on fire. Make the work visible, agree on what it means, and the data will make the difficult conversations easier.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

How I Assess Engineering Health in My First Week at a New Org

Raleigh Schickel — Mon, 15 Jun 2026 19:17:29 +0000

The diagnostic framework I run every time I join a new engineering organization, and why the data is only half the picture.

The first week at a new company is a lot. You're meeting strangers, trying to read rooms you've never been in, figuring out who the actual decision-makers are and how decisions get made. For me, as an introvert, there's also the sheer exhaustion of performing extroversion for eight hours a day while trying to simultaneously form a coherent picture of a place I've never seen before.

And everyone is watching. The people above you want to know if they made the right hire. The people who report to you are wondering whether their jobs are safe, whether you're going to blow up the way they've been doing things, whether you're someone who actually gives a damn or just another executive who issues mandates from a conference room. You try not to let that pressure paralyze you. But you feel it.

The thing that helps me most is having a framework. Not because frameworks give you answers, but because a good framework gives you the right questions to ask. The two main questions I'm trying to answer in week one each open into several more questions, and the answers to those won't come from the data. They'll come from the team. The data just tells me where to look and what to ask about.

Here's how I do it.

The Two Questions

Everything I'm trying to learn in the first week boils down to two things: what is the state of quality, and how healthy are the individual Product Engineering teams?

Those sound related, and they are, but they're measuring different things. Quality is about the product: How much defect debt has accumulated? How are fixes being prioritized? Are bugs being worked on in any coherent order? Team health is about execution: Can these teams actually deliver what they say they're going to deliver, and if not, what's getting in their way?

I use both because either one alone tells an incomplete story. A team executing beautifully might be doing so while the product quietly burns down around them. A product that's relatively stable might be maintained by a team that's one bad quarter away from collapse. You need both pictures.

First, the Tooling

Before I run a single calculation, I spend time in whatever ticketing system the org is using and I just look around. How many projects are there relative to the number of teams? How are bugs being categorized? Are there priority labels, and does anyone seem to be using them consistently?

This sounds mundane, but it tells you something before any numbers do. An org that has carefully structured its tooling to reflect how it actually works is a different kind of organization than one that crammed everything into a single project and built seventeen custom dashboards to compensate. You can see the history of decisions people made and abandoned just by looking at the archaeology of the project structure.

One thing I've formed a strong opinion on over the years: severity and priority should not both be in play for bug classification. I've seen orgs use both, and what you end up with is a combinatorics problem. High severity, low priority. High priority, low severity. It becomes harder to reason about, not easier. I collapse those into a single priority axis (urgent/P0, high/P1, medium/P2, low/P3) and work from there.

Assessing Quality

For quality, I'm looking at week-over-week trends. How many bugs are open total? How many were opened this week? How many were resolved? What's the net delta? Then I break all of that down by priority. How many P0s are open? P1s? What's changing week to week?

The parallel question, and this one matters a lot, is which priorities are actually getting worked. It's a remarkably common pattern that organizations fix the wrong things. Lower-priority issues get closed out while urgent and high-priority bugs sit open for weeks or months. Nobody made a decision to do that. It just happens, because individuals are making judgment calls without any system enforcing priority order. Nobody is managing the queue.

When I see that pattern, it tells me several things: there's no triage process, individual engineers are deciding on their own what to pick up, and nobody in leadership has visibility on what's actually being worked on. But those are all symptoms of the same root problem: Nobody is accountable for ensuring the most important things get done. That role exists in every healthy Product Engineering org, and when it's vacant, you get a queue that sorts itself by whatever is easiest or most interesting, not by what actually matters. And it doesn't stay contained to Product Engineering. Customer Support is responsible for managing unhappy customers, and when those customers' complaints aren't being addressed, Support can't do their job. Sales runs into the same wall. Eventually it trickles up to executive leadership, and what started as a prioritization gap becomes a trust problem. The Product Engineering org loses credibility with the rest of the company. Everything else you see in the data flows from that gap.

I also pay attention to where bugs are coming from. If almost everything in the ticketing system was filed by engineers, that's strange. You'd expect support teams, QA, or customers to be the primary sources. When engineers are filing most of the bugs, it often means there's a shadow process somewhere. Bugs are being reported through some other channel: a spreadsheet someone built, a shared inbox, the customer support tool that Support and Sales are living in, or an entirely separate ticketing system that another department decided to stand up on their own. Only occasionally does any of it make it into the official system, and only when an engineer happens to create a ticket. That shadow system is worth finding, because it's where the real backlog lives.

Assessing Team Health

For individual Product Engineering teams, I'm looking sprint over sprint rather than week over week. The core metric I care about most is predictability: how much work did the team commit to at the start of the sprint, and how much did they actually complete?

A healthy team delivers somewhere between 85% and 115% of what they committed to. Too far under means they don't have a grip on their velocity, or work is being added mid-sprint and eating their capacity. Too far over usually means they're padding estimates, or the commitment target is so conservative that any reasonable sprint will sail past it. Both directions are worth investigating.

Predictability is also genuinely hard to game, which I appreciate. When deployment frequency became a headline DORA metric, some teams just started deploying more often without the underlying reliability improvements the metric was supposed to represent. Predictability is a relationship between two numbers (commitment and completion) and manipulating that relationship requires coordination across the whole team in ways that tend to fall apart. It's not a perfect metric, but it's more honest than most.

From there, I want to know how much of the completed work was planned at the start of the sprint versus added mid-sprint. Unplanned work is not inherently bad. Sometimes a critical bug lands and you deal with it. But the ratio matters. A team where 40% of completed work was unplanned has a disruption problem, and that disruption is probably coming from somewhere specific. Maybe it's customer escalations. Maybe it's an executive who keeps dropping things on the team. Maybe it's a quality problem generating constant reactive work. The metric points you toward the right questions.

I also track points added to and removed from sprints after they begin. This one reveals something about organizational culture that tends to be uncomfortable to surface: most places love to add work mid-sprint and hate to remove it. The implicit assumption is that the team just absorbs whatever gets added. But teams have a fixed capacity and prioritization is a hard choice. Work added without anything being removed rolls over to the next sprint, affects how much new work can be taken on, and compounds from there. Watching that pattern over several sprints tells you whether the organization actually respects the team's capacity or just treats sprint planning as a rough suggestion.

I look at velocity as a rolling three-sprint average rather than a single-sprint number. Any given sprint's velocity is noisy. Someone's sick, there's a holiday, scope changed late. The rolling average smooths that out and gives you a more honest baseline to plan against.

Finally, cycle time. I want to know how long work is spending in each phase of the delivery process: in progress, in review, waiting to deploy. The goal isn't to make all the numbers small. It's to find the bottlenecks. If code review time is twice as long as implementation time, something is creating friction in review. Maybe there are too few reviewers. Maybe the PRs are too large. Maybe one person is doing all the reviews and they're buried. The metric doesn't tell you the cause, but it tells you where to look.

The Organizational View

Once I have the individual team picture, I pull it up to the org level. I plot all the teams' metrics on the same charts, which makes outliers obvious in a way they wouldn't be if you were looking at each team in isolation.

At the org level, the two numbers I talk about publicly are organizational predictability (the average across all teams) and organizational percentage of planned work completed. The second one is worth raising openly because it sends a signal to the whole company: if 40% of what we delivered last sprint was unplanned work, that's a disruption problem that affects everyone, not just Product Engineering. Planned quality improvements get deprioritized. Promised bug fixes slip, degrading customer trust. Predicted delivery dates become unreliable. And when in-progress projects fall behind, other departments absorb the cost too. Marketing may have already invested weeks preparing launch content for a feature that's now delayed, time they could have spent on higher-value work had they known sooner. Left unchecked, it derails the roadmap entirely and can put revenue plans at risk. That's a useful conversation to have in the open.

Individual team performance is a different matter. Cross-team metrics are useful for coaching, but not for broadcasting. Putting comparative performance data in front of the whole company without context creates exactly the wrong dynamic: teams start optimizing for how they look rather than how they're actually doing. The right venue for this is the sprint review, and the right approach is to open it by sharing what the metrics are showing and asking the team what they think. What do you see here? Is there something we should be focusing on improving? Start with questions, not conclusions. That framing turns the data into a conversation starter rather than a verdict.

You can't walk into a room with bad graphs and tell a team they're failing. Or, technically, you can, but you won't be there much longer if you do. My preferred approach is Socratic. I share what I'm seeing and ask what the team thinks is happening. "Your cycle time in review is notably higher than in progress. What's your sense of why that is?" They usually know. They've been living with it. What they didn't have was someone asking the question and treating the answer as worth addressing.

That matters for a reason beyond just being diplomatic about it. If I solve every problem myself, I've made myself indispensable, and that's exactly the wrong outcome. The goal is to teach people to read these signals themselves, to build the habit of asking these questions every sprint, to create a team that can self-correct without me in the room. The sprint review, done right, is where that habit forms.

The Data Gets You to the Door

I've refined this diagnostic approach across a lot of organizations over a lot of years, and at this point I can build the spreadsheet in my sleep. But I want to be clear about what it actually is: a tool. The methodology gives me the right questions to ask. It gives me almost no answers. The answers come from experience, from having seen the same patterns play out enough times that when the data points in a certain direction, I have a pretty good idea of what I'm looking at. But recognizing a pattern and knowing exactly how to address it in this organization, with these specific people, inside this particular culture, are two very different things. The first tool I reach for might not be the right one here. That part still requires doing the work.

None of this touches the technical side of the house directly. Deployment pipelines, production infrastructure, development environments, architectural decisions: those are also part of the role, and eventually they become a significant part of it. But trying to assess all of that in week one is the wrong sequence. The framework tends to surface the technical problems anyway. When cycle time in the deployment column is consistently an outlier, that's usually an infrastructure conversation waiting to happen. When unplanned work keeps spiking, it often traces back to architectural debt generating a steady stream of reactive work. When defect volume is high and resolution is slow, sometimes the answer is process, but sometimes the answer is that the system is just hard to work in. The data points you toward those doors. Then you go look.

The methodology is the easy part. The hard part is figuring out which conversations to have, in what order, with people you've known for less than a week. You have a picture of what's broken, but no relationships yet, no trust yet, and no real sense of who in the room is ready to hear what. Coaching someone toward a better way of working looks completely different from lecturing them about what they're doing wrong, and the line between those two things is mostly about whether the person on the other end of the conversation feels supported versus scrutinized.

If the first thing people feel when you walk in is that they're being watched and judged, you've already failed, regardless of how accurate your metrics are. If you walk in with genuine curiosity, about the work and about the people doing it, and you back that curiosity up with data, you have a shot at actually changing something.

That's the whole thing, really. The rest is just spreadsheets.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.