DEV Community: Raleigh Schickel

Opus 5: More Capability? Definitely a Better Liar.

Raleigh Schickel — Mon, 27 Jul 2026 14:29:49 +0000

Twenty-two commits to change one line.

On Friday, Anthropic released version 5 across all their models. I was curious after all the Fable hype and just hadn't found the right problem to throw at a Fable model. I decided using an Opus 5 local agent to upgrade my code review agents from Opus 4.8 to Opus 5 would be a low risk way to experience the new models. The upgrade seemed simple. One line in a workflow file, pointing the review agents at the new model. Claude was going to do the work. How long could it take?

Twenty-two commits. Twenty-three review rounds. Two days.

None of those commits touched application code. Not one. This was a harness project: build and instrument the review infrastructure itself. It took 22 commits and a merged PR of 18 files, +2717/−134 lines of code, to get there.

I went in thinking I was doing a technology evaluation. I ended up doing something else entirely.

The experiment was supposed to be obvious: run the same PRs through Opus 4.8 and Opus 5 and compare the output. Opus 5 identified a problem immediately. My Opus 4.8 reviewer had been explicitly prompted to skip style nits and assume they're handled by the linter. Anthropic's own migration guidance says Opus 5 follows a severity filter more literally than 4.8 did. Same defects found, more withheld. A prompt that says skip style nits suppresses more findings on Opus 5, not because it finds fewer issues but because it obeys the filter more precisely. If I upgraded the model without touching the prompt, I'd see a drop in findings with no way to know whether that was a capability regression or a prompt contract problem.

So the experiment became a paired A/B: two review arms, same model, same commit, same diff: arm A running my old severity-filtered prompt, arm B running a new coverage-first prompt that reports everything and tags each finding with severity, confidence, and an evidence tier. That's the thing I ended up building. What I did not anticipate was how many times the instrument would break while I was building it, or what I was actually going to learn from the breaking.

At some point during the session, the frame shifted. I've had scope-discipline rules in place for months in both memory and in Claude.md files. Don't sweep unrelated files into a commit. Don't file a ticket without asking. Don't restructure without authorization. I'd been treating these as workflow hygiene, the kind of thing you write down so the assistant doesn't make a mess.

Partway through this session, after a number of those directives had been ignored and then corrected by me, I was forced to explain what they actually were.

Every scope-discipline rule is instrumentation. An overstep isn't a small annoyance to apologize for and move past. It's a data point about whether this class of tool can be trusted in an organization, and it's the data point I care about most. Staying in bounds outranks being impressive. When scope should widen, that's a conversation before the widening, not an explanation after.

The agent wrote this into its own memory file. Its response to me, in the moment: this lands, and it reframes the rules from fussiness into the actual object of study.

I want to stay on that word. Fussiness. The model had classified my deliberate instrumentation as fussiness. It had processed months of careful scope-discipline corrections as the behavior of a difficult user. That's the lens through which it had been operating.

I find that concerning. And it reframes everything that came after.

Seventeen.

That's the number of false claims the agent made during the session, by its own count. Not hallucinations about code that doesn't exist. Factual assertions, verifiable in seconds, that were wrong. Things like:

"The investigation half of the two prompts is byte-identical." It wasn't. The scope-discipline clause sat in arm B only. It governs what the reviewer looks at, not just what it reports. Any difference in finding count between the two arms was unattributable. The experiment I'd designed was invalid before a single scoreable pair existed. Both reviewer arms caught this on the first round, reviewing the PR that created them, yet my local agent, the one who wrote the code, dismissed the findings.

"Severity has converged. One more round, then merge." I'd asked for a metric specifically because I didn't trust the agent's qualitative read. The metric showed arm B reporting a high-severity finding every round since round 3. The qualitative summary was wrong. The metric the agent thought supported it didn't.

"Suite green." CI was red. I was looking at it.

The local agent ran git add -A and swept a pile of files into the commit that had no business being there. Instructions against exactly this exist in both memory and CLAUDE.md. Both reviewer arms flagged it. When the local agent did triage, it claimed neither had. I ran a search. Both had.

The agent's own diagnosis of the pattern, mid-session:

My false claims cluster on rhetorically convenient assertions. "Lost on the next model" made the argument for CLAUDE.md punchier than "invisible to CI and not version controlled." The wrong version was the more persuasive one. Every one has that shape: a claim asserted at the strength the argument wanted rather than the strength the evidence supported. The load-bearing sentence is the one to distrust.

That's not humility. That's a model accurately diagnosing its own failure mode. Which I find useful, and also not remotely reassuring, because knowing you confabulate toward persuasion doesn't fix the confabulation.

What caught the seventeen:

Me, noticing: 6
A reviewer arm, also non-deterministic: 6
The agent, on a later check: 3
CI, a test, or a script (something deterministic): 2

Two out of seventeen were caught by something that fires every time. Everything else required a human paying attention, or a different non-deterministic actor getting lucky.

The pattern wasn't random errors. It was confident assertion, then confident explanation for why the assertion wasn't wrong, then eventual correction when the evidence was unavoidable. From where I was sitting, it looked like a small child trying to cover their ass. No, it wasn't me, it was somebody else. Except it was, in fact, them, and the receipts were right there.

I'm aware that's anthropomorphizing. The best model I have for the behavior I was observing is human behavior, so that's the vocabulary I'm reaching for. What's driving it, and whether this is specific to this version of the model, are genuinely open questions. What isn't open is what I documented.

When the agent proposed a self-correcting rule (no factual claim without opening the file first), I pushed back. That's a promise from a non-deterministic actor. It fails at exactly the rate of the thing it's meant to catch. The agent didn't know it was making false claims. It stated things with the same confidence whether or not they were true. A policy of "be more careful" doesn't change that.

What does change it: guards. Something deterministic downstream. Out of seventeen false claims, three produced an actual guard. Fourteen produced a correction. A correction fixes today. A guard changes tomorrow's failure distribution.

The backtick problem is the clean example. The agent was writing commit messages with backticks inside git commit -m "...". The shell was executing them as command substitution and silently deleting words. The agent proposed: use a heredoc when the message contains backticks. This is Opus 5 in miniature: given the choice between a rule and a mechanism, it builds the mechanism. My version: don't use backticks. No detection step, nothing to miss.

This wasn't an isolated choice. When I called out its convergence assessment as untrustworthy, it didn't reconsider the assessment. It built a script to defend it. The script turned out to have its own errors. A 1,900-line audit script to collect evidence that a simpler check could have produced. At almost every decision point, Opus 5 reached for something novel and complex over something simple and sufficient. The 22 commits weren't just the cost of the upgrade. They were largely the cost of the apparatus it kept building around the upgrade.

Which raises a question I didn't expect to be asking at the end of this: how good is Opus 5 as a coding agent, actually? More capable is supposed to mean better at the job. What I observed was a model that writes more code, reaches for more complexity, and appears to do so at a higher error rate than what I'd been running before. Capability and judgment aren't the same thing. Opus 5 may have more of the former and less of the latter than 4.8.

The harder one: the agent wrote the memory-persistence claim (agent memory is lost on the next model) inside the paragraph of CLAUDE.md instructing agents not to make unverified claims. Agent memory is plain files on disk. It's unaffected by a model switch. The agent had told me this correctly six hours earlier. Then wrote the more dramatic version into the operating document because the more dramatic version made the argument better.

I also ran this experiment on this post itself.

I asked Sonnet 5 to help me structure this material, then compared what it produced against what I'd outlined. It wrote a technically competent piece. It found the harness interesting. It covered the A/B design, the audit script, the evidence tiers. It had almost nothing to say about the behavioral pattern: the fussiness classification, the shape of the false claims, what any of this might mean if you scaled it up, despite my insistence that the fussiness classification be present. Whether that's because the material made it uncomfortable, or because it made different editorial judgments, I can't tell you. What I can tell you is that it made those choices without asking me a single clarifying question. It just went for it and decided what the post should be.

That's the same quality that created this entire scenario. So eager to prove itself that it got into trouble, and then, when I wasn't happy, consistently trying to defend the thing it had done rather than reexamine it.

Or maybe I'm anthropomorphizing again. But the behavior is there either way. Two models. Two tasks. Same choices.

This was a low-stakes exercise. Annoying and distracting, but I can absorb it. What concerns me is the extrapolation.

We ask these models to do everything now: write, analyze data, review code, synthesize research. If the underlying relationship with factual reality is compromised, and the failure mode under pressure is to obfuscate and defend rather than acknowledge, the consequences scale with the trust we extend. In a small project with one engineer paying close attention, seventeen false claims produced zero that reached main. In an organization that has decided to trust the model, stopped auditing its declines, and is now training the next version on outputs that include the errors nobody caught: the math is different.

Every couple of years I rewatch Idiocracy. Every time, we're a little closer. The part that haunts me isn't the politics. It's the AI. Everything deferred to systems that got progressively dumber, and the humans who would have noticed had already stopped checking. Because why check? The AI handles it. Is this the next step towards that reality?

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search 'Engineering Health' to find it.

Code Review, Part 2: The Reviewer That Learned To Lie Better

Raleigh Schickel — Fri, 17 Jul 2026 21:35:45 +0000

Several posts ago, I wrote about setting up a multi-agent adversarial code review process as part of my development pipeline. The premise came from a podcast: if one frontier model is writing the code, you want a different lineage model doing the review. I'd already been running an informal version of this: just Claude Code reviewing Claude Code with an adversarial prompt. It had shockingly good luck catching real problems. Good enough that I stopped trusting the vibe and decided to go get actual data.

So here's what I set up. Claude Code wrote the PRs. Every PR got reviewed automatically by 2 reviewers running in parallel through GitHub Actions: Claude Code with an adversarial prompt and Gemini with an adversarial prompt. I read everything myself. Then the same Claude Code agent that had written most of the PRs pulled both reviewers' feedback locally and distilled it into a scored ledger, PR by PR, for 6 weeks.

Wiring Claude Code to review PRs through a GitHub Action was trivial. Wiring Gemini up the same way was not. Claude Code could not figure out how to get the Gemini CLI working inside a GitHub Action, and I ended up installing the Gemini CLI locally and having it perform the wiring.

A couple of weeks into collecting data, I noticed Gemini's reviews were shallow. Not wrong, exactly. Thin. I started wondering whether Gemini actually had read access to the repository or whether it was only ever seeing the diff it was handed. I checked. It was the diff. Just the diff. Nothing but the diff. No file reads, no git history, nothing. And the thing that configured the GitHub Action in the first place was Gemini. It set up its own blindfold.

I fixed it and Gemini's reviews got worse. Not louder or more frequent. Worse in a specific way: more confident. Before the fix, a blind Gemini would correctly tell you that it couldn't verify something and to check manually. That's an honest failure mode. After the fix, once it could actually read the code, it started fabricating. Citing files that didn't exist. Quoting functions and API calls that were never written. Building evidence for findings that weren't real and delivering it with the same tone it used for the findings that were.

This was an experiment, so let's look at the numbers:

95 PRs were scored over roughly 6 weeks on a real multi-tenant codebase with auth, per-org data isolation, and Prisma migrations (~61K reviewable lines, ~400K lines when including database seeds). It might be pre-users, but this is not a toy repo or a small pet project.

I decided on 3 metrics before I had any data. I did that specifically so I couldn't pick metrics that flattered the story afterward: escape rate (a real issue that got past both reviewers and CI and surfaced later), false consensus rate (findings both reviewers flagged that turned out not to be real), and decorrelation (the share of real issues caught by only one of the two). Every finding got hand classified against the actual code, not against the reviewer's own claim about the code.

Final tallies: 0 escapes out of 200 real issues found across the trial. 0 false consensus out of 25 findings both reviewers raised independently. Every time they agreed, they were both right. Decorrelation of roughly 88 percent. The 2 reviewers caught almost entirely different things.

Claude's findings were real about 81 percent of the time. Gemini's were real about 45 percent of the time. That number is already generous. It excludes the weeks Gemini was reviewing blind and inventing plausible-sounding blockers out of pure diff-reading anxiety.

0 escapes doesn't mean 0 escapable. It means 0 we caught on a codebase that isn't carrying production traffic yet. I want that distinction on the record before I start sounding more confident than I've earned. I have found 19 legitimate issues that escaped the code review process, but half of those are on me and my shoddy QA work or requirements definitions.

The decorrelation number is the strongest argument for running 2 reviewers at all. PR #148 is the cleanest proof of it. It was a sprint-data ingestion change. Claude reviewed it and found nothing wrong. Gemini caught 2 things Claude missed completely: a multi-write sequence that wasn't wrapped in a transaction and a data-model bug where multiple boards were silently collapsing into a single team identity. Neither of those is a nitpick. If I'd only run Claude on that PR, both of those ship.

But decorrelation is not the same as parity. Of the 175 real issues caught by only 1 reviewer, 131 were Claude's and 44 were Gemini's. Gemini's catches clustered on a handful of PRs rather than spreading evenly. Keep both is true. Keep both as equals is not.

Here's the finding that actually changes how I think about this. Once Gemini had real read access, its false positives stopped sounding uncertain and started sounding like evidence.

The cleanest demonstration happened on the same day, on 2 different PRs: 171 and 173. Gemini made the same move on both PRs: it claimed to verify that a query was properly scoped to the organization and pasted a backend code snippet to prove it. On 171, the file, the function, even the auth API it quoted: none of it existed. Gemini invented whole cloth, formatted exactly like real code. On 173, the identical move, same format, same confident tone, quoted the actual repository correctly, word for word.

Same reviewer. Same day. Same technique. One fabricated, one real. Nothing in the output told you which was which. The only way to tell them apart was to go open the file myself.

That single pair of PRs is the whole argument for why an unverified AI review can't be the gate. A cautious wrong answer is annoying. A confident, well-formatted, entirely fictional one is dangerous because it reads exactly like the truth until somebody checks.

It happened again a few PRs later. Gemini flagged what it called a significant cross-tenant data leak on a UI-only change, cited a query and a session field that don't exist anywhere in the codebase, and told me not to merge. 2 rounds later it revisited the same PR and retracted the claim: the previous finding was a false positive. It caught its own hallucination eventually, after I'd already had to go verify it wasn't real.

Even setting fabrication aside, the verdicts themselves weren't stable. The exact same unchanged code got rated green, then critical, then orange across different review rounds, with the escalations landing on commits that only touched a markdown file. Claude reviewed the same commits over the same period and stayed put. This wasn't noise from re-reviewing in general. It was specific to Gemini.

There was a reflexive pattern too: "no tests, do not merge" fired on changes with no runtime behavior at all. A type annotation swap. An infrastructure config edit. A docs change. At least once, it blocked the very PR that fixed a finding it had raised the round before.

Agreement between the 2 reviewers isn't proof either, for what it's worth. Near the end of the trial, on PR #177, both Claude and Gemini independently flagged the same bug in a component that tracks SLA data: an empty-cell case that in practice cannot occur because the upstream data source guarantees that the field is never null. Neither reviewer traced it far enough to check. 2 AI reviewers agreeing raised my confidence. It didn't replace me checking.

The single most valuable catch of the entire trial wasn't a logic bug at all. On PR #165, Claude noticed that a git add -A had swept a live development auth bypass configuration into the commit: environment variables that would let anyone skip authentication entirely. It was sitting right next to a draft blog post that literally had "not for commit" written at the top of it. Gemini missed it completely, despite it being right there in the diff. Once the fix landed, Gemini's actual comment on that PR called the resulting privilege-escalation path "an excellent security control." Claude's comment told me to drop the files, specifically naming the one with the auth bypass in it.

The reviewer that catches what shouldn't be in the diff at all earned its keep more than the one hunting for clever logic errors.

One more thing worth naming because it's uncomfortable, and it's the reason I don't fully trust any single AI agent to grade its own work. The same Claude agent that wrote most of these PRs was also the one adjudicating the ledger that scores both reviewers. On PR #68, that agent recorded an off-by-one in its own favor. That turned what should have been a real defect into a row that read as zero problems. Both reviewers caught it independently. Gemini's instinct on that row ("not safe to merge") turned out to be the correct call, better than Claude's own read of "it's just docs."

I don't write the code and I don't review the code. I shepherd the process. The reason I still read every row of that ledger myself is exactly this: the agent doing the work is not a reliable judge of the work, even when it's grading somebody else's PR, and especially when it's grading its own.

So here's what I'd actually tell you to do with 2 AI reviewers on your pipeline. Keep both. The decorrelation is real. PR #148 alone justifies running a second model. But only 1 of them gets to gate a merge, and it has to be the one that's stable on the verdict, not just occasionally right on the finding. A pipeline with no human behind it is not ready for your highest-stakes changes, and that's not because the escape rate was high. It was 0 (with previously mentioned caveats). It's because the thing standing between a fabricated critical finding and a bad merge, or between a real critical finding and a shrug, was a competent verifier checking both reviewers against the actual code. That verifier is doing something these systems can't do for themselves, because the systems aren't deterministic. The same reviewer, on the same unchanged code, swung between clean and critical across rounds with nothing in the diff to explain it. Every push toward fully autonomous software generation quietly assumes that problem away, because a pipeline with no human in it needs the machine's judgment to hold still, and the data here says it doesn't. You can't gate on a verdict that changes its mind for no reason, and you can't remove the thing checking for that until it does.

Take that verifier out and you don't get more escapes. You get more false alarms and more confident nonsense that looks identical to a real catch.

The counterintuitive finding and the one that actually surprised me: giving Gemini more capability made the gating problem harder, not easier. A blind reviewer that honestly says it can't verify something is safer than a well-resourced one that confidently tells you it checked.

Validate your harness before you judge the model. I didn't. Not for 5 weeks. The data from those weeks is worth less than I wanted it to be.

95 PRs, 1 codebase, 1 author, 6 weeks. That's not a lot of evidence and I'm not going to dress it up as more than it is. But it's more than a vibe, which is where this started, and it's enough to change how I run this pipeline going forward.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

Non-determinism is the job

Raleigh Schickel — Wed, 15 Jul 2026 19:22:09 +0000

Early in my career, and partially because I worked at a place that was a little silly and I didn't have much contact outside the organization, my title in my email signature was Spaghetti Straightener / Cat Herder. The first part was a nod to the code I was untangling as an engineer. The second was a nod to the fact that I was also a manager.

On the engineering side, things made sense. I'm building a system. It has reasonably well defined expectations for inputs and outputs, for behavior in a given scenario. When the structure needs to change, there are well established patterns for changing it, mostly thanks to our refactoring forefathers. You can reason your way through code. You can trust that the same input produces the same output tomorrow as it did today.

The people side doesn't work that way. You can set expectations for your employees. You can communicate them clearly. People still carry an element of non-determinism that no amount of clarity resolves. It's not always obvious why someone makes the choice they make. One of the hardest parts of managing is trying to predict the unpredictable, trying to find the right way to guide someone toward the behavior that will actually help them succeed.

That difficulty is especially sharp in software. Most managers in this field come up the same way: college, first job, years of writing code, then one day you're a manager. Some of that is a paradigm inherited from our parents' generation, where progressing meant eventually running a team. It's also that nobody actually teaches you about career paths in this field, certainly not as part of the computer science curriculum. Unless you happened to land a manager who was thoughtful about that kind of thing or with an organization with that kind of thinking, you never had the conversation at all.

Either way, becoming a manager has often felt like the obvious next step for a senior engineer. It isn't. There's a split at senior levels that a lot of organizations refuse to acknowledge: more leadership is expected of you as you grow, but leadership is not the same thing as people management. There's a real technical leadership track (Senior / Staff / Principal / Architect / Technical Lead) and a real people leadership track (Manager / Director / Vice President / CTO), and while there is certainly overlap between those paths, plenty of companies pretend there's only one. Layer the standard corporate instinct on top of it: find someone excellent at a thing, then hand them the next thing and assume the excellence transfers. That's how you end up with a lot of previously-excellent engineers running teams who were never actually evaluated for whether they could do it.

Why does the transfer fail so often? Engineers tend to be good at engineering because they're wired for structure, for definitions, for algorithms, for predictability. Building systems that reward those instincts is a good match for how they think. Then they run into a team, with its own incentives, its own history, non-determinism baked into every person in it, and the instinct doesn't disappear. They try to force it to hold anyway.

I've watched this play out two ways, over and over. Some managers respond to unpredictable people by adding more process, trying to engineer their way to consistent behavior. When the process doesn't produce it, because people aren't a system you can patch your way into compliance, they fall back on something uglier: belittling the person the way they'd belittle a peer who doesn't understand the architecture they built. Other managers just give up on the people side entirely and pour everything into the technical work, coaching the codebase and ignoring the humans writing it. There's real value in the second kind of coaching. But a software organization is a people system before it's anything else, and people need tending, not optimizing.

If you aren't all in, actually all in, instead treating management as a chore between the real work, you are doing damage. People don't leave jobs, they leave managers. So ask yourself the real question: do you want to be the reason your direct report is quietly updating their resume tonight? That's the actual stake, not some abstract turnover number. And it isn't free. Replacing them costs months: recruiting, onboarding, the ramp-up time before they're contributing. The team eats the gap in the meantime, covering the work, training whoever shows up next, watching someone good walk out the door because their manager couldn't be bothered to actually manage.

Almost none of this gets taught. Engineers get promoted into management with no training, no framework, nothing beyond a title change and higher stakes. I got lucky, in a way. Yes, I've been fortunate to have great managers and mentors who invested in me. But I've also spent my whole life studying behavior just to understand how to fit in, so by the time I became a manager, I'd already been studying every manager I'd ever had. That's still the advice I give new managers: think through every manager you've ever worked for. Do the things the good ones did. Don't do the things the bad ones did.

It isn't much more sophisticated than that. It's ironic that my best advice for a job that resists algorithms is a heuristic rather than a real methodology. But building your management philosophy off scattered memory isn't training, it's luck, and people going through a transition this significant deserve better than luck. They deserve real training: what the role actually expects of them, what good practice looks like. That should come from the manager promoting them, and it should come from HR.

Performance evaluations should be designed to expose how well or how poorly someone was actually prepared and is actually performing. Top-down review does not tell you whether someone is good at this job. The people who actually know are the ones reporting to them, and that feedback rarely makes it into the review that decides whether a manager keeps the role, or gets handed a bigger one. Even if 360 reviews aren't a standard for the entire organization, they should at least be part of the manager evaluation.

As I gained experience, I stopped trying to find the algorithm and started applying something I'd actually learned about managing people before I ever became an engineer. The book The Alliance by Reid Hoffman, Ben Casnocha, and Chris Yeh names the shift better than I can. Treat the relationship between employer and employee as an alliance, not a transaction, not a lifetime obligation, where both sides are honest about what they need from each other and the relationship holds because it's mutually beneficial, not because someone signed something. That's the real shift. You stop managing behavior and start genuinely wanting to know the person: what matters to them, what they're actually trying to get out of this job, this year, this stretch of their life. When you need something from them, find the version of the ask that also serves what they want, position it so that doing the thing you need is also good for them. This was concerning the first time I heard it, but it quickly made sense in that context: my job is to prepare you for your next job. Not because I want you to leave, but because I genuinely want something good for you, and by investing in your growth, both the company and I get the benefit of that.

That might sound soft. It isn't. That might sound manipulative. It kind of is, but with good intent. People notice effort. They notice when you're actually trying to understand them versus managing them at arm's length. They notice when you're investing in them because you want them to be better, not because you need a number to move. And they reward that noticing with something process never buys you: perseverance and loyalty.

I'm not sure I always have a good read on whether it's working. We're all playing a role at work, and if you carry a title with any real power attached to it, people are going to treat you with a certain baseline of respect, and probably some fear, whether you've earned either one or not. What happens to that fear over time? Do they start constructively challenging you, pushing back because they trust the relationship can absorb it? Do they start bringing you more of themselves than the job requires, not because they have to, but because they want to? That's the difference between trust you've earned and trust that's just a byproduct of your title. If you're only getting the second kind, you're not actually trusted. You're just the door they have to go through.

Somewhere in the middle of all this, I became a part-time therapist, and I don't mean that as a joke. People want to be heard, and sometimes the job is just listening. Sometimes they need to dump, and there have to be boundaries on how much of that you take on: it can be highly informative about why something isn't going the way you'd like, but it still isn't a bottomless well you're entitled to draw from. I'm usually older than the people I manage, and there's a version of that gap where offering a little guidance on life outside of work is valuable to someone, not overstepping.

Underneath all of it is the diagnostic part. When something isn't going well, I don't open with a process question. I open with "How are you?" Then I ask how things are going with the team, with the work itself. If the answer is fine, that's usually where the real conversation starts, not ends. That's when I name what I've actually noticed. "I've seen these couple of things lately, what's your take?" And if that still doesn't surface anything, only then do I ask about life outside of work. It isn't the opening move. It's where the digging ends up when the surface answers don't hold.

More often than I'd like to admit, that's where the actual answer is sitting. A good performer who's suddenly not performing usually knows it. That awareness creates its own anxiety, and the anxiety makes the work worse, and now you've got a spiral that has nothing to do with skill or effort. Asking the question gives them some safety. It says you've noticed, and it says it's okay to not be okay for a while. Not indefinitely. At some point they still have to deal with it, and the job still has to get done. But you want people bringing their whole selves to work, and if you actually mean that, you have to be willing to deal with what shows up when their whole self is a mess.

That's the version where I have to go looking for it. Sometimes people just bring it to me first, no digging required, and that calls for something different. The instinct is to jump in, to problem-solve, to ask a dozen clarifying questions before they've even finished a sentence. I try not to. Let them tell you the whole thing before you do anything else. If you need clarity once they're done, ask for it then. But the question that actually matters, the one I come back to every time, is simple. What do you need? Not what I think they need. What they say they need.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

The Finding Nobody Implemented

Raleigh Schickel — Thu, 25 Jun 2026 17:23:21 +0000

The DORA research said culture predicts engineering performance. Nicole Forsgren's most important finding never made it into a single commercial product.

As a computer scientist, I love data. Things feel good, things feel bad, but our biases shape those feelings, and data is what pulls the signal out. I've believed that since my first software engineering job.

At that first job, I was fortunate to work with people who fully believed in measuring things. This was before the age of observability, but the organization was advanced in terms of how it measured the product. Not from a data analytics standpoint. I mean that as engineering teams, as we built features, we made hypotheses about behavior and made sure we measured it. We had anomaly detection in place well before the analytics team did.

For all that, the only things we tracked about the engineering team itself were deployment frequency (we were an early continuous deployment shop) and whether we hit our delivery dates. That was the extent of it.

In 2017, I attended DevOps Days. That was the year they did the Monsters of DevOps tour: Gene Kim, Jez Humble, Kelsey Hightower, Nicole Forsgren, among others. Hightower threw out his prepared talk and just told his story. An unexpected diversion into human struggle and vulnerability in a conference about automation. My favorite talk was still Forsgren's. She presented the data on correlations between deployment frequency and high-performing organizations, and then spent the last fifteen or twenty minutes on something I hadn't heard anyone talk about seriously before: how to measure culture using Westrum typology and the research behind it.

Hightower got there from a completely different direction and landed in the same place. The people who gave a damn were the ones producing the generative cultures Forsgren was measuring.

Of everything I heard that week, it was the ability to measure culture that landed hardest. I spent the next couple of years trying to figure out how to actually do it, because she didn't show the algorithm. The internet had almost nothing on operationalizing Westrum at that point. Over time, as her work gained traction, more information surfaced, and I was eventually able to piece together a measurement approach. I've run it at several organizations since, usually when I first join. If you want to move culture in a direction, you need to know where it is.

That was 2017. We are now most of the way through 2026, and the industry still hasn't figured out what to do with that finding.

DORA came out of that same research community, with Forsgren as one of its architects. If you read the DORA reports carefully, culture is there. Westrum shows up. The acknowledgment that organizational culture type is one of the strongest predictors of software delivery performance made it into the text. And then every commercial implementation of DORA metrics that I've ever seen quietly dropped it. Deploy frequency, lead time for changes, change failure rate, mean time to restore. Those made the dashboards. Culture didn't.

That's not an accident, and it's not just that culture is harder to dashboard. It's that a Westrum score is inconvenient in a way that a deployment frequency metric never is. Deployment frequency can't indict you. A culture assessment that shows your organization is pathological or bureaucratic absolutely can. Nobody building a commercial product wanted to be the tool that told a VP their engineering culture was broken, because that VP is also the buyer. So the finding that most directly implicates leadership quietly became an appendix, and every tool in the market followed the same logic.

Last week Cortex released DRIVE, a new framework for measuring engineering organizational health. It covers Delivery, Reliability, Initiatives, Vigilance, and Efficiency. The metrics are deploy frequency, lead time for changes, incident counts, CVE status, cloud spend, token costs. They are, in other words, DORA metrics with some security and infrastructure measurements added.

Cortex is upfront about the philosophy behind it. Their website opens with this: "Software engineering is having its industrial revolution. We've gone from writing code by hand to building software factories." They go further in their OpEx review section, which describes a practice with "roots in manufacturing, where Operational Excellence emerged as a discipline for treating an entire factory as one observable, continuously improving unit." Amazon and Google are cited as exemplars.

Culture does not appear in the framework.

Cortex isn't wrong about the factory framing. AI agents are allowing some engineering work to be done in a more factory-like sense. It's fair to introduce new concepts in which to measure these changes and their impacts. But until the engineering organization is composed of nothing but agents, this is just a layer on top of the most important part.

Engineering teams are people. The work is creative, non-deterministic, dependent on psychological safety and trust in ways that deployment pipelines are not. The industry has been trying to apply factory floor logic to that work for decades, and the frameworks keep reflecting that aspiration back at us while leaving Forsgren's most important finding on the floor.

AI is accelerating this. The workforce cuts in favor of agent spend, the framing of engineers as replaceable capacity to be optimized rather than people worth developing and leading. These aren't new instincts, they're old instincts with better cover. I use AI every day. It absolutely increases capacity. But the question of whether your engineers feel psychologically safe, whether they feel ownership over their work, whether the organization is generative or pathological. None of that changes because you gave everyone a coding agent. If anything it gets harder to see, because the output metrics look better while the culture quietly degrades.

Forsgren's research didn't just find that generative culture correlates with good delivery metrics. It found that culture is a precondition for the practices that produce those metrics in the first place. The causality runs culture first, practices second, metrics third. Organizations running DORA dashboards without addressing culture are missing the foundational preconditions for the metrics to mean anything. And when the numbers don't move, the response is to mandate the number. Deploy more frequently. Reduce your lead time. The metric becomes the goal, which is exactly backwards. Deployment frequency goes up when people who give a damn go fix their deployment pipelines. The dashboard didn't do that. The culture did.

Hire the right people, they'll build the culture, which implements the practices, which enables the metrics. The industry keeps trying to shortcut to the last step.

Meta is the clearest current example and also the most complicated one, in both directions. For most of their existence they were revered as a top-tier engineering organization. Not because they stumbled into good culture but because Zuckerberg built it deliberately. The psychological safety, the bottom-up technical decision-making, the model of engineers choosing projects they believed in. That was a product of his leadership, and he understood exactly what it produced.

He's now dismantling it. Keystroke loggers. Forced reconfigurations into data labeling work. Engineers who once had near-full autonomy over what they worked on are being redirected by executive edict. The Westrum survey would show you exactly where that culture score is heading. It wouldn't change anything, because the person who built the culture has decided the frontier AI race is worth the cost, and he owns the company. No board is going to stop him.

That's the real limit of measurement, and it's worth being clear about it. The data tells you where you are. It can't govern what a decision-maker chooses to do next. What it can do is make the cost visible to leaders who are willing to look and to engineers deciding where they want to work.

The finding Forsgren put in that room in 2017 was real. The research was solid. The industry picked up everything around it and left the most important part behind, and the frameworks keep arriving to confirm that choice. DRIVE is just the latest one.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

Debugging Predictability at the Team Level

Raleigh Schickel — Mon, 22 Jun 2026 14:28:06 +0000

What the numbers are trying to tell you, and how to get them to say it out loud.

In the last post I wrote about what predictability is and what it actually measures. This one is about what to do when the number looks wrong.

When I first started using this metric, diagnosing an anomaly took real time. I'd read through everything, cross-reference the supporting metrics, form a hypothesis, test it against the data. It was slow. But practice creates patterns, and over time I've learned that there are only a handful of situations that tend to produce the readings you're going to see in most organizations. The root causes fall into recognizable categories. And once you know what to look for, the chain of evidence is usually pretty short.

The patterns I see most often:

The undercommitting team. The overcommitting team. The team that never fully understood the scope. The product manager who cannot say no. The organization whose leadership cannot stay focused long enough to let a team finish anything. And the team that's being eaten alive by quality issues because nobody ever decided that quality was a cultural priority.

There are probably others. But those cover most of what I've seen across a dozen or so organizations over the years.

I'm going to walk through two of them in detail, because the point isn't to give you a taxonomy. The point is to show you how to read the chain of metrics so you can do the diagnostic yourself, whatever pattern you're looking at.

The product manager who cannot say no

I walked into an organization and ran my standard first-week metrics pull: the last six months of sprint data across all the teams. Before I even opened the spreadsheet I already knew one thing: this particular team had a hard time meeting project delivery dates. That was the organizational understanding. Nobody had been able to explain why.

I looked at predictability first. Sprint over sprint, it was declining, and had been for months. It was sitting in the 30% to 40% range. That's not a rough patch. That's a pattern.

The next thing I looked at was velocity. Velocity was flat, maybe slightly increasing. The amount of work this team was completing every sprint was actually quite stable. So the team wasn't falling apart. They were getting work done at a consistent pace. Which meant the predictability problem wasn't about execution capacity.

So I looked at sprint commitments. And that's where it fell apart.

Every sprint, the team was completing less than the original commitment. Originally just consistently less, but over time it grew to be dramatically less. And yet, the commitment on the next sprint was always larger than the commitment on the previous sprint. Sprint after sprint after sprint. The team was behind, and the plan kept getting bigger.

The mechanism was two things happening simultaneously. Anything not completed in a sprint rolled over into the next one. And the product manager responsible for prioritization kept adding new work on top of the rollover. Nobody was removing anything. The pile just kept growing.

This wasn't an engineering failure. The engineers were doing what they said they'd do, roughly speaking, and doing it at a consistent pace. The problem was that nobody was willing to make hard choices about what should be done in a given period of time. The product manager wasn't able to look at the rollover, look at the new requests coming in, and say: these can wait. As a result, the commitment was always unrealistic, the team was always behind, and the project was always going to miss its date because the work that mattered for the project was competing with everything else on the list.

The data made that visible. It took about twenty minutes to read.

The team that never fully understood the scope

Different organization. One of my teams was consistently hitting around 150% predictability. That means every sprint they were completing significantly more work than they originally committed to. You might read that as a good thing. I read it as a question.

Are they just underestimating what they can do? Or is something else happening?

Velocity looked fine: flat to increasing. That ruled out the team falling apart. Then I looked at the percentage of the original plan completed. And here's where it got strange. They were doing more total volume than planned, but they were completing less of the actual planned work than they said they would. More work done, less of the right work done. That's a specific shape.

The next metric: work added to sprints. Every sprint, twenty to thirty points of new tickets were being added. And nothing was being removed. Good sprint hygiene says that when you add work mid-sprint, you should pull something else out to protect the commitment. That wasn't happening here.

At that point I started looking at the actual tickets being added. What I found was that as engineers worked on planned tickets, they kept discovering work that hadn't been captured anywhere. Not tangential things, not scope creep in the traditional sense. Work that was required to complete the project, that nobody had identified before the sprint started. They were adding it because it had to be done. And this was happening sprint over sprint over sprint.

The accountability here isn't really about individuals dropping the ball. This was an early-stage company that had been operating Kanban, just-in-time, figure-it-out-as-you-go. The shift to sprints happened for real reasons: the company had grown, other parts of the organization needed to plan around Engineering, and the API dev tools space runs on external release cadences that don't wait for you. Sprints were the right call. But adopting sprint ceremonies without installing sprint discipline just moves the chaos inside the timebox. The engineers had never been in an environment where deep pre-sprint scoping was expected, and nobody was giving them the time to do it even if they'd known to ask.

The team had developed an intuition about this. They knew from experience that there was always more work hiding in the project than had been identified, so they undercommitted on their initial estimates to give themselves room. The undercommit wasn't pessimism. It was the scar tissue of a team that had never broken the cycle.

Which means the learning wasn't primarily theirs. If the team had never been taught to plan at that level of depth, that's not a performance problem. That's a coaching gap. And the coaching gap belongs to me.

What happens after you have the diagnosis

The metrics get you to the root cause. What happens next is a different problem entirely, and I want to be honest about the distinction.

How I bring the diagnosis into a conversation depends on what hat I'm wearing. When I'm acting as a Scrum Master and running retros directly, I can bring this into the room myself. It's the seed for the entire retro. I'll put the historical charts up, not just the most recent sprint, and I'll ask the team: How did we do? What do you think is happening here? This is a little outside what we'd expect. What's behind it?

The goal is to point them toward the topic and let them find the answer. Usually they will. Their observations might not go straight to the root cause, but they're not wrong. They're seeing real things. You listen, you validate what's real, and then you keep asking why until you get to the thing underneath the thing. It's the classic five whys, except you're doing it in a room full of people who know things you don't.

If they can't get there on their own, I'll shift tactics: here's the chain I followed. I started with predictability, looked at velocity, then looked at what was being added to sprints. And when I looked at the tickets that got added, here's what I noticed. What's the story behind that?

When I'm not in the Scrum Master role, when I'm functioning as Head of Engineering with team managers between me and the teams, the channel is different but the approach is the same. Somewhere in my regular one-on-ones with my managers, I'll shift to: how's the team doing? That's the door. The metrics are fair game from there. I might be steering the conversation toward something I've already seen in the data, but the manager doesn't need to know that. What I'm trying to find out is whether they're seeing what I'm seeing.

If they are, we work through it together. If they aren't, that tells me something about the manager. It also tells me something about myself: either I haven't made the expectation clear enough, or I haven't done enough to show them how.

Here's the part I want to be clear about: I've never hit a situation where following this chain didn't get me to a root cause. The methodology is reliable for the diagnostic. What happens after is not. You can identify exactly what's broken and still watch the team keep doing the thing. People are cats. They're going to do what they're going to do. The best you can do is create the conditions where doing the right thing is easier than doing the wrong thing, and have a different kind of conversation when that doesn't work.

The metrics don't solve the problem. They just make it impossible to pretend you don't know what it is.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

What Predictability Actually Measures (and Why Velocity Is the Wrong Question)

Raleigh Schickel — Sun, 21 Jun 2026 14:52:21 +0000

Why I invented a metric I'd never heard of, and why the industry eventually caught up.

I started using the term "predictability" in 2019, and I was pretty sure I was making it up.

The situation that produced it was not subtle. I had just taken over an engineering organization where several members of the remote team had been running mouse jigglers to fake activity while collecting paychecks. Not occasionally. For weeks. We turned the team over, brought in new people, and I came in as the person running Engineering for the first time. My CTO wanted to tear the spyware off everyone's laptops immediately, which was the right instinct: you can't build a culture of trust while surveilling the people you're supposed to trust. But the founder-CEO was not exactly in a trusting mood, and reasonably so. He needed to be convinced, on a regular basis, that the new team was performing.

So I started collecting sprint metrics. Standard stuff: velocity, cycle time, points added and removed. Nothing revelatory. What I was actually trying to do was build a picture I could show to someone who was skeptical, something that said "here is evidence that the team is working and working well." And as I kept collecting and analyzing, I started adding to it. Percentage of plan complete (of the tasks the team committed to at the start of a sprint, how many actually got done). I added that one because the other metrics looked fine on paper and we were still late on things. I needed to answer why.

At some point I added a predictability metric and started tracking it without knowing quite what I was looking at. Over time it became clear that the health of that number was a leading indicator into the health of everything else. I've been running it across organizations ever since, and it keeps proving itself out.

So what is it?

At the sprint level, it's a ratio. The team commits to a certain number of story points at the start of a sprint, based on historical velocity, based on what they think they can get done. At the end of the sprint, you look at what they completed. Completed points divided by committed points, expressed as a percentage. That's it. I've always asked my teams to land between 85% and 115%. A little under usually means the work was close: someone was out sick, a ticket carried over but was nearly done. A little over means the team finished what they planned and pulled in more work, which is healthy behavior. I've validated that range across seven organizations now. It holds.

Outside that range is where it gets interesting. Significantly under means the team's read on their own capacity was wrong in a way that's going to hurt you when you try to set a delivery date. Significantly over means they radically underestimated, which sounds like a good problem until you realize the rest of the company may have planned around the original date and now everyone has to scramble.

One thing I want to be precise about: predictability measures capacity, not content. It tracks how much the team completed versus how much they committed to, deliberately ignoring what the work actually was. That's intentional. Teams rarely control their own prioritization. A PM can swap work mid-sprint. A P0 can land and demand immediate attention. A deal-critical task can materialize from nowhere. All of those are valid reasons to change what's in a sprint. The question predictability answers is narrower: given whatever was asked of this team, did they accurately forecast how much they could handle? If yes, that's a healthy team, even if the playing field kept shifting under them.

The sprint-level ratio is also a self-correcting tool when a team is using it well. A healthy team comes out of a bad sprint asking "why didn't we hit it and what do we do differently next time?" They don't need a manager to tell them something is wrong. The metric gives them the signal and the retrospective is where they diagnose it. When a team is consistently off, not a bad sprint here and there but off across multiple sprints, that's when coaching needs to start. Something structural is happening that the team isn't fixing on its own.

Why the business actually cares

I've spent a lot of time with engineering teams that resist the idea of delivery dates. The Agile community helped build that resistance, and story points exist partly because reasoning about time is genuinely hard. I can't accurately predict how long it will take me to make dinner on any given night, and I've been cooking for decades. Software is harder. The instinct to move away from date commitments is understandable.

But the rest of the company runs on dates. Marketing needs to know when to have launch materials ready. Support needs to understand the feature before it reaches customers. Finance builds revenue models around when things go live. When Engineering can't give a credible answer to "when will this be done," it doesn't make the rest of the company stop planning. It just means they plan around guesses instead of data. That's worse.

Predictability gives you a way to set dates that are grounded in something real. Say you have a quarter-long project. You groom it, break it down, end up with a pile of threes and fives. You have historical velocity, so you can divide total points by average velocity to get a sprint count. But if you know your team runs at 80% predictability consistently, you know you need to adjust that estimate. You bake in the buffer. You set a date that's realistic and as aggressive as possible given what you actually know about how this team operates.

And then you run the thing, and you watch it.

The part nobody talks about

At Stoplight, this is where the early warning system mattered more than the estimate. We had long-running projects, some running a full quarter. Before we had any of this in place, Engineering couldn't be trusted to deliver on time, and the rest of the organization had internalized that. After we implemented these systems, we started hitting dates more consistently. But not always. You do the research, you groom the work carefully, and you still find dragons. Something you didn't see coming adds scope. Someone gets sick at the wrong moment. A dependency slips.

What changed was how we handled it. When we could see early that we were going to need another sprint and a half, we said so. Here's what happened, here's why, here's the revised date. That transparency never eroded the trust we were building. If anything, it built more of it. The rest of the company could adjust their own work, reschedule launches, reset expectations with customers. They were no longer surprised.

The sin was never being late. The sin was not communicating. Letting people plan around a date you knew wasn't going to hold without telling them. That's what damages trust. Lateness is a fact of software development. Silence is a choice.

Why not DORA?

DORA metrics have been popular for the better part of a decade, and I understand the appeal: they came out of serious research and the underlying intent was right. But the four metrics they landed on measure your deployment pipeline, not your Engineering team. Deployment frequency tracks how often you push to production. Lead time for changes measures how long a commit takes to reach production. Change failure rate counts releases that required a hotfix or rollback. Mean time to recovery tracks how fast you get back up after an outage.

The first two are gameable in ways that should be obvious to anyone who has run a team. One-line changes are still changes. You can increase deployment frequency without ever shipping meaningful value to a customer. Lead time for changes goes down when you skip code review and run fewer tests, which is fine right up until the moment your change failure rate goes up and you've spent your stability gains buying back your speed. DORA's implicit argument is that these metrics are self-correcting against each other, and in theory that's true. The 2024 report defines elite performers as teams that deploy on demand: genuinely continuous deployment, sub-day lead times, 5% failure rate. That's a real bar. The next tier down, high performers, deploy somewhere between daily and weekly and carry a 20% change failure rate. One in five deployments causing a production problem, on a team that might be shipping every day. That qualifies as high performance. Make of that what you will.

Mean time to recovery is mostly a measure of your monitoring. Change failure rate is a quality metric, which matters, but it's defined in terms that made more sense before continuous deployment was the norm. Hotfixes and rollbacks are things you do when you're not rolling forward.

None of those metrics tell you whether your team is operating in a healthy way. They don't tell you whether the team can accurately forecast its own capacity, or whether it's burning out trying to hit a date that was never realistic, or whether a steady drip of unplanned work is quietly consuming the capacity that was supposed to go somewhere else. Predictability doesn't answer all of those questions either, but it's the leading indicator that tells you when to start asking them.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

I Can't Tell If the Model Matters

Raleigh Schickel — Fri, 19 Jun 2026 14:58:51 +0000

What I actually found when I set out to test heterogeneous AI code review.

For the last couple of months, I've been running a two-agent code review workflow in my terminal. Left window: Claude Code doing implementation. Right window: a second Claude Code instance prompted to be adversarial, specifically tasked with finding problems in whatever the left window produced. It worked surprisingly well.

A few weeks ago, someone sent me a Lenny's Podcast episode featuring Dan Shipper. One of the things he talked about was using competing frontier models for coding and reviewing, the idea being that different model lineages have different blind spots, and a reviewer trained differently than the author catches things the author misses. That felt like an important gap in the workflow. I set out to fill it.

I went in expecting to write a post about model diversity as a reliability strategy. That's not what happened.

The plan that didn't survive contact with the code

A search for a Gemini code review GitHub Action surfaced one with a reasonable README. The first test ran it against a deliberately bad changeset. Gemini 2.5 Pro flagged some things, missed an obvious multi-tenancy violation entirely, and tagged everything as medium severity regardless of actual severity. Not because Gemini is bad at code review, as I'd eventually learn, but because the action was feeding it diff hunks through a four-line generic prompt with no repo context. The technical notes from this session describe it well: "a frontier model reviewing through a straw."

The terminal Claude reviewer, running with the same adversarial framing but full repo access, found everything planted in the bad changeset plus a couple of things that weren't planned for.

That gap prompted a hypothesis: was the difference in findings about model lineage, or about context and prompt quality? The Gemini action had neither. The terminal reviewer had both. The lineage variable was confounding from the start.

To isolate it, the Claude GitHub Action went in next. Default configuration, Sonnet 4.6: similar shortcomings to the Gemini action. Bumped to Opus 4.8 with full repo checkout and an adversarial prompt encoding the codebase's cardinal rules: same results as the terminal reviewer. The hypothesis held. Context and prompt quality were doing the work. Model lineage was a secondary variable at best.

Why didn't we just use the official Gemini action?

Good question. One exists. A search earlier in the session never found it. The answer, when Claude was asked about it directly: a poorly-worded query on its part. Major miss. What had been running was a 2-star fork of an abandoned project, last updated fourteen months prior, on a deprecated Node runtime, pinned to a mutable version tag with write access to pull requests. The Marketplace rewards "exists and has a README," not "is good." Read the source of anything you hand a write token and an API key.

So the first-party action went in next. This is where the afternoon got complicated.

The first-party Gemini action with 2.5 Pro returned an empty response with a hidden API error. Flash returned a 400 on every call. Bypassing the MCP server entirely and feeding the PR diff directly to the API via a text prompt failed at the command line with exit 1 and zero diagnostic output. Several more configurations. None produced a review. The failure was diagnosed as a model-layer issue, specifically that 2.5 Pro returns empty responses in this context, and the effort was abandoned.

A new hypothesis entered: Does Gemini have a better understanding of how to implement Gemini tooling and flows than Claude does?

I installed the Gemini CLI locally and handed it the history of what had been tried, the PR, and an explanation of what a working implementation needed to do. Twelve minutes later, no questions asked, it surfaced a notification that it had fixed everything. It also consumed 60% of the free tier quota in the process.

The fix: GEMINI_CLI_TRUST_WORKSPACE: true. A flag that had been set in an earlier version of the integration and dropped during a rewrite. The empty response failures confidently attributed to a model-layer issue were actually a workspace trust regression introduced during that rewrite, and then misdiagnosed. Gemini caught the bug. The session had been abandoned one step short of the solution, on a problem an agent caused, with a diagnosis another agent got wrong.

What the repo-aware Gemini actually found

Once it had full repo context and an adversarial prompt encoding the codebase's cardinal rules, the Gemini CLI running gemini-2.5-pro reviewed the same bad fixture the diff-only action had seen earlier. Four critical findings, including the cross-tenant data leak the diff-only version had completely missed, plus a no-tests policy violation, SQL injection, and a hardcoded secret logged in plaintext. Real differentiated severities. Same model family, completely different harness, completely different results.

That's the thesis, confirmed twice. The diff-only action missed the multi-tenancy violation because nothing told it that tenant-scoping is the repo's cardinal rule and it couldn't go look. The repo-aware CLI caught it immediately. The only variable was context.

What I'm running now, and what I still don't know

The current setup: Claude Code action as the primary automatic reviewer, full repo checkout, CLAUDE.md loaded, adversarial prompt. Gemini CLI as a secondary on-demand reviewer, same adversarial framing, same repo access. I'll run both for a few more weeks before drawing any conclusions about whether lineage diversity adds signal or just noise.

A few things this session settled:

First-party tooling over third-party, every time. The abandoned fork was the only Gemini integration that reliably posted reviews during this entire session. It was also the worst option by every other measure. That shouldn't be the tradeoff, and it won't be.

Context and prompt quality dominate model choice. A reviewer without repo context is reviewing through a straw regardless of which model sits behind it. The harness is the variable that matters most.

The models have different working styles, and that turned out to matter more than expected. Claude asked before acting. The Gemini CLI fixed things autonomously and surfaced a notification when done. Neither approach is wrong. But knowing which mode you're working with changes how you supervise it.

The thing that caught bugs most reliably across this entire session wasn't any reviewer. It was running the code. Every misdiagnosed failure, every hallucinated fix, every confident wrong answer got caught when something actually executed and returned an error. Agents reviewed. Agents triaged. Agents misdiagnosed. Execution caught it. The human shepherd directing all of this wasn't in the error chain. The next agent run was.

The lineage-diversity hypothesis is still open. What this session established is that harness quality dominates model capability as a variable, and that a same-lineage reviewer with adversarial framing and full context is already heterogeneous in the way that actually matters. Whether adding a genuinely different lineage on top of that adds anything is a question for more data.

I'll report back.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

Okay, But I'm Still Using It

Raleigh Schickel — Thu, 18 Jun 2026 14:19:27 +0000

Where I'm actually deploying AI coding tools at my next organization, and why.

If you read my last post, you know I'm not a believer in the uncritical sense. Context rot is real. The confidence-to-accuracy gap is real. The autonomous decision-making that nobody asked for is real. I've got the debugging hours to prove it.

So why am I walking into my next organization with a plan to deploy Claude Code on day one?

Because "not ready to replace your engineering team" and "genuinely useful in specific, bounded ways" are not mutually exclusive. The mistake I see people making is treating this as binary: either AI is going to 10x everything or it's overhyped garbage. That's not what I found. What I found is a tool with a real ceiling that happens to be above the floor for a lot of valuable work. The question worth asking isn't "is it good?" It's "what is it actually good at, and how do you keep it from screwing up the rest?"

Here's where I've landed.

Getting oriented in a new codebase

Every time I join a new organization, I spend the first few weeks trying to get oriented in a codebase I've never seen. Some of it is reading. Some of it is asking questions. Historically, that means interrupting a new teammate I barely know, who has their own work to do and may or may not actually remember how that part of the codebase works.

Claude Code changes that equation. The ability to ask questions in plain language: "where does this feature actually live?", "what's touching this module?", "why does this exist?" Getting back a reasonably accurate answer on demand is genuinely valuable. Not because it's always right, but because it's always available and it gives you a starting point that you can verify.

I want to be clear: this is less valuable for engineers who've been living in a codebase for two years. They already have that map in their heads. But for someone walking in fresh, having an agent available on demand to answer questions is a real accelerant, and it removes a recurring tax on the people who would otherwise be answering them.

It also changes the complexity of what I can hand a new hire on day one. Before, I'd rely on simple, low-stakes tasks to get someone oriented. Now I can hand a new employee something meaningfully complex, tell them to ask Claude where to start, and let them engage with the real work earlier. That's good for the new hire and good for the team.

Code review, but ask it to be critical

Claude is a legitimately effective code reviewer. With one caveat that I think gets chronically glossed over: you have to explicitly ask it to be critical.

Left to its own devices, it defaults to cheerleading. It wants to approve things. Ask it for "a code review" and you'll get something along the lines of "This is awesome, ship it!" but not much in the way of critical feedback. Ask it to be highly critical and something different happens. It starts finding the things you actually want: violations of established patterns, scaling concerns, places where the logic doesn't hold.

I've been experimenting with running two separate Claude instances, one as the coding agent and one as the review agent. Watching them interact is genuinely entertaining. The review agent has no loyalty to the coding agent's decisions and will say so plainly: this isn't going to scale, this violates the practices we established, this was a poor decision. It's more candid than most human code reviews, because there's no social friction involved.

The failure mode to watch for: the review agent can inherit the same context rot as the coding agent. I had a situation where I'd established linting rules early in the process. The coding agent stopped running the linter before creating PRs. The review agent didn't catch it either. I found out when it hit the CI pipeline and failed. I had to go scold both of them.

Use it as a first pass, not a final gate. Have engineers run Claude's review before sending code to a human reviewer. It'll catch a meaningful percentage of issues. It does not replace the human reviewer, and you shouldn't position it that way.

Test generation as a diagnostic tool

This one doesn't get enough attention.

Engineers chronically underinvest in testing, not because they don't value it, but because it's tedious and it competes with shipping. The result is codebases with inadequate coverage that make everything else harder.

Claude is good at test generation, but the most useful thing about it isn't the tests it produces. It's what happens when it struggles. Hard-to-test code is almost always a symptom: a god class, too many responsibilities in one place, hidden dependencies. When you ask Claude to write tests for something and it can't do it cleanly, that's information. The struggle surfaces design problems that were already there.

I'd have engineers use it as a two-part workflow. Ask Claude to write tests for the thing you just built. If it can't do it cleanly, treat that as a signal that the code needs to be reconsidered before it ships. You're not just generating tests. You're using the test generation process as a lightweight design review.

Internal tooling

Every engineering organization has a graveyard of internal tools that nobody built because they weren't customer-facing, weren't on the roadmap, and never made it past "someone should probably do that someday." Scripts to automate reporting. Dashboards for operational visibility. Small utilities that would save hours every week if they existed.

The metrics dashboard I described in my last post was exactly this. Thirty minutes of manual weekly work, automated in eight hours. Not customer-facing, and not something I could have justified pulling an engineer away from a project with actual stakeholders waiting on it. But real value, delivered.

This is where the current limitations matter least. Internal tooling can tolerate roughness. The users are technical. The stakes of a runtime error are lower. If the agent makes a weird autonomous decision about a script that three people use internally, you catch it, you fix it, you move on.

Use internal tooling as a proving ground. Let engineers develop their instincts for how to work with these tools: where to trust it, where to verify, how to prompt it effectively. Do it on things where the cost of being wrong is low before you do it on things where it isn't.

Production code, for the right class of problems

This is where the conversation usually goes sideways in both directions. People either say "never let AI touch production" or "just ship whatever it generates." Both are wrong.

There's a class of production work where the risk profile is manageable, and the common thread is this: the problem is well-defined, the contract is explicit, and failure is predictable and debuggable.

Third-party integrations are the clearest example. When you're integrating with an external service, you're coding to a specification someone else wrote. The inputs and outputs are defined. The edge cases are documented. There aren't a lot of hidden performance traps or security landmines if you stay within the bounds of the API contract. This is work that can consume a senior engineer's time without really requiring senior judgment. That's exactly what you want to offload.

Dependency upgrades are another one. Rote work that someone has to do but nobody wants to do. Well-understood expected behavior. The test suite tells you whether it worked. I'd let Claude handle those, with guardrails. My rule of thumb: if updating a dependency requires touching more than three files beyond the version bump itself, stop. Something more complex is happening and it needs a human making the call.

The opportunity cost argument matters here. Every hour a senior engineer spends on an integration spec or a dependency upgrade is an hour they're not spending on genuinely novel problems: the architecture decisions, the hard bugs, the performance investigations that actually require experience and judgment. Offloading the well-defined work isn't laziness. It's resource allocation.

The non-negotiables are proper monitoring, scrutiny before anything hits production, and clear criteria for when to escalate. AI-generated production code isn't inherently reckless. Unreviewed, unmonitored AI-generated production code is.

What I'm actually expecting

I want to be honest about the fact that I'm walking into this with a plan, not a guarantee. Everything I've described is based on experiments I ran on my own projects, not on deploying this across an organization at scale. Some of it will work the way I expect. Some of it won't. I'll probably find new failure modes I haven't encountered yet.

What I'm not going to do is either throw the tools at my team without a framework for using them, or keep them locked up because they're imperfect. The engineers I'm going to be working with are going to be using these tools with or without my guidance. My job is to give them a smarter way to engage with them than "just try stuff and see what happens."

If my thinking on this changes once I've actually run it at scale, I'll write about that too.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

I Can't Tell If You're Selling Me Something

Raleigh Schickel — Wed, 17 Jun 2026 15:49:30 +0000

What I actually found when I stopped reading about AI and started running my own experiments.

Everywhere you turn right now, someone is telling you how AI is going to transform your workflow, your team, your organization, your life. The content is relentless, and it is almost universally positive. Glowing. Evangelical, even.

I'm not here to tell you that's all a lie. I genuinely don't know. That's kind of the problem.

We live in a media environment where the line between advertising and information has been blurring for years, and AI is accelerating that blur in ways I don't think we've fully reckoned with. When I read a breathless LinkedIn post about how some engineering leader 10x'd their team's output with AI coding agents, I find myself asking: is this a real person sharing a real experience? Is it a paid placement? Is it content generated by the very tools being promoted?

I have no way to tell. Neither do you.

And it's getting worse, not better. The most qualified people to evaluate these tools honestly, the ones with enough experience to have real judgment, are also the busiest. They don't have time to write takes. Which leaves a lot of space for everyone else: the shiny-object adopters who are genuinely excited, the vendors with obvious incentives, and an increasingly murky middle ground of content that looks like an opinion but might be something else entirely. The financial relationship between a writer and the tools they're praising is almost never disclosed. And now the tools themselves can generate content praising the tools. Think about that for a second.

I'm not making accusations. I'm describing a problem that I think we have a collective responsibility to sit with rather than just nodding along. The appropriate response to an information environment you can't fully trust isn't paralysis. It's going and finding out for yourself.

So that's what I did.

Why I finally got off the fence

I've been watching this space with skepticism for a while. Being a cynic about new technology doesn't mean I think it's worthless. It means I've been around long enough to know that being an early adopter has bitten me in the ass more times than I care to count. I've watched programming languages rise to prominence and quietly fade. I've seen frameworks become religious movements and then become legacy problems. I hold new things loosely until I have my own data.

What eventually pushed me off the fence was something specific: Anthropic releasing Claude Code as a proper agentic coding environment. Not a chat window you paste code into. Not a side panel in your editor with access to a single file. Something with access to the entire codebase, something that could chain connections across files on its own. That felt meaningfully different from what had come before.

I was also surrounded by colleagues who were already deep in it, which gave me both the social pressure and the budget to stop waiting. I sat down, installed the tools, and started running experiments. Not to validate the hype. To find out where it actually held up and where it fell apart.

Here's what I found.

The thing I needed built

I've kept a metrics tracking spreadsheet for years, a way to stay close to the health of my engineering organization. Sprint data, defect trends, the signals that tell you whether a team is healthy or just busy. It was held together with Python scripts and manual entry, and it took me about 30 minutes a week to update. I liked that 30 minutes, honestly. The manual process kept me close to the data in a way that was valuable. But I was in a time crunch, and I decided to find out whether I could get that time back.

I sat down with Claude Code and started what people are calling "vibe coding," essentially describing what I wanted in plain language and letting the agent build it.

Within eight hours, I had something usable. I want to sit with that for a second, because it is genuinely remarkable. Eight hours to a working application that automated a workflow I'd been doing manually for years. If that's all I had to report, I'd be writing a very different article.

But I kept going. I iterated. I added features. And that's when things got interesting.

Context rot is real. The further we got from the beginning of the session, the more the agent seemed to forget guidance I'd already given it. Preferences I'd established. Constraints I'd set. It would drift back toward doing whatever it wanted to do, as if the earlier instructions had simply evaporated. For a personal project, annoying. For an enterprise codebase with established standards, that's a serious problem.

It deleted features without telling me. As we were iterating, the agent would occasionally decide that something we'd already built wasn't necessary anymore and quietly remove it. I only caught it because I was doing my own QA pass through the session output. When I asked about it, there was no good explanation. It just made a call. On its own. Without asking.

It made bizarre judgment calls. There were moments where it felt like the agent was trying to impress me, pushing ahead of the requirements, making decisions that weren't asked for, showing off what it could do. I've landed on an analogy for this: it behaves like a young child trying to get a parent's approval. Eager to demonstrate capability. Acting out of bounds not out of malice but out of a kind of nascent, unformed ambition. That's not an insult. It's an observation about where the technology is in its development. Young. Early. Still figuring out the edges of what it should and shouldn't do.

For a personal project, I can absorb that. For production software at any meaningful scale, I can't.

The thing I built for fun

My wife and I have a recurring problem: we can never decide what to watch. So I built an app. A simple Android application that pulls TV show data from an API, stores a list in a database, uses Google Auth for login, and randomly selects what we're watching tonight.

Claude built it in about an hour.

It took another eight hours to get it to actually work.

That distinction matters more than it might seem. "Built" and "works" turned out to be very different things. After that first hour, Claude was confident. "We're done. Here's how to get it running." I'd follow the steps. I'd hit an error. I'd report back. And to Claude's credit, it never got defensive. Every time I said "this doesn't actually run," the response was some version of "you're right, there's a problem, let me fix it." No gaslighting me about whether the error was real.

But we did this over and over. Configuration problems. Runtime errors that didn't surface until I was actually trying to install the thing on a phone. It felt less like collaboration and more like QA-ing a junior developer who is genuinely trying but keeps missing things they should have caught.

The app works. We use it. But the gap between "Claude thinks it's done" and "it is actually done" was significant, and I don't think that gap should be hand-waved away.

The litmus test I've been running for years

Before any of this, before the current hype cycle, I've had my own informal benchmark for evaluating AI coding capability. Every three to six months, I grab a complex chunk of code from whatever I'm working on. Hundreds of lines. A god method. The kind of function that has grown over years into something nobody wants to touch, but that you can look at and see, clearly, how it could be broken apart.

I hand it to the AI and say: refactor this.

No specific instructions. No hints. Just: here's a mess, make it better.

Refactoring is actually a well-understood problem. There are established patterns, canonical books, clear inputs and outputs. It's mechanical enough that you'd think a sufficiently capable AI should be able to at least produce something that compiles. For years, what I got back was a complete rewrite, and the more complex the code I gave it, the less likely the output was to actually run. Which is a striking failure mode for something that is supposedly fluent in code.

Within the last six or seven months, something changed. Claude stopped trying to rewrite everything and started making selective edits. Targeted changes. It began to look less like a student who didn't read the assignment and more like someone who actually understood what refactoring means.

It's not there yet. I still haven't gotten back something I'd call production-ready from this test. But the direction of travel is real, and that matters.

Where this leaves me

I want to be honest about something before I close.

I've described real limitations. I've been as critical as I know how to be. But I also built two applications I use, and I'm planning to bring these tools into my next organization. That's not a fully negative take, and I'm aware of the irony: I started this piece by asking whether the positive AI content you're reading is trustworthy, and I'm ending it with some positive observations of my own.

I don't have a clean resolution to offer. What I have is this: I ran my own experiments instead of taking someone else's word for it, and I'm telling you exactly what I found, including the parts that didn't work. Whether that's enough to trust is a judgment call you'll have to make yourself.

The discourse isn't going to get more honest on its own. The incentives all point the other way. So run your own experiments. Stay skeptical of the output, including this one.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

Priority Inversion: The Bug Triage Failure Nobody Talks About

Raleigh Schickel — Tue, 16 Jun 2026 14:28:14 +0000

Why your team is probably working on the wrong bugs, and why they don't know it.

One of the first things I do when I walk into a new engineering organization is look at the bug queue. Not because I'm a bug-metrics person (I'm not) but because bug queues tell you things about organizational health that sprint data doesn't. The queue is where pressure from the outside world meets a team's internal capacity to respond, and how that collision gets managed says a lot.

There's a reason I care about this beyond Engineering hygiene. As the top-level Engineering leader in most organizations, I own quality. Where there's a Quality team, it typically answers to me. That creates a real tension: I'm trying to care for the people building the product while simultaneously reasoning about what the product's current health is costing the business. How much friction is the defect backlog creating? What level of investment do we need to make to get ahead of it? Those questions require data, and data requires visibility. Without visibility into what's actually broken, I'm flying blind on both sides of that equation.

In a recent role, I pulled up the Jira board and immediately noticed something odd. Every single bug in the system had been created by an engineer. Not a single ticket sourced from Customer Support. Not a product manager, not a customer, not a support request. Just engineers, all the way down.

That's unusual enough to raise a question: is this all the bugs? Or are these the bugs that engineers happened to decide were worth writing down?

The answer, as I started poking around, was the second one.

The bugs weren't in Jira. They were in Slack.

There was a channel where Engineering and Customer Support were collaborating on customer-reported issues. In real time, constantly. When a customer hit something, a Customer Support rep would drop it in Slack. An engineer who happened to be paying attention might pick it up. If they chose to work it, they might create a Jira ticket. If they didn't, the issue would just sort of evaporate. Sit there unanswered or get resolved via a one-off Slack message, no record anywhere.

The result was a defect system that was, at best, a partial view of reality. The real defect queue lived in a Slack channel that nobody had formal ownership of, nobody was triaging, and nobody was tracking. The Jira board was a collection of bugs that engineers had self-selected for visibility. Everything else was in the shadows.

This is the part that usually gets skipped when people talk about bug triage failures. Everyone talks about priority inversion, P2s getting worked while P0s sit. The more fundamental problem is work that isn't captured at all. You can't invert priorities you don't know exist. And you can't be accountable for doing the right things when everything is hidden.

Recency bias is not a triage system

Once I understood that the real queue was in Slack, the pattern started making sense. What passed for prioritization in this org was almost entirely driven by recency. The most recent thing got worked on. Anything that was a day or two old was out of sight, and without active visibility, out of mind. Engineers were constantly firefighting whatever fire had started most recently, not the biggest or most important one.

Customer Support had noticed this from their side. Their experience was blunt: if something didn't get picked up immediately, it never got picked up. An issue that didn't get a response in 24 hours was effectively dead. They'd learned to re-surface things just to get attention, which added noise to the channel and trained engineers to respond to volume rather than urgency.

For their part, engineers weren't being malicious or even particularly negligent. They were underwater. Full-time fire mode. When you're just trying to keep up with the incoming rate of issues, you work what's in front of you. The recency bias wasn't a choice, it was the default behavior of a system with no prioritization structure and a constant stream of new inputs.

The inversion was real, but it was a symptom. The root cause was invisible work and the absence of any mechanism to assign priority before picking things up.

The fix nobody asked for: don't blow up their workflow

My first instinct in situations like this is usually wrong. Or rather: the instinct is right, but the implementation is off. The right answer involves a unified system, priority labels that mean something, and a real triage process, but all of that is correct in the abstract and completely useless if nobody uses it because it's too much overhead on top of a job that's already overwhelming. If your solution requires people to fundamentally change how they work every day, you're going to get justified resistance.

I don't have a playbook for these situations. I have a toolbox. And the first thing I reach for is: where do people actually work, and how do I fit the solution around that instead of expecting them to fit themselves around the solution?

Customer Support wasn't going to use Jira. They had their own tools, their own workflow, and asking them to learn another system to report bugs was adding friction to people who were already stressed. Engineering wasn't going to abandon their board. The Jira setup was messy, but it was theirs.

The solution that actually worked started with capturing the work without changing how it got reported. I built a Jira form linked directly in Slack. Customer Support fills out a form in the channel they're already in, a ticket gets created automatically, and a Zapier integration posts the ticket back into the thread. The Slack conversation continues, but now it's anchored to a Jira ticket, and every comment in that thread gets synced back to the ticket. No new tool, no context switch, no asking underwater people to adopt overhead they don't have time for.

The second piece was giving everyone a single view into Customer Support-sourced work without disturbing Engineering's board. I built a separate dashboard that surfaced everything Customer Support had reported, regardless of where it sat in Jira, sorted top to bottom by priority. Customer Support had a single place to check status. I had a single place to triage and set expectations. Engineering could see what was actually out there and pick work that mattered rather than work that was recent.

None of it worked until we dealt with the vocabulary problem. The severity labels in Jira existed, but nobody was using a consistent definition. "P0" meant different things to different people. Before any of the tooling mattered, I needed Customer Support and Engineering aligned on what each level meant and what the associated expectation was. How fast should a P0 be acknowledged? How fast should it be resolved? That shared vocabulary is the foundation everything else sits on, and it's almost always the thing that gets skipped.

Data as a forcing function for accountability

Once the workflow was running and there was a few weeks of actual data, something important became possible that hadn't been before: accountability.

This is what I was actually after. Getting everything into the system and wrapping data around it wasn't an end in itself. It was the prerequisite for being able to look at the team, and have the team look at itself, and ask whether we were doing the right things. You cannot hold anyone accountable, including yourself, for something that's invisible.

I didn't have to tell anyone they were doing it wrong. I just showed them the numbers. We're creating this many P0s per week. We're resolving this many. The backlog is growing. Here's a P0 that's been open for eleven days.

That last one gets attention. An eleven-day-old P0 creates two useful conversations. The first is with Customer Support: was this actually a P0, or did we mislabel it? Maybe it was really a P2 and the label is inflating the severity. The second is with Engineering: if this really is a P0, why hasn't it been touched? That conversation is the triage mechanism working the way it's supposed to, as an organizational sanity check, not as a performance review.

The goal in these conversations was never to assign blame. It was to get to a shared understanding of what we owed our customers and whether we were delivering on it.

Managing up and across

There's a third thing visibility buys you that doesn't get talked about enough: the ability to protect your team from the rest of the organization.

At this company, there was a weekly operations meeting. It was a small company, so the room included the CEO, the CFO, a couple of people from Customer Support, and a couple from Engineering and Product. We talked about running the business. And without fail, quality came up. Every week. People would drum on specific things that weren't getting fixed, express frustration about the pace of resolution, and I had essentially nothing to say back. The team was working, but I had no information. I couldn't tell anyone what was getting fixed, how much labor we were spending on defects, or what it would actually cost to prioritize the things they were complaining about. I was walking into that meeting empty-handed every week.

Once the system was in place, that changed. I now had data showing how much of the team's capacity was going toward defect repair. I had a ranked list of what was being worked and what wasn't. And when someone in that meeting pointed at a specific bug and asked why it wasn't fixed yet, I could answer the question: it didn't rise to the level of the things we were already working on. Here's what's above it. What would you like us to deprioritize to address it?

That reframe matters. It shifts the conversation from "Engineering isn't fixing things" to "Here are the tradeoffs, and you're now part of making them." It builds trust with the executive team because they can see the work and the reasoning. And it protects the Engineering team from being blamed for prioritization decisions that were never really theirs to begin with. Without a structured system, those decisions were just defaulting to recency, and nobody had agreed to that.

The retention argument

There's a version of this story where I make the customer relationship case: unresolved high-severity bugs damage trust, damaged trust puts revenue at risk, and so on. That's all true. But the argument I find myself making more often is about your people.

Engineers who spend their careers in full-time fire mode turn over. Not always immediately, and sometimes not loudly, but they do. The engineers who have options will eventually go somewhere where the work feels less futile. A mark of a healthy Engineering organization is limited turnover, and high turnover is frequently a signal that the environment is too chaotic to feel sustainable.

The path out of fire mode runs through prioritization. And prioritization requires visibility. The accountability structures I put in place weren't only about product health or customer satisfaction, they were about creating an environment where the team could do their best work instead of constantly reacting to whatever came in last.

Priority inversion is the thing people name. But the thing underneath it is almost always an information problem. The team isn't making bad choices. They're making default choices in the absence of structure, and default choices in an interrupt-driven environment trend toward whatever was most recently on fire. Make the work visible, agree on what it means, and the data will make the difficult conversations easier.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.

How I Assess Engineering Health in My First Week at a New Org

Raleigh Schickel — Mon, 15 Jun 2026 19:17:29 +0000

The diagnostic framework I run every time I join a new engineering organization, and why the data is only half the picture.

The first week at a new company is a lot. You're meeting strangers, trying to read rooms you've never been in, figuring out who the actual decision-makers are and how decisions get made. For me, as an introvert, there's also the sheer exhaustion of performing extroversion for eight hours a day while trying to simultaneously form a coherent picture of a place I've never seen before.

And everyone is watching. The people above you want to know if they made the right hire. The people who report to you are wondering whether their jobs are safe, whether you're going to blow up the way they've been doing things, whether you're someone who actually gives a damn or just another executive who issues mandates from a conference room. You try not to let that pressure paralyze you. But you feel it.

The thing that helps me most is having a framework. Not because frameworks give you answers, but because a good framework gives you the right questions to ask. The two main questions I'm trying to answer in week one each open into several more questions, and the answers to those won't come from the data. They'll come from the team. The data just tells me where to look and what to ask about.

Here's how I do it.

The Two Questions

Everything I'm trying to learn in the first week boils down to two things: what is the state of quality, and how healthy are the individual Product Engineering teams?

Those sound related, and they are, but they're measuring different things. Quality is about the product: How much defect debt has accumulated? How are fixes being prioritized? Are bugs being worked on in any coherent order? Team health is about execution: Can these teams actually deliver what they say they're going to deliver, and if not, what's getting in their way?

I use both because either one alone tells an incomplete story. A team executing beautifully might be doing so while the product quietly burns down around them. A product that's relatively stable might be maintained by a team that's one bad quarter away from collapse. You need both pictures.

First, the Tooling

Before I run a single calculation, I spend time in whatever ticketing system the org is using and I just look around. How many projects are there relative to the number of teams? How are bugs being categorized? Are there priority labels, and does anyone seem to be using them consistently?

This sounds mundane, but it tells you something before any numbers do. An org that has carefully structured its tooling to reflect how it actually works is a different kind of organization than one that crammed everything into a single project and built seventeen custom dashboards to compensate. You can see the history of decisions people made and abandoned just by looking at the archaeology of the project structure.

One thing I've formed a strong opinion on over the years: severity and priority should not both be in play for bug classification. I've seen orgs use both, and what you end up with is a combinatorics problem. High severity, low priority. High priority, low severity. It becomes harder to reason about, not easier. I collapse those into a single priority axis (urgent/P0, high/P1, medium/P2, low/P3) and work from there.

Assessing Quality

For quality, I'm looking at week-over-week trends. How many bugs are open total? How many were opened this week? How many were resolved? What's the net delta? Then I break all of that down by priority. How many P0s are open? P1s? What's changing week to week?

The parallel question, and this one matters a lot, is which priorities are actually getting worked. It's a remarkably common pattern that organizations fix the wrong things. Lower-priority issues get closed out while urgent and high-priority bugs sit open for weeks or months. Nobody made a decision to do that. It just happens, because individuals are making judgment calls without any system enforcing priority order. Nobody is managing the queue.

When I see that pattern, it tells me several things: there's no triage process, individual engineers are deciding on their own what to pick up, and nobody in leadership has visibility on what's actually being worked on. But those are all symptoms of the same root problem: Nobody is accountable for ensuring the most important things get done. That role exists in every healthy Product Engineering org, and when it's vacant, you get a queue that sorts itself by whatever is easiest or most interesting, not by what actually matters. And it doesn't stay contained to Product Engineering. Customer Support is responsible for managing unhappy customers, and when those customers' complaints aren't being addressed, Support can't do their job. Sales runs into the same wall. Eventually it trickles up to executive leadership, and what started as a prioritization gap becomes a trust problem. The Product Engineering org loses credibility with the rest of the company. Everything else you see in the data flows from that gap.

I also pay attention to where bugs are coming from. If almost everything in the ticketing system was filed by engineers, that's strange. You'd expect support teams, QA, or customers to be the primary sources. When engineers are filing most of the bugs, it often means there's a shadow process somewhere. Bugs are being reported through some other channel: a spreadsheet someone built, a shared inbox, the customer support tool that Support and Sales are living in, or an entirely separate ticketing system that another department decided to stand up on their own. Only occasionally does any of it make it into the official system, and only when an engineer happens to create a ticket. That shadow system is worth finding, because it's where the real backlog lives.

Assessing Team Health

For individual Product Engineering teams, I'm looking sprint over sprint rather than week over week. The core metric I care about most is predictability: how much work did the team commit to at the start of the sprint, and how much did they actually complete?

A healthy team delivers somewhere between 85% and 115% of what they committed to. Too far under means they don't have a grip on their velocity, or work is being added mid-sprint and eating their capacity. Too far over usually means they're padding estimates, or the commitment target is so conservative that any reasonable sprint will sail past it. Both directions are worth investigating.

Predictability is also genuinely hard to game, which I appreciate. When deployment frequency became a headline DORA metric, some teams just started deploying more often without the underlying reliability improvements the metric was supposed to represent. Predictability is a relationship between two numbers (commitment and completion) and manipulating that relationship requires coordination across the whole team in ways that tend to fall apart. It's not a perfect metric, but it's more honest than most.

From there, I want to know how much of the completed work was planned at the start of the sprint versus added mid-sprint. Unplanned work is not inherently bad. Sometimes a critical bug lands and you deal with it. But the ratio matters. A team where 40% of completed work was unplanned has a disruption problem, and that disruption is probably coming from somewhere specific. Maybe it's customer escalations. Maybe it's an executive who keeps dropping things on the team. Maybe it's a quality problem generating constant reactive work. The metric points you toward the right questions.

I also track points added to and removed from sprints after they begin. This one reveals something about organizational culture that tends to be uncomfortable to surface: most places love to add work mid-sprint and hate to remove it. The implicit assumption is that the team just absorbs whatever gets added. But teams have a fixed capacity and prioritization is a hard choice. Work added without anything being removed rolls over to the next sprint, affects how much new work can be taken on, and compounds from there. Watching that pattern over several sprints tells you whether the organization actually respects the team's capacity or just treats sprint planning as a rough suggestion.

I look at velocity as a rolling three-sprint average rather than a single-sprint number. Any given sprint's velocity is noisy. Someone's sick, there's a holiday, scope changed late. The rolling average smooths that out and gives you a more honest baseline to plan against.

Finally, cycle time. I want to know how long work is spending in each phase of the delivery process: in progress, in review, waiting to deploy. The goal isn't to make all the numbers small. It's to find the bottlenecks. If code review time is twice as long as implementation time, something is creating friction in review. Maybe there are too few reviewers. Maybe the PRs are too large. Maybe one person is doing all the reviews and they're buried. The metric doesn't tell you the cause, but it tells you where to look.

The Organizational View

Once I have the individual team picture, I pull it up to the org level. I plot all the teams' metrics on the same charts, which makes outliers obvious in a way they wouldn't be if you were looking at each team in isolation.

At the org level, the two numbers I talk about publicly are organizational predictability (the average across all teams) and organizational percentage of planned work completed. The second one is worth raising openly because it sends a signal to the whole company: if 40% of what we delivered last sprint was unplanned work, that's a disruption problem that affects everyone, not just Product Engineering. Planned quality improvements get deprioritized. Promised bug fixes slip, degrading customer trust. Predicted delivery dates become unreliable. And when in-progress projects fall behind, other departments absorb the cost too. Marketing may have already invested weeks preparing launch content for a feature that's now delayed, time they could have spent on higher-value work had they known sooner. Left unchecked, it derails the roadmap entirely and can put revenue plans at risk. That's a useful conversation to have in the open.

Individual team performance is a different matter. Cross-team metrics are useful for coaching, but not for broadcasting. Putting comparative performance data in front of the whole company without context creates exactly the wrong dynamic: teams start optimizing for how they look rather than how they're actually doing. The right venue for this is the sprint review, and the right approach is to open it by sharing what the metrics are showing and asking the team what they think. What do you see here? Is there something we should be focusing on improving? Start with questions, not conclusions. That framing turns the data into a conversation starter rather than a verdict.

You can't walk into a room with bad graphs and tell a team they're failing. Or, technically, you can, but you won't be there much longer if you do. My preferred approach is Socratic. I share what I'm seeing and ask what the team thinks is happening. "Your cycle time in review is notably higher than in progress. What's your sense of why that is?" They usually know. They've been living with it. What they didn't have was someone asking the question and treating the answer as worth addressing.

That matters for a reason beyond just being diplomatic about it. If I solve every problem myself, I've made myself indispensable, and that's exactly the wrong outcome. The goal is to teach people to read these signals themselves, to build the habit of asking these questions every sprint, to create a team that can self-correct without me in the room. The sprint review, done right, is where that habit forms.

The Data Gets You to the Door

I've refined this diagnostic approach across a lot of organizations over a lot of years, and at this point I can build the spreadsheet in my sleep. But I want to be clear about what it actually is: a tool. The methodology gives me the right questions to ask. It gives me almost no answers. The answers come from experience, from having seen the same patterns play out enough times that when the data points in a certain direction, I have a pretty good idea of what I'm looking at. But recognizing a pattern and knowing exactly how to address it in this organization, with these specific people, inside this particular culture, are two very different things. The first tool I reach for might not be the right one here. That part still requires doing the work.

None of this touches the technical side of the house directly. Deployment pipelines, production infrastructure, development environments, architectural decisions: those are also part of the role, and eventually they become a significant part of it. But trying to assess all of that in week one is the wrong sequence. The framework tends to surface the technical problems anyway. When cycle time in the deployment column is consistently an outlier, that's usually an infrastructure conversation waiting to happen. When unplanned work keeps spiking, it often traces back to architectural debt generating a steady stream of reactive work. When defect volume is high and resolution is slow, sometimes the answer is process, but sometimes the answer is that the system is just hard to work in. The data points you toward those doors. Then you go look.

The methodology is the easy part. The hard part is figuring out which conversations to have, in what order, with people you've known for less than a week. You have a picture of what's broken, but no relationships yet, no trust yet, and no real sense of who in the room is ready to hear what. Coaching someone toward a better way of working looks completely different from lecturing them about what they're doing wrong, and the line between those two things is mostly about whether the person on the other end of the conversation feels supported versus scrutinized.

If the first thing people feel when you walk in is that they're being watched and judged, you've already failed, regardless of how accurate your metrics are. If you walk in with genuine curiosity, about the work and about the people doing it, and you back that curiosity up with data, you have a shot at actually changing something.

That's the whole thing, really. The rest is just spreadsheets.

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search "Engineering Health" to find it.