DEV Community: Neil Giarratana

Measuring AI's Impact on Engineering Teams: The Metrics Already Exist

Neil Giarratana — Tue, 17 Mar 2026 14:15:00 +0000

There's a question that keeps coming up in engineering leadership circles: how do we measure whether AI is actually helping?

It's a fair question. Most of us are investing real time and money into AI tooling for our teams and at some point we need to understand what we're getting for it. The challenge is that the most visible metrics...PR counts, commits, code suggestions accepted...are measuring activity, not outcomes.

We've been down this road before. There was a time when lines of code was the go-to measure of engineer productivity, and it was a terrible idea. People padded their stats and quality suffered. Same thing happened with test coverage. Once it became the goal instead of a signal, teams chased the number instead of what we actually wanted which was confidence in our code.

The 2025 DORA Report puts some real numbers to this. AI helps developers complete 21% more tasks and merge 98% more pull requests...but organizational delivery metrics stayed flat.

More activity, same outcomes.

The report also notes that AI "magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones." It's an amplifier, not a fix.

So if activity metrics aren't the answer, what is?

I keep coming back to DORA. For those unfamiliar, DORA (DevOps Research and Assessment) is a research program, now part of Google Cloud, that has studied software delivery performance across thousands of organizations. Their findings, published in the book Accelerate and in annual State of DevOps reports, identified four key metrics that correlate with high-performing engineering organizations.

What's interesting about these metrics is that none of them measure activity. They measure outcomes.

Lead time is the time from when an idea is conceived to when it's in production and in front of customers. Pragmatically, most teams measure this as first commit to first production deployment...which doesn't tell the whole story, but sometimes you have to start with what you can actually measure and refine from there.

This is where AI's leverage really shows because if we can compress that loop, we're not just shipping faster...we're learning faster. We're getting real data about whether we built the right thing, and we're able to iterate on it sooner.

That feedback loop is everything.

Deployment frequency tells us how often our teams are releasing to production. More frequent, smaller deployments are a signal of a healthy delivery pipeline and a team that's iterating confidently.

With AI in the mix, we should see this go up...not because we're pushing more code, but because we're moving through the development cycle with less friction.

Change failure rate is the percentage of deployments that result in a failure requiring remediation. This one should go down with AI since it can help catch bugs, flag regressions, and improve code quality before things hit production.

If our change failure rate isn't trending in the right direction, that's a signal worth paying attention to because it might mean we're shipping faster without the guardrails to match.

Mean time to resolution is traditionally about how quickly we recover from incidents. But I think this metric gets more interesting when we broaden it a bit.

Most teams have low-priority security findings they can't get to, edge-case bugs that keep slipping down the backlog, tech debt that never wins the prioritization fight. AI gives us the horsepower to finally tackle that work.

Think about a backlog of tech debt that's been piling up for months because there was always a feature or a bug that took priority...that's the kind of work AI can help burn down. It's not just that MTTR improves; it's that the scope of what we can resolve expands.

Now here's the thing that's always been true about DORA and is worth restating: any one of these metrics in isolation can be misleading. We can deploy like crazy and ship nothing of value. We can have a great change failure rate because we're barely deploying at all.

It's the composition of these signals together that tells the real story. A team that's deploying frequently, with short lead times, low failure rates, and fast recovery...that's a team that's delivering well. And that picture doesn't change just because AI entered the equation.

The piece I'd add on top of DORA is product metrics. At the end of the day, don't tell me how many PRs went up; tell me whether the change actually mattered to the people that use it. Did the feature hit its goals? Did users behave differently? Did we move a number that the business cares about?

DORA tells us whether we're delivering well. Product metrics tell us whether we're delivering the right thing. Together, they give us the full picture.

None of this is particularly new thinking. DORA has been around for years and the Accelerate book laid out these principles back in 2018. What's new is that AI is tempting us to forget them in favor of shinier, easier-to-measure activity metrics.

The tools for measuring what actually matters are already there. We just have to use them.

Building a Leadership Development Program, step-by-step

Neil Giarratana — Wed, 01 Nov 2023 14:28:53 +0000

Transforming an engineering organization starts with changing hearts and minds which sounds really simple...it's not.

While it'd be great to simply reveal a treasure map and proceed methodically to the treasure, the truth is that people don't invest themselves in maps all that well and the map is different for every organization.

To get to the treasure, we must take the journey together and define the path as we go, one step at a time...1% better every day.

So without a map, where do we start?

Here is what's worked for me as a foundational program that provides a shared vocabulary, leads us to discover trends in the industry, and is an easy vehicle to discuss what we believe collectively.

Getting started…

If you’re starting a new program from scratch, start with a single group (aka cohort)…maybe your leadership team to get a feel for the program.

Start with “why?”

Here’s an example of my why’s:

We have a deep desire to learn and grow as an organization
We have varying levels of knowledge and often misunderstand each other based on vocabulary
Everyone agrees we need to move faster with less friction but we also want to be safe and raise the bar on quality

Grounding ourselves in a common vocabulary and discovering best practices in our industry so that we can discuss and decide what we value together will help us achieve our goals.

We start the group with two books, Accelerate and Implementing Lean Software Development. See the syllabus below for an exact template schedule.

Week to week…

📖 Every week, we read a set of assigned chapters, typically < 30 pages.

💬 We meet on the same day and time each week to discuss what we read, how it applies to us (or doesn’t) and what it would take to apply what we liked.

Although the discussion is based on what we read that week, the secret sauce is that we dynamically decide the specific topics and how long to spend on them using a format modeled on Lean Coffee. Here’s how:

Brainstorm Topics - We start the meeting with a 3 minute brainstorm. Each person puts sticky notes on the board with proposed topics that were interesting to them from the book.
Group Topics - We spend a minute reading the sticky notes and grouping like things together.
Vote! We spend another minute voting on the topics that interest us. Each person gets three votes.
Start talking - Set a timer for 8 minutes and start discussing the highest voted topic.
Check-in After 8 minutes, we do a thumbs up, neutral, or thumbs down visual vote to decide if we should keep talking about that topic or move on to the next.

👍 If we’re mostly thumbs up, we’ll put 4 minutes on the timer and keep going.

👎 If we’re mostly thumbs down, we’ll put 8 minutes on the timer again and start the next topic

If everyone is neutral, then no one cares enough to keep discussing and we move on to the next topic

We spend the hour working our way through the prioritized topic discussions, capturing epiphanies and action items on separate stickies. Don’t worry if you don’t get through everything, much like in our work we must prioritize. At the end of the hour, we summarize and assign any actions that came out.

👩‍💻 What about remote teams?

All of my teams are remote so I feel you. My favorite tool for this is Figjam but Miro is nearly as good. Everyone connects to a virtual board and puts stickies on the board. The software includes a timer and even voting tools that are easy to use and visual for everyone. Figjam is one of the best tools available for getting remote team member to actively participate in discussions, brainstorming, etc.

Now what?

Ok. We started our leadership group, read the first two books and we’re on board with one another with organizational change. Let’s get started with implementing.

Hold on! Cultural and organizational change requires doing things together, not dictating from above. The next step here is to begin to expand the program.

We have two options:

Each leader replicates the program in their own domain. Note that each group is free to discuss the topics from the two books that interest them most and they can come to their own conclusions. (You’d be amazed at how often the conclusions are similar, if not the same).
Begin forming blended, cross-domain cohorts where there’s a person or two from each leader’s domain. This promotes breaking down silos, creating new relationships and forming holistic organizational values.

Either way, start slow…don’t feel like 100% of the org needs to be in day one. You can grow the program over time, starting new cohorts every quarter.

What do we do at the end of the two books? Is that it?

Nope, after a cohort completes the first two books, they choose their own adventure, voting on what book should be next. We provide a list of known good books for them to choose from or they can insert their own.

By each group choosing its own books after the first two, we keep it fresh and continually insert fresh ideas. We also try to keep like roles together, e.g. managers in one cohort and individual contributors in another. The managers might choose a leadership book like Radical Candor while the IC group might choose something like Domain Driven Design for their next book.

Our Syllabus

Here's a template for our first two books and the cadence with which we read them:

Book 1 - Accelerate

Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations

by Nicole Forsgren PhD, Jez Humble, Gene Kim

"We strongly recommend this book to anyone involved in a digital transformation for solid guidance about what works, what doesn't work, and what doesn't matter."
-Tom and Mary Poppendieck, authors of the Lean Software Development Series

Paperback | Kindle | Audible

Week 1 - Measuring Unit (50 pages)

Preface
Ch 1 - Accelerate
Ch 2 - Measuring Performance
Ch 3 - Measuring & Changing Culture

Week 2 - Technical Unit (32 pages)

Ch 4 - Technical Practices
Ch 5 - Architecture
Ch 6 - Integrating Infosec Into the Delivery Cycle

Week 3 - Lean Software Process (27 pages)

Ch 7 - Management Practices for Software
Ch 8 - Product Development
Ch 9 - Making Work Sustainable

Week 4 - Leadership (28 pages)

Ch 10 - Employee Satisfaction, Identity, and Engagement
Ch 11 - Leaders and Managers

Book 2 - Implementing Lean Software Development

Implementing Lean Software Development: From Concept to Cash

by Mary and Tom Poppendieck

"This remarkable book combines practical advice, ready-to-use techniques, anda deep understanding of why this is the right way to develop software. I haveseen software teams transformed by the ideas in this book."

Mike Cohn, author of Agile Estimating and Planning

Paperback

Week 5 - History & Principles (42 pages)

Ch 1 - History
Ch 2 - Principles

Week 6 - Value (22 pages)

Ch 3 - Value

Week 7 - Waste (28 pages)

Ch 4 - Waste

Week 8 - Speed (22 pages)

Ch 5 - Speed

Week 10 - People (32 pages)

Ch 6 - People

Week 11 - Knowledge (28 pages)

Ch 7 - Knowledge

Week 12 - Quality (30 pages)

Ch 8 - Quality

Week 13 - Journey (24 pages)

(Skip Ch 9 - Partners - outdated)
Ch 10 - Journey

Frequently Asked Questions

How much time will this take?

Most people read about 30 pages / hour for non-fiction so I would recommend budgeting 90 minutes for most weeks to include note-taking, re-reading, etc. + 60 minutes for the meeting.

How should I prioritize this?

Simple answer: Unless the house is burning down, you should be there. 😉

The participants of your group are a cohort that depend on each other and, therefore, this should be prioritized above most other things. If there is a meeting that needs someone from your team (most often you), it’s a good time to delegate to a team member and follow up after.

Does this end after this book?

Nope. This is an ongoing series. After the first few books of the curriculum that we use as foundational, we’ll select books as a group to continue our development together.

Am I required to read on my own time?

Nope. This is part of your professional development here so please create an appointment on your calendar for reading time. (Some people do the audiobook or read while at the gym so there’s nothing forbidding you from reading outside of work hours of course).

🙌 I'd be remiss if I didn't give a shout out to Braedon McCoy and Bill Kinnard. They started a program, Leadership Development Kit, within Asurion upon which this program builds. Using book readings to start discussions is a core part of the program.

I'd run technical book clubs for years but their approach to targeted learning and discussion was instrumental to LDP.

Key aspects that I've added include:

Prescribing only the first two books and letting the group choose their own adventure for future books
Running the meeting in a Lean Coffee format so that the group is engaged in discussion more than reviewing text

Photo by Mitchell Luo on Unsplash

Five tips to become a Pragmatic Programmer

Neil Giarratana — Wed, 21 Dec 2022 21:10:05 +0000

The Pragmatic Programmer, 20th Anniversary Edition your journey to mastery by David Thomas & Andrew Hunt is one of those must-read books for any software engineer's bookshelf. Comprised of eight chapters that each stand on their own, it reads like a mentor with decades of experience is revealing best practices they’ve collected over a career. There are seventy "tips" that detail the characteristics and behaviors of a pragmatic programmer.

Here are five of my favorite tips:

Tip 4: Don’t Live with Broken Windows

Photo by Tara Evans on Unsplash

Don't leave "broken windows" (bad designs, wrong decisions, or poor code) unrepaired. Fix each one as soon as it is discovered. If there is insufficient time to fix it properly, then board it up.

Every engineering team should have the opportunity to improve their code, to leave the campground cleaner. This tip is about the personal responsibility of keeping your code clean and also the leadership responsibility of creating an environment where teams can make things better.

Tip 11: Don’t Repeat Yourself

Photo by Possessed Photography on Unsplash

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system

In today's world of distributed applications spread across an enterprise, it’s easy to fall into the trap of shotgun blasting duplicative (at best) knowledge across the system. Examples can be found in simple implementations of single-page-apps + backend-for-frontend + backend-apis. Three different layers that all have a notion of the domain, e.g. a sales order, but may create individual representations in each.

Tip 14: There are No Final Decisions

Photo by Johann Walter Bantz on Unsplash

The mistake lies in assuming that any decision is cast in stone—and in not preparing for the contingencies that might arise. Instead of carving decisions in stone, think of them more as being written in the sand at the beach. A big wave can come along and wipe them out at any time.

So many of these tips are related to avoiding apathy of software engineers toward the applications they work on. This one is chief among them. With the assumption that things will always change and that we can continually improve on not just our code but everything involved in our work, we actively invest ourselves in seeking that improvement. Our software architecture, our underlying systems, even the way we work as a team are all fair game for incremental or even dramatic changes to make them better.

We design and code differently in that environment. We experiment more freely; we purposely decouple systems and abstract interfaces. We build in more testing because we know we’ll need to ensure stability with future change. In short, we make better software.

Tip 20: Keep Knowledge in Plain Text

Photo by Ken Suarez on Unsplash

As Pragmatic Programmers, our base material isn't wood or iron, it's knowledge. We gather requirements as knowledge, and then express that knowledge in our designs, implementations, tests, and documents. And we believe that the best format for storing knowledge persistently is plain text. With plain text, we give ourselves the ability to manipulate knowledge, both manually and programmatically, using virtually every tool at our disposal.

I’m a bit of a productivity tools geek who likes to play with different apps to write, plan, message, and code. I was bitten many times by tools that became obsolete before I learned my lesson. I kept finding myself in a place where I had to abandon collected knowledge because the binary content was not portable to another tool. Plain text is the key to portability and the output should be, ideally, human readable. Even things like diagrams can be represented in plain text and then parsed to produce visual designs.

It still seems strange that I write most of my posts first in Notion. There are two primary reasons: 1. I find it super convenient to use across all of my devices and 2. I leverage markdown (e.g. using ## to start secondary headings and _word_ to italicize a word) when writing…and it can all be exported simply in markdown.

Tip 55: Don’t Think Outside the Box - Find the Box

Photo by Erda Estremera on Unsplash

You must challenge any preconceived notions and evaluate whether or not they are real, hard-and-fast constraints. It's not whether you think inside the box or outside the box. The problem lies in finding the box—identifying the real constraints.

Particularly in large enterprises, hunting down the “real” requirements often combines researching regulations, tribal knowledge, and rumor. Getting down to brass tacks on what’s an actual requirement vs. what’s “the way we’ve always done it” takes effort and requires a willingness (and environment) to question everything. That said, it’s critically important to creating value for the customer.

Cover photo by Michal Matlon on Unsplash