I understand the wish to find a number for a developer's work, and I agree velocity is the wrong one. But the problem is older than AI. Every developer KPI I've seen broke the same way:
Lines of code? Pay for lines and you'll get lines, whatever the task needs.
Number of bugs? That depends on the whole team, not one developer: how well everyone understood the business need, whether the infrastructure was set up right, UX requirements nobody discussed, a third-party service failing that we can't fix at all.
Number of tests? We can write them endlessly.
So AI didn't create the measurement problem. It made the wrong numbers grow faster.
I like that your three tests are questions, not numbers. Test 3 is actually a rephrasing of a principle every professional developer knows: dependency inversion from Clean Architecture, where business logic doesn't depend on frameworks, databases or vendors. To me that's the best part of your argument. The principles that made code last before AI are the same ones that make it last now. AI just makes it easier to skip them.
To answer your question, the signal I trust most is how long it takes a new person to make a safe change, and whether someone on the team can still explain why the code is the way it is.
That last point is probably the best answer to the question. How long does it take a new person to make a safe change, and can someone explain why the code works this way? I agree the measurement problem existed long before AI. AI just makes it much easier to produce impressive numbers while hiding the actual cost of maintaining what we build.
I think we should put more effort into automating testing and review, because that's where the hidden cost gets caught early. You said AI makes it easy to produce impressive numbers while hiding the maintenance cost. A test or a review check that runs on every change can't be impressed by a demo. I've started asking the agent to write more types of tests, not just unit ones. Golden screen tests, and e2e tests for the MCP server (when tested, it is connected as a client). Maybe that helps in your field too.
Absolutely, especially the point about testing beyond unit tests. Golden screen tests and end-to-end tests for the MCP server can catch issues that unit tests won't, particularly when components work fine in isolation but fail when connected. I think the key is making sure the tests cover real user flows and meaningful failure cases, not just increasing test coverage for its own sake. Automating that feedback loop is probably one of the best ways to keep AI-generated changes maintainable as the codebase grows.
Test 3 is the one I'd push on, because the interface fixes the compile-time half of a swap and not the half that bites. Summarizer.summarize() keeps the same signature across vendors, but output length, tone and what gets dropped all change, and nothing in the type system notices. So the signal I'd trust more than velocity is whether a dependency swap comes with something that can fail: a dozen saved inputs with properties the output must keep, run against both providers before the switch. Without that, a clean boundary mostly makes it easier to swap in a regression.
Fair point. An interface makes the swap easier mechanically, but it doesn't guarantee equivalent behavior. I like the idea of keeping representative inputs and checking what must be preserved across providers. I'd add that to the boundary itself, because otherwise we've only solved half the problem.
One signal I’d track is how often a system needs human intervention to recover from routine failures. A service can have clean architecture and strong test coverage yet still depend on someone manually replaying jobs, clearing queues, or correcting inconsistent state. That operational work often stays invisible in velocity metrics because the system technically remains available. Tracking intervention frequency, recovery time, and recurring failure causes could reveal whether the architecture is genuinely becoming more resilient or simply shifting the maintenance burden onto the team. With AI making implementation cheaper, reducing the amount of human babysitting required in production feels like a much more meaningful measure of engineering progress.
Absolutely, this is a great signal to track. I’ve seen systems that look perfectly healthy on paper but still need someone to keep things moving behind the scenes. That work rarely shows up in the usual metrics, but it’s still engineering effort being spent.
I think that’s where the real difference shows up with AI. Shipping faster is great, but if someone has to spend the next six months babysitting what you shipped, did you really save that much time?
The "year two" framing is the right test. Almost any codebase looks healthy in the first month, and AI assistance mostly compresses that first month further. The bill arrives later, when someone has to change a feature nobody fully understood when it was generated.
In client builds I have found it helps to treat maintainability as an explicit acceptance criterion rather than a hope. If the definition of done for a piece of work includes tests for the critical path, a short note on why the main decisions were made, and a deploy that someone other than the author can run, the speed gains from AI stop being borrowed against the future.
The other thing that helps is keeping each unit of work small and bounded. A single, well defined problem with a clear outcome is much easier to review and hand over than a sprawling feature that grew one prompt at a time. Curious whether you track any metric for this, such as time to first change by a new contributor.
I like the idea of making maintainability part of the definition of done instead of something we hope for later. Small, bounded changes also make a huge difference when AI makes it so easy to generate more than anyone can realistically review. I don't track time to first change by a new contributor yet, but honestly, that's a metric worth trying.
Đúng trọng điểm. Dashboard đẹp mắt (PRs/week, cycle time giảm, prototype nhanh) chỉ đo lường velocity, không đo sustainability. Year two mới bộc lộ chi phí thực: technical debt tích lũy từ code gen không review kỹ, test coverage giả vờ, architecture bị ép vào pattern LLM phù hợp demo chứ không phù hợp domain, và team mất khả năng debug khi black-box model thay đổi behavior.
Thấy nhiều team celebrate "shipped in 2 days" rồi month 6 vẫn đang chase flaky test, refactor spaghetti prompt chain, và explain cho stakeholder tại sao feature "done" lại break khi model update minor version. Engineering maturity là biết khi nào không dùng AI, và xây guardrail cho những chỗ dùng — không phải maximize metric dashboard (site: labagent .tech)
Exactly, especially the part about knowing when not to use AI. Shipping in two days means very little if the next six months go into debugging and patching it. The tool should help us make better engineering decisions, not become an excuse to skip them.
I don't think there's a hard signal for this, but rather a "soft" one. Ask the developers what they created and why it matters to the business. If they can clearly explain what they built, why it exists, and the value it provides, that's probably a pretty good sign.
I like this take. If a developer can explain what they built, why they made those choices, and how it helps the business, that tells you a lot more than a PR count. The real test is whether they can still explain it six months later when something breaks or the requirements change.
The manual fixes per week metric hits home. I've shipped enough small tools to know that "it works" and "I never have to touch it" are very different things. The second one takes more thought upfront.
Exactly. Getting something to work is often the easy part. Getting it to keep working without someone constantly checking on it is a completely different engineering problem. The hidden maintenance hours are what make the real difference.
We're a place where coders share, stay up-to-date and grow their careers.
I understand the wish to find a number for a developer's work, and I agree velocity is the wrong one. But the problem is older than AI. Every developer KPI I've seen broke the same way:
So AI didn't create the measurement problem. It made the wrong numbers grow faster.
I like that your three tests are questions, not numbers. Test 3 is actually a rephrasing of a principle every professional developer knows: dependency inversion from Clean Architecture, where business logic doesn't depend on frameworks, databases or vendors. To me that's the best part of your argument. The principles that made code last before AI are the same ones that make it last now. AI just makes it easier to skip them.
To answer your question, the signal I trust most is how long it takes a new person to make a safe change, and whether someone on the team can still explain why the code is the way it is.
That last point is probably the best answer to the question. How long does it take a new person to make a safe change, and can someone explain why the code works this way? I agree the measurement problem existed long before AI. AI just makes it much easier to produce impressive numbers while hiding the actual cost of maintaining what we build.
I think we should put more effort into automating testing and review, because that's where the hidden cost gets caught early. You said AI makes it easy to produce impressive numbers while hiding the maintenance cost. A test or a review check that runs on every change can't be impressed by a demo. I've started asking the agent to write more types of tests, not just unit ones. Golden screen tests, and e2e tests for the MCP server (when tested, it is connected as a client). Maybe that helps in your field too.
Absolutely, especially the point about testing beyond unit tests. Golden screen tests and end-to-end tests for the MCP server can catch issues that unit tests won't, particularly when components work fine in isolation but fail when connected. I think the key is making sure the tests cover real user flows and meaningful failure cases, not just increasing test coverage for its own sake. Automating that feedback loop is probably one of the best ways to keep AI-generated changes maintainable as the codebase grows.
Test 3 is the one I'd push on, because the interface fixes the compile-time half of a swap and not the half that bites. Summarizer.summarize() keeps the same signature across vendors, but output length, tone and what gets dropped all change, and nothing in the type system notices. So the signal I'd trust more than velocity is whether a dependency swap comes with something that can fail: a dozen saved inputs with properties the output must keep, run against both providers before the switch. Without that, a clean boundary mostly makes it easier to swap in a regression.
Fair point. An interface makes the swap easier mechanically, but it doesn't guarantee equivalent behavior. I like the idea of keeping representative inputs and checking what must be preserved across providers. I'd add that to the boundary itself, because otherwise we've only solved half the problem.
One signal I’d track is how often a system needs human intervention to recover from routine failures. A service can have clean architecture and strong test coverage yet still depend on someone manually replaying jobs, clearing queues, or correcting inconsistent state. That operational work often stays invisible in velocity metrics because the system technically remains available. Tracking intervention frequency, recovery time, and recurring failure causes could reveal whether the architecture is genuinely becoming more resilient or simply shifting the maintenance burden onto the team. With AI making implementation cheaper, reducing the amount of human babysitting required in production feels like a much more meaningful measure of engineering progress.
Absolutely, this is a great signal to track. I’ve seen systems that look perfectly healthy on paper but still need someone to keep things moving behind the scenes. That work rarely shows up in the usual metrics, but it’s still engineering effort being spent.
I think that’s where the real difference shows up with AI. Shipping faster is great, but if someone has to spend the next six months babysitting what you shipped, did you really save that much time?
The "year two" framing is the right test. Almost any codebase looks healthy in the first month, and AI assistance mostly compresses that first month further. The bill arrives later, when someone has to change a feature nobody fully understood when it was generated.
In client builds I have found it helps to treat maintainability as an explicit acceptance criterion rather than a hope. If the definition of done for a piece of work includes tests for the critical path, a short note on why the main decisions were made, and a deploy that someone other than the author can run, the speed gains from AI stop being borrowed against the future.
The other thing that helps is keeping each unit of work small and bounded. A single, well defined problem with a clear outcome is much easier to review and hand over than a sprawling feature that grew one prompt at a time. Curious whether you track any metric for this, such as time to first change by a new contributor.
I like the idea of making maintainability part of the definition of done instead of something we hope for later. Small, bounded changes also make a huge difference when AI makes it so easy to generate more than anyone can realistically review. I don't track time to first change by a new contributor yet, but honestly, that's a metric worth trying.
Đúng trọng điểm. Dashboard đẹp mắt (PRs/week, cycle time giảm, prototype nhanh) chỉ đo lường velocity, không đo sustainability. Year two mới bộc lộ chi phí thực: technical debt tích lũy từ code gen không review kỹ, test coverage giả vờ, architecture bị ép vào pattern LLM phù hợp demo chứ không phù hợp domain, và team mất khả năng debug khi black-box model thay đổi behavior.
Thấy nhiều team celebrate "shipped in 2 days" rồi month 6 vẫn đang chase flaky test, refactor spaghetti prompt chain, và explain cho stakeholder tại sao feature "done" lại break khi model update minor version. Engineering maturity là biết khi nào không dùng AI, và xây guardrail cho những chỗ dùng — không phải maximize metric dashboard (site: labagent .tech)
Exactly, especially the part about knowing when not to use AI. Shipping in two days means very little if the next six months go into debugging and patching it. The tool should help us make better engineering decisions, not become an excuse to skip them.
I don't think there's a hard signal for this, but rather a "soft" one. Ask the developers what they created and why it matters to the business. If they can clearly explain what they built, why it exists, and the value it provides, that's probably a pretty good sign.
I like this take. If a developer can explain what they built, why they made those choices, and how it helps the business, that tells you a lot more than a PR count. The real test is whether they can still explain it six months later when something breaks or the requirements change.
The manual fixes per week metric hits home. I've shipped enough small tools to know that "it works" and "I never have to touch it" are very different things. The second one takes more thought upfront.
Exactly. Getting something to work is often the easy part. Getting it to keep working without someone constantly checking on it is a completely different engineering problem. The hidden maintenance hours are what make the real difference.