DEV Community: Leonid Bugaev

What is the new engineering bottleneck?

Leonid Bugaev — Thu, 07 May 2026 07:05:53 +0000

Something I keep thinking about:

Maybe AI is not exposing a new problem in engineering. Maybe it is exposing an old one that we were already bad at.

We talk about AI like the bottleneck is still writing code. But honestly, writing code has not been the hard part for a long time. The hard part is all the surrounding context.

Why are we making this change? Who asked for it? Which customer depends on the current behavior? Was this weird edge case intentional? Is this a product decision or just an implementation accident? Did security already review something similar before? Is the documentation describing the current behavior or the behavior we wish we had?

And the uncomfortable part is that most of this context is not in one place.

It is all over GitHub, Slack, docs, tests, head of the engineer who left six months ago.

This was already a problem. AI just makes it harder to ignore. Because now we can create more code from less context, more tests, more docs, more confident explanations.

But if the context is incomplete, then all of that output is built on sand. This is the part I find interesting.

Not “will AI replace engineers?” I don’t think that is the most useful question.

The more interesting question is: What happens when engineering teams can generate implementation faster than they can preserve intent?

Because that is where things get messy.

You can have a clean PR. You can have passing tests. You can have updated docs. You can even have a very convincing AI-generated explanation.

And still nobody can answer the basic question: “Is this actually the right change?”

That question is much more expensive than people admit. I have felt this many times in open source and infrastructure work.

You look at a small change and think, “This should be simple.” Then you start pulling the thread. There is a backwards compatibility issue. There is some behavior that looks wrong but someone depends on it. There is a test that protects the implementation but not the real promise. There is a doc page that says one thing and production behavior says another. There is a customer workaround that became part of the product without anyone naming it. Suddenly the small change is not small.

And this is why I think “AI will make everyone ship faster” is only half true. AI can make the creation part faster. But creation is not the same as shipping. Shipping means the organization understands the change well enough to stand behind it.

That is a different problem. I don’t have the perfect answer yet.

But I think “AI coding” is the wrong frame. The real problem is not coding. The real problem is engineering memory.

And most teams’ engineering memory is held together with Slack search, old PR comments, and someone saying: “I think I remember why we did that.”

That does not scale.

Trust Is the Bottleneck

Leonid Bugaev — Thu, 07 May 2026 07:04:16 +0000

Everyone is asking the same question now: if AI can help us create much more code, why aren’t engineering teams suddenly moving much faster?

I think the question is right, but the answer usually stops too early.

AI does make some things dramatically faster. MVPs are faster. Prototypes are faster. The time to validate an idea is reduced a lot. You can explore directions that previously weren’t worth the effort. This is real, and I don’t want to pretend otherwise. But creating the first version of something isn’t the same as maintaining a product, and creating more pull requests isn’t the same as creating more trusted change.

This is where the economics breaks. If your team can create ten times more pull requests, your product doesn’t automatically move ten times faster. Your company doesn’t become ten times faster. The economy doesn’t double or triple. Because the expensive part of mature engineering was never only the typing of code.

The expensive part is trust.

Can I trust this change? Does it match the intent? Does it break a hidden customer flow? Does it affect backwards compatibility? Are the docs updated? Are the tests proving the right thing? Did we think about security, performance, malformed input, error states, release notes, migration, support?

A pull request doesn’t answer all of this.
A pull request is just something asking to be trusted.

So I don’t think the interesting question is “can AI create more code?” It can. The interesting question is: what needs to exist around the code so we can safely absorb more change?

If we can scale trust, we can unlock the real scaling of AI. But not by sending maintainers ten times more PRs. That only moves the bottleneck. What I want is a pull request that comes with enough context that I can actually believe it: why this change exists, what it affects, which tests prove it, which docs changed, what can break, and what still needs a human decision.

AI made implementation cheaper, but trust is still expensive. Join me on journey to find what to trust in new, post AI world.

If that is the kind of engineering problem you care about, subscribe.

If your trust model is green CI, you are in trouble

AI isn’t going back in the box. Even if you personally didn’t join the hype train, people around you probably did. Engineers use it to write code. PMs use it to write specs. Someone asks it to validate the plan, then write the code, then write the tests, then check everything against the same plan again.

I do it too. I ask AI to help me write the spec. Then I ask it to validate the plan. Then I ask it to write the code. Then validate its own code. Then write the tests. Then check everything against the original plan. It’s tempting because it works surprisingly well. In a lot of cases, it feels almost magical.

But this is exactly why it becomes dangerous.

For a long time, our basic engineering trust model was something like this: write the code, write the tests, pass CI/CD, review the pull request, ship. It was never perfect, but it was much better than nothing. Green CI never meant the product was correct. It meant the code passed the checks we had.

The problem is that those checks don’t prove intent. They don’t prove the requirement was correct. They don’t prove the tests were testing the right thing. They don’t prove the documentation was complete. They don’t prove the change matched the real product behavior we needed.

They prove that the current artifacts agreed with the current checks.

With AI, the whole chain can be generated. The spec can be wrong, the code can follow the wrong spec, the tests can validate the wrong code, the docs can describe the wrong behavior, and CI can still be green. Everything agrees with everything, but the intent is wrong.

That’s not trust. That’s a consistent mistake.

This is why high coverage isn’t enough either. In my previous article about jsonparser, the painful part wasn’t that I had no tests. I had near-100% coverage in the area that mattered. The problem was that malformed input behavior was never properly described. So the tests proved what existed, not what should have existed.

You cannot test what you never described.

Security makes this even less optional. For years, many teams survived with some quiet version of security by obscurity. Not officially, of course. Everyone says security matters. But in practice, a lot of software depended on nobody looking too closely, or on attackers moving slowly enough that maintainers had time to react.

That assumption is breaking. VulnCheck reported that in the first half of 2025, 32.1% of known exploited vulnerabilities had exploitation evidence on or before the day the CVE was issued. This doesn’t mean every vulnerability becomes an exploit in hours, but it does mean the old time cushion isn’t something you can build your product around anymore.

So things that felt optional before become normal engineering requirements: malformed input, authorization boundaries, resource limits, timeout behavior, error states, data exposure, public API behavior. These aren’t enterprise extras. They’re product requirements.

This is the uncomfortable part: the trust problem is now everyone’s problem. Even if your company hasn’t “adopted AI,” your people probably have. Even if your CI is green, it may be green against the wrong intent. Even if your coverage is high, it may cover the behavior you remembered to describe, not the behavior the product actually needs.

So we need a different source of truth. Not instead of CI/CD, not instead of tests, not instead of code review. Above them. Something that says what the system is supposed to do, which obligations apply, what evidence proves them, and what becomes suspicious when something changes.

Otherwise AI won’t only help us move faster.
It will help us move faster with a false feeling of safety.

The outside structure is not the product

I know this problem from open source. For the last 12 years at least, I worked a lot in open source. I had my own popular open source projects, and today at Tyk we build an open source API Gateway.

Open source is hard. Not because people are bad. Usually it’s the opposite. Someone from the outside sends you a pull request. Maybe it’s a bug fix. Maybe a new feature. Maybe it’s useful. Maybe it’s technically correct. Maybe they spent their evening on it.

But as a maintainer, you still need to get inside the context. You need to understand what’s happening and why this person is doing it. You can be fast and accept too much, or stay picky and make people unhappy. Neither option really solves the trust problem.

The real issue isn’t that contributors are bad. The issue is that they see the outside structure. They see the code. Maybe they see the tests. Maybe they see the docs. But they don’t see the intent in the same way the owner of the project sees it. They don’t know all the small product promises made over the years. They don’t know which ugly thing is accidental and which ugly thing is load-bearing. They don’t know which customer flow depends on some behavior that looks strange from the outside.

They are not inside of this bubble.

And this isn’t only open source. The same thing happens inside a company. Someone from support knows the product very well. They see customer pain every day. They may even be technical enough to raise a pull request. Someone from solutions architecture can do the same. Another team can contribute to your service. AI makes all of this easier.

But internal doesn’t automatically mean trusted.

A support engineer may understand the product from the customer side, but not the architecture. Another team may understand code, but not the local history. AI may generate something that looks clean, but it has no real ownership unless someone gives it context and checks it.

These contributions can become shallow. Not useless. Shallow. They touch the visible layer of the system, but they aren’t backed by the deep intent of the people who own this part of the product.

We tried relaxing quality gates a few times. More people contributing sounds obviously good, especially when every company has more backlog than humans. But we had cases where a simple line, a simple fix, broke everything. We had other cases where the fix was so big in scope that it was too dangerous to move there.

The conclusion wasn’t “only engineers can write code.”
The conclusion was: if you want to scale engineering, it’s always about trust.

This is also why “move fast” changes meaning when you have customers. When you’re still searching for an MVP, you can break things and call it learning. But when customers put your product inside their infrastructure, the product is no longer fully yours. They pay you for stability, security, and predictable behavior. In a way, you give away part of the ownership.

At Tyk, this is very real. We build software used by banks, governments, and large enterprises. Quality assurance isn’t some internal slogan. It’s part of the relationship with customers. Every software has bugs; I don’t want to pretend otherwise. But the price of a bug isn’t the same everywhere. Sometimes it’s legal. Sometimes it’s regulatory. Sometimes it’s very big money. Forget even the money for a second: what if the bank goes down? It can become a national-level issue.

Speed isn’t how quickly you can make a change.
Speed is how quickly you can safely absorb change.

Lehman’s software evolution work has a phrase that fits here: “The safe rate of change per release is constrained by the process dynamics.” In the same passage, he says that as the number, size, and architectural distance of changes increase, complexity and fault rate grow more than linearly.

This sounds academic, but it matches product reality. You can move only as fast as your safety norms allow. Your current team, your architecture, your customer base, your process, your quality gates, your review culture — all of this defines your real speed.

If AI gives you more change than your trust system can absorb, you aren’t scaling engineering. You’re scaling incoming work.

Temporary specs become archaeology

One of the deeper problems is how we treat specifications in consumer engineering.

Most of what we call a specification is a temporary artifact. You start with all the best practices. Maybe an RFC. Then it becomes a detailed Jira ticket. Maybe later there is an ADR. There are comments in GitHub. A Slack thread. A Confluence page. A few decisions made during review because reality was different from the original assumption.

At the moment, this feels normal. This is how software gets built. But after some time, all these artifacts pile up. If you want to understand how a component works, you need to dig through history. You need to understand why it ended up in this final state. Why this was done and not that. You may be lucky and find the exact explanation. In most cases it’s lost in someone’s head.

This is archaeology, not development.

The bigger problem is that these artifacts are independent. The RFC isn’t connected to all the code. The Jira ticket isn’t connected to all the tests. The docs are scattered across ten pages. The final implementation isn’t connected back to the original assumptions. It’s not a graph.

So we trust a person — engineer, architect, lead, PM — to hold the high-level picture in their head. We trust them to find all dependencies. We trust them to notice backwards compatibility issues. We trust them to know which docs need updating. We trust them to remember which customer flow can break.

And of course people forget. Not because they’re careless. Because this is too much context for one person to carry.

You fix a bug and break some other flow. You build a feature and forget a dependency with another service. You update two documentation pages and miss the other eight. The feature exists, but it’s unusable for one group of users. The implementation works, but not in the real production shape.

The spec was supposed to create clarity.
But because it was temporary, it becomes one more historical artifact.

This is also why spec-first isn’t enough. Spec-driven development is better than no spec. Planning before coding is obviously better than jumping into implementation. But if the spec is still treated as a temporary artifact, after a few iterations you end up in the same position, with intent chaos.

During development, the spec always changes. You start with assumptions. Researched assumptions, but still assumptions. Then implementation begins and reality appears. The architecture doesn’t work. A limitation appears. A reviewer notices a security issue. QA finds a case. A customer dependency changes the direction. And in many teams, the spec isn’t updated.

The real knowledge moves into GitHub comments, Slack messages, review threads, and people’s heads. The Jira ticket becomes stale. The implementation says one thing. The ticket says another.

Imagine someone from QA comes back from vacation and needs to test the feature. They see the ticket. They see the implementation. They have no idea what is happening. Why this? Why not that? Is this intended?

It’s so common. I bet a lot of you feel the same.

How I can know what I don’t know?

A lot of bugs are not in the code first. They are in the missing specification.

Have you actually described what should happen if the input is malformed? Have you described that this functionality must not allow SQL injection? What happens if the third-party service times out? What is the error state? Have you described authorization boundaries? Resource limits? Performance boundaries? What happens when something goes from ten requests per second in a test environment to thousands in production?

There are also more subtle cases. Concurrency. Non-deterministic behavior. Map iteration. Merge order. I’m looking at you, Go.

Have you described that the behavior should be deterministic? Did you write a test for it? Did the test prove the requirement, or did it just execute the code?

This is where checklists, obligations, processes, discipline, and all the boring stuff come in. I know people hate boring process. I hate fake process too. Documents nobody reads. Boxes people tick after the fact. Quality theatre.

But the useful version is different. An obligation is not a test case. It’s a category of behavior you are required to describe: malformed input, boundary behavior, error handling, access denied, determinism, idempotency, atomicity, nil safety, overflow safety, encoding safety.

The obligation doesn’t tell you the answer.
It forces you to ask the question.

That is why I like it. It turns “maybe someone remembers” into a deterministic process. The checklist itself is human judgment. But checking whether the spec covered the checklist can be mechanical.

This is also where AI can help without pretending to own the product. If the code uses goroutines, the system can ask where cancellation, lifecycle, and error propagation are described. If code depends on map iteration or merge logic, it can ask whether determinism or commutativity matters. If code reads time directly, it can ask whether time is part of the behavior and how this is tested. If code changes a public API, it can ask where compatibility and documentation obligations are.

This isn’t AI judging architecture taste. It’s tooling surfacing missed questions.

That is the “how I know what I don’t know” loop. Spec obligations force code and test evidence. Code shape can reveal missing spec questions.

What regulated industries got right

Consumer engineering and regulated engineering live in different worlds. Different tools. Different conferences. Different language. Some of it is archaic. Some of it is bureaucracy. I don’t want every SaaS team to become an avionics certification team.

But we shouldn’t ignore what they learned.

I expected to find paperwork. Annoyingly, I found a lot of things our world forgot to learn.

In aviation, automotive, medical devices, space systems, the spec isn’t treated as a temporary note. It’s a source of truth that lives together with the software. Requirements have IDs. They have layers. They are linked to documentation, tests, implementation, verification evidence. You can see blast radius. You can see what a change affects. During review, if implementation differs from the spec, the spec must be updated.

The useful idea is not the paperwork.
The useful idea is that intent is durable, traceable, and connected to evidence.

NASA’s FRET is one example of this direction. It lets users enter hierarchical system requirements in structured natural language, gives those requirements unambiguous semantics, and can show them as natural language, formal logic, diagrams, and interactive simulation.

That doesn’t mean every product team needs FRET or formal methods everywhere. It means the requirement is not just a document. It’s something you can analyze, link, verify, and keep alive.

This is where requirement management becomes interesting again for consumer engineering. Not the old heavy version copied blindly from regulated industries. Not paperwork for paperwork. But the useful part: a source of truth, cross-links, invalidation, traceability, and evidence.

Combined with everything consumer engineering learned over the years: CI/CD, pull requests, fast feedback, developer experience, automated tests, observability, docs, release automation.

We should not throw away modern engineering.
We should add the missing trust layer.

From pull request to evidence pack

Today a pull request usually gives me code, maybe tests, maybe a description. But it doesn’t give me the whole chain.

It doesn’t tell me the original intent. It doesn’t tell me which obligations apply. It doesn’t show the blast radius. It doesn’t show which docs changed or should have changed. It doesn’t show which specs this conflicts with. It doesn’t show what changed during implementation compared to the plan.

So the reviewer has to reconstruct all of that.

Again, archaeology.

What I want instead is an evidence pack. Not enterprise theatre. Not documents for the sake of documents. A practical package that makes the change reviewable.

Here is the intent. Here are the requirements. Here are the obligations. Here are the tests that witness them. Here are the docs. Here is the blast radius. Here is how it aligns with existing specs and where we checked for conflicts. Here is what changed during implementation. Here is what still needs human judgment.

Then the pull request isn’t only code. It’s the full chain of development.

This matters for open source. It matters for support engineers contributing fixes. It matters for other internal teams. It matters for AI agents. You don’t trust the contributor blindly. You don’t trust AI blindly. You trust the evidence chain, and then you still apply human judgment where judgment is needed.

This will feel slower at first. Writing obligations is slower than writing a vague ticket. Linking tests to requirements is slower than writing random tests. Updating docs through the graph is slower than pushing a change and hoping someone remembers. But not all friction is bad.

The question is whether the friction creates trust.

Bureaucracy gives you friction without trust.
Evidence gives you friction that lets more people move safely.

If this trust exists, then AI can actually help us scale. Not by dumping more pull requests into the same review bottleneck, but by making more changes reviewable, traceable, and safe to absorb in parallel.

Without trust, maintainers become managers of incoming things. Instead of thinking about architecture, future, and vision, they review an endless stream of pull requests, fixes, and generated artifacts.

That is not the scaling I want.

Why I am building Proof

This is why I am building Proof.

I don’t want another tool whose main purpose is to create more code. We already have many of those. The problem isn’t that we can’t produce enough artifacts. The problem is that the artifacts don’t preserve intent.

I want specs to stop being temporary. I want requirements to live with the software. I want obligations to force the boring questions before they become production bugs. I want code, tests, docs, and requirements to invalidate each other when they drift. I want a reviewer to see the evidence chain instead of rebuilding it from memory.

AI will make engineering faster. That part is already happening. But faster without trust is not enough.

For me, the real question is this: how can I end up in the position where it’s not just a pull request coming from someone from the outside, but a well-thought evidence pack that makes me believe I can merge it as soon as possible?

That is the scaling I care about.

Not just more code.

More trusted change.

I Had Near 100% Test Coverage. It Didn't Matter.

Leonid Bugaev — Wed, 29 Apr 2026 18:17:34 +0000

You cannot test for what you never described.

I woke up and saw a wall of emails in my personal account. Then logged into my corporate Slack, and it was filled with Zendesk messages from customers. Everyone was looking for me.

The library I wrote, jsonparser, which got used by a lot of projects, got its very own public CVE. So everyone started freaking out looking at their scanners.

"That's what the fame is," was my first thought.

Now I remember some notifications I kept ignoring from the Google OSS Fuzz project, I signed up multiple years ago.

This lib was written in the pre-AI-agents era (so weird to say that now!). Every piece was handcrafted manually, using best practices, with full test coverage. I checked the function which had the issue, and it literally had near 100% test coverage. But it did not matter, because the issue was in handling of malformed input data. One of the edge cases which was missed. In other words, the issue was in the specification of what this function should do and how it should behave in edge cases.

But it opened one more can of worms. I wrote this library like 6 years ago. I don't remember anything. And my only source of truth is the code and the tests, which is rather cryptic and looks more like archaeology.

The issue is fixed now. But how do I prevent such issues happening in the future? And if 100% code coverage is not the answer, what is? And what is my source of truth?

So I started digging. And it went way deeper than I expected, and changed the way I look at software engineering forever.

Down the rabbit hole

I started thinking about what the gold standard of software quality is. My first answer was NASA. How does NASA solve these kinds of issues?

AI now produces so much code that I feel like I am losing ownership of it. Not only of the code. Of the intent.

I wanted to understand how people work when tests passing is still not enough and the price of being wrong is huge.

The surprising thing is that a lot of NASA's work is public. Their software engineering requirements are public. FRET is public. Kind2 is public. A lot of the case studies are public. There are papers about aircraft, Mars rovers, superconducting magnets, and formal requirements that found bugs before code existed.

I started reading all of this not as an academic exercise, but because I had a very dumb practical problem: my tests were green, my coverage looked fine, and still one missed edge case was enough to create a public CVE.

Then I went deeper into automotive and aerospace. It opened a whole new world of software engineering for me. For some reason, our world of consumer software engineering and regulated software engineering in those industries almost do not intersect. Different tools, different conferences, different language. Sometimes it feels like they live in a parallel universe.

Some of it looks archaic. Some methodologies are weird.

Our engineering progressed a lot too. We got very good at moving fast and catching damage quickly. CI/CD, linters, tests, canaries, observability, rollbacks. I don't want to pretend every SaaS product should behave like avionics certification.

But we optimized for speed. They optimized for evidence.

Their industry spends much more time asking what evidence they need before they are allowed to trust the change. Some of it is painful. Some of it is bureaucracy. But the idea underneath is not stupid: if you claim the system should behave in some way, you need a durable chain from that statement to tests, code, and evidence.

There is real proof there, but it is not the fantasy version I had in my head, where every line of every product is mathematically proven end-to-end. They prove specifications. They use model checking. They simulate models, like with Simulink, against many input/output cases. They measure structural coverage. They use formal proof where the criticality justifies it.

And they still use testing, code review, static analysis, and all the normal engineering work around it. The difference is that proof and evidence are attached to the parts where being wrong is not acceptable.

That actually made the idea useful for normal engineering.

This is a huge topic, which I will cover in future articles. But the first concrete thing I found was MC/DC. It is one of the ways safety-critical industries look at coverage, and it made standard line coverage look very weak to me.

Line coverage says a line was touched at runtime. It does not say that the decision was tested.

Why 90% line coverage can still mean 60% real coverage

I still use line coverage. I still look at it.

But line coverage is bullshit. You should not trust it. Not on its own.

In Go, when you run:

go test -cover ./...

you mostly get statement coverage. The tool tells you whether a statement executed during the test run. That's useful. But it doesn't tell you whether the decision was tested.

Take a tiny parser-style example:

func isDigit(c byte) bool {
    return c >= '0' && c <= '9'
}

Now test it like this:

func TestIsDigit(t *testing.T) {
    if !isDigit('5') {
        t.Fatal("5 should be a digit")
    }
    if isDigit('x') {
        t.Fatal("x should not be a digit")
    }
}

Looks fine. The line ran. The function returned true once. The function returned false once. Your coverage report can look perfect.

But what did you actually prove?

You tested '5'. You tested 'x'. You didn't prove the lower boundary. You didn't prove that '/' fails because it's before '0'. You didn't prove that ':' fails because it's after '9'.

The line is covered. The boundary is not.

MC/DC stands for Modified Condition/Decision Coverage. It asks the question line coverage does not ask: did each condition independently affect the outcome?

When your code says if a && b, line coverage tells you the if was hit. MC/DC asks whether a alone can change the result, and whether b alone can change the result.

For this line:

return c >= '0' && c <= '9'

there are two conditions:

c >= '0'
c <= '9'

A simplified MC/DC table looks like this:

Case	`c`	`c >= '0'`	`c <= '9'`	Result	What it proves
lower edge	`'0'`	true	true	true	lower edge accepted
below lower edge	`'/'`	false	true	false	lower bound independently blocks
above upper edge	`':'`	true	false	false	upper bound independently blocks
nominal digit	`'5'`	true	true	true	normal digit works

The table is just a way to say: these are the cases that matter. This is the part ordinary coverage does not force you to say.

This used to be mostly a safety-critical tooling conversation. DO-178C requires MC/DC for the highest-criticality aviation software. The tooling was expensive, slow, and hard for normal teams to justify.

That changed. GCC 14 has -fcondition-coverage. Clang 18 has -fcoverage-mcdc. Rust is moving in the same direction with richer branch and condition coverage work, even if I would not call Rust MC/DC stable yet. Go does not have native MC/DC support, so I ended up adding code-level Go MC/DC measurement to Proof, and we have been extending the same direction to JavaScript and TypeScript as well.

What aerospace and automotive had because they were slow and diligent is now becoming available to normal engineering teams because AI changed the economics. You don't need a certification lab to ask a harder question about your tests. You also don't need to apply all of this to the whole company on day one. Start with the part where wrong behavior actually hurts.

The jsonparser numbers weren't subtle

After the CVE fix, I wanted to understand why my previous approach didn't make this kind of missing behavior obvious enough.

So I applied the MC/DC and requirements approach to jsonparser in a later public PR: buger/jsonparser#281.

Again: this PR didn't fix the original CVE. It was the follow-up work after the CVE fix. But it was not just a paperwork exercise. The hardening pass found and fixed more real issues and removed dead code that my previous process had not made obvious.

That was the uncomfortable part for me. I started by asking: what did my tests actually prove?

On the main branch before that work, ordinary Go statement coverage was already decent:

Metric	Before / main	After / PR
Standard Go statement coverage	85.3%	99.4%
MC/DC decisions	138/209 = 66.0%	203/203 = 100%
MC/DC conditions	175/253 = 69.2%	244/244 = 100%
MC/DC gaps	71 incomplete decisions, 78 missing condition proofs	0

85.3% coverage isn't bad. Most teams would see that and move on. But decision coverage told a different story: only 66% of decisions were fully covered, and only 69.2% of conditions were proven independently.

And the more interesting part: some functions already looked perfect by ordinary coverage.

Examples from the before state:

parseInt                     100% statement coverage
Unescape                     100% statement coverage
decodeSingleUnicodeEscape    100% statement coverage

But MC/DC still found missing independent-condition evidence:

bytes.go:21   parseInt missing proof for c < '0'
escape.go:148 Unescape missing proof for len(in) > 0
escape.go:47  decodeSingleUnicodeEscape missing proof for h1 == badHex
escape.go:47  decodeSingleUnicodeEscape missing proof for h2 == badHex
escape.go:47  decodeSingleUnicodeEscape missing proof for h3 == badHex

100% line coverage can still leave a condition unproven.

The code ran. The decision wasn't tested.

The bug was in what I forgot to describe

Coverage does not paint the whole picture. Even MC/DC. The bug can still be in the spec.

That is what happened with jsonparser. It was a classical case: you are building something, moving forward, and not looking back. You don't know what you don't know. I did not think about what would happen if this edge case appeared. I think most of us do not think about it this way.

I did not have any specs driving development or anything that forced me to think about the edge cases before writing the code. So of course I did not test for them. You cannot test for what you never described.

Testing assumes the specification is correct. That is the NASA/formal-methods lesson that changed how I think about this. The hard part is not testing the implementation. The hard part is questioning the specification itself.

This is where I found two different questions that I had been mashing together.

The first question starts from my specification: if this is what I claim the system should do, which logical cases need to be witnessed?

Not the code. The intent.

NASA built an open-source tool called FRET (Formal Requirements Elicitation Tool) that lets you write requirements in structured English and translates them into formal logic.

FRET includes an algorithm called FLIP (FuLl Independence Pair). FLIP takes a formalized requirement and generates the minimum set of test cases proving each boolean variable independently affects the outcome. Not every possible combination. Just the ones that matter.

I still have to write the requirement. I still have to decide what malformed input, boundaries, errors, and edge cases mean. FLIP does not do that for me.

But once the requirement is formalized, FLIP tells me exactly which test cases that requirement needs.

I built a tool called Proof that implements this approach.

That is the part I care about: how many tests are enough for this requirement?

Not "how many tests did I happen to write?" Enough for what I described.

The second question starts from my actual code: did my tests exercise every boolean condition in the implementation so each one independently affects the outcome?

This side does not care what I meant. It looks at what I wrote.

And sometimes it shows that my code has many more logical cases than my spec. So maybe my spec is not accurate enough.

Or my spec says this edge case matters, but my tests don't witness it.

Or my tests cover implementation details, but the behavior is under-described.

I learned this the hard way on jsonparser. The spec side and the code side kept disagreeing in useful ways, and that is where code drift and spec drift become visible.

The gap goes in both directions. Sometimes the code is wrong. Sometimes the tests are weak. Sometimes the spec is too vague.

Sometimes all of it combined badly.

Checklists, not memory

What can be more deterministic than a checklist? In aerospace and automotive, everything has its own checklist. The price of a mistake is too high to rely on someone's memory. I think checklists are the driving force behind quality engineering in those industries.

When you do not have specifications, it is very hard to create a checklist. When you are building a feature, you can have test cases, but that is a moving target. The items are constantly changing. You need something that will be the same all the time.

You cannot rely on humans here. Even on me, to be frank. I can miss these items too. You need deterministic checklists.

In practice, the questions are very simple:

What will happen if this is malformed data? What will happen if this is slow and the request times out? What will happen if the database is down? What will happen if you have a very large object? What will happen if the function returns different values with the same inputs?

These are the cases where security issues and data bugs tend to live. For jsonparser, these are the exact cases I had not thought about.

Without obligations, edge cases depend on memory. Maybe I remember to test malformed data. Maybe the AI remembers. Maybe a reviewer notices. Maybe no one does.

At the moment, it is just a matter of whether someone forgets or not forgets to test it.

This is where the CVE fix actually changed how I work. The fix itself was mechanical. But the obligations I wrote afterward forced me to think about the cases I had skipped. Every one of those became an explicit question I had to answer. Not "did someone remember to test this?" but "here is the list, and each item needs a witness."

Obligations turn edge cases from "someone remembered to test this" into a deterministic process.

When I first started writing obligations for jsonparser, it was actually quite easy with modern AI tooling. I reviewed all of the specs. The flow is: you cannot pass this check until the checklist is green, until you define obligations for all of those cases, and until you define test cases for all of those cases as well.

This is what the double link looks like in practice:

// In the code — annotated with the requirement it implements:
// SYS-REQ-863
func (s *Service) lookupCache(req Request) (*Result, bool) {
    // ...
}

// In the test — annotated with both the requirement AND the specific MC/DC row:
// Verifies: SYS-REQ-863
// MCDC SYS-REQ-863: cache_lookup_requested=T, component_inputs_unchanged=F,
//                    cached_component_result_reused=F => TRUE
func TestMCDC_SYS_REQ_863_Row1(t *testing.T) {
    evalVerifyScenario(t, "SYS-REQ-863", map[string]bool{
        "cache_lookup_requested":         true,
        "component_inputs_unchanged":     false,
        "cached_component_result_reused": false,
    }, true)
}

Each test is not just "test the function." Each test is: "prove that this specific variable independently affects the outcome of this specific requirement."

If I change the spec, I can see exactly which MC/DC rows are affected and which tests need to be reviewed. If I change a test, I can see which spec requirement it was proving and check whether the spec still says the same thing. If I add a new variable to the requirement, FLIP will generate new witness rows, and the missing tests become immediately visible.

This is the double link. Change the spec, review the tests. Change the tests, review the spec. If you have not touched the spec, why would you touch the test?

This is where the "how many tests are enough?" question changed for me. Before, the answer was always vibes. Write enough tests. Cover important paths. Don't overdo it. Be pragmatic.

All true, and also not very helpful.

Now I think about it differently. Enough tests means enough evidence that every condition I described, or every condition my code actually contains, can independently affect the behavior I care about.

It is not about how many tests I have. It is about whether I really, really trust my system and whether it actually does what I described.

The true challenge is legacy

You can always start a new project and have a really nice experience with all of this. But the true challenge lies in the big legacy projects. They make up like 90% of all software. They bring the majority of the money. And they are the ones where wrong behavior actually hurts.

I work with very complex software. At Tyk, we build API gateway software used by banks, governments, and other serious enterprise customers. I am a very sceptical person. I always want some proof. At the same time, I understand that software is always about compromises.

But the game is changing. What was not possible in the past is now possible for small teams in terms of quality and processes. The wind is changing with AI.

The true power happens when you can apply some of those approaches to legacy large enterprise codebases. If it works there, it will work everywhere.

I know how challenging it is. You cannot do it in one go. You cannot just make a switch and start using a new process.

This is not only about the technical part. It is also about the people part. Even at the size of Tyk, with like a hundred people, it is not about the implementation. It is about the processes and the people. The technical part is the easiest one.

In order to convince people that you can actually make it, you need to be able to do it in parts. Start small, then scale.

Can you take small parts, turn them into a repeatable process, and then start scaling? That is how it works in the majority of cases.

So I picked the policy engine. Authorization and gateway policy decisions are obviously critical. If the policy engine behaves incorrectly, you are not talking about a cosmetic bug.

I applied the same kind of thinking to the Tyk policy package in a public PR: TykTechnologies/tyk#7932.

Metric	Before / main	After / PR
Standard Go statement coverage	81.0%	99.2%
MC/DC decisions	74/115 = 64.3%	111/111 = 100%
MC/DC conditions	95/142 = 66.9%	137/137 = 100%
MC/DC gaps	41 incomplete decisions, 47 missing condition proofs	0

81% ordinary coverage. 64.3% decision coverage. The normal coverage number says most statements ran. The MC/DC number says a lot of policy decisions still do not have independent evidence.

For a policy engine, the second number is the one I care about.

Code coverage is not about a metric

It is about trust.

What do we trust? In classical software engineering, we say: here is the code and here are the tests, the tests are the source of truth. If you want to know how the system works, read the tests.

I do not believe that anymore. Not with AI writing code. Not with AI writing tests. Not with AI validating its own assumptions.

The source of truth cannot just be tests anymore. AI can write those too.

A passing test can prove that the code agrees with the test. It cannot prove that both agree with my intent.

So I moved the source of truth up. For me, it has to be the specification: the static description of what I expect the system to do.

Then code implements it. Tests witness it. Coverage measures evidence around it. Traceability keeps the chain from silently rotting.

I started this whole journey because of one CVE in a library I wrote six years ago. I ended up in a completely different place.

I thought the problem was in the code. It was in what I forgot to describe.

I thought coverage was the answer. It was the wrong question.

The first article was about losing intent. This one is about binding intent back to code.

Originally published on substack: https://blog.reqproof.com/p/i-had-near-100-test-coverage-it-didnt

AI Made Implementation Faster. Verification Is Still the Bottleneck

Leonid Bugaev — Thu, 23 Apr 2026 15:35:12 +0000

AI made implementation dramatically faster.

Trust did not.

I live in two different worlds now.

In one, I build my own projects with AI and ship more software than ever. I have written more software in the last two years than across the rest of my career, and I have barely written any code manually in the last year.

In the other, I lead engineering for software used by banks, governments, and other regulated environments, where mistakes are expensive and confidence matters more than speed.

In both worlds, I keep hitting the same wall:

Implementation got dramatically faster. Trust did not.

That is the part I think the industry still keeps smoothing over.

Faster code generation is not faster engineering

The current AI coding conversation often assumes that if code generation speeds up, engineering speeds up too.

That is not what I see.

On my own projects, I can build much faster than before. AI helps me move quickly, clean things up, write tests, refactor, and push ideas further in less time.

But it also asks me to trust more.

I am not just delegating typing.

I am delegating thinking, validation, and judgment too.

And I am still not sure where the safe line is.

In enterprise software, the picture is different but the problem is the same.

AI absolutely helped us in some areas. It reduced noise. It reduced interruption-based work. It helped other teams answer questions about system behavior without constantly pulling senior engineers into ad hoc investigations.

That mattered.

People were less interrupted. Context switching got better. Engineers were happier.

But it did not suddenly make us ship features 2x faster.

Not even close.

Because implementation was never the whole job.

Verification is the bigger slice.

The verification gap

The phrase I keep coming back to is:

verification gap

By that I mean the distance between what I intend the software to do and what I can actually prove about its behavior.

Between intended behavior and demonstrated behavior.

That gap always existed.

AI did not invent it.

It amplified it.

Why AI makes this problem worse

When humans wrote the code, the same brain often held the intent, the implementation, and the validation loop together.

Not perfectly. People still shipped bugs. Specs were incomplete. Tests missed things.

But there was at least one place where the system could be understood as a whole: the person writing it.

That is no longer the default.

Now the human writes the prompt.

The model writes the code.

The model writes the tests.

The human skims the diff.

The model writes the cleanup.

The CI passes.

The feature ships.

And if the original intent was slightly wrong, incomplete, or misunderstood, that mistake does not stay in one place anymore.

It gets propagated through the whole stack.

The plan is based on the wrong assumption.
The implementation is based on the wrong assumption.
The tests are based on the wrong assumption.
The documentation often reflects the same wrong assumption.
The "manual validation" is often the same model being asked to sanity-check itself.

At that point, what exactly are we proving?

Often just that the system is internally consistent with the assumption it invented for itself.

Not that it matches our intent.

Bug free is not the same as intent-correct

This is why I think a lot of AI productivity discourse still misses the real problem.

People say: just write better tests.

I do write tests.

AI writes tests for me too.

That is not the point.

Tests verify behavior for cases somebody thought of.

That somebody used to be a human.

Now it is often a human plus a model.

That is still not the same thing as verifying intent.

You can have 100% line coverage and still miss the thing that matters.

You can have a green CI run and still not know whether the software behaves the way you intended.

A green pipeline can still be a polished misunderstanding.

Bug free is not the same as intent-correct.

Software is not flat. It is layers.

This gets worse as software gets bigger.

Software is not flat.

It is layers.

It is wide, deep, and full of interacting components, hidden assumptions, old decisions nobody remembers, backwards compatibility constraints, and behavior that only makes sense if you know four other subsystems.

Any project that lives long enough eventually reaches a point where one brain is no longer enough.

That was true before AI.

It is still true now.

AI does not remove that limit.

In some cases it makes you hit it faster, because you can generate change faster than you can understand its consequences.

A lot of our engineering process exists because of this:

CI/CD
QA
RFCs
architecture reviews
team boundaries
approval workflows

These are not random rituals.

They are patches over the same underlying problem: software complexity grows beyond what one brain can safely manage.

Where does intent actually live?

I think mainstream software engineering is still missing something fundamental.

We do not maintain a real source of truth for intent.

If I ask where the intended behavior of a system lives right now, the honest answer in most teams is:

all of it combined badly

Some of it is in source code.

Some of it is in tests.

Some of it is in RFCs.

Some of it is in Jira tickets.

Some of it is in Confluence.

Some of it is in the heads of senior engineers.

None of those is the place where I can go and see, clearly, how the system is supposed to behave right now.

That is not a source of truth.

That is archaeology.

And that feels like a major difference between mainstream software and more regulated domains like aerospace or automotive, where intended behavior is at least treated as a first-class artifact.

In mainstream software, especially in large, complex systems, we mostly reconstruct intent after the fact from scattered artifacts.

And then we act surprised when regressions keep happening.

So what is the actual bottleneck now?

If a feature can be implemented in hours instead of weeks, why have so many teams not seen the full payoff?

Because implementation was never the only bottleneck.

The harder part is deciding what should be built, making that intent explicit enough, and then verifying that the resulting system still matches it after the code, tests, and surrounding context have all changed.

That is where the time goes.

That is why I think AI did not remove the hard part of engineering.

It moved it from writing to verification.

If you want the next essays on this topic, subscribe on Substack: https://blog.reqproof.com/p/ai-writes-your-code-nobody-verifies