DEV Community: Marx Jenes

Verification vs Validation in Software Testing: A Practical Breakdown

Marx Jenes — Thu, 20 Aug 2026 08:06:12 +0000

A few years back I worked on a reporting feature that hit every acceptance criterion in the ticket. Every field matched the spec, every calculation was correct according to the documented formula, every edge case in the ticket was handled. QA signed off. Then it shipped, and the first support ticket came in within a day: the numbers were "technically right" but useless, because the spec itself had been written against an old version of how the business actually calculated that metric. Nobody had checked with finance before writing the ticket.

That's verification and validation in one story. We verified the software matched the spec perfectly. Nobody validated that the spec matched what the business actually needed. Both matter, they're not interchangeable, and mixing them up is one of the more expensive mistakes a team can make without realizing it's happening.

Verification: are we building it right?

Verification asks whether the software conforms to its specification, requirements, or design documents. It's internal-facing - you're checking the product against what was written down, not against what anyone actually wants.

This covers most of what people think of as "testing" day to day:

Code reviews checking implementation against a design doc
Unit and integration tests checking behavior against documented requirements
Static analysis, linting, spec-conformance checks
QA walking through a ticket's acceptance criteria one by one

Verification is good at catching a specific kind of bug: the gap between what was specified and what got built. It's terrible at catching a different kind of bug: the gap between what was specified and what should have been specified in the first place. My reporting story is a verification success and a validation failure, and verification alone had no way of flagging that.

Validation: are we building the right thing?

Validation asks whether the software actually satisfies the real-world need it's meant to serve, regardless of what the spec says. It's external-facing - checking against reality, users, and business intent, not against a document.

This looks more like:

User acceptance testing with actual users, not just QA reading a checklist
Beta releases and real usage data
Product and stakeholder review against the actual problem being solved, not just the ticket
Watching what happens when real traffic hits the thing, not just the scenarios someone thought to write down

Validation is where "it passed every test" and "it's ready to ship" turn out to be different sentences. A feature can be verified into oblivion - every acceptance criterion green, and still fail validation the moment a real user tries to do something the spec author never considered.

Why the mix-up is expensive, not just academic

Teams that only verify tend to over-trust their test suite. A hundred passing tests feels like safety, but if every one of those tests was written against a spec that had a gap, you've built a very well-tested version of the wrong thing. The bug isn't in the code. It's upstream, in the requirements, and no amount of additional unit testing against those same requirements will ever catch it.

This is also why "we have great test coverage" and "our users are happy" aren't the same claim, and teams get bitten when they treat the first as proof of the second. Coverage measures verification. It says nothing about validation.

The verification toolbox, and what each piece actually catches

It's worth breaking "verification" down further, because the different techniques catch genuinely different classes of mismatch:

Static verification happens before anything runs: code reviews, design reviews, static analysis, linting, type checking. These catch structural problems - a function that doesn't match its documented contract, a data flow that violates an assumption elsewhere in the system, a design that technically satisfies a requirement but contradicts another one nobody cross-checked. Cheap to run, cheap to fix at this stage, and the earlier a mismatch is caught here, the less it costs later.

Dynamic verification is everything that runs the code against expected behavior: unit tests, integration tests, contract tests, regression suites. This is where most engineering time goes, and for good reason - it's fast, repeatable, and automatable in a way static review isn't. But it's bounded by the same limitation every time: it can only check the code against what someone wrote down as "expected." A dynamic test suite with high branch coverage against an incomplete spec is still just thoroughly verifying the wrong thing.

Formal verification - proving a program correct against a formal specification using mathematical methods, sits at the far end and is mostly reserved for places where a bug is catastrophic: aerospace, medical devices, cryptographic protocols, safety-critical embedded systems. Most teams will never need it, but it's worth knowing it exists, because it makes the boundary clearer: even a mathematically proven-correct program is only proven correct relative to its spec. If the spec encodes the wrong requirement, formal verification will prove the wrong thing flawlessly.

The pattern across all three: verification techniques get more rigorous as you move from static review to formal proof, and none of them, at any level of rigor, can tell you whether the specification itself reflects reality. That gap only closes through validation.

The validation side, and why it resists automation

Validation techniques look different because they're checking against something inherently fuzzier than a spec:

User acceptance testing (UAT) puts the software in front of actual users or close proxies (product owners, domain experts) and asks whether it does what they need - not whether it matches a document. UAT frequently surfaces requirements gaps that were invisible during verification, because the people testing aren't checking a checklist, they're trying to accomplish something.

Beta programs and staged rollouts extend this to real usage at scale, which matters because individual UAT sessions still can't replicate the variety of how a broad user base actually behaves. A feature can sail through UAT with five friendly testers and still collapse against the messiness of real users doing things nobody anticipated.

Production monitoring and incident analysis is validation that never stops. Every bug report, every confused support ticket, every unexpected usage pattern is a live signal about whether the system satisfies the real need - arguably a more honest signal than any pre-release testing phase, because it's coming from reality instead of a simulation of it.

Domain and stakeholder review - going back to the people who understand the actual problem, not just the ticket, and asking whether this solves what they actually needed - is the least technical and most frequently skipped form of validation, and it's often the one that would have caught my reporting-feature story before it reached a user.

The common thread: validation techniques resist full automation because they're checking against a moving, real-world target rather than a fixed document. You can automate a regression suite. You can't fully automate whether something still makes sense to the humans using it, and teams that try to substitute more verification for that keep rediscovering the gap the hard way.

A rough way to think about where each one belongs

Verification tends to live close to the code - the earlier and more automatable, the better, because it's checking against something explicit and stable (a spec, a contract, a set of requirements). This is where automated test suites earn their keep: fast, repeatable checks that the implementation still matches what was agreed on.

Validation tends to need something verification can't provide on its own: exposure to reality. Real users, real traffic, real edge cases nobody thought to write into a ticket. This is also where a lot of teams under-invest, because it's harder to automate and harder to feel "done" with - there's no green checkmark for "definitely building the right thing," only ongoing signal from actual usage.

Verification and validation in software testing, does not substitute each other. A team that's excellent at verification and weak at validation ships well-tested products nobody needed exactly as specified. A team that's strong on validation but sloppy on verification ships the right idea built on a shaky, bug-prone foundation. The healthiest setups treat them as two different questions, asked continuously, rather than one "testing" phase that's supposed to cover both.

Next time a feature passes every test and still gets a confused reaction from users, it's worth asking which of the two actually failed. Usually it's not the tests.

How to Build a First Test Suite From Scratch for a New Project?

Marx Jenes — Wed, 12 Aug 2026 06:24:22 +0000

The worst test suite I ever inherited had 400 tests, and I trusted about six of them. The rest were either testing implementation details nobody cared about, duplicating each other, or so tightly coupled to internal function names that a harmless refactor broke thirty tests for no real reason. Reading that codebase taught me more about what not to do than any greenfield project ever has.

So when you're starting from zero, the goal isn't "write a lot of tests fast." It's building a suite you'll still trust a year from now. If you're new to this, getting the software testing basics right early matters more than covering everything - learning how to build a first test suite from scratch teaches you what to prioritize in a way that inheriting someone else's bloated suite never will. Here's roughly how I'd approach it.

Start with what would actually hurt if it broke

Before writing a single test, list the handful of things that would be genuinely bad if they silently broke - checkout completing, auth working, the core thing your product does actually happening. Not every function, not every branch. Just the stuff where a silent failure costs you money, users, or trust.

This list is usually shorter than people expect. Five to ten flows for most early-stage products. That's your actual test suite's job in the first few months, not "100% coverage."

Unit tests for logic, not for plumbing

Unit tests are for things with actual decision-making in them - pricing calculations, validation rules, state transitions, anything where "given this input, is the output correct" is a real question with a wrong answer possible. They're fast, they're cheap, and they should make up the bulk of your suite.

Skip unit-testing pure plumbing: a function that just calls another function and returns its result doesn't need its own test. That's the kind of test that pads a coverage number without catching anything real, and it's exactly the kind of test that made that 400-test suite so hard to trust.

Integration tests for the seams

This is where most early suites are too thin. Unit tests tell you your pricing function is correct in isolation. They don't tell you the checkout flow actually calls it correctly, passes the right currency, or handles what happens when the payment provider returns an error you didn't expect.

The seams between your code and everything external - your database, third-party APIs, other internal services are where bugs actually like to hide, because that's where two sets of assumptions meet and don't always agree. This is also the hardest category to get right early on, because hand-writing every plausible external response (success, timeout, malformed payload, rate limit) is a lot of manual work, and most teams just don't do it, which is part of why integration bugs are so common in production even in codebases with decent unit coverage. Tools that can record real API traffic and replay it as test cases - Keploy is one - exist specifically to close that gap without hand-writing every scenario, which is worth knowing about before you spend a week writing mocks by hand.

Don't test the framework

A depressing amount of early test-writing time goes into testing things the framework already guarantees - does this React component render, does this route return a 200. If your framework is reasonably mature, trust it. Test your logic, not its plumbing.

Make flaky tests a same-week problem, not a someday problem

The first flaky test in a new suite is a fork in the road. Fix it now, while the codebase is small enough that the cause is easy to find, or let it slide and watch the team's trust in the whole suite erode as more join it. I've written about this before - flaky tests are almost never actually about the test itself, they're usually pointing at shared state, a race condition, or an environment assumption nobody wrote down.

Let the suite grow with real usage, not just imagination

Whatever you write on day one will cover the bugs you could imagine. It won't cover the malformed payload from a partner's outdated client, or the auth token format some old integration still sends. As the project gets real traffic, treat production incidents as test-suite input - every bug that reaches production and wasn't caught is a gap in the suite, and turning it into a regression test is the cheapest way that gap ever gets closed.

The suite you actually want in six months

Not the biggest one. The one where a red result still means something, where the team fixes failures instead of quietly muting them, and where adding a new feature doesn't mean an afternoon fighting tests that were testing the wrong thing to begin with. Start small, aim it at what actually matters, and let it grow from real usage rather than a coverage target.

What AI Actually Helps With in Test Automation

Marx Jenes — Mon, 10 Aug 2026 04:30:11 +0000

What AI Actually Helps With in Test Automation (and Where It Falls Apart)

A teammate spent an afternoon last month feeding our checkout flow into an AI coding assistant and asking it to generate a test suite. Twenty minutes later he had forty tests. Good coverage on paper, clean syntax, decent naming. Then we ran them against a known bug we'd fixed the week before, and every single test passed anyway. The AI had written tests that matched what the code currently did, not what the code was supposed to do. It had no way of knowing the difference.

That's the honest version of where AI sits in test automation right now, and it's a lot less dramatic than most of what gets written about it.

Where it's genuinely useful

Scaffolding, fast. Writing the boilerplate for a new test file, setting up fixtures, wiring up assertions for a straightforward CRUD endpoint - this is exactly the kind of repetitive, low-judgment work AI is good at. It saves real time, especially early in a project when you're writing a lot of similar tests.

Spotting edge cases you didn't think of. Describe a function's inputs and ask what could break it, and you'll usually get a decent list back: empty strings, null values, boundary numbers, unicode weirdness. Not exhaustive, and not always relevant to your actual domain, but a reasonable starting checklist that's faster than staring at a blank test file.

Explaining existing test failures. Pointing an AI assistant at a stack trace and a failing assertion and asking "what's likely going on here" is one of the more underrated uses. It's fast at pattern-matching common failure modes - off-by-one errors, type mismatches, obvious null pointer issues, even if it can't always diagnose the deeper cause.

Refactoring test code itself. Cleaning up duplicated setup logic, converting a pile of copy-pasted tests into a parameterized suite, modernizing old assertion syntax - mechanical work that AI handles well because it's about the shape of the code, not what the code is supposed to prove.

Where it falls apart

It doesn't know what "correct" means for your system. This is the checkout example above. AI can write a test that passes against your current code, but it has no independent way of knowing whether your current code is right. It's not testing against your business logic, it's testing against whatever's already there - which means it's very good at locking in bugs, not just catching them.

It doesn't know your actual traffic patterns. A generated test suite tends to cover the inputs a human (or an AI) can imagine, not the inputs your system actually receives. Real API traffic is messier than anyone's imagination: malformed payloads from a partner's outdated client, an auth token format from three integrations ago that somehow still works, timing patterns nobody would think to write by hand. Software test automation built entirely from imagined scenarios misses a category of bug that only shows up under real usage.

Maintenance is still a human problem. AI can generate forty tests in twenty minutes. It doesn't show up six months later when the API contract changes and half of them need updating. Automated testing has always had a maintenance cost, and generating tests faster just means you're generating maintenance debt faster too, unless something is tracking how the underlying system actually behaves over time and flagging what's drifted.

It can't tell you what to test, only how. The genuinely hard part of test automation was never syntax. It's deciding what matters: which flows are business-critical, which failure modes are acceptable, where the risk actually lives. That's a judgment call rooted in what your product does and who depends on it. No amount of prompting hands that over.

Where this is heading

The realistic near-term picture isn't "AI writes your test suite," it's AI doing more of the mechanical work - scaffolding, edge-case suggestions, failure triage - while the actual test strategy, the ground-truth of what correct behavior looks like, and the ongoing maintenance stay firmly a team's responsibility. The tools that will matter most in this space long-term probably aren't the ones that generate the most tests fastest, but the ones that keep tests grounded in what a system genuinely does in production, rather than what someone (human or AI) guessed it should do.

Worth remembering next time a demo makes AI-driven test automation look like a solved problem: the demo is testing against a scenario someone already knew the answer to. Production isn't that generous.

Why Flaky Tests Are Rarely About the Test

Marx Jenes — Thu, 06 Aug 2026 12:42:52 +0000

We had a checkout test at my last job that everyone called "the coin flip." Green for a week, red twice on a Tuesday, green again. Someone eventually wrapped it in a retry and it sat like that for eight months before anyone looked at it again. Turned out the real bug was a webhook that occasionally fired before the order record finished writing to the DB - a two-hundred-millisecond gap that only showed up under load. The test wasn't broken. It was the only thing in the entire pipeline that noticed.

That's usually the story. Someone blames the test - bad selector, missing wait, a sleep(2) some intern left in there three years ago, and half the time they're right. But when a test flakes repeatedly and nobody can explain why, the test is rarely the actual problem. It's just the part of the system rude enough to say something.

A few places I keep finding the real cause hiding.

Tests that quietly depend on each other

Test A writes a row, Test B reads it and never knew it needed to. Run B by itself, it passes. Run the suite in a different order, or in parallel, and B fails for no reason anyone can point to. I've lost a full afternoon to this exact thing more than once - a cache value from Test 12 leaking into Test 47.

The actual fix is annoying and unglamorous: every test gets its own fixtures, its own scoped data, no assumptions about what ran before it. If your suite only goes green in one specific order, you don't have a flaky test. You have an undocumented dependency graph, and it's going to bite someone eventually.

The app is racing, not the test

Click a button, immediately assert on the result - that's a bet that the UI update lands the instant the click handler returns. It usually does, on your machine, on a good day. Add a debounce, a background job, or just enough network latency and that bet stops paying off.

This one's frustrating because the test isn't being paranoid. The app genuinely has a race condition. The test just runs the interaction often enough, across enough machines, that it eventually catches the app mid-race - something a human clicking through the same flow once or twice would probably never notice.

Clocks, timezones, and CI boxes that aren't your laptop

Date.now() and anything DST-adjacent will happily pass for months and then fail on one specific day of the year, or the moment your CI runner's timezone doesn't match what the original author assumed (usually their own laptop, usually not documented anywhere). Same idea with tests that were written assuming reasonably fast hardware - throw them on a throttled CI box and timing assumptions that never mattered locally suddenly do.

Mocks that stopped matching reality

This is the sneaky one, because it doesn't look like flakiness at all - it looks like a test that's passing. Someone mocks a third-party API, gets it right on day one, and then the real API changes: a field gets renamed, an error shape changes, rate limiting gets added. The mock has no idea. It keeps returning exactly what it always returned, the test stays green, and production quietly starts failing in a way nothing in CI can see.

If anything this is worse than a red test, because it erodes trust in the wrong direction - teams end up more confident in a mock-backed test than they should be, right up until it fails somewhere that actually matters.

The twenty-minute question

It's easy to treat a flaky test as noise to manage - quarantine it, slap a retry on it, mute it and move on. But almost every one of them is pointing at something real: state nobody's tracking, a race condition in the app itself, an environment assumption that was never written down, or a mock that fell out of sync with the thing it's supposed to represent.

Muting it doesn't make any of that go away. It just makes the pipeline quieter while whatever's actually wrong keeps happening, unwatched, in production.

Next time a test earns a reputation, it's worth the twenty minutes to ask why before reaching for a retry decorator. Most of the time it wasn't lying to you.