A green checkmark can lie to you
Here’s a scene that keeps repeating in engineering teams that use AI to write code at scale: the number of changes goes up, the number of automated tests goes up with it, everything shows green, and everyone relaxes. The problem is that green never meant “correct.” It only ever meant “the paths someone thought to check didn’t break.”
When a large part of the code is no longer written only by humans, the gap between “tests pass” and “the system actually works” grows wider. AI is very good at writing automated tests that mirror the same code it just wrote. It checks that the code does what it does, not that it does what the business actually needs. In any system, that’s a risk. In a fintech company, that risk is money walking out the door.
Automated testing was never meant to replace human judgment. It was meant to free up time so people could focus that judgment where it matters most. When test automation becomes the goal instead of the tool, you end up with a lot of tests and very little real coverage of what could actually go wrong.
Why the risk is bigger in fintech
Business rules in a fintech company are rarely simple “if this, then that” statements. They’re full of exceptions built from real regulations, limits that change depending on the country, fraud patterns that depend on a user’s past behavior, and compliance rules that exist because a real case of abuse once forced the company to add them. This kind of domain complexity doesn’t show up by running the same happy path over and over. It shows up when someone asks: “what happens if this amount comes in as zero, or negative, or from a user who was flagged before but the flag already expired?”
That kind of question doesn’t come from typing a prompt. It comes from someone who understands the business, has seen this type of bug before, and knows where money problems tend to hide. That’s judgment, not repetition.
Testing with the situation in mind, not a fixed checklist
This idea has a name: Context-Driven Testing. (I’ve written about this concept, make sure to check it out) It’s simple: there’s no single “best way” to test software that works for every situation. Testing a basic calculator and testing a system that decides in real time whether a transaction is fraud call for completely different approaches, even if both show the same green checkmark at the end.
Applied to a world full of AI-written code, this idea shifts the question from “how much can we automate” to “what, in this specific situation, can only a person with real knowledge of the business actually catch.” This isn’t against automation. It’s automation used with a purpose, aimed at the places where the real risk actually lives.
Deciding what to test first based on real risk, not leftover time
This is called Risk-Based Testing, and the idea is simple to say and hard to follow under deadline pressure: decide what to test first based on how likely something is to break and how bad it would be if it did, not based on whatever order tasks happen to show up in.
In fintech, the questions that matter most when making this call are usually:
- How much money moves if this part fails
- Would this failure need to be reported to a regulator, or bring fines, or put the company’s ability to operate at risk
- How many users or transactions would be affected before anyone notices (what’s often called the “blast radius”)
- Can it be undone, or is the money already gone
It’s what turns “what do we test first” from a gut feeling into a choice you can explain and defend when someone asks why one area got more attention than another.
Giving structure to exploratory testing
Exploratory testing without any structure just becomes “I clicked around for a while and it seemed fine.” That doesn’t scale, and there’s no record of what was actually checked. The fix is a method called Session-Based Test Management (SBTM): give the exploration a clear goal (“check what happens to the fraud decision when the risk score changes in the middle of a transaction”), a time limit (usually one to two hours), and a short summary at the end of what was checked, what was found, and what was left out on purpose.
This connects directly to Risk-Based Testing: the areas with the most money and regulatory risk are the ones that get this kind of focused exploratory session first. Lower-risk areas get a quick pass, or get skipped this round, on purpose and written down, not by accident.
I saw this play out myself while testing a feature in a fintech system. Every written test case had already passed and the script said the feature was ready. Instead of stopping there, I spent a few minutes exploring paths that weren’t in that script, just following what felt worth poking at. In under ten minutes, I found a real bug, one that no written test case had ever been designed to catch, because nobody had thought to write a case for it in the first place. That’s the real proof of why exploratory testing matters: it’s the only place where you find what you didn’t know to look for. A script can only catch what someone already imagined, exploration catches the rest.
This isn’t just a one-off story. Test Partners ran scripted and exploratory testing side by side on the same website UAT project, splitting the team so both approaches covered the same system under the same conditions. Over a two-day test window, the exploratory side raised twice as many bugs as the scripted side. Their explanation lines up with what I saw myself: the scripted testers were working off test cases that were already outdated by the time testing started, while the exploratory testers learned the actual system as they went and used that understanding to chase down the areas that mattered most. Scripts can only be as good as the documentation they were written from, and documentation is always the thing furthest behind.
Putting these two things together, a risk-based view and planned exploratory sessions, is what lets a team say, with evidence, where human attention actually went and why. Without that link, exploratory testing is just a nice-sounding name for “we didn’t have time to automate this.”
What still needs a human
AI is a real help here. It speeds up writing the obvious tests, writing repetitive automation code, and even drafting a plan for what to check by hand. The mistake is treating that speed as a replacement for judgment.
What still can’t be replaced is the ability to notice that a business rule has an exception nobody wrote down; to sense that something “feels off” even when there’s no automated check for it; and to decide, based on real money and regulatory risk, where the next two hours of hands-on testing are best spent.
Automation without that judgment behind it isn’t real progress. It’s just volume. And in fintech, volume without judgment is exactly what creates a false sense of security, the kind that lets a serious business logic bug reach real users while every check shows green.
Top comments (0)