DEV Community: Rebeca

Beyond Test Automation: Judgment, Risk, and the Power of Exploratory Testing in Fintech

Rebeca — Sat, 29 Aug 2026 17:05:06 +0000

A green checkmark can lie to you

Here’s a scene that keeps repeating in engineering teams that use AI to write code at scale: the number of changes goes up, the number of automated tests goes up with it, everything shows green, and everyone relaxes. The problem is that green never meant “correct.” It only ever meant “the paths someone thought to check didn’t break.”

When a large part of the code is no longer written only by humans, the gap between “tests pass” and “the system actually works” grows wider. AI is very good at writing automated tests that mirror the same code it just wrote. It checks that the code does what it does, not that it does what the business actually needs. In any system, that’s a risk. In a fintech company, that risk is money walking out the door.

Automated testing was never meant to replace human judgment. It was meant to free up time so people could focus that judgment where it matters most. When test automation becomes the goal instead of the tool, you end up with a lot of tests and very little real coverage of what could actually go wrong.

Why the risk is bigger in fintech

Business rules in a fintech company are rarely simple “if this, then that” statements. They’re full of exceptions built from real regulations, limits that change depending on the country, fraud patterns that depend on a user’s past behavior, and compliance rules that exist because a real case of abuse once forced the company to add them. This kind of domain complexity doesn’t show up by running the same happy path over and over. It shows up when someone asks: “what happens if this amount comes in as zero, or negative, or from a user who was flagged before but the flag already expired?”

That kind of question doesn’t come from typing a prompt. It comes from someone who understands the business, has seen this type of bug before, and knows where money problems tend to hide. That’s judgment, not repetition.

Testing with the situation in mind, not a fixed checklist

This idea has a name: Context-Driven Testing. (I’ve written about this concept, make sure to check it out) It’s simple: there’s no single “best way” to test software that works for every situation. Testing a basic calculator and testing a system that decides in real time whether a transaction is fraud call for completely different approaches, even if both show the same green checkmark at the end.

Applied to a world full of AI-written code, this idea shifts the question from “how much can we automate” to “what, in this specific situation, can only a person with real knowledge of the business actually catch.” This isn’t against automation. It’s automation used with a purpose, aimed at the places where the real risk actually lives.

Deciding what to test first based on real risk, not leftover time

This is called Risk-Based Testing, and the idea is simple to say and hard to follow under deadline pressure: decide what to test first based on how likely something is to break and how bad it would be if it did, not based on whatever order tasks happen to show up in.

In fintech, the questions that matter most when making this call are usually:

How much money moves if this part fails
Would this failure need to be reported to a regulator, or bring fines, or put the company’s ability to operate at risk
How many users or transactions would be affected before anyone notices (what’s often called the “blast radius”)
Can it be undone, or is the money already gone

It’s what turns “what do we test first” from a gut feeling into a choice you can explain and defend when someone asks why one area got more attention than another.

Giving structure to exploratory testing

Exploratory testing without any structure just becomes “I clicked around for a while and it seemed fine.” That doesn’t scale, and there’s no record of what was actually checked. The fix is a method called Session-Based Test Management (SBTM): give the exploration a clear goal (“check what happens to the fraud decision when the risk score changes in the middle of a transaction”), a time limit (usually one to two hours), and a short summary at the end of what was checked, what was found, and what was left out on purpose.

This connects directly to Risk-Based Testing: the areas with the most money and regulatory risk are the ones that get this kind of focused exploratory session first. Lower-risk areas get a quick pass, or get skipped this round, on purpose and written down, not by accident.

I saw this play out myself while testing a feature in a fintech system. Every written test case had already passed and the script said the feature was ready. Instead of stopping there, I spent a few minutes exploring paths that weren’t in that script, just following what felt worth poking at. In under ten minutes, I found a real bug, one that no written test case had ever been designed to catch, because nobody had thought to write a case for it in the first place. That’s the real proof of why exploratory testing matters: it’s the only place where you find what you didn’t know to look for. A script can only catch what someone already imagined, exploration catches the rest.

This isn’t just a one-off story. Test Partners ran scripted and exploratory testing side by side on the same website UAT project, splitting the team so both approaches covered the same system under the same conditions. Over a two-day test window, the exploratory side raised twice as many bugs as the scripted side. Their explanation lines up with what I saw myself: the scripted testers were working off test cases that were already outdated by the time testing started, while the exploratory testers learned the actual system as they went and used that understanding to chase down the areas that mattered most. Scripts can only be as good as the documentation they were written from, and documentation is always the thing furthest behind.

Putting these two things together, a risk-based view and planned exploratory sessions, is what lets a team say, with evidence, where human attention actually went and why. Without that link, exploratory testing is just a nice-sounding name for “we didn’t have time to automate this.”

What still needs a human

AI is a real help here. It speeds up writing the obvious tests, writing repetitive automation code, and even drafting a plan for what to check by hand. The mistake is treating that speed as a replacement for judgment.

What still can’t be replaced is the ability to notice that a business rule has an exception nobody wrote down; to sense that something “feels off” even when there’s no automated check for it; and to decide, based on real money and regulatory risk, where the next two hours of hands-on testing are best spent.

Automation without that judgment behind it isn’t real progress. It’s just volume. And in fintech, volume without judgment is exactly what creates a false sense of security, the kind that lets a serious business logic bug reach real users while every check shows green.

Stop asking AI to generate test cases: how to integrate AI into your engineering workflow

Rebeca — Sun, 19 Jul 2026 18:55:57 +0000

Every time a new feature or fix comes up, some people do the same thing. They open their favorite AI chat, copy the requirement or the PR description, paste it in, and ask "generate test cases for this feature". Then they copy the output back into their test management tool or their notes.

There is nothing wrong with using AI this way, it works. I do think there is something wrong with having a manual flow around it. You have to remember to do it, you have to copy and paste the context every single time, and the AI only knows what you decided to paste. If you forget a detail, or if the feature touches something you did not think to mention, the AI will not know either.

Instead of manually going to an AI chat every time, you can integrate that same LLM into your workflow, so it runs automatically and already has the context it needs.

One example I built recently at work is a GitHub workflow that runs on the PR level.

Every time a developer opens a PR implementing a feature or a fix, we have a label called "ai:generate-test-cases". When someone adds this label, it triggers a GitHub workflow that runs a script calling an LLM. This script reads the full PR diff, the comments, and even external links using MCPs, to gather as much context as possible. Then it drops a comment on the PR with a list of possible test scenarios, the risk level of that change, and whether the developer already added automated tests for it. This way we can focus on the areas that matter the most.

Traditional workflow
Requirement
    ↓
Copy
    ↓
Paste into AI chat
    ↓
Generate Test Cases

Integrated workflow
Pull Request
    ↓
Label added
    ↓
GitHub Action
    ↓
LLM reads diff + context
    ↓
PR Comment with:
- Test scenarios
- Risk level
- Test coverage insights

Test case suggestions are just one output of this. The same context, the diff, the comments, the linked docs, could just as easily be used to flag risk, or check if something important was missed. Generating test cases is just the first thing I built with it.

This is opt-in by design. It will not generate anything on infra related PRs, or any PR where the label is not added. You choose which PRs need test case suggestions. This is useful not only for QAs, but also for developers who want to make sure they covered the main scenarios, and it can surface an outcome or edge case you had not thought about.

Of course this does not replace human judgment. It is a tool to help organize and prioritize scenarios, not a guarantee that every hidden path has been covered. We should always use our own experience to decide what is worth testing and what is not, since it is impossible to test everything.

AI is much more valuable when it becomes part of your engineering workflow instead of another browser tab. Test case generation is just one example of that.

If this is something you want to try or adapt to your own workflow, the repo is public and I am happy to talk through how it works.

What Does 'QA' Really Mean? (Hint: It's Not a Person)

Rebeca — Wed, 27 May 2026 14:30:14 +0000

I'm tired of seeing people refer to QA as simply "the person who tests." And honestly, it's exhausting. As a tester (or QA engineer, if you prefer) I feel the need to clear a few things up.

QA is not a person. It's an entire discipline.

Quality Assurance is about quality as a whole. There are many types of QA across different industries: software, hardware, manufacturing, and beyond. So when someone says a task is "just waiting on QA," it tells me something: that person doesn't take their own code seriously enough.

QA engineers are not the sole guardians of quality.

Saying that "QA is responsible for the quality of the system" implies that developers can simply ignore quality while building it. That's a dangerous assumption. Quality is a shared responsibility and it belongs to the entire team.

And when a company says "we don't need QA," what are they actually saying? That their software doesn't need quality? More likely, what they mean is they don't need testers, and those are very different things. Conflating them is part of the problem.

Replacing testers with developers doesn't work.

Some companies are delegating the "QA role" to developers themselves. I could expand on this on another post, but the short version: developers have confirmation bias. They build things to work, so they test to confirm they work. That's not the same as testing to find where things break. Saying "let's get rid of the QA people on the team" isn't a quality strategy, it's a blind spot.

The "QA responsibilities" debate is muddier than it needs to be.

Everyone seems to have a different opinion on what QA is supposed to do. You've probably heard things like "being in QA isn't just about testing" or "QA needs to wear multiple hats." And sure, testers do much more than find bugs. Documentation, reporting, risk analysis, process improvement, it's all part of the work.

But here's what bothers me: the false promises. Saying that QA is responsible for "ensuring quality and preventing all bugs" is unrealistic and, frankly, unachievable. It sounds good in a job description, but reality is far less dramatic. What we are primarily responsible for is finding problems, and those problems reveal the actual state of the product's quality. That's not a lesser role. That's an honest one.

So where does that leave us?

The solution starts with language. If we're more precise about what we mean: tester vs. QA, shared responsibility vs. sole ownership, risk reduction vs. zero-defect guarantees then conversations about quality become more honest and more productive. Teams make better decisions when they understand what they're actually asking for.

I won't pretend the industry is close to getting this right. But what keeps me going is knowing that good professionals exist, people who genuinely understand these distinctions, who treat testers as strategic partners rather than a final checkpoint. And despite all the noise, I keep showing up and doing my best work. Because the craft matters, even when the title is misunderstood.

Context-Driven Testing: What It Is and Why It Matters Now

Rebeca — Fri, 22 May 2026 01:41:17 +0000

Context-driven testing rejects the notion that there's a universal "best practice" for QA. Instead, it holds that the value of any testing approach depends entirely on the context you're operating in. What works at a regulated bank with 50 QA engineers is actively harmful at a startup with one contractor and no formal QA process.

And right now, that distinction matters more than ever.

AI is creating a false sense of coverage. Teams are generating thousands of test cases with LLMs and mistaking volume for coverage.
Without proper context, you end up with beautifully written test cases that miss the actual risk entirely.

In startup environments especially: apply heavyweight process to every feature and you'll be ignored. Test everything equally and you'll miss what actually matters to your customers.

Imagine a scenario where you have limited institutional power, the company you work for has customers with high ARR who can escalate over your head, and has no established QA process to lean on. Context driven testing gives you a better vision for every decision: rather than just testing the feature, you can ask things like "What's the actual risk if it breaks?" or "Who is most likely to be harmed if we miss this?". Those two questions do more work than any test plan template.

Context driven testing is not an excuse to do less testing, it's actually the opposite, since this is way harder, it requires to develop critical thinking and really understanding the user needs and areas with greater risks, not just tooling.

Do you apply context-driven thinking in your daily work? What's the highest-risk area in your system right now?