DEV Community: Alan Richardson

How I AI - Automated AI Interactive Review

Alan Richardson — Tue, 30 Jun 2026 11:01:00 +0000

TLDR; I’ve been using AI to interact with a web application and find defects. And it does interact with the application, keep focus on the specific story context, then finds and raises defects.

The experiment is: how much value can I get from a single Goal oriented prompt in Codex which spins up sub-agents to review a story, and associated PR, to try various interactions on the the deployed site and report defects.

Is that testing? I’m not ready to call it testing yet, but I use a lot of the words associated with testing to describe it. At the moment I think of it as an automated interactive review.

Video Overview

In the video I provide an overview of the experiment, where to find the prompt and examples of the types of artifacts that it creates.

Artifacts:

readable logs from agents
defect reports
defect replication videos
summary reports
issue screenshots

What is this?

I’ve seen a lot of people experimenting with Agentic technology. Letting the AI loose on the application for coding, review, raising issues, prioritising issues, fixing issues, etc.

A good example of Agentic technology in the Testing process can be found here:

I’m not quite ready to that yet.

I’m trying to look at how AI can fit into a normal team process with multiple team members and existing development tools.

Because those are the type of teams I work and consult with.

As such, I’m building a process around Github:

Github issues to represent Stories, Defects, Tasks
Planning using Github Project
Github PR for reviews prior to accepting code
CI pipeline using Github Actions
Deployed Test Environment (using Github pages)
a live environment

You could replace Github with Jira or other tools, but the concept here is - human tools and processes for planning and communication. AI uses those tools to communicate back to humans.

I’ve been using AI Coding Tools to create code. I wanted to see if I could use AI tooling to identify defects through interacting with the application. And then… could it communicate those defects using the Human oriented process.

I’ve been surprised at how successful this has been.

I am documenting all the execution sessions and the evolution of the goal prompt and tooling over at:

https://github.com/eviltester/anywaydata-ai-testing

The main experiment is:

ai-review-goal.md

And session execution results in:

/docs/testing

Process

Create Github Issues to track work
- have AI work on the Github issues using branches for the stories
When ‘done’ create a PR to review the work
- I also have various review tools scanning the PR but that requires a different blog post.
Trigger the experimental Goal to review the story, review the PR, interact with the application on the test environment using Playwright and report back finding relevant to the issue and PR.

The prompt is a little larger than this, but that’s the gist.

The process requires that the AI coding tooling can access Github.

I’m using Codex so it has Github capabilities built in, but it also uses HTTP to access issue JSON feeds, and it uses local Git and Github tooling for pulling code and writing comments on Github.

I’ve also setup a variety of MCPs. Playwright MCP and Chrome Dev Tools MCP - this is to allow interaction with the application using a browser.

Playwright, node etc. are all installed on my machine so it also writes adhoc scripts to automated the application and record screenshots and videos as it works.

I currently run this on Codex using 5.5 on High reasoning in fast mode - so if I was using an API plan to do this it might be quite expensive.

At the end of running the session the AI tooling finishes by:

collating reports and logs into PDF
reporting the results by creating a Github issue in the target folder
creating sub-issues for any defects found

All reports are written to the anywaydata-ai-testing project.

Videos and support files are left on my local machine - because I’m trying not to clutter the Github repo with too much temporary information. But if this was working with Jira or a team environment then I’d either have the defect videos as attachments or find a way to share the videos across the team via some video sharing tool.

Hmmmm, idea, have a prompt that uses ffmpeg to create a video summarizing the execution session and including all defect reports and upload to YouTube or some other video hosting. This isn’t far fetched because I have used ffmpeg to create videos of defect replication, and I do have other experiments for creating videos automatically.

I initially had the AI creating replication videos with ffmpeg but then had it use the Playwright APIs and now uses the Playwright screencast API (which has automated chapter points to explain what is happening).

Results

Every time I’ve run this, it has found useful defects.

Some that I might not have fixed in the past because they seem low priority, but with AI, a fix is a prompt away so it isn’t too much work to fix most items.

Some I might not have fixed because they are low priority design decisions, but… AI seems biased in its exploration patterns and keeps identifying the same type of issues, so I fix them or re-design the system to make it easier to use and understand. Something I might not have done for primarily human use - because humans can adapt more easily.

I do not let the defects flow directly into another AI session. I triage them on Github and close those that are unimportant. I also group them to identify which ones are actually related to ‘design’ and which are issues.

I don’t let the AI act on them directly because I don’t know what decisions it would make to ‘fix’ those reported issues which are design related. I did try this in the past but it made what I considered to be the ‘wrong’ design decision.

Each execution session takes about 40 minutes for the AI tooling to complete and has resulted in 4-6 issues being raised each session.

Human in the loop for decision making.

Logs

Each agent writes log files e.g.

These vary in style and consistency, some are slightly malformed for viewing as Markdown in github. But they are intended for Human consumption.

The AI tooling is prompted to write timestamps, intent, actions and results.

I tend to sample these reports. Some are more interesting than others.

I find the logs most useful when they have tables of sections of data because I can often spot issues in the data and results that the AI doesn’t identify as an issue.

Evolution

The goal prompt I’m using evolves after each execution because I see something that didn’t work well enough, or I want a different output.

When I have an idea of what I want it to do I experiment with prompts before adding into the goal. e.g. when I wanted to have videos of defect replication I interacted with the AI tooling to find different approaches.

a simple - create a video of the defect worked
but then I wanted captions and explanation
it first used ffmpeg to edit an existing video, and it worked but took more time, felt risky for flakiness and I thought there should be a better way
then tried Playwright with the CDP API and that created good videos with the intent clearly visible but seemed complex
then found Playwright screencast API which creates slides with a delay through a simple supported API and so settled on this. The output was not as good as the direct CDP API control, but it seemed simpler and more reliable.

I’ve also added an agents.md to try and prevent the AI from committing support material and videos. It tends to bypass guardrails like .gitignore and so I’ve also add pre-commit hooks.

Comparison to commercial tooling

I haven’t used a lot of AI Commercial Testing Tooling. The evaluations are too small or limited, the tools are in beta and don’t work effectively on some environments, they don’t fit the style of approach I want to use.

Of the tools I’ve tried. I’ve had 1 or two points of action, but not as much value as I’ve had from this goal.

Commercial tooling offers a nicer UI, but I can’t imagine it is using as heavyweight an AI model so it has to have a better surrounding set of co-ordination activities, prompts and supporting tools to get the most out of this.

The tools I tried did not hook into Github and a normal human process… or if they did I didn’t see this in the limited evaluation time or it wasn’t accessible during an evaluation.

Most commercial tools seem to be described in terms of ’testing’ and ‘self-healing’ i.e. they identify paths through the application and repeat those paths, ‘self-healing’ where the paths fail due to changes, then they might add a few more ’tests’ for new functionality in the app.

I use AI for all of the above, but I create normal test execution coverage using Playwright, Selenium, Testing Library, API calls, Domain/Payload/Page objects, and maintain this as a normal part of development.

It is pretty easy to keep your automated execution coverage up to date and running clean if it is updated in sync with the application updates.

What I really want from the additional tooling is information. Insights related to the context of the work that is not covered by the automated execution coverage.

I hope to evaluate more commercial tools in the future because I hope they have a more optimized process than using a prompt.

But, I need to know the capabilities of the existing AI Coding tools prior to adopting or recommending any commercial tool so that I know the commercial offering has benefits.

Try it Yourself

Since this is just a /goal prompt it can be used on any AI Coding or CLI system.

They all have their own nuances, which hopefully you’ll know how to adapt the prompt for.

I use Codex, so this is built around a goal which is the current Codex syntax which is most ’loop’-like.

If your AI Tooling can:

Use Playwright MCP
Use Chrome Dev Tools MCP
Make HTTP calls
Interact with Git and Github

Then you’ll be able to use this concept or adapt the prompt to your needs.

I’m making all my results and experimentation public here:

https://github.com/eviltester/anywaydata-ai-testing

While I build and develop AnyWayData:

https://github.com/eviltester/grid-table-editor
https://eviltester.github.io/grid-table-editor/
https://anywaydata.com
Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

How I AI - A Working Position on 20260626

Alan Richardson — Fri, 26 Jun 2026 11:01:00 +0000

TLDR; I’ve been using AI. I’ve changed my approach a lot as I’ve gained experience. AI has allowed me to experiment with more tools and process, and build more, than I could possibly have done without it. I’m experimenting to find new ways to use it better.

My use of AI evolves. Every day.

I use AI. I learn what worked and what didn’t. I change the way I work. How I use AI is not fixed.

This post is a snapshot summary of what I’m currently doing and I’ll provide links to experiments and results of those experiments later. I’ll also point to a prompt I’ve been using to interact with an application and find defects. And I’ve listed some of the AI tools I’m gaining most value from.

I hope to create more posts that describe the path I used to get the the current way I use AI, lessons learned, and how I’m using AI in more detail in future posts.

This post was written by a human. Based on the human’s experiences of using AI to get things done.

I’m The Human. Me. I wrote this.

How I used AI this Morning

I don’t think I have time to list all the activities, and products, and ways, that I’ve used AI over the last 6 - 8 months so it is probably easier to start with a snapshot of how I used it this morning 26th June 2026.

This morning I used AI for:

work on Project A
- automatically reviewing a PR
- respond to and fix PR request comments
- interactively use Playwright to explore changes related to a PR and create some defects
- write code to fix the defects raised by the earlier AI process
- auto create videos when I asked AI to check if a defect was replicable
work on Project B
- fix some feature nuances and make the app work the way I want to
- investigate why downloads for some content were not working as well as they should
personal development
- summarize 20 new podcast episodes so I can review them and see if anything is useful
- search the web and summarize news stories on a variety of topics

My role on Project A this morning - Product Owner & Manager

My role on Project A? On Project A today I was the Product Owner and Engineering Manager.

I triaged the PR comments and closed any I thought were not relevant prior to the AI review
I triaged the defects and closed any that I thought were not relevant and added ‘fix it this way’ comments for those that could be approached ambiguously
created new stories to expand functionality and move the project forwards
adjusted the way the AI interactive review process works and I’ll see if it is improved later as a result
this afternoon I hope to test and review the changes (But it’s so hot! And I’m writing this post.)

Project A - this morning was primarily interactive review coverage for work that yesterday I was the Staff Engineer. Yesterday, I reviewed the code and architecture decisions, and specified the work since it was a technical re-architecture story.

Product A is AnyWayData.com and I’m approaching this as a professional development project and trying to learn how AI can work for a Software Development Team.

My Role on Project B this morning - User

My role on Project B? On Project B - I was the user. The product is fairly mature now and is in daily production usage, by me. I use the product, and when I see something that can be improved, I tap the AI on the shoulder and say - can you improve this… then it does as I continue working… then I test the improvement and keep going.

Product B is an internal tool, but it is the basis for the podcast summaries on:

This is ’less professional team’ and more ‘solo dev’ but has automated coverage, code reviews, CI, database, CDN, etc. etc. I interact with AI differently when building these products.

Multiple Roles

How I use AI varies from day to day and I’m still learning what it is capable of.

I have to adopt all the different Software Development roles when I develop with AI.

I do not believe we will one day all just be Project Owners. It seems more likely that to get the most out of AI for Software Development we will need to learn how to do all of the processes in Software Development to some extent.

Teams that adopt AI can have multiple people and each person will adopt multiple ‘senior’ roles, while the AI takes on the junior role.

The smaller you make the teams. The more roles each team member has to learn how to do.

The faster you move, the more you have to trust the AI processes you put in place. So only move as fast as you trust the process.

As the Product Owner for the AI:

I come up with the ’next thing to do’
I write the acceptance criteria
I plan the scope of the product

I have to act as Senior Engineer:

I’m responsible for the architecture and coding principles that are used.
I’m pushing for more tooling to help improve quality and speed of delivery
I’m pushing for a better build process and better environment release process
I review the code because the AI cheats and takes short cuts when it writes code and if I don’t spot those in a review it impacts long term development

I have to act as Tester because:

the AI is getting better at finding defects by performing focused exploration on the changes made but still not good at managing coverage
the AI still operates at a fairly junior level and has biases in its approach
the AI often makes mistakes when communicating a defect and I have to identify the root cause
the AI can simulate some of the activities that a Tester performs, but it’s an AI, not a Tester.

I have to act as Senior Automator because:

AI writes horrible automated execution code without guidance and review
I have to own the standards that we automate to

I have to act as Manager to prioritise the build work because:

The AI doesn’t know how to prioritise
The AI doesn’t know how to interpret nuance

When I say “the AI doesn’t know how to…” I’m really saying “I haven’t found a way to delegate this to the AI such that I would trust the output without getting involved.”

I’m still learning how to work with AI and I think I’m learning how to use it better every day.

Can AI be a team sport?

When I first started working with AI, and when other teams around me were working with AI, it didn’t feel like it would fit in well to a team process.

People using AI had a hard time getting AI generated PRs through a Human Review.
It was hard to reign in AI and keep it focussed on a single section of code so PRs would expand and merge conflicts were plentiful.
AI seemed to rely on specs and docs in the code and didn’t integrate with any Human Tooling.

Now, having used AI for some time. I’m trying to add more ’team’ structure to the process.

This means I’m forcing AI to adapt to a human communication process more than me adapt to what it needs.

My current approach to AI on AnyWayData is one that I think can be used in teams.

Create Github Issues for the work to do with acceptance criteria
‘plan’ with the AI around that issue - AI tooling can read the Github issue directly so I don’t need to cut and paste into the system
AI can write the code
AI can write supporting automated coverage
AI comments on Github with Summaries and artifacts from the work done
Create PRs for changes
AI Tooling automatically reviews the PRs and leaves comments
AI and Humans can review the comments and triage them
AI can read the PR and resolve comments
Frequent commits mean we can have multiple restore points or commits to cherry pick from
AI can be pointed at an issue and a PR to provide a narrower context to interact with the application and identify defects in the running system, not just code review

As a human, I’m involved in all those steps. And I’m moving some of my review work into Github, just like I would do on a human team. I’m trying to learn how to fit the AI into the process so that Human work remains central for evaluation and decision making.

Can Humans keep up?

I see a lot of tool vendor statements that AI is now producing code so fast that humans can’t keep up.

And therefore we need to have a new breed of tools that do other things fast.

Really?

If I can’t keep up with the code creation, how will I find time to also keep up with the AI generated output from the other tools?

And if the other tools can do things fast, won’t that allow the code creation to move even faster? A continually accelerating cycle that will eventually move so quickly the output will evolve faster than anyone can use. A process limited only by the heat sinks on our machines and the speed of our deployment process.

Humans can keep up.

If the humans make the AI work at the speed and level of quality that they can handle.

We can choose to create with AI in ways that we can keep up.

There has to be a better marketing tag line.

I am available for marketing consultancy, contact me here.

If Humans can keep up then we aren’t maxing out our use of tokens and are wasting money!

We might then argue that if puny humans can keep up, then we aren’t making the most use of AI and are letting tokens go to waste.

The response is not - then everything must go fast - it could be - “you’re paying $200 a month, that’s less than $10 a working day. and if you are getting more than $10 a day in value from the AI tooling, then you’re doing fine”.

Decide on the monetary return in value before deciding you are wasting money. Experiment to increase the return in value and improve gradually.

Perhaps use those extra tokens that you are ‘wasting’ for side investigations that you don’t have to keep up with?

Here are some side-investigations I performed this week. Because I wanted to and not just to ‘prevent tokens going to waste’.

AI evaluated an ‘open source’ test tool that I’d seen mentioned while researching things I heard on a podcast. Tried to get it working. Amended code to bypass some hard environment dependencies that limited its use as open source. Provided in depth analysis of its features and how it worked.
AI evaluated 5 or 6 static analysis tools that I don’t have on the project. Downloaded, installed, used, timed the performance, compared the features, described the impact of adopting them, ruled out those that wouldn’t work.
AI investigated and implemented CDN caching for one of my apps to reduce hosting cost.

I now have decided not to spend time on the ‘open source’ test tool. And have made some decisions about tooling migrations that I will prioritise next week. And I think I’ve saved a whole $4.93 in hosting costs per month.

There are ‘other things’ the AI can do, in parallel, that don’t force the humans to go faster.

Focus and Experiment at the same time

Our focus on projects has often been - prioritise, and focus on 1 or 2 achievable clear goals, because we can’t do everything.

Now we can do more.

Experiment.

Have the AI tooling do things in parallel that we would never get the chance to do. While we maintain a focus on 1 or 2 achievable clear goals that we can pursue at a sustainable pace.

Use the extra tokens to experiment:

Some of those experiments will probably end up being throwaway.
Some you will forget to look at - I can see in my Codex threads that I have an experiment from a month ago that I have yet to try (no, I did not just inadvertently provide evidence that humans can’t keep up).
Some you will prioritise to take forward at a sustainable pace.

Some of those experiments will also be on how far we can push AI and our process.

My use of AI is evolving. I might even learn to trust the more Agentic styles of development. I currently don’t. But I also haven’t had time to experiment with them because I’m getting things done with my current evolving approach.

Where can I see some evidence and output?

I’ve been using AI to expand AnyWayData.com and I’m pushing AI to see how much I can use AI, how effective AI is for development and experimentation, and not just - how fast can I expand the functionality.

Prior to May 2026 AnyWayData was a legacy human written project with no real automated execution coverage.

I initially used AI to finish migrating the tool’s data grid from AG Grid to Tabulator. I can see from commit history that I had started work on that in June 2025 and intermittently worked on it until December 2025. Then stopped. I remember it only took a weekend to finish the migration when I started back working on it with AI - of course, I, The ‘Puny’ Human, had done the hard work to architect the abstraction layer required to make the AI step easy!

And then I started expanding AnyWayData to experiment with the process and technology of AI and learn how to build more professionally with AI.

Using AI I’ve added:

Storybook - to help refactor to UI components and introduce a lower level of interaction coverage.
Playwright for automated API and deployed UI coverage
Dockerisation of the MCP, and API
An auto-deployed test environment
- https://eviltester.github.io/grid-table-editor/
Improved Github pipelines
Features that I had been experimenting with but never finished e.g n-wise data generation
Experiments with WebMCP and in browser AI APIs
An experimental AI interactive review process which reports defects
- https://github.com/eviltester/anywaydata-ai-testing
and more…

Looking through Github commits I’m surprised that I only started using AI on the project extensively at the start of May (end of April) 2026. Because I’ve done more in that time, than I could possibly have done on my own without the support of AI. I didn’t realise I had expanded the functionality quite so much and quite so quickly - that’s a pretty good secondary gain.

Over that time I’ve used:

all the free AI chat systems
OpenCode
Continue.dev
Co-Pilot
Claude Code
Codex
OpenRouter

I currently use OpenCode (for manually triggered reviews) and Codex (for everything else). And for the first half of the month I use Co-Pilot credits to review PRs until they run out (I suspect co-pilot might not be in my tool list much longer).

I use OpenRouter as my AI API marketplace, rather than run all models locally.

I also use the open source plans of the following tools for AI Automated PR code reviews:

You can see review comments from both of the above on my recent PRs.

I’m also using the project to try and experiment with other tooling.

I’ve even tried a few commercial AI test execution tools on the project but the trials are often too limited or I just can’t pull value from them. But since the tools evolve quickly I look forward to trying more commercial tools in the future and revisiting those that I have tried earlier.

I’m trying to make as much of this work as public as I can, and as visible as I can, to make it easier to pull lessons from.

https://github.com/eviltester/grid-table-editor

If you look at the PRs in the project then you’ll see the comments generated by the AI review tools that I’ve been using.

And I think, if you looked through the most recent closed issues and PRs you can get a feel for how I’m trying to use AI in a process that could work in a team environment.

I haven’t really been blogging about this.

I’ve been describing progress in Patreon posts because I’ve been experimenting as much as I can, rather than reporting on work in progress that was too early to comment on properly.

Can AI Test Stuff?

I’ve been evolving a prompt that I have found very useful as part of my testing process over at:

https://github.com/eviltester/anywaydata-ai-testing

It’s just a prompt. It is not a new revolutionary AI Testing framework.

It was easy to install:

configure Playwright MCP
or Chrome Dev Tools MCP
paste in prompt to Codex

I wouldn’t call it testing. My mind frames it as an Automated Interactive AI Review.

I’m adding the output from the process in the repo to make it easy to see the output and you can review to see if it might add value. Hint: it’s in the main readme.md

In the prompt I use the language associated with testing because I’m trying to set the context for the AI such that it communicates and operates as a simulation of a Tester and I’ll see how much value I can extract from it.

I will say that in the last few days, this has generated more valuable defects for me than any of the AI Test Tools I’ve used.

Have a look through the repo, experiment with the approach and see if it adds value.

Can AI Test Stuff? On its own? No.

Can we use AI as part of our test process? Yes. Hopefully the repo above provides a simple example of an experiment to try as part of your test process and you can use it in any tool.

Have a look through the AnyWayData code:

https://github.com/eviltester/grid-table-editor

And you’ll see I’m using AI to generate a lot of execution coverage, using multiple tools.

I use Page Objects to model the UI. I find that works better when the AI tooling expands coverage.

I did try and use the Playwright MCP to generate execution code but it was… not acceptable to me.

Hopefully there are some jumping off points here to see some examples to learn from and some of the experiments I’ve been running.

How to Learn AI?

The current approach that I’m using now feels very different to the experiments I was running when I started to learn AI.

I documented a few of those experiments here:

https://github.com/eviltester/ai-supported-testing-experiments

So the way to learn AI is:

pick an AI tool
use it to try and do something
look at what didn’t work
look at what did work
try to do the thing again, but better

Experiment and iterate.

And… the more you can share what you are doing, the more everyone else can learn as well.

Don’t be afraid to pursue a different direction to everyone else.

Your natural approach will probably be completely different to someone else’s. You’ll do experiments that other people won’t.

I’m learning from the excellent work that Dragan Spiridonov is doing, and the comprehensive information he is sharing.

But… I’m experimenting with approaches that fit my natural approach, and seeing which add value to my process.

Everyone working with AI at the moment is learning.

Start with the first experiment that makes sense to you and keep going.

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

A Web Testing, Automating and Tooling Masterclass

Alan Richardson — Thu, 05 Feb 2026 08:30:00 +0000

TLDR; We can only test to the level supported by our Ability, and the degree to which we are supported by tooling to Observe, Interrogate and Manipulate the System.

Video

Watch on YouTube

00:00 Introduction
00:26 Application Under Test
02:51 Live Testing
03:41 Testing vs Automating
06:03 Testability vs Automatability
09:25 Workarounds
10:55 Technical Knowledge
15:44 Test Approach - JavaScript
17:02 Exercise - Test Approach
17:24 Automated Execution Approach
18:34 Exercise - Code Review
20:35 Exercise - Your Project
20:59 Server Side
24:20 Exercise - Test Approach - API
24:52 API Tooling
29:14 Exercise - API Testing
29:34 Automating the API
31:54 API Interacting with UI
37:09 End Notes

Introduction

A customer has reported that one of your pages doesn’t work. But your automated execution coverage tells you that it does. What went wrong?

That’s what we’re going to explore now.

difference between testing and automating,
testability vs automatability,
we’ll think about test coverage from a technology perspective,
and we’ll look at some tools that can help you test.

Here’s the Application Under Test.

https://testpages.eviltester.com/challenges/synchronization/infinite-scroll/

Infinite Scroll Challenge on the TestPages

Many sites have infinite scroll. You scroll to the bottom and it loads new items.

That functionality works. We have have automated coverage. That passes.

What’s the issue?

Oh yeah, sorry I forgot to mention. I’m going to encourage you to think. Ideally we would do this as a workshop, as hands on training, but it’s a video and text workbook.So all I can do is prompt you into thinking.

Contact me if you are interested in live training for your team or organization.

So what’s the issue?

Well, I can’t click the button. The page refreshes and scrolls too quickly and the button isn’t enabled for long enough when it is visible.

This happens with a lot of infinite scroll sites, there are footers that can’t be clicked, stuff at the bottom of the screen that you can’t see.

This did work at some point, because we tested it, but I guess someone changed the time out value and the automated execution assertions didn’t flag this as an issue.

So let’s start thinking about this from a Testing perspective.

Live

If this was a live project we could just stop here and raise a defect.

Obviously the timeout is wrong. Raise the defect. Move on to the next issue.

But wait.

We have automated coverage, and it didn’t highlight this issue.

We should think about the difference between Testing and Automating.

Testing vs Automating

Testing is what we do. As humans, we interact with the software and we observe, we build models, we learn, we compare the actual interaction with the models. We report information derived from the difference between our models and the observations.

When we investigate an issue and expand our models we interrogate the system more deeply to learn what’s going on.

Testing is the human activity of interacting, observing, thinking, learning, experimenting.
Automating, is the human activity of making the interaction and observation of the system an automated process. And that includes the comparing of observed results with the expected results.

Automating is a human activity which results in the output of an automated execution process. The only involvement with the human after the activity has been automated is to investigate the failure reports and maintain it when it fails.

So both Automating and Testing are human processes. But testing leads to more human processes, automating leads to an automated process that only involves humans when it fails.

NOTE: AI might change how I view the process of automating. Certainly I’ll need more distinctions in how I describe nuances around Automating, but for now, all the automating I do is Human Initiated, or Directed, and results in some automated execution process.

Testability vs Automatability

And let’s just quickly look at Testability and Automatability because these words are used badly when describing software.

Is this system testable?

Yes. I can access it in the browser, I can see it, I can interact with it. The button toggles too quickly but I can test the application.

But when we talk about testability we often talk about, adding ids to the elements, make it it more observable, etc.

Well the ids, are really for automatability, not testability.

In the browser, the browser tooling makes it observable because of the technology used, not the system.

Most of the time when we are talking about testability we are really talking about automatability.

This application has been built with automatability in mind.

It has ids, there are classes, the JavaScript source is visible, it is easy to change the state and configuration variables, this thing is so easy to automate and observe. But none of that was required to help me test it.

Testability is not the same as automatability. Keep that in mind as we continue through this process, particularly when we move to the server side interaction.

With the JavaScript Infinite Scroll system, the main thing that impacts my Testability, is the bug, preventing me from testing the button functionality.

Workarounds

Let’s quickly consider workarounds.

Your ability to find workarounds will impact your ability to test the system.

So your ability impacts the Testability.

Testability is as much about your ability to test the application as it is the application supporting you in testing it.

Depending on how deep you want to go. You can only test the application to the limit that you can observe, manipulate and interrogate the application.

Your ability to test the application is often impacted by the usability of the application, and that is true here. The application is not usable for one main function so it is hard to test that function.

But, we can work around that, for this application, with more Technical Knowledge.

Technical Knowledge

When working with the web, we need to understand our tool capabilities.

My tool at this point is the browser.

What can I do with it?

I can view the page source.

I need to be able to understand HTML. Some CSS. Some JavaScript.

Having the tooling ability to look at the source doesn’t help me unless I understand what I’m looking at.

So if you’re testing web applications, you probably want to understand:

How a browser works,
How HTML works,
What is CSS and how we use it,
What is JavaScript and how it works.

At the very least to have a reading ability of these technology artifacts.

So at this level, at the source, with the technical knowledge I have… I can see in the JavaScript some variables, I can see they are amendable.

But, I don’t have the tooling to amend it yet.

So, what else do I have available?

I can look at the dev tools.

I can see the DOM view shows me much the same as the source. But… it can be different.

The source is what we gave the browser to work with.
The DOM is what the browser created after interpreting and executing the source.

In the DOM I can see the JavaScript as well.

I also have the ability now, to interact and manipulate the JavaScript variables that I observed in the source.

There are a few timeout and millisecond variables there.

Perhaps the bug is the scrollAfterMillis variable?

Let me change that to 3000.

scrollAfterMillis

And… now it just takes longer to refresh, but I still can’t click the button.

Let me try the preLoadTimeout:

preLoadTimeout=2000

Now, the button stays active for 2 seconds before it loads the next set of data. That’s my workaround to make this application testable.

It also means that I can go a little deeper in the bug report and say that the root cause is the preLoadTimeout being too small a value so the user doesn’t have time to click the button.

You might be interested in this video and blog post covering more details of Chrome Dev Tools for Testing.

Test Approach

So what is my test approach for this?

load the page
scroll down
see that it refreshes and adds the data I expect
click the button
see that it doesn’t refresh when I scroll

Is that it?

Do I need to reload the page and check that it starts auto refreshing again?
How long do I need to wait after stopping it, and trying to scroll to make sure it doesn’t start auto-refreshing again?
What else?

Exercise - What is your test approach?

After this video I encourage you to think through what your Test Approach for this application would be.

Test Approach vs Automated Execution Approach

Once you’ve figure out how to test it, how would you automate it?

I’ll show you what we have here.

And this test passes.

@Test  
public void scrollToStopLoadingAndClick(){  
WebDriverWait(driver, Duration.ofSeconds(10)).until(  
        ExpectedConditions.elementToBeClickable(page.getStopLoadingButton()  
    ));  

    page.getStopLoadingButton().click();  

    new WebDriverWait(driver, Duration.ofSeconds(10)).until(  
        ExpectedConditions.textToBePresentInElement(  
            driver.findElement(By.id("statusMessage")),"Clicked")  
    );  
}

Is that good enough?

It works. It passes.

But it didn’t highlight the fact that the halting functionality is unusable for a human.

This is one of the issues when we automate something.

We automate the functionality.

We don’t automate the experience.

If you are interested in more details about WebDriver with Java then have a look at this video Masterclass on the basics of WebDriver with Java

Exercise - Critique the code

So after the video, or now, pause the video.

Critique this code:

Understand what it does
What does it not do?
- I can see it doesn’t try to re-scroll the page to make sure that clicking the button stopped the auto-scroll.
- What else does it not do?
- What conditions does it not check?
Should it match the user experience?
- How could it?
- What would you change to match the user experience?

Much automated execution coverage does not match the user experience. When it doesn’t then there is a risk that the automated execution passes, but the user experience doesn’t.

And there is a risk we don’t notice.

Testing will hopefully identify those issues. But automated execution often does not.

Exercise - Consider Your Project

Is there a risk on the projects you work with that the automated execution covers the functionality, but not the user experience of that functionality?

Server Side

OK, so that was the JavaScript version of the Infinite Scroll.

It offered us some scope for thinking like the tester.

But let’s push this a little further.

We also have the server side infinite scroll.

https://testpages.eviltester.com/challenges/synchronization/infinite-scroll/server-side/

It looks the same. It has the same bug, the button is too fast. And my automated execution passes.

But this page, when it needs new items to display it calls the server to get the information back.

Do you trust that statement?

How do you know it calls the server?

Visibly, in the browser, when I test it and observe the output. It looks the same.

We need to be able to be able to interact with the application from a technical perspective.

What if I just think it is connecting to the server, but I accidentally release the wrong version? What if I just renamed the file?

So we need to check.

We need to observe the application at multiple technical levels.

So let’s look in the dev tools again. And this time we’ll look in the network tab.

Filter by Fetch/XHR to see the requests made by JavaScript and I can observe a call to moreitems

Let’s interrogate that request

I can see a JSON response:

[
    {
        "id": 1,
        "text": "This is content item number 1. Scroll down
         to automatically load more content when the
          \"Click to Stop\" button becomes visible."
    },
    {
        "id": 2,
        "text": "This is content item number 2. Scroll down
         to automatically load more content when the
          \"Click to Stop\" button becomes visible."
    },
    ...
]

Great.

So I have tooling to increase my ability to test this.

But I need to know about:

HTTP Requests
Fetch and XML Http Request
JSON

My ability to test this application will be limited if I do not have that technical knowledge and if I do not know how to use the Dev tools network tab.

But at least I know now that it is making server requests.

I didn’t have to trust anyone. I could verify that this was true.

So now, our testing scope just expanded.

Exercise - Server Side Test Approach

Now, what do you have to test?

Is it good enough to just test the front end now but scrolling up and down and clicking the button?

After watching this video, take some time to think through what you want to test to cover the Server Side infinite scroll.

Do we also have to test that HTTP call?
How much do we have to test it?

Server Side Test Tooling

And… do you know how to do that?

How can you amend the HTTP requests?

We can do some of that from Chrome.

I could:

copy as cURL
copy as fetch

with cURL I can use the command line.

Or fetch I can amend it in the console.

I could take the cURL and paste it into an API tool.

I can do that in Bruno by creating a new request from cURL.

Or in postman I can paste in the cURL command into the new request.

I can experiment with it in these tools.

It is important to test the API on its own.

For example, when I built this, it was only when I was automating and testing the API, that I realised that I really need a limit on the count. If I let people make a request and ask for 6 million items back. That could easily bring down my server.

NOTE: you can find a list of API Tools on the API Challenges site.

Exercise - API Testing

As an exercise, think through what conditions you would want to test on the API, then you can use any of the tooling approaches mentioned to experiment.

Automating The API

I would probably want to automate the API as well.

This does remove us from the user experience because the responses from this API are normally handled by the UI, so just because we see something working in the API we can’t assume that it works with the API interacting with the UI.

I used RestAssured to automate the API.

The coverage runs at the same time as the web UI coverage.

And I have a lot more coverage at the API level, than I do at the UI level.

Think about what coverage you would add for the API.

Then try to automate it. I used Java, with RestAssured, but you could use any library or programming language you want.

Or you could even have a set of canned requests in Postman or Bruno or any of the other API tools.

For production API automating, I primarily use code and HTTP or API libraries. I don’t tend to use tools like Bruno or Postman.

But for production API Testing, I do use Bruno and other tools. Because I like the flexibility and the ability to see the requests being made.

Testing and Automating often sound like the same thing. But they have different aims and are supported by different tooling.

API Interacting with UI

And now I’m in the position where I’m testing the API in isolation of the UI.

Is that a risk?

For example, I don’t know how the front end handles a 500 error response. I can see it is the same JSON format but it is a different status code, does that make a difference?

[
    {
        "id"=0,
        "text"="For input string: '-1.02'"
    }
]

Would you test that?

Do you know how to test that?

One way to do that is to use a Proxy.

Intercept the request, amend it to be one that triggers an error and play it through to the front end.

And then I can see if the system handles error responses or not.

I would either use Zap or BurpSuite.

For this exercise I would use ZAP.

open a session using Edge
create a new context for testpages.eviltester.com
filter the history to “Show only URLs in Scope”
Set the breakpoint on all requests and responses
trigger a scroll through the UI
amend the request
see the result in the UI

I can use tooling to increase my ability to observe the system and manipulate the system at different technology levels.

NOTE: A list of recommended HTTP Proxy Tools is available on the API Challenges site

End Notes

It is often surprising how much depth we have to test and automate when testing even simple pieces of functionality.

The more that we extend our technical ability to cover the multiple levels of the application, and we learn how to use tooling to help us observe, interrogate and manipulate at those different technology levels, the more we can expand the coverage of our testing.

Automating is not the same as Testing. Both are human processes, but the output of automating is not the same as the output of testing.

But yes… we can be testing, as we are automating, we may well learn things during the process of automating the application. But we should not confuse the continued execution of the output from automating with testing.

Testing can miss things because we are human, or we may not have covered the conditions, or we may not have gone deep enough into the system.

Automated execution can miss things because we humans forgot to add the conditions, or we didn’t assert enough. But automated execution can also miss the human user experience and tell us things are working, when they clearly are not.

I hope that you do now go off and do the exercises yourself. You may not have all the skills to do this yet, you may not have tried all the tools. That just means you can revisit this exercise and application multiple times as you grow your knowledge and skill set. And repeat it when you want to evaluate or learn new tools.

That’s why I created the Test Pages, and that’s also why this is a fairly high level overview.

Remember to work through the exercises.

Exercises

The Applications Under Test

Exercise - What is your test approach for JavaScript Infinite Scroll?

think through what your Test Approach for the JavaScript Infinite Scroll application would be.
Do you need to reload the page and check that it starts auto refreshing again?
How long do you need to wait after stopping it, and trying to scroll to make sure it doesn’t start auto-refreshing again?
What else?
- what conditions would you cover?
- how would you approach the testing?

Exercise - Critique the code

Understand what it does
What does it not do? - I can see it doesn’t try to re-scroll the page to make sure that clicking the button stopped the auto-scroll. - What else does it not do? - What conditions does it not check?
Should it match the user experience? - How could it? - What would you change to match the user experience?

Critique this code:

public class InfiniteScrollTest {  

    static WebDriver driver;  
    static InfiniteScrollPage page;  

    @BeforeAll  
    static void setupWebDriver(){  
        driver = DriverFactory.getNew();  
        page = new InfiniteScrollPage(driver);  
    }  

    @BeforeEach  
    public void reload(){  
        page.open();  
    }  

    @Test  
    public void scrollToStopLoadingAndClick(){  
            ExpectedConditions.elementToBeClickable(page.getStopLoadingButton()  
        ));  

        page.getStopLoadingButton().click();  

        new WebDriverWait(driver, Duration.ofSeconds(10)).until(  
            ExpectedConditions.textToBePresentInElement(  
                driver.findElement(By.id("statusMessage")),"Clicked")  
        );  
    }  

    @AfterAll  
    public static void closeDriver(){  
        driver.close();  
    }  

}

Supporting Abstractions:

public class DriverFactory {  
    public static WebDriver getNew() {  

        WebDriver driver;  
        ChromeOptions options = new ChromeOptions();  
        options.addArguments("--disable-smooth-scrolling");   
        driver = new ChromeDriver(options);   
        return driver;  
    }  
}

public class InfiniteScrollPage {  

    private final WebDriver driver;  

    public InfiniteScrollPage(WebDriver driver) {  
        this.driver = driver;  
    }  

    public void open() {  
        String url = SiteConfig.SITE_DOMAIN +
         "/challenges/synchronization/infinite-scroll/";  
        driver.get(url);  
    }  

    public WebElement getStopLoadingButton() {  
        return driver.findElement(By.id("loadMoreBtn"));  
    }  
}

public class SiteConfig {  

    public static final String SITE_DOMAIN = "https://testpages.eviltester.com";  

}

Exercise - Consider Your Project

Is there a risk on the projects you work with, that the automated execution covers the functionality, but not the user experience of that functionality?

Because if that’s a risk, you might want to revisit your test approach and your automated execution approach.

How might your approach need to change?

Exercise - Server Side Test Approach

What do you have to test now that the Server Side calls are involved?

Is it good enough to just test the front end now but scrolling up and down and clicking the button?
Do we also have to test that HTTP call?
How much do we have to test it?

Exercise - API Testing

As an exercise, think through what conditions you would want to test on the API, then you can use any of the tooling approaches mentioned to experiment.

Suggested Tools:

Browser Dev Tools - fetch requests in console
Browser Dev Tools - generate cURL and use from CLI
Bruno
Postman

A List of API Tools is available on API Challenges site

Exercise - UI and API Interactive Testing

Use a Proxy to allow you to observe and interrogate the traffic from the Web Site to the Internal API backend.
Amend the Request to trigger an error response and see how the front end handles it.
Try testing the API from within the proxy.

A list of recommended HTTP Proxy Tools is available on the API Challenges site

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Test Interaction with HTML form fields

Alan Richardson — Thu, 15 Jan 2026 09:00:00 +0000

TLDR; When testing web apps we need to test the interaction between the browser implemented controls and our system. We don’t test the browser.

Testing HTML Form Fields

Watch on YouTube

Testing Web Technology

I like to use the term Technical Testing.

Many people don’t. I think other people view the term as implying that some people are technical and other people are not.

I use the term to cover “Testing which is informed by Technical knowledge”.

So if you’re testing a web application, you need to understand web technology, you need technical understanding of how browsers and the web works, otherwise you can’t test it effectively.

In this video I try to explore some of the reasons why this is important.

If I’m testing a web application and it has a form, and in the form is a date field which has a date control calendar pop up which lets me set a date.

What should I test?

Should I test that…

the calendar popup works on mobile and every web browser?
the calendar control works with keyboard?
the calendar control correctly switches years and months when I click the control buttons?

Well, no. Not if the HTML looks like this:

<input id="datetime-local-input" 
 type="datetime-local" name="datetime-local">

All of the complex UI interaction is implemented by the browser.

All our application does is use an off-the-shelf browser provided control.

Do I not test the functionality?

When we are using off-the-shelf controls like this we have to test the interaction between the control and our application.

If we added JavaScript event listeners to the control to perform extra validation or functionality then… yes we have to test that.

If we use the date entered in the control by the user, then we have to test that our application can handle the data supplied by the value of that control.

If we are not using the date then we should question why we have the control on our application UI.

Learn to spot technology risks

I can see from the HTML above that there might be an accessibility risk, because I don’t see any ARIA attributes associated with that control.

So we might not have configured the control in the page properly to implement all the accessibility requirements we might want.

If we had created a custom control, or it is specialised React control, then we do want to test it because we just introduced a technology risk that might impact mobile and different browsers. We might also have functional bugs.

But if we are using built in browser controls we don’t need to test as much functionality related to the control. We need to test the functional interaction with our application and custom code.

Test the Domain Configuration

We shouldn’t spend a long time testing validation messages from custom controls, but if a control has been configured then we need to test that the configuration is correct.

<input id="password-input" type="password" name="password"
 pattern="^[A-Za-z0-9_]{4,12}$" maxlength="14" required="true">

I don’t need to test that the password control above shows dots instead of letters and hides the input. But I do need to check that the configuration we supplied for the validation pattern and length are correct. And I need to make sure that the password is actually required (because we configured that it is).

NOTE: eagle-eyed readers, who have a technical understanding of how the configuration works will probably have spotted a bug in the above configuration. Yes. We should find that bug in our testing. And we know to look for it because we understand that the control has been configured with our application domain rules, and that’s what we are testing.

Don’t Trust the Client

The greater the technical understanding we have of the web technology, the less we will trust any data that the server receives.

We can edit the DOM to bypass domain configuration.

We can use JavaScript to enter data that bypasses browser validation.

We can bypass the browser altogether and just communicate directly with the server.

A greater technical understanding allows us to understand the risks of the technology more effectively

Testing Web Apps

All of the examples above are from:

testpages.eviltester.com

I also have a guide to testing HTML elements to describe some of the risks and issues that we might have when using or configuring off-the-shelf HTML fields and elements.

Testing Input Elements - Notes and Exercises

I had to learn web technologies when testing web applications.

I have seen people functionally test the browser implemented controls because they didn’t learn web technologies.

This means they wasted time, because they didn’t understand the technology they were testing.

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

New Version of CounterString Extension

Alan Richardson — Sun, 04 Jan 2026 09:00:00 +0000

TLDR; Generate CounterStrings, random data from Regex, ranges of data, and strings made by concatenating repeated data.

I released a new version of my CounterString generation Chrome Extension with new features and a new UI.

CounterString Extension

Watch on YouTube

Source and Download Links

I have a tool page for the CounterString extension with more details.

https://www.eviltester.com/page/tools/counterstringjs/

But you can find the tool:

Overview

I first released the CounterString Extension in 2019 (I think), and at that point it was a simple input field that created a string and added it as a value to the input field.

Now it can:

Generate CounterStrings
Configurable length and delimiters
Generate Random Data From Regex
Generate Character Ranges
Repeat text, characters and regex
Add generated data as value, with input event, to an input field
Trigger Key events to ’type’ the data with configurable speed
Binary Chop Calculator for field length exploration

Some of this was a result of migrating code from my Java Test Tool Hub (which I think I worked on about 12 years ago):

https://github.com/eviltester/testtoolhub

From the Test Tool Hub I took the:

Robot - I had to find a different way of triggering events, but this allows ’typing’ the data into the field rather than just amending the value.
Increased customization of the CounterStrings
Ranges of Data
Repeated Data
Binary Chop Range Calculation

And the Data from Regex was repurposing some code from my AnyWayData.com test data generator which uses the same JavaScript library to generate data from a Regex - RandExp

CounterStrings

I wrote about CounterString Algorithms in an earlier blog post.

This extension uses both reverse generation and forward generation.

I initially learned about CounterStrings from James Bach. James has a tool called PerlClip which is a command line tool to generate data directly into the clipboard. This uses a reverse algorithm where the full string is generated and then reversed to paste into a field.

A forward algorithm allows ’typing’ the String character by character without generating the full string first.

While I don’t generate the full String first, I do precompute a Schema which I then follow to generate the String, so it takes about the same processing time, but uses less memory and allows streaming of the result.

I created a different algorithm this time, which is much simpler to understand, but probably takes longer to execute. I’ll need to update the algorithm page at some point, but you can see the algorithm implemented in the source code.

github.com/eviltester/counterstringjs/blob/master/extension/js/generateSchema.js

Other CounterString Tools

A quick hunt around the web revealed a few more CounterString tools that I don’t think I had found all of these before. So these might be interesting for anyone thinking of creating their own implementation.

PERL: https://www.satisfice.com/download/perlclip
TYPESCRIPT: https://github.com/j19sch/counterstring/
RUBY: https://github.com/jamesmartin/counterstring
JAVA: https://github.com/eviltester/testtoolhub
PYTHON: https://github.com/deefex/pyclip
RUST: https://github.com/thomaschaplin/rust-counter-strings

Another Chrome Extension:

https://github.com/Pawel-Albert/utilities-for-testing-extension
- and some associated Test Pages that I haven’t seen before
- https://pawel-albert.github.io/utilities-for-testing-extension/

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Software Testing Podcast - AI Optimism or Pessimism - The Evil Tester Show Episode 028

Alan Richardson — Thu, 18 Dec 2025 09:00:00 +0000

Rather than retrospect on the year in general. This podcasts looks at how I’ve been approaching the learning of AI. Where the industry has gone wrong and what to look forward to with AI.

audio links and subscribe here

AI Optimism or Pessimism

I sat down to think what am I optimistic about, and what am I pessimistic about with the Software Development industry’s use of AI.

I see a lot of positive benefits for AI use in the Software Development Process.

There are many risks involved with adding AI into products and we should rightly be concerned about those and try to identify how to test AI enabled applications properly.

Many of the statements concerning how Software Testing has to evolve concern me. e.g. more technical skills, more focus on risk, point out flaws in process, identify architecture risk. I thought Software Testing already did this. So perhaps one good thing from AI is an evaluation of what part Software Testing plays in the Development Process and what are the fundamental skills and knowledge we need in order to Test software effectively.

AI Productivity

I’m incredibly optimistic about what I can personally do with AI. It’s massively increased what I can get done, especially in programming, content creation, and marketing.

But companies are cutting jobs and blaming AI. But companies have always looked for scapegoats when trimming staff AI is just the latest excuse.

If leadership truly saw their people as the reason for their growth, they’d use AI to amplify productivity and expand the team’s capabilities, not use it as a reason to shrink down.

“If companies viewed the staff as a core engine for growth, they wouldn’t want to get rid of people when AI comes in. They’d want to use AI to make the staff more effective and grow the company. Clearly they don’t.”

Many management teams see staff as a cost center, not as an engine for growth. Sales gets viewed as the engine of growth and is often paid on commission, so it seems cheap. The product is a ’thing to be sold’. And management… are always essential.

Some organizations don’t think “how do we help our people do more?” They take the position: “how can we do the same (or less) with fewer people?”

AI is the scapegoat. Not the cause.

Short-Term Pessimism

AI is overhyped. Leadership decisions are being made based on hope, not the technology’s actual capabilities.

In the short term, I’m a bit pessimistic, not about AI itself, but about how people (management) are choosing to react to it. Too many seem to want to use AI to restrict and constrain, instead of to grow.

But our work is already changing. New tools exist. We can’t afford to be complacent. We need to experiment and figure out where AI makes sense.

How Should We Use AI?

Not every product needs AI enablement and features. Too much risk, particularly if you do not know how to test the results.

I do NOT use AI for direct human-to-human communication.
- If you get an email from me, I wrote it. If I get one from you, I read it, not an AI.
I would not use AI for company procedures or performance reviews.
- Adding AI to these human, recognition-based workflows is a mistake.
I won’t use AI as a buffer between people.
- Don’t use AI to hide. Engage.

I do use AI to summarize public podcasts, blog posts, etc, Stuff I choose to consume, but not personal, direct communication.

Management is about people, and putting an AI buffer between you and your team leads to bad communication and alienation.

“Your boss should already know what you’re doing. They should be helping you explain the value that you’ve added. This is a human process of recognizing value. It’s not something that we add AI into the middle of.”

How I learned AI

When I started playing with AI, I created a podcast summarizer. My process for learning anything new is to find a project and use the new technology to build something.

I wanted to consume more podcasts but couldn’t keep up. So, I wrote a tool using:

Hugging Face libraries
Whisper AI for transcription
Different LLM models for summarization
Prompt engineering
Ollama for running models locally
OpenRouter for connecting to cloud-based models

I did everything locally as much as possible, partly to learn the limits before relying on cloud or paid models. Only after learning the limits did I see real value in some of the paid services.

Don’t believe in ‘magic’ from big models until you know what smaller, free models can actually do.

Step By Step Adoption

Step 1: Chat Interfaces

At first, I used AI the same way most people, from a chat interface, like ChatGPT or Claude. It worked but had limits. I wanted something embedded into my IDE.

Step 2: Embedding into the IDE

Now, I use a plugin called Continue, which works in both IntelliJ and Visual Studio Code.

Initially connected to Ollama (local code models)
Later I Switched to OpenRouter for bigger cloud models

So now, I can:

Jump into a chat interface via my IDE
Still use chatbots like Claude/ChatGPT for generic things
Never look at Stack Overflow anymore, everything’s in the IDE chat

Step 3: Agentic Coding

I started using agentic tools, like OpenCode, on the command line. They can scan my whole codebase, not just what’s open in the IDE.

I use it to:

Create a page object model for a URL
Write a test for that model
Generate K6 Scripts
Create swagger files
Write scripts
Write API Abstractions
… so much

Now instead of:

Writing the test
Building abstractions and models manually
Iteratively improving

I let the AI generate code, then I come in and change or adapt what I need.

“I’m using it as a tool to help me create things faster. Then I come in and use my knowledge and experience to review the code or expand it.”

Step 4: OpenSpec for Better Requirements

OpenSpec helps generate and maintain evolving and up to date requirements documentation.

Specs are continuously updated as requirements change
OpenCode has documentation to work from, in addition to my prompts

When I use OpenSpec, I’m letting the Agentic Coding Assistant write even more code, than when I use it iteratively at the CLI.

This has helped me:

write a new application faster
Keep requirements and documentation in sync
Automate coverage incrementally

I’m not sure how well it would scale to a full team, but it works well for a solo dev.

But…

AI sometimes makes strange design choices
If you don’t review and refactor, your codebase can fall apart fast
Human review is still necessary

AI saves time, but still requires fundamental core skills to evaluate their output and fix them when they go wrong.

I use AI for programming all the time, but not for actual testing.

Agentic AI in Testing

I’m beginning to look into Agentic AI tools specifically designed for testing.

I’m currently experimenting with AQE Fleet from Dragan Spiridonov

Pros and Cons

Right now, I am optimistic about AI empowering individual development team members. It makes us faster, and the tools are getting better.

But I am pessimistic about the way companies choose to adapt to AI. Focus on headcount reduction instead of increased effectiveness and growth.

What is Testing?

“I keep seeing posts that testers need to evolve into quality engineers and risk analysts and customer experience advocates because we’ll be reviewing the systems for risk more than we’re and testing them… but I thought we already did those things.”

People keep saying testers now need to be risk analysts, customer experience advocates, technical experts, etc. But that’s what good testers have always done. It almost feels like AI is forcing companies to rediscover what software testing really is, and always has been.

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Free Technical Web Testing Course Released To Youtube

Alan Richardson — Thu, 18 Dec 2025 09:00:00 +0000

TLDR; Technical Web Testing course has been released to YouTube as a 6 hour video.

Free Technical Web Testing 101 course

Watch on YouTube

Slides etc. can be found via the course page

Practice Sites Now

The sites I use in the course are a little old.

If you want to practice with the techniques in the course then I maintain lists of sites to practice with here:

Course History

The course was created in 2013 or 2014. Originally as a free course, then I added a bunch of content and it turned into 6 hours plus 3 hours of additional webinar content.

Every course provider I hosted it on has changed terms and conditions multiple times making it painful to maintain and support.

Since the examples and videos are now showing their age, and one course provider changed their terms and conditions so much that I now no longer have access to their course platform and can’t support it.

I decided to make the course free.

The course is too big to make free on Udemy so I’ve combined all the videos into one and made them free on Youtube.

Thanks to everyone who supported the course over the years.

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

An overview of the Basic Shopping Cart Practice Web Page

Alan Richardson — Mon, 15 Dec 2025 11:19:00 +0000

TLDR; an overview of the basic shopping cart that you can use to explore, automate through the UI or REST API and perform security testing on.

The Shopping Cart Application offers a lot of possibility for testing due to the multi-state and multi-page nature. With the added bonus of a REST API which is documented using OpenAPI spec there are even more ways to interact with it.

Find the Shopping Cart Application Here

Overview Video

The video provides an overview of the shopping cart:

Watch on YouTube

Shopping Carts for Practicing

This is not the first Shopping Cart practice Application.

I know of:

SwagLabs Saucelabs demo application.
Toolshop from this has multiple versions
Restful Booker could be classed as a shopping cart
and my Basic Shopping Cart

SwagLabs is pure JavaScript so there are no HTTP calls, but you can explore and test the interaction with local browser storage, and the various users that you can login with change the display so you can use it for visual testing and admin access.

Toolshop, Restful Booker and my Shopping Cart all have APIs and the data is cleared regularly. I think Restful Booker clears data every 15 minutes. Shopping Cart clears data on a rolling basis by limiting the amount stored, so if there are a lot of users at any one time (about 200+), you might notice your old orders clearing from the backend.

When I was designing the Basic Shopping Cart I very deliberately chose to make sure that there was no possible way of having any PI data in the system. This meant that the checkout process does not ask for email addresses, credit card details, or even passwords. The users are ‘canned’ with pre-defined passwords. This means you can perform security testing without any issues.

Really you shouldn’t add any PI data to any testing practice site, but when applications use forms, the auto-complete might be switched on, and your details might be auto-populated into the form.

I do like the other Shopping Carts and have used all of them when practicing my testing.

Why A Shopping Cart?

Shopping Cart applications offer a lot of scope.

There tend to be multiple pages offering navigation scope.
Pagination if there are enough products.
API and server side interaction.
Checkout Process has state - tracked locally via cookies or browser storage, and possibly server side as well. Having state in two places offers scope for state moving out of sync and creating mismatch between server and client.
Usually a registration or login process. Which also lends itself to security testing.
Tend to be content rich.

Content rich applications are interesting. The first time we automate them we tend to hard code product names and ids. Then we realise that there is usually a CMS (Content Management System), and users can edit content, which then causes our automated execution to fail.

This might lead us to using the API and UI as a consistency Oracle for content. Does the API serve data consistently. Is the data rendered correctly. So we don’t automate or test exclusively through the UI, we use the UI and API in combination when we want to assert on data conditions.

Creation of the Shopping Cart

To build the application I started API first. The API was completely hand coded. This meant that I could test the flows and the functionality before spending any time on the UI. I was able to experiment with various authentication flows and data representations until I had one that I thought was rich enough and removed any PI risks.

Once I had the backend covered I used OpenCode to incrementally add basic REST API automated coverage, and then expanded the coverage.

Frond End Creation

I wanted the front-end to be simple JavaScript, HTMl and CSS, and avoid the use of frameworks.

My Test Applications are designed to be entry points. They will be challenging enough naturally, so I didn’t want to clutter the learning effort for people practicing by using React.

The Front End was creating by prompting through OpenCode and then I would amend and refactor where necessary. This allowed me to create the front-end quickly. But because I prompt in very small increments the code is more closely aligned with the style of coding I use, and makes it easier for me to amend in the future.

The models used were spread across free models from OpenCode Zen.

Grok Code Fast 1

Or I used Kat Coder Pro (free) on OpenRouter.

I try to use a mix of models in the CLI, and then a different set of models in the IDE. In the IDE I have Continue configured to use OpenRouter models.

Creating Content

Al the content was generated via AI tooling.

I prompted GTP-OSS-20B via OpenRouter to create 100 ’thing’ name variations. I fed it the sample ’thingy’ words and adjectives. I could have varied this more, and I could easily have used a normal combinatorial algorithm to do this but the AI interface was fast enough.

With a set of 100 names, I then asked for one line descriptions, using the same process. The are all output as JSON to make it easy for me to amend in the code.

Since this data is ‘content’ I can vary it over time. For the first release I just need 100 different products to add to the product list.

Creating Product Images

Generating the images was a little more interesting.

I am deliberately trying to keep my AI cost as low as possible to see how far I can push my use of AI with minimal cost.

My biggest cost on OpenRouter was using Nano Banana to create 2 experimental images when it was first released. The Nano Banana images cost 50c each and they didn’t end up the way I wanted. This was not an experiment I wanted to repeat with the product images. I’d need at least 100 and I might want to regenerate them if they didn’t work out.

First I tried image generation models through the local install of localai.io but I don’t think my machine is powerful enough. The process was slow and the output was basically a random png and completely unsuitable.

Rather than spend another couple of hours experimenting to improve this, I decided to try one of the image models on OpenRouter. These are easy to automate but past experience made me hesitant. However, as I was looking at the costs I noticed that the image pricing was quite cheap for one of the models, but this was only shown in the comparison page.

https://openrouter.ai/compare/openai/gpt-5-image-mini/google/gemini-3-pro-image-preview

GPT 5 Image Mini was estimated at $0.002 per K generated.

I didn’t expect to have good results with this but I tried it through a simple prompt and was surprised at the image quality.

It took me about 20 minutes to write code to call the API and experiment with prompts. This led to some good random product images and they were about $0.007 per image, so I could generate 100 images for about $0.70.

I used the product descriptions generated earlier and fed them through into OpenRouter with my script to generate the images you can see in the product pages.

Generating Image Descriptions

Rather than use Lorem Ipsum I wanted to generate fanciful and long descriptions for each of the products.

I wanted to use AI to generate a fuller paragraph style description of each product.

I found that the lava:7b model will run nicely on Ollama locally and it is very small. It has vision capabilities, so it can read images.

https://ollama.com/library/llava:7b

It only consumes about 5gb when running so it fits comfortably in my GPU memory.

I used this via api calls to describe each of the images in turn. I was very surprised at the quality of the descriptions when running locally.

This worked out well enough for my experiment and in total cost around $0.70 to generate and describe 100 images.

I thought that 100 was enough to allow pagination experiments and build on over time.

In total to do all this programmatically took about an hour to write code and 4 hours to run.

I think it would probably have taken longer for me to search the web, find a tool to do all this, if a single tool even existed at all.

Tool Vendors Take Note: it is becoming easier and easier for people to create their own tooling, cheaply and with the exact functionality they need.

Automating the Web UI

I used OpenCode to generate the Page Object Models and an initial set of coverage, I then amended and refactored to expand the coverage as I required.

This has become my normal approach to generating automated execution code now. This is also different from the way I code manually.

When coding manually:

I write the test first,
and grow the Page Object Model organically through refactoring,
this way I only create the abstraction code that I need in the test.

When coding with AI, I reverse this:

I generate the Page Object
- I give OpenCode access to the Chrome Dev Tools MCP so that it can read the DOM and it generates accurate locators. Sometimes I refine them, but they are usually good enough.
During the Page Object creation, in the same prompt I ask for
- “A set of basic tests which use the Page Object Model methods to cover the basic functionality. Do not test the Page Object, use the Page Object to avoid putting any locators for ‘driver.’ methods in the test code.”.
I then review and refactor.
Then I iteratively expand with small prompts, targeting specific coverage with each prompt.
I’ve found this works well, avoids rework and is fast.

Automating the REST API

I prompted OpenCode to read the routing code that I had written in Java for the API and create basic coverage for the each of the API end points in turn. This led to the creation of the payload objects and incrementally build up an API abstraction.

By targeting each endpoint individually I could gradually grow the code, keep control of it, make sure it was using my style of coding and automating and didn’t need a lot of refactoring later.

I then prompted to recreate the UI automated execution flows using only the API abstractions that I had just built up.

Testing

The general rule we are told is that you test first, then automate. You can see from the above description that I do not abide by this rule.

I used the automated coverage to handle the basic flows and flush out any simple errors. This creates an automated safety net that can help me when I’m fixing any issues that I find from testing. Without the basic automated coverage I might introduce more errors when fixing bugs than I remove.

Testing was performed via exploration using Chrome, heavily reliant on the dev tools and viewing requests in BurpSuite.

I then moved on to testing the REST API directly.

This helps me build up a list of issues, some of which I fix, but since it is a testing app, some of them I don’t.

The automated coverage tells me that it can work. My testing convinces me where it fails to work.

Security Testing

I coded in some security issues, so I made sure to test these.

I used Bruno as my API client to explore all these issues.

Some of them I removed because they would be annoying to the user if exploited when they were trying to test the application, so they are primarily information leakage rather than amendment. But… it might be possible if you test deep enough.

The only tool used during this process was Bruno to issue requests.

Performance Testing

I didn’t want to go live with the cart unless I could convince myself that it could handle multiple users.

I used OpenCode to convert my API automated execution scenarios into K6 scripted scenarios such that I could scale Virtual Users and hit the application functionality.

I do not recommend the Shopping Cart as a target for performance testing, that isn’t what it is designed for.

Butch Mayhew maintains a list of sites designed for practicing Performance testing against in his Awesome Sites To Test On

Two from Blazemeter and one from OctoPerf:

This took me about an hour in total, because I had to fix a couple of errors in the generated k6 script, but I think using OpenCode saved me a couple of hours since I’m not a K6 expert and have to re-learn it every time I use it.

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Software Testing Podcast - BrowserStack Community QnA - The Evil Tester Show Episode 028

Alan Richardson — Fri, 12 Dec 2025 09:00:00 +0000

The answers given during an AMA session held on Discord on the 11th of December 2025, following a live LinkedIn video stream. The session focused on “Mastering Automatability for Test Automation”. The main theme is the concept of Automatability, which I view as the ability to automate , this personal skill is more critical than reliance on specific tools. The discussion covers various topics, including how to separate automation problems from application design issues, dealing with slow UIs and non-automation friendly third-party widgets, evaluating automation readiness, and addressing common architectural failings related to large-scale UI automation.

Links to audio version of the podcast here

Discord And LinkedIn

This was a Browserstack hosted event. The initial Q&A session started on Linkedin with a conversation between Alan Richardson and David Burns.

Recording can be found here:

LinkedIn Event

The session then moved to Discord. The Browserstack discord has many AMA and interviews so is worth signing up to have a look at.

Other AMA sessions include:

Join the BrowserStack Community on Discord and discover many more sessions, videos and conversations.

https://www.browserstack.com/community

Q&A Session Questions and Summaries

I’ve listed the questions and summary answers. Full answers can be found in the podcast, audio or video, or on the Discord AMA chat.

Q&A 1: Understanding Automatability for a First Automation Framework

Question: If I’m building my first test automation framework, what’s the one thing about automatability I should understand?

Summary of Answer:

The most important thing to understand is that automatability refers to your ability to automate. By having a strong ability to automate, you become less dependent on specific tools, making it easier to create workarounds and choose from multiple tools. Developing experience in how to automate allows you to succeed more often, and means you are not reliant on a tool interacting with your system, which makes workarounds harder. Automating is fundamentally about your understanding of what and how to automate, and practicing the application of that ability.

Q&A 2: Separating Automation Problems from Application Design Problems

Question: How do you separate automation problems from application design problems?

Summary of Answer:

If an issue causes problems when you are automating, I would call it an automation problem. While this problem might be triggered by an application design problem (such as a state-based system that is hard to track, or features that are harder to automate), the issue itself remains distinct. If the team cannot change the application design, they must figure out how to automate the application as it is. This might involve absorbing the issue, figuring out how to automate it at a different level (not end-to-end), or handling it through testing processes using observability tools like DataDog.

Q&A 3: Slow UIs and Testability/Automatability

Question: When dealing with slow UIs is the slowness a testability issue an automatability issue or both?

Summary of Answer:

Slowness is likely both, and more , because it is also a usability/user experience issue. If the slow UI impacts the user experience, it is more likely to be addressed than if it only impacts testing or automation. In cybernetics terms, testers or automators must possess the “requisite variety” to handle the variety (slowness) in the system being tested, which means knowing how to synchronize or potentially cleaning the environment to improve speed. The focus would be on the impact of the slowness, rather than the slowness itself, and whether the team or its tools can absorb the variation in response times.

Q&A 4: Handling Non-Automation Friendly Third-Party Widgets

Question: How do you handle third party widgets like payment gateways that are inherently not automation friendly?

Summary of Answer:

If a third-party widget is “not automation friendly” for one tactic (e.g., UI automation), it might become easier to automate by adopting different tactics, such as issuing HTTP requests using cookies collected from the UI. Teams may not need to automate the full flow of the widget, but instead focus on ensuring the widget is wired up correctly within their own application. This can involve only testing partway through the flow, or using a mock or stub in the environment so that the full widget flow doesn’t need to be tested constantly.

Q&A 5: Evaluating Automation Readiness and Consultancy Frameworks

Question: How do you evaluate an application’s automation readiness during consulting? Do you follow a framework?

Summary of Answer:

I do not use a formal consulting frameworks. The closest methodology used is the meta model from Neuro-Linguistic Programming, which involves asking questions to build a model of the client’s environment and processes, comparing it to the reality they face.

learn more about NLP and Testing in this paper I wrote.

Regarding automation readiness, an application is considered ready to automate “as soon as someone wants to automate it”. Readiness is judged by whether the client is prepared to commit to whatever it takes to automate the application at that specific point in time to achieve their desired outcomes, regardless of the application’s current state.

Q&A 6: Architectural Patterns Failing in Large-Scale UI Automation

Question: What architectural patterns do you see repeatedly failing when it comes to large scale UI automation?

Summary of Answer:

Recurring issues often stem from the team lacking the ability to automate and consequently blaming the tool for problems, rather than creating necessary workarounds. Common process anti-patterns, not strictly architectural patterns, include deploying differently into test and production environments (not using the same install process).

A major failure point is test data maintenance , especially trying to use production data or any data that the team does not control. Automating against specific data conditions without control over that data causes random test failures. This can be worked around by hardcoding tests against data conditions instead of specific data, and dynamically selecting the required data during execution.

Q&A 7: Prioritizing Testability and Automation in Sprint Planning

Question: If testability improves debugging and automation improves scale, how do we prioritize them during sprint planning?

Summary of Answer:

Prioritization can be based on what the team wants to achieve (the outcomes) by the end of the sprint, specifically focusing on the expected coverage from testing and automation. It is beneficial to plan for features that need extensive testing to be delivered early in the sprint. Ideally, testing and automating occur in parallel, and teams automate at lower levels (like unit level) to reduce the necessary coverage at the higher UI level. Issues often arise when teams are divided into isolated roles, creating process problems that hinder effective interaction and prioritization.

Q&A 8: Playwright and the Illusion of Reduced Automatability Design Needs

Question: Do modern frameworks like Playwright reduce the need for high automatability design or is that an illusion?

Summary of Answer:

It is an illusion. Frameworks like Playwright are designed to absorb application variability through features like retry mechanisms (for synchronization) and locator strategies (like visible text), which reduces the need for constant notification when minor changes occur. This absorption capability makes Playwright effective for agent-based automation where the goal is checking an end-to-end path and the final result.

However, this absorption can hide issues that a team might want exposed. Even when using Playwright, developers must still understand how to automate and structure their code using abstraction layers (like page objects, domain objects) to ensure long-term maintainability and efficiency.

Q&A 9: Explaining Automatability as an Investment to Leadership

Question: How do I explain to leadership that improving automatability is an investment not a delay?

Summary of Answer:

The explanation depends on what is being improved. If improving automatability means increasing the team’s ability to automate, it can be presented as an investment in staff training. If it involves adding technical aids (like IDs in the UI), leadership might perceive it as a delay because they may not value UI execution coverage or may already be confident in unit-level automation. To convince leadership, the team could demonstrate the return on investment by showing the alternative world. This involves comparing the current reality to a scenario where improved automatability allows the team to do beneficial things they otherwise couldn’t, thereby highlighting the value gained.

Q&A 10: AI Agents Dealing with Dynamic Elements

Question: We are exploring AI agents for our teams and I want to know how does the AI agent deal with dynamic elements like rotating banner banners, third party widgets or AB tests?

Summary of Answer:

How the agent deals with dynamic elements depends on how it works (e.g., building high-level BDD scripts or generating code). Agents often operate on first principles. If an agent uses a BDD approach, it works from a runtime specification and handles dynamic elements because it works from scratch for each execution, constantly aiming to fulfill the objective. For example, if an unexpected pop-up appears, the agent clears it and continues.

If the agent writes code, it uses what is often called “autohealing”. This process automatically amends the script based on the current application state, prioritizing the achievement of the objective regardless of whether the change is “right”.

Q&A 11: Early Signals of Flaky Features

Question: What early signals tell you that a feature will become flaky once automated?

Summary of Answer:

Early signals involve understanding the synchronization points of the page. A feature is likely to be flaky if the application is populated or amended over time by JavaScript and the automation tool is not synchronizing properly on the DOM buildup. If a page is being constantly updated in the background without clear visual indicators (like spinners, which are easy to synchronize on), flakiness is more likely.

The signal is the update process itself, particularly when it is non-deterministic (e.g., how totals are updated in a shopping cart). The core question is whether synchronization is required to prevent flakiness, and if it is difficult to synchronize, that is a strong signal. If necessary, automatability might be enhanced by adding an extra flag to the DOM to signal when the update is complete.

Q&A 12: Layers to Focus on in Microservices for Automatability

Question: In microservices setups which layers should teams focus on first to increase overall automatability?

Summary of Answer:

The foundational layer to focus on is the human understanding of the architecture and the requirements for automation. In microservices specifically, teams would typically focus on the interface layer and their ability to automate it while keeping the interface standard.

If microservices are communicating via HTTP messages compliant with a version standard, automation can be relatively easy. If interfaces are internal and change randomly, issues may arise, requiring attention to managing event-based queues if applicable. Strategies include using versioned interfaces or having a process to update automated coverage and abstraction layers when microservice interfaces change. It is crucial to avoid replicating interface objects (like payload objects) directly into test code, as this can prevent tests from spotting issues when new or removed fields occur in the application interface.

Q&A 13: Budget-Limited Automatability Fixes for Fastest ROI

Question: For a team with limited budget which problems around automatability should we fix first to get the fastest return on investment?

Summary of Answer:

The fastest return on investment comes from enhancing the team’s ability to automate. This improvement allows teams to develop workarounds, find alternative solutions, and identify when to use different techniques. It is not about purchasing multiple or expensive tools. Instead, investment could be placed in training, practicing, exploring the capabilities of existing tools, and eliminating fundamental issues like test flakiness by fixing the root causes.

Q&A 14: Collaborating Earlier to Avoid Automatability Rework

Question: How can developers and testers collaborate earlier to avoid expensive rework on automatability issues?

Summary of Answer:

Collaboration is achieved by removing the barriers that cause people to be isolated into silos, such as separate programming, testing, or test automation teams. The core issue is often that a “development team” is defined only as a programming team, instead of encompassing responsibility for design, programming, product suitability, testing, and production.

Practical steps include involving the programming team in the automated execution maintenance. When programmers contribute, they often ensure data IDs are present, which removes many hard-to-automate issues typically found during end-to-end testing. Sharing the responsibility for maintenance ensures people understand and resolve related issues earlier.

Q&A 15: Automatability in Continuous Delivery and Trunk-Based Development

Question: How should teams think about automatability when shifting towards continuous delivery and trunk-based development?

Summary of Answer:

This environment requires high automated execution coverage that runs quickly, often with features being merged multiple times a day. Automatability is achieved by ensuring the person or pair responsible for creating a feature is also responsible for adding automated execution coverage (unit tests). These tests demonstrate that the feature has been tested and provide future checks against accidental changes impacting the functionality.

Teams could focus on structuring unit tests at the domain level (e.g., focusing on users or orders) rather than strictly class level. This approach results in a degree of internal end-to-end flow tests without needing extensive external system testing. Furthermore, application architecture can be designed so that interfaces (like HTTP interfaces) can be tested primarily at the domain level, reducing the need for numerous actual HTTP calls.

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Convert Chrome Dev Tools Recording to WebDriver Java Code Using Free OpenCode AI Agent

Alan Richardson — Mon, 24 Nov 2025 10:19:00 +0000

TLDR; Record and Playback is a notoriously bad way to automate. But what if we record, then have AI write code from the recording? Answer: it is a little better.

I performed an experiment to try tand demonstrate how to replicate the AI-powered test automation features commonly found in commercial SaaS testing tools using open source solutions. The process involved recording user interactions in an application with Chrome DevTools Recorder, exporting the recording as JSON, and then using an AI tool to convert the JSON into JUnit tests with WebDriver and page objects.

Overview Video

Watch on YouTube

In this video I:

record some web actions using Chrome Dev Tools recorder
fail to play them back because record and playback is notoriously bad
feed the recorded JSON file into OpenCode with a prompt
check the results of the AI process writing Automated Execution using WebDriver and Java

We do get working automated coverage out the back. I would need to edit it a little, and if I wanted to do this longer term I would need to amend the prompt to meet my coding preferences.

process

I used the triangle application on Test Pages:

https://testpages.eviltester.com/apps/triangle/

Test Pages has a deliberately hard to automate side bar. The easy automation is through the top level menu.

But I recorded the Triangle application by:

clicking on the side menu bar to open the Triangle application
and the instructions
reading the instructions
clicking back on the triangle application
interacting with the app to generate invalid and valid triangles

When I tried to playback the recording in the Chrome Dev Tools, it failed.

Most of us experience failure with record and playback tooling.

AI tooling

I have OpenCode, configured to use Chrome Dev Tools MCP, and working with Kat Code Pro Free in OpenRouter, but could just as easily have used any of the free coding LLMS on Open Code Zen or with Ollama locally.

I used a basic prompt

The process took about 5 minutes to run.

It opened the web page a few times,
created some Page Objects.
Found that the menu interaction was hard.
Decided to jump straight to the triangle page and automate that way
It did replicate the functional interaction included in the recording
it did generate assertions for the results
I could have used the test code generate with some small amendments.

Lessons Learned

SAAS tools which take this approach will use Agents, which are essentially longer prompts. I should have added into the prompt more conditions:

“given the json attached, convert the json into a set of junit tests using webdriver and page objects to recreate the flows recorded by the user. The json file is a recording made in Chrome Dev Tools recorder. In the test code, do not write any findElement code, always abstract the location and interaction into a page object. Use existing page objects where possible and try to re-use existing methods. Make sure that any page object methods you use are used in a test so we have coverage of the interactions.”

Over time, if I wanted to continue to automate using this process, I would refine the prompt further until I got the results I wanted, and possibly encode this prompt as an agent.

Would I do this?

Unlikely that I would do this in the real world.

This seems like the replication of a record and playback approach.

I do create Page Objects and have created initial execution coverage using AI.

I have some empathy for AI tooling scanning applications and building flows through the application to automatically interact and generate a wide variety of data, to increase the scope of coverage across a few simple flows. I can see potential in augmenting a team to allow the team to go off and do more detailed stuff. Provided the AI tooling doesn’t get noisy and keep distracting the team.

For any functionality as simple as I experimented with here, I hope that when people evaluate SAAS tools, that they also compare the paid tools to the capabilities of Open Source tools.

Nothing wrong with paid tools. They are incredibly valuable when they make hard things simple, and add value when they make hard things easier.

But, when the activity is simple and the functionality differences between Open Source and paid are small, I hope companies consider “what if we upskilled our staff a little such that we could use the Open Source tools”.

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

What tool to choose for Web UI Test Automation?

Alan Richardson — Thu, 25 Sep 2025 09:00:00 +0000

TLDR; I tried to objectively assess the web automation tool market. And as a new development team I’d probably choose Playwright, but as someone experienced in automating the web I’d still choose WebDriver.

Reviewing Automation Tools

I reviewed the market for Open Source and free Web Automation Tools so that I could more objectively answer questions like “What tool should we use for Web Test Automation”.

I tried to approach this like any development team would:

web searches for tools
read forums and comments
read the documentation

But I also tried to project scalability. If the team starts to create a lot of Web UI coverage and wants to have many different versions of browsers available then which tools also let them use cloud providers as the execution agent.

This was a thought experiment so I didn’t prototype execution with each tool I have used both tools so for the purpose of this write up I skipped this step.

Web Searches

There are a lot of spam articles on “17 Best Web Automation Tools for 2025”. You could create random variations of this title and probably find an article using your generated title on the web.

What you find are random lists which lump paid no-code tools with unsupported niche code libraries mixed with commercial AI based tools. Also automation tools (Webdriver, Playwright) would be mixed with execution and design frameworks (Robot, Cucumber).

An impossible combination to search through and most of those pages offered me no value.

So the first thing a team has to decide are filtering criteria:

paid or free and open source?
code based or no-code based?
AI or non-AI?

This will depend on the team. Development teams with coding skills are probably best served by looking for free code based tools. They will have the skill sets to take advantage of the flexibility provided.

Test teams with no coding skills will have a harder time choosing tools because there is so much competition for paid, no-code, AI tooling.

I’m focused on the free and code based tooling because I code, and I work with, Development teams with coding skills.

I also recommend that teams who are automating without these skills, try to develop these skills as it can help avoid tool lock-in and ultimately opens up more approaches for automating.

Community Sentiment Analysis

If I didn’t have much experience with the tools I’d check sentiment analysis and see what people were recommending on Community sites like Reddit.

Looking at the community comments Playwright is promoted more than Selenium WebDriver.

WebDriver seems to require a PR boost because it is seen as outdated. This is unfortunate because it isn’t.

At this point I’d have a smaller list of tools:

Selenium WebDriver
Playwright
WebdriverIO
Cypress
Puppeteer

And as I start looking through them I realise that if I want the option to use multiple languages I really only have two options: Selenium WebDriver and Playwright. All the other tools require automating in TypeScript.

Multi-language support is only offered by Selenium WebDriver and Playwright, all other tools are aimed at JavaScript/Typescript. This is probably a natural side-effect that ‘web applications’ have a front end that is a mix of HTML, CSS and JavaScript so the obvious choice to automate the front end is using the language space of the web: which is TypeScript and JavaScript. But, many teams are polyglot with a mix of skill sets. The core development skill for a large application is going to center on the main code base, which is likely the back-end, and that might well be written in TypeScript, but could equally be Java or Python.

A library with support for multiple languages is likely to the be choice for most polyglot environments.

My two main choices for Web Automated Execution driven by coding would be Selenium WebDriver and Playwright.

The community comments might lead me to think that WebDriver is flaky and hard to use. But I’ve never found WebDriver hard to use, or to be the the root cause of flakiness in my execution. I’ve never needed self-healing locators or auto-waits for stability.

Some comments against WebDriver: “you need to have the right driver installed for the browser” were removed a long time ago, first by the Selenium Manager library and now this is built in to WebDriver so it is very easy to get started with WebDriver since it downloads drivers and browsers when required. Selenium Manager

If I read community comments about WebDriver driver and browser management I might be concerned, but I’d then see in the Playwright documentation that I have to manage the browsers for playwright using maven or npm commands see browers docs. The documentation for Playwright actually makes this seem harder than it needs to be because I’m pretty sure I’ve never done this with Playwright and it downloaded what it needed automatically.

Scalability Support - Online hosts for Web Automation Tools

Reviewing the online Web Automation SAAS tools can give a good indication of which libraries and tools are popular and scalable.

If you start using a tool heavily then you might want to expand the range of browsers that you automate against, this can be time consuming to setup locally so Online hosts might help by pointing the test code at a cloud instance for a specific browser.

BrowserStack
- https://www.browserstack.com/
- Selenium, Playwright, Cypress, Puppeteer, WebDriverIO
LambdaTest
- https://www.lambdatest.com/
- Selenium, Playwright, Cypress, Puppeteer, WebDriverIO
SauceLabs
- https://www.lambdatest.com/
- Selenium, Playwright, Cypress, WebDriverIO
TestingBot
- https://testingbot.com/
- Selenium, Cypress, Puppeteer, Playwright, WebDriverIO

Reading the documentation for these tools I’d learn of other tools which are mentioned when looking through docs of online hosts e.g. TestCafe, Nightwatch. But again these are not polyglot so wouldn’t really expand my options.

At this point I’d also see that with my short-list of tools I have options around scalability which don’t require me setting up my own multi browser environments.

Playwright vs Selenium WebDriver

I narrowed my choice down to Playwright vs Selenium due to multi-language support.

So now I have to go through the thought process of comparing the tools.

Perceived Execution Speed and Coverage

Playwright has the advantage that it is newer and only targets a subset of browsers and browser communication protocols, it has no support for legacy browsers, it has never had to concern itself with backwards compatibility.

If you need to use legacy browsers then you’ll probably use WebDriver.

Both WebDriver and Playwright are converging on the WebDriver BiDi W3C standard, so when this happens a choice between WebDriver and Playwright is likely to come down to the programmer oriented API.

Documentation

Comparing the Playwright and WebDriver documentation and we can see the advantage that a tool with Big Company support has.

Playwright is provided by Microsoft and has people paid to work on the documentation and examples. WebDriver is community driven and so naturally the Playwright documentation is currently more attractive than WebDriver.

For an experienced team this isn’t really an issue because the WebDriver API is very well documented at the code level and both projects have example code on Github.

For a team just starting out the documentation page and community comments could well swing people over to Playwright immediately but I would hope this wouldn’t be a deciding factor for experienced teams.

Auto Waits and Retries

Playwright is probably also viewed as ’easier’ because of the auto-waits. I have some experience with automating so this actually makes me nervous, I’ve seen auto-waiting and auto-synchronisation cover up application bugs.

The auto-wait conditions are documented on the Playwright site under actionability Here I can see the automatic synchronisation on check, click, etc. I checked the Playwright documentation and see that I can configure the auto-wait action timeouts so I can take control over that behaviour when necessary.

Playwright has auto-retrying assertions expect(locator).toBeEditable() which seem analogous to the WebDriverWait

API Coverage

The addition of API coverage from a single tool might push teams towards Playwright.

I personally prefer tools and libraries to be focused on ‘one thing’ and do it well

API coverage can easily be achieved on projects using WebDriver by adding specialised libraries for the coverage required e.g. HTTP libraries for API testing and mocking libraries for APIs.

The dynamic mocking that Playwright supports where network traffic from the current page are mocked is harder to achieve with WebDriver and requires Network access. But also, is that the ‘right’ place to test that functionality? I’d expect to see that covered pre-deployment at the code level rather than running integration level and would expect that to be covered by testing-library.com level component tests with a mock API.

Future Project Scope

For me, this API Coverage might act as a Risk that the scope of Playwright expands beyond what I would require.

The scope of Playwright might be trying to evolve to cover deployed application execution and component development testing (like Cypress or Testing Library).

I can also see that there are differences in functionality between the language variants with the Node.js version offering a web server for local execution. This also makes me concerned about future viability of some of the languages. Would Java always be a first class citizen in the Playwright world?

Since the feature set between Playwright and WebDriver isn’t too different, I probably wouldn’t worry too much about this because most of my library usage would be behind a set of abstraction layers and I’d be able to migrate the implementation of those layers to other tools if necessary.

What to choose?

If I was new to automating I’d probably choose Playwright.

I’d be nervous about the differences in language support, but it probably wouldn’t be enough to stop me. If I was automating in TypeScript it probably wouldn’t concern me because I suspect TypeScript is the primary supported language for Playwright. For .Net I probably wouldn’t be concerned because .Net is Microsoft and they’ll keep supporting it.

Both tools are very good, they are maintained, they are both fast, they are both converging on BiDi. Playwright has extra features, many of which can be added to WebDriver projects using other libraries.

But this is a hard choice. And is probably determined by the feature set that you think you need for automating.

A Minority View

I feel that I’m in a Minority at this point. For production web automating I’d still choose Selenium WebDriver with Java or Kotlin.

I have a strong preference for single-focus libraries that do not try to extend their functional remit too far so that helps sway me towards WebDriver so I personally would still choose WebDriver for production work, but I acknowledge that I am biased based on my previous exposure to WebDriver.

I’m pretty sure that when the WebDriver BiDi standard is moved out of draft and supported fully by all browsers that the differences between WebDriver and Playwright will be minimal, but WebDriver may have lost a lot of the installed user-base by that point as people have moved towards Playwright.

The weight of public opinion at the moment is pushing towards Playwright which I think is also a pity because the standards that make Playwright possible were created and pushed by the WebDriver project team.

There is obviously more money behind Playwright, but it’s free and Big Companies often choose not to support tools for business reasons. This risk feels unlikely to happen but it’s in the back of my mind.

I think WebDriver remains an essential part of the Web Automation eco-system and it is unfortunate that the Big Browser Companies like Microsoft, Google, Apple are not listed on the Selenium WebDriver sponsors page. Despite this the WebDriver team manage to continue to push for open standards that make automating the web easier and better, and maintain tooling across languages and browsers that other tools do not support.

Multiple Language Support Tools Summary

Selenium WebDriver

https://www.selenium.dev/documentation/webdriver/
Programming languages supported: Java, Python, CSharp (.Net), Ruby, JavaScript, Kotlin(Java)
Browsers: Chrome, Edge, Firefox, IE, Safari
Functionality is aimed at Browser Automation.

Playwright

https://playwright.dev/java/
Programming languages supported: Java, Python, JavaScript/Typescript (Node.js), .Net
Documentation has some odd choices
Functionality covers Browser, API Automation, API Mocking, Assertions, auto-waits, videos, screenshots, traces

TypeScript/JavaScript Language Tools Summary

There are a lot of tools in this category which I didn’t really look at e.g. Nightwatch, TestCafe, and a host of others. In some ways I made life easy by only considering a polyglot team. If you specialise in Typescript and JavaScript then you have a lot of options to evaluate.

WebdriverIO

https://webdriver.io/
a JavaScript based abstraction on top of WebDriver protocols and Appium and Puppeteer (https://pptr.dev/)
Programming languages supported: Node.js (JavaScript/Typescript)

Cypress

https://www.cypress.io/
Chrome, Edge, Firefox, WebKit (experimental)
Programming languages supported: Node.js (JavaScript/Typescript)

Puppeteer

https://pptr.dev/
Chrome Dev Team maintained JavaScript library that is an abstraction on top of the WebDriver Bidi protocol, works with Chrome, Chromium and Firefox
Not recommended for normal Automated Coverage use cases, but if you want something lightweight that only targets a specific browser set, in JavaScript then have a look.
Programming languages supported: Node.js (JavaScript/Typescript)
Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Software Testing Podcast - Episode 026 - Portfolio Projects - The Evil Tester Show

Alan Richardson — Wed, 17 Sep 2025 09:00:00 +0000

Should you have an online portfolio showcasing your skills and abilities to help get a job?

It really depends on the recruitment process. But… if I’m recruiting, and you have a profile then I will have looked at it. So it better be good.

Welcome to The Evil Tester Show! Covering the common question of “Should we have portfolio projects?” Alan shares his experience as both a creator and reviewer of portfolios, covering everything from the dos and don’ts of GitHub repos, to which skills actually benefit from a showcase, and how a well-crafted readme can make a difference. Whether you’re considering building a portfolio, looking to upgrade your current one, or wondering where to focus your efforts, this episode has practical advice to help you stand out as a top 1% applicants and avoid creating a portfolio that could hurt more than help.

audio and subscribe links here

Episode Summary

A practical look at online portfolios for developers and testers. Covering both trying to find your next role and also looking to switch careers. Most people don’t have public portfolios so that means you can really stand out if you build a good one.

We explore what actually matters in a portfolio: the difference between learning repos, personal projects, and true showcase projects designed for job hunting. We break down the content that impresses, and the stuff that doesn’t. We emphasize the need for simplicity and quality. Expect to find tips about managing forks on GitHub, writing an effective README, documenting your learning process, and promoting your work across LinkedIn and blogs to maximize visibility.

Tailor your projects to communicate exactly what you want and make your portfolio a focused sales tool rather than a catch-all for everything you’ve ever coded. Plus, advice on how to keep your repo clean, how to handle learning projects when switching roles or languages, and why you should never be afraid to show that you’re still learning.

Most recruiters won’t look at your portfolio, but when they do, you’ll want to be in the top 1%. If you do decide to build a portfolio, make sure it represents your best work.

Outline

00:00 Portfolio Value

The episode starts by discussing the real-world value of online portfolios. Most recruiters don’t look at them, but if the right person does, yours better be impressive. The host shares personal experiences with portfolios and offers perspective on how useful these tools can really be.

02:59 Stand Out Skills

We dig into what actually makes you, and your portfolio, stand out. Typical skills tested during interviews like defect writing, test cases, or strategies do not need to be shown off in a repo. Instead, focus on things that make you unique, such as personal opinions, in-depth case studies, or examples of thought leadership. This is often best done through blogs or articles, linked to your GitHub on your LinkedIn profile.

09:19 Project Types

A breakdown of project types:

Learning Projects: Messy, experimental, used to learn new tools, languages, or skills. It’s okay if they’re not polished, but document what you learned.
Personal Projects: Hobbies or interests that show personality and initiative. These don’t need to be perfect but should reflect genuine enthusiasm and curiosity.
Portfolio Projects (Showcase Projects): Your best, most complete work. These should be polished, well-documented, and focused on the skills you’re trying to sell.

Making these distinctions helps recruiters judge your work appropriately. Always clarify the purpose in your README files.

12:27 Showcase Projects

Showcase projects are your sales pitch—they need to be as close to perfect as possible. The host shares advice on what to include in a GitHub repo:

Use your profile README to introduce yourself professionally.
Only include original repos, not forks, to keep things clean.
Don’t commit IDE or compiled files; use .gitignore.
Use static analysis tools and add unit tests to show you care about quality.
Readme documentation should cover project intent, installation instructions, known limitations, and choices made during development.
Whenever possible, leverage GitHub Actions to show your code works.

Recency and maintenance matter when looking for work. Keep your portfolio projects updated and relevant.

19:39 Promoting Yourself

Once you’ve created a portfolio project, get the word out. Add it to your LinkedIn profile, feature it as a project, and share articles about your work. Updating and promoting your portfolio regularly keeps it visible and demonstrates your ongoing growth.

21:44 Final Advice

Most professionals don’t bother with portfolios, a missed opportunity for many, but if you go the extra mile to create one you can stand out. Make sure it’s good, you don’t want to stand out because it’s bad. Even if recruiters don’t look at it most of the time, the right one will, and that’s when it really helps. Sharing your learning and even your mistakes is crucial for personal development, especially as you gain more experience.

Key Takeaways

Most developers and testers don’t have a portfolio, so building one sets you apart when it’s high quality.
A strong online portfolio acts as a sales tool, focusing on your best, most relevant skills for the job you want.
Differentiate between learning projects, personal hobby repos, and polished, focused showcase repositories.
Keeping documentation clear, and up-to-date is just as important as good code in your portfolio.
Promoting your portfolio on platforms like LinkedIn and your blog can drive the right attention.

Quotes

“Should you have an online portfolio showcasing your skills and abilities and will that help you get a job? Well, I got an online portfolio, I’ve had one for years. But I also know that people don’t look at it.”

“If you’ve got a portfolio and it’s bad, it might put interviewers off. If you’ve got a learning project and you’ve been copying someone on YouTube… When people look at that, they’re gonna go, ‘This person doesn’t know what they’re doing.’”

“What you want to showcase is something that makes you different, a reason why someone would pick you up as opposed to someone else.”

“Portfolio project says, ‘I know how to do this, therefore judge me on the work.’ A learning project is, ‘I learned from these other people.’”

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.