DEV Community: Katja

My testing agent remembers what it learned. Mostly.

Katja — Thu, 01 Oct 2026 14:53:40 +0000

TL;DR: An open-source agent that reads a Jira ticket, writes test steps, runs them in a real Chrome window, judges the result from screenshots, and reports back to Jira. The new version has a memory: it reuses browser automation code that worked and drops code that broke. Setup is at the end.

A little while ago I wrote about an AI QA agent that tests your Jira tickets in a real browser. You give it a ticket ID and a URL. It reads the bug, writes test steps, drives Chrome through them, judges the result from screenshots, and reports back to Jira. If everything passes, it moves the ticket to Done.

I've updated it since. The big change: it has a memory now.

The old version started from zero on every ticket. Every step meant an LLM writing fresh browser code, even for a click it had already done a hundred times. That works, but it's slow, it costs tokens, and it's a bit like a colleague who forgets everything over the weekend.

The new version remembers automation code that worked and reuses it when a similar step comes up again, even on a different ticket and phrased differently. It also forgets code that broke, so it doesn't keep trusting something broken.

Here's how the updated version works and why I built it this way. If you want to run it yourself, skip to How to try it.

Two agents, one phone call between them

There are two agents.

The main one, the orchestrator, turns a Jira ticket into a list of test steps and coordinates running them. The second one, step_execution, does exactly one thing: it executes a single test step against a live browser and tells you how it went.

The orchestrator calls the sub-agent as a tool, the same way it would call any function. Give it a step, get back a verdict. It never sees a selector, a screenshot comparison or a line of browser code. That separation turned out to matter more than I expected: I can rework the sub-agent without touching the planner.

Here's the rough shape:

Jira ticket ID ──▶ [Main agent]
                     ├─ Start browser session (BaaS)
                     ├─ Log in to the app under test
                     ├─ Fetch Jira issue + comments
                     ├─ LLM: generate ordered test steps
                     └─ Orchestrator LLM
                          │  one tool call per step
                          ▼
                   [Sub-agent: step_execution]
                     ├─ Screenshot + HTML snapshot
                     ├─ LLM: which page is this? check memory
                     ├─ Reuse a stored program OR write a new one
                     ├─ Run the program in the browser
                     ├─ LLM: compare screenshots with expected result
                     └─ Memory: keep good programs, drop bad ones
                          │  verdict per step
                          ▼
                   back to main agent
                     ├─ Broken step? Retry it
                     ├─ Test finished -> Post results as a Jira comment
                     ├─ All passed? Move ticket to Done
                     └─ Stop browser session

The main agent: from ticket to test plan

The input is small: a URL, a Jira ID and a Jira user email.

It starts a real browser. The agent calls our Browser-as-a-Service (BaaS) API to get an isolated browser session, then logs in. BaaS accepts a program string in a tiny automation language, with commands like navigate(...), clickN(...), sendKeysToElement(...) and takeScreenshot(...), sent as JSON to the BaaS runner. What comes back is the result: a sessionID, which travels through the whole run, and a screenshot of the freshly logged-in app.

It reads the whole ticket, comments included. Two calls to the Jira REST API, one for the issue and one for its comments. Small JavaScript nodes clean both up. The comments matter. That's often where the real reproduction steps are hiding.

It plans like a QA engineer. The LLM gets the bug summary, the description, the comments, the current screenshot, and a library of reusable procedures for the app we test, which is a platform for running EOS/L10 meetings. For example, "start an L10 meeting" has two variants, depending on whether a Resume button is already on screen.

It has to return strict JSON, an ordered list of steps:

{ "stepDescription": "...", "stepExpectedResult": "...", "sessionID": "..." }

The sessionID is filled in by a template, and the prompt forbids the model from inventing or changing it.

An LLM runs the loop. The orchestrator gets the whole list and one tool: the sub-agent. Its instructions are short:

Execute each step by calling the tool. Wait for the result before proceeding. If status is test_broken, retry once. If still broken, or if failed, stop execution.

It reports back. The orchestrator's transcript of tool results goes to Jira as a comment. A JavaScript node checks the status of the last step. If it passed, the agent finds the ticket's "Done" transition and fires it. If not, the ticket stays open and waits for me. At the end the browser session is shut down.

The sub-agent: run one step, remember it

This is the part I'm happy about. step_execution gets a session ID, a step description and an expected result. It does five things.

1. Look before touching anything. One BaaS call returns a screenshot and the page's HTML. A second call asks BaaS which automation commands it currently supports. That list goes straight into the code-generation prompt, so the language isn't frozen into my prompt text. If we add a command to BaaS tomorrow, the agent can use it tomorrow.

2. Work out where it is, then check its memory. An LLM classifies the screenshot into one of a fixed set of pages, the old Page Object idea: Active Meeting, Issues, Scorecard and so on. Then it has to search a memory collection called test_steps, filtered by that page. It's looking for a step that means the same thing, the same action with the same expected outcome, even if it's worded differently. It returns the step plus one flag: from_memory, true or false.

3. Reuse or write. A branch splits on that flag.

Found: pull the stored program from memory. No code generation at all.
Not found: a second LLM writes a new program from scratch. It can only use the commands discovered in step 1, and it has house rules. Use clickN for repeated selectors. Fall back to llmClickElement when there's no reliable selector. Output one line, no template literals.

That's the bet. The first time the agent sees something like "click Raise Issue on a to-do," an LLM writes the code. The next time a similar step shows up, on a different ticket, phrased differently, it should come straight from memory.

4. Run it and look again. The program, stored or new, goes to the browser, followed by another screenshot. A third LLM compares before and after against the expected result. It returns one of four verdicts: success, failed, broken or test_broken.

5. Remember or forget. A final branch decides what happens to memory.

success or failed: store the program that was used, tagged by page.
broken or test_broken: delete the stored program that just produced a bad result.

So code that works gets reused, and code that misbehaves gets thrown out. The agent shouldn't keep trusting automation that just let it down. That's the theory, anyway.

How to try it

The whole thing is open source and runs on Docker. It's built from two pieces. Agent Factory is the studio where you design, edit and chat with agents. BaaS (Browser as a Service) is what actually drives Chrome: it opens pages, clicks, waits and takes screenshots for the agent.

Ursa-Minor-Beta / agent-factory-docker-api-ui

Docker deployment for Agent Factory (agent-factory + agent-factory-ui).

Agent Factory Deploy

Docker deployment for Agent Factory (agent-factory + agent-factory-ui).

Quick Start

Copy environment file:

cp .env.example .env

Edit .env with your configuration (at minimum set ADMIN_EMAIL and ADMIN_PASSWORD)
Run with local MongoDB:

docker-compose --profile with-db up --build

Or with external MongoDB (set MONGODB_URI in .env):

docker-compose up --build

Access
- UI: http://localhost:8080
- API: http://localhost:3000

Deploy Specific Version

API_VERSION=<branch/tag> UI_VERSION=<branch/tag> docker-compose up --build

Update After Branch Changes

docker-compose --profile with-db build --no-cache && docker-compose --profile with-db up

Configuration

Variable	Description	Default
`API_VERSION`	API branch/tag to deploy	`master`
`UI_VERSION`	UI branch/tag to deploy	`master`
`API_PORT`	API exposed port	`3000`
`UI_PORT`	UI exposed port	`8080`
`MONGO_PORT`	MongoDB exposed port	`27017`
`MONGO_USER`	MongoDB root username	-
`MONGO_PASSWORD`	MongoDB root password	-
`JWT_SECRET`	JWT signing secret	-
`ENCRYPTION_KEY`	Encryption key (32 bytes)	-
`MONGODB_URI`	MongoDB connection string	`mongodb://mongo:27017/agent-factory`
`VITE_API_URL`	API URL for UI	`http://localhost:3000`

…

View on GitHub

Ursa-Minor-Beta / baas

Browser as a Service — browser automation and document extraction over HTTP

BaaS — Browser as a Service

BaaS drives a real Chrome instance behind an HTTP API. You POST a small JavaScript-like program describing what to do in the browser; BaaS runs it and returns the page, a screenshot, cookies, a downloaded file, or Markdown extracted from whatever the page served you.

It also parses documents without a browser: PDF, DOCX, XLSX, PPTX, CSV and plain text all convert to Markdown through the same API.

Browser automation — navigate, click, type, wait, upload, download screenshot, run arbitrary JS, all through ChromeDP.
LLM-assisted actions — llmClick('the login button') llmLogin(user, pass), llmText('one-time code'), for pages whose selectors you do not know in advance. Backed by OpenAI, Azure OpenAI or Ollama.
Sync and async sessions — one-shot programs, or a long-lived session you send commands to over time.
Readability extraction — article text, tables and metadata from a URL.
Document conversion…

View on GitHub

The short version:

Start Agent Factory. Clone the repo, set an admin email and password in .env, and run:
```
docker-compose --profile with-db up --build
```
Run BaaS. Clone the repo and set an API key, BROWSER_HEADFUL=true and your LLM credentials in its .env. If you run BaaS locally, also set your Chrome path. Locally it operates your own browser, and you get to watch the agent click around in a real window. Map host.docker.internal to 127.0.0.1 so the two can find each other, then start BaaS. It will complain about missing Pandoc and friends. Ignore it, that's what success looks like.
Connect and smoke test. In the studio at localhost:8080, add BAAS_API_KEY under Secrets and your LLM key under Provider. Then run the Browser Screenshot agent on google.com. If you get a screenshot back, you're wired up.
Hook up Jira. Create an Atlassian API token, base64-encode email:token, and save it as the Jira_auth secret:
```
echo -n "you@example.com:your-api-token" | base64
```
In the Test Orchestrator agent, set your test environment URL and Jira subdomain, and hardcode your login in the http-1 node. Login rarely changes, so there's no point paying tokens for it every run.
Run it and tune it. Chat with Test Orchestrator, give it a ticket ID like SCRUM-165 and your URL, and watch. Then teach it your app: describe it in the llm-generate-steps prompt (your user documentation works well here), and adjust the three Test Step Executor prompts. llm-0 adapts steps to the screen, llm-1 writes the BaaS commands, and llm-2 decides pass, fail or broken.

Every command and config value is in my previous post: Building an AI QA Agent That Tests Your Jira Tickets in a Real Browser.

This is a side project, and I'd honestly like to know whether to keep building it. If it's useful, star the repo, send me a DM, or leave a comment. If it breaks on your app, tell me that too. That's often the more useful message.

Keep humans on the loop, and don't ship on Fridays.

Building an AI QA Agent That Tests Your Jira Tickets in a Real Browser

Katja — Fri, 25 Sep 2026 17:05:32 +0000

Why I built this

Give it a Jira ticket ID and a test environment URL, and this agent reads the ticket, writes test steps, and runs them in a real Chrome window while you watch.

It's built from two open-source pieces. Agent Factory is the studio where agents are designed, edited, and chatted with. BaaS (Browser as a Service) is the part that actually drives the browser: it opens pages, clicks, waits, and takes screenshots on the agent's behalf.

This post walks through setting both up locally and running the bug check agent against your own app.

What you'll need

Required:

A GitHub account
An OpenAI API key (Anthropic works too, but I recommend starting with OpenAI)
Google Chrome

Recommended:

Docker. The instructions below assume it, but a bare-metal setup works as well.
Go, so you can run BaaS locally.

Step 1: Start Agent Factory

Clone the repo: Ursa-Minor-Beta/agent-factory-docker-api-ui.

Then create your env file and bring everything up with the bundled database:

cp .env.example .env
# edit .env – at minimum set the admin email and password
docker-compose --profile with-db up --build

The admin email and password in .env are what you'll use to log in to the factory later, so pick something you'll remember.

Step 2: Run BaaS locally

I recommend running BaaS locally while you debug and in Docker while you test, because locally you can watch exactly what the agent does in the browser. If the local setup fights you (OS permissions and the like), fall back to Docker. It's stable.

Start MongoDB and create the env file:

cd baas
docker compose up -d mongodb
cp .env.example .env
vim .env

Set these values in .env:

API_KEY=<pick-a-key>

BROWSER_EXECUTABLE='/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
BROWSER_HEADFUL=true

LLM_CLIENT=openai
OPENAI_TOKEN="<your token here>"
OPENAI_ORGANIZATION="<your organization here>"

BROWSER_HEADFUL=true is what makes the Chrome window visible. The path above is for macOS; point it at your own Chrome binary on other systems. Remember your API_KEY, because the studio needs it in Step 3.

Next, map host.docker.internal to localhost so the Dockerized factory and the local BaaS can find each other:

sudo vim /etc/hosts
# add this line:
127.0.0.1       host.docker.internal

Finally, load the env and start BaaS:

set -a && . ./.env && set +a
go run ./cmd/baas

You'll see warnings about missing Pandoc, PDF-to-image, and wmctrl. Ignore them: you don't need those tools, and this is what a successful start looks like.

Step 3: Connect the studio and run a smoke test

Open the studio at http://localhost:8080 and log in with the admin credentials from Step 1. Then:

Go to Secrets and add BAAS_API_KEY, set to the same value as API_KEY in the BaaS .env.
Go to Provider and add your OpenAI (or Anthropic) API key.

That's the whole setup. To check the browser actually works, go to Agents → Browser Screenshot and enter https://google.com.

A new Chrome window will open. It may ask for permissions, so allow them. If everything is wired up, the window loads Google and the chat returns a browser session ID plus a screenshot of the page.

Step 4: Wire up Jira and the Test Orchestrator

Create a Jira auth secret

echo -n "you@example.com:your-api-token" | base64

Add the output to the studio's Secrets under the name Jira_auth.

Configure the orchestrator

In the factory, go to Agents, find Test Orchestrator, and click Edit. Two things to change:

In the input-1 node, replace the default URL (https://google.com) with your test environment URL, and set your jiraSubdomain.
Decide how to handle login in the http-1 node.

For login, I recommend hardcoding it in http-1. Login flows rarely change, so scripting them once saves tokens on every run.

If you'd rather skip a separate login step, set the body of http-1 to the following. It starts a browser session, opens the URL, and takes a screenshot:

"body": "{\"program\":\"navigate('{{node:input-1.url}}'); sleep('3s'); waitReady(body,'timeout:20s'); takeScreenshot('screenshot', 'timeout:10s');\", \"sessionID\": \"{{node:http-0.response.result.sessionID}}\", \"stopSession\": false}",

Step 5: Run your first test, then tune it

Open Test Orchestrator and click Chat. Send it your Jira ticket ID (something like SCRUM-165) and your test environment URL. The agent takes it from there and runs the first test.

Teach it about your app

The prompt in llm-generate-steps is where test steps get written. Replace the example details there with what you know about your application. Pasting in your user documentation, if you have any, works well.

Tune the Test Step Executor

The orchestrator uses Test Step Executor as a skill, and its prompts are worth adapting to your app and its quirks. Each LLM node has one job:

llm-0 adapts each test step to what's actually on screen, using the screenshot. Tell it about any layout or navigation it should expect.
llm-1 writes the BaaS commands. Tell it which selectors to prefer.
llm-2 decides whether a step passed, failed, or is broken. Tell it anything to check closely, like whether the logo is visible.

Over to you

I hope this agent saves you some manual testing. If you run into problems or have ideas, get in touch.

One honest ask: this is a side project, and I'd like to know whether to keep building it. If it's useful to you, star the repo on GitHub, send me a DM, or leave a comment below. Any of those tells me the time is well spent.

Keep humans on the loop and don’t ship on Fridays!