DEV Community

Katja
Katja

Posted on

Building an AI QA Agent That Tests Your Jira Tickets in a Real Browser

#ai #testing #automation #opensource

Why I built this

Give it a Jira ticket ID and a test environment URL, and this agent reads the ticket, writes test steps, and runs them in a real Chrome window while you watch.

It's built from two open-source pieces. Agent Factory is the studio where agents are designed, edited, and chatted with. BaaS (Browser as a Service) is the part that actually drives the browser: it opens pages, clicks, waits, and takes screenshots on the agent's behalf.

This post walks through setting both up locally and running the bug check agent against your own app.

What you'll need

Required:

  • A GitHub account
  • An OpenAI API key (Anthropic works too, but I recommend starting with OpenAI)
  • Google Chrome

Recommended:

  • Docker. The instructions below assume it, but a bare-metal setup works as well.
  • Go, so you can run BaaS locally.

Step 1: Start Agent Factory

Clone the repo: Ursa-Minor-Beta/agent-factory-docker-api-ui.

Then create your env file and bring everything up with the bundled database:

cp .env.example .env
# edit .env – at minimum set the admin email and password
docker-compose --profile with-db up --build
Enter fullscreen mode Exit fullscreen mode

The admin email and password in .env are what you'll use to log in to the factory later, so pick something you'll remember.

Step 2: Run BaaS locally

I recommend running BaaS locally while you debug and in Docker while you test, because locally you can watch exactly what the agent does in the browser. If the local setup fights you (OS permissions and the like), fall back to Docker. It's stable.

Start MongoDB and create the env file:

cd baas
docker compose up -d mongodb
cp .env.example .env
vim .env
Enter fullscreen mode Exit fullscreen mode

Set these values in .env:

API_KEY=<pick-a-key>

BROWSER_EXECUTABLE='/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
BROWSER_HEADFUL=true

LLM_CLIENT=openai
OPENAI_TOKEN="<your token here>"
OPENAI_ORGANIZATION="<your organization here>"
Enter fullscreen mode Exit fullscreen mode

BROWSER_HEADFUL=true is what makes the Chrome window visible. The path above is for macOS; point it at your own Chrome binary on other systems. Remember your API_KEY, because the studio needs it in Step 3.

Next, map host.docker.internal to localhost so the Dockerized factory and the local BaaS can find each other:

sudo vim /etc/hosts
# add this line:
127.0.0.1       host.docker.internal
Enter fullscreen mode Exit fullscreen mode

Finally, load the env and start BaaS:

set -a && . ./.env && set +a
go run ./cmd/baas
Enter fullscreen mode Exit fullscreen mode

You'll see warnings about missing Pandoc, PDF-to-image, and wmctrl. Ignore them: you don't need those tools, and this is what a successful start looks like.

Step 3: Connect the studio and run a smoke test

Open the studio at http://localhost:8080 and log in with the admin credentials from Step 1. Then:

  1. Go to Secrets and add BAAS_API_KEY, set to the same value as API_KEY in the BaaS .env.
  2. Go to Provider and add your OpenAI (or Anthropic) API key.

That's the whole setup. To check the browser actually works, go to Agents → Browser Screenshot and enter https://google.com.

A new Chrome window will open. It may ask for permissions, so allow them. If everything is wired up, the window loads Google and the chat returns a browser session ID plus a screenshot of the page.

Step 4: Wire up Jira and the Test Orchestrator

Create a Jira auth secret

Log in to Atlassian API tokens and choose Create API token with scopes. Then encode your email and token together:

echo -n "you@example.com:your-api-token" | base64
Enter fullscreen mode Exit fullscreen mode

Add the output to the studio's Secrets under the name Jira_auth.

Configure the orchestrator

In the factory, go to Agents, find Test Orchestrator, and click Edit. Two things to change:

  1. In the input-1 node, replace the default URL (https://google.com) with your test environment URL, and set your jiraSubdomain.
  2. Decide how to handle login in the http-1 node.

For login, I recommend hardcoding it in http-1. Login flows rarely change, so scripting them once saves tokens on every run.

If you'd rather skip a separate login step, set the body of http-1 to the following. It starts a browser session, opens the URL, and takes a screenshot:

"body": "{\"program\":\"navigate('{{node:input-1.url}}'); sleep('3s'); waitReady(body,'timeout:20s'); takeScreenshot('screenshot', 'timeout:10s');\", \"sessionID\": \"{{node:http-0.response.result.sessionID}}\", \"stopSession\": false}",
Enter fullscreen mode Exit fullscreen mode

Step 5: Run your first test, then tune it

Open Test Orchestrator and click Chat. Send it your Jira ticket ID (something like SCRUM-165) and your test environment URL. The agent takes it from there and runs the first test.

Teach it about your app

The prompt in llm-generate-steps is where test steps get written. Replace the example details there with what you know about your application. Pasting in your user documentation, if you have any, works well.

Tune the Test Step Executor

The orchestrator uses Test Step Executor as a skill, and its prompts are worth adapting to your app and its quirks. Each LLM node has one job:

  • llm-0 adapts each test step to what's actually on screen, using the screenshot. Tell it about any layout or navigation it should expect.
  • llm-1 writes the BaaS commands. Tell it which selectors to prefer.
  • llm-2 decides whether a step passed, failed, or is broken. Tell it anything to check closely, like whether the logo is visible.

Over to you

I hope this agent saves you some manual testing. If you run into problems or have ideas, get in touch.

One honest ask: this is a side project, and I'd like to know whether to keep building it. If it's useful to you, star the repo on GitHub, send me a DM, or leave a comment below. Any of those tells me the time is well spent.

Keep humans on the loop and don’t ship on Fridays!

Top comments (0)