DEV Community: shankar subramanian

I turned plain English (and any live website) into API tests

shankar subramanian — Sun, 26 Jul 2026 22:39:06 +0000

Hi 👋 This is my first post here, and I want to share a side project I built as a QA engineer — because it scratches an itch I think a lot of us have.

Here's the itch: for one business flow, I'd end up building the same thing three times — a Postman collection, a Playwright API test, and a JMeter plan. Same requests, same variables, same assertions, just typed out three different ways. It's slow, it's boring, and the three copies drift apart the moment anything changes.

So I built TestFlow Agent to do it once. It works in two ways, and I want to walk through both — because they solve two pretty different situations.

Part 1 — When it's your service: just describe the flow

The first mode is for when you have control over the API you're testing — it's your team's service, you know the endpoints, maybe it even lives in the same repo.

In that case, you don't need to record anything. You just type the test in plain English:

Create an enrollment for a Texas member.
Get available plans for Texas. Select a Silver plan.
Submit enrollment with effective date 01/01/2026.
Validate enrollment status is ACTIVE.

…and TestFlow Agent reads the intent — the state, the plan, the date, whether it's a happy path or a negative case — and produces the API sequence:

POST {{baseUrl}}/auth/token
POST {{baseUrl}}/members
GET  {{baseUrl}}/plans?state=TX
POST {{baseUrl}}/enrollments
GET  {{baseUrl}}/enrollments/{{enrollmentId}}

From that, it generates the Postman collection, the Playwright spec, and the JMeter plan — all sharing the same variables. And because the repo ships a small mock API, you can hit Run and watch the flow actually execute end-to-end (enrollment created, status ACTIVE, IDs chained). It's not just static text; it runs.

This part is great for teams testing their own services: you already know the flow, so describing it is faster than clicking through it.

But it has a limit — it works best when the flow is known and the service is "yours." Which led me to the part I'm actually more excited about.

Part 2 — When it's any app: just record it

The second mode needs zero knowledge of the API. You point it at any website, click through your flow by hand, and it figures out the tests for you.

You enter a URL, hit Start Discovery, and a real browser opens. You log in and do whatever you'd normally do — I tested it against a live OrangeHRM demo — then hit Stop. Behind the scenes it recorded all the network traffic, and now it hands you the same three artifacts (Postman / Playwright / JMeter), built from what you actually did.

The catch is that raw browser traffic is messy, and making it into tests that run again is where most of the real work went. A few of the things it has to get right:

Figuring out what's a variable. IDs, tokens, employee numbers — it detects them and chains them across steps, so step 4 uses the ID that step 2 created.
Naming them sensibly. It names a variable from the JSON field it came from (employee.id → employeeId), not from the URL — because on real apps every URL says "employees" whether or not that's what the ID means.
Handling login tokens. Captured CSRF tokens are dead by the next run, so instead of replaying a stale one, it re-fetches a fresh token before logging in. That's the difference between a test that logs in and one that just 403s.
Cutting the noise. Fonts, analytics, i18n bundles, tracking pixels — all filtered out, so you're left with the actual business API calls.
Not corrupting things. A tiny one that bit me: an ID of 7 must never get find-and-replaced inside a number like 4700. Short IDs are only matched on whole tokens.

You don't have to think about any of that — you click through the app, and you get runnable tests. That's the part I think anyone can benefit from, on any app, without touching the source code.

A quick before/after

Here's one real request it captured from that OrangeHRM session:

What the browser sent:

PUT /web/index.php/api/v2/performance/reviews/17/evaluation/employee
{ "ratings": [ { "kpiId": 1, "rating": null } ] }

What it generated:

PUT {{baseUrl}}/web/index.php/api/v2/performance/reviews/{{reviewId}}/evaluation/employee
{ "ratings": [ { "kpiId": "{{kpiId}}", "rating": null } ] }

reviewId and kpiId are now variables that resolve from a generated environment file — so you import it and run, no manual filling-in.

Try it

It's open source (MIT) — Node + Express + React — with a one-command Docker start:

git clone https://github.com/sshankar07/test-flow-agent
cd test-flow-agent
docker compose up --build     # then open http://localhost:5173

(The live-recording part opens a real browser, so run that bit natively — the README shows both ways.)

👉 Repo: https://github.com/sshankar07/test-flow-agent

That's it — my first dev.to post! If you're in testing/automation, I'd love to know: what would you want it to generate next — k6? RestAssured? A CI pipeline? Drop a comment 🙌

Clearing a false positive: my error guard matched too much

shankar subramanian — Sun, 26 Jul 2026 19:31:12 +0000

Project Overview

I built TestFlow Agent — an open-source tool that turns plain-English test cases (or a live browser recording) into runnable Postman, Playwright, and JMeter tests.

Its live-discovery feature launches a headed Chromium via Playwright to record real traffic. That only works where there's a display, so when someone runs the backend inside Docker (no X server), the launch fails. To make that failure friendly, I added a guard that catches the error and returns a clear 503 — "live discovery needs a headed browser; run the backend natively."

Good idea. Buggy execution.

Bug Fix or Performance Improvement

My guard decided "this is a no-display environment" by string-matching the launch error message. I included has been closed in the match:

The problem: Playwright throws "Target page, context or browser has been closed" for a whole range of unrelated failures (a crashed context, a closed page mid-call, a killed browser). Every one of those would now be misdiagnosed as "you're in Docker, run natively" — pointing the user at completely the wrong problem.

An AI code review (GitHub Copilot) flagged it on the PR, and it was dead right: the heuristic was too broad.

Code

Before (backend/services/discoveryService.js):

if (/XServer|X server|\$DISPLAY|has been closed/i.test(message)) {
  const error = new Error('Live discovery needs a headed browser ... run natively');
  error.statusCode = 503;
  throw error;
}
throw launchError;

After:

// Match only display/X-server markers — NOT generic Playwright phrases like
// "Target page, context or browser has been closed", which are unrelated failures.
if (/XServer|X server|\$DISPLAY/i.test(message)) {
  const error = new Error('Live discovery needs a headed browser ... run natively');
  error.statusCode = 503;
  throw error;
}
throw launchError;

🔗 PR: https://github.com/sshankar07/test-flow-agent/pull/5

I verified both directions with a quick check:

const re = /XServer|X server|\$DISPLAY/i;
re.test('...without having a XServer running. Missing X server or $DISPLAY'); // true  ✅ real no-display error still caught
re.test('Target page, context or browser has been closed');                  // false ✅ unrelated error no longer misfires

My Improvements

The guard now fires only on genuine display/X-server errors — no more false "run it natively" advice when the real cause was something else entirely.
Unrelated Playwright failures once again surface as themselves, instead of being swallowed by a misleading 503.
Bonus from the same review pass: the Docker builds moved from npm install to npm ci (reproducible installs, no accidental lockfile drift).

Best Use of Sentry

Not used in this submission.

Best Use of Google AI

Not applicable — though it's worth noting the bug was caught by an AI code review on the pull request, which is a nice argument for keeping AI reviewers in the loop for exactly this kind of over-broad heuristic.

I Dockerized my app and one button broke — thanks, localhost

shankar subramanian — Sun, 26 Jul 2026 19:30:40 +0000

Every developer has met this bug. I just met it again in the most on-brand way possible.

The setup

My side project, TestFlow Agent, is three little services:

a mock API (the thing under test),
a backend that generates and runs test flows,
a React frontend.

One of the headline features is a "Run Enrollment Flow" button: the backend fires a real sequence of requests at the mock API and reports back "enrollment created, status ACTIVE." Locally, it worked every single time.

Then I added a one-command Docker setup so anyone could try it without juggling three terminals. docker compose up --build, open the app, click around… analyze works, generate works, and then — "Run Enrollment Flow" fails. Only that button. The one that talks between services.

The hunt

The frontend calls the backend fine (both ports are published to the host). Generation is pure computation — fine. The only thing different about "Run Flow" is that the backend itself makes an outbound HTTP call to the mock API.

I opened the backend and there it was:

const baseUrl = 'http://localhost:4000';   // 👈

Natively, localhost:4000 is the mock API — same machine. But inside the backend container, localhost means the container itself. There's no mock API on port 4000 in there. The backend was cheerfully calling itself and getting nothing.

Classic "works on my machine," except the machine changed out from under it the moment it went into a container.

The fix

In Docker Compose, services reach each other by service name, not localhost. So I made the target configurable and pointed it at the service name inside the compose network — while keeping localhost as the default so native runs don't change at all:

// backend/routes/analysisRoutes.js
const baseUrl = process.env.MOCK_API_URL || 'http://localhost:4000';

# docker-compose.yml
backend:
  environment:
    - MOCK_API_URL=http://mock-api:4000   # reach the mock by its service name
  depends_on:
    - mock-api

🔗 PR: https://github.com/sshankar07/test-flow-agent/pull/3

Before / after

Before: Run Flow in Docker → backend calls localhost:4000 → nothing there → failure.
After: Run Flow in Docker → backend calls mock-api:4000 → cross-container SUCCESS. I watched it create an enrollment (ENR-4C554E55, status ACTIVE) from inside the compose stack.

What I took away

Hardcoded localhost is a time bomb the day you containerize. It's invisible until the network boundary moves.
The fix isn't "hardcode the service name" either — it's make it configurable with a sane default, so both native and Docker runs work with zero edits.
Container-to-container = service-name DNS; host-to-container = published ports. Mixing them up is the root of a huge share of "it works locally but not in Docker" bugs.

Small diff, satisfying fix, and a reminder that the oldest bugs are the ones that keep showing up in new outfits.

Repo (open source, MIT): https://github.com/sshankar07/test-flow-agent