DEV Community

Cover image for 1 human check, 8 scripted steps: QA in our coding agent pipeline
Panth Patel
Panth Patel

Posted on Originally published at panth.vardayinitech.in

1 human check, 8 scripted steps: QA in our coding agent pipeline

After a developer drags a ticket to Git Push, eight steps run on their own, from the commit to the health check on our dev VM. No agent is involved in any of them. The step right before them is always a person: the developer who asked for the change opens a preview and decides whether it is what they asked for.

I'm Panth, and I lead the software team at Oizom, where Envizom is the platform. This post is about that one human step in our agent pipeline: where it sits, what it checks, and why we kept it on purpose.

Where QA sits

A developer writes what they want in a ticket, in plain English, and drags it to Approved. Our orchestrator, orch, gives the ticket its own stack: a branch in every repo, its own services, a copy of the dev database and a private preview URL. A coding agent does the work inside that stack.

When the agent is done, it posts the preview link in the ticket thread and moves the ticket to QAQC. That is as far as it goes. The agent never marks its own work as done.

Approved ──▶ agent works in the ticket's own stack
                │
                ▼
             QAQC  ◀──── feedback in the thread, ticket moved back
                │            (same agent, same context)
                │ a person tests the preview
                ▼
            Git Push ──▶ 8 scripted steps, no agent ──▶ dev VM
                                                       (rollback if unhealthy)
Enter fullscreen mode Exit fullscreen mode

What the person does

The developer who asked opens the preview and tests it. If something is off, they reply in the same thread and move the ticket back. The same agent picks it up with the same context and carries on.

In our demo video I asked for a camera layout change on a device page. The agent shipped it and posted the preview. My QA reply was: "also handle more than 4 cameras with a horizontal scroll, and set up 8 test cameras on this device". The agent did both, and it created the test data in the ticket's own copy of the database, so nothing it added touched anyone else's data.

That reply is a QA note and a new requirement in one message, and it went to the agent the same way the original ask did: as words in the ticket thread.

Why it stays human

The QA step is where someone checks that the thing that was built is the thing that was asked for.

If the agent graded its own work, the one review that matters most would disappear. Every step after it is built to be boring and repeatable, and none of them asks whether the change was the right change. QA is the only place in the pipeline where that question gets asked, so it is the one place we did not automate.

It also keeps the record honest. What was asked, what was built, which preview was tested and what feedback went back all sit in one ticket thread, and the decision to ship is a person's action in that thread, not the agent's.

Everything after QA is a script

When the developer is happy, they drag the ticket to Git Push. From there orch:

# Step
1 commits the work on the ticket branches
2 merges the current dev branch into them
3 opens a pull request in each affected repo
4 merges the pull requests
5 posts the PR and commit links back into the ticket thread
6 waits for CI
7 rolls the new images onto the dev VM
8 runs health checks against the VM

If the VM is not healthy afterwards, orch rolls it back to the previous images, reverts the merges, and tells the thread what happened. When it is healthy, the ticket moves to Completed and the developer gets a notification.

None of this is an agent. It is deterministic, it is tested, and when it breaks I read a log and fix it. That is what lets me leave the merge and the deploy to a machine: the only judgement call in the chain has already been made by a person.

It stops at dev

All of that automation, the auto-merge, the VM update, the rollback, stops at the dev environment. Production goes through a separate path where people make the call, on a slower cadence.

This is deliberate too. We do not want automation deciding what reaches production, and we do not want a fast feature list to become the goal. A human QA step on dev and a human decision for production are two different checks, and we kept both.

My own review moved earlier

QA is the developer's step. Mine happens before anything is built. I read the architecture docs, the plans and the prompts, and they tell me about 90% of what is coming. The other 10% is reviewing the output and writing QAQC cases.

So the agent sits between two kinds of human attention: the plan and the prompt going in, and the preview test coming out. It does the engineering in the middle, inside a stack that exists only for its ticket.

If you are setting this up

Give the agent a place to stop that is not "done". For us that is a column called QAQC, a preview link and a thread where the reply goes. Make the person who asked for the change the one who tests it, because they are the only one who knows what they meant. Then script everything after that decision, and keep it out of production.

If your team runs coding agents, who decides that a task is done: the agent, or the person who asked for it?

Top comments (0)