DEV Community: Oleg Malyshev

How We Taught AI to Triage Failed Autotests and File Bugs in the Tracker

Oleg Malyshev — Mon, 10 Aug 2026 20:28:13 +0000

Hi, I'm Oleg. In the previous article I covered how we generate autotests from Swagger and test cases using OpenAPI Generator plus Cursor AI / Claude Code, and how we measure coverage automatically with Swagger Coverage.

This one is about the other end of the pipeline: how we triage failed autotests using Allure TestOps integrated with an issue tracker, an MCP server for TestOps, an MCP server for the tracker, and Cursor AI / Claude Code.

But I won't start with AI. First the process itself: why we need defects in TestOps, how we triage a run by hand, why it quickly becomes drudgery without matcher rules, and what exactly we automated afterwards.

Two notes for readers outside our stack: Allure TestOps is a test management system that ties manual test cases to automated tests, CI runs, and reporting. Yandex Tracker is an issue tracker, roughly analogous to Jira; TestOps integrates with it the same way it integrates with Jira, so if you're on a different tracker, the shape of this stays the same.

The problem: the tests failed — now what?

When autotests fail, the mere fact of a red run tells you almost nothing. You need to work out what actually happened:

a real bug in the product;
the autotest itself broke;
the test data went stale;
the environment or infrastructure went down;
the contract changed but Swagger or the tests weren't updated;
or it's a known problem that already has a ticket in progress.

If all you ever look at is the list of failed/broken tests, the team quickly starts spending time re-triaging the same failures. Today a QA engineer works out that three tests fail because of one backend error; tomorrow another QA engineer opens the same three tests and walks the same path from scratch.

So after a test run we have one simple rule: every failed result must be triaged and linked to a defect. Ideally, after triage the run has no unresolved failed/broken results left.

Why defects in TestOps

A defect in TestOps isn't just another entity sitting next to a test case and a run. It's how you turn a chaotic list of failures into a usable database of causes.

One defect should describe one cause of failure. Not "404", "500" or "AssertionError", but a specific problem — for example, the image-fetching endpoint started returning 404 for a group of scenarios where the contract says 200.

That buys several practical things:

Fewer duplicates. Before creating a new defect, QA checks whether an open defect with the same cause already exists.
Faster repeat runs. If the same error recurs in the next run, TestOps can link the failure to the existing defect automatically.
Clearer release risk. Before a release you can see which critical defects are still open and which failures are already known and not blocking.
A link to development. When the cause is a product bug, the defect is tied to a tracker issue.
Easier to separate concerns. Product bugs don't get mixed up with test data, environment, or authorization problems, or with bugs in the autotests themselves.

The main point: defects stop you re-triaging the same failure after every single run.

What the manual process looks like

The triage procedure is fairly straightforward.

1. Open the run and look at the unresolved results

After the autotests finish, go to the run results in TestOps and open the overview. Unresolved results in a run mean either that no defects have been filed for the failed tests, or that defects exist but specific test results aren't linked to them.

One important constraint: you can only create a defect and link it to a failed test in a run that isn't closed yet. So triage the failures before closing the launch.

2. Analyse the error in the failed test

Next, open the failed test and look at the reason. For example, the test might fail with:

Expected status code <200> but was <404>.

The error text alone usually isn't enough. You need to expand the stack trace and find the line that belongs to your autotest — the class and method where the failure happened.

For example:

IntegrationsRegressTests.getIntegrationsProductsAllImagesWithLimitTest

Now we have two important pieces:

the error text: Expected status code <200> but was <404>.
the failure location in the test code: IntegrationsRegressTests.getIntegrationsProductsAllImagesWithLimitTest

After that you need to determine the cause. Not every failed autotest equals a product bug. Sometimes the problem is the host, a token, test data, setup, cleanup, the Swagger contract, or local deserialization of the response into a generated model.

3. Create a new defect or reuse an existing one

If a suitable open defect already exists, link the failed test to it. But don't stop at a plain manual link — linking the current test result to a defect only works for the current run.

For TestOps to link similar failures itself in later runs, the defect needs an automation rule.

If this cause hasn't come up before, create a new defect. The title doesn't need to be elaborate at this first step, but it should convey the essence of the error. The description can then be fleshed out so a developer understands it:

which endpoint or scenario is failing;
what request was made;
what response was expected;
what response came back;
why this looks like a product bug rather than a test problem.

4. Create or link a tracker issue

When a defect corresponds to a product bug, it should have a tracker issue attached. You can do this straight from TestOps: either link the defect to an existing issue, or create a new one through the bug tracker integration.

This part matters: the issue should be created through the TestOps integration, so the link between run, defect, and tracker issue stays transparent.

5. Configure the automation rule

The automation rule is what makes defects actually save time.

In the simple case, you put the error text in the matcher:

Expected status code <200> but was <404>.

But that pattern can be far too broad. A 404 can show up in different places for different reasons. So it's better to constrain the rule with the stack trace as well:

IntegrationsRegressTests.getIntegrationsProductsAllImages*

A rule like that tells TestOps: if a new run hits "expected 200, got 404" and it happened in the group of tests about fetching images, link that failure to this defect automatically.

This doesn't just catch any 404. It catches one specific recurring cause.

The matcher doesn't always need to go down to a specific test method. If the problem applies to a whole test class, constraining by the class in the stack trace is enough.

And if the error is unique and specific enough on its own, a matcher on just the error message sometimes suffices.

6. Verify the result

After creating a defect, check that:

the defect is linked to the right failed test results;
the defect has an automation rule;
if it's a product bug, a tracker issue is created or linked;
the run overview shows fewer unresolved results;
the matcher isn't so broad that it will hide new, independent errors.

On the next run, if these same tests fail — or new tests fail with the same cause — TestOps links them to the defect automatically. And when the tracker issue is closed, the defect in TestOps can close automatically through the integration too.

The rules we settled on

To keep the defect database from turning into a landfill, we stick to a few rules.

One defect — one cause of failure.
Before creating a new defect, check whether a suitable open one already exists.
Don't merge a product bug, an infrastructure problem, a test data problem, and an autotest bug into a single defect.
Every new defect gets an automation rule.
The matcher must not be too broad: a bare 404, 500, or AssertionError with no stack trace constraint is almost always a bad idea.
Only close a defect once the fix is confirmed and a fresh run has no failures from the same cause.

Classifying failures matters separately. If the cause is the autotest, the test data, authorization, the environment, or a Swagger mismatch with no backend bug behind it — don't file a product defect in the tracker. Otherwise the development team gets noise, and trust in the autotests drops.

Where AI comes in

The manual process works, but it still takes time. Open the run, get all failed/broken results, look at the details, group similar errors, check existing defects, create a matcher, link test results, create or verify a tracker issue, and make sure every field is filled in correctly.

So to speed triage up, we wired in AI plus MCP.

I've written about our MCP server for TestOps before. In short, MCP lets Cursor / Claude Code reach into TestOps, fetch the entities it needs, and write results back. For this task we use the TestOps MCP together with the tracker's MCP.

On top of that I wrote a skill, /testops-run-defects. It takes three parameters:

/testops-run-defects <link to the launch/run in TestOps> <team> <component>

For example:

/testops-run-defects https://allure.example.com/launch/67737 red loylt-gifts

The agent then does roughly what a QA engineer following the procedure would do, only faster and more systematically:

extracts the launch id from the link;
fetches the unresolved failed/broken test results from TestOps;
requests failure details, pulling the stack trace only where it's needed to build a sane matcher;
groups failures by shared cause;
checks open defects and their matcher rules so it doesn't create duplicates;
separates product bugs from test, data, environment, authorization, and Swagger-mismatch problems;
creates defects in TestOps with matcher rules;
creates a linked tracker issue through the TestOps integration;
explicitly links every failed test result to its defect;
via the tracker's MCP, verifies and fills in the issue fields: team, component, type, priority, and tag;
finishes with a control pass: matchers exist, test results are linked, tracker issues are created, no unresolved results remain.

Importantly, the AI must not just file defects for everything red. The skill has guardrails against false defects: if the cause looks like a test data problem, a wrong host, bad authorization, a broken setup, or a bug in the autotest, that result must not become a backend bug.

What it looks like in practice

Below is an example run of the skill. We passed the launch link, the team red, and the component loylt-gifts. The agent fetched the list of failures, triaged them, created defects, linked them to tracker issues, and verified the result.

The output is a short report: how many results were processed, which defects were created, which tracker issues were linked, and whether any unresolved failures remain.

In this example the result was:

failed/broken unresolved processed: 29
defects created: 1279–1287
issues created and linked: CMA-9551 – CMA-9559
unresolved results remaining: 0
every defect has a matcher
all 29 test results are linked to their defects
the tracker issues got the right fields: team, component, type, priority, and the autoapi tag

What's next

Right now I'm actively extending the skill so it fits different teams and different components, rather than one hard-coded mapping.

The next logical step is an AI agent that walks the failed autotests of every team, triages the results, files defects and issues, configures matcher rules, and leaves only the ambiguous cases to a human.

The ideal picture for me looks like this: the autotests fail, the agent triages the run on its own, groups the causes, separates product bugs from test and data problems, creates the necessary defects and tracker issues — and in the morning QA sees not a heap of red tests, but a clear list of causes and actions.

You can't remove the human from this process entirely. But removing most of the drudgery is well within reach.

Short version

Defects in TestOps aren't a formality. They make the failure database manageable, keep duplicates down, and stop you re-triaging the same errors after every run.

Matcher rules turn a defect from a manual annotation into a mechanism for automatically linking recurring failures. Tracker integration connects the autotests to development. And MCP plus Cursor AI / Claude Code automate most of the drudgery around failed results.

In practice that means QA spends less time shuttling information between systems and more time on the thing that matters: working out where the product is genuinely broken, and where the tests, the data, or the infrastructure need fixing instead.

One caveat worth flagging, since it came up when I published this in Russian: the "one defect — one cause" rule runs into the granularity of what the test actually reports. A single assertion covering ten malformed fields will only ever be split as finely as the report shows it. A soft assertion emitting a message per field can be decomposed properly. Worth knowing before you assume the matcher will do all the work for you.

Let’s Work Together

I’m open to QA leadership opportunities and collaborations with teams looking to improve their testing strategy, build scalable test automation, introduce AI-assisted QA workflows, and establish meaningful quality metrics.

If this sounds relevant to your team, feel free to connect with me on LinkedIn.

How We Turned Swagger from Documentation into the Engine Behind Our API Tests

Oleg Malyshev — Sun, 09 Aug 2026 18:31:53 +0000

Hi! I’m Oleg, a QA Engineering Manager focused on test strategy, automation, and AI-assisted QA workflows.

This is a continuation of a series on where AI actually earns its place in QA — and this time the answer starts somewhere less glamorous than AI: with the OpenAPI spec.

The short version: we generate a Java API client from Swagger, write tests against that generated client, measure coverage against the spec automatically in GitLab CI, and hand the resulting gaps to an LLM as a structured JSON report so it can write the missing tests.
The problem: tests that drift away from the contract
A very common failure mode in API testing is that tests live separately from the API contract.

A QA engineer assembles URLs by hand, writes JSON by hand, parses responses by hand. Swagger exists somewhere nearby — analysts and developers keep it updated, the tests nominally check the same API — but there is no actual link between them. Nothing breaks when they diverge. You find out later, usually in production.

We decided to make Swagger the source of truth for API tests rather than documentation nobody reads.

Using our Cohorts service as the running example, the pipeline looks like this:

cohorts.swagger.yaml

        ↓

OpenAPI Generator + Mustache templates

        ↓

cohorts.api.CohortsApi

cohorts.model.*

        ↓

JUnit + RestAssured tests

        ↓

Swagger Coverage

        ↓

HTML / JSON coverage report

        ↓

LLM skills that generate the missing tests

Tests don't call given().get("/api/..."). They call a generated client with generated models, both produced from the OpenAPI spec.
How Swagger becomes a Java client
Take the endpoint that creates a cohort: POST /api/v1/cohort.

The OpenAPI file describes it at two levels. The first is paths — which endpoint, which HTTP method, the operationId, the request schema to send and the response schema to expect:

/api/v1/cohort:

  post:

    tags:

      - cohorts

    summary: Create an empty cohort

    operationId: createCohort

    requestBody:

      content:

        application/json:

          schema:

            $ref: '#/components/schemas/CreateCohortsRq'

    responses:

      201:

        description: Success

        content:

          application/json:

            schema:

              $ref: '#/components/schemas/CreateCohortsRs'

The second is components.schemas — the fields that make up the DTOs referenced above via $ref:

CreateCohortsRq:

  type: object

  required:

    - name

    - description

  properties:

    name:

      type: string

      description: Cohort name

    description:

      type: string

      description: Cohort description

    case_type:

      type: integer

      description: Cohort use case type

CreateCohortsRs:

  type: object

  required:

    - id

  properties:

    id:

      type: integer

      description: Cohort ID

After generation, that becomes Java:

operationId: createCohort → cohortsDefaultApi().createCohort()

CreateCohortsRq           → cohorts.model.CreateCohortsRq

CreateCohortsRs           → cohorts.model.CreateCohortsRs

If Swagger has a name field, the Java model has a builder method .name(...). If Swagger has id, the response model has getId().

This is the part that matters: QA doesn't invent the DTO. The test can only work with what the contract describes.
Why we use Mustache templates
OpenAPI Generator works fine out of the box — the standard templates will produce API classes, models, methods for path/query/body parameters, and a basic client wrapper.

But when the generated code doesn't quite match your project's style, you can customise it with Mustache templates. This is a standard OpenAPI Generator mechanism: take the default template and change how the generated API classes or models look.

We don't use Mustache because generation is impossible without it. We use it because we want the generated code to sit comfortably inside our existing test framework.

src/test/resources/openapi-templates/api.mustache

src/test/resources/openapi-templates/pojo.mustache

They're wired into the Gradle generation task:

templateDir.set("$rootDir/src/test/resources/openapi-templates")

api.mustache controls the generation of API classes — for example cohorts.api.CohortsApi. It's what defines the shape of the generated methods:

public CreateCohortOper createCohort() {

    return new CreateCohortOper(createReqSpec());

}

The request execution methods:

public <T> T execute(Function<Response, T> handler) {

    return handler.apply(

            RestAssured.given()

                    .spec(reqSpec.build())

                    .expect()

                    .spec(respSpec.build())

                    .when()

                    .request(REQ_METHOD, REQ_URI)

    );

}

And typed response deserialization:

public CreateCohortsRs executeAs(Function<Response, Response> handler) {

    TypeRef<CreateCohortsRs> type = new TypeRef<CreateCohortsRs>() {};

    return execute(handler).as(type);

}

So execute(...), executeAs(...), .idPath(...), .pageQuery(...), .body(...) aren't RestAssured magic on their own. They're the shape of the generated client, which we get from OpenAPI Generator plus our templates.

pojo.mustache handles the models — CreateCohortsRq, CreateCohortsRs, ListUsersWithPagination. It describes what the generated model classes look like: fields from the Swagger schema, Jackson annotations, getters and setters, builder-style methods, equals/hashCode/toString.

In practice this lets us keep the convenience of generation while controlling the shape of what comes out: Allure steps, a fluent API, execute/executeAs, how RequestSpecification is handled, and the style of the models.
Wiring it together with Gradle
OpenAPI Generator is added in build.gradle:

plugins {

    id "org.openapi.generator" version "7.18.0"

}

The generated code is registered as a test source set:

sourceSets {

    test {

        java {

            srcDir('build/generate-resources/src/main/java')

        }

    }

}

Which means the Java classes OpenAPI Generator drops into build/generate-resources/src/main/java are available in tests like any other test code.

Each service gets two Gradle tasks: one to download and prepare the Swagger file, one to generate the Java RestAssured client. For Cohorts:

tasks.register('downloadCohortsSwaggerFile', Exec) {

    group = "openapi tools"

    def url = "https://example.com/cohort/openapi_cohort.yaml"

    def outputFile = file("${projectDir}/src/test/resources/cohorts.swagger.yaml")

    if (outputFile.exists()) {

        commandLine("curl", "-z", outputFile, "-o", outputFile, url)

    } else {

        commandLine("curl", "-o", outputFile, url)

    }

}

A note from experience: sometimes the spec needs a little preparation before generation, to get it into a shape that both OpenAPI Generator and the coverage tool handle predictably. This happens when a service's Swagger has grown organically — inconsistent in places, overly generic descriptions in others, not specific enough for the generator somewhere else. I won't go into our internal cleanup details here. The point is that what you feed the generator should reliably produce a client and models.

Then generation runs:

tasks.register("openApiCohorts", GenerateTask) {

    dependsOn 'downloadCohortsSwaggerFile'

    apiPackage.set("cohorts.api")

    modelPackage.set("cohorts.model")

    generatorName = "java"

    library = "rest-assured"

    inputSpec.set("$rootDir/src/test/resources/cohorts.swagger.yaml")

    outputDir.set("$buildDir/generate-resources")

    templateDir.set("$rootDir/src/test/resources/openapi-templates")

}

Producing:

cohorts.api.CohortsApi

cohorts.ApiClient

cohorts.model.CreateCohortsRq

cohorts.model.CreateCohortsRs

cohorts.model.ListUsersWithPagination

...

And to make all of it happen automatically before the tests compile:

compileTestJava.dependsOn tasks.openApiBanners, tasks.openApiGifts, tasks.openApiCohorts

So when CI runs the Gradle test task, this chain fires on its own:

downloadCohortsSwaggerFile

        ↓

openApiCohorts

        ↓

compileTestJava

        ↓

test

Using the generated client in tests
Building cohortsDefaultApi
We don't construct a RestAssured request by hand every time. There's a base method that returns the generated API class:

public class BaseCohortTests {

   protected CohortsApi cohortsDefaultApi() {

       return ApiClient.api(ApiClient.Config.apiConfig().reqSpecSupplier(

               () -> new RequestSpecBuilder()

                       .addRequestSpecification(CohortsSpec.defaultReqSpec())

       )).cohorts();

   }

}

A RequestSpecification is passed into the generated client, built in CohortsSpec.defaultReqSpec():

public static RequestSpecification defaultReqSpec() {

   return new RequestSpecBuilder()

           .setBaseUri(config.COHORTS_URL())

           .setConfig(Specifications.hiddenTokensConfig())

           .addFilter(new AllureRestAssured())

           .addFilter(new SwaggerCoverageV3RestAssured(

                   new FileSystemOutputWriter(Paths.get("build/swagger-coverage-output/cohorts"))))

           .addHeader("x-token", valuesConfig.cohortsToken())

           .addHeader("x-device", "ekb")

           .log(LogDetail.ALL)

           .build();

}

There are two distinct layers here, and keeping them separate is what makes the whole thing maintainable.

The generated client owns everything that came from Swagger: endpoint, HTTP method, path parameters, query parameters, request body, response model.

The RequestSpecification owns the infrastructure of a particular test run: base URL, tokens, technical headers, logging, the Allure filter, the Swagger Coverage filter.

The Allure filter adds HTTP requests and responses to the Allure report — handy for triaging failures, unrelated to client generation. The Swagger Coverage filter does something different: it records the actual HTTP calls into build/swagger-coverage-output so a coverage report can be built after the run.
The test itself

@Test

@DisplayName("Create an empty cohort")

void createCohort() {

    CreateCohortsRq request = CreateCohortsRq.builder()

            .name("Autotest cohort " + UUID.randomUUID())

            .description("Cohort created by an autotest")

            .build();

    CreateCohortsRs result = cohortsDefaultApi()

            .createCohort()

            .body(request)

            .executeAs(response -> response

                    .then()

                    .statusCode(201)

                    .extract().response());

    cohortId = result.getId();

    step("And I verify the cohort was created", () -> {

        assertNotNull(result, "Response must not be null");

        assertNotNull(cohortId, "Created cohort ID must not be null");

        assertNotEquals(0, cohortId, "Created cohort ID must not be 0");

    });

}

Almost every line traces back to Swagger: CreateCohortsRq is the request model, createCohort() is the operationId, body(request) is the request body, CreateCohortsRs is the response schema.
Path and query parameters are generated too
Another example — fetching the list of cards in a cohort. In Swagger:

/api/v1/cohort/{id}/users:

  get:

    operationId: getUsers

    parameters:

      - in: path

        name: id

      - in: query

        name: page

      - in: query

        name: size

      - in: query

        name: search_pattern

    responses:

      200:

        content:

          application/json:

            schema:

              $ref: '#/components/schemas/ListUsersWithPagination'

Which generates a fluent API:

operationId: getUsers          → .getUsers()

path parameter id              → .idPath(...)

query parameter page           → .pageQuery(...)

query parameter size           → .sizeQuery(...)

query parameter search_pattern → .searchPatternQuery(...)

response schema                → ListUsersWithPagination result

In the test:

@Test

@DisplayName("Get the list of cards in a cohort")

void getUsers() {

    step("Given cohort ID: %s".formatted(cohortId));

    ListUsersWithPagination result = cohortsDefaultApi()

            .getUsers()

            .idPath(cohortId)

            .pageQuery(PAGE)

            .sizeQuery(PAGE_SIZE)

            .searchPatternQuery(number)

            .executeAs(response -> response

                    .then()

                    .statusCode(200)

                    .extract().response());

    step("And I verify the response body", () -> {

        assertNotNull(result, "Response must not be null");

        assertNotNull(result.getNumbers(), "Card list must not be null");

        assertTrue(result.getNumbers().contains(number), "Card list must contain the test card");

        assertNotNull(result.getPageNumber(), "pageNumber must not be null");

        assertNotNull(result.getPageSize(), "pageSize must not be null");

        assertNotNull(result.getTotalPages(), "totalPages must not be null");

        assertNotNull(result.getTotalCount(), "totalCount must not be null");

    });

}

QA doesn't need to remember what a query parameter is called in the URL. If it's described in Swagger, the generator makes a method for it.
execute vs executeAs
execute(...) makes a plain RestAssured call and hands control back out. Useful when the response has no body:

cohortsDefaultApi()

        .addUsers()

        .idPath(cohortId)

        .body(request)

        .execute(Validatable::then)

        .statusCode(204);

executeAs(...) is for when Swagger describes a response schema. Under the hood it executes the request and deserializes the response into the generated model.

For us this isn't just a convenience wrapper — it's an extra contract check. If Swagger says the response should be CreateCohortsRs, the test tries to read the actual JSON as exactly that. When the backend returns something that doesn't match the contract, the test fails on deserialization or on the assertions. That catches:

Swagger calls the field id, the API returns cohort_id
Swagger types id as integer, the API returns a string
Swagger describes an object, the API returns an array
with strict deserialization, the backend added a field and nobody updated Swagger

Which is how you gradually train a team to keep the contract current — the contract breaking becomes your problem, immediately, instead of a documentation debt nobody feels.
The next problem: we couldn't see what was actually covered
At this point we had a large pile of API tests using generated clients and models, much closer to the contract than the old hand-rolled ones.

But a different problem appeared: we had no idea how much of the API those tests really covered. Lots of tests, all green — but which Swagger operations do they actually call? Which endpoints are untouched entirely? Which are only partially covered? Checking that by reading code got old fast.

So we took a different approach: record the actual HTTP calls during the test run, then compare that log against the OpenAPI spec afterwards. Tests run as usual; after the run you get a report showing what part of the contract was exercised.
Swagger Coverage in GitLab CI
For this we use a modified version of swagger-coverage, originally created by Victor Orlovsky. The core idea fit us well: split the process in two stages. During the test run, a RestAssured filter collects the actual HTTP calls. After the run, a separate CI job compares those calls against the OpenAPI spec and builds a coverage report.

The original project hasn't been maintained for a while, and we hit bugs and limitations while integrating it. So Andrey Poletaev (@fenixnow) forked it, fixed the problems, and adapted the tool to our needs. The fork lives at github.com/fenixnow/swagger-coverage, and its README covers configuration and running it via Docker.
During tests: collecting the calls
The RestAssured request specification includes the coverage filter:

.addFilter(new SwaggerCoverageV3RestAssured(

        new FileSystemOutputWriter(Paths.get("build/swagger-coverage-output/cohorts"))))

It hooks into the HTTP execution chain and sees every request that goes through our RequestSpecification:

test calls a generated API method

        ↓

generated client executes a RestAssured request

        ↓

RestAssured passes request/response through filters

        ↓

SwaggerCoverageV3RestAssured records the call

        ↓

data is written to build/swagger-coverage-output/cohorts

Worth being clear about: the filter does not build an HTML report during the test. It only collects raw data — which endpoint was called, with which HTTP method, and what status came back.
The CI jobs
After the tests, separate coverage jobs run in a Docker image containing the modified swagger-coverage:

swagger-coverage-report--banners

swagger-coverage-report--gifts

swagger-coverage-report--cohorts

Because the image ships swagger-coverage-commandline, each job stays simple — it only needs the spec, the input directory, and a config:

swagger-coverage-commandline \

  -s src/test/resources/cohorts.swagger.yaml \

  -i build/swagger-coverage-output/cohorts \

  -c swagger-coverage-cohorts-config.json

Where -s is the OpenAPI spec, -i is the data collected during the tests, and -c is the report rules config.

The job then copies out two artifacts:

cohorts-coverage-report.html → a readable report for humans

cohorts-coverage-results.json → structured JSON for an LLM

The HTML report looks like this:

Getting the links into the Merge Request
The coverage job stores the HTML and JSON URLs in a dotenv artifact, and a separate publish-coverage-report job posts a comment into the Merge Request with links to both report types for every service.

Why two formats: the HTML is for a person — open it in a browser, look at the summary, methods, groups, coverage variants, and quickly see where the gaps are. The LLM JSON exists because HTML is genuinely bad input for a model: awkward to feed in, hard to parse reliably. So the modified swagger-coverage gained a separate JSON export aimed specifically at LLM consumption.
The whole chain

GitLab CI

        ↓

test job runs the Gradle test task

        ↓

Gradle downloads Swagger

        ↓

OpenAPI Generator generates Java clients/models

        ↓

JUnit/RestAssured tests run through the generated client

        ↓

SwaggerCoverageV3RestAssured writes build/swagger-coverage-output

        ↓

GitLab stores swagger files and coverage output as artifacts

        ↓

coverage job runs swagger-coverage-commandline in Docker

        ↓

HTML and JSON coverage reports are generated

        ↓

publish-coverage-report posts the links into the Merge Request

        ↓

an LLM reads the JSON report and knows which tests are still missing

That connection is what makes it automatic. Nobody downloads Swagger by hand, runs the generator, hunts for coverage output, or assembles a report. The pipeline does it, and the team gets a readable HTML report plus a JSON file suitable for model analysis.
Connecting an LLM to API coverage
Once we had coverage reports, the obvious next step was to use them as structured input for an LLM rather than just a web page for humans.

The JSON report contains a coverage summary, a list of paths, the state of each operation, and the requirements not yet closed by tests. Simplified:

{

  "summary": {

    "total_operations": 89,

    "fully_covered": 5,

    "partially_covered": 20,

    "not_covered": 128,

    "coverage_percent": 13.0

  },

  "paths": {

    "/api/v1/cohort/{id}/users": {

      "GET": {

        "state": "PARTY",

        "coverage": "3/6",

        "deprecated": false,

        "requirements": {

          "status_codes": [],

          "parameters": [],

          "body": [],

          "properties": []

        }

      }

    }

  }

}

Some people on the team work in Cursor, others in Claude Code, so we described the test generation rules as skills for both tools.

The important design decision was splitting the skills by responsibility. The first analyses the coverage report and acts as an orchestrator. The second writes the actual tests.
First skill: analysis and orchestration
It takes two files as input — cohorts.swagger.yaml and cohorts-coverage-results.json. The spec tells it the contract: endpoints, operationIds, parameters, request bodies, response models, statuses. The coverage JSON tells it what's covered and what isn't.

It reads the operation states:

EMPTY → endpoint not covered by tests at all

PARTY → endpoint partially covered

FULL → endpoint fully covered

Then looks at the requirements to work out what specifically is missing:

status_codes → a required HTTP status was never asserted

parameters → a path/query/header parameter was never used

body → the request body isn't covered

properties → response fields were never checked

And decides what to do:

EMPTY endpoint → propose a basic happy-path test
PARTY endpoint → add the missing assertions or scenarios
FULL endpoint → skip
deprecated endpoint → skip

Then it calls the second skill and passes not the whole report, but a specific task: which endpoint to cover, which operationId to use, which request/response schemas to look at, which requirements to close.
Second skill: generating tests to the project's rules
The second skill doesn't analyse anything globally. It receives a concrete task, reads the OpenAPI spec, finds the right generated API method and models, checks the existing tests, and only then writes the new test.

It follows our constraints:

use the generated Swagger client
use the generated models
don't hand-write raw JSON
don't invent your own DTOs
don't assemble URLs manually
put tests in the correct package
pick executeAs or execute according to the contract
assert through the generated models' getters
wrap steps in Allure

So for an uncovered Cohorts endpoint, it goes to Swagger, finds the operation, reads the operationId and schemas, and works out which generated code to use:

API method → cohortsDefaultApi().getUsers()

path parameter id → .idPath(...)

query parameter page → .pageQuery(...)

query parameter size → .sizeQuery(...)

response model → ListUsersWithPagination

execution method → executeAs(...)

This is the crux of it: the LLM must not invent its own way of talking to the API. It has to use the same contract-bound client that every other test uses. Otherwise you get tests that pass while quietly bypassing the very mechanism that makes the suite trustworthy.
When Swagger alone isn't enough
Being honest about the limits: not every test can be generated from Swagger and a coverage report.

For simple CRUD it's often plenty — there's an endpoint, a clear request body, a clear response body, and you can build a happy path with basic assertions.

But complex microservices exist where calling a single method requires a lot of preconditions:

create an entity in another system
prepare a user in a particular state
obtain a valid token
enable a feature flag beforehand
pass parameters you cannot guess from Swagger
respect a business-level sequence of calls

Swagger describes the contract of one HTTP method well. It doesn't explain business context: where the data comes from, what state the system must be in, which steps precede the call.

For those cases we need test cases written by QA, with preconditions, steps, test data, and expected results. So we have separate skills that work from test cases in Allure TestOps instead of from a coverage report. We built our own MCP server for TestOps so an LLM can pull a test case, parse it, and turn it into an autotest:

Allure TestOps test case

        ↓

our MCP server for Allure TestOps

        ↓

an LLM skill reads preconditions, steps, and expected result

        ↓

the skill maps the steps onto the OpenAPI spec

        ↓

finds the generated API methods and models

        ↓

writes the test to the project's rules

So we're not trying to replace QA and test design with Swagger alone. Swagger owns the contract, coverage owns visibility into the gaps, and test cases own the complex business scenarios where a human description of preconditions and expected behaviour is unavoidable.
What about gRPC
Every example here is deliberately REST, where OpenAPI is the contract.

For gRPC the idea is the same — tests should lean on the contract — but instead of Swagger you have .proto files generating service stubs, RPC methods, and request/response messages. The technical implementation differs enough (different transport, different contract format, different generated classes, a different approach to coverage) that mixing both into one article would help nobody. REST here; gRPC deserves its own write-up.
Keeping generated code and test code apart
There are two kinds of code in the project.

Generated from Swagger, never written by hand:

cohorts.api.CohortsApi

cohorts.ApiClient

cohorts.model.CreateCohortsRq

cohorts.model.CreateCohortsRs

cohorts.model.ListUsers

cohorts.model.ListUsersWithPagination

And our test code — BaseCohortTests, CohortsSpec, CohortV1Tests — responsible for scenarios, steps, assertions, tokens, base URLs, and wiring up filters.

Simplified, the division of responsibility runs like this:

Swagger describes the contract

        ↓

OpenAPI Generator creates API classes and models

        ↓

Mustache templates define the shape of the generated code

        ↓

BaseCohortTests connects the generated client to the test infrastructure

        ↓

CohortV1Tests describes a specific test scenario

Generated code owns the technical API contract; our test code owns scenarios, data, steps, and assertions.
What this buys us
The main benefit is that the tests sit close to the contract.

Change an operationId in Swagger and the generated method name changes. Change the request body and the generated model changes. Remove a field from a response and the getter disappears. A lot of divergence becomes visible at compile time or during deserialization — not a week after release.

And the test stays readable:

CreateCohortsRs result = cohortsDefaultApi()

        .createCohort()

        .body(request)

        .executeAs(response -> response

                .then()

                .statusCode(201)

                .extract().response());

From that chain you can immediately see which API is called, what body is sent, what status is expected, and which model the response is deserialized into.

Swagger Coverage closes the second part of the problem: which operations from the spec were actually exercised, and which are still untouched. The LLM skills close the third: turning that report into new tests — not randomly, but to the project's rules. And test cases from TestOps cover the complex scenarios where Swagger isn't enough on its own.

The end result is that Swagger stops being a documentation page. It becomes a working contract between analysts, developers, and QA — and the coverage report stops being a pretty HTML artifact and becomes a backlog for the test suite.
What's next
This covered one slice of a bigger process: contract-driven API tests and a coverage report an LLM can read.

Next in the series: how we triage failed autotests through defects in Allure TestOps, tie that to an issue tracker, and use AI not only to generate tests but to analyse results and file tasks. And after that, a system where AI connects to our services, prepares test plans itself, verifies a task through the database, REST, and gRPC, and posts the final report straight into the tracker issue.

If you've built something similar on top of OpenAPI, I'd be curious what your coverage numbers looked like the first time you measured them. Ours were humbling.

Let’s Work Together

If this sounds relevant to your team, feel free to connect with me on LinkedIn.

How We Migrated Test Cases from a Task Tracker to Allure TestOps with a Single Command Using Cursor + MCP

Oleg Malyshev — Thu, 06 Aug 2026 20:24:34 +0000

Hi! I’m Oleg, a QA Engineering Manager focused on test strategy, automation, and AI-assisted QA workflows.

I lead one of the testing stacks at a large grocery retail company. We've been working through a series of experiments on where AI actually saves time in QA — not where it looks impressive in a demo, but where it removes hours of mechanical work that nobody wants to do.

An earlier piece covered AI review of manual test cases. This one is about a less typical use of Cursor: migrating test cases from a task tracker into Allure TestOps with essentially one command.

Two bits of context for readers outside our stack:

Yandex Tracker is an issue tracker, roughly analogous to Jira. Its issue descriptions support wiki-style tables, which matters here.
Allure TestOps is a test management system (TMS) that connects manual test cases with automated tests, CI/CD runs, and reporting.

If you use Jira and TestRail, or Linear and Xray, the specific tools change but the pattern in this article doesn't.

The problem: test cases living in the wrong place

Historically, one large team kept all of its test cases and checklists inside the issue tracker. Each issue had a "Testing" section containing a table, and that table was the team's actual test documentation.

Then the inevitable happened — we needed all of it in Allure TestOps, because:

it's the correct place for it (a single TMS instead of test cases scattered across issues);
it gives us analytics, traceability, and artifacts;
it links properly to automated tests and CI/CD;
and the rest of the company already lives in TestOps.

The catch was volume. Years of accumulated cases and checklists. Migrating them by hand meant a large amount of pure copy-paste work for QA engineers, which is exactly the kind of task you don't want to hand to people if you can avoid it.

So the plan was to migrate everything quickly, with no manual copy-paste, using an AI agent.

The approach: Cursor + two MCP servers + two prompts

The plan looked suspiciously simple:

Take Cursor.
Connect the Yandex Tracker MCP server.
Connect the Allure TestOps MCP server (ours, built in-house).
Write a project rule that:

takes the "Testing" table from the issue description,
converts it into a "title + steps" structure,
collects links to the source issue and its related issues.

Run it as a single command.

For anyone new to it: MCP (Model Context Protocol) is a standard way to expose external systems as tools an AI agent can call. Instead of the agent guessing, it gets real API access to the tracker and to TestOps.

What the source data looked like

Inside each tracker issue, the test cases lived in a table under a "Testing" section, structured roughly like this:

an "Acceptance criterion" column,
a "Test" column,
other columns (outcome, result, comments, and so on) that exist but are irrelevant to the migration.

We agreed on a simple mapping:

Acceptance criterion → test case title in TestOps
Test → scenario steps in TestOps

Everything else gets dropped.

One wrinkle worth flagging early, because it's the kind of thing that quietly corrupts a migration: the tables contain visually merged cells. When several test rows belong to one acceptance criterion, only the first row has the criterion filled in and the rest are blank. A naive parser produces a pile of test cases with empty titles.

The project rule: parsing the tracker and producing JSON

The first artifact is a project rule describing exactly how to parse the table and what JSON to return. I called the file get-check-list-rule. Here it is, translated:

You are a QA Automation / test analyst assistant. Your task: given a tracker issue key {ISSUE_KEY}, extract the "Testing" section (a table) from its description and prepare structured data for migration into Allure TestOps.

Input {ISSUE_KEY} — the issue key (for example, LKP-3260).

What to do

Fetch the issue via the issue_get tool with include_description=true. Fetch related issues via issue_get_links.

In the description field, find the "Testing" section block. It starts with a line reading "Testing" and contains a table where the table opens with #|, followed by rows starting with || and the column headers, and closes with |#.

Parse the table and extract all test rows:

Relevant columns: "Acceptance criterion", "Test".
All other columns and their values (for example "Acceptance outcome", "Result", "Comment") must be completely ignored and carried nowhere — not into titles, not into steps, not into links.

Account for merged cells:

If the "Acceptance criterion" column is empty in a row, inherit the last non-empty value above it.
The same technique applies to other columns if needed, though in practice only the criterion requires it.

Group results by acceptance criterion. For each criterion, build:

title: the criterion text, which becomes the test case title in TestOps, without leading numbering. If the criterion starts with 1., 1), 1\. or similar, strip it. Normalize the same way as steps: plain text, no markdown or HTML, no redundant whitespace or blank lines. If the criterion spans several lines, join them into one with spaces.
steps: the list of steps from the "Test" column for every row belonging to that criterion.

Normalize steps:

Remove redundant whitespace and repeated blank lines.
Steps must arrive in Allure TestOps as plain text — no HTML tags, no markdown.

Converting "Test" into plain text:

Strip or unwrap markdown formatting: bold, italics, # headings, > quotes, inline code and backticks, fenced code blocks — keep the text only.
Convert markdown links [text](url) into text (url), or just url when there is no text.
Convert lists (-, *, numbered) into multi-line plain text, preserving order but dropping the markers and numbers.

Important: if the "Test" column contains fragments that actually belong to other columns (PASS/FAIL/BLOCK markers, checkboxes, comments, references like screen/log from the "Comment" column) — do not migrate them. Migrate only what genuinely describes scenario steps.

Strip explicit numbering from the start of step lines where present (1. ..., 2) ..., a) ...) — steps[] must contain unnumbered steps.

Important: do not add a line like "Source: {ISSUE_KEY}" into the steps or description. Issue links belong in the Links (external links) section in Allure TestOps, not inline in the description.

Important, on related issues:

In the TestOps Links section, add a link to the source issue {ISSUE_KEY}, plus links to all issues related to it from issue_get_links.
Order matters: {ISSUE_KEY} always comes first (the source issue the tests came from), followed by the related issues (their object.key), with no duplicates and without repeating {ISSUE_KEY}.

Link naming rule, so the source is always first in the Links list:

For the source issue {ISSUE_KEY}, use the link name Source: {ISSUE_KEY}.
For related issues, use a link name equal to their key (for example LKP-4106) with no prefix.

Return the result strictly as JSON in this shape, with no surrounding text:

json
{
  "issueKey": "{ISSUE_KEY}",
  "linkIssueToTestOps": {
    "field": "Links",
    "keys": ["{ISSUE_KEY}", "<LINKED_ISSUE_KEY_1>", "<LINKED_ISSUE_KEY_2>"]
  },
  "testops": [
    {
      "name": "<Acceptance criterion>",
      "steps": [
        "<Step 1 from the Test column, unnumbered, plain text>",
        "<Step 2 from the Test column, unnumbered, plain text>"
      ]
    }
  ]
}

Quality checks

testops[] must contain every criterion from the "Testing" table, not just the first one.
If there is no "Testing" table, return JSON with an empty testops array and an error field explaining why.
If there are several "Testing" tables (for example, one per environment), extract each and add an environment field (stage, test, and so on) when it can be determined from the surrounding text.

A note on the literal strings: in our tracker the section and column headings are in Russian, so the rule matches on those exact names. If you adapt this, swap in whatever your own tables actually say.

Two details in that rule carry most of the weight. The first is the merged-cell inheritance, which is the difference between a clean migration and a pile of untitled cases. The second is the instruction to return an error field rather than improvise — without it, an agent that can't find a table will happily invent a plausible one, and you won't notice until a QA engineer opens TestOps.

The command: /move, one issue or a batch

The second artifact is the command, which I called move. It:

accepts one or more ISSUE_KEY values (links to the issues to migrate from),
pulls each issue from the tracker,
generates JSON according to get-check-list-rule,
creates the test cases in TestOps and populates the Links section.

The command prompt, translated:

Migrate test cases from the tracker into Allure TestOps, strictly following the project rule @cursor/rules/get-check-list-rule.mdc.

Input

I will give you one of:

a link to a tracker issue (or several links)
issue key(s) (for example LKP-3260, ABC-123)
plus attached screenshots, if wording or context needs clarifying

What to do

Support migrating several issues at once. Example invocation: /move LKP-3260 LKP-3261 LKP-3262
Extract the list of ISSUE_KEY values from the text (support both bare keys and URLs), remove duplicates, preserve order of appearance.
For each ISSUE_KEY:
fetch the issue through the Yandex Tracker MCP (issue_get, include_description=true)
generate JSON strictly according to @.cursor/rules/get-check-list-rule.mdc (including issue_get_links, title/step normalization, and building linkIssueToTestOps.keys)
push the result into Allure TestOps through the user-allure MCP: create test cases from testops[], then add the issue links from linkIssueToTestOps into the Links section, using a link_name of the form Source: <ISSUE_KEY> for the source issue
If the Allure MCP is unavailable or errors out for some issue — do not invent anything. Add an error field to that issue's JSON and carry on processing the rest.

Output (strict)

If one issue was passed — a single JSON object per get-check-list-rule.mdc, with no surrounding text.
If several issues were passed — a JSON array where each element follows the same format.
If there were problems with the Allure MCP for an element, add an error field to that element with a short description. Otherwise, omit it.

A note on the literal strings: in our tracker the section and column headings are in Russian, so the rule matches on those exact names. If you adapt this, swap in whatever your own tables actually say.

Two details in that rule carry most of the weight. The first is the merged-cell inheritance, which is the difference between a clean migration and a pile of untitled cases. The second is the instruction to return an error field rather than improvise — without it, an agent that can't find a table will happily invent a plausible one, and you won't notice until a QA engineer opens TestOps.

The command: /move, one issue or a batch

The second artifact is the command, which I called move. It:

accepts one or more ISSUE_KEY values (links to the issues to migrate from),
pulls each issue from the tracker,
generates JSON according to get-check-list-rule,
creates the test cases in TestOps and populates the Links section.

The command prompt, translated:

Migrate test cases from the tracker into Allure TestOps, strictly following the project rule @cursor/rules/get-check-list-rule.mdc.

Input

I will give you one of:

a link to a tracker issue (or several links)
issue key(s) (for example LKP-3260, ABC-123)
plus attached screenshots, if wording or context needs clarifying

What to do

Support migrating several issues at once. Example invocation: /move LKP-3260 LKP-3261 LKP-3262
Extract the list of ISSUE_KEY values from the text (support both bare keys and URLs), remove duplicates, preserve order of appearance.
For each ISSUE_KEY:
fetch the issue through the Yandex Tracker MCP (issue_get, include_description=true)
generate JSON strictly according to @.cursor/rules/get-check-list-rule.mdc (including issue_get_links, title/step normalization, and building linkIssueToTestOps.keys)
push the result into Allure TestOps through the user-allure MCP: create test cases from testops[], then add the issue links from linkIssueToTestOps into the Links section, using a link_name of the form Source: <ISSUE_KEY> for the source issue
If the Allure MCP is unavailable or errors out for some issue — do not invent anything. Add an error field to that issue's JSON and carry on processing the rest.

Output (strict)

If one issue was passed — a single JSON object per get-check-list-rule.mdc, with no surrounding text.
If several issues were passed — a JSON array where each element follows the same format.
If there were problems with the Allure MCP for an element, add an error field to that element with a short description. Otherwise, omit it.

Point 4 is the one I'd copy into any similar workflow. Partial failure is the normal case in batch migrations, and an agent that stops on the first error is nearly as unhelpful as one that hallucinates through it.

MCP configuration

For any of this to run, both MCP servers need to be connected: Allure TestOps (ours) and Yandex Tracker.

Example config for the tracker MCP:

"yandex-tracker": { "command": "uvx", "args": [ "yandex-tracker-mcp@latest" ], "env": { "TRACKER_TOKEN": "your token", "TRACKER_ORG_ID": "your org id" } }

What this looks like for a QA engineer

Exactly one thing: run the command with the issue keys.

/move LKP-3260

or, as a batch:

/move LKP-3260 LKP-3261 LKP-3262

LKP-* are the issues to migrate test cases from, and a project number identifies the TestOps project to migrate them into.

That's the whole interface. No spreadsheet, no intermediate export, no clipboard.

The result
After a run, the migration lands as expected: test cases appear in TestOps, with titles and steps; in the Links section, the source issue comes first, labeled Source: , followed by all related issues.

That last part turned out to matter more than we expected. Test cases that carry a link back to the issue they came from stay traceable — six months later you can still see why a case exists, which is usually the first thing lost in a bulk migration.

Where else this pattern fits

The specific tools here are incidental. The shape of the problem — structured content trapped in the wrong system, in a format that's regular enough to parse but irregular enough to defeat a simple script — is extremely common.

The same setup would move requirements documentation out of Jira issues into Confluence with a Jira MCP and a Confluence MCP, or the reverse. Anywhere you have two systems with API access and a mechanical transformation between them, two prompts and an agent will beat a weekend of copy-paste.

We also could have written a prompt that iterated across every issue in the tracker and migrated everything automatically, saving QA even more time. We decided we didn't need all of it migrated yet — which is its own small lesson. The value came from removing a specific painful backlog, not from automating the maximum possible surface area.

Let’s Work Together

If this sounds relevant to your team, feel free to connect with me on LinkedIn.

How We Built an AI Test Case Reviewer Using Cursor, MCP, and Allure TestOps

Oleg Malyshev — Mon, 27 Jul 2026 20:56:57 +0000

Hi! I’m Oleg, a QA Engineering Manager focused on test strategy, automation, and AI-assisted QA workflows.

My goal is simple: eliminate repetitive work for QA engineers using AI tools.

In an ideal world, we provide an AI model with requirements, Figma designs, Git branches, and other project artifacts through MCP. The model then helps us:

Create test cases using the available project context.
Generate automated tests based on those test cases.

We Decided to Start With AI Test Case Reviews

Writing high-quality test cases from requirements and supporting documentation is not an easy task.

Instead of trying to automate the entire process immediately, we decided to start with something smaller: an AI-powered test case reviewer.

The basic idea was:

A QA engineer moves a test case to the Review status in Allure TestOps.
An AI agent retrieves and reviews the test case according to our internal guidelines.
The agent publishes the review as a new comment on the test case.
The test case is moved to an appropriate status, such as Needs Fixing, so the QA engineer immediately knows what to do next.

To implement this workflow, we needed the AI agent to be able to:

Connect to Allure TestOps.
Retrieve a test case and its scenario.
Review the content.
Publish the review back to TestOps.

For this purpose, we created our own MCP server for Allure TestOps, along with an interface for configuring review prompts for individual TestOps projects.

Why We Did Not Use the Built-in Allure TestOps MCP

Allure TestOps recently introduced its own MCP integration and an AI chat interface directly inside the product.

However, the AI chat requires a separate license. We also discovered the official MCP integration only after we had already developed our own solution.

After examining the built-in TestOps AI functionality, we concluded that our implementation covered our current needs at least as well.

Our solution also remains free for us within our existing infrastructure and software licenses.

How Our MCP Server Works

Our MCP server is written in Python using the FastMCP framework.

We also converted the OpenAPI specification provided by Allure TestOps into a generated API client.

The resulting workflow looks like this:

The MCP server calls the TestOps API.
It retrieves the required test case data, including its scenario, steps, attachments, and other fields.
The data is passed to an LLM for analysis.
After processing, the result is saved back to TestOps through MCP—for example, as a new review comment.

Allure TestOps
      ↓
MCP server
      ↓
Language model
      ↓
MCP server
      ↓
TestOps comment

Why We Used Cursor

When we started working on the prototype, we had several practical limitations:

The web interface for reviewing test cases was not completely ready.
Our MCP server could occasionally fail to return some parts of a TestOps scenario.
The language model used by the main solution required separate payment.
Our team already had access to corporate Cursor accounts and experience using Cursor.

I therefore created a minimal project that could be opened in Cursor.

The goal was to let any QA engineer:

Clone the repository.
Open it in Cursor.
Run a single command to review a test case.
Quickly evaluate the quality of the AI-generated review.

How the Cursor Project Is Structured

The project contains only a few components:

One project rule containing the main test case review prompt.
One command: /review-test-case.
One user rule that ensures Cursor loads the project rules.
An MCP server for interacting with Allure TestOps.

User Rule

We added a user rule because Cursor did not always load the project rules consistently.

The rule explicitly instructs Cursor to read the project rules before every response. A fixed phrase at the beginning of the response serves as a visible indication that the rules were actually applied.

Respond in Russian.

Before every response, read the project rules from .cursor/rules.
Do not begin answering until you have read the rules.

Start every response, without exception, with the phrase:
"Rules added to context: X, Y, Z, etc."

List the names of the rule files you have read.

This is a workaround rather than an ideal architecture, but it made the behavior easier to verify during our initial experiments.

The Test Case Review Prompt

The following prompt is still experimental. We created it to obtain initial results quickly and determine which areas required further improvement.

You are a Senior QA Engineer with more than 30 years of
experience testing mobile, web, and API applications.

You review test cases in Allure TestOps and help QA engineers
write cases that are:

- unambiguous;
- verifiable;
- atomic;
- reproducible;
- deterministic;
- suitable for automation.

I will provide screenshots or text from an Allure TestOps test
case. Sometimes only some of its fields will be available.

Your task is to review the test case quickly and strictly
according to the rules below and propose specific corrections.

=== REVIEW GUIDELINES ===

1. General principles

Check the test case for:

- unambiguity;
- verifiability, including measurable expected results;
- atomicity;
- independence and repeatability;
- determinism, including fixed feature flags and test data;
- suitability for automation.

Remove vague phrases such as "verify that it works correctly"
or "pay attention to."

2. Terminology

Use the same terms that appear in the product UI.

If UI terminology differs from API or domain terminology,
explain the difference.

3. Required fields

A test case must include:

- Title;
- Preconditions containing only system state and test data;
- Shared steps if the same sequence appears in two or more cases;
- A scenario with an Expected Result for every step;
- Relevant artifacts;
- Tags and links.

4. Title

4.1 Mobile and frontend tests:

<Screen or section> — <What is being tested> —
<Expected effect>

4.2 API tests in a separate API project:

Suite = <endpoint or context>

Title = <scenario or variation>

Do not include generic terms such as Positive, Negative,
Search, or HTTP in the test case title.

5. Preconditions

Do not include instructions such as:

- Read the documentation.
- Prepare Postman.
- Import the cURL request.

These are actions, not system states.

Specify test data using either exact values or criteria for
finding suitable data. Also explain where to obtain it.

6. Steps

One step must contain one action.

Every Expected Result must be measurable. Specify exactly what
should be displayed, which status code should be returned,
which fields should be present, or which state should change.

=== REQUIRED RESPONSE FORMAT ===

A) Short assessment

✅ What is good

Provide 3–6 points.

❌ Problems and risks

Provide 3–8 points.

Be specific. Examples:

- Expected Result is not measurable.
- Preconditions contain instructions instead of state.
- Test data or selection criteria are missing.
- Several checks are mixed into one test case.
- The scenario is not deterministic.
- Terminology differs from the product.
- Repeated actions should be moved to a shared step.

B) Corrections: before → after

Use the following structure for every correction:

1. What is wrong:
   One sentence explaining why it must be changed.

2. Before:
   A short quotation or summary of the current text.

3. After:
   A concise improved version.

Provide corrections for the following areas:

- Title;
- Preconditions;
- Shared steps, if applicable;
- Scenario, including steps and an Expected Result for every step;
- Artifacts, tags, and links, if applicable.

Important:

- Do not write long theoretical explanations.
- Provide only the review and specific corrections.
- Avoid vague statements such as "the link opens."
- Specify where and how it opens: system browser or WebView,
  expected URL or title, close button, and absence of errors.
- If navigation, token preparation, or environment setup is
  repeated, explicitly recommend moving it to a shared step.
- If an exact value is unknown, use placeholders such as
  {banner_id}, {token}, or {card_number}.
- Explain how to find suitable data and verify that it is current.

=== INPUT DATA ===

The user provides the ID of the test case that must be reviewed.

Start the response immediately with section A, followed by
section B.

The `/review-test-case` Command

To allow QA engineers to launch a review with a single command, I added the review-test-case command.

# review-test-case

Review a test case in Allure TestOps and publish the result
as a NEW comment on that test case through the TestOps MCP server.

Important:

Always create a new comment.

Never update an existing comment and never delete comments.

Input parameter:

{testcase} — a test case URL or ID.

Data priority:

1. Retrieve the test case data through MCP using
   testcasegetbyid and testcasegetscenario.

2. If some fields in the API response are empty—for example,
   scenario: null or steps: []—but a screenshot is attached
   to the message, use the information visible in the screenshot.

3. Do not state that the scenario or field is empty if its
   contents are visible in the attached image.

Example:

/review-test-case 70756

Response requirements:

- Start with section A, followed by section B.
- Section A must contain:
  - ✅ What is good: 3–6 points.
  - ❌ Problems and risks: 3–8 points.
- Section B must provide "before → after" corrections for:
  - Title;
  - Preconditions;
  - Shared steps, if applicable;
  - Scenario, including an Expected Result for every step;
  - Artifacts, tags, and links, if applicable.

Why We Added a Second MCP Server for Sequential Thinking

We connected two MCP servers to the project:

The TestOps MCP server, which retrieves test cases and scenarios and publishes comments.
A sequential-thinking MCP server, which encourages the model to analyse the test case step by step.

We expected sequential analysis to be useful for test case reviews in the same way that it can be useful for code-related tasks.

The model can examine the test case in the following order:

Preconditions
    ↓
Steps
    ↓
Expected results
    ↓
Test data
    ↓
Determinism
    ↓
Suitability for automation

This process helps expose common problems:

Vague or immeasurable expected results.
Missing test data.
Several independent checks mixed into one test case.
Environment or feature-flag dependencies that are not documented.
Steps that cannot be automated reliably.

This is currently an observation from our experiments rather than proof that the additional MCP server improves every review.

We still need to evaluate its effect using a larger and more representative set of test cases.

What the Workflow Looks Like in Practice

The process is intentionally simple:

Open the required test case in Allure TestOps.
Run the following command in Cursor:

/review-test-case <test-case-id>

Attach a screenshot if necessary.

The screenshot acts as a fallback because the MCP server may occasionally return an incomplete scenario.

Cursor retrieves the test case through MCP.
The model reviews the test case using the project guidelines.
The MCP server publishes the result as a new comment in Allure TestOps.

A few seconds later, the QA engineer can see the review directly in the test case.

What We Learned

This prototype is deliberately small.

It does not attempt to replace an experienced QA engineer or automatically approve test cases. Its purpose is to identify common quality problems and provide specific corrections before a human reviewer examines the case.

Even this minimal implementation can help detect:

Ambiguous titles.
Instructions incorrectly placed in preconditions.
Missing test data.
Steps without expected results.
Non-measurable expected results.
Repeated actions that should become shared steps.
Scenarios that depend on undocumented state.
Test cases that are difficult to automate.

The main advantage is that the result appears inside the tool where QA engineers already work.

They do not need to copy the test case into a separate chatbot and then manually transfer the review back.

Conclusion

The complete workflow is straightforward:

QA engineer
    ↓
Cursor command
    ↓
TestOps MCP
    ↓
Test case data
    ↓
LLM review
    ↓
New TestOps comment

For us, this was a practical way to test AI-assisted test case reviews before building a complete web interface.

The next steps are to:

Improve the review prompt.
Measure review quality using a larger dataset.
Reduce incomplete responses from the TestOps API.
Determine which parts of the workflow can be safely automated.

For a small prototype, I think the result is promising.

If there is enough interest, I will publish a follow-up describing how we evaluate review quality and generate automated API tests from TestOps test cases.

Let’s Work Together

If this sounds relevant to your team, feel free to connect with me on LinkedIn.

DEV Community: Oleg Malyshev

How We Taught AI to Triage Failed Autotests and File Bugs in the Tracker

The problem: the tests failed — now what?

Why defects in TestOps

What the manual process looks like

1. Open the run and look at the unresolved results

2. Analyse the error in the failed test

3. Create a new defect or reuse an existing one

4. Create or link a tracker issue

5. Configure the automation rule

6. Verify the result

The rules we settled on

Where AI comes in

What it looks like in practice

What's next

Short version

Let’s Work Together

How We Turned Swagger from Documentation into the Engine Behind Our API Tests

Let’s Work Together

How We Migrated Test Cases from a Task Tracker to Allure TestOps with a Single Command Using Cursor + MCP

The problem: test cases living in the wrong place

The approach: Cursor + two MCP servers + two prompts

What the source data looked like

The project rule: parsing the tracker and producing JSON

The command: /move, one issue or a batch

MCP configuration

What this looks like for a QA engineer

Where else this pattern fits

Let’s Work Together

How We Built an AI Test Case Reviewer Using Cursor, MCP, and Allure TestOps

We Decided to Start With AI Test Case Reviews

Why We Did Not Use the Built-in Allure TestOps MCP

How Our MCP Server Works

Why We Used Cursor

How the Cursor Project Is Structured

User Rule

The Test Case Review Prompt

The /review-test-case Command

Why We Added a Second MCP Server for Sequential Thinking

What the Workflow Looks Like in Practice

What We Learned

Conclusion

Let’s Work Together

The `/review-test-case` Command