DEV Community

Jessica Sales Melo
Jessica Sales Melo

Posted on Originally published at github.com

AI-Powered Test Case Generation: Turning a Feature Description into Executable Gherkin

#ai #testing #playwright #llm

AI-Powered Test Case Generation: Turning a Feature Description into Executable Gherkin

A practical look at plugging an LLM into the QA cycle — and why a controlled vocabulary keeps AI-generated tests runnable.

As a QA Engineer, one of my biggest experiments this year was: can AI write tests that actually run? The short answer is yes — if you constrain the "creativity." In this article I share a working approach where you describe a feature in plain text and an AI generator produces Gherkin scenarios executed by Playwright.

This is a write-up of a portfolio project. The full, runnable source is on my GitHub.

The pipeline 

feature description (.md)  →  🧠 AI generator  →  Gherkin (.feature)  →  Playwright runs it
Enter fullscreen mode Exit fullscreen mode

1. The goal: text → runnable Gherkin

The user writes a natural-language spec (what the feature does and its acceptance criteria). The generator uses AI to produce a .feature file. Then playwright-bdd runs it for real.

2. The key design decision: a pluggable IAProvider

The heart is a simple interface:

interface IAProvider {
  generateFeature(spec: string, glossary: string): Promise<string>;
}
Enter fullscreen mode Exit fullscreen mode

Two implementations:

Provider Used when What it does
deterministic No API key Generates scenarios from curated templates, detected by keywords in the spec. Runs on any environment (local and CI), no secret needed.
openai OPENAI_API_KEY set Uses a real LLM to derive new scenarios from the description.

Selection is automatic: if a key exists, use the LLM; otherwise fall back to deterministic. This keeps the pipeline run-ready even with no credentials.

3. The trick that makes AI tests executable: a controlled glossary

This is the most important lesson. If you let an LLM invent any step text, you end up with beautiful scenarios that cannot run — because no step definition matches them.

Solution: give the model a glossary of supported steps and ask it to generate only within that vocabulary:

PASSOS DISPONÍVEIS (use exactly these):
- Given I am on the page "PAGINA"
- When I check checkbox number N
- When I select the option "OPÇÃO" in the dropdown
...
Enter fullscreen mode Exit fullscreen mode

The prompt also includes app context (credentials, expected messages). Result: AI-generated Gherkin always matches the step definitions.

4. Prompt engineering for deterministic-enough output

  • temperature: 0.3 → low creativity, high adherence.
  • System prompt: "Generate ONLY the Gherkin file, starting with # language: pt. Do not add commentary."
  • Post-process: strip markdown fences the model sometimes wraps in.

5. Validation

bddgen (missing-step detection) runs as a quality gate — if the AI invents a step, generation fails early instead of at runtime. Then Playwright executes the scenarios.

Takeaways

  • AI accelerates authoring, but the executable contract (glossary ↔ step definitions) is what keeps quality high.
  • A pluggable provider makes the tool work everywhere: deterministic in CI, real LLM locally.
  • This is a realistic template for "AI in QA" that isn't just a demo — it produces tests that run.

The source (with the deterministic provider, OpenAI provider, and the glossary) is public on my GitHub — a good starting point if you want to experiment with AI-assisted test generation.

Jessica Sales — QA Engineer, exploring AI applied to the test automation lifecycle.

Top comments (0)