DEV Community: Nikita Tsvetkov

Always Up-to-Date API Docs Are Real (And No, It’s Not AI)

Nikita Tsvetkov — Fri, 15 Aug 2025 18:18:34 +0000

API documentation is broken. You've likely seen it: endpoints that don't exist anymore, missing response codes, or a lingering "TODO" in a YAML spec. And that's assuming there's a spec at all.

The core issue? Most developers don't enjoy maintaining documentation. Manually editing OpenAPI specs or annotating endpoints with docstrings is tedious, repetitive, and error-prone.

Modern tools try to ease this pain, but each comes with its own trade-offs.

Where Existing Tools Fall Short

1. Export-Based Tools (Postman, Insomnia, etc.)

Tools like Postman allow you to build collections and export them as OpenAPI specs. This is certainly better than hand-writing YAML, but it's still manual work. You need to remember to update your collections every time you change an endpoint, add parameters, or modify response structures.

The workflow looks like this: change code → update Postman collection → export spec → and hope nothing was missed. It's not error-prone in the traditional sense, but it requires discipline and creates another maintenance burden that developers often skip when deadlines loom. You're always just one forgotten update away from spec drift.

2. Annotation- and Decorator-Based Tools

A step up from manual exports are tools that let you annotate or decorate your routes directly in code. Flask-RESTX, Django REST Framework, and similar libraries fall into this category. You add decorators or docstrings to your endpoints, and the spec gets generated automatically.

from rest_framework import viewsets
from drf_spectacular.utils import extend_schema, OpenApiParameter
from drf_spectacular.types import OpenApiTypes

class UserViewSet(viewsets.ModelViewSet):
    serializer_class = UserSerializer

    @extend_schema(
        summary="List all users",
        description="Returns a paginated list of all users in the system",
        parameters=[
            OpenApiParameter(
                name='search',
                description='Search users by username',
                required=False,
                type=str
            ),
        ],
        responses={
            200: UserSerializer(many=True),
            403: {"description": "Forbidden"}
        },
    )
    def list(self, request, *args, **kwargs):
        return ...

This approach keeps documentation closer to the code, which is good. But it's still fundamentally manual work. If a developer forgets to add or update an annotation, the spec compiles successfully but becomes incorrect. Maybe missing a 422 validation error response or an optional query parameter. The spec generation works perfectly; the human memory doesn't.

3. Auto-Generation from Type Hints (FastAPI and Friends)

The most sophisticated approach uses type hints and models to automatically generate documentation. FastAPI is the poster child here - it introspects your Pydantic models and function signatures to build comprehensive OpenAPI specs with minimal manual input.

from typing import List, Optional
from fastapi import FastAPI, Query

@app.get("/users/", response_model=List[UserResponse])
async def list_users(
    skip: int = Query(0, ge=0, description="Number of records to skip"),
    limit: int = Query(10, ge=1, le=100, description="Max number of records to return"),
    role: Optional[UserRole] = Query(None, description="Filter by user role"),
    is_active: Optional[bool] = Query(None, description="Filter by active status"),
    search: Optional[str] = Query(None, min_length=1, description="Search in username or email")
):
    return ...

This can automate a significant portion of the work and produces impressively detailed documentation. But edge cases still require manual intervention: middleware logic, custom response codes, specific media types, and detailed examples. You're also locked into a specific framework from day one of your API development, which is a significant architectural commitment.

The Uncomfortable Truth

Even with these advanced tools, the problem persists at scale. According to Nordic APIs research, 43% of APIs don't have a public specification at all, and among those that do, 75% of production APIs have variances from their published OpenAPI specs.

It's not because developers are lazy or tools are bad. It's because they're fighting against the fundamental nature of software development: code changes constantly, and any manual process, no matter how streamlined, will eventually fall behind.

Every manual step is a potential point of failure. Every annotation that needs updating is a chance for drift. Every example that needs refreshing is an opportunity for confusion.

A different approach is needed. One that doesn't rely on developers remembering to update documentation. One that can't drift because it's generated from the single source of truth about API behavior.

That source of truth? It's been in codebases all along.

Tests Are Already Your API Specification

Here's a thought experiment: What if you're already writing comprehensive API documentation every day, you just don't realize it?

Think about what your API tests actually do. They:

Make real HTTP requests to every endpoint
Send actual request bodies with realistic data
Include all required headers and authentication
Receive and validate real responses
Check different status codes and error conditions

Your test suite is essentially a living, executable specification of your API. It knows exactly how your API behaves because it's constantly verifying that behavior. The only problem? This knowledge is trapped in test code, invisible to the developers who need it most.

The Insight That Changes Everything

What if you could capture all those test requests and responses and transform them into OpenAPI documentation? Not by parsing test code or adding annotations, but by simply observing what actually happens when tests run?

This isn't a pipe dream. It's exactly what vedro-httpx does - an official plugin for the Vedro testing framework that not only makes API testing effortless but also generates OpenAPI specs from your test runs.

Here's the beautiful part: you don't write documentation. You don't annotate anything. You don't maintain separate files. You just write good tests (which you're doing anyway), and documentation appears. Always accurate. Always complete. Always up-to-date.

How It Actually Works

The Interface Pattern

vedro-httpx encourages wrapping external services in lightweight interface classes. This pattern provides three key benefits:

Centralized configuration for reusable HTTPX clients (base URL, headers, TLS, timeouts)
Clear, type-annotated methods for each endpoint that keep tests readable
A unified Response object that Vedro renders clearly when assertions fail

Here's a typical interface:

from vedro_httpx import Response, SyncHTTPInterface

class AuthAPI(SyncHTTPInterface):
    def __init__(self, base_url: str = "http://localhost") -> None:
        super().__init__(base_url)

    def login(self, username: str, password: str) -> Response:
        return self._request("POST", "/auth/login", json={
            "username": username,
            "password": password
        })

With the AuthAPI interface defined, you can use it in test scenarios:

from vedro_fn import scenario
from contexts import registered_user
from interfaces import AuthAPI

@scenario
def login_as_registered_user():
    user = registered_user()
    response = AuthAPI().login(user["username"], user["password"])
    assert response.status_code == 200

Now vedro-httpx tracks every request and response flowing through your tests. The documentation generation works through two straightforward steps.

Step 1: Record Requests

With a simple CLI flag, vedro-httpx records all HTTP requests and responses during test execution:

$ vedro run --httpx-record-requests

After the test run, all request/response data is saved to .vedro/artifacts in HAR format , a widely-used standard that's human-readable JSON. HAR files can be opened in browser developer tools, analyzed with tools like Fiddler, or viewed online with tools like Google HAR Analyzer.

The format looks like this:

{
  "log": {
    "entries": [
      {
        "request": {
          "method": "POST",
          "url": "<URL>",
          "queryString": [],
          "headers": ...
        },
        "response": {
          "status": 200,
          "statusText": "OK",
          "headers": ...,
          "content": ...
        }
      }
    ]
  }
}

Step 2: Generate Documentation

Based on the HAR files, vedro-httpx generates a ready-to-use OpenAPI spec with a single command:

$ vedro_httpx .vedro/artifacts > spec.yaml

The generated spec captures everything your tests actually did:

openapi: 3.0.0
info:
  title: API
  version: 1.0.0
servers:
- url: https://chat-api-tutorial.vedro.io/tl3mzuetbb
paths:
  /chats/{chat_id}/messages:
    post:
      summary: Endpoint for POST /chats/{chat_id}/messages
      operationId: post_chats_chat_id_messages
      parameters:
      - name: x-auth-token
        in: header
        required: true
        description: X auth token header
        schema:
          type: string
      responses:
        '200':
          description: OK
        '400':
          description: Bad Request
        '401':
          description: Unauthorized

Render it with your favorite tool:

$ docker run -p 8080:8080 -v $(pwd):/app -e SWAGGER_JSON=/app/spec.yaml swaggerapi/swagger-ui

That's it. Your tests run, and documentation is generated. No manual steps, no drift, no outdated examples.

The Magic? There Is No Magic

This approach works because it's based on a simple truth: your tests are already doing the work. They're already making real requests and receiving real responses. The tool is just capturing that information and presenting it in a different format.

No guessing about what your API does. No forgetting to document edge cases. No drift between documentation and reality. If your tests pass, your documentation is accurate. If your API changes, your tests fail, you fix them, and your documentation updates automatically.

It's not AI trying to understand your code. It's not complex static analysis. It's just recording what actually happens and turning it into documentation. Simple, reliable, and impossible to get wrong.

The Hidden Benefits You Didn't Expect

When you start generating documentation from tests, something interesting happens. You don't just solve the documentation problem; you accidentally improve your entire API development workflow.

Zero Specification Drift (Really, Zero)

This isn't hyperbole. With traditional approaches, drift is inevitable because documentation and code are separate artifacts that must be manually synchronized. With test-generated docs, drift is literally impossible.

Your tests are your specification. Break one, you break the other. The documentation can't lie about your API's behavior because it's generated from actual API behavior. This is a fundamentally different guarantee than "trying really hard to keep docs updated."

Living Documentation That Updates Itself

Imagine documentation that refreshes with every CI run. No "documentation sprints". No tickets titled "Update API docs for Q3 changes." No archeology expeditions to figure out when the docs stopped matching reality.

Set up a simple GitHub Action with automatic publishing to GitHub Pages and your documentation stays current without any human intervention. Every merge to main updates your documentation. Your API docs are as fresh as your latest test run. Documentation becomes a build artifact, not a maintenance burden.

Instant API Coverage Metrics

Here's a benefit that surprises everyone: gaps in your generated documentation reveal untested endpoints. It's an instant, visceral API coverage metric.
Running the generator and notice /users/{id}/preferences isn't in the spec? That endpoint isn't tested. The absence of documentation becomes a todo list for test coverage. You can even fail builds if certain endpoints disappear from the generated spec, catching accidentally untested code before it ships.

Friction-Free Contract Testing

Your generated OpenAPI spec isn't just documentation - it's a contract. Frontend teams can:

Generate TypeScript clients automatically
Spin up mock servers with realistic responses
Validate their requests against your actual API behavior

Partner integrations become smoother too. Instead of maintaining separate "partner documentation", you hand them the same OpenAPI spec your tests generate. They're coding against your API's actual behavior, not your best guess at what it does.

Design Feedback Loop

This might be the most underrated benefit: bad documentation often reveals bad API design, but only after it's too late. With test-generated docs, you see your API's real interface while you're building it.

Endpoint naming inconsistencies jump out. Weird authentication patterns become obvious. Response format discrepancies glare at you from the generated spec. It's like having an API design reviewer that never sleeps and never misses anything.

The Compound Effect

Each of these benefits reinforces the others:

Better test coverage leads to better documentation →
Better documentation reveals design issues earlier →
Better design makes writing tests easier →
Easier tests mean more coverage

It's a virtuous cycle where improving any part improves the whole. You're not just solving documentation - you're systematically improving API quality.

How to Use It

The process has two stages: recording requests (1) and generating documentation (2). The beauty is that the second stage works with any language or framework. You just need HAR files as input.

If You're Using Vedro and vedro-httpx

This is the simplest path. Just run your existing tests with the recording flag:

$ vedro run --httpx-record-requests

Then generate the spec:

$ vedro_httpx .vedro/artifacts > spec.yaml

If You're Using Another Python Framework

You can still use vedro-httpx in your tests, even if you're not using Vedro as your main testing framework. The HTTP interface pattern works anywhere.

If You're Using Another Language

You need to record requests and responses in HAR format. Several approaches can help you capture this data:

Playwright allows recording HAR files for browser-based testing
Browser developer tools can export network activity as HAR
HTTP proxies like mitmproxy can intercept and export traffic as HAR
Some HTTP clients have extensions or middleware for HAR export

Once you have HAR files, the vedro_httpx command-line tool can generate OpenAPI specs from them regardless of how they were created.

You could even capture production traffic (carefully!) and generate documentation from real-world usage patterns.

The Bigger Picture

This approach represents a fundamental shift in how to think about documentation. Instead of treating it as a separate artifact that requires maintenance, it recognizes that documentation is just another view of a system's behavior. And behavior is already captured in tests.

Today it's API documentation. Tomorrow? Maybe infrastructure documentation from deployment tests. Configuration documentation from integration tests. Architecture diagrams from service interaction tests. Once you start seeing tests as executable specifications, you see opportunities everywhere.

The tools and techniques are here. The only question is: how much more time do you want to spend manually updating YAML files?

Preventing Flaky Tests: Best Practices for Reliable Automation

Nikita Tsvetkov — Sat, 24 Feb 2024 13:03:13 +0000

What are Flaky Tests?

Flaky tests are software tests that exhibit unpredictable outcomes, switching between passing and failing, despite no changes in code or environment. This inconsistency can mislead by masking the presence or absence of bugs and may be caused by various factors, including timing issues, external dependencies, and stateful tests.

The Impact of Flaky Tests

Flaky tests not only disrupt the testing process but also have broader implications on the overall software development lifecycle. The effects extend far beyond mere annoyance, undermining the foundation of trust, efficiency, and speed within the development and testing teams. Let's explore the key impacts of flaky tests to fully understand their ramifications.

1. Erosion of Trust in Test Results

The primary impact of flaky tests is the erosion of trust in the testing process. When tests yield inconsistent results, developers and testers start to question the validity of all test outcomes, not just the flaky ones. This skepticism can lead to a dismissive attitude toward failing tests, potentially allowing real issues to slip through unnoticed. Over time, this erodes confidence in the testing suite as a reliable quality assurance tool.

2. Wasted Time and Resources

Flaky tests are notorious time sinks. Engineers often find themselves rerunning tests to determine if a failure is genuine or just another instance of flakiness. This repetitive process consumes valuable time and resources that could be better allocated to more productive tasks, such as feature development or addressing actual bugs.

3. Delayed Feedback Loop

In continuous integration and continuous deployment (CI/CD) environments, flaky tests can cause significant delays in the feedback loop. The uncertainty caused by these tests means that code changes cannot be reliably and swiftly moved through the pipeline. These delays impede the agility of the development process, slowing the delivery of new features and fixes. In fast-paced development environments, these delays can be especially detrimental, impacting the overall time-to-market and the ability to quickly adapt to user requirements.

The First Line of Defense

The best approach to handle flaky tests is to prevent them from occurring in the first place. Here’s how Vedro contributes to this:

🛠️ Boosting Test Reliability with Vedro's Best Practices

Vedro promotes best practices that not only make tests maintainable but also stable, like writing clear tests, avoiding dependencies between tests, and ensuring each test is self-contained. By encouraging and facilitating good practices, Vedro significantly reduces the chances of introducing flaky tests in the first place.

🔄 Achieving Test Consistency Through Multiple Runs

Another effective strategy employed by Vedro to combat the issue of flaky tests is the implementation of multiple test runs. This approach is rooted in the principle that consistency in test results is key to establishing their reliability. Running a new test multiple times before integrating it into the codebase greatly enhances the chances of catching flaky behavior early in the development process.

Vedro simplifies this process with straightforward command-line options. For example, executing a test multiple times is achievable using the command:

$ vedro run -N 3

Output example:

Scenarios
*
 ✔ register via email
 │
 ├─[1/3] ✔ register via email
 │
 ├─[2/3] ✔ register via email
 │
 ├─[3/3] ✔ register via email


# --seed fb0095bc-db57-4976-9368-a356dfb1ff94
# repeated x3
# 1 scenario, 1 passed, 0 failed, 0 skipped (0.49s)

🔍 Unveiling Hidden Dependencies with Random Test Ordering

A critical factor in ensuring the robustness of a test suite is the independence of each test. Tests should not rely on the execution sequence or the side effects of other tests to pass. However, in reality, hidden dependencies between tests can exist, often unnoticed until they manifest as flaky behavior.

Running tests in a randomized order is an effective strategy to expose these hidden dependencies. If a test suite consistently passes when executed in a specific order but fails under a randomized sequence, it indicates that some tests are not truly independent. This scenario is a red flag, signaling the presence of interdependencies that can lead to unpredictable test outcomes.

Vedro makes it easy to run tests in a random order, which can be done using a simple command:

$ vedro run --order-random

Ensuring Test Reliability

While the features and best practices provided by Vedro significantly enhance test reliability, there is an inherent challenge: ensuring that all team members consistently follow these guidelines and run new or modified tests before integrating them into the codebase. Vedro addresses this challenge through automation and integration tools designed to enforce best practices and efficiently detect flakiness.

🤖 Automating Flakiness Detection

Vedro offers a plugin, vedro-git-changed, which automatically identifies and runs test scenarios that have changed relative to a specified git branch. This targeted approach ensures that newly introduced or modified tests are scrutinized for flakiness before being merged.

To install the plugin, use the command:

$ vedro plugin install vedro-git-changed

To run tests that have changed against the main branch, execute:

$ vedro run --changed-against-branch=main

To further bolster flaky test detection and ensure test suite robustness, the plugin can be seamlessly integrated into CI pipelines. Here are example configurations for GitLab CI and GitHub Actions.

GitLab CI Configuration:

# .gitlab-ci.yml
run_changed_tests:
  stage: test
  image: python:3.11
  before_script:
    - pip install -r requirements.txt
  script:
    - vedro run --changed-against-branch=main -N 10 --order-random
  only:
    refs:
      - branches
    changes:
      - scenarios/**/*

GitHub Actions Workflow:

# .github/workflows/run_changed_tests.yml
on:
  push:
    branches-ignore:
      - 'main'
    paths:
      - 'scenarios/**'

jobs:
  run_changed_tests:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout Code
      uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Install Dependencies
      run: pip install -r requirements.txt

    - name: Run Changed Tests
      run: vedro run --changed-against-branch=main -N 10 --order-random

The vedro-git-changed plugin not only simplifies the process of running new or changed tests locally but also establishes a robust quality gate within the CI pipeline. Automatically running tests multiple times in random order reduces the likelihood of undetected flakiness.

✅ Enforcing Best Practices

Flake8-Vedro extends the capabilities of the popular Flake8 linting tool by introducing rules and checks specifically tailored for Vedro test scenarios. Although it might not cover every possible best practice, it focuses on enforcing a set of rules that significantly contribute to improving test quality.

To install flake8-vedro, run the following command:

$ pip install flake8-vedro

To check for linting errors, use the command below:

$ flake8

Integrating Flake8-Vedro into the CI pipeline adds another layer of quality control. By automatically running this specialized linter on each commit, it detects deviations from established best practices early in the development cycle. Alongside code reviews, this automated check acts as a vigilant gatekeeper, ensuring that all tests remain maintainable and stable. This approach upholds the quality and reliability of the test suite.

Up Next: Tackling Flaky Tests in CI

Despite comprehensive preventive measures, flaky tests can still sneak into continuous integration (CI) environments. The next article will focus on strategies for dealing with flaky tests that have bypassed initial defenses. To ensure staying updated on this topic and more, follow Vedro on Telegram, Instagram, and Twitter.

Avoid Ifs in Tests

Nikita Tsvetkov — Wed, 11 Oct 2023 18:44:52 +0000

Like any code, test code also has its set of best practices and pitfalls. One common but easily overlooked pitfall is the use of conditional statements, particularly "if" statements, in tests.

At first glance, using "if" statements in tests might seem harmless or even necessary. After all, the conditional logic allows tests to adapt based on different parameters, making it seem more flexible. However, this apparent benefit is misleading. Introducing conditional statements into tests can lead to problems, including unclear test results, gaps in test coverage, and increased code complexity.

The Problem with Ifs

Let's start with an example to illustrate the problem. Imagine a movie streaming platform that offers age-restricted movies like "John Wick", which should be available only to viewers 17 years old or above.

The initial test for this feature might look like this:

import vedro
from contexts import logged_in_user, open_age_restricted_movie

class Scenario(vedro.Scenario):
    subject = 'open age-restricted movie'

    def given_logged_in_user(self):
        # `logged_in_user` provides a user with an age
        self.user = logged_in_user()

    def when_user_opens_movie_page(self):
        self.page = open_age_restricted_movie(self.user)

    def then_it_should_check_user_age(self):
        if self.page.text('.warning') == 'You must be 17+ to watch this movie':
            assert self.page.exists('.movie-title') is False
        else:
            assert self.page.exists('.movie-title') is True

On the surface, this test may look okay. It checks whether the user sees a warning message when trying to view an age-restricted movie and also checks if the movie is actually displayed. However, the test has a crucial flaw: it could pass for the wrong reasons.

Frontend Ignores Age Restriction: Suppose the frontend application completely disregards the age restriction and allows everyone to watch "John Wick." In this situation, the test would still pass (the absence of a warning message would trigger the else condition).
Frontend Restricts the Movie for All Users: On the other hand, if the frontend application restricts the movie for all users, regardless of their age, the test will still pass (the warning message would display for every user, satisfying the if condition).

Both cases demonstrate that the test might not effectively verify the intended behavior. A passing test does not necessarily confirm that the application is correctly implementing age restrictions.

The core of this issue is the test's reliance on output data, in this case, the warning message. Verifying outcomes based solely on output, such as a user-facing warning message, introduces significant risk. Numerous factors, including future changes in the UI, refactoring of the underlying codebase, and adjustments in internationalization and localization, can influence output data. This makes it an unreliable criterion for evaluating the correct functionality of the code being tested.

An Attempt to Refine the Test

class Scenario(vedro.Scenario):
    subject = 'open age-restricted movie'

    def given_logged_in_user(self):
        self.user = logged_in_user()

    def when_user_opens_movie_page(self):
        self.page = open_age_restricted_movie(self.user)

    def then_it_should_check_user_age(self):
        if self.user['age'] >= 17:
            assert self.page.exists('.movie-title') is True
        else:
            assert self.page.exists('.movie-title') is False

In this version, the "if" condition is based on the age of the logged-in user, providing a more reliable foundation for the test. The age is a part of the input data, which the test assumes is under control and therefore reliable.

However, this refined test still has significant drawbacks. The test heavily relies on the logged_in_user function to dictate its path. What if, due to future code refactoring or changing business requirements, logged_in_user consistently returns users aged 17 or older? In such a case, the test would provide coverage only for one specific scenario—that users aged 17 or above can view the movie. It would fail to verify that younger users are restricted from watching age-sensitive content, essentially defeating the purpose of the test.

Even though the test appears to be improved, it still lacks comprehensive coverage. By relying on conditional statements, it opens the door to scenarios where it might not fully validate the functionality it is intended to test.

The Dangers of Using Ifs

As seen from the examples above, even a well-intentioned effort to make the test more robust can falter when "if" statements are involved. But why are "if" statements so problematic in tests? Let's delve into some key dangers.

Unnecessary Complexity

Introducing an "if" statement into a test adds branching logic to test code. In other words, the test can take multiple paths based on varying conditions. While this might seem like a flexible approach, it actually introduces unnecessary complexity. A test should have a straightforward flow so that it clearly either passes or fails based on predefined conditions. If there are multiple branching paths, understanding why a test failed—or even worse, why it passed—becomes difficult.

Business Logic in Tests

Tests are meant to validate business logic, not contain it themselves. When a test starts to include its own branching logic and conditions, it essentially becomes a mini-program replete with its own business logic. This poses a significant problem because if the test itself is flawed, it might require its own set of tests to validate. This results in a recursive issue, leading away from the primary purpose of a test, which is to serve as a reliable indicator of code quality.

Strategies for Improvement

Parameterized Testing

One effective strategy to address the challenges of conditional logic in tests is through parameterized testing. This approach allows the same test logic to run multiple times with different sets of parameters. Instead of using branching logic for various data combinations, parameterized tests ensure a linear flow, which enhances their clarity and reliability.

import vedro
from vedro import params
from contexts import logged_in_user, open_age_restricted_movie

class Scenario(vedro.Scenario):
    subject = 'open age-restricted movie'

    @params(age=13, can_view_movie=False)
    @params(age=17, can_view_movie=True)
    def __init__(self, age, can_view_movie):
        self.age = age
        self.can_view_movie = can_view_movie

    def given_logged_in_user(self):
        self.user = logged_in_user(age=self.age)

    def when_user_opens_movie_page(self):
        self.page = open_age_restricted_movie(self.user)

    def then_it_should_check_user_age(self):
        assert self.page.locator('.movie-title').exists() == self.can_view_movie

In the example above, the @params decorator is used to provide different sets of data for the test. Each set represents a different test scenario.

Splitting Tests

Another method to eliminate conditional statements from tests is to split them into separate tests. Instead of one test trying to cater to both positive and negative scenarios, it's often clearer to have individual tests, each tailored for a specific responsibility.

One test would handle the positive scenario where the user, aged 17, should be able to open an age-restricted movie.

import vedro
from contexts import logged_in_user, open_age_restricted_movie

class Scenario(vedro.Scenario):
    subject = 'open age-restricted movie'

    def given_logged_in_user(self):
        self.user = logged_in_user(age=17)

    def when_user_opens_movie_page(self):
        self.page = open_age_restricted_movie(self.user)

    def then_it_should_show_movie(self):
        assert self.page.locator('.movie-title').exists()

Conversely, another test would deal with the negative scenario, ensuring that a 13-year-old user cannot access age-sensitive content.

import vedro
from contexts import logged_in_user, open_age_restricted_movie

class Scenario(vedro.Scenario):
    subject = 'try to open age-restricted movie'

    def given_logged_in_user(self):
        self.user = logged_in_user(age=13)

    def when_user_opens_movie_page(self):
        self.page = open_age_restricted_movie(self.user)

    def then_it_should_restrict_movie(self):
        assert self.page.locator('.movie-title').not_exists()

Combining Approaches

Both methods, splitting tests and parameterized testing, offer unique advantages when applied in isolation. However, their true potential is realized when they are combined, leading to enhanced test coverage and clarity.

By splitting tests, each test is tailored with a specific responsibility, providing clear intent. It becomes straightforward to identify what each test is set out to achieve. Meanwhile, parameterized testing applies consistent test logic across different data sets. This ensures that the application behaves as expected for a wide range of scenarios. For example, adding more test scenarios, such as a 16-year-old user for a negative test and an 18-year-old user for a positive test, ensures that the age "17" isn't just hard-coded into the application logic.

Scenario-Based Testing: The Key to Maintainable Automation

Nikita Tsvetkov — Fri, 15 Sep 2023 14:40:25 +0000

The Nature of Interactions

Interactions, whether between a user and software or between different software components, follow a distinct pattern. They begin with intent, proceed through a sequence of steps, and culminate in an expected outcome.

Consider, for instance, the act of shortening a URL. The intent is clear: to convert a lengthy URL into a shorter, more manageable one. The sequence of steps involves opening the URL shortener service, entering the original lengthy URL, clicking the "shorten" button, and then receiving the shortened URL. The expected outcome? Acquiring a shortened version of the original URL that redirects to the original webpage.

In software testing, recognizing this structure is crucial. It not only reflects our natural thought processes but also provides a blueprint for creating tests. By designing tests that mimic this structure, engineers can craft scenarios that align closely with real-world use cases. It promotes clarity and understandability in test scenarios, making it easier to discern the purpose and flow of each test.

Anatomy of a Scenario

Vedro takes this principle to heart. It presents interactions in the form of scenarios, making test creation intuitive and straightforward. Each scenario starts with the subject (which represents the intent) and consists of a sequence of steps. These steps are divided into three key phases:

Given Step(s). The given steps form the first phase. These steps prepare the system's state and any necessary data for the main action being tested. They set up the conditions under which the when step, the main action, takes place.
When Step. Following the given steps is the when step. This is the primary action being tested. It represents the user or system interaction we aim to examine. This step performs the crucial action that will have some sort of effect on the system state, which is subsequently examined in the then steps.
Then Step(s). The final phase consists of the then steps. Here, we verify the system's responses (or side-effects) against the expected outcomes of the primary action performed in the when step.

Consider the following example as an illustration:

import vedro
from contexts import opened_url_shortener

class Scenario(vedro.Scenario):
    subject = "shorten url"

    def given_opened_shortener(self):
        self.page = opened_url_shortener()

    def given_original_url(self):
        self.page.fill_url("https://vedro.io/docs/quick-start")

    def when_user_shortens_url(self):
        self.page.click_shorten_button()

    def then_it_should_shorten_url(self):
        shortened_url = self.page.get_shortened_url()
        assert shortened_url

It's worth noting that Vedro's scenario anatomy directly follows the AAA (Arrange-Act-Assert) pattern.

The Power of the Scenario-Based Approach

Adopting a scenario-based approach provides significant advantages. It emphasizes simplicity, readability, and maintainability, allowing tests to be easily written and understood. Even complex test cases can be broken down into simpler, manageable scenarios, making it easier for teams to collaborate, understand each other's work, and maintain high code quality over time.

To harness the full potential of the scenario-based approach, Vedro builds upon the inverted pyramid concept. This concept mirrors the way we naturally perceive and process information, moving from a broad overview to specific details. It takes into account the reader's cognitive load, presenting the most vital information first and then descending into the intricate details.

Let's break it down.

1. Project Level

Here, scenarios offer a high-level view, organized neatly into files and directories. This bird's-eye view quickly reveals the behavior of the system and the scenarios being tested.

* chat_api/
└── scenarios/
    ├── auth/
    │   ├── register_new_user.py
    │   ├── login_as_registered_user.py
    │   └── try_to_login_as_nonexisting_user.py
    ├── send_message/
    │   ├── send_message.py
    │   └── try_to_send_message_as_unauthorized_user.py
    └── get_messages/
        ├── get_messages.py
        └── try_to_get_messages_as_unauthorized_user.py

This not only offers an instant overview without delving into individual tests but also promotes modularity. As the system grows, integrating new tests under relevant directories becomes seamless, ensuring the test suite remains scalable.

2. Scenario Level

Delving deeper, the scenario level provides detailed insights without overwhelming the reader with technicalities. Following the steps of a scenario reveals the core interaction and expected outcomes.

class Scenario(vedro.Scenario):
    subject = "login as registered user"

    def given_registered_user(self):
        ...

    def when_user_logs_in(self):
        ...

    def then_it_should_return_success_response(self):
        ...

    def and_it_should_return_created_token(self):
        ...

An added benefit of this structure is that steps can be integrated into reporting systems, like Allure. This makes the tracking of test execution and results more visual and organized, adding an extra layer of utility and traceability to the testing process.

3. Step Level

For those who seek a granular understanding, the step level delves into the specifics. Here lies the essence of each test, with the code ensuring no room for ambiguity.

    def when_user_logs_in(self):
        self.response = ChatApi().login(self.user)

    def then_it_should_return_success_response(self):
        assert self.response.status_code == 200

4. Beyond the Steps

Exploration doesn't stop at the step level. Delving deeper into classes, methods, and even underlying libraries reveals the mechanics that drive the tests. This depth offers insights into technical aspects such as properties of the HTTP client (for example, default request timeout), granting a comprehensive understanding of the underlying test machinery.

from vedro_httpx import Response, SyncHTTPInterface

class ChatApi(SyncHTTPInterface):
    def login(self, username: str, password: str) -> Response:
        return self._request("POST", "/auth/login", json={
            "username": username,
            "password": password
        })

Conclusion

Scenario-based testing, with its structured and intuitive approach, is an effective way to manage and maintain tests. Its high readability caters to various levels of engagement, allowing readers to delve as deep as they prefer.

Furthermore, this methodology integrates seamlessly with other best testing practices, uncovering new avenues for effective testing.