DEV Community

Cover image for No PASS, No PR: A Small Python Tool for Safe API Drift Repair
Burak
Burak

Posted on • Originally published at itnext.io

No PASS, No PR: A Small Python Tool for Safe API Drift Repair

#python #testing #api #github

APIs change. Tests break. That part is normal.

The annoying part starts when a tiny contract change turns into repetitive debugging.

A backend field gets renamed.

The OpenAPI schema is updated.

But an old YAML test case still sends the previous field name.

I built a small Python demo around that exact problem.

The tool is called API Drift Healer.

It detects a simple OpenAPI/test mismatch, generates a healed YAML test case, validates the fix locally, and opens a GitHub Pull Request only after the healed test passes.

The rule is simple:

No PASS, No PR.

No direct push to main.

No blind test rewriting.

No PR without validation.

The drift scenario

In this demo, the OpenAPI schema requires this field:

email_address
Enter fullscreen mode Exit fullscreen mode

But the existing YAML test case still sends the old field:

userEmail
Enter fullscreen mode Exit fullscreen mode

So the API rejects the request:

[FAIL] Expected 201, got 400
Error: missing required field: email_address
Enter fullscreen mode Exit fullscreen mode

OpenAPI schema showing raw `email_address` endraw as a required field
Image 1: OpenAPI schema showing email_address as a required field.

This is a small example, but it represents a common testing problem:

The contract changed, but the test did not.

Software has a special talent for turning one renamed field into a tiny disaster.

Before and after

The original test case sends:

body:
  name: Test User
  userEmail: qa_user@example.com
Enter fullscreen mode Exit fullscreen mode

The healed version sends:

body:
  name: Test User
  email_address: qa_user@example.com
Enter fullscreen mode Exit fullscreen mode

Before and after YAML test case
Image 2: Before and after YAML test case

The goal is not to rewrite tests blindly.

The goal is to detect a clear field-level mismatch and produce a small, reviewable fix.

V0.1: local healing

The first version was a local one-command flow.

It followed this sequence:

Run original test
Detect failure
Compare request body with OpenAPI schema
Map old field to new field
Generate healed YAML
Run healed test
Enter fullscreen mode Exit fullscreen mode

In this case, it detected:

userEmail -> email_address
Enter fullscreen mode Exit fullscreen mode

The core idea was intentionally deterministic.

This version does not ask an LLM to guess the fix. It only applies a patch when the mismatch is narrow enough to prove safely.

Instead of guessing, the tool checks for one narrow condition:

# Only heal when there is exactly one missing required field
# and exactly one invalid field in the request body.

if len(missing_required_fields) == 1 and len(invalid_existing_fields) == 1:
    old_field = invalid_existing_fields[0]
    new_field = missing_required_fields[0]

    # Safety guardrail:
    # Do not map unrelated fields into an email field.
    if "email" in new_field.lower() and "email" not in old_field.lower():
        print(f"[!] Risky match flagged: '{old_field}' -> '{new_field}'")
        return None

    # Apply the patch in memory.
    request_body[new_field] = request_body.pop(old_field)
Enter fullscreen mode Exit fullscreen mode

This keeps the first version conservative.

If the mismatch is simple and obvious, the tool heals it.

If the drift is more complex, it does nothing and leaves the decision to a human.

Then it writes the patched test case to:

api_test_case.healed.yaml
Enter fullscreen mode Exit fullscreen mode

The original test failed.

The healed test passed.

V0.1 terminal flow showing original failure, healed YAML generation, and passing validation
Image 3: V0.1 terminal flow showing original failure, healed YAML generation, and passing validation.

That proved the basic idea:

A simple API contract drift can be detected and healed automatically.

But local healing alone is not enough. A safer workflow needs human review.

V0.2: Pull Request flow

In V0.2, I added a GitHub Pull Request flow.

The tool still starts locally.

It runs the failing test, detects the drift, generates the healed test file, and validates the fix before touching GitHub.

Only after the healed test passes:

[PASS] Expected 201, got 201
Enter fullscreen mode Exit fullscreen mode

does the PR flow unlock:

SECURITY LOCK RELEASED: PASS received. Initiating GitHub PR flow...
Enter fullscreen mode Exit fullscreen mode

V0.2 terminal flow showing fail, heal, validate, and PR creation.
Image 4: V0.2 terminal flow showing fail, heal, validate, and PR creation.

Opening a PR is not the interesting part.

Refusing to open one until the healed test passes is the actual safety mechanism.

Automation that changes tests without validation is not helpful.

It is just a very confident bug delivery system.

The generated PR

After validation passes, the tool creates a new branch, commits only the patched test case, pushes the branch, and opens a Pull Request.

The PR includes a short report:

Root Cause:
OpenAPI requires email_address, but the test case was sending userEmail.

Applied Fix:
userEmail -> email_address

Validation:
Original test: Failed with 400
Healed test: Passed with 201

Safety:
This PR was created only after the healed test passed locally.
Enter fullscreen mode Exit fullscreen mode

Generated Pull Request report
Image 5: Generated Pull Request report

This makes the fix easier to review.

The tool proposes the change.

The reviewer still owns the merge.

A small, reviewable diff

The generated PR changes only one line:

- userEmail: qa_user@example.com
+ email_address: qa_user@example.com
Enter fullscreen mode Exit fullscreen mode

GitHub PR diff showing one changed field
Image 6: GitHub PR diff showing one changed field

This is the part I like most.

One changed line.

A documented reason.

A passing validation result.

A human review before merge.

That makes the tool less of a black box and more of a review assistant that explains its work.

What this demo solves

This demo intentionally focuses on one narrow problem:

API field name changes
Old test still sends the previous field
Test fails
Tool detects the mismatch
Tool proposes a validated fix through a PR
Enter fullscreen mode Exit fullscreen mode

It does not try to solve every API drift problem.

This version does not handle:

  • nested objects
  • multiple field changes
  • large schemas
  • authentication issues
  • CI-based validation

That is intentional.

I wanted the first version to be safe before making it broad.

A small, safe automation is better than a large one that quietly creates more work.

Next steps

The next version could add:

  • GitHub Actions validation
  • PR labels
  • risk scoring
  • nested JSON support
  • multiple test case support
  • better drift classification

The most important next step is moving validation from local execution to CI.

A stronger workflow would look like this:

Original Test FAIL
        ↓
Drift Detected
        ↓
Healed Test Generated
        ↓
Local PASS
        ↓
PR Opened
        ↓
GitHub Actions Validation
        ↓
CI PASS
        ↓
Human Review
        ↓
Merge
Enter fullscreen mode Exit fullscreen mode

Final thought

The idea is simple:

Fail -> Heal -> Validate -> Pull Request
Enter fullscreen mode Exit fullscreen mode

No direct push to main.

No blind test rewriting.

No PR without proof.

The tool does not merge code.

It prepares a small, validated fix for review.

Or shorter:

No PASS, No PR.

Top comments (0)