DEV Community

AI and QA Leaders profile image Let's Automate 🛡️
Let's Automate 🛡️ for AI and QA Leaders

Posted on • Originally published at Medium on

I Built an AI-Powered Cypress Framework That Analyses Test Failures for Free

#llm #langchain #ai #testautomation

Cypress test debugging is painful. This free AI-powered framework analyses failures instantly and tells you exactly what went wrong.


AI-Powered Cypress Framework That Analyses Test Failures for Free

Ever stared at a cryptic Cypress error message wondering what broke? 😩 We’ve all been there. That’s why I built something that changed my debugging workflow forever.

Introducing v2.1 of my Cypress Natural Language Test Framework — now featuring 🔍 AI Failure Analysis that costs you absolutely nothing.

😤 The Problem Every QA Engineer Knows

Picture this: Your CI pipeline fails and error be like:

CypressError: Timed out retrying after 4000ms: Expected to find element: '#submit-btn', but never found it.
Enter fullscreen mode Exit fullscreen mode

Now you’re left guessing:

🤔 Did the selector change?

⏳ Is the page loading too slowly?

✏️ Did someone rename the button?

⚡ Is it a timing issue?

You spend the next hour digging through logs, comparing commits, and testing locally. Sound familiar?

💡 The Solution: AI That Debugs For You

With v2.1, debugging becomes a one-liner:

python qa_automation.py --analyze "CypressError: Timed out retrying: Expected to find element: #submit-btn"
Enter fullscreen mode Exit fullscreen mode

Output:

🔍 Analyzing...
REASON: Element #submit-btn not found - selector likely changed during recent UI update
FIX: Use cy.get('[data-testid="submit"]') or add cy.wait() before the click action
Enter fullscreen mode Exit fullscreen mode

✅ Two lines. Problem identified. Solution provided. Done.

🏗️ System Architecture

Here’s how the entire framework fits together:

⚙️ How It Works Under The Hood

The implementation is surprisingly simple. Here’s the core function:

def analyze_failure(log: str) -> str:
    response = requests.post(
        url="https://openrouter.ai/api/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {os.getenv('OPENROUTER_API_KEY')}",
            "Content-Type": "application/json"
        },
        json={
            "model": "deepseek/deepseek-r1-0528:free",
            "messages": [{"role": "user", "content": f"Analyze this Cypress test failure. Reply ONLY:\nREASON: (one line)\nFIX: (one line)\n\n{log}"}],
            "max_tokens": 150
        }
    )
    return response.json()["choices"][0]["message"]["content"]
Enter fullscreen mode Exit fullscreen mode

That’s it. About 15 lines of code that leverage OpenRouter’s free tier with DeepSeek R1. 🆓

🛠️ Three Ways To Use It

1️⃣ Direct from command line:

python qa_automation.py --analyze "Your error message here"
Enter fullscreen mode Exit fullscreen mode

2️⃣ From a log file:

python qa_automation.py --analyze -f cypress-output.log
Enter fullscreen mode Exit fullscreen mode

3️⃣ Piped from another command:

cat error.log | python qa_automation.py --analyze
Enter fullscreen mode Exit fullscreen mode

🔄 CI/CD Integration

The real power comes when you integrate this into your pipeline. Here’s how the updated GitHub Actions workflow looks:


- name: Run Cypress tests
  id: tests
  continue-on-error: true
  run: |
    npx cypress run --spec "cypress/e2e/generated/**/*.cy.js" 2>&1 | tee test-output.log- name: AI Failure Analysis
  if: steps.tests.outcome == 'failure'
  env:
    OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
  run: |
    echo "Analyzing failures with AI..."
    python qa_automation.py --analyze -f test-output.log
Enter fullscreen mode Exit fullscreen mode

When tests fail, your CI logs now include actionable insights instead of just error dumps. 📋

🚀 Setting It Up

Step 1: Get your free API key from openrouter.ai 🔑

Step 2: Add to your .env:

OPENROUTER_API_KEY=your_key_here
Enter fullscreen mode Exit fullscreen mode

Step 3: Add requests to requirements.txt (if not already there) 📦

Step 4: Start analyzing 🎉

That’s the entire setup. No complex configurations. No paid subscriptions.

🖥️ Local Development Flow

For local development, the flow is just as smooth:

📦 What’s In v2.1

Here’s everything new in this release:

Feature Description

🔍 AI Failure Analyzer Instant debugging with free LLM

🌐 OpenRouter Integration Uses DeepSeek R1 at zero cost

💻 CLI Flag Simple --analyze command

📁 File Input Analyze entire log files with -f

⚙️ CI/CD Ready Updated GitHub Actions workflow

Combined with v2.0 features:

🤖 Natural language test generation

🔄 cy.prompt() self-healing tests

📊 LangGraph workflow orchestration

📚 Vector store documentation context

🌍 Real World Example

Old approach: Manual Investigation 😓

New approach:

python qa_automation.py --analyze -f nightly-run.log
Enter fullscreen mode Exit fullscreen mode

Result:

REASON: Login button selector changed from #login-btn to .auth-button
FIX: Update selector to cy.get('.auth-button') or use data-testid

REASON: API response timeout - server took 6s, test timeout was 4s
FIX: Increase timeout with cy.request({timeout: 10000}) or add retry logic

REASON: Element detached from DOM after React re-render
FIX: Add cy.wait() after state change or use {force: true} option
Enter fullscreen mode Exit fullscreen mode

🔗 Try It Yourself

The framework is open source and available now:

🔗 GitHub: github.com/aiqualitylab/cypress-natural-language-tests

Clone it, set up your API keys, and start generating tests and debugging failures with AI.

💭 Final Thoughts

AI shouldn’t just generate code. It should help maintain it too. This failure analyzer is my attempt at closing that loop — from requirements to tests to debugging, all AI-assisted.

The best part? It’s completely free to use. 🆓

Give it a try and let me know how much time it saves you! 💬

If this helped you, consider ⭐ starring the repo. It helps others discover it.

Top comments (0)