DEV Community

Cover image for From CI Failure to Fix in Under a Minute — QAI Agent Now Closes the Full Loop
Tomer Lihovetsky
Tomer Lihovetsky

Posted on

From CI Failure to Fix in Under a Minute — QAI Agent Now Closes the Full Loop

#github #testing #devops #playwright

QAI Agent now does more than cluster failures and score PR risk. It alerts your team in Slack, explains why tests failed with AI-powered RCA, and generates the fix — all without leaving the PR.

A few weeks ago I wrote about QAI Agent — a GitHub Action that clusters test failures and scores PR risk.

The feedback was clear: clustering is useful. But developers wanted more. Not just what broke — why it broke, and how to fix it.

So we closed the loop.

Here's what happens now when your CI fails.

Step 1 — Slack finds you

You don't check CI. CI comes to you.
The moment a high-risk PR is detected, your team gets a Slack alert:

🔴 High Risk PR #28 — Do not merge
Risk: 0.60 · 8 test failures · 8 clusters
Recommendation: investigate failures first
No polling. No tab switching. It just appears in your team channel.

Step 2 — The PR comment tells you everything

You click "View in QAI Platform." The PR already has a full breakdown:

  • 8 failures listed with exact errors
  • 8 unique failure clusters identified
  • RCA Analysis table — cause, confidence score, suggestion
  • 💡 AI fix suggestions available

The RCA table is new. For each Playwright trace, QAI detects the failure category and confidence.

Rule-based detection, no cloud required. It runs locally on the GitHub Actions runner.

Step 3 — AI generates the fix

Click "View in QAI" → open the failing test → hit "Suggest fix."

The AI explains exactly what went wrong:

The price locator matches multiple elements (paragraph and span both showing $54.95), causing a strict mode violation that prevents the visibility check.

Then generates the fix:

await expect(page.locator('span').filter({ hasText: /\$\d+/ })).toBeVisible();
Enter fullscreen mode Exit fullscreen mode

One click. Ready to apply.

The full loop

CI fails
→ Slack alert fires to your team channel
→ PR comment posts: clusters + RCA + confidence scores
→ AI fix suggestion on demand
→ Merge verdict: go or no-go

Before QAI: open CI → read logs → guess → fix → repeat.
After QAI: open Slack → click link → apply fix → merge.

Setup — still one step

Nothing changed on the setup side. Add one step after your tests run:

- name: QAI Agent
  uses: useqai/qai-agent@v1
  if: always()
  with:
    junit-path: 'test-results/results.xml'
    trace-path: 'test-results/**/*.zip'   # optional, enables RCA
Enter fullscreen mode Exit fullscreen mode

For Slack alerts and AI fix suggestions, connect the cloud platform with two more lines:

    qai-url: https://ingest.useqai.dev
    qai-api-key: ${{ secrets.QAI_API_KEY }}
Enter fullscreen mode Exit fullscreen mode

Get your free API key at useqai.dev — 30 seconds, no credit card.

GitHub Action — fully open source:

  • PR comment with risk score ✅
  • Failure clustering ✅
  • Playwright trace RCA (rule-based, runs locally) ✅
  • Block merges on high risk ✅

Cloud platform (useqai.dev):

  • AI fix suggestions (LLM-powered) ✅
  • Slack alerts for high-risk PRs ✅
  • Historical trends + flakiness tracking ✅
  • Cross-repo visibility ✅

Try it

🔧 GitHub Action: useqai/qai-agent on the Marketplace
📦 Source: github.com/useqai/qai-agent
📊 Dashboard: useqai.dev
💬 Live PR comment demo: PR #2

If you try it and hit any issues — open an issue or drop a comment here. Especially interested in non-Playwright frameworks and edge cases in JUnit parsing.

Top comments (0)