DEV Community

mage0535
mage0535

Posted on

I Let an AI Write My Tests for 30 Days: Coverage Went 38% to 71%

#ai #testing #opensource #devtools

Here's the number that surprised me: 30 days, zero tests written by hand, coverage from 38% to 71%.

I handed test-writing to an open-source AI agent (search the-agent on GitHub) and let it generate, run, and maintain my tests from natural-language descriptions. This is the full account — the workflow, the configs, the pitfalls, and the honest trade-offs.

Why I Tried This

Last month I broke 35 tests by changing one function signature. Fixing them took until lunch. The pain wasn't writing tests — it was maintaining them: normal inputs, edge cases, error branches, and the worst part — that false confidence of "all green" when critical paths were never covered.

I saw the-agent trending on GitHub (a prompt-based test automation tool that uses AI agents to generate, run, and maintain tests) and decided to run a real 30-day experiment.

The Setup 

# Install
npm install -g the-agent

# Init project config
the-agent init --project ./my-app --language typescript
Enter fullscreen mode Exit fullscreen mode

Generated config:

{
  "project": "my-app",
  "language": "typescript",
  "testFramework": "vitest",
  "coverageTarget": 70,
  "asyncDetection": true,
  "compatibilityNotes": "legacy endpoints keep original format"
}
Enter fullscreen mode Exit fullscreen mode

Key flags:

  • coverageTarget — CI gate threshold
  • asyncDetection — catches missing async waits (critical, see pitfalls)
  • compatibilityNotes — tells the agent about legacy constraints

First Real Test

I asked it to test an order module's calculateTotal:

the-agent test --describe "calculateTotal receives product array, computes total, supports coupon discount, 100 off 20"
Enter fullscreen mode Exit fullscreen mode

It generated cases covering: normal totals, empty arrays, discount thresholds, coupon stacking, and negative-price exceptions. First run, I was genuinely impressed.

The CI Integration

The trick is patch-style generation, not full-suite generation:

name: AI Test Agent
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  ai-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: |
          the-agent test --diff origin/main...HEAD \
            --config ./the-agent.config.json \
            --report ./ai-test-report.json
      - uses: actions/upload-artifact@v4
        with:
          name: ai-test-report
          path: ai-test-report.json
Enter fullscreen mode Exit fullscreen mode

Only tests changed files. ~5-8 minutes per PR, and every PR gets an AI-generated coverage patch plus a coverage gate.

Results After 30 Days

  • Coverage: 38% → 71%
  • New-feature test time: from "half a day" to ~40 minutes (including review)
  • Regressions caught by CI: 3 (would have shipped)
  • Redundant test cases deleted: ~40%

The Pitfalls (the honest part)

1. Bad description = wrong tests. I forgot to mention an async confirmation step; it generated all-sync cases that passed falsely. Fix: explicitly state async in the description.

2. Legacy compatibility. It writes "best-practice" tests that fail against old formats. Fix: declare constraints in compatibilityNotes.

3. Async gaps. Timers, callbacks, external calls — occasionally missed timing. asyncDetection: true helps but doesn't fix everything. Review async cases manually.

4. It won't think for you. It guarantees tests run, not that your business logic is right. Wrong description → confidently wrong tests. My rule: AI generates, I review semantics.

Who Should Use It

  • Devs fighting test maintenance — huge time saver
  • Test engineers — great for exploratory coverage, but own the business logic
  • Anyone expecting "install and forget" — skip it; it needs tuning, and the payoff comes after tuning

What's Next

Hard coverage gates (block merge below 60%), Python/Go support, and documenting the prompt templates I've collected. AI-assisted testing is becoming mainstream — start now and you'll have a workflow ready when the tooling matures.

The tool is free and open source. Search "the-agent" on GitHub. Save this for when you wire it into your CI.

Top comments (1)

Collapse
 
alexshev profile image
Alex Shev •

The coverage jump is interesting, but the more useful metric is whether the tests caught behavior you actually cared about. AI-generated tests can improve breadth quickly, but they still need mutation checks, failure review, and a way to prevent high coverage from becoming false confidence.