The AI testing space moves fast. Test generators, self-healing frameworks, MCP servers, LLM evaluators, visual AI platforms, new tools every month. Most of them claim to "use AI" without saying what that actually means.
Existing awesome lists either don't cover AI testing or mix it into broader test automation directories. The result: hard to compare options, hard to find open source alternatives, hard to see what's mature versus experimental.
awesome-ai-testing is a new curated list focused on this gap. It currently has around 100 tools across 20 categories, with each entry tagged as open source, commercial, or open core. I'm sharing it early, not polished, to get feedback from people who actually run tests for a living.
What's in it
Categories are organized by what people actually search for, not academic taxonomies:
- Test Generation: Qodo-Cover, EvoMaster, EvoSuite, Diffblue, plus AI editors like Cursor and Claude Code that can write Playwright, Cypress, and Selenium tests with the right prompting.
- MCP-Based Testing: Playwright MCP, Chrome DevTools MCP, and community variants. This is the area moving fastest right now.
- Self-Healing Test Frameworks: Healenium for open source, Testim and Functionize on the commercial side.
- Natural Language Test Authoring: Magnitude, Passmark, Shortest, Auto Playwright, Midscene.js, all open source.
- LLM-as-Judge Evaluation: Promptfoo, DeepEval, Ragas, Inspect AI, TruLens, Arize Phoenix.
- Mock and Service Virtualization: Mountebank, WireMock, Mockoon, MSW, Pact, Hoverfly. Useful when mocking LLM APIs in tests.
- Visual AI, Mobile AI, Performance, Accessibility, API testing: Each with its own section.
Each entry has a badge:
- 🆓 Open source
- 💰 Commercial
- 🆓💰 Open core (free tier or open source with paid features)
Open source options are listed first in each category. The commercial space is well-promoted already; open source needs visibility.
What's excluded
To keep the list useful:
- Tools abandoned for 2+ years
- Marketing pages without a real product
- Generic test runners with weak AI claims (no Selenium with bolted-on plugins)
- Closed source tools without a free trial or transparent pricing
Honest disclaimers
The list is new. A few caveats:
- Some categories are denser than others. AI Code Review didn't make the cut yet because it doesn't clearly belong in a testing-focused list.
- This is a curated list, not a benchmark. Tools are included based on quality signals (active maintenance, real users, transparent pricing), not personal testing of every entry.
- Categorization will shift as the list grows. Promptfoo sits in two categories because it genuinely does both.
Three workflows the list supports
- "I need to test my AI app" → LLM and AI System Testing + Mock and Service Virtualization
- "I want AI to write my tests" → Test Generation + MCP-Based Testing + Natural Language Test Authoring
- "My existing test suite is brittle" → Self-Healing Test Frameworks + Test Analytics and Triage
What would help right now
If you maintain or use an AI testing tool that's missing, open a PR. The contribution guide is two screens long.
If you spot something miscategorized or inaccurate, open an issue.
If you tried a tool from the list and it was great or terrible, drop it in the comments here. That kind of signal is exactly what turns a list into a useful list.
Top comments (0)