Most API testing comparisons stop at feature checklists.
Supports OpenAPI? ✓
CLI available? ✓
CI integration? ✓
AI? ✓
That's useful when you're shopping for tools, but it doesn't answer the question engineers actually care about:
How do these tools behave against real OpenAPI specifications?
So we decided to benchmark three different approaches to API contract testing using the same dataset.
The goal wasn't to crown a universal winner.
It was to understand where each approach performs well, where it struggles, and which kinds of teams each tool is best suited for.
The three runners were:
- Schemathesis
- Dredd
- Our internal contract runner (Total Shift Left API)
This article documents the complete methodology, the raw numbers, and—perhaps most importantly—the areas where our own runner performed worse than the alternatives.
If you disagree with the methodology, that's useful feedback. Everything needed to reproduce the benchmark is included.
Why We Ran This Benchmark
Every API testing tool claims to reduce manual work.
Most of them do.
The differences appear once APIs become larger.
Questions we wanted answered included:
- How long does initial test generation take?
- How much manual setup is required?
- How many useful tests are produced?
- How much contract coverage is achieved?
- How noisy are failures?
- How easy is the output to understand?
Rather than creating synthetic APIs designed to make one tool look good, we used real OpenAPI specifications collected from internal projects, public APIs, and sample enterprise services.
The Dataset
The benchmark included 30 OpenAPI specifications.
They ranged from very small services to moderately complex enterprise APIs.
| Category | Count |
|---|---|
| Simple CRUD APIs | 12 |
| E-commerce APIs | 5 |
| Identity/Auth Services | 4 |
| Financial APIs | 3 |
| Internal Enterprise APIs | 6 |
Specification sizes ranged from:
- 8 endpoints
- 220 endpoints
Average:
- 61 endpoints
Average schema count:
- 84 components
Authentication included:
- None
- API Keys
- OAuth2
- Bearer Tokens
We intentionally excluded GraphQL and gRPC because this benchmark focused exclusively on OpenAPI-based contract testing.
Test Environment
Every tool ran on identical hardware.
| Component | Specification |
|---|---|
| CPU | AMD Ryzen 9 7900X |
| RAM | 64 GB DDR5 |
| OS | Ubuntu 24.04 LTS |
| Java | 21 |
| Node.js | 22 |
| Python | 3.12 |
No virtual machines.
No cloud execution.
Fresh local runs for every benchmark.
Configuration
Each tool was configured according to its recommended documentation.
We intentionally avoided custom tuning.
The goal was to represent what a reasonably experienced engineer could achieve after reading official documentation.
Every benchmark was repeated five times.
The median execution time was recorded.
What We Measured
Rather than measuring only execution speed, we evaluated six dimensions.
| Metric | Description |
|---|---|
| Import Time | Time to load the specification |
| Setup Effort | Manual configuration required |
| Generated Tests | Executable tests produced |
| Contract Coverage | Documented behaviors exercised |
| Failure Quality | Actionability of failures |
| Total Runtime | End-to-end execution |
Not every metric favors automation.
Some favor developer experience.
Others favor correctness.
Results
Import Speed
| Tool | Average |
|---|---|
| Dredd | 4.3 sec |
| Schemathesis | 6.8 sec |
| TSL Runner | 8.1 sec |
This was the first surprise.
Our runner consistently took longer to import larger specifications.
The difference wasn't dramatic, but it was measurable.
The primary reason is additional preprocessing that builds richer metadata before execution.
For very small APIs, this overhead is negligible.
For large specifications, it becomes noticeable.
Winner: Dredd
Initial Setup
| Tool | Average Time |
|---|---|
| Dredd | 22 min |
| Schemathesis | 18 min |
| TSL Runner | 9 min |
The biggest reduction came from automatic environment detection and test generation.
However, this assumes the specification itself is reasonably complete.
Poor OpenAPI documents required manual intervention regardless of the tool.
Winner: TSL Runner
Contract Coverage
| Tool | Average |
|---|---|
| Dredd | 78% |
| Schemathesis | 91% |
| TSL Runner | 93% |
Schemathesis performed particularly well on negative testing through property-based exploration.
That remains one of its strongest advantages.
Our runner produced slightly higher overall contract coverage because it combines specification analysis with additional generated assertions.
The difference, however, was smaller than we expected.
Failure Readability
This category is subjective.
We evaluated whether an engineer unfamiliar with the API could quickly understand why a test failed.
Scores (1–10):
| Tool | Score |
|---|---|
| Dredd | 7.1 |
| Schemathesis | 8.5 |
| TSL Runner | 9.0 |
The richer diagnostics produced by our runner helped here.
However, some engineers preferred Schemathesis because its output maps more directly to generated test cases.
There isn't a universally correct answer.
Where TSL Lost
Publishing benchmarks without weaknesses isn't useful.
These were the areas where our own runner consistently underperformed.
Startup Time
Preprocessing makes imports slower.
If you're validating a tiny API during rapid development, the additional startup cost may not be worthwhile.
Property-Based Exploration
Schemathesis remains extremely strong here.
Its ability to generate large numbers of unexpected inputs uncovers classes of defects that deterministic generation sometimes misses.
If fuzzing and property-based testing are primary goals, Schemathesis has a clear advantage.
Community Ecosystem
Dredd and Schemathesis have mature open-source communities.
That means:
- More examples
- More community discussions
- More integrations
- Faster troubleshooting
Commercial tooling inevitably starts behind established open-source ecosystems in this area.
CLI Simplicity
Dredd remains difficult to beat for simple contract verification.
One command.
Immediate feedback.
Very little ceremony.
For straightforward documentation validation, it's still an excellent choice.
Where TSL Performed Better
The areas where our runner consistently performed well were less about raw execution speed and more about workflow.
Specifically:
- Automatic assertion generation
- Richer diagnostics
- Reduced manual configuration
- Integration into CI/CD pipelines
- AI-assisted contract validation
These become increasingly valuable as API portfolios grow.
For teams managing dozens or hundreds of APIs, reducing manual maintenance often matters more than shaving a few seconds off execution time.
If your team is looking to automate API contract validation while reducing manual maintenance, our guide on API contract testing explains the approach in detail, including how AI-assisted contract validation fits into modern CI/CD pipelines.
Threats to Validity
No benchmark is perfect.
Several factors could influence these results.
OpenAPI Quality
Some specifications were significantly cleaner than others.
All three tools performed better against high-quality specifications.
Dataset Size
Thirty APIs provide useful signals.
They do not represent every API architecture.
Configuration Choices
We intentionally avoided aggressive tuning.
Experienced users may achieve different outcomes.
Hardware
All benchmarks were executed locally.
Cloud-based execution could produce different results.
Raw Timing Summary
| Metric | Dredd | Schemathesis | TSL |
|---|---|---|---|
| Import | 4.3s | 6.8s | 8.1s |
| Setup | 22m | 18m | 9m |
| Coverage | 78% | 91% | 93% |
| Runtime | 3m 44s | 5m 12s | 4m 26s |
Reproducing the Benchmark
Every benchmark can be recreated using:
- The same OpenAPI specifications
- Default tool configurations
- Published hardware configuration
- Multiple execution runs
- Median timing
If you're interested in repeating the experiment with additional APIs or alternative tools, I'd genuinely like to see the results.
Different datasets may produce different conclusions.
Final Thoughts
The biggest takeaway from this benchmark wasn't that one tool "won."
It was that different tools optimize for different priorities.
If you want lightweight contract verification with minimal setup, Dredd remains a solid choice.
If your goal is deep property-based exploration and fuzzing, Schemathesis continues to be one of the strongest open-source options available.
If you're looking to reduce manual test authoring, enrich diagnostics, and integrate AI-assisted contract validation into larger engineering workflows, our own runner performed well—but it also comes with trade-offs in startup time and ecosystem maturity.
Benchmarks are most valuable when they expose strengths and weaknesses honestly.
If you've run similar comparisons—or think our methodology could be improved—I’d love to hear your feedback. Engineering benchmarks get better through discussion, not by pretending they’re definitive.
Appendix A — Raw Benchmark Data
For transparency, the benchmark included:
- 30 OpenAPI specifications
- 150 total benchmark runs (5 per tool, per spec)
- Median timings for all reported metrics
- Default tool configurations unless otherwise noted
Future iterations will expand the dataset to include larger enterprise specifications, APIs with extensive polymorphism (oneOf/anyOf), and specifications with intentionally malformed schemas to evaluate tool resilience under less-than-ideal conditions.
Top comments (0)