"It writes code" is no longer impressive. The question in 2026 is whether it writes code you'd actually merge into a production codebase without spending more time reviewing and fixing than you'd spend writing it yourself.
We gave three autonomous coding agents the same production task and evaluated the output on the only metric that matters: would we ship this?
The task: add rate limiting to an existing Express.js API with Redis backing, configurable per-endpoint limits, proper HTTP response headers, graceful degradation when Redis is unavailable, and comprehensive tests.
This isn't a toy benchmark. It's a feature request that lands in a real team's sprint every quarter.
The agents: Devin (via Devin Desktop, Cognition's rebranded Windsurf platform at $20/month), Claude Code (on Max plan at $100/month with Opus 4.8), and Cursor Agent mode (Pro at $20/month with Composer 2.5).
Same repo. Same requirements document. Same evaluation rubric. No hand-holding during execution, we gave each agent the task and let it work.
The task specification
We provided each agent with identical requirements:
Add rate limiting middleware to the Express API.
Requirements:
- Redis-backed sliding window algorithm
- Configurable limits per endpoint via route decorator
- Standard HTTP headers: X-RateLimit-Limit,
X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After
- Graceful fallback when Redis is unavailable
(allow requests with warning log, don't block)
- Tests covering: normal operation, rate exceeded,
Redis failure, concurrent requests, config validation
- Follow existing project patterns for middleware,
error handling, and test structure
The existing codebase had established patterns for middleware registration, error response formatting, and test organisation. We wanted to see which agents respected those patterns versus imposing their own conventions.
Devin's output
Devin worked autonomously for approximately 22 minutes. It explored the codebase, identified the existing middleware patterns, created the implementation files, and ran the test suite to verify its work.
The implementation used a sliding window algorithm with Redis sorted sets, a solid technical choice. The per-endpoint configuration used a decorator pattern that was consistent with the codebase's existing middleware approach.
What worked well: the Redis interaction code was clean, the sorted set implementation was correct, and the HTTP headers were properly formatted. The decorator pattern matched the existing codebase conventions. Devin clearly analysed the existing middleware structure before writing its own.
What didn't ship: the graceful degradation on Redis failure had a critical issue. When Redis was unavailable, the middleware threw an unhandled promise rejection instead of falling through to the allow-with-warning behaviour the requirements specified. The error handling path was partially implemented, there was a try-catch, but the catch block called a logging function that itself tried to write to Redis, creating a recursive failure.
The tests covered the happy path and the rate-exceeded path well. The Redis failure test existed but tested the wrong behaviour, it asserted that requests were blocked when Redis was down, contradicting the requirement for graceful degradation. Two tests had hardcoded timing assertions that would produce flaky failures on slower CI runners.
Code quality score: 3.5/5. Time-to-PR-ready: 45 minutes after fixing the Redis fallback, updating the test assertions, and removing the timing dependencies.
Claude Code's output
Claude Code completed the task in approximately 18 minutes. It loaded the full project context, asked no clarifying questions (the specification was detailed enough), and produced the implementation in a single pass.
The sliding window implementation used Redis MULTI/EXEC transactions for atomicity, a more robust approach than Devin's sorted set method for high-concurrency scenarios. The per-endpoint configuration used the existing decorator pattern.
What worked well: the Redis failure handling was the best of the three. A circuit breaker pattern, after three consecutive Redis failures, the middleware switches to allow-all mode for 30 seconds before retrying. This is more sophisticated than the requirements asked for, and it's the correct production pattern because it prevents cascading retry storms when Redis is genuinely down. The tests included a concurrent request test using Promise.all that actually exercised the race condition path.
What needed review: the circuit breaker added complexity that the requirements didn't specify. We debated whether to keep it. We kept it, it's the right engineering decision even though it wasn't requested. One test used a mock Redis client that didn't perfectly replicate the real client's error throwing behaviour, causing a false positive on the error path test.
Code quality score: 4.5/5. Time-to-PR-ready: 15 minutes, fixing the mock client issue and adding a comment explaining the circuit breaker design decision.
Cursor Agent's output
Cursor's Composer agent mode completed the task in approximately 14 minutes, the fastest of the three. The inline IDE experience meant we could watch the implementation happen in real time and the iteration cycle was the most fluid.
The sliding window used a Redis INCR with TTL approach, simpler than the other two implementations, which is both a strength (less code, easier to understand) and a limitation (less precise window boundaries at high concurrency).
What worked well: the code was the most readable of the three implementations. Clear variable names, good comments, consistent with the existing codebase style. The test structure followed the project's existing patterns exactly. The HTTP headers were correctly implemented.
What didn't ship: the rate limiting algorithm had a subtle issue at window boundaries. Because it used INCR with a fixed TTL rather than a true sliding window, a burst of requests at the end of one window and the beginning of the next could exceed the intended limit. At low traffic this doesn't matter. At high traffic with strict rate requirements, it's a bug.
The Redis failure handling worked but was simplistic, a try-catch that logged a warning and allowed the request through. No circuit breaker, no backoff, no tracking of Redis health. Functional but not production-hardened.
The tests were well-structured and covered the main paths but didn't include a concurrent request test. The boundary condition issue wouldn't have been caught by the generated tests.
Code quality score: 3.5/5. Time-to-PR-ready: 35 minutes after replacing the INCR approach with a proper sliding window and adding a concurrent test.
The comparison matrix
What this actually tells us
Claude Code produced the only output we would have merged with minimal changes. The circuit breaker pattern, the atomic transactions, and the concurrent test demonstrated understanding of production concerns that the other two agents didn't exhibit.
Cursor was fastest to initial output but the window boundary issue meant more human review time. The code was the most readable and the most consistent with existing style, which matters for long-term maintainability.
Devin's recursive Redis failure is the kind of bug that would have caused a production incident if it shipped. It's also the kind of bug that's hard to catch in review because the error handling code looks correct at first glance, you have to trace the execution path through the logging function to see that it re-enters the failing Redis connection.
The pattern across all three: the "happy path" implementation is solid from every agent. The differentiation is in error handling, edge cases, and production hardening. These are the dimensions where engineering experience matters most, and they're the dimensions where the gap between agents is widest.
Autonomous coding agents are advancing fast, but "shipping production code" is a much higher bar than passing SWE-bench. The benchmarks test whether the agent can solve a defined problem. Production readiness tests whether it handles the problems that aren't defined, the edge cases, the failure modes, the concurrent access patterns that the requirements didn't specify because experienced engineers handle them implicitly.
We wrote the full CTO comparison with ten tools for teams evaluating autonomous coding agents at enterprise scale, covering not just output quality but the procurement criteria that matter for teams of 20-200 engineers.
Published by Dextra Labs, AI Consulting and Enterprise Agent Development
Top comments (0)