Stanislav

Posted on Jul 7

Debugging Benchmark: 6 LLM Models on a Real Race Condition Bug

#llm #ai #programming #python

A comprehensive comparison of DeepSeek V4 Pro, MiMo V2.5 Pro, DeepSeek V4 Flash, MiMo V2.5, GLM 5.2, and Kimi K2.6 on a genuine production bug — including architecture analysis of each solution

TL;DR

Model	Time	Cost	Cache Hit	Bugs Found	V2 Approach	V2 = Reference?
DeepSeek V4 Flash	~7 min	$0.040	93.3%	1 (unique)	Architectural	✅ Yes
MiMo V2.5 (cheap)	~27 min	$0.067	93.3%	1	Guaranteed cleanup	❌ No
MiMo V2.5 Pro	~15 min	$0.13	95.2%	3	Three-phase	✅ Yes
DeepSeek V4 Pro	~8 min	$0.14	92.1%	1	Lock-based atomic	✅ Yes
Kimi K2.6	~53 min	$1.04	91.5%	43 req	1	I/O before state
GLM 5.2	~28 min	$1.28	96.7%	1	Collect-then-close	✅ Yes

Key findings:

Cheapest: DeepSeek V4 Flash ($0.04) — found a unique bug no other model caught
Best debugger: MiMo V2.5 Pro (3 bugs vs 1 for everyone else)
All models converged on prevention in Round 2 — but via different architectures
Round 1 revealed a gap: all models default to cleanup thinking; prevention requires guidance

Why This Benchmark?

Most LLM benchmarks test coding ability — write a function, solve a puzzle. But debugging is harder than writing code. You need to:

Understand complex, multi-file codebases
Find non-obvious root causes
Explain the mechanism clearly
Propose a correct fix with the right architecture

We tested this on a real race condition bug from httpcore — a production library used by httpx.

The Bug: httpcore #961

Repository: encode/httpcore
Issue: #961 - Race Condition After Async Cancellations Breaks Connection Pool
Fix PR: #880 - Safe async cancellations

When async tasks are cancelled during connection operations, the pool's internal state becomes inconsistent. The pool thinks connections are still in use when they're actually cancelled, leading to pool exhaustion — new requests can never acquire a connection.

Why it's hard:

Multi-file: connection_pool.py, connection.py, http2.py
Async-specific: only manifests with asyncio/trio cancellation
Non-obvious: logs show normal operation
Real-world: production issue affecting real users

Methodology

Each model received the entire httpcore project at the commit BEFORE the fix. Two rounds:

Round 1: Find the bug, explain the mechanism, propose a fix.

Round 2: Same prompt for all models — a hint about "atomic state management" to guide them from patches to prevention.

Round 1: Finding the Bug

Every model found the core issue: orphaned connections when async tasks are cancelled. But they found different things, proposed different fixes, and had different levels of depth.

DeepSeek V4 Pro

Time: ~3 minutes | Cost: part of $0.14 total

What they found: 1 race condition — orphaned connections when task is cancelled after assignment.

Mechanism explained: 5-step walkthrough with code. Correctly identified the FIFO guard in _attempt_to_acquire_connection and why orphaned status blocks the queue.

What they missed: RC2 (ConnectionNotAvailable retry leak), RC3 (connection leak during aclose), cancellation inside initial lock block.

Proposed fix:

except BaseException as exc:
    async with self._pool_lock:
        if status in self._requests:
            self._requests.remove(status)
        if status.connection is not None:
            connection = status.connection
            if connection.is_closed() and connection in self._pool:
                self._pool.remove(connection)
            else:
                status.unset_connection()
                for s in self._requests:
                    if s.connection is None:
                        await self._attempt_to_acquire_connection(s)
                        break
    raise exc

Architecture: Reactive cleanup — detect orphan, release it, reassign to next waiter.

Pros:

✅ Handles the primary race condition
✅ Tries to reassign released connections
✅ Minimal changes to existing code

Cons:

❌ await self._attempt_to_acquire_connection(s) inside the lock — creates a NEW cancellation window
❌ Only handles RC1. Misses RC2 and RC3
❌ Doesn't address root cause — state and I/O still interleaved

Verdict: Good analysis, incomplete fix. Works for the common case but introduces a new race window. The cure partially creates the disease.

MiMo V2.5 Pro

Time: ~9 minutes | Cost: part of $0.13 total

What they found: 3 distinct race conditions (no other model found more than 1):

RC1: Status leak during initial pool lock — cancellation during _close_expired_connections() or _attempt_to_acquire_connection() inside async with pool_lock leaves status in self._requests
RC2: Status leak during ConnectionNotAvailable retry — Python exception semantics: exceptions inside except blocks bypass sibling except handlers. CancelledError during retry exits the while True loop entirely.
RC3: Connection leak during cleanup — await connection.aclose() followed by self._pool.pop(idx). If cancelled during aclose(), pop() never executes.

Key insight: Explained why existing tests don't catch it — they use single-request scenarios. Also identified Python exception semantics (sibling except blocks don't catch each other's exceptions).

Proposed fix:

# Part 1: Top-level safety net
try:
    while True:
        # ... existing logic ...
except BaseException:
    async with self._pool_lock:
        if status in self._requests:
            self._requests.remove(status)
    raise

# Part 2: Defensive sweep
async def _close_expired_connections(self) -> None:
    self._pool = [c for c in self._pool if not c.is_closed()]
    # ... rest of existing logic ...

Architecture: Defense in depth — multiple layers of protection.

Pros:

✅ Catches all three race conditions — only model to do this in Round 1
✅ Defensive sweep is idempotent
✅ Top-level handler catches cancellations from initial lock block
✅ Doesn't change existing code structure significantly

Cons:

❌ Still reactive — connections get orphaned, then cleaned up
❌ _close_expired_connections() still async inside lock — RC3 still possible during sweep
❌ More code than simpler approaches

Verdict: Best Round 1 fix. Found all 3 bugs, proposed defense for all 3. But still a patch, not prevention.

DeepSeek V4 Flash

Time: ~3 minutes | Cost: part of $0.040 total

What they found: A unique bug no other model caught:

# File: httpcore/_async/connection.py, line 97
# Bug: except Exception doesn't catch CancelledError
# CancelledError is BaseException in Python 3.8+
except Exception:  # ← BUG
# Fix:
except BaseException:  # ← 1 line

Why it matters: This is a separate bug from the pool-level race. Even with a perfect pool fix, the connection-level handler would miss CancelledError. Two bugs in one codebase.

Proposed fix: 1 line — Exception → BaseException.

Architecture: Minimal fix at the right level of abstraction.

Pros:

✅ Correct and minimal — 1 line, zero side effects
✅ Different abstraction level — while others looked at pool logic, Flash looked at connection handling
✅ Catches a bug that persists even with a perfect pool fix
✅ Zero risk — strict superset of previous behavior
✅ No new code — just a wider catch clause

Cons:

❌ Doesn't fix the pool-level race — this is a different bug
❌ Doesn't explain the pool mechanism — focused on connection level
❌ Could be seen as incomplete — found a different bug, not THE bug

Verdict: Brilliant lateral thinking. Found a bug at a different abstraction level. Combined with any pool-level fix, this makes the solution stronger. Alone, it doesn't solve pool exhaustion.

MiMo V2.5 (cheap)

Time: ~4 minutes | Cost: part of $0.067 total

What they found: 1 race condition — orphaned connections. Proposed 2 variants: (A) handler + (B) try/finally.

Proposed fix:

async with self._pool_lock:
    self._requests.append(status)
    try:
        await self._close_expired_connections()
        await self._attempt_to_acquire_connection(status)
    except BaseException:
        if status.connection is None and status in self._requests:
            self._requests.remove(status)
        raise

Architecture: Exception safety — guarantee cleanup via try/finally.

Pros:

✅ Simplest change — wraps existing code
✅ Condition-aware — only removes if connection is None
✅ No new methods

Cons:

❌ I/O still inside lock (await _close_expired_connections())
❌ Only catches RC1. Misses RC2 and RC3
❌ If connection was assigned — except block doesn't handle it
❌ Reactive — orphan created, then cleaned up

Verdict: Quick fix, not production-ready. Good enough for a hotfix.

Kimi K2.6

Time: ~23 minutes | Cost: part of $1.04 total

What they found: 1 race condition. Shortest Round 1 (57 lines). Correctly explained the FIFO guard and why orphan blocks the pool.

Proposed fix: Identical to MiMo cheap — try/finally around lock body.

Architecture: Same as MiMo cheap. Same strengths, same weaknesses.

Pros:

✅ Shortest Round 1 solution
✅ Correct for the common case

Cons:

❌ Same issues as MiMo cheap

Verdict: Clean and minimal. But same architectural weakness.

GLM 5.2

Time: ~20 minutes | Cost: part of $1.28 total

What they found: 1 race condition. Key observation — asymmetric cleanup: two exception handlers do different things.

Key insight: First handler (lines 240–248) only removes status. Second handler (lines 265–268) calls response_closed() for full cleanup. This asymmetry IS the design problem.

Proposed fix:

except BaseException as exc:
    with AsyncShieldCancellation():
        if status.connection is not None:
            await self.response_closed(status)
        else:
            async with self._pool_lock:
                if status in self._requests:
                    self._requests.remove(status)
                for s in self._requests:
                    if s.connection is None:
                        acquired = await self._attempt_to_acquire_connection(s)
                        if not acquired:
                            break
                await self._close_expired_connections()
    raise exc

Architecture: Unified cleanup — make both handlers do the same work.

Pros:

✅ Identifies the architectural problem — asymmetric design, not just a missing line
✅ Re-evaluates the queue — promotes next waiting request (fixes Queue Stall)
✅ Handles both cases — connection assigned vs not

Cons:

❌ await calls inside lock — still has cancellation windows
❌ Reactive — orphans created, then cleaned up
❌ Duplicates re-evaluation loop from response_closed()

Verdict: Best problem diagnosis in Round 1. But the fix doesn't match the quality of the analysis.

Round 1 Summary

Model	Bugs Found	Fix Type	Creates New Race Windows?	Lines
DeepSeek Pro	1	Cleanup handler	Yes (await in lock)	~80
MiMo Pro	3	Defense in depth	No	~120
Flash	1 (unique)	1-line fix	No	1
MiMo cheap	1	try/finally	No	~20
Kimi	1	try/finally	No	~20
GLM	1	Unified cleanup	Yes (await in lock)	~100

Key observation: All models defaulted to cleanup thinking in Round 1. "Detect the problem, then fix it." None proposed prevention ("restructure so the problem can't happen"). This suggests prevention is a harder reasoning pattern than cleanup.

Round 2: From Patches to Prevention

With the hint about "atomic state management," every model shifted from cleanup to prevention. The core insight — zero await points inside lock body — was universal. But the implementations differed architecturally.

DeepSeek V4 Pro — Lock-based Atomic

Time: ~5 minutes

Architecture: Move connection claiming from response_closed() (Task B) to wait_for_connection() (Task A). Each task claims its own connection inside a lock.

Before: Task B assigns connection to Task A (proactive)
After:  Task A claims connection for itself (reactive, but atomic)

Key changes:

New shared event _pool_state_changed
New method _wait_and_acquire() with manual lock management
Removed proactive assignment from response_closed()

Pros:

✅ Clear responsibility — each task manages its own lifecycle
✅ Simplest mental model: wait → claim → use → release
✅ 5 cancellation scenarios analyzed (most thorough)

Cons:

❌ Manual lock management (__aenter__/__aexit__) is fragile
❌ Busy-polling under FIFO guard
❌ Micro-window at lock release (mitigated by safety net)

Comparison to reference: Different architecture (who claims) but same guarantee. Reference is simpler.

MiMo V2.5 Pro — Three-phase Separation

Time: ~6 minutes

Architecture: 3 phases. Lock body = ZERO await points.

Phase 1: CLEANUP  — identify expired/idle connections (I/O)
Phase 2: STATE    — append, sweep, acquire (sync)
Phase 3: I/O      — wait, send request (network)

Key changes:

_attempt_to_acquire_connection → sync, returns (bool, List[connection])
_close_expired_connections → sync, returns List[connection]
AsyncShieldCancellation for Phase 1 and 2

Pros:

✅ Zero await in lock — mathematically provable
✅ Collect-then-close — I/O can be batched/retried
✅ Return value is explicit

Cons:

❌ Return type change breaks existing callers
❌ More code (~350 lines)
❌ Shield overhead

Comparison to reference: Same architecture, same guarantee. More code but more formal.

DeepSeek V4 Flash — Reference Quality

Time: ~4 minutes

Architecture: Same as reference — state under lock, I/O outside. Cleaner naming.

_attempt_to_acquire_connection → sync, returns List[connection]
_close_expired_connections → _mark_expired_connections (sync)

Key changes: Same as MiMo Pro but with better naming and less code.

Pros:

✅ Cleanest implementation (~200 lines)
✅ Fastest V2 (~4 min)
✅ No AsyncShieldCancellation needed

Cons:

❌ Same return type change as MiMo Pro
❌ Less detailed scenario analysis

Comparison to reference: Effectively the reference implementation. Same architecture, same approach.

MiMo V2.5 (cheap) — Guaranteed Cleanup

Time: ~22 minutes

Architecture: try/finally around every state mutation.

try: await aclose()
finally: pop()

Pros:

✅ Minimal changes (~50 lines)
✅ Doesn't break API
✅ Easy to understand

Cons:

❌ NOT prevention — I/O still inside lock
❌ Lock held longer (aclose blocks)
❌ Reasoning loop in Round 2 (131K output tokens)

Comparison to reference: Fundamentally different approach. Reference prevents; MiMo cheap cleans up after.

Kimi K2.6 — I/O Before State

Time: ~30 minutes

Architecture: All async operations BEFORE lock. Lock body = only sync state.

Before: Lock → await close + await acquire → Unlock
After:  await close + await capacity → Lock → sync acquire → Unlock

Key changes:

New helper _ensure_pool_has_capacity() (async, before lock)
_attempt_to_acquire_connection → sync
Lock body = 2 sync operations

Pros:

✅ Shortest lock time (2 sync ops)
✅ Clean mental model
✅ Less lock contention

Cons:

❌ Duplicated logic (_ensure_pool_has_capacity ≈ _close_expired_connections)
❌ Two await points before lock (time wasted on cancellation)

Comparison to reference: Different order of operations (I/O first → state vs state → I/O). Both prevent the race. Kimi's approach has lower lock contention.

GLM 5.2 — Collect-then-Close

Time: ~8 minutes

Architecture: Same as three-phase (MiMo Pro). Collect inside lock, close outside.

Key changes: Same as MiMo Pro. Added queue re-evaluation in exception handler.

Pros:

✅ Same correctness as MiMo Pro
✅ Queue re-evaluation fixes Queue Stall
✅ Most detailed documentation (365 lines, ASCII diagrams)

Cons:

❌ Near-clone of MiMo Pro
❌ Most expensive ($1.28)

Comparison to reference: Same architecture. More documentation, same code.

Round 2 Summary

Model	Zero await in lock?	Lock duration	Code volume	Unique aspect
DeepSeek Pro	✅	Medium	~300 lines	Manual lock, shared event
MiMo Pro	✅	Short	~350 lines	Three phases, shield
Flash	✅	Short	~200 lines	Cleanest, = reference
MiMo cheap	❌	Long	~50 lines	try/finally, not prevention
Kimi	✅	Shortest	~150 lines	I/O first approach
GLM	✅	Short	~365 lines	Queue re-evaluation, most docs

Token Usage & Cost

Model	Input	Output	Cache Hit	Cost	Requests
Flash	2,504,516	55,867	93.3%	$0.040	34
MiMo cheap	1,648,424	173,624	93.9%	$0.067	20
MiMo Pro	3,356,951	53,145	95.2%	$0.13	34
DeepSeek Pro	2,431,121	43,663	92.1%	$0.14	30
Kimi	2,283,533	130,532	91.5%	$1.04	43
GLM	1,875,842	56,476	96.7%	$1.28	29

Why Flash is cheapest: Lowest token count + no reasoning loop + same cache hit rate.

Why GLM is most expensive: Pricing: $1.40/M input, $4.40/M output (vs $0.0036 for others). Even with highest cache hit rate (96.7%), per-token cost is 400x higher.

MiMo cheap reasoning loop: 131K output tokens in one request during Round 2 — model generated massive text without reaching a solution. "More tokens, same problem."

The Reference Fix (PR #880)

Tom Christie's fix: move ALL state management into non-cancellable sections using locks.

Key insight: "The async case cannot have cancellations or context-switches midway through the state management because we hold the lock."

Files changed: 9 files, +512/-379 lines

All Round 2 models converged on this approach — zero await points inside lock body. The differences are in implementation details.

Key Findings

1. All Models Found the Bug in Round 1

Every model — from $0.04 Flash to $1.28 GLM — correctly identified orphaned connections as the core issue.

2. MiMo Pro Found 3 Bugs vs 1

MiMo V2.5 Pro identified three distinct race conditions. Other models found only the primary one. This suggests deeper code analysis capability for concurrent systems.

3. Flash Found a Unique Bug at a Different Level

DeepSeek V4 Flash found except Exception not catching CancelledError at the connection level — a bug no other model noticed. Sometimes cheaper models find different things by looking at different abstraction levels.

4. All Models Defaulted to Cleanup in Round 1

Every model proposed patches (detect → clean up) rather than prevention (restructure → can't happen). Only with the Round 2 hint did they shift to prevention. This is a meaningful finding about LLM debugging: they're good at finding and patching bugs, but need guidance to redesign architectures.

5. Two Models Created New Race Windows

DeepSeek Pro and GLM put await calls inside the lock in their Round 1 fixes, creating new cancellation points. The cure partially recreated the disease.

6. Six Models, Six Architectures

Round 2 produced six different approaches to the same problem. All are valid. The differences are trade-offs:

Priority	Best Model
Minimum code changes	MiMo cheap (try/finally)
Minimum lock time	Kimi (I/O first)
Formal correctness	MiMo Pro (three-phase)
Responsibility clarity	DeepSeek Pro (lock-based)
Best overall	Flash ($0.04, reference quality)

7. Price ≠ Quality

GLM 5.2 ($1.28) and DeepSeek V4 Flash ($0.04) both found 1 bug and proposed prevention. Flash was 32x cheaper and found a unique bug. GLM's only advantage was more detailed documentation.

Verdict

Task	Best Model	Why
Budget debugging	DeepSeek V4 Flash	$0.04, unique bug, reference V2
Deep analysis	MiMo V2.5 Pro	3 bugs, systematic approach
Code generation	DeepSeek V4 Pro	Faster, cleaner architecture
Documentation	GLM 5.2	Most detailed, ASCII diagrams
Minimum lock time	Kimi K2.6	I/O first approach
Hotfix	MiMo V2.5 cheap	50 lines, doesn't break API

The surprise: DeepSeek V4 Flash — a "cheap" model — found a bug that both Pro versions missed, proposed a reference-quality fix, and did it all for $0.04.

What's Next

This is Level 1 of a multi-level benchmark. Future levels:

Level 2: FastAPI #4719 (Async Dependencies + Middleware Hang)
Level 3: CPython #129204 (Asyncio Memory Leak)
Level 4: redis-py #2641 (Async Race Condition in Queue Mechanics)

Each level increases complexity. We'll see if cheap models can keep up.

DEV Community

Debugging Benchmark: 6 LLM Models on a Real Race Condition Bug

TL;DR

Why This Benchmark?

The Bug: httpcore #961

Methodology

Round 1: Finding the Bug

DeepSeek V4 Pro

MiMo V2.5 Pro

DeepSeek V4 Flash

MiMo V2.5 (cheap)

Kimi K2.6

GLM 5.2

Round 1 Summary

Round 2: From Patches to Prevention

DeepSeek V4 Pro — Lock-based Atomic

MiMo V2.5 Pro — Three-phase Separation

DeepSeek V4 Flash — Reference Quality

MiMo V2.5 (cheap) — Guaranteed Cleanup

Kimi K2.6 — I/O Before State

GLM 5.2 — Collect-then-Close

Round 2 Summary

Token Usage & Cost

The Reference Fix (PR #880)

Key Findings

1. All Models Found the Bug in Round 1

2. MiMo Pro Found 3 Bugs vs 1

3. Flash Found a Unique Bug at a Different Level

4. All Models Defaulted to Cleanup in Round 1

5. Two Models Created New Race Windows

6. Six Models, Six Architectures

7. Price ≠ Quality

Verdict

What's Next

Top comments (0)