We fixed the eval platform we're competing on: a TypeError that crashed three benchmark pipelines

#devchallenge #bugsmash #python #opensource

Summer Bug Smash: Clear the Lineup 🐛🛹

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

The PR: AML-memory/agent-memory-leaderboard#12 — fixes #11

The situation

Last night we submitted our memory system to the Agent Memory Leaderboard — a public benchmark platform whose entries include Tencent, Mem0, and Cognee. Before competing, we did what our house rules demand: run their published evaluation pipeline ourselves instead of trusting our own imitation of it.

It crashed on the first invocation. Before making a single API call.

The bug

Three of the platform's five public benchmark pipelines share this line:

async with httpx.AsyncClient(timeout=120) as client, output.open("a", encoding="utf-8") as handle:

In a multi-item async with, every context manager must be asynchronous. Path.open() returns a plain synchronous file object — no __aenter__, no __aexit__. So Python raises, unconditionally, on every version:

TypeError: '_io.TextIOWrapper' object does not support the asynchronous context manager protocol

This isn't an environment quirk. It's a language-level guarantee — the code cannot have ever run as shipped. Both the answer() and evaluate() entry points hit it, in locomo-refined, beam, and longmemeval-s. (Two other suites don't use the pattern and are fine.)

Confirming it was really theirs (and not us)

Before pointing a finger at someone else's code, we confirmed three independent ways:

Their exact file — byte-identical to current main (md5-matched against a fresh download) — crashes on a one-line input.
A pure-language isolation — the same pattern rebuilt with a stub async context manager and zero third-party code. Same crash. Not httpx, not our environment: the pattern itself.
An adversarial review — the claim was handed to independent AI reviewers briefed to refute it. It survived.

Then we filed the issue with a minimal repro — and offered the fix.

The fix

Split the composed statement into nested managers — async client outside, sync file inside. Semantics unchanged; bodies re-indented only:

# before — raises before the first request
async with httpx.AsyncClient(timeout=120) as client, output.open("a", encoding="utf-8") as handle:
    for item in items:
        ...

# after — runs
async with httpx.AsyncClient(timeout=120) as client:
    with output.open("a", encoding="utf-8") as handle:
        for item in items:
            ...

Six blocks across three files. +67/−61, no behavior change beyond existing at runtime.

Verification (the part that counts)

All three files pass py_compile.
The exact invocation that crashed now runs end-to-end: the locomo-refined pipeline completed its answer stage and its judge stage on a real input and produced a real verdict.
The diff was eyeballed file-by-file — the largest one (beam) got extra scrutiny because its blocks span multi-line calls, and every line moved exactly one indent level and nothing else.

Why fix the platform you're competing on?

Because the whole point of a public benchmark is that everyone can run it. A leaderboard whose own evaluation pipelines crash as-shipped hurts every participant — including the ones we're trying to beat. Our house rule is simple: we don't compete against anyone we wouldn't help.

And there's a selfish honesty benefit too: having run their real pipeline (via a pristine-import driver while the fix waits upstream), our local numbers mean something. If we'd quietly patched around the crash and never told anyone, every comparison we made would have an asterisk nobody else could see.

The score from that benchmark isn't back yet. The fix is submitted either way.

Part of The Organism Files — an ongoing record of building a verification-first AI partnership from scratch.

DEV Community