Memdoor: How a Team of 5 Qwen‑3.8‑27B Models Beats Fable‑5 on LiveCodeBench Hard

1. What was released / announced

The Memdoor project (github.com/guregodevo/memdoor) introduced a new workflow where five copies of the Qwen‑3.8‑27B model cooperate to solve programming problems. In the LiveCodeBench Hard benchmark, this “team‑solve” strategy outperformed Meta’s Fable‑5, a state‑of‑the‑art code‑generation model. The repo ships a ready‑to‑run example that spins up the five agents, shares a mutable memory store, and aggregates their answers.

2. Why it matters

a) Scaling LLM reasoning without bigger models

Most of us have been chasing larger context windows or more parameters to get better code generation. Memdoor shows that orchestrating multiple smaller models can be more effective than a single monolithic model. For engineers, this opens a path to higher quality results without the massive GPU budget a 70B+ model would demand.

b) Real‑world collaborative coding

Think of code review, pair programming, or a “rubber‑duck” that can ask follow‑up questions. Memdoor’s architecture mirrors that workflow: each model writes a snippet, stores it, reads what the others produced, and refines the solution. That pattern is directly reusable for internal tooling, CI‑assistants, or even educational bots.

c) Open‑source reproducibility

Everything lives in a public repo, with Dockerfiles and a simple run.sh. No vendor lock‑in, no API key throttling, and you can swap Qwen‑3.8‑27B for any other model that respects the OpenAI‑compatible chat API. This makes the approach future‑proof as new models appear.

3. How to use it

Below is the minimal path to get the Memdoor team running on a single machine with Docker. I tested it on an AWS g5.12xlarge (8×A100) but you can scale down to a single 40GB GPU for experimentation – just reduce the parallelism.

Step 1 – Clone and build the image

# Clone the repo
git clone https://github.com/guregodevo/memdoor.git
cd memdoor

# Build the Docker image (includes Qwen‑3.8‑27B weights via huggingface hub)
docker build -t memdoor:latest .

Tip: The Dockerfile pulls the model from modelscope – make sure you have a HuggingFace token set as HF_TOKEN in your environment.

Step 2 – Spin up the memory store (Redis) and the agents

# Start Redis (used as the shared mutable memory)
docker run -d --name memdoor-redis -p 6379:6379 redis:7-alpine

# Launch the five agents (each in its own container for isolation)
for i in {0..4}; do
  docker run -d --name memdoor-agent-$i \
    --link memdoor-redis:redis \
    -e AGENT_ID=$i \
    -e MODEL_NAME="Qwen/Qwen3.8-27B" \
    memdoor:latest
done

Each container runs python -m memdoor.agent which:

1. Pulls the model.
2. Registers itself in Redis under agents.
3. Listens for a task key, generates a partial solution, writes back to memory:<task_id>.

Step 3 – Submit a LiveCodeBench problem

import redis, json, time

r = redis.Redis(host='localhost', port=6379)

# Example problem (taken from LiveCodeBench Hard)
problem = {
    "id": "lcbench_hard_001",
    "prompt": "Write a Python function `is_palindrome(s: str) -> bool` that returns True for palindrome strings, handling Unicode correctly."
}

# Push the task to Redis
r.set(f"task:{problem['id']}", json.dumps(problem))

# Give agents ~30 seconds to collaborate
time.sleep(30)

# Retrieve the aggregated answer
answer = r.get(f"memory:{problem['id']}")
print(answer.decode())

The agents will read the same task key, generate their own snippets, and iteratively refine each other’s output. The final aggregated answer is stored under memory:<task_id>.

Step 4 – Replace Qwen with your favourite model (optional)

If you have a local Llama‑3.1‑8B or an OpenAI‑compatible endpoint, just change the MODEL_NAME env var:

export MODEL_NAME="openai/gpt-4o-mini"
# restart containers with the new env var

The rest of the workflow stays the same because the agent code talks to the model via the ChatCompletion interface.

4. My take

Having built AI pipelines for Griffin AI Tech for the past three years, I’m constantly weighing cost vs. capability. The Memdoor experiment hits a sweet spot: instead of allocating a single 80‑GPU node to a 70B model, you can spin up a fleet of 27B models that collectively outperform a larger competitor on a hard coding benchmark.

From an infrastructure perspective, the pattern aligns with the micro‑service mindset we champion in our MLOps platform:

• Stateless agents – each container can be autoscaled behind a Kubernetes Deployment.
• Shared Redis memory – a cheap, low‑latency store that can be swapped for a more durable backend (e.g., DynamoDB) if you need persistence.
• Observability – instrument each agent with OpenTelemetry; you’ll instantly see how many iterations each model contributes and where the bottleneck lies.

In practice, I see two immediate use‑cases for our customers:

1. Automated code review bots – run the team‑solve on a PR diff, let the agents suggest improvements, and surface a consensus patch.
2. Educational tutors – a student asks a programming question; the agents collaboratively generate a step‑by‑step answer, and the memory store can be queried for the reasoning trace, which is a gold‑mine for learning analytics.

That said, there are trade‑offs. Running five 27B models still eats a lot of VRAM, and the coordination overhead adds latency (typically 15‑30 s per problem). For latency‑critical services, a single, well‑tuned model may still be preferable. Also, the current implementation assumes synchronous collaboration – an area ripe for research, especially with asynchronous message passing or reinforcement‑learning based role assignment.

Overall, Memdoor is a practical proof‑of‑concept that collaborative LLMs can beat larger singular models on complex code tasks. It gives us a concrete blueprint to build more resilient, cost‑effective AI services that scale horizontally rather than vertically. If you’re tinkering with AI‑augmented development tools, I recommend cloning the repo, running the workflow on a modest GPU, and then iterating on the orchestration layer to fit your own product needs.

---

Happy hacking!