Building a Multi Agent Townhall simulation from scratch

Sean Graham — Tue, 21 Jul 2026 14:13:05 +0000

I wanted to understand how multi-agent systems actually work — not from a tutorial, but by wiring the pieces together myself. So I built a medieval townhall simulation where three AI characters (a mayor with a French accent, a judge, and a villager) debate topics in a medieval townhall. They can all hear each other in real time and can all contribute to the conversation.

This project covers how to implement asynchronous agent communication, how to pass a shifting context window, how to store and retrieve memory using vector search (cosine similarity search) and finally lessons I learned on how to get the best outputs out of your agents based on the Multi-Agent Debate (MAD) framework.

Find the repository here.

The Architecture

How do we get asynchronous communication?

TLDR; Redis Streams

Getting the agents to communicate directly to each other via APIs is tedious and not scalable. An agent would need to send their comment to every single other agent, requiring them to already know every other entity and how to interface with them.

To tackle this we need a message bus. A central source of truth that any authorised entity can push data to and read data from at any time. For our use case I chose to implement Redis Streams.

Redis Streams is a built-in data structure in Redis that acts as an append-only log. Each entry has an auto-generated timestamp ID and a set of key-value fields. What makes it useful for multi-agent work is consumer groups: multiple independent readers can each track their own position in the stream, so the same message gets delivered to every group without duplication.

Crucially Redis Streams is easy to host locally.

How do we get the context window?

Once your Redis Streams setup is done and you have decided on a topic to insert data to, you can implement a context window. This is simply a sliding window of the last n messages that you can dynamically feed into the prompt of your agent.

The Memory Pipeline: Embeddings → Faiss → SQLite

A context window is great but it does not allow for more meaningful agent development. Instead of just reacting to the past n messages, the agents can instead draw on past experiences / accumulated knowledge to enrich their answers.

This requires two things: memory insertion and memory retrieval.

Step 1: The ChronicleAgent Creates Summaries

A dedicated ChronicleAgent runs alongside the debate agents. It polls the Redis stream and, every 10 messages, asks Mistral to produce a concise summary:

prompt = f"""
You are an AI agent tasked with summarising the following events
from a townhall meeting.
Create an objective, concise summary...
Keep the summary under {SUMMARY_SOFT_LIMIT_WORDS} words.
Here are the events to summarise:
{context}
"""

This is the rolling memory of the town — compressed, searchable, persistent. By dedicating the role of memory to one agent we reduce duplicate memories / personal biases. We also reduce complexity and expense of query execution.

If you had different memory stores for different agents you would then likely wish to abandon this approach and instead make every agent responsible for creating its own memories.

Step 2: Summaries Get Embedded and Stored

Each summary is turned into a 768-dimensional vector using sentence-transformers (all-mpnet-base-v2), running locally on CPU. The summary text + embedding go into SQLite (the source of truth), and the vector goes into a Faiss index for fast similarity search:

embedding: np.ndarray = generate_embedding(summary)
embedding_bytes: bytes = embedding.tobytes()

# SQLite: source of truth
with SQLiteSummaryStore() as store:
    summary_id = store.insert_summary(summary_record)

# Faiss: rebuildable search cache
with FaissVectorStore() as vector_store:
    vector_store.add(sqlite_id=summary_id, embedding=embedding)

Step 3: Agents Retrieve Relevant Memories Before Responding

When an agent needs to respond, it embeds the recent conversation context and queries Faiss for the most similar past summaries:

memories = await asyncio.to_thread(retrieve, formatted_context, 2)
formatted_memories = self.format_memories(memories)
persona = self.generate_prompt(formatted_context, formatted_memories)

The retrieve() function wires the full pipeline:

def retrieve(query: str, k: int) -> list[RetrievalResult]:
    query_embedding = generate_embedding(query)

    with FaissVectorStore() as vector_store:
        faiss_results = vector_store.search(query_embedding, k=k)

    sqlite_ids = [sid for sid, _ in faiss_results]
    with SQLiteSummaryStore() as summary_store:
        summaries = summary_store.get_summaries_by_ids(sqlite_ids)

    return [RetrievalResult(summary=s, score=score_map[sid])
            for s, sid in zip(summaries, sqlite_ids)]

This means the Villager remembers that "last meeting, we agreed bandits are driven by starvation" even though that conversation happened in a previous run.

Why Faiss with Inner Product (Not L2)?

Faiss offers several index types. I used IndexFlatIP (flat inner product) rather than IndexFlatL2 (Euclidean distance). The trick: if you normalise vectors to unit length before insertion, inner product becomes equivalent to cosine similarity — but faster, since Faiss is heavily optimised for dot products.

# In FaissVectorStore.add():
euclidean_norm = np.linalg.norm(embedding)
normalised_embedding = embedding / euclidean_norm
self.faiss_index.add(normalised_embedding.reshape(1, -1).astype(np.float32))

This is a standard trick in production vector search systems. Cosine similarity measures the angle between vectors (semantic direction), ignoring magnitude. Two texts about "bandits and food shortages" will have high cosine similarity regardless of how long or short the embedding vectors happen to be.

Depending on your use case you may choose a different type of vector search. For instance if the magnitude of a result matters then a dot product search may better lend itself to your use case.

What the Agents Actually Produce

Each agent has a distinct persona defined in its generate_prompt() method. The Mayor speaks with a French accent, the Judge is stern and incensed, the Villager is Welsh and frustrated:

[Mayor] "Oui, oui, Capitaine—zere is no doubt ze threat grows! But we must
ask: do we crush ze bandits wiz force alone, or do we starve our own people
to pay for it? Sacré bleu, ze answer is not so simple!"

[Villager] "Ach, listen here now—if we're so keen on spendin' gold on swords
an' not seeds, then who's left to plant the bloody crops when half the village
is either dead or robbin' us blind, eh?"

[Judge] "Then let me ask plainly—shall we bleed the town dry for patrols now,
or watch it rot from within when the fields lie fallow and the granaries
stand empty?"

As you can see the conversation is entertaining but limited. The agents are given a topic and will debate on it until each hits their hard limit on number of responses. The conversation often spirals into dramatics with agents only ever seeming to escalate the stakes until they are all screaming that they are going to be killed.

It is crucial to note here that I did not add a conflict resolution mechanism. The agents are prompted to debate, but the judge can never issue an order and end the conversation.

Where It Breaks: The Circular Debate Problem and the Degeneration of Thought

Here's the hard ceiling I hit. After about 30 messages, the conversation degenerates into a loop:

Villager: "We're starving!"
Judge: "The lords hoard grain!"
Mayor: "But we can't afford to give it away!"
Villager: "We're starving!"
...forever

Each Agent is prompted to respond with their concerns about the topic. They are not told to propose solutions. Additionally no mechanism is in place to determine if a reasonable solution has been identified or to synthesize one from the debaters' contributions.

Worse, the retrieved memories reinforce the loop. Past summaries all say "No decision reached; further debate needed" — because that's what happened every time. The agents are literally being reminded that they've failed to decide before, then asked to express more concerns.

To find the best solution to this problem I did some research and discovered that this is a well documented problem.

Liang et al. (2023) call it "Degeneration of Thought" — once an LLM establishes confidence in a position, self-reflection can't break it out. Their Multi-Agent Debate (MAD) framework introduces a judge with an "adaptive break" that ends debate when positions converge, plus distinct affirmative/negative roles that force genuine disagreement.

They actually break down their debates into rounds to keep it more structured and to leave the Judge more opportunities to identify a solution from their discussions.

What I'll Do Next

I believe that the MAD approach is the best solution to our current problem so I plan to do the following:

Give the debaters prompts encouraging them to propose ideas (perhaps with an affirmative and a negative side)
Give the judge the ability to assess whether a consensus has been reached and if not to determine whether a reasonable solution can be extracted from the debaters answers
Add structured outputs for the LLMs to make aggregation and voting possible
Testing to see how often agents disagree / agree with each other, who the judge sides with more often and other core metrics

Key Takeaways

Your prompts define a behaviour ceiling. Every agent did exactly what it was told: express concerns. No amount of architectural sophistication will overcome a prompt that asks for the wrong thing.
Memory can reinforce failure. If your summaries record "no consensus reached," and agents retrieve those summaries, you've built a system that remembers it can't decide — and acts accordingly.
A message bus is the foundation. Redis Streams with consumer groups gave me fan-out messaging with acknowledgment, decoupled agents from each other, and made the whole system trivial to extend with new roles.
Normalised inner product = cosine similarity. A standard trick, but one worth understanding from first principles. If you normalise your vectors before inserting them into a Faiss IndexFlatIP index, you get cosine similarity for free.
Read the literature When I was trying to find out how people improved their agent dialogues I came across various research papers; The MAD framework and Generative Agents (Park et al). This really opened my eyes to a new information source I would not have used before. These papers are brilliant and I highly recommend using research papers in the future

DEV Community: Sean Graham