DEV Community: ammorick

Can Multi-Model Discussion Actually Solve AI Hallucination? A Reflection from an MVP Practitioner

ammorick — Fri, 10 Jul 2026 02:16:59 +0000

I'm a beginner currently transitioning into tech, and I've been building an MVP of a multi-expert AI system.

The current core mechanism is:

1.User asks a question

2.System calls multiple expert models in parallel (medical, law, strategy, general)

3.Each model responds independently

4.A director brain aggregates the output

5.Supports random multi-brain self-interaction mode

6.A simple memory module

After the user asks a question, multiple expert models respond in parallel, and the outputs are aggregated. More perspectives should be more reliable than a single one — that logic sounded reasonable.

But after running it for a while, something started to feel off.

I began to wonder: does this actually achieve what I was hoping for — multi-brain cross-validation that improves information accuracy? Can this kind of AI really be trusted?

If each individual model is itself unreliable, then what does their "consensus" actually represent? Is it converging toward correctness, or just making unreliability look more convincing?

The core problem: the validator itself is unreliable

What I'm doing, in essence, is calling multiple off-the-shelf models and having them answer the same question concurrently. Accuracy and certainty still depend on each model's own knowledge base — and AI models aren't entirely trustworthy. They make things up. They hallucinate with confidence.

It doesn't eliminate uncertainty. It just spreads the source of uncertainty from one model to multiple models.

My original intent was to build "cross-validation" — multiple brains questioning and testing each other, eventually converging on more reliable conclusions. But what I've built so far feels more like "multiple models talking past each other, then stitched together."

This isn't true cross-validation. It's simulating the appearance of discussion.

The difference between known and unknown problems

When we test these systems, we almost always use questions we already know the answers to.

But those questions are already in the training data. We're not testing reasoning ability — we're testing memory. True "reliability" should be demonstrated when facing unknown problems. And on unknown problems, we have no reference to verify whether the model's answer is correct.

If the validator itself is unreliable, what does the validation result even mean?

The design direction is right, but the foundation is wrong

The idea of "multi-brain cross-validation" itself is sound — it's a reasonable response to the problem of "AI being unreliable." The issue lies in its precondition: the participating models themselves need to have a baseline level of reliability. If the models themselves are unreliable, then no amount of discussion will move things in the right direction.

So my MVP isn't meaningless — it got me to the point where I can see this problem clearly. But it also means that if I want this direction to truly work, I need to think about not just "how models discuss," but "how models become more reliable."

A new possibility: should validation happen inside the model itself?

Recently I've started thinking about a question:

Should multi-brain cross-validation happen inside the model itself?

If this direction could work, it might mean: models would no longer need such massive amounts of training data. This could be a fundamental solution to reducing data dependency, and potentially another form of AI self-learning.

This is different from another "AI self-learning" path I've been considering, but I see them as parallel possibilities.

Choices at the current stage

If I want this MVP to have real differentiating value, I need to think about several things:

Actually solving AI reliability — not just simulating it

Structural design is the path, but it only has value if it actually solves a real problem

To be different, the model itself needs to be different — otherwise validation itself can't be validated

The direction is right, but the foundation is wrong. This leads to:

Should I build my own wheels?

Or use existing wheels to build the framework first?

What are the risks and returns of each choice?

Would this become over-engineering? After all, it's just an MVP

But an MVP still needs to prove something

Current status

I don't have an answer yet.

Should I pivot? Should I continue in the current direction? I'm still evaluating. At this stage, I lack the ability to test these ideas, but at least I can now see where the problem lies.

This article is a record of my current thinking, not a conclusion. I'll update it as I continue experimenting.

If you're also working on similar multi-agent systems, or have your own views on this direction, feel free to share in the comments.

Links:

GitHub: https://github.com/ammorick/ai-learning-journey

Idea Archive: https://github.com/ammorick/Future-exploration-direction.to-be-sorted

From a Sketch to a Working Multi‑Brain Dialogue Module — Design, Iteration, and Lessons Learned

ammorick — Mon, 22 Jun 2026 04:25:31 +0000

Background

I've been building a multi‑expert AI Q&A system. The core logic is straightforward: when a user asks a question, the system calls multiple domain experts (medical, legal, strategy, etc.) in parallel, collects their responses, and synthesizes a final answer.

But there's a longer‑term direction I've been thinking about: if a large number of expert brains engage in random dialogue, it might generate knowledge beyond what we currently know. That's what I decided to think through today.

I drew a sketch to capture the idea at that moment.

Module Design

I called it the "random brain" module.

The core flow works like this:

Select participants: Randomly pick 2–3 experts from a roster. Either fully random, or user‑specified.

Generate entry point: The system auto‑generates an open topic — no user input needed.

Multi‑round dialogue: Experts discuss the topic across multiple rounds. Each round, they can see the full conversation history.

Separate storage: Discussion records are saved independently, not mixed with normal user conversation history.

Mode switching is handled via command line:

/random — pure random selection

/random medical strategy — specify a particular combination

/normal — switch back to normal dialogue mode

This module doesn't depend on user input — it runs as a system‑initiated behavior, outside of the user's view.

Testing and Iteration

Getting this right took three distinct iterations.

First test: each expert just listed their own directions

Two experts were randomly selected and asked to speak on an open topic.

The output was stable — but nothing unexpected. Both experts basically listed five directions from their own fields. The medical expert listed medical cross‑domain ideas; the strategy expert listed strategic ones. No interaction, no response to each other's points.

First revision: added context summaries

In later rounds, each expert received a summary of what other experts had said in previous rounds.

This improved things slightly, but not enough. Occasionally there were light references — "thanks for the previous expert's perspective" — but overall it still felt like parallel monologues.

Second revision: layered prompts

First round stayed open. From the second round onward, experts were explicitly asked to respond to other experts' points.

This finally produced what I was looking for. The two experts started referencing each other's ideas and building on them.

One expert picked up a direction from the other and added a concrete concept. The other took the first expert's idea and expanded it with more detailed proposals.

Instead of each producing a separate list, they actually formed a structured discussion. This was the first time I saw "dialogue" rather than "parallel output" from this module.

Current Status

The core mechanism works, but there's still a long way to go before it's stable enough to ship.

What works:

Multi‑round dialogue framework

Random or specified expert combinations

Separate storage for discussion records

Command‑line mode switching

Known limitations:

Prompts are still hard‑coded — needs to be externalized for future expansion

Valuable content from round 2 can sometimes get diluted or overwritten in round 3 — a risk of reverse optimization

Response time is slow — waiting for all API calls across multiple rounds creates noticeable latency

Safety monitoring isn't wired in yet — random discussions don't pass through the safety filter

Output quality varies across different expert combinations — consistency still needs work

The thread pool is created per call — works fine for 2‑3 experts, but needs refactoring for scaling

Discussion records are saved with timestamps but not indexed — retrieval will become harder over time

All of these are fixable. They don't invalidate the core functionality — they just mean there's more work ahead.

Some Thoughts

The value of this module isn't in solving a specific problem — it's in exploring possibilities that aren't already in the expected range. When the system doesn't rely on user input and doesn't have a fixed direction, it can generate intersections you wouldn't have thought of yourself.

This direction is still early. The next challenge is stability and consistency. If I can get that right, potential use cases include cross‑industry exploration, inspiration for academic research, or a component in a larger exploratory system.

Technical Details

Layered prompt design: round 1 stays open; round 2 and later include summaries of other experts' points as context

Each round calls expert APIs concurrently using ThreadPoolExecutor

Discussion records are saved as JSON with full conversation history

File naming pattern: random_discussion_{timestamp}_{expert_combo}.json

Mode switching is handled via command line

Repo: https://github.com/ammorick/ai-learning-journey

The public repository currently keeps the design sketch and module structure visible, but the full implementation is private for now.

Open Question

If you've worked on multi‑agent collaboration or exploratory modules, what challenges did you run into with stability and consistency? How did you handle them?

Feel free to share your experiences — I'd love to hear how others are approaching similar problems.

Postscript

This article is a record of the development process for the random brain module. It's still an experimental feature — not a finished product, not a productized feature. If you're working on something similar, I'd be curious to hear about your approach.

From Delivery Rider to Building My First AI System — Here's My Story

ammorick — Sat, 20 Jun 2026 16:50:21 +0000

Hi everyone, I'm a self-taught developer from China. I work as a delivery rider from 8 PM to 8 AM, and I code in every spare moment between shifts.

In June 2025, I finished a shift, lay down in bed, and started scrolling through videos. I came across a course on AI Agents. The structure was: receive input → call tools → output → memory. And a question popped into my head:

"Why can't we have two AI brains talking to each other, cross-validating information?"

A whole string of ideas followed. And I had this strange feeling — I don't know why, but it felt like this was what I was supposed to do with my life.

I didn't know how to write a single line of code back then. But the thought never went away.

On May 24, 2026, I decided to start learning to code. I'm a delivery rider — I work from 8 PM to 8 AM, and I have about 4 hours a day for self-study. So I started building. Two weeks later, I had a working MVP:

Multi-expert parallel execution: simultaneously calling medical and legal "expert brains" (powered by LLM APIs) using ThreadPoolExecutor

Safety brain: input + output filtering, with black/whitelist loaded from an external JSON and separate violation logs

Memory module: persistent conversation history (JSON), keeping the last 20 turns, with support for "recall what we talked about last time"

Director brain: aggregates responses from multiple experts into a clear, user-friendly final answer

Engineering habits: environment variable management, modular design (dispatcher/summarizer/memory/safe/safety_logger), requirements.txt, etc.

This sketch was drawn after the MVP was running, to map out the next steps. At that point, I had three main directions in mind: implementing true multi-brain dialogue through the memory module; optimizing the director brain (layered routing, value system, aggregation logic); and evolving the safety brain from static filtering toward "ground-level safety."

Today, my MVP has these capabilities:

Multiple experts (medical, legal, strategy) running in parallel, each powered by a different API — Zhipu, Aliyun, and OpenRouter (which gives flexible access to multiple models through a single gateway)

Multi-round debate — they can agree, disagree, or supplement each other, all within a shared memory framework

Memory — it remembers the last 20 turns, so context carries over

Safety — input and output filtering with violation logs (there's still a substring-matching false-positive issue, but it doesn't affect the core value of the MVP, so I'm leaving it for now)

A brief note on how it works:

When a user asks a question, the system matches keywords against available experts, with a general expert as fallback. If multiple experts are triggered, the system initiates a multi-brain dialogue — it creates an independent history log that records each round of discussion, and passes that log as context to every expert in subsequent rounds. Each expert sees the full discussion thread, not just the other side's last response. This allows them to build on or push back against each other's arguments, rather than answering in isolation. Finally, the director brain aggregates everything into a single output.

This approach does consume more tokens than a standard setup, but I already have some ideas for optimization. For now, I'm focused on getting the core functionality right.

Test case:

"I have a cold. I can't understand the medication instructions from the doctor, and my company won't approve my sick leave. What should I do?"

This triggers all four experts at once (medical, law, strategy, and the general expert as fallback). Each expert sees the full conversation history from previous rounds, so they can build on each other's arguments or push back where they disagree.

It's far from perfect. The director brain's output is still just simple concatenation. Response speed is slow. The timeout logic hasn't been refined yet. All of these need work. But the core functionality works — the rest is just time.

The code is here: https://github.com/ammorick/ai-learning-journey

I also keep a "Museum of Ideas" — a separate repo where I capture the bigger, longer-term directions I'm not ready to build yet: multi-brain dialogue, AI that helps design new AI brains, safety as the underlying "ground" instead of a fence around the system. Some of these may take years. Some may never happen. But they're my starting point, and they're my North Star.

https://github.com/ammorick/Future-exploration-direction.to-be-sorted

It's not polished. It's still rough around the edges. But it runs, and I'm iterating on it every day.

I've written a couple of articles about this journey — in Chinese for now, but the code and the process are universal:

From Single Expert to Multi-Expert Debate — a technical recap of the core mechanism

Night-Shift Delivery Rider, Coding in Spare Time — the origin story and overall architecture

While testing, I realized something: to an end user, this system probably looks just like any other Q&A bot — even though the underlying logic has completely changed.

But I'm not worried about that right now. As a developer, my energy should go into the technology itself: improving my skills, sharpening my judgment, and refining the system architecture. User feedback matters — but that's for a different stage. Right now, presenting the project as a work-in-progress is itself a form of demonstration. What people who know this space will see is the thinking process and the judgment behind it, not just the final output.

I don't know if what I'm building will ever be "good enough" for the market. But at least it's already proof of some of my capabilities — and that certainty alone might open other doors further down the road.

To be honest, I don't have many people in my circle who work in this space, and it's hard to find others in a similar situation online. Most of the time, I'm learning and building on my own. And I often ask myself: Am I doing this right? Is my progress on track? What does the industry actually need? That kind of uncertainty can be more draining than the technical challenges themselves. But I keep going — because I choose to trust myself.

One thing I always keep in mind: there is no tomorrow without today. Only by putting in the work every single day can a real tomorrow arrive. I'll keep building this project — unless it gets crushed by the wheel of history.

I'm new to this field, and my fundamentals are still weak — I'm working on them. So I'm here to share my progress, and also to learn. If you've been through something similar, I'd love to hear how you navigated it.

One question I'd like to ask:

For Python fundamentals, I've been learning on the go — picking up whatever I need for the project. I used to spend about 30 minutes a day on practice exercises, and recently I've switched to a weekly schedule: MVP development on Mon/Tue/Wed, fundamentals practice on Thu/Fri/Sat, and reviews plus writing on Sundays. Does this rhythm make sense? Any advice on how to improve more effectively?

Also:
I'd love to hear your thoughts — what would you add to a system like this?

Thanks for reading.

P.S. I spent quite a while trying to include two images in this post — a sketch of my early design and a terminal screenshot of the test case running — but neither of them would display properly after upload. I've tried different formats, compression, and even external hosting like Imgur, but nothing seems to work. If you've run into this issue before and know how to fix it, I'd really appreciate any advice. Thanks!