When you build a knowledge base powered by an LLM, you eventually have to answer a practical question: which model should the query agent actually use?
This post shares our findings from a structured evaluation across five models. The short version: the cheapest viable option also won on accuracy.
Background
Synthadoc is an open-source LLM wiki compilation engine. It reads your raw source documents — PDFs, spreadsheets, web pages, Word files — and synthesizes them into a structured, cross-referenced wiki at ingest time. Knowledge is compiled once, not re-summarized on every query.
The QueryAgent sits at the front of every question. It decomposes the query, runs parallel BM25 retrieval across the compiled wiki, assembles a proportional context window, and calls the configured LLM to synthesize an answer with inline citations. It's the most token-intensive path in the system, and the one where model choice has the most visible impact on both quality and cost.
Why We Ran This Evaluation
Synthadoc supports a wide range of LLM providers: Anthropic, MiniMax, DeepSeek, Qwen, Gemini, Groq, and local Ollama. Choosing between them used to be a matter of intuition. We wanted data.
The specific question: is a more expensive model meaningfully more accurate, or is there a sweet spot that gives you most of the quality at a fraction of the cost?
The Evaluation Setup
We used the AquaFlow Capital scenario, a private equity M&A due diligence wiki built from eight deal documents (LBO model mechanics, quality of earnings, covenant framework, ESG standards, legal DD process, and exit valuation benchmarks).
We ran 15 structured questions across three complexity tiers:
- English - Medium (Q1–Q5): factual retrieval, single-page lookups
- English - High (Q6–Q10): multi-source synthesis, numerical reasoning
- Cross-lingual Chinese (Q11–Q15): same questions in Chinese, requiring CJK retrieval
Five models were evaluated: MiniMax-Think (M3), Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1 (V3), and Qwen Plus.
Each answer was graded on a 0–100 scale against a ground-truth rubric. PASS threshold: 85%.
What We Found
The scatter tells the story clearly.
MiniMax-Think (M3) sits alone in the top-left, highest accuracy at the lowest cost. That's the ideal quadrant, and no other model comes close to occupying it.
| Model | Accuracy | Est. cost / 1K queries |
|---|---|---|
| MiniMax-Think (M3) | 92% | ~$25–40 |
| Claude Opus 4.8 | 89% | ~$1,050 |
| Claude Sonnet 4.6 | 86% | ~$210 |
| DeepSeek-R1 (V3) | 78% | ~$35–55 |
| Qwen Plus | 73% | ~$30–50 |
Claude Opus 4.8 scored 89%, just 3 percentage points below MiniMax-Think, but at roughly 5× the cost of Sonnet and 26× the cost of MiniMax. For a query workload measured in thousands of questions, that gap is significant.
Claude Sonnet 4.6 at 86% is the best option if you're locked to the Anthropic Claude or its API and need reliable citation quality.
DeepSeek-R1 at ~$40/1K queries is a solid budget pick for exploratory or conceptual synthesis queries, where synthesis depth matters more than precision.
Takeaway
For the AquaFlow domain - dense financial documents, multilingual queries, numerical reasoning - MiniMax-Think (M3) is the clear default. It wins on accuracy and cost simultaneously. That's a rare combination.
Model choice isn't just a quality decision. At scale, it's a cost architecture decision.
The full report includes per-question breakdowns, cost math, WARN pattern analysis (where each model fails and why), and a per-question line chart across all 15 questions:
Top comments (0)