Every team running LLMs in production eventually asks the same question: should we switch to a Mixture-of-Experts model? The MoE architecture activates fewer parameters per token, which means cheaper inference on shared infrastructure. But switching blindly is a gamble. Your prompts, your traffic volume, and your latency SLA determine whether the tradeoff actually pays off.
MoE Cost Analyzer is a benchmarking tool that answers that question with hard numbers rather than theory. It takes a JSON file of your real production prompts, runs live API calls against both a dense and a MoE model via OpenRouter, measures actual latency and cost per query, and returns a concrete recommendation against SLA thresholds you define. No simulations. No estimates. Real calls, real results, one decision.
To validate the tool, it was run on a 100-task sentiment-analysis benchmark against two Gemma variants: the dense google/gemma-4-31b-it and the MoE google/gemma-4-26b-a4b-it. Both models received identical prompts via live OpenRouter API calls. Every response was timed, every token counted, every dollar logged. The results below come from those 200 real API calls, with zero errors.
The Report: Validated Live Results
Benchmark: 100 sentiment-analysis tasks
API: OpenRouter (live calls, not simulated)
Total calls: 200 (100 per model) | Errors: 0
Verdict: USE MoE. It met both SLA targets (latency < 3 000 ms, cost < $0.02/1K queries).
Reading the Results
The MoE model is cheaper because it activates fewer parameters per token.
Both models produced identical token counts (4 939 total). The output quality is equivalent, but the MoE model costs 20% less per token on OpenRouter because inference is cheaper on sparse architectures.
The MoE model is faster at median load (P50: -27%) but the gap narrows under stress (P95: -13%).
P50 shows typical behaviour: MoE is noticeably snappier. P95 shows tail latency under OpenRouter's shared infrastructure, and both models slow down similarly under peak load, so MoE still wins but by a smaller margin. For latency-sensitive SLAs check your own P95 against your threshold.
Token count is identical. Output quality is maintained.
Average prompt tokens (~49) and completion tokens (~20) were the same across both models, confirming the architecture change does not degrade output length or structure.
Zero errors across 200 live API calls.
The retry logic (3x exponential back-off on 429/5xx) was never triggered. Both models were stable throughout.
Extrapolated Production Impact
Based on real measured average of ~49.4 tokens/query.
Your numbers will differ, which is exactly why you should run this tool on your own prompts.
The Tool: Run It on Your Own Workload
The report above is a fixed snapshot. Your production prompts are longer, your SLA is different, your traffic pattern is unique. Run the analyzer on your own benchmark to get a recommendation that actually applies to you.
Setup
git clone https://github.com/dakshjain-1616/MoE-Cost-Analyzer.git
cd MoE-Cost-Analyzer
pip install -r requirements.txt
cp .env.example .env
# add your key: OPENROUTER_API_KEY=sk-or-v1-...
Prepare your benchmark
Create a JSON file with your real production prompts:
{
"name": "My Production Benchmark",
"tasks": [
{
"id": "task_001",
"prompt": "Extract all dates mentioned in this support ticket: ...",
"expected_labels": ["2026-01-15", "2026-02-03"]
},
{
"id": "task_002",
"prompt": "Classify the urgency of this message as Low / Medium / High: ...",
"expected_labels": ["High"]
}
]
}
Use 50–200 representative prompts. The included benchmark.json has 100 sentiment tasks you can use as a starting point.
Run
# Live run against OpenRouter
python analyze.py my_benchmark.json
# With custom SLA thresholds
python analyze.py my_benchmark.json \
--sla-latency-ms 1500 \
--sla-cost-per-1k 0.005 \
--output my_results.csv
# Dry-run (no API calls, uses simulated data — good for testing the pipeline)
python analyze.py benchmark.json --dry-run
What You Get
1. A Rich terminal table: colour-coded decision matrix printed to stdout:
╭────────────────────────┬──────────────────────┬──────────────────────┬──────────╮
│ Metric │ Dense │ MoE │ MoE vs Dense │
├────────────────────────┼──────────────────────┼──────────────────────┼──────────┤
│ Avg Latency (ms) │ 1721.3 │ 1283.2 │ -25.5% │
│ P50 Latency (ms) │ 833.2 │ 605.1 │ -27.3% │
│ P95 Latency (ms) │ 6748.1 │ 5879.2 │ -12.9% │
│ Avg Cost / Query (USD) │ $0.000005 │ $0.000004 │ -20.0% │
│ Total Cost (USD) │ $0.0005 │ $0.0004 │ -20.0% │
│ Total Tokens │ 4939 │ 4939 │ 0.0% │
│ Error Rate │ 0.0% │ 0.0% │ N/A │
╰────────────────────────┴──────────────────────┴──────────────────────┴──────────╯
Summary: MoE is 20.0% cheaper and 25.5% faster than the dense model.
✓ Recommendation: USE MoE — latency 1283ms < 3000ms SLA and cost $0.0040/1K < $0.0200/1K SLA
2. A per-query CSV: every task, every model, every metric:
task_id,model_id,latency_ms,prompt_tokens,completion_tokens,total_tokens,cost_usd,error
task_001,google/gemma-4-26b-a4b-it,816.6,60,20,80,0.0000064,
task_001,google/gemma-4-31b-it,1384.2,60,20,80,0.0000080,
task_002,google/gemma-4-26b-a4b-it,897.0,60,20,80,0.0000064,
Load it in pandas for deeper analysis:
import pandas as pd
df = pd.read_csv("results.csv")
print(df.groupby("model_id")[["latency_ms", "cost_usd"]].describe())
3. A recommendation string: one of three outcomes:
✓ USE MoE: MoE meets both your latency and cost SLAs. Switch.
~ MARGINAL: MoE meets one SLA but not both. Review the failing dimension.
✗ STICK WITH DENSE: MoE fails one or both SLAs at your thresholds. Do not switch yet.
CLI Reference
python analyze.py <benchmark_file> [options]
Options:
--dense-model Dense model ID (default: google/gemma-4-31b-it)
--moe-model MoE model ID (default: google/gemma-4-26b-a4b-it)
--sla-latency-ms Avg latency cap (ms) (default: 2000)
--sla-cost-per-1k Cost cap per 1K reqs (default: 0.01)
--output CSV output path (default: results.csv)
--dry-run Simulate API calls, no cost incurred
Edge Cases and Reliability
Empty benchmark file: Prints "No tasks to evaluate." and exits cleanly.
Invalid model ID: Prints "Error: Invalid model ID" and exits with code 1.
Missing API key: Suggests --dry-run and exits with code 1.
Rate limit (429): Retries up to 3x with exponential back-off (1s, 2s, 4s).
Server error (5xx): Same retry logic.
Partial failures: Error rate shown in matrix; successful tasks still analysed.
How It Works
your_benchmark.json
│
▼
analyze.py ─── validates model IDs + task list
│
▼
runner.py ─── asyncio.gather (semaphore = 5 concurrent)
├── google/gemma-4-31b-it ──► OpenRouter API
└── google/gemma-4-26b-a4b-it ─► OpenRouter API
│
▼
analyzer.py ─── per-model stats: avg / p50 / p95 latency, cost
─── decision matrix DataFrame
─── recommend() against your SLA thresholds
─── save_csv()
│
▼
reporter.py ─── Rich terminal table + coloured recommendation
Concurrent requests are capped at 5 to stay within OpenRouter rate limits. Adjust asyncio.Semaphore(5) in src/runner.py if your plan allows higher throughput.
Pricing
Pricing is approximate and subject to change. Check OpenRouter for current rates. Update src/pricing.py when OpenRouter rates change. The ModelPricing dataclass also stores context_window and model type for future extensions.
Tests
pytest tests/ -v
# 18 passed in 5.11s
test_analyzer.py: Stats computation, decision matrix shape, recommendation logic, error-rate tracking.
test_runner.py: Pricing calculation, model ID validation, cost ordering.
test_cli.py: Empty benchmark → clean exit, invalid model ID → error message, dry-run → CSV written.
Project Structure
moe-cost-analyzer/
├── analyze.py ← entry point
├── benchmark.json ← 100-task example benchmark
├── requirements.txt
├── .env.example
├── conftest.py
├── docs/
│ └── benchmark_results.svg ← chart generated from live results
├── src/
│ ├── pricing.py ← ModelPricing dataclass + compute_cost()
│ ├── runner.py ← async OpenRouter calls, retry logic, dry-run
│ ├── analyzer.py ← stats, decision matrix, CSV export
│ └── reporter.py ← Rich terminal output
└── tests/
├── test_analyzer.py
├── test_runner.py
├── test_cli.py
└── fixtures/
└── empty_benchmark.json
How I Built This Using NEO
This project was built using NEO. NEO is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.
The requirement was a tool that runs live API calls against both a dense and a MoE model on a user-supplied benchmark, measures real latency and cost, and returns a data-backed recommendation against configurable SLA thresholds. NEO planned and produced the files in this repository: the CLI entry point in analyze.py, the async OpenRouter runner with retry logic and dry-run mode in src/runner.py, the stats and decision matrix engine in src/analyzer.py, the Rich terminal reporter in src/reporter.py, the pricing dataclass in src/pricing.py, three test files covering stats computation, pricing, and CLI behavior, and the 100-task example benchmark in benchmark.json.
The result is a fully working analyzer that takes any JSON benchmark, makes live API calls to both models concurrently, and returns a colour-coded terminal table, a per-query CSV, and a single recommendation string in seconds.
How You Can Use This With NEO
Run it on your own production prompts before switching models.
Replace benchmark.json with 50–200 of your actual production prompts, set your own --sla-latency-ms and --sla-cost-per-1k thresholds, and get a recommendation grounded in your real workload rather than a generic benchmark.
Use --dry-run to test the pipeline without spending API budget.
The dry-run mode uses simulated data and produces the same CSV and terminal output as a live run, making it safe to validate the setup and SLA threshold logic before committing to real API calls.
Load the per-query CSV in pandas for deeper analysis.
Every task, every model, every latency and cost measurement is written to a CSV. Grouping by model_id and running describe() on latency_ms and cost_usd surfaces distribution details that the summary table alone does not show.
Point it at any two OpenRouter models, not just Gemma.
The --dense-model and --moe-model flags accept any valid OpenRouter model ID. The same benchmark, the same SLA thresholds, and the same recommendation logic work across any pair of models you want to compare.
Final Notes
The question "should we switch to a MoE model?" comes up every time a team hits a cost or latency wall. The answer is almost always "it depends on your prompts", which is why a tool that runs on your actual workload and returns a gated recommendation is more useful than any benchmark you did not run yourself.
The code is at https://github.com/dakshjain-1616/MoE-Cost-Analyzer
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code
Top comments (0)