The Numbers Are Real. The Comparison Isn’t.
How to read AI benchmark tables — using GigaChat 3.5 Reasoning vs. DeepSeek V4 Flash Preview as a case study
TL;DR: GigaChat 3.5 Reasoning is a serious open-weight model with a serious online-RL training pipeline. Its published numbers are real.
But the headline claim — “on par with DeepSeek V4 Flash while using 37% fewer tokens” — compresses a much messier benchmark profile into a very convenient story.
GigaChat wins some evaluations convincingly.
It loses others by double-digit margins.
It uses dramatically fewer reasoning tokens on the four math benchmarks used for the efficiency claim — including two benchmarks where its score is substantially lower.
And the published “Average” combines heterogeneous tasks while comparing models with unequal benchmark coverage.
None of that requires accusing anyone of fabricating data.
The more interesting problem is framing.
The benchmark table is doing rhetorical work that the raw numbers do not fully justify.
That is not a GigaChat-specific problem. It is a benchmark-literacy problem.
1. The claim
The release presents three ideas that are easy to fuse into a single headline:
- GigaChat 3.5 Reasoning is a new Russian open-weight reasoning model trained with online RL.
- It is comparable to DeepSeek V4 Flash Preview on benchmarks.
- It uses 37% fewer reasoning tokens.
The first claim is the least controversial.
The architecture, post-training pipeline, domain-specific RL experts, and on-policy distillation are all described in considerable technical detail. The model card also publishes the benchmark methodology and evaluation setup.
The second claim is where the word “comparable” starts doing a lot of work.
The third claim is mathematically reproducible — but much narrower than the sentence makes it sound. The 37% efficiency figure comes from four mathematical evaluations:
- AIME 2025
- AIME 2026
- HMMT 2025
- IMOAnswerBench
That distinction matters.
Because:
“37% fewer reasoning tokens” is not the same claim as “37% cheaper for equivalent quality across the benchmark suite.”
Those are two very different engineering statements.
2. First, let's give GigaChat its due
Before criticizing the framing, it is worth understanding the thing being framed.
GigaChat 3.5 Reasoning is described as a 432B-parameter MoE with 28B active parameters, using a hybrid architecture that combines Multi-head Latent Attention (MLA) and GatedDeltaNet-style linear attention. The model supports a 262K context and was released with open weights.
The post-training story is arguably more interesting than the headline.
The team describes training six domain-specific RL experts and then combining their capabilities using on-policy distillation (OPD). The domains include mathematics, code, agents, dialogue, and instruction following.
The technical write-up is also refreshingly candid about reward hacking.
The authors describe cases where a reward function incentivized behavior that looked successful to the scorer without solving the underlying task. They also describe length bias in judge-based rewards and the need for adaptive penalties to prevent reasoning from simply becoming longer and longer.
That is not a toy pipeline.
It is serious RL engineering.
So this article is not:
“GigaChat is fake.”
It is:
GigaChat is real, technically interesting, and more nuanced than the marketing frame suggests.
In fact, I think the engineering story is more interesting than the headline.
3. Red flag #1: “37% fewer tokens”
Here is the published reasoning-efficiency comparison:
| Benchmark | GigaChat tokens | DeepSeek tokens | Reduction | GigaChat score | DeepSeek score |
|---|---|---|---|---|---|
| AIME 2025 | 13,980 | 19,129 | 27% | 89.00 | 88.95 |
| AIME 2026 | 13,635 | 17,697 | 23% | 92.00 | 90.40 |
| HMMT | 13,311 | 19,553 | 32% | 83.13 | 95.21 |
| IMOAnswerBench | 17,074 | 29,041 | 41% | 73.00 | 85.75 |
The aggregate 37% token-reduction figure is therefore perfectly defensible as a summary of these four tests.
So don't attack the arithmetic.
Attack the interpretation.
GigaChat is slightly ahead on both AIME sets.
On HMMT, it uses 32% fewer reasoning tokens and scores 12.08 points lower.
On IMOAnswerBench, it uses 41% fewer tokens and scores 12.75 points lower.
That changes the meaning of the word efficiency.
A model that uses fewer tokens while producing a weaker answer is certainly cheaper in tokens.
That does not automatically mean it is more efficient in the sense most engineers care about.
The real question is:
How much quality did we buy with those savings?
The missing picture: quality vs. reasoning cost
This is the graph I wish more model releases included.
X-axis: average reasoning tokens — lower is cheaper.
Y-axis: benchmark score — higher is better.
Each line connects the two models on the same benchmark.
Score vs. Reasoning Tokens — GigaChat 3.5 Reasoning vs. DeepSeek V4 Flash Preview 
The picture is much harder to reduce to a slogan.
On AIME 2025 and AIME 2026, GigaChat sits slightly above DeepSeek while also using fewer tokens.
That is a genuine efficiency win.
On HMMT and IMOAnswerBench, the lines point in the other direction.
GigaChat still uses fewer tokens — but it also scores substantially lower.
So the interesting quantity is not simply:
token reduction
It is:
quality at a given reasoning budget.
Or, more precisely, the model's quality/cost frontier.
That is what a production engineer actually has to choose between.
4. A length penalty is not automatically a flaw
There is a subtle point here that is easy to miss.
The developers explicitly describe an adaptive reasoning-length penalty. The intention is not simply “make the model stop thinking.”
The idea is to penalize excessive reasoning more aggressively when the model is already likely to solve the task and less aggressively when the task is genuinely difficult.
That is a perfectly reasonable optimization objective.
There is no virtue in spending 20,000 tokens to solve a problem that can be solved reliably in 5,000.
The criticism is therefore not:
“They used a length penalty, therefore the model is worse.”
The real question is:
Did the length penalty move the model to a better quality/cost frontier?
The benchmark data suggests that the answer depends heavily on the task.
That is exactly the kind of information the headline leaves out.
5. Red flag #2: the dash
Now look at one tiny cell:
| Benchmark | GigaChat | DeepSeek |
|---|---|---|
| MERA-2.0 | 42.3 | — |
A dash is not a score.
It means the benchmark is not directly comparable for that model in the published table.
There is an important correction here, however.
The dash does not appear to be treated as zero in the published average.
In fact:
- GigaChat averages 18 reported benchmark values → 68.88
- DeepSeek averages 17 available benchmark values → 72.71
So the correct criticism is not:
“They gave DeepSeek a zero.”
They did not.
The correct criticism is:
The two averages have different benchmark coverage.
That's a much more precise problem.
The table effectively says:
GigaChat 18 / 18 benchmarks scored
DeepSeek 17 / 18 benchmarks scored
and then places both numbers under the same label:
Average
A reader scanning quickly is unlikely to notice the denominator changed.
That is exactly why benchmark tables need metadata, not just numbers.
6. Red flag #3: the “Average” score
This is where the table becomes much more interesting.
The published average combines:
- olympiad mathematics
- instruction following
- structured output
- function calling
- agents
- software engineering
- terminal use
- preference arenas
The arithmetic is fine.
The interpretation is not obvious.
Suppose a model scores:
- 95 on math
- 90 on coding
- 50 on tool use
- 75 on Russian preference data
What is its “overall capability”?
There is no natural answer.
Someone has to decide:
- which benchmarks belong in the aggregate,
- which benchmarks matter,
- how much each matters,
- whether a preference arena should count as much as an exact-match math benchmark,
- and whether all benchmark families deserve equal weight.
Once you do that, you have created an index.
And there is nothing inherently wrong with an index.
But then be honest about what it is.
An average of heterogeneous benchmarks is itself a benchmark design decision.
Or, less politely:
The moment you average GPQA, SWE-bench, Terminal-Bench, instruction following and Russian preference arenas into one number, you have stopped merely reporting measurements and started manufacturing an index.
The index may be useful.
It is not a law of nature.
7. The “home turf” problem
The Russian-language arena results are real and important.
GigaChat is considerably ahead on several of them:
| Arena | GigaChat | DeepSeek |
|---|---|---|
| Pollux | 67.9 | 49.0 |
| Arena Hard Logs V3 | 56.5 | 53.7 |
| Arena Hard Ru | 60.7 | 36.8 |
| Ru LLM Arena | 64.0 | 48.5 |
If you're building a Russian-language assistant, these results matter.
But something else is happening rhetorically.
A deployment-specific strength is being placed next to general capability benchmarks and then folded into a single narrative about parity.
That creates an easy inference:
“GigaChat matches DeepSeek.”
But the more defensible conclusion is:
GigaChat is particularly strong on these Russian-language preference evaluations.
Those are not equivalent claims.
Nothing is wrong with winning on Russian-language benchmarks.
The problem begins when a deployment-specific advantage is silently promoted into a general frontier-capability claim.
That distinction matters for anyone actually choosing a model.
If you're serving Russian-speaking users, the Russian arena results might deserve substantial weight.
If you're buying a model for autonomous software engineering, they probably shouldn't.
Deployment-specific strength ≠ general capability parity.
8. Where the gap becomes impossible to ignore
Now remove the arena rows and look at some of the workloads that are increasingly relevant to production AI systems:
| Domain | Benchmark | GigaChat | DeepSeek | Δ |
|---|---|---|---|---|
| Tool use | Function Calling V4 | 58.59 | 68.06 | -9.47 |
| Agents | TAU3-bench | 47.80 | 67.70 | -19.90 |
| Coding | SWE-bench Verified | 64.70 | 78.60 | -13.90 |
| Terminal agents | Terminal-Bench 2 | 30.30 | 56.60 | -26.30 |
| Math | HMMT 2025 | 83.13 | 95.21 | -12.08 |
These are not rounding errors.
A 26-point gap on Terminal-Bench is not what most readers mean by “on par.”
Neither is a 20-point gap on TAU3-bench.
And the official technical write-up itself acknowledges that the model still trails in agentic programming and olympiad mathematics.
Again, this is why “the vendor hid the losses” is the wrong criticism.
The losses are actually in the table.
The more interesting criticism is:
The headline selects a few favorable dimensions and lets the reader mentally fill in the rest.
9. Red flag #4: different sampling budgets
There is another subtle issue:
- AIME 2025 → mean@32
- AIME 2026 → mean@32
- HMMT 2025 → mean@8
That means the score depends partly on how many samples were drawn.
This matters because k changes the statistical opportunity for a model to find a successful solution.
A model that is highly consistent behaves differently under mean@8 and mean@32 from a model that produces occasional high-quality answers.
Different k values are not automatically wrong.
But they make casual cross-row comparison less clean.
And unless the evaluation protocol is predetermined and justified, sampling budget is another degree of freedom that can materially affect rankings.
I would not claim that the authors selected k after seeing the results. The published information does not establish that.
You don't need that accusation.
The methodological point is enough:
If the score depends on the evaluation budget, the budget is part of the result.
10. The hidden evaluation stack
Here is the biggest conceptual mistake in benchmark discourse.
A score like:
78.6
looks like a property of the model.
It isn't.
It is a property of an experiment.
Something closer to:
MODEL
│
▼
system prompt
│
▼
sampling strategy
│
▼
harness
│
┌───────┴───────┐
▼ ▼
tools timeout
│ │
└───────┬───────┘
▼
evaluation
│
┌─────┴─────┐
▼ ▼
judge verifier
│ │
└─────┬─────┘
▼
scorer
│
▼
aggregate
│
▼
78.6
And the published GigaChat evaluation notes make this especially obvious.
For example:
- IMOAnswerBench uses Qwen-3-235B-Instruct-2507 as the judge.
- TAU3-bench is averaged across Airline, Retail, Telecom and Banking.
- Natural Plan uses a corrected scorer that normalizes UTF-8 characters to ASCII.
- SWE-bench Verified and Terminal-Bench 2 use mini-swe-agent with a three-hour timeout.
- Arena evaluations use MiniMax-M2.7 as the judge and GPT-5.2 as the baseline.
- Benchmarks without a methodology-defined system prompt were run with an empty system prompt.
These details are not decoration.
They are part of the measurement.
The right mental model is not:
model → score
It is:
model + protocol → score
That distinction should be taught every time someone publishes a benchmark table.
11. The fair table
So what would a more honest comparison look like?
Not another magic aggregate.
A table that makes the important caveats impossible to miss.
Benchmark results
| Domain | Benchmark | GigaChat 3.5 Reasoning | DeepSeek V4 Flash Preview | Δ |
|---|---|---|---|---|
| Math | AIME 2025, mean@32 | 89.00 | 88.95 | +0.05 |
| AIME 2026, mean@32 | 92.00 | 90.40 | +1.60 | |
| HMMT 2025, mean@8 | 83.13 | 95.21 | -12.08 | |
| IMOAnswerBench | 73.00 | 85.75 | -12.75 | |
| GPQA-Diamond | 82.32 | 87.40 | -5.08 | |
| General | IFBench | 77.00 | 73.33 | +3.67 |
| StructEval | 85.00 | 80.19 | +4.81 | |
| MERA-2.0 | 42.30 | N/A | — | |
| Function Calling V4 | 58.59 | 68.06 | -9.47 | |
| TAU3-bench | 47.80 | 67.70 | -19.90 | |
| Natural Plan | 80.19 | 88.00 | -7.81 | |
| Code | LiveCodeBench v6 | 85.40 | 87.87 | -2.47 |
| SWE-bench Verified | 64.70 | 78.60 | -13.90 | |
| Terminal-Bench 2 | 30.30 | 56.60 | -26.30 | |
| Preference / Arena | Pollux | 67.90 | 49.00 | +18.90 |
| Arena Hard Logs V3 | 56.50 | 53.70 | +2.80 | |
| Arena Hard Ru | 60.70 | 36.80 | +23.90 | |
| Ru LLM Arena | 64.00 | 48.50 | +15.50 |
The raw values are reproduced from the published GigaChat model card.
Notice what I deliberately did not do.
I did not color one column green and the other red.
I did not crown an overall winner.
I did not create another “fair average.”
And I did not pretend that 18 heterogeneous benchmarks measure one thing.
Instead, the table answers a more useful question:
Which model looks stronger for the workload I actually care about?
12. Coverage should be part of the headline too
The published averages are:
GigaChat: 68.88
DeepSeek: 72.71
But the denominators differ:
| Model | Reported benchmark values | Missing |
|---|---|---|
| GigaChat 3.5 Reasoning | 18 | 0 |
| DeepSeek V4 Flash Preview | 17 | 1 |
The average itself can therefore be numerically correct while still being methodologically asymmetric.
This is a useful general rule:
Same formula does not mean same comparison if the input coverage differs.
A responsible table should show both.
13. Don't replace one bad aggregate with another
It would be tempting to fix all of this by inventing a new weighted score.
Don't.
You would simply move the argument one level up.
Why should SWE-bench count twice as much as GPQA?
Why should Russian Arena count half as much as HMMT?
Why should Terminal-Bench count more than StructEval?
Those are legitimate questions.
There is no universal answer.
So rather than pretending the weights don't exist, show the profile.
Across the 17 directly comparable benchmark rows:
GigaChat wins 7.
DeepSeek wins 10.
That is already more informative than a single average.
And the distribution matters.
Largest GigaChat wins
- Arena Hard Ru: +23.9
- Pollux: +18.9
- Ru LLM Arena: +15.5
- StructEval: +4.81
- IFBench: +3.67
Largest DeepSeek wins
- Terminal-Bench 2: +26.3
- TAU3-bench: +19.9
- SWE-bench Verified: +13.9
- IMOAnswerBench: +12.75
- HMMT: +12.08
The exact win/loss count is not a universal capability metric either.
But it tells you what the average deliberately hides:
The model has a shape.
And that shape is often more important than its mean.
14. Quality vs. reasoning cost
The efficiency claim deserves its own table.
| Benchmark | GigaChat score | DeepSeek score | GigaChat tokens | DeepSeek tokens | Token reduction |
|---|---|---|---|---|---|
| AIME 2025 | 89.00 | 88.95 | 13,980 | 19,129 | 27% |
| AIME 2026 | 92.00 | 90.40 | 13,635 | 17,697 | 23% |
| HMMT | 83.13 | 95.21 | 13,311 | 19,553 | 32% |
| IMOAnswerBench | 73.00 | 85.75 | 17,074 | 29,041 | 41% |
Now the marketing sentence becomes much harder to repeat without qualification.
The headline says:
37% fewer tokens.
The actual profile says:
23–41% fewer tokens on these four tests, with quality ranging from slightly better to substantially worse.
That is a much more useful statement.
It does not make the efficiency achievement disappear.
It puts it in the correct place.
15. The model is more interesting than the claim
This is where I think the story gets genuinely interesting.
The strongest argument for GigaChat is not necessarily that it “matches DeepSeek.”
The strongest argument is that the team built a large open-weight model around an unusually explicit online-RL pipeline, discovered real reward-design failure modes, trained specialized experts, and combined those capabilities through on-policy distillation.
The technical write-up describes:
- CISPO-based online RL
- cascade reward design
- six domain-specific RL experts
- adaptive reasoning-length control
- on-policy distillation
- agentic training environments
- reward-hacking failures and mitigations
That's a much more interesting engineering story than a leaderboard headline.
In fact:
The model is more interesting than the claim being made about it.
And this is important because a critique of the marketing frame should not accidentally erase the engineering achievement.
16. This is not evidence that GigaChat is a DeepSeek fine-tune
Another common reaction to a strong benchmark comparison is:
“It probably just copied DeepSeek.”
The published material does not establish that.
The model is described as a distinct 432B MoE architecture with its own training pipeline and a distinct post-training mixture.
More importantly, the benchmark profile itself is not a carbon copy.
GigaChat is unusually strong on several instruction-following and Russian-language preference evaluations while falling behind on several agentic and difficult reasoning tasks.
That looks much more like a model with its own optimization priorities than a model that simply inherited someone else's capabilities.
And that leads to another useful benchmark lesson:
A distinctive weakness profile can be evidence of distinctive training, not evidence of failure.
17. “On par” is not a property of a model
This phrase deserves special attention.
When someone says:
“Model A is on par with Model B”
the immediate question should be:
On what axis?
Capability?
Latency?
Cost?
Russian-language preference?
Math?
Coding?
Agent autonomy?
Token efficiency?
There is no single scalar called parity.
A model can be:
- ahead on Russian chat,
- roughly tied on AIME,
- behind on HMMT,
- behind on coding agents,
- and ahead on reasoning-token efficiency.
All of these can be true simultaneously.
Therefore:
“On par” is not a property of a model. It is a property of a comparison criterion.
Once you see that, many benchmark headlines become much easier to decode.
18. Benchmark tables are photographs, not permanent rankings
There is one more problem with this kind of comparison: time.
A benchmark table is a snapshot of specific versions of specific models evaluated under specific protocols.
That matters a lot in September 2026.
DeepSeek has already released V4.1-Flash, a new model generation with a different architecture, new reasoning and agentic claims, and new serving characteristics. DeepSeek says that the previous V4-Flash endpoints are being retired, with compatibility routes temporarily pointing to V4.1-Flash.
So this article should not be read as a permanent statement about:
“GigaChat vs. DeepSeek.”
It is a statement about:
GigaChat 3.5 Reasoning vs. DeepSeek V4 Flash Preview Reasoning, using the published evaluation protocols.
That distinction is not pedantry.
It is reproducibility.
A benchmark result should always have a timestamp attached to the thing being compared.
19. What a responsible benchmark release should publish
A genuinely useful model-release table should expose the entire experiment.
At minimum:
Model version
Exact checkpoint and reasoning mode.
Benchmark version
Benchmarks change.
Prompt
System and user instructions.
Sampling protocol
Temperature, k, seed policy, and other relevant parameters.
Tool configuration
Which tools were available and how they were implemented.
Harness
Especially important for agents.
Timeout / compute budget
Particularly important for coding and terminal tasks.
Judge
Model name, version, and ideally judge prompt or calibration methodology.
Missing values
N/A should remain N/A.
Sample count
Tell us how many examples generated the reported statistic.
Token / compute cost
Efficiency claims without quality are incomplete.
Uncertainty
Where sampling or benchmark size makes variance material, report it.
And one final principle:
Aggregate scores should be optional, not obligatory.
A reader should be able to understand the model without trusting a single number invented by the presentation layer.
20. A practical checklist for the next benchmark table
The next time an AI lab publishes a beautiful green-and-blue spreadsheet, ask:
What exactly is being compared?
Same model generation?
Same checkpoint?
Same reasoning mode?
Same prompt?
Are the metrics comparable?
mean@32 and mean@8 are not the same protocol.
A deterministic verifier and an LLM judge are not the same measurement.
Are missing values visible?
Does N/A mean not evaluated, unavailable, or simply unpublished?
What exactly does the “average” average?
Are the tasks commensurable?
Are the denominators equal?
Why those weights?
Where was the efficiency claim measured?
Across the whole suite?
Or only on the rows where it looked best?
Who judged the result?
A benchmark score is partly a property of its evaluator.
What does the benchmark predict?
Your workload matters more than somebody else's leaderboard ordering.
What does the headline emphasize?
Now go back and read the headline again.
After you understand the table.
21. So, is GigaChat 3.5 Reasoning “on par with DeepSeek”?
If by “on par” you mean approximately equivalent across the published benchmark suite, the table does not support that reading.
If you mean competitive on a substantial subset of evaluations, absolutely.
If you mean particularly strong on Russian-language preference tasks, yes.
If you mean more reasoning-token-efficient on the four reported mathematical benchmarks, yes.
If you mean equivalent on agentic coding, terminal use, and the harder olympiad-style tasks, the published results say no.
All of those statements can be true at the same time.
That's the point.
The model doesn't need to be mediocre for the benchmark framing to be misleading.
The benchmark doesn't need to be fabricated for the comparison to be selective.
The arithmetic doesn't need to be wrong for the conclusion to be too broad.
The numbers are real. The comparison is curated. Those are different things.
22. The bigger lesson
The real lesson here is not about GigaChat.
It is about how we consume AI benchmarks.
A benchmark table can contain nothing but correct numbers and still produce a misleading impression.
No individual cell has to be false.
The distortion can emerge from:
- what gets highlighted,
- what gets averaged,
- what gets omitted,
- how metrics are normalized,
- how sampling is chosen,
- which benchmarks are grouped together,
- and which conclusion is placed above the table.
That is more interesting than simply calling something “fake benchmarks.”
Because it is universal.
Every model lab has incentives.
Every benchmark suite has blind spots.
Every leaderboard compresses a multidimensional capability space into something a human can scan in five seconds.
The solution is not cynicism.
It is reconstruction.
Ask:
What experiment generated this number?
Then ask:
What conclusion does that experiment actually justify?
Those two questions will protect you from a surprising amount of AI benchmark marketing.
23. Final takeaway
The most useful description of GigaChat 3.5 Reasoning is not:
“A Russian model that matches DeepSeek while using 37% fewer tokens.”
It is something closer to:
“A large open-weight model trained with online RL that is highly competitive on some reasoning and instruction-following benchmarks, particularly strong on several Russian-language evaluations, substantially behind on some agentic and olympiad tasks, and demonstrably more token-efficient on the specific mathematical workloads reported.”
That sentence is less exciting.
It is also much more informative.
And that is what a benchmark table should do.
It should make the reader better informed, not merely more impressed.
So the next time a model release tells you who won, don't just inspect the score.
Inspect the race.
Before you ask which model won, ask who chose the race.
Top comments (0)