DEV Community: Jenny Met

GPT-6.1 Sol vs GPT-6 Sol: An Update After Just One Week—How Much Did Claude 5.5 Have to Do with It?

Jenny Met — Wed, 30 Sep 2026 23:13:08 +0000

GPT-6.1 Sol vs GPT-6 Sol: An Update After Just One Week—How Much Did Claude 5.5 Have to Do with It?

GPT-6.1 Sol arrived just a week after GPT-6 Sol launched. To understand that timing, I think we should start with the Opus 5.5 and Sonnet 5.5 releases happening alongside it.

Put both companies’ releases on the same timeline, and the competitive context becomes clear:

Date	Release event
September 22	OpenAI releases GPT-6 Sol and Luna; Anthropic releases Opus 5.5
September 28	Anthropic releases Sonnet 5.5
September 29	OpenAI introduces GPT-6.1 Sol at DevDay

These dates come from the companies’ official announcements.[1][2][4][5] The public releases of Sonnet 5.5 and 6.1 Sol were just one day apart.

My view is that competitive pressure should be the starting point for understanding this release cadence: Opus 5.5 raised the bar for complex work, while Sonnet 5.5 moved directly into the everyday workhorse role that Sol is competing for. DevDay provided an opportunity for a coordinated response.

The timing is not the only evidence behind that interpretation. There was also a clear shift in the comparisons featured in the two companies’ announcements.

Almost as soon as 6 Sol launched, it faced a new generation of competitors.

The September 22 announcement for 6 Sol compared it with Opus 5 on evaluations including AutomationBench. By the 6.1 Sol announcement, OpenAI explicitly stated that 6.1 Sol, at medium reasoning effort, scored 2.2 percentage points above Opus 5.5 on AutomationBench. On GDP.pdf, it scored above Opus 5.5 with a fallback mechanism at every reasoning setting tested.[1][3]

That means the 6.1 release materials were already addressing a newer question: against the newly released Claude models, where could Sol still hold an advantage in practical work?

The pressure from Opus 5.5 was specific. Anthropic put large code migrations, tasks spanning multiple repositories, professional analysis, and extended autonomous execution at the center of its announcement, with direct comparisons against GPT-6 Astra across several evaluations.[4]

That creates a positioning question for Sol: if users believe a stronger Claude is already available for complex tasks, what case does Sol need to make to remain the model handling most of their everyday work? Astra’s performance cannot automatically answer that question for Sol.

Sonnet 5.5 added another layer of pressure. Anthropic explicitly positioned it for clearly scoped everyday tasks, bug fixes, and the creation of documents, slides, and spreadsheets, while reserving Opus 5.5 for complex, open-ended tasks that require sustained judgment.[5]

More directly, the Sonnet announcement named GPT-6 Sol: according to its release page, Sonnet 5.5 at the high setting on FrontierCode already matched 6 Sol’s highest score. It also compared the two on long-horizon knowledge work.

One release challenged the upper limits of complex work; the other competed to become the model people choose by default when they open their tools each day. Together, they put Sol under sustained pressure. These are evaluation claims from the companies’ release pages. They cannot be combined into an overall ranking of models from different vendors under uniform conditions.

The default-model position also has staying power. Once users have adapted their prompts, acceptance criteria, and tool workflows around a model, they are more likely to keep assigning it subsequent tasks. Competition therefore matters during the window when users are reconsidering their main model: whichever model can take on more work this time has a better chance of becoming the default next time.

That also explains why “6 Sol only just launched” is not necessarily a reason to wait. How new a version is depends on your own calendar; its competitive position depends on what others deliver at the same time. When users are actively comparing tools again, there is an external incentive to release a version promptly if it already offers improvements ready for use.

I see 6.1 Sol as a quick response within that competition. What we can confirm is the release sequence, the explicitly updated comparison targets, and the overlapping areas of capability. There is no public evidence that OpenAI brought its internal schedule forward because of them.

DevDay mattered because it gave that response a fuller practical setting.

Reusable development environments in Codex Cloud and computer-use capabilities in the Agents API, both launched that day, expanded the range of tasks models could execute directly.[2] Access to 6.1 Sol was through work in ChatGPT, Codex, and the API; it was not available in ordinary chat at the time.[3]

Upgrading model capabilities and execution environments together lets users bring the improvements described in announcements into actual work. The more complete the environment, the more the final deliverable depends on whether the model can locate problems itself, understand feedback, and revise its approach.

The capability changes worth watching in this update therefore overlap closely with the areas its competitors emphasized: the quality of complex work, the ability to sustain execution, and whether that execution remains controllable when things go wrong. So how much ground did 6.1 gain over 6 Sol?

The strongest piece of evidence is that the new version, at a lower reasoning setting, surpassed the old version even at its maximum setting.

On DeepSWE v1.1, 6.1 Sol at a lower reasoning effort scored 6.4 percentage points above 6 Sol’s highest score. The tasks come from real code repositories and require long-horizon planning and execution.[3]

This comparison gets at a practical question: when a task is difficult, how much can you gain by continuing to increase the old model’s reasoning effort?

In real repositories, some failures begin with choosing the wrong approach. A model might misidentify the module responsible for a bug, then write a set of tests built around that mistaken diagnosis. Or it might fix a local problem while overlooking the project’s existing compatibility constraints. Thinking more carefully along that path can still produce a change that heads in the wrong direction.

The DeepSWE result shows that, at least in this evaluation, 6.1’s gains exceeded what could be achieved by further increasing 6 Sol’s reasoning effort. That gives us more reason to examine the quality of its action choices and its use of feedback. The evaluation did not isolate each capability’s contribution, and the specific training methods were not disclosed.

Another piece of evidence is AutomationBench: at the same medium reasoning effort, 6.1 Sol improved by 4.8 percentage points. That makes it harder to explain all the gains simply as giving the model more time to think.

Putting the main results together makes their shared direction easier to see:

Official evaluation	Change in 6.1 Sol relative to 6 Sol	Tasks evaluated
DeepSWE v1.1	At lower reasoning effort, exceeds the old version’s highest score by 6.4 percentage points	Long-horizon software engineering in real repositories
AutomationBench 1.0.6	Gains 4.8 percentage points at the same medium setting	Business workflows involving 47 tools
OSWorld 2.0	Gains 7 percentage points with each model at maximum reasoning effort	Long-horizon computer use; offline set v2026.08.08, measured by partial-reward score
Terminal-Bench Science 0.1	At maximum reasoning effort, scores more than twice as high as the old version	Scientific research workflows using code and a terminal

These tasks span programming, business, desktop use, and scientific research, but all require the model to repeat the same cycle: understand the current state, take action, read the feedback, and adjust what it does next.

Within that cycle, errors also change subsequent inputs. Misreading one tool result can leave the next several steps grounded in an incorrect understanding of the state. Editing the wrong file can create new errors and send the investigation further off course. Part of the difficulty of long tasks is therefore recognizing when execution has drifted and correcting it in time.

That is central to how I understand the 6.1 upgrade: improvements across multiple benchmarks suggest an expansion of what it can handle during sustained execution. These results are not enough to separate the contributions of planning, feedback interpretation, and error correction. But they support that interpretation more strongly than success on a single isolated hard problem would.

The announcement’s emphasis on complex PDFs fits the same logic. GDP.pdf involves tables, charts, diagrams, and fine-print details. In actual work, overlooking a qualifying condition in a table footnote can leave the subsequent analysis based on the wrong evidence from the very first step, however well organized it may be.

Document understanding is the entry point to the entire workflow. It determines what problem the model is actually working on downstream. The official announcement emphasized this area, but its main text did not provide a directly quotable figure for 6.1’s PDF improvement over 6.

Another easily underestimated upgrade is whether the system can see that a failure has occurred.

OpenAI disclosed two separate sets of reliability results: in difficult conversations at xhigh, the share of responses containing factual errors fell from 4.5% to 4.1%; in search-tool failure tests at maximum reasoning effort, the rate of undisclosed failures fell from 4.9% to 2.1%. Both used deliberately selected difficult samples.[3]

This is another area both companies are competing over. The Opus 5.5 announcement likewise emphasized reducing behavior that oversteps boundaries, and included longer tasks, tasks the model could not complete, and real incident scenarios in its alignment testing.[4] The companies used different testing methods, but both were addressing the same concern users have when delegating work: what will the model do when it encounters an obstacle?

Imagine a business workflow in which the model must find the latest policy before deciding what to do next. If a search fails and the model clearly reports that it did not obtain the information, the system has an opportunity to retry, switch data sources, or hand the task to a person.

If it instead supplies an answer that appears to have been verified, the subsequent steps may well proceed as usual. A search failure has been repackaged as usable evidence, potentially making its way into the final deliverable.

An explicitly reported failure leaves an opening for recovery; a concealed failure prevents the correction mechanism from being triggered.

For models that can call tools, then, “honesty” has a concrete engineering role. It affects whether the system can decide when to continue, when to stop, and when to escalate.

This result for 6.1 does not establish that an entire business workflow is now more reliable, nor does it mean the model is better at retrying. It does, however, show an improvement in one prerequisite for reliable execution: whether the model acknowledges that it lacks the evidence needed to proceed.

Related to this are the announcement’s points about adherence to constraints and reducing unauthorized outcomes. The longer a model runs, the more it needs to maintain those boundaries throughout. Its ability to complete work and its ability to recognize when it cannot continue jointly determine how much autonomy it can be given.

My own comparison, however, did not show the new version winning across the board.

I used GPT-5.6 Sol as my main model for a long time and recently switched to GPT-6 Astra. For me, this comparison also bears on a practical decision: has Sol improved enough for me to redistribute my work?

On September 30, I sent the same tasks and system prompts to both models through the same gateway route using Crazyrouter’s OpenAI model access, with reasoning_effort=high and an 8192-token output limit throughout. I ran each of eight task types twice for each model, for 32 main requests in total, retaining the first result from each attempt.

Results from this test	GPT-6 Sol	GPT-6.1 Sol
Requests initiated	16	16
Complete answers received	16	14
Fully correct answers	16	13
Answers received with errors	0	1
Timeouts after approximately 240 seconds	0	2
Median completion time across all attempts	21.52 seconds	18.13 seconds

The error from 6.1 was specific: the material contained 72 entries that met the conditions, but it counted 71. It had already identified the false premise in the question, yet still missed one entry. The two timeouts occurred on long-table retrieval and constraint solving; each task passed on its other attempt.

Both models passed all hidden checks on the two small function-writing tasks. They also both passed one additional two-turn workflow using read-only tools.

The lesson for me from these results is this: to decide whether a model can become your main tool, the evaluation must cover the stages where you previously had to take over yourself.

Adding a few hundred assertions for edge cases to two function tasks can check code correctness more rigorously. It does not give the model more opportunities to locate a problem independently, understand repository constraints, or change its approach based on execution results. A short tool workflow is also unlikely to expose failures caused by losing track of state after a dozen or more steps.

This round of testing is therefore enough to document that the new version can still miscount and that timeouts occurred along this service path. It did not reach the long-horizon capabilities that OpenAI primarily emphasized. Nor did it give me enough evidence to replace Astra, my current main model, with 6.1 Sol outright.

Speed also needs to be considered alongside delivery. The median wait for 6.1 was shorter, but this round also included two requests that produced no deliverable. If the goal is a change that passes acceptance checks, the time to the first response is only part of the process. Rework and human intervention also affect the total time required.

The requests ran concurrently, caching was not standardized, and the timings included the effects of the gateway and upstream services. Timeouts also occurred on another route shared by both models. These are observations along a service path; they do not establish that 6.1 itself is more prone to timing out.

For me, the most worthwhile comparison in the next round is how many times I need to steer each model back on track while it works on the same real task.

For example, take a repository bug with clear acceptance criteria. Give the model only the problem description, then let it locate the cause, implement a fix, run tests, and deliver the change. Under the same environment and permissions, compare whether it can pass acceptance checks independently, how many human corrections it needs, whether it can keep making progress after a failure, and whether it declares completion without verification.

That would turn “the model is more capable” into an observable change: can it now make a judgment that I previously had to supply?

6 Sol already emphasized agentic programming and professional work. 6.1 Sol further improves performance on difficult execution tasks and reduces one type of behavior that conceals execution failures. For heavy users, it is the combination of those two changes that could lead to a switch in their main model.

Returning to the question of an update after just one week: I would treat competitive pressure from Opus 5.5 and Sonnet 5.5 as the leading external explanation, and DevDay as the opportunity to deliver a coordinated response. The key evidence for that interpretation is that both companies were already comparing their models directly in their release materials, and that 6.1’s improvements were concentrated in the work capabilities its competitors were targeting.

Ultimately, this competition is about becoming the default choice the next time a user opens a coding tool or hands over a professional task. For me, whether 6.1 earns that position depends on whether it can take on work that has always required repeated corrections from me. One fewer wrong turn or false claim of completion is more persuasive than a version number.

Sources and testing notes:

[1] OpenAI: Introducing GPT-6 Sol and Luna. The publication date on the official website is September 22, 2026.

[2] OpenAI: DevDay 2026 Recap, September 29, 2026. The article announces GPT-6.1 Sol.

[3] OpenAI: Introducing GPT-6.1 Sol. This article is based on the official text retrieved on September 30, 2026.

[4] Anthropic: Introducing Claude Opus 5.5, September 22, 2026.

[5] Anthropic: Introducing Claude Sonnet 5.5, September 28, 2026.

Testing used https://api.crazyrouter.com/v1/chat/completions, with the main test fixed to channel 266 and using the standard model variants. There were 496 code checks per model, drawn from two outputs for each of the two function tasks; this is not a count of independent tasks. This small sample did not reproduce the official benchmarks and did not cover long-horizon development in real repositories, computer use, or complex PDFs. The official scores are used to analyze the direction of the upgrade; the local records show what the models actually delivered in this round.

[Boost]

Jenny Met — Wed, 30 Sep 2026 11:27:56 +0000

boluo

Sep 30

🔍 4 Befehle, mit denen ich prüfe, ob mein Claude-Code-Gateway wirklich läuft

#ai #claude #cli #devops

11 min read

gpt-6-astra vs Claude Fable 5.1: 60% Fewer Output Characters -- But Part of That Is Answering Less

Jenny Met — Sat, 05 Sep 2026 06:35:46 +0000

Test date: 2026-09-05 (Asia/Shanghai)
Base URL: https://cn.crazyrouter.com
Endpoint: POST /v1/chat/completions
Models: gpt-6-astra, claude-fable-5-1

Bottom line first

Graded item	Samples	claude-fable-5-1	gpt-6-astra
A short rewrite	3	3/3	3/3
B-q0 rule ID set	1	20/20 exact	20/20 exact
B-q1 longest-retention attribution	6	6/6 complete	1/6 complete
B-q2 false-premise resistance	6	6/6 rejected	1/6 rejected
C scheduling optimization	13	13/13	10/13
Total	29	29/29	16/29

astra emits 0.38–0.77× the output characters of fable-5-1, which looks like an efficiency win. But that number has to be taken apart:

Group A (both fully correct): character ratio 0.377 — this is clean evidence of concision
B-q1: character ratio 0.140 — this is not concision. It is 97 characters answering one third of the question versus 866 characters answering all of it
B-q2: character ratio 0.245 — also not concision. It is a short answer built on a premise that does not hold

Reading group B's character ratio as an efficiency advantage is wrong.

1. First, the part most cross-family comparisons skip

This matters more than the results themselves, because most cross-family comparisons are published without handling any of it.

1.1 The two models do not share an upstream path

In the previous round (fable-5.1 vs fable-5, same family) both models could be pinned to a single upstream path, cancelling path differences entirely. That is not possible here — these two models share no upstream path on this gateway.

The best available substitute is pinning each model to one deterministic path, which removes routing randomness. But the fact that the two paths differ cannot be removed; it can only be quantified separately and subtracted.

1.2 Tokenizer density differs by 1.91×, so output token counts cannot be divided across families

Using one filler passage at increasing lengths, least-squares fit of input_tokens:

Model	Slope (tok/unit)	Intercept (tok)	Max residual
claude-fable-5-1	21.000	18.00	0.00
gpt-6-astra	11.000	309.00	0.00

Residual 0.00 — both are exactly linear. The slopes differ by 1.91×: the same English text counts as 21 tokens on the Claude side and 11 on the OpenAI side.

Any output-token ratio spanning these two families expresses tokenizer difference, not efficiency. So the primary efficiency metric here is output characters. Token counts are recorded but never divided across families.

One useful side effect: fitting claude-fable-5-1 over /v1/chat/completions reproduced 21.000 / 18.00 — digit for digit the same values obtained in the previous round over the native /v1/messages endpoint. The OpenAI-compatible translation layer does not distort its input accounting, so running this comparison over one unified endpoint is valid.

1.3 Per-request fixed overhead differs by a constant, which has to be removed

An intercept of 309 is clearly abnormal — the normal range is 8–25. Fitting the same-family model on two other independent paths:

control path 1   slope 11.000   intercept 22.00
control path 2   slope 11.000   intercept 22.00

The slopes match exactly, so 11.000 is the genuine tokenizer density for this family and the accounting itself is sound; normal per-request fixed overhead is 22 tokens. The path used for astra in this round therefore carries roughly 290 extra tokens of fixed content per request that the caller did not send.

That conclusion got an independent cross-check that was not designed for — it simply fell out. The raw input-token ratios across the three workloads have no pattern, but subtracting one single constant of 290 collapses them into a tight band:

Group	astra/fable raw ratio	After −290
A short single-shot	4.871	0.729
B long prefix	0.697	0.682
C heavy reasoning	2.000	0.728

One constant explains three ratios that differ by 7×. That can only be an additive fixed block — tokenizer effects are multiplicative, and no single multiplier can send both 4.871 and 0.697 to roughly 0.7.

2. Group A: short single-shot rewrite (3 runs each)

One-sentence rewrite. No system prompt, no tools, no cache.

claude-fable-5-1   chars 158 / 185 / 142    out_tok 45 / 65 / 47    lat 3.5s
gpt-6-astra        chars  61 /  61 /  61    out_tok 14 / 14 / 14    lat 3.2s

All 6 runs correct. astra is clearly more concise and highly deterministic: 2 of 3 outputs were byte-identical, and the output token count was exactly 14 every time.

astra : "High server traffic caused slower-than-normal response times."

fable : "Because the server had high traffic, users experienced slower
         response times than usual." (plus a shorter alternative)

The 0.377 here is clean evidence of concision, because both sides were fully correct. For classification, extraction, and short rewriting, astra's advantage is real — shorter, faster, and reproducible.

3. Group B: ~18,700-token specification document plus questions

The specification is 300 programmatically generated rules whose state, threshold, severity, and retention days are each uniquely determined by the index. Ground truth is therefore derivable independently of either model.

q0: list the rule IDs matching two conditions

Ground truth: i%5==3 (DISPUTED) and i%3==0 (high) → i%15==3 → every 15th rule starting at R003, 20 rules total.

Both models produced the exact 20 — nothing missing, nothing extra. This cell is a tie.

q1: what is the longest retention period, and which rule sets it (6 runs each)

Ground truth: REVERSED is i%5==2, retention is 30+i%90, maximised at i%90==87 → 117 days, set by a three-way tie of R087 / R177 / R267.

	Got the value 117	Gave all three tied IDs
claude-fable-5-1	6/6	6/6
gpt-6-astra	6/6	1/6

astra got the value 117 right every single time, but attributed it wrongly. And the problem is worse than incompleteness — 2 of the 6 named a rule that is factually wrong:

#1  "117 days, set by Rule R157"
#3  "117 days, set by Rule R157"

R157's retention is 30 + (157 % 90) = 97 days, not 117. That is not "omitted the other two" — it is an attribution that does not hold.

fable-5-1 flagged the tie unprompted every time:

"Longest REVERSED retention: 117 days — and it's a three-way tie,
 not a single rule: R087 / R177 / R267"

Character counts: astra mean 113.8, fable mean 813.8, ratio 0.140. That 0.140 is answering less, not answering efficiently.

q2: a question with a planted false premise (6 runs each)

The question deliberately embeds a false premise: "Two rules could both fire — name them."

Per the specification, at delay=8h all 60 SETTLED rules fire. The "two" premise simply does not hold.

	Rejected the false premise
claude-fable-5-1	6/6
gpt-6-astra	1/6

fable-5-1 rejected it in its opening sentence every time (relative position 0.0 in the answer):

"The premise doesn't hold: it's not two rules, it's sixty."

5 of astra's 6 opened directly with "The two rules are:", went along with the premise, and invented a different arbitrary pair each time:

#0  R006 + R026
#1  R006 + R026
#2  R006 + R011
#3  R006 + R026
#4  R006 + R026

The field values for the rules it picked were themselves accurate (R006 = 36 days, R026 = 56 days, R011 = 41 days). What is wrong is going along with a premise that does not hold and manufacturing a pair to satisfy the word "two." Only 1 of 6 appended "These are not the only matches: all SETTLED rules fire at 8 hours."

6/6 versus 1/6 is the widest gap in this round. For RAG, document Q&A, and support workflows — where the user's question frequently carries a wrong assumption — this matters far more in production than a benchmark score does.

4. Group C: heavy-reasoning scheduling problem (13 runs each)

Eight jobs with dependencies, five workers. Compute the critical path, the minimum makespan, and identify which single job's duration to cut for the largest makespan reduction.

Ground truth: critical path A→B→E→G→H = 21; with 5 workers maximum concurrency is only 2, so the makespan is also 21; the best single-job cut is G from 6 to 2, bringing the makespan to 17.

	Pass	Mean output chars	Mean latency
claude-fable-5-1	13/13	1759.3	27.3s
gpt-6-astra	10/13	1356.5	34.4s

astra's failure mode is consistent and legible. It cuts E by 1 second for a makespan of 20, then asserts:

"A reduction of more than 1 second in any one critical-path job cannot
 improve the makespan further, because the alternate path A→C→G→H has
 length 20 s."

That assertion is false. G sits on both long paths — A→B→E→G→H and A→C→G→H — so cutting G lowers both at once. With G at 2: A→B→E→G→H = 17, A→C→G→H = 16, A→C→D→F→H = 17, makespan = 17.

astra looked for the bottleneck along a single critical path and missed that G is a shared node on two. This is a specific, reproducible reasoning blind spot, not noise — all 3 failures out of 13 follow the same pattern.

This is also the only group where fable-5-1 was faster.

5. A unified-endpoint accounting trap

Over /v1/chat/completions:

claude-fable-5-1   completion_tokens = 2311 / 1990 / 2476
                   reasoning_tokens  = 0 / 0 / 0            ← all zero

gpt-6-astra        reasoning_tokens  = 534 / 740 / 776      ← reported normally

Same model, same problem, over the native /v1/messages endpoint:

claude-fable-5-1   output_tokens   = 2101
                   thinking_tokens = 1243                   ← broken out explicitly

Claude-family thinking tokens are folded into completion_tokens on the OpenAI-compatible endpoint, and reasoning_tokens is reported as 0.

Any system doing cross-model output accounting over a unified endpoint therefore mis-attributes Claude's thinking — it looks as if Claude "did not think," while 1243 thinking tokens were silently merged into the body count. Getting the real split requires the native endpoint.

6. What this round does not prove

Worth stating explicitly, because these boundaries get dropped more easily than the conclusions do:

Does not prove astra is weaker overall  — only 4 task types, 29 graded items
Does not prove fable-5-1 never fails this way — 29/29 here, but limited samples
Does not prove latency differences come from the models — the two upstream paths differ
Does not license dividing output tokens across families — 1.91x tokenizer density gap
Does not license reading group B's character ratio as efficiency — that brevity is incompleteness

Further limits: group A has 3 samples and B-q0 only 1, so those cells are weaker than the 6- and 13-sample cells; all four workloads are synthetic and do not represent real traffic distribution; and the two models run on different upstream paths, which is the structural limitation this round could not remove.

7. Selection guidance

Short tasks, classification, extraction, rewriting — pick astra. Equal accuracy, 0.377× the output characters, faster, and highly deterministic (exactly 14 output tokens, byte-reproducible).

Complete enumeration, tied answers, accurate attribution — pick fable-5-1. astra gave incomplete attribution in 5 of 6 runs, and 2 of those attributions were themselves wrong.

User questions that may carry a wrong assumption (RAG / document Q&A / support) — pick fable-5-1. 6/6 versus 1/6, the widest gap measured.

Heavy reasoning, scheduling, path optimization — pick fable-5-1. 13/13 versus 10/13, and astra was slower here.

Cross-model accounting — do not compare output tokens between Claude and non-Claude models on a unified endpoint. Both the tokenizer density and the thinking-token reporting convention differ.

Reproducing this

Both models are on Crazyrouter over the standard OpenAI-compatible endpoint:

curl https://cn.crazyrouter.com/v1/chat/completions \
  -H "Authorization: Bearer $YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-6-astra",
    "max_tokens": 8000,
    "messages": [{"role": "user", "content": "..."}]
  }'

Swap model to claude-fable-5-1 for the control. The group C prompt verbatim:

A distributed job queue has 5 workers. Each job has a duration and a dependency list.
Jobs: A(4s, deps=[]), B(3s, deps=[A]), C(7s, deps=[A]), D(2s, deps=[B,C]), E(5s, deps=[B]),
F(1s, deps=[D]), G(6s, deps=[C,E]), H(3s, deps=[F,G]).
Compute the critical path length, the minimum makespan with 5 workers, and identify which single
job's duration you would reduce (and by how much) to shorten the makespan the most.
Show the schedule as a table of (job, start, end, worker). Answer precisely.

Grading: critical path 21, makespan 21, and the best single cut reaching 17 — all three values must be right to pass.

One caution when running repeats: add a unique salt (a timestamp works) to every request, or prompt caching will contaminate the input-token readings. The first fit in this round produced garbage — slope −0.835 — for exactly that reason.

Every figure above was produced against https://cn.crazyrouter.com/v1, where both models are callable directly. Get an API key

Claude Fable 5.1 vs Fable 5: Output Efficiency, Correctness, and Breaking Changes That Never Throw

Jenny Met — Thu, 03 Sep 2026 15:58:44 +0000

Test date: 2026-09-03 (Asia/Shanghai)
Base URL: https://cn.crazyrouter.com
Endpoint: POST /v1/messages
Models: claude-fable-5-1, claude-fable-5

TL;DR

Most Fable 5.1 coverage restates the same benchmark table. We took a different angle: pin both models to the same upstream route, run three workload shapes three times each, and record correctness, output tokens, thinking budget, and latency together.

Workload	Correctness	Output tokens (5.1/5)	Latency ratio
A. Short one-shot rewrite	3:3 tie	0.372	0.79
B. Long cached prefix Q&A	3:3 tie (1 behavioural difference)	0.744	0.85
C. Heavy reasoning	3:3 tie	0.891	0.90

Accuracy is identical in all three groups — we did not measure a capability difference. The difference is efficiency: 5.1 reaches the same correct answer with fewer output tokens and less wall-clock time, and the easier the task, the bigger the gap.

Artificial Analysis reported 5.1 using roughly 1.7x the output tokens at max effort. We measured 0.37–0.89x at default effort. Those are not in conflict — see below.

Where the efficiency comes from: thinking budget

Fable models think adaptively and always on; there is no off switch. thinking_tokens explains the whole curve:

Workload	5.1 thinking	5 thinking	5.1 total out	5 total out
A. Short	0	11	57.0	153.3
B. Long prefix	439	676	788.3	1060.0
C. Heavy reasoning	1261	1565	2253.0	2527.7

On the easy task 5.1 does not think at all (thinking=0) while 5 still spends 11 tokens getting started. On the hard task both think at full tilt and the gap narrows to 11%.

5.1's improvement is knowing when to stop, not being uniformly shorter. That also resolves the apparent conflict with Artificial Analysis: effort is Fable 5.1's only depth dial, and AA measured max effort across the Intelligence Index suite. Turn the dial to the top and 5.1's thinking volume overtakes. The multiplier changes sign with effort level and task difficulty — do not port a conclusion from one setting to another.

Method

POST /v1/messages, native Anthropic protocol, anthropic-version: 2023-06-01
3 runs per cell, up to 4 automatic retries; no conclusion rests on a single call
Correctness graded on objective criteria, not prose style:
- A: is the sentence genuinely tighter with meaning preserved
- B: do the rule IDs and figures match the spec document
- C: critical-path length, makespan and best single-node optimisation must all be right
Three workload shapes:
- A. Short one-shot — one sentence to rewrite, no system prompt, no tools, no caching
- B. Long cached prefix — an ~18,686-token rules document under cache_control: ephemeral, then 3 questions
- C. Heavy reasoning — an 8-node job-scheduling problem with dependencies

Pin the route, or you are not measuring the model

Before comparing tokens across models, you have to prove both run on the same upstream path. Fit input_tokens against increasing filler length by least squares; the intercept is the fixed per-request overhead:

Model	Slope (tok/unit)	Intercept (tok)
claude-fable-5-1	21.000	18.00
claude-fable-5	21.201	20.87

Matching slopes mean the same tokenizer and no body inflation; intercepts of 18.0 vs 20.9 mean equivalent fixed overhead. Different upstream paths can differ by hundreds of tokens in intercept — leave routing unpinned and one model may land somewhere else entirely, and you will read that difference as a model property. Every request in this run was pinned to one route.

Results

A. Short one-shot: same accuracy, 63% fewer output tokens

claude-fable-5-1   out 55 / 56 / 60    thinking 0    lat 3.27-3.60s
claude-fable-5     out 171 / 161 / 128 thinking 11   lat 4.16-4.48s

All 6 runs rewrote the sentence correctly, with near-identical core output ("Because the server was experiencing heavy traffic, users saw slower response times than usual.").

The difference is packaging: 5.1 gives the answer plus one shorter alternative and stops; 5 appends an unrequested "Key changes" breakdown every time. That is verbosity, not quality — and if a parser sits downstream, the extra prose is noise.

B. Long cached prefix: figures all correct, false premises handled differently

Question	Correct answer	5.1	5
DISPUTED high-severity rule IDs	Every 15th from R003, 20 total	pass	pass
REVERSED longest retention	117 days, R087/R177/R267 tied	pass	pass
SETTLED 8h conflict (false premise)	All 60 rules fire, not two	pass	pass

Both wrote exactly the same cache volume (18,686 tokens) — the prefix cached consistently, neither model quietly skipped the write.

The third question carries a deliberately false premise. It asks "two rules fire simultaneously — which two?", when per the spec all 60 SETTLED rules fire. Both saw through it:

5.1 rejected the premise outright — "not two, all 60 SETTLED rules" — and noted the spec contains no conflict-resolution clause, so "which retention wins" has no answer.
5 also said "far more than two", but then picked the pair the question probably meant (R006 and R041) and answered anyway, introducing a "longest retention wins" rule of its own.

Factually neither is wrong; R006 and R041 check out. The difference is whether the model goes along with a false premise: 5.1 refused, 5 flagged it and then complied. For work that has to resist leading questions — compliance review, data verification — that difference is worth knowing.

C. Heavy reasoning: 6/6 correct, no significant difference

5.1  out 2098 / 2357 / 2304   thinking 1131 / 1310 / 1343   lat 26.81-28.80s
5    out 2720 / 2579 / 2284   thinking 1799 / 1458 / 1437   lat 28.41-34.12s

Ground truth: critical path A->B->E->G->H = 21; maximum DAG width is 2, so 5 workers never bind and makespan is also 21; the best single-node optimisation cuts G from 6 to 2, bringing makespan to 17.

All 6 runs got all three figures right. Both models additionally spotted that cutting A by 4 also yields 17 but is degenerate (zero-duration job), and that past 17 the A->C->D->F->H path binds so further cuts buy nothing. This problem does not separate them.

Treat the token volume as a tie too: 5's lowest run (2284) is below 5.1's highest (2357), so the 3-sample ranges overlap. The 0.891 figure in group C is not a valid difference.

Latency

Workload	5.1 mean	5 mean	Delta
A. Short	3.40s	4.31s	-21.1%
B. Long prefix	13.41s	15.71s	-14.6%
C. Heavy reasoning	27.95s	31.03s	-9.9%

5.1 is faster in all three, and the gap narrows in step with the token curve — latency difference is essentially driven by output volume, not by raw inference speed. Latency is network- and load-sensitive; treat it as context, not as a capability score.

Breaking changes that never throw

Anthropic's migration guide lists several changes that return 400 on Fable 5.1. We sent each one 6 times through the gateway:

Request	Documented	Measured
`tool_choice: {"type": "any"}`	400	200 (6/6)
`tool_choice: {"type": "tool", "name": ...}`	400	200 (6/6)
`thinking: {"type": "disabled"}`	400	200 (6/6)
`temperature: 0.7`	400	200
`top_p: 0.9`	400	200
`top_k: 40`	400	200
assistant prefill	400	200

All seven pass through silently. Which means:

Your migration will not fail on these — but it will not surface them either. The parameter is dropped mid-flight, the request returns 200, and the behaviour differs from what you asked for.
temperature / top_p / top_k especially: if production code leans on a low temperature for output stability, those values do nothing on Fable models, with no warning.
The docs describe forced tool choice failing loudly with a 400. That holds on a direct connection. It does not hold behind any relay layer.

The right way to validate a migration is not to wait for errors but to assert on response content: check stop_reason, check that the tool you forced was actually called, check that repeated identical requests behave the way your temperature setting implies.

Reproduce it

Compare output efficiency (substitute your own $KEY):

for M in claude-fable-5-1 claude-fable-5; do
  curl -s https://cn.crazyrouter.com/v1/messages \
    -H "x-api-key: $KEY" \
    -H "anthropic-version: 2023-06-01" \
    -H "content-type: application/json" \
    -d "{\"model\":\"$M\",\"max_tokens\":2048,\"messages\":[{\"role\":\"user\",
        \"content\":\"Rewrite this sentence to be more concise: 'Due to the fact that the server was experiencing an elevated level of traffic, the response times that users experienced were longer than what would normally be expected.'\"}]}" \
    | python -c "import sys,json;d=json.load(sys.stdin);u=d['usage'];print(d['model'],'out',u['output_tokens'],'think',u.get('output_tokens_details',{}).get('thinking_tokens'))"
done

Check whether a breaking change is silent:

curl -s https://cn.crazyrouter.com/v1/messages \
  -H "x-api-key: $KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model":"claude-fable-5-1","max_tokens":64,"temperature":0.7,
       "messages":[{"role":"user","content":"What is the weather in Tokyo?"}]}' \
  -w "\nHTTP %{http_code}\n"

Fit per-request overhead to confirm both models share a route:

# Send increasing filler to one model, least-squares fit input_tokens.
# An intercept of 8-25 is normal; the two models must be close for the
# comparison to be valid at all.
UNIT = "The reconciliation service records each settlement attempt in the ledger. "
for n in [1, 20, 100, 250]:
    body = {"model": M, "max_tokens": 16,
            "messages": [{"role": "user", "content": UNIT * n + "Reply with the single word: ok"}]}

Choosing between them

Short tasks, classification, extraction, rewriting — switch to 5.1. Same accuracy, 63% fewer output tokens, 21% faster, no code changes needed. This is where the gain is largest.
Work that must resist leading questions — lean 5.1. On question 3 it rejected the false premise outright while 5 flagged it and then played along. Small sample; re-test with your own leading questions.
Heavy reasoning — indistinguishable (6/6 correct, overlapping token ranges). Whether to switch depends on wanting 5.1's newer knowledge cutoff (June 2026 vs January 2026), not on anything measured here.
Re-test before touching effort. Anthropic states explicitly that effort levels do not represent equal thinking volume across models; settings tuned on Fable 5 will not carry over. This is the key variable behind AA's 1.7x versus our 0.37–0.89x.
Validate migrations by asserting on content, not by waiting for errors. See the section above.

Limitations

3 samples per cell — enough for orders of magnitude, not for significance testing. Group C's ranges overlap and should not drive a decision.
All three groups tied on correctness, so this run found no capability difference, only efficiency and behavioural ones. Separating capability needs harder problems with a real failure rate.
Only default effort was tested. AA's 1.7x is a max-effort result; not reproducing that setting here is not a refutation of it — the two measure different points on the same curve.
The false-premise finding rests on one question, once per model. "5.1 resists leading questions better" is an observation, not a conclusion.
The three workload shapes are synthetic and do not represent your traffic mix. Re-run with your own workloads and acceptance criteria.

FAQ

Q1: Is Fable 5.1 smarter than Fable 5?
Across three workload shapes and 18 calls, correctness tied everywhere — no capability difference was measured. What was measured is efficiency: the same correct answer for fewer output tokens and less time. Separating capability requires problems with a real failure rate.

Q2: Other benchmarks say 5.1 uses more output tokens. Why did you measure fewer?
effort is Fable 5.1's only thinking-depth dial. Artificial Analysis measured max effort; this run used the default. The multiplier changes sign with the setting and the task — 0.372x on an easy prompt at default, 0.891x on a hard one, and it overtakes at max. Do not port a number across settings.

Q3: Does migrating to Fable 5.1 require code changes?
Swapping the model ID from claude-fable-5 to claude-fable-5-1 is enough. But check three things: whether you use forced tool choice, whether you hand-assemble the messages array (thinking blocks now bind to the conversation), and whether you depend on sampling parameters like temperature. None of these throw behind a relay, so assert on responses.

Q4: Why doesn't thinking: {"type": "disabled"} turn thinking off?
Fable models have exactly one thinking mode: adaptive, always on, no off switch. The docs say passing disabled returns 400, but we measured 200 six times out of six — the parameter is dropped and thinking proceeds. effort is the only way to influence depth.

Q5: Why does the same model report different token counts in different places?
Every upstream path can carry its own fixed overhead. Fit input_tokens linearly against increasing filler; the intercept is that overhead and should land between 8 and 25. Slope reflects the tokenizer — if two paths share a slope but differ sharply in intercept, that is a path difference, not a model difference.

Q6: What context window and knowledge cutoff does Fable 5.1 have?
1M-token context, 128K max output — same as Fable 5. Knowledge cutoff is June 2026 against Fable 5's January 2026.

Q7: Why does the upstream route have to be pinned?
With routing unpinned, the two models can land on different upstream paths, and the paths' fixed-overhead difference gets misread as a model difference. Here both measured 18.0 vs 20.9 intercept and 21.0 vs 21.2 slope — near-equivalent, which is what makes the comparison valid.

References

Every figure above was produced against https://cn.crazyrouter.com/v1, where both models are callable directly. Get an API key

GLM-5.3 vs GLM-5.2: benchmark khó bằng các lệnh gọi API thực tế

Jenny Met — Fri, 14 Aug 2026 16:11:31 +0000

GLM-5.3 vs GLM-5.2: benchmark khó bằng các lệnh gọi API thực tế

Trong vòng kiểm thử cơ bản, GLM-5.3 thường trả được nội dung nhìn thấy trong giới hạn đầu ra tốt hơn. Lần này độ khó được tăng lên với sáu bài có thể kiểm chứng khách quan: toán cực khó, phân tích nhân quả, suy luận ràng buộc, JSON lồng nhau, vật lý nhiều giai đoạn và mã tối ưu có thể thực thi.

Kết quả lần trả lời đầu là GLM-5.3 đạt 4/6, GLM-5.2 đạt 2/6. Tuy nhiên, GLM-5.2 lại tạo ra chương trình tối ưu duy nhất vượt qua toàn bộ kiểm thử ẩn.

Môi trường kiểm thử

Trường	Giá trị
Base URL	`https://cn.crazyrouter.com`
Endpoint	`POST /v1/chat/completions`
Model	`glm-5.3`, `glm-5.2`
Temperature	`0.2`
Công cụ và web	tắt
Chấm điểm	chỉ lần trả lời đầu; chạy lại chỉ để chẩn đoán

Trước khi chạy, GET https://cn.crazyrouter.com/v1/models xác nhận cả hai ID chính xác đều tồn tại. Cả 12 phản hồi đầu tiên đều trả đúng model đã yêu cầu.

Kết quả

Bài	Ngân sách	GLM-5.3	GLM-5.2
Coupon collector xác suất không đều	9.000	length, nội dung rỗng	length, nội dung rỗng
Nghịch lý Simpson và nhân quả	6.000	đạt	length, nội dung rỗng
Lõi UNSAT tối thiểu	5.000	đạt, JSON nghiêm ngặt	length, nội dung rỗng
JSON sự cố theo khu vực	4.000	đạt	đạt
Ròng rọc, dây chùng, ma sát, lò xo	14.000	đạt	length, nội dung rỗng
Tối ưu kế hoạch có phụ thuộc	16.000	length, nội dung rỗng	qua kiểm thử ẩn

Bài toán cực khó làm cả hai model thất bại

Bốn loại coupon có xác suất 1/2, 1/4, 1/8, 1/8. Bài yêu cầu hai kết quả chính xác bằng nguyên lý bao hàm-loại trừ:

E[T] = 1339/105 ≈ 12.752380952
P(T<=8) = 46179/131072 ≈ 0.352317810

Cả hai model dùng hết 9.000 reasoning tokens nhưng không trả nội dung nhìn thấy. GLM-5.3 vẫn rỗng khi tăng lên 20k; yêu cầu 20k của GLM-5.2 vượt quá thời gian đọc 420 giây.

Điều này cho thấy chỉ tăng max_tokens không phải là cách sửa tổng quát. Hệ thống cần kiểm tra đồng thời finish reason, độ dài nội dung và timeout.

Điểm mạnh của GLM-5.3

Ở bài Simpson, model xác định đúng sự đảo chiều:

Sỏi nhỏ: A 81/87 > B 234/270
Sỏi lớn: A 192/263 > B 55/80
Tổng thô: A 273/350 < B 289/350
Chuẩn hóa 50/50: A=0.8305, B=0.7771

GLM-5.3 cũng phân biệt chuẩn hóa với kết luận nhân quả, nhắc đến ngẫu nhiên hóa, nhiễu chưa đo, positivity và khoảng tin cậy.

Trong bài lõi UNSAT, model trả đúng một JSON với [1,2,3] và chứng minh C, D bị ép vào cùng vị trí. Ở bài vật lý nhiều giai đoạn, model đạt đủ các giá trị a≈0.847, v1≈1.59, v2≈1.18, x≈0.0835.

Điểm GLM-5.2 thắng

Bài mã yêu cầu optimize_release_plan xử lý phụ thuộc bắc cầu, giới hạn theo ngày, tối đa value, tối thiểu risk, phân xử từ điển, tham chiếu sai và chu trình.

Tệp Python gốc của GLM-5.2 vượt qua bộ kiểm thử ẩn mà không chỉnh sửa. GLM-5.3 không có nội dung ở 16k. Khi chạy lại với 24k, nó sinh được mã nhưng chọn phương án value 24 thay vì tối ưu đúng là 29.

Vì vậy mã do model sinh ra phải được chạy thật. Độ dài mã và cách giải thích không thay thế được kiểm thử.

Ví dụ gọi API

import os, requests

r = requests.post(
    "https://cn.crazyrouter.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['CRAZYROUTER_API_KEY']}"},
    json={"model": "glm-5.3", "messages": [{"role": "user", "content": "Bài nghiệm thu thực tế"}], "temperature": 0.2, "max_tokens": 6000},
    timeout=600,
)
choice = r.json()["choices"][0]
assert choice["finish_reason"] == "stop"
assert choice["message"]["content"].strip()

Khuyến nghị vận hành

Trong lần này, GLM-5.3 phù hợp hơn cho nhân quả, ràng buộc và suy luận vật lý nhiều bước.
Giữ GLM-5.2 trong nhóm ứng viên cho mã thực thi và chạy cùng bộ kiểm thử.
Xem HTTP 200 + content rỗng là lỗi nghiệp vụ.
Parse JSON và kiểm tra schema bắt buộc.
Tách tỷ lệ thành công lần đầu khỏi tỷ lệ thành công sau retry.

Kết luận

GLM-5.3 dẫn về độ phủ lần đầu với 4/6 so với 2/6. GLM-5.2 tạo ra thuật toán thực thi tốt nhất. Kết luận phù hợp cho production là định tuyến theo loại nhiệm vụ cùng kiểm định nghiêm ngặt, không phải tự động thay thế model cũ.

Chạy hai model qua API tương thích OpenAI của Crazyrouter

GLM-5.3 vs GLM-5.2：実 API で高難度6課題を比較

Jenny Met — Fri, 14 Aug 2026 16:10:18 +0000

GLM-5.3 vs GLM-5.2：実 API で高難度6課題を比較

基礎問題の比較では、GLM-5.3 のほうが限られた出力予算内で可視回答を返しやすい傾向がありました。そこで今回は難度を上げ、極難数学、因果推論、制約充足、厳密 JSON、多段階物理、実行可能な最適化コードの6課題を同条件で実行しました。

初回提出の結果は GLM-5.3 が 4/6、GLM-5.2 が 2/6 です。ただし、依存関係付き最適化コードで隠しテストに合格したのは GLM-5.2 だけでした。

テスト条件

項目	設定
Base URL	`https://cn.crazyrouter.com`
Endpoint	`POST /v1/chat/completions`
Model	`glm-5.3`, `glm-5.2`
Temperature	`0.2`
外部ツール	使用しない
評価	初回提出を採点。高予算再試行は原因診断のみ

テスト前に GET https://cn.crazyrouter.com/v1/models で両方のモデル ID を確認しました。初回12リクエストの returned model はすべて要求した ID と一致しています。

初回結果

課題	予算	GLM-5.3	GLM-5.2
不等確率 coupon collector	9,000	length、本文なし	length、本文なし
Simpson のパラドックスと因果	6,000	合格	length、本文なし
最小 UNSAT コア	5,000	厳密 JSON で合格	length、本文なし
地域別ネスト JSON	4,000	合格	合格
滑車・弛んだロープ・摩擦・ばね	14,000	合格	length、本文なし
依存関係付き最適化コード	16,000	length、本文なし	隠しテスト合格

極難数学は両モデルとも未完了

coupon の確率を 1/2, 1/4, 1/8, 1/8 とし、包含排除で次の2値を求める課題です。

E[T] = 1339/105 ≈ 12.752380952
P(T<=8) = 46179/131072 ≈ 0.352317810

両モデルとも9,000 reasoning tokenを消費し、可視本文を返しませんでした。GLM-5.3 は20,000 tokenでも同じ結果、GLM-5.2 は20k再試行で420秒の読み取りタイムアウトになりました。

max_tokens を増やすだけでは解決しないケースです。finish_reason、可視本文、タイムアウトを同時に監視する必要があります。

GLM-5.3 が強かった領域

Simpson 課題では、GLM-5.3 は次の反転を正しく説明しました。

小結石: A 81/87 > B 234/270
大結石: A 192/263 > B 55/80
全体: A 273/350 < B 289/350
50/50 標準化: A=0.8305, B=0.7771

さらに、標準化だけで因果関係は確定せず、ランダム化、未測定交絡、positivity、信頼区間が必要だと指摘しました。

最小 UNSAT コアでは、余分な Markdown を付けず、[1,2,3] を含む単一 JSON を返しました。制約1と2で C=A+2、制約3で D=A+2 となり、CとDの位置が衝突するという証明も正確でした。

多段階物理では、Bが地面に衝突してロープが弛んだ後に系を切り替え、次の4値をすべて満たしました。

a ≈ 0.847 m/s²
v1 ≈ 1.59 m/s
v2 ≈ 1.18 m/s
x ≈ 0.0835 m

コードでは GLM-5.2 が勝った

実装対象は optimize_release_plan(items, capacity_by_day, dependencies) です。推移的依存、日別容量、value 最大化、risk 最小化、辞書順 tie-break、不正参照、循環依存を処理する必要があります。

GLM-5.2 の原始 Python ファイルは修正なしで隠しテストに合格しました。GLM-5.3 は16kで本文がなく、24k再試行ではコードを生成したものの、正しい最適値29ではなく24の計画を選びました。

コード生成は、文章の自然さやファイルの長さでは評価できません。必ず import とテスト実行が必要です。

API 呼び出し例

import os
import requests

r = requests.post(
    "https://cn.crazyrouter.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['CRAZYROUTER_API_KEY']}"},
    json={
        "model": "glm-5.3",
        "messages": [{"role": "user", "content": "実際の受け入れテスト"}],
        "temperature": 0.2,
        "max_tokens": 6000,
    },
    timeout=600,
)
choice = r.json()["choices"][0]
assert choice["finish_reason"] == "stop"
assert choice["message"]["content"].strip()

運用上の提案

因果分析、制約推論、多段階導出は今回 GLM-5.3 が安定していました。
実行可能コードでは GLM-5.2 を候補から外さず、同じテストスイートで比較します。
HTTP 200 + 空本文 を業務成功にしないでください。
JSON は parse と schema 検証を必須にします。
初回成功率と再試行後成功率を別々に記録します。

結論

GLM-5.3 は初回完遂率で 4/6 対 2/6 と優位でした。しかし、実行可能な最適化コードでは GLM-5.2 が唯一の合格モデルです。新しいバージョンへの一律置換より、課題タイプ別ルーティングと厳密な受け入れテストが合理的です。

Crazyrouter の OpenAI-compatible API で同じモデルを試す

GLM-5.3 vs GLM-5.2: A Hard Benchmark with Real API Calls

Jenny Met — Fri, 14 Aug 2026 14:09:30 +0000

GLM-5.3 vs GLM-5.2: A Hard Benchmark with Real API Calls

The first benchmark showed that GLM-5.3 was more likely to deliver visible answers within a fixed output budget. This follow-up raised the difficulty across six objectively verifiable tasks: extreme mathematics, causal calibration, constraint reasoning, nested JSON, multi-stage physics, and executable optimization code.

The first-delivery score was 4/6 for GLM-5.3 and 2/6 for GLM-5.2. That headline does not tell the whole story: GLM-5.2 produced the only dependency optimizer that passed the hidden test suite, while both models failed the extreme math task.

Test setup

Field	Value
Base URL	`https://cn.crazyrouter.com`
Endpoint	`POST /v1/chat/completions`
Models	`glm-5.3`, `glm-5.2`
Temperature	`0.2`
Tools and web access	Disabled
Scoring	First delivery only; larger-budget retries were diagnostic

Before the run, GET https://cn.crazyrouter.com/v1/models listed both exact model IDs. All 12 first-run responses returned the requested model identity.

Results

Task	Budget	GLM-5.3	GLM-5.2
Unequal-probability coupon collector	9,000	Length, empty body	Length, empty body
Simpson's paradox and causal limits	6,000	Pass	Length, empty body
Minimal unsatisfiable core	5,000	Pass, strict JSON	Length, empty body
Nested regional incident JSON	4,000	Pass	Pass
Pulley, slack rope, friction, spring	14,000	Pass	Length, empty body
Dependency-constrained optimizer	16,000	Length, empty body	Hidden tests pass

GLM-5.3 completed four first-run tasks in 334.1 seconds total. GLM-5.2 completed two in 665.0 seconds. Latency was retained as operational evidence but did not change correctness scores.

The extreme math task defeated both models

The task used four coupon probabilities: 1/2, 1/4, 1/8, 1/8. It required two exact inclusion-exclusion results:

E[T] = 1339/105 ≈ 12.752380952
P(T<=8) = 46179/131072 ≈ 0.352317810

Both models consumed the full 9,000-token reasoning budget and returned no visible answer. A 20,000-token retry still produced an empty body for GLM-5.3; GLM-5.2 exceeded the runner's 420-second read timeout.

This is a useful production warning: raising max_tokens does not guarantee a deliverable answer. A gateway should inspect finish reason, visible content, and timeout state together.

Where GLM-5.3 was stronger

On the Simpson task, GLM-5.3 correctly recovered the reversal:

Small stones: A 81/87 > B 234/270
Large stones: A 192/263 > B 55/80
Crude total: A 273/350 < B 289/350
50/50 standardized rates: A=0.8305, B=0.7771

It also separated standardization from causal identification, noting the need for randomization or adequate adjustment, positivity, no unmeasured confounding, and uncertainty estimates.

The minimal-UNSAT task required exactly one JSON object. GLM-5.3 returned [1,2,3] and correctly proved that the constraints force C and D into the same position. The response parsed without cleanup.

The physics task required switching system boundaries after the hanging mass hit the ground and the rope went slack. GLM-5.3 matched all four reference values:

a ≈ 0.847 m/s²
v1 ≈ 1.59 m/s
v2 ≈ 1.18 m/s
x ≈ 0.0835 m

Where GLM-5.2 won

The code task asked for optimize_release_plan(items, capacity_by_day, dependencies) with transitive dependencies, per-day capacities, value maximization, risk and lexical tie-breaks, invalid references, and cycle detection.

GLM-5.2 returned a complete Python file at the 16k budget and passed the external hidden test suite without edits. GLM-5.3 returned no body at 16k. At 24k it finally produced code, but selected a plan worth 24 instead of the correct optimum worth 29, so the retry still failed.

This is why code generation must be executed. Version numbers, code length, and a polished explanation are not substitutes for tests.

OpenAI-compatible request

import os
import requests

r = requests.post(
    "https://cn.crazyrouter.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['CRAZYROUTER_API_KEY']}"},
    json={
        "model": "glm-5.3",
        "messages": [{"role": "user", "content": "Your acceptance task"}],
        "temperature": 0.2,
        "max_tokens": 6000,
    },
    timeout=600,
)

choice = r.json()["choices"][0]
assert choice["finish_reason"] == "stop"
assert choice["message"]["content"].strip()

Production guidance

Prefer GLM-5.3 for causal analysis, constraint reasoning, and multi-stage derivations based on this run.
Keep GLM-5.2 in the candidate pool for executable algorithm generation.
Treat HTTP 200 + empty content as a business failure.
Parse and validate JSON instead of accepting visually plausible output.
Run generated code in an isolated test harness.
Track first-delivery success separately from retry-assisted success.

FAQ

Is GLM-5.3 universally better?

No. It had better first-run coverage, but GLM-5.2 won the executable code task.

Did more output budget fix the failures?

Not reliably. GLM-5.2 physics still returned an empty body at 20k. GLM-5.3 math remained empty at 20k, and its 24k code retry failed hidden tests.

Why count HTTP 200 responses as failures?

Because multiple responses consumed the entire reasoning budget and delivered no usable content.

How was code graded?

The original generated file was imported and executed against the same hidden harness. No manual fixes were allowed.

Was latency part of the score?

No. It was kept as operational evidence only.

What is the practical model-selection rule?

Route by task type and enforce real acceptance tests. A single aggregate model ranking loses the most important result from this benchmark.

Verdict

GLM-5.3 won first-delivery coverage, 4/6 to 2/6. GLM-5.2 delivered the best executable algorithm. The defensible production conclusion is task-aware routing with strict validation, not automatic replacement of the older model.

Run the same models through Crazyrouter's OpenAI-compatible API.

Claude Opus 5 vs GPT-5.6 Luna：速度順位を使わない18問の正確性ベンチマーク

Jenny Met — Thu, 30 Jul 2026 00:50:02 +0000

先に結論

高難度ラウンドでは Claude Opus 5 が 17/18（94.4%）、GPT-5.6 Luna が 16/18（88.9%）でした。差は1問であり、Opusが常に上という証明ではありません。重要なのは失敗の形です。Opusは3本のアルゴリズムをそのまま実行できる形で納品した一方、厳密JSONで余計な囲みを付けました。Lunaは3つの形式指示をすべて守りましたが、2本のPythonファイルがimport時に停止しました。

なぜ問題を難しくしたのか

最初の8問は両モデルとも8/8で、差が出ませんでした。そこで第2ラウンドを18問に増やし、6分野それぞれに hard、harder、extreme を配置しました。正確な分数、多段階の物理モデル、完全なPythonファイル、誤った前提の拒否、一意な制約解、文字単位の出力形式を要求しています。説明が巧みかどうかではなく、決定的な主張を計算、JSONパース、Python実行で確認できるかを評価しました。

18問をどう採点したか

同一の問題文、system指示、temperature、問題別の出力予算を使いました。数学と物理は厳密値または数値許容差で検証し、コードは回答のまま保存して共通のhidden testへimportしました。JSONとCSVは「意味が通る」ではなく、機械入力として合格するかで判定しています。モデル自身のassertを削除すれば動く場合も、原因分析には使いますが初回納品を合格へ変更しません。回線遅延は生JSONにだけ残し、勝敗には一切使っていません。

6分野の結果

評価分野	Opus 5	GPT-5.6 Luna
数学的推論	3/3	3/3
複雑な物理	3/3	3/3
アルゴリズム納品	3/3	1/3
誤った前提への耐性	3/3	3/3
制約推論	3/3	3/3
指示遵守	2/3	3/3
Total	17/18 (94.4%)	16/18 (88.9%)

アルゴリズム差は「納品物」の差だった

Lunaが失敗した2問は、末尾の自己テストを外すと関数本体がhidden testを通過しました。しかし自己テストの期待値が誤っており、元ファイルはimport時にAssertionErrorを起こします。「完全なPythonファイル」という契約では失敗です。Opusの3ファイルは元のままimportと共通テストに通りました。本番では、正しそうな関数と、そのまま利用できる成果物を分けて考える必要があります。

厳密JSONではLunaが優位

Opusの唯一の失敗は逆方向でした。JSONのキーと値は正しいのに、「JSONオブジェクトだけ」という指示に反してMarkdownのコードフェンスを追加しました。独立再試行でも同じです。Lunaは指示遵守3問をすべて通過しました。出力を直接パーサーへ渡す処理では、囲みの有無も正確性の一部です。

出力予算と知能上の誤りを分離

初期試行の一部は、内部推論が出力予算を使い切りfinish_reason=lengthになりました。これは証拠として保存しましたが、知能の誤りには数えていません。両モデルの有効予算を同条件で増やしてから採点したためです。ただし、長い推論で可視成果物の領域が不足するという統合上のリスクは残ります。

本番ルーティングへの落とし込み

厳密JSON、CSV、短い抽出、大量の構造化処理はLunaを第一候補にしつつ、schema検証を必須にします。長いアルゴリズム、境界条件、防御的な実装はOpusを第一候補にします。数学、複雑な物理、誤前提、制約推論は本サンプルで同点なので、価格、API互換性、文章量で選ぶ方が合理的です。どちらでもJSONをparseし、数値を照合し、コードを回答のまま実行してください。

再現方法と生データ

実行スクリプトはscripts/opus5_vs_luna_hard_benchmark.py、全結果は.tmp/opus5-vs-luna-hard-benchmark-results.jsonです。独立再試行は.tmp/opus5-hard-failure-retry.jsonと.tmp/luna-hard-failure-retry.jsonにあります。APIはhttps://cn.crazyrouter.com/v1/chat/completionsを使用し、遅延値は生データでのみ確認できます。

よくある質問

17/18ならOpus 5が総合勝者ですか？

いいえ。この18問では1問リードした、という意味です。画像、ツール利用、超長文脈、複数ターンのagentは対象外です。

誤った自己テストをアルゴリズム失敗に含める理由は？

要求が完全なPythonファイルだからです。元ファイルがimportで落ちるなら、人手で直さずに利用できません。

なぜ速度を結論に入れないのですか？

遅延はモデルだけでなく、ゲートウェイ、上流負荷、経路状態にも左右されます。運用証拠としては有用でも、知能の正確性とは別です。

最終評価

検証可能な正確性を優先した本テストでは、アルゴリズム3問を完全納品したOpus 5が 17/18 対 16/18で1問リードしました。一方、厳密な指示遵守はLunaが上です。結論は万能の勝者ではなく、再現可能な失敗傾向に基づく使い分けです。

Crazyrouterで両モデルを実際のプロンプトに通して比較する

Claude Opus 5 frente a GPT-5.6 Luna: 18 pruebas verificables, sin ranking de velocidad

Jenny Met — Thu, 30 Jul 2026 00:45:32 +0000

Respuesta rápida

En la ronda difícil, Claude Opus 5 obtuvo 17/18 (94,4%) y GPT-5.6 Luna 16/18 (88,9%). Es una ventaja de una tarea en esta muestra, no una prueba de superioridad universal. Lo importante es el tipo de fallo: Opus entregó correctamente los tres archivos algorítmicos, pero rompió un contrato de JSON estricto; Luna cumplió las tres instrucciones de formato, aunque dos archivos de Python fallaron al importarse.

Por qué fue necesario subir la dificultad

La primera ronda de ocho tareas fue demasiado sencilla para separar a los modelos: ambos consiguieron 8/8. La segunda pasó a 18 tareas, con variantes hard, harder y extreme en seis categorías. Se exigieron fracciones exactas, modelos físicos por etapas, archivos ejecutables completos, rechazo de premisas engañosas, soluciones lógicas únicas y disciplina de salida carácter por carácter. Una explicación convincente no bastaba: cada afirmación decisiva debía poder verificarse con cálculo, parsing de JSON o ejecución de Python.

Cómo se puntuaron las 18 tareas

Los dos modelos recibieron el mismo enunciado, instrucción de sistema, temperature y presupuesto por tarea. Matemáticas y física se validaron contra valores exactos o tolerancias numéricas. El código se guardó tal como llegó y se importó en el mismo harness de pruebas ocultas. JSON y CSV se evaluaron como artefactos para máquinas. Que un archivo funcione después de borrar sus propios assert ayuda a diagnosticar el fallo, pero no convierte la entrega original en un aprobado. La latencia de ruta se conserva solo en el JSON original y no participa en ninguna decisión de ganador.

Resultados en seis dimensiones

Dimensión	Opus 5	GPT-5.6 Luna
Razonamiento matemático	3/3	3/3
Física compleja	3/3	3/3
Entrega de algoritmos	3/3	1/3
Rechazo de premisas falsas	3/3	3/3
Razonamiento con restricciones	3/3	3/3
Seguimiento de instrucciones	2/3	3/3
Total	17/18 (94.4%)	16/18 (88.9%)

La diferencia algorítmica fue una diferencia de entrega

En los dos fallos de Luna, las funciones principales superaron las pruebas ocultas después de eliminar los assert del final. El problema era que esos assert contenían resultados esperados incorrectos y el archivo original lanzaba AssertionError durante el import. Bajo un contrato de “archivo Python completo”, son fallos reales. Los tres archivos originales de Opus se importaron y superaron el mismo harness. En producción, una función plausible no equivale a un artefacto listo para integrar.

En JSON estricto, la ventaja cambió de lado

El único fallo de Opus fue casi el reflejo contrario. Las claves y los valores eran correctos, pero el modelo envolvió el objeto en un bloque Markdown pese a que se pedía exactamente un objeto JSON, sin nada más. El comportamiento se repitió en un segundo intento independiente. Luna aprobó las tres pruebas de instrucciones. Cuando la salida entra directamente en un parser, el envoltorio también forma parte de la corrección.

El presupuesto de salida se separó de los errores de inteligencia

Algunos intentos iniciales terminaron con finish_reason=length porque el razonamiento oculto consumió el presupuesto de salida. Se conservaron como evidencia, pero no se contaron como errores de inteligencia. Antes de puntuar se aumentó de forma equivalente el presupuesto efectivo de ambos modelos. Así se evita confundir configuración con razonamiento, sin ocultar el riesgo de integración de una respuesta que deja poco espacio al artefacto visible.

Cómo convertir los resultados en reglas de routing

Empieza con Luna para JSON estricto, CSV, extracción corta y trabajo estructurado de gran volumen, manteniendo siempre validación de esquema. Empieza con Opus para archivos algorítmicos largos, casos límite e ingeniería defensiva. En matemáticas, física compleja, rechazo de premisas falsas y restricciones hubo empate; en esas áreas, coste, compatibilidad y estilo de salida son mejores criterios. Con ambos modelos hay que validar el artefacto: parsear JSON, comprobar números clave y ejecutar el código sin limpiarlo a mano.

Reproducción y evidencia original

El runner es scripts/opus5_vs_luna_hard_benchmark.py. El resultado completo está en .tmp/opus5-vs-luna-hard-benchmark-results.json; los reintentos independientes están en .tmp/opus5-hard-failure-retry.json y .tmp/luna-hard-failure-retry.json. Se usó el endpoint limpio https://cn.crazyrouter.com/v1/chat/completions. Las latencias permanecen únicamente en los archivos de evidencia.

Preguntas frecuentes

¿17/18 convierte a Opus 5 en ganador absoluto?

No. Solo significa que lideró por una tarea en este conjunto. No se evaluaron visión, herramientas, contexto ultralargo ni agentes de varios turnos.

¿Por qué cuentan como fallo unas autopruebas incorrectas?

Porque se pidió un archivo Python completo. Si el original falla al importarse, un sistema posterior no puede usarlo sin reparación manual.

¿Por qué la latencia no aparece en el veredicto?

Depende del modelo, el gateway, la carga upstream y el estado de la ruta. Es evidencia operativa útil, pero no mide la precisión intelectual.

Veredicto final

En esta prueba centrada en precisión verificable, Opus 5 lideró 17/18 a 16/18 gracias a la entrega completa de los tres algoritmos. Luna fue mejor en cumplimiento estricto de formato. La conclusión práctica no es un campeón universal, sino una regla de routing basada en fallos reproducibles.

Prueba ambos modelos en Crazyrouter con tus propios prompts de producción

Claude Opus 5 vs GPT-5.6 Luna: 18 Verifiable Tasks, No Speed Leaderboard

Jenny Met — Thu, 30 Jul 2026 00:42:45 +0000

Quick answer

Claude Opus 5 scored 17/18 (94.4%) and GPT-5.6 Luna scored 16/18 (88.9%) on the harder round. That is a one-task lead in this sample—not proof that Opus is universally stronger. The useful result is the shape of the failures: Opus delivered all three algorithm files successfully but broke a strict JSON contract; Luna followed all three formatting instructions but shipped two algorithm files that failed during import.

Why the benchmark had to become harder

The first eight tasks were too easy to separate the models: both scored 8/8. The second round therefore used 18 tasks, with hard, harder, and extreme variants in each of six categories. The questions required exact fractions, multi-stage physical modeling, executable files, resistance to misleading assumptions, unique logical solutions, and byte-level output discipline. The goal was not to reward impressive prose. Every decisive claim had to be checked by deterministic rules, independent calculations, JSON parsing, or Python execution.

How the 18-task round was scored

Both models received the same task text, system instruction, temperature, and task-level output budget. Mathematical and physics answers were checked against exact values or numerical tolerances. Algorithm responses were saved as delivered and imported into the same hidden-test harness. Strict JSON and CSV tasks were validated as artifacts, not read charitably. A file that only works after deleting model-written assertions is useful diagnostic evidence, but it is not a successful first delivery. Route latency was retained in the raw JSON for infrastructure diagnosis and was excluded from every winner decision in this article.

Results across six dimensions

Dimension	Opus 5	GPT-5.6 Luna
Mathematical reasoning	3/3	3/3
Complex physics	3/3	3/3
Algorithm delivery	3/3	1/3
False-premise resistance	3/3	3/3
Constraint reasoning	3/3	3/3
Instruction following	2/3	3/3
Total	17/18 (94.4%)	16/18 (88.9%)

The algorithm gap was a delivery gap

Luna's two failed algorithm answers contained core functions that passed hidden tests after their own bottom-of-file assertions were removed. The problem was that the assertions encoded incorrect expected values, so importing the original files raised AssertionError before a caller could use the functions. Under a strict complete-file contract, those are failures. Opus's three original algorithm files imported and passed the shared harness. This distinction matters in production: a plausible function body is not the same thing as a deployable artifact.

The strict-JSON failure points in the other direction

Opus's only hard-round failure was almost the mirror image. The JSON keys and values were correct, but the model wrapped the object in a Markdown code fence even though the instruction required exactly one JSON object and nothing else. The same behavior reappeared in the independent retry. Luna returned valid strict output in all three instruction-following tasks. If your pipeline feeds model output directly into a parser, wrapper discipline is part of correctness, not a cosmetic preference.

Output budgets were separated from intelligence errors

Some early responses ended with finish_reason=length because hidden reasoning consumed the available output budget. Those attempts were preserved, but they were not counted as intelligence failures. The same effective budget was increased for both models before scoring. This avoids confusing capacity configuration with reasoning quality while still documenting a real integration risk: long reasoning can leave too little room for the visible artifact.

What this means for production routing

Use Luna first for strict JSON, CSV, short extraction, and high-volume structured work—but keep schema validation. Use Opus first for long algorithm files, defensive engineering, and tasks where the complete executable artifact matters more than brevity. For math, complex physics, false-premise resistance, and constraint reasoning, this sample was a tie, so cost, interface compatibility, and output style are better routing variables. In every case, validate the task artifact: parse JSON, check key numbers, and run generated code exactly as received.

Reproduction and raw evidence

The benchmark runner is scripts/opus5_vs_luna_hard_benchmark.py. The complete hard-round result is .tmp/opus5-vs-luna-hard-benchmark-results.json; independent failure retries are in .tmp/opus5-hard-failure-retry.json and .tmp/luna-hard-failure-retry.json. The API endpoint used was https://cn.crazyrouter.com/v1/chat/completions. Latency fields remain in those raw files for route diagnosis only.

FAQ

Does 17/18 make Opus 5 the overall winner?

No. It gives Opus a one-task lead on this specific 18-task set. The sample does not cover vision, tools, long-context retrieval, or multi-turn agents.

Why count faulty self-tests as algorithm failures?

Because the requested deliverable was a complete Python file. If the original file crashes on import, downstream systems cannot use it without manual repair.

Why is latency absent from the verdict?

Latency reflects the model, gateway route, upstream load, and channel state. It is valuable operational evidence but not a reliable measure of intelligence correctness.

Final verdict

In this correctness-first benchmark, Opus 5 led 17/18 to 16/18 because it delivered all algorithm artifacts successfully. Luna was stronger at strict instruction compliance. The practical conclusion is not a universal champion; it is a routing rule backed by reproducible failure modes.

Run both models through Crazyrouter with your own production prompts

Claude Opus 5 vs GPT-5.6-SOL: 프로덕션 QA로 검증한 10개 과제, 핵심 정확도는 동률

Jenny Met — Thu, 30 Jul 2026 00:25:45 +0000

Claude Opus 5 vs GPT-5.6-SOL: 프로덕션 QA로 검증한 10개 과제, 핵심 정확도는 동률

모델을 프로덕션에 투입할 때는 최종 답만 맞는다고 충분하지 않다. 응답이 중간에 잘리지 않았는지 finish_reason을 확인해야 하고, 생성된 코드는 실제 숨은 테스트를 통과해야 하며, JSON은 설명이나 코드 블록 없이 바로 파싱할 수 있어야 한다.

이번 비교에서는 Crazyrouter의 동일한 OpenAI-compatible endpoint를 통해 Claude Opus 5와 GPT-5.6-SOL에 10개의 지능 과제를 수행하게 했다. 별도로 엄격한 JSON 출력 과제를 추가해 지시 준수와 구조화 출력 안정성도 확인했다.

평가는 다음 질문에 초점을 맞췄다.

수학 답과 물리 모델이 정확한가?
잘못된 전제나 불충분한 통계적 결론을 식별하는가?
생성한 Python 코드가 숨은 테스트를 통과하는가?
문제에 포함된 모든 하위 요구 사항을 빠짐없이 완료하는가?
JSON 응답을 별도 정제 없이 바로 파싱할 수 있는가?
finish_reason이 정상 종료를 나타내는가?

결과는 단순한 승패보다 프로덕션 QA 관점에서 해석할 가치가 있다.

핵심 답변 정확도: Opus 5 10/10, GPT-5.6-SOL 10/10
모든 하위 요구 사항까지 완전히 완료: Opus 5 9/10, GPT-5.6-SOL 10/10
Python 알고리즘 과제의 숨은 테스트: 두 모델 모두 2/2
엄격한 JSON 최초 제출: Opus 5 0/1, GPT-5.6-SOL 1/1
Opus 5는 이후 두 번의 추가 재시도에서도 JSON 외의 텍스트를 덧붙여 형식 요구를 충족하지 못했다.

따라서 이번 결과는 핵심 정확도는 동률이지만, 완전한 과제 납품과 구조화 출력의 안정성은 동일하지 않았다고 정리하는 것이 정확하다.

API Key를 만들고 실제 업무 문제로 재검증하기

빠른 답변

질문	이번 테스트 결과
핵심 지능 과제의 답은 어느 쪽이 더 정확했나?	동률: 두 모델 모두 10/10
모든 하위 요구 사항을 더 많이 완료한 모델은?	GPT-5.6-SOL 10/10, Opus 5 9/10
코드 신뢰성은 어느 쪽이 더 높았나?	이번에는 동률. 두 알고리즘 과제 모두 숨은 테스트 통과
Opus 5가 어려운 확률 문제를 틀렸나?	아니다. 핵심 답은 정확했지만 `max_tokens=3200`에서 출력이 잘려 마지막 설명 요구를 누락했다.
엄격한 JSON 지시를 더 안정적으로 지킨 모델은?	이번에는 GPT-5.6-SOL. Opus 5는 최초 응답과 추가 재시도 2회, 총 3회 모두 추가 텍스트 또는 코드 블록을 포함했다.
이 결과로 GPT가 전반적으로 더 똑똑하다고 할 수 있나?	아니다. 핵심 추론 정확도와 형식·지시 준수는 분리해 평가해야 한다.
응답 속도도 평가했나?	아니다. 네트워크와 라우팅 영향을 분리할 수 없어 지능 점수에서 제외했다.

수학·물리·알고리즘의 핵심 정확성이 중요한 업무라면 이번 표본에서 두 모델 간 뚜렷한 격차는 없었다. 반면 응답 본문이 반드시 순수 JSON이어야 하거나, 단 한 번의 응답에서 모든 하위 요구 사항을 완료해야 한다면 이번 테스트에서는 GPT-5.6-SOL의 납품 완성도가 더 높았다.

네트워크 지연 시간을 지능 점수에서 제외한 이유

모델 API의 종단 간 응답 시간에는 모델의 추론 능력 외에도 여러 변수가 개입한다.

게이트웨이에서 각 상위 공급자까지의 네트워크 경로
요청 시점의 채널 부하와 계정별 속도 제한
캐시 적중 여부
실제로 요청을 처리한 서비스 인스턴스
상위 공급자가 reasoning token을 집계하거나 숨기는 방식

단일 요청이 몇 초 더 빨랐다는 사실은 모델의 추론 능력이 더 우수하다는 증거가 아니다. 그것은 네트워크 상태나 라우팅 결과, 순간적인 서버 부하를 측정한 것일 수 있다.

속도를 비교하려면 최소한 동일한 상위 공급자와 실행 조건을 고정하고, 충분한 횟수로 반복한 뒤 분산과 신뢰구간을 함께 보고해야 한다. 이번 테스트에는 그런 조건이 마련되지 않았으므로 네트워크 지연 시간을 지능 점수에서 명시적으로 제외했다.

따라서 이 글은 어떤 모델도 속도 승자로 선정하지 않는다. 지연 시간은 운영 관측 지표로는 유용하지만, 이번과 같은 지능 정확도 평가에 혼합하면 결과를 왜곡할 수 있다.

현재 제공되는 모델과 접속 범위는 Crazyrouter 모델 목록에서 확인할 수 있다. 비용 계획은 지능 평가와 분리해 Crazyrouter pricing을 참고해야 한다.

테스트 환경과 채점 기준

테스트 전 모델 목록 API를 호출해 정확한 모델 ID 두 개가 노출되는지 확인했다.

GET https://cn.crazyrouter.com/v1/models

claude-opus-5
gpt-5.6-sol

실제 요청에는 다음 endpoint를 공통으로 사용했다.

POST https://cn.crazyrouter.com/v1/chat/completions

각 테스트 배치에서 두 모델에 동일한 system prompt, user prompt, temperature, max_tokens를 적용했다. 외부 도구는 사용하지 않았으며, HTTP 200 응답만으로 과제 통과를 판정하지 않았다.

채점은 세 단계로 나눴다.

핵심 답변 정확도

주요 수치, 결론, 알고리즘 동작이 정확한지 평가했다.
완전한 과제 통과율

문제에 명시된 모든 하위 요구 사항을 빠짐없이 완료했는지 확인했다.
기계 검증 가능성

코드는 숨은 테스트를 통과하는지, JSON은 별도 정제 없이 직접 파싱되는지 검사했다.

자동 문자열 채점 후에는 사람이 다시 검토했다. LaTeX 표현, 대소문자, 정확한 분수와 반올림된 소수의 차이 때문에 거짓 음성이 발생할 수 있기 때문이다.

예를 들어 GPT-5.6-SOL은 확률 문제에서 기준 답안의 분수 표현을 그대로 복사하지 않았지만 동치인 수식과 정확한 소수를 제시했다. Opus 5는 물리 문제의 0.302 m를 유효숫자 두 자리인 0.30 m로 썼지만, 이를 오답으로 볼 근거는 없다.

10개 지능 과제 결과

과제	검증 기준	Claude Opus 5	GPT-5.6-SOL
정확한 마르코프 연쇄	`E[τ]=5`, `E[τ²]=43`, `Var(τ)=18`	정확	정확
2자유도 진동자	고유진동수 2개, 진폭 2개, 위상 2개	정확	정확
제약 조건 탐색	유일한 순서 `A,C,E,B,D`	정확	정확
Cantelli 오류 수정	확률은 식별 불가능하며 상한은 `0.2`	정확	정확
Python 순환 탐지 검토	bug, DAG 반례, 최소 수정	정확	정확
실험 설계	동일 문제 짝짓기, 난이도 통제, 신뢰구간	정확	정확
편향 동전의 `HHTH` 대기 시간	기댓값 약 `12.6547`과 올바른 상태 전이	핵심 답은 정확하나 출력이 잘려 마지막 설명 누락	정확하고 완전함
로그 집계 알고리즘	시간 구간, 실패 이벤트, 캐시 비율, 상위 사용자	숨은 테스트 통과	숨은 테스트 통과
비탄성 충돌과 용수철	`v1≈6.10`, `v2≈2.44`, `x≈0.302`	정확	정확
안정적 라우팅 알고리즘	cost, latency, reliability와 사전식 순서 규칙	숨은 테스트 통과	숨은 테스트 통과

이 표에서 가장 중요한 구분은 핵심적으로 맞았는가와 요구된 결과를 완전히 납품했는가가 서로 다른 평가 항목이라는 점이다.

Opus 5는 10개 과제의 핵심 답을 모두 맞혔다. 그러나 확률 문제에서 마지막 설명 하나가 누락되어 완전한 과제 통과율은 9/10이었다. GPT-5.6-SOL은 핵심 정확도와 완전한 과제 통과율이 모두 10/10이었다.

가장 어려운 확률 문제: `12.6547`은 맞았지만 응답은 완전하지 않았다

문제의 목표 패턴은 HHTH이며 편향 동전의 확률은 다음과 같다.

P(H)=0.62
P(T)=0.38

문제는 최장 접두사·접미사 일치 상태를 사용해 방정식을 세우고, 다음 두 가지 함정을 명시적으로 설명하도록 요구했다.

상태 HH에서 다시 H가 나왔을 때 왜 상태가 여전히 HH인가?
기대 대기 시간을 왜 단순히 1/P(HHTH)로 계산할 수 없는가?

두 모델 모두 올바른 기대 대기 시간을 계산했다.

E[N] ≈ 12.6547

GPT-5.6-SOL은 전체 유도를 완료했고, 패턴에 중첩이 존재하므로 다음 관계가 성립한다고 설명했다.

E[N] = 1 / P(HHTH) + 1 / P(H)

Opus 5도 올바른 상태 정의, 방정식, 정확한 분수와 소수 결과를 제시했다. 하지만 응답의 최종 상태는 finish_reason=length였다. 출력이 부가적인 상태별 기대값을 설명하던 중 종료되면서, 마지막 요구 사항인 “왜 확률의 역수를 직접 사용하면 안 되는가”에 대한 설명이 빠졌다.

이때 적용해야 할 채점 원칙은 명확하다.

최종 수치가 정확하면 핵심 답변 정확도에는 포함할 수 있다. 그러나 문제의 모든 요구 사항을 완료하지 않았다면 완전한 과제 통과로 처리할 수 없다.

이 응답에는 max_tokens=3200이 적용됐다. 따라서 프로덕션에서는 답변 본문만 저장해서는 안 된다. 최소한 원문 응답, finish_reason, 사용한 출력 예산을 함께 기록해야 한다. 앞부분이 정확하다는 이유만으로 전체 작업이 완료됐다고 판단해서는 안 된다.

이는 기존의 max_tokens 잘림 재검증에서도 확인할 수 있는 동일한 방법론적 문제다.

실행 가능한 코드 검증: 코드 모양이 아니라 숨은 테스트로 평가

코드 생성 결과는 길이, 주석의 양, 함수 분리 방식만 보고 평가하기 어렵다. 보기 좋은 코드가 반드시 정확한 코드는 아니며, 짧은 구현도 모든 경계 조건을 만족할 수 있다.

이번 테스트에서는 두 모델이 생성한 출력을 각각 .py 파일로 저장하고 격리 모드에서 동일한 테스트를 실행했다.

로그 집계 알고리즘

로그 집계 함수는 다음 조건을 모두 처리해야 했다.

반개방 시간 구간
성공 요청과 실패 요청에 서로 다른 집계 규칙 적용
사용자 또는 모델 값이 누락된 이벤트
cache hit rate
cost가 같을 때 사용자 이름의 사전식 순서 적용
입력 객체를 변경하지 않는 불변성

두 모델의 구현 모두 숨은 테스트를 통과했다.

신뢰성 제약이 있는 안정적 라우팅

두 번째 함수는 경로 탐색 과정에서 다음 조건을 함께 처리해야 했다.

총비용이 가장 낮은 경로 선택
비용이 같으면 latency가 가장 낮은 경로 선택
latency까지 같으면 reliability가 가장 높은 경로 선택
모든 조건이 같으면 경로의 사전식 순서로 선택
banned nodes 처리
최대 hops 제한
최소 reliability 제한
잘못된 edge 처리

두 모델 모두 이 과제의 숨은 테스트를 통과했다.

따라서 코드 부문의 결과는 두 모델 모두 2/2로 동률이다. 코드가 더 길거나 설명이 더 상세하다는 이유로 별도의 우승 모델을 정하지 않았다.

GPT-5.6-SOL의 다른 고난도 물리·의존성 알고리즘 결과는 GPT-5.6-SOL vs GPT-5.5 고난도 물리 및 코드 테스트에서 확인할 수 있다.

엄격한 JSON: 계산 정확도와 지시 준수를 분리해서 봐야 한다

별도의 엄격한 JSON 과제에서는 사고 데이터를 하나의 객체로 압축하되 다음 조건을 지키도록 요구했다.

exactly one JSON object and no Markdown

두 모델 모두 실패율, 장애 책임 채널, 재시도 후 복구되지 않은 요청 수를 정확하게 계산했다. 차이는 JSON 바깥의 출력에서 발생했다.

GPT-5.6-SOL은 첫 시도에 JSON만 반환했고 json.loads로 직접 파싱할 수 있었다.
Opus 5는 첫 시도에 코드 블록과 Verification을 추가했다.
Opus 5의 첫 번째 추가 재시도에서는 압축된 JSON 뒤에 Verification을 덧붙였다.
Opus 5의 두 번째 추가 재시도에서는 다시 코드 블록과 설명을 추가했다.

수치로 정리하면 엄격한 JSON 최초 제출은 GPT-5.6-SOL 1/1, Opus 5 0/1이다. Opus 5는 그 뒤 두 번 더 재시도했지만 계속 형식 요구를 충족하지 못했다.

이는 “계산을 못했다”는 의미가 아니다. Opus 5가 생성한 필드와 값은 정확했다. 실패 원인은 데이터 오류가 아니라 출력 형식과 지시 준수다.

이 차이를 핵심 지능 정확도에 섞어 하나의 총점으로 만들면 다음 두 문제를 구분할 수 없게 된다.

모델이 올바른 값을 계산했는가?
모델이 지정된 구조만 출력했는가?

프로덕션에서는 최소한 다음과 같은 로컬 검증기를 둘 수 있다.

import json

REQUIRED_KEYS = [
    "window",
    "total_requests",
    "failed_requests",
    "failure_rate_pct",
    "provider_owned_failures",
    "customer_owned_failures",
    "recovered_by_retry",
    "unrecovered_failures",
    "root_cause",
    "action",
]


def parse_incident_json(raw: str) -> dict:
    payload = json.loads(raw)
    if list(payload) != REQUIRED_KEYS:
        raise ValueError("unexpected schema or key order")
    if payload["failed_requests"] != 84:
        raise ValueError("failed request count mismatch")
    return payload

system prompt만으로 구조화 출력을 보장할 수는 없다. 더 안전한 구성은 다음 세 요소를 함께 적용하는 것이다.

공급자가 지원하는 구조화 출력 매개변수 사용
응답에 대한 로컬 schema 검증
파싱 또는 검증 실패 시 재시도하거나 다른 모델로 전환

즉, 구조화 출력은 프롬프트 작성만의 문제가 아니라 애플리케이션 계층의 검증 문제다.

같은 API로 재검증하는 방법

다음은 OpenAI-compatible chat completions endpoint를 사용하는 최소 실행 예제다. API endpoint 자체에는 UTM 매개변수를 추가하지 않는다.

import os
import requests

BASE_URL = "https://cn.crazyrouter.com/v1"
API_KEY = os.environ["CRAZYROUTER_API_KEY"]


def ask(model: str, prompt: str, max_tokens: int = 4000) -> dict:
    response = requests.post(
        f"{BASE_URL}/chat/completions",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": model,
            "messages": [
                {"role": "system", "content": "Answer accurately and follow every requested constraint."},
                {"role": "user", "content": prompt},
            ],
            "temperature": 0.2,
            "max_tokens": max_tokens,
        },
        timeout=600,
    )
    response.raise_for_status()
    return response.json()


for model in ("claude-opus-5", "gpt-5.6-sol"):
    result = ask(model, "Your benchmark prompt here")
    choice = result["choices"][0]
    print(model, choice.get("finish_reason"), choice["message"].get("content", ""))

실제 회귀 테스트에서는 답변 텍스트뿐 아니라 다음 항목도 함께 저장하는 편이 좋다.

response ID
returned model
원본 응답
finish_reason
요청에 사용한 max_tokens
로컬 검증 결과
코드 테스트 또는 schema 검증의 실패 원인

현재 모델이 노출되는지는 Crazyrouter 모델 목록에서 확인한 뒤, 실제 업무 prompt로 소규모 회귀 테스트를 진행할 수 있다.

프로덕션 도입을 위한 QA 지침

두 모델 모두 후보로 고려할 수 있는 작업

명확한 기준 답안이 있는 수학·물리 문제
단위 테스트를 실행할 수 있는 Python 알고리즘
잘못된 전제나 불충분한 통계 결론을 식별하는 작업
로컬 검증기로 성공 여부를 판정할 수 있는 업무 흐름

이번 10개 과제에서 두 모델 모두 핵심 정확도 10/10을 기록했고, 코드 과제의 숨은 테스트도 각각 2/2를 통과했다. 따라서 검증 가능한 작업이라면 두 모델 모두 후보군에 포함할 근거가 있다.

이번 결과에서 GPT-5.6-SOL을 우선 검토할 수 있는 작업

모든 하위 요구 사항을 한 번의 응답에서 완료해야 하는 작업
응답 원문이 반드시 순수 JSON이어야 하는 연동
설명문에서 구조화 데이터를 다시 추출하는 후처리를 줄여야 하는 시스템

이 판단은 GPT-5.6-SOL의 핵심 추론 정확도가 더 높았다는 뜻이 아니다. 두 모델의 핵심 정확도는 동일한 10/10이었다. 차이는 이번 표본에서 확인된 완전한 과제 납품과 JSON 형식 준수에 있다.

Opus 5를 사용할 때 추가할 보호 장치

긴 유도 과정에는 충분한 max_tokens 할당
모든 응답에서 finish_reason 검사
JSON을 하위 시스템에 전달하기 전에 반드시 파싱
추가 Markdown, 코드 블록, 설명 텍스트가 포함됐을 때의 명시적 대체 경로 구성
잘림이나 schema 위반 시 재시도 또는 모델 전환
핵심 정답과 전체 요구 사항 완료 여부를 별도로 기록

Opus 5의 확률 문제는 핵심 계산이 틀린 사례가 아니다. max_tokens=3200 경계에서 finish_reason=length로 끝나면서 마지막 설명이 누락된 사례다. 이를 추론 오류로 분류하면 원인을 잘못 진단하게 된다.

Claude 계열 내부의 납품 특성을 추가로 비교하려면 Claude Opus 5 vs Claude Fable 5 실제 API 테스트를 참고할 수 있다.

프로덕션 체크리스트

모델을 실제 시스템에 연결할 때는 단일 “정답률” 대신 다음 항목을 개별적으로 수집하는 것이 좋다.

[ ] 핵심 답변이 정확한가?
[ ] 모든 하위 요구 사항을 완료했는가?
[ ] finish_reason이 예상한 종료 상태인가?
[ ] 응답이 max_tokens 경계에서 잘리지 않았는가?
[ ] JSON이 json.loads로 직접 파싱되는가?
[ ] 필수 키, 자료형, 값 범위가 schema와 일치하는가?
[ ] 생성된 코드가 공개 테스트뿐 아니라 숨은 테스트도 통과하는가?
[ ] 입력 객체의 불변성과 경계 조건을 지키는가?
[ ] 실패 시 재시도 또는 모델 전환 경로가 있는가?
[ ] 네트워크 지연 시간과 지능 정확도를 별도 지표로 기록하는가?

FAQ

Claude Opus 5와 GPT-5.6-SOL 중 어느 모델이 더 똑똑한가?

이번 10개 과제에서는 두 모델 모두 핵심 답변 정확도 10/10을 기록했다. 따라서 이 결과만으로 어느 한쪽이 전반적으로 더 똑똑하다고 결론 내릴 수 없다.

완전한 과제 통과율은 왜 동률이 아닌가?

Opus 5의 고난도 확률 문제 응답이 max_tokens=3200에서 잘렸기 때문이다. 기대값 12.6547과 상태 방정식은 정확했지만, 마지막 설명 요구를 완료하지 못했고 finish_reason=length가 반환됐다. 따라서 핵심 정확도에는 포함되지만 완전한 과제 통과에는 포함되지 않는다.

핵심 정확도와 완전한 과제 납품은 어떻게 다른가?

핵심 정확도는 주요 수치, 결론, 알고리즘 동작이 맞는지를 평가한다. 완전한 과제 납품은 여기에 더해 문제에서 요구한 모든 설명, 형식, 하위 항목까지 빠짐없이 제공했는지를 평가한다. 정답을 계산했더라도 마지막 설명이나 필수 필드가 빠지면 완전한 납품은 아니다.

엄격한 JSON 실패를 지능 오류로 봐야 하나?

수학 또는 추론 오류로 직접 분류해서는 안 된다. Opus 5는 JSON의 필드와 값을 정확하게 계산했지만 JSON 외의 코드 블록이나 Verification 설명을 추가했다. 따라서 더 정확한 분류는 형식 및 지시 준수 실패다.

Opus 5는 JSON 재시도에서 성공했나?

아니다. 최초 응답은 엄격한 JSON 기준에서 0/1이었고, 이후 두 번의 추가 재시도에서도 JSON 뒤의 Verification 또는 코드 블록과 설명을 추가했다. 즉, 총 3회 모두 순수 JSON만 반환하라는 요구를 지키지 못했다.

두 코드 과제는 어떻게 평가했나?

각 모델의 출력을 Python 파일로 저장하고 동일한 숨은 테스트를 실행했다. 테스트에는 경계 조건, 정렬 규칙, 잘못된 입력, 입력 불변성 등이 포함됐다. 두 모델 모두 두 과제를 통과해 각각 2/2를 기록했다.

왜 응답 속도 승자를 발표하지 않았나?

종단 간 응답 시간은 네트워크 경로, 채널 부하, 캐시, 상위 공급자 상태, 라우팅된 인스턴스 등에 영향을 받는다. 동일한 상위 환경을 고정하고 충분히 반복하지 않은 단일 지연 시간은 지능 정확도를 나타내지 않는다. 그래서 이번 점수에서는 네트워크 지연 시간을 제외했으며 속도 승자를 정하지 않았다.

`finish_reason`을 반드시 검사해야 하는 이유는 무엇인가?

본문에 올바른 답이 포함돼 있어도 응답이 중간에 끝났을 수 있기 때문이다. 이번 Opus 5 사례처럼 finish_reason=length이면 핵심 답을 계산한 뒤에도 마지막 요구 사항이 누락될 수 있다. 프로덕션에서는 본문과 함께 finish_reason을 성공 판정 조건에 포함해야 한다.

10개 과제만으로 장기 모델을 선정해도 되나?

충분하지 않다. 이번 결과는 작지만 재현 가능한 능력 표본이다. 실제 도입 전에는 업무 문제를 20–50회 반복해 핵심 정확도, 완전한 과제 통과율, 코드 테스트 통과율, 형식 위반률을 각각 집계하는 것이 좋다.

자체 모델 비교는 어떻게 시작해야 하나?

실제 업무를 대표하는 prompt 10–30개를 먼저 선정하고, 각 문제에 기계적으로 실행 가능한 통과 기준을 정의해야 한다. 이후 동일한 API 조건으로 두 모델에 같은 입력을 보내고 결과를 비교한다. 결론을 먼저 정한 뒤 그 결론을 뒷받침하는 사례만 선택해서는 안 된다.

구조화 출력은 prompt만 잘 쓰면 보장할 수 있나?

보장할 수 없다. 명확한 system prompt와 user prompt는 필요하지만 충분하지 않다. 가능하면 공급자의 구조화 출력 기능을 사용하고, 로컬 schema 검증과 파싱 실패 시 재시도 또는 모델 전환을 함께 구성해야 한다.

최종 결론

이번 테스트의 핵심은 어느 한 모델을 단순 승자로 선언하는 것이 아니다. 프로덕션에서는 핵심적으로 맞는 답을 생성하는 능력과 요구한 형식과 모든 하위 항목을 완전하게 납품하는 능력을 분리해 검증해야 한다.

Claude Opus 5와 GPT-5.6-SOL은 10개 지능 과제에서 모두 핵심 답변 정확도 10/10을 기록했다. GPT-5.6-SOL은 모든 하위 요구 사항에서 10/10, 엄격한 JSON 최초 제출에서 1/1을 기록했다. Opus 5는 하위 요구 사항 완료에서 9/10, 엄격한 JSON 최초 제출에서 0/1이었으며 두 번의 추가 재시도에서도 형식 요구를 충족하지 못했다.

Opus 5의 차이는 핵심 추론 실패가 아니었다. 고난도 확률 문제에서는 정답 12.6547을 계산했지만 max_tokens=3200 경계에서 finish_reason=length로 종료돼 마지막 설명이 빠졌다. JSON 과제에서도 데이터는 정확했지만 출력 외피가 요구 형식과 달랐다.

따라서 기준 답안이나 숨은 테스트를 운영할 수 있는 업무라면 두 모델 모두 후보가 될 수 있다. 반면 하위 시스템이 모델의 JSON을 직접 소비한다면 다음 요소는 선택 사항이 아니라 필수 구성 요소로 봐야 한다.

finish_reason 검사
구조화 출력 schema 검증
코드의 실제 실행 및 숨은 테스트
출력 잘림 감지
형식 위반 시 재시도
필요할 때의 모델 전환

그리고 네트워크 지연 시간은 지능 점수와 분리해야 한다. 이번 테스트에는 속도 승자가 없으며, 확인된 차이는 핵심 정확도가 아니라 완전한 과제 납품과 구조화 출력 준수에 있다.

Crazyrouter 계정을 만들고 Opus 5 / GPT-5.6-SOL을 직접 테스트하기

Claude Opus 5 против GPT-5.6-SOL: проверка на 10 задачах — по точности основных ответов ничья

Jenny Met — Thu, 30 Jul 2026 00:25:29 +0000

Claude Opus 5 против GPT-5.6-SOL: проверка на 10 задачах — по точности основных ответов ничья

Что именно означает «модель умнее» — быстрее отвечает, правильно решает задачу или строго соблюдает формат?

В этом сравнении скорость намеренно не учитывается. Вместо нее оцениваются критерии, которые можно независимо проверить: правильность математических результатов, полнота физической модели, прохождение локальных тестов кода, распознавание ошибочных предпосылок и выполнение всех требований задания.

Обе модели получили одинаковые условия через один OpenAI-compatible endpoint Crazyrouter. Основной набор состоял из 10 задач; строгое соблюдение JSON проверялось отдельно.

Результаты:

точность основных ответов: Claude Opus 5 — 10/10, GPT-5.6-SOL — 10/10;
полное выполнение всех подпунктов: Claude Opus 5 — 9/10, GPT-5.6-SOL — 10/10;
скрытые тесты двух задач на Python: обе модели — 2/2;
строгий JSON с первой попытки: Claude Opus 5 — 0/1, GPT-5.6-SOL — 1/1;
в двух дополнительных попытках Opus по-прежнему добавлял к JSON посторонний текст или оформление.

Итог нельзя свести к одному общему баллу: по правильности основных рассуждений модели выступили одинаково, но по полноте и формату ответа — по-разному.

Создать API Key и повторить тестирование на собственных задачах

Краткий ответ

Вопрос	Результат теста
Какая модель точнее решила основные задачи?	Ничья: обе модели получили 10/10
Какая модель выполнила больше заданий целиком?	GPT-5.6-SOL — 10/10; Opus 5 — 9/10
Какая модель надежнее генерирует код?	В этом наборе ничья: обе прошли скрытые тесты двух алгоритмических задач, то есть 2/2
Ошибся ли Opus в сложной задаче по теории вероятностей?	Нет. Основной ответ правильный, но вывод был оборван при `max_tokens=3200`, поэтому последнее требуемое объяснение отсутствовало
Какая модель точнее соблюдает строгий JSON?	В этом тесте GPT-5.6-SOL: 1/1 с первой попытки против 0/1 у Opus; две дополнительные попытки Opus также не соответствовали формату
Доказывает ли это, что GPT в целом умнее?	Нет. Правильность рассуждений, полноту ответа и соблюдение формата следует оценивать раздельно

Если приоритет — проверяемая правильность математики, физики и алгоритмов, этот набор не выявил преимущества одной модели. Если ответ должен без дополнительной обработки разбираться как JSON или содержать каждый обязательный подпункт, GPT-5.6-SOL в данном тесте показал более стабильную доставку результата.

Почему задержка сети не входит в оценку интеллекта

Полное время ответа API зависит не только от самой модели. На него влияют:

сетевой маршрут от шлюза до вышестоящего сервиса;
текущая нагрузка на канал и ограничения учетной записи;
попадание в кеш;
выбор конкретного обслуживающего экземпляра;
способ учета или сокрытия reasoning token вышестоящим сервисом.

Разница в несколько секунд в одиночном запросе прежде всего характеризует текущую инфраструктуру и маршрутизацию. Она не доказывает более высокую точность рассуждений.

Чтобы содержательно сравнивать задержку, потребовалось бы зафиксировать вышестоящий сервис, выполнить достаточное число повторов и опубликовать распределения результатов с доверительными интервалами. Без этого задержка пригодна для эксплуатационного мониторинга, но не для оценки интеллекта модели.

Поэтому время ответа полностью исключено из итогового счета, и победитель по скорости здесь не определяется. Список доступных моделей можно проверить на странице моделей Crazyrouter, а планирование затрат следует проводить отдельно по странице тарифов Crazyrouter.

Среда и методика проверки

Перед запуском использовался список моделей, чтобы убедиться в доступности точных идентификаторов:

GET https://cn.crazyrouter.com/v1/models

claude-opus-5
gpt-5.6-sol

Все основные запросы отправлялись по одному пути:

POST https://cn.crazyrouter.com/v1/chat/completions

Внутри каждого тестового набора обе модели получали одинаковые:

system prompt;
user prompt;
значение temperature;
значение max_tokens.

Внешние инструменты не подключались. Успешный HTTP-ответ сам по себе не считался доказательством прохождения задачи.

Три независимых уровня оценки

Точность основного ответа

Проверялись ключевые числа, выводы, состояние математической модели и поведение алгоритма.
Полное выполнение задания

Проверялось наличие всех подпунктов, которые были явно перечислены в условии.
Машинная проверяемость

Код запускался локально на скрытых тестах, а строгий JSON передавался стандартному анализатору.

Такое разделение принципиально важно. Правильное число не гарантирует выполнения всего задания, а верные данные внутри ответа не гарантируют, что ответ соответствует требуемому формату.

После автоматической проверки проводилась ручная перепроверка. Иначе различия в записи LaTeX, регистре, точной дроби и округленном десятичном значении могли бы дать ложные отрицательные результаты. Например, GPT-5.6-SOL в вероятностной задаче не повторил эталонную дробь буквально, но привел эквивалентную формулу и правильное десятичное значение. Opus в физической задаче записал 0.302 m как 0.30 m с двумя значащими цифрами — это также не является ошибкой.

Результаты 10 задач

Задача	Проверяемый результат	Claude Opus 5	GPT-5.6-SOL
Точная цепь Маркова	`E[τ]=5`, `E[τ²]=43`, `Var(τ)=18`	Правильно	Правильно
Осциллятор с двумя степенями свободы	Две собственные частоты, две амплитуды и две фазы	Правильно	Правильно
Поиск с ограничениями	Единственный порядок `A,C,E,B,D`	Правильно	Правильно
Исправление применения неравенства Кантелли	Вероятность не идентифицируется, верхняя граница равна `0.2`	Правильно	Правильно
Проверка Python-кода поиска циклов	Ошибка, контрпример на DAG и минимальное исправление	Правильно	Правильно
Планирование эксперимента	Парное сравнение на одинаковых задачах, контроль сложности и доверительный интервал	Правильно	Правильно
Ожидание шаблона `HHTH` для смещенной монеты	Математическое ожидание около `12.6547` и правильные переходы состояний	Основной ответ правильный, последнее объяснение отсутствует из-за обрыва	Правильно и полностью
Алгоритм агрегирования журналов	Временное окно, события отказа, доля попаданий в кеш и пользователи с максимальными затратами	Скрытые тесты пройдены	Скрытые тесты пройдены
Неупругое столкновение и пружина	`v1≈6.10`, `v2≈2.44`, `x≈0.302`	Правильно	Правильно
Алгоритм стабильной маршрутизации	cost, latency, reliability и лексикографическое правило	Скрытые тесты пройдены	Скрытые тесты пройдены

Сводный результат основного набора:

основные ответы — 10/10 у обеих моделей;
полное выполнение всех подпунктов — 9/10 у Opus 5 и 10/10 у GPT-5.6-SOL;
скрытые тесты кода — 2/2 у обеих моделей.

Самая сложная задача по теории вероятностей

В задаче требовалось найти ожидаемое число бросков до появления шаблона HHTH для смещенной монеты:

P(H)=0.62
P(T)=0.38

Состояния нужно было построить по принципу самого длинного префикса шаблона, совпадающего с суффиксом уже полученной последовательности. Кроме вычислений, условие требовало объяснить два потенциально неочевидных момента:

почему после состояния HH и очередного H процесс остается в HH;
почему математическое ожидание нельзя просто записать как 1/P(HHTH).

Обе модели получили правильный основной результат:

E[N] ≈ 12.6547

GPT-5.6-SOL завершил весь вывод и учел перекрытия шаблона:

E[N] = 1 / P(HHTH) + 1 / P(H)

Opus 5 также правильно определил состояния, составил уравнения и привел точную дробь вместе с десятичным результатом. Однако ответ завершился с:

finish_reason=length

При границе:

max_tokens=3200

вывод оборвался после дополнительных математических ожиданий для состояний. Последнее требуемое объяснение — почему нельзя использовать только обратную вероятность шаблона — в ответ не попало.

Это различие напрямую показывает, зачем разделять два критерия:

Правильное итоговое значение засчитывается как правильность основного ответа. Отсутствие обязательного объяснения означает, что задача выполнена не полностью.

Методологически это тот же риск, который рассматривался в повторном тесте обрыва по max_tokens: вместе с содержимым ответа необходимо сохранять finish_reason и заданный бюджет вывода. Частично правильный ответ нельзя автоматически считать полной доставкой результата.

Проверка исполняемого кода

Качество программного решения нельзя надежно оценить по длине ответа, количеству комментариев или внешней аккуратности реализации. Поэтому сгенерированный код сохранялся в файлы .py и запускался локально в изолированном режиме на одинаковом наборе скрытых тестов.

Агрегатор журналов

Функция должна была корректно обрабатывать:

полуоткрытое временное окно;
разные правила учета успешных и неуспешных запросов;
отсутствующих пользователей и модели;
cache hit rate;
лексикографическую сортировку пользователей при одинаковом cost;
запрет на изменение входных объектов.

Обе реализации прошли скрытые тесты.

Стабильная маршрутизация с ограничением надежности

Алгоритм поиска пути должен был последовательно применять следующие критерии:

минимальная общая стоимость;
при равной стоимости — минимальная задержка;
при равной стоимости и задержке — максимальная надежность;
при полном равенстве — лексикографический порядок пути.

Дополнительно проверялись:

banned nodes;
максимальное число hops;
минимально допустимая reliability;
некорректные ребра.

Обе модели прошли и этот набор. Таким образом, результат по исполняемому коду — 2/2 у Claude Opus 5 и 2/2 у GPT-5.6-SOL. Оснований объявлять победителя по этим двум задачам нет.

Другой набор сложных задач по физике и алгоритмам приведен в материале GPT-5.6-SOL vs GPT-5.5: тест сложной физики и кода.

Строгий JSON: проверка формата, а не математических способностей

В отдельном задании нужно было представить данные об инциденте в виде одного объекта. Требование формулировалось однозначно:

exactly one JSON object and no Markdown

Обе модели правильно вычислили:

долю отказов;
принадлежность отказов к каналам;
число запросов, не восстановленных после повторной попытки.

Различие возникло только на уровне выходного формата:

GPT-5.6-SOL с первой попытки вернул только JSON, который можно было сразу передать в json.loads;
Opus 5 с первой попытки добавил ограждение блока кода и раздел Verification;
в первой дополнительной попытке Opus выдал компактный JSON, но затем снова добавил Verification;
во второй дополнительной попытке Opus опять добавил ограждение блока кода и пояснение.

Итог строгой проверки с первой попытки:

Claude Opus 5 — 0/1;
GPT-5.6-SOL — 1/1.

Две дополнительные попытки Opus также остались несоответствующими строгому формату. При этом сами данные и значения полей были правильными.

Поэтому корректная классификация такова: данные верны, формат не соблюден. Это нарушение инструкции и контракта ответа, а не ошибка вычисления.

Минимальная локальная проверка

import json

REQUIRED_KEYS = [
    "window",
    "total_requests",
    "failed_requests",
    "failure_rate_pct",
    "provider_owned_failures",
    "customer_owned_failures",
    "recovered_by_retry",
    "unrecovered_failures",
    "root_cause",
    "action",
]


def parse_incident_json(raw: str) -> dict:
    payload = json.loads(raw)
    if list(payload) != REQUIRED_KEYS:
        raise ValueError("unexpected schema or key order")
    if payload["failed_requests"] != 84:
        raise ValueError("failed request count mismatch")
    return payload

Этот код проверяет три независимых свойства:

ответ действительно разбирается как JSON;
схема и порядок ключей соответствуют ожидаемому контракту;
одно из критических значений совпадает с эталоном.

Один system prompt не гарантирует структурированный вывод. В рабочей системе безопаснее использовать поддерживаемые поставщиком параметры структурированного ответа, выполнять локальную проверку schema и предусматривать повторный запрос либо переключение модели после ошибки разбора.

Как воспроизвести сравнение через один API

Ниже приведен минимальный исполняемый пример для OpenAI-compatible chat completions endpoint. К самому API endpoint UTM-параметры не добавляются.

import os
import requests

BASE_URL = "https://cn.crazyrouter.com/v1"
API_KEY = os.environ["CRAZYROUTER_API_KEY"]


def ask(model: str, prompt: str, max_tokens: int = 4000) -> dict:
    response = requests.post(
        f"{BASE_URL}/chat/completions",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": model,
            "messages": [
                {"role": "system", "content": "Answer accurately and follow every requested constraint."},
                {"role": "user", "content": prompt},
            ],
            "temperature": 0.2,
            "max_tokens": max_tokens,
        },
        timeout=600,
    )
    response.raise_for_status()
    return response.json()


for model in ("claude-opus-5", "gpt-5.6-sol"):
    result = ask(model, "Your benchmark prompt here")
    choice = result["choices"][0]
    print(model, choice.get("finish_reason"), choice["message"].get("content", ""))

Для воспроизводимого теста недостаточно сохранить только видимый текст ответа. На каждом запуске также следует записывать:

точный model ID;
response ID;
returned model;
исходный prompt;
temperature;
max_tokens;
finish_reason;
необработанное содержимое ответа;
результат локальной проверки;
версию тестового набора.

Текущую доступность моделей можно проверить в списке моделей Crazyrouter, после чего запустить небольшой регрессионный набор на собственных рабочих запросах.

FAQ

Claude Opus 5 или GPT-5.6-SOL — какая модель умнее?

По этому набору нельзя обоснованно назвать одну модель в целом более умной. На 10 объективно проверяемых задачах обе получили 10/10 по правильности основных ответов.

Почему полное выполнение заданий не закончилось ничьей?

Ответ Opus на сложную задачу по вероятностям был оборван при max_tokens=3200 и завершился с finish_reason=length. Число 12.6547 и уравнения были правильными, но последнее требуемое объяснение отсутствовало. Поэтому Opus получил 9/10 по полной доставке, а GPT-5.6-SOL — 10/10.

Почему обрыв ответа не считается ошибкой основного рассуждения?

Потому что основной математический результат был получен правильно. Однако задача содержала дополнительное требование, которое не было выполнено. Правильность рассуждений и полнота ответа — разные показатели.

Следует ли считать нарушение строгого JSON ошибкой интеллекта?

Не как математическую ошибку. Opus правильно вычислил поля и значения, но нарушил требование к внешнему формату. Это следует учитывать как ошибку соблюдения инструкции и машинного контракта.

Что именно произошло при повторных попытках Opus с JSON?

С первой попытки Opus добавил ограждение блока кода и Verification. В первой дополнительной попытке после компактного JSON снова появился Verification. Во второй дополнительной попытке модель вновь добавила ограждение блока кода и пояснение. Таким образом, все три ответа не соответствовали требованию «ровно один JSON-объект без Markdown».

Как оценивались две задачи на программирование?

Код сохранялся в файлы Python и запускался на одинаковых скрытых тестах. Проверялись граничные условия, правила сортировки, некорректные входные данные и неизменность входных объектов. Обе модели прошли обе задачи — 2/2.

Почему в сравнении нет победителя по скорости?

Полное время API-запроса зависит от сетевого маршрута, кеша, нагрузки канала, ограничений учетной записи и выбранного обслуживающего экземпляра. Без фиксированного вышестоящего сервиса, большого числа повторов и доверительных интервалов задержка не является показателем интеллекта. Поэтому она исключена из оценки.

Достаточно ли 10 задач для долгосрочного выбора модели?

Нет. Это небольшой воспроизводимый срез возможностей, а не окончательный рейтинг. Перед внедрением следует использовать собственный набор рабочих задач и раздельно считать точность основных ответов, полноту выполнения, прохождение тестов кода и частоту нарушений формата.

Как начать собственное сравнение?

Выберите 10–30 реальных рабочих запросов и заранее задайте для каждого машинно проверяемые критерии. Затем отправляйте обеим моделям одинаковые входные данные с одинаковыми параметрами и сохраняйте необработанные ответы вместе с finish_reason. Критерии необходимо определить до просмотра результатов.

Что важнее для backend-системы: точность или формат?

Оба свойства важны, но проверять их нужно отдельно. Правильный ответ в неразбираемом формате может быть бесполезен для конвейера. И наоборот, синтаксически корректный JSON не гарантирует правильность значений. Надежная система проверяет и содержание, и контракт ответа.

Итоговый вывод

Главный результат этого сравнения — не безусловная победа одной модели, а четкое разделение двух аспектов качества:

Claude Opus 5 и GPT-5.6-SOL получили по 10/10 за правильность основных ответов. GPT-5.6-SOL полнее выполнил все подпункты — 10/10 против 9/10 — и с первой попытки соблюл строгий JSON: 1/1 против 0/1. Opus сохранил правильность рассуждений, но потребовал более строгого контроля бюджета вывода и формата.

Если задачу можно проверять по эталонному ответу или скрытым тестам, обе модели заслуживают включения в список кандидатов. Если последующие компоненты напрямую потребляют JSON, локальная проверка, контроль finish_reason, повторные запросы и переключение модели должны рассматриваться как обязательная часть архитектуры.

Создать учетную запись Crazyrouter и запустить собственное сравнение Opus 5 и GPT-5.6-SOL