Originally published by Tokenhot Team on 2026-10-11. The DEV edition was formatted with AI assistance.
A useful Sonnet 5.5 vs Opus 5.5 comparison answers a specific question: which configuration completes your tasks at an acceptable cost and level of quality? To answer it, give both candidates the same work, decide what counts as success, and include failed attempts and human corrections in your records.
This is a practical evaluation guide. It contains no live model benchmark or measured winner. You can follow the method with a spreadsheet; the optional Python and JSON files help organize your own observations. They do not call an API, grade responses, or supply real model results.
1. Pick the work and define a pass
Start with one workload you actually need to handle, such as extracting fields from support tickets. Use a fixed set of representative inputs, remove sensitive data, and keep the same set for both candidates. Include difficult cases as well as routine ones.
Write the acceptance checks before looking at model responses. For example:
| Task | A useful pass condition | A failure worth recording |
|---|---|---|
| Extract a support ticket | Return the requested fields and preserve missing values as null. |
Inventing a renewal date that the ticket never supplied. |
| Fix a calculation function | Pass the declared tests without changing the input data or adding dependencies. | Correcting the happy path while breaking zero quantities or empty inputs. |
| Answer using a tool | Use the required tool and support the answer with its returned data. | Guessing the right answer without making the required lookup. |
The downloadable cases illustrate these three checks with synthetic inputs. Use them to understand the recording process, then build a separate task set from your own work. Three toy cases cannot establish production reliability.
2. Make the comparison repeatable
Record the exact model ID, provider, endpoint, prompt, tools, output limit, effort setting and retry limit. Start each candidate with a fresh conversation. Keep these settings with the results so that a later change does not silently become part of the comparison.
Effort needs particular care. As checked on October 12, 2026, Anthropic documents high as the Sonnet 5.5 default and medium as the Opus 5.5 default; Opus 5.5 has adaptive thinking always on. See the model comparison and effort reference. Leaving settings unspecified therefore does not create the same test condition.
The supplied recording template labels both candidates medium. This is a starting configuration to document, not a claim of equal compute or an optimal setting. Check that your actual route supports the model and settings, and make the request configuration match what you record. Editing the template alone does not configure or run a model.
Decide the retry rule in advance, such as one initial attempt and at most one retry under the same stated conditions. Apply it to both candidates. If you later tune prompts or settings separately, record that as another comparison instead of mixing the results.
3. Keep a record of every attempt
For each task, save the response, grade it against the checks, and record elapsed time and review time. A completed API request can still produce an unacceptable answer. Add a new row for each retry rather than replacing the original failure.
If you use the optional template, these fields have distinct jobs:
| Fields | What to enter |
|---|---|
candidate, model, provider, endpoint, effort
|
The configuration that actually produced the response. Keep separately tuned configurations under separate candidate names. |
task_id, attempt
|
The task identifier and attempt number, starting at 1. For retries, add rows numbered 2, 3, and so on without gaps. |
status, accepted
|
Use completed, failed, or unknown for an observed attempt. Mark acceptance from the grading result; leave uncertain outcomes unresolved. Untested rows remain not_run with accepted: null. |
artifact_path, grader, review_minutes
|
Where the response is saved, who checked it, and the time spent reviewing or correcting it. |
request_id, http_status, stop_reason, elapsed_ms, returned_usage
|
The actual request and response details available from your client or provider. |
charge, currency, charge_reconciled, billing_evidence
|
The charge and a reference to its billing record. Set reconciliation to true only after checking that record. Missing cost remains unknown. |
Keep credentials and sensitive response data out of any records you share. If a coding task produces executable code, inspect and test it in an isolated environment; the supplied summarizer never executes model output.
4. Include retry costs in the decision
For each candidate, calculate:
cost per accepted task =
total verified charges for all observed attempts
/ number of distinct tasks accepted
Include unsuccessful attempts in the numerator. Count a task once in the denominator even if a retry was needed. When no task is accepted, the ratio is undefined. When charges are missing, report the cost as unknown.
Arithmetic example only, not measured model performance: suppose you test 10 tasks, accept 8, and verify $2.40 in charges including retries. The cost per accepted task is $2.40 / 8 = $0.30. Dividing by all 10 tasks would give $0.24 and understate the cost of an accepted result.
Compare that figure alongside acceptance, first-attempt success, latency and human review time. The summarizer reports task counts, cost and recorded review minutes; it does not calculate latency statistics. Review the recorded timings separately. Check that both candidates attempted the same planned tasks before drawing a conclusion.
Apply your requirements in order. A candidate that fails a mandatory quality or latency threshold does not qualify merely because it is cheaper. If both qualify, compare the cost of accepted work and the review burden. If neither qualifies, change the workflow or test conditions and evaluate again. A retry that switches models should also be assessed as a combined workflow, including both models' charges.
Try the optional local tools
Use these files if you want a reusable record format and a local summary. Download them into the same folder, keeping the filenames shown:
- evaluation_cases.json — three synthetic tasks and their acceptance checks. Use these as examples when defining your own tests; they contain no model responses.
- evaluation_template.json — six unrun records: two candidates, each with three planned tasks. Save a copy to record your observations.
- summarize_eval.py — reads your JSON records and prints counts, verified cost and recorded review time. It uses Python's standard library and makes no network requests.
With Python installed, open a terminal in that folder and check the untouched template:
python summarize_eval.py evaluation_template.json
This command requires no API key and makes no model calls. It prints JSON. With the supplied template, the important values are:
-
planned_tasks: 3per candidate: three task records are present. -
observed_attempts: 0andaccepted_tasks: 0per candidate: no test results have been entered. -
all_planned_tasks_attempted: falseper candidate: the planned comparison has not been run. -
reconciled_charge: nullandcharge_per_accepted_task: nullper candidate: actual costs are unknown. -
winner: "not_determined": the script always leaves the decision to you.
Those zero counts confirm that the empty template can be read; they are not scores for either model. The script only reads the file and prints a summary. It does not fill in missing data or verify that a response or billing record is genuine.
To use it with your work, save a copy of the template as my-results.json, replace the task list and configuration as needed, and enter observations after running and grading your own requests. Keep one planned row per task and candidate; add rows for retries. Then run:
python summarize_eval.py my-results.json
Use the resulting counts and costs with your saved responses and acceptance checks to make the decision. A completed comparison needs those observations; downloading the files or running the empty template does not provide them.
Top comments (0)