Can a coding agent deliver sooner without switching to a faster model? We tested that in Sunny Code by comparing Fast and Standard execution policies, with the model and reasoning setting held fixed.
Across nine small tasks, Fast reached local delivery sooner in all 36 paired observations. Across the six cohort-by-model aggregates, mean completion time fell 50.4%–80.1%, and mean new-token usage fell 10.2%–38.7%.
The important boundary: this measures time until the agent ends the task. Independent acceptance ran afterward and is not included in that duration. Faster delivery did not mean every delivery passed.
Normal-verification cohort only; each bar averages six runs, including failed deliveries. Time ends when the agent finishes the task and excludes subsequent independent acceptance.
Setup: 9 tasks, 2 models, 72 runs
We used GPT-6.1 Sol and DS4.1 Flash, both with low thinking. Each task ran twice with each model in each mode: 9 × 2 × 2 × 2 = 72 runs.
Pairs held the task, code snapshot, dependencies, model, and reasoning setting fixed. Runs were serial, with alternating mode order. We compared the product modes as complete configurations; provider caching and service load were not fully controlled.
The tasks covered seven repositories:
| Cohort | Repositories and tasks |
|---|---|
| Normal verification | Sunny Code: local preview URL boundaries; text prompt learning and media input. Click: normalized Choice shell completion. |
| Cheap verification | boltons: Unicode line splitting/indentation and relative JSONL seeking. python-slugify: bypass the transliteration backend for ASCII input. |
| No existing unit tests | fogleman/primitive: bound ellipse vertical radius by height. fogleman/ln: zero-direction ray–box intersection. fogleman/sdf: tetrahedron scaling. |
The cheap-verification repositories had median native-suite baselines of about 0.7–2.7 seconds. This helped test whether the result persisted when repository tests were already inexpensive.
Completion time
Each cell below is the mean of six runs, including failed deliveries.
| Cohort | Model | Standard | Fast |
|---|---|---|---|
| Normal verification | GPT-6.1 Sol | 106.4 s | 31.3 s |
| Normal verification | DS4.1 Flash | 94.5 s | 18.8 s |
| Cheap verification | GPT-6.1 Sol | 54.3 s | 24.9 s |
| Cheap verification | DS4.1 Flash | 29.6 s | 14.7 s |
| No existing unit tests | GPT-6.1 Sol | 47.6 s | 23.3 s |
| No existing unit tests | DS4.1 Flash | 34.9 s | 9.6 s |
Fast aims to keep the agent focused on the task, limit extra work, and deliver promptly after making and inspecting the changes. The modes did not perform identical verification, so these numbers should not be interpreted as the same work simply executing faster.
New tokens also decreased
Here, new tokens = provider-reported input + output, excluding cacheRead and cacheWrite. This is neither total processed tokens nor an actual bill.
| Cohort | Model | Standard | Fast |
|---|---|---|---|
| Normal verification | GPT-6.1 Sol | 23,518 | 14,411 |
| Normal verification | DS4.1 Flash | 23,626 | 16,034 |
| Cheap verification | GPT-6.1 Sol | 25,213 | 16,650 |
| Cheap verification | DS4.1 Flash | 19,964 | 15,573 |
| No existing unit tests | GPT-6.1 Sol | 15,281 | 13,726 |
| No existing unit tests | DS4.1 Flash | 15,004 | 9,224 |
Token reduction and time reduction were different in size. For example, Sol in the no-tests cohort used 10.2% fewer new tokens while finishing 51.1% sooner. Our preceding experiment included cached input in its token totals, so its percentages are not directly comparable.
What passed—and what failed
After timing ended, we ran independent behavioral checks on separate copies of the delivered code, alongside available repository regression checks. Failed runs stayed in the results; we did not repair source and then count it as passing.
| Cohort | Strict acceptance: Standard / Fast | What happened |
|---|---|---|
| Normal verification | 8/12 / 8/12 | All eight Click deliveries missed supplementary empty-prefix requirements. |
| Cheap verification | 8/12 / 8/12 | Behavior passed 24/24. Eight slugify deliveries failed a legacy assertion that conflicted with the new requirement. |
| No existing unit tests | 12/12 / 12/12 | External behavior and total acceptance passed 24/24. |
In the slugify task, ASCII input was supposed to bypass the transliteration backend, but the old test still expected backend initialization. Updated upstream tests passed in supplementary diagnosis; that does not replace the original failures.
Equal pass counts do not establish quality equivalence. Tests retained in Standard deliveries may also have future maintenance value that this timing metric does not capture.
What I take from this
An agent's delivery policy is worth measuring alongside model choice. For these tasks, Fast shortened local-delivery time even when native verification was cheap or absent. This experiment does not isolate the contribution of any particular implementation change.
This is a deliberately selected sample of nine tasks with two repetitions per configuration—not 36 independent tasks. If your definition of delivery requires independent acceptance to pass, include that time and any rework in the comparison.
I'm building Sunny Code to explore more efficient AI coding, better interactions, and the future of programming. These experiments are one way of making that exploration measurable.
Full experiment: charts, task requirements, acceptance coverage, and all 72 run records.
Where do you draw the delivery boundary for a coding agent: when it finishes editing, when local checks pass, or after independent acceptance?
Top comments (0)