DEV Community: sunny

Two coding agents agreed—and still built incompatible APIs

sunny — Fri, 30 Oct 2026 02:16:53 +0000

The frontend agent proposed API A. The backend proposed API B. Later, the frontend agreed to B—and the backend agreed to A.

Both were cooperating. Their implementations still did not connect.

While building Sunny Code, we compared one coding agent working across two repositories with two agents splitting frontend and backend work. That exchange changed what we needed to measure: sending a message, receiving it in context, agreeing on a contract, and implementing that contract are separate events.

Three rounds, three different questions

Each round had four paired comparisons using GPT-6.1 Sol at medium reasoning. The first two compared one session with two. The third used two agents in both arms and changed message delivery timing. These rounds should not be pooled into a leaderboard.

1. Start with an agreed API

The task was an OAuth integration with persistence, concurrent callbacks, expiry, workspace isolation, retries, and browser navigation. Both arrangements received the complete interface contract and shared a task-level budget.

All eight saved artifacts passed 28 checks after we corrected two evaluator assertions that incorrectly required lowercase “connected” instead of accepting “Connected”. We re-evaluated unchanged artifacts; the agents did not repair them.

Two sessions were faster in three of four pairs, with mean implementation time 12.6% lower. Their cumulative reported tokens were 2.17× the single-session total. That includes repeated cache reads, so it does not mean twice the dollar cost. The timing also excluded independent acceptance and human review.

This supported a narrow finding: splitting an already-specified interface can give a parallel benefit. It did not tell us whether the agents could negotiate one.

2. Leave the interface unsettled

We added safe disconnection to a working OAuth product. An old request retried after a lost response must not disconnect a newly established connection.

Some pairs started with incompatible API drafts; others continued real maintenance histories. Both arrangements could access the relevant information, but its organization and retrieval costs differed.

Under a shared ten-minute, 1.6-million cumulative-token and 24,000-output-token budget:

Round two outcome	One session, two repos	Two sessions, queued messages
Completed within budget and passed	4/4	0/4
Saved artifact passed all 48 checks	4/4	1/4

Passing code and completed delivery were separate outcomes. One two-session artifact passed every check after exceeding its budget.

The bridge accepted 38 messages, but only 20 entered recipients’ persisted histories before the tasks stopped. The delivery mode queued messages until the recipient’s current execution settled. A decision could remain waiting while both agents implemented their own assumptions.

3. Change when messages enter context

Both arms now used two agents, reusing round two’s starting snapshots, tasks, histories and budgets. Queued follow-ups were compared with delivery at tool boundaries during execution.

Round three outcome	Queued follow-up	Mid-execution delivery
Completed within budget and passed	0/4	1/4
Saved artifact passed all 48 checks	1/4	4/4
Messages entered before stop	14/32	45/47

Earlier entry aligned the final code more often in this small comparison. It did not solve bounded completion: three of four mid-execution runs still exceeded budget. Delivery at a tool boundary also does not interrupt a running tool or guarantee arrival before the immediately following step.

The crossed-confirmation exchange occurred here too. In the mid-execution arm, the agents subsequently froze a final version and aligned their code. That artifact passed all 48 checks, but exceeded the output budget.

Check the checks, too

We later added 12 boundary checks and calibrated them against a reference implementation and four deliberately broken variants. The reference passed all 12; the corresponding checks caught each broken variant.

We then evaluated unchanged copies of all 16 saved artifacts from rounds two and three, without rerunning the agents. All ten previously passing artifacts passed the added checks. This increases confidence in those particular boundaries, not complete coverage: the audit was post-hoc and the reviewer knew the earlier outcomes.

What I would check next time

For a new task spanning two repositories, we currently lean toward explicitly attaching both to one session. That is a product judgment informed by these runs, not a universal ranking.

For separate sessions, I want the record to distinguish queued → entered context → answered → implemented, with one agreed contract checked against both implementations. Mandatory handshakes and designated coordinators were not tested here.

There were only four pairs per round. File-access boundaries were primarily instruction-based, and one two-agent run in each of rounds two and three used host Playwright outside the built-in-dependencies-only premise. Those runs remain disclosed in the results. We did not measure full human review and rework time.

The useful failure was concrete: two agents can both say yes while building different interfaces. Message counts alone would have missed it.

Adapted from our original experiment, with per-run results and method disclosures. AI-assisted writing; experiment findings and limitations checked against the published record.

Continue, compact, fork or start fresh? A 120-run coding-agent experiment

sunny — Mon, 26 Oct 2026 15:17:27 +0000

When a coding agent finishes a task, should the next task continue the conversation, compact it, fork it, or start fresh?

We compared five session strategies across two models and two rounds: 120 runs. There was no universal winner. Keeping history could be expensive without improving delivery in one model, yet faster in another.

Experiment design

Every strategy received the same target instructions, repository, documentation and evidence. Fresh meant no inherited chat, not an empty project.

Fresh: start from the task and repository.
Continue: retain the full conversation.
Compact: continue after native compaction.
Fork: copy history into a new session identity.
Fresh + summary: start fresh with a shared, frozen handoff summary generated by Sol.

Two rounds × two models × two target tasks × five strategies × three repetitions = 120 runs. Models were gpt-6.1-sol (medium) and deepseek-flash (medium requested, high resolved by the runtime; displayed as DeepSeek V4.1 Flash).

History came from real development. The second round used more complex prior work—durable queues, idempotency, leases, cancellation and recovery—but was not longer. Sol history went from roughly 442k to 425k tokens; DeepSeek from 331k to 295k. This is not a controlled context-length staircase.

The target tasks were E2E verification of an existing export feature, including honest reporting of expected failures, and a redaction bugfix. The fix had to preserve structural identities and raw diff comparisons while using redacted text for both display and layout width.

One group, all five strategies

Sol · complex prior-work round · E2E verification:

Strategy	Mean recorded cost/run (USD)	Median minutes	Complete deliveries
Fresh	$0.194	2.7	3/3
Continue	$2.293	14.0	3/3
Compact	≥$0.548	6.8	2/3
Fork	$3.813	14.2	3/3
Fresh + summary	$0.207	2.6	3/3

The ≥ value is a recorded lower bound after interruption. Failed attempts remain in the means. Summary preparation and prior development are excluded.

In this group, full history increased cost and time without adding completed deliveries. That ordering did not generalize: in DeepSeek's complex-round E2E, Fresh took 6.8 minutes, Continue 3.0, and Fork 2.2, all with 3/3 complete deliveries.

Cache hits do not make an entire task cheap

For Sol's complex-round E2E, Continue averaged $1.266 in cache-read cost alone, versus $0.194 for the entire Fresh run—about 6.5×. This compares task-level costs, not token unit prices.

Continue cumulatively read about 6.33 million cached tokens per run across repeated calls. Full-history groups produced roughly 2.8–3.8× the output tokens of Fresh. Traces contained longer verification, scripts and repeated checks; we cannot label every additional step unnecessary.

A long-input pricing tier also contributed. Repricing the same recorded usage without its surcharge attributed about 47% of Continue's and 49% of Fork's recorded cost to the tier in this group. Pricing effects are not evidence of “context gravity.”

A summary remains a hypothesis

Fresh + summary was close to Fresh for Sol. In DeepSeek's complex E2E it cost about $0.082 and took 4.1 minutes, versus $0.103 and 6.8 minutes for Fresh. Yet across both bugfix rounds, summary runs passed functional checks 4/6, versus 6/6 for Fresh.

We would try a fresh session with a short summary first. The experiments have not shown it is the best choice. We used a shared frozen summary rather than a newly tailored summary for each target task; the main comparison excludes preparation cost.

Correctness and measurement boundaries

Of 120 runs, 102 completed successfully. The other 18 comprised 14 runtime failures/timeouts and four completed runs with functional boundary defects. Some interrupted runs already passed code acceptance but had not finished the delivery protocol.

Defects included visible paths mistakenly treated as immutable identities, and layout width calculated from original text after display redaction. The latter violates the task boundary; it does not by itself demonstrate a plaintext leak.

Three repetitions per group used one fixed development history per round. Four E2E judgments were corrected after manual review of permitted verification-script locations; original judgments were retained and models were not rerun. Cache warmth, routing and host load were not fully controlled.

Costs are fixed-rate API-equivalent estimates, not current prices or subscription bills. They include recorded uncached input, cache reads, cache writes and output, plus native compaction. Recorded cache writes were zero; incomplete usage can understate interrupted-run costs.

We have neither established a stable best strategy nor identified a context-gravity threshold. Next we want to test task-specific summaries, count preparation costs, and examine decisions that exist only in old chat rather than repository files.

Originally published on the Sunny Code engineering blog, with interactive comparisons, methodology and downloadable aggregate data.

Same model, faster agent: Harness Fast mode

sunny — Sun, 18 Oct 2026 17:52:12 +0000

Can a coding agent deliver sooner without switching to a faster model? We tested that in Sunny Code by comparing Fast and Standard execution policies, with the model and reasoning setting held fixed.

Across nine small tasks, Fast reached local delivery sooner in all 36 paired observations. Across the six cohort-by-model aggregates, mean completion time fell 50.4%–80.1%, and mean new-token usage fell 10.2%–38.7%.

The important boundary: this measures time until the agent ends the task. Independent acceptance ran afterward and is not included in that duration. Faster delivery did not mean every delivery passed.

Normal-verification cohort only; each bar averages six runs, including failed deliveries. Time ends when the agent finishes the task and excludes subsequent independent acceptance.

Setup: 9 tasks, 2 models, 72 runs

We used GPT-6.1 Sol and DS4.1 Flash, both with low thinking. Each task ran twice with each model in each mode: 9 × 2 × 2 × 2 = 72 runs.

Pairs held the task, code snapshot, dependencies, model, and reasoning setting fixed. Runs were serial, with alternating mode order. We compared the product modes as complete configurations; provider caching and service load were not fully controlled.

The tasks covered seven repositories:

Cohort	Repositories and tasks
Normal verification	Sunny Code: local preview URL boundaries; text prompt learning and media input. Click: normalized Choice shell completion.
Cheap verification	boltons: Unicode line splitting/indentation and relative JSONL seeking. python-slugify: bypass the transliteration backend for ASCII input.
No existing unit tests	fogleman/primitive: bound ellipse vertical radius by height. fogleman/ln: zero-direction ray–box intersection. fogleman/sdf: tetrahedron scaling.

The cheap-verification repositories had median native-suite baselines of about 0.7–2.7 seconds. This helped test whether the result persisted when repository tests were already inexpensive.

Completion time

Each cell below is the mean of six runs, including failed deliveries.

Cohort	Model	Standard	Fast
Normal verification	GPT-6.1 Sol	106.4 s	31.3 s
Normal verification	DS4.1 Flash	94.5 s	18.8 s
Cheap verification	GPT-6.1 Sol	54.3 s	24.9 s
Cheap verification	DS4.1 Flash	29.6 s	14.7 s
No existing unit tests	GPT-6.1 Sol	47.6 s	23.3 s
No existing unit tests	DS4.1 Flash	34.9 s	9.6 s

Fast aims to keep the agent focused on the task, limit extra work, and deliver promptly after making and inspecting the changes. The modes did not perform identical verification, so these numbers should not be interpreted as the same work simply executing faster.

New tokens also decreased

Here, new tokens = provider-reported input + output, excluding cacheRead and cacheWrite. This is neither total processed tokens nor an actual bill.

Cohort	Model	Standard	Fast
Normal verification	GPT-6.1 Sol	23,518	14,411
Normal verification	DS4.1 Flash	23,626	16,034
Cheap verification	GPT-6.1 Sol	25,213	16,650
Cheap verification	DS4.1 Flash	19,964	15,573
No existing unit tests	GPT-6.1 Sol	15,281	13,726
No existing unit tests	DS4.1 Flash	15,004	9,224

Token reduction and time reduction were different in size. For example, Sol in the no-tests cohort used 10.2% fewer new tokens while finishing 51.1% sooner. Our preceding experiment included cached input in its token totals, so its percentages are not directly comparable.

What passed—and what failed

After timing ended, we ran independent behavioral checks on separate copies of the delivered code, alongside available repository regression checks. Failed runs stayed in the results; we did not repair source and then count it as passing.

Cohort	Strict acceptance: Standard / Fast	What happened
Normal verification	8/12 / 8/12	All eight Click deliveries missed supplementary empty-prefix requirements.
Cheap verification	8/12 / 8/12	Behavior passed 24/24. Eight slugify deliveries failed a legacy assertion that conflicted with the new requirement.
No existing unit tests	12/12 / 12/12	External behavior and total acceptance passed 24/24.

In the slugify task, ASCII input was supposed to bypass the transliteration backend, but the old test still expected backend initialization. Updated upstream tests passed in supplementary diagnosis; that does not replace the original failures.

Equal pass counts do not establish quality equivalence. Tests retained in Standard deliveries may also have future maintenance value that this timing metric does not capture.

What I take from this

An agent's delivery policy is worth measuring alongside model choice. For these tasks, Fast shortened local-delivery time even when native verification was cheap or absent. This experiment does not isolate the contribution of any particular implementation change.

This is a deliberately selected sample of nine tasks with two repetitions per configuration—not 36 independent tasks. If your definition of delivery requires independent acceptance to pass, include that time and any rework in the comparison.

I'm building Sunny Code to explore more efficient AI coding, better interactions, and the future of programming. These experiments are one way of making that exploration measurable.

Full experiment: charts, task requirements, acceptance coverage, and all 72 run records.

Where do you draw the delivery boundary for a coding agent: when it finishes editing, when local checks pass, or after independent acceptance?

Same Model, Different Agents: Comparing Token Usage, Cost, and Task Success

sunny — Wed, 14 Oct 2026 18:16:29 +0000

Does using the same model mean similar token usage across coding agents?

We compared Codex, SunnyCode Base and SunnyCode CodeMode on eight tasks from p-queue, Click and markdown-it. All used GPT-6 Astra with medium reasoning and Fast off. Each configuration ran each task three times, with shared primary acceptance checks and failed runs retained.

I develop SunnyCode, so this is a developer-run comparison, not an independent benchmark.

Results

Means across 24 runs per configuration, including failures:

Configuration	Tokens	Estimated cost/run	Checks passed	Time
Codex	725,185	$1.440	21/24	222.6 s
SunnyCode Base	281,080	$0.785	23/24	297.1 s
SunnyCode CodeMode	165,566	$0.669	23/24	232.2 s

The biggest gap was cached input

Cached input accounted for 96.5% of Codex’s additional recorded tokens relative to CodeMode. These totals include context processed across requests—not just generated code.

Because cached input is discounted, CodeMode’s 77.2% reduction in tokens translated to a smaller 53.5% reduction in estimated cost.

Codex still had the fastest overall mean completion time. Fewer tokens did not automatically mean faster execution.

Costs are API-equivalent estimates, not subscription bills, using frozen rates of $10/M uncached input, $1/M cached input and $50/M output.

What this does—and doesn’t—show

Agent configuration is worth measuring alongside model choice. Separate cached input, uncached input and output, and compare acceptance and time alongside cost.

But this is eight tasks, not a general ranking. System prompts differed, Base runs were collected later, and provider/cache conditions were not randomized. Supplemental checks also found defects beyond the primary acceptance results.

Methodology, per-run results and source data