DEV Community: Kristiyan Stoyanov

How does clean architecture impact token costs and execution times?

Kristiyan Stoyanov — Thu, 17 Sep 2026 12:10:51 +0000

I have been wondering whether the economics of software architecture are changing as agents write more of our code. If an agent does most of the implementation, does the structure we put in place still save enough work later to justify its cost?

As a principal engineer, I started with a fairly strong expectation. Clear boundaries, architectural guidance and tests should help an agent make changes without having to understand everything at once. I expected that investment to become more useful as a project grew.

So I built an experiment around two versions of the same service. I started with feature additions, tried harder challenges, grew both repositories, and finally tested a change specifically chosen to benefit hexagonal architecture.

This follows my first post, but I will go through the setup and the results here so you do not need to read that first.

The experiment and the setup

The application is ChargeLedger, a Java service for an EV charging network. It turns charging sessions into invoices, with energy charges, idle charges, tax and stored totals. Over the experiment it gained effective-dated tariffs, refunds, tenant isolation and other billing behavior.

The rules matter. Old invoices must keep their original prices. Retrying an operation must not refund twice. Financial changes and their audit records must stay consistent.

I prepared two versions with the same baseline business requirements:

Version	Starting approach
Hexagonal	Separate domain and application code, ports and adapters, architectural guidance, and architecture tests
Flat	The same functional requirements, with no prescribed architecture

“Flat” is the experiment's label. I did not ask the model to write bad code or put everything in one file. It made its own choices, including shared pricing and persistence code. Both versions had tests.

The stack started with Java 21, Javalin, SQLite and Gradle. A local Qwen model ran on my DGX Spark through vLLM. OpenCode ran on my Mac. ChatGPT/Codex helped prepare requirements, build the external evaluator and orchestrate the runs. The local model implemented the measured application changes. The original hexagonal baseline received both local-model and OpenAI assistance during preparation; that preparation was outside the measured time.

Each task had a written contract and external validation. The agent worked in a fresh session with the current repository. After implementation, the controller ran the project's tests and the external checks. A failure went back to the agent for repair within a fixed budget. Only one participant ran at a time, and each accepted feature got a commit.

The main measure is time to acceptance: agent work, project checks, external checks and repairs. It excludes initial project preparation and controller preparation before dispatch. Both sides of a task had the same functional contract and budget, but their architecture guidance and internal test suites differed. This compares two complete setups, not architecture as an isolated variable.

I also recorded input and output tokens, model steps, tool calls and controller-requested repairs. Input includes context sent repeatedly across model steps; it is not unique source code, GPU work or a bill. Output and reasoning are separate recorded fields. A controller repair is a feedback round after validation, not every correction the agent makes while working.

First, nine cumulative features: F1–F9

The first sequence began with CSV export and an offline invoice CLI. It then changed idle billing while preserving historical invoices, added issuance auditing and rollback behavior, and introduced effective-dated tariffs, partial refunds and tenant routing.

Each feature built on the accepted version of the previous one. I expected the architectural separation to help as more rules and entry points were added.

F1–F9 total	Flat	Hexagonal	Hexagonal compared with flat
Time to acceptance	165.93 min	228.57 min	37.8% more
Input tokens	31.25 million	53.40 million	70.9% more
Output tokens	164,177	217,296	32.4% more
Reasoning tokens	154,599	236,416	52.9% more
Model steps	426	612	43.7% more
Tool calls	520	749	44.0% more
Controller-requested repairs	2	2	Same

Both implementations passed all nine tasks. Hexagonal took longer overall and logged more tokens, steps and tool calls. The original architectural preparation was not included, so these figures describe subsequent work rather than the full investment.

Then, six harder challenges: C1–C6

My next question was whether the tasks had been demanding enough. I added six challenges:

Challenge	Requirement
C1	Calculate a quote without persisting an invoice
C2	Void a refund while preserving its history
C3	Import charging sessions from NDJSON through the CLI
C4	Create and deliver transactional outbox events
C5	Reconcile daily tenant balances
C6	Paginate invoices with a tenant-bound keyset cursor

These were independent branches. Each challenge started from its condition's accepted F9 version, not from the previous challenge. They cannot be treated as six more cumulative features.

C1–C6 total	Flat	Hexagonal	Hexagonal compared with flat
Time to acceptance	161.55 min	174.24 min	7.9% more
Input tokens	33.69 million	51.23 million	52.1% more
Output tokens	120,784	179,600	48.7% more
Reasoning tokens	180,707	153,479	15.1% less
Model steps	357	546	52.9% more
Tool calls	442	611	38.2% more
Controller-requested repairs	0	1	One additional repair

Both passed all six challenges. Hexagonal was faster on batch import and transactional outbox delivery; flat was faster on the other four. The overall time difference was smaller, and reasoning tokens went in the other direction from input and output.

That was a more mixed result. It did not establish that hexagonal was catching up as the code grew, because these were different tasks on independent branches.

Growing the repositories: S01–S15

A fair criticism of the first results was that the repositories were small. The matched C2 versions had 4,072 production Java lines in flat and 4,939 in hexagonal. A focused service is a relevant setting, but that does not settle an argument about larger systems.

I continued from those accepted C2 checkpoints and added fifteen changes in sequence. The first five added ordinary resource management across HTTP, CLI, validation and persistence. The later tasks crossed those resources and changed earlier requirements.

Stages	What changed
S01–S05	Customers and contacts, charging sites, chargers, fleet vehicles and versioned billing profiles
S06–S07	Assign customers to draft invoices and freeze billing details when an invoice is issued
S08–S10	Allow draft reassignment with history, transfer chargers between sites, and manage maintenance windows
S11–S12	Import fleet vehicles with per-row results, then add an all-or-nothing import mode
S13–S15	Redact customer data while preserving financial evidence, generate monthly statements, then change how statement addresses are selected

The changes of direction were intentional. A design has to handle the business changing its mind as well as adding functionality that fits the original plan.

This sequence inherited F9 plus C2, followed by each accepted S task. It did not contain all six earlier challenges.

S01–S15 total	Flat	Hexagonal	Hexagonal compared with flat
Time to acceptance	298.86 min	389.45 min	30.3% more
Input tokens	83.04 million	126.86 million	52.8% more
Output tokens	278,069	417,686	50.2% more
Reasoning tokens	203,985	292,081	43.2% more
Model steps	998	1,219	22.1% more
Tool calls	1,027	1,505	46.5% more
Controller-requested repairs	1	1	Same

Both implementations passed all fifteen stages. Hexagonal had small time wins on S08 and S15, but took longer overall.

These points connect tasks of different difficulty with a growing validation suite. They are not a controlled growth curve or evidence of a specific size threshold.

What the repositories looked like by then

By S15, these were substantially larger applications than the C2 starting points.

Repository measure	Flat at C2	Hexagonal at C2	Flat at S15	Hexagonal at S15
Production Java lines	4,072	4,939	11,709	15,205
Production Java files	37	85	84	239
Test Java lines	4,480	5,836	10,305	13,724
Test Java files	15	34	29	74

These are physical source lines, including comments and blanks. They exclude the external evaluator, dependencies and generated build output.

They were still single services, not large enterprise systems. But they had grown beyond the original few thousand lines and accumulated more behavior. That addressed part of the size criticism without turning line count into a measure of quality or difficulty.

At this point I was surprised. I had expected hexagonal to start winning more consistently as the application grew and requirements changed. It had individual wins, but the larger cumulative sequence still took 30.3% longer. More structure had not produced the general implementation advantage I expected.

So I chose a one-off task designed to give the existing boundaries a strong opportunity to help: replace the persistence backend.

S16: a change chosen to favor hexagonal

Both implementations had to support PostgreSQL across the service while keeping SQLite working. This directly exercises the replaceable persistence boundary that ports and adapters are supposed to provide.

It was more than changing a connection string. PostgreSQL had to preserve the accumulated behavior, survive restarts and pass checks involving separate JVMs, concurrent invoice operations, refunds, tenant isolation and invalid configuration. The old SQLite regression suite still had to pass.

This was not a historical data migration or a zero-downtime rollout. PostgreSQL and the same JDBC dependency were prepared for both projects before timing. Each run had a three-hour budget. Hexagonal ran first, then flat.

S16 only	Flat	Hexagonal	Hexagonal compared with flat
Time to acceptance	71.25 min	38.92 min	45.4% less
Agent work	63.64 min	31.10 min	51.1% less
External evaluation	7.56 min	7.77 min	2.8% more
Input tokens	24.53 million	14.75 million	39.9% less
Output tokens	55,523	32,504	41.5% less
Reasoning tokens	61,492	23,011	62.6% less
Model steps	174	120	31.0% less
Tool calls	173	119	31.2% less
Controller-requested repairs	0	0	Same

Both passed their first controller submission. That does not mean the agents made no mistakes or ran no internal test-and-fix loops.

Here, hexagonal had a clear advantage in time and recorded agent activity. Most of the time difference came from implementation work; external evaluation took roughly the same time on both sides.

What changed in the code

The hexagonal implementation added PostgreSQL repositories and supporting connection, schema and transaction code. Existing domain code, application service, ports, HTTP and CLI adapters, and SQLite adapters stayed unchanged. Among existing production files, only configuration and startup wiring changed.

Flat reused its central repository, introduced database-specific branches and locking behavior, and translated its SQLite schema into PostgreSQL form. Its HTTP and business value classes stayed unchanged too. Both designs accommodated the change, but they did so differently.

Production code in S16	Flat	Hexagonal
New files	3	11
Existing files modified	4	2
Lines added	390	4,527
Lines removed	83	11
Net new lines	307	4,516
Final production lines	12,016	19,721
Final production files	87	250

Much of the hexagonal PostgreSQL repository code closely follows the SQLite implementations. The separation kept the existing core stable, but left two sets of persistence logic to maintain.

Flat avoided most of that duplication. It also put more database-specific decisions into an already large repository. Its schema translation uses text substitutions, which makes future schema changes something to review carefully.

The faster clock and the smaller diff each describe part of the result. Neither settles which implementation will be easier to maintain.

The results across the phases

The broad pattern was higher accumulated time and input/output tokens for hexagonal in the feature and challenge phases, followed by a reversal on the adapter task.

Within the cumulative S extension, the adapter win did not recover all the earlier time difference.

Time accounting	Minutes
Extra hexagonal time across S01–S15	90.59
Time saved by hexagonal on S16	32.33
Extra hexagonal time still remaining	58.26

S16 recovered 35.7% of that gap. Across S01–S16, flat totaled 370.12 minutes and hexagonal 428.37 minutes, or 15.7% more.

This is measured run time, not a complete return-on-investment calculation. Original preparation, future maintenance and human debugging are outside it. One replacement also cannot establish the cost of the next one.

What the evidence can support

This is one service and one run per condition per task. The starting implementations, guidance and internal tests differ. I alternated which version ran first during S01–S15; S16 had a fixed order. Server-wide prefix-cache counters during S16 showed about 97% hits for both intervals, but those counters can include other clients and cannot establish each run's exact benefit.

The first S01 attempt also exposed error-response expectations that the written contract had not specified clearly. I clarified the contract and restarted both sides from their original C2 checkpoints. All S01–S15 figures here use that fresh paired rerun. The invalidated attempts are kept separately.

The results show how these implementations handled these tasks. They do not establish a scale threshold, long-term maintainability or production readiness. Passing the external suite means passing the behavior it covers. This study also did not compare agents with human developers, so it cannot by itself prove that agents changed the economics.

The version 2 technical report and supporting data cover the full experiment described here, including S01–S16, the methods, source checkpoints and limitations.

AI systems assisted with planning, orchestration, analysis, visuals and drafting. The measured application changes were made by the local coding agent.

My opinion: the architectural investment still has value

The following is my interpretation, informed by the results and my experience as a principal engineer.

The adapter task is a concrete example of architectural investment helping. The existing boundary gave the agent a place to implement PostgreSQL without changing the application core. It reached acceptance with less time, fewer tool calls and fewer tokens. That is useful value, even though it had not recovered the earlier time difference.

I no longer expect that same investment to make every kind of change cheaper. For the feature work we tested, the structured version usually took longer. The benefit appeared most clearly when the task matched a boundary the architecture had already established.

But how often do you change your database?

If database replacement is the reason we give for a design, we should ask how often we expect to use that flexibility. Most of the work on a service may be new fields, validation rules, reports and changes to existing behavior. Those changes deserve a place in the calculation too.

A replacement that becomes cheaper does not automatically justify making frequent work more expensive. In this experiment, the adapter task saved about 32 minutes after the preceding fifteen changes had accumulated about 91 extra minutes. That is a tradeoff I want to discuss explicitly when choosing a design.

Database replacement is also only one reason to separate persistence. Testing, ownership, dependency isolation and keeping business decisions understandable can matter even if the database never changes. If those are the benefits we need, we should name them and assess whether the design actually delivers them.

What I am questioning is the habit of accepting the ongoing cost of an architecture without being clear about which of its benefits the project needs.

Project timelines change

There is an equally important problem with planning only for the work in front of us: we are often wrong about how long a project will live.

The proof of concept becomes production. A temporary integration becomes a permanent dependency. A service meant for one customer ends up serving several. By the time a boundary becomes necessary, introducing it may be much harder.

That is why I would not turn these results into advice to remove structure and let the agent do whatever it wants. Someone still has to understand the system, review a financial change or investigate a failed refund during an outage.

My default would be to keep important responsibilities clear and behavior well tested, then add separation where the cost of introducing it later would be high. Pricing rules should have an identifiable home. Transaction ownership should be clear. A second real integration is a stronger reason for an abstraction than a diagram that says every service needs one.

A larger experiment could test that approach directly: a third implementation with a few deliberate boundaries, followed by both routine changes and disruptive ones. We have not established whether that would offer the best balance.

Conclusion: choose the tradeoff, not the diagram

Architectural investment can make a major change easier while adding cost to smaller, more frequent changes. That is the tradeoff this experiment made concrete for me. The existing persistence boundary helped with PostgreSQL; the same overall setup took longer across the preceding feature work.

This does not mean that every big change favors hexagonal or that every small change favors flat. Both implementations completed the adapter task, and some earlier tasks also favored hexagonal. The useful question is whether the design fits the changes and risks of this particular service.

For me, evaluating that case by case means being specific: what are we likely to change, what behavior must remain safe, what does the structure cost during ordinary work, and how difficult would it be to introduce later? We also need to allow for a longer project lifetime than the current plan suggests.

Agents are making writing and shipping code faster in my own work. I believe that makes it more important to revisit the practices we apply by default. Strong behavioral validation and understandable responsibilities remain worth investing in. Additional layers and abstractions should have a reason tied to the system we are building.

I would start with the simplest design that protects those needs, and let further architectural investment earn its place. I would not follow an architecture diagram blindly.

I am hyped to keep exploring this. Which boundary in your own service has earned its cost, and what change made that clear?

I expected hexagonal architecture to help my coding agent. It didn't.

Kristiyan Stoyanov — Tue, 15 Sep 2026 20:39:41 +0000

I am a principal engineer, and I started this experiment trying to defend something I have believed for years: invest in a good structure now, and later changes should get easier.

Two or three years ago, I would have pushed back on the claim that hexagonal architecture was mostly overhead. I wanted to show that the investment would also help a coding agent.

Across nine features, the hexagonal version took 38% longer to reach acceptance. That was not the result I had hoped for.

I still care about clean code. But this experiment made me much less willing to prescribe an architecture before I can explain what it buys us.

The question behind the experiment

In my work, agents are changing how quickly we can write and ship code. I think that changes which engineering practices deserve our time. This experiment did not compare agents with humans; it tested a question about the code we ask agents to work in.

I am less interested in whether a method has crossed a line-count threshold than in whether I can find the pricing rule, trace a state change, and understand where a transaction begins and ends. I still want good names. I still want business rules to have a clear home. I still want tests that exercise the behavior that matters.

What I am questioning is indirection added by habit: the interface with one implementation and no clear boundary, the pass-through method, or the extraction that scatters one operation across several files without making it easier to understand or change. Every layer has a cost for the person and the agent that must read it. I now think that cost needs a concrete justification.

ChargeLedger gave me a way to test that belief. It is a Java service for an EV charging network. It turns a completed session into an invoice: energy used, time spent plugged in after charging completes, tax, and a total that must remain stable once stored. It is small, but its requirements were not trivial. A tariff change must not rewrite an old invoice. A retry must not refund twice. If an audit write fails, issuing the invoice must roll back with it.

Those are the kinds of guarantees I want software structure to make easier to preserve.

Two starting points, one acceptance loop

I prepared two implementations with the same baseline business contract, Java 21, Javalin, SQLite, Gradle, and dependency pins.

The hexagonal condition was asked to use a domain isolated from HTTP and SQLite, application use cases, ports, adapters, edge composition, and architecture tests. The flat condition received the same functionality requirements but an architecture-neutral prompt. “Flat” does not mean deliberately messy. It became a direct, coherent application with shared pricing, persistence, transactions, HTTP, and CLI behavior.

I compared two complete setups. Their starting code, architecture guidance and test suites differed. I used both the local model and OpenAI assistance to prepare the hexagonal baseline; the flat baseline was generated by the local model. I excluded that preparation time from the measured clock. The results therefore cannot tell us that hexagonal architecture alone caused anything, or whether either approach has a better total lifecycle cost.

I served the local model on my DGX Spark and ran OpenCode on my Mac. ChatGPT/Codex helped prepare the tasks, build the evaluator and orchestrate the runs. Only one participant job ran at a time, so the two projects were not competing for the model.

For every measured task, the agent got a frozen public contract in a fresh session. The controller then ran project tests and a separate public-interface evaluator. If a product-level check failed, it returned feedback for a bounded repair attempt. A run counted as accepted only when both gates passed.

The timing is time to acceptance: agent work, project tests, external checks, and any repair. The clock stopped only after validation passed.

The baseline suite had 14 original tests. Across the study, all 30 measured runs passed their project and external acceptance checks. That means the implementations met the covered behavior. The suite can still miss things.

The result I did not expect

I gave both implementations nine cumulative feature tasks. They began with CSV export and an offline invoice CLI, then changed idle billing while keeping historic invoices fixed. They added issuance audit events and atomic rollback behavior, then effective-dated tariffs, partial refunds with concurrent balance protection, and tenant routing across the service.

I expected the architectural investment to become useful as that surface area grew.

Instead, the flat trajectory reached acceptance in 165.93 minutes. The hexagonal trajectory took 228.57 minutes: 37.8% longer. Upfront preparation was outside both clocks.

The time budget was 30 minutes per task for F1–F6 and 90 minutes for F7–F9, with the same budget for each side of a pair. Each task occurred once per condition, so those lines are not a statistical trend. They connect outcomes from a cumulative sequence whose task difficulty and validation scope changed over time. But they are still the result that made me pause. There were individual wins for hexagonal, but no overall speed advantage.

The trace data did not offer a simple explanation. Across F1–F9, the flat runs logged 31.25 million input tokens and the hexagonal runs 53.40 million. Those totals include context repeatedly supplied across model steps. They are not billed dollars, GPU work, or unique repository text. Hexagonal also logged more model steps and tool calls. Those are observations about this setup, not a verdict on an architecture.

I tried harder work

My first response was that the feature sequence might have been too easy. So I added six harder contracts: calculate a quote without creating a row, void a refund while preserving history, import NDJSON offline, create and deliver transactional outbox events, reconcile daily tenant balances, and paginate invoices with a tenant-bound keyset cursor.

These were not F10 through F15. Each challenge started independently from its condition’s accepted F9 checkpoint. None inherited the previous challenge’s implementation.

Across the six challenge pairs, flat took 161.55 minutes and hexagonal took 174.24 minutes, about 8% longer for hexagonal. Hexagonal was faster on two tasks: batch import and transactional outbox delivery. Flat was faster on the other four.

That is a more mixed result. I cannot treat the smaller gap as evidence that hexagonal would eventually overtake flat: these were different tasks on separate branches, with one run per condition.

The input-token totals were also higher in the structured condition: 33.69 million flat and 51.23 million hexagonal across the challenges. Again, repeated context is part of those values. They are evidence about recorded agent activity, not a price tag or a quality metric.

What the accepted code looked like

I then looked at the largest matched challenge versions: C2, refund voiding.

The flat version had 4,072 production Java lines in 37 files. The hexagonal version had 4,939 lines in 85 files. Flat’s largest production file accounted for about 36% of its production lines; hexagonal’s largest was about 18%.

Flat concentrated more code in one place; hexagonal spread it across more files. That gives us something concrete to inspect, but a file count cannot tell us which system will be easier to maintain.

The code review added some context. The flat code put shared pricing in Pricing and issuance in SessionRepository.issue. Hexagonal used PricingPolicy, LedgerService and ports. At F9, more persistence and migration work sat in the flat repository, while hexagonal separated schema migration and tenant wiring across roles. Both had shared business operations and transaction handling. Neither was a caricature.

The tradeoff is real. The direct path can be easier to trace. The separated path can make dependency direction and role boundaries explicit. I no longer think I should assume either benefit wins before I can name the problem it solves.

These are small applications. That limits the conclusion, but focused services are a relevant setting too. A microservice does not need to become a large codebase before we can ask whether its structure is helping. What we have here is a few thousand production lines with some demanding rules, not evidence about a whole enterprise system.

What I think changes in the agentic age

My conclusion is an opinion informed by the experiment, not a rule established by it.

Start simple. Add a boundary when it isolates a dependency, separates a real responsibility, protects a volatile integration, or makes an ownership decision clearer. Do not add it because every service is supposed to have another layer.

At the same time, simplicity cannot mean “throw it all in one file and trust the agent.” I want one place to change the tariff-selection rule. I want an idempotency key to have a clearly understood scope. I want invoice issuance and auditing to have a traceable all-or-nothing transaction. I want names that tell the next reader where those decisions live.

And I want stronger behavioral evidence. When code is cheaper to generate, a common external acceptance suite, regression checks, and explicit failure cases become more valuable. Tests only establish what they cover, but that is still much better than treating a confident agent response as proof.

This also changes how I think about reviews. A nitpick about a line break or another mechanical convention is often a good candidate for automation. A blocking review comment should explain the concrete consequence it prevents: duplicated state, an unsafe retry, an unbounded query, a broken tenant boundary, or a transaction that can partially commit.

There is another possibility: larger context windows and better retrieval may help agents make use of modular structures more effectively, not just direct code. My hypothesis is that stronger context may expand the range where a simple implementation works well, but this experiment did not test that boundary.

Proofs of concept also become production more often than we plan. If someone has to investigate an incorrect invoice during a SEV1, they need to find the decision and understand the state change. The benchmark clock stops at acceptance; ownership does not. I want to spend engineering effort where I can explain the behavior it protects or the change it makes easier.

The limits are part of the result

This was one model/harness setup, one service, and 15 paired tasks, each run once per condition. The local service was labeled Qwen3.8FlashNext for the project, but frozen records identify the served alias as qwen; the exact checkpoint and server settings were not independently verified. The evaluator changed through documented amendments, baseline preparation was excluded, and the treatments differed in more than architecture.

The result changed my mind about a default I used to defend. It leaves plenty of questions open.

The full method, frozen contracts, acceptance data, figures, and limitations are in the technical report:

Read the technical report and download the data

Stoyanov, K. (2026). Hexagonal architecture and coding agents: an exploratory study of change acceptance in ChargeLedger (Version 1.0). Zenodo.

AI systems assisted with planning, orchestration, analysis, figures, and drafting; the report’s method section describes the coding-agent participant and the limits of the evidence.

I am hyped to share this and hear from people working through the same questions. In particular: what layer, interface, or clean-code rule has earned its cost on your team—and what evidence convinced you?

How I Trained a DFlash Drafter for Speculative Decoding

Kristiyan Stoyanov — Mon, 07 Sep 2026 16:17:10 +0000

If you prefer video format

How I Trained a DFlash Drafter for Speculative Decoding

Running a capable local LLM is often easy. Making it responsive enough for interactive use, coding, or agent workflows is much harder.

I wanted to improve the decode throughput of a Qwen3.8 27B target model running in my local setup. Rather than changing the target model, I trained a small DFlash draft model and used speculative decoding. This post focuses on the practical workflow: how to prepare training data, choose an online or offline pipeline, train and inspect a drafter, export it, serve it, and benchmark it honestly.

This is not a universal copy-paste recipe. Exact package names, flags, configurations, and supported architectures move quickly. Treat the commands as a concrete template, pin the versions that work in your environment, and verify the current documentation for your chosen framework.

What we are building

A speculative decoding system uses two models:

A target model. This is the model whose output behavior you want to preserve.
A draft model, or drafter. This smaller model proposes several next tokens quickly.

At generation time, the drafter proposes a block of tokens. The target model verifies that proposal. It accepts the matching prefix and corrects the first disagreement. Because the target can verify multiple proposed tokens together, speculative decoding can improve output throughput without changing the target model's output distribution.

The important point is that a drafter is not a general-purpose replacement for the target model. It is a specialized accelerator coupled to a particular target model and serving implementation.

Why DFlash is interesting

Many speculative approaches draft autoregressively: token one, then token two, then token three. A DFlash drafter uses block diffusion to predict a block of future tokens in one parallel forward pass.

A simplified view looks like this:

Prompt
  |
  +--> Target model produces context features
  |
  +--> DFlash drafter proposes a token block in parallel
  |
  +--> Target model verifies the proposed block
  |
  +--> Accept matching prefix and continue

DFlash uses the target model's internal context features to help the small draft model make better proposals. In particular, its design fuses hidden states from the target and injects that context into the drafter's attention keys and values. This lets the drafter spend its capacity on predicting the next block rather than reconstructing the entire context from scratch.

This is why two metrics must always be considered together:

Acceptance. How many consecutive draft tokens the target accepts.
Draft speed. How quickly the drafter can create proposed blocks.

A slow but accurate drafter may not help. A fast drafter with poor acceptance may add overhead. The metric that matters at the end is measured end-to-end output throughput under a realistic workload.

Prerequisites and planning

Before starting a run, confirm these items.

1. A compatible target model

Your target must be supported by the training code and the serving runtime. The draft model is architecture-specific in practice: tokenizer vocabulary, hidden sizes, RoPE behavior, attention layout, and target hidden-state interfaces all matter.

Do not assume that a drafter trained for one Qwen version can be paired with another. Pair the exact target checkpoint named by the draft model or training recipe.

2. A suitable GPU environment

Drafter training can be memory-bandwidth and storage intensive. You will commonly have all of these on disk at once:

The target model
Source prompt data
Regenerated or distilled responses
Training checkpoints
Exported Hugging Face artifacts
Logs and temporary caches

Provision more storage than your first estimate. For large models, 1 TB is a safer starting point than trying to expand a rented instance halfway through a run.

Use a persistent remote session for every long operation:

tmux new -s dflash-train

# Detach with Ctrl-b then d
# Reattach later
tmux attach -t dflash-train

Back up progress outside the rented machine at regular intervals:

rsync -avP /workspace/dflash-runs/ user@your-local-host:/data/dflash-backups/

3. A reproducible software stack

Do not blindly install the latest version of every library. The intersection of PyTorch, CUDA, Transformers, FlashAttention, vLLM, SGLang, and a speculative-decoding training framework can be fragile.

Start with an isolated environment and record the working versions immediately.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

python --version
pip freeze | tee requirements.lock.txt
nvidia-smi

If the project provides a Docker image, development container, or pinned lockfile, prefer it. A known-good container can save days of dependency debugging.

The training pipeline

At a high level, the workflow has five stages:

1. Collect prompts
2. Generate target-model responses
3. Train the drafter
4. Export the checkpoint
5. Serve and benchmark

The training data should resemble the work you actually ask the target model to do. A coding assistant benefits from coding prompts. An agent server benefits from tool calls, structured output, planning traces, and the system prompts it sees in production.

A generic instruction dataset can produce a usable drafter, but task-aligned prompts are more likely to improve acceptance on your own workload.

Step 1: Prepare prompt data

A simple starting format is JSONL: one JSON object per line.

{"messages":[{"role":"user","content":"Write a Python function that retries an HTTP request with exponential backoff."}]}
{"messages":[{"role":"user","content":"Explain the difference between Kubernetes Deployment and StatefulSet."}]}

For chat models, preserve the structure that the target model expects. Do not flatten messages into ad hoc strings unless you also control and validate the exact chat template.

Here is a small Python script that samples a larger JSONL dataset reproducibly:

import json
import random
from pathlib import Path

source = Path("prompts-all.jsonl")
destination = Path("prompts-sample-30k.jsonl")
sample_size = 30_000
seed = 42

lines = source.read_text().splitlines()
random.Random(seed).shuffle(lines)
selected = lines[:sample_size]

with destination.open("w") as f:
    for line in selected:
        json.loads(line)  # Fail early if the input is malformed.
        f.write(line + "\n")

print(f"Wrote {len(selected)} examples to {destination}")

Before spending GPU money, inspect examples manually. Look for malformed conversations, duplicate prompts, giant contexts, private data, template artifacts, and prompts that do not represent your intended workload.

head -n 3 prompts-sample-30k.jsonl | jq .
wc -l prompts-sample-30k.jsonl

Use the target model to create labels

The drafter should learn the behavior of the target model, not the behavior of whatever dataset originally supplied an answer. Send each prompt to the target model and save the target's generated response. This is the distillation stage.

First launch an OpenAI-compatible target server. This SGLang command is illustrative; use the version and flags validated for your target model.

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --host 0.0.0.0 \
  --port 30000 \
  --tp-size 1 \
  --dtype bfloat16 \
  --mem-fraction-static 0.80

Then run the framework's data-regeneration script in a second terminal. The command below shows the inputs you generally need: source prompts, output path, target endpoint, concurrency, generation settings, and resumability.

python scripts/regenerate_train_data.py \
  --input prompts-sample-30k.jsonl \
  --output distilled-qwen3.8-27b.jsonl \
  --base-url http://127.0.0.1:30000/v1 \
  --model Qwen/Qwen3.8-27B \
  --concurrency 64 \
  --temperature 0.7 \
  --max-tokens 2048 \
  --resume

The most important practical feature here is --resume or its equivalent. Distillation can run for many hours. Spot instances can disappear, servers can crash, and local networks can drop. A resumable output turns a disaster into an interruption.

If your target has a supported built-in speculative mode, enabling it during data generation may reduce the time needed to build the distilled dataset. Validate that doing so preserves the target behavior you want to imitate.

Step 2: Choose offline or online training

There are two broad ways to feed target-model information into a drafter.

Offline training

Offline training separates the work into two phases:

Distilled data
  -> Run target model and save hidden states
  -> Train drafter from saved hidden states

Typical workflow:

python scripts/prepare_hidden_states.py \
  --model Qwen/Qwen3.8-27B \
  --input distilled-qwen3.8-27b.jsonl \
  --output-dir hidden-states-qwen3.8-27b

torchrun --nproc_per_node=8 train_dflash.py \
  --config configs/qwen3.8-27b-dflash.yaml \
  --data-path distilled-qwen3.8-27b.jsonl \
  --hidden-state-path hidden-states-qwen3.8-27b

Advantages:

Easier mental model: prepare features, then train.
Target and drafter do not have to share live runtime memory during the training step.
Easier to retry only one stage.

Disadvantages:

Hidden states can be very large.
You pay for a separate extraction pass.
Data preparation can become the bottleneck.

Online training

Online training keeps the target and drafter live together. The training process obtains target features as it trains, often through a shared KV-cache or shared-memory mechanism.

Distilled data
  -> Live target model
  -> Shared context features or KV cache
  -> Live drafter training

A generic layout looks like this:

# Terminal 1: start the target model with the shared-cache integration.
python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --port 30000 \
  --enable-shared-kv-cache

# Terminal 2: run training.
torchrun --nproc_per_node=8 train_dflash.py \
  --config configs/qwen3.8-27b-dflash-online.yaml \
  --data-path distilled-qwen3.8-27b.jsonl \
  --target-base-url http://127.0.0.1:30000/v1

Advantages:

Avoids a separate hidden-state extraction workflow.
Can use a large GPU more efficiently.
Faster iteration when the environment is already working.

Disadvantages:

More moving parts.
Greater runtime memory demand.
More sensitive to version compatibility between the training process, serving runtime, and shared cache.

My practical rule is simple: choose offline when you want the least complicated path or have limited runtime memory. Choose online when you have enough GPU headroom and are willing to spend time validating the shared-cache setup.

Step 3: Configure DFlash training

A DFlash configuration usually controls the drafter architecture, block size, target feature layers, optimizer settings, dataset handling, and checkpoint cadence.

Here is an illustrative YAML configuration. Field names vary by framework, so use it as a map of the decisions you need to make.

model:
  target_model: Qwen/Qwen3.8-27B
  drafter_type: dflash
  num_hidden_layers: 5
  hidden_size: 2048
  num_attention_heads: 16
  num_key_value_heads: 4

training:
  output_dir: runs/qwen3.8-27b-dflash
  max_steps: 20000
  per_device_train_batch_size: 1
  gradient_accumulation_steps: 8
  learning_rate: 0.0001
  warmup_ratio: 0.03
  bf16: true
  gradient_checkpointing: true
  save_steps: 1000
  logging_steps: 20

sequence:
  max_length: 4096
  block_size: 16
  num_anchors: 256

loss:
  self_logit_distillation: true
  loss_decay_factor: 7.0
  answer_only_loss: true

data:
  train_file: distilled-qwen3.8-27b.jsonl
  chat_template: templates/qwen3-train.jinja

The parameters that deserve attention

Block size

Block size is the number of token positions in a draft block. Larger blocks create more opportunity for acceleration, but later tokens are generally harder to predict and may be rejected.

Start with the block size recommended by the implementation and the corresponding reference checkpoint. Do not change it casually at serving time. The drafter is trained around a particular block structure.

Drafter depth and size

A larger drafter may improve acceptance, but it consumes more memory and can reduce draft speed. DFlash is attractive because it can predict a block in parallel, but that does not eliminate the cost of a deeper network.

Start with an upstream architecture configuration that is known to export and serve correctly. Only tune drafter capacity after you have a benchmark harness.

Number of steps

A small experimental run is useful for validating the pipeline, but it is not a production training schedule. In my experiment, I used a short staged run: an initial phase, then a lower learning rate phase. That was enough to prove the full path worked, but not necessarily enough to maximize generalization.

Save checkpoints often enough that a failed rental instance does not waste a day of work:

training:
  save_steps: 1000
  save_total_limit: 3

Assistant-only loss and chat templates

If you only want to train on assistant responses, ensure that the data pipeline can identify those tokens accurately. Some frameworks require chat templates with explicit generation markers in order to produce an assistant token mask.

Do not assume a model's default template has the required markers. Test tokenization before beginning a large run.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.8-27B")
messages = [
    {"role": "user", "content": "Explain a binary search."},
    {"role": "assistant", "content": "Binary search repeatedly halves a sorted search space."},
]

encoded = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    return_assistant_tokens_mask=True,
)

print(encoded.keys())
print(encoded.get("assistant_masks"))

If no assistant mask is created, fix the template before training. Otherwise, you may waste substantial compute learning to predict user messages, system prompts, or formatting tokens that do not improve generation behavior.

Step 4: Launch and monitor training

A typical multi-GPU launch uses torchrun.

torchrun \
  --standalone \
  --nproc_per_node=8 \
  train_dflash.py \
  --config configs/qwen3.8-27b-dflash-online.yaml

For distributed jobs, log the exact command, config, git commit, container tag, and hardware details alongside every run.

git rev-parse HEAD | tee runs/qwen3.8-27b-dflash/git-commit.txt
pip freeze | tee runs/qwen3.8-27b-dflash/requirements.txt
nvidia-smi -q | tee runs/qwen3.8-27b-dflash/nvidia-smi.txt
cp configs/qwen3.8-27b-dflash-online.yaml runs/qwen3.8-27b-dflash/

What to monitor

At minimum, watch these signals:

Training loss should trend downward without exploding or becoming NaN.
Draft token accuracy should generally rise.
GPU utilization should be high enough to justify the rental.
Memory usage should leave a small safety margin.
Checkpoints should be written successfully.
Data loading should not starve the GPU.

Accuracy in a training log is useful, but it is not the final serving metric. A per-token training accuracy number does not directly tell you how many consecutive tokens will be accepted in a live speculative loop.

Also watch for overfitting. An implausibly perfect score on a narrow dataset can mean the drafter memorized a distribution that does not match your real prompts.

A simple live log filter can help during debugging:

tail -f runs/qwen3.8-27b-dflash/trainer.log | \
  grep --line-buffered -E "step|loss|acc|accuracy|grad_norm|nan"

Step 5: Export to Hugging Face format

Training checkpoints are often framework-specific. Serving runtimes usually need a Hugging Face compatible export containing at least a config.json and one or more model.safetensors files.

A typical export command looks like this:

python scripts/export_hf_checkpoint.py \
  --model_path runs/qwen3.8-27b-dflash/checkpoint-20000 \
  --export_path exports/qwen3.8-27b-dflash

Inspect the exported directory before attempting to serve it:

find exports/qwen3.8-27b-dflash -maxdepth 2 -type f -printf "%f\n" | sort
ls -lh exports/qwen3.8-27b-dflash
jq . exports/qwen3.8-27b-dflash/config.json | head -n 60

Important configuration fields vary by implementation, but generally include architecture metadata, vocabulary size, block size, attention dimensions, and the target pairing assumptions.

Do a cheap load test before copying a large checkpoint to another machine:

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "exports/qwen3.8-27b-dflash",
    trust_remote_code=True,
    device_map="cpu",
)
print(type(model).__name__)

Step 6: Serve the target and drafter

Serving integration changes quickly. Always begin with the current documentation for your runtime and DFlash implementation.

An example vLLM-style configuration commonly looks like this:

vllm serve Qwen/Qwen3.8-27B \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --speculative-config '{
    "method": "dflash",
    "model": "/models/qwen3.8-27b-dflash",
    "num_speculative_tokens": 15
  }'

An SGLang-style deployment can look like this:

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path /models/qwen3.8-27b-dflash \
  --tp-size 1 \
  --dtype bfloat16 \
  --trust-remote-code

The draft block is often represented as an anchor token plus masked positions internally. A runtime flag named num_speculative_tokens may count only proposed tokens, while the training block_size can include the anchor. Check the implementation's terminology and use the reference serving command where possible.

Test correctness before measuring speed:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.8-27B",
    "messages": [{"role": "user", "content": "Give me three concise Linux troubleshooting tips."}],
    "temperature": 0,
    "max_tokens": 128
  }'

Look for failed remote-code imports, incompatible model configs, tokenizer mismatches, cache-layout errors, and unsupported attention backends. Solve those before chasing benchmark numbers.

Benchmark the right way

Do not benchmark one prompt once and call the result a speedup. A useful benchmark compares baseline and speculative runs under the same conditions.

Keep these fixed:

Target model and quantization
Hardware and GPU count
Runtime version and container image
Prompt dataset
Prompt length distribution
Output length limit
Sampling parameters
Concurrency
Warm-up procedure
Measurement window

Record both decode throughput and request-level behavior. Depending on your use case, useful outputs include:

Output tokens per second
Time to first token
End-to-end latency
Accepted draft tokens per verification step
Acceptance length
GPU memory use
Error rate

Here is a small benchmark client that sends prompts concurrently and estimates generated output tokens. It is not a replacement for a full benchmark tool, but it is useful for a controlled smoke test.

import asyncio
import time
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
prompts = [
    "Write a Python function to merge two sorted lists.",
    "Explain how a Kubernetes readiness probe differs from a liveness probe.",
    "Show a PostgreSQL query that finds duplicate email addresses.",
]

async def run_one(prompt):
    started = time.perf_counter()
    response = await client.chat.completions.create(
        model="Qwen/Qwen3.8-27B",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=256,
    )
    elapsed = time.perf_counter() - started
    text = response.choices[0].message.content or ""
    approx_tokens = len(text.split()) * 1.3
    return elapsed, approx_tokens, text

async def main():
    results = await asyncio.gather(*(run_one(p) for p in prompts))
    total_seconds = sum(elapsed for elapsed, _, _ in results)
    total_tokens = sum(tokens for _, tokens, _ in results)
    print(f"Approximate aggregate throughput: {total_tokens / total_seconds:.2f} tokens/sec")

asyncio.run(main())

For serious measurements, use the tokenizer rather than an approximate token count, use enough prompts to smooth out variance, and run the exact same harness against baseline and speculative servers.

A result table should disclose enough context to be interpreted:

Target model: Qwen3.8 27B
Drafter: custom DFlash checkpoint
Hardware: one GPU, exact model recorded in run metadata
Prompt set: held-out mixed coding and agent prompts
Temperature: 0
Max output tokens: 512
Concurrency: 1

Mode                    Output tokens/sec    Notes
Baseline                14.0                 No speculative decoding
DFlash via SGLang       18.5                 Runtime version recorded
DFlash via vLLM         20.0                 Runtime version recorded

The numbers themselves matter less than the methodology. If the runtime versions or kernels differ, say so. If you compiled a runtime from a pull request, say so. If the speculative path only helps at a particular concurrency or output length, show it.

Common failure modes

The run dies after an SSH disconnect

Use tmux or a similar terminal multiplexer. Make data generation and checkpointing resumable. Keep logs on disk.

You run out of disk

Check disk space before and during the run:

df -h
du -sh /workspace/* | sort -h

Checkpoints, model caches, and hidden states can grow much faster than expected. Clean only files you fully understand. Never delete the latest known-good checkpoint until the export has loaded successfully.

Version mismatches

Symptoms include broken imports, serialization failures, CUDA kernel errors, incompatible model classes, and runtime errors while serving an exported checkpoint.

Fix this by pinning working versions, recording git commits, and rebuilding from a known-good environment. Avoid partial upgrades in the middle of an experiment.

High training accuracy but weak serving gain

This is the classic drafter failure mode. Possible causes include:

The training prompts do not match serving prompts.
The drafter is too slow relative to its acceptance benefit.
The draft block is too long for the model's quality level.
You measured a training proxy rather than live acceptance.
The serving runtime is not using the expected optimized speculative path.
The benchmark is dominated by prefill or short responses rather than decode.

Treat the entire pipeline as the system under optimization, not just the checkpoint.

Serving works but results differ across runtimes

vLLM and SGLang may have different implementations, kernels, versions, cache behavior, batching choices, and speculative support maturity. Benchmark each runtime separately, record exact versions, and avoid claiming a direct runtime comparison unless the setups are truly matched.

Practical next steps

Once the generic pipeline works, the most interesting improvement is data specialization.

A general-purpose drafter learns from a broad prompt distribution. But a local assistant may spend most of its time doing only a few things:

Coding and debugging
Tool calling
Shell commands
Structured JSON responses
Infrastructure troubleshooting
A fixed set of system prompts

You can log and sanitize your own requests and target outputs, deduplicate them, split them into train and held-out evaluation sets, and continue training a strong base drafter on that distribution.

Be careful with privacy. Strip secrets, tokens, personal data, customer data, and proprietary code before any dataset leaves your trusted environment. Keep a fixed held-out set so you can distinguish real generalization from memorization.

Final checklist

Before publishing or relying on a custom drafter, verify the following:

[ ] Target and draft checkpoints are an intended compatible pair
[ ] Training and serving dependency versions are recorded
[ ] Prompt data resembles the real serving workload
[ ] Distillation is resumable and outputs have been validated
[ ] Chat template and assistant-only loss behavior have been tested
[ ] Checkpoints are backed up off the rented instance
[ ] Hugging Face export loads successfully
[ ] Baseline and speculative servers use the same benchmark harness
[ ] Benchmark settings are documented
[ ] You measure end-to-end throughput, not acceptance alone
[ ] You have tested held-out prompts, not only training-like examples

Resources

Closing thoughts

Training a drafter is a useful systems exercise because it touches the whole LLM stack: data design, target-model behavior, GPU infrastructure, distributed training, checkpoint formats, serving runtimes, and benchmark methodology.

The best lesson is not that every model needs a custom drafter. It is that speculative decoding is only valuable when the complete system improves. Build a reliable pipeline first, measure the real workload, and then specialize the drafter where it makes sense.

Speculative Decoding on Local Hardware: Benchmarking N-Gram, MTP, EAGLE3, and DFlash on the NVIDIA DGX Spark

Kristiyan Stoyanov — Sat, 18 Jul 2026 18:20:03 +0000

Speculative Decoding on Local Hardware: Benchmarking N-Gram, MTP, EAGLE3, and DFlash on the NVIDIA DGX Spark

TL;DR: We benchmark four speculative decoding methods — N-Gram, MTP, EAGLE3, and DFlash — running Qwen3.5-122B-A10B-hybrid-int4-fp8 on a single NVIDIA DGX Spark (GB10, SM121). MTP-2 delivered the best balance of throughput and stability (~49 tok/s avg, ~51 tok/s peak). DFlash achieved the highest individual peaks (~78 tok/s) but with significant variance. Configurations shared.

Why Speculative Decoding Matters for Local AI

Squeezing every token per second out of expensive local hardware is not just a nice-to-have — on machines like the DGX Spark where you've invested in a system to run frontier-scale models locally, inference throughput directly translates to latency in agentic pipelines, coding assistants, and multi-turn conversations.

Speculative decoding is one of the few optimization techniques that can deliver meaningful speedups without any change to output quality. Because the target model's final verification step guarantees mathematically identical output distributions to standard autoregressive decoding, there is zero quality trade-off.

LLM Inference: The Memory-Bandwidth Bottleneck

Before diving into speculative decoding mechanics, it's worth understanding why LLM inference is slow in the first place.

Every time an LLM generates a single token, the GPU must load the full model weight matrix from VRAM into compute units. For a 120B+ parameter model, this means moving tens of gigabytes of data — often for just a handful of floating-point operations per weight. This makes inference fundamentally memory-bandwidth bound, not compute-bound. The GPU's tensor cores sit largely idle while the memory bus does the heavy lifting.

Each inference step consists of two distinct phases:

Prefill (Prompt Processing): The input prompt tokens are processed in parallel. This phase is compute-bound and relatively fast, even for long contexts.
Decode (Generation): The model generates one token at a time, autoregressively. Each step requires a full forward pass through the model, loading all weights from memory. This is the slow phase — and it scales linearly with output length.

On the DGX Spark with the Qwen3.5-122B-A10B-hybrid-int4-fp8 checkpoint, the unoptimized baseline sits around 36–37 tok/s generation throughput. That's the number we're trying to beat.

How Speculative Decoding Works

The core idea is to break the serial token-by-token bottleneck. A small, fast draft model proposes a sequence of K candidate tokens in rapid succession. The large target model then verifies all K tokens in a single forward pass — because verification (running the full model over a known input) is significantly cheaper than independent generation.

The effective speedup depends on two competing factors:

Draft speed: How fast the draft model can propose tokens
Acceptance rate (α): What fraction of draft tokens the target model accepts

A very cheap but low-quality drafter (low α) will generate tokens faster than the target can accept, wasting compute. A very accurate drafter that nearly matches the target model will have a high α but won't be meaningfully faster to run. The sweet spot is a lightweight drafter that accurately mimics the target model's distribution for the task at hand.

The expected number of tokens produced per target-model forward pass is:

where α is the per-token acceptance rate and K is the number of speculative tokens. This means even modest acceptance rates (e.g., 0.8) with K=2 yield ~2.4 tokens per target pass — a meaningful multiplier.

The Test Setup

All tests were run on a single NVIDIA DGX Spark (GB10, SM121, 128 GB unified memory) with the following base setup:

Model: Qwen3.5-122B-A10B-hybrid-int4-fp8 (Intel AutoRound INT4 MoE experts + FP8 dense shared expert layers)
Inference engine: vLLM 0.19.1 compiled for SM121
Attention backend: FlashInfer
MTP weights patch: Required for the Intel AutoRound checkpoint (see below)

The hybrid checkpoint is worth noting: the MoE expert weights are quantized to INT4 (via Intel's AutoRound), while the dense shared expert layers use FP8 from the official Qwen FP8 checkpoint. This hybrid approach avoids the accuracy penalty of INT4-quantizing high-utilization attention layers.

Albond Recipe for Hybrid Checkpoint

Method 1: N-Gram Speculative Decoding

Concept: N-Gram matching is the simplest possible speculative decoding approach — and the most instructive for understanding the acceptance rate dynamic. It uses no learned model at all. Instead, it scans the current context window for recurring token n-grams and proposes continuations based on frequency co-occurrence.

Speculative configuration in vLLM:

{
  "method": "ngram",
  "num_speculative_tokens": 5,
  "prompt_lookup_num_tokens": 4
}

prompt_lookup_num_tokens controls how many tokens form the lookup key. The algorithm finds all positions in the context where the last N tokens appeared before and proposes what came after them.

Why it fails for open-ended generation: N-gram matching works well when the output is highly repetitive relative to the input (e.g., the model is echoing back parts of the prompt, or doing summarization). For free-form generation, the context has no meaningful prefix matches, so the acceptance rate drops sharply.

docker run -it --name vllm-qwen35 \
  --gpus all --net=host --ipc=host \
  -v /home/krisitown/AI/models:/models \
  vllm-qwen35-v2 \
  serve /models/qwen35-122b-hybrid-int4fp8 \
  --served-model-name qwen/qwen3.5 \
  --max-model-len 196608 \
  --max-num-batched-tokens 32768 \
  --gpu-memory-utilization 0.88 \
  --port 8000 \
  --host 0.0.0.0 \
  --load-format fastsafetensors \
  --attention-backend FLASHINFER \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --generation-config auto \
  --override-generation-config '{"temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 0.0, "repetition_penalty": 1.0}' \
  --speculative-config '{"method":"ngram","num_speculative_tokens":4,"prompt_lookup_min":2,"prompt_lookup_max":4}'

Result: ~24–30 tok/s — below the 36–37 tok/s baseline. The overhead of proposing tokens that get rejected outweighs any gain. N-Gram speculative decoding is only practical in highly repetitive/retrieval-heavy workloads.

Method 2: MTP (Multi-Token Prediction)

Concept: MTP is a fundamentally different approach. Rather than using a separate external draft model, MTP leverages a lightweight head that was trained jointly with the target model and ships as part of its checkpoint. Because the MTP head has been optimized to match the target model's own output distribution — sharing its internal representations — acceptance rates are dramatically higher than any external drafter.

The MTP head adds extra transformer layers at the end of the main model that are trained with an auxiliary loss to predict the next token in the sequence, using the intermediate hidden states of the main model. During inference, these layers run as a cheap drafting pass that reuses the KV cache already computed by the main model.

The MTP weights patch for Intel AutoRound:

The Intel AutoRound INT4 checkpoint for Qwen3.5-122B ships the MTP weights in model_extra_tensors.safetensors (785 tensors, ~4.8 GB BF16) but does not register them in model.safetensors.index.json. vLLM reads the index to discover weights, so it never loads the MTP head unless you patch the index manually. The recipe at albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4 includes a script that registers all 785 tensor mappings:

python patches/02-mtp-speculative/add-mtp-weights.py \
    --source "$INTEL_DIR" \
    --target "$MODEL_DIR"

Without this patch, adding --speculative-config will silently have no effect.

vLLM configuration:

vllm serve /models/qwen35-122b-hybrid-int4fp8 \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

⚠️ MTP requires --attention-backend FLASHINFER. The PyTorch backend does not support MTP in vLLM ≤0.19.

Tuning num_speculative_tokens:

The number of speculative tokens is the key tuning parameter. Testing with 1, 2, and 4 tokens revealed a clear pattern:

`num_speculative_tokens`	Avg tok/s	Peak tok/s	Notes
0 (baseline)	~36–37	~38	No speculative decoding
1	~44–45	~46	Solid improvement
2	~48–49	~51	Best balance
4	~34–35	~38	Acceptance rate drops, worse than baseline

The degradation at num_speculative_tokens=4 is expected. Each additional speculative token compounds the error: if the MTP head diverges from the target model at position i, all subsequent positions i+1 ... K are already wrong, so the overhead of generating and then discarding them hurts throughput. The acceptance rate at position 4 was measured in the low 50s%, meaning roughly half of fourth-position tokens were rejected — and the overhead of running the head + verification + token correction exceeded the benefit.

Result: MTP-2 delivers ~49 tok/s average, ~51 tok/s peak — approximately +32% over baseline with stable, consistent throughput.

Method 3: EAGLE3

Concept: EAGLE3 is the third iteration of the EAGLE family of speculative decoding drafters. Unlike MTP, the EAGLE drafter is a separately trained model — but it compensates for this by conditioning on the target model's internal hidden states, not just the output token embeddings.

The intuition is: if the draft model has access to the target model's intermediate representations at the current position, it already "knows" most of what the target was going to compute before producing the next token. This dramatically improves acceptance rates compared to a purely external drafter.

EAGLE3 architectural changes from EAGLE1/2:

Multi-layer hidden state inputs: The drafter consumes hidden states from multiple layers of the target model (not just the final layer). Earlier layers carry richer semantic information that the final layer collapses into a logit distribution.
Removal of next-feature prediction loss: EAGLE-2 trained the drafter to predict both the next token and the next hidden state. EAGLE3 drops the hidden state prediction auxiliary loss, freeing model capacity for the actual goal.
Inference-aligned training data augmentation: During training, the drafter is exposed to its own autoregressive rollouts (not ground-truth hidden states), reducing the distribution shift between training and inference.

Practical constraint: EAGLE3 heads are model-specific and must be trained for a particular target model checkpoint. For this test, no EAGLE3 head was available for the 122B MoE model, so the test was conducted on a separate Qwen3-30B-A3B baseline (~33 tok/s without speculative decoding) using an EAGLE3 head pulled from Hugging Face.

vLLM configuration:

{
  "method": "eagle",
  "model": "model-name/eagle3-head",
  "num_speculative_tokens": 1
}

Results on Qwen3-30B-A3B:

`num_speculative_tokens`	Avg tok/s	Notes
0 (baseline)	~33	Qwen3-30B-A3B baseline
1	~38–40	~+20% improvement
12	~15–17	Severe degradation (8-11% acceptance rate)

The num_speculative_tokens=12 result is instructive. Even though the EAGLE3 drafter was running at 85–86 tok/s draft speed, an acceptance rate of 8–11% means the target model was rejecting the vast majority of drafts. The net result was roughly halved generation throughput compared to baseline.

This is the fundamental constraint of autoregressive drafters: each additional speculative token must be generated sequentially by the drafter. Drafting 12 tokens takes 12 forward passes through the draft model, and if most are rejected, you've spent significant time generating tokens the target will discard.

Result: EAGLE3 with num_speculative_tokens=1 delivered a ~+20% improvement on the 30B-A3B model. The method scales poorly with higher speculative token counts unless the acceptance rate is very high (>85%).

Method 4: DFlash (Block Diffusion Speculative Decoding)

Concept: DFlash is the most architecturally novel method in this comparison, and addresses the core limitation of autoregressive drafters: the serial bottleneck. Instead of generating draft tokens one at a time, DFlash uses a lightweight block diffusion model that generates a block of up to 16 tokens in a single forward pass.

The method draws inspiration from image generation diffusion models. In image diffusion, all pixels are generated in parallel via iterative denoising. DFlash adapts this concept to token generation: the drafter generates an entire block of masked tokens simultaneously, denoised in a single step.

Architecture:

The target model processes the prompt and extracts hidden context features
These features are injected directly into the KV cache of the draft diffusion block
The diffusion head generates K tokens in one forward pass, conditioned on the target model's deep representations
The target model verifies the proposed block with a single forward pass

The key difference from EAGLE3: because the block is generated in parallel, the marginal cost of generating additional speculative tokens approaches zero. Generating 16 tokens takes roughly the same time as generating 4. This means DFlash can use far higher speculative token counts without the exponential cost increase that punishes autoregressive drafters.

Performance ceiling: Published benchmarks from the original DFlash paper (Chen et al., arXiv:2602.06036) show over 6x lossless acceleration on Qwen3 models, delivering up to 2.5x higher speedup than EAGLE3 on reasoning-heavy tasks. On coding benchmarks, acceptance lengths of 6–7 tokens per step were reported — double what EAGLE3 achieves.

vLLM configuration (DFlash):

vllm serve /models/qwen35-122b-hybrid-int4fp8 \
  --attention-backend FLASHINFER \
  --speculative-config '{
    "method": "dflash",
    "model": "z-lab/Qwen3.5-122B-DFlash",
    "num_speculative_tokens": 16
  }'

The DFlash head for the Qwen3.5-122B model was created by ZLab and is available on Hugging Face. Setup requires following a specific GitHub recipe (link in video description).

Observed behavior: DFlash showed high throughput variance. When the acceptance rate was strong (high-structure output like code), peak throughput hit ~78 tok/s — significantly exceeding MTP-2. However, for general-purpose prompts, throughput fell to ~30–33 tok/s average, with peaks to ~50 tok/s.

The variance is explained by DFlash's acceptance semantics. Unlike autoregressive drafters where individual token rejections are isolated, DFlash accepts a prefix of the diffusion block. If the target model accepts tokens 1–3 but rejects token 4, tokens 5–16 are discarded entirely even if they would have been accepted. This means low acceptance rate at any position in the block wastes the entire remaining draft. When the task is highly structured (code, JSON, math), the acceptance rate is consistently high across positions, and DFlash dominates. For open-ended generation, variance is high.

Result: DFlash achieved the highest peak throughput (~78 tok/s) but also the highest variance. Average throughput for general prompts was lower than MTP-2. For structured-output tasks (coding agents, JSON generation), DFlash is compelling. It is also a relatively new method and actively improving.

Results Summary

Method	Model	Avg tok/s	Peak tok/s	Acceptance Rate	Notes
None (baseline)	Qwen3.5-122B	~36–37	~38	—	Reference
N-Gram	Qwen3.5-122B	~28–30	~31	Very low	Worse than baseline
MTP-1	Qwen3.5-122B	~44–45	~47	Mid-to-high 80s%	Solid improvement
MTP-2	Qwen3.5-122B	~48–49	~51	~80%+	Best consistency
MTP-4	Qwen3.5-122B	~34–35	~38	Drops significantly	Worse than MTP-2
None (baseline)	Qwen3-30B-A3B	~33	~40	—	Separate model
EAGLE3 (1 token)	Qwen3-30B-A3B	~38–39	~41	Good	~20% improvement
DFlash	Qwen3.5-122B	~48–50 avg*	~78	Variable	High peaks, high variance

*DFlash average varies significantly by task type (structured vs. open-ended).

Practical Guidance: Choosing the Right Method

Use MTP if your model ships with a native MTP head (Qwen3.5, DeepSeek V3, Gemma 4). It is the lowest-friction option — no separate model download, no architecture mismatch risk, and acceptance rates are consistently high because the head was jointly trained. num_speculative_tokens=2 is a strong default for most Qwen3.5 deployments.

Use EAGLE3 if no MTP head is available and a well-matched EAGLE3 drafter exists for your target model on Hugging Face. Keep num_speculative_tokens conservatively at 1–4; beyond that, acceptance rates drop sharply for most tasks.

Use DFlash for structured-output workloads: code generation, function calling, JSON mode, math. The parallel drafting architecture gives DFlash a fundamental advantage when the output has strong syntactic structure and acceptance rates across the block are consistently high. Avoid it for open-ended chat where variance makes latency unpredictable.

Avoid N-Gram for general inference. It is primarily useful as a pedagogical example or in specialized summarization/extraction pipelines where output closely mirrors input.

Key Takeaways

Speculative decoding is lossless. Output quality is mathematically identical to standard autoregressive decoding. The only variable is throughput.
Acceptance rate is everything. More speculative tokens are not always better. The acceptance rate at each position compounds — even a single bad position in an autoregressive chain invalidates all subsequent drafts.
DFlash's parallel drafting breaks the serial constraint. By generating blocks in a single forward pass, DFlash eliminates the per-token cost overhead that limits autoregressive drafters at high K. This is the architectural reason its peak throughput is so high.
Task type matters. Structured output (code, JSON, math) consistently benefits more from speculative decoding than open-ended generation, because the conditional distribution is sharper and easier for any drafter to model.
Hardware matters. These numbers are specific to the DGX Spark GB10 (SM121, 128 GB unified memory). Results on H100, A100, or consumer GPUs will differ due to memory bandwidth, compute, and kernel optimization differences.

If you want more detail, here is a YT video I did, going through the tests:

Resources

albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4 — Hybrid checkpoint + MTP patch recipe
DFlash paper: arXiv:2602.06036 — Block Diffusion for Flash Speculative Decoding
ZLab DFlash models on Hugging Face — DFlash heads for Qwen3 models
vLLM Speculative Decoding docs — Configuration reference
EAGLE3 architecture reference

The AI agent habit that was quietly wasting my time and tokens

Kristiyan Stoyanov — Tue, 23 Jun 2026 20:39:55 +0000

The AI Agent Habit That Was Quietly Wasting My Time and Tokens

Tags: ai, localai, machinelearning, productivity, agents

I realized I had been using AI agents in a way that looked smart but was actually pretty wasteful.

The pattern was simple: I would ask the agent for something useful, it would go off and figure it out, and eventually I would get an answer. The problem is that if you keep asking the agent to rediscover the same process over and over, you are paying for repeated reasoning, repeated tool usage, and repeated trial and error. That means more tokens, more latency, and more opportunities for the agent to fumble.

What finally clicked for me was this: use LLM inference for decisions, not for repetition.

If a task has already been figured out once, I do not want the model burning context and tool calls to solve it again every time. I want the model to recognize the task, use a reliable tool, and move on.

That is the pattern I have been using with Hermes, and it has made my local agent setup much more useful.

The Setup I Am Running

Right now I am running Hermes on a DGX Spark. In the video, I show the machine with 128 GB of unified memory, and at that moment I had about 1 GB free because I had a quantized Qwen 3.5 model loaded locally.

Hermes is my current agent framework of choice. I have tried other options, but Hermes has been easy to install and easy to live with. One thing I especially like is that it supports Telegram through a gateway, so I can talk to my agent from my phone instead of only from a terminal window.

On the tool side, the ones that matter most for this workflow are:

Web search and scraping
Terminal access
File operations
Code execution
Sub-agent delegation

For web search, I am using Tavily. In the video, I mention the free tier gives about 1,000 requests per month, which is enough for experimentation but still limited enough that I notice when an agent wastes calls.

That matters, because this whole post is really about reducing unnecessary tool usage.

The Wasteful Version

I started with a normal prompt:

What is the weather going to be like in Sofia this weekend? Help me plan some activities based on it.

That is exactly the kind of thing I would send to an agent from Telegram while I am on the move.

Hermes did eventually answer, but watching the trace was the important part. It did a web search, then a web extract, then checked time and date, then stumbled a bit, then searched again after not getting what it wanted the first time. In the video, I call out the real cost: this simple request filled about 20k tokens of context.

And that is the issue.

The answer was fine. The process was not.

If I ask for weather and activity suggestions regularly, I do not want the model improvising a mini research project every single time.

The Better Version: Research Once, Automate Once, Reuse Forever

Instead of asking the agent the end question again, I switched to building a capability.

First, I asked Hermes to research free weather APIs that did not need a key and were easy to automate:

Research free and open APIs that give you weather forecasts. Look for APIs that do not need an API key and can be easily automated with a Python script. Do not write the script yet. Let me choose the API first.

Hermes went off, searched around, and came back with several options, including Open-Meteo, WeatherAPI, and met.no. It recommended Open-Meteo, and that was good enough for me.

So I moved to the next step and told it to build something concrete:

Let's use Open-Meteo. I want you to spawn an open code sub-agent and create a directory. Inside of that directory the sub-agent must implement an Open-Meteo API client wrapped by a CLI. Use Python. Make sure it uses real data. Use mocks only for unit tests. Report back when ready.

That “use real data, mocks only for unit tests” line is one I use a lot. If the agent can run against reality, it can verify its own work much better.

Hermes delegated the task to a coding-focused sub-agent, created the project, and implemented the CLI.

Then came the part that matters most.

Never Trust the Agent

When the agent said the project was complete, I did not just accept it.

I tested it with a real request:

Now let's test it with the real API. I want you to use the script to give me the weather forecast for Berlin.

This is the rule I keep coming back to: never trust your agent.

Read the code. Run the script. Verify the output. Make sure it is using the real API. Make sure it is not doing anything unexpected. Only after that should it move from “experiment” to “capability.”

In the demo, the script returned a 7-day Berlin forecast in about 0.4 seconds. That is the moment where the whole pattern becomes obvious. The slow, token-heavy part was discovering how to do the task. Once that is solved, the best move is to package it.

Turning a One-Off Script into a Permanent Skill

Once the weather CLI worked, I asked Hermes to wrap it as a reusable skill:

Now let's create a skill for you that wraps around this CLI script and uses it whenever I ask you about the weather in future sessions.

Hermes created the skill using its skill management flow, and that became part of its permanent skill set.

Then I started a completely new session.

That is the real test, because a fresh session has fresh context. No hidden memory from the earlier chat. No cheating.

I asked the same question again:

What is the weather going to be like in Sofia this weekend? Help me plan some activities based on it.

This time Hermes checked its skills, found the weather skill, executed the script, and gave me a clean answer with activity suggestions.

The difference was huge.

The first time, it burned through web searches, including two Tavily searches, and spent a lot of tokens figuring out how to answer. The second time, it reduced the whole thing to essentially one tool call to the script I had already verified.

That is the pattern in one line:

Explore once. Automate once. Wrap it as a skill. Reuse forever.

This Gets More Interesting Than Weather

Weather is a toy example, but it is useful because the waste is easy to see.

The more interesting example from the video is one I built from my phone over Telegram. I use my agent pretty often for stock-related questions, so I had it create a stock analyzer script that fetches stock or index data from an open API.

Same pattern:

Create a stock analyzer Python script that fetches stock or index data from a popular and open API. Use real data. Use mocks only for unit tests.

When it finished, I verified it with a real run for Microsoft one year back. The output included about 250 trading days of data, the latest price, some moving averages, technical indicators, and a short interpretation.

Then I turned that into a skill too.

In a brand-new session, I asked a vague question about the USO ETF. I did not mention the script. I did not explain the workflow again. Hermes picked the stock analyzer skill on its own and returned a useful summary with current data.

That is where this starts to feel less like chatting with a model and more like growing a personal assistant over time.

The Security Rule I Want to Keep

The core safety idea here is simple: verify before you automate.

If an agent writes a script, read it. If it claims something works, test it. If it needs access to real systems, expose only the operations you actually want it to perform.

For anything sensitive, I would keep the agent on the narrowest possible rails. In practice, that means preferring read-only capabilities where possible, using small purpose-built tools instead of broad access, and only promoting a workflow into a permanent skill after I have seen it behave correctly.

The more capable the agent gets, the more important this becomes.

The Bigger Picture

What I like most about this pattern is that it compounds.

Every time I notice a repeated agent task, I have a choice:

Keep paying for the agent to rediscover the solution.
Or turn the solution into a reusable capability.

Over time, that changes the shape of the whole setup.

I stop treating the model like a universal improviser and start treating it like a coordinator that knows when to call reliable tools. The model still provides the intelligence, but the repetitive parts move into code.

That opens the door to more domain-specific assistants too. A natural next step is something like a private realtor assistant that checks listings, pulls mortgage news, summarizes changes, and sends a Telegram update on a schedule. Same principle, just applied to a workflow that actually matters to someone day to day.

That is the part I find exciting. Not AI magic, but a steadily improving assistant that gets more useful because I keep teaching it durable skills.

If you want to see the full walkthrough, including the Hermes session, the weather skill build, and the Telegram-based stock example, watch the YouTube video here:

If you are experimenting with agents locally, I would love to hear what repeated task you would turn into a skill first.