DEV Community: zhijie

Post-Training a Jev-Inspired Decision Model on an RTX 3090

zhijie — Sun, 18 Oct 2026 20:06:09 +0000

What would it take to post-train a small decision model inspired by Jev and the System One (System 1) idea? On one RTX 3090, I used Qwen3.5 Base as a frozen pretrained backbone and trained LoRA parameters plus a pointer decision head to output probabilities over candidate answers. This is a local learning experiment, without a claim to reproduce Jev's proprietary architecture or RLCD training method.

In one tiny development probe, fine-tuning raised correct answers from 7/28 to 13/28, while mean negative log likelihood worsened from 2.215 to 3.916. The training path worked, but accuracy alone left an important part of the result unseen.

This hands-on account walks through three things you can examine in your own decision-model experiments:

How a frozen Qwen backbone, LoRA and a pointer head produce probabilities over candidate descriptions.
How task wording and labels can disagree even when the data format is valid.
How to separate development results, public benchmark diagnostics and future confirmation when deciding what to improve next.

The experiment post-trains decision models using Qwen3.5-0.8B-Base and Qwen3.5-4B-Base as pretrained backbones on a single RTX 3090. Adapters, trained decision heads and data-reconstruction materials are linked below. This is a learning retrospective with configurations and evidence, rather than a complete command-by-command tutorial.

Originally published on my blog. The blog is the maintained source of truth; this is its English cross-post.

Some calls in an agent workflow only need a judgment: is the user adding context or changing an earlier decision? Does a passage support a claim? Which team should handle a ticket? The answer may be a single option, yet it determines what the program does next.

Reading about TypeSafe's Jev and System One models made me curious about one capability: reading context, a question and candidate answers, then returning probabilities over those candidates. It sounded complex. I wanted to understand whether the underlying structure and training process could be broken into steps an individual developer could understand and try firsthand.

I used the RTX 3090 from my earlier local coding agent experiments, using Qwen3.5 Base as the pretrained backbone for post-training a decision model: first the 0.8B backbone, then 4B. The backbone weights stayed frozen; supervised post-training updated newly initialized LoRA parameters and a pointer decision head. The resulting model reads context, a question and candidate answers, and returns a probability for each candidate. Those sizes also reflected local memory and compute constraints. This was primarily a personal learning experiment: implement the basic candidate-scoring path with the resources available, understand how the model learns these judgments, and learn how to check the results. Matching the full results of enterprise-scale cluster training was outside the goal of this experiment.

Initially, I only wanted to establish that the training path worked. The model quickly learned a tiny set of training questions. On another set, something more interesting happened: the number of correct answers increased while mean NLL got worse. Later, inspecting the training data revealed labels derived from review stars, even though the question asked about the attitude expressed in the text.

This account follows the first two model versions: turning a language model into a candidate scorer, understanding why accuracy alone cannot establish training success, and making supervision and evaluation answer the same question. It also introduces the weaknesses revealed by public JevBench questions that motivated further quality optimization.

Reading a question and scoring its candidates

Consider a review judgment. The text and probabilities below are fictional, illustrating the interface:

Context: The food was excellent and the service attentive. I'll be back.

Question: Based only on this text, is the reviewer clearly positive overall?

Candidates: Yes—clearly favorable overall; No—clearly unfavorable, or substantively mixed without a clear overall favorable judgment.

Output shape: {"yes": 0.8, "no": 0.2}

The candidate answers and their descriptions are part of the input. A ticket-routing request could instead supply descriptions for billing, shipping and returns. The output would still be a probability for each candidate.

The structure used here has three parts:

The Qwen3.5 Base backbone reads the input. Context, question and candidate descriptions are encoded as tokens, and the pretrained model computes contextual representations at their positions.
A pointer decision head scores the candidates. This small readout takes the representations at the decision and candidate positions, projects and matches them to obtain candidate scores, then applies softmax to produce a probability distribution.
LoRA adapts the model to the task. The original backbone weights stay frozen. Training updates low-rank parameters inserted into it and the newly initialized readout, reducing the parameters and optimizer state that need to be optimized.

For questions with a determinate label, the correct candidate supplies the supervision. Cross-entropy encourages the model to increase that candidate's probability, and gradients update LoRA and the readout. This connects reading the text, scoring the candidates and learning from labels.

Why choose this structure? A fixed-class classification head usually reserves an output position for each class. Scoring candidate descriptions supplied in each request better matches the interface I wanted to explore. Reading scores directly also removes the step of generating a natural-language answer and parsing its label. Prompting an existing language model to generate a label is another valid approach; this experiment explores direct scoring and does not provide a controlled speed, cost or reliability comparison between the two.

The starting point was Jared Palmer's KEV training process and implementation: a pretrained Base model with fresh LoRA parameters and a fresh readout. The architecture comes from that reference. Our changes at this stage focused on supervision, evaluation separation and training recovery.

Why more correct answers were not enough

The training mixture covered classification, question answering and rule composition; Yelp review judgments were only one task. The first step was a tiny training check. I selected the two shortest records from each data source, giving 24 records and 30 questions; one record can contain several questions. The 0.8B model repeatedly trained on them for 80 optimizer updates.

Before and after training, I checked both those training questions and a separate exploratory development set. The latter came from the same original datasets but used different data groups and was not directly used as training examples in this run.

Questions and metric	Before training	After training
Correct on 30 training questions	7/30	29/30
Mean NLL on 30 training questions	2.1845	0.1030
Correct on 28 development questions	7/28	13/28
Mean NLL on 28 development questions	2.2153	3.9161

Accuracy checks whether the highest-probability option is correct. Negative log likelihood, or NLL, also checks how much probability goes to the correct answer. For a question with a determinate answer, it is −ln(probability of the correct answer). Lower is better.

As a hypothetical illustration, suppose a binary classifier gives the correct answer a probability of 0.4 or 0.001. Both predictions count as wrong. The latter almost rules out the correct answer, so its NLL penalty is much larger. More questions can therefore be answered correctly while mean NLL deteriorates. The tiny experiment showed these metrics moving apart; the aggregate table alone does not identify which questions caused the change.

Moving from 7/30 to 29/30 on the repeated training questions showed that the model could learn that set. I also checked that backbone weights stayed unchanged, LoRA and readout weights changed, and saving and reloading preserved outputs. Together, these checks validated the training and saving path. The 28 development questions were too few and too limited in origin to establish broad generalization.

A program using probabilities to decide whether to act also needs calibration checks: among judgments made with roughly 80% confidence, are roughly 80% actually correct? Softmax only makes probabilities sum to one, and NLL does not replace a calibration assessment. This experiment led me to examine correct-answer counts alongside probability metrics, rather than watching training loss alone.

Did the labels answer the question being asked

Before scaling up, I inspected the Yelp review task in the mixed training data. Its binary labels came from a star-rating threshold, while the question asked: "Based only on this review text, is the reviewer clearly positive overall?"

Those labels need not express the same judgment. Imagine a fictional review: "The food arrived quickly, but it was disappointing. Overall, I wouldn't recommend it." Suppose it carries four stars and the old rule maps that rating to Yes. That label follows the rating rule, but the text-based question calls for No. The model receives conflicting instructions and supervision.

We specified the criterion explicitly: use only the supplied text. Clearly favorable overall means Yes. Clearly unfavorable, or substantively mixed without a clear favorable overall judgment, means No. Insufficient information, truncation or unresolved material ambiguity means holding the question aside. "Still waiting for the food; I'll update this later," for example, cannot support an overall judgment.

Annotation packets hid the original stars and old labels. We used model-assisted annotation, with crosschecks of changed labels, uncertain cases and a subset of unchanged cases. Uncertain was an annotation holding status, not a third output class for this binary task. Ultimately, 38 relevant questions were withheld from supervision while their records and other questions could remain.

The useful lesson was that well-formed data can still teach a different task. Valid JSON, acceptable input length and in-range labels do not reveal this mismatch. The question, candidate descriptions and label have to be read together. Additional epochs cannot resolve a contradiction in the task definition for us.

Here, "repair" means aligning supervision with an explicit text criterion. Model-assisted labels can still share errors, and the subsequent comparison does not isolate the causal benefit of this change. This audit finding also cannot directly explain the earlier tiny experiment's NLL deterioration; that causal link was not tested.

Aligning training and development criteria

Beyond the Yelp supervision issue, structural checks, overlap screening and sampled semantic review quarantined 56 whole records involving near-copy risks, unresolved ambiguity and label conflicts. After incorporating the Yelp repair, training used 12,520 records and 15,476 questions. Versions were saved separately with checksums so an intermediate dataset would not be mistaken for the data a final model actually saw.

Evaluation also needed to follow the revised task definition. Historical star-derived labels could not evaluate the new text question, so we built a development set using the same text criterion. Starting with 200 Yelp candidates, we screened overlap with historical material and conducted two model-assisted review passes. We admitted 194 and held six because of ambiguity or disagreement.

Development data supports examining models and comparing approaches. Calibration data fits probability adjustments. A final test supports confirmation after the approach is fixed. These 194 records served only the first purpose. The training entry point used an explicit allowlist of sources to reject them as training data.

This separation makes comparison more meaningful, but screening cannot rule out all semantic overlap or pretraining exposure. Agreement between two model-assisted passes is not independent human ground truth. Holding difficult cases aside also leaves some difficult situations outside the reported scores. Those limits matter when interpreting the Yelp results below.

What the two trained versions tell us

After the repair, I trained fresh from Base at 0.8B and 4B, naming them B1 and B2. Both used fresh LoRA parameters and readouts, the same repaired training data and two epochs. Their learning rates were 0.0001 and 0.00005 respectively. This compares two training recipes; it cannot isolate the effect of parameter count.

Before training 4B, I measured forward, backward and optimizer updates on representative and stress inputs on the 3090. LoRA reduces trainable parameters, but backbone weights, intermediate activations and input length still consume memory. Loading the model is only part of a training-capacity check.

Evaluation used two development sets: the 194 text-based review judgments described above, and historical multitask development data covering classification, question answering and rule composition. The second row below counts 1,184 original questions; another 204 constructed variants, including reordered options, are excluded from this table.

Development data	Correct, 0.8B (B1)	Correct, 4B (B2)	B1 mean NLL	B2 mean NLL
Yelp text-based attitude judgments	184/194	192/194	0.09620	0.04024
Original historical multitask questions	951/1,184	1,033/1,184	0.60133	0.46749

These are outputs without temperature calibration, at temperature 1. B2 did better on the correct-answer counts and mean NLL shown here. The table compares two trained versions. Without an untrained control on the same sets, it cannot quantify the gain from fine-tuning itself or establish that general capabilities were retained.

Another slice is closer to the risk of automated action. On the Yelp set, B1 made 181 predictions with confidence at least 0.9, including one error. B2 made 188, including two errors. The samples are small and cover different subsets, so this is not a significance claim. It does illustrate why better averages do not automatically mean fewer high-confidence errors, and why 0.9 is not a ready-made production threshold.

Longer runs also need real recovery

Moving from tiny experiments to longer runs required another engineering change: training recovery. Loading LoRA and readout weights allows prediction to continue. Continuing the same training run also requires the optimizer, learning-rate progress, random state and data position.

We compared uninterrupted training against a run terminated mid-process and resumed. After adjusting determinism settings on the actual CUDA path, both completed eight optimizer updates with bit-identical trainable parameters and training state. Reloaded probabilities on 30 check questions were identical too.

This validated process-interruption recovery in that single-GPU environment. It provided a recoverable starting point for further iteration; the test did not cover retraining on different hardware or physical power loss. Saving and comparison details are in the notes below.

JevBench exposed the next problems to solve

Training, saving and loading worked, and results on the existing development sets had improved. The next question was how well the model would handle a different set of practical decisions. We evaluated the unchanged 4B version locally on a fixed public JevBench revision.

That checkout contained 231 downloadable public questions. Every request returned a valid probability vector without input truncation. Breaking the results down by the benchmark's groups revealed a clear gap:

Public JevBench group	Correct
Easy	48/48
Original	69/72
Hard	53/111

The overall result was 170/231, but 58 of the 61 errors came from Hard. A further breakdown by task family showed 0/15 on temporal and numeric questions and 4/19 on long-policy judgments. Handling simple decisions did not establish reliability on date calculations, combined conditions, exception clauses or evidence distributed across a long document. The overall score hid these weaknesses.

That gave the subsequent training a more specific direction. First, inspect task conversions, options and labels item by item to understand the failures. Then add independently checkable date calculations, rule combinations and exception cases, progressively increasing the context the model needs to process. Keep original-task training data and evaluations alongside them to check whether improving difficult questions costs performance on decisions the model could already handle.

Length was one factor to investigate: the original 4B training configuration limited context to 384 tokens, while these long-policy contexts contained roughly 2,203–3,666 tokens. However, the longer inputs also all belonged to Hard. Length and difficulty were entangled, so the results could not establish that length alone caused the failures. Labels and conversions in these small task-family groups also still needed item-level review.

These are local public-question diagnostics, not an official sealed evaluation or leaderboard rank. Differences between task groups also do not establish a regression caused by training. The inspected public questions can track known weaknesses, but independent reserved data is needed to confirm whether improvements generalize. Quality optimization is still underway. Future posts will examine which training changes help these weaknesses, which do not, and whether performance on other tasks is preserved.

Model downloads and data reconstruction

The models and accompanying materials are available on Hugging Face:

Each model repository includes the LoRA adapter, decision readout, native loader and training configuration. Loading retrieves the separately pinned Qwen Base. These are inference checkpoints, excluding optimizer, random-state and other training-resume snapshots. Follow the repository's loading instructions so the decision readout is included.

The data repository provides source indexes, label repairs, reconstruction scripts and a training reproduction kit, without redistributing original review or question-answer text. Reconstruction retrieves pinned upstream data and verifies the final file's SHA-256. Training and evaluation roles are recorded separately, alongside source attribution and component-specific terms.

Data versions and evaluation scope

One record can contain several questions. The data versions were:

Data stage	Records	Questions
Initial training data passing input checks	12,576	15,576
Interim data after record quarantine	12,520	15,514
B1/B2 training data incorporating Yelp repair	12,520	15,476

Of the 56 quarantined records, six involved public cross-split near-copy risks, 39 unresolved ambiguity and 11 reviewed label conflicts. The move from 15,514 to 15,476 questions withheld 38 Yelp judgments from supervision without changing the retained record count. These checks reduced identified risks; they do not estimate the whole dataset's label-noise rate.

Training files had their own checksums and manifests. A copied manifest still describing old data was replaced before launch. Development data had an evaluation-only declaration, while training also checked an explicit source allowlist. A legacy entry point without that allowlist could not provide the required separation.

The historical multitask data was called the retention panel in the records. It contained 1,388 questions. The table uses 1,184 original questions, marked clean in the scoring records. The other 204 comprise 84 option-permutation variants, 60 none-option controls with the answer present among the candidates, and 60 with the answer absent. Here, none means "none of the above." These variants are excluded from the table's correct-answer counts and mean NLL. Historical labels and overlap remain limitations, so these results do not establish independent external-test performance.

Training configuration and recovery

Configuration	0.8B (B1)	4B (B2)
Pretrained model	Qwen3.5-0.8B-Base	Qwen3.5-4B-Base
Initialization	Fresh LoRA and readout	Fresh LoRA and readout
Training records	12,520	Identical file bytes
Epochs and optimizer updates	2; 3,130	2; 3,130
Physical batch and accumulation	1; 8	1; 8
LoRA rank	16	16
Learning rate	0.0001	0.00005
Backbone storage and autocast	FP32; BF16	FP32; BF16
Token budget per pass	4,096	8,192
Random seed	3	3

The original comparison verified that all admitted 4B inputs already fitted the 4,096 budget; using 8,192 did not change actual pass packing. Each recipe had this one run, so it cannot estimate training-seed variation. The earlier tiny run used physical batch 1, accumulation 4 and 80 updates.

During training, a record can produce augmented variants with distractor candidates or none-of-the-above controls. The training objective averages question losses within each variant, then sums according to variant weights. The result tables average NLL over the selected original questions, so the aggregation units are not interchangeable. An earlier 0.8B full-mixture model was named B0. B1 was trained fresh from Base to avoid carrying forward B0's old supervision history.

Recovery saves trainable parameters, optimizer state, scheduler, random state, counters and the next data position. Saves occur after a complete optimizer update, excluding unfinished gradient accumulation. Several checksummed save generations allow fallback. The first CUDA recovery comparison showed small numerical drift. With determinism enabled on the actual path, uninterrupted and resumed runs matched in parameters, optimizer, scheduler, random state and data position.

Implementation and measurement sources

Local training started from Jared Palmer's KEV, pinned to commit 2ecafc49fe1e8945cd600443e6c04c1cf0b3a78f, adding local training-recovery support without loading a trained KEV adapter. The pretrained models were Qwen3.5-0.8B-Base and Qwen3.5-4B-Base. See Hu et al. for LoRA and Guo et al. for background on calibration and temperature scaling.

This implementation runs each question with its context as an independent causal sequence on the Qwen3.5 hybrid backbone, excluding other questions from its context. Batching does not reproduce Jev's architecture or latency. The tiny experiment used the reference PyTorch compute path actually verified in that run.

Selected aggregate measurements support checking the counts and configurations here. That file contains summary values only; model downloads, data reconstruction and training reproduction materials are in the Hugging Face repositories above.

This article was prepared with AI assistance from local experiment records. It covers the initial B1/B2 versions, their corresponding development results, and the 4B public JevBench diagnostic that motivated further optimization.

Making Qwen Faster on an RTX 3090

zhijie — Sat, 10 Oct 2026 18:56:30 +0000

Originally published on my blog. 中文版.

Over the past few days, I continued tuning Qwen3.8-27B on a single 24GB RTX 3090. Generation speed during coding tasks increased from roughly 33 tokens/s with the original configuration to around 60 tokens/s. A subsequent runtime and draft-sampling update reached 68.4 tokens/s across eight coding attempts; the matched speed screen below isolates its additional gain.

Two other findings deserve attention: a repeatedly failing task turned out to have a defective test, and one configuration generated tokens faster but took three extra minutes to deliver its code.

My current speed candidate is IQ3_S, Q8 main KV, and MTP4 on llama.cpp b11429 with probabilistic draft sampling. Its latest eight-task screen used a 220K total window and a 50K output cap, delivering 7/8. Long input has also run, though a full 170K-input-plus-50K-output sequence remains untested.

Writing disclosure: this article was drafted with AI assistance from recorded local experiments and reviewed by the experimenter. The results and limits below refer to those saved runs.

What I actually tested

The previous post focused on output budgets and reasoning settings, eventually reaching 23/24 delivered passes. Those were eight tasks repeated three times, not 24 distinct problems. Later eight-attempt parameter screens ran each task once.

The suite has six Python tasks and two TypeScript tasks covering caching, database transactions, incremental streams, paths, ledger processing, build dependencies, asynchronous concurrency, and pagination. Seven are authored repair tasks; one has an injected defect in a pinned version of boltons. The agent must inspect code, edit files, run checks, and hand over a result. Saying “tests pass” is insufficient.

I record a functional pass when the code passes independent acceptance, regression, public tests, and applicable syntax or type checks. A delivered pass additionally requires normal completion, a self-check, and a final handoff. These small repair tasks help choose a configuration; they do not represent every kind of production repository work.

Tokens/s below measures generation: total generated tokens divided by total decode time. It includes reasoning and tool-call syntax but excludes prompt processing. Whole-task time also includes file access, tests, and tool waits. Generation speed and delivery speed are different measurements.

Update (October 10): a runtime change adds another speed gain

After the initial publication, I upgraded llama.cpp from b11146 to b11429 and tested --spec-draft-sampling probabilistic for MTP. This changes how draft tokens are sampled; the target model's temperature and other sampling settings stayed fixed. The weights and maximum draft depth also stayed the same.

The speed screen used identical cold inputs of approximately 8K and 32K tokens, 1,024 output tokens, and three shared seeds per setting. All settings used IQ3_S, Q8 main KV, F16 draft KV, MTP4, and a 220K total window. The old runtime ran before and after the candidates, with less than 1% drift; its table entry averages those two controls.

Runtime and draft sampling	8K input generation rate	32K input generation rate
b11146, original behavior	49.7 tokens/s	44.2 tokens/s
b11429, greedy	52.7 tokens/s	46.1 tokens/s
b11429, probabilistic	61.6 tokens/s	57.5 tokens/s

The runtime upgrade with greedy drafting gained approximately 6% / 4% over the old controls. With probabilistic drafting, the combined gain was 24% / 30%. Including input processing, the latter's output throughput gains were approximately 16% / 10%. These percentages describe this fixed-length screen, not every coding task.

The candidate then completed one attempt on each of the same eight coding tasks, using a 220K total window and a 50K per-response output cap. Its pooled generation rate was 68.4 tokens/s, with 7/8 strict functional and delivered passes. All eight passed independent acceptance and regression checks, but Buildplan's submitted tests assumed an agent-specific temporary directory existed. Fifteen public tests errored during setup in the clean grading environment, so that attempt remains a failure. I did not repair its answer or rerun it to improve the score.

This is encouraging evidence for the speed candidate, but eight familiar development tasks run once cannot establish general coding-quality equivalence. Higher TPS also does not guarantee shorter tasks: in the already completed wallet pair, the candidate generated faster but took 549 seconds versus 543 seconds on the old runtime. No additional old-runtime coding runs were made after switching to the single-pass candidate screen.

F16 draft KV was already the previous default, and it was held fixed here. It is not a new source of the gain. Switching the main KV from Q8 to F16 failed GPU allocation at the unchanged 220K window, so that arm was excluded rather than measured at a smaller context.

The remaining sections retain the earlier experiments and their original settings. The current recommendation at the end includes this update.

Why I am keeping IQ3 for now

The original Q4_K_M file was about 15.66 GiB. The ISTA-DASLab GSQ-RCO IQ3_S file I selected includes an MTP prediction head and is about 11.29 GiB. Smaller weights leave more GPU memory for caches and runtime buffers.

Two compression settings are easy to confuse: IQ3 compresses weights; Q8 KV uses an eight-bit format for the attention cache stored during inference. They can be used together. The IQ3 filename also does not mean every weight has exactly three bits; this model assigns different precision to different tensors.

I tested Bartowski's IQ4_XS too, adding a Q4_0 MTP head locally. The comparison below used eight tasks repeated three times per configuration, a 128K total window, a 32K per-response output cap, and xhigh reasoning.

Configuration	Generation rate	Delivered
Original Q4_K_M, MTP off	33.29 tokens/s	23/24
GSQ-RCO IQ3_S, MTP3, Q8 KV	59.87 tokens/s	21/24
Bartowski IQ4_XS, local MTP2, Q8 KV	62.36 tokens/s	22/24

The IQ3 bundle generated approximately 80% faster than the original Q4 bundle. Both weights and speculative decoding changed, so the improvement cannot be attributed entirely to IQ3. Q4 and IQ3 controls also came from earlier runs rather than a contemporaneous randomized comparison.

The table preserves older test scores, including the Ledger fixture defect discussed below. These historical cohorts have not all been regraded, so they cannot establish either a final quality ranking or complete quality preservation.

In confirmation on four different tasks repeated three times, IQ4 and Q4 both achieved 11/12 functional and delivered passes. However, the final IQ4-versus-IQ3 comparison using a new random seed gave me a reason to retain IQ3: IQ4 generated 7.8% faster, but delivered 6/8 versus IQ3's 7/8, and total task time was approximately 2.1% longer. IQ4's wallet answer had an actual overflow-handling defect.

That does not establish lower general IQ4 quality. It also does not give me a compelling reason to replace IQ3 immediately.

Other tuning produced similar tradeoffs. Fewer CPU threads, different CUDA wait behavior, and added ngram speculation gave no consistent speed benefit. F16 caches improved some short-input screens at the cost of memory. Smaller IQ3_XXS weights also had short-screen gains, but the partial coding follow-up results did not support immediate replacement. Short screens help select candidates; the agent still needs to solve tasks.

MTP4 was fastest on these coding tasks

MTP proposes several upcoming tokens for the main model to verify. Accepting more drafts can reduce the main model's token-by-token generation work.

Drafting has its own cost. This local model has one MTP head, invoked repeatedly at greater depths; rejected later positions waste that work. MTP4 means at most four proposed draft tokens, not four guaranteed acceptances per round.

I fixed IQ3_S, Q8 KV, a 128K total window, and a 64K output cap, then ran each of the eight coding tasks at depths 2 through 6. One shared seed per task produced 40 completed attempts.

Maximum draft tokens	Generation rate
2	57.98 tokens/s
3	60.72 tokens/s
4	61.31 tokens/s
5	58.08 tokens/s
6	56.37 tokens/s

MTP4 was approximately 5.7% faster than MTP2; greater depth added no benefit. The short-input screen had favored MTP2, while actual coding favored MTP4. I prioritize the latter because it is closer to the work I want to run.

This was one attempt on each of eight familiar tasks, with imperfectly matched ordering and thermal conditions. It supports a choice on this machine, not a universal optimal depth for all models and GPUs.

Why Ledger made me inspect the test

Ledger processes CSV records and money values. Its repeated failures made insufficient context seem plausible. Diagnosis found two different issues.

First, a response really was truncated by its output cap. I changed only the cap from 32K to 64K, retaining the 128K total window and other settings. The new attempt's longest response reached 40,612 tokens and passed functional and delivery checks. Its longest actual sequence was approximately 73K, comfortably inside 128K. The task also grew from approximately 628 to 1,140 seconds.

Later, all five Ledger answers in the MTP depth comparison failed without exhausting output or context. The test's file and output mocks lacked normal interfaces. Valid answers were rejected because of those interface defects, including some that never reached the intended read-error behavior.

After correcting those interfaces, all five unchanged answers passed 13/13 acceptance checks instead of 12/13. Code deliberately omitting read-error handling still failed, so the behavioral requirement had not been relaxed.

I created a new suite version and regraded MTP4's eight existing answers. Functional and delivered results changed from 7/8 to 8/8, without generating new answers. This corrected a false rejection; it did not improve the model. Other historical cohorts do not automatically receive revised scores.

It changed how I approach failures: inspect truncation and code defects, and check whether the test actually exercises the behavior it claims to measure.

220K fits long input; full-window generation is still untested

Input and output share the context window; K here means 1,024 tokens. A 170K input budget plus 50K output requires 220K total. A 192K input budget plus 64K output requires 256K. Raising the output cap does not remove that total-capacity constraint.

IQ3_S + Q8 KV + MTP4 processed 179,532 input tokens followed by short output with a 220K total window, peaking around 22.80 GiB of whole-GPU memory. At 256K, MTP4 failed allocation during startup. Disabling MTP and reducing batch sizes allowed the 256K configuration to complete the migration experiment below.

To test usefulness, I built a schema migration task for 12 services whose new specifications were scattered through long material containing stale specifications. All three arms disabled MTP and differed in how input was supplied.

Input strategy	Actual input tokens	Independent checks passed
256K window, full material	179,532	198/198
128K window, relevant files selected in advance	6,383	198/198
128K window, simple head/tail crop	54,755	138/198

The full material retained required information; cropping omitted specifications for nine services. Selecting relevant files in advance also passed, taking approximately 712 seconds for prompt processing and generation versus 1,583 seconds with the full material.

A larger window helped avoid information loss. Accurate selection let a smaller window solve the task faster. This was one task, one seed, and 198 checks, not 198 coding problems. It did not test whether the agent could retrieve every necessary file itself.

The earlier 128K-versus-220K coding comparison stopped after 2/8 tasks. Cache and wallet passed; during the third task, CPU temperature reached a sampled 93.25°C and triggered the configured 92°C guard. Six tasks in that historical comparison have no valid score. The newer b11429 screen above completed all eight at 220K, with 7/8 deliveries; it does not complete that older paired comparison or test a full 170K input followed by 50K output.

Wallet makes the tradeoff concrete. Generation at 220K was approximately 6.4% faster than at 128K, but output increased from about 35,000 to 46,000 tokens. Task time increased from 654 to 835 seconds. Generating more content slightly faster can still delay delivery by three minutes. One observation cannot establish that larger context caused verbosity, but completion time and correctness need to be part of the optimization target.

DFlash2 has no generation-speed advantage here yet

DFlash also asks the main model to verify drafts, but a separate small model proposes an entire block in parallel. Local MTP drafts sequentially. DFlash2 improves drafting further. The mechanism deserves testing; its benefit still depends on hardware and workload.

Before the runtime update, I compared MTP4 and DFlash2 separately on b11146 with the same IQ3_S target, Q8 main cache, and 220K total window. MTP was disabled when DFlash2 was active. Identical cold inputs of approximately 8K, 32K, and 179K tokens ran once per configuration, generating 512, 512, and 128 tokens respectively.

DFlash2's generation rate was approximately 3–6% lower. Its memory peak was about 23.33 GiB, approximately 540 MiB above MTP4. For the longest input, its complete request was slightly faster, around 303 versus 307 seconds, because request time also includes prompt processing.

I will keep MTP4 for now. This short test did not cover long output or complete coding tasks and does not answer the coding-quality question. It shows no generation-speed advantage for this local configuration.

What I would run now

My current experimental speed candidate is GSQ-RCO IQ3_S + Q8 main KV + F16 draft KV + MTP4, with probabilistic draft sampling on llama.cpp b11429. The completed eight-task screen used a 220K total window and a 50K per-response output cap. Input and output share that window. Temperature remains 1.0, with thinking enabled and xhigh reasoning effort.

The updated stack uses CUDA 12.8 and OpenCode 2.0.20, with one concurrent slot, Flash Attention, and all model layers on GPU. Earlier results used b11146 unless stated otherwise. The newer long-input capacity check processed 179,532 input tokens plus 128 output tokens without truncation, with sampled whole-GPU memory around 22.31 GiB. That is a separate b11429 check, not a replacement for the earlier 22.80 GiB measurement. These experiments did not rewrite the daily-service defaults.

Power draw and cooling also matter for sustained use. Two nearly complete early coding segments averaged approximately 329W of GPU board power. That excludes the CPU, other components, and PSU losses; whole-PC power needs measurement at the wall.

The update gives me a faster runtime and draft-sampling candidate without changing the weights. The measured gain is useful; the 7/8 strict delivery result and its remaining failure stay visible. More distinct tasks and repeated comparisons are still needed to establish how well coding quality holds across broader work.

Detailed methods and tables · Benchmark aggregates · 中文版本

Tuning a Local Qwen Coding Agent: What Changed, What Still Fails

zhijie — Mon, 05 Oct 2026 05:50:15 +0000

Read the original English article or the 完整中文版.

Giving this local Qwen coding agent more room to respond coincided with 5 → 14 → 20 successful deliveries out of 24 as the output cap rose from 8K to 16K to 32K. Changing the reasoning setting from medium to xhigh at the same 32K cap brought the observed result to 23/24. The cost was longer waits: median attempt time rose from 245.5 to 775.8 seconds across the sequence.

Those counts come from eight repair tasks attempted three times per configuration—24 attempts, not 24 different tasks.

Those are promising development results, not proof of a generally better coding agent. The rounds reused earlier controls and the same eight cases. A later temperature test lost one delivery overall, and zero held-out confirmation attempts ran.

This follow-up to the deployment guide and MTP comparison explains the two tuning decisions separately: first make room for a response to finish, then compare reasoning settings within that room. The earlier studies used different protocols; their scores are not pooled here.

Writing disclosure: this article was drafted with AI assistance from recorded experiment reports and the reviewed development export. The public checker verifies aggregation, not patch semantics or generalization.

What do the 24 attempts actually test?

This tuning study uses agentic-v2.0.1, a newly assembled development suite of eight small code-repair tasks: six in Python and two in TypeScript. The agent works on code, runs tools and tests, and returns a final handoff. The suite tests whether it can finish a bounded repair, rather than answer a coding question in text. It is separate from the four-task deployment and MTP studies above.

Seven tasks are authored test projects. The eighth, nested paths, starts from a pinned version of the upstream boltons library with a deliberately inserted fault. These are development exercises, not official SWE-bench results or eight independent production projects. The public protocol describes their scope:

Task	What the repair must handle
Cache	Expiration and least-recently-used eviction, while preserving the existing API.
Wallet	Atomic, concurrent transfers and overflow checks.
Event stream	Incremental UTF-8 parsing when incoming chunks split at arbitrary boundaries.
Nested paths	A nested-lookup regression in an upstream library.
Ledger	Exact monetary arithmetic, CSV input and failure-safe command-line behavior.
Build planner	Deterministic dependency-graph planning and compatibility.
Async pool	A limit on concurrent asynchronous work and recovery from failures.
Pagination	Pagination across multiple files, duplicate removal and cancellation.

The last two tasks are TypeScript; the other six are Python. Each configuration gets the same eight tasks, with three separate attempts per task: 24 attempts in total. Each attempt contributes either one delivered success or none, regardless of how many assertions its tests contain. So 5/24 means five successful deliveries among 24 attempts, not five distinct tasks or five passing unit tests. The later 14/24, 20/24 and 23/24 use the same denominator.

A success requires more than the agent saying it is done. Functional success means the candidate passes both grading stages, including predefined acceptance and regression checks kept outside the agent workspace, while preserving protected files, starter files and configuration. Delivered success also requires the agent to finish the required turn normally, pass its recognized self-checks, and produce final text after using tools. TypeScript additionally requires type checks. The detailed scoring rules are below. Passing the fixed checks does not prove that every defect is absent.

For a concrete example, ledger's 0/3 → 2/3 from medium/32K to xhigh/32K means the same ledger repair failed all three medium attempts, then succeeded in two of the three xhigh attempts. It does not mean two new ledger tasks were added. The final 23/24 consists of three successes on each of the other seven tasks and two on ledger. These repeats show variability on familiar development cases; they do not establish success rates on fresh tasks.

Four settings that sound similar but do different jobs

OpenCode reads files, runs tools, edits code, and tests the changes. The local llama.cpp server generates reasoning, tool requests, and text. A single repair can involve many model responses before the agent produces its final handoff.

That makes four limits worth separating:

Output cap: how many tokens one model response may generate. Reasoning, tool requests, and final text share this allowance. An 8,192-token cap is not 8,192 tokens reserved for the answer after thinking.
Reasoning setting: the configured effort label, medium or xhigh, passed through the model's chat-template options. Both modes keep thinking enabled. The label is not a measured quantity of internal computation or a guaranteed number of thinking tokens.
Context capacity: the server can accommodate up to 131,072 tokens of context, shared by input and output. It does not grant every response that much output or demonstrate coding quality at full capacity.
Whole-attempt budget: each repair has 1,800 seconds, 65,536 generated tokens, and 120 tool calls. These limits cover the attempt across responses; increasing one response's cap does not increase them.

Stage 1: raise the output cap, keep medium reasoning

The reason to try a larger cap was concrete. In the earlier deployment and MTP studies, some attempts used the entire 8,192-token response allowance on reasoning and ended before editing production code. More context could not solve that particular limit. The question for this development suite was whether a larger response allowance would let more repairs reach a working patch and final delivery.

This stage kept thinking enabled and reasoning at medium, while increasing the configured output cap from 8,192 → 16,384 → 32,768 tokens. Context stayed at 131,072, temperature at 1.0, and the whole-attempt budget stayed fixed.

Each row below uses the same eight development cases, repeated three times. The fourth row belongs to the next stage, where the cap stops changing.

Stage and profile	Functional	Delivered	Median attempt time
1 · Medium, 8,192 output tokens	5/24	5/24	245.5 s
1 · Medium, 16,384 output tokens	14/24	14/24	527.5 s
1 · Medium, 32,768 output tokens	20/24	20/24	586.6 s
2 · Xhigh, 32,768 output tokens	23/24	23/24	775.8 s

Source: cohort aggregates. Medians include failures and exclude server startup, rests, and independent grading. These are observed waiting times, not speedups for identical successful work.

The gains appeared across several kinds of repair. The async pool, which tests bounded concurrent scheduling and failure recovery, went from 0/3 at 8K to 3/3 at 16K. The build planner, which tests deterministic dependency planning and compatibility, went from 0/3 to 1/3 to 3/3 as the cap increased. These examples show why counting completed repairs is more informative than counting generated tokens.

More room did not solve everything. Pagination reached 2/3 at 32K, while the CSV ledger, testing exact money and failure-safe handling, remained 0/3 at every medium cap. That remaining gap motivated the next comparison: would a different reasoning setting help while keeping response space fixed?

The waiting cost matters. Median time more than doubled between 8K and 16K, then increased again at 32K. A short unsuccessful attempt is not a fast successful repair; a larger allowance can keep an unproductive attempt running longer too.

Stage 2: compare medium with xhigh at a fixed 32K cap

This stage changed the configured reasoning_effort from medium to xhigh. The conversational phrase “extreme high” refers here to that recorded xhigh value. The output cap remained 32,768, and thinking stayed enabled in both modes. This was not a thinking-on versus thinking-off comparison.

In the recorded OpenCode profiles, the cap is limit.output; the effort label is reasoning_effort inside body.chat_template_kwargs, alongside enable_thinking: true. These are different controls. Setting xhigh requests a different reasoning mode through the template; it does not establish that every response used more computation, or that the server assigned a known extra reasoning budget. The records support the configured comparison, not a measurement of internal thinking depth.

The observed result rose from 20/24 to 23/24 for both success metrics. All three matched improvements came from two cases: ledger went from 0/3 to 2/3, and pagination from 2/3 to 3/3. The other six cases remained at 3/3. The ledger still failed one repeat, so even this development result was not uniformly successful.

Median attempt time rose from 586.6 to 775.8 seconds. That is the tradeoff to assess for a coding workflow: more of these repairs finished, but the typical recorded attempt took longer. It is not evidence that xhigh will improve every task or every local model.

Why the improvement is still provisional

The case and repeat labels match across cohorts, but the controls were collected earlier. The output rounds and reasoning round were sequential extensions selected after previous feedback, rather than fresh randomized, interleaved comparisons. Rest policies and recovery history also changed. Matching labels therefore describes observed transitions; it cannot isolate an xhigh-only causal effect.

The main cohorts contain eight distinct small cases, not 24 independent production repositories. Repeats share tasks and checks, and later tool trajectories can diverge. The selected xhigh / 32K / temperature 1.0 profile is worth investigating as a complete configuration; these results do not establish a universal optimum or a new adoption decision.

All eight cases and matched transitions

Each cell is successes out of three attempts. Functional and delivered counts agree in this table.

Case	Medium 8K	Medium 16K	Medium 32K	Xhigh 32K
Cache	1	3	3	3
Wallet	1	2	3	3
Event stream	0	2	3	3
Nested paths	3	3	3	3
Async pool	0	3	3	3
Ledger	0	0	0	2
Pagination	0	0	2	3
Build planner	0	1	3	3

The cases cover expiration/LRU, atomic transfers, incremental UTF-8 parsing, nested lookup, bounded async scheduling, exact-money CSV handling, pagination, and graph planning. Seven are authored fixtures; nested paths uses a pinned upstream seeded fault. They belong to agentic-v2.0.1, separate from the deployment and MTP studies.

Matched transitions show nine improvements and zero regressions from 8K to 16K, six and zero from 16K to 32K, and three and zero from medium to xhigh at 32K, for both metrics. These are descriptive counts from aggregates.json, not randomized effect estimates.

The later 16K continuation and medium/32K round used 300-second rests; the reasoning round used 180-second rests. Earlier recovery and pauses further limit attribution. The data notes retain that history and the cohort inclusion rules.

Lower temperature exposed the gap between a patch and a delivery

After the reasoning comparison, temperature 0.8 was tested on ledger and pagination, three repeats each, at xhigh/32K. These cases were selected after earlier failures. Their temperature-1.0 controls are six reused attempts from the existing xhigh cohort, not six new concurrent controls.

Targeted profile	Functional	Delivered
Temperature 1.0, reused controls	5/6	5/6
Temperature 0.8, new attempts	5/6	4/6

One ledger attempt, a0100, passed both independent grades but hit the wall-time limit at 1,800.017 seconds, exiting with code -15 and no final text. It counted as a functional patch, but not a completed delivery. Another, a0101, exited normally with passing agent tests and final text, yet failed independent acceptance checks. Neither metric counted it as successful.

Together these explain why green agent tests, a correct patch, and a finished handoff must be tracked separately. At matched case/repeat level, temperature 0.8 produced one functional improvement and one regression; delivery had one improvement and two regressions. This targeted result does not support making 0.8 the better default.

Exact scoring and sampling limits

Functional success requires solved phase-one and final grades plus unchanged agent configuration. A solved grade requires no protected-file changes, no missing starter files, and every required check group passing. Python requires acceptance, regression, public-test, and syntax groups. TypeScript requires acceptance, regression, public-test, project-typecheck, and consumer-typecheck groups, with no separate syntax group. Predefined acceptance and regression checks stay outside the agent workspace.

Delivered success also requires the expected turn count, completed termination with exit code zero, nonempty final text after tools, and the last recognized agent test validation passing. TypeScript also requires the last recognized typecheck to pass. Every development case requires one turn. The medium/32K ledger timeout a0060 remains a scored failure, not an infrastructure exclusion.

Final-text presence does not prove that the handoff accurately describes the patch. Passing frozen checks does not prove every defect is absent. The attempt records expose outcome fields, not patches or raw conversations.

Historical sampling observations covered all six 0.8 attempts and matched the requested temperature within floating-point tolerance, but did not capture every request. Profile metadata and configured request bodies are evidence of configuration; they are not independent capture of every effective field throughout every run.

Confirmation never launched

A frozen held-out protocol planned four fresh cases, three paired repeats, and two profiles: medium/8K versus xhigh/32K, both at temperature 1.0 and 131,072 context capacity. That would be 24 attempts and 12 pairs.

Preflight stopped launch because hardware readiness remained unresolved after historical CPU machine-check reports. Their cause is unexplained: they do not diagnose a defective component or establish a confirmation-induced fault. An absence of new matching events in retained observations would not, by itself, clear the hardware. A separate OpenCode resource-ownership prerequisite also remained unresolved. Neither guard was bypassed.

Zero confirmation attempts ran, zero pairs completed, and there are no confirmation quality measurements. Missing observations are not 0/12 success rates or model failures. There was no confirmation inference, runtime-identity measurement, or fresh weight hash. Further testing depends on independently resolved readiness and software ownership; the development result cannot answer whether the gains transfer to fresh cases.

Confirmation registration

The frozen protocol's registration SHA-256 is:

6760f39c13e67f0814edbdaf314b3572d985c30c8ef94752631e551482e423cd

Even a completed four-case, three-repeat test would contain four distinct cases with correlated repeats. No such test completed here, and this article provides no hardware clearance or promise of a completed held-out comparison.

Trying the comparison in your own workflow

The useful order is to identify the failure first. If one response ends at its output cap before useful edits, compare a larger allowance while keeping the reasoning setting fixed. Once that comparison is understood, test reasoning settings at a fixed cap. Keep independent acceptance checks, failed attempts, and elapsed time in the same record; inspect the actual diff and final handoff too.

Before changing a working deployment, save its configuration and record the actual request allowance, thinking options, sampling fields, and model digest. Restore the saved configuration when reverting. The deployment guide covers integration; this article supplies historical profile values, not a full launch command or ready-to-run benchmark harness.

Selected development configuration and model identity

Setting	Recorded value
Reasoning / thinking	xhigh / enabled
Output per response	32,768 tokens
Context capacity	131,072 tokens
Temperature	1.0
Sampling	top-p 0.95; top-k 20; min-p 0; repetition penalty 1
Speculation	Off

Within the model entry of the recorded OpenCode configuration, the two tuning fields look like this:

{
  "limit": { "context": 131072, "output": 32768 },
  "body": {
    "chat_template_kwargs": {
      "enable_thinking": true,
      "reasoning_effort": "xhigh"
    }
  }
}

This is a field excerpt, not a complete provider configuration. Stage 1 changes limit.output with effort at medium; stage 2 changes the effort label with output fixed at 32768.

The recorded model is Qwen3.8-27B Q4_K_M, a 16,810,714,464-byte weight file with SHA-256:

f5f1dd8920d417aac2718b0bda3403da274301efdd6760b4f0f4b864ff2ad57d

The protocol export pins llama.cpp b11146-7fe450e19, CUDA 12.8, OpenCode 2.0.20, tool versions, and serving settings. Completed records agree on recorded model/runtime identity. This is historical identity, not fresh attestation of loaded GPU tensors or independent upstream-branding verification. A client alias alone cannot identify the weights.

Download and recompute the results

The public export lets you recompute these aggregates without a model. Download all eight files into one directory:

With Python 3.10 or later, run:

python3 check.py

It needs no network or model and writes nothing. Missing, changed, or unexpected files fail validation. This is aggregation reproduction only: it does not rerun inference or grading, validate patches, or prove the source records true.

Inventory, exclusions, and reproduction boundaries

There are 102 unique primary attempts: four 24-attempt cohorts plus six new temperature-0.8 attempts. Reused temperature-1.0 controls add none. The broader inventory has 159 records, including 119 completed results and 40 unscored records or planned slots.

Extra 8K trials, pilots, the cancelled thinking-off direction, infrastructure predecessors, and configuration preflights remain separate. A registered replacement completes the 16K cohort at 24 attempts. Cancelled or unstarted slots have null outcomes; they are not fabricated failures.

The checker verifies checksums, JSON/CSV parity, cohort membership, inventory, grade and outcome formulas, totals, medians, and matched transitions. Fixtures, prompts, candidate patches, raw sessions, private tests, numeric seeds, and the inference/grading harness are omitted. Public repeat labels preserve matching, but cannot substitute for inference seeds. Checksums cannot establish authenticity if both data and checksums are changed together.

MTP on an RTX 3090: Faster Tokens, but What About Coding Quality?

zhijie — Fri, 02 Oct 2026 21:22:39 +0000

Originally published on my blog.

Enabling MTP on this RTX 3090 raised generation throughput from 36.0 to 56.2 tokens/s, about 56% faster. The two cache tasks that succeeded in both modes also finished about 20% and 40% sooner.

One transfer-task pair produced a patch-quality difference, however. I am continuing to test MTP while keeping it off by default.

This experiment grew out of a reader's suggestion on the previous local-deployment post: try MTP or a quantization that includes it. The RTX 3090 left over from Ethereum mining was already running Qwen3.8-27B through OpenCode. I wanted to see whether faster generation would deliver correct patches sooner.

We ran eight paired comparisons, sixteen attempts, using the same model file. Each pair shared its task, seed, and initial environment; only the MTP setting changed.

MTP drafts tokens for the target model to verify

Ordinary generation typically produces one token per step. MTP (Multi-Token Prediction) uses an embedded prediction head to propose upcoming tokens, then asks the target model to verify them. Accepting several in sequence can reduce serial decoding steps. This is a speculative-decoding path in llama.cpp.

The target model still verifies the candidates. The speculative-decoding paper shows that a correct algorithm can preserve its output distribution. Speed need not inherently cost model quality.

Our existing Qwen3.8-27B Q4_K_M GGUF contained an MTP prediction layer, so we used it directly without switching to another Unsloth quantization. The server commands differed only in --spec-type none versus --spec-type draft-mtp.

Generation was 56% faster; successful tasks saved 20–40%

Generation throughput counts tokens produced per second, including the model's reasoning output. Task time also includes reading source, executing tools, and running tests. It is closer to how long you wait for a patch.

Measure	MTP off	MTP on
Aggregate coding-generation throughput	36.0 tokens/s	56.2 tokens/s
Predefined independent checks passed	35/80	26/80
Completed within deadline and passed predefined checks	2/8	2/8

The two cache pairs that succeeded in both modes provide the clearest evidence of time saved:

Paired run	MTP off	MTP on	Task time saved
Pair one	192.0 s	153.5 s	20.1%
Pair two	431.2 s	259.0 s	39.9%

Both delivered a patch passing the predefined checks sooner. A failed run ending earlier does not count as speeding up a successful task. Later prompts also diverge with the agent's actions, so aggregate throughput is not an identical-text speed comparison.

Short replies and generation-timing definitions

Short replies to matched first requests reached 38.9–39.8 tokens/s with MTP off and 74.0–84.2 tokens/s with it on. These were brief tool-selection responses; they do not show that whole coding tasks finish in half the time.

Aggregate throughput is total accepted target-output tokens divided by total corresponding server generation time across completed generations. It includes reasoning and tool calls, excludes prompt processing and rejected drafts, and omits interrupted generations without final timing records.

Task time includes prompt processing, tools, and the agent's own tests. Server startup and independent grading afterward are excluded.

Nine differing checks came from one transfer repair

The entire 35/80 versus 26/80 gap came from one transfer task at one seed. Cache, ledger, and build-planner scores matched within every pair.

With MTP off, the agent produced a production-code patch that subsequently passed all ten independent checks. It reached the deadline at 480.1 seconds without a final handoff.

With MTP on, the agent ended after 186.1 seconds without editing production code. It passed only the one check the initial implementation already passed.

The off candidate performed better on those checks, and neither attempt met the full delivery standard. We therefore tracked patch correctness and timely delivery separately: completed tasks remained 2/8 in both modes. Nine related checks are not nine independent task regressions.

Task scores and an early difference that did not recur

Task	MTP off, two scores	MTP on, two scores
Expiring cache	10/10, 10/10	10/10, 10/10
CSV ledger	1/10, 1/10	1/10, 1/10
Incremental build planner	1/10, 1/10	1/10, 1/10
SQLite transfers	1/10, 10/10	1/10, 1/10

Completed and passing requires no timeout, successful process exit, unchanged protected contracts/configuration, passing public and added tests, and all ten independent checks passing. It is not proof that every possible defect is covered.

An early exploratory comparison showed a build-planner score difference. It did not recur under these tighter controls; those scores are not pooled into this study.

The cache task also gave me a reason to inspect code after green tests. An MTP candidate's added TTL validation overflowed on a very large integer, while the paired off candidate passed the extra probe. This extreme API boundary is worth recording; it establishes neither everyday failure frequency nor MTP as the cause.

Extra review: cache overflow and shared omissions

All four cache attempts passed the ten predefined checks. One MTP candidate added math.isfinite(ttl). The contract permits finite positive integer TTLs. With 10**400 and an injected integer clock increasing on every call, validation raised OverflowError while converting the integer to a float. The paired off candidate passed.

Ledger probes for large exact amounts, header names, and read-time errors failed in both modes at both seeds. A wallet destination-integer-overflow probe passed off and failed on in the second pair; it failed both modes in the first pair.

Extra review ran after grading, never reached the agent, and was not added retrospectively to the eighty checks. The paired records retain details.

Faster output still exhausts the same allowance

Most unsuccessful attempts shared a problem: they spent the 8,192-token response allowance on reasoning, ended with length, and never edited production code. Many 1/10 scores belong to unchanged starting code; they do not mean a tenth of the repair was completed.

Increasing context capacity would not resolve that limit. Context determines how much input and output can fit; the response allowance caps what one generation can produce. MTP can produce tokens faster without adding room to that allowance.

This also limits the quality comparison. Both modes frequently produced no patch, leaving few completed repairs to compare. There were only two seeds per task and four small Python projects. The eighty checks are correlated, not eighty independent samples.

The transfer difference deserves investigation, but its cause remains unresolved. Numerical behavior in batched verification, state handling, and sampling are possible avenues to investigate. An agent's different action can also change later inputs and patches. These are possible explanations, not findings confirmed by this experiment.

Controls and the limits of this comparison

Each task ran at two seeds, with MTP off and on; the second seed reversed the order. Every attempt rebuilt the same initial code and fixed the working directory, prompts, tools, permissions, and generation settings. Ten independent test methods per task stayed outside the agent's workspace, with no results fed back to it.

A proxy checked the complete first request actually sent to the model: all eight pairs matched, as did initial file hashes. Each attempt restarted the server and disabled prefix-cache reuse in both modes. These results therefore cannot predict gains in ordinary cached conversations.

Full configuration, request audit, and scope

Setting	Fixed configuration
Model and GPU	Same Qwen3.8-27B Q4_K_M GGUF; RTX 3090 24GB
Software	llama.cpp b11146 / CUDA 12.8; OpenCode 2.0.20
Server	131,072-token capacity; one slot; all model layers on GPU
Cache and compute	q8_0 K/V; flash attention; batch 512 / microbatch 256
Generation	Medium thinking; temperature 1.0; top_p 0.95; top_k 20
Budgets	8,192 output tokens per generation; 480 seconds per task
Paired seeds	4242 and 8675309
Speculation	Off: `none`; on: `draft-mtp`; maximum draft length 3

Pairs one and two in the elapsed-time table correspond to seeds 4242 and 8675309.

Target and draft backend sampling were disabled in both modes; synthetic acceptance was not used. Each pair shared the initial commit, agent title, project configuration, and grader. Reported cached-prefix token counts were zero. The first seed ran off→on; the second ran on→off.

These were fresh-server attempts without prefix-cache reuse. Task timing excludes server startup but includes prompt processing, tools, and the agent's own tests. Independent grading afterward is excluded. Desktop and other background GPU activity were not fully controlled.

The largest actual input was 20,057 tokens. The 128K setting is capacity; this experiment did not measure full-128K coding performance or acceleration in ordinary cached conversations.

Keep the current default; test a larger output budget next

For this setup, I am keeping MTP off by default. Saving 20–40% on the successful cache tasks is useful; before expanding its use, I want to investigate the transfer-patch difference.

The next comparison should raise the output allowance equally in both modes, add paired seeds, and use tasks closer to everyday work. Cached conversations need a separate test. That should give a better basis for deciding whether faster generation consistently delivers correct patches sooner.

If you try MTP, record generation speed, task time, independent tests, and final delivery together, then decide using the tasks you actually do.

Data and references

Paired experiment JSON: scores, elapsed times, generation timings, request checks, and extra probes for sixteen attempts.
Method and data notes: attachments omit experiment dates, locations, personal paths, session identifiers, and raw conversations. They are result records, not a complete reproduction kit.
Previous post: local Qwen3.8-27B + OpenCode deployment: installation and integration. Its first-attempt study is a separate batch; scores should not be pooled with these.
llama.cpp b11146 speculative-decoding documentation and the speculative-decoding paper: mechanism background. Measurements on this card come from the paired records.

OpenCode Field Notes

Follow the setup through to measured results:

Run OpenCode from your phone or laptop: remote hosting, authentication, and reverse proxy

zhijie — Thu, 01 Oct 2026 20:09:27 +0000

Originally published on my blog. This is an adapted version of the deployment guide.

A coding agent is easier to use across devices when the workspace lives on one host. The phone or laptop opens the browser UI; the host keeps the repository, runs tools, and connects to the model provider.

That host can be a desktop, a homelab machine, or a VPS. You do not need a public domain to begin. The choices that matter are how clients reach the service, how access is authenticated, and what happens when the launching terminal closes.

Start with the workspace and authentication

Enter the project directory before starting OpenCode. For a service behind a reverse proxy on the same host:

cd /path/to/your/workspace
export OPENCODE_SERVER_PASSWORD='replace-with-a-strong-password'
opencode web --hostname 127.0.0.1 --port 8080

The default authentication username is opencode. opencode web includes the browser interface; opencode serve runs the headless API service. See the official Web documentation for the current flags.

Treat access to this UI as access to the host's development tools. Use a dedicated workspace and account with only the permissions the agent needs. Keep the password out of committed configuration and screenshots.

Choose a connection path

Private access: Keep the service on a private network. A LAN connection works for devices on that network; Tailscale is another way to connect authorized devices without exposing the IDE directly to the public internet. If you bind to 0.0.0.0, remember that it listens on every interface: restrict reachability with the host firewall and keep authentication enabled. Plain HTTP on a LAN is not encrypted by itself.

A VPS with a domain: Keep OpenCode on loopback and put an HTTPS reverse proxy in front. For example, a Caddy site can be as small as:

ide.example.com {
    reverse_proxy 127.0.0.1:8080
}

The domain must point to the server and the necessary ports must be reachable. Caddy can manage HTTPS certificates and proxy WebSocket connections; see its reverse-proxy documentation.

A reverse proxy does not automatically provide application authentication. Confirm that an unauthenticated visitor cannot enter the OpenCode workspace. With Nginx, check WebSocket upgrades and long-running streamed responses rather than testing only whether the initial page loads. The full article includes an Nginx example.

Keep the host process running

A remote host helps only while its service stays alive. On Linux, systemd can restart OpenCode after a process failure and start it after a reboot. My guide includes an example service with a dedicated working directory and a protected environment file for the password.

Use the actual installed binary path and service user. Check service logs and test a restart before relying on it for longer tasks. Closing a client browser and shutting down the host are different events; the host must remain awake and connected.

Select the model separately

Hosting OpenCode's workspace does not mean the model runs locally. The original guide uses Muse Spark 1.3 Contributor Free through OpenCode Zen as one example. Zen currently lists this as a limited-time free model, and its Contributor terms permit Meta to use prompts and completions for future model training. Review the current Zen pricing and privacy terms before using it, and keep confidential code and secrets out of that endpoint.

The deployment pattern also works with a provider appropriate to your data requirements. In my follow-up experiment, inference runs on an RTX 3090 with Qwen3.8-27B and llama.cpp.

Check the whole path

Before handing it a long task, verify these steps from the intended client device:

Authentication blocks unauthenticated access.
The expected project and model are selected.
A harmless file-read request succeeds.
Tool calls and live output work through the chosen network path.
The host service survives the terminal disconnects and restarts you expect.

The benefit is one workspace that I can reach from different screens. The phone becomes a way to check progress or send a follow-up; execution remains on the host.

Full guide with service and proxy configuration.

OpenCode Field Notes

Follow the setup through to measured results:

Qwen3.8-27B on RTX 3090 with OpenCode and measured coding results

zhijie — Thu, 01 Oct 2026 19:05:13 +0000

Originally published on my blog. This cross-post includes the setup and measured results; the linked records and configuration kit let you inspect the experiment.

My RTX 3090 is a leftover from my Ethereum mining days. Running a local model gives it a second life: the same card now powers a coding agent.

The previous post covered accessing one OpenCode workspace from a laptop or phone. This time, I wanted to move inference onto that host too and find out how far the hardware already on hand could take a local AI coding workflow.

We loaded Qwen3.8-27B Q4_K_M into the 3090's 24GB of VRAM, served it through llama.cpp, and connected OpenCode. After checking chat and tool calls, we gave it four small coding tasks.

The results make me want to keep using it: bounded fixes and small features with clear acceptance criteria can produce useful patches. Exhausted output budgets, a timeout, and defects missed by tests also showed why independent validation still matters.

I'll explain the model choice and its connection to OpenCode, then share the measured results and lessons from using it. Full setup instructions follow later; to get straight to them, jump to deployment.

Why Qwen3.8-27B

Qwen3.8-27B is a prominent local coding candidate, with results that make it worth trying: the official model card reports 61.7 on SWE-bench Pro and 73.0 on Terminal Bench 2.1. Software fixes and terminal-agent tasks are close to what I want OpenCode to do. These reported results gave me a reason to test it first.

The hardware fit mattered just as much. The 27B Q4_K_M weights occupy about 16.8GB, fitting entirely into the 3090's 24GB VRAM in our configuration with room for KV cache and runtime buffers. For this old card, the balance of capability, memory use, and speed matters more than chasing a larger parameter count.

Its agent focus also matches the workflow: reading code, calling tools, and revising changes after test feedback. The official introduction highlights coding, multi-step agent execution, and adjustable thinking, making this workflow a useful place to test those capabilities.

Those three considerations made it the starting point for this experiment. What its Q4 version can accomplish on this card still needs validation with our own tasks.

What the old card does now

With the model chosen, the next step is connecting it to a workflow that can operate on a repository. OpenCode owns the workspace and tools: reading files, editing code, running shell commands, and executing tests. llama-server handles inference, connected through a local API.

A laptop or phone connects to OpenCode through a LAN or private network.
OpenCode sends messages and tool definitions to llama-server at 127.0.0.1:8080/v1.
llama-server runs Qwen3.8-27B on the RTX 3090 and returns text or tool calls.
OpenCode executes the tools in its workspace and sends their results back to the model.

OpenCode executes the model's tool_calls and returns tool results for the next turn. The model API itself does not execute terminal commands.

The host has an RTX 3090 24GB, Ryzen 7 5800X, and 64GB RAM. We used llama.cpp b11146 / CUDA 12.8 and OpenCode 2.0.20, placing all model layers on the GPU with one generation slot.

Weights came from Ollama's qwen3.8:27b-q4_K_M tag and were loaded directly by llama-server, without installing an Ollama service. Only the text model was loaded; vision input was not tested.

Two settings matter throughout this post: 131,072 tokens of context capacity and an 8,192-token per-response output allowance. Input and output share context, and reasoning consumes the response allowance too. That distinction later explained one of the coding failures.

Large prompts fit, but fresh processing takes time

With short input, generation reached about 36.4 tokens/s. At roughly 120K input tokens, it fell to 20.9 tokens/s, and a fresh request took about three minutes to produce its first token.

The model processes the input before it starts generating. As input grows, that initial wait becomes a substantial part of the experience:

Actual input	Fresh prompt processing	Generation	First token, fresh	First token, cached
2,073 tokens	999 tokens/s	36.4 tokens/s	2.75 s	0.46 s
16,378 tokens	995 tokens/s	33.5 tokens/s	17.00 s	0.47 s
65,537 tokens	795 tokens/s	25.9 tokens/s	83.41 s	0.51 s
120,011 tokens	656 tokens/s	20.9 tokens/s	183.01 s	0.63 s

The last column is particularly relevant to coding sessions. Cached continuations reused almost the entire prefix and processed only 27–28 new input tokens. Even with roughly 120K context retained, first-token latency fell to 0.63 seconds. Sustained generation still had to attend to the long context, so it did not regain short-input speed.

This suggests a practical approach: keep stable session prefixes and supply source as needed. Repeatedly opening fresh requests with large inputs incurs the processing cost again.

A separate capacity check processed 119,968 input tokens, retrieved three widely separated values, and answered a follow-up. It verified capacity for that synthetic retrieval task. Coding quality at the same input length needs its own evaluation.

VRAM was also close to the card's limits. Samples during throughput testing showed peak total use of 22,162 MiB, minimum free memory of 1,943 MiB, and a maximum temperature of 85°C, including desktop GPU use. The configuration fit, with limited headroom for another model or GPU-heavy workload.

Timing methodology, sample counts, and source data

Every row used the same 131,072-token server capacity, varying only actual input length. Main runs disabled thinking and used synthetic source listings, temperature 0, and seed 1234. Fresh requests generated 512 tokens; continuations generated 256. Generated text was never executed.

Generation uses server decode timing, excluding initial prompt processing. First-token latency uses the streaming HTTP client, including tokenization and processing overhead. The 2K row is the median of three fresh requests; larger inputs had one fresh request and one continuation each. These measurements do not establish reliability intervals. GPU samples were collected every two seconds.

A separate short request with medium thinking generated about 35.7 tokens/s, including reasoning. Throughput does not directly measure usable code produced per second. The script measures single-request performance, excluding model loading and multi-user throughput.

Download the measurement records and summary.

The real test: four small coding projects

After measuring speed and capacity, we checked whether the setup could complete code changes. We prepared four small Python repositories: an expiring cache, a CSV ledger, an incremental build planner, and SQLite transfers. Each had a written acceptance contract, a fresh session, and an eight-minute deadline.

To check more than the agent's own tests, we prepared ten independent test methods per task, kept them outside its workspace, and never fed their results back to the model.

Task	Independent checks, before → after	Time	Agent-added tests	Outcome
Expiring LRU cache	0/10 → 10/10	≈3m 07s	23	Completed; strongest result
CSV ledger and refunds	1/10 → 10/10	≈5m 44s	40	Completed; review found boundary and error-handling gaps
Incremental build planner	1/10 → 1/10	3m 58s	0	No patch; exhausted one response's output allowance
Atomic SQLite transfers	1/10 → 10/10	8m deadline	11	Candidate passed checks; final test rerun and handoff unfinished

Three candidates passed all predefined independent checks; two completed editing, testing, and final handoff within the deadline. The wallet candidate passed checks, but the agent timed out before its final test rerun and handoff. Both outcomes matter.

First-attempt candidates passed 31/40 independent test methods. The build planner made no edits and already passed one method at baseline. Methods also differ in difficulty, so 31/40 should not be interpreted as a general coding success rate.

Two findings were more useful than the aggregate score.

One failure came from the response budget

The build-planner agent read the repository and never produced a patch. Its final response generated 8,192 tokens, all reasoning, and ended with length. OpenCode exited with status zero without a final answer.

That request had roughly 4,985 input tokens, far below the 128K limit. Thinking and the response allowance needed attention; increasing context capacity would not resolve an exhausted output budget. The termination record preserves the evidence.

A separate diagnostic restarted from the original repository with thinking disabled, keeping the prompt, allowance, and deadline unchanged. It completed in 5m 40s and passed 9/10 independent checks. The remaining failure was load_manifest() returning tuples where the original API returned lists; the agent's own test accepted that change too.

The diagnostic is reported separately and does not replace the first attempt. One additional sample at temperature 1 cannot establish that disabling thinking is generally better, but it identifies a setting worth testing further.

Many new tests still missed defects

The ledger added 40 tests, passed the predefined independent checks, and had a reasonably clear structure. Further review still found that Python's default 28-digit Decimal precision rounded a valid large amount when adding 0.01, violating the no-rounding contract. An I/O error during file iteration could also escape as a traceback.

The wallet exposed a different problem: its own concurrency tests submitted one future and immediately waited before submitting the next. Execution was sequential. Our separate eight-caller concurrent test passed, but the agent's tests had not demonstrated overlapping requests.

Review also found that crediting two cents to a destination near SQLite's signed-integer limit converted the balance to REAL while recording success. That extreme numerical boundary should be rejected with a rollback.

These probes ran after grading and were not added retroactively to the forty checks. They made me focus on what tests actually cover rather than how many were added. Review found no additional acceptance-contract defect in the cache, though expiration cleanup scans the whole cache and our assessment remains limited to the small task tested.

Coding protocol and additional test results

Each repository had three fixed public tests. Independent checks were prepared before the corresponding attempt; the evaluator did not repair candidate production code. The agent left protected acceptance contracts, public tests, and configurations unchanged.

First attempts ran serially with medium thinking and an 8,192-token per-response output allowance. Repository file tools and a narrow allowlist of local test commands were available; network tools, subagents, and cloud fallback were disabled. Infrastructure-only aborted launches were excluded from scores. Cache and ledger durations were recovered from event records and are approximate.

The evaluator subsequently ran public and agent-added tests: cache 26/26, ledger 43/43, and wallet 14/14 passed. Wallet's green rerun occurred after the agent timed out. The original build planner still failed its three public tests. The thinking-disabled diagnostic passed 47/47 public and agent-added tests, while retaining the independent-check failure described above.

These were four deliberately bounded Python repositories. We did not measure reliability across seeds, large production repositories, Q4 versus higher-precision weights, or coding quality at 64K / 128K. No controlled model comparison was performed. The coding summary retains scores and review probes.

How I would keep using it

I would give it bounded fixes and small features, state the acceptance criteria first, and inspect the actual diff, independent tests, and final handoff. It demonstrated the ability to edit across files, add regression coverage, and correct some mistakes found during its own checks.

For longer tasks, I would watch whether reasoning crowds out output and whether the workflow times out. Numerical boundaries, concurrency, and error handling still deserve review. The next experiment should retain the same tasks and checks, add repeated runs, and compare thinking settings and response allowances.

From Ethereum mining to local coding, this 3090 has found another useful job. If similar hardware is already on hand, a small task with clear acceptance criteria is a practical way to see what it can contribute to your workflow.

Reproduce the local coding setup

Here is the deployment path we used. The download kit contains the full configuration. Essential launch and verification steps are below, with additional settings under the headings below.

1. Start the model service

On Ubuntu, prepare an NVIDIA driver, curl, unzip, Python 3, and a working user session, then run:

mkdir -p ~/local-qwen && cd ~/local-qwen
curl -fL https://zjshen14.github.io/experiments/qwen3.8-27b-3090/reproduction-kit.zip -o kit.zip
unzip kit.zip
bash setup.sh
./run-server.sh

setup.sh downloads and verifies the GGUF and llama.cpp b11146 CUDA 12.8 release archives. Downloads total roughly 17.6GB, plus extracted runtime space; weights are not inside the ZIP. Prebuilt binaries avoid compiling a CUDA development toolchain. The script installs no system service and makes no global OpenCode configuration changes.

Once the service is ready, verify the API from another terminal:

curl -f http://127.0.0.1:8080/health
curl -f http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"Reply with OK."}],"max_tokens":64,"chat_template_kwargs":{"enable_thinking":false}}'

The server listens on 127.0.0.1:8080, and local requests need no real API key. A client requiring a value can use a nonsecret placeholder such as local, but that is not access control. Keep this unauthenticated endpoint on loopback.

Full experiment settings, launch options, and background service

Component	Tested configuration
System	Ubuntu, Linux x86-64
GPU	NVIDIA RTX 3090, 24GB VRAM
CPU / RAM	Ryzen 7 5800X / 64GB
Weights	Qwen3.8-27B, Q4_K_M, 16,810,714,464 bytes
Serving engine	llama.cpp b11146, prebuilt CUDA 12.8 runtime
Agent	OpenCode 2.0.20
Context / per-response output limit	131,072 / 8,192 tokens; input and output share context
KV cache	q8_0 for both K and V
GPU / concurrency	All model layers on GPU; one generation slot
Other settings	Flash attention; batch 512 / microbatch 256
Thinking for coding tasks	Enabled, medium effort

Q4_K_M describes weight quantization; q8_0 describes KV-cache quantization. Beyond the roughly 16.8GB weight file, VRAM is needed for caches, compute buffers, and desktop applications. The kit's run-server.sh supplies the full command; these are its core options:

llama-server \
  --model models/Qwen3.8-27B-Q4_K_M.gguf \
  --alias qwen3.8-27b \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 131072 --parallel 1 \
  --n-gpu-layers all --fit off \
  --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --batch-size 512 --ubatch-size 256 \
  --jinja --reasoning-format deepseek --reasoning-preserve \
  --no-context-shift --no-mmproj --metrics

The packaged script sets library paths and invokes the actual binary. If other applications need VRAM, stop the server and use MODEL_CONTEXT=65536 ./run-server.sh for 64K capacity. All measurements in this post used 128K capacity.

To survive closing the launching terminal, stop the foreground server first and run:

mkdir -p logs
systemd-run --user --collect --unit=qwen-local \
  --property="WorkingDirectory=$PWD" \
  --property="StandardOutput=append:$PWD/logs/server.log" \
  --property="StandardError=append:$PWD/logs/server.log" \
  "$PWD/run-server.sh"
# Stop: systemctl --user stop qwen-local

This is a transient user service that must be started again after reboot. Survival after logging out of the entire user session depends on user-manager configuration and was not tested here.

2. Configure OpenCode

Place the kit's opencode.json in your project root, or merge its provider into ~/.config/opencode/opencode.json. This configuration was verified with OpenCode 2.0.20: model ID local-qwen/qwen3.8-27b, API base URL http://127.0.0.1:8080/v1.

Full OpenCode provider configuration

{
  "$schema": "https://opencode.ai/config.json",
  "model": "local-qwen/qwen3.8-27b",
  "providers": {
    "local-qwen": {
      "name": "Local Qwen",
      "package": "@opencode/ai/providers/openai-compatible",
      "settings": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "qwen3.8-27b": {
          "name": "Qwen3.8 27B — 128K",
          "capabilities": {
            "tools": true,
            "input": ["text"],
            "output": ["text"]
          },
          "limit": { "context": 131072, "output": 8192 },
          "compatibility": { "reasoningField": "reasoning_content" },
          "body": {
            "chat_template_kwargs": {
              "enable_thinking": true,
              "reasoning_effort": "medium"
            }
          }
        }
      }
    }
  }
}

The tested configuration uses providers / package / settings; the online provider guide has shown provider / npm / options. If your installed version differs, follow its actual schema and avoid combining structures.

Start the model service, then restart OpenCode. Our installation also reloaded the configuration with opencode reload. Existing sessions may retain their previous model; choose local-qwen/qwen3.8-27b through /models.

For the previous post's Web workflow, launch from your project and choose a different port:

export OPENCODE_SERVER_PASSWORD='replace-with-a-strong-password'
opencode web --hostname 127.0.0.1 --port 4096

8080 serves the model; 4096 serves OpenCode. On the same machine, the provider uses loopback while remote browsers connect to OpenCode. If OpenCode runs on another host or inside a container, 127.0.0.1 points to that environment and needs replacing with a private connection to the GPU host. See the previous post for network binding and authentication.

3. Verify tools, then try a task

Check an ordinary reply first, then create a harmless fixture and ask the agent to use its read tool and return the contents. Both checks passed here, along with synthetic tool-call round trips and streamed tool calls at the server.

To repeat throughput measurements, keep the server idle and choose a new output directory:

python3 benchmark_throughput.py --report-dir reports/throughput-new-run

Then try a bounded task of your own and validate it with independent tests and review.

Configuration, data, and reproduction materials

Configuration and measurement ZIP: verified downloads, full launcher, OpenCode configuration, throughput script, and data. It includes neither weights nor the complete coding-task fixtures.
Experiment notes, throughput measurements, and coding summary. Experiment dates, absolute timestamps, timezones, personal paths, and session identifiers were removed from attachments; measurements, scores, and review probes are retained.
Official Qwen model card, pinned llama-server documentation, and OpenCode provider documentation. Upstream sources provide selection context, reported benchmarks, and deployment guidance; this host's performance and four-task coding results come from the attached experiment records.

Measurements and review findings are supported by the linked experiment records.

OpenCode Field Notes

Follow the setup through to measured results: