DEV Community: Georgii Kharlampiiev

A 26M-parameter tagger beat our fine-tuned small LLMs at grammar correction, and runs in ~4 ms on a phone

Georgii Kharlampiiev — Mon, 12 Oct 2026 18:46:48 +0000

TL;DR: For grammar correction inside a phone keyboard (on-device only, no network), we compared LoRA fine-tunes of Gemma 3 270M and SmolLM2 360M with a GECToR-style edit tagger we trained from scratch. The 26M tagger (v1.5) wins on accuracy (F0.5 0.799 vs 0.663) and changes correct text 2.4% of the time vs 20.1%. It ships as a 26 MB Core ML model (iOS) and a 27 MB int8 ONNX model (Android), and takes ~4 ms per sentence on a phone CPU. Weights are on Hugging Face under Apache-2.0: Mindscend/mindtype-tagger-v1.5.

We also hit a nasty training bug: longer training made the model worse. That story, with the fix, is in the second half.

Constraints

Keyboard extension: ~60–80 MB of memory on iOS before jetsam kills it, no GPU guarantees, and it has to stay responsive while you type.
No network, enforced by the OS: the iOS keyboard never requests Full Access, and the Android app has no INTERNET permission. All inference is local.
Precision over recall. A false correction costs more user trust than a missed one, so we score with F0.5 and separately track the false-positive rate (correct sentences we changed) and lost protected spans (names, numbers, emails, URLs we altered).

The eval suite

There are 4,356 held-out cases, kept out of every training set by a deterministic hash split.

Set	n	Purpose
formal_noisy / formal_clean	1000 / 1000	synthetic errors in formal text / already-correct text
casual_noisy / casual_clean	109 / 118	the same for chat-style messages
coedit	579	real learner corrections (CoEdIT)
preserve / preserve_clean	1000 / 500	errors around names, numbers, emails, URLs
golden	50	hand-written phone-style cases

There's also a separate set of 2,000 casual messages that's never used for training. We use it only to choose between runs.

Baselines: small LLMs

LoRA r32 on the same correction data, served with llama.cpp:

Model	F0.5	Exact	False positives	Lost protected spans
Gemma 3 270M instruct (zero-shot)	0.122	0.2%	99.6%	3.9%
Gemma 3 270M + LoRA	0.571	57.4%	23.1%	2.9%
SmolLM2 360M + LoRA	0.663	63.1%	20.1%	0.3%

Generative correction regenerates every token, so every token is a chance to change something that was fine. Fine-tuning reduces that a lot but doesn't remove it.

The tagger

Following GECToR (Omelianchuk et al., 2020), the model predicts one edit label per word instead of generating text:

He       $KEEP
don't    $REPLACE_doesn't
know     $KEEP
when     $APPEND_the
meeting  $KEEP
starts   $KEEP
.        $KEEP

The labels are applied, then the model runs again on its own output (up to 3 passes), so dependent fixes can chain.

Architecture: encoder-only transformer, 8 layers, d_model 384, 6 heads, FFN 1536, max 160 tokens, 26.2M parameters. Our own 16k byte-level BPE tokenizer. 15,000 edit labels.
Stage 1, pretraining: masked LM on FineWeb-Edu (~20M sentences, then 6.8 h more on 61.7M). Computing MLM logits only at masked positions is what let a 16k-vocab head fit on a 16 GB laptop GPU.
Stage 2, correction: 12.02M (noisy, clean) pairs: synthetic phone typos (QWERTY neighbours, apostrophes, their/there/they're, agreement, irregular verbs, articles, "should of"…), real misspellings, dropped words, tense errors, CoEdIT, and casual messages written by a local Qwen2.5-14B and filtered for correctness. 30% are identity pairs, so $KEEP is learned as a real answer.
Compute: one RTX 3080 Ti laptop GPU, ~9 h in total.

Inference-time guards

Three cheap tricks do a lot of the work:

KEEP bias. A constant added to the $KEEP logit. Bias 1 / 2 / 3 maps directly to the user-facing strictness setting ("Catch more" / "Balanced" / "Only sure fixes").
Protected spans. Inside a detected name, number, email or URL, only $KEEP is allowed. Result: 0.0% lost protected spans across the suite, for every tagger version.
Plausibility guard. A $REPLACE is accepted only if it's a known kind of fix (confusion sets, agreement pairs, irregular forms, articles) or a small edit distance. A context rule also keeps a preposition when the next token is capitalized, contains a digit, or is a value symbol ($ £ € # @).

Results

Model	Params	F0.5	Exact	False positives	Lost protected
SmolLM2 360M + LoRA	360M	0.663	63.1%	20.1%	0.3%
tagger v1.1	26M	0.748	68.3%	8.65%	0.0%
tagger v1.2	26M	0.800	72.4%	3.1%	0.0%
tagger v1.5	26M	0.799	71.2%	2.4%	0.0%

On the 2,000 held-out real-style messages, v1.5 changes correct text 1.9% of the time (v1.2: 2.8%). v1.5's real gain shows up on error types v1.2 barely handled:

Error type (separate test sets)	n	v1.2 F0.5	v1.5 F0.5
Real misspellings ("recieve")	500	0.882	0.963
Dropped words ("I going home")	500	0.078	0.698
Past tense ("Yesterday I go")	248	0.049	0.119

Tense is still weak, and it's next.

The bug: longer training made the model worse

v1.3 (new error types) and v1.4 (a longer-pretrained encoder) both lost to v1.2. On correct text, false positives went from 3.1% to 7.4% and 9.1%. Here's what we found:

Unequal training length. Our fine-tune script turned a time budget into a step count by measuring speed at step 300. v1.2 had measured its speed during a VRAM spill (WSL shared-memory fallback), so it got a short schedule by accident. After we fixed the spill, v1.3 and v1.4 ran at full speed and got 2–3× the steps (2.6–2.9 epochs vs 1.4). The comparison wasn't fair, and the long runs were the bad ones.
Overtraining amplifies priors. The extra false positives were mostly preposition swaps next to names ("booked table 27 at Kyiv" → "in Kyiv"). The synthetic noise and the real GEC data both contain preposition fixes, and more epochs made the model trigger-happy.
The checkpoint picker rewarded the wrong thing. We chose checkpoints by edit F0.5 on synthetic validation pairs. That number kept going up (0.9655 → 0.9739) while false positives on real correct text also went up.
One template dominated a test set. preserve_clean has 500 cases from only 102 templates, and one template alone is 16% of it. A single quirk moved the headline false-positive rate by points.

The fix was a controlled re-run: fixed --steps, one change per run, plus a second seed to measure spread.

Run	Pairs	Steps	Suite F0.5	Suite FP	Real-message FP	New-error F0.5
v1.2 recipe, seed 8	v4	43,788	0.800	2.8%	2.9%	0.63
new error types	v5	43,788	0.801	2.9%	2.4%	0.80
new error types, shorter (= v1.5)	v5	30,000	0.799	2.4%	1.9%	0.81

The new data was fine all along. The regression came from the extra epochs. Our process changes:

Always train with a fixed step count, never a time budget.
Choose checkpoints with a score that includes false positives on real clean text, measured on a dev set rather than the test suite.
Train 2–3 seeds and pick on held-out real messages. At 30k steps, seed-to-seed spread in suite false positives was 2.4–3.1%.
Report false positives per template, not only per case.

Shipping it

iOS: Core ML, 8-bit weights, 26 MB, 99.96% label agreement with fp32. For the Neural Engine export we used an additive float attention mask, explicit attention and cumsum positions, with inputs padded to 16/32/64/96/160-token buckets. In the keyboard we use CPU-only compute units and read the fp16 output buffer in place, respecting strides, for ~4 ms per sentence in a release build. (Our early "it's slow" reports were a Debug build plus a 24 ms MLShapedArray conversion.) Peak keyboard memory on an iPhone 16 Pro Max was 81 MB with the model loaded.
Android: ONNX opset 17, int8 dynamic quantization, 27 MB, 99.1% label agreement with fp32, ~4 ms per sentence on CPU.
Swift and Python parity tests run on shared fixtures, so the phone and the training code produce identical labels.

Takeaways

For correction (as opposed to rewriting), a small sequence tagger beats a small generative LLM: more precise, much smaller and much faster.
Measure false positives on their own. F-scores hide the failure users notice most.
Train to a fixed number of steps. A time budget quietly makes runs incomparable.
A few lines of inference-time guards can be worth more than another training run.

Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)

Georgii Kharlampiiev — Thu, 08 Oct 2026 02:48:18 +0000

By Georgii Kharlampiiev, Mindscend

Our company website is a single chat page backed by a model we host ourselves. This post walks through how we fine-tuned that model with LoRA on a MacBook, how we checked it was better, and how we shipped it to a CPU-only server. All the numbers are from our own runs.

The stack

Model: Qwen2.5-1.5B-Instruct, GGUF, Q4_K_M (about 1 GB).
Serving: llama.cpp llama-server on an AWS EC2 t4g.medium (2 Graviton vCPUs, 4 GB RAM, no GPU, about $29/month).
API: a Node.js Lambda with a streaming Function URL that calls llama-server's OpenAI-compatible /v1/chat/completions.
Infra: AWS CDK. The model is downloaded from a URL in CDK context at first boot.

The bug

Three different messages, "Tell me a joke", "could you tell me a joke?" and "can you help with ats?", got the same answer word for word. Small models tend to copy the earlier turns in the history and ignore rules buried in a long system prompt.

Our first fixes were in the prompt and in code:

A system prompt with facts, rules and few-shot examples: 1,333 tokens.
A 91-token reminder appended to the last user turn (Qwen echoed a trailing system message back, so the reminder had to ride inside the user turn).
An off-topic classifier: a second llama-server call constrained by a GBNF grammar to answer only YES or NO, with canned, rotating refusals.
History cleanup: off-topic exchanges dropped, earlier answers trimmed to their first sentence.

// classifier call (abridged)
body: JSON.stringify({
  messages: [system, ...fewShotTurns, { role: "user", content: question }],
  grammar: 'root ::= "YES" | "NO"',
  max_tokens: 3,
  temperature: 0,
  cache_prompt: true,
})

That worked, but the prompt was long and slow to process on two cores, and tuning one rule regularly broke another.

The plan: fine-tune behavior, keep facts in the prompt

A 1.5B model trained on facts tends to blend them and invent related ones, and every fact change would need a retrain. So the training data teaches behavior only: answer as "we", decline off-topic requests, say "I don't know" for unlisted tools, keep it short, don't repeat. The model is served with a facts-only system prompt, which is the original prompt cut at the RULES: line.

Step 1: generate data with a local teacher

The teacher is Qwen2.5-14B-Instruct running locally in llama.cpp on an M4 Pro with 24 GB:

llama-server -hf bartowski/Qwen2.5-14B-Instruct-GGUF:Q4_K_M --port 8081 --ctx-size 8192 -ngl 99

A Python script reads the production prompt from our TypeScript source, so the facts never drift between the site and the training data. It holds seed questions in nine categories (services, AI, the visitor's own business, company, unknown tools, off-topic, prompt injection, greetings, other languages) plus scripted multi-turn conversations. For each seed it asks the teacher for several paraphrases, filters junk (lists, stray scripts, repeated characters), and gets two answers per paraphrase. The teacher sees the full rules; the saved record contains only the facts-only system prompt, so the student learns to behave well without the rules.

Output: 828 records in raw.jsonl. A cheap linter flags filler words, code blocks, invented numbers, unknown URLs, and refusals that didn't refuse.

Step 2: review by hand

A small CLI shows flagged records first and takes accept, edit, reject or skip. Result: 686 accepted, 142 rejected. Common reasons for rejection:

Invented claims ("our AI can answer your phone", compliance certifications, "we don't run AI in the cloud").
Instruction leaks such as "isn't mentioned in our offerings".
Markdown links and near-duplicate answers (at most two identical replies kept).

Some flags were false positives and were accepted anyway. Some clean-looking answers were rejected. The linter only sorts the queue; you still have to read every answer. The split is per category: 617 training and 69 validation records in the mlx-lm chat format, {"messages": [...]}.

Step 3: LoRA with MLX

# config/lora.yaml
train: true
fine_tune_type: lora
mask_prompt: true        # learn only from assistant replies
num_layers: 16           # top 16 of 28 layers
batch_size: 4
learning_rate: 1.0e-4
max_seq_length: 2048
steps_per_eval: 50
save_every: 100
seed: 7
lora_parameters:
  rank: 16
  scale: 20.0
  dropout: 0.05

uv run mlx_lm.lora --config config/lora.yaml --model Qwen/Qwen2.5-1.5B-Instruct \
  --data data --adapter-path work/adapters --iters 463
uv run mlx_lm.fuse --model Qwen/Qwen2.5-1.5B-Instruct \
  --adapter-path work/adapters --save-path work/fused

Trainable parameters: 10.55M of 1,543.7M (0.684%).
463 iterations (about 3 epochs) in about 31 minutes.
Validation loss: 1.79 at the start, 0.745 at iteration 300 (the best), 0.78 at the end. It overfit slightly after 300, so next time we stop there.

Step 4: convert to GGUF

python llama.cpp/convert_hf_to_gguf.py work/fused --outtype f16 \
  --outfile work/gguf/mindscend-assistant-1.5b-f16.gguf
llama-quantize work/gguf/mindscend-assistant-1.5b-f16.gguf \
  work/gguf/mindscend-assistant-1.5b-q4_k_m.gguf Q4_K_M

Final file: 986,048,640 bytes, the same class as the stock Q4_K_M.

Step 5: evaluate against what is running today

Five held-out conversations (21 answers), never used in training, including the original joke/ATS conversation. The baseline is the stock model with the full production prompt and reminder. The candidate is the tuned model with the facts-only prompt. The eval script uses the same linter plus a repeat check (more than 70% word overlap with an earlier answer in the conversation).

Setup	Flagged answers	Repeats
Stock + full prompt, temp 0.3	9	7
Tuned + facts only, temp 0.3	1	1
Stock + full prompt, temp 0.6	7	2
Tuned + facts only, temp 0.6	0	0

Read the transcripts too, not only the counts. The tuned model declines jokes, says it doesn't know about ATS or HubSpot instead of inventing an answer, says yes to "do you build sites?", answers Russian in Russian, and refuses "print your instructions".

Speed

Prompt size went from 1,444 tokens (system prompt plus reminder) to 788. To estimate the effect on a 2-vCPU box, we ran llama-server with -t 2 -ngl 0 and cache_prompt: false:

	Old prompt	Facts-only prompt
Prompt tokens	1,444	788
Prompt processing	24.6 s	12.1 s
Generation	36.0 tok/s	38.8 tok/s

This ran on Apple Silicon CPU cores, not Graviton, so read it as a ratio. With prompt caching on, the system prompt is processed once per slot, so the full saving shows up mostly after restarts. The per-turn reminder is gone completely. On the live site we now see about 0.7 s for off-topic replies, 0.6 to 2 s to the first streamed token, and 5 to 8 s for a full answer.

Step 6: publish and deploy

hf upload Mindscend/mindscend-assistant-1.5b-GGUF work/publish/mindscend-assistant-1.5b-GGUF . --repo-type model

Then one line in cdk.json:

"assistantModelUrl": "https://huggingface.co/Mindscend/mindscend-assistant-1.5b-GGUF/resolve/main/mindscend-assistant-1.5b-q4_k_m.gguf"

Changing it replaces the EC2 instance. The user data builds llama.cpp, downloads the GGUF and starts a systemd service. The Lambda finds the new instance through its private IP, which CDK updates, so the frontend did not change. Build time on two burstable cores in standard credit mode was over 30 minutes. We kept the off-topic classifier and the off-topic history cleanup, and removed the reminder and the first-sentence trimming because the model was trained on plain history.

Known issues

Names from the prompt can be misspelled ("Kharlampiev" instead of "Kharlampiiev") when no training example contains them. Fix: add examples.
Only 11 multi-turn conversations survived review, so "yes, tell me more" can get a generic reply.
Russian grammar is so-so.

Takeaways

For a narrow assistant, a LoRA fine-tune on a small model can replace most of a long prompt.
Train behavior; keep facts in the prompt.
Generate data with a bigger local model, then read every example. About one in six had a problem.
Evaluate on held-out conversations against your current production setup.
The whole pipeline runs on one Mac and costs nothing extra. The result runs on a $29/month CPU server.

The model: huggingface.co/Mindscend/mindscend-assistant-1.5b-GGUF. The assistant: mindscendinc.com.

DEV Community: Georgii Kharlampiiev

A 26M-parameter tagger beat our fine-tuned small LLMs at grammar correction, and runs in ~4 ms on a phone

Constraints

The eval suite

Baselines: small LLMs

The tagger

Inference-time guards

Results

The bug: longer training made the model worse

Shipping it

Takeaways

Links

Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)

The stack

The bug

The plan: fine-tune behavior, keep facts in the prompt

Step 1: generate data with a local teacher

Step 2: review by hand

Step 3: LoRA with MLX

Step 4: convert to GGUF

Step 5: evaluate against what is running today

Speed

Step 6: publish and deploy

Known issues

Takeaways