Why my from-scratch Japanese LLM says "That sounds tough" to good news

Kate — Thu, 08 Oct 2026 16:12:18 +0000

TL;DR

Lilas is a ~40M-parameter Japanese chat model I built from scratch, tokenizer included. It kept answering cheerful remarks with "That sounds tough, please don't overdo it." A raw count of the training data pointed at one over-used phrase. Weighting the counts by how the data is actually mixed at training time changed the picture. The real gap is that the plain positive reply, "I'm glad to hear that" (それは良かった), is the rarest positive reaction the model sees, 3–5x rarer than the other positive replies and 4x rarer than "That sounds tough."

This post covers the diagnosis and the fix plan. The retraining has not been run yet, so there are no after-numbers.

Code and data: github.com/keito118/lilas-llm-from-scratch
Me: github.com/keito118

Background: what Lilas is

Lilas is a hobby project to build a Japanese conversational AI without any pretrained model. Everything runs on one consumer GPU (RTX 3060, 12 GB).

Part	Details
Model	GPT-style decoder, ~40M parameters (n_embd=512, n_layer=10, n_head=8, block_size=1024)
Tokenizer	Own byte-level BPE, vocab 16,000
Pretraining	Aozora Bunko + Wikipedia excerpts + public instruction data, ~960M characters (~610M tokens)
Fine-tuning	`User:` / `Lilas:` dialogue files in `index/`
Tools	Calculator, date, holidays, weather and Wikipedia lookups run in code and are injected into the prompt as reference text (RAG)

One detail matters for this post. The hand-made persona and dialogue files are tiny next to the public instruction data, so lilas/data.py copies them persona_repeat times into the training mix. The production model ("round2") used persona_repeat=21. Without this, the model forgot its own name.

The production model was picked after eight rounds of experiments in one day, logged in SCALE_UP_PLAN.md. The common lesson of those rounds: adding data helps only near the training examples and rarely generalizes to new phrasings.

The symptom

A typical failure:

User:  今日は天気がいいから洗濯物がよく乾きそう
       (The weather's nice today, so the laundry should dry well.)
Lilas: それは大変そうですね、無理なさらないでください
       (That sounds tough. Please don't overdo it.)

It was not a one-off. The held-out evaluations had been showing the same pattern:

Emotion-matching accuracy rose from 30–40% to 66.7%, but about a third still failed. The most common failure was a flipped polarity: a cheerful reply to bad news, or the reverse.
Adding more emotion data (round 8) did not help with close paraphrases. Training on "They sent my draft back for a rewrite" did nothing for "They told me to rewrite it again." Asked "Looking forward to the party?", the model started talking about curry.

So instead of adding more data again, I counted what was already there.

First pass: raw counts

The first pass took three steps and no training at all.

Find where the stock phrases come from. Grepping the data-generation scripts for common reaction phrases turned up five generators (emotion, greetings, long-context, arithmetic and seed dialogues) that all reuse the same handful of replies.
Count them. grep -o <phrase> index/*.txt | wc -l for each phrase over the full local training set (20 files, 68,955 Lilas: lines).
Check how the files are mixed. Reading lilas/data.py confirmed which files are copied persona_repeat times.

The headline number: "That sounds tough" (それは大変) appeared 364 times, while "I'm glad to hear that" (それは良かった) appeared only 25 times, about 15x fewer.

The first hypothesis followed directly. "That sounds tough" is the model's default escape hatch, and persona_repeat multiplies it further. That turned out to be half right.

Second pass: count what the model actually sees

Raw file counts are not what the model trains on, so I redid the count with each file multiplied by its real factor. The result overturned half of the first hypothesis.

The public repo holds only the curated dialogue files (16 files, 3,971 Lilas: lines). The large public instruction files stay local. Of the 364 "That sounds tough" hits, only 132 are in the curated files. 102 of those 132 come from a single template generator (generated_longcontext_ja.txt). The other 232 sit in the large instruction files, which are not upsampled.

Raw: grep over the full local training set (20 files). Weighted: repo files × their factor in lilas/data.py, plus the 232 non-repo occurrences ×1.

After weighting, "That sounds tough" drops from 31% of the reaction phrases to 17% and is no longer the most common. "I'm glad to hear that" stays the rarest positive reply. The upsampling did amplify something, just not the phrase I first suspected.

Revised hypothesis

The problem is not that one negative phrase dominates. It is that the model has almost no plain way to say "good for you."

The small-good-news reply is missing. "I'm looking forward to it" belongs to future plans. "That's wonderful" belongs to achievements. "I'm glad to hear that" fits everyday small wins like dry laundry, and it is the rarest positive reply in the training mix.
"That sounds tough" is a fixed two-sentence chunk. 43 of the 132 Lilas: lines containing it also say "please don't overdo it" (無理なさらないで). The failing reply is exactly that chunk.
The templates attach it to anything that isn't clearly happy. In the long-context generator, 65 of the 102 "tough" replies follow just two user lines (spilling coffee, a late train). The other 37 are the same "sounds tough, don't overdo it" line attached to mostly neutral remarks such as "I took my watch in for repair" or "I'm thinking of getting a massage." A small model can learn "anything that isn't clearly happy gets sympathy" without learning polarity at all.
When polarity is unclear, the model falls back to a high-frequency chunk. With no generic positive acknowledgement to fall back on, the negative chunk wins. This step is still a guess, and the retraining is where it gets tested.

This also explains why round 8 failed. More emotion examples written in the same style reproduce the same gap.

The plan

Five steps, roughly 6–9 hours. The order comes from what failed in earlier rounds.

Inventory conversational moves (0.5–1 h). A script that counts reactions, back-channel replies, topic changes, clarifying questions and follow-up questions per file, so gaps are visible as numbers.
Add polarity-matching data (2–3 h). Examples where the reply depends on whether the input is positive, negative or neutral, with "I'm glad to hear that" filled in across many topics. Neutral remarks get neutral replies, not sympathy.
Fill other thin conversational skills (1–2 h) found in step 1.
Add a polarity-mismatch check to the eval (1 h). Flag a negative reaction to a positive input automatically, not just by eye.
Retrain from the clean pretrained checkpoint and compare on held-out conversations (1.5–3 h) against the production model, with a fixed seed.

Lessons carried over from earlier rounds

What happened before	How it shapes this plan
Testing on conversations close to the training data made memorization look like generalization	Decide only on a frozen held-out set that never enters training (`eval/`)
Widening a recall task from 8 to 32 topics made it work 8 times in a row on unseen names	Use the same topic × slot recipe, with wide topic coverage, for polarity data
Training on 27, then 97, then 70 fixed facts kept blending entities together	Avoid a few memorized "answers"; spread reactions over many phrasings
Making only the model bigger made it worse	Keep the model size fixed and change only the data distribution
Run-to-run noise hid real effects	Fix the random seed for every comparison

Takeaways

A small model repeats the habits of its data. Before blaming model capacity, grep the corpus. The whole diagnosis took minutes of counting and zero GPU time.
Count what the model actually trains on. Raw file counts and the weighted training mix told different stories. Upsampling changes which phrases dominate.
Look at the shape, not the totals. Positive replies were never scarce overall: they match negative-leaning ones in the raw files and outnumber them in the training mix. Stopping at "too much negativity" would have led to deleting sympathy data, which fixes nothing.
Template generators leak their defaults. Most of the "tough" replies come from one generator, often attached to neutral remarks.

Limits

The counts are substring matches. They do not separate User: lines from Lilas: lines, except where noted.
The 232 occurrences outside the repo files are assumed to be used once (they are not upsampled by persona_repeat in lilas/data.py).
The repo holds the latest version of the dialogue files, which may differ slightly from what the production model trained on. That this distribution causes the bug is still a hypothesis. Step 5 is the test. I will add the results to this post.

Reproduce

From a clone of the repo, the curated files can be counted like this:

for p in それは大変 それは良かった それは楽しみ; do
  echo "$(cat index/*.txt | grep -o "$p" | wc -l) $p"
done

Multiply each file's count by its factor from lilas/data.py (persona_repeat=21 for the persona files, ×3 extra for knowledge files, ×5 extra for greetings) to get the weighted numbers.

DEV Community: Kate