TL;DR: I wanted a model small enough to bundle inside a memory tool (tens to hundreds of MB, CPU-only) that could judge whether two Japanese sentences make the same claim, contradict each other, or are unrelated — a Natural Language Inference (NLI) task. I benchmarked two NLI cross-encoders on 16 hand-labeled pairs written in the register of real work logs. The small Japanese model (111M params) failed paraphrase detection outright on this set (0/5). The multilingual one (279M) did better (10/16) and — with a confidence threshold — works today as a zero-false-positive contradiction detector. Neither solves "same claim" out of the box. Numbers, traps, and a possible way forward below.
Why I wanted this
I run a local, self-hosted memory system: every conversation turn with my AI assistant gets appended verbatim to SQLite. The tape never lies, but it accumulates. The obvious next layer is consolidation — merging entries that state the same fact, flagging entries that contradict each other.
The judgment at the core of consolidation is a three-way classification:
Given two sentences: same claim / contradiction / unrelated?
That's not generation. That's NLI, a task BERT-class models were solving before LLMs were cool. So instead of paying an LLM (API cost, GPU memory, or both) to babysit my logs, could a bundle-sized model (tens to hundreds of MB) do it? My constraints:
- CPU only, imperceptibly light (the user should not be able to tell it's running)
- Bundleable: same deployment shape as the OCR and embedding models my tool already ships (40–150MB each)
- Fail-safe direction: false merges are worse than no merges. Low recall is acceptable; false positives are not.
- Wall-clock time is unlimited (it's a sleep-time job)
The test set
16 hand-labeled Japanese pairs, written deliberately in the register of my actual logs — casual, technical, elliptical. Not textbook NLI sentences. Three axes:
- 5 paraphrase pairs (expected: entailment) — e.g. "検索ツールを構成から外した" vs "検索ツールは構成の一覧から削除された"
- 5 contradiction pairs — e.g. "バックアップはそのうちやろう" vs "バックアップは今すぐ実装する"
- 6 unrelated pairs, including 2 adversarial ones: same topic, different claims — e.g. "このツールは検索には悪くない" vs "このツールでは意味の蒸留までは厳しい". These are exactly what embedding cosine similarity gets wrong, and exactly what consolidation must not weld together.
(Tool names in the examples above are lightly anonymized; sentence structure and register are unchanged. Yes, n=16 is small — this is a smoke test to decide whether deeper investment is justified, not a paper.)
Hardware: consumer desktop CPU, single-pair inference, PyTorch, no batching.
Candidate 1: Japanese-only, 111M params
akiFQC/bert-base-japanese-v3_nli-jsnli-jnli-jsick — tohoku BERT-base v3, fine-tuned on JSNLI + JNLI + JSICK.
Trap #1 before any evaluation: the model's config.json has no id2label. You get LABEL_0/1/2 and nothing else. The mapping (0=entailment, 1=neutral, 2=contradiction) exists only in the README. If you guess the standard-but-wrong order, your accuracy numbers are garbage and you may not notice. I burned 20 minutes on this; a probe set of trivial pairs ("犬が走っている / 動物が走っている") is the fast way to pin the mapping empirically.
Results (16 pairs):
| axis | correct |
|---|---|
| paraphrase (E) | 0/5 |
| contradiction (C) | 2/5 |
| unrelated (N) | 5/6 |
| total | 7/16 |
Latency: 23.7 ms/pair (CPU, single pair, no batching).
(Counts are under the corrected label mapping — and yes, I first miscounted this table as 8/16 at 5 a.m.; recounting from raw logs before publishing is part of the job.)
On this set, the paraphrase failure is total: every single paraphrase pair — including near-trivial ones like "外した / 削除された" — collapsed to neutral. The model only caught contradictions that contained explicit negation ("使っていない" vs "毎日使っている"). A pragmatic contradiction ("そのうちやろう" vs "今すぐ実装する") sailed through as neutral.
Why: JSNLI is machine-translated SNLI, i.e. translated image captions. JNLI's pairs also derive from image captions (YJ Captions). The model has essentially never seen technical, colloquial, or elliptical Japanese. This is textbook distribution shift, and at 111M params there's no reserve of general knowledge to absorb it. The latency is spectacular — the semantics are not.
Candidate 2: multilingual, 279M params
MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 — mDeBERTa-v3-base fine-tuned on ~2.7M NLI pairs across many datasets. Japanese is mostly cross-lingual transfer.
Results (same 16 pairs):
| axis | correct |
|---|---|
| paraphrase (E) | 2/5 |
| contradiction (C) | 4/5 |
| unrelated (N) | 4/6 |
| total | 10/16 |
Latency: 389 ms/pair (CPU, single pair, no batching).
Better across the board except the two N misses (it saw contradictions that weren't there). Still weak on paraphrase, and it also missed the pragmatic "later vs now" contradiction. But here's the interesting part:
Threshold analysis. If I only act when P(contradiction) ≥ 0.95:
- 3 contradictions detected (at 0.976, 0.998, 0.984)
- 0 false positives in this set (the wrong contradiction calls scored 0.69 and 0.54 — comfortably below the bar)
That's exactly the fail-safe shape I need: a low-recall, high-precision contradiction alarm. Not a consolidator — an alarm. "This new log line disagrees with what your knowledge base says" is a useful, safe, shippable feature even when full consolidation isn't.
16× slower than the 111M model, but 389 ms on CPU is still nothing for a sleep-time job, and int8 ONNX quantization is commonly reported to buy back around 4× (I haven't measured that here).
What I concluded
- These two bundle-sized NLI models did not solve same-claim detection on my logs. The core judgment of memory consolidation — "are these two sentences saying the same thing?" — is precisely the axis where both models are weakest. Hypothesis refuted, with numbers. (My embedding model had already failed this differently: cosine similarity says "same topic", which is emphatically not "same claim" — that conflation is how you weld contradictions together.)
- Threshold-gated contradiction detection is usable today. High precision, low recall, zero LLM involvement, CPU-trivial. Deployed as an alarm, it fails silently — which is safe.
-
Read the README, not just the config. Missing
id2labelis a silent accuracy killer. Probe with trivial pairs before trusting any numbers. - The failure is specific, therefore fixable. The diagnosis is distribution shift, not model capacity. Which suggests:
The way forward (untested)
The one path I haven't run yet: domain distillation. Sample a few thousand sentence pairs from my actual logs, have a large LLM label them once, offline (the LLM appears at training time, never at runtime), and fine-tune the 111M Japanese model on pairs that look like my data. CPU fine-tuning of a model this size is slow but feasible, and sleep-time jobs have unlimited wall-clock budget.
Encouragingly, a paper at the Japanese NLP conference this year (ANLP 2025, Q4-2) studied domain adaptation for English-Japanese medical translation and found that supervised fine-tuning alone acquired domain knowledge about as well as continued pretraining + SFT — and that mixing in over 50% pretraining data actually degraded scores on low-frequency knowledge. Their domain isn't mine, so the caveat may or may not transfer, but it suggests I can try skipping the pretraining stage and go straight to labeled pairs. If the experiment happens, the follow-up post will have the numbers, whichever way they fall.
Repro notes
- Python 3.11,
transformers+torchCPU wheels +fugashi+unidic-lite(candidate 1 needs the MeCab tokenizer deps; candidate 2 needssentencepiece) - On Windows, watch out for: pip long-path failures inside deep venv paths, and
cp932console encoding clobbering UTF-8 output (PYTHONIOENCODING=utf-8) - Both models load with
AutoModelForSequenceClassification.from_pretrained(...); classify with softmax over logits of the(premise, hypothesis)pair
Drafted with my local AI pipeline; all measurements are from real runs on my machine and reproducible from the notes above. Opinions mine, benchmarks yours to check.
Top comments (0)