TL;DR — We released ja-punct-restore-v1, a small model that restores 、。?! into punctuation-free Japanese text, one prediction per character. On a real ASR benchmark (650 FLEURS Japanese utterances transcribed by Whisper, punctuation stripped, then restored) it scores punctuation F1 0.8881, against 0.6062 for Whisper's own punctuated output. On a sealed held-out test it scores F1 0.8156. ONNX included, ~16 ms per utterance on a desktop CPU, Apache 2.0.
- Post-processing beats hoping the ASR gets it right. Whisper's own punctuation on Japanese is inconsistent — fine for captions you read once, painful for transcripts you search, summarize, or feed to an LLM. A dedicated restoration step on the plain transcript measured far better than the recognizer's built-in punctuation on the same audio.
- Measured on real ASR output, not just clean text: FLEURS Japanese test set, transcribed with Whisper large-v3-turbo, punctuation removed, then restored by this model — F1 0.8881 (precision 0.8537, recall 0.9255) vs 0.6062 for Whisper's punctuated output as-is. Periods are nearly solved (F1 0.9806); commas carry most of the remaining error (0.8268).
- The honest weak point: exclamation marks. ! scores F1 0.4884 on the sealed test, with recall 0.3920 — the model is conservative and under-fires. And on the real ASR benchmark, ? and ! barely appear in the references at all, so their real-world ASR performance is effectively unevaluated. Details below.
The problem
Japanese ASR output often arrives as a wall of unpunctuated text, or with punctuation that follows the recognizer's guesses rather than the sentence structure. Everything downstream suffers: readability, subtitle breaks, sentence splitting for summarization, and any NLP pipeline that expects sentences.
Punctuation restoration is a small, well-bounded job: given the characters, decide where 、。?! go. That is exactly the kind of single-purpose part a small encoder can do on CPU, as a post-processing step after any ASR system.
The model
- ja-punct-restore-v1 — token classification over characters: for each character, predict one of none / 、 / 。 / ? / !
- Base model: sbintuitions/modernbert-ja-130m
- Apache 2.0, ONNX (fp32) included. The class-bias adjustment tuned on development data is baked into the classification head, so inference needs no extra correction step
- Speed: ONNX on a Core i7-11700F averaged 16.3 ms per utterance over 200 real ASR transcripts — cheap enough to run on every transcript
- Training data contains no real personal information: concatenated windows from the Japanese subset of FineWeb-2 (ODC-By 1.0), plus 15,000 synthetic colloquial sentences to cover spoken style
Use it
from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch
repo = "japan-opendata/ja-punct-restore-v1"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo)
model.eval()
PUNCT = {"COMMA": "、", "PERIOD": "。", "QUESTION": "?", "EXCLAIM": "!"}
def restore(text: str) -> str:
enc = tok(text, return_offsets_mapping=True, add_special_tokens=False)
with torch.no_grad():
logits = model(input_ids=torch.tensor([enc["input_ids"]]),
attention_mask=torch.tensor([enc["attention_mask"]])).logits[0]
pred = logits.argmax(-1).tolist()
id2label = model.config.id2label
out = []
for ch, (start, _), p in zip(text, enc["offset_mapping"], pred):
out.append(ch)
lab = id2label[p]
if lab in PUNCT:
out.append(PUNCT[lab])
return "".join(out)
print(restore("今日は良い天気ですね明日も晴れるといいですね"))
# 例: 今日は良い天気ですね。明日も晴れるといいですね。
Measured results
Sealed test (5,003 held-out windows, used for neither training nor tuning):
- Micro-F1 (punctuation only): 0.8156 (precision 0.8154, recall 0.8157), MCC 0.8284
- Per-symbol F1: 、 0.7591 / 。 0.9188 / ? 0.7300 / ! 0.4884
Real ASR benchmark (FLEURS Japanese, 650 utterances, Whisper large-v3-turbo transcripts with punctuation stripped, restored by this model, compared against the references):
- This model: F1 0.8881 (precision 0.8537, recall 0.9255)
- Whisper's own punctuated output, untouched: F1 0.6062
- Per-symbol F1 (this model): 、 0.8268 / 。 0.9806
One reading note: the sealed test is written-style text, the ASR benchmark is read speech. The two numbers measure different distributions, which is why both are printed.
What didn't work / Limitations
- Exclamation marks are weak. F1 0.4884 and recall 0.3920 on the sealed test — the model under-fires ! rather than over-firing it. If your transcripts are lively spoken content, expect missing exclamation marks, not spurious ones.
- ? and ! are unevaluated on real ASR output. The FLEURS references contain almost no instances (?: 0, !: 1), so the 0.8881 headline says nothing about those two symbols in the wild. Their only measurement is the sealed test above.
- Written text dominates the training data. Synthetic colloquial sentences patch the gap, but messy spontaneous speech — false starts, fragments, overlapping repairs — may score lower than either benchmark here.
- It only restores punctuation. No typo correction, no rewriting, no sentence reordering. Recognition errors from the ASR pass straight through.
This is the third in our line of small, single-purpose Japanese models — a PII detector that runs in the browser, a prompt-injection guard, and now punctuation restoration for ASR output. Same rules each time: free, ONNX included, measured on a sealed or public test set, failures printed on the card. Code for the series lives in one repository (linked below).
If you post-process ASR transcripts today, where does punctuation come from in your pipeline — the recognizer, a rules pass, or a model like this?
- Model on Hugging Face: https://huggingface.co/japan-opendata/ja-punct-restore-v1
- Code (all three models): https://github.com/japan-opendata-lab/ja-small-models
- Model page with usage notes: https://jpopendata.jp/models/ja-punct-restore-v1/
Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ (newsletter signup in the footer).
Top comments (0)