DEV Community

Japan Open Data Lab
Japan Open Data Lab

Posted on Edited on

Putting the punctuation back into Japanese ASR transcripts (F1 0.89 vs Whisper's own 0.61)

TL;DR — We released ja-punct-restore-v1, a small model that restores 、。?! into punctuation-free Japanese text, one prediction per character. On a real ASR benchmark (650 FLEURS Japanese utterances transcribed by Whisper, punctuation stripped, then restored) it scores punctuation F1 0.8881, against 0.6062 for Whisper's own punctuated output. On a sealed held-out test it scores F1 0.8156. ONNX included, ~16 ms per utterance on a desktop CPU, Apache 2.0.

  1. Post-processing beats hoping the ASR gets it right. Whisper's own punctuation on Japanese is inconsistent — fine for captions you read once, painful for transcripts you search, summarize, or feed to an LLM. A dedicated restoration step on the plain transcript measured far better than the recognizer's built-in punctuation on the same audio.
  2. Measured on real ASR output, not just clean text: FLEURS Japanese test set, transcribed with Whisper large-v3-turbo, punctuation removed, then restored by this model — F1 0.8881 (precision 0.8537, recall 0.9255) vs 0.6062 for Whisper's punctuated output as-is. Periods are nearly solved (F1 0.9806); commas carry most of the remaining error (0.8268).
  3. The honest weak point: exclamation marks. ! scores F1 0.4884 on the sealed test, with recall 0.3920 — the model is conservative and under-fires. And on the real ASR benchmark, ? and ! barely appear in the references at all, so their real-world ASR performance is effectively unevaluated. Details below.

The problem

Japanese ASR output often arrives as a wall of unpunctuated text, or with punctuation that follows the recognizer's guesses rather than the sentence structure. Everything downstream suffers: readability, subtitle breaks, sentence splitting for summarization, and any NLP pipeline that expects sentences.

Punctuation restoration is a small, well-bounded job: given the characters, decide where 、。?! go. That is exactly the kind of single-purpose part a small encoder can do on CPU, as a post-processing step after any ASR system.

The model

  • ja-punct-restore-v1 — token classification over characters: for each character, predict one of none / 、 / 。 / ? / !
  • Base model: sbintuitions/modernbert-ja-130m
  • Apache 2.0, ONNX (fp32) included. The class-bias adjustment tuned on development data is baked into the classification head, so inference needs no extra correction step
  • Speed: ONNX on a Core i7-11700F averaged 16.3 ms per utterance over 200 real ASR transcripts — cheap enough to run on every transcript
  • Training data contains no real personal information: concatenated windows from the Japanese subset of FineWeb-2 (ODC-By 1.0), plus 15,000 synthetic colloquial sentences to cover spoken style

Use it

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

repo = "japan-opendata/ja-punct-restore-v1"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo)
model.eval()

PUNCT = {"COMMA": "、", "PERIOD": "。", "QUESTION": "?", "EXCLAIM": "!"}

def restore(text: str) -> str:
    enc = tok(text, return_offsets_mapping=True, add_special_tokens=False)
    with torch.no_grad():
        logits = model(input_ids=torch.tensor([enc["input_ids"]]),
                       attention_mask=torch.tensor([enc["attention_mask"]])).logits[0]
    pred = logits.argmax(-1).tolist()
    id2label = model.config.id2label
    out = []
    for ch, (start, _), p in zip(text, enc["offset_mapping"], pred):
        out.append(ch)
        lab = id2label[p]
        if lab in PUNCT:
            out.append(PUNCT[lab])
    return "".join(out)

print(restore("今日は良い天気ですね明日も晴れるといいですね"))
# 例: 今日は良い天気ですね。明日も晴れるといいですね。
Enter fullscreen mode Exit fullscreen mode

Measured results

Sealed test (5,003 held-out windows, used for neither training nor tuning):

  • Micro-F1 (punctuation only): 0.8156 (precision 0.8154, recall 0.8157), MCC 0.8284
  • Per-symbol F1: 、 0.7591 / 。 0.9188 / ? 0.7300 / ! 0.4884

Real ASR benchmark (FLEURS Japanese, 650 utterances, Whisper large-v3-turbo transcripts with punctuation stripped, restored by this model, compared against the references):

  • This model: F1 0.8881 (precision 0.8537, recall 0.9255)
  • Whisper's own punctuated output, untouched: F1 0.6062
  • Per-symbol F1 (this model): 、 0.8268 / 。 0.9806

One reading note: the sealed test is written-style text, the ASR benchmark is read speech. The two numbers measure different distributions, which is why both are printed.

What didn't work / Limitations

  • Exclamation marks are weak. F1 0.4884 and recall 0.3920 on the sealed test — the model under-fires ! rather than over-firing it. If your transcripts are lively spoken content, expect missing exclamation marks, not spurious ones.
  • ? and ! are unevaluated on real ASR output. The FLEURS references contain almost no instances (?: 0, !: 1), so the 0.8881 headline says nothing about those two symbols in the wild. Their only measurement is the sealed test above.
  • Written text dominates the training data. Synthetic colloquial sentences patch the gap, but messy spontaneous speech — false starts, fragments, overlapping repairs — may score lower than either benchmark here.
  • It only restores punctuation. No typo correction, no rewriting, no sentence reordering. Recognition errors from the ASR pass straight through.

This is the third in our line of small, single-purpose Japanese models — a PII detector that runs in the browser, a prompt-injection guard, and now punctuation restoration for ASR output. Same rules each time: free, ONNX included, measured on a sealed or public test set, failures printed on the card. Code for the series lives in one repository (linked below).

If you post-process ASR transcripts today, where does punctuation come from in your pipeline — the recognizer, a rules pass, or a model like this?

Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ (newsletter signup in the footer).

Top comments (0)