How I built a browser-based Chinese tone trainer with pitch-curve scoring

Chinese tones are hard to learn because feedback is hard to get. Most tools give you a model recording and ask you to imitate it. Very few tell you what your pitch actually did.

I wanted a free browser tool that records one syllable, extracts the pitch contour, and draws it on top of the target tone. No app, no signup, and no audio upload.

This is a technical write-up of how the tone trainer works: the browser audio pipeline, YIN pitch detection, contour normalization, tone scoring, and the extra work needed to score whole words and sentences.

You can try the result here: NihaoLearn tone trainer.

The constraints

The project is a static Nuxt site deployed on Cloudflare Pages. There is no audio backend. That forced a few decisions early:

• recording and analysis must happen entirely in the browser;
• the result must be understandable without a linguistics degree;
• the same code should work on a laptop and a phone;
• the user's voice should never leave the device.

The audio path is:

getUserMedia -> MediaRecorder -> Blob -> decodeAudioData
  -> Float32Array -> YIN frames -> normalized contour
  -> compare with reference contours -> score + explanation

Recording without destroying the signal

The first version used the default microphone constraints. That was a mistake.

Browser audio processing is designed for calls, not pitch analysis. Automatic gain control flattens the dynamic range, noise suppression can distort the fundamental frequency, and echo cancellation can smear short syllables.

The recorder now asks for a raw signal:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: false,
    noiseSuppression: false,
    autoGainControl: false,
  },
})

Recording uses MediaRecorder. A single syllable has a hard ceiling of 2.5 seconds, and a silence watcher stops the recording automatically after the learner stops speaking. Without that, a 0.5-second syllable would leave the user waiting for the full timeout.

Once the recording stops, decodeAudioData gives us a Float32Array of samples. From that point on, everything is pure TypeScript.

Pitch detection with YIN

The naive approach is autocorrelation. The problem is octave errors: a rising tone can be detected as a flat tone, or a low third tone can jump down an octave. For a tone trainer, that is not a small numerical error. It changes the feedback completely.

I used the YIN algorithm (de Cheveigné & Kawahara, 2002). The important part is the cumulative mean normalized difference function, which suppresses the octave and half-octave errors that plain autocorrelation is prone to.

The implementation has five steps:

1. compute the difference function for candidate periods;
2. convert it to the cumulative mean normalized difference (CMND);
3. find the first period below an absolute threshold;
4. refine that period with parabolic interpolation;
5. reject frames that are too quiet or outside the expected human range.

The current parameters are:

const opts = {
  sampleRate: buffer.sampleRate,
  frameSize: 2048,       // about 46 ms at 44.1 kHz
  hopSize: 512,          // about 11.6 ms
  minF0: 70,             // low male voice
  maxF0: 400,            // female / child voice
  threshold: 0.2,
  rmsFloor: 0.008,
}

Each frame produces a frequency and a clarity value:

interface PitchFrame {
  time: number
  f0: number | null
  clarity: number
}

f0 is sampleRate / period. Frames below 0.5 clarity are treated as unvoiced. This matters for silence and for consonants: including them in the contour would flatten the curve at the beginning and end of every syllable.

From frames to a tone contour

A contour is not just a list of frequencies. Pitch perception is logarithmic, so the first transformation is:

const semitones = hz.map((value) => 12 * Math.log2(value / medianHz))

After subtracting the median, the curve represents the speaker's shape relative to their own voice, not their absolute vocal range.

The contour is then resampled to 32 points. The 5th and 95th percentiles define the range, which keeps a single bad frame from stretching the whole chart. If the range is under 1.5 semitones, the contour is treated as flat and drawn as a horizontal line.

That "flat" rule is important. A nearly level line is usually a first tone, not a dramatic rise followed by a fall. Without it, a small amount of microphone noise becomes a fake mountain range.

const frames = detectPitch(samples, opts)
const contour = buildContour(frames, 32)

buildContour also returns rangeSemitones and voicedRatio. Those two values are useful later: a contour can be the right shape but too narrow, or it can be based on too little voiced audio.

The third-tone problem

Mandarin third tone often ends in vocal fry. YIN can lock onto a subharmonic there, producing a sudden drop of an octave. In one real example, the low part of jiǎng huà was estimated at -17 semitones relative to the word's median.

For whole-word scoring I clean those spikes with a 12-semitone deviation limit. For the isolated tone trainer I keep the raw contour, because changing that behavior would silently change the exercise.

Scoring an isolated tone

The reference tones use Chao tone letters:

Tone	Contour	Meaning
1	55	high and level
2	35	rising
3	214	dipping, with the turning point early
4	51	falling

The trainer does not compare Hz directly. Both the user and the reference are converted to the same normalized 0–1 shape. Then the learner's contour is compared with all four tones:

const ranking = TONES
  .map((spec) => ({
    tone: spec.tone,
    score: scoreAgainst(userShape, referenceShape(spec.tone, 32)),
  }))
  .sort((a, b) => b.score - a.score)

The distance-to-score mapping uses a calibration divisor of 0.45:

score = 100 * (1 - meanAbsoluteDifference / 0.45)

The most useful output is not the number. It is the ranking:

This sounds closer to tone 3 than to the target tone 2.

That tells the learner what to change. A score of 62 by itself does not.

The trainer also adds specific remarks:

• the pitch is almost flat;
• the rise or fall is too small;
• the first tone should stay level;
• the recording was too short for the tone to unfold.

Why words and sentences need a different scorer

The isolated tone trainer compares against ideal tone values. That is the right model for a single syllable, but it is wrong for real speech.

Tone sandhi changes the actual pitch. In 你好, the first syllable is often pronounced closer to a rising tone. Neutral tones do not have a fixed tone value at all.

So the shadowing scorer uses a different reference: the pitch contour extracted from our own model recording. The model already contains the sandhi, rhythm, and neutral tone. We do not need to invent a second linguistic model.

The analysis parameters are shared by both sides:

const ANALYSIS = {
  sampleRate: 16000,
  frameMs: 32,
  hopMs: 8,
  points: 32,       // words
  sentencePoints: 64,
}

Both the reference and the learner's recording go through the same resampling and frame settings. If one side uses a different frame length, the comparison measures the parameters instead of the pronunciation.

Compare semitones, not normalized shapes

This was a counterintuitive lesson.

For an isolated tone, normalized shape is the right space. For a whole word, it is not.

One model recording of mā had a small 2.2-semitone wobble from the TTS voice. Min-max normalization turned that wobble into a full 0-to-1 ramp. A learner who read the first tone as a straight line was then penalized for being more level than the model.

The fix was to compare semitone values after subtracting each side's median pitch. That preserves the size of the movement. A 2.2-semitone wobble stays 2.2 semitones. A learner who moves 1.5 semitones and a learner who moves 6 semitones no longer get the same score.

Timing needs to be allowed, but not free

People do not speak with the same timing as a model recording. A 10% timing shift should not make a correct contour look wrong.

The scorer uses a weighted banded DTW (Sakoe-Chiba) with a ±15% alignment band. The cost of matching frame i to frame j is:

const cost =
  Math.abs(reference[i] - user[j]) +
  3 * Math.abs((i + 0.5) / n - (j + 0.5) / m)

The second term is a warp penalty. Without it, the algorithm can "fix" a contour that only moves half as much by stretching the time axis. With the penalty, timing can absorb small alignment differences, but it cannot hide a missing rise or fall.

The final score is a weighted combination:

score =
  0.6 * melodyScore +
  0.2 * rangeScore +
  0.2 * durationScore

• Melody compares the semitone curve after alignment.
• Range checks whether the learner moved enough, measured in semitones.
• Duration compares speaking rate in log2 space, so speaking twice as fast and twice as slow are penalized symmetrically.

The word-level duration tolerance is deliberately loose: 0.45 in log2 space, with a decay of 0.9. A learner who reads carefully at 1.3–1.5× the model duration should not be heavily punished for being clear. A learner who takes twice as long still loses points.

What the user sees

The chart uses the five-level tone grid, not the normalized 0–1 shape. A first tone should look high, not like a line through the middle of the graph.

The learner's curve is mapped into the target tone's pitch range for display. Its absolute pitch is not meaningful, but its shape and its range are.

The text feedback is intentionally concrete:

• "Your pitch moves about half as much as the model."
• "Syllable 2 drifts the most from the model."
• "The model stays almost level here; keep it steadier."

The goal is to make the next attempt obvious.

What I would improve next

• Better handling of noisy laptop microphones and room echo.
• More work on third-tone vocal fry and creaky voice.
• Connected-speech scoring for longer passages.
• A clearer distinction between "wrong tone" and "right tone, unstable pitch."
• More calibration data from real learners, not only model recordings.

What I learned

The hard part of a pronunciation tool is not the FFT. It is deciding what to compare and how to explain the difference.

Three lessons stood out:

1. Calibrate with real recordings. Synthetic test cases are useful, but real voices expose octave errors, vocal fry, and timing differences that a clean sine wave never will.
2. Separate pitch from timing. If timing is part of the pitch score, the same mistake gets penalized twice. A separate duration score makes the feedback more actionable.
3. A score needs an explanation. "72" is not feedback. "This sounds closer to tone 3" is.

The tone trainer and the shadowing scorer are both free and work without an account. If you are learning Mandarin, try a few syllables and see whether the curve matches what you expected:

• Tone trainer
• Shadowing scorer
• Graded reading with HSK coverage

If you are building something similar, I would be interested in how you handle noisy microphones, tone sandhi, and the difference between a useful score and a misleading one.