Generating realistic exam listening tests with TTS: voices, pauses and timestamps

Listening is the part of a language exam where you can't slow down. The recording plays, you answer, and the next one starts. If you're building practice material for it, a plain audio file isn't enough: it has to behave like the real thing, including the instructions, the pauses to read the questions, and the second play.

I'm building ExamReady, a platform for preparing for German B1/B2 exams, and listening practice is one of the parts I had to figure out from scratch. This post is about how I generate the audio with text-to-speech, assemble it with ffmpeg, and get timestamps out of the build so the app can show transcripts and highlight text while the audio plays.

ExamReady is an independent project and is not endorsed by ÖSD, ÖIF or the Goethe-Institut.

The problem

I needed listening practice that:

• has several speakers who sound different from each other
• includes the exam framing: instructions, time to read the questions, and a second play
• lets the learner see a transcript afterwards, and ideally see which line is being spoken right now

I didn't want to hand-edit audio in a DAW for every recording. Every change to a script would mean re-exporting by hand, and I'd lose track of what was where. I wanted the audio to be a build artifact, produced from a text script.

Step 1: describe the recording as segments

Each recording is a script made of segments. A segment is either speech (a voice and some text), a silence, or a chapter marker.

type Segment =
  | { kind: "speech"; voice: string; text: string }
  | { kind: "silence"; seconds: number }
  | { kind: "chapter"; title: string };

const recording: Segment[] = [
  { kind: "chapter", title: "Instructions" },
  { kind: "speech", voice: "narrator", text: "You will hear a short conversation twice." },
  { kind: "silence", seconds: 5 }, // example value
  { kind: "chapter", title: "Conversation" },
  { kind: "speech", voice: "speakerA", text: "..." },
  { kind: "speech", voice: "speakerB", text: "..." },
];

The important idea is that everything the learner hears is in this list, including the silences. Silence is not something I leave to chance or to the TTS engine. If the exam gives you time to read the questions, that time is an explicit segment in the script.

Voices are just names mapped to a text-to-speech voice configuration, so a conversation between two people uses two different voices, and the narrator uses a third.

Step 2: generate speech per segment

For speech I use ElevenLabs text-to-speech. Each speech segment becomes its own audio clip. Here is the shape of the build step, in pseudo-code:

clips = []
for seg in script.segments:
    if seg.kind == "speech":
        clips.append(tts(voice=VOICES[seg.voice], text=seg.text))
    elif seg.kind == "silence":
        clips.append(make_silence(seg.seconds))
    # chapters produce no audio, only a marker

Generating clip by clip, instead of one long request for the whole conversation, has a practical benefit: I control exactly where each voice starts and stops, and I can insert silence between any two lines.

Step 3: assemble with ffmpeg and record the timestamps

Once I have the clips, ffmpeg concatenates them into a single file. The part that turned out to matter most is that the build also prints the start time of every segment.

To do that, the build measures the duration of each clip and keeps a running total:

t = 0.0
timeline = []
for seg, clip in zip(script.segments, clips):
    timeline.append({"segment": seg.id, "start": round(t, 2)})
    t += duration_of(clip)  # measured from the real audio, not estimated

The key detail is "measured from the real audio, not estimated". If you guess durations from the text length, the timestamps drift and the highlighting looks wrong after the first few lines. Reading the actual duration of each generated clip keeps the timeline honest.

Then the concatenation itself is a single ffmpeg call over a list of the clips in order:

ffmpeg -f concat -safe 0 -i clips.txt -c copy listening.mp3

Using stream copy only works if all clips share the same format and parameters, so it's worth making the silence clips match the speech clips. If they don't, re-encoding is the safer fallback.

What the timestamps are used for

The printed start times become data attached to the recording. The app uses them in two ways:

1. Transcripts. After the exam-style run, a learner can read the transcript, line by line, with each line linked to its position in the audio.
2. Highlighting. While the audio plays, the app compares the current playback time to the start times and highlights the matching text.

Because the timeline comes from the same script that produced the audio, the transcript and the audio can't drift apart. If I edit a line and rebuild, everything is regenerated together.

Baking in the exam framing

I put the exam framing into the file itself: the instructions, the reading pauses and the second play. That was a deliberate choice. A learner doing practice should experience the same rhythm as in the exam, not a clean recording with the pacing left to them.

The tradeoff is that the file is less flexible. If I wanted a version with no pauses, I'd have to build a second file. For exam practice I think that's the right tradeoff, and since it's all generated from the script, a second build isn't much work.

What I'd watch out for

A few cautions rather than hard-won certainties:

• TTS voices are not real exam audio. They're clear and consistent, which is good for practice, but real recordings can have different accents, speeds and background noise. I wouldn't claim generated audio is identical to what a learner will hear on exam day.
• Check pronunciation. Names, numbers, dates and times matter a lot in listening tasks, and TTS can read them in a way you didn't expect. It's worth listening to the output instead of trusting it.
• Keep the script as the source of truth. Don't patch the audio by hand. If something is wrong, fix the script and rebuild, otherwise the timestamps stop matching.

Takeaways

1. Describe the recording as data: segments with a voice and text, silences and chapters.
2. Make silences explicit. They are part of the exam.
3. Measure clip durations from the generated audio, never from the text.
4. Print the timeline from the build. Transcripts and highlighting come almost for free.
5. Listen to the output. A pipeline that runs without errors can still read a number wrong.

If you've built something similar, for exams or for any audio that needs to stay in sync with text, I'd be curious how you handled the timeline.

I'm building this at examready.at, feedback welcome.