We shipped a "chat with your transcript" feature on Longscribe (longscribe.com, free long-form transcription) recently, and the interesting engineering problems weren't the ones I expected going in.
The naive version doesn't survive contact with real transcripts
The obvious first approach: stuff the whole transcript into the prompt, let the model answer from it. That works fine for a 5-minute clip. It falls apart fast once users start uploading 2-hour interviews, lectures, and webinar recordings, which is most of what actually gets uploaded to a tool built for long recordings.
A 2-hour transcript is easily 20-30k tokens once you include speaker labels and timestamps. On a free-tier LLM API, that's either not going to fit in context at all, or it fits but eats your entire rate-limited budget on a single question, before the user has asked a second one.
Chunking a transcript is not like chunking a document
Most RAG chunking advice assumes prose: split on paragraphs, maybe respect headings. A transcript has none of that structure. What it has instead:
- Speaker turns of wildly different lengths (one-word answers next to five-minute monologues)
- Topic drift that doesn't line up with speaker changes at all
- Timestamps that are actually valuable to preserve, because "at what point does he mention the budget" is a completely reasonable question users ask
Splitting on a fixed token window ignores all of that and routinely cuts a sentence — or an answer — in half, right at chunk boundary. The fix that actually helped was chunking on speaker-turn boundaries with a target size, not a hard token cutoff, and keeping the nearest timestamp attached to every chunk so an answer can point back to "around 47:30" instead of just asserting something.
Grounding matters more than people expect
The failure mode nobody warns you about enough: a free-tier model, given a question it can't quite answer from the retrieved chunks, doesn't say "I don't know" — it blends the transcript with its training data and answers confidently anyway. For a transcript Q&A feature, a plausible-sounding wrong answer is worse than no answer, because the whole pitch is "trust what this says about your recording."
What worked: an explicit instruction to answer only from the provided chunks and say so when they don't cover the question, plus keeping retrieved chunk count visible in the UI so users can sanity-check the "why" behind an answer instead of taking it on faith.
Free tier changes the design, not just the budget
Running this on a free API tier (Groq, in our case) isn't just "the same feature but slower." It forces retrieval to actually be good, because you can't paper over sloppy chunking by just throwing a bigger context window and more tokens at the problem. That constraint turned out to be a feature-quality forcing function, not just a cost one.
Curious if anyone else building transcript/long-document Q&A has hit the same chunking-on-speaker-turns problem, or found a better heuristic than what I landed on.
Top comments (0)