Transcription is a solved problem. You pipe audio into any decent ASR model and get text back at 90%+ accuracy. If you think that's the hard part of building an "AI meeting assistant," you'll ship something nobody keeps using.
We learned this the slow way while building KenzNote, and I want to walk through the parts that actually turned out to be hard; because most of them aren't about the model, they're about the product decisions around it.
"Who said what" is a harder problem than "what was said"
Raw transcripts are a wall of text. The first thing that makes a transcript useful is speaker attribution, knowing that line 47 was the VP of Sales, not the engineer who joined five minutes late. Diarization (splitting audio into "who spoke when") is its own error-prone pipeline stage, separate from ASR, and the two don't always agree with each other.
A speaker change mid-sentence, two people talking over each other, someone joining from a noisy coffee shop; all of it degrades attribution quality in ways that compound downstream.
Once you have decent diarization, you get something more interesting for free: speaking-time distribution and participation patterns. Turns out "who talked for 80% of a meeting that was supposed to be a discussion" is a data point people actually want, not just a novelty stat.
Summarization is a compression problem with a business logic layer on top
A generic LLM summary of a call is fine. A useful summary requires deciding, ahead of time, what categories of information matter: decisions made, open questions, and action items. That last one is the deceptive part. "Extract action items" sounds like a prompt-engineering exercise. In practice it's an entity-resolution problem:
- Assignee detection has to resolve "can you take that" to an actual person, which means tracking who "you" refers to across turns of dialogue, not just NER on names.
- Deadline detection has to normalize "let's circle back Thursday" against the actual meeting date, which sounds trivial until the meeting happens the night before a holiday and everyone means different Thursdays.
- Not every commitment is phrased as a commitment. "I guess I'll look into it" is an action item. "That would be nice" is not. Getting that boundary right is most of the tuning work.
We treat this less like a single summarization call and more like a small pipeline: segment the transcript, classify segments, extract candidate items, then resolve entities against meeting metadata (attendee list, calendar context if available, date). Each stage is boring on its own. The value is in not skipping any of them.
"Chat with your meetings" is a retrieval problem, not a chat problem
The feature people ask for once they trust the transcripts is: let me just ask a question instead of reading anything. "What did we agree on pricing last Tuesday?" That's not a prompting problem, it's an indexing and retrieval problem, you need per-meeting and cross-meeting retrieval that respects timestamps and speaker context, not just semantic similarity over chunks of text.
Naive RAG over transcripts gives you answers that are technically relevant and practically wrong, because it'll happily surface a similar-sounding tangent from three meetings ago instead of the actual decision from the one you meant.
Getting retrieval to respect which meeting, when, and who turned out to matter more than which LLM sits at the end of the pipeline.
The architecture decision that actually mattered: privacy as a constraint, not a feature
Early on we had to decide whether meeting audio and transcripts would ever be used to improve our models, the default assumption for most AI products, and usually buried in a ToS nobody reads.
We went the other way: it's contractually excluded, for us and for any vendor in the pipeline, and everything is encrypted in transit and at rest.
This wasn't a marketing decision, it was an architecture decision made early, because "we'll bolt on privacy later" doesn't really work once training pipelines exist.
It constrains vendor choice and shrinks your options, but for a category where people discuss compensation, layoffs, and legal matters out loud, I think it's the correct trade.
We also skipped the assumption that we need standing calendar access. Most competitors want an OAuth grant to your calendar so a bot can auto-join everything.
We built the primary flow around local recording upload instead, you control what gets captured, full stop. A calendar-bot mode exists for people who want it, but it's opt-in, not the default posture.
Where that leaves us
None of this is exotic. ASR, diarization, structured extraction, and retrieval are all "solved" in the sense that there are libraries and APIs for each.
The actual product is in the seams between them; attribution errors that need graceful degradation, extraction that respects the difference between a commitment and a nice thought, retrieval that understands time and identity, and a privacy posture that's a constraint from day one rather than a patch afterward.
If you want to see how it holds up on your own meetings, KenzNote has a free two-meeting trial, no card required. I'd genuinely rather hear what breaks than get a compliment.
Top comments (0)