Previous posts in this series covered the infrastructure: running three AI models sequentially on one server and why I run speech-to-text locally instead of calling a cloud API. This one is about the business problem I'm building toward.
The problem
I run a solo consulting practice. Client communication arrives over three channels: phone calls (including voicemails I check hours later), messaging apps, and email. None of these talk to each other.
After a busy stretch, I found myself doing a manual audit: scrolling back through four or five apps, trying to reconstruct what commitments I'd made, who I hadn't responded to, what questions were still open. It's time-consuming and I kept missing things.
The expensive miss is a phone call. Someone calls, leaves a voicemail, doesn't follow up in email. If I'm mid-task when the call comes in and don't process the voicemail until the next day, that lead has likely moved on. Industry data suggests small businesses miss 40–60% of incoming calls. I don't know exactly what my number is, but I've been that business.
The tools I evaluated each handle one channel. Transcription services for calls. Email summary tools for inbox. Nothing I found reads all three and produces a single output. The integration gap is where the problem actually lives.
What I'm building
The goal is a daily summary: here's who contacted you, here's what they wanted, here's what's still open, here are the likely follow-ups.
The infrastructure pieces are working individually. Whisper running locally handles call transcription—audio stays on my server, never goes to an external API. bge-m3 handles embeddings for semantic search. Gemma handles images and screenshots. All running sequentially on one box.
What I'm building is the layer on top: intake → process → report.
Two problems I've been figuring out this week:
1. What the output should actually contain
Before building the generator, I spent time designing the output format. A daily report isn't useful if it's just a list of transcripts. The items that actually need to surface:
- Missed contacts: someone reached out and I haven't replied
- Commitments: explicit statements I made that need follow-through ("I'll send that over by Friday")
- Scheduling candidates: date/time mentions that might be pending appointments
- Open questions: things they asked that I said I'd look into
The hard part wasn't picking the categories—it was making each one specific enough to actually implement. "Show me missed follow-ups" is ambiguous. "Flag any contact where they reached out and I have no outbound reply within 48 hours" is implementable. I went through the format item by item and rewrote vague categories as concrete conditions.
2. Linking the same person across channels
If someone calls Monday and emails Tuesday, those land in two separate systems with no shared ID. To produce a per-contact summary, I need to link them.
This is an entity resolution problem—the same class of problem I work on in my main client work, building ER pipelines for corporate data. The structure is identical: match records that don't share a unique identifier.
For comms data, the linking fields are usually phone number or email address. Complications:
- People use multiple numbers (mobile, work, home)
- Email aliases (
john.smith@company.comandjsmith@company.com) - Display names don't match across apps
My current sketch: normalize phone numbers (strip formatting, expand country codes), match email with tolerance for common alias patterns, and handle name variants as a fallback signal rather than a primary match key. I haven't implemented this yet—this is still the design phase.
Where things stand
Working: Local transcription (Whisper), local embeddings (bge-m3), local VLM (Gemma). All running sequentially on one server. Audio, documents, and images stay local.
Building: The pipeline that takes call transcripts + email content + message content and produces the daily report. The output format is designed. The entity resolution approach is sketched. Nothing is integrated end to end.
The gap between "pieces working individually" and "system producing something I'd actually use daily" is what I'm working through now. Next concrete step: get the report generator to produce output I'd actually want to read, not just a proof of concept that technically runs.
Previously in this series: Running three AI models on one local server | Why local STT instead of a cloud API
Top comments (3)
The part I would be strict about is entity matching. False merges are worse than missed links in this setup, because one bad merge can make the daily summary confidently attach a private voicemail to the wrong client.
I'd probably start boring here. Stable identifiers win first, then phone and email, then fuzzy name matches only when the evidence is narrow enough to show in the report.
Nice writeup! Two things I liked:
Question: for entity resolution, are you more worried about false merges (wrong people linked) or missed links?
Suggestion: tag each report item with its source channel/snippet so you can quickly spot-check accuracy before trusting it.
Good luck getting it end-to-end!
Local-first is the right instinct for this use case. The business risk is not just latency. It is where the raw call audio, transcript, and customer intent end up sitting after the missed call is “handled.”