I wrote before about why I run speech-to-text locally—for me it started as a personal data hygiene decision. But when I've talked to other people about local AI pipelines, a different version of the same constraint comes up: not "I'd prefer the audio to stay local" but "the audio legally cannot leave our environment."
That's a different problem. A preference shapes your tooling choices. A compliance constraint shapes your architecture.
What the constraint actually looks like
The most common version I hear: enterprise customer agreements. A company builds a software tool that serves clients in regulated industries. Those clients have data handling requirements that specify data can only be processed in specific jurisdictions or on approved infrastructure. When that company wants to add AI-powered call analysis, every cloud transcription API is immediately out of scope—not because of policy preference, but because the customer contract requires it.
Attorney-client privilege is another version. Law firms and their clients have specific expectations about where privileged communications go. Using a third-party transcription service means the audio passes through systems outside that privileged relationship. Whether that technically breaks privilege is a legal question that most firms don't want to be the test case for.
Healthcare is the most formalized version: PHI (protected health information) that appears in patient calls needs to stay within HIPAA-compliant infrastructure. Some cloud providers offer BAAs (Business Associate Agreements) that extend that coverage, but plenty of healthcare organizations have decided the simplest answer is a processing pipeline that stays entirely on infrastructure they control.
The common thread: the constraint isn't about the transcription output (text) but about the audio itself, during processing. It's a data residency and custody question.
The architectural implication: it's not just transcription
Here's where this gets interesting from a systems perspective. If you accept the constraint that audio can't leave your infrastructure, and you're trying to build something useful from that audio—summaries, action items, entity extraction, follow-up flags—the constraint propagates through every step that touches the content.
You transcribe locally. But then what?
If you send the transcript to a cloud LLM for summarization, you've sent the content of the call off your infrastructure. The audio stayed local, but the information left. For most compliance constraints, this doesn't actually satisfy the requirement.
So the constraint ends up requiring:
- Local transcription (Whisper or equivalent)
- Local embeddings if you're doing semantic search across transcripts
- Local LLM inference if you're doing summarization, extraction, or classification
- Local storage, obviously
You end up needing a full local inference stack, not just a transcription component. The compliance requirement that started with "we can't use cloud STT" turns into "we need to run the whole pipeline on infrastructure we control."
This is the constraint I'm designing around
The setup I've been building runs Whisper for transcription, bge-m3 for embeddings, and Gemma for image/document processing—all on a single local server. I covered the resource management side of running three models on one box in an earlier post.
The "nothing leaves the box" constraint wasn't an afterthought. It was the first design decision, and it determined the whole stack. Every model choice, every pipeline component was evaluated against it: does this require sending data somewhere I don't control? If yes, it's out.
The tradeoffs are real. Local inference is slower than cloud APIs for equivalent hardware. You own the updates and maintenance. When a better Whisper model ships, you're updating your own deployment. The TCO calculation looks different depending on how much you're processing.
But for the use cases where the compliance constraint is genuine—where "it's in the customer agreement" or "it's a regulatory requirement"—those tradeoffs aren't optional. The architecture has to start from the constraint.
What I haven't built yet
The local inference stack is working. What I'm building toward is the layer that makes it actually useful: intake from multiple channels (calls, email, messages), process everything locally, surface what matters in a daily report.
The compliance constraint applies there too—not just transcription. Every step processes content that, in an NDA or regulated environment, can't transit external APIs. So the design for the report generation layer has to be local all the way through.
That's the hard part I'm working on now: not "can I transcribe calls locally" (yes, that's working), but "can I build the summarization and reporting layer locally at quality that's actually useful."
Other posts in this series: Running three AI models on one server | Why local STT over cloud API | Missed calls, missed revenue
Top comments (0)