How We Built a GDPR-Compliant AI Receptionist for Small Businesses

Building AI voice applications for real-world businesses is full of interesting challenges, especially when it comes to compliance and latency. When we started building Hallodesk, our goal was to create a reliable, production-ready AI receptionist that small businesses and medical clinics could trust with sensitive caller data.

In this post, we’ll dive into the technical details of how we built our system, managed low latency in voice streaming, and ensured strict GDPR compliance from day one.

The Architecture

A voice-based AI receptionist has a few core components:

1. Telephony Integration: Handling incoming calls, SIP trunking, and streaming audio.
2. Speech-to-Text (STT): Translating real-time audio streams into text.
3. Dialogue Engine (LLM): Generating natural responses based on business context and user inputs.
4. Text-to-Speech (TTS): Converting responses back into high-quality, natural audio.

Here is a high-level view of our architecture:

• We use WebSockets for real-time bi-directional streaming of audio between the telephony provider and our core backend server.
• The backend orchestrates the transcription, LLM generation, and speech synthesis.

Solving the Latency Problem

In human conversation, a delay of more than 500–800ms feels unnatural. However, chaining STT -> LLM -> TTS sequentially can easily exceed 2 seconds of latency. To solve this, we implemented several optimizations:

• Streaming STT: We stream audio chunks to our STT model and use endpointing algorithms to detect when the user has finished speaking.
• LLM Streaming & Chunking: We stream the LLM response token-by-token. Instead of waiting for the full response, we split sentences as they are generated and feed them to the TTS engine.
• Audio Pre-caching: Common filler phrases (like "Let me look that up for you...") are pre-cached to immediately respond while the main logic executes.

Ensuring GDPR Compliance

For European businesses, GDPR compliance is non-negotiable—especially for medical clinics handling patient names, appointment details, and health concerns. We built privacy into our architecture:

1. Zero-Data Retention (ZDR) APIs: We use LLM providers that guarantee data is not stored or used to train models.
2. Local Transcription & Synthesis: Wherever possible, we deploy self-hosted open-source STT and TTS models in European data centers to prevent data from leaving Germany.
3. Automatic PII Redaction: Before storing call logs for analytics, we pass them through a Named Entity Recognition (NER) pipeline to redact names, phone numbers, and other PII.
4. User-Controlled Data Policies: Business owners can configure automatic deletion of call recordings and transcripts after a set retention period.

Try It Out

If you want to see our voice agent in action or implement a similar widget on your own website, check out the Hallodesk Website Voice Agent.

We are constantly refining our pipeline to bring latency under 500ms while keeping security airtight. What are your experiences with building real-time voice applications? Let us know in the comments below!