Building a conversational AI platform that serves SMBs across 20 industries sounds straightforward until you hit the real constraints: latency, cost, privacy, and reliability — all at once. This is the story of how we architected SARA (our AI agent layer for WhatsApp), the decisions we made, and why we open-sourced the entire thing.
Repo: https://github.com/Alessandro114/sara
The core problem: 20 verticals, one conversation interface
SARA powers AI agents for restaurants, hotels, law firms, clinics, retail, logistics, and 14 other verticals — all through WhatsApp. Each vertical needs domain-specific tool access (a restaurant agent books tables; a clinic agent checks appointment slots). But the conversation interface is identical: a WhatsApp message in, a useful response out.
The naive approach is one giant prompt per vertical. That breaks fast: context windows fill up, prompts drift, and debugging is a nightmare.
Our approach: one core agent loop, 20 sets of tool definitions. Function calling is the vertical differentiation layer, not the prompt.
Architecture decision 1: Multi-provider LLM failover chain
We chain free-tier LLM providers in priority order: Groq → Cerebras → SambaNova → Mistral. Each is tried in sequence; on timeout or error, the next kicks in. The caller never knows which backend responded.
// ai-providers.ts — simplified provider failover chain
const PROVIDER_CHAIN: AIProvider[] = [
{
name: 'groq',
client: new Groq({ apiKey: process.env.GROQ_API_KEY }),
model: 'llama-3.1-70b-versatile',
maxTokens: 8192,
},
{
name: 'cerebras',
client: new Cerebras({ apiKey: process.env.CEREBRAS_API_KEY }),
model: 'llama3.1-70b',
maxTokens: 8192,
},
{
name: 'sambanova',
client: new SambaNova({ apiKey: process.env.SAMBANOVA_API_KEY }),
model: 'Meta-Llama-3.1-70B-Instruct',
maxTokens: 4096,
},
{
name: 'mistral',
client: new MistralClient(process.env.MISTRAL_API_KEY),
model: 'mistral-large-latest',
maxTokens: 8192,
},
];
async function getAIResponse(
messages: ChatMessage[],
tools?: ToolDefinition[]
): Promise<AIResponse> {
for (const provider of PROVIDER_CHAIN) {
try {
const response = await provider.client.chat.completions.create({
model: provider.model,
messages,
tools,
tool_choice: tools?.length ? 'auto' : undefined,
max_tokens: provider.maxTokens,
});
return { provider: provider.name, response };
} catch (err) {
console.warn(`[AI] Provider ${provider.name} failed, trying next...`, err);
}
}
throw new Error('All AI providers exhausted');
}
Key insight: the tool definitions are passed identically to every provider. OpenAI-compatible function calling is now a de-facto standard — Groq, Cerebras, SambaNova, and Mistral all support it, so the failover is truly transparent.
Architecture decision 2: Tool dispatcher pattern
Function calling gives you the LLM's intent ({ name: 'book_table', arguments: {...} }). You still need something to actually execute it. We built a centralized dispatcher with 30+ handlers — one per tool — all registered in a single map.
The key design principle: tool handlers are pure functions. They take validated arguments, hit the DB or an external API, and return a structured result. The agent loop feeds results back as role: 'tool' messages until the LLM stops requesting tool calls.
// tool-dispatcher.ts — dispatcher core + agent loop
type ToolHandler = (
args: Record<string, unknown>,
context: AgentContext
) => Promise<ToolResult>;
const TOOL_REGISTRY: Record<string, ToolHandler> = {
book_table: handlers.bookTable,
check_availability: handlers.checkAvailability,
get_menu: handlers.getMenu,
create_ticket: handlers.createTicket,
get_appointment: handlers.getAppointment,
// ... 25+ more across 20 verticals
};
async function dispatchTool(
toolCall: ToolCall,
context: AgentContext
): Promise<ToolResult> {
const handler = TOOL_REGISTRY[toolCall.function.name];
if (!handler) return { error: `Unknown tool: ${toolCall.function.name}` };
const args = JSON.parse(toolCall.function.arguments);
const validated = validateToolArgs(toolCall.function.name, args);
if (!validated.ok) return { error: validated.error };
return handler(validated.args, context);
}
// Agent loop: keep calling tools until the LLM stops asking for them
async function chatChainWithTools(
messages: ChatMessage[],
tools: ToolDefinition[],
context: AgentContext
): Promise<string> {
let response = await getAIResponse(messages, tools);
while (response.tool_calls?.length) {
const toolResults = await Promise.all(
response.tool_calls.map(tc => dispatchTool(tc, context))
);
messages.push(
{ role: 'assistant', content: null, tool_calls: response.tool_calls },
...toolResults.map((r, i) => ({
role: 'tool' as const,
tool_call_id: response.tool_calls![i].id,
content: JSON.stringify(r),
}))
);
response = await getAIResponse(messages, tools);
}
return response.choices[0].message.content ?? '';
}
Why function calling over prompt-only approaches?
- Structured output by default — no regex parsing of free-text responses.
- Composable — the LLM can chain multiple tools in a single turn.
- Auditable — every tool invocation is logged with exact arguments, giving a full per-conversation audit trail.
Architecture decision 3: PII anonymization before the LLM sees anything
WhatsApp conversations contain real names, phone numbers, fiscal codes, addresses. Sending that raw to third-party LLM APIs is a GDPR problem and a trust problem.
We anonymize PII before the message reaches any LLM, then deanonymize in the dispatcher layer — only when a tool actually needs the real value to write to the DB.
// pii-anonymizer.ts
interface PIIMap {
[placeholder: string]: string; // '[PHONE_1]' -> '+39 333 1234567'
}
function anonymizeMessage(text: string): { anonymized: string; map: PIIMap } {
const map: PIIMap = {};
const counter = { PERSON: 0, PHONE: 0, EMAIL: 0, ID: 0 };
const anonymized = text
.replace(FISCAL_CODE_REGEX, (match) => {
const key = `[ID_${++counter.ID}]`;
map[key] = match;
return key;
})
.replace(PHONE_REGEX, (match) => {
const key = `[PHONE_${++counter.PHONE}]`;
map[key] = match;
return key;
})
.replace(EMAIL_REGEX, (match) => {
const key = `[EMAIL_${counter.EMAIL++}]`;
map[key] = match;
return key;
});
return { anonymized, map };
}
// Restore real values in tool args before execution
function deanonymizeToolArgs(
args: Record<string, unknown>,
map: PIIMap
): Record<string, unknown> {
return JSON.parse(
JSON.stringify(args).replace(
/\[(?:PERSON|PHONE|EMAIL|ID)_\d+\]/g,
(ph) => map[ph] ?? ph
)
);
}
The LLM sees [PHONE_1] and reasons about it correctly. The book_table handler gets the real phone number only at execution time. The LLM API provider never sees actual PII.
Architecture decision 4: Autonomy gate — risk-tiered execution
Not all tool actions are equal. Reading a menu is safe. Cancelling a reservation or triggering a refund is not. An autonomy gate sits between the dispatcher and execution, classifying each tool call by risk level.
// autonomy-gate.ts
const TOOL_RISK: Record<string, 'low' | 'medium' | 'high'> = {
get_menu: 'low',
check_availability: 'low',
book_table: 'medium',
cancel_reservation: 'high',
process_refund: 'high',
send_notification: 'medium',
};
async function gateToolExecution(
toolName: string,
context: AgentContext
): Promise<{ allowed: boolean; reason?: string }> {
const risk = TOOL_RISK[toolName] ?? 'medium';
if (risk === 'low') return { allowed: true };
if (risk === 'high' && !context.userConfirmed) {
return {
allowed: false,
reason: `Action "${toolName}" requires explicit user confirmation.`,
};
}
if (risk === 'medium' && context.autonomyLevel < 2) {
return {
allowed: false,
reason: `Action "${toolName}" requires autonomy level >= 2.`,
};
}
return { allowed: true };
}
The autonomy level is set per-tenant at onboarding. A conservative restaurant owner sets level 1 (read-only AI, all writes need confirmation). A high-trust logistics operator sets level 3 (full autonomous execution). Same agent code, different gate behaviour.
What ships in the open-source repo
github.com/Alessandro114/sara includes:
- Multi-tenant WhatsApp session management (WAHA-based)
- Full provider failover chain (Groq / Cerebras / SambaNova / Mistral)
- Tool dispatcher with 30+ handlers across 20 verticals
- PII anonymization + deanonymization pipeline
- Autonomy gate with configurable per-tenant risk levels
- RAG layer (pgvector + Jina embeddings v3) for per-tenant knowledge bases
- Dream cycle: proactive agent behaviours that fire outside conversation threads
- White-label support: each tenant gets their own agent identity and persona
Licensed AGPL-3.0 — free to self-host, cloud hosting available for teams that prefer managed infrastructure.
Why open-source this?
Most WhatsApp AI demos are either toy FAQ bots or closed-source commercial black boxes. We wanted to show what a production-grade, multi-vertical agent architecture actually looks like — the provider failover, the PII layer, the autonomy controls, the tool dispatch loop.
The growing platform already serves SMBs across 20 industries. Open-sourcing the core means the community can audit, extend, and build on what we learned.
If this architecture helps you ship faster or think differently about agent design, that is the win.
If this is useful, please star the repo: https://github.com/Alessandro114/sara
Open an issue to discuss architecture decisions, request a vertical definition, or contribute a new tool handler — PRs welcome.
Top comments (0)