Building a German Speaking Partner with ElevenLabs Agents and Cloudflare Workers

Anyone who has learned a foreign language knows the gap between a passed exam and real fluency with native speakers. Closing it takes practice with someone at native level, and not everyone has a native-speaking partner or the budget and time for endless lessons.

So I asked myself: what if an AI voice agent could be that partner? Someone to have casual conversations with, like a buddy who speaks at native level, never loses patience, and points out your mistakes when they matter.

I built it with ElevenLabs agents and Cloudflare Workers.

What it does

• Holds a casual conversation in German, like chatting with a friend.
• Corrects real grammar and word-choice mistakes, then keeps the conversation going instead of stopping at every slip.
• Saves mistakes and new vocabulary after each call to a review page, where learned items can be deleted.
• Remembers earlier sessions, so you don't start from scratch every time.
• Greets you by name and keeps the tone personal.

How it fits together

An ElevenLabs agent handles the voice, and a Cloudflare Worker (Hono) does everything else: it serves a small Vite + TypeScript page, stores data in D1, and exposes the endpoints the agent calls.

1. The browser requests the page. The Worker authenticates the user and rewrites the HTML before sending it.
2. The widget starts a call with a signed user ID as a dynamic variable.
3. During the call the agent calls two tools on my Worker: recall, which returns what happened in recent sessions, and store, which saves items from the conversation.

1. When the call ends, ElevenLabs sends a post-call webhook. The Worker verifies it and writes the mistakes and vocabulary to D1.
2. The review page reads those tables.

Decisions worth explaining

Memory. When I built this, ElevenLabs agents had no built-in memory across sessions, so I added it myself: the webhook stores each session, and the agent calls recall when a call starts. I load only the last two or three sessions. Pulling in more makes the agent sound robotic. Nobody remembers a hundred earlier conversations with a friend.

No secrets in the browser. The Worker puts the user's identity into the page before it reaches the browser, as an HMAC-signed ID. The signing key only exists in the Worker.

const rewritten = new HTMLRewriter()
  .on('elevenlabs-convai', {
    element(el) {
      el.setAttribute(
        'dynamic-variables',
        JSON.stringify({ user_id: token, name: displayName }),
      );
    },
  })
  .transform(response);

rewritten.headers.set('Cache-Control', 'no-store');

The no-store header matters, because this page is personalized and must never be served from a shared cache.

The agent's endpoints are not public. The agent never decides whose data it touches. The widget passes the signed user ID to the agent as a dynamic variable, and every tool call sends it back. The tool routes verify the signature before reading or writing anything, so an invented or altered ID is rejected. The post-call webhook has its own check. Route wiring, simplified:

app.get('/tools/recall', recallHandler);   // verifies the signed user_id first
app.post('/tools/store', storeHandler);    // same check before any write
app.post('/api/webhook', saveSession);     // webhook signature check

The browser sees the signed ID but can't create a new one.

Routing. Static files come straight from the asset bundle. Only /, /api/* and /tools/* invoke the Worker.

What I'd do next

• Replace plain webhook tools with MCP tools, for cleaner and more standard tool access.
• Add a demo mode with a throwaway user, call-length limits and a daily spend cap, so it can be public without burning credits.
• Show progress over time, like which error categories are shrinking.

If you're building voice agents and want to know more about my agent - drop a comment, I'm happy to share details.