DEV Community

Sonam Gupta
Sonam Gupta

Posted on

Build a Voice Assistant That Remembers Every Caller

#ai #webdev #programming

Most voice assistants begin every call with a blank slate. The caller gives their name again. They explain why they called last time. They repeat their preferences. Any context gathered during the previous conversation is trapped in a transcript, CRM record, or database that the assistant cannot immediately use.

In this tutorial, we will build a different pattern: one durable intake dossier per caller. When someone calls, the assistant loads that caller’s history before the conversation begins. It can greet them by name, reference their provider, and include relevant account context. After the call, it files the new reason, summary, and next step back into the same dossier.

Sensitive credentials are encrypted before they reach the assistant, and the complete backend runs as a TypeScript application on Telnyx Edge Compute. The complete sample is available here:

View the per-caller intake code on GitHub

What we are building

The sample uses a fictional healthcare intake workflow, but the underlying architecture is useful anywhere conversations need durable customer context.

The flow looks like this:

Incoming call
    |
    v
Telnyx AI Assistant
    |
    | initialization webhook
    v
Edge Compute router
    |
    | actor ID derived from caller number
    v
One Stateful Actor per caller
    |
    | embedded SQL + durable state
    v
Dynamic variables + encrypted credential
Enter fullscreen mode Exit fullscreen mode

After the conversation:

Post-conversation webhook
    |
    v
Same caller actor
    |
    v
File visit reason, summary, and next step
Enter fullscreen mode Exit fullscreen mode

The example exposes four routes:

POST /webhook/initialization
POST /webhook/post-conversation
GET  /dossier/:phone
GET  /health
Enter fullscreen mode Exit fullscreen mode

The two webhook routes form the main lifecycle. The initialization webhook retrieves context before the call, and the post-conversation webhook saves what happened afterward.

Route each caller to one actor

The key design decision is how actors are identified.

The initialization payload includes telnyx_end_user_target, which represents the caller for this interaction. The application normalizes that value and uses it as the Stateful Actor ID:

const phoneDigits = normalizePhoneDigits(
  event.telnyx_end_user_target
);

const actorId = env.DOSSIERS.idFromName(phoneDigits);
const dossier = env.DOSSIERS.get(actorId);
Enter fullscreen mode Exit fullscreen mode

Every request associated with the same normalized phone number reaches the same actor.

That actor owns the caller’s:

  • Identity information
  • Provider preference
  • Account balance
  • Visit history
  • Most recent activity
  • Filing state

This removes the need for application code that repeatedly queries a shared database, reconstructs a profile, and coordinates simultaneous updates.

The routing key is also the ownership boundary.

A phone number is useful for routing, but it should not automatically be treated as verified identity. Production systems should apply their own authentication and trust policies before revealing sensitive information.

Store relational history inside the actor

Each actor has a private embedded SQL database. The sample creates three tables:

CREATE TABLE IF NOT EXISTS identity (
  patient_name TEXT NOT NULL,
  balance_due TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS prefs (
  provider TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS visits (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  at TEXT NOT NULL,
  reason TEXT NOT NULL,
  summary TEXT NOT NULL,
  next_step TEXT NOT NULL
);
Enter fullscreen mode Exit fullscreen mode

Why SQL instead of one large JSON object?

Visit history is naturally relational. The application may need the most recent visit, a chronological timeline, or filtered records. SQL provides those operations without requiring the entire history to be loaded and rewritten for every update.

The sample seeds demonstration data for a fictional patient named Sarah with a preference for Dr. Lee. Replace this with your own onboarding, CRM, scheduling, or account-data integration.

The important part is that the data belongs to one actor. It is not a globally shared table containing every caller.

Verify the initialization webhook

The initialization endpoint receives a request from Telnyx before the assistant begins the conversation:

if (
  request.method === "POST" &&
  url.pathname === "/webhook/initialization"
) {
  // Verify signature, parse payload, route to actor.
}
Enter fullscreen mode Exit fullscreen mode

Before using the payload, the application verifies its Ed25519 signature using:

  • The raw request body
  • The Telnyx signature header
  • The Telnyx timestamp header
  • The public key stored in TELNYX_PUBLIC_KEY

The endpoint fails closed if the public key is not configured or the signature cannot be verified.

That matters because the request determines which caller dossier will be loaded. An unsigned request should never be able to select an identity and retrieve its context.

After verification, the application confirms that the event is an assistant.initialization event and extracts telnyx_end_user_target.

You can learn more about Telnyx webhook signing in the documentation.

Give the assistant runtime context

Once the request reaches the correct actor, handleInitialization() reads the caller’s information and returns two groups of values:

{
  "dynamic_variables": {
    "patient_name": "Sarah",
    "provider": "Dr. Lee",
    "last_visit": "Annual wellness visit",
    "balance_due": "$0.00"
  },
  "encrypted_dynamic_variables": {
    "portal_token": "..."
  }
}
Enter fullscreen mode Exit fullscreen mode

Regular dynamic variables can be referenced by the assistant during the conversation:

Hello {{patient_name}}. I see your preferred provider is {{provider}}.
How can I help you today?
Enter fullscreen mode Exit fullscreen mode

They can also provide concise context without dumping an entire conversation history into the prompt.

Telnyx sends initialization requests with a default timeout of 1.5 seconds. The timeout can be configured up to 10 seconds. This sample uses an 8-second timeout to leave room for initialization and cold starts.

See the Dynamic Variables guide for the payload format and assistant configuration.

Keep credentials out of the prompt

Not every runtime value should become a normal dynamic variable.

Regular dynamic variables can appear in prompts and logs. That makes them unsuitable for API keys, portal credentials, access tokens, or other sensitive values.

For the fictional portal integration, the actor generates a fresh random token and encrypts it using AES-256-GCM:

const tokenBytes = crypto.getRandomValues(new Uint8Array(32));
const nonce = crypto.getRandomValues(new Uint8Array(12));

const ciphertext = await crypto.subtle.encrypt(
  {
    name: "AES-GCM",
    iv: nonce,
  },
  encryptionKey,
  tokenBytes
);
Enter fullscreen mode Exit fullscreen mode

The nonce and ciphertext are combined and encoded as a base64url value. The encrypted result is returned under encrypted_dynamic_variables, not dynamic_variables.

The encryption key must decode to exactly 32 bytes and must be configured in two places:

  1. As the Edge Compute secret PORTAL_ENC_KEY
  2. As the matching Telnyx integration secret portal_enc_key

This lets the assistant use an encrypted credential through the intended integration flow without adding its plaintext value to the conversational prompt.

Telnyx documents this pattern in its Per-Caller Credentials guide.

File the conversation afterward

Once the conversation ends, Telnyx calls the post-conversation endpoint:

POST /webhook/post-conversation
Enter fullscreen mode Exit fullscreen mode

The request contains structured values such as:

{
  "telnyx_end_user_target": "+15551234567",
  "visit_reason": "Prescription refill",
  "follow_up": "Confirm pharmacy details",
  "next_step": "Send request to provider"
}
Enter fullscreen mode Exit fullscreen mode

The endpoint first checks the bearer token stored in DOSSIER_WEBHOOK_AUTH. It then normalizes the caller number and routes the request to the same actor used during initialization.

Inside the actor, the record is written to the visits table:

INSERT INTO visits (
  at,
  reason,
  summary,
  next_step
) VALUES (?, ?, ?, ?)
Enter fullscreen mode Exit fullscreen mode

The demo also checks for an existing record with the same reason and next step before inserting. If the webhook is retried, the actor can return the existing visit ID rather than filing an obvious duplicate.

For a production system, I would use a stable event or tool invocation ID as the idempotency key whenever one is available. Matching text fields is useful for a sample, but it is not a universal deduplication strategy.

Run the project locally

You will need:

  • Node.js 20 or newer
  • Telnyx Edge CLI 0.2.2 or newer
  • A Telnyx API key
  • A configured Telnyx AI Assistant

Clone the repository and enter the example:

git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/per-caller-intake
Enter fullscreen mode Exit fullscreen mode

Install the dependencies:

npm install
Enter fullscreen mode Exit fullscreen mode

Check the TypeScript:

npx tsc --noEmit
Enter fullscreen mode Exit fullscreen mode

Then run the included smoke tests:

npm run smoke
Enter fullscreen mode Exit fullscreen mode

The smoke test covers several security-sensitive paths:

  • AES-GCM encryption and decryption
  • Valid Ed25519 signatures
  • Modified request bodies
  • Stale timestamps
  • Incorrect public keys
  • Initialization payload parsing

Configure the secrets

Generate a 32-byte encryption key:

KEY=$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '=')
Enter fullscreen mode Exit fullscreen mode

Store it in Edge Compute:

telnyx-edge secrets add PORTAL_ENC_KEY "$KEY"
Enter fullscreen mode Exit fullscreen mode

Store the same value in the corresponding Telnyx integration secret as portal_enc_key.

Next, retrieve your Telnyx public key:

PUBLIC_KEY=$(curl -s \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  https://api.telnyx.com/v2/public_key \
  | jq -r '.data.public')
Enter fullscreen mode Exit fullscreen mode

Add it as another Edge secret:

telnyx-edge secrets add TELNYX_PUBLIC_KEY "$PUBLIC_KEY"
Enter fullscreen mode Exit fullscreen mode

Finally, create a strong token for the post-conversation webhook and save matching copies as:

DOSSIER_WEBHOOK_AUTH
dossier_webhook_auth
Enter fullscreen mode Exit fullscreen mode

The first belongs to your Edge function. The second belongs to the assistant integration that invokes the webhook.

Deploy and connect the assistant

Deploy the application:

telnyx-edge ship
Enter fullscreen mode Exit fullscreen mode

After deployment, use the generated function URL for the assistant’s dynamic variables webhook:

https://your-function.telnyxcompute.com/webhook/initialization
Enter fullscreen mode Exit fullscreen mode

Configure post-conversation processing to call:

https://your-function.telnyxcompute.com/webhook/post-conversation
Enter fullscreen mode Exit fullscreen mode

The sample uses an 8-second initialization timeout:

{
  "dynamic_variables_webhook_timeout_ms": 8000,
  "post_conversation_settings": {
    "enabled": true
  }
}
Enter fullscreen mode Exit fullscreen mode

You can inspect a demonstration dossier through:

GET /dossier/:phone
Enter fullscreen mode Exit fullscreen mode

That route is convenient for development, but it should be authenticated or removed before production use.

Where this pattern fits

The example uses patient intake, but the architecture is not healthcare-specific.

A support line could keep one dossier per customer, including open cases, product ownership, and previous troubleshooting steps.

A property-management assistant could remember the caller’s unit, maintenance history, and access preferences.

A financial-services assistant could load account context after authentication and store structured outcomes from each conversation.

A delivery assistant could maintain one durable customer profile with preferred drop-off instructions and recent shipment issues.

The reusable pattern is:

  1. Derive a stable actor ID from the interaction.
  2. Load only the context needed for the current conversation.
  3. encrypt credentials that should not enter the prompt.
  4. Store the structured outcome after the conversation.
  5. Route the next interaction back to the same actor.

With that lifecycle in place, a voice assistant stops behaving like a stateless demo and starts acting like part of the application.

Production considerations

Before adapting the sample for real customer data:

  • Authenticate callers before exposing sensitive context.
  • Define retention and deletion policies.
  • Collect only the data the assistant needs.
  • Protect or remove diagnostic dossier routes.
  • Use stable idempotency keys for webhook retries.
  • Rotate both copies of shared encryption keys together.
  • Review consent, privacy, and regulatory requirements for your industry.
  • Decide whether a phone number is sufficient as an actor key for your threat model.

Resources

Top comments (0)