DEV Community

Harpreet Seehra
Harpreet Seehra

Posted on Edited on

Build an AI Grocery Delivery Assistant on WhatsApp with Telnyx

Grocery delivery apps have a retention problem. Users download the app, order once, forget their password, and go back to the shop. The interface that every phone already has — a messaging app — never gets used for commerce because wiring up a conversational AI agent to a messaging channel is traditionally a stack of three or four vendors: a messaging gateway, an LLM provider, a transcription service, and a phone validation API.

This walkthrough builds FreshCart, a WhatsApp-based grocery delivery assistant powered by an AI agent named Mia. A customer messages a WhatsApp number, browses products, builds a cart, places an order, tracks delivery, handles substitutions, and reorders previous deliveries — all inside a single chat thread. Voice notes work too: say "add bananas, eggs, and sourdough" and Mia transcribes, parses, and adds each item to the cart.

The whole system runs on four Telnyx products through a single unified API — no third-party LLM, no external transcription service, no separate phone validation vendor.

The canonical code example lives in the Telnyx code examples repo:

https://github.com/team-telnyx/telnyx-code-examples/tree/main/whatsapp-grocery-delivery-nodejs

What This Example Builds

A Node.js Express server with a WhatsApp webhook handler, an agentic AI tool-calling loop, and a 3-panel real-time dashboard:

Component Purpose
WhatsApp webhook Receives inbound messages (text, interactive replies, voice notes)
AI agent (Mia) 12-tool agentic loop — browse, cart, order, track, substitute, reorder
Dashboard Product catalog, WhatsApp conversation mirror, delivery pipeline
SSE stream Real-time event push to the browser dashboard

Why This Matters

WhatsApp has 2 billion monthly active users. For many markets — grocery, pharmacy, food delivery — WhatsApp is where customers already are. Building a full commerce experience inside WhatsApp means no app download, no account creation, no forgotten passwords.

The technical pattern here — an agentic tool-calling loop over a messaging channel — is the foundation for any conversational commerce application:

  • Grocery and food delivery — Browse, order, track, substitute, reorder
  • Pharmacy — Prescription refills, drug interaction checks, delivery scheduling
  • Restaurant ordering — Menu browsing, customization, delivery tracking
  • Retail — Product search, size/color selection, order tracking, returns
  • Service booking — Appointment scheduling, rescheduling, cancellations

The system prompt and tool definitions are the only things that change.

Products Used

telnyx_products: [WhatsApp Business API, AI Inference, Number Lookup, Audio Transcription]
Enter fullscreen mode Exit fullscreen mode

WhatsApp Business API — Sends and receives messages via POST /v2/messages/whatsapp. Supports text, interactive list messages (for product browsing), and quick-reply buttons (for substitution decisions and delivery confirmations).

AI Inference — Powers the agentic tool-calling loop via POST /v2/ai/chat/completions with OpenAI-compatible function calling. The model (GLM-5.3-Flash) receives the conversation history and tool definitions, decides which tool to call, and the server executes it.

Number Lookup — Validates and normalizes customer phone numbers at registration via GET /v2/number_lookup/{phone}. Returns E.164 format so the app never stores mismatched phone formats.

Audio Transcription — Transcribes WhatsApp voice notes via POST /v2/ai/audio/transcriptions using the Whisper model.

Architecture

  WhatsApp Message (text / voice note / button tap)
        |
        v
  +-----------------------+
  | /webhooks/whatsapp    |  Express webhook handler
  +----------+------------+
             |
             v
  +-----------------------+
  | AI Inference           |  Agentic tool-calling loop
  | (GLM-5.3-Flash)       |  12 tools available
  +----------+------------+
             |
     +-------+-------+
     |               |
     v               v
  [Tool execution]  [WhatsApp reply]
  - browse_products  - sendText()
  - add_to_cart      - sendList()
  - place_order      - sendButtons()
  - track_delivery
             |
             v
  +-----------------------+
  | SSE -> Dashboard      |  Real-time 3-panel view
  +-----------------------+
Enter fullscreen mode Exit fullscreen mode

The Tool-Calling Loop

The core of the AI agent is a loop that runs until the model returns a final text response instead of a tool call:

async function processInbound(db, ssePublish, fromNumber, text) {
  const messages = await getConversationHistory(db, fromNumber);
  messages.push({ role: "user", content: text });

  for (let turn = 0; turn < MAX_TURNS; turn++) {
    const response = await chatCompletion(messages);
    const choice = response.choices[0];

    if (choice.finish_reason === "tool_calls") {
      for (const toolCall of choice.message.tool_calls) {
        const result = await executeTool(db, ssePublish, fromNumber, toolCall);
        messages.push({
          role: "tool",
          content: JSON.stringify(result),
          tool_call_id: toolCall.id,
        });
      }
    } else {
      const reply = choice.message.content;
      await sendText(fromNumber, reply);
      await storeMessage(db, fromNumber, "assistant", reply);
      break;
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The 12 AI Tools

Tool Purpose WhatsApp Output
browse_products Browse by category or search Interactive list message
add_to_cart Add items (supports bulk from voice) Text confirmation
view_cart Show cart contents and total Formatted text
update_cart Change quantity or remove items Text confirmation
get_delivery_slots Available delivery windows Interactive list
place_order Checkout from cart Text with order number
track_delivery Delivery status and ETA Status timeline
reschedule_delivery Change delivery window Quick-reply buttons
handle_substitution Accept/reject replacements Quick-reply buttons
reorder Reorder past delivery items Quick-reply buttons
register_customer New customer + Number Lookup Text
send_whatsapp Send formatted message Varies

Voice Note Ordering

When a customer sends a WhatsApp voice note, the webhook detects audio and routes it to Telnyx Audio Transcription:

const resp = await fetch("https://api.telnyx.com/v2/ai/audio/transcriptions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: audioUrl, language: "en", model: "whisper" }),
});
const data = await resp.json();
return data.text;
Enter fullscreen mode Exit fullscreen mode

The transcription feeds into the AI agent as text. "Add bananas, eggs, and sourdough" becomes three separate add_to_cart calls.

Phone Number Validation

New customers register with their phone number. The server calls Telnyx Number Lookup to validate and normalize to E.164:

const nlResp = await fetch(
  `https://api.telnyx.com/v2/number_lookup/${encodeURIComponent(phone)}`,
  { headers: { Authorization: `Bearer ${apiKey}` } }
);
const nlData = await nlResp.json();
if (nlData.data.valid_number === false) {
  return res.status(422).json({ error: "Invalid phone number" });
}
const e164Phone = nlData.data.phone_number;
Enter fullscreen mode Exit fullscreen mode

This prevents the common bug where local format (0626781895) and international format (+31626781895) create duplicate customer records.

Run It

git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/whatsapp-grocery-delivery-nodejs
cp .env.example .env  # add your keys
npm install
node server.js
Enter fullscreen mode Exit fullscreen mode

The server starts on http://localhost:3000. Expose it for webhooks:

ngrok http 3000
Enter fullscreen mode Exit fullscreen mode

Configure the webhook URL in the Telnyx Portal under your WhatsApp Messaging Profile -> Webhook URL -> https://<id>.ngrok.io/webhooks/whatsapp.

Open http://localhost:3000 in a browser to see the 3-panel dashboard. Register with your phone number, then message the WhatsApp number from your phone.

The Dashboard

The 3-panel dashboard updates in real time via Server-Sent Events:

Left — Product Catalog. Nine categories, 70+ products with prices and dietary tags.

Center — WhatsApp Phone Mockup. Live conversation mirror with typing indicators and tool activity.

Right — Delivery Pipeline. Order cards progressing through Confirmed -> Picking -> Packed -> Out for Delivery -> Delivered. An Advance button lets you manually push orders through stages, sending WhatsApp notifications at each step.

Going to Production

  • Database — The app includes a Telnyx Edge SQL adapter for serverless deployment
  • Webhook security — Add signature verification for inbound webhooks
  • Conversation limits — Cap conversation history sent to AI Inference
  • Error recovery — Send retry messages on inference failures
  • Multi-instance — Move session state to Redis for horizontal scaling

Frequently Asked Questions

Do I need a WhatsApp Business Account? Yes. You need a Telnyx account connected to a WABA with an approved phone number and a messaging profile.

Can customers order items not in the catalog? Yes. The AI creates ad-hoc products when items aren't in the seed catalog.

How does voice note ordering work? Voice notes are transcribed via Telnyx Audio Transcription (Whisper), then fed to the AI agent as text. The agent parses items naturally and calls add_to_cart for each one.

Resources


If you build something with this pattern — pharmacy, restaurant, booking agent — let me know in the comments. The agentic loop is more general than the grocery example.

Top comments (0)