DEV Community

Harpreet Singh Seehra
Harpreet Singh Seehra

Posted on

Build a Chunked SMS Conversation Exporter with Edge SQL, Cloud Storage, and the Agent SDK

Export SMS conversation history from Edge SQL to Cloud Storage as chunked JSON files, with a 4-stage Agent SDK pipeline that handles 10k+ messages — and texts you when the export is done. All on Telnyx Edge Compute, in under 350 lines of TypeScript.

What You'll Build

An SMS conversation exporter that:

  • Ingests SMS messages via Telnyx Messaging webhooks into a per-actor SQL database
  • Counts and chunks messages for export (configurable chunk size, default 500)
  • Uploads each chunk as a separate JSON file to Cloud Storage
  • Writes a manifest file listing all chunks with metadata
  • Sends an SMS notification when the export is complete (zero-credential binding)
  • Handles 10,000+ messages via non-blocking, self-requeuing pipeline stages

The entire system runs on Telnyx Edge Compute — no external database, no separate upload service, no cross-cloud latency.

Architecture 

                    POST /export
                         │
                         ▼
         ┌──────────────────────┐
         │  ExportAgent         │  (one actor per export job)
         │                      │
         │  1. countMessages()  │──► SQL DB: SELECT COUNT(*)
         │  2. exportChunk()    │──► SQL DB: SELECT chunk
         │                      │──► Cloud Storage: PUT JSON chunk
         │     (re-queues       │    (repeats until all chunks done)
         │      until done)     │
         │  3. writeManifest()  │──► Cloud Storage: PUT manifest.json
         │  4. notifyComplete() │──► SMS via [telnyx] binding
         └──────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Quick Start

1. Clone and install 

git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/sms-conversation-exporter
npm install
Enter fullscreen mode Exit fullscreen mode

2. Configure

Edit telnyx.toml:

[storage.cloudstorage.EXPORT_STORAGE]
bucket_name = "<your-storage-bucket-name>"
region = "us-central-1"

[env_vars]
ALERT_PHONE = "+18005559876"
SENDER_PHONE = "+18005551234"
CHUNK_SIZE = "500"
Enter fullscreen mode Exit fullscreen mode

3. Deploy 

telnyx-edge secret set TELNYX_API_KEY KEY0123456789ABCDEF
telnyx-edge ship
Enter fullscreen mode Exit fullscreen mode

4. Test the export 

# Simulate 10,000 messages
curl -X POST https://your-deployment.telnyxcompute.com/simulate-bulk \
  -H "Content-Type: application/json" \
  -d '{"count": 10000}'

# Start the export
curl -X POST https://your-deployment.telnyxcompute.com/export

# Check progress
curl https://your-deployment.telnyxcompute.com/export/{exportId}
Enter fullscreen mode Exit fullscreen mode

How the Pipeline Works

Agent SDK queue() — Non-Blocking Stages

Each export job gets its own ExportAgent actor instance. The pipeline uses this.queue() to chain stages without blocking the HTTP request:

async start(params: { exportId: string; conversationFilter: string | null }): Promise<void> {
  await this.setState({ exportId: params.exportId, status: "counting" });
  await this.queue("countMessages");
}
Enter fullscreen mode Exit fullscreen mode

Self-Requeuing Chunks

The key pattern is the self-requeuing exportChunk() stage:

async exportChunk(): Promise<void> {
  const state = await this.getState();
  const offset = state.chunkIndex * chunkSize;

  const rows = this.ctx.storage.sql.exec(
    "SELECT * FROM messages ORDER BY timestamp ASC LIMIT ? OFFSET ?",
    chunkSize, offset
  ).toArray();

  await this.env.EXPORT_STORAGE.put(
    `exports/${state.exportId}/chunk-${state.chunkIndex}.json`,
    JSON.stringify({ messages: rows })
  );

  if (state.chunkIndex + 1 < state.totalChunks) {
    await this.queue("exportChunk");  // schedule next chunk
  } else {
    await this.queue("writeManifest");
  }
}
Enter fullscreen mode Exit fullscreen mode

Each chunk runs as a separate, non-blocking stage. The actor isn't blocked, progress is visible at any time via GET /export/:id, and a chunk failure is captured in state.

Zero-Credential SMS Notification 

async notifyComplete(): Promise<void> {
  await this.env.TELNYX.messages.send({
    from: this.env.SENDER_PHONE,
    to: this.env.ALERT_PHONE,
    text: `Export complete: ${state.exportedMessages} messages in ${state.uploadedChunks.length} chunk(s).`
  });
}
Enter fullscreen mode Exit fullscreen mode

No API key in code. The [telnyx] binding in telnyx.toml gives the actor access to the Messaging API — the platform handles authentication.

Chunked Output in Cloud Storage 

exports/export-1234567890-abc123/
├── chunk-0000.json    (messages 0–499)
├── chunk-0001.json    (messages 500–999)
├── chunk-0002.json    (messages 1000–1499)
├── ...
└── manifest.json      (metadata: total count, chunk list)
Enter fullscreen mode Exit fullscreen mode

Each chunk is self-contained with metadata — exportId, chunkIndex, totalChunks, totalMessages. Process chunks independently in Spark, BigQuery, or any data pipeline.

API Endpoints

Method Path Description
POST /webhooks/messaging Messaging webhook (ingests SMS into SQL)
POST /export Start a chunked export job
GET /export/:id Get export status and progress
GET /messages List messages in SQL DB
GET /messages/count Get total message count
POST /seed Add a single test message
POST /simulate-bulk Bulk insert test messages (default 10k)

Use Cases

  • Compliance archival — Export SMS conversation history for regulatory compliance
  • Data migration — Move SMS data from Edge SQL to a data warehouse via Cloud Storage
  • Backup — Periodic JSON exports of all conversations to Cloud Storage
  • Analytics — Export conversation data for offline analysis

Resources

Top comments (0)