DEV Community

LION ZHANL
LION ZHANL

Posted on

How to Design a Replaceable Gateway for Real-Time Multimodal AI

#ai #webdev #javascript #architecture

Real-time multimodal AI feels like one feature in the interface, but it is really a chain of tightly coupled systems: camera capture, audio transport, model inference, streaming responses, and user feedback. If any model-specific detail leaks into the client, replacing the provider later becomes expensive.

While building an independent SeedRealtime demo, I used a replaceable gateway pattern so the browser, server, and model adapter can evolve independently. The live fallback can use one provider today while the same interface stays ready for another approved model endpoint later.

Start with a stable contract

The client should not know a provider's URL, authentication scheme, or raw event names. It should talk to your own server using a small contract:

type RealtimeRequest = {
  sessionId: string;
  audio?: ArrayBuffer;
  frame?: string;
  instruction?: string;
};

type RealtimeEvent =
  | { type: "transcript.delta"; text: string }
  | { type: "response.delta"; text: string }
  | { type: "audio.delta"; data: string }
  | { type: "session.error"; message: string };
Enter fullscreen mode Exit fullscreen mode

This contract becomes the boundary between product behavior and provider behavior. The browser only renders events it understands. Each provider adapter translates its native stream into this format.

Keep credentials and routing on the server

A browser bundle is the wrong place for API keys. It also makes provider migration harder because every change requires shipping a new client. A server-side gateway solves both problems:

  1. The browser creates a session with your backend.
  2. The backend authenticates the user and applies rate limits.
  3. A router chooses the active model adapter.
  4. The adapter signs the upstream request.
  5. Provider events are normalized before they reach the client.

This gives you one place for logging, safety filters, quotas, and graceful fallback behavior. The SeedRealtime API integration guide shows the same separation in a practical project structure.

Design adapters around capabilities

Different real-time models rarely expose identical features. One may accept continuous video, while another expects sampled images. One may stream synthesized audio, while another only returns text.

Avoid pretending that every provider is identical. Define explicit capabilities:

type ModelCapabilities = {
  inputAudio: boolean;
  inputVideo: boolean;
  outputAudio: boolean;
  toolCalls: boolean;
  interruption: boolean;
};

interface RealtimeAdapter {
  capabilities: ModelCapabilities;
  connect(session: SessionConfig): Promise<AdapterSession>;
}
Enter fullscreen mode Exit fullscreen mode

The UI can disable unsupported controls or display a clear fallback state. Capability checks are easier to maintain than scattered provider-name conditions.

Treat latency as a budget

"Real time" is not a single number. It is the sum of capture, upload, queueing, inference, synthesis, and playback. Instrument each boundary with a shared request or session ID.

Track at least:

  • time to session ready
  • time to first transcript token
  • time to first response token
  • time to first audio byte
  • interruption-to-stop delay
  • reconnect count and reason

A fast model cannot rescue an application that uploads oversized frames or buffers too much audio. The real-time multimodal AI guide covers the product-level tradeoffs among responsiveness, context, and cost.

Build interruption into the protocol

Voice interfaces feel slow when users cannot interrupt them. Treat interruption as a first-class event rather than a UI hack.

When the microphone detects new speech, the client should send an interrupt signal, stop local playback immediately, and let the gateway cancel upstream generation.

Make failure visible and recoverable

A good real-time interface explains what is happening:

  • connecting
  • listening
  • processing
  • responding
  • reconnecting
  • permission blocked

Use bounded retries with backoff, and preserve enough state to resume without replaying an entire conversation. If an upstream model is unavailable, route to a compatible fallback only when the capability difference is clear to the user.

Why the gateway pays off

The replaceable gateway adds a small amount of structure at the beginning, but it protects the rest of the product from provider churn. It also creates a clean place for observability, access control, and cost management.

You can try the independent SeedRealtime real-time audio-visual AI experience and inspect the linked guides for implementation details. The site is an independent project and is not an official ByteDance website.

The durable lesson is simple: design the product around a stable event contract, and isolate every model-specific decision behind an adapter.

Top comments (0)