Deploying LLMs on Streaming Platforms: A Step-by-Step Guide

#engineering #oxlo #ai

Live streaming platforms generate thousands of chat messages per minute, and manual moderation does not scale. In this guide, I will walk you through building a real-time moderation agent that ingests a stream of chat messages and uses an LLM to classify, filter, and respond to them. We will run the inference on Oxlo.ai so costs stay flat per request rather than scaling with token count, which you can verify at https://oxlo.ai/pricing.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client

We start by importing the OpenAI SDK and pointing it at Oxlo.ai. This is a drop-in replacement, so the only change is the base URL and your API key.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

Step 2: Define the moderation policy

The system prompt is the agent's rulebook. It instructs the model to return structured JSON for every chat message so our pipeline can act on it without regex parsing.

SYSTEM_PROMPT = """You are a moderation bot for a live streaming platform.
Evaluate each chat message and respond with a JSON object containing:
  action: one of "allow", "delete", "timeout"
  reason: one sentence explaining the decision
  reply: a short friendly reply if action is "allow", otherwise null

Keep responses strictly in JSON. No markdown fences."""

Step 3: Simulate the live chat stream

To keep this tutorial self-contained, we will use a generator that yields synthetic chat messages at a fixed interval. In production, this would be replaced by a WebSocket connection to Twitch or YouTube Live.

import random
import time

SAMPLE_MESSAGES = [
    "Love the stream today!",
    "Check out my channel www.example.com",
    "You are terrible at this game",
    "What keyboard are you using?",
    "First time here, hello everyone",
    "Spam spam spam",
    "Can you play Cyberpunk next?",
]

def chat_stream(limit=20):
    for _ in range(limit):
        msg = random.choice(SAMPLE_MESSAGES)
        user = f"viewer_{random.randint(1, 999)}"
        yield {"user": user, "text": msg}
        time.sleep(1)

Step 4: Build the moderation function

This function sends each message to Oxlo.ai with the system prompt. We use Llama 3.3 70B because it handles structured outputs reliably and keeps latency low.

import json

def moderate_message(text: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
    )
    raw = response.choices[0].message.content.strip()
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return {"action": "allow", "reason": "malformed json", "reply": None}

Step 5: Wire the processing loop

Now we connect the stream to the moderator. The loop prints the original message, the moderation decision, and any automated reply. This is the core of the deployed service.

def run_moderator():
    for msg in chat_stream(limit=10):
        decision = moderate_message(msg["text"])
        print(f"User: {msg['user']}")
        print(f"Message: {msg['text']}")
        print(f"Action: {decision['action']}")
        print(f"Reason: {decision['reason']}")
        if decision.get("reply"):
            print(f"Reply: {decision['reply']}")
        print("-" * 40)

if __name__ == "__main__":
    run_moderator()

Run it

Save the complete script as moderator.py, set your API key, and run it. You should see output similar to the following.

$ export OXLO_API_KEY="sk-oxlo.ai-..."
$ python moderator.py
User: viewer_42
Message: Love the stream today!
Action: allow
Reason: Positive community engagement.
Reply: Thanks for the love, viewer_42!
----------------------------------------
User: viewer_88
Message: Check out my channel www.example.com
Action: delete
Reason: Unauthorized self-promotion.
----------------------------------------

Next steps

Replace the chat_stream generator with an async WebSocket reader from your streaming platform of choice. You can also switch to Qwen 3 32B on Oxlo.ai for multilingual chat rooms, or use Kimi K2.6 if you need vision moderation for image-based super chats.

DEV Community