We are building a low-latency FAQ agent that streams concise answers to technical questions. It is designed for support teams that need sub-second responses without running their own inference stack. Because Oxlo.ai charges a flat rate per request, we can include a detailed system prompt and few-shot examples without pushing cost higher as the prompt grows.
What you'll need
- An Oxlo.ai API key
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai
Step 1: Configure the client and system prompt
First, I create the client pointing at Oxlo.ai and define a strict system prompt that forces brevity, which is the single biggest win for generation latency. Because Oxlo.ai uses flat per-request pricing, I can pack detailed instructions and few-shot examples into the prompt without the cost scaling you see on token-based providers.
SYSTEM_PROMPT = """You are a terse technical support agent for a developer platform.
Rules:
- Answer in one sentence whenever possible.
- Never exceed 30 words.
- If you do not know the answer, reply exactly: "I need to escalate this."
Examples:
User: How do I rotate an API key?
Agent: Navigate to Settings > API Keys and click Regenerate.
User: What regions do you support?
Agent: We support us-east, eu-west, and apac-syd."""from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY",
timeout=10.0,
)Step 2: Stream tokens to the terminal
Instead of waiting for the full completion, we print each chunk as it arrives. This cuts perceived latency and lets users start reading immediately.
import sys
def stream_answer(user_message):
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
stream=True,
temperature=0.1,
max_tokens=100,
)
for chunk in response:
delta = chunk.choices[0].delta.content
if delta:
sys.stdout.write(delta)
sys.stdout.flush()
sys.stdout.write("\n")Step 3: Add latency tracking and retries
Network blips happen. I wrap the call in a small retry loop and measure wall-clock time from request to final token so we can verify we are hitting our latency budget.
import time
def ask_agent(user_message, max_retries=2):
for attempt in range(max_retries + 1):
start = time.perf_counter()
try:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
stream=True,
temperature=0.1,
max_tokens=100,
)
answer = ""
for chunk in response:
delta = chunk.choices[0].delta.content
if delta:
answer += delta
print(delta, end="", flush=True)
elapsed = time.perf_counter() - start
print(f"\n[Latency: {elapsed:.2f}s]")
return answer
except Exception:
if attempt == max_retries:
raise
time.sleep(0.5 * (attempt + 1))
return ""Step 4: Build the interactive loop
Finally, I wire everything into a simple REPL that keeps the client alive across questions. Reusing the same OpenAI client instance avoids repeated TLS handshakes.
if __name__ == "__main__":
print("Low-latency FAQ agent running. Type 'exit' to quit.")
while True:
user_input = input("\nUser: ").strip()
if user_input.lower() in ("exit", "quit"):
break
print("Agent: ", end="", flush=True)
ask_agent(user_input)Run it
Save the script as faq_agent.py, export your key, and run it:
export OXLO_API_KEY="sk-..."
python faq_agent.pyExample session:
Low-latency FAQ agent running. Type 'exit' to quit.
User: How do I reset my password?
Agent: Visit the login page and click Forgot Password to receive a reset link.
User: Do you offer SAML SSO?
Agent: Yes, SAML SSO is available on Enterprise plans.Next steps
Swap in qwen-3-32b if you need multilingual answers, or add function calling to query live account data without adding per-token cost to the context. If throughput becomes the bottleneck, remember that Oxlo.ai has no cold starts on popular models, so scaling concurrency is simple.
Top comments (0)