Integrating the OpenAI API the Right Way — Streaming, Rate-Limiting, and Prompt

Integrating the OpenAI API the Right Way
Calling openai.chat.completions.create() is the easy 10% of building an AI-powered product. The hard 90% is making it feel instant, not breaking under load, and getting consistent output from a model that's fundamentally non-deterministic. I ran into all three of these while building an AI tool that generates formal letters from user input using GPT.
Here's what actually mattered.

1. Streaming isn't optional for UX If you're generating anything longer than a sentence, a blocking request feels broken — users stare at a spinner for 5-10 seconds with no feedback. Streaming turns that dead time into something that reads like the model is "thinking out loud," which massively changes perceived performance even though total latency is the same. javascriptconst stream = await openai.chat.completions.create({ model: "gpt-3.5-turbo", messages: [{ role: "user", content: prompt }], stream: true, });

for await (const chunk of stream) {
const token = chunk.choices[0]?.delta?.content || "";
process.stdout.write(token); // or push to a WebSocket/SSE connection
}
On the frontend, this pairs naturally with Server-Sent Events or a WebSocket connection so tokens render as they arrive instead of waiting for the full response.

1. Rate-limiting protects you, not just OpenAI It's tempting to think rate-limiting is only about respecting OpenAI's API limits. In practice, it's just as much about protecting your infrastructure and your bill from a single user (or a bug in your own retry logic) hammering the endpoint. A simple token-bucket approach per user goes a long way: javascriptconst buckets = new Map();

function allowRequest(userId, limit = 5, windowMs = 60_000) {
const now = Date.now();
const bucket = buckets.get(userId) || { count: 0, resetAt: now + windowMs };

if (now > bucket.resetAt) {
bucket.count = 0;
bucket.resetAt = now + windowMs;
}

if (bucket.count >= limit) return false;

bucket.count += 1;
buckets.set(userId, bucket);
return true;
}
For production, swap the in-memory Map for Redis so limits hold up across multiple server instances — an in-memory bucket resets the moment you scale horizontally.

1. Prompt templates need version control, not vibes Once you have more than two or three prompts, "tweak it until it feels right" stops scaling. What worked well was treating prompt templates as versioned, parameterized objects rather than inline strings scattered across the codebase: javascriptconst templates = { formalLetter: { version: "v3", system: "You are a professional correspondence writer. Be concise, formal, and grammatically precise.", build: ({ recipient, purpose, tone }) => Write a formal letter to ${recipient} regarding ${purpose}. Tone: ${tone}., }, // ...9 more templates }; This made it possible to A/B test prompt changes, roll back a regression instantly, and keep a clean diff history of why a prompt changed — something you lose completely when prompts are just strings buried in route handlers.
2. Treat the model's output as untrusted input This one's easy to skip until it bites you. Model output can include unexpected formatting, occasional hallucinated details, or — if you're not careful with your system prompt — instructions the user tried to sneak into the input. Always:

Sanitize before rendering (especially if output ever touches HTML)
Validate structure if you expect JSON (models drift; don't trust it blindly)
Set a max_tokens ceiling so a runaway generation doesn't blow your latency or cost budget

javascripttry {
const response = await openai.chat.completions.create({
model: "gpt-3.5-turbo",
messages,
max_tokens: 800,
});
const content = response.choices[0].message.content;
// sanitize/validate before using content
} catch (err) {
// handle rate limit (429), timeout, and malformed-response cases separately
}
The takeaway
None of this is exotic — it's the same discipline you'd apply to any third-party API. But it's easy to skip when a demo works fine with one user and no load. The gap between "works in a demo" and "works in production" for LLM features is almost entirely streaming, rate-limiting, and prompt versioning — not the model call itself.

I write about backend systems, applied AI, and the engineering details that don't make it into the docs.