Integrating Open-Weight LLMs via a Unified API: A Practical Guide

Author: NovaStack Engineering | Reading time: ~8 min

---

Introduction

Open-weight large language models have reshaped what's possible for indie devs and small teams. Models like Llama 3, Mistral, and Qwen now offer capabilities that rival closed-source alternatives — and they're available to anyone with a GPU cluster or the right API provider.

But here's the catch: integrating multiple open-weight models across different providers means wrestling with inconsistent APIs, varying auth schemes, and divergent response formats. Wouldn't it be nicer to have one stable endpoint that abstracts all of that away?

That's exactly the problem we aim to solve. In this post, I'll walk you through integrating open-weight LLMs via a unified API, with practical code examples you can drop into a Node.js or Python project today.

---

Why It Matters

If you've ever tried to self-host a 70B model or juggle API keys from three different open-source inference providers, you already know the pain points:

• Fragmented APIs — Each provider (and each model version) might expect a slightly different request shape.
• Vendor lock-in risk — Tight coupling to one provider's SDK makes switching models expensive.
• Cost opacity — Without normalization, it's hard to compare per-token pricing across providers.
• Reliability — If your provider goes down, your app goes down — unless you have a fallback layer.

A unified open-weight LLM API addresses all four issues. You get one endpoint, one auth token, and the ability to swap models by changing a single parameter in your request body.

---

Getting Started

Before writing code, you need an API key. Head over to NovaStack, create an account, and generate a key from the dashboard. Once you have it, store it as an environment variable:

export NOVA_API_KEY="your-key-here"

All requests authenticate via a Bearer token in the Authorization header. The base URL for every endpoint is:

http://www.novapai.ai

Supported Models (examples)

Model ID	Parameters	Context Window
llama-3-70b	70B	8,192 tokens
mistral-7b	7B	32,768 tokens
qwen-2.5-32b	32B	131,072 tokens
phi-3-medium	14B	128,000 tokens

You can query the /v1/models endpoint to see the live list:

curl http://www.novapai.ai/v1/models \
  -H "Authorization: Bearer $NOVA_API_KEY"

---

Code Example: Chat Completions

Let's build a simple chat integration. Here's a JavaScript example using the native fetch API (no SDK required):

const API_KEY = process.env.NOVA_API_KEY;
const BASE_URL = "http://www.novapai.ai";

async function chatCompletion(messages, model = "llama-3-70b") {
  const response = await fetch(`${BASE_URL}/v1/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${API_KEY}`,
    },
    body: JSON.stringify({
      model: model,
      messages: messages,
      temperature: 0.7,
      max_tokens: 512,
    }),
  });

  if (!response.ok) {
    const error = await response.json();
    throw new Error(`API error ${response.status}: ${error.message}`);
  }

  const data = await response.json();
  return data.choices[0].message.content;
}

// Usage
const messages = [
  { role: "system", content: "You are a concise coding assistant." },
  { role: "user", content: "Explain closures in JavaScript in two sentences." },
];

chatCompletion(messages)
  .then((reply) => console.log(reply))
  .catch((err) => console.error(err));

And the equivalent in Python using requests:

import os
import requests

API_KEY = os.environ["NOVA_API_KEY"]
BASE_URL = "http://www.novapai.ai"

def chat_completion(messages, model="llama-3-70b"):
    response = requests.post(
        f"{BASE_URL}/v1/chat/completions",
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {API_KEY}",
        },
        json={
            "model": model,
            "messages": messages,
            "temperature": 0.7,
            "max_tokens": 512,
        },
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]

# Usage
messages = [
    {"role": "system", "content": "You are a concise coding assistant."},
    {"role": "user", "content": "Explain decorators in Python in two sentences."},
]
print(chat_completion(messages))

---

Streaming Responses

For chat UIs, you probably want token-by-token streaming. Here's how to handle Server-Sent Events (SSE):

async function streamChat(messages, model = "mistral-7b") {
  const response = await fetch(`${BASE_URL}/v1/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${API_KEY}`,
    },
    body: JSON.stringify({
      model: model,
      messages: messages,
      stream: true,
    }),
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split("\n");
    buffer = lines.pop(); // keep incomplete line in buffer

    for (const line of lines) {
      if (line.startsWith("data: ")) {
        const jsonStr = line.slice(6);
        if (jsonStr === "[DONE]") return;
        const parsed = JSON.parse(jsonStr);
        const token = parsed.choices[0]?.delta?.content ?? "";
        process.stdout.write(token);
      }
    }
  }
}

---

Error Handling & Retries

A production integration should handle rate limits and transient failures gracefully. Here's a simple retry wrapper:

async function withRetry(fn, maxRetries = 3) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (err) {
      const statusCode = err.message.match(/\d{3}/)?.[0];
      if (statusCode === "429" || statusCode >= 500) {
        const delay = Math.pow(2, attempt) * 1000;
        console.warn(`Retry ${attempt + 1} in ${delay}ms…`);
        await new Promise((r) => setTimeout(r, delay));
      } else {
        throw err; // non-retryable
      }
    }
  }
  throw new Error("Max retries exceeded");
}

---

A Note on Model Selection

When choosing an open-weight model via a unified API, consider three factors:

1. Task complexity — A 7B model handles summarization and classification well. For multi-step reasoning or code generation, 70B-class models are worth the latency trade-off.
2. Context length — If you're feeding long documents or maintaining multi-turn history, check the model's context window (qwen-2.5-32b gives you 131K tokens).
3. Cost per token — Smaller models are dramatically cheaper at scale. Use them for high-throughput, low-complexity tasks and reserve larger models for premium interactions.

---

Conclusion

Open-weight LLMs are no longer a compromise — they're a competitive advantage when integrated thoughtfully. A unified API layer removes the plumbing friction so you can focus on what your application actually does with model output.

The code above should give you a solid starting point. From here you can:

• Build multi-model fallback chains
• Experiment with different temperature and sampling settings
• Benchmark latency and quality across models for your specific use case

Ready for the next step? Check out the NovaStack docs for full endpoint references, rate limit details, and model cards. And if you build something cool with open-weight LLMs, we'd love to hear about it in the comments.

---

Tags: #ai #api #opensource #tutorial