DEV Community: Mundo Ghose

Building an AI Model Router in Node.js with One Unified API

Mundo Ghose — Tue, 09 Jun 2026 11:18:13 +0000

From Chatbots to Personal AI Agents: The Infrastructure Developers Actually Need

Mundo Ghose — Mon, 08 Jun 2026 09:15:44 +0000

title: Your AI Agent Should Not Be Locked to One LLM Provider
published: false
description: Why serious AI agents need a provider-agnostic architecture, model routing, fallback, and a unified API gateway.

tags: ai, llm, agents, architecture

Your AI Agent Should Not Be Locked to One LLM Provider
Most AI agent prototypes start the same way.

You pick one model provider.
You install one SDK.
You write a few prompts.
You add tool calling.
You build a demo.

It works.

Until it does not.

The moment you want to try another model, reduce cost, add fallback, improve latency, or support different task types, your simple agent starts turning into a messy collection of provider-specific logic.

That is when you realize something important:

A real AI agent should not be locked to one LLM provider.

If you are building a personal AI agent, coding assistant, research assistant, internal workflow agent, or AI-native product, the model should be replaceable infrastructure — not a hardcoded dependency.

The Problem with Single-Provider Agents
A simple agent architecture often looks like this:

CopyUser
↓
Agent
↓
One LLM Provider
↓
Response
This is fine for a proof of concept.

But real-world agent systems need more flexibility.

Different tasks often need different models:

Task Better Model Strategy
Quick summarization Fast, low-cost model
Complex coding Strong coding model
Long document analysis Long-context model
Reasoning-heavy planning Reasoning model
Multilingual writing Model strong in that language
Background automation Cheap and reliable model
Production fallback Backup provider
If your agent is deeply coupled to one provider, every optimization becomes harder.

You cannot easily answer questions like:

What happens if the provider is down?
What if latency spikes?
What if another model is cheaper for simple tasks?
What if a new model is better for coding?
What if a user wants Claude for writing but GPT for structured reasoning?
What if you want to route Chinese tasks to a different model than English tasks?
This is not just a model problem.

It is an infrastructure problem.

The Better Pattern: Provider-Agnostic Agents
A more scalable architecture looks like this:

CopyUser
↓
Agent Runtime
↓
Model Router
↓
AI API Gateway
↓
Multiple Model Providers
In this design, your agent does not talk directly to every model provider.

Instead, it talks to a unified gateway.

The gateway handles access to multiple models, while your agent focuses on:

user intent,
planning,
tool use,
memory,
task execution,
result evaluation.
This keeps your core agent logic clean.

Why OpenAI-Compatible APIs Matter
One of the easiest ways to build provider-agnostic agents is to use an OpenAI-compatible API format.

Many developers already understand this request shape:

Copy{
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": "Explain model routing for AI agents."
}
]
}
If your gateway supports this format, your agent code can stay mostly the same even when the underlying model changes.

That is the idea behind platforms like OpenRain.

OpenRain provides an AI API Gateway with OpenAI-compatible access to many model providers through a unified API layer.

Instead of wiring your agent directly to each provider, you can call one endpoint and manage model access behind the gateway.

A Simple Example
Here is a minimal Python example using an OpenAI-compatible gateway:

Copyimport os
import requests

API_KEY = os.getenv("OPENRAIN_API_KEY")

response = requests.post(
"https://openrain.ai/v1/chat/completions",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
},
json={
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": "Explain why AI agents need model routing."
}
],
},
timeout=60,
)

print(response.json()["choices"][0]["message"]["content"])
The important part is not the specific model name.

The important part is that your application talks to a stable interface.

That gives you room to experiment, route, fallback, and optimize later.

Add a Simple Model Router
Once your agent uses a unified API, you can introduce routing logic.

For example:

Copydef choose_model(task_type: str) -> str:
if task_type == "coding":
return "code-strong-model"

if task_type == "summary":
    return "fast-low-cost-model"

if task_type == "reasoning":
    return "reasoning-model"

return "general-purpose-model"

Then your agent can do this:

Copytask_type = "summary"
model = choose_model(task_type)

Send request to the gateway with the selected model

This is a small change, but it unlocks a better architecture.

Your agent can now choose models based on task type instead of being hardcoded to one provider.

Add Fallback
Production agents need fallback.

LLM providers can fail.
Requests can timeout.
Rate limits happen.
Models can be temporarily unavailable.

A basic fallback function might look like this:

Copydef call_with_fallback(client, models, messages):
last_error = None

for model in models:
    try:
        return client.chat(model=model, messages=messages)
    except Exception as error:
        last_error = error
        print(f"{model} failed. Trying next model...")

raise last_error

Now your agent has a recovery path.

Instead of failing immediately, it can try another model.

This is especially useful for personal agents that run background jobs, scheduled tasks, research workflows, or production automation.

The Real Shift: From Model Usage to Model Orchestration
The next generation of AI applications will not be built around a single model.

They will be built around model orchestration.

A serious agent should be able to decide:

which model to use,
when to use it,
how much budget to spend,
when to retry,
when to fallback,
when to use a faster model,
when to use a more capable model.
This is why a model gateway becomes a foundational layer.

It turns model access into infrastructure.

Final Thought
Hardcoding one LLM provider is fine for a weekend demo.

But if you are building a real AI agent, you probably want:

provider flexibility,
model routing,
fallback,
usage tracking,
cost control,
lower integration complexity.
That is why I believe every serious AI agent framework should start with a provider-agnostic model layer.

This is also the direction I am exploring with OpenRain: a unified AI API Gateway for developers building agents, tools, and AI-native applications.

If your agent can switch models without changing core application code, you are already designing it the right way.

You can check out OpenRain here:

https://openrain.ai

Copy

Article 2

---
title: The Missing Layer in Most AI Agent Frameworks: A Model Gateway
published: false
description: AI agents need more than prompts, tools, and memory. They need reliable multi-model infrastructure.
tags: ai, agents, llm, devtools
---
The Missing Layer in Most AI Agent Frameworks: A Model Gateway
When developers talk about AI agent frameworks, they usually talk about:

planning,
tool calling,
memory,
RAG,
workflows,
multi-agent collaboration.
These are important.

But there is one layer that often gets ignored:

the model gateway.

Most agent frameworks assume model access is simple.

Call an LLM.
Get a response.
Continue the workflow.

In reality, model access is one of the most important infrastructure decisions in an agent system.

If your agent depends on a single provider, a single endpoint, or a single model, the rest of your architecture becomes fragile.

Agents Are Not Normal Chatbots
A chatbot usually answers one user message at a time.

An agent may need to perform a sequence of tasks:

CopyUnderstand goal
 ↓
Create plan
 ↓
Call tools
 ↓
Read files
 ↓
Search web
 ↓
Use model
 ↓
Validate result
 ↓
Retry if needed
 ↓
Return final answer
This process may involve many model calls.

Some calls are simple.

Some calls are expensive.

Some calls require long context.

Some calls need strong reasoning.

Some calls need fast latency.

Using the same model for every step is often wasteful.

A Personal Agent Needs Multiple Model Modes
Imagine a personal general-purpose agent.

It helps you:

summarize documents,
draft emails,
write code,
prepare reports,
compare products,
translate content,
search information,
organize tasks,
automate workflows.
Should all these tasks use the same model?

Probably not.

A better system uses different model strategies:

CopySimple classification → cheap model
Long document summary → long-context model
Code generation → coding model
Strategic planning → reasoning model
Polished writing → writing model
Fallback path → backup model
That means the agent needs a routing layer.

And the routing layer needs a stable way to access many models.

That is where a model gateway fits.

What Is a Model Gateway?
A model gateway sits between your application and model providers.

CopyYour Application
      ↓
Model Gateway
      ↓
OpenAI / Claude / Gemini / DeepSeek / Qwen / Mistral / Others
The gateway gives your application one consistent way to access multiple models.

A good gateway can help with:

unified API access,
provider abstraction,
model routing,
automatic failover,
usage tracking,
latency optimization,
cost visibility,
API key management.
OpenRain is an example of this type of infrastructure.

It provides an AI API Gateway with OpenAI-compatible APIs, access to many model providers, global routing, failover, and usage statistics.

Why This Matters for Agent Frameworks
An agent framework usually has several internal modules:

CopyIntent Classifier
Planner
Tool Caller
Memory Manager
Evaluator
Final Response Generator
Each module may have different model requirements.

For example:

Agent Module    Model Requirement
Intent classifier   fast and cheap
Planner strong reasoning
Tool-call formatter reliable structured output
Memory summarizer   low-cost summarization
Evaluator   accurate judgment
Final writer    strong language quality
If your framework can route each module to a different model, your agent becomes more efficient and reliable.

Without a model gateway, this gets messy quickly.

A Simple Architecture
Here is a practical architecture:

CopyUser
 ↓
Agent API
 ↓
Task Classifier
 ↓
Planner
 ↓
Tool Runtime
 ↓
Memory Runtime
 ↓
Model Router
 ↓
OpenRain AI Gateway
 ↓
Model Providers
The key idea:

The agent should not care which provider serves the final model response.

The agent should only care about capability.

For example:

CopyNeed fast classification
Need strong code generation
Need long-context reasoning
Need low-cost summarization
Need high-quality writing
The model router maps capability to model.

The gateway handles the access layer.

Example: Capability-Based Routing
Instead of writing code like this:

Copymodel = "some-specific-provider-model"
You can design around capabilities:

CopyCAPABILITY_TO_MODEL = {
    "fast_classification": "fast-low-cost-model",
    "code_generation": "code-strong-model",
    "long_context": "long-context-model",
    "reasoning": "reasoning-model",
    "writing": "general-writing-model",
}
Then:

Copydef select_model(capability: str) -> str:
    return CAPABILITY_TO_MODEL.get(capability, "general-writing-model")
Your agent can say:

CopyI need reasoning.
The router decides which model to use.

This makes your framework more maintainable.

Add Failover at the Gateway Layer
Failover should not be an afterthought.

If your agent is executing a multi-step workflow, one failed model call can break the whole task.

A better design has fallback candidates:

CopyMODEL_FALLBACKS = {
    "reasoning": [
        "primary-reasoning-model",
        "backup-reasoning-model",
        "general-purpose-model",
    ],
    "writing": [
        "primary-writing-model",
        "backup-writing-model",
    ],
}
Then your runtime can try the next model if the first one fails.

This is much easier when all models are accessed through a unified API.

Usage Tracking Is Part of Agent Design
Agents can be expensive.

A single user request may trigger:

planning call,
search query generation,
document summarization,
tool result analysis,
final answer generation,
evaluation call.
That means one visible user action may contain many hidden model calls.

Without usage tracking, you may not know:

which tasks are expensive,
which models are overused,
where latency comes from,
how often fallback happens,
which users or workflows consume the most tokens.
This is why a gateway with usage statistics is valuable.

It gives developers operational visibility.

The Gateway Is Not the Agent
It is important to separate responsibilities.

A model gateway does not replace your agent framework.

It supports it.

The agent framework handles:

goals,
tools,
memory,
planning,
execution,
evaluation.
The model gateway handles:

model access,
provider abstraction,
routing support,
reliability,
usage visibility.
Together, they form a much better foundation.

Final Thought
Most AI agent discussions focus on intelligence.

But production agents also need infrastructure.

A reliable agent needs:

the right model for each task,
fallback when models fail,
cost control,
usage tracking,
provider flexibility,
clean API abstraction.
That is why I think the model gateway is one of the missing layers in many AI agent frameworks.

If you are building agents, do not just ask:

Which model should I use?

Ask:

How will my agent access, route, monitor, and fallback across models?

That question leads to a much better architecture.

I am building OpenRain to make this layer easier for developers.

OpenRain provides an OpenAI-compatible AI API Gateway for accessing multiple model providers through one unified interface.

https://openrain.ai

Copy
---

# Article 3

markdown

title: A Practical Architecture for Building a Personal General-Purpose AI Agent
published: false
description: A simple layered architecture for building personal AI agents with tools, memory, planning, routing, and multi-model access.

tags: ai, agents, architecture, productivity

A Practical Architecture for Building a Personal General-Purpose AI Agent
A useful personal AI agent is not just a chatbot.

It is a system that can understand your goals, remember your preferences, use tools, choose the right model, and complete tasks across different contexts.

That sounds complex.

But the architecture can be simple if you break it into layers.

In this article, I will describe a practical architecture for building a personal general-purpose AI agent.

The goal is not to build a fully autonomous sci-fi assistant.

The goal is to build something useful, reliable, and extensible.

The Core Idea
A personal AI agent should work like this:

CopyUser gives a goal
↓
Agent understands intent
↓
Agent creates a plan
↓
Agent chooses tools
↓
Agent chooses models
↓
Agent executes steps
↓
Agent checks result
↓
Agent returns useful output
This requires more than one prompt.

It requires an agent runtime.

The 7-Layer Architecture
Here is the architecture I recommend:

Copy1. Interface Layer

User Profile Layer
Intent Layer
Planning Layer
Tool Layer
Memory Layer
Model Gateway Layer
Let’s go through each one.
Interface Layer
This is where the user interacts with the agent.

It could be:

web app,
CLI,
browser extension,
mobile app,
Slack bot,
Telegram bot,
desktop assistant.
The interface should be thin.

It should not contain your model routing, memory logic, or tool execution logic.

A good structure is:

CopyFrontend
↓
Agent API
↓
Agent Runtime
This keeps your agent portable across different interfaces.

User Profile Layer A personal agent needs to know basic user preferences.

For example:

Copy{
"language": "English",
"tone": "clear and practical",
"timezone": "Asia/Shanghai",
"preferred_format": "markdown",
"technical_stack": ["Python", "TypeScript", "PostgreSQL"]
}
This helps the agent personalize output.

For example, if the user often publishes on DEV.to, the agent can format drafts in Markdown with front matter, tags, headings, and code blocks.

Personalization should be useful, not invasive.

The user should be able to inspect, edit, or delete stored preferences.

Intent Layer The intent layer classifies what the user is asking for.

Examples:

User Request Intent
“Summarize this document” document_summary
“Fix this bug” coding_help
“Write a blog post” content_generation
“Compare these tools” research
“Plan my week” planning
“Extract action items” information_extraction
Intent classification helps the agent decide what workflow to run.

A coding task should not use the same workflow as a writing task.

A research task should not use the same model strategy as a quick classification task.

Planning Layer The planning layer breaks a goal into steps.

Example:

CopyGoal:
Write a DEV.to article about AI model gateways.

Plan:

Identify target audience.
Explain the problem with single-provider agents.
Introduce model gateway architecture.
Add a simple code example.
Explain routing and fallback.
End with a practical takeaway. Not every task needs planning.

If the user asks:

CopyTranslate this sentence into French.
The agent can answer directly.

But if the user asks:

CopyCreate a launch content plan for my AI API gateway.
Planning becomes useful.

Tool Layer Tools allow the agent to interact with the world.

Examples:

Copysearch_web(query)
read_file(path)
write_note(title, content)
query_database(sql)
send_email(to, subject, body)
create_calendar_event(data)
call_internal_api(endpoint, payload)
Tools should be explicit and permissioned.

The model can suggest an action, but the runtime should decide whether the action is allowed.

For personal agents, this is especially important.

You probably do not want an agent sending emails, deleting files, or charging a credit card without confirmation.

Memory Layer Memory gives the agent continuity.

There are several types of memory:

Short-Term Memory
Current conversation context.

CopyWhat is the user asking now?
What files are being discussed?
What steps have already been completed?
Long-Term Memory
Persistent user preferences.

CopyThe user prefers Markdown.
The user writes for developers.
The user is building OpenRain.
Project Memory
Context for a specific project.

CopyProject: OpenRain content strategy
Audience: developers building AI agents
Positioning: AI API Gateway for multi-model access
Operational Memory
Execution traces.

CopyWhich model was used?
How many tokens were consumed?
Which tools were called?
Did fallback happen?
This last type is often ignored, but it is very important for improving agent reliability.

Model Gateway Layer The model gateway layer handles model access.

This is where a platform like OpenRain fits naturally.

OpenRain provides an AI API Gateway with:

OpenAI-compatible API access,
access to many AI models,
unified model calling,
smart routing,
automatic failover,
usage statistics,
developer-friendly integration.
For an agent framework, this means you do not need to hardcode every provider directly into your agent runtime.

Instead of this:

CopyAgent → OpenAI SDK
Agent → Anthropic SDK
Agent → Gemini SDK
Agent → DeepSeek SDK
Agent → Qwen SDK
You can design this:

CopyAgent → OpenRain Gateway → Multiple Model Providers
That makes your agent easier to maintain.

Example Workflow
User request:

CopyWrite a technical article about why personal AI agents need model routing.
The agent runtime may process it like this:

CopyInterface:
Receive request

User Profile:
User prefers practical developer writing

Intent:
content_generation

Planning:
Create article outline

Tool Layer:
Optionally fetch product context

Memory:
Recall OpenRain positioning

Model Gateway:
Use a strong writing model

Final:
Return DEV.to-ready Markdown article
The result feels simple to the user.

But internally, the system is layered.

That is what makes it extensible.

Why the Model Gateway Should Be Designed Early
Many developers start with memory or tools first.

That is understandable.

But I think model access should be designed early.

Why?

Because everything depends on it.

Your planner needs a model.
Your summarizer needs a model.
Your evaluator needs a model.
Your tool-call generator needs a model.
Your final response writer needs a model.

If model access is messy, the whole agent becomes messy.

A gateway gives you a clean foundation.

Final Thought
A personal general-purpose AI agent is not one giant prompt.

It is a layered system.

The model is important, but the infrastructure around the model is just as important.

A practical agent needs:

interface,
user profile,
intent detection,
planning,
tools,
memory,
model routing,
fallback,
usage visibility.
If you build these layers cleanly, your agent becomes easier to extend over time.

That is the kind of architecture I believe more AI applications will move toward.

And it is the kind of infrastructure OpenRain is designed to support.

https://openrain.ai

Copy

Article 4

---
title: From Chatbot to Agent: Why Tool Use Is Not Enough
published: false
description: Tool calling is only one part of building reliable AI agents. You also need routing, memory, budget control, fallback, and observability.
tags: ai, agents, llm, python
---
From Chatbot to Agent: Why Tool Use Is Not Enough
Tool calling is one of the most exciting features in modern AI applications.

It allows a model to interact with external systems:

search the web,
read files,
query databases,
call APIs,
create calendar events,
send messages,
run workflows.
Because of this, many developers think:

If my chatbot can call tools, it is now an agent.

Not quite.

Tool use is important, but it is not enough.

A real agent also needs routing, memory, budget control, fallback, and observability.

Chatbot vs Agent
A chatbot usually does this:

CopyUser message
 ↓
Model response
An agent does this:

CopyUser goal
 ↓
Understand intent
 ↓
Plan steps
 ↓
Select tools
 ↓
Select model
 ↓
Execute actions
 ↓
Validate result
 ↓
Retry or fallback
 ↓
Return final output
The difference is not just tool use.

The difference is workflow.

Tool Use Without Control Is Risky
Imagine an agent that can call tools but has no strong runtime control.

That can create problems.

For example:

It may call the wrong tool.
It may call a tool too many times.
It may spend too much money.
It may use an expensive model for simple tasks.
It may fail if one provider is unavailable.
It may produce inconsistent results.
It may be hard to debug.
This is why agent engineering is not just prompt engineering.

It is systems engineering.

The Five Capabilities Agents Need
A practical agent should have at least five capabilities:

Copy1. Tool use
2. Memory
3. Model routing
4. Budget control
5. Failover and observability
Let’s look at each.

1. Tool Use
Tools allow the agent to act.

A tool can be simple:

Copydef create_note(title: str, content: str):
    return {
        "status": "created",
        "title": title,
        "content": content
    }
But the important part is not the function itself.

The important part is the control layer around it.

Before executing a tool, your runtime should know:

Is this tool allowed?
Does the user need to confirm?
Are the inputs valid?
Is there a rate limit?
Should the action be logged?
Can this action be undone?
The model should not be the only decision-maker.

2. Memory
Memory helps the agent become personal.

For example:

Copy{
  "preferred_output": "markdown",
  "writing_style": "technical and practical",
  "current_project": "OpenRain",
  "audience": "developers building AI agents"
}
Memory helps the agent avoid asking the same questions repeatedly.

But memory should be scoped.

A good memory system should support:

user preferences,
project context,
task state,
execution history,
deletion and editing.
Do not store everything blindly.

Store what improves future work.

3. Model Routing
Different tasks need different models.

A simple model router might look like this:

Copydef select_model(task_type: str, priority: str) -> str:
    if priority == "high":
        return "premium-reasoning-model"

    if task_type == "classification":
        return "fast-low-cost-model"

    if task_type == "coding":
        return "code-strong-model"

    if task_type == "writing":
        return "writing-model"

    return "general-purpose-model"
This allows your agent to optimize for the task.

Using the same model for everything is simple, but often inefficient.

4. Budget Control
Agents can trigger multiple model calls for one user request.

For example:

CopyUser asks for a research summary
 ↓
Generate search queries
 ↓
Analyze search results
 ↓
Summarize sources
 ↓
Compare claims
 ↓
Write final answer
 ↓
Evaluate answer
That may be six or more model calls.

Without budget control, costs can grow quickly.

A practical agent should track:

input tokens,
output tokens,
model used,
task type,
cost estimate,
number of retries,
fallback events.
This is why usage visibility is important.

OpenRain includes usage statistics so developers can better understand how their AI applications consume model resources.

5. Failover and Observability
LLM providers are not perfect.

Requests fail.

Models timeout.

Rate limits happen.

Network issues happen.

A reliable agent needs fallback.

Example:

Copydef call_models_with_fallback(client, messages, candidates):
    errors = []

    for model in candidates:
        try:
            return client.chat(model=model, messages=messages)
        except Exception as error:
            errors.append({
                "model": model,
                "error": str(error)
            })

    raise RuntimeError(f"All models failed: {errors}")
This is simple, but powerful.

The agent can continue operating even if the primary model fails.

Observability then helps you understand what happened:

Copy{
  "task_type": "writing",
  "primary_model": "model-a",
  "fallback_model": "model-b",
  "fallback_used": true,
  "latency_ms": 3200,
  "tokens": 1800
}
Without logs, you are guessing.

With logs, you can improve the system.

Where OpenRain Fits
OpenRain is useful as the model infrastructure layer.

Instead of building direct integrations with many providers, your agent can call an OpenAI-compatible gateway.

CopyAgent Runtime
 ↓
OpenRain AI Gateway
 ↓
Multiple Model Providers
This helps with:

unified API access,
multi-model support,
smart routing,
automatic failover,
usage tracking,
simpler developer experience.
For agent builders, this means less time maintaining provider glue code and more time improving the actual agent experience.

Final Thought
Tool calling is important.

But tool calling alone does not make a reliable agent.

A real agent needs a runtime.

It needs memory.
It needs routing.
It needs budget awareness.
It needs fallback.
It needs observability.
It needs clean infrastructure.

That is the shift from chatbot building to agent engineering.

If you are building agents, do not only ask:

What tools can my model call?

Also ask:

How does my agent choose models, recover from failure, track cost, and improve over time?

That is where the real engineering begins.

I am exploring this infrastructure layer with OpenRain, an AI API Gateway for developers building multi-model AI applications and agents.

https://openrain.ai

Copy
---

# Article 5

markdown

title: Building AI Apps with One API Key Across Multiple Models
published: false
description: A practical introduction to using a unified AI API gateway to simplify model access for AI applications and agents.

tags: ai, api, llm, devtools

Building AI Apps with One API Key Across Multiple Models
One of the annoying parts of building AI applications is managing model access.

You may want to use:

OpenAI for general reasoning,
Claude for writing or long-context tasks,
Gemini for multimodal workflows,
DeepSeek for reasoning,
Qwen for Chinese and open-source model use cases,
Mistral for coding,
other providers for specialized needs.
Each provider may have different:

API keys,
SDKs,
request formats,
rate limits,
billing dashboards,
model names,
error formats,
availability patterns.
For small experiments, this is manageable.

For real applications, it becomes operational overhead.

That is why I like the idea of using a unified AI API gateway.

The Problem
A typical multi-model application can quickly become messy:

CopyApp
├── OpenAI client
├── Anthropic client
├── Gemini client
├── DeepSeek client
├── Qwen client
├── Mistral client
├── custom retry logic
├── custom usage tracking
└── custom fallback logic
This is not where most developers want to spend their time.

If you are building an AI product, you probably want to focus on:

user experience,
workflows,
prompts,
tools,
evaluation,
product logic.
Not provider glue code.

The Gateway Pattern
A cleaner architecture looks like this:

CopyYour App
↓
Unified AI API Gateway
↓
Multiple Model Providers
Your app calls one API.

The gateway handles the model access layer.

OpenRain is built around this idea.

It provides an AI API Gateway with OpenAI-compatible APIs, access to many models, smart routing, automatic failover, and usage statistics.

For developers, this means simpler integration.

Why OpenAI-Compatible Matters
OpenAI-compatible APIs are useful because many developers and tools already understand the request format.

A common chat request looks like this:

Copy{
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": "Write a product description for an AI API gateway."
}
]
}
If your gateway supports this style, you can keep your application code simple.

You can switch models by changing configuration instead of rewriting integrations.

Minimal Python Example
Copyimport os
import requests

api_key = os.getenv("OPENRAIN_API_KEY")

response = requests.post(
"https://openrain.ai/v1/chat/completions",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
json={
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": "Give me 5 ideas for an AI agent app."
}
],
},
)

data = response.json()
print(data["choices"][0]["message"]["content"])
This is enough to start building.

Minimal JavaScript Example
Copyconst response = await fetch("https://openrain.ai/v1/chat/completions", {
method: "POST",
headers: {
"Authorization": Bearer ${process.env.OPENRAIN_API_KEY},
"Content-Type": "application/json"
},
body: JSON.stringify({
model: "gpt-4o",
messages: [
{
role: "user",
content: "Explain AI API gateways in one paragraph."
}
]
})
});

const data = await response.json();
console.log(data.choices[0].message.content);
Again, the point is not just the code.

The point is the abstraction.

Your app talks to one API layer.

Use Case: AI Agent Model Routing
Suppose you are building a personal AI agent.

You may define model roles like this:

Copy{
"fast": "fast-low-cost-model",
"reasoning": "reasoning-model",
"writing": "writing-model",
"coding": "coding-model",
"fallback": "backup-model"
}
Then your agent can choose based on task type:

Copydef model_for_task(task_type):
if task_type == "coding":
return "coding-model"
if task_type == "reasoning":
return "reasoning-model"
if task_type == "summary":
return "fast-low-cost-model"
return "writing-model"
This keeps your app flexible.

Use Case: Fallback
If a model fails, your app can try another one.

Copydef call_with_fallback(messages, models):
for model in models:
try:
return call_model(model, messages)
except Exception:
continue

raise Exception("All models failed")

With a unified gateway, fallback is easier because all candidates can share the same API interface.

Use Case: Cost Control
Not every task needs a premium model.

For example:

CopyTag generation → cheap model
Daily summary → cheap model
Blog post draft → balanced model
Legal analysis → premium model
Complex coding → premium model
A unified gateway plus usage statistics makes it easier to understand and optimize model spend.

Why This Matters
AI applications are moving from simple chat interfaces to complex workflows.

That means developers need better infrastructure.

A modern AI app may need:

multiple models,
multiple providers,
retries,
routing,
fallback,
usage tracking,
team-level API keys,
cost visibility.
Building all of that yourself takes time.

Using a gateway lets you start with a cleaner foundation.

Final Thought
The future of AI development is multi-model.

No single model will be the best choice for every task.

Developers need a simple way to access, route, monitor, and fallback across models.

That is why the gateway pattern is becoming increasingly important.

If you are building AI apps or agents, start by asking:

Can my application switch models without changing core code?

If yes, your architecture is already in a better place.

OpenRain is an AI API Gateway designed around this idea: one OpenAI-compatible API to access multiple AI models with routing, failover, and usage visibility.

https://openrain.ai

What Building a Multi-Model AI Gateway Taught Me About Reliability

Mundo Ghose — Sat, 06 Jun 2026 12:11:30 +0000

I’m building OpenRain, an OpenAI-compatible AI API gateway. I originally thought the hard part would be integrating more providers. I was wrong. The hard part is absorbing inconsistency — and still giving developers something stable enough to trust in production.

Building an OpenAI-Compatible Multi-Model Gateway Taught Me What “Unified API” Really Means

When I first started thinking about a multi-model gateway, I had a fairly simple mental model.

Take a few major model providers. Put a clean API in front of them. Normalize the request shape. Forward the response. Add billing and routing. Done.

That picture survives for about five minutes.

The moment you try to make the system feel reliable to real developers, the problem changes completely. You stop thinking like someone stitching together vendor APIs, and start thinking like someone who is responsible for a production contract. That shift is where most of the complexity comes from.

A “unified AI API” sounds like an interface design problem. In practice, it turns into a systems problem: inconsistent semantics, unstable tail latency, streaming edge cases, partial failures, fuzzy usage data, and policy enforcement that has to happen in the hot path.

This post is a reflection on what surprised me most while building an OpenAI-compatible gateway, and why the word “compatible” hides much more engineering work than it seems.

1. I learned very quickly that “OpenAI-compatible” is an overloaded phrase

At first glance, compatibility looks straightforward. Many providers accept a similar request shape:


json
{
  "model": "some-model",
  "messages": [
    { "role": "user", "content": "Hello" }
  ]
}
But request-shape compatibility is the easy part.

What breaks much sooner is behavior.

Different providers interpret system prompts differently. Some support tool calling, but only in a narrow sense. Some stream clean deltas; others stream in a way that is technically valid but operationally awkward. Some report usage consistently. Some are mostly compatible until you hit a multimodal edge case, or a timeout, or a high-concurrency burst.

This was one of the first lessons that changed how I thought about the architecture: if you want one API surface outside, you need a much stricter abstraction inside.

For me, that meant treating the gateway as four separate problems:

defining one canonical internal request model
maintaining a capability map for each provider and model
translating requests into provider-specific formats
normalizing responses and failures back into one stable contract
Without that separation, the abstraction leaks everywhere.

And once it leaks, developers feel it immediately.

2. I thought routing would be about model selection. It turned out to be about constraints.
A lot of people talk about “access to many models” as if the number itself is the value.

I don’t think that’s how developers actually experience it.

In real usage, they are usually not asking for “many models.” They are asking for a model that fits a constraint: lower latency, better reasoning, stronger coding, longer context, better Chinese, lower cost, more stable uptime.

That changes the design of routing.

If routing is just if model_name == x, then you are not really building a gateway. You are building a switch statement with branding around it.

A usable gateway needs a model registry that knows more than names. It needs to know what each model can do, how expensive it is, how it behaves under load, what regions are healthy, and what kinds of traffic it is appropriate for.

A simplified version looks something like this:

Copy{
  "model_id": "provider/model-version",
  "supports_stream": true,
  "supports_tools": true,
  "supports_vision": false,
  "supports_reasoning": true,
  "context_window": 128000,
  "latency_profile": {
    "p50_ms": 900,
    "p95_ms": 2400
  },
  "availability_score": 0.997
}
Once you have that, routing becomes something more honest. It becomes a policy engine.

You can send long-document workloads to long-context models. You can bias interactive traffic toward lower p95. You can control which keys can access which classes of models. You can choose fallback targets that are operationally safe, not just syntactically similar.

That was an important mindset shift for me: the real job of routing is not choosing a model. It is choosing a compromise.

3. The latency problem was less about speed than about unpredictability
I used to think “fast enough” was mostly a p50 question.

It isn’t.

In user-facing AI systems, what people remember is variance.

A model that usually responds in one second but occasionally takes twelve feels unreliable, even if its average latency looks fine in a dashboard. This becomes especially obvious in chat, copilot-like experiences, and any workflow where the user is waiting synchronously.

So over time I became much more interested in p95 and p99 than in average speed.

That has consequences.

First, timeout budgets can’t be one-size-fits-all. A lightweight chat model and a heavier reasoning model should not inherit the same expectations.

Second, “provider is up” is not a very useful health model. A provider can be technically available and still be degraded enough to damage user experience. Health has to be dynamic and contextual: recent timeouts, regional instability, error rate, queueing behavior, tail latency.

Third, failover is not just a reliability feature. It is a semantics feature.

If you retry too aggressively across providers, you can create duplicate outputs, ambiguous billing, inconsistent tone, or broken streaming. By the time I appreciated this fully, I had stopped thinking of failover as “just try the next one.” It became a question of whether the substitution is safe, whether the request is still idempotent in practice, and whether the user experience survives the handoff.

That is a much less glamorous problem than “smart routing,” but in production it matters more.

4. Streaming is where the abstraction gets stress-tested
Normal request-response flows are relatively easy to make look clean.

Streaming is where the truth comes out.

This was one of the most humbling parts of the system for me. On paper, many providers support streaming. In reality, they do it differently enough that “supported” doesn’t mean “interchangeable.”

The differences show up everywhere:

chunk structure
delta semantics
finish reasons
heartbeat behavior
tool call emission
usage timing
connection endings
partial failures
You can hide some of this, but not by pretending it isn’t there. The only sustainable answer I found was to explicitly build a stream normalization layer:

Copyprovider stream -> parser -> event normalizer -> policy filter -> client stream
Even then, the problem doesn’t end at format normalization.

Once streaming traffic grows, you start worrying about backpressure, client disconnects, memory growth, timeout propagation, cleanup of half-open streams, and whether a stream that already emitted tokens can still be retried without making the product feel broken.

A lot of “compatible API” claims sound convincing until streaming enters the picture. That’s where the operational differences become impossible to ignore.

5. Usage accounting ended up being much closer to distributed state management than to finance
Before building this, I would probably have described usage accounting as a billing concern.

Now I think that framing is incomplete.

In a multi-model gateway, usage is part of runtime truth. It is not just something you summarize later for a dashboard.

You have to decide what the system believes happened when a request is admitted, forwarded, partially streamed, completed, retried, interrupted, or compensated. You have to deal with providers that report usage at different times. You have to handle client disconnects, partial outputs, ambiguous timeouts, and request paths that were rejected before they ever reached upstream.

At some point, I realized the only sane way to think about this was as an internal ledger with explicit states, not as a single “token in / token out” record.

Something like:

received
admitted
forwarded
partially streamed
completed
failed
compensated
That model is less elegant than a neat billing summary, but it is closer to what actually happens in a live system.

And if your internal truth is fuzzy, your billing, rate limiting, dashboards, and support tooling will all drift in different directions.

6. Per-key control sounds administrative until you have to enforce it in real time
From the outside, things like unified billing, key-level restrictions, and model allowlists can sound like console features.

From the inside, they are request-path logic.

A real platform quickly accumulates rules like:

this key can use only lower-cost models
this project cannot call reasoning models
this tenant has a spend ceiling
this environment must stay within a region or provider set
this customer can access multimodal features, but not image generation
this internal service can stream, another cannot
None of that can be bolted on cleanly at the end. It has to be designed into the gateway itself.

That was another lesson I didn’t appreciate enough early on: authorization in AI infrastructure is no longer just about identity. It is about identity under constraints.

Who is making the call is only the first question. The more difficult question is: what are they allowed to invoke, under which policy, with which budget, and with what operational risk?

7. Error handling is one of the most human parts of the system
There is something deceptively important about error design.

Raw upstream errors are often inconsistent, noisy, or not actionable. Different vendors disagree on status codes, error shapes, retry semantics, and what even counts as a timeout.

If you pass all of that directly through, the user ends up integrating not with your platform, but with the chaos behind it.

So I gradually came to see error normalization as a product decision, not just an engineering cleanup task.

If the external API is unified, then the failure experience has to be unified too. Developers should not need to reverse-engineer five providers through your gateway.

A stable error contract does more than improve DX. It gives support, observability, and incident response a common language. That matters a lot once the system starts handling real traffic.

8. Observability turned out to be the part that makes everything else possible
If I had to pick one area that became more important over time, it would be observability.

Not because dashboards are nice to have, but because once multiple providers, regions, models, retries, and fallback paths sit behind one endpoint, you lose intuition very quickly.

You need to see what was requested, what was actually routed, which region it hit, whether it streamed, how many retries happened, how long it spent in each stage, whether the provider was degrading, whether the fallback rate is climbing, whether structured outputs are failing more often for one adapter than another.

Without that, routing is guesswork.

Without that, failover is guesswork.

Without that, debugging becomes a sequence of anecdotes.

Over time, I started to think of observability as the thing that converts a gateway from a thin aggregation layer into infrastructure you can improve deliberately.

9. The hardest part is that the target never stops moving
One thing I underestimated was how often the ecosystem changes beneath you.

Providers update model versions. Pricing changes. Context windows change. Rate limits shift. Streaming behavior changes. Response fields evolve. A provider can remain “compatible” in broad marketing terms while becoming materially different in an edge case that breaks your assumptions.

That means architecture needs to assume motion.

The patterns that have felt most durable to me are fairly unglamorous:

isolate providers behind adapters
keep one canonical internal schema
keep routing policy configurable
centralize capability metadata
make quirks declarative when possible
preserve enough telemetry to detect silent regressions
None of this is exciting to describe. All of it becomes important once you are no longer integrating a demo, but operating a surface that other developers are relying on.

10. What people call “aggregation” is often really a search for reliability
From the outside, a multi-model gateway looks like a convenience layer.

Inside the system, it feels more like an attempt to absorb volatility.

Yes, unified access matters. So does lower switching cost. So does having one integration surface instead of many.

But the part that keeps revealing itself as the real value is reliability: consistent behavior, safer fallbacks, clearer errors, better visibility, better policy control, and fewer vendor-specific surprises leaking into application code.

That’s the part I did not fully understand before building this.

The hard part is not exposing more models.

The hard part is standing between a messy, fast-changing provider ecosystem and a developer who just wants their production system to behave predictably.

That, more than anything else, is what “unified API” has come to mean to me.

If you’re building in this space too, I’d genuinely be curious how you think about normalization, routing, streaming, and failover. I’ve found that those four topics reveal most of the real architecture very quickly.

Why Integrating Multiple LLM Providers Gets Messy Fast

Mundo Ghose — Fri, 05 Jun 2026 08:41:43 +0000

What Building AI Products Taught Me About Model Fragmentation

When I first started building with large language models, I assumed the hard part would be choosing the best model.

I was wrong.

The real pain started after that.

Once you move beyond a prototype, "just integrate another model provider" quickly turns into an infrastructure problem:

different APIs
different auth patterns
different billing dashboards
different failure modes
different latency profiles
different regional availability
custom fallback logic everywhere

In theory, using multiple models gives you flexibility.

In practice, it often gives you fragmentation.

The hidden cost of multi-model AI apps

A lot of discussion in AI focuses on model quality.

Which model is better at reasoning?

Which one is cheaper?

Which one supports multimodal input better?

Which one has the best context window?

Those are important questions.

But if you're actually shipping an AI product, the harder question is usually:

How do I keep my stack flexible without turning my backend into a mess?

This becomes obvious the moment you need to do any of the following:

switch providers because pricing changed
add failover because one upstream becomes unstable
route certain workloads to different models
compare quality across providers
enforce cost controls across teams
maintain one product while the model landscape keeps shifting

The model layer is evolving faster than most application stacks can absorb.

Why I started thinking about this differently

At some point I realized the real problem wasn't "lack of model access."

It was switching cost.

If your app is tightly coupled to one provider's API shape, reliability behavior, or billing workflow, every future decision gets more expensive:

trying a better model
migrating to a cheaper model
adding redundancy
supporting multiple regions
giving teams controlled access to different models

That cost doesn't always show up on day one.

It shows up later, when your product needs to move faster than your integration layer allows.

What I wanted instead

I wanted something simpler:

one consistent API style
one key
one integration pattern
access to multiple major models
better reliability primitives
cleaner billing visibility

That idea became OpenRain.

OpenRain is an OpenAI-compatible AI API gateway designed to help developers access major AI models through a unified interface.

Instead of integrating providers one by one, the goal is to give developers a more stable control layer for model access.

Today, the product focuses on:

OpenAI-compatible API access
50+ AI models
global smart routing
automatic failover
low-latency, high-availability infrastructure
unified billing
pay-as-you-go usage

If you're curious, the product is here: https://openrain.ai/

The important part isn't "50+ models"

Honestly, I don't think "more models" is the real value.

The real value is reducing the operational burden of optionality.

Most developers don't want infinite AI choices.

They want:

flexibility without chaos
redundancy without custom glue code
experimentation without major rewrites
billing visibility without five dashboards
a way to adapt as the model ecosystem changes

That distinction matters a lot.

Because once you stop treating model access as a feature list and start treating it as infrastructure, you begin designing for resilience instead of novelty.

A simple example

If you've already built against an OpenAI-style interface, the integration should feel familiar.


python
import requests

url = "https://openrain.ai/v1/chat/completions"
headers = {
    "Authorization": "Bearer sk-your-api-key",
    "Content-Type": "application/json",
}
payload = {
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}],
}
response = requests.post(url, json=payload, headers=headers)
print(response.json()["choices"][0]["message"]["content"])
Copy
The point of a layer like this isn't to be flashy.

The point is to reduce migration friction.

What I've learned from building in this space
A few things now feel increasingly true to me:

1. Model quality changes fast, so abstraction matters
The "best" model today may not be the best one six months from now. Hard coupling is a tax on future learning.

2. Reliability is part of product experience
Users don't care which upstream failed. They care that your product stopped working.

3. Billing fragmentation is also an engineering problem
Once multiple teams or workloads are involved, cost visibility becomes part of platform design.

4. Optionality only helps if it's usable
More choices are only useful if developers can actually switch, compare, and operate them without excessive complexity.

Why I’m sharing this on DEV
I'm posting this here because DEV is one of the few places where developers still talk openly about tradeoffs, implementation pain, and lessons learned — not just polished product pages.

And I think this is a conversation worth having.

A lot of AI products today are being built on top of assumptions that may not age well:

that one provider will stay best
that direct integration is always simplest
that switching later will be easy
that reliability can be bolted on afterward
Maybe sometimes that's true.

But I increasingly think AI builders need to think about flexibility earlier than they expect.

I'm curious how others are handling this
If you're building AI apps, agents, internal tools, or SaaS products:

Are you integrating providers directly?
Are you already dealing with failover or routing logic?
How painful has model switching been for your team?
Do you optimize first for cost, quality, latency, or reliability?
I'd genuinely love to hear how other developers are thinking about this problem.

How to Choose the Right LLM for Your AI Application

Mundo Ghose — Thu, 04 Jun 2026 10:00:26 +0000

How to Choose the Right LLM for Your AI Application
Large Language Models are becoming the core infrastructure behind many modern AI applications.

From chatbots and AI agents to SaaS tools, coding assistants, customer support systems, and workflow automation platforms, LLMs are changing how software is built and used.

But as more models become available, developers face a new challenge:

Which LLM should I use for my application?

There is no single best model for every use case. Some models are better at reasoning. Some are faster. Some are cheaper. Some are better for coding, writing, multilingual tasks, or high-volume production usage.

In this article, we will look at how developers can choose the right LLM based on cost, latency, quality, scalability, and flexibility.

Why Choosing an LLM Is Not Simple
A few years ago, choosing an AI model was relatively straightforward. There were fewer options, fewer providers, and fewer production use cases.

Today, the LLM ecosystem is much more complex.

Developers may need to compare models based on:

Output quality
Response speed
API cost
Context window size
Coding ability
Reasoning ability
Multilingual performance
Stability
Rate limits
Tool calling support
Embedding support
Vision or multimodal capability
This makes model selection a real engineering decision, not just a product decision.

For example, a customer support chatbot may need fast and affordable responses.
An AI coding assistant may need stronger reasoning and code generation ability.
A document analysis tool may need a larger context window.
An AI agent may need low latency and reliable tool calling.

Different tasks require different models.

Understand Your Use Case First Before choosing an LLM, you need to clearly define your use case.

Ask yourself:

Is the application user-facing or internal?
Does it require real-time responses?
Is accuracy more important than speed?
Will the application process long documents?
Do you need code generation?
Do you need multilingual support?
How many requests will you handle per day?
What is your maximum acceptable cost per request?
For example:

Chatbot
A chatbot usually needs:

Low latency
Stable response quality
Affordable pricing
Good conversational ability
AI Agent
An AI agent usually needs:

Strong reasoning
Tool calling
Reliable instruction following
Fast response time
Content Generation Tool
A content generation product usually needs:

Good writing quality
Creativity
Style control
Low cost for high-volume usage
Code Assistant
A coding assistant usually needs:

Strong code understanding
Good debugging ability
Support for technical explanations
Larger context window
Once you understand your use case, it becomes much easier to compare models.

Compare Quality and Cost Together One common mistake is choosing a model only because it has the highest benchmark score.

In production, the best model is not always the most powerful one.

Sometimes, a smaller or cheaper model is good enough for the task. If your application handles thousands or millions of requests, even a small price difference can have a big impact.

For example:

Use a powerful model for complex reasoning.
Use a faster, cheaper model for simple classification.
Use a lightweight model for short customer support replies.
Use a larger-context model only when long documents are required.
This strategy can significantly reduce your LLM costs without hurting user experience.

The key is to match the model to the task.

Latency Matters More Than You Think For many AI applications, latency is part of the product experience.

If a chatbot takes too long to respond, users may leave.
If an AI agent is slow, the entire workflow feels inefficient.
If a SaaS feature depends on real-time AI output, slow responses can reduce product value.

When testing LLMs, developers should measure:

Time to first token
Total response time
Average latency
Latency under high traffic
Stability during peak usage
A model with slightly lower quality but much faster response time may be a better choice for interactive applications.

Speed matters.

Avoid Vendor Lock-In Another important consideration is flexibility.

If your application is tightly coupled to a single model provider, switching models later can become difficult.

You may need to rewrite API logic, update prompt formats, change error handling, modify billing workflows, and test everything again.

This creates vendor lock-in.

A better approach is to design your AI infrastructure in a model-agnostic way.

That means your application should be able to switch between different models without major code changes.

This is where a unified LLM API can be very useful.

Use a Unified API for Multiple Models Instead of integrating multiple LLM providers one by one, many developers are now using AI model aggregation platforms.

A model aggregation platform allows you to access multiple LLMs through one API.

This gives developers several advantages:

One integration for many models
Easier model switching
Lower development cost
Faster testing and comparison
More pricing options
Better flexibility
Reduced maintenance work
At [openrain.ai], we are building a unified AI model platform that helps developers access multiple leading LLMs with lower cost, low latency, and higher efficiency.

Instead of managing different accounts, API keys, pricing rules, and documentation from many providers, developers can connect once and use multiple models from one place.

This is especially useful for teams building:

AI chatbots
AI agents
SaaS AI features
Developer tools
Customer support automation
Content generation platforms
Internal AI workflows

Test Multiple Models Before Production Do not choose a model only based on marketing pages or benchmark results.

The best way to choose an LLM is to test it with your own real-world data.

You can create a simple evaluation set with examples from your actual product.

For each model, compare:

Accuracy
Response quality
Speed
Cost
Failure rate
Formatting consistency
Instruction following
User satisfaction
For example, if you are building a support chatbot, test each model with real customer questions.

If you are building a coding assistant, test with real code issues.

If you are building a summarization tool, test with real documents.

Your own use case is the most important benchmark.

Use Different Models for Different Tasks A single application does not have to use only one LLM.

In many production systems, using multiple models is actually more efficient.

For example:

A cheaper model can classify user intent.
A stronger model can handle complex reasoning.
A fast model can generate short replies.
A long-context model can process large documents.
A coding model can handle programming-related tasks.
This multi-model strategy helps balance quality, cost, and performance.

However, managing multiple models manually can become complex. That is another reason why a unified model API is useful.

With [Platform Name], developers can experiment with different models and select the best one for each task through a single platform.

Monitor Performance After Launch Choosing an LLM is not a one-time decision.

Models change.
Pricing changes.
User behavior changes.
New models are released.
Your product requirements evolve.

After launching your AI application, you should continue monitoring:

API cost
Latency
Error rate
User feedback
Output quality
Token usage
Model performance by task
This helps you optimize your AI system over time.

The most successful AI products are not just built with powerful models. They are built with flexible infrastructure that can adapt quickly.

Final Thoughts
Choosing the right LLM is about more than selecting the most famous or most powerful model.

A good decision requires balancing:

Quality
Cost
Latency
Scalability
Flexibility
Developer experience
For many teams, the best solution is not using one model forever. It is building a flexible AI infrastructure that allows you to use the right model for the right task.

That is why unified LLM APIs and model aggregation platforms are becoming increasingly important.

With [openrain.ai

], developers can access multiple AI models through one simple API, reduce integration complexity, lower costs, improve response speed, and build AI applications more efficiently.

If you are building an AI product and want more model choices with better pricing and lower latency, you can try [openrain.ai].