Your app sends a text-only request. It also sends an image-with-a-question request. Right now, both go to the same model — usually the most expensive one that can handle both. That's the wrong unit of integration.
The provider is the wrong unit of integration. The model group is the right one. And once you think in model groups, a question you couldn't answer before becomes obvious: why does a text-only request need to hit a vision-capable backend at all?
Multimodal-aware routing means your model group filters its targets by the request's modalities. Text requests go to a cheap text tier. Image requests skip incompatible backends and land on a vision-capable one. You get lower cost, fewer failed requests, and no code changes in your app.
Here's how to build it with ModelPlane.
The problem: one model for every modality is expensive and fragile
Most LLM apps are multimodal in practice even when they don't think of themselves that way. A support bot sends text. A document analyzer sends images. A coding assistant sends screenshots of error messages. A data pipeline sends charts.
If you hard-code model="gpt-4o" in your client, every one of those requests — text and image alike — goes to the same backend. That's the default, and it's wrong for two reasons.
First, cost. Vision-capable flagship models are priced at a premium over text-only tiers. A text-only request that gets routed to a vision model is paying for capability it doesn't use. When 80% of your traffic is text, that's a tax on the majority of your requests.
Second, fragility. Not every model accepts images. If you route an image request to a text-only model, you get a 400 error — or worse, the model silently ignores the image and answers from text alone. You've shipped a bug you can't see.
The fix isn't to write modality-checking logic in every service. It's to make the routing layer aware of what the request contains.
Model groups: the abstraction that makes this possible
A model group is a name you control — prod-chat, vision-pipeline, whatever — that maps to a routing strategy and a list of backend targets. Your client sends model="prod-chat" to https://modelplane.dev/v1, and the gateway expands that name into a routing config and runs the request through the routing engine.
The key insight: request.model is a name you control, not a provider model id. You can change what that name resolves to without touching your code. That's what makes multimodal-aware routing possible — the routing config, not the client, decides which backend answers.
The routing engine supports four strategy modes: single, fallback, loadbalance, and conditional. For multimodal-aware routing, you combine fallback (for reliability) with target-level filtering based on request content.
How multimodal-aware routing works
ModelPlane's routing engine walks a target tree for each request. The fallback strategy tries targets in order and advances to the next on specified status codes — that's your high-availability backbone. What multimodal-aware routing adds is a filter at the target level: if the request contains an image, skip targets that can't handle images.
Concretely, your model group's routing config looks like this:
{
"strategy": { "mode": "fallback" },
"targets": [
{
"kv_ref": "cred:<userId>:<credId>",
"override_params": { "model": "deepseek-v4-flash" },
"modalities": ["text"]
},
{
"kv_ref": "cred:<userId>:<credId2>",
"override_params": { "model": "gpt-5.4-mini" },
"modalities": ["text", "image"]
},
{
"kv_ref": "cred:<userId>:<credId3>",
"override_params": { "model": "gpt-5.6-sol" },
"modalities": ["text", "image"]
}
]
}
The gateway inspects the incoming request. If it's text-only, the first target (cheap, fast, text-only) handles it. If it contains an image, the router skips the first target and goes straight to a vision-capable backend. If that backend fails, it falls through to the next vision-capable one.
No client-side logic. No modality checks in your app code. The routing layer handles it.
A concrete example: DeepSeek-first, vision-capable fallback
Let's build a real model group. The CTA for this post is "Build a DeepSeek-first, vision-capable-fallback group." Here's why that's the right shape.
DeepSeek's V4 family is cost-efficient and has a 1M-token context window with a 384K max output — excellent for long-generation text workloads. But it's not a vision model. If you route an image request there, you get an error.
So you build a group that's DeepSeek-first for text, with vision-capable fallbacks for image requests:
from openai import OpenAI
client = OpenAI(
base_url="https://modelplane.dev/v1",
api_key="gw-...", # your ModelPlane gateway key
)
# Text-only request → routes to deepseek-v4-flash
text_response = client.chat.completions.create(
model="prod-chat",
messages=[
{"role": "user", "content": "Summarize this changelog in three bullets."}
],
)
# Image request → skips text-only targets, routes to a vision-capable backend
image_response = client.chat.completions.create(
model="prod-chat",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What's the error in this screenshot?"},
{
"type": "image_url",
"image_url": {
"url": "https://example.com/error-screenshot.png"
},
},
],
}
],
)
Same client. Same model group name. Different backends, chosen automatically by the router based on what the request contains.
The first request hits DeepSeek V4 Flash — cheap, fast, text-only. The second request skips it and lands on a vision-capable model. If that model is rate-limited or returns a 5xx, the fallback strategy advances to the next vision-capable target. Your app never sees a failure.
Why routing beats client-side modality checks
You could implement modality detection in your app. Check if the message contains an image, then pick a model. It's not hard — a few lines of code.
But it's the wrong place for that logic, for the same reason system prompts belong at the router, not in your app: behavior should live with the route, not copy-pasted into every client.
Client-side modality checks have three problems:
They drift. You have five services calling models. Each one implements its own modality check. One service forgets. Another uses a different image format. Now you have inconsistent behavior across your stack.
They can't react to provider changes. Your text-only model gets a vision update. Your vision model gets deprecated. With client-side checks, you're updating code and redeploying. With router-side filtering, you edit the model group config and you're done.
They can't enforce policy. A client can always send an image to a text-only model. The router can't stop it. With multimodal-aware routing, the gateway enforces the filter — an image request physically cannot land on a text-only target.
The router is the single enforcement point. That's where modality awareness belongs.
Combining multimodal filtering with other routing strategies
Multimodal-aware routing isn't a replacement for the other strategies — it composes with them.
Fallback + multimodal. Your vision-capable primary target fails. The router advances to the next vision-capable target. Text requests never trigger the vision fallback chain, so you're not paying vision prices for text traffic.
Loadbalance + multimodal. You want to split traffic across two vision-capable models by weight — 70% to one, 30% to another — while text-only requests go to a third, cheaper tier. The modality filter narrows the candidate pool; the load-balance strategy picks among the survivors.
Conditional + multimodal. You route by customer tier: free-tier users get the cheap vision model, paid users get the flagship. The modality filter runs first, then the conditional router picks among the vision-capable targets.
The routing engine's target tree handles all of this. Multimodal filtering is just another constraint on which targets are eligible for a given request.
What this means for your LLM stack
Multimodal-aware routing changes how you think about your model inventory. Instead of one model per use case, you have a pool of backends with different capabilities, and the router matches requests to the cheapest capable target.
The practical wins:
- Lower cost. Text requests stop paying vision premiums. If 80% of your traffic is text, that's a direct reduction in your bill.
- Fewer failures. Image requests never hit text-only backends. No more silent image-ignoring bugs.
-
One client. Your app sends
model="prod-chat"and never thinks about modalities again. - Provider flexibility. Swap DeepSeek for another text tier, add a new vision model, rebalance weights — all in the model group config, no deploys.
The provider is the wrong unit of integration. The model group is the right one. Multimodal-aware routing is what happens when you take that seriously — your routing layer knows what your requests contain, and it routes accordingly.
Build a DeepSeek-first, vision-capable-fallback group and watch your text traffic stop paying vision prices.
Ready to route smarter? Start free — $5 credits, no card. Create your first model group in minutes and see multimodal-aware routing in action.
This post is part of a series on building production-grade LLM infrastructure with ModelPlane. Stay tuned for deep dives into routing strategies, model groups, unified thinking, system-prompt injection, high availability, coding-plan routing, price-aware routing, billing, the provider catalog, teams, and the routing engine itself.
Top comments (0)