DEV Community

ModelPlane
ModelPlane

Posted on • Originally published at modelplane.dev

Embedding Groups: Dimension-Aware Routing for /v1/embeddings

#engineering #product #embedding

Embedding Groups: Dimension-Aware Routing for /v1/embeddings

If you're building on embeddings, you already know the pain: you pick a provider, bake model="text-embedding-3-large" into your code, and pray the pricing and latency stay acceptable. The moment you want to switch or add a fallback, you discover the dirty secret of embedding vectors — they're meaningless unless every vector in your store has the same dimension. A 3072-dimension vector from one model and a 1536-dimension vector from another can't be compared, clustered, or searched. Your vector database will happily store both, and then your similarity queries return garbage.

ModelPlane's embedding groups solve this by treating dimension as a first-class routing constraint. The gateway validates that every backend in an embedding model group produces vectors with the same dimension — at configuration time, before you route a single request, and at runtime, so a provider change can't silently corrupt your index.

The abstraction: model groups, not model IDs

The core ModelPlane abstraction is the model group: request.model is a name you control, not a provider model ID. You define a group like prod-embeddings, point it at one or more backends, and your application code never changes when you swap providers.

For chat and completion models, a model group can contain wildly different backends — gpt-4o and claude-3-5-sonnet have different tokenizers, different pricing, different latency profiles, but they all produce text. The routing engine doesn't care about the output shape; it just needs to get a string back.

Embeddings break that assumption. The output of an embedding call is a vector, and vectors are only useful when they share a dimensional space. You can't fall back from a 3072-dimension model to a 1536-dimension model without silently breaking every downstream consumer of those vectors.

This is why embedding groups are dimension-aware. The gateway enforces a contract that your code can't — that every backend in a group produces vectors your vector store can actually use.

Why dimension mismatch is a silent production killer

Let's be concrete about the failure mode. You have a RAG pipeline. You chunk documents, embed them with text-embedding-3-large (3072 dimensions), and store them in Pinecone or pgvector. Your app queries with the same model, gets a 3072-dimension query vector, and everything works.

Now your embedding provider has an outage. You have a fallback configured — good engineering instinct. The fallback is text-embedding-3-small (1536 dimensions). The gateway routes your query to the fallback, returns a 1536-dimension vector, and your vector store... accepts it. Pinecone doesn't reject vectors with the wrong dimension. pgvector doesn't either. Your similarity search now compares 3072-dimension stored vectors against a 1536-dimension query vector.

The results are nonsense. Not obviously broken — just subtly wrong rankings, missing relevant documents, and a production incident that takes hours to diagnose because the error messages are unhelpful or nonexistent.

This is the kind of bug that doesn't show up in staging. Staging has one provider. Production has fallbacks. And fallbacks without dimension validation are landmines.

How embedding groups enforce the contract

ModelPlane's routing engine, tryTargetsRecursively, walks a target tree according to a routing strategy: single, fallback, loadbalance, or conditional. For embedding groups, the gateway adds a dimension check on top of that tree.

At configuration time, when you create or update an embedding model group, the gateway validates that all targets resolve to models with the same embedding dimension. It uses the provider catalog's model metadata to check this before the group is saved. If you try to mix a 3072-dimension model with a 1536-dimension model, the gateway rejects the configuration.

At runtime, the gateway verifies the dimension of each embedding response against the group's expected dimension. If a provider returns a vector with the wrong dimension — whether due to a model change, a provider-side error, or a misconfigured override — the gateway treats it as a failed target and advances to the next backend in the fallback chain.

This means your fallback chain is safe by construction. You can configure prod-embeddings with text-embedding-3-large as primary and text-embedding-3-small as fallback — but the gateway will reject that group at creation time because the dimensions don't match. You'll need to pick backends that actually produce compatible vectors.

What you can build with dimension-aware groups

The constraint isn't a limitation — it's what makes the abstraction useful. Here's what becomes possible:

True embedding HA. Pick two or more providers that offer models with the same dimension. For example, OpenAI's text-embedding-3-large (3072) and a compatible model from another provider. Configure them in a fallback group. When one provider has an outage or rate-limits you, the gateway routes to the next, and every vector that comes back is guaranteed to be queryable.

Load-balanced embedding pipelines. If you have a high-volume ingestion pipeline, spread embedding requests across multiple backends with weighted load balancing. The gateway distributes traffic, and you never have to worry about a fraction of your vectors being incompatible with the rest.

Conditional routing by request metadata. Route embedding requests based on customer tier, region, or feature flag — but only among backends that produce the same dimension. The dimension check applies regardless of which strategy mode you use.

Provider migration without re-embedding. Want to move from one embedding provider to another? If they offer same-dimension models, you can run both in a load-balanced group, migrate traffic gradually, and re-embed your corpus at your own pace. The gateway ensures every new vector is compatible with your existing store.

A concrete example

Here's what an embedding group looks like in practice. First, create the group via the ModelPlane API:

curl -X POST https://modelplane.dev/api/model-groups \
  -H "Authorization: Bearer $MODELPLANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "prod-embeddings",
    "routing_config": {
      "strategy": { "mode": "fallback" },
      "targets": [
        { "kv_ref": "cred:user:openai", "override_params": { "model": "text-embedding-3-large" } },
        { "kv_ref": "cred:user:voyage", "override_params": { "model": "voyage-3-large" } }
      ]
    }
  }'
Enter fullscreen mode Exit fullscreen mode

If text-embedding-3-large and voyage-3-large have the same dimension, the group is created. If not, the gateway returns a validation error telling you the dimensions don't match.

Now your application code is trivial. Point your OpenAI client at ModelPlane and use the group name as the model:

from openai import OpenAI

client = OpenAI(
    base_url="https://modelplane.dev/v1",
    api_key="your-modelplane-api-key",
)

# This routes to prod-embeddings, which handles fallback and dimension validation
response = client.embeddings.create(
    model="prod-embeddings",
    input="The vector database is the new hotness, but only if your vectors are compatible.",
)

vector = response.data[0].embedding
print(f"Dimension: {len(vector)}")
Enter fullscreen mode Exit fullscreen mode

That's it. Your code doesn't know which provider answered, doesn't know about fallbacks, and doesn't need to check dimensions. The gateway handles all of it.

The dimension check is the feature

Most routing gateways treat embeddings as an afterthought — just another endpoint to proxy. ModelPlane treats them as a distinct workload with a distinct contract. The dimension check isn't a validation nicety; it's the difference between a fallback that saves you from an outage and a fallback that silently corrupts your production data.

Think about what your vector store actually needs. It needs every vector to have the same dimension. It needs that invariant to hold across provider outages, model deprecations, and traffic spikes. Your application code can't enforce that invariant — it's too far from the routing decision. Your vector store can't enforce it — it accepts whatever you give it. The only place to enforce it is at the routing layer, where the gateway sees every request and every response.

That's what embedding groups do. They make dimension compatibility a property of the route, not a hope about the provider.

Beyond embeddings: the same principle applies

The dimension check is a specific instance of a general principle: the routing layer should enforce the contracts your application depends on. For chat models, that means thinking normalization — one thinking parameter, encoded per provider, so reasoning output comes back in a consistent shape. For system prompts, it means injection at the router, so safety and formatting instructions live with the route, not copy-pasted into every client. For embeddings, it means dimension validation, so your vector store never receives a vector it can't use.

ModelPlane's provider catalog covers the major embedding providers, and the same routing engine that handles your chat traffic handles your embedding traffic. One endpoint, one API key, one dashboard for usage and billing across all of it.

Start with a free account

If you're building on embeddings, you know the pain of provider lock-in and the fear of silent corruption. Embedding groups give you the safety of a validated contract and the flexibility of routing — without changing your application code.

Create your first embedding group — free $5 credits, no card required. Point your OpenAI client at https://modelplane.dev/v1, define a model group, and let the gateway handle the rest.

This post is part of a series on building production-grade AI infrastructure with ModelPlane. Check out the rest:

Top comments (0)