Your LLM gateway is throwing away the data that would improve your prompts

Most LLM gateways stop at routing. They take your request, pick a provider,
forward it, and hand back the response. Useful, but static. The prompt you shipped
is the prompt that runs forever, and the only person improving it is whoever
remembers to open the file.

Meanwhile you are generating exactly the data that would improve it. Every
request is a labelled example of what your prompt produced in production. Most
gateways log it for a dashboard and then drop it.

route-switch is a Go gateway that closes that loop. It routes across providers
through one OpenAI-compatible API, captures every invocation as a dataset, and
runs MIPROv2-style optimization against those captured traces. Routing plus a
feedback loop, behind a drop-in endpoint. MIT, and the repo is here:
https://github.com/Skelf-Research/route-switch

Routing is table stakes, the loop is the point

The gateway speaks the OpenAI API, so adopting it is changing a base URL. It
routes across OpenAI, Anthropic, Google, Ollama, Cohere, Mistral and others via
the gollm library. That half is a commodity.

The differentiated half is that prompts are treated as things that improve, not
constants. The documented loop is six steps:

1. Capture, every invocation is stored in a per-prompt SQLite database
2. Bootstrap, load recent records as calibration data
3. Generate, create instruction candidates through provider calls
4. Optimize, run Bayesian optimization across combinations
5. Evaluate, replay dataset rows with the configured evaluation strategy
6. Deploy, update the gateway with the optimized prompt

Step 1 is the part other gateways skip, and it is the part that makes steps 2
through 6 possible at all.

The storage split is the interesting design choice

There are two stores, and conflating them is the mistake I made when I first read
the architecture.

The dataset store is SQLite, one database per prompt under
dataset.base_path. It records rendered input, output, variables, cost and
success flags, timestamps. Retention is bounded by dataset.max_records. This is
the training material.

The analytics store is DuckDB. It collects request and response summaries,
latency, error rates, cost, and powers the /status and /v1/system/analytics
endpoints. This is the dashboard.

Splitting them makes sense once you see the access patterns. The optimizer needs
to replay individual rows for one prompt, which is a point-lookup workload that
SQLite handles fine and keeps naturally partitioned per prompt. Analytics needs
to aggregate across everything, which is the columnar workload DuckDB exists for.
One store would have been worse at one of those jobs.

Running it

Configure providers and a strategy in YAML, then start the gateway:

model_providers:
  openai:
    api_key: "sk-..."
    models: ["gpt-4o", "gpt-4", "gpt-3.5-turbo"]

gateway:
  addr: ":8080"
  strategy: "round_robin"

dataset:
  base_path: "data/prompts"

./route-switch --config config.yaml --gateway

Now point any OpenAI-compatible client at it:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Your existing SDK, your existing request shape, one changed URL. That is what
makes it cheap to trial.

How does it know a prompt got better?

This is the question I care about most, because "automatic prompt optimization"
is usually where a project waves its hands. Route-switch ships three concrete
evaluation strategies:

• Similarity, the default. Token overlap and length heuristics, default threshold 0.7. Meant for creative or long-form output.
• Exact match. Trims whitespace, binary pass or fail. For deterministic answers and unit-test-style validation.
• Keyword match. Accepts predefined keywords or derives them from expected output, scores partially on presence.

You pick per prompt, and the optimizer's tunables are real config, not magic:

mipro_v2:
  num_candidates: 5
  num_trials: 20
  num_instruction_candidates: 4
  max_bootstrapped_demos: 3
  evaluation_strategy: "Similarity"

Bayesian search over instruction candidates, scored by the strategy you chose,
using goptuna underneath. Optimization also runs continuously if you want it to:
the gateway has a background optimizer gated by OptimizationEnabled and an
OptimizationInterval.

That specificity is the part worth stealing even if you never run this gateway.
"Optimize my prompt" is meaningless without a scoring function. Naming three and
letting you choose is the honest version of the feature.

Where it does not fit

The value is backloaded. Optimization is only as good as the traces it has.
On day one there is nothing to optimize against. If your prompt runs ten times a
week, the loop may never gather enough signal to beat a human editing the file.
It pays off on high-volume prompts.

Optimization costs tokens. Read the config above literally: 20 trials, 5
candidates, 4 instruction candidates, and generating candidates means provider
calls. That is real spend on top of your inference bill, before any gain. The
quality improvement has to be multiplied across enough requests to clear it.

The default evaluator is a heuristic. Similarity scoring is token overlap and
length, not semantic judgment. For long-form output that is a rough proxy, and
you can absolutely optimize toward a prompt that scores well and reads worse.
Exact and keyword match are sharper but only apply when your task has a
checkable answer.

One advertised strategy is not implemented yet. The README lists
least-connections load balancing. In internal/gateway/load_balancer.go that
case returns round-robin, with the comment "For now, treating all as equal until
we implement connection tracking." Round-robin, weighted round-robin,
performance-based and random are real. Worth knowing before you set
strategy: least_connections and assume you got something.

It is in your critical path. Every request flows through it, and the dataset
plus analytics stores are state you now own. For a low-traffic app that just
needs multi-provider failover, a thin proxy is less to carry.

Takeaways

• Routing is the commodity. Capturing production traffic as a per-prompt dataset is the thing that makes everything downstream possible.
• Split your stores by access pattern. SQLite per prompt for replay, DuckDB for aggregate analytics.
• Never accept "automatic optimization" without asking for the scoring function. Three named evaluators beats one unnamed one.
• Read the code before you trust the feature table. The gap I found took one file.

Repo, and the docs live under documentation/docs in-tree:
https://github.com/Skelf-Research/route-switch

If you have run trace-driven prompt optimization in production, I want the number
I could not get from the code: how many captured traces before the optimized
prompt actually beat your hand-tuned one? That threshold is the whole business
case, and I have not seen anyone publish it.