Introducing QueryFlux: Multi-Engine Query Router and Universal SQL Proxy

Efficiently routing multiple query engines is a critical challenge.

QueryFlux is a universal SQL proxy and multi-engine query router written in Rust. It sits between clients and query engines. Clients connect to QueryFlux using a protocol they already know. QueryFlux routes each query to the right backend, translates SQL dialects when needed, enforces concurrency limits, and gives you a unified observability surface.

Open table formats unified the data. QueryFlux unifies the access.

If you already run more than one query engine, you know the problem is not only where data lives. The harder part is how query access works in practice.

• Which engine should this run on?
• Which client should connect where?
• How do you protect low-latency traffic from batch workloads?
• What happens when one cluster is saturated?
• How much routing logic ends up hardcoded across the stack?

That is the problem QueryFlux is built to solve.

Why QueryFlux exists

Modern data platforms are multi-engine by design.

A team may use Trino for federated queries, DuckDB for embedded analytics, StarRocks for low-latency serving, and Athena for pay-per-scan workloads on cold data. That mix is not a sign of architectural drift. In many cases, it is the right shape of the system.

Open table formats made this possible. With Apache Iceberg, Delta Lake, or Hudi, multiple engines can read the same data in object storage without duplicating it. That solved storage interoperability.

What it did not solve is compute access.

Each engine still comes with its own protocol, its own SQL dialect, its own connection handling, and its own operational behavior. Clients still need to know where to connect. Routing logic still leaks into notebooks, applications, dashboards, and team conventions. Capacity management is still fragmented across backends.

QueryFlux adds the missing layer above the table format: one access layer in front of the engine fleet.

What QueryFlux does

At a high level, QueryFlux handles three things:

• protocol ingestion
• routing
• dispatch and dialect translation

Clients connect using protocols they already speak. QueryFlux supports:

• Trino HTTP
• PostgreSQL wire
• MySQL wire
• Arrow Flight SQL
• Admin REST API

On the backend side, it already supports:

• Trino
• DuckDB
• StarRocks
• Athena

That gives it a very specific place in the stack. It is not trying to replace engines, and it is not introducing a custom client model. It is making a heterogeneous engine fleet look coherent from the access layer.

How a query flows through the system

A client connects to QueryFlux using a native protocol.

The query is evaluated against an ordered routing chain.

The first matching rule selects the cluster group that should handle the query.

From there, QueryFlux selects a healthy cluster in that group, optionally rewrites the SQL into the target dialect using sqlglot, and dispatches the query.

If the group is already at its concurrency limit, the query can queue at the proxy instead of failing immediately.

That is the important design move. QueryFlux is not just a forwarder. It is the runtime layer where access, routing, translation, and capacity handling meet.

Architecture

Client (psql / Trino CLI / mysql / BI tool)
    │
    │ native protocol
    ▼
┌─────────────────────────────────────────────┐
│                 QueryFlux                   │
│                                             │
│  Frontend ──► Router ──► Dialect translation│
│                    │                        │
│              Cluster group                  │
│         (concurrency limit + queue)         │
└──────────────────┬──────────────────────────┘
                   │
      ┌────────────┼────────────┐
      ▼            ▼            ▼
   Trino       StarRocks      Athena
      └────────────┴────────────┘
          Apache Iceberg / Delta / Hudi

The architecture is simple enough to understand quickly, but deep enough to be useful in real environments.

The simplicity is at the edge. Clients keep using the protocols they already know.

The depth is inside the routing and dispatch path, where QueryFlux can apply routing policy, translation, concurrency limits, queueing, health-aware selection, and load balancing without pushing that complexity back into every client.

Routing is where the value becomes obvious

QueryFlux evaluates each query against an ordered router chain.

Routing can be based on:

• protocol
• HTTP headers
• SQL text using regex
• client tags
• Python script logic
• compound rules
• fallback routing

That matters because real routing logic is rarely a single condition. In practice, you may want to steer PostgreSQL wire traffic to a low-latency group, send ETL-tagged traffic to a batch-oriented cluster, and use query patterns to catch common fast-path cases.

A simple example looks like this:

routes:
  - name: fast_queries
    match:
      query_regex: "SELECT .* LIMIT \\d+"
    target: duckdb_group

  - name: dashboard_queries
    match:
      protocol: mysql
    target: starrocks_group

  - name: heavy_analytics
    match:
      query_regex: "JOIN|GROUP BY|WINDOW"
    target: trino_group

  - name: fallback
    fallback: true
    target: athena_group

The point is not that every deployment should use this exact policy. The point is that the policy becomes explicit, traceable, and shared.

That alone removes a surprising amount of hidden operational drag.

Cluster groups make routing operational

Once a route resolves to a cluster group, QueryFlux handles execution there.

It supports these load-balancing strategies:

• roundRobin
• leastLoaded
• failover
• engineAffinity
• weighted

It also supports:

• per-group concurrency limits
• proxy-side queueing when groups are full
• health-aware cluster selection
• background health checks

This is where QueryFlux starts to feel deeper than a typical proxy.

It is not only deciding where a query should go. It is also giving operators a place to control how traffic behaves when systems are under load, how overflow is absorbed, and how healthy capacity is chosen.

That is the part that makes the system practical.

SQL translation is built into the path

Multi-engine routing is much more useful when SQL dialect differences do not immediately get in the way.

QueryFlux integrates dialect-only translation through sqlglot. When needed, it can rewrite SQL into the target engine’s dialect during dispatch.

That means:

• clients can keep speaking the SQL they naturally emit
• QueryFlux can normalize for the backend that will actually execute the query
• teams do not need to maintain multiple versions of the same query only because engines differ

The current design is disciplined here. What is implemented today is dialect-only translation. Schema-aware translation is explicitly on the roadmap.

That is a good balance: the system is already useful now, and the path to deeper translation is clear.

Observability is part of the product, not an add-on

A routing layer only works if operators can see what it is doing.

QueryFlux includes:

• Prometheus metrics
• Grafana dashboard
• Admin REST API
• QueryFlux Studio

The current observability surface covers:

• query counts
• query duration
• translation metrics
• running queries
• queued queries

It also supports routing traces, which matters in practice. When you introduce a routing layer, one of the first questions engineers ask is: why did this query land there? QueryFlux has a real answer to that.

Where this becomes useful quickly

The value of QueryFlux is easier to see in real scenarios than in abstract feature lists.

A multi-engine platform

• BI tools connect through one access layer
• different workloads are routed to the engines they fit best
• backend topology becomes configuration instead of client code

Dashboard SLA protection

• low-latency groups can be protected with concurrency limits
• overflow can queue or spill instead of degrading the serving path

Incremental engine migration

• weighted routing makes gradual traffic shifts possible
• clients do not need to change while the migration happens

Mixed workloads on shared data

• batch, interactive, and exploratory traffic can be separated by policy
• routing intent lives in one place instead of being spread across the stack

These are practical benefits. They show up immediately once a platform becomes multi-engine.

Getting started is intentionally simple

One of the nice things about the project is that the first run experience is straightforward.

A minimal setup looks like this:

git clone https://github.com/lakeops-org/queryflux.git
cd queryflux/examples/minimal-trino
docker compose up -d --wait

That gives you:

• QueryFlux on http://localhost:8080
• Trino direct on http://localhost:8081
• Admin API on http://localhost:9000
• Studio on http://localhost:3000
• Postgres on localhost:5433

You can then send a simple query through the Trino HTTP frontend:

curl -X POST http://localhost:8080/v1/statement \
  -H "X-Trino-User: dev" \
  -d "SELECT 42"

There are also examples for:

• a minimal in-memory setup
• a Prometheus + Grafana stack
• a full stack with Trino, StarRocks, and Iceberg-related services

That combination is important. The system is conceptually ambitious, but the on-ramp is short.

It feels like deep infrastructure without feeling heavy to try.

What is already shipped

QueryFlux already includes a substantial set of capabilities on the main branch:

• Trino HTTP frontend
• PostgreSQL wire frontend
• MySQL wire frontend
• Arrow Flight SQL frontend
• Admin REST API
• Trino backend
• DuckDB backend
• StarRocks backend
• Athena backend
• ordered router chains and routing fallback
• route tracing support
• per-group concurrency limits
• proxy-side queueing
• multiple load-balancing strategies
• health-aware cluster selection
• dialect-only translation through sqlglot
• in-memory persistence
• PostgreSQL persistence
• authentication providers including none, static, OIDC, and LDAP
• authorization modes including allow-all, simple policy, and OpenFGA
• Prometheus metrics
• Grafana dashboard
• QueryFlux Studio
• dynamic config reload from Postgres

This matters because the project already feels like infrastructure, not just an idea.

What comes next

The roadmap extends the same core design.

Near-term work includes:

• schema-aware SQL translation
• ClickHouse backend and HTTP frontend
• richer routing telemetry in Studio

Medium-term work includes:

• cost- and performance-aware routing
• Snowflake backend
• BigQuery backend
• Redis persistence

That roadmap makes sense. It deepens the same access layer instead of changing the project’s center of gravity.

Solving the data ccess side

The most interesting thing about QueryFlux is not that it is a proxy.

It is that it is a carefully placed layer in a part of the modern data stack that is still surprisingly underbuilt.

Open table formats solved the data side.

QueryFlux is solving the access side.

That creates an appealing combination:

• conceptually clean architecture
• obvious operational benefits
• room for sophisticated policy and routing logic
• low-friction adoption path

It feels like the kind of infrastructure that becomes more valuable as the rest of the stack becomes more heterogeneous.

Getting started

Once a data platform becomes multi-engine, the missing piece is usually not another engine.

It is the access layer.

Clients still need to know where to connect. Routing still leaks into tools and applications. SQL dialect differences still show up at the edges. Capacity handling is still fragmented.

QueryFlux gives that layer a shape.

It makes multi-engine access easier to reason about, easier to operate, and easier to evolve.

That is why it is a compelling project: the idea is deep, the benefits are immediate, and the first experience is simple.

To try it out visit: https://queryflux.dev/