Tried apache/datafusion Today: A Fast SQL Engine for AI Data Workloads

Apache DataFusion is an extensible query engine written in Rust. It provides a SQL and DataFrame API for querying CSV, Parquet, JSON, and custom data sources without requiring a full database server. The project is currently gaining attention, with +7 GitHub stars today, likely reflecting continued interest in lightweight analytics infrastructure for data and AI applications.

Why It Matters

DataFusion is useful when an application needs embedded SQL, columnar execution, or custom query planning. It is especially attractive for AI pipelines that need to filter and aggregate large document, embedding, or evaluation datasets before sending selected context to a language model.

The architecture is also modular: developers can add custom catalogs, functions, data sources, and execution plans. That makes it a practical foundation for retrieval evaluation, agent analytics, and local data processing.

Quick Test-Drive Architecture

A simple pattern is to use DataFusion for local analytics and route the model request through an OpenAI-compatible gateway:

import os
from openai import OpenAI

# DataFusion can prepare structured context before this request.
client = OpenAI(
    api_key=os.environ["BLOST_API_KEY"],
    base_url="https://b-lost.com/v1",
)

response = client.chat.completions.create(
    model="claude-fable-5",
    messages=[
        {
            "role": "system",
            "content": "Summarize the SQL query results accurately."
        },
        {
            "role": "user",
            "content": "Query results: ... "
        }
    ],
)

print(response.choices[0].message.content)

For larger prompts, B-Lost’s native Anthropic /v1/messages support and Prompt Caching can be relevant: cache hits receive a stated 90% discount, which may reduce repeated-context costs in evaluation or agent workflows. The relay also advertises 20% off official list pricing, although actual spend still depends on model usage and token volume.

Performance Takeaway

DataFusion is not a replacement for every warehouse, but it is compelling for embedded analytics and Rust-based systems. My evaluation checklist would include query latency, Parquet scan throughput, memory usage, and the cost of downstream model calls. The strongest combination is predictable local SQL execution plus a configurable AI gateway, keeping data preparation fast and model spending measurable.