DEV Community: Borivoj Grujicic

Elusion v8.3.0 is out!

Borivoj Grujicic — Sun, 05 Apr 2026 19:25:38 +0000

Data Engineering Library - Elusion -, now has a built-in Medallion Architecture pipeline framework (Bronze / Silver / Gold) for building production data pipelines in pure Rust.
No Python. No dbt. No Airflow.
✅ DAG-based execution with parallel processing
✅ Auto materialization to Parquet or Delta per layer
✅ Microsoft Fabric / OneLake ready
✅ Config-driven — elusion.toml + connections.toml
✅ One file per model, clean separation of layers
Single binary. Docker ready. Compile and ship.

👇 Download Starter Template Project from the link bellow! 👇

🔗 Crates.io
🔗 GitHub Reporistory
🚀 Starter template:

Elusion v8.0.0 is the best END-TO-END Data Engineering library writen in RUST

Borivoj Grujicic — Tue, 21 Oct 2025 14:53:51 +0000

Elusion v8.0.0 just dropped with something I'm genuinely excited about: native SQL execution and CopyData feature.

Functional API still going strong:

Write queries however you want - Unlike SQL, PySpark, or Polars, you can chain operations in ANY order. No more "wait, does filter go before group_by or after?" Just write what makes sense:

use elusion::prelude::*;
#[tokio::main]
async fn main() -> ElusionResult<()> {
    let sales = CustomDataFrame::new("sales.csv", "sales").await?;

    let result = sales
        .select(["customer_id", "amount", "order_date"])
        .filter("amount > 1000")
        .agg(["SUM(amount) AS total", "COUNT(*) AS orders"])
        .group_by(["customer_id"])
        .having("total > 50000")
        .order_by(["total"], ["DESC"])
        .limit(10)
        .elusion("top_customers")
        .await?;

    result.display().await?;
    Ok(())
}

Raw SQL when you need it - Sometimes you just want to write SQL. Now you can:

There is small macro sql! to simplify usage and avpid using &[&df] for each Dataframe included in query.

use elusion::prelude::*;
#[tokio::main]
async fn main() -> ElusionResult<()> {
    let sales = CustomDataFrame::new("sales.csv", "sales").await?;
    let customers = CustomDataFrame::new("customers.csv", "customers").await?;
    let products = CustomDataFrame::new("products.csv", "products").await?;

    let result = sql!(
        r#"
        WITH monthly_totals AS (
            SELECT 
                DATE_TRUNC('month', s.order_date) as month,
                c.region,
                p.category,
                SUM(s.amount) as total
            FROM sales s
            JOIN customers c ON s.customer_id = c.id
            JOIN products p ON s.product_id = p.id
            GROUP BY month, c.region, p.category
        )
        SELECT 
            month,
            region,
            category,
            total,
            SUM(total) OVER (
                PARTITION BY region, category 
                ORDER BY month
            ) as running_total
        FROM monthly_totals
        ORDER BY month DESC, total DESC
        LIMIT 100
        "#,
        "monthly_analysis",
        sales,
        customers,
        products
    ).await?;

    result.display().await?;
    Ok(())
}

COPY DATA:
Now you can read and write between files in true streaming fashion:

You can do it in 2 ways: 1. Custom Configuration, 2. Simplified file conversion

// Custom Configuration
copy_data(  
    CopySource::File {
        path: "C:\\Borivoj\\RUST\\Elusion\\bigdata\\test.json",
        csv_delimiter: None,
    },
    CopyDestination::File {  
        path: "C:\\Borivoj\\RUST\\Elusion\\CopyData\\test.csv",
    },
    Some(CopyConfig {
            batch_size: 500_000, 
            compression: None,
            csv_delimiter: Some(b','), 
            infer_schema: true,  
            output_format: OutputFormat::Csv,
    }),
).await?;

// Simplified file conversion
copy_file_to_parquet(
    "input.json",
    "output.parquet",
    Some(ParquetCompression::Uncompressed), // or Snappy
).await?;

If you hear for Elusion for the first time bellow are some core features:

🏢 Microsoft Fabric - OneLake connectivity

☁️ AzureAzure BLOB storage connectivity

📁 SharePoint connectivity

📡 FTP/FTPS connectivity

📊 Excel file operations

🐘 PostgreSQL database connectivity

🐬 MySQLMySQL database connectivity

🌐 HTTP API integration

📈 Dashboard Data visualization

⚡ CopyData High-performance streaming operations

Built-in formats: CSV, JSON, Parquet, Delta Lake, XML, EXCEL

Plus:

Redis caching + in-memory query cache

Pipeline scheduling with tokio-cron-scheduler

Materialized views

To learn more about the crate, visit: https://github.com/DataBora/elusion

Elusion Celebrates 50K+ Downloads: A Modern Alternative to Pandas and Polars for Data Engineering

Borivoj Grujicic — Wed, 17 Sep 2025 08:53:36 +0000

The Rust data ecosystem has reached another significant milestone with Elusion DataFrame Library surpassing 50,000 downloads on crates.io. As data engineers and analysts, that love SQL syntax, continue seeking alternatives to Pandas and Polars, Elusion has emerged as a compelling option that combines the familiarity of DataFrame operations with unique capabilities that set it apart from the competition.

What Makes Elusion Different

While Pandas and Polars excel in their respective domains, Elusion brings several distinctive features that address gaps in the current data processing landscape:

1. Native Multi-Format File Support Including XML

While Pandas and Polars support common formats like CSV, Excel, Parquet, and JSON, Elusion goes further by offering native XML parsing capabilities. Unlike Pandas and Polars, which require external libraries and manual parsing logic for XML files, Elusion automatically analyzes XML file structure and chooses the optimal processing strategy:

`// XML files work just like any other format

let xml_path = "C:\path\to\sales.xml";

let df = CustomDataFrame::new(xml_path, "xml_data").await?;`

2. Flexible Query Construction Without Strict Ordering

Unlike DataFrame libraries that enforce specific operation sequences, Elusion allows you to build queries in ANY order that makes sense to your logic. Whether you want to filter before selecting, or aggregate before grouping, Elusion ensures consistent results regardless of function call order.

`// Write operations in the order that makes sense to you

sales_df

.filter("amount > 1000")

.join(customers_df, ["s.CustomerKey = c.CustomerKey"], "INNER")

.select(["c.name", "s.amount"])

.agg(["SUM(s.amount) AS total"])

.group_by(["c.region"])

Same result is achieved with different function order:
`sales_df

.join(customers_df, ["s.CustomerKey = c.CustomerKey"], "INNER")

.select(["c.name", "s.amount"])

.agg(["SUM(s.amount) AS total"])

.group_by(["c.region"])

.filter("amount > 1000")`

2. Built-in External Data Source Integration

While Pandas and Polars require additional libraries for cloud storage and database connectivity, Elusion provides native support for:

Azure Blob Storage with SAS token authentication
SharePoint integration for enterprise environments
PostgreSQL and MySQL database connections
REST API data ingestion with customizable headers and pagination
Multi-format file loading from folders with automatic schema merging

3. Advanced Caching Architecture

Elusion offers sophisticated caching capabilities that go beyond what's available in Pandas or Polars:

Native caching for local development and single-instance applications

Redis caching for distributed systems and production environments

Materialized views with TTL management

Query result caching with automatic invalidation

4. Production-Ready Pipeline Scheduling

Unlike Pandas and Polars which focus primarily on data manipulation, Elusion includes a built-in pipeline scheduler for automated data engineering workflows:

`let scheduler = PipelineScheduler::new("5min", || async {

// Your data pipeline logic here

let df = CustomDataFrame::from_azure_with_sas_token(url, token, None, "data").await?;

df.select(["*"]).write_to_parquet("overwrite", "output.parquet", None).await?;

Ok(())

}).await?;`

5. Interactive Dashboard Generation

While Pandas requires additional libraries like Plotly or Matplotlib for visualization, Elusion includes built-in interactive dashboard creation:

Generate HTML reports with interactive plots (TimeSeries, Bar, Pie, Scatter, etc.)

Create paginated, filterable tables with export capabilities

Combine multiple visualizations in customizable layouts

No additional dependencies required

6. Streaming Processing Capabilities

Elusion provides streaming processing options for handling large datasets for better performance while reading and writing data:

`// Stream processing for large files

big_file_df

.select(["column1", "column2"])

.filter("value > threshold")

.elusion_streaming("results").await?;

// Stream writing directly to files

df.elusion_streaming_write("data", "output.parquet", "overwrite").await?;`

7. Advanced JSON Handling

Elusion offers specialized JSON functions for columns with json values, that simplify working with complex nested structures:

Extract values from JSON arrays with pattern matching
Handle multiple JSON formats automatically
Convert REST API responses to JSON files than to DataFrames

`let path = "C:\RUST\Elusion\jsonFile.csv";

let json_df = CustomDataFrame::new(path, "j").await?;

let df_extracted = json_df.json([

"ColumnName.'$Key1' AS column_name_1",

"ColumnName.'$Key2' AS column_name_2",

"ColumnName.'$Key3' AS column_name_3"

])

.select(["some_column1", "some_column2"])

.elusion("json_extract").await?;`

Performance and Memory Management

Elusion is built on Apache Arrow and DataFusion, providing:

Memory-efficient operations through columnar storage
Redis caching for optimized query execution
Automatic schema inference across multiple file formats
Parallel processing capabilities through Rust's concurrency model

`let sales = "C:\RUST\Elusion\SalesData2022.csv";

let products = "C:\RUST\Elusion\Products.csv";

let customers = "C:\RUST\Elusion\Customers.csv";

let sales_df = CustomDataFrame::new(sales, "s").await?;

let customers_df = CustomDataFrame::new(customers, "c").await?;

let products_df = CustomDataFrame::new(products, "p").await?;

// Connect to Redis (requires Redis server running)

let redis_conn = CustomDataFrame::create_redis_cache_connection().await?;

// Use Redis caching for high-performance distributed caching

let redis_cached_result = sales_df

.join_many([

    (customers_df, ["s.CustomerKey = c.CustomerKey"], "RIGHT"),

    (products_df, ["s.ProductKey = p.ProductKey"], "LEFT OUTER"),

])

.select(["c.CustomerKey", "c.FirstName", "c.LastName", "p.ProductName"])

.agg([

    "SUM(s.OrderQuantity) AS total_quantity",

    "AVG(s.OrderQuantity) AS avg_quantity"

])

.group_by(["c.CustomerKey", "c.FirstName", "c.LastName", "p.ProductName"])

.having_many([

    ("total_quantity > 10"),

    ("avg_quantity < 100")

])

.order_by_many([

    ("total_quantity", "ASC"),

    ("p.ProductName", "DESC")

])

.elusion_with_redis_cache(&redis_conn, "sales_join_redis", Some(3600)) // Redis caching with 1-hour TTL

.await?;

redis_cached_result.display().await?;`

Getting Started with Elusion: Easier Than You Think

For SQL Developers

If you write SQL queries, you already have 80% of the skills needed for Elusion. The mental model is identical - you're just expressing the same logical operations in Rust syntax:

`// Your SQL thinking translates directly:

df.select(["customer_name", "order_total"]) // SELECT

.join(customers, ["id = customer_id"], "INNER") // JOIN

.filter("order_total > 1000") // WHERE

.group_by(["customer_name"]) // GROUP BY

.agg(["SUM(order_total) AS total"]) // Aggregation

.order_by(["total"], ["DESC"]) // ORDER BY
`

For Python/Pandas Users

Elusion feels familiar if you're coming from Pandas:

`sales_df

.join_many([

    (customers_df, ["s.CustomerKey = c.CustomerKey"], "INNER"),

    (products_df, ["s.ProductKey = p.ProductKey"], "INNER"),

])

.select(["c.name", "p.category", "s.amount"])

.filter("s.amount > 1000")

.agg(["SUM(s.amount) AS total_revenue"])

.group_by(["c.region", "p.category"]) 

.order_by(["total_revenue"], ["DESC"])

.elusion("quarterly_report")

.await?`

Installation and Setup

Adding Elusion to your Rust project takes just two lines:

`[dependencies]

elusion = "6.2.0"

tokio = { version = "1.45.0", features = ["rt-multi-thread"] }
`

Enable only the features you need to keep dependencies minimal:

Start simple, add features as needed

elusion = { version = "6.2.0", features = ["postgres", "azure"] }

Then, your first Elusion program would look like this:
`use elusion::prelude::*;

[tokio::main]

async fn main() -> ElusionResult<()> {

// Load any file format - CSV, Excel, JSON, XML, Parquet

let df = CustomDataFrame::new("data.csv", "sales").await?;

// Write operations that make sense to you

let result = df

    .select(["customer", "amount"])

    .filter("amount > 100")

    .agg(["SUM(amount) AS total"])

    .group_by(["customer"])

    .elusion("analysis").await?;

result.display().await?;

Ok(())

}
`

Perfect for SQL Developers and Python Users Ready to Embrace Rust

If you know SQL, you already understand most of Elusion's power. The library's approach mirrors SQL's flexibility - you can write operations in the order that makes logical sense to you, just like constructing SQL queries. Consider this familiar pattern:

SQL Query:

`SELECT c.name, SUM(s.amount) as total

FROM sales s

JOIN customers c ON s.customer_id = c.id

WHERE s.amount > 1000

GROUP BY c.name

ORDER BY total DESC;`

Elusion equivalent:

`sales_df

.join(customers_df, ["s.customer_id = c.id"], "INNER")

.select(["c.name"])

.agg(["SUM(s.amount) AS total"])

.filter("s.amount > 1000")

.group_by(["c.name"])

.order_by(["total"], ["DESC"])`

The 50,000 download milestone reflects growing recognition that modern data processing needs tools designed for today's distributed, cloud-native environments. SQL developers and Python users that are discovering that Rust doesn't have to mean starting from scratch - it can mean taking your existing knowledge and supercharging it.

Rust DataFrame Alternatives to Polars: Meet Elusion v4.0.0

Borivoj Grujicic — Mon, 11 Aug 2025 04:44:48 +0000

The Rust ecosystem has seen tremendous growth in data processing libraries, with Polars leading the charge as a blazingly fast DataFrame library.

However, a new contender has emerged that takes a fundamentally different approach to data engineering and analysis: Elusion.

While Polars focuses on pure performance and memory efficiency with its Apache Arrow-based columnar engine, Elusion positions itself as equaly dedicated for performance and memory efficiency, also with Appache Arrow and DataFusion, as well as a comprehensive data engineering platform that prioritizes flexibility, ease of use, and integration capabilities alongside high performance.

Architecture Philosophy: Different Approaches to the Same Goals

Polars: Performance-First Design

Polars is written from scratch in Rust, designed close to the machine and without external dependencies. It's based on Apache Arrow's memory model, providing very cache efficient columnar data structures and focuses on:

Ultra-fast query execution with SIMD optimizations
Memory-efficient columnar processing
Lazy evaluation with query optimization
Streaming for out-of-core processing

Elusion: Flexibility-First Design

Elusion takes a different approach, prioritizing developer experience and integration capabilities:

Core Philosophy: "Elusion wants you to be you!"

Unlike traditional DataFrame libraries that enforce specific patterns, Elusion offers flexibility in constructing queries without enforcing specific patterns or chaining orders. You can build your queries in ANY SEQUENCE that makes sense to you, writing functions in ANY ORDER, and Elusion ensures consistent results regardless of the function call order.

Loading files into DataFrames:

Regular Loading: ~4.95 seconds for complex queries on 900k rows

CustomDataFrame::new()

Streaming Loading: ~3.62 seconds for the same operations

CustomDataFrame::new_with_stream()

Performance improvement: 26.9% faster with streaming approach

Polars approach:

let df = LazyFrame::scan_csv("data.csv", ScanArgsCSV::default())?
    .filter(col("amount").gt(100))
    .select([col("customer"), col("amount")])
    .collect()?;

Elusion approach - flexible ordering:

let df = CustomDataFrame::new("data.csv", "sales").await?
    .filter("amount > 100")           
    .select(["customer", "amount"]) 
    .elusion("result").await?;

// Or reorder as you find fit - same result
let df = CustomDataFrame::new("data.csv", "sales").await?
    .select(["customer", "amount"])   // Select first
    .filter("amount > 100")           // Filter second
    .elusion("result").await?;

Polars Basic file loading:

let df = LazyFrame::scan_csv("data.csv", ScanArgsCSV::default())?
    .collect()?;

// Parquet with options
let df = LazyFrame::scan_parquet("data.parquet", ScanArgsParquet::default())?
    .collect()?;

Elusion Data Loading - Comprehensive Sources:

use elusion::prelude::*;

Local files with auto-recognition

let df = CustomDataFrame::new("data.csv", "sales").await?;
let df = CustomDataFrame::new("data.xlsx", "sales").await?;  // Excel support
let df = CustomDataFrame::new("data.parquet", "sales").await?;

Streaming for large files (currently only supports .csv files)

let df = CustomDataFrame::new_with_stream("large_data.csv", "sales").await?;

Load entire folders

let df = CustomDataFrame::load_folder(
    "/path/to/folder",
    Some(vec!["csv", "xlsx"]), // Filter file types or `None` for all types
    "combined_data"
).await?;

Azure Blob Storage (currently supports csv and json files)

let df = CustomDataFrame::from_azure_with_sas_token(
    "https://account.blob.core.windows.net/container",
    "sas_token",
    Some("folder/file.csv"), //or keep `None` to take everything from folder
    "azure_data"
).await?;

SharePoint

let df = CustomDataFrame::load_from_sharepoint(
    "tenant-id",
    "client-id", 
    "https://company.sharepoint.com/sites/Site",
    "Documents/data.xlsx",
    "sharepoint_data"
).await?;

REST API to DataFrame

let api = ElusionApi::new();

api.from_api_with_headers(
    "https://api.example.com/data",
    headers,
    "/path/to/output.json"
).await?;

let df = CustomDataFrame::new("/path/to/output.json", "api_data").await?;

Database connections

let postgres_df = CustomDataFrame::from_postgres(&conn, query, "pg_data").await?;

let mysql_df = CustomDataFrame::from_mysql(&conn, query, "mysql_data").await?;

Polars: Structured Approach

Polars requires logical ordering

let result = df
    .lazy()
    .filter(col("amount").gt(100))
    .group_by([col("category")])
    .agg([col("amount").sum().alias("total")])
    .sort("total", SortMultipleOptions::default())
    .collect()?;

Elusion: Any-Order Flexibility

All of these produce the same result:

Traditional order:

let result1 = df
    .select(["category", "amount"])
    .filter("amount > 100")
    .agg(["SUM(amount) as total"])
    .group_by(["category"])
    .order_by(["total"], ["DESC"])
    .elusion("result").await?;

Filter first

let result2 = df
    .filter("amount > 100")
    .agg(["SUM(amount) as total"])
    .select(["category", "amount"])
    .group_by(["category"])
    .order_by(["total"], ["DESC"])
    .elusion("result").await?;

Aggregation first

let result3 = df
    .agg(["SUM(amount) as total"])
    .filter("amount > 100")
    .group_by(["category"])
    .select(["category", "amount"])
    .order_by(["total"], ["DESC"])
    .elusion("result").await?;

All produce identical results!

Advanced Features: Where Elusion Shines

Built-in Visualization and Reporting Create interactive dashboards

let plots = [
    (&line_plot, "Sales Timeline"),
    (&bar_chart, "Category Performance"),
    (&histogram, "Distribution Analysis"),
];

let tables = [
    (&summary_table, "Summary Stats"),
    (&detail_table, "Transaction Details")
];

CustomDataFrame::create_report(
    Some(&plots),
    Some(&tables),
    "Sales Analysis Dashboard",
    "dashboard.html",
    Some(layout_config),
    Some(table_options)
).await?;

Automated Pipeline Scheduling Schedule data engineering pipelines

let scheduler = PipelineScheduler::new("5min", || async {
    // Load from Azure
    let df = CustomDataFrame::from_azure_with_sas_token(
        azure_url, sas_token, Some("folder/"), "raw_data"
    ).await?;

    // Process data
    let processed = df
        .select(["date", "amount", "category"])
        .agg(["SUM(amount) as total", "COUNT(*) as transactions"])
        .group_by(["date", "category"])
        .order_by(["date"], ["ASC"])
        .elusion("processed").await?;

    // Write results
    processed.write_to_parquet(
        "overwrite",
        "output/processed_data.parquet",
        None
    ).await?;

    Ok(())
}).await?;

Advanced JSON Processing
Can handle complex JSON structures with Arrays and Objects

let df = CustomDataFrame::new("complex_data.json", "json_data").await?;

If you have json fields/columns in your files you can explode them:

Extract simple JSON fields:

let simple = df.json([
    "metadata.'$timestamp' AS event_time",
    "metadata.'$user_id' AS user",
    "data.'$amount' AS transaction_amount"
]);

Extract from JSON arrays:

let complex = df.json_array([
    "events.'$value:id=purchase' AS purchase_amount",
    "events.'$timestamp:id=login' AS login_time",
    "events.'$status:type=payment' AS payment_status"
]);

When to Choose Which

Choose Polars When:
Pure performance is the top priority
You prefer structured, optimized query patterns
Memory efficiency is critical
You need minimal dependencies
Choose Elusion When:
You need integration flexibility (cloud storage, APIs, databases)
Developer experience and query flexibility matter
You want built-in visualization and reporting
You need automated pipeline scheduling
Working with diverse data sources (Excel, SharePoint, REST APIs)
You prefer intuitive, any-order query building

Installation and Getting Started

Polars

[dependencies]
polars = { version = "0.50.0", features = ["lazy"] }

Elusion [dependencies]

elusion = "4.0.0"
tokio = { version = "1.45.0", features = ["rt-multi-thread"] }

Elusion With specific features

elusion = { version = "4.0.0", features = ["dashboard", "azure", "postgres"] }

Rust version requirement:

Polars: >= 1.80
Elusion: >= 1.81

Real-World Example: Sales Data Analysis
Polars Implementation:

use polars::prelude::*;

let df = LazyFrame::scan_csv("sales.csv", ScanArgsCSV::default())?
    .filter(col("amount").gt(100))
    .group_by([col("category")])
    .agg([
        col("amount").sum().alias("total_sales"),
        col("amount").mean().alias("avg_sale"),
        col("customer_id").n_unique().alias("unique_customers")
    ])
    .sort("total_sales", SortMultipleOptions::default().with_order_descending(true))
    .collect()?;

println!("{}", df);

Elusion Implementation:
use elusion::prelude::*;

#[tokio::main]
async fn main() -> ElusionResult<()> {
    // Load data (flexible source)
    let df = CustomDataFrame::new("sales.csv", "sales").await?;

    // Build query in any order that makes sense to you
    let analysis = df
        .filter("amount > 100")                
        .agg([                                    
            "SUM(amount) as total_sales",
            "AVG(amount) as avg_sale", 
            "COUNT(DISTINCT customer_id) as unique_customers"
        ])
        .group_by(["category"])                
        .order_by(["total_sales"], ["DESC"])       
        .elusion("sales_analysis").await?;

    // If you like to display result
       analysis.display().await?;

    // Create visualization
    let bar_chart = analysis.plot_bar(
        "category",
        "total_sales", 
        Some("Sales by Category")
    ).await?;

    // Generate report
    CustomDataFrame::create_report(
        Some(&[(&bar_chart, "Sales Performance")]),
        Some(&[(&analysis, "Summary Table")]),
        "Sales Analysis Report",
        "sales_report.html",
        None,
        None
    ).await?;

    Ok(())
}

Conclusion

Elusion v4.0.0 represents a paradigm shift in DataFrame libraries, prioritizing developer experience, integration flexibility, and comprehensive data engineering capabilities.
The choice between Polars and Elusion depends on your priorities:

For raw computational performance and memory efficiency: Polars
For comprehensive data engineering with flexible development: Elusion

Elusion's "any-order" query building, extensive integration capabilities, built-in visualization, and automated scheduling make it particularly attractive for teams that need to work with diverse data sources and want a more intuitive development experience.
Both libraries showcase the power of Rust in the data processing space, offering developers high-performance alternatives to traditional Python-based solutions. The Rust DataFrame ecosystem is thriving, and having multiple approaches ensures that different use cases and preferences are well-served.

Try Elusion v4.0.0 today:

cargo add elusion@4.0.0

For more information and examples, visit the Elusion github repository: Elusion repository and join the growing community of Rust data engineers who are discovering the flexibility and power of any-order DataFrame operations.