DEV Community: Mukhtar

Meet Surveilr: The SQL-First Platform That Turns Any Resource Into Compliance Gold

Mukhtar — Tue, 26 Aug 2025 10:59:46 +0000

Part 1 of 6 in the "Surveilr Deep Dive" series

Picture this: It's Monday morning, and your organization is facing a compliance audit. The auditors want to know about every file that contains sensitive data, every email with external attachments, every system configuration change in the last quarter. Your team scrambles across dozens of tools, exports from multiple systems, and spends weeks manually correlating data that should be instantly queryable.

What if I told you there's a tool that turns ANY file, email, or system resource into queryable SQL data in minutes? Meet Surveilr – the open-source platform that's revolutionizing how organizations approach compliance, security monitoring, and resource surveillance.

The Compliance Data Problem

Before we dive into the solution, let's acknowledge the problem. Modern organizations generate digital evidence across countless systems:

Files scattered across file systems with varying permissions and metadata
Emails buried in different mailboxes with attachments and complex threading
Project management data spread across GitHub, Jira, GitLab
System configurations and logs in proprietary formats
Network resources and databases with their own query languages

When audit time comes, teams spend 80% of their effort just finding and correlating data, leaving only 20% for actual analysis. This is backwards.

Enter Surveilr: The SQL-First Revolution

Surveilr flips this equation. Its core philosophy is elegantly simple:

Walk resources. Capture everything. Query anything.

Instead of learning dozens of tools and APIs, you learn SQL once and query everything. Instead of fragmented data silos, you get unified Resource Surveillance State Databases (RSSDs) that contain all your organizational evidence in a single, queryable format.

Here's what makes Surveilr different:

🎯 SQL-First Architecture

Every piece of data – files, emails, system info, project data – becomes rows and columns in SQLite databases. If you know SQL, you can analyze anything.

🏠 Local-First, Edge-Based

Your data stays on your infrastructure. Surveilr creates local SQLite databases that can be queried without internet connectivity or cloud dependencies.

🔓 Tool-Independent Results

Once created, RSSDs are standard SQLite files. You can query them with any SQL tool, language, or platform. Surveilr doesn't hold your data hostage.

🧩 Universal Resource Processing

Files, emails, APIs, executables, documents – if it exists on your systems, Surveilr can ingest it and make it queryable.

The RSSD: Your Universal Evidence Database

At the heart of Surveilr is the Resource Surveillance State Database (RSSD). Think of it as a universal translator that converts any resource into structured, queryable data.

Here's the magic: every RSSD contains a standard schema with tables like:

uniform_resource – Every file, email, or resource becomes a row
device – Information about the systems being surveilled
ur_ingest_session – Audit trail of when and how data was collected
uniform_resource_transform – Processed and enriched data
Plus dozens more for specialized data types

The beauty is consistency. Whether you're analyzing a Windows server, a Linux container, or a MacBook, the RSSD schema remains the same. Your SQL queries work everywhere.

Your First Surveilr Experience

Let's get hands-on. I'll walk you through creating your first RSSD and running your first surveillance queries.

Installation

Surveilr provides a one-liner installation script:

curl -sL https://raw.githubusercontent.com/opsfolio/releases.opsfolio.com/main/surveilr/install.sh | sh

Or if you prefer a custom installation path:

SURVEILR_HOME="$HOME/bin" curl -sL https://raw.githubusercontent.com/opsfolio/releases.opsfolio.com/main/surveilr/install.sh | sh

Verify your installation:

surveilr --version
surveilr --help

Your First Ingestion

Let's start simple – surveilling your current directory:

# See what would be ingested (dry run)
surveilr ingest files --dry-run

# Actually ingest the files
surveilr ingest files

# Check what we created
ls -la resource-surveillance.sqlite.db

That's it! You've just created your first RSSD. Surveilr walked through your current directory, captured metadata about every file, and stored it all in a SQLite database.

Your First Queries

Now comes the magic. Let's query our surveillance data:

# Connect to the database
sqlite3 resource-surveillance.sqlite.db

# See what tables were created
.tables

# Look at the uniform_resource table structure
.schema uniform_resource

# Find all files modified in the last 7 days
SELECT 
    uri,
    size_bytes,
    datetime(last_modified_at, 'unixepoch') as modified_date
FROM uniform_resource 
WHERE last_modified_at > strftime('%s', 'now', '-7 days')
ORDER BY last_modified_at DESC;

# Find the largest files
SELECT 
    uri,
    size_bytes,
    ROUND(size_bytes/1024.0/1024.0, 2) as size_mb
FROM uniform_resource 
WHERE size_bytes IS NOT NULL
ORDER BY size_bytes DESC 
LIMIT 10;

# Look for configuration files
SELECT uri, nature, size_bytes
FROM uniform_resource 
WHERE uri LIKE '%.conf%' 
   OR uri LIKE '%.config%' 
   OR uri LIKE '%.json%'
   OR uri LIKE '%.yaml%'
   OR uri LIKE '%.yml%';

What Just Happened?

In less than 5 minutes, you:

Installed a powerful surveillance platform
Ingested file system metadata from your current directory
Queried that data using standard SQL
Generated insights about file modifications, sizes, and types

But this is just the beginning. That RSSD you just created is a standard SQLite database. You can:

Query it from any programming language with SQLite support
Import it into business intelligence tools
Share it with team members
Merge it with other RSSDs
Archive it for compliance purposes

The Architecture Behind the Magic

Before we wrap up this introduction, let's peek under the hood. Surveilr's architecture is elegantly modular:

Core Components

surveilr ingest – The data collection engine that walks resources and populates RSSDs
surveilr admin – Administrative commands for managing and merging databases
surveilr notebooks – Built-in SQL notebooks for common queries and reports
surveilr shell – Interactive SQL interface with enhanced capabilities

Resource Types

Surveilr doesn't just handle files. It can ingest:

File systems with full metadata and content analysis
Email systems via IMAP (Gmail, Outlook, Microsoft 365)
Project management platforms (GitHub, GitLab, Jira, OpenProject)
Executable scripts that generate dynamic data
Cloud storage systems via OpenDAL integration
Custom data sources through capturable executables

SQL-Centric Philosophy

Everything in Surveilr revolves around SQL:

Configuration is stored in SQL tables
Business logic is implemented as SQL queries
Reports are SQL result sets
Extensions are SQL functions and procedures
Migrations are SQL scripts

This means you can understand, modify, and extend Surveilr using the same SQL knowledge you already have.

What's Coming Next in This Series

We've just scratched the surface. Over the next five articles, we'll build on this foundation:

Article 2: "From Chaos to Queries" – We'll expand our RSSD with email ingestion, project management data, and capturable executables. You'll see how multiple data sources become a unified, queryable dataset.

Article 3: "Talk to Your Data" – We'll add AI-powered natural language querying, turning your RSSD into a conversational interface. Ask questions in plain English, get SQL results.

Article 4: "Zero-Code Dashboards" – We'll build web-based compliance dashboards using SQLPage, creating visual interfaces that non-technical stakeholders can use.

Article 5: "Breaking SQLite Barriers" – We'll enable remote team access using Surveilr's PostgreSQL wire protocol implementation, making your local RSSD accessible to distributed teams.

Article 6: "Enterprise Mastery" – We'll explore advanced features like document processing, data deidentification, and building compliance automation workflows.

Try It Yourself

The best way to understand Surveilr is to use it. Here are some exercises to try before the next article:

Install Surveilr and create your first RSSD
Experiment with different directories – try surveilling /etc (on Unix systems) or your Documents folder
Explore the schema – run .schema in sqlite3 to see all available tables
Write custom queries – look for files by type, size, or modification date
Check out the help – run surveilr ingest files --help to see advanced options

The SQL-First Future

Surveilr represents a fundamental shift in how we think about organizational data. Instead of learning new tools for each data source, we leverage the universal language of SQL. Instead of fragmented compliance efforts, we build unified evidence databases. Instead of vendor lock-in, we create portable, standards-based data assets.

In a world where data is the new oil, Surveilr is the refinery that turns raw organizational resources into compliance gold.

Ready to dive deeper? In the next article, we'll expand our surveillance capabilities beyond simple files to include emails, project management data, and dynamic executable content. Your RSSD is about to become a lot more interesting.

Coming up in Article 2: "From Chaos to Queries: Ingesting Files, Emails, and Everything Else"

Implementing OpenDAL with Filesystem (FS) In Rust

Mukhtar — Sat, 19 Jul 2025 18:55:47 +0000

Introduction to OpenDAL with SQLite Virtual Tables

OpenDAL is a powerful and unified data access layer that provides an abstraction for different storage backends such as local filesystems, cloud storage, and object stores. It simplifies file and metadata operations by offering a unified API, allowing seamless interaction with different storage solutions.

This guide explains the concepts behind integrating OpenDAL with SQLite virtual tables, allowing you to query filesystem metadata using SQL. The code examples demonstrate key concepts rather than complete implementations.

Core Concepts

1. OpenDAL Operator

The foundation of any OpenDAL integration is the Operator - your interface to the storage backend.

Concept: Create a configured operator for your storage type

// Conceptual example - actual implementation needs error handling
let fs_builder = Fs::default().root("/");
let operator = Operator::new(fs_builder)?.finish();

Key Ideas:

The operator abstracts away storage-specific details
Different services (Fs, S3, Azure, etc.) use the same Operator interface
Configuration happens through service-specific builders

2. SQLite Virtual Table Architecture

Virtual tables in SQLite allow you to present non-SQL data as queryable tables.

Concept: Bridge between OpenDAL and SQLite

// Conceptual structure - real implementation much more complex
struct FileSystemTable {
    base: sqlite3_vtab,           // Required SQLite base
    operator: Rc<Operator>,       // Your OpenDAL operator
}

Key Ideas:

Virtual tables implement a specific SQLite interface
They translate SQL queries into storage operations
The table schema defines what data is queryable

3. Schema Design

Design your virtual table schema to expose useful file metadata.

Concept: Map file attributes to SQL columns

CREATE TABLE filesystem_view(
    name TEXT,                    -- File name
    path TEXT,                    -- Full path
    last_modified TEXT,           -- When file was changed
    content BLOB,                 -- File contents (lazy-loaded)
    size INTEGER,                 -- File size in bytes
    content_type TEXT,            -- MIME type
    digest TEXT,                  -- File hash/checksum
    arg_path HIDDEN              -- Query parameter (hidden column)
);

Key Ideas:

Hidden columns can accept query parameters
BLOB columns can hold binary data
Schema should balance utility with performance

4. Query Translation Flow

The magic happens in translating SQL queries into OpenDAL operations.

Concept: SQL WHERE clause becomes OpenDAL list operations

// Conceptual flow - real implementation involves cursor management
fn handle_query_with_path_filter(path: &str) -> Vec<FileMetadata> {
    // 1. Use OpenDAL to list files in the specified path
    let lister = operator.lister_with(path)
        .metakey(Metakey::ContentLength | Metakey::LastModified)
        .call()?;

    // 2. Convert OpenDAL entries to your table format
    let mut results = Vec::new();
    for entry in lister {
        if entry.metadata().mode() == EntryMode::FILE {
            results.push(convert_to_table_row(entry));
        }
    }

    results
}

Key Ideas:

SQL filters become OpenDAL query parameters
Metadata is fetched on-demand to avoid performance issues
Results are converted to SQLite-compatible format

5. Lazy Loading Strategy

For performance, expensive operations (like reading file contents) should be lazy.

Concept: Only load data when specifically requested

// Conceptual approach to lazy loading
fn get_column_value(&self, column: i32) -> SqliteValue {
    match column {
        0 => SqliteValue::Text(self.file_name.clone()),
        1 => SqliteValue::Text(self.file_path.clone()),
        2 => SqliteValue::Integer(self.file_size),
        3 => {
            // Expensive operation - only do when column is requested
            let content = self.operator.read(&self.file_path)?;
            SqliteValue::Blob(content)
        }
        _ => SqliteValue::Null,
    }
}

Key Ideas:

Separate metadata operations from content operations
Use OpenDAL's stat() for metadata, read() for content
Only perform expensive operations when columns are actually selected

6. Error Handling Patterns

Bridge OpenDAL errors to SQLite errors appropriately.

Concept: Convert storage errors to SQL errors

// Conceptual error mapping
fn map_opendal_error(err: OpenDalError) -> SqliteError {
    match err.kind() {
        ErrorKind::NotFound => SqliteError::NotFound,
        ErrorKind::PermissionDenied => SqliteError::Auth,
        ErrorKind::Unexpected => SqliteError::Internal,
        _ => SqliteError::Internal,
    }
}

Key Ideas:

OpenDAL errors need translation to SQLite error codes
Some errors should be handled gracefully (missing files)
Others should propagate up to the SQL layer

Usage Patterns

Once implemented, your virtual table enables powerful queries:

-- List all files in a directory
SELECT name, size FROM filesystem_view WHERE arg_path = '/documents';

-- Find large files
SELECT path, size FROM filesystem_view 
WHERE arg_path = '/media' AND size > 1000000;

-- Search by file type (if you implement content_type detection)
SELECT name, path FROM filesystem_view 
WHERE arg_path = '/projects' AND content_type LIKE 'text/%';

Benefits of This Approach

Why combine OpenDAL with SQLite virtual tables?

Unified Query Interface: Use SQL to query any storage backend
Flexibility: Same queries work across local files, S3, Azure, etc.
Integration: Easy to embed in existing SQLite-based applications
Performance: SQLite's query optimization benefits your file operations
Ecosystem: Leverage SQLite's rich ecosystem of tools and extensions

Next Steps

To implement this approach:

Study SQLite's virtual table interface documentation
Explore OpenDAL's service implementations for your storage needs
Design your schema based on your specific use cases
Implement the virtual table interface with proper error handling
Add performance optimizations like caching and pagination

This pattern opens up powerful possibilities for unified data access across different storage systems while maintaining the familiar SQL interface.

Conclusion

This guide demonstrated how to use OpenDAL for filesystem operations in Rust and integrate it with SQLite virtual tables. By leveraging OpenDAL, you can seamlessly interact with local storage while maintaining a structured query interface via SQLite.

For more details, check out OpenDAL's documentation.

Getting Started with Rust in 2025: Why Now Is a Great Time to Learn Rust 🦀

Mukhtar — Sun, 13 Jul 2025 20:48:12 +0000

Getting Started with Rust in 2025 🦀

Rust has been growing steadily over the years, and in 2025, it's more relevant than ever. If you've been on the fence about learning it, now's the perfect time to jump in.

🧠 Why Learn Rust?

Rust is a modern systems programming language focused on:

Performance (like C/C++)
Safety (no nulls, no data races)
Modern developer experience (great tooling, compiler messages)

Companies like Microsoft, Amazon, Cloudflare, and Dropbox use Rust in production. It's built to help you write fast, reliable, and maintainable software.

🌟 What Makes Rust Special?

Memory safety without garbage collection
Fearless concurrency – build multi-threaded apps without race conditions
Helpful compiler – the compiler actually teaches you!
Powerful tooling – cargo, clippy, rustfmt, and more
Zero-cost abstractions – safety and performance, no tradeoffs

If you've dealt with C/C++ bugs or hit performance walls in Python or JavaScript, Rust is the best of both worlds.

🆚 Rust vs Other Languages

Rust sits in a unique position compared to other programming languages:

Performance – As fast as C/C++ but much safer
Memory Management – Manual control without the crashes
Type Safety – Catches bugs at compile time, not runtime
Concurrency – Built-in support for safe multi-threading
Learning Curve – Steeper initially, but pays off long-term

Unlike garbage-collected languages, Rust gives you control. Unlike manual memory management languages, Rust prevents common bugs.

🚀 Code Examples

Hello World

Every Rust program starts with a main() function. This is your entry point!

// This is the main function - where your program starts
fn main() {
    // println! is a macro that prints to the console
    println!("Hello, world!");
}

Variables and Types

Rust is statically typed but can often infer types. Variables are immutable by default - you need mut to change them.

fn main() {
    // Variables are immutable by default
    let name = "Alice";           // string slice (&str)
    let age: u32 = 30;           // unsigned 32-bit integer
    let mut score = 100;         // 'mut' makes it mutable (changeable)

    // We can change mutable variables
    score += 50;

    // {} are placeholders for variables in println!
    println!("{} is {} years old with a score of {}", name, age, score);
}

Functions

Functions in Rust are defined with fn. The last expression (without semicolon) is the return value.

// Function that takes two i32 parameters and returns an i32
fn add(a: i32, b: i32) -> i32 {
    a + b  // no semicolon = this is the return value
}

fn main() {
    // Call the function and store the result
    let result = add(5, 3);
    println!("5 + 3 = {}", result);
}

Ownership (Rust's Superpower)

Ownership is Rust's way of managing memory safely. Each value has one owner, and when the owner goes out of scope, the value is dropped.

fn main() {
    // Create a String on the heap
    let s1 = String::from("hello");

    // This MOVES s1 to s2. s1 is no longer valid!
    let s2 = s1;

    // This would cause a compile error because s1 was moved
    // println!("{}", s1); 

    // Only s2 is valid now
    println!("{}", s2);
}

Borrowing

Borrowing lets you use a value without taking ownership. Think of it like borrowing a book - you can read it, but you have to give it back!

// This function borrows a String reference (&String)
// It doesn't take ownership, just looks at the data
fn print_length(s: &String) {
    println!("Length: {}", s.len());
}

fn main() {
    let my_string = String::from("hello");

    // Pass a reference (&) to the function
    print_length(&my_string);

    // We can still use my_string because we only borrowed it
    println!("{}", my_string);
}

Structs & Methods

Structs are like classes in other languages - they group related data together. You can add methods to them using impl blocks.

// Define a struct (like a class in other languages)
struct Person {
    name: String,
    age: u32,
}

// Implementation block - where we define methods
impl Person {
    // Associated function (like a constructor)
    fn new(name: String, age: u32) -> Person {
        Person { name, age }  // shorthand for name: name, age: age
    }

    // Method that takes &self (borrows the instance)
    fn greet(&self) {
        println!("Hi, I'm {}", self.name);
    }
}

fn main() {
    // Create a new Person instance
    let person = Person::new("Bob".to_string(), 25);

    // Call the method
    person.greet();
}

Error Handling

Rust doesn't have exceptions. Instead, it uses Result to handle errors explicitly. This forces you to deal with potential failures.

// Function returns Result<T, E> - either Ok(value) or Err(error)
fn divide(a: f64, b: f64) -> Result<f64, String> {
    if b == 0.0 {
        // Return an error
        Err("Cannot divide by zero".to_string())
    } else {
        // Return the successful result
        Ok(a / b)
    }
}

fn main() {
    // Use match to handle both success and error cases
    match divide(10.0, 2.0) {
        Ok(result) => println!("Result: {}", result),
        Err(error) => println!("Error: {}", error),
    }
}

🛠️ Getting Started

Install Rust: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Create new project: cargo new my_project
Build and run: cargo run
Read "The Rust Book" online for free

🔥 What You Can Build

Web servers (Actix, Rocket frameworks)
Command-line tools (ripgrep, bat)
Game engines (Bevy)
Blockchain projects (Solana, Polkadot)
WebAssembly applications
Operating systems (Redox OS)

💡 Why 2025 is Perfect

Mature ecosystem – tons of quality crates
Industry adoption – major companies using it in production
Great learning resources – excellent documentation and community
Job market – high demand, good salaries
Future-proof – language designed for next-generation computing

Rust's combination of performance, safety, and growing ecosystem makes it an excellent choice for modern development in 2025!