NexusPipeline: Enterprise Data Ingestion Engine with AI Content Querying

Brian — Wed, 23 Sep 2026 09:49:18 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

NexusPipeline is a high-performance, resilient data ingestion and processing engine built with .NET 8, RabbitMQ, and SQLite (Write-Ahead Logging mode). Designed for high-throughput enterprise data routing, it ensures strict multi-tenant isolation, immutable audit logging, and automated error quarantine. To bring intelligence to the pipeline, I integrated an AI Content Query Agent that exposes a secure endpoint (/api/agent/query). This allows autonomous agents to query, filter, and inspect structured telemetry and data streams in real time.

Demo

Architecture Highlights: Built as a multi-threaded C# solution featuring asynchronous message brokering and dead-letter quarantine routing.
API Integration: Securely guarded via custom API Key attributes to ensure isolated tenant data access.

Code

Core & Controllers:
- AgentQueryModels.cs: Defines request/response DTOs for structured content queries.
- AgentController.cs: Handles secure, tenant-isolated search queries against the SQLite database using Entity Framework Core.
Repository/Project: View the NexusPipeline Repository on GitHub

How I Used Sanity / Structured Content

NexusPipeline leverages structured content by treating every incoming payload as an immutable audit record with strict data schemas, file hashing, and metadata tracking. The AI Content Query Agent connects directly to this structured content layer, enabling automated systems to run precision queries on pipeline health, message statuses, and transaction logs without risking data corruption or cross-tenant leaks.

How I Built a Multi-Threaded File Organizer and Duplicate Finder in C#

Brian — Sun, 13 Sep 2026 18:40:32 +0000

// Example snippet structure of the parallel scanner logic
Parallel.ForEach(Directory.GetFiles(targetPath, ".", SearchOption.AllDirectories), file =>
{
using (var md5 = MD5.Create())
{
using (var stream = File.OpenRead(file))
{
var hash = BitConverter.ToString(md5.ComputeHash(stream)).Replace("-", "").ToLowerInvariant();
// Register or track hash collisions for duplicate detection
}
}
});
The Problem
Sorting through massive directory structures manually is a time sink. I wanted a lightweight, native Windows utility built in C# that could handle background monitoring via FileSystemWatcher and instantly sweep directories for redundant files using parallel MD5 hashing.

Technical Implementation

Fast Duplicate Detection with Parallel Processing
To keep the scan times down on large drives, synchronous file reading doesn't cut it. By leveraging Parallel.ForEach combined with buffered stream reading, we can compute cryptographic hashes concurrently across multiple threads without choking system memory.
Real-Time Directory Monitoring
Using FileSystemWatcher, the application hooks directly into target folders to catch file creations, renames, or movements on the fly, instantly applying sorting logic based on custom rules without requiring constant manual triggers.
Generating Audit Reports
Once operations finish, the app structures the logs and writes out a clean, readable HTML summary report so you have a complete audit trail of what was sorted or cleaned.

Wrapping Up
If you're looking to grab the pre-compiled, standalone Windows executable version of this tool or want to check out how the UI and automated rules are packaged together, I've listed it over on Gumroad: bryjackson.gumroad.com/l/zxluba.