DEV Community: syamaner

Part 1: The Architecture & The Agent - Spec-Driven ML Development With Warp/Oz

syamaner — Tue, 14 Apr 2026 07:55:00 +0000

Last year I built a prototype coffee first crack detector and wrote about it in a 3-part series. The prototype works - I have been running it on my own roasts since November - but it carries the technical debt of something built to prove a concept rather than to last.

This series is the production rebuild. The outcome: an Audio Spectrogram Transformer at 97.4% accuracy and 100% precision on first crack detection, running on a Raspberry Pi 5 at 2.09 seconds per 10-second window. The full pipeline - data preparation, training, evaluation, ONNX INT8 export, edge validation, and a Gradio UI - shipped in two evenings.

I didn't build this by brute-forcing the codebase myself. I acted strictly as the engineering lead, while Warp and its AI agent, Oz, handled the implementation from inside my terminal. My responsibilities were entirely architectural:

Designing the workflow: Setting the strict rules of engagement between the agent and the codebase.
Defining the science: Dictating the specs, testing strategy, evaluation metrics, and dataset annotation approach.
Directing the execution: Guiding the agent through the implementation and reviewing the output.

Operating this way over the weekend, Warp/Oz executed an 18-story (at the time) epic across 10 pull requests. That resulted in 11,087 lines of Python across 75 files, with 52 of those commits explicitly co-authored by the agent. Copilot reviewed every PR, flagging 111 individual issues across 28 review batches. The model is published on Hugging Face, the dataset is open-sourced, and the source is on GitHub.

This post is about the system that made that possible - not the model itself. The ML science comes in Posts 2 and 3. Here, I want to show the exact architecture I used to direct an AI agent through a complex, multi-phase ML project without losing control of the engineering decisions that matter.

Before the agent could train anything, I had to build the training data from scratch. There is no public audio dataset for coffee roasting first crack - not on Hugging Face, not on Kaggle, not in academic literature. That meant recording roasting sessions, annotating them in Label Studio, and architecting a recording-level data pipeline to prevent the chunk-level leakage that silently inflates test metrics in time-series audio ML. The full data engineering story is in Post 2 (coming soon).

From Prototype to Production

The prototype had accumulated real technical debt. The code was monolithic, the model had no reusable packaging, the MCP server architecture had flaws I had been working around, and nothing ran on edge hardware. I had to use my laptop for every roast.

This series covers the production rebuild. Same domain, completely new architecture:

A standalone, Hugging Face-native training repository.
Strict data engineering to prevent audio leakage.
ONNX INT8 quantization for Raspberry Pi 5 edge deployment.
A live Gradio Space for public inference.

The Director/Coder Dynamic

The core pattern was a strict, enforced separation of concerns between three actors:

I (the human) owned:

The Architecture: Defining repository structure, module boundaries, and enforcing Hugging Face's save_pretrained/from_pretrained as the standard packaging contract.
The ML Science: Model selection (AST over CNN), data split strategy (recording-level to prevent leakage), class weighting, and hyperparameter math.
The Workflow Constraints: Defining the project rules, writing the parameterised skills, and managing the state of the epic.
The Quality Gates: Reviewing every PR, interpreting the evaluation metrics, and deciding when to retrain versus when to ship.

Oz (Warp's terminal-native agent) owned:

Terminal Execution: Running training loops, evaluations, ONNX exports, and SSH sessions directly on the Raspberry Pi.
Code Generation: Writing the boilerplate-WeightedLossTrainer subclasses, CLI argument parsers, pytest scaffolds, and audio data loaders.
Skill Invocation: Executing parameterised skill files (e.g., .claude/skills/train-model/SKILL.md) that encoded exact command sequences and validation checks.
State Management: Reading the epic document, updating context, and checking off stories after completing a phase.

GitHub Copilot owned:

Async Code Review: Flagging type safety issues, API misuse, missing error handling, and dependency hygiene across all 10 PRs.
The Reality Check: Copilot never once caught a machine learning logic error. Every data leakage fix, hyperparameter correction, and precision/recall tradeoff decision came from me. Copilot acts as an aggressive linter for code, not a reviewer for ML science.

This three-way split wasn't a gentleman's agreement-it was hardcoded into the project via an AGENTS.md file. Whenever Oz started a task, it was forced to read this rulebook first.

The Agentic Setup: AGENTS.md, Epics, and Skills

Three files controlled the entire project.

1. `AGENTS.md` - The Rulebook

This file sits at the repository root. The agent is instructed to read it before starting any task. It contains the project rules, quick commands, codebase architecture, and platform-specific constraints. Here is the exact rules section from this project:

## Rules

- Python 3.11+ with full type hints on all public functions and methods
- Google-style docstrings
- `ruff check` and `ruff format` must pass before marking code complete
- `pyright` must pass with no errors on new code
- All dependencies declared in `pyproject.toml` - never install ad-hoc
- Large files (WAV, checkpoints, ONNX models) go to Hugging Face Hub - never commit to git
- `data/`, `experiments/`, and `exports/` are `.gitignore`'d - keep them that way
- Seed all RNG using `configs/default.yaml` seed value
- One PR per story, branch: `feature/{issue-number}-{slug}`
- Before starting a task: read `docs/state/registry.md` → open epic file → check GitHub issue

That last line is the critical one. It forces the agent into a state-reading loop before writing any code. Without it, the agent starts generating based on stale context.

The file also includes a codebase architecture map, quick commands for every operation (training, evaluation, export, benchmarking), and platform-specific notes for MPS, CUDA, and the RPi5. The full file is on GitHub.

2. Epic State Management - The Checklist

A registry file (docs/state/registry.md) points to the active epic. The epic file itself (docs/state/epics/coffee-first-crack-detection.md) contains 18 stories grouped into 6 phases, each linked to a GitHub issue. Before and after every task, the agent reads the epic state and updates it according to this protocol:

Before starting any task:
1. Read docs/state/registry.md to find the active epic
2. Open the epic file - check story status
3. Open the GitHub story issue - read comments for latest requirements
4. Work on a branch: feature/{issue-number}-{slug}

After completing a story:
1. Check off the story in the epic doc
2. Update Active Context section with what was built
3. Comment on the GitHub story issue, then close it
4. Tick the checkbox in GitHub epic issue #1
5. Open a PR referencing the story issue

This is how 18 stories were delivered without losing track of what was done, what was next, or what had changed. The agent maintained its own project state.

Here is Oz running the full data preparation pipeline - chunking 973 audio segments, performing the recording-level split, and then invoking the /train-model skill:

3. Parameterised Skills - The Playbooks

Skills are markdown files under .claude/skills/ that encode exact command sequences for common operations. Each skill defines the prerequisites, the commands, and the validation steps. I wrote four:

train-model/SKILL.md - End-to-end training with data validation and checkpoint saving.
evaluate-model/SKILL.md - Test-set evaluation with metrics report generation.
export-onnx/SKILL.md - ONNX export (FP32 + INT8) with size and latency benchmarking.
push-to-hub/SKILL.md - Publish model and dataset to the Hugging Face Hub.

When I told Oz to "train the model," it didn't improvise. It read the skill file and followed the exact sequence I defined. This eliminated an entire class of errors where the agent guesses at flags, skips validation steps, or forgets to save the feature extractor configuration alongside the model weights.

Here is Oz chaining the /export-onnx and /push-to-hub skills to export the model and publish everything to Hugging Face Hub in a single sequence:

A Generalised AGENTS.md Template

Here is a stripped-down version you can drop into any project. Replace the placeholders with your domain-specific rules.

This file is not documentation for humans. It is a system prompt for your codebase. Every rule you omit is a decision the agent will make on its own-and it will make it differently every time.

# AGENTS.md - [Project Name]

Project rules and context for AI coding agents.

## Rules
- [Language] [version]+ with [typing/linting requirements]
- [Formatter] and [linter] must pass before marking code complete
- All dependencies declared in [manifest file] - never install ad-hoc
- Large files go to [remote storage] - never commit to git
- Before starting a task: read `docs/state/registry.md` → open epic → check issue

## Quick Commands
### Setup
[environment setup commands]

### Build / Test / Deploy
[the exact commands for each operation]

## Codebase Architecture
[directory tree with one-line descriptions per module]

## Epic State Management
Before starting any task:
1. Read docs/state/registry.md
2. Check story status in the epic file
3. Read the GitHub issue for latest requirements
4. Branch: feature/{issue-number}-{slug}

After completing a story:
1. Check off the story in the epic doc
2. Update Active Context
3. Close the GitHub issue
4. Open a PR

The Build & The Fails

The first commit after the initial scaffold was feat(S5/S6/S8): implement train.py, evaluate.py, inference.py. In a single pass, Oz generated the training pipeline, evaluation harness, and sliding-window inference module. It followed the AGENTS.md rules, used the correct base model (MIT/ast-finetuned-audioset-10-10-0.4593), and wired up the WeightedLossTrainer subclass with class-weighted CrossEntropyLoss exactly as I specified.

Then training failed.

The `input_features` vs `input_values` Bug

Oz wrote the dataset adapter to return input_features as the tensor key - a reasonable guess if you have seen other Hugging Face audio pipelines. But ASTFeatureExtractor returns input_values, not input_features. The model silently received no input and the loss exploded.

Here is the exact diff from the fix commit (75bbb4b):

# src/coffee_first_crack/train.py - _HFDatasetAdapter.__getitem__
-            "input_features": inputs["input_features"].squeeze(0),
+            "input_values": inputs["input_values"].squeeze(0),

It was a one-line bug. The kind of bug that costs you an hour of staring at training logs if you don't know what to look for. Oz pattern-matched from Whisper examples - the most common audio model in Hugging Face tutorials - where input_features is correct. For ASTFeatureExtractor, the key is input_values. This is a known, unresolved inconsistency in the Hugging Face audio API.

The same commit also added accelerate>=0.26.0 to pyproject.toml - a dependency the Hugging Face Trainer requires at runtime but doesn't explicitly import at the top level. Oz didn't catch it during code generation because it never triggered an ImportError until actual training.

Here is the model evaluated on a Raspberry Pi 5 - 191 test samples, INT8 quantised, 4 threads, via SSH from Warp:

This is what the validation loop looks like in practice - Oz hitting a pyright failure, diagnosing the type issues, fixing them, then running the full ruff check → ruff format → pyright → pytest chain until all checks pass:

Copilot as the Third Actor

Across the 10 PRs in this project, Copilot submitted 28 review batches containing 111 individual comments. Here is how they broke down by PR:

PR #23 (RPi5 ONNX validation): 36 comments across 6 review rounds - the most reviewed PR by far.
PR #17 (Export, scripts, tests): 26 comments across 5 rounds.
PR #27 (Data prep + mic-2 expansion): 16 comments across 3 rounds.
PR #16 (Train, eval, inference): 10 comments.
PR #28 (Gradio Space): 10 comments.

The pattern was consistent. Copilot caught:

Type safety: Missing type hints, incorrect return types, untyped function signatures.
Unused imports: Dead code left behind after refactoring.
API misuse: Deprecated parameters, missing synchronisation calls, incorrect exception handling.
Dependency hygiene: Missing explicit dependencies, version pinning issues.
Docs and copy: Misleading docstrings, inaccurate UI text in the Gradio Space.

However, Copilot did not catch the core machine learning logic issues. To be fair, this is largely because my workflow required me to intercept them before they ever reached a PR:

The input_features vs input_values key mismatch: This was fixed locally during the active dev loop before opening the PR.
Data leakage from chunk-level splitting: This is the biggest ML risk in this project, but this was addressed architecturally during the setup phase.
Hyperparameter choices: Overfitting issues were identified and corrected interactively by reading the local training logs.
The precision/recall tradeoff: The class weighting strategy was a deliberate human decision delivered prior to code review.

This is not a criticism of Copilot. It is doing exactly what it should: catching code-level defects at review time. But if you are relying on AI code review to validate your ML pipeline logic, you will ship broken models with clean code.

By the Numbers

Metric	Result
Wall-clock time	Two evenings (Fri → Sat)
Stories completed	18 across 6 phases
Pull requests	10 merged
Total commits	~65 (55 non-merge)
Oz co-authored	52 commits
Lines of code	11,087 insertions across 75 files
Copilot reviews	28 batches, 111 individual comments
Model accuracy	97.4% test / 100% precision
Edge latency	2.09s per 10s window (RPi5, INT8, 4 threads)
Dataset	First public coffee roasting audio dataset - 973 chunks, 15 roasts

The model is live at huggingface.co/syamaner/coffee-first-crack-detection. The dataset is at huggingface.co/datasets/syamaner/coffee-first-crack-audio. The source is on GitHub.

The piece of this workflow I wouldn't give up: the state-reading loop in AGENTS.md. Without it, agent context drifts within two or three tasks and it starts generating against stale assumptions. If you've run a long-form agentic project and solved the context problem differently, I'd be interested in the specifics.

Next up: Post 2 - The Data(coming soon) covers how I built the first public audio dataset for coffee roasting first crack detection, and the data engineering decisions that got us to zero false positives.

Try it - upload a 10-second roasting clip or use an existing sample:

Links

Project:

Tools:

Part 3: From Neural Networks to Autonomous Coffee Roasting - Orchestrating MCP Servers with .NET Aspire 13 and n8n Agents

syamaner — Sun, 16 Nov 2025 18:48:48 +0000

Introduction

In Part 1, we have fine tuned a neural network to detect coffee first crack from audio using PyTorch and the Audio Spectrogram Transformer. In Part 2, we have built two MCP (Model Context Protocol) servers - one to control my Hottop KN-8828B-2K+ roaster and another to detect first crack using a microphone in real-time.

This is where put it all together. But first: can .NET Aspire orchestrate Python MCP servers and n8n workflows to autonomously roast coffee?

Spoiler alert: Yes, it can. And the coffee tastes spot on.

The Challenge

Autonomous coffee roasting isn't just about detecting when first crack happens. It's a complex orchestration problem involving:

Multiple systems: Python MCP servers to interact with hardware, an agent layer for orchestration (n8n workflows to begin with), containerised services.
Real-time decision making: Monitoring sensors every few seconds and deciding on actions depending on the status.
Safety-critical control: Managing heat and fan speed to avoid burning / wasting green beans.
Precise timing: Detecting bean charge event (when beans were added during pre heating stage), first crack, and hitting target development time percentage by adjusting controls available.
Observability: Tracking telemetry across Python, n8n, and .NET components.

The solution? .NET Aspire 13 orchestrating everything.

Why Aspire 13?

Aspire 13.0 (released with .NET 10) brings significant improvements for Python integration and container orchestration—perfect for this use case:

Simplified Python Hosting with `AddPythonModule`

Aspire 13 replaces the old AddPythonApp API with three specialized methods:

AddPythonModule: Runs Python modules with -m flag (e.g., python -m src.mcp_servers.roaster_control.sse_server).
AddPythonScript: Runs standalone Python scripts.
AddPythonExecutable: Runs executables from virtual environments (e.g., uvicorn, gunicorn).

For MCP servers running as modules, AddPythonModule is cleaner and more explicit:

// Old way (Aspire 9)
builder.AddPythonApp("roaster-control", projectRoot, "-m", venvPath)
    .WithArgs("src.mcp_servers.roaster_control.sse_server")

// New way (Aspire 13)
builder.AddPythonModule("roaster-control", projectRoot, "src.mcp_servers.roaster_control.sse_server")
    .WithVirtualEnvironment(venvPath)

Cleaner AppHost Project Structure

The new Aspire.AppHost.Sdk/13.0.0 simplifies project files:

No separate <Sdk Name="..." /> element needed.
Aspire.Hosting.AppHost package included automatically.
Removes IsAspireHost property (implicit).

Enhanced Container Orchestration

Better lifecycle management for containers (n8n in this project).
Improved health check support.
More granular control over container runtime arguments.

Built-in OpenTelemetry Integration

Out-of-the-box observability with .WithOtlpExporter() for:

Structured logging from Python processes.
Distributed tracing across MCP calls.
Real-time metrics in the Aspire dashboard.

Architecture Overview

Here's what .NET Aspire orchestrates:

Why Aspire?

Single command startup: dotnet run starts all 3 services with proper dependency ordering
Shared configuration: Environment variables, Auth0 credentials, OpenTelemetry
Python support: Built-in virtual environment management with AddPythonModule
Container orchestration: Manages n8n container
Observability: Unified dashboard with logs, traces, and metrics from all components
Development velocity: Changes to Python code auto-reload, no container rebuilds needed

The n8n Autonomous Roasting Workflow

As a first step, n8n is selected for the agent layer. The visual workflow setup and the constructs provided by n8n allowed for radio verification of the agentic roasting process. The heart of the system is an n8n workflow that acts as the "roasting brain." Here's what it does:

Phase 1: Initialisation & Preheating (Preheating Agent)

Start → Read Roaster Status → Start Roaster → Monitor Temperature

The workflow:

Connects to both MCP servers via SSE (Server-Sent Events).
Starts the roaster at 100% heat, 30% fan.
Monitors bean temperature rising toward ~170°C during preheating.
Uses an AI Agent node (Preheating Agent) with custom instructions to detect preheating completion.

Key metrics tracked:

Bean temperature.
Rate of Rise (°C/min).
Fan speed (%).
Heat level (%).

Phase 2: Bean Charge Detection (Preheating Agent)

Monitor Temp → Detect Temperature Delta threshold → Mark T0 Timestamp

When green beans are added to the hot roaster, the temperature suddenly drops (e.g., from 170+°C → less than 90°C). Then the workflow:

Tracks rolling temperature averages.
Detects sudden drops > 40°C.
Marks "T0" - the beginning of roast time.
All subsequent metrics are relative to T0.

From the logs:

{
  "t0_detected": true,
  "beans_added_temp_c": 96,
  "t0_timestamp_utc": "2025-11-15T21:21:56.490259+00:00"
}

Phase 3: First Crack Detection (Roast Agent)

Loop: Poll First Crack MCP → Check Status → Wait

The workflow continuously calls the First Crack Detection MCP server:

Streams microphone audio to the PyTorch model.
Uses sliding window inference (10-second windows).
Implements "pop-confirmation" logic (minimum 3 pops within 30 seconds).
Reports when first crack is confirmed.

Detection event:

{
  "first_crack_temp_c": 184.0,
  "first_crack_time_display": "08:42",
  "roast_elapsed_seconds": 522
}

Phase 4: Development Time Management (Roast Agent)

This phase is important as it can lead to under-roasted or over-roasted beans. The agent's objective is to adjust fan and heat to extend development time.

Development time percentage is the percentage of the time spent between first crack and end of roast compared to the overall roasting time. The goal is to get this period around 15-20%. On my machine, I have noticed that this needs to be achieved before bean temperatures go above 196°C to get the results I prefer.

Loop: Adjust Heat/Fan → Monitor Development % → Check Target

Once first crack is detected, the critical development phase begins. The workflow's AI agent:

Monitors:

Current bean temperature
Rate of Rise (to prevent stalling or rushing)
Development time percentage (target: 15-20%)
Time since first crack

Controls:

Reduces heat (100% → 60% → 40%)
Increases fan speed (30% → 50% → 70%)
Slows the roast to extend development time

Decision logic (via AI agent):

IF development_time_percent >= 15% AND development_time_percent <= 20%:
    IF bean_temp_c >= 190 AND bean_temp_c <= 195:
        → DROP BEANS (optimal light roast)
    ELSE IF bean_temp_c > 195:
        → DROP BEANS (approaching medium roast)
ELSE:
    → CONTINUE MONITORING

Actual output from workflow:

{
  "phase": "development",
  "action": "monitor",
  "bean_temp_c": 191,
  "message": "Development: 191°C, 8.9%"
}

Then moments later:

{
  "phase": "cooling",
  "action": "drop",
  "bean_temp_c": 193,
  "message": "Optimal! Dropping beans."
}

Phase 5: Completion

Drop Beans → Set Cooling Fan to 100% → Stop Heat → Cool

The workflow:

Commands the roaster to drop beans into cooling tray.
Sets the cooling fan to 100% for maximum cooling.
Cuts heat to 0%.
Records final metrics for analysis.

Final roast profile:

{
  "roast_elapsed_seconds": 584,
  "roast_elapsed_display": "09:44",
  "beans_added_temp_c": 175.0,
  "first_crack_temp_c": 184.0,
  "first_crack_time_display": "08:42",
  "development_time_seconds": 62,
  "development_time_display": "01:02",
  "development_time_percent": 10.6,
  "total_roast_duration_seconds": 584
}

The Aspire Orchestration Code

Here's how .NET Aspire 13 makes this all work (from Program.cs):

Python MCP Servers

// Roaster Control MCP Server
var roasterControl = builder.AddPythonModule(
        "roaster-control",
        projectRoot,
        "src.mcp_servers.roaster_control.sse_server")
    .WithVirtualEnvironment(sharedVenvPath)
    .WithHttpEndpoint(port: 5002, env: "ROASTER_CONTROL_PORT")
    .WithEnvironment("AUTH0_DOMAIN", auth0Domain)
    .WithEnvironment("AUTH0_AUDIENCE", auth0Audience)
    .WithEnvironment("USE_MOCK_HARDWARE", useMockHardware)
    .WithEnvironment("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc")
    .WithOtlpExporter();

// First Crack Detection MCP Server
var firstCrackDetection = builder.AddPythonModule(
        "first-crack-detection",
        projectRoot,
        "src.mcp_servers.first_crack_detection.sse_server")
    .WithVirtualEnvironment(sharedVenvPath)
    .WithHttpEndpoint(port: 5001, env: "FIRST_CRACK_DETECTION_PORT")
    .WithEnvironment("AUTH0_DOMAIN", auth0Domain)
    .WithEnvironment("AUTH0_AUDIENCE", auth0Audience)
    .WithEnvironment("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc")
    .WithOtlpExporter();

What's happening here:

AddPythonModule: New in Aspire 13, replaces the old AddPythonApp API
WithVirtualEnvironment: Points to shared Python 3.11 venv at repo root
WithHttpEndpoint: Configures SSE endpoints for n8n to connect
WithOtlpExporter: Sends telemetry to Aspire dashboard
Modules run with -m flag implicitly (e.g., python -m src.mcp_servers.roaster_control.sse_server)

Container Services

// n8n Workflow Engine
var n8n = builder.AddContainer("n8n", "n8nio/n8n", "latest")
    .WithHttpEndpoint(port: 5678, targetPort: 5678, name: "n8n-ui")
    .WithBindMount("./n8n-data", "/home/node/.n8n")
    .WithEnvironment("N8N_HOST", "0.0.0.0")
    .WithEnvironment("N8N_PORT", "5678")
    .WithEnvironment("WEBHOOK_URL", "http://localhost:5678/")
    .WithEnvironment("N8N_METRICS", "true");

Key features:

Bind mount for persisting workflows and credentials.
Exposes port 5678 for web UI.
Metrics enabled for observability.
Auto-restarts on failure.

Real-World Results

The First Autonomous Roast

Stats:

Total roast time: 9:44 (584 seconds)
First crack: 8:42 at 184°C
Development time: 1:02 (10.6% - slightly under target but acceptable)
Final temperature: 193°C
Result: Light roast, consistent colour and smooth taste.

What worked:

Temperature drop detection caught bean addition instantly..
First crack detection was accurate (within 20 seconds of my ears). This is why 10% development percentage is not an issue.
Heat/fan adjustments prevented burning.
Development % monitoring kept roast in safe zone.

What could improve:

Development time was 10.6% instead of target 15-20%.
Could start reducing heat earlier after first crack.
Rate of Rise could be smoother in final phase.

The Aspire Dashboard Experience

The unified Aspire dashboard shows:

Services:

roaster-control (Python) - Running
first-crack-detection (Python) - Running
n8n (Container) - Running

Metrics:

Lessons Learned

1. MCP Server Design Matters

Current design has two MCP servers. One for roaster control, one for first crack detection. The original idea was, the roaster control MCP server could run on a low powered device connected to the roaster and the First Crack Detector could run on the laptop due to hardware requirements.

This design adds coordination overhead to the agent and makes it more complicated than necessary. A unified MCP server that returns all metrics in a single call would simplify the agent logic and likely lead to more predictable behaviour. Before moving onto multiple agent framework comparison, this will be one area to improve.

2. Aspire's Python Support is Production-Ready

Before Aspire:

Multiple terminal windows or docker compose
Manual venv activation
Additional effort to add open telemetry collectors and dashboards

With Aspire:

One command: dotnet run.
Automatic venv management.
Shared configuration.
Structured logging and tracing.
Custom metrics.

3. n8n is Powerful for Agent Orchestration

Why n8n worked well:

Visual debugging: See workflow execution in real-time.
Built-in AI Agent node: Uses OpenAI with tool calling.
MCP client support: Native SSE connections.
Error handling: Built-in retry logic and error branches.
State management: Workflow variables persist between runs.

4. MCP Protocol Makes Tool Integration a Breeze

The MCP servers exposed simple HTTP/SSE endpoints:

Roaster Control Tools:

read_roaster_status → Returns current sensors + metrics
adjust_heat(level: int) → Sets heat 0-100%
adjust_fan(speed: int) → Sets fan 0-100%
stop_roaster() → Emergency stop
start_roaster() → Begin roast

First Crack Tools:

start_first_crack_detection() → Start audio monitoring
get_first_crack_status() → Check if first crack detected
stop_first_crack_detection() → Stop monitoring

The n8n AI Agent called these tools naturally:

Agent: "I need to check the roaster status"
→ Calls read_roaster_status
→ Receives JSON with temp, fan, heat, metrics
→ Makes decision
→ Calls adjust_heat(60) to reduce heat

5. Observability is Critical

When the roast is in progress, you need:

Real-time monitoring: See temperature changing every 2 seconds
Error visibility: Know immediately if MCP server crashes
Performance metrics: Ensure control commands complete in <500ms
Historical data: Review roast profile after completion

Aspire's OpenTelemetry integration gave us all of this for free.

The Development Experience with Warp Agent

Throughout this project, I used Warp Agent extensively:

For Aspire upgrade (9 → 13):

Warp Agent searched Microsoft Learn docs via MCP
Found breaking changes in AddPythonApp → AddPythonModule
Generated migration plan with test steps
Verified builds and runtime behavior

For n8n workflow debugging:

Analyzed MCP server logs to diagnose connection issues
Suggested retry logic for transient network errors
Helped structure AI agent prompts for decision-making

For Python model optimization:

Profiled inference latency
Suggested caching strategies for feature extraction
Optimized sliding window parameters

What made Warp Agent effective:

Context awareness: Understood the full stack (C#, Python, n8n)
MCP integration: Could fetch latest Microsoft docs and Context 7 for n8n.
Iterative debugging: Quickly test → analyse → fix cycles.
Code generation: Created boilerplate while I focused on logic.

Next Steps

Short Term

Roast profile tuning: Adjust heat/fan curves to hit 15-20% development consistently.
Data collection: Log every roast for analysis (temp curves, timestamps, outcomes).
- Add support for automatically exporting roast statistics and ability to rate roasts later.
Improved first crack detection: Capture manual recording sessions using different environmental setup to improve detection. What we have is impressive given we only had 9 roasting sessions for fine tuning. But we can do better.
Implement multiple agent frameworks and compare pros and cons.
Test MCP servers running on a Raspberry PI 5.

Medium Term

Train an emulation roast model using historical roast logs.
- This will allow experimentation without using actual hardware and will also allow realistic response taking heat, fan and time variables to emulate roaster heating process.
Machine learning on roast profiles: Train model to predict optimal heat/fan adjustments once there is enough roast samples and ratings.
Custom UI: Build dedicated roasting interface (replace n8n for end users) to allow unified experience across agent frameworks.
Multi-origin support: Adjust profiles based on bean origin (Kenya vs Brazil)

Conclusion

Can .NET Aspire roast coffee? Absolutely.

More importantly, it provided:

Unified orchestration for polyglot services (C#, Python, Node.js containers)
Developer productivity with single-command startup and hot reload
Production observability with unified logs, traces, and metrics
Flexibility to iterate quickly on both code and workflows

The combination of:

PyTorch model for first crack detection (Part 1)
MCP servers for hardware control and detection (Part 2)
.NET Aspire orchestration with n8n workflows (Part 3)

...resulted in a fully autonomous coffee roasting system that produces genuinely good coffee.

From 9 raw audio recordings to autonomous coffee roasting—all orchestrated with a single command: dotnet run

The coffee tastes great. The code is open source. And yes, .NET Aspire can definitely roast coffee.

For reference, Today's roast incurred $0.76 OpenAI API usage cost.

Resources

Code and Articles

Code repository: Bean Agent
Part 1: Training the Audio Detection Model
Part 2: Building MCP Servers

.NET Aspire Documentation

Aspire Overview: .NET Aspire documentation
Upgrade to Aspire 13: Upgrade guide
Python Hosting in Aspire: Orchestrate Python apps

Tools and Protocols

n8n Workflow Automation: n8n.io
Model Context Protocol: modelcontextprotocol.io
OpenTelemetry: opentelemetry.io

Model & ML:

Audio Spectrogram Transformer (AST) - Pre-trained model
AST Documentation
Fine-Tuning AST Tutorial
Original AST Paper - Gong et al., 2021

The first roast:

Part 2: Building MCP Servers to Control a Home Coffee Roaster - An Agentic Development Journey with Warp Agent

syamaner — Sun, 02 Nov 2025 14:58:02 +0000

Introduction

In this 3-part series, we are building an autonomous coffee roasting agent with Warp. The first part covered how we fine-tuned a model to detect first crack — a critical phase in the roasting process. This was a nice warm-up implementing a key component for our end goal, but detection alone isn't enough. Now we need to expose this functionality so the agent we'll build can both detect first crack and control the roasting process.

This post focuses on:

The objective: Turning ML predictions into real-world roaster control actions.
Solution overview: Model Context Protocol (MCP) servers as the bridge between AI agents and hardware
Implementation: The two MCP servers we built—First Crack Detector MCP + Hottop Controller MCP

📊 TL;DR

Connect trained ML model to physical roaster control using an agent to achieve autonomous coffee roasting
Build two MCP servers — FirstCrackDetector + HottopController
- Stack: Python MCP SDK, pyserial, pyhottop, Auth0 authentication
Real-time detection + safe roaster control via AI agents (208+ tests passing)
Using Warp Agent mode, Context7 MCP Server, Auth0 MCP Server during development
Next Part: Part 3 orchestrates both servers with Microsoft Agent Framework

Integrating software, hardware and agents without reinventing the wheel

Traditional approach: Build custom APIs, handle authentication, manage state

Write integration code using imperative/declarative patterns to manage task lifecycles.
Homegrown specifications make it harder to leverage emerging ML/AI technologies.
Each new AI model or agent requires custom integration work.

The MCP option: Standardised protocol for AI <-> tool communication

Provides deterministic tools for non-deterministic AI systems.
Benefits: Discoverability, type safety, streaming support, composability, and interoperability across AI models and agents.
Write once, connect to MCP-compatible AI (Claude, ChatGPT, custom agents).

🤖 Warp Agent Contributions

Provided MCP server scaffolding and tool definitions using standard MCP SDK.
Helped integrate pyhottop library for Hottop serial protocol communication.
Debugged serial communication timing issues and state synchronization.
Generated Auth0 authentication middleware with role-based access control.
Created comprehensive test suites (208+ tests across both MCP servers).
Suggested testing strategies including simulation mode for hardware free development.

What is MCP (Model Context Protocol) and Why Use It?

What is MCP?

MCP is a protocol that enables AI assistants like Claude, ChatGPT, Warp Agent Mode to connect to external resources through a client-server architecture following standard protocols. This allows using the same MCP server with various agent technologies without having to modify the code.

Client-Server Concept

MCP Server

A program that exposes specific data and tools (functionality) that an AI Agents can use. For example, a server might provide access to a database, file system, or even as in our case specific hardware.

MCP Client

The application that connects to MCP servers and makes their capabilities available to the AI. As an example, Claude /ChatGPT acts as an MCP client.

Transport Types

MCP supports different ways for clients and servers to communicate:

stdio (Standard Input/Output)
- Most common for local integrations
- Server runs as a subprocess, communicating via stdin/stdout
- Simple and works well for local tools
SSE (Server-Sent Events) -Used for remote servers over HTTP
- Server pushes updates to client
- Good for web-based integrations
Custom transports can also be implemented
- The protocol is designed to be transport-agnostic

Flow

When user asks Claude (MCP Client) something that is exposed by an MCP Server, the client can call the MCP server to retrieve data or execute tools, then use that information in its response.

Server 1: First Crack Detector MCP

In this section, we will briefly cover how the detector we have trained in the previous article is exposed as an MCP Server.

The following diagram illustrates how the components are exposed as an MCP Server:

Implementation Details

MCP SDK setup: Server initialisation using standard Python MCP SDK with stdio and SSE transports.
Tool definitions: start_detection, stop_detection, get_status with Auth0 role-based authorisation.
Session management: Thread-safe singleton pattern with idempotency enforcement.
Real-time monitoring: Streaming detection events via SSE for live status updates.
Error handling: Audio device enumeration failures, model loading issues, thread crashes, timeout scenarios.

Code Walkthrough

# Key implementation components:
# - MCP SDK decorators for tool registration
# - Auth0 JWT validation middleware
# - Session manager with thread-safe state
# - OpenTelemetry tracing integration

# First Crack Detection MCP Server setup
from mcp.server import Server
from mcp.types import Tool, TextContent

mcp_server = Server("first-crack-detection")

@mcp_server.list_tools()
async def list_tools() -> list[Tool]:
    return [
        Tool(
            name="start_first_crack_detection",
            description="Start monitoring audio for first crack events",
            inputSchema={
                "type": "object",
                "properties": {
                    "audio_source_type": {
                        "type": "string",
                        "enum": ["audio_file", "usb_microphone", "builtin_microphone"]
                    }
                },
                "required": ["audio_source_type"]
            }
        )
    ]

@mcp_server.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
    if name == "start_first_crack_detection":
        # Thread-safe session management
        result = session_manager.start_session(
            audio_config=AudioConfig(**arguments)
        )
        return [TextContent(
            type="text",
            text=json.dumps(result, indent=2)
        )]

Testing Approach

Custom test scripts: Python scripts using stdio communication (test_mcp_roaster.py)
Shell integration tests: Bash scripts for end-to-end workflows (test_roaster_server.sh)
Unit test coverage: 86 passing tests with pytest, mocking audio devices and model inference
Manual hardware testing: Real USB microphone validation with live roasting sessions

Server 2: Hottop Roaster Controller MCP

The Second MCP Server is to expose the status of the roaster and also to run commands to set heat, fan as well as start / stop commands for the roast.

This is implemented as a separate MCP server due to different hardware requirements which might require running these on different hosts.

The Hottop KN-8828B-2K+ Protocol

Initial approach with pyhottop library

Initially, we have attempted to use the pyhottop library for serial communication with the Hottop KN-8828B-2K+ roaster. However, we encountered compatibility issues that prevented reliable operation and had to consider an alternative approach.

Adapting to Artisan's protocol

After pyhottop proved unreliable, we analyzed the Artisan roasting software source code — a mature, widely-used open-source application for coffee roaster control. Being a user of Artisan's Hottop integration, I knew it has been working well for me. Given it has been battle-tested by the roasting community for a long time, it was an obvious next choice.

Warp Agent Mode has successfully analysed and adapted Artisan's serial protocol implementation.

Implementation Details

Serial connection management: USB serial at 115200 baud, continuous 0.3s command intervals (required by Hottop)
Command encoding: Artisan-compatible 36-byte protocol with checksums
Status parsing: Real-time temperature readings (bean + chamber) from serial responses
Input validation:
- Heat/fan values: 0-100% in 10% increments
- Connection state checks before commands
- Thread-safe state management with locks
Authentication: Auth0 JWT with role-based access control
- read:roaster - Status monitoring only
- write:roaster - Full hardware control
- Per-user audit logging

Code Architecture

# Key implementation layers:
# 1. MCP Server (sse_server.py) - Auth0 + tool definitions
# 2. SessionManager - Thread-safe orchestration
# 3. HardwareInterface - Artisan serial protocol
# 4. Continuous command loop - 0.3s intervals with temperature polling

Testing Strategy

MockRoaster: Realistic thermal simulation for development without hardware. Unfortunately, due to time restrictions, this has provided limited utility.
Hardware verification: Validated with physical Hottop KN-8828B-2K+ (October 2025).
Test coverage: 122 passing unit tests.
Manual test scripts: test_hottop_interactive.py, test_hottop_auto.py.
Integration tests: SSE transport, Auth0 authentication, command sequences.

Hardware verification

The implementation was verified with physical Hottop KN-8828B-2K+ hardware on October 25, 2025:

Drum motor control
Heat control (0-100%)
Fan control (0-100%)
Bean drop sequence
Cooling system
Continuous temperature readings (Bean & Chamber)

Security and Safety Considerations

Transport and Security Architecture

Why SSE (Server-Sent Events)?

Although both MCP servers currently run on the same machine as the agent (hint hint: stdio transport would be simpler), we designed for a distributed architecture from the start. The plan is to eventually deploy:

Roasting MCP servers need to run close to the hardware - Running on the machine physically connected to the roaster and microphone.
Agent on a separate device - A cloud server or different local machine for orchestration.

This approach means we could potentially use a low powered computer (such as RaspBerry PI) for the servers and also make it easier to plugh and start instead of having to use a laptop next to roaster every time.

SSE benefits: Real-time streaming, works over standard HTTP/HTTPS, firewall-friendly
MCP compatibility: Follows MCP specification for HTTP+SSE transport
Future-proof: Easy transition from localhost to remote deployment

Authentication and Authorisation with Auth0

Once MCP servers are exposed over the Internet / network, security becomes critical—especially for hardware control. This section briefly covers our approach for integrating Auth0 for authentication and authorisation.

Why Auth0?

Ease of integration: Well-documented SDKs and middleware
OAuth 2.0 Client Credentials: Perfect for machine-to-machine authentication
Role-based access control: Granular permissions via scopes
Bonus: Auth0 MCP Server for Warp Agent Mode to perform configuration tasks.

Security implementation:

JWT validation: Every MCP request validates Auth0 JWT tokens
Scope-based authorization:
- read:roaster - Status monitoring only (observer role)
- write:roaster - Full hardware control (operator role)
- admin:roaster - Administrative functions (future)
Token expiration: JWTs expire, requiring regular re-authentication

This architecture ensures that only authorised clients can control the roaster, with full traceability of who did what and when—essential for safety-critical hardware operations.

Lessons Learned

It has been a fun side project seeing how far Warp Agent Mode (Coding Agent) and emerging MCP Servers for developer documentation (Context 7) and service access (Auth0 MCP Server) cam in terms of speeding up development process.

It is still required have a clear architecture, requirements and a final picture before starting. When these are used in conjunction with Agentic development tools, tasks could take several days can be completed in a day or so.

What Worked Well and Lessons Learned

Standard MCP SDK: Provided solid foundation for both stdio and SSE transports.
SSE transport: Future-proofed the architecture for distributed deployment.
- Tested working with N8N, LangFlow and Python based local agents.
Auth0 integration: Straightforward OAuth 2.0 implementation with role-based access control.
Warp Agent Mode assistance: Accelerated MCP protocol understanding, test generation, and Auth0 middleware implementation.
Context7 and Auth0 MCP servers: Using MCP servers to build MCP servers (via Warp) streamlined development.
pyhottop library: Initial attempt failed, but pivoting to Artisan's proven protocol worked as expected.
MockRoaster simulation: Due to time restrictions, this was not explored sufficiently and ended up manually testing using local agents.
- This is needs to be revisited in the future especially to be able to test agent loop.

MCP-Specific Insights

Transport choice matters: SSE enables remote deployment but requires careful auth implementation.
Tool vs resource patterns: Tools for actions (hardware control), resources for data streams (status monitoring) so far working well with agents.
Idempotency is critical: Start/stop commands must be safe to call multiple times.
- Initial attempts caused hardware to go into start stop loop and required manual intervention.
Scope-based authorization: Fine-grained permissions (read:roaster vs write:roaster) essential for hardware safety.
- Although in the current integration mode, only a single client is used.
Observability: Using Open Telemetry helped troubleshooting issues quickly.
Testing strategies: Combination of unit tests (208 total), MockRoaster simulation, and manual hardware validation.

Real-World Usage Examples

Warp Agnt with Roaster Control (initial testing)

The following are sample prompts intended for testing the MCP server from Warp.

User: "Start the roaster"
Warp Agent Mode: [calls roaster_control MCP: start_roaster()]
         "Roaster started. Ready to add beans."

User: "Check the roaster status"
Warp Agent Mode: [calls roaster_control MCP: read_roaster_status()]
         "Status:
         • Drum: Running
         • Bean temp: 172°C
         • Chamber temp: 185°C
         • Heat: 0%
         • Fan: 0%"

User: "Stop the drum"
Warp Agent Mode: [calls roaster_control MCP: stop_roaster()]
         "Roaster drum stopped. Heat automatically turned off for safety."

What's Next: Part 3 Preview

Now that we have:

A trained first crack detection model (Part 1).
Built two MCP servers with hardware control + ML inference (Part 2).
Used Auth0 for security and SSE transport for distributed deployment (Part 2).

In Part 3: Building an Autonomous Roasting Agent, we'll bring it all together:

Comparing multiple Agent frameworks and assessing their suitability for such long running workflows.
- N8N
- LangFlow
- Python based
- Using Microsoft Agent Framework
- Using OpenAI Python SDK
Real-time decision making: Agent analyses temperature trends, RoR, and first crack to adjust roast.
Extending OpenTelemetry observability: Distributed tracing across agent + MCP servers + hardware.
Safety systems:
- Temperature bounds monitoring.
- Emergency stop on anomalies.
- Human override via UI.
Full end-to-end test: Press start -> Preheat -> Add beans -> Hands off -> First crack -> Development -> Drop -> (hope for) Perfect roast :)

The goal: An AI that roasts coffee consistently, safely, and (hopefully) better than manual control.

Resources

MCP & Standards:

Model Context Protocol Spec
Official MCP Python SDK - Used in this project

Project Code:

Coffee Roasting Repository - Complete source code
First Crack Detector MCP
Roaster Control MCP

Hardware & Serial Communication:

Artisan Roaster Scope - Source of Hottop protocol implementation
pyserial Documentation - USB serial communication
Hottop KN-8828B-2K+ - Roaster hardware

Authentication:

Auth0 Documentation - Identity provider
OAuth 2.0 Client Credentials - Machine-to-machine auth

Tools:

Warp Terminal - AI-assisted development environment
Auth0 MCP Server - Used via Warp for Auth0 configuration
Warp MCP Documentation
Context7 MCP Server - Used for documentation lookup during development

Part 1: Training a Neural Network to Detect Coffee First Crack from Audio - An Agentic Development Journey with Warp Agent

syamaner — Mon, 27 Oct 2025 20:59:05 +0000

When it comes to coffee, everyone has their preferences. I usually prefer smooth, naturally sweet coffee with nice fragrance - no bitter or smoky flavours.

There is a challenge though: achieving that perfect roast at home requires split-second timing. Miss the "first crack" by 30 seconds? You've got bitter, over-roasted beans. Finish the roast early? Enjoy your grassy / earthy tasting coffee.

This post is about teaching a neural network to detect that critical moment from audio alone.

While home roasting has been niche, over the recent years there are more options available for roasting coffee at home. These devices usually have smaller capacity ~ 250 / 500g and compact and lightweight enough to run over a counter.

To achieve my desired roast level I generally aim for a light / medium roast which requires development phase to be about 10% - 15% of the roast time. Development phase is the duration from the first crack starting until the end of roast where beans are ejected from the roaster.

First crack is the audible popping sound that occurs when coffee beans rapidly expand and release moisture and CO2 due to the buildup of internal pressure during roasting. Many light roast profiles end just after first crack begins, while medium roasts continue for 1-3 minutes beyond this point. On my setup, first crack typically begins around 170°C-180°C, and I aim to finish the roast at approximately 195°C. This gives me 1-3 minutes of development time after first crack starts. This value is based on my observations on a Hottop KN8828B-2K+ home roaster.

Detecting the First Crack event is important for the end goal as we need to adjust heat and fan from that point to slow down the roast and stretch the development phase.

The current series of posts will cover the following:

Training a Neural Network to Detect Coffee First Crack from Audio - An Agentic Development Journey
Part 2: Building an MCP server to control a home coffee roaster
Part 3: Building a Coffee roasting Agent with Aspire to automate coffee roasting

I have been recording coffee roasting audio during summer and have been looking into fine-tuning an existing model to be able to train and run inference on an arm based laptop. The task is performing binary classification on an audio stream to identify either first crack happened in the sample or not. For example a common baseline benchmark is making a random choice to predict class a or b (coin toss) which makes the baseline random performance 50% for any binary classification problem when using random guessing. Our goal is to beat this with minimal data available for fine-tuning.

My initial objective has been utilising a pre-trained AST (Audio Spectrogram Transformer) model from Hugging face that was originally trained on AudioSet and fine-tuning it for first crack vs not first crack binary classification task. In this approach, the model architecture remains the same, but we are updating the weights through training on our coffee roasting audio data.

To tackle this challenge systematically, I decided to leverage modern development tools and adopted an AI-first development approach. In the next section, the details of the setup will be discussed.

📊 TL;DR

Problem: Detect coffee "first crack" from audio to optimize roast profiles
Solution: Fine-tune MIT's AST model on 9 recording sessions
Results: 93.3% accuracy, 0.986 ROC-AUC with minimal data
Tools: Warp Agent, Label Studio, PyTorch, Hugging Face
Next: Part 2 builds MCP servers, Part 3 creates autonomous roasting agent

Why Automated First Crack Detection?

Manual first crack detection requires constant attention during a 10-12 minute roast. Environmental factors (noisy extractors, ambient sounds) can mask the cracks and pops.

This project aims to:

Free the roaster to multitask during the roast
Provide consistent detection regardless of ambient noise
Enable data-driven roast profile development
Lay groundwork for fully autonomous roasting (Part 3)

🤖 Warp Agent Contributions

Throughout development, Warp's Agent Mode:

Suggested Label Studio over manual Audacity annotation
Generated data preprocessing pipeline architecture
Created train, test eval split logic
Debugged overfitting with annotation strategy advice
Auto-generated evaluation scripts

Setting Up the Development Environment with Warp Agent

Having used Warp Agent Mode at work for the past few months and how it transformed my development flow, it was a natural choice for this project.

I have started with creating a readme file and shared my starting requirements and setup. I have included links to the tutorials of interest, the libraries I intend to use and the model I would like to use for fine-tuning.

Warp's Agent Mode helped me structure the project, suggest tools, and iterate on the implementation approach from training scripts, evaluation to inference and manual testing scripts.

Project Evolution and Documentation

The readme above was pretty much all I shared with Warp and then asked to focus on Phase 1 and create an implementation plan for Phase 1.

I have lost the recordings I made over summer and therefore I had to start with minimal data - only 4 recording sessions ~10 minutes each. This was enough to build the initial workflow with Warp.

Data Collection Strategy

For data collection, I have used a USB microphone pointed towards the roaster and recording each roasting session. A session takes about 10 - 12 minutes. At the time of starting, I only had 4 recording sessions available.

Recordings have the following properties:

Sample rate: 44.1kHz (recommended for compatibility)
Format: WAV (uncompressed)
Bit depth: 16-bit minimum
Channels: Mono sufficient
Recording duration: Full roast cycle (10-15 minutes)

Data Annotation and Labeling

When I started, I was intending to do manual annotation using a free and open source audio editor and recording application Audacity. However Warp Agent pointed me towards Label Studio and even provided the configuration snippets and described how to use it.

With the initial 4 recordings, I have used sparse labels and proceeded to training and evaluation. This has led to overfitting and the results were not reliable.

Initial Results with Sparse Labeling

With only 4 recording sessions and sparse annotation (marking only obvious first crack events), the model showed signs of overfitting:

Metric	Value	Issue
Validation Accuracy	100% (epochs 2-7)	Perfect scores = memorisation
Training Accuracy	100% (epochs 3-7)	No learning after epoch 3
Test Precision	75%	High false positive rate
Class Imbalance	15% / 85%	Severe imbalance

The problem: The model memorised the limited training data rather than learning generalisable acoustic features of first crack.

The solution: Expanding to 9 sessions with balanced annotation (equal first_crack and no_first_crack samples) dramatically improved precision from 75% → 95.2% while maintaining excellent recall.

As I had increased the recordings to 9, I spent more time annotating and aimed at building a balanced data with enough samples for first crack and no first crack. Each annotated sample was 3 - 6 seconds.

Warp Agent Mode has also provided the configuration snippet for Label Studio

<View>
  <Header value="Coffee Roast First Crack Detection"/>
  <Text name="instructions" value="Listen to the audio and mark regions where first crack occurs. Mark other regions as no_first_crack."/>
  <Audio name="audio" value="$audio" zoom="true" speed="true"/>
  <Labels name="label" toName="audio">
    <Label value="no_first_crack" background="#3498db"/>
    <Label value="first_crack" background="#e74c3c"/>
  </Labels>
  <TextArea name="notes" toName="audio" 
            placeholder="Optional notes about this region (e.g., 'very clear pops', 'subtle', etc.)"
            editable="true"/>
</View>

Data Preprocessing Pipeline

Coffee roasting is driven by many variables ranging from ambient temperature to the bean type, machine, Heating type and so on. A basic electric roaster like the one used here is slow to respond to change in controls as the heating element needs to warm up and cool down depending on the command. Accurately identifying the current phase of the roast is crucial and this can be done by audio analysis, visual, a combination of time and temperature to varying degree of success. In my manual roasts, recently I have been getting better results by adjusting the parameters once first crack is reached and therefore decided to fine tune a model to detect these.

So given we have a microphone pointing at the roaster during roasting process and a relatively controlled environment, how do we get the recording and convert it into the format needed to support our fine-tuning process.

Challenges:

Raw audio files are captured from multiple roasting sessions of varying length.
- Additionally ~ 10 previously recorded sessions lost accidentally.
First crack events are sparse. Happens around about 12-25% of the whole duration. And they are also not continuous.
- This leads to an imbalance in samples.
We need a workflow and a pipeline to process these and end up with a balance dates for training evaluation and test.
At the beginning we also have a limited number of sessions recorded (9 at the time of writing)
Labelling should be easy and repeatable to avoid user errors.

Labelling Process

While the fine tuning approach, and the base model was instructed to Warp, Label Studio was not in the original requirements. And Warp has not only recommended using Label Studio but also provided detailed steps about running and configuring and get going. These worked out of the box.

┌─────────────────────────────────────────────────────────────────┐
│                    Label Studio (Web UI)                        │
│          Manually annotate audio files                          │
│          Mark "first crack" / "not first crack" time regions    |
└────────────────────────┬────────────────────────────────────────┘
                         │
                         │ Export JSON
                         ▼
        📄 project-1-at-2025-10-18-20-44-9bc9cd1d.json
                         │
                         │
    ╔════════════════════▼═══════════════════════════════════════╗
    ║  STEP 1: convert_labelstudio_export.py                     ║
    ║  • Strip hash prefixes from filenames                      ║
    ║  • Compute audio durations                                 ║
    ║  • Extract labeled time regions from the raw files         ║
    ║  • Output one JSON per audio file                          ║
    ╚════════════════════╦═══════════════════════════════════════╝
                         │
                         ▼
              📁 data/labels/
              ├── roast-1.json
              ├── roast-2.json
              └── roast-3.json
                         │
                         │
    ╔════════════════════▼═══════════════════════════════════════╗
    ║  STEP 2: audio_processor.py                                ║
    ║  • Read annotation JSONs                                   ║
    ║  • Load raw audio files (44.1kHz mono)                     ║
    ║  • Extract time segments (start→end)                       ║
    ║  • Save chunks as WAV files by label                       ║
    ║  • Generate processing_summary.md                          ║
    ╚════════════════════╦═══════════════════════════════════════╝
                         │
                         ▼
              📁 data/processed/
              ├── first_crack/
              │   ├── roast-1_chunk_000.wav
              │   └── roast-1_chunk_001.wav
              └── no_first_crack/
                  ├── roast-1_chunk_002.wav
                  └── roast-2_chunk_000.wav
                         │
                         │
    ╔════════════════════▼═══════════════════════════════════════╗
    ║  STEP 3: dataset_splitter.py                               ║
    ║  • Collect all chunks by label                             ║
    ║  • Train, validation and test split                        ║
    ║    (70% train, 15% val, 15% test)                          ║ 
    ║  • Copy files to split directories                         ║
    ║  • Generate split_report.md                                ║
    ╚════════════════════╦═══════════════════════════════════════╝
                         │
                         ▼
              📁 data/splits/
              ├── train/     (70%)
              │   ├── first_crack/
              │   └── no_first_crack/
              ├── val/       (15%)
              │   ├── first_crack/
              │   └── no_first_crack/
              └── test/      (15%)
                  ├── first_crack/
                  └── no_first_crack/

Once the steps above are complete, we are ready for training and evaluation.

Dataset Overview

Total Samples: 298 chunks from 9 roasting sessions

Overall Class Balance

Class	Count	Percentage	Avg Duration
first_crack	145	48.7%	4.5s
no_first_crack	153	51.3%	4.0s

Split Distribution

Split	Total Samples	first_crack	no_first_crack	Split Ratio
Train	208	101 (48.6%)	107 (51.4%)	69.8%
Validation	45	22 (48.9%)	23 (51.1%)	15.1%
Test	45	22 (48.9%)	23 (51.1%)	15.1%

Class Balance Across Splits

Class	Train	Validation	Test	Total
first_crack	101	22	22	145
no_first_crack	107	23	23	153

Per-Session Breakdown

Recording Session	first_crack	no_first_crack	Total	Balance
25-10-19_1103-costarica-hermosa-5	13	14	27	48.1% / 51.9%
25-10-19_1136-brazil-1	19	19	38	50.0% / 50.0%
25-10-19_1204-brazil-2	20	15	35	57.1% / 42.9%
25-10-19_1236-brazil-3	18	17	35	51.4% / 48.6%
25-10-19_1315-brazil4	15	14	29	51.7% / 48.3%
roast-1-costarica-hermosa-hp-a	16	17	33	48.5% / 51.5%
roast-2-costarica-hermosa-hp-a	16	19	35	45.7% / 54.3%
roast-3-costarica-hermosa-hp-a	13	19	32	40.6% / 59.4%
roast-4-costarica-hermosa-hp-a	15	19	34	44.1% / 55.9%

Key Observations:

Nearly balanced dataset (48.7% vs 51.3%)
Stratified split maintains balance across train/val/test
9 recording sessions, mix of Costa Rica and Brazil beans
Average chunk duration: 4.2 seconds
Total annotated audio: ~21 minutes

Evaluation Metrics

For this binary classification task, we use multiple metrics to evaluate the model performance:

Accuracy

The proportion of correct predictions (true positives and true negatives) among all predictions.

Accuracy = (TP + TN) / (TP + TN + FP + FN)

This metric provides an overall sense of model correctness. However, accuracy alone can be misleading with imbalanced datasets.

Precision

Of all samples predicted as first_crack, what proportion actually were first crack events?

Precision = TP / (TP + FP)

High precision means fewer false alarms. Critical when we don't want to prematurely adjust roaster settings based on incorrect detections.

Recall (Sensitivity)

Of all actual first_crack events, what proportion did the model correctly identify?

Recall = TP / (TP + FN)

High recall means we catch most first crack events. Missing first crack (false negative) is likely to result in over-roasting.

F1 Score

The harmonic mean of precision and recall, providing a single balanced metric.

F1 = 2 × (Precision × Recall) / (Precision + Recall)

Balances precision and recall. Useful when both false positives and false negatives are costly.
In case of roasting, these could mean under roasted or dark roast which is not desirable from this project perspective.

ROC-AUC (Area Under the Receiver Operating Characteristic Curve)

Measures the model's ability to distinguish between classes across all classification thresholds.

Confusion Matrix

The confusion matrix visualises the model's predictions versus actual labels:

                    Predicted
                    first_crack  no_first_crack
Actual  first_crack      TP            FN
        no_first_crack   FP            TN

Where:

TP (True Positive): Correctly predicted first crack
TN (True Negative): Correctly predicted no first crack
FP (False Positive): Predicted first crack, but was actually no first crack (false alarm)
FN (False Negative): Predicted no first crack, but was actually first crack (missed detection)

Training and Evaluation

With our dataset properly split and balanced and or metrics defined, we're ready to fine-tune the Audio Spectrogram Transformer (AST) model for first crack detection.

Model Architecture

The project uses MIT's pre-trained AST model (MIT/ast-finetuned-audioset-10-10-0.4593) from Hugging Face, which was originally trained on AudioSet. The model architecture:

Input: Audio spectrograms (16kHz, 10-second windows)
Architecture: Vision Transformer adapted for audio
Transfer Learning: We keep the pre-trained weights and fine-tune for binary classification
Output: Two classes - first_crack vs no_first_crack

Training Configuration

The training process uses the following configuration (defined in models/config.py):

TRAINING_CONFIG = {
    'batch_size': 8,
    'learning_rate': 1e-4,
    'num_epochs': 50,
    'device': 'mps',  # Apple Silicon GPU
    'sample_rate': 16000,
    'target_length_sec': 10.0
}

Key training features:

Class-weighted loss: Addresses class imbalance
AdamW optimizer: With cosine annealing learning rate schedule
Early stopping: Based on validation F1 score
TensorBoard logging: Real-time metrics visualization

Training Process

To start training:

./venv/bin/python src/training/train.py \
  --data-dir data/splits \
  --experiment-name baseline_v1

The training script:

Loads train/val data using AudioDataset (automatic resampling to 16kHz)
Applies class weights to handle imbalance
Trains with early stopping (patience: 10 epochs)
Saves best model based on validation F1 score
Writes checkpoints to experiments/runs/<experiment_name>/

Results: Exceeding Expectations

With only 9 recording sessions (~21 minutes of annotated audio):

Metric	Baseline (Random)	Our Model	Improvement
Accuracy	50.0%	93.3%	+86.6%
Precision	50.0%	95.2%	+90.4%
Recall	50.0%	90.9%	+81.8%
F1 Score	50.0%	93.0%	+86.0%
ROC-AUC	0.50	0.986	+97.2%

Translation: The model correctly identifies first crack 93 times out of 100,
with only 1 false alarm and 2 missed detections across the test set.

Confusion Matrix
                    Predicted
                    no_first_crack  first_crack
Actual  no_first_crack     22            1
        first_crack         2           20

This is excellent performance for a model trained on just 9 recording sessions! The higher overlap (70% vs previous experiments) likely contributed to the improved results. This demonstrates the power of transfer learning with pre-trained audio models.

Performance breakdown:
• Only 1 false alarm (FP) - down from 2
• Only 2 missed detections (FN) - same as before
• 22/23 correct no_first_crack predictions (95.7%)
• 20/22 correct first_crack predictions (90.9%)

This balanced performance is crucial for real-time roasting control where both missing first crack and triggering false adjustments have consequences.

Evaluation on Test Set

To evaluate the final model:

./venv/bin/python src/training/evaluate.py \
  --checkpoint experiments/final_model/model.pt \
  --test-dir data/splits/test

This generates:

Classification report with per-class metrics
Confusion matrix visualization
ROC curve analysis
Detailed results saved to text files

Key Learnings

What Worked:

Transfer learning from AudioSet significantly reduced data requirements
Balanced annotation (equal first_crack/no_first_crack samples) improved performance
10-second windows captured enough context for accurate detection
Class-weighted loss handled remaining imbalance effectively

Challenges:

Initial sparse labelling with only 4 sessions led to overfitting
Limited training data (9 sessions) required careful annotation strategy
Environmental noise kept to a minimum under a controlled environment

Future Improvements:

Collect more diverse roasting sessions (different beans, temperatures, extractor configuration)
Experiment with data augmentation (time stretching, pitch shifting)
Test shorter inference windows for faster real-time detection

Real-Time Inference

The trained model can now detect first crack in real-time from either audio files or live microphone input:

# File-based detection
./venv/bin/python src/inference/first_crack_detector.py \
  --audio data/raw/roast-1.wav \
  --checkpoint experiments/final_model/model.pt

# Live microphone detection
./venv/bin/python src/inference/first_crack_detector.py \
  --microphone \
  --checkpoint experiments/final_model/model.pt

The detector uses sliding window inference with "pop-confirmation" logic:

Analyzes 10-second audio windows with 70% overlap (3-second hop between windows)
Requires minimum of 3 positive detections (pops) within a 30-second confirmation window
Maintains detection history to filter false positives
Returns timestamp in MM:SS format when first crack is confirmed

This forms the foundation for Part 2, where we'll wrap this detector in an MCP server for integration with AI agents.

Real-Time Performance

Hardware: Apple M3 Max (MPS - Metal Performance Shaders)

Speed Metrics

Metric	Value
Real-Time Factor (RTF)	87.64x
Per-window Latency	70-90ms
Throughput	~18 windows/second
Processing Speed	1 hour of audio in ~41 seconds

Batch Inference Results

File	Duration	Processing Time	RTF
roast-1	10:39 (639.7s)	7.67s	83.46x
roast-2	10:16 (616.6s)	6.92s	89.06x
roast-3	10:25 (625.6s)	7.05s	88.74x
roast-4	9:44 (584.8s)	6.55s	89.29x
Total	41.1 min	28.2s	87.64x

Latency Breakdown (per 10s window)

Audio loading: 1-2ms
Feature extraction: 20-30ms
Model inference: 50-60ms
Total: ~70-90ms per window

Resource Usage

CPU Usage: 5-10% during inference
Memory: ~1.5GB for model + 100MB working
GPU Memory: ~2GB on MPS
Latency overhead: 0.9% (90ms used / 10,000ms available)

Key Insight: The model processes audio 87x faster than real-time, providing a 111x headroom for real-time streaming detection. A 10-minute roast is fully processed in just ~7 seconds, making real-time monitoring easily achievable even with additional processing overhead.

The Warp Agent Advantage

Throughout this project, Warp's Agent Mode was instrumental in:

Rapid Prototyping - From idea to working pipeline in hours, not days

Best Practice Guidance - Suggested Label Studio and evaluation workflows
Code Generation - Created complete scripts for data processing, training, and inference

Iterative Refinement - Helped debug overfitting issues and improve annotation strategy

Documentation - Generated summaries, reports, and README documentation automatically

The development workflow felt more like pair programming with an engineer who knew PyTorch and audio processing.

What's Next?

The first crack detector is working well, but it's just the beginning.

In Part 2: Building MCP Servers for Coffee Roasting, we'll:

Wrap the detector in an MCP server for real-time streaming
Build a second MCP server to control the Hottop roaster (heat, fan, cooling)
Implement authentication and safety controls
Test end-to-end detection → action loop

In Part 3: Creating an Autonomous Roasting Agent, we'll bring it all together:

Use .NET Aspire to orchestrate multiple MCP servers
Build AI agents that make real-time roasting decisions
Implement safety rails and human override
Roast a batch fully autonomously and compare against manual profiles

The goal: Press start, add beans when ready, hand off and observe and enjoy perfectly roasted coffee.

Follow along on GitHub or subscribe for Part 2!

Resources

Project & Tools:

Project Repository - Complete code and documentation
Warp Terminal - AI-assisted development environment
Label Studio - Audio annotation tool

Model & ML:

Audio Spectrogram Transformer (AST) - Pre-trained model
AST Documentation
Fine-Tuning AST Tutorial
Original AST Paper - Gong et al., 2021

Audio Processing:

LibROSA - Audio analysis library
PyTorch Audio - Audio I/O and transforms

Coffee Roasting Context:

First Crack Explained - For readers unfamiliar with roasting

Beyond Basic RAG: Measuring Embedding and Generation Performance with RAGAS

syamaner — Sat, 12 Apr 2025 14:33:03 +0000

Introduction

In the previous post, we looked at a basic Retrieval Augmented Generation (RAG) example using .Net for both retrieval and generation. This is built using out of the box components offered by Semantic Kernel and used an out of the box chunking approach.

The barriers of entry to achieve this is low which helps to democratise access to Large Language Models (LLMs) in wider ecosystems and drive innovation. For instance; in .Net, it is possible to use Microsoft Semantic Kernel or Aspire.Azure.AI.OpenAI (OpenAI, Azure OpenAI as well as compatible local options such as Ollama). There is even an emerging open source .NET port of Langchain with JetBrains being an official supporter. For those who would like to run inference in process (CPU or GPU) without HTTP APIs, there is also LLamaSharp which is a .Net wrapper around llama.cpp supporting CPU and GPU inference.

However, given that there are many parameters / tweaks to ingestion, retrieval and generation, how can we measure the quality and outcome when building such applications?

The following Google Trends chart compares search terms LLM (Blue), RAG (green), RAG Evaluation (Red) and langchain (Yellow) between January 2023 and April 2025.

We observe that:

Earlier in 2023, RAG was a more popular search term.
Around January 2024, LLMs started to takeover in popularity.
langchain remained in a steady position during the time frame.
RAG Evaluation has negligible existence in the trends.

Given Google is a public and general purpose search engine, the results do not mean there is no interest in evaluation but the general public may not be thinking about these aspects yet.

Another look from academic papers perspective comparing "RAG" and "RAG Evaluation" yields different results. The publications has grown from 14 papers on "RAG" / 3 papers on "RAG Evaluation" during 2022 to 1041 on "RAG" / 454 on "RAG Evaluation" in 2024 (Source: ArXiv Trends, 2025). As RAG topic becomes mainstream and popular, the evaluation methods also become a popular topic of research.

This post will cover the following sections:

RAG Evaluation
System under Evaluation
RAGAS
Evaluation Approach
Results
Conclusion

RAG Evaluation

Evaluating Retrieval-Augmented Generation (RAG) systems is a crucial aspect of solutions that incorporate such technologies.

Unlike traditional software where testing involves a deterministic process (given the input, we know the expected outcome), RAG outputs depend on two probabilistic / non deterministic components:

Retrieval accuracy (finding relevant source data, rewriting user query, and similar approaches)
Generation quality (producing coherent, factual responses)
Variation in ingestion and generation: Chunking strategies, using metadata or not, tweaking / versioning prompts or inference parameters.

Without systematic evaluation:

How can we tell the difference in results from random noise?
Hallucinations and irrelevant answers can go undetected.
Optimisation by guesswork: Let me change this parameter and see.
How do we deal with regressions?
Cost / benefits.
- Given runtime costs include input / output tokens, for production applications, these are also crucial metrics.
- If these are not included evaluation and comparison, a well performing system that might be too expensive to run could be the end result. Given this is a hobby project, this step is excluded from the current experiments.

There are established data sets and benchmarks for Retrieval Augmented Generation systems. These benchmarks typically include datasets for tests including the query, expected answer, expected context. These are then used against the RAG system under test to evaluate the results using various metrics as defined below. Google Frames Benchmark is one example that provides dataset based on Wikipedia Articles to evaluate. metrics such as factuality, retrieval accuracy, and reasoning.

These approaches introduce some challenges as following:

Such datasets are generic and do not necessarily factor domain specific nuances in the target use case.
It is possible that the test data might have been included in the training data set.
There can be bias agains specific metrics.

In this post, we will focus on how to measure both retrieval performance (e.g., Context Precision) and generation quality (e.g., faithfulness, semantic similarity) using RAGAS evaluation framework.

We will be utilising RAGAS and LLM as judge approach to generate evaluation data from our documents and then run evaluation using Jupyter Notebooks running on Aspire to see the results.

System under Evaluation

We are ingesting Markdown documentation from official Microsoft .NET Aspire repository.
Using Semantic Kernel for ingestion and a very simple chunking approach.
Using Semantic Kernel for search.
We register a dedicated Qdrant Vector store for each embedding model we use for evaluation. We also register an embedding model with semantic kernel for each model we will evaluate.
Lastly we also register chat completion models for each LLM we are evaluating using model name as key.

This approach allows us to use correct vector store at runtime for ingestion and retrieval depending on the request parameters used. This also ensures the request can select the LLMs for generation aspect when we are running evaluation.

System Overview

Ingestion and Query

RAGAS

Ragas is one of the libraries that simplify the evaluation of Large Language Model (LLM) applications. It provides the necessary tools to generate test data as well as evaluate the results using various approaches and metrics.

RAGAS Metrics

In this section a brief overview of the metrics used in this post will be provided. For more details, please refer to RAGA Metrics documentation.

The following metrics are summarised from official RAGAS documentation metrics section.

Semantic Similarity

Measures the similarity between the answer from the LLM and the reference answer in the test dataset.

Starts with the answer embeddings and the reference embeddings. Then computes cosine similarity between the two vectors.

Answer Relevancy

Answer relevancy measures how relevant the response is to the user input. This is calculated as the following:

Using an LLM, and the response under evaluation, generate a set of (3) artificial questions.
Compute cosine similarity between the embedding of the user input and the embedding of the generated questions.
Average of the scores will determine the Answer Relevancy.

Factual Correctness

Factual Correctness metric is used to evaluate the factual accuracy of the generated response against the reference. This metric uses the LLM to first break down the response and reference into claims and then uses natural language comparison to determine the factual overlap between the response and the reference. This overlap is quantified using precision, recall, and F1 score.

Precision: Measured number of positive predictions that were correct. This metric is higher when there are low false positive.

Recall: Measured how many of the actual positives were correctly identified. Recall is higher when false negatives are low.

F1: When the difference between Precision and Recall is large, F1 score can be used to balance. F1 score will be closer to the lower of two other metrics. F1 can only be high if both precision and recall are high.

Faithfulness

Faithfulness metric can be used to measure how factually consistent a response is with the retrieved context.

A faithful response is a response where all claims included in the response are consistent with the retrieved context from vector store.

This is a measure that addresses the hallucination detection. If the provided answer can be backed up by context, then it means there are no additions from the generative model.

Context Recall

Context recall measures the number of relevant documents retrieved from the vector store. If retrieval ensures that no important information is missed at this stage, then the Context Recall is considered high.

First, the reference from the evaluation dataset is broken down into claims. Then each claim in the reference answer is analysed to determine whether it can be attributed to the retrieved context or not. Ideally, all claims in the reference answer should be attributable to the retrieved context.

Evaluation Approach

Our evaluation approach involves using multiple embedding and generation models. So the process goes as following:

Configuration

Full evaluation dataset.
Embedding models:
- mxbai-embed-large (335M parameters, Ollama local)
- text-embedding-3-large OpenAI
For each embedding model, we evaluate using the following generative models:
- phi4 (14B parameters)
- chatgpt-4o-latest
- qwen2.5:32b
- deepseek-r1 (7B parameters)
- llama3.2 (3B parameters)
- llama3.3 (70b parameter)
- deepseek-r1:70b
- gemma3:12b
- mistral-small3.1 (24B parameters)
- gemma3 (4B parameters)
- gemma3:27b
We then save evaluation results a csv file.

Test Data Generation (RAGAS, GPT-4o, Jupyter Notebooks)

The selected approach is LLM As a judge and we use RAGAS to generate test dataset from our documents (.NET Aspire documentation)

We use TestsetGenerator class from RAGAS as documented in basic usage. The steps are:

Load our documents.
Define the personas to be used for generating queries (technical, novice, expert, ...)
Declare the distribution for question types (simple, complex or reasoning)
Generate test datasets.

We are using GPT-4o for generative model and text-embedding-ada-002 (default) for the embedding model. This is based on the assumption that using state of the art models will provider better quality test data generation.


# Initialise personas and generator

#https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_persona_generator/#personas-in-testset-generation

personas = [
    Persona(
        name="Technical Analyst",
        role_description="Focuses on detailed system specifications and API documentation"
    ),
    Persona(
        name="Novice User",
        role_description="Asks simple questions using layman terms and basic functionality"
    ),
    Persona(
        name="Security Auditor",
        role_description="Focuses on compliance, data protection, and access control aspects"
    ),
    Persona(
        name="Docker expert",
        role_description="Has in depth experience with Docker and DSocker compose and expert at cloud native concepts"
    )
]

generator_llm = LangchainLLMWrapper(ChatOpenAI(model=openai_model, temperature=0.1))  
generator_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())
generator = TestsetGenerator(
    llm=generator_llm,
    embedding_model=generator_embeddings,
    persona_list=personas
)

# Initialise query distribution and generate dataset.

# https://docs.ragas.io/en/stable/references/synthesizers/

from ragas.testset.synthesizers import (
    SingleHopSpecificQuerySynthesizer,
    MultiHopAbstractQuerySynthesizer,
    MultiHopSpecificQuerySynthesizer
)
query_distribution = [
    (SingleHopSpecificQuerySynthesizer(), 0.4),  # Simple questions
    (MultiHopSpecificQuerySynthesizer(), 0.4),   # Complex questions
    (MultiHopAbstractQuerySynthesizer(), 0.2)    # Reasoning questions
]

dataset = generator.generate_with_langchain_docs(
    docs,
    testset_size=100,
    query_distribution=query_distribution
)

Generated test dataset contains the following columns:

user_input : The question we will pass to our RAG system.
reference_contexts: The reference context that would be retrieved in the ideal case for the given question.
reference: The ideal response to the user input.
synthesizer_name: The type of synthesiser used to generate the row. (single hop, multiple hop abstract, multi hop specific.

SingleHopSpecificQuerySynthesizer

RAGAS uses a knowledge graph based approach to passing the documents to create test dataset. A single hop specific query synthesiser will use only one node from the graph(headlines or key phrases) to generate query. Single hop in this context means using a single node from a knowledge graph built from the input document.

Example question:

"How does .NET Aspire manage launch profiles for ASP.NET Core service projects?"

MultiHopSpecificQuerySynthesizer

Similar to previous synthesiser, this also uses specific properties. However this is achieved by using multiple chunks that overlap with each other. These would require the retrieval process to be able to retrieve multiple documents or sections to build the expected context.

Example question:

How does the Azure SDK impact the ability to run Azure services locally in containers and provision infrastructure using .NET Aspire?

MultiHopAbstractQuerySynthesizer

Intends to provide generalised (abstract) queries using multiple notes of the document.

Example question:

How can a custom command be created and tested in .NET Aspire to clear the cache of a Redis resource?

Generated test data available via Github Repository

The notebook to generate the test data is also accessible via GitHub Repository

RAG Pipeline (.NET)

Ingestion:
- For each Embedding model:
- Create a vector store for persistence.
- Generate embeddings using the current Embedding model and add them to the given Vector store matching the embedding model name.
Retrieval and Generation
- Request specifies embedding model and generation model
- Retrieve using the vector store named after the requested embedding model
- Use the contest with the desired Generative model from the request.
- Semantic Kernel simplified this by allowing keyed registration support multiple embedding and chat models via .NET Dependency Injection.

The vector stores for embedding models can be seen below:

Evaluation Run

This is achieved using Python as RAGAS is a Python library. Evaluation is performed as following:

Pick n (5) random entries from eval dataset
For each embedding model
- For each generative model
- Call our API to get embeddings (context) using vector search
- Call our API to run RAG query and return answer.
- Set the retrieved_contexts and response in the eval dataset.
- Generate the data and run evaluation.

RAG Evaluation Dataset Structure

Column Name	Description
user_input	Generated question used to query the system
reference_contexts	Reference context documents generated prior to evaluation
retrieved_contexts	Actual context documents returned from the API during runtime
reference	Reference answer generated prior to evaluation (ground truth)
response	Actual response generated by the RAG system during evaluation
embedding_model	The embedding model used for retrieval (e.g., text-embedding-3-large)
chat_model	The generative model used to produce the final response (e.g., ChatGPT-4o)

code example:

# Retrieve context for evaluation (vector search) using eval input from the current dataset row.

endpoint = f"{base_url}/vector-search"
params = {
    "query": query,
    "embeddingModel": embedding_model
}

# RAG query using the eval query, current embedding model and current generative model

endpoint = f"{base_url}/chat-with-context"

params = {
    "query": query,
    "embeddingModel": embedding_model,
    "chatModel": chat_model
}

Full code available at GitHub Repository](https://github.com/syamaner/moonbeans/blob/bulk-performance_evaluation/src/AspireRagDemo.AppHost/Jupyter/Notebooks/evaluation.ipynb)

Results

Embedding and LLM Model Performance Comparison

Edit 14/04/2025 - Results for the full dataset with 101 questions and answers

As seen below, when using the full evaluation dataset (101 questions and answers), top performers for faithfulness are:

text-embedding-3-large + gemma3: 80.42%
mxbai-embed-large + chatgpt-4o-latest: 79.37%
text-embedding-3-large + phi4: 79.28%

In above cases, we have at least one component that is not open source. However, this also means we can optimise our choices on models if needed finding a hybrid approach.

Considering top performers for semantic similarity, OpenAI embeddings come on top. However, open source chat models can be competitive when combined with text-embedding-3-large as seen below.

text-embedding-3-large + gemma3:12b: 93.68%
text-embedding-3-large + deepseek-r1:70b: 93.64%
text-embedding-3-large + phi4: 93.59%

Observations

text-embedding-3-large generally achieves higher semantic similarity scores compared to mxbai-embed-large
The combination of embedding model and chat model significantly impacts performance metrics
Larger models don't always outperform their smaller counterparts (e.g., gemma3:12b vs gemma3:27b)

Embedding Model	Chat Model	Faithfulness Score	Semantic Similarity
text-embedding-3-large	gemma3	80.42%	93.41%
mxbai-embed-large	chatgpt-4o-latest	79.37%	92.71%
text-embedding-3-large	phi4	79.28%	93.59%
mxbai-embed-large	deepseek-r1	78.62%	92.42%
mxbai-embed-large	qwen2.5:32b	78.26%	93.00%
text-embedding-3-large	gemma3:12b	78.21%	93.68%
text-embedding-3-large	llama3.2	77.85%	93.46%
mxbai-embed-large	gemma3	77.83%	92.22%
mxbai-embed-large	gemma3:12b	77.73%	92.98%
text-embedding-3-large	llama3.3	77.64%	93.58%
mxbai-embed-large	gemma3:27b	76.63%	92.44%
mxbai-embed-large	llama3.3	76.59%	92.91%
text-embedding-3-large	deepseek-r1:70b	76.46%	93.64%
mxbai-embed-large	mistral-small3.1	76.40%	93.01%
text-embedding-3-large	deepseek-r1	75.76%	93.18%
mxbai-embed-large	llama3.2	75.29%	92.38%
mxbai-embed-large	deepseek-r1:70b	75.19%	92.60%
text-embedding-3-large	chatgpt-4o-latest	74.56%	93.56%
text-embedding-3-large	gemma3:27b	74.53%	92.83%
text-embedding-3-large	qwen2.5:32b	74.25%	93.40%
text-embedding-3-large	mistral-small3.1	74.12%	93.10%
mxbai-embed-large	phi4	73.50%	92.28%

Key Takeaways

Open source models can be competitive

mxbai-embed-large outperformed OpenAI's premium text-embedding-3-large + chatgpt-4o-latest combination in faithfulness metrics.
Small models can be mighty

Local 3B-parameter models like llama3.2 achieved 93.46% semantic similarity, proving even quantised and smaller models have recently become more powerful.
The Hidden Cost of Accuracy

OpenAI's top-performing combination (text-embedding-3-large + ChatGPT-4o) had 74.56% faithfulness vs. open source alternatives 80.42% - a critical tradeoff between commercial API provider costs and local precision.
Microsoft is investing on AI with .Net Platform

We can now build a RAG system and achieve decent performance using nearly out of the box implementation. .NET Aspire takes this even further giving a flexible local development environment where we can mix and match the hosts for local inference as a container, host machine or over local network.

Future

The results are based on out of the box code for ingestion, generation as well as test data generation and first step towards establishing a baseline.

Given the evaluation process uses OpenAI API, it is costly to perform evaluation on a large dataset for hobby purposes. The next steps I would ideally follow are:

Start experimenting with prompts and versioning.
Run evaluation again.
Consider further tests using different chunking, retrieval / reranking strategies and compare the results.
If using a local llm as a judge proves effective, carry on the experiments using local models on a larger scale.

Jupyter AI & .NET Aspire: Building an LLM-Enabled Jupyter Environment

syamaner — Mon, 17 Feb 2025 20:58:44 +0000

In this post, we will cover installing and configuring Jupyter AI with Jupyter while driving configuration from .NET Aspire. The approach documented here provides out of the box solution using Jupyter AI without having to manually set it up.

What we will cover?

What is Jupyter AI?
Adding Jupyter AI to Jupyter image
Adding Microsoft.dotnet-interactive Jupyter kernel to add c# support to Jupyter Notebooks
.NET Aspire Configuration to specify code and embedding models / model providers
Running the custom image using .NET Aspire
And a quick Python demo

Jupyter AI

Jupyter AI is an extension that adds generative AI support to JupyterLab. With this extension it is possible to use the chat interface to ask about our code in Jupyter Notebooks, the source files and documentation accessible. It can also be used to generate code and inject to a cell.

To get Jupyter AI working the following steps are necessary:

Install and activate the extension.
Configure the embedding and language model (model name, endpoint, api keys if needed)
Access the extension and interact with it

Installing Jupyter AI and .NET Interactive kernel using a custom Dockerfile

In this section we will cover the Dockerfile used as well as the configuration via custom entry point script.

Creating the Dockerfile

This part is straightforward. We start with an appropriate Jupyter base image and then:

Install .NET 9
Install Python dependencies using requirements.txt
Install Microsoft.dotnet-interactive (so we can install .Net Interactive Kernel)
Copy our entry point file (run.sh)
Call entry point as a non root user

Yes with this minimal Dockerfile we have JupyterLabs server, Jupyter AI and even .Net Interactive Kernel allowing us using C#, F# and even PowerShell in nor notebooks.

FROM jupyter/base-notebook:ubuntu-22.04
ENV PYTHONDONTWRITEBYTECODE=1

USER root
RUN apt-get update && apt-get install software-properties-common cmake build-essential  libc6  -y

RUN add-apt-repository ppa:dotnet/backports \
    && apt-get update \
    && apt-get install -y dotnet-sdk-9.0  libgl1-mesa-dev  libglib2.0-0 \
    && apt-get clean && rm -rf /var/cache/apt/archives /var/lib/apt/lists/*

USER ${NB_UID}

RUN dotnet tool install -g Microsoft.dotnet-interactive
ENV PATH="${PATH}:/home/jovyan/.dotnet/tools"

COPY ./requirements.txt /home/jovyan/requirements.txt
RUN pip install --no-cache-dir -r /home/jovyan/requirements.txt
RUN dotnet interactive jupyter install

USER root
COPY ./run.sh /home/jovyan/run.sh
RUN chmod +x /home/jovyan/run.sh

USER ${NB_UID}

ENTRYPOINT ["/home/jovyan/run.sh"]

Configuring Jupyter AI via entry point script

This is achieved by our run.sh file as following:

We know that .NET Aspire injects the connection strings and additional config as environment variables. So we will utilise this:

Pass '' as token so we do not have auth for local Jupyter server.
We know that Jupyter AI extension is configured at startup by passing --AiExtension.* arguments to Jupyter lab command.
- We inject relevant --AiExtension.* arguments as passed from our Aspire host using environment variables.
- Pass EMBEDDING_MODEL and CODE_MODEL as language and embedding models using relevant arguments.
- Optionally set Embedding and Language model urls (if not using OpenAI).
- Inject the API Keys (required ig using OpenAI)
Execute the built entry command.

#!/bin/bash
CODEMODELURL=$(echo "${ConnectionStrings__codemodel}" | cut -d'=' -f2 | cut -d';' -f1)
EMBEDDINGMODELURL=$(echo "${ConnectionStrings__embeddingmodel}" | cut -d'=' -f2 | cut -d';' -f1)

# Base command
CMD="jupyter lab --NotebookApp.token=''"
echo "code model: $CODEMODELURL"
echo "embedding model: $EMBEDDINGMODELURL"

# Add embedding model
CMD="$CMD --AiExtension.default_embeddings_model=$EMBEDDING_MODEL"
# Add code model
CMD="$CMD --AiExtension.default_language_model=$CODE_MODEL"
CMD="$CMD --AiExtension.default_api_keys='{\"OPENAI_API_KEY\":\"${OPENAI_API_KEY}\", \"HUGGINGFACEHUB_API_TOKEN\":\"$HUGGINGFACEHUB_API_TOKEN\"}'"
CMD="$CMD --AiExtension.default_max_chat_history=12"
#,

# Add embedding model URL if specified
if [ ! -z "$EMBEDDINGMODELURL" ]; then
    CMD="$CMD --AiExtension.model_parameters $EMBEDDING_MODEL='{\"base_url\":\"$EMBEDDINGMODELURL\"}'"
fi

# Add code model URL if specified
if [ ! -z "$CODEMODELURL" ]; then
    CMD="$CMD --AiExtension.model_parameters $CODE_MODEL='{\"base_url\":\"$CODEMODELURL\"}'"
fi

# Execute the command
eval "$CMD"

The entry command above will provide us the following when we run out Aspire host:

.NET Aspire configuration and execution

In this section, we will cover the structure of launchSettings.json and the Aspire code putting it all together.

Configuration

Provided example has 3 profiles as following:

http-ollama-host : Using Ollama running on host with ollama:qwen2.5-coder:32b as code model and ollama:nomic-embed-text as embedding model.
http-ollama-local : ollama:qwen2.5-coder:14b as code model and ollama:nomic-embed-text as embedding model. and
http-openai : openai-chat:chatgpt-4o-latest as code model and openai:text-embedding-3-large as embedding model.

As an example, the following is how the settings is configured within launchsettings.json:

    "http-ollama-host": {
      ... 
      "environmentVariables": {
        ... 
        "CODE_MODEL": "ollama:qwen2.5-coder:32b",
        "CODE_MODEL_PROVIDER": "OllamaHost",
        "EMBEDDING_MODEL": "ollama:nomic-embed-text",
        "EMBEDDING_MODEL_PROVIDER": "OllamaHost",
        "EXTERNAL_OLLAMA_CONNECTION_STRING": "Endpoint=http://host.docker.internal:11434;"
      }
    }

Aspire Code

There is not much new here. We are spinning up a Jupyter container using the Dockerfile and optionally spinning up Ollama if the configuration requires so. For more reference the source file is available here

The Dockerfile is run as a container as below:

var jupyter = builder
    .AddDockerfile(Constants.ConnectionStringNames.JupyterService, "./Jupyter")
    .WithBuildArg("PORT", applicationPorts[Constants.ConnectionStringNames.JupyterService])
    .WithBindMount("./Jupyter/Notebooks/", "/home/jovyan/work")
    .WithHttpEndpoint(targetPort: applicationPorts[Constants.ConnectionStringNames.JupyterService], env: "PORT", name:"http")
    .WithLifetime(ContainerLifetime.Session)
    .WithOtlpExporter()
    .WithEnvironment("OTEL_SERVICE_NAME", "jupyterdemo")
    .WithEnvironment("OTEL_EXPORTER_OTLP_INSECURE", "true")
    .WithEnvironment("PYTHONUNBUFFERED", "0")
    .WithEnvironment("CODE_MODEL", chatConfiguration.CodeModel)
    .WithEnvironment("EMBEDDING_MODEL", chatConfiguration.EmbeddingModel);

Python Demo

For the demo the following use case is considered:

"Using the coding assistant, write code to extract SIFT features from two images and match them using approximate nearest neighbour approach (ANN). Then guide the assistant to implement RANSAC using Homography to improve the matches and eliminate false positives."

The initial prompt was straightforward and got a working code without much effort. However this approach is also full of false positives as seen below:

Initial Matches

From prompt two, we start asking improving matches by RANSAC and things start going wrong. however after a number of prompt and /fix commands, we actually get working code without human interaction

Improved matches

The conversation can be seen by inspecting the notebook snapshot:
src/AspireJupyterAI.AppHost/Jupyter/Notebooks/py-qwen-2-5-coder-32b-ollama-host.ipynb

C# Demo

Same use case was tried with a C# notebook however it took GPT-4o to come up with the solution. Given that OpenCV support in .Net is a bit of a niche, it is not surprising the models are not as effective. In addition as we are using .NET interactive in Jupyter, we are also in a niche territory there.

Here is an example notebook with c# (the prompts that led to the final code are missing unfortunately. csharp-openai-gpt-40-latest.ipynb

To get this working on an ARM laptop, the Dockerfile is also more complicated as we actually build OpenCv and OpenCVSharp as a stage in pour Docker file then copy native libraries and bindings to our final stage. Modified Dockerfile to support OpenCvSharp on ARM

The changes to this file are adopted from one of my older posts - Docker multi-architecture, .NET 6.0 and OpenCVSharp

Ingesting documents using .NET to build a simple Retrieval Augmented Generation (RAG) system

syamaner — Sun, 16 Feb 2025 18:55:12 +0000

Here is a quick post summarising how to use .NET Semantic Kernel, Qdrant and .Net to ingest markdown documents. One of the comments a recent post related to the topic was about why using Python for ingestion instead of .NET. That was a personal preference at the time but also using .NET with Semantic Kernel to ingest documents for a simple pipeline is not necessarily any more work.

In this post, we will go through the ingestion process utilising high level libraries available to us in .NET ecosystem.

.NET Semantic Kernel and related connectors for managing vector store
LangChain .NET for chunking
.NET Aspire to bring it all together using one of the Inference APIs. (Ollama on host, Ollama as container managed by ASPIRE or OpenAI)

Use case

In the Python version, we can either pull the documents from a GitHub Repository or use a file generated by GitIngest UI. HitIngest is an open source library allowing consumers to integrate ability to scrape public repositories from GitHub or manually downloading a file using the Web UI linked earlier.

In this case, we have a single File that contains markdown and .yml files from Official .NET Aspire Documentation Repository. This file is generated by GitIngest UI and contains around 180 files concatenated into a single text file.

Ingestion Process

File Format.

The ingestion process in this example is straightforward and we follow the steps illustrated below.

Splitting actual files

As we are using a single file containing multiple .md and .yml files as described above, first step is to split them into filename, file content pairs.

The files are separated by headers as following:

... content
================================================
File: README.md
================================================
... content

Given this is a throw away example, code below is just enough to demonstrate the process without much distractions.

public static class GitIngestFileSplitter
{
    private const string SeparatorLine = "=====================";
    private const string FilePrefix = "File:";

    public static Dictionary<string, string> ParseContent(string content)
    {
        // declarations omitted 
        foreach (var line in lines)
        {
            if (line.Trim().Contains(SeparatorLine))
            {
                if (currentFileName != null && isCollectingContent && !skipNextSeperatorLine)
                {
                    result[currentFileName] = contentBuilder.ToString().TrimEnd();
                    contentBuilder.Clear();
                    currentFileName = null;
                    isCollectingContent = false;
                    skipNextSeperatorLine = false;
                    continue;
                }
            }
            switch (isCollectingContent)
            {
                case false when line.StartsWith(FilePrefix):
                    currentFileName = line.Replace(FilePrefix,"").Trim();
                    isCollectingContent = true;
                    skipNextSeperatorLine = true;
                    continue;
                case true when currentFileName != null:
                {
                    skipNextSeperatorLine = false;
                    if (!line.Trim().Contains(SeparatorLine) && !string.IsNullOrWhiteSpace(line))
                    {
                        contentBuilder.AppendLine(line);
                    }

                    break;
                }
            }
        }

        // Don't forget to add the last file if there is one
        if (currentFileName != null && contentBuilder.Length > 0)
        {
            result[currentFileName] = contentBuilder.ToString().TrimEnd();
        }
        return result;
    } 
}

Chunking

Now that we have a Dictionary of file names and file content, we now need to get chinks for the file contents.

In this case, I have opted to experiment with LangChain .NET project
We are using MarkdownHeaderTextSplitter and CharacterTextSplitter from LangChain .NET.

...
public class GitIngestChunker : IChunker
{
    // declarations / constructor omitted.
    public async IAsyncEnumerable<FileChunks> GetChunks(string gitIngestFilePath)
    {
        // Read the text file (this is the single file containing all markdown files)
        var gitIngestFileContent = await File.ReadAllTextAsync(gitIngestFilePath);
        // Split the files as discussed earlier
        var files = GitIngestFileSplitter.ParseContent(gitIngestFileContent);
        // Start chunking each split file.
        foreach (var file in files)
        {
            using var chunkingTimer = new MetricTimer(_metrics, MetricNames.Chunking);            
            // omitted: get TextSplitter for given file type.            
            var fileChunks = new FileChunks(file.Key, []);
            var chunks = splitter.SplitText(file.Value);
            // we are using markdown header splitter. So if generated chinks are large, we need to keep chunking them.
            if(chunks.Any(x=>x.Length>600))
            {
                foreach (var chunk in chunks)
                {
                    if(chunk.Length>600)
                    {
                        var subChunks = _characterSplitter.SplitText(chunk);
                        fileChunks.Chunks.AddRange(subChunks);
                    }else{
                        fileChunks.Chunks.Add(chunk);
                    }
                }
            }
            else
            {
                foreach (var chunk in chunks)
                {
                    fileChunks.Chunks.Add(chunk);
                }
            }
            // return the chunks representing the current markdown or yml file
            yield return fileChunks;
        }
    }
    public bool CanChunk(DocumentType documentType)
    {
        return documentType == DocumentType.GitIngest;
    }
}

Getting embedding for the chunks

We are using Semantic Kernel so this part is straightforward and will work with whichever API we chose to use. Given we have so far split the file, and got the chunks for each document, we can use the registered ITextEmbeddingGenerationService (this is driven by app and aspire configuration) to compute the embeddings using the inference approach we have configured.

We also have some custom metrics we are tracking that are visible on Aspire Dashboard as we perform ingestion.

...
public class IngestionPipeline(
    Kernel kernel, ...
{
    private readonly ITextEmbeddingGenerationService _embeddingGenerator =
        kernel.GetRequiredService<ITextEmbeddingGenerationService>();

    public async Task IngestDataAsync(string filePath, DocumentType documentType)
    {
        ... get chunks
        await foreach (var fileChunk in documentChunker.GetChunks(filePath))
        {
            IList<ReadOnlyMemory<float>>? embeddings = null;

            using (new MetricTimer(metrics,
                       MetricNames.Embedding, new KeyValuePair<string, object?>("File", filePath),
                       new KeyValuePair<string, object?>("EmbeddingModel", configuration.Value.EmbeddingModel)))
            {
                embeddings = await _embeddingGenerator.GenerateEmbeddingsAsync(fileChunk.Chunks);
            }
            ... rest of the method
        }
    }    
    ... rest of the class
}

Inserting the vectors

Now that we have the embeddings, we need to insert them. This process involves a few steps:

Mapping a .NET class to a vector store document
Ensuring the Collection exists (optionally recreated)
Using correct dimensions for the collection which depends on what embedding model we use.

Mapping

Microsoft has good documentation on how to build custom mappers for Vector Store Connectors so I will not repeat it here. However at a high level, it is important to cover some of the aspects.

We can use attributes for mapping but in this demo we can use multiple embedding models and they have different dimensions for embedding vectors so using attributes would mean hardcoding these.

We can however define our VectorStoreRecordDefinition in code so that we can at runtime chose the correct dimensions for our collection.

So our mapping can be as simple as the following snippet from QdrantCollectionFactory.cs:

    private static readonly Dictionary<string, int> EmbeddingModels = new()
    {
        { "mxbai-embed-large", 1024 },
        { "nomic-embed-text", 768 },
        { "granite-embedding:30m", 384 }
    };


    private readonly VectorStoreRecordDefinition _faqRecordDefinition = new()
    {
        Properties = new List<VectorStoreRecordProperty>
        {
            new VectorStoreRecordKeyProperty("Id", typeof(Guid)),
            new VectorStoreRecordDataProperty("Content",
                typeof(string)) { IsFilterable = true, StoragePropertyName = "page_content" },
            new VectorStoreRecordDataProperty("Metadata", typeof(FileMetadata))
            {
                IsFullTextSearchable = false, StoragePropertyName = "metadata"
            },
            new VectorStoreRecordVectorProperty("Vector", typeof(float))
            {
                Dimensions = EmbeddingModels.ContainsKey(embeddingModel) ? EmbeddingModels[embeddingModel] : 384,
                DistanceFunction = DistanceFunction.CosineSimilarity, IndexKind = IndexKind.Hnsw,
                StoragePropertyName = "page_content_vector"
            },
        }
    };

When bootstrapping we can then use our factory and register it with .NET Semantic Kernel so whenever we inject and IVectorStore we will have our mappers integrated in the pipeline.

        var options = new QdrantVectorStoreOptions
        {
            HasNamedVectors = true,
            VectorStoreCollectionFactory = new QdrantCollectionFactory(embeddingModelName)
        };
        kernelBuilder.AddQdrantVectorStore(options: options);
    }

Inserting vectors to our collection

Once we handle the registration and configuration, we are ready to consume IVectorStore in our code and make use of it. So in our IngestionPipeline.cs we need to perform the following:

Ensure collection exits:
- Create if it does not or recreate if required.
Insert the vectors as below:

// .NET Semantic Kernel is experimental so we need to opt in to use it.
#pragma warning disable SKEXP0001
.... code omitted
public class IngestionPipeline(
    IVectorStore vectorStore,
    AspireRagDemoIngestionMetrics metrics)
{
    private readonly IVectorStoreRecordCollection<Guid, FaqRecord> _faqCollection = vectorStore.GetCollection<Guid, FaqRecord>(configuration.Value.VectorStoreCollectionName );
    public async Task IngestDataAsync(string filePath, DocumentType documentType)
    {
        await EnsureCollectionExists(true);
        var documentsProcessed = 0;
        .... code omitted
        using var ingestionTimer = new MetricTimer(metrics,
            MetricNames.DocumentIngestion, new KeyValuePair<string, object?>("File", filePath),
            new KeyValuePair<string, object?>("EmbeddingModel", configuration.Value.EmbeddingModel));
        await foreach (var fileChunk in documentChunker.GetChunks(filePath))
        {               metrics.RecordProcessedChunkCount(fileChunk.Chunks.Count);
            for (var i = 0; i < fileChunk.Chunks.Count; i++)
            {
                    try
                    {
                        var faqRecord = new FaqRecord()
                        {
                            Id = Guid.NewGuid(),
                            Content = fileChunk.Chunks[i],
                            Vector = embeddings[i],
                            Metadata = new FileMetadata()
                            {
                                FileName = new StringValue() { Value = fileChunk.FileName }
                            }
                       };
                        await _faqCollection.UpsertAsync(faqRecord);
                    }
                }
            }
            documentsProcessed++;
        }
        metrics.RecordProcessedDocumentCount(documentsProcessed);
    }

    private async Task EnsureCollectionExists(bool forceRecreate = false)
    {
        var collectionExists = await _faqCollection.CollectionExistsAsync();
        switch (collectionExists)
        {
            case true when !forceRecreate:
                return;
            case true:
                await _faqCollection.DeleteCollectionAsync();
                break;
        }

        await _faqCollection.CreateCollectionAsync();
    }
}

Summary

In this quick post we have covered using TextSplitters from LangChain .NET, Vector Stores and Embedding models via .NET Semantic Kernel and some custom metrics captured during ingestion.

Without much code, we can get impressive results using what is available to us in .NET world and if you would like to see the results here is how to:

Clone the repository
Use http-ollama-local configuration in the AppHost Project.
Run the aspire project
Wait for models to one downloaded and started
Then use the src/AspireRagDemo.API/AspireRagDemo.API.http and execute http://localhost:5026/ingest?fileName=dotnet-docs-aspire.txt call. Depending on model size and CPU, tis can take somewhere between 30 seconds to 15 minutes.
Once ingestion completed, access the UI from Aspire Dashboard and run some Aspire Related queries.

In addition, feel free to explore the metrics as below:

Building a simple Retrieval Augmented Generation system using .Net Aspire

syamaner — Sun, 02 Feb 2025 15:31:43 +0000

In this post, we will look into building a simple Retrieval Augmented Generation (RAG) system where we use Jupyter Notebooks for ingestion and .NET Web API for retrieval and generation part using .NET Aspire and having telemetry from both Python and C# components of the system.

We will be looking into the following components to build our system:

Vector store: Qdrant with Aspire.Hosting.Qdrant package.
Ingestion: Jupyter Notebooks
- Langchain for ingestion and OpenTelemetry to ensure
Experimental UI: Streamlit.
Embeddings and Generative models:
- Ollama using CommunityToolkit.Aspire.Hosting.Ollama package.
- Ollama hosted on the development machine (without Docker)
- OpenAI
- HuggingFace
API: Asp.Net Web API with .Net 9
- Microsoft Semantic Kernel.

There are several posts about how to integrate Ollama, OpenAI, Semantic Kernel and emerging open source models. This post will focus on how Aspire 9 networking enhancements help us to build and debug the systems where we might be using multiple languages and frameworks as well as being able to switch our models and model providers with a few lines of configuration change. In additional, we will also look into how to utilise hardware acceleration when such improvements are not available via Docker (mainly MacOS devices)

This post will focus on the following:

How we can use .Net Aspire for polyglot solutions where some components might be better off in different programming languages.
How improved Docker network support in Aspire 9 helps us?
How to utilise power of configuration in Aspire to be able to run Ollama as a container or as an application on the host machine without changing any code.
- Likewise, how to swap Ollama with OpenAI or HuggingFace inference endpoints.

The use case in this post is ingesting .Net Aspire documentation repository and using a RAG approach to answer questions about .Net Aspire. Building such a system is easy but not necessarily helpful if we don't have any metrics to measure success. We are not covering evaluation on this post and that will be the main subject of the next post on the topic. To achieve our use case, we will be utilising Gitingest which is a Python library that helps scraping Github repositories in a format easy for parsing and ingesting. The python code can use the library directly or can consume a text file produced by Gitingest.

Retrieval Augmented Generation (RAG)

There is almost universal awareness that the technology and architecture behind Large Language Models (LLMs) is prone to being creative and making things up. Likes of Gary Marcus and Grady Booch have been trying to raise awareness on what the architectures enabling LLMs are and what they are not.

So if a given technology is good at some areas and have well known limitations on other areas such as being creative with facts, how can we utilise the strengths of such technologies?

One of the approaches is Retrieval Augmented Generation. One of the earlier papers using the term "Retrieval Augmented Generation" is “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” by Lewis, Patrick et al. 2020

A simple RAG system involves in context learning where a general purpose LLM can be used to summarise / extract the answer to a question given related context retrieved using a vector store. In the next section we'll cover these building blocks.

Embeddings and Vector Storage

R in RAG stands for Retrieval. This is where the strength of the approach comes from. Given a repository of data, if we can get relevant answers (context - from a vector store), then we can use a generative model to provide the answer we are looking for given a number of matches to our query.

The ingestion process and impact of chunking

Ingestion process involves:

Enumerating the documents (in this case we are dealing with text only where multiple markdown and yml files are merged into a single text file)
Breaking them down into chunks to make them manageable (chunking).
- Typically, there is a lot to consider here. Some documents have hierarchy which can be utilised when chunking and some documents might be ok with simple breaking down by a fixed size.
Getting the embeddings using embedding model (this needs to be use for the retrieval stage later too)
Adding them to our Vector Store.

For more information on chunking, follow the links at the bottom of the post.

Retrieval and Generation

Once the data is ingested and vector store is up to date, then we can query our RAG system as illustrated in the above diagram.

The steps are:

Get embeddings for the query using the same embedding model utilised for ingestion.
Query the vector store for n nearest results matching our input.
Build the context and run our prompt against our generative model.

.Net Aspire, .NET and Python

There is experimental support for running Python projects as executable in an Aspire Application Host. However, it is also possible to run containers from a Dockerfile which can provide more flexibility.

Jupyter Notebooks

We could run directly using a prebuilt image. However, if we need additional modules or any customisation, then using our Dockerfile and requirements file will ensure our notebook is available immediately (once built) so that we don't have to install same packager each time container is recreated.

FROM quay.io/jupyter/minimal-notebook:python-3.12.8

USER root
RUN apt-get update && apt-get install -y libmagic-dev

RUN mkdir /app
COPY requirements.txt /app

RUN pip install -r /app/requirements.txt

USER ${NB_UID}
ENTRYPOINT ["start-notebook.sh"]

We can then run this as container in our AppHost project:


var jupyter = builder
    .AddDockerfile(Constants.ConnectionStringNames.JupyterService, "./Jupyter")
    .WithBuildArg("PORT", applicationPorts[Constants.ConnectionStringNames.JupyterService])    
    .WithArgs($"--NotebookApp.token={jupyterLocalSecret.Resource.Value}")
    .WithBindMount("./Jupyter/Notebooks/","/home/jovyan/work")

Replit UI

Given this was intended as a quick experimentation to understand how the pieces plug together, using Replit made sense.

However, as Replit is started using a CLI it seemed easier to run it as a container too.

FROM python:3.9-slim
ARG PORT=8501
ENV APP_PORT=$PORT
WORKDIR /app

COPY requirements.txt /app
RUN pip3 install -r /app/requirements.txt

COPY main.py /app
COPY TraceSetup.py /app
COPY entrypoint.sh /app
RUN chmod +x /app/entrypoint.sh

EXPOSE ${PORT}

RUN groupadd -r -g 65532 replitui && useradd --create-home --shell /bin/bash --uid 65532 -g replitui ui_user
USER 65532:65532

HEALTHCHECK CMD curl --fail http://localhost:${PORT}/_stcore/health

ENTRYPOINT [ "bash", "/app/entrypoint.sh"]

Using the framework, it only takes a few lines of code to have the basic components needed for our UI. The Python and bash code are linked at the corresponding branch for this article.

Web API with Semantic Kernel

To utilise Semantic Kernel, we need to define our prompt as well as the Prompt Template. For the RAG query, it is defined as below:

In our prompt, we define the input placeholders for context and question. Then in the PromptTemplateConfig, we link the prompt as well as defining two input arguments for runtime.

    private const string RagPromptTemplate = """
                                             You are a helpful AI assistant specialised in technical questions and good at utilising additional technical resources provided to you as additional context.
                                             Use the following context to answer the question. You always bringing necessary references.
                                             You prefer a good summary over a long explanation but also provide clear justification for the answer.
                                             If the question has absolutely no relevance to the context, please answer "I don't know the answer."
                                             Please do not include the question in the answer. You can sometimes make educated guesses if the context can imply the answer.

                                             Context:
                                             {{$context}}

                                             Question:
                                             {{$question}}                                             
                                             """;

    /// <summary>
    /// To answer the question, the AI assistant will use the provided context.
    /// </summary>
    public static readonly PromptTemplateConfig RagPromptConfig = new()
    {
        Template = RagPromptTemplate,
        InputVariables =
        [
            new InputVariable { Name = "context" },
            new InputVariable { Name = "question" }
        ]
    };

With the configuration out of the way, we can build a compact C# class that puts it all together for us as below. The notable sections are:

GetContextFromVectorStore where we query our Vector Store by getting embeddings for the user's question.
In the method AnswerWithAdditionalContext, we then create a kernel function and execute it by passing arguments containing user's question and additional context retrieved from our vector store.

// omit using
#pragma warning disable SKEXP0001
public class ChatClient(
    Kernel kernel,
    IVectorStore vectorStore, 
    IOptions<ModelConfiguration> configuration,
    ILogger<ChatClient> logger) : IChatClient
{
    private const short TopSearchResults = 20;
    private readonly ITextEmbeddingGenerationService _embeddingGenerator = ...

    private readonly IVectorStoreRecordCollection<Guid, FaqRecord> _faqCollection = ....

    // additional methods omitted for brevity.

    private async Task<string> AnswerWithAdditionalContext(string context, string question)
    {
        var arguments = new KernelArguments
        {
            { "context", context },
            { "question", question }
        };

        var kernelFunction = kernel.CreateFunctionFromPrompt(PromptConstants.RagPromptConfig);
        var result = await kernelFunction.InvokeAsync(kernel, arguments);
        return result.ToString();
    }

    /// <summary>
    /// Get context from the vector store based on the question.
    ///  This method uses the vector store to search for the most relevant context based on the question:
    ///      1. Retrieve the embeddings using the embedding model
    ///      2. Search the vector store for the most relevant context based on the embeddings.
    ///      3. Return the context as a string.
    /// </summary>
    /// <param name="question"></param>
    /// <returns>Vector Search Results.</returns>
    private async Task<string> GetContextFromVectorStore(string question)
    {
        var questionVectors =
            await _embeddingGenerator.GenerateEmbeddingsAsync([question]);

        var stbContext = new StringBuilder();

        var searchResults = await _faqCollection.VectorizedSearchAsync(questionVectors[0],
            new VectorSearchOptions() { Top = TopSearchResults });

        await foreach (var item in searchResults.Results)
        {
            stbContext.AppendLine(item.Record.Content);
        }

        return stbContext.ToString();
    }
}

With little code, we have a RAG system functioning even with a barebones UI for local testing.

Aspire - Docker Networking: Communication Between Components

There are three different ways for application components to communicate:

Container to container
Container to host
Host to container

Aspire 9 creates a Docker network which supports all these communication options.

Container to container

When using Docker compose, we can use service names to connect from one container to another container on the same Docker network.

In the demo application Jupyter Notebook container can connect to Ollama (if running as container - more on this later) and Qdrant container using their service names.

Container to host

Aspire Dashboard in our project runs as an executable as opposed to container. which means, if containers to Use OpenTelemetry, then OTLP endpoint telemetry on Aspire Dashboard running as executable on our host machine needs to be accessible from the containers.

In this case, it is not possible to use localhost as destination in containers so we can use host.docker.internal as the OTEL collector url from containers. This way containers can reach services running on host machine too.

Host to container

This is the case where our .Net Web Api project which is running as an executable process on our host machine and can access all services using localhost and corresponding ports.

Running Ollama as a container or not?

Just because we can run everything as containers does not mean we should run it as a container.

Hardware accelerated Docker

Currently, it is possible to utilise the GPU on Docker hosting NVIDIA Docker. This setup requires device running Linux (or Windows + WSL 2 configured correctly).

When this is the case, running Ollama as a container makes sense.

There are times, when the host machine supports hardware acceleration for Ollama when running on the host but not when running as containers.

For instance:

ARM based MacBook Pro and other MacOS devices.
- Ollama supports acceleration and depending on the specs, can make a huge difference.
- However as hardware acceleration is not supported by Docker in these operating systems, running Ollama in Docker (with or without Aspire) will end up being much slower.
Similarly on Windows devices where there is a dedicated NVIDIA GPU but no NVIDIA Docker support, running Ollama on the host OS will provide better performance.

Our example project also allows for the following set up with a configuration change:

Switching Models and Model providers

Given .NET Aspire shines as a development time orchestration framework, it is no wonder the configuration system is also powerful but simple.

In this project we can conditionally spin up Ollama (if needed) or inject connection string for Ollama running on Host using launchSettings.json on App Host. In addition, the models used for embeddings and generation can be easily swapped too and both Python and .NET based components will use whichever values injected via configuration at runtime.

In addition, it is also possible to use OpenAI for both embeddings and generation using the configuration. In such case, we do need to set up developer secrets to contain a valid OpenAI key.

The main driver for our solution is the launchsettings.json file included with Aspire AppHost project. By modifying this, all our components will utilise the desired models or providers.

{
  "$schema": "https://json.schemastore.org/launchsettings.json",
  "profiles": {
    "http": {
      "commandName": "Project",
      "dotnetRunMessages": true,
      "launchBrowser": true,
      "applicationUrl": "http://localhost:15062",
      "environmentVariables": {
        ....        
        "EMBEDDING_MODEL": "nomic-embed-text",
        "EMBEDDING_MODEL_PROVIDER": "Ollama",        
        "CHAT_MODEL": "mistral",
        "CHAT_MODEL_PROVIDER": "Ollama",        
        "VECTOR_STORE_VECTOR_NAME": "page_content_vector",
        ...
      }
    }
  }
}

In this project, we can use the following values for EMBEDDING_MODEL_PROVIDER:

Ollama : Spin up an Ollama Container using aspire and inject the connection string.
OllamaHost : Do not spin up Ollama container and inject host.docker.internal to containers or localhost to the application executables.
OpenAI: Inject the API key from secrets and use default OpenAI urls.
HuggingFace: Inject API key from developer secrets and use default HuggingFace inference urls.

To use OpenAI or huggingFace, the following user secrets need to be set with valid keys:
Please note, nothing Python and .Net components will use default endpoints for these services and therefore connection strings are not used.

{
  "Parameters:OpenAIKey": "",
  "Parameters:HuggingFaceKey": ""
}

Conclusion

This was a warm up to building a metadata driven retrieval system to query photographs using multi-modal computer vision models which I have been posting about.

Even with quantised and smaller models that run on CPU, we can get decent results asking about .Net Aspire based on the markdown and yml files in the official documentation repository. In this case, we did not utilise any metadata but that will be part of the photo search project.

Models used for testing

Embedding model: granite-embedding
Generative model: qwen2.5:1.5b

Here is a simple question with a relevant answer when using Rag query:

And a made up answer when the question is sent directly to the llm:

Runtime performance

In addition, if using Ollama and working with a laptop that has some level of hardware acceleration but acceleration is not available in Docker, then using Ollama installed locally vs running as a container via Aspire gives much better runtime performance. Here is a comparison using a small model:

Running as a container

We can see that ingestion took 1 minute and running two questions took about 33 seconds.

Running natively on host machine where host has acceleration

When running Ollama natively on the laptop and using host.docker.internal to connect to it from containers, we get around 15 seconds for ingestion and 4 seconds for two queries.

Next Steps

With the available technologies, we can rapidly build question and answer solutions. However if we don’t define our performance metrics and a suitable evaluation approach, there is little value in building such systems.

For instance, we can use different embedding models and generative models. We can change our chunking method or use additional metadata to query and extract relevant chunks more effectively. We can also change model parameters and the list goes on.

With so many variables, how do we compare the outcome? The next post on this topic will include the following:

Generating evaluation data using LLMs.
Defining our metrics.
Performing evaluation using the evaluation data and our target metrics.
Collecting the results from our experiments.
Visualising and comparing the performance of the evaluation process.

Links and References

Chunking

Tools and Frameworks

Comparing Open-Source Vision Models for Photo Description Tasks Using .NET Aspire

syamaner — Mon, 16 Dec 2024 07:00:00 +0000

In our ongoing series about building a local image summarisation system, we have explored how to combine various open-source technologies to generate meaningful descriptions of photos. Today, we'll tackle a crucial question: How do we choose the best vision model for our needs?

This article we focus on a simple approach: using OpenAI's GPT-4o as an automated judge to evaluate the quality of summaries generated by different open-source models. In the next sections, we will explore the following:

Setting up an evaluation pipeline with .NET Aspire
Using GPT-4o to score model outputs
Visualising and analysing the results using Jupyter notebooks

Our evaluation covers six prominent open-source vision models:

llama3.2-vision: Latest iteration of Meta's multimodal model
llava-llama3: Vision-language model built on LLaMA architecture
llava:7b: Compact vision-language model suitable for local deployment
llava:13b: Larger variant offering enhanced capabilities
Florence-2-large-ft: Microsoft's vision model known for detailed scene understanding
llava-phi3: Recent addition combining efficiency with strong performance

These models run locally through our Aspire based infrastructure, which handles:

Model inference and serving
Reverse geocoding for location context
Experiment tracking and result storage

Now that we can generate summaries, which model should we use for summarising our photo library? This post will cover a simple approach how that can be achieved using help of a commercial model.

Evaluation Process

As a weekend project this post covers the following idea: "How about using a commercial model to judge the output generated by Open Source models?"

Why GPT-4o?

GPT-4o (released May 2024) offers several advantages as our evaluation model:

Multimodal capabilities for analysing both images and text.
Consistent scoring methodology.
Cost-effective solution.

Besides being a good choice for the task, pricing was also an advantage for a fun project where there is no budget at all. For instance, 300 evaluation requests (50 images x 6 open source models) cost around $0.8. Open AI API Pricing is available here

Approach

Our approach can be described simply as below:

Input parameters:
- Original photo (scaled to 256px width).
- Model-generated summary.
- Model used.
- Categorisation predictions.
- Top 10 detected objects.
Scoring Criteria:
- Quality and accuracy of the summary (0-100).
- Accuracy of category predictions.
- Precision of object detection.
- Consistency with image content.
Result Collection:
- Structured score and justification storage.
- Integration with existing MongoDB database.

Setting up

For this task, we utilise OpenAIClient from Aspire.OpenAI as seen in code sample below.

Key Implementation Decisions:

Temperature Setting (0.1f):
- Chosen for consistent, deterministic evaluations.
- Reduces random variation in scoring.
JSON Schema Format:
- Ensures structured, parseable responses.
- Simplifies result processing and storage.
Image Preprocessing:
- 256px width limitation balances detail and API costs.
- Consistent sizing ensures fair comparisons.


public class OpenAiPhotoSummaryEvaluationClient([FromKeyedServices("openaiConnection")] OpenAIClient client)
    : IPhotoSummaryEvaluator
{
    private const string SystemPrompt =
        "You are a highly accurate and fair image summarisation evaluation model. "
        + " Your job is to evaluate the quality of summaries generated from images by different computer vision models. \n\n"
        + " When evaluating a summary of the provided image:\n\n"
        + " - Provide a single score ranging between 0 and 100 combining the following properties: \n\n"
        + "    - Quality and accuracyof the summary.\n\n"
        + "    - Quality and accuracy of the categories predicted for the image.\n\n"
        + "    - Quality and accuracy of the objects predicted to be in the image.\n\n"
        + "  - Be fair and consistent when evaluating. \n\n";
    private const string PromptSummary =
        "Please score the provided image summary based on the quality and accuracy of the summary, categories, and objects predicted in the image.";

    public async Task<PhotoSummaryScore> EvaluatePhotoSummary(string base64Image, ImageSummaryEvaluationRequest summary)
    {
        .. omitted: resize the image to me max 256 px wide
        var img = ChatMessageContentPart.CreateImagePart(new BinaryData(memStream.ToArray()), "image/jpeg",
            ChatImageDetailLevel.Auto);
        List<ChatMessage> messages =
        [
            new UserChatMessage(PromptSummary, JsonSerializer.Serialize(summary), img),
            new SystemChatMessage(SystemPrompt)
        ];

        var options = new ChatCompletionOptions()
        {
            Temperature = 0.1f,
            ResponseFormat = ChatResponseFormat.CreateJsonSchemaFormat(jsonSchemaFormatName: "image_summary_result",
                jsonSchema: BinaryData.FromString("""
                {
                    "type": "object",
                    "properties": {
                        "Score": { "type": "number"},
                        "Justification": { "type": "string"}
                    },
                    "required": ["Score", "Justification"],
                    "additionalProperties": false
                }
                """),
            jsonSchemaIsStrict: true)
        };
        var completion = await client.GetChatClient("gpt-4o").CompleteChatAsync(messages, options);
        using var structuredJson = JsonDocument.Parse(completion.Value.Content[0].Text);
        var score = structuredJson.RootElement.GetProperty("Score").GetDouble();
        var justification = structuredJson.RootElement.GetProperty("Justification").GetString();
        return new PhotoSummaryScore(score, justification!, "OpenAI");
    }
}

The results from this process are then stored in the database with the summaries against the original image. OpenAI API has certain rate limits and therefore it is important to manage how often these are being Called.

Analysis and visualisation

Our analysis notebook provides:

Data Collection:
- MongoDB query and result aggregation
Visualisation Components:
- Model comparison table.
- Example evaluation cases
Validating the outcome:
- Filter results by best model.
- Visualise evaluation justifications.
Use Aspire Command to download and upload the notebook between development machine and Jupyter Server on Docker host.

Below is an example of the evaluation process where GPT-4o correctly identifies the inaccuracies in the generated summaries. The results look fair and accurate making it easier to introduce more open source models and then use the notebook to evaluate the performances.
This also allows to tweak the prompts to get better results from the models. For example wrong location information is likely from including the address resolved from the GPS tag of the photo which leads some models to be more creative with their description.

Further results can be seen on the notebook

Results and Remarks

Following the process outlined earlier, llava:13b is on top with an average of 85.6 score with Florence-2-large-ft being second as below:

Observations

Providing too much address detail can lead models to make up location information.
Larger models provide more detailed summaries.
OpenAIClient from Aspire.OpenAI works well with Ollama Server as well.
The Aspire Command for Jupyter notebook made it was for me to pull and push the notebook from my machine to wherever Aspire is running containers on.
- As a next step, it makes sense to consider periodic downloading of the notebook.

Conclusion and what's next

It is easy enough to utilise APIs that allow inference on image inputs. However, making a decision on what model to use is not so straightforward given the need for a large number of test images against each model. This is what makes the evaluation process crucial to make the most out of such technology.

In this post, we have looked into using OpenAI GPT-4o model to evaluate open source model performance to assess the quality of the image summaries generated by open source models.

Our evaluation framework using GPT-4o provides a systematic approach to comparing vision model performance. Key takeaways include:

Automated Evaluation Benefits:
- Consistent scoring methodology.
- Scalable to large image sets.
- Cost-effective solution.
Implementation Insights:
- Aspire.OpenAI simplifies integration.
- Jupyter notebooks enable flexible analysis.
- .Net Aspire makes local development orchestration a breeze.

Next Steps

Model Expansion:
- Integration of newer vision models
- Prompt engineering optimisation
- Performance benchmarking
Feature Development:
- Natural language image search implementation
- Enhanced evaluation metrics
- Automated testing pipeline

The notebook can be accessed in GitHub with rest of the code.

Welcoming .NET Aspire 9.0 : Photo Summary Project

syamaner — Mon, 18 Nov 2024 08:27:22 +0000

The project so far makes it possible to scan photos in a directory, run them through various vision models and store the details in a database. Using Aspire instead of Docker Compose for local development as I would usually do has been fun so far.

Aspire 9.0 has handy new features, that are interesting for the next stage of the project. This post will summarise them with some examples.

As the next stage in the project is evaluating the performance of the models I am using, at this point I needed to make Jupyter Notebooks easily accessible inside my codebase and development environment. With .Net Aspire 9.0 this becomes a convenient process.

Getting Ready for Next Steps

Given we have several options from open source when it comes to computer vision models go generate photo summaries, we need to be able to evaluate the results from these models to be able to choose one that suits our domain.

One workflow to do this effectively is using Jupyter Notebooks where we can retrieve our results from the database and then compare with results obtained from commercial models.

Introducing Jupyter Notebooks that runs on a remote host to our Project means the following would be important:

Containers are running on remote host so we need to be able to include the notebooks in version control
- However docker volumes are on a remote host, no easy way to copy them
Also we need container to container connection Jupyter Server to MongoDB both of which are running on a remote server and Jupyter Server needs to be able to speak to MongoDb.

As we will see below, .NET Aspire 9.0 takes care of these.

Here is the list of features in .Net Aspire 9.0 that are relevant to this project and will be covered in this post

Tooling
- No longer relying on workloads: Now we can set up .Net Aspire using packages and project templates.
- Templates can also be installed as following: dotnet new install Aspire.ProjectTemplates::9.0.0
Dashboard and UX
- Managing Resource Lifecycles: Start, Stop, Restart from the dashboard.
- Browser Telemetry Support.
App Host (Orchestration)
- Waiting for dependencies.
- Resource health Checks
- Persistent Containers
- Resource commands
- Container networking
One from .Net 9.0
- Enabling DI registration of metrics using IMeterFactory

Browser Telemetry Support

Earlier on, I was curious how to integrate traces from the front end and see the distributed traces. .Net Aspire 9.0 brings an out of the box way for this as below.

It is important to remember that Open Telemetry client instrumentation on the browser is experimental.

Define DOTNET_DASHBOARD_OTLP_HTTP_ENDPOINT_URL environment variable for Apphost launch settings. '"DOTNET_DASHBOARD_OTLP_HTTP_ENDPOINT_URL": "http://localhost:16175"'
I still needed to inject DOTNET_DASHBOARD_OTLP_ENDPOINT_URL to the .Net applications.
Front end required the HTTP endpoint as well as the following environment variable: "OTEL_EXPORTER_OTLP_PROTOCOL","http/protobuf"

With these in place, I was able to follow Microsoft examples to make it work on a Stencil Js application.

...
//https://www.honeycomb.io/blog/opentelemetry-browser-instrumentation
//https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-instrumentation-xml-http-request
//https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/web/opentelemetry-instrumentation-user-interaction
//https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/web/opentelemetry-instrumentation-long-task
//https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/web/opentelemetry-instrumentation-long-task

export default async () => {
  const otlpOptions = { omitted };
  const attributes = { omitted }
  provider.addSpanProcessor(new SimpleSpanProcessor(new OTLPTraceExporter(otlpOptions)));
  provider.register({contextManager: new StackContextManager()});
  registerInstrumentations({
    instrumentations: [
      getWebAutoInstrumentations({
        '@opentelemetry/instrumentation-xml-http-request': {
          clearTimingResources: true,
        }
      }),
      new LongTaskInstrumentation({
        observerCallback: (span, longtaskEvent) => {
          span.setAttribute('location.pathname', window.location.pathname)
        }
      }),
      new FetchInstrumentation({
        propagateTraceHeaderCorsUrls: [new RegExp('\\/api\\/*')],
        ignoreUrls: [new RegExp('\\/tile\\/*')],
      })],
  });
...
};

With the changes above and the existing setup in our backend, we can see the end to end traces below. The use case below is for selecting a model and then seeing a request to extract summaries from all 50 images in the db. We can see the durations of db calls, calls to inference endpoints as well as transport and our backend components as they all been triggered by the ui.

Resource Health Checks

If we would like to specify resource dependencies to control startup-process, it is important to be able to define what health check meant to various components.

If no health checks defined, a resource is considered healthy if it is in running state
If the Resource exposes an http health check, we can register with one call .WithHttpHealthCheck("/health")
Sometimes external resources do not provide health checks suitable for us, we can also define, register and use our health checks. This is the method that will be discussed here.

Checking if Nominatim Resource is healthy:

public class NominatimHealthCheck : IHealthCheck
{
... ignored
    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
        CancellationToken cancellationToken = new CancellationToken())
    { 
        var ready = await IsServerReady(cancellationToken);
        return ready
            ? HealthCheckResult.Healthy()
            : HealthCheckResult.Unhealthy();
    }

    private async Task<bool> IsServerReady(CancellationToken cancellationToken = default)
    {
        const string searchUrl = "/search.php?q=avenue%20pasteur";
        // check if we have success result or not. Code omitted.
    }
}

// we also need to register this as below:

builder.Services.AddHealthChecks()
    .AddTypeActivatedCheck<NominatimHealthCheck>("nominatim-healthcheck",..);
var nominatimResourceBuilder = builder.AddResource(nominatimResource)
    .WithHealthCheck("nominatim-healthcheck")
....);

And once all registered, we can also see the health in the dashboard:

Defining Resource Dependencies

Once we have defined our health checks, we can declare our dependencies so that our applications will not start before services they might depend on at start up.

Prior to .NET Aspire 9.0, it was possible to achieve this following an example provided by David Fowler in Issue #921 in Aspire repository as linked below.

So now, once all the additional code deleted, all we need is to use .WaitFor(resource) from the framework as below:

var apiService = builder.AddProject<Projects.PhotoSearch_API>("apiservice") 
    .WithReference(ollamaContainer)
    .WithReference(mongodb)
    .WithReference(messaging)
    .WithReference(openai)
    .WaitFor(ollamaContainer)
    .WaitFor(mongodb)
    .WaitFor(messaging);

This ensures all dependencies spin up first and become healthy and then the application will start. This also helps in cases where the containers need to download data on first run, which might take several minutes.

Persistent Containers

In this project we have some containers that take time to lad and get ready such as OSM Map Tile Server, Nominatim container for reverse geocoding and Ollama is starting the first time as it needs to download the model.

So if we make code changes, containers would stop and then we would need to wait for all dependencies again.

This is another area where Aspire 9.0 comes to the rescue with a single method call as below:

var nominatimResourceBuilder = builder.AddResource(nominatimResource)
    .WithLifetime(ContainerLifetime.Persistent)
    ... other calls omitted;

With this in place, we can start debugging stop and start again lightning fast without having to wait for the containers.

Resource Commands

Resource commands allow the developers register commands that can be accessed from the Aspire Dashboard against a resource.

As I have added Jupyter Notebooks container to the project this weekend, command have helped sole one problem when running the Jupyter Server on a remote Docker host: How do we manage the notebook files? Ideally we manage them in the same repository.

With a download and upload command, we can upload the notebook from our local drive where our git repository is and then we can download the model when modified inside the Jupyter Notebook container.

Downloading the notebook in regular intervals will be one of the next steps.

    public static IResourceBuilder<ContainerResource> WithUploadNoteBookCommand(
        this IResourceBuilder<ContainerResource> builder, string jupyterToken, string jupyterUrl)
    {
        builder.WithCommand(
            name: "upload-notebook",
            displayName: "Upload Notebook",
            executeCommand: context => OnUploadNotebookCommandAsync(builder, context, jupyterToken, jupyterUrl),
            updateState: OnUpdateResourceState,
            iconName: "ArrowUpload",
            iconVariant: IconVariant.Filled);

        return builder;
    }
    private static async Task<ExecuteCommandResult> OnUploadNotebookCommandAsync(... params omitted)
    {
       var notebookData = read from disk
       // setup omitted
        var result = await httpclient.SendAsync(uploadNoteBookHttpRequestMessage);
       // handle the response
    }

// And command is registered as below:
var jupyter = builder.AddContainer(name, image)
    .WithUploadNoteBookCommand(token, "http://localhost:8888")
    .WithDownloadNoteBookCommand(token, "http://localhost:8888")
    ... other calls omitted;

From a high level the process looks as below:

Although the above illustrates the case when we are using a remote Docker daemon, the experience is the same if we were using a local Docker daemon.

Container Networking

This is another feature that made life easy. With workloads running on other machines on local network, so far container to container access have not been that important.

With the Jupyter Notebooks container, as our data source is a MongoDb container on the same docker host, it was necessary to be able to access the database.

This is another change where improvements to Aspire have been transparent to the developer and gained the benefits without extra work as below:

There is not much to do besides ensuring the connection string are injected as below:

builder.AddJupyter("jupyter", !string.IsNullOrWhiteSpace(dockerHost), "secret",portMappings["JupyterPort"].PublicPort)
    .WithReference(mongodb);

Once that is done, we can access the connection string in python as following: connection_string = os.environ.get('ConnectionStrings__photo-search')

Custom Metrics via Dependency Injection

This is not aspire but as a colleague pointed last week on elf the features in .Net 9.0 was out of the box ability to use Dependency Injection (DI) for registering consuming metrics.

We can achieve this by injecting IMeterFactory to a utility class where we manage our meters.

Here is an example from this project for measuring number of models summarised as well as durations for each image:

using System.Diagnostics.Metrics;
namespace PhotoSearch.ServiceDefaults;
public class ConsoleMetrics
{
    private readonly Counter<int> _photosSummariesCounter;
    private readonly Histogram<double> _photosSummaryHistogram;
    public ConsoleMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("PhotoSummary.Worker");
        _photosSummariesCounter = meter.CreateCounter<int>("photosummary.summary.generated");
        _photosSummaryHistogram = meter.CreateHistogram<double>("photosummary.summary.durationseconds");
    }
    public void PhotoSummarised(string model, int quantity)
   {
        _photosSummariesCounter.Add(quantity,
            new KeyValuePair<string, object?>("photosummary.summary.model", model));
    }

    public void PhotoSummaryTiming(string model,string photo, double durationSeconds)
    {
        _photosSummaryHistogram.Record(durationSeconds,
            new KeyValuePair<string, object?>("photosummary.summary.model", model),
            new KeyValuePair<string, object?>("photosummary.summary.photo", photo));
    }
}

// Register:
builder.Services.AddSingleton<ConsoleMetrics>();
// Ensure our meter is added when configuring OpenTelemetry:
builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics.AddMeter(InstrumentationOptions.MeterName);
        // other calls
    })
// Inject and use: 
builder.Services.AddSingleton<ConsoleMetrics>();

As we see below, total 70 images summarised using two models so far.

Finally in the following screen capture, we can see the timings of each photo summary request. While generally they are 5 - 10 seconds, there are some outliers taking about 5 minutes. We can dig into the metrics, find out which photo / model combination causes the spikes at that stage using the traces and then understand if it is a random GPU issue or a consistent delay under some circumstances.

If we investigate the traces, the results are interesting.

To summarise a photo, currently there are 3 calls to Ollama container:

Get the overall summary
Using the context, get a list of objects
Again using the context so far, get a list of possible categories

And the traces show us, we spent 4 minute 27 seconds waiting for categories to be generated.

This is worth investigation and as we have a notebook, it is also easier to experiment with the same prompt / image combinations.

Conclusion

.Net Aspire 9.0 changes make Aspire a great alternative to using Docker Compose. By adopting standard container technologies, managing a local development environment using Aspire is worth having a go.

It has also been great witnessing how Aspire 9.0 features have been discussed in Github issues and ended up ready to consume for the masses with the new release. The transparency and the speed of improvement makes it a great choice for development.

Now that upgrade is out of the way, next step will be generating the summaries using Open API models and then comparing / ranking each summary generated locally against it and evaluating results. I have also come across a paper that proposes a more systemic approach to evaluation using state of the art models and will be experimenting with that too.

Adding Map Based Photo Viewer to .Net Aspire Project with Stencil and OpenStreetMap Tile Server

syamaner — Sun, 01 Sep 2024 13:50:44 +0000

Here is the next post on my journey building a personal photo search application using open source technologies as a testbed for .Net Aspire.

Please note that, while these posts cover somehow to's, these are not necessary intended a tutorials on specific topics and instead an overview of how we can integrate these technologies to build something interesting.

The following areas have been covered so far:

.NET Aspire
- Declaring resource dependencies so that our applications can wait until all referenced services have started successfully and fully initialised.
- How to use remote Docker daemon for the containers we depend when we don't want to overload our current development machine using SSH.
- How we can use SSH port forwarding in our App Host.
Machine Learning
- How to use MultiModal ML Models for summarising and extracting information from photos.
- How we could integrate local models using Ollama or a simple Python project using Hugging Face hosted models.
Incorporated reverse geocoding into our solution with .Net Aspire and Open Street Map (OSM) Nominatim containers.

In today's post the following topics will be explored:

Building a simple web component using Stencil that will:
- Provide a map Web Component to display Photos on a database.
- Provide a summary Web Component that will show the generated summaries of the photo using multiple models as well as the address, location and the predicted categories of the images using given models.
A new .Net Aspire resource for Open Street Map (OSM) Tile Server so that our web component can render maps using the local (or using a remote docker daemon) resources.

What does it look like?

So far, we have been able to import our Photos into MongoDB including GeoLocation and Metadata. In addition, reverse geocoding applied so that the GeoData is converted to the nearest address based on OSM data using Nominatim resource.

Once the images are imported, then a background worker has generated Summary, Category and Content using a number of open source multi modal models and stored them in a dictionary against the photos.

So the recent changes in the project mean, we can visualise these on the map and see what the models generated. We have not yet started looking into model evaluation so right now, we will see a number of accurate results as well as totally made up information which makes evaluation a critical part of this project.

If the image is no clear enough here is the text content:

The image captures a lively scene of a band performing on stage. The stage is bathed in warm yellow lights, creating an atmosphere of excitement and energy. At the heart of the stage, three musicians are immersed in their performance. On the left, a guitarist strums his instrument with passion, his fingers moving over the strings as he plays a melody that fills the room. In the center, a singer belts out a tune, her voice echoing off the walls of the auditorium. To the right, a drummer beats out a rhythm on his drum set, his hands striking the drums in a steady beat. In the background, a large screen displays an image of the band, amplifying their presence and engaging with the audience. The stage is surrounded by a sea of spectators, some of whom are captured in the foreground of the image, their faces turned towards the performers. The perspective of the photo suggests it was taken from the viewpoint of someone standing close to the stage, immersing themselves in the concert experience. The image is a snapshot of a moment filled with music and energy, encapsulating the spirit of live performance.

Photo Categories

Concert, music, performance

Audience, stage, band

Instruments, lighting, yellow lights.

Photo Contents

Guitarist, singer, drummer

Guitar, drum set, drums

Screen, audience, stage lights

Yellow light bulbs, spectators.

Web Components and Stencil

Web components are a set of standardised technologies that allow developers to create reusable custom elements with encapsulated functionality.

The key parts of Web Components can be summarised as the following:

Custom Elements: Define new HTML elements.
Shadow DOM: Encapsulates styles and markup to prevent them from affecting the rest of the page.
HTML Templates: Define chunks of markup that can be reused without rendering immediately.

Why Use Web Components?

Web components offer lightweight reusability, allowing developers to create components that can be used across multiple projects without being tied to specific frameworks. They provide encapsulation by isolating styles and scripts, which helps prevent conflicts and bugs in the global scope.

Additionally, web components are highly interoperable, which means they can be integrated in applications built with frameworks such as React, Vue.js, Angular as well as vanilla HTML / JavaScript, making them a future-proof solution for developers.

Web Components vs. Frameworks

Unlike traditional frameworks, web components are not dependent on any specific framework since they are built on browser-native APIs. This foundation on web standards guarantees their longevity and stability, ensuring long-term support without the need to adapt to changes in framework-specific updates. Web components are also known for their performance benefits, as they can be lightweight and optimised without the runtime overhead that frameworks often introduce.

For developers already familiar with HTML, CSS, and JavaScript, web components offer a more straightforward learning curve compared to adopting an entire new framework.

Stencil

Stencil is web component compiler that simplifies the process of building scalable, performant, and framework-agnostic web components. It is one of the many options we have for simplifying the build process for Web Components.

The web components in this Project are built using Stencil.

Adding a Mapping UI to our Aspire Project as a Web Component.

.NET Aspire already have support for NPM applications so adding aStencil Startup Application is as simple as following:

Create a hello world application as outlined in Stencil documentation in the same repository as our Aspire application.
Restore the packages, add your components
Then use AddNpmApp to register the application in our Aspire App Host.


builder.AddNpmApp("stencil", "../photosearch-frontend")
    .WithReference(apiService)
    .WithReference(osmTileService)
    .WithHttpEndpoint(port: portMappings["FEPort"].PublicPort, 
        targetPort: portMappings["FEPort"].PrivatePort, 
        env: "PORT", 
        isProxied:false)
    .PublishAsDockerFile();

This even supports auto reloading so we can keep modifying the UI source code and see the changes reflected quickly. I have been using Rider for the Aspire Project and VS Code for the Stencil or Python code.

The map component is simple and responsible for the following:

Calling our .Net API endpoint to get photos.
Render the photos on map.
As the user makes a selection and views a photo on the map, it will also raise an event so that summary component can deploy details of the selected photo and summaries.

The component encapsulates and open source library for map rendering called "MapLibre GL JS" and uses our OSM Map Tile Server container to render the tiles. As we can see below, the service discovery is handled by Aspire and we do not have to worry about manually updating urls.


... imports omitted 

@Component({ tag: 'map-component', styleUrl: 'map-component.css', shadow: true })
export class MapComponent {

  mapElement: HTMLElement;
  photoSummaries: Array<PhotoSummary> = [];
  map: Map | undefined;

  markers: {
    [name: string]: Marker
  } = {};

  loadPhotos = async () => {

    const response = await fetch(Env.API_BASE_URL + "/photos");
    this.photoSummaries = await response.json();
    this.photoSummaries.forEach((photo) => {
      const marker = new Marker({
        draggable: false
      }).setLngLat([photo.Longitude, photo.Latitude]);

      const imgUrl = `${Env.API_BASE_URL}/image/${photo.Id}/1280/1280`;
      marker.setPopup(new Popup({ className: "apple-popup" })
        .setHTML(`<img src='${imgUrl}' data-id="${photo.Id}" loading="lazy"></img>`));
      marker.getPopup().setMaxWidth("300px");

      let popupElem = marker.getElement();
      popupElem.addEventListener('click', () => {
        PubSub.publish(EventNames.PhotoSelected, photo);
      });
      this.markers[photo.Id] = marker;
    });
  };

  componentWillLoad = async () => {
    await this.loadPhotos();
  }

  disconnectCallback = () => {
    this.markers = null;
    this.map = null;
  }

  componentDidLoad = async () => {
    const style: StyleSpecification = {
      version: 8,
      sources: {
        osm: {
          type: 'raster',
          tiles: [`${Env.MAP_TILE_SERVER}/tile/{z}/{x}/{y}.png`],
          tileSize: 256,
          attribution: '.....'
        }
      },
      layers: [{
        id: 'osm',
        type: 'raster',
        source: 'osm',
      }],
    };

    this.map = new Map({
      container: this.mapElement,
      style: style,
      center: [
        this.photoSummaries[0].Longitude, this.photoSummaries[0].Latitude
      ],
      zoom: 14
    });
    this.photoSummaries.forEach((photo) => {
      this.markers[photo.Id].addTo(this.map);
    });
  }

  render() {
    return <div id="map" ref={(el) => this.mapElement = el as HTMLElement}></div>
  }
}

Service discovery for the Stencil Application

Given Aspire injects connection strings of services we depend on, the Stencil configuration can read these variables and inject into components as following:

  ...,
  env: {
    API_BASE_URL: process.env.services__apiservice__http__0,
    MAP_TILE_SERVER: process.env.ConnectionStrings__OSMMapTileServer
  }

How to use the component?

Once the components ready we can use them just like any html elements:

        <div class="flex mb-4 map-container" >
          <div class="w-1/2 h-120" >
            <map-component></map-component>
          </div>
          <div class="w-1/2   h-120">
            <photo-summary-view></photo-summary-view>
          </div>
        </div>

Open Street Map (OSM) Tile Server .NET Aspire Resource

We have a mapping web component but without a Map Tile server, we will not be Abe to render the maps. although there are some free tile servers for demo, it would be better not to overload those resources and ensure we consume our own computing power instead.

OSM Tile Server Container makes this a simple job. And once we integrate with Aspire, we have a Tile Server running on demand without a complicated setup.

Adjustments to the Tile Server Container

As per documentation, we would need to run the container once to download the maps into our volume and then use the same volume ad run the container to host the map. The changes made in this post will allow to perform both actions at startup.

The image built in this project will execute the following startup script:

#!/bin/bash

cd /
./run.sh import
./run.sh run

Besides the above there is no change to the original OSM Tile Server Image.

Creating an OSM Tile Server Resource

The process for defining the resource is similar to Nominatim resources covered in one of the previous posts so will not be covered here but the source code is available at the repository as always.

The sample images are all across London only therefore we are using London maps to ensure quick downloading of maps and ability to setup the map database quickly at the first start of the container.

Where We are and What Next

So far we have a means of importing photos and then processing them using various multi modal machine learning models. Generative models will often generate what is requested but the results are not often as desired. Before deciding how to use these models, it is important to have an evaluation approach.

Now that we have a very basic UI, we can now focus on evaluating these models to find the best prompt / model combination for our search application.

This means we will need to version our results to include the prompt used as well as the models used and then work out some metrics that will help choosing the combinations that offer highest success rate.

Initial method will be looking into comparing results to those generated by models such as GPT-4 and see how or small / local models compare.

This will be the focus of next post.

Simplifying Remote Docker Container Connections in .NET Aspire with SSH.Net

syamaner — Sun, 04 Aug 2024 18:44:27 +0000

Previously, I have shared my experience so far in my journey towards .NET Aspire using Photo Search use case by the help of multi-modal models. The key part for me was being able to use remote docker host which has worked well out of the box for most part. My focus so far is on the local development environments and therefore exploring different ways to achieve the original goal of running containers remotely on local network using SSH.

As a recap, it all started with the question whether or not the scenario illustrated in the following image could be supported out of the box or not:

In this brief post, we will start with some of the challenges of the approach used so far, and then introduce the SSH Port forwarding method and how it improves on the shortcomings of the initial approach.

Previous Approach: Overriding Connection Strings When Using a Remote Host

The approach so far has been as following:

For the containers:
- Ensure the container port binding uses 0.0.0.0 to listen on all interfaces so that it can be reached from the local network.
- container.WithContainerRuntimeArgs("-p", $"0.0.0.0:{publicPort}:{publicPort}");
For Aspire AppHost
- Ensure DOCKER_HOST environment on the development machine is set.
  - The remote Docker host allows ssh connections from the development machine (e.g. ssh-copy-id is run to add our public key to the remote server)
- If creating a custom resource, ensure the connection strings and exposed endpoints use the IP of the Docker Host instead of localhost.
- For built in resources:
  - Either override the injected connection string environment variables.
  - Or use connection string redirection.

The challenges with this approach are:

Having to modify connection strings at runtime when using remote docker host can get messy as we add more resources.
Connection string redirection succeeded for PostgreSQL but not for RabbitMQ so had to fall back to environment variables override.
When we update the connection strings, the exposed endpoints in the Aspire Dashboard can still point to localhost or removed as they are no longer valid.

For instance, this has worked for PostgreSQL connection string redirection to remote host:

var pgConnectionStringRedirection =
            new CustomPostgresConnectionStringRedirection(dbName, host, publicPort.ToString(), pgUsername, pgPassword);

postgresContainer.WithConnectionStringRedirection(pgConnectionStringRedirection);

SSH.Net to the rescue

As Docker CLI is using SSH to run the containers on the remote Docker Daemon, instead of making containers listen on all interfaces, we can instead do SSH port forwarding from our development machine into Docker Host machine.

Wouldn't it be cool to orchestrate this from Aspire AppHost project so that it happens for us when the development orchestration starts and stopped when we spin down our development environment.

The following diagram is a simplified illustration of this concept. The ports used for containers are forwarded using SSH from development machine to Docker Host.

Implementation

Given we have already had a successful SSH connection which allows managing containers on a remote host, forwarding container ports vis SSH not only made sense but also simplified the setup.

When the AppHost starts we need to forward the ports as illustrated in the code snipped below. Instead of calling WithContainerRuntimeArgs("-p", $"0.0.0.0:{publicPort}:{publicPort}"); on the resources, we need to ensure the Endpoints are not proxied as we are taking care of this using SSH Port forwarding instead.

Some advantages are:

Connection strings and endpoints can be used without having to transform or injecting connection strings.
No more having to call WithContainerRuntimeArgs("-p", $"0.0.0.0:{publicPort}:{publicPort}"); the containers will listen on localhost and no need to worry abut exposing on other network interfaces.
Able to use the links on aspire Dashboard to access container resources.
- Localhost on local machine is forwarded to the right host / port so this works transparently for us.


// ... usings

var builder = DistributedApplication.CreateBuilder(args);

var dockerHost = StartupHelper.GetDockerHostValue();
var enableNvidiaDocker = StartupHelper.NvidiaDockerEnabled();

// ... resources

var apiService = builder.AddProject<Projects.PhotoSearch_API>("apiservice") 
    .WithReference(ollamaContainer)
    .WithReference(postgresDb)
    .WaitFor(messaging)
    .WithReference(messaging);

var backgroundWorker = builder.AddProject<Projects.PhotoSearch_Worker>("backgroundservice")
    .WithReference(ollamaContainer)
    .WithReference(postgresDb)
    .WithReference(florence3Api)
    .WithReference(nominatimContainer)
    .WithReference(messaging)
    .WaitFor(ollamaContainer)
    .WaitFor(nominatimContainer)
    .WaitFor(messaging);

// add ssh_user and ssh_key_file (path to the key file) for ser secrets.
using var sshUtility = new SShUtility(dockerHost, builder.Configuration["ssh_user"]!,  builder.Configuration["ssh_key_file"]!);

if (!string.IsNullOrWhiteSpace(dockerHost))
{
    // Forwards the ports to the docker host machine
    sshUtility.Connect();
    // PgAdmin
    sshUtility.AddForwardedPort(8081, 8081);
    // Postgres
    sshUtility.AddForwardedPort(5432, 5432);
    // RabbitMQ
    sshUtility.AddForwardedPort(5672, 5672);
    // RabbitMQ Management
    sshUtility.AddForwardedPort(15672, 15672);
    // Nominatim
    sshUtility.AddForwardedPort(8180, 8180);
    // Ollama
    sshUtility.AddForwardedPort(11438, 11438);
}

builder.Build().Run();

As we can see below. the dashboard URLs point to localhost and they are fully functional even though the containers are running remotely. Using SSH port forwarding improved this where previously the links in dashboard were either blank (removed endpoints as they were not valid) or listed incorrectly as localhost even though we had to access container services using the IP address of the Docker host.

With port forwarding this issue has been resolved.

Summary

When using remote docker host, we can also do port forwarding via SSH which simplifies the setup and requires less code compared to overriding connection strings for applications. When we are running the containers locally, we don't do the port forwarding and let Aspire Endpoints do the job as intended.

This also means this setup supports using GPU hosting providers such as Lambda Labs GPU Cloud for running containers for a few hours to experiment with large models that require GPUs with large VRAM and pay per use.

DEV Community: syamaner

Part 1: The Architecture & The Agent - Spec-Driven ML Development With Warp/Oz

From Prototype to Production

The Director/Coder Dynamic

The Agentic Setup: AGENTS.md, Epics, and Skills

1. AGENTS.md - The Rulebook

2. Epic State Management - The Checklist

3. Parameterised Skills - The Playbooks

A Generalised AGENTS.md Template

The Build & The Fails

The input_features vs input_values Bug

Copilot as the Third Actor

By the Numbers

Links

Part 3: From Neural Networks to Autonomous Coffee Roasting - Orchestrating MCP Servers with .NET Aspire 13 and n8n Agents

Introduction

The Challenge

Why Aspire 13?

Simplified Python Hosting with AddPythonModule

Cleaner AppHost Project Structure

Enhanced Container Orchestration

Built-in OpenTelemetry Integration

Architecture Overview

The n8n Autonomous Roasting Workflow

Phase 1: Initialisation & Preheating (Preheating Agent)

Phase 2: Bean Charge Detection (Preheating Agent)

Phase 3: First Crack Detection (Roast Agent)

Phase 4: Development Time Management (Roast Agent)

Phase 5: Completion

The Aspire Orchestration Code

Python MCP Servers

Container Services

Real-World Results

The First Autonomous Roast

The Aspire Dashboard Experience

Lessons Learned

1. MCP Server Design Matters

2. Aspire's Python Support is Production-Ready

3. n8n is Powerful for Agent Orchestration

4. MCP Protocol Makes Tool Integration a Breeze

5. Observability is Critical

The Development Experience with Warp Agent

Next Steps

Short Term

Medium Term

Conclusion

Resources

Code and Articles

.NET Aspire Documentation

Tools and Protocols

The first roast:

Part 2: Building MCP Servers to Control a Home Coffee Roaster - An Agentic Development Journey with Warp Agent

Introduction

Integrating software, hardware and agents without reinventing the wheel

Traditional approach: Build custom APIs, handle authentication, manage state

The MCP option: Standardised protocol for AI <-> tool communication

🤖 Warp Agent Contributions

What is MCP (Model Context Protocol) and Why Use It?

What is MCP?

Client-Server Concept

MCP Server

MCP Client

Transport Types

Flow

Server 1: First Crack Detector MCP

Implementation Details

Code Walkthrough

Testing Approach

Server 2: Hottop Roaster Controller MCP

The Hottop KN-8828B-2K+ Protocol

Implementation Details

Code Architecture

Testing Strategy

Hardware verification

Security and Safety Considerations

Transport and Security Architecture

Why SSE (Server-Sent Events)?

Authentication and Authorisation with Auth0

Lessons Learned

What Worked Well and Lessons Learned

1. `AGENTS.md` - The Rulebook

The `input_features` vs `input_values` Bug

Simplified Python Hosting with `AddPythonModule`