DEV Community: Siddarth NS

Building an AI Trading Incident Response Agent with TrueForge and MCP

Siddarth NS — Sun, 30 Aug 2026 19:00:00 +0000

---
title: "Building an AI Trading Incident Response Agent with TrueForge and MCP"
description: "How we built Sentinel Trading Desk, an AI-powered trading incident response system using TrueForge, MCP, Alpaca, and Qodo."
tags: ai, agents, hackathon, mcp
---

# Building an AI Trading Incident Response Agent with TrueForge and MCP

Hackathons are usually about building something quickly.

For us, the interesting challenge was not simply:

> Can we build an AI agent?

It was:

> Can we build an AI agent that can investigate a trading incident, understand what happened, and safely recommend or execute a response?

That question led us to build **Sentinel Trading Desk**, an AI-powered trading incident response system using TrueForge, MCP, Alpaca, React, and Qodo.

## The Problem

Trading systems generate a large amount of information:

- Account balances
- Open positions
- Recent orders
- Portfolio performance
- Strategy activity
- Unexpected changes in equity

When something goes wrong, a trader or developer may need to manually inspect several sources to understand what happened.

For example:

> Why did the portfolio suddenly lose 3%?

Answering this might require checking positions, recent orders, account state, and historical portfolio performance.

We wanted an agent that could perform this investigation automatically.

However, an AI system with unrestricted access to trading actions introduces another problem: safety.

Our design principle became:

> Investigate automatically. Act with approval.

---

## What We Built

Sentinel Trading Desk is an AI-powered incident response dashboard for paper-trading environments.

A typical investigation follows this workflow:

```text
Trading anomaly detected
        |
        v
AI agent starts investigation
        |
        v
Check account state
        |
        v
Check positions
        |
        v
Inspect recent orders
        |
        v
Analyze portfolio history
        |
        v
Determine likely cause
        |
        v
Recommend remediation
        |
        v
Human approval
        |
        v
Execute approved action

The goal is to turn a complicated trading investigation into a structured workflow.

Architecture

The system consists of three main layers:

+-----------------------------+
|       Sentinel UI            |
|       React + Vite           |
+--------------+--------------+
               |
               v
+-----------------------------+
|          TrueForge           |
|       AI Agent Runtime      |
+--------------+--------------+
               |
               | MCP
               v
+-----------------------------+
|       Sentinel MCP Server    |
+--------------+--------------+
               |
               v
+-----------------------------+
|       Alpaca Trading API     |
|          Paper Account      |
+-----------------------------+

The separation between these components was important.

The UI does not directly manipulate the trading account.

Instead, the system follows:

UI -> TrueForge -> MCP -> Alpaca

This provides the agent with a controlled interface to the trading environment.

How We Used TrueForge

TrueForge is the agent orchestration layer of Sentinel.

We configured a TrueForge agent with access to our Sentinel MCP server.

The MCP server exposes tools including:

get_account
get_positions
get_recent_orders
get_portfolio_history
flatten_position
disable_strategy

The agent starts with read-only investigation tools.

For example:

Agent:
"Investigate the recent portfolio anomaly."

        |
        v

get_account()

        |
        v

get_positions()

        |
        v

get_recent_orders()

        |
        v

get_portfolio_history()

        |
        v

Analyze evidence

        |
        v

Determine likely cause

After analyzing the available information, the agent can recommend a remediation.

Potentially destructive operations require human approval.

This allows the agent to perform investigations autonomously while keeping a human responsible for consequential trading actions.

Why We Used MCP

We used the Model Context Protocol (MCP) as the interface between TrueForge and the trading environment.

Instead of exposing our entire backend to the agent, we expose a limited set of explicit tools.

For example:

const tools = [
  "get_account",
  "get_positions",
  "get_recent_orders",
  "get_portfolio_history",
  "flatten_position",
  "disable_strategy",
];

This gives the agent a clear capability boundary.

The MCP server is independently deployed and authenticated.

The resulting architecture is:

TrueForge Agent
      |
      | MCP
      v
Sentinel MCP Server
      |
      +---- get_account
      +---- get_positions
      +---- get_recent_orders
      +---- get_portfolio_history
      +---- flatten_position
      +---- disable_strategy
      |
      v
Alpaca

Safety and Human Approval

Trading automation requires stronger safety controls than a typical chatbot.

A wrong chatbot response may be inconvenient.

A wrong trading action can have financial consequences.

Therefore, Sentinel follows:

Read-only investigation
          |
          v
      AI analysis
          |
          v
  Proposed remediation
          |
          v
   Human approval
          |
          v
    Action execution

Actions such as:

flatten_position
disable_strategy

require explicit approval.

This gives us a human-in-the-loop control point before consequential operations.

The MCP server is also protected using a shared authentication secret.

Requests to the MCP endpoint must include:

Authorization: Bearer <MCP_SHARED_SECRET>

Using Qodo for Code Quality

We used Qodo as an additional AI-powered code review and quality-assurance layer during development.

Sentinel contains several interacting components:

React UI
   |
   +-- TrueForge SDK
   |
   +-- TrueForge Agent
   |
   +-- MCP Server
   |
   +-- Alpaca API
   |
   +-- Authentication

Because failures can occur at the boundaries between these components, code review was particularly important.

Qodo helped us identify issues around:

Error handling
Authentication
API integration
Input validation
Edge cases
Reliability

We applied the suggested fixes and tested the affected functionality again.

This gave us an additional review layer beyond our own testing.

Deployment

We deployed the components as separate services.

Sentinel UI

https://sentinel-ui-pqwl.onrender.com/

TrueForge

https://trueforge.onrender.com/

Sentinel MCP

https://sentinel-alpaca-mcp.onrender.com/mcp

The MCP server also exposes a health endpoint:

https://sentinel-alpaca-mcp.onrender.com/health

A successful health response looks like:

{
  "status": "ok"
}

This allows us to independently verify that the MCP service is running.

Debugging During the Hackathon

One of the most useful parts of the project was debugging the deployed system.

The local version worked, but deployment introduced several new problems involving:

Render configuration
MCP authentication
CORS
TrueForge API routes
Frontend environment variables
MCP connector configuration
SDK compatibility

For example, at one point the browser showed:

sessions    404
preflight   CORS

The MCP server itself was healthy and the TrueForge MCP connector showed the available Sentinel tools.

The actual problem was the communication between the Sentinel UI and the hosted TrueForge API.

This was an important lesson:

In distributed AI applications, the model is only one part of the system.

The surrounding infrastructure matters just as much.

Environment Configuration

The deployed Sentinel UI connects to the hosted TrueForge instance using:

VITE_TRUEFORGE_BASE_URL=https://trueforge.onrender.com

TrueForge then connects to the Sentinel MCP endpoint:

https://sentinel-alpaca-mcp.onrender.com/mcp

The MCP server uses a shared secret for authentication:

MCP_SHARED_SECRET=<secret>

The resulting request from TrueForge to the MCP server is:

POST /mcp
Host: sentinel-alpaca-mcp.onrender.com
Authorization: Bearer <MCP_SHARED_SECRET>

This keeps the authentication boundary between TrueForge and the MCP server.

What We Learned

1. Agentic systems need boundaries

Giving an agent access to tools is powerful.

Giving it unrestricted access to consequential actions is risky.

Tool-level permissions and human approval provide important safety boundaries.

2. MCP simplifies integrations

MCP gives us a structured interface between the agent and external capabilities.

Instead of implementing custom integrations for every agent, we can expose well-defined tools through an MCP server.

3. Deployment is part of the engineering problem

A system can work perfectly on localhost and fail after deployment.

Environment variables, authentication, CORS, API routes, and service-to-service communication all become important.

4. Debugging distributed agents requires observability

When an agent fails, "something went wrong" is not enough.

We need to determine:

Which service failed?
Which endpoint was called?
Was authentication successful?
Was the tool available?
What response was returned?
Was the failure in the agent, MCP server, or UI?

This debugging approach was essential during the hackathon.

What's Next

Sentinel currently focuses on paper trading, but the architecture can be extended.

Potential improvements include:

More sophisticated anomaly detection
Strategy-level monitoring
Automated incident classification
Multi-agent investigations
Portfolio risk analysis
More granular approval policies
Detailed audit logs
Additional trading integrations

The long-term goal is not simply to build an AI that can trade.

It is to build an AI-powered trading operations layer that helps humans understand what is happening in their systems and respond safely.

Final Thoughts

This hackathon showed us that building an AI agent is not just about connecting an LLM to a few tools.

The difficult engineering happens around the agent:

Tool design
    +
Permissions
    +
Authentication
    +
Deployment
    +
Observability
    +
Error handling
    +
Human oversight

With Sentinel Trading Desk, we wanted to explore what an AI-native trading incident response system could look like.

TrueForge provides the agent orchestration layer.

MCP provides a structured interface to the trading environment.

Alpaca provides the trading infrastructure.

Qodo provides an additional layer of code review and quality assurance.

The result is an AI agent that can investigate trading incidents, explain what happened, and help determine what to do next while keeping humans in control of consequential actions.

Project

GitHub:

https://github.com/siddarth709/trading-desk-sentinel

Live UI:

https://sentinel-ui-pqwl.onrender.com/

MCP Server:

https://sentinel-alpaca-mcp.onrender.com/mcp

ChronicleOps: Autonomous Chaos Engineering & Self-Healing Telemetry Engine published

Siddarth NS — Sun, 09 Aug 2026 17:55:44 +0000

What I Built

ChronicleOps is an autonomous cloud chaos engineering, real-time observability, and self-healing platform built natively on Zerops.

Instead of manually troubleshooting downtime, ChronicleOps lets developers spin up target application stacks, inject controlled fault vectors (such as process kills or container crashes), observe real-time telemetry degradation, and let the system automatically recover services while generating Gemini AI root-cause diagnoses.

The Problem & Solution

The Problem: Microservice failures in cloud environments are often chaotic and hard to diagnose. Reproducing fault states safely without breaking persistent infrastructure is difficult, and measuring exact Mean Time to Recovery (MTTR) requires heavy monitoring setups.
The Solution: ChronicleOps automates the entire chaos-to-recovery loop. With a single click, it provisions isolated Zerops environments, kills target processes, measures recovery latency down to the millisecond, and feeds real-time container log streams into vector-matched AI for automated incident reports.

Architecture & Tech Stack

ChronicleOps is structured as a multi-service architecture running directly on Zerops infrastructure:

Component	Stack	Description
Orchestrator	FastAPI (Python 3.11), Uvicorn	Primary API gateway, environment lifecycle orchestrator, and Gemini AI diagnosis engine.
Worker	Python 3.11, Zerops CLI (`zcli`)	Background polling daemon that monitors target health, triggers recovery restarts, and runs TTL resource pruning.
Ingestion Engine	FastAPI, Pub/Sub	Log collector and real-time event pipeline for telemetry streams.
Database & Cache	PostgreSQL, Valkey	Vector store for historical incident embeddings and real-time state synchronization.
Frontend UI	HTML5, Modern CSS3, Canvas, Vanilla JS	Zero-dependency Command Center featuring CRT scanline aesthetic, real-time Canvas telemetry charts, and interactive controls.

                     ┌─────────────────────────────────────────┐
                     │          ChronicleOps Dashboard         │
                     │    (Vanilla JS + HTML5 Canvas RTC)      │
                     └────────────────────┬────────────────────┘
                                          │
                                          ▼
                     ┌─────────────────────────────────────────┐
                     │          Orchestrator Service           │
                     │          (FastAPI / Python 3.11)        │
                     └───────┬─────────────────────────┬───────┘
                             │                         │
                             ▼                         ▼
┌─────────────────────────────────────────┐   ┌─────────────────────────────────┐
│              Worker Service             │   │       Ingestion & AI Engine     │
│   (Zerops CLI / Health & TTL Poller)    │   │  (PostgreSQL + Gemini Vector RAG)│
└─────────────────────────────────────────┘   └─────────────────────────────────┘

Key Features

1-Click Demo Quickstart: Instantly provisions an isolated application stack on Zerops using a demo repository without local configuration.
Controlled Chaos Injection: Injects process termination signals (SIGKILL / 502 Bad Gateway) into active containers to test fault tolerance.
Real-time Canvas Telemetry: Tracks 1.0s RTC uptime signals with visual telemetry drop and step-up curves.
Autonomous Self-Healing & MTTR: Detects service outages in real time, executes automated restarts via zcli, and calculates exact Mean Time to Recovery metrics.
Gemini AI Root-Cause Diagnosis: Correlates log tails against historical incident vector embeddings stored in PostgreSQL/Valkey to generate root-cause reports, confidence scores, and recommended fixes.
TTL Environment Pruning: Automatically cleans up expired demo environments and Zerops project resources to keep platform quotas clean.

How Zerops Powers ChronicleOps
ChronicleOps relies on Zerops for its deployment pipeline and multi-service orchestrations:

Isolated Multi-Service Deployments: Deploys orchestrator, worker, and ingestion services independently via zerops.yaml.
Dynamic Infrastructure Provisioning: Uses Zerops CLI (zcli) inside the worker service to spin up and tear down temporary user environments on demand.
Internal Service Discovery & Public Ingress: Connects microservices using internal Zerops environment variables (VALKEY_HOST, POSTGRES_URL) while exposing public ingress endpoints for target apps.

YAML
zerops:
  - setup: orchestrator
    build:
      base: python@3.11
      buildCommands:
        - pip install -r orchestrator/requirements.txt
      deployFiles:
        - ./orchestrator
    run:
      base: python@3.11
      ports:
        - port: 8080
          httpSupport: true
      start: uvicorn orchestrator.app.main:app --host 0.0.0.0 --port 8080

Links & Live Demo
Live Dashboard: https://orchestrator-2b86-8080.prg1.zerops.app
Main GitHub Repository: https://github.com/siddarth709/ChronicleOps
Demo App Target Repository: https://github.com/siddarth709/ChronicleOps_Demo

Peering Inside the Quant Black Box: How I Instrumented an 8-Stage ML/DL Pipeline Using OpenTelemetry & SigNoz

Siddarth NS — Sat, 25 Jul 2026 23:49:01 +0000

In high-frequency quantitative modeling and algorithmic trading, standard logging mechanisms fall drastically short. When an automated prediction pipeline produces an erratic trading signal or suffers from an unexpected latency spike, line-by-line terminal logs offer little help. Correlating raw financial tick data with downstream deep learning inferences quickly turns into an operational nightmare.

To solve this transparency deficit, I built AlphaTrace—a fresh, 7-stage stock forecasting engine powered by FinBERT, GARCH, Kalman Filters, LSTM, Transformer, XGBoost, and Black-Scholes pricing models. By instrumenting every layer end-to-end with OpenTelemetry (OTel) and routing metrics, traces, and logs through SigNoz, AlphaTrace eliminates the "black box" nature of quantitative ML/DL workflows.

Here is an architectural walkthrough of how full-stack observability transforms quantitative modeling from guesswork into a deterministic, real-time engineering discipline.

1. The Operational Challenge: The Algorithmic Black Box

Standard microservices process requests by reading from a database and returning a response. A quantitative prediction pipeline, by contrast, executes a series of heavy mathematical transformations where failure in an early stage silently degrades all downstream steps:

News Retrieval: Fetching live, current headlines for the given ticker directly from Yahoo Finance, filtered and deduplicated before scoring.
Data Ingestion: Fetching high-frequency price and volume data using yfinance.
Sentiment Analysis: Scoring news headlines via FinBERT and VADER.
Volatility Modeling: Fitting GARCH / EGARCH(1,1) time-series models to compute dynamic conditional volatility.
State Estimation: Removing market noise through a discrete Kalman Filter.
Ensemble Modeling: Synthesizing weighted outputs from LSTM, Transformer, and XGBoost architectures.
Options Pricing: Valuing synthetic options via Black-Scholes equations coupled with Monte Carlo Geometric Brownian Motion (GBM) simulations.
Signal Generation: Emitting a final BUY, SELL, or HOLD action alongside an overall confidence score.

When a model yields a low-confidence decision, traditional setups can't easily identify the root cause. Was the ingested pricing data stale? Did the GARCH optimization fail to converge? Did the deep learning ensemble exhibit extreme variance? Without distributed tracing, pin-pointing the bottleneck requires tedious manual debugging.

2. Infrastructure Deployment with SigNoz Foundry

To ensure local reproducibility, I deployed SigNoz using SigNoz Foundry. Foundry automates the environment provisioning in Docker and outputs configuration locks (casting.yaml and casting.yaml.lock) directly into the repository root.

# Provisioning SigNoz and its native MCP server via Foundry
Install signoz using foundry
foundryctl cast
foundry cast --lock

With the observability backend live, I constructed a reusable initialization module (otel_setup.py) to configure the OpenTelemetry TracerProvider and send OTLP gRPC payloads to the local collector at localhost:4317.

# otel_setup.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Define service identity
resource = Resource.create({"service.name": "alphatrace-pipeline"})
provider = TracerProvider(resource=resource)

# Configure batch exporter to local SigNoz collector
exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))

trace.set_tracer_provider(provider)
tracer = trace.get_tracer("alphatrace.pipeline")

3. Engineering the End-to-End Span Waterfall

Instead of emitting disconnected logs, every prediction cycle runs inside a root span (pipeline.run) containing eight individual child spans. This design renders every ticker analysis as a clear, multi-stage execution waterfall in the SigNoz UI.

Embedding Domain Attributes directly into Spans

Rather than pushing unstructured text strings into logs, key algorithmic parameters are bound directly to span attributes:

# Example: Volatility Estimation Stage with Fallback Instrumentation
with tracer.start_as_current_span("stage.volatility") as span:
    try:
        volatility = fit_garch(returns)
        span.set_attribute("model.garch_converged", True)
    except ConvergenceError:
        # Graceful fallback to historical volatility calculation
        volatility = historical_volatility(returns)
        span.set_attribute("model.garch_converged", False)
        span.add_event("garch_fallback_triggered")  # Explicit span event

    span.set_attribute("model.volatility", float(volatility))

By tagging model.garch_converged, model.finbert_score, and ensemble.confidence directly onto spans, algorithmic metrics become instantly searchable and queryable within SigNoz.

Correlating Metrics and Logs via Trace Identifiers

Alongside spans, OTel meters collect histograms of stage execution times, counter metrics for total runs, and gauge values measuring prediction drift (predicted price vs. actual close price). Furthermore, by routing application events through the OpenTelemetry logging bridge, runtime warnings and error messages automatically attach the active trace_id. Clicking an entry in SigNoz lets you jump instantly from a log line to its exact position in the execution waterfall.

4. Operational Control Panels via SigNoz Query Builder

Using SigNoz’s native Query Builder, I built three dedicated operational dashboards to track systemic and statistical performance:

Pipeline Latency & Health: Displays max execution time per stage, and it surfaced something I wouldn't have guessed by reading the code. In a real run, pipeline.run totaled 12.77 seconds — and stage.ingestion alone accounted for 11.56 of them. Every model in the pipeline — GARCH, the Kalman filter, the LSTM/Transformer/XGBoost ensemble, both pricing methods — combined took roughly a second. The bottleneck was never the machine learning. It was a single network call to Yahoo Finance, and the trace made that a one-line, provable diagnosis instead of a guess.
Model Health & Fallback Frequency: Tracks GARCH convergence rate, Kalman filter residuals, ensemble disagreement, and Black-Scholes/Monte Carlo pricing divergence — four panels that, on their own, look like routine time series. Together, they told a story none of them could tell alone: during one window, the GARCH fallback rate spiked, the Kalman residual spiked, and the pricing divergence between Black-Scholes and Monte Carlo spiked — all in the same few minutes. Three independently-coded models, none of which know the others exist, all reacted to whatever happened in the underlying price data at that moment. That correlation is invisible from any single dashboard or any one model's logs. It only shows up when you can put all three side by side and watch the timestamps line up.
Data Ingestion Quality: Tracks data.freshness_seconds and ingested row counts, surfacing network timeouts or stale exchange data before it reaches the deep learning layer.

Where the Models Disagree:
Observability isn't just for finding bottlenecks — it's also for catching the moments your own models don't agree with each other. Running AlphaTrace live on AAPL against real, current headlines ("4 Big Tech earnings reports, a Fed meeting, and $100 oil," among others), FinBERT scored the batch at a flat 0.0, while VADER read the same headlines at +0.075. Two sentiment models, same real input, meaningfully different conclusions.
Rather than averaging that disagreement away, sentiment.finbert_spread and sentiment.vader_spread are their own span attributes — so a flat blended score can never quietly hide two models pulling in different directions. I don't have a fully satisfying explanation for the gap yet. What I have, because of tracing, is visibility into exactly when and how often it happens, which is a more honest place to start than pretending it doesn't.
On the pricing side, the two independent methods agreed closely — Black-Scholes valued a sample AAPL option at $6.91, Monte Carlo (10,000 simulated paths) at $6.81, about 1.4% apart. Small, reassuring, and — because both numbers are span attributes on the same trace — instantly verifiable rather than asserted.

5. Conversational Observability with the SigNoz MCP Server

AlphaTrace also integrates the SigNoz MCP Server paired with the Claude Code agent-skills plugin. This setup allows developers and operators to inspect telemetry and configure alerts using natural language prompts directly from the CLI:

This turns SigNoz querying into a conversation instead of a dashboard hunt. A developer can ask something like "show me the latency breakdown across all pipeline stages over the last hour and list any failed GARCH convergences," and the MCP server answers from the live trace and metric data itself — no manual Query Builder construction required. In practice, this is where the ingestion bottleneck and the GARCH fallback pattern above were first noticed, before they became dashboard panels.

Key Takeaways

Rich Attributes Over Unstructured Text: Attaching quantitative metadata (model.confidence, pricing.black_scholes, data.freshness) to spans turns observability tools into real-time model auditing platforms.
Parent-Child Span Hierarchies: Wrapping sequential ML workflows in structured parent/child relationships makes identifying bottlenecks straightforward.
Automated Guardrails Keep Models Safe: Setting automated alerts on data staleness and model non-convergence prevents bad data from triggering automated trades.
Correlated Signals Beat Isolated Metrics: No single dashboard panel told the real story on its own. It was only visible by tracing enough of the pipeline, in enough detail, to watch three unrelated models react to the same event at the same time.

By combining OpenTelemetry with SigNoz, complex machine learning systems no longer need to be black boxes. You get complete visibility into the health, efficiency, and accuracy of every prediction your models make.

GitHub Repository: github.com/siddarth709/alphatrace
Tech Stack: SigNoz, OpenTelemetry, PyTorch, XGBoost, Arch, React