DEV Community: Siddarth NS

TRACEFORGE: Building a Graph-Powered Security Intelligence Platform with FalkorDB

Siddarth NS — Sun, 18 Oct 2026 18:39:51 +0000

Turning connected security data into explainable attack-path insights

Security incidents rarely happen in isolation. A compromised workstation, an exposed service account, and a server with excessive access privileges might appear to be separate issues. But when these entities are connected, they can reveal a potential path an attacker could take toward a critical system.

That is the idea behind TRACEFORGE, a graph-powered security intelligence platform I built for FalkorDB Graph Hacks 2026, Track 01: Agents That Act on Connected Data.

The project explores how graph databases can help security teams understand relationships between assets, investigate potential attack paths, and make better-informed defensive decisions.

The Problem: Security Risks Hide in Relationships

Modern environments contain workstations, servers, identities, applications, and service accounts. Understanding each asset individually is useful, but it does not always reveal the bigger picture.

Consider a simplified scenario:

An attacker gains an initial foothold on a workstation.
That workstation can reach another system.
A service account has access to an application server.
The application server is connected to a critical server.

Each relationship might look harmless on its own. Together, they could form a potential attack path.

Security analysts need a way to explore these connections, identify paths toward critical assets, and understand why a particular route deserves attention.

TRACEFORGE was designed to explore this problem using graph-based security intelligence.

What Is TRACEFORGE?

TRACEFORGE is a security intelligence dashboard prototype designed to help security analysts and security operations teams:

Explore relationships between connected assets.
Identify candidate attack paths toward critical systems.
Discover graph entities and inspect observed relationship types.
Present graph-derived information through a dashboard.
Provide a foundation for evidence-based investigations and defensive simulations.

The goal is to make complex relationships easier to understand and give analysts a clearer starting point for investigating potential risks.

Why FalkorDB?

I chose FalkorDB because the central problem in TRACEFORGE is about relationships and traversal, not just storing individual records.

A conventional data model can store workstations, accounts, and servers in separate tables. However, exploring multi-step connections across those entities can require several joins and additional application logic.

With a graph database, entities can be represented as nodes and their connections as relationships. This makes it natural to ask questions such as:

What possible paths connect an initial foothold to a critical server?

FalkorDB serves as the graph data layer for TRACEFORGE, allowing the backend to query connected security entities and return results for analysis and visualization.

Modeling the Security Environment as a Graph

The graph represents a simplified security environment.

Nodes represent entities such as:

Workstations
Service accounts
Application servers
Critical servers
Other security-relevant assets

Relationships represent connections between those entities. Examples in the project include:

USES
AUTHENTICATES_TO
ACCESSES
REACHES

These relationships allow the backend to explore how one entity connects to another.

For example, a path might look like:

Workstation A → Service Account → Application Server → Critical Server

This is an illustrative example of the kind of route the platform is designed to investigate. Whether a path represents a genuine security risk depends on the accuracy and meaning of the underlying relationship data.

That distinction matters: a graph can reveal connectivity, but the application still needs reliable evidence and appropriate analysis to interpret what that connectivity means.

How FalkorDB Fits into the Architecture

TRACEFORGE uses a simple architecture that separates graph storage, backend processing, and dashboard presentation.

1. FalkorDB — Graph data layer

FalkorDB stores the connected entities and relationships used by the prototype. It provides the graph-query capabilities needed to explore connections and candidate paths.

2. Backend API — Query and response layer

The backend communicates with the graph database and exposes HTTP endpoints for the frontend.

Two important endpoints in the current prototype are:

/graph — retrieves graph nodes and relationships for inspection.
/paths — accepts source entities and a target, then returns candidate paths found by the backend.

A /health endpoint is also used to check backend availability.

3. TRACEFORGE dashboard — Visualization layer

The frontend calls the backend endpoints to display graph inventory and attack-path results. This separation allows the UI to consume graph data without needing to communicate directly with FalkorDB.

The dashboard also distinguishes graph-derived information from illustrative metrics and sample records. This is important because security interfaces should not present demonstration data as if it were live telemetry.

Exploring Potential Attack Paths

One of the core workflows is querying candidate paths between selected source entities and a target.

In the demonstration dataset, the backend accepts sources such as Workstation A, Secondary Foothold, and Service Account, with Critical Server as the target.

The backend then returns the candidate paths it finds in the graph. During testing, the configured query returned three paths for this synthetic dataset.

This demonstrates how graph traversal can help answer a useful security question:

Which connected routes in the modeled environment could lead toward a critical asset?

The results can provide analysts with a starting point for further investigation. In a production system, each path would need supporting evidence, clear relationship semantics, and appropriate confidence information before it could be treated as a meaningful security finding.

Beyond Path Discovery: Toward Agent-Assisted Defense

The broader direction of TRACEFORGE is to explore how an agent could use connected graph data to support security workflows.

A potential workflow would involve:

Querying the graph for relevant entities and candidate paths.
Collecting evidence that supports each relationship.
Explaining why a path may matter.
Recommending a defensive action for analyst review.
Testing the proposed change in an isolated simulation environment.
Independently checking whether the candidate path remains after the simulated change.
Recording the evidence, decision, and verification result in an audit trail.

The important principle is that finding a path, recommending a defense, and verifying a defense are different tasks.

An agent should not treat a plausible path as confirmed compromise, and a proposed remediation should not automatically become a real-world action. Analyst approval, controlled testing, and independent verification are essential parts of a trustworthy security workflow.

These are the design goals for extending TRACEFORGE beyond its current graph-query and dashboard prototype.

Challenges and Lessons Learned

Building TRACEFORGE reinforced several lessons for me.

1. Graph structure is only as useful as the data

A graph makes relationships easier to explore, but incorrect or incomplete relationships can produce misleading results. Evidence provenance and data quality are critical.

2. A candidate path is not proof of an attack

A connected route identifies a possibility in the modeled environment. It does not, by itself, prove that an attacker used the route or that every relationship is exploitable.

3. Keep the data layer separate from the interface

Using a backend API between FalkorDB and the dashboard creates a clear boundary. It also makes it easier to test graph queries independently from UI behavior.

4. Verification should be independent

If defensive simulations are added, the system should verify their outcomes by querying the resulting graph again instead of simply assuming that the proposed change worked.

5. Demonstrations should be transparent

The current prototype uses synthetic security data. Clearly labeling sample metrics, illustrative topology, and live API results helps avoid confusing a working demonstration with a fully deployed security product.

What's Next?

The next steps for TRACEFORGE are to strengthen the evidence behind graph relationships, improve the explainability of path results, and develop a more complete analyst-in-the-loop workflow.

Potential improvements include:

Evidence-backed explanations for each relationship in a candidate path.
Clearer confidence and uncertainty reporting.
Analyst approval before any defensive action.
Isolated, disposable simulation graphs for testing proposed changes.
Independent post-simulation path verification.
Auditable records of findings, approvals, and test outcomes.

These improvements would help move the project toward a more trustworthy security intelligence workflow.

Final Thoughts

FalkorDB provided a natural foundation for modeling connected security entities and exploring potential paths between them. Instead of looking at workstations, identities, and servers as isolated records, TRACEFORGE treats their relationships as an important part of the security picture.

The project has also helped me think more carefully about the difference between graph connectivity, evidence-based reasoning, and verified defensive outcomes.

For me, the exciting opportunity is not just discovering connections. It is using those connections to help analysts understand risk, evaluate options, and make better-informed decisions.

TRACEFORGE is an ongoing prototype, built with FalkorDB as its graph data layer, with the longer-term goal of making security investigations more connected, explainable, and verifiable.

FalkorDB #GraphDatabase #Cybersecurity #AI #BuildInPublic

Building an AI Trading Incident Response Agent with TrueForge and MCP

Siddarth NS — Sun, 30 Aug 2026 19:00:00 +0000

---
title: "Building an AI Trading Incident Response Agent with TrueForge and MCP"
description: "How we built Sentinel Trading Desk, an AI-powered trading incident response system using TrueForge, MCP, Alpaca, and Qodo."
tags: ai, agents, hackathon, mcp
---

# Building an AI Trading Incident Response Agent with TrueForge and MCP

Hackathons are usually about building something quickly.

For us, the interesting challenge was not simply:

> Can we build an AI agent?

It was:

> Can we build an AI agent that can investigate a trading incident, understand what happened, and safely recommend or execute a response?

That question led us to build **Sentinel Trading Desk**, an AI-powered trading incident response system using TrueForge, MCP, Alpaca, React, and Qodo.

## The Problem

Trading systems generate a large amount of information:

- Account balances
- Open positions
- Recent orders
- Portfolio performance
- Strategy activity
- Unexpected changes in equity

When something goes wrong, a trader or developer may need to manually inspect several sources to understand what happened.

For example:

> Why did the portfolio suddenly lose 3%?

Answering this might require checking positions, recent orders, account state, and historical portfolio performance.

We wanted an agent that could perform this investigation automatically.

However, an AI system with unrestricted access to trading actions introduces another problem: safety.

Our design principle became:

> Investigate automatically. Act with approval.

---

## What We Built

Sentinel Trading Desk is an AI-powered incident response dashboard for paper-trading environments.

A typical investigation follows this workflow:

```text
Trading anomaly detected
        |
        v
AI agent starts investigation
        |
        v
Check account state
        |
        v
Check positions
        |
        v
Inspect recent orders
        |
        v
Analyze portfolio history
        |
        v
Determine likely cause
        |
        v
Recommend remediation
        |
        v
Human approval
        |
        v
Execute approved action

The goal is to turn a complicated trading investigation into a structured workflow.

Architecture

The system consists of three main layers:

+-----------------------------+
|       Sentinel UI            |
|       React + Vite           |
+--------------+--------------+
               |
               v
+-----------------------------+
|          TrueForge           |
|       AI Agent Runtime      |
+--------------+--------------+
               |
               | MCP
               v
+-----------------------------+
|       Sentinel MCP Server    |
+--------------+--------------+
               |
               v
+-----------------------------+
|       Alpaca Trading API     |
|          Paper Account      |
+-----------------------------+

The separation between these components was important.

The UI does not directly manipulate the trading account.

Instead, the system follows:

UI -> TrueForge -> MCP -> Alpaca

This provides the agent with a controlled interface to the trading environment.

How We Used TrueForge

TrueForge is the agent orchestration layer of Sentinel.

We configured a TrueForge agent with access to our Sentinel MCP server.

The MCP server exposes tools including:

get_account
get_positions
get_recent_orders
get_portfolio_history
flatten_position
disable_strategy

The agent starts with read-only investigation tools.

For example:

Agent:
"Investigate the recent portfolio anomaly."

        |
        v

get_account()

        |
        v

get_positions()

        |
        v

get_recent_orders()

        |
        v

get_portfolio_history()

        |
        v

Analyze evidence

        |
        v

Determine likely cause

After analyzing the available information, the agent can recommend a remediation.

Potentially destructive operations require human approval.

This allows the agent to perform investigations autonomously while keeping a human responsible for consequential trading actions.

Why We Used MCP

We used the Model Context Protocol (MCP) as the interface between TrueForge and the trading environment.

Instead of exposing our entire backend to the agent, we expose a limited set of explicit tools.

For example:

const tools = [
  "get_account",
  "get_positions",
  "get_recent_orders",
  "get_portfolio_history",
  "flatten_position",
  "disable_strategy",
];

This gives the agent a clear capability boundary.

The MCP server is independently deployed and authenticated.

The resulting architecture is:

TrueForge Agent
      |
      | MCP
      v
Sentinel MCP Server
      |
      +---- get_account
      +---- get_positions
      +---- get_recent_orders
      +---- get_portfolio_history
      +---- flatten_position
      +---- disable_strategy
      |
      v
Alpaca

Safety and Human Approval

Trading automation requires stronger safety controls than a typical chatbot.

A wrong chatbot response may be inconvenient.

A wrong trading action can have financial consequences.

Therefore, Sentinel follows:

Read-only investigation
          |
          v
      AI analysis
          |
          v
  Proposed remediation
          |
          v
   Human approval
          |
          v
    Action execution

Actions such as:

flatten_position
disable_strategy

require explicit approval.

This gives us a human-in-the-loop control point before consequential operations.

The MCP server is also protected using a shared authentication secret.

Requests to the MCP endpoint must include:

Authorization: Bearer <MCP_SHARED_SECRET>

Using Qodo for Code Quality

We used Qodo as an additional AI-powered code review and quality-assurance layer during development.

Sentinel contains several interacting components:

React UI
   |
   +-- TrueForge SDK
   |
   +-- TrueForge Agent
   |
   +-- MCP Server
   |
   +-- Alpaca API
   |
   +-- Authentication

Because failures can occur at the boundaries between these components, code review was particularly important.

Qodo helped us identify issues around:

Error handling
Authentication
API integration
Input validation
Edge cases
Reliability

We applied the suggested fixes and tested the affected functionality again.

This gave us an additional review layer beyond our own testing.

Deployment

We deployed the components as separate services.

Sentinel UI

https://sentinel-ui-pqwl.onrender.com/

TrueForge

https://trueforge.onrender.com/

Sentinel MCP

https://sentinel-alpaca-mcp.onrender.com/mcp

The MCP server also exposes a health endpoint:

https://sentinel-alpaca-mcp.onrender.com/health

A successful health response looks like:

{
  "status": "ok"
}

This allows us to independently verify that the MCP service is running.

Debugging During the Hackathon

One of the most useful parts of the project was debugging the deployed system.

The local version worked, but deployment introduced several new problems involving:

Render configuration
MCP authentication
CORS
TrueForge API routes
Frontend environment variables
MCP connector configuration
SDK compatibility

For example, at one point the browser showed:

sessions    404
preflight   CORS

The MCP server itself was healthy and the TrueForge MCP connector showed the available Sentinel tools.

The actual problem was the communication between the Sentinel UI and the hosted TrueForge API.

This was an important lesson:

In distributed AI applications, the model is only one part of the system.

The surrounding infrastructure matters just as much.

Environment Configuration

The deployed Sentinel UI connects to the hosted TrueForge instance using:

VITE_TRUEFORGE_BASE_URL=https://trueforge.onrender.com

TrueForge then connects to the Sentinel MCP endpoint:

https://sentinel-alpaca-mcp.onrender.com/mcp

The MCP server uses a shared secret for authentication:

MCP_SHARED_SECRET=<secret>

The resulting request from TrueForge to the MCP server is:

POST /mcp
Host: sentinel-alpaca-mcp.onrender.com
Authorization: Bearer <MCP_SHARED_SECRET>

This keeps the authentication boundary between TrueForge and the MCP server.

What We Learned

1. Agentic systems need boundaries

Giving an agent access to tools is powerful.

Giving it unrestricted access to consequential actions is risky.

Tool-level permissions and human approval provide important safety boundaries.

2. MCP simplifies integrations

MCP gives us a structured interface between the agent and external capabilities.

Instead of implementing custom integrations for every agent, we can expose well-defined tools through an MCP server.

3. Deployment is part of the engineering problem

A system can work perfectly on localhost and fail after deployment.

Environment variables, authentication, CORS, API routes, and service-to-service communication all become important.

4. Debugging distributed agents requires observability

When an agent fails, "something went wrong" is not enough.

We need to determine:

Which service failed?
Which endpoint was called?
Was authentication successful?
Was the tool available?
What response was returned?
Was the failure in the agent, MCP server, or UI?

This debugging approach was essential during the hackathon.

What's Next

Sentinel currently focuses on paper trading, but the architecture can be extended.

Potential improvements include:

More sophisticated anomaly detection
Strategy-level monitoring
Automated incident classification
Multi-agent investigations
Portfolio risk analysis
More granular approval policies
Detailed audit logs
Additional trading integrations

The long-term goal is not simply to build an AI that can trade.

It is to build an AI-powered trading operations layer that helps humans understand what is happening in their systems and respond safely.

Final Thoughts

This hackathon showed us that building an AI agent is not just about connecting an LLM to a few tools.

The difficult engineering happens around the agent:

Tool design
    +
Permissions
    +
Authentication
    +
Deployment
    +
Observability
    +
Error handling
    +
Human oversight

With Sentinel Trading Desk, we wanted to explore what an AI-native trading incident response system could look like.

TrueForge provides the agent orchestration layer.

MCP provides a structured interface to the trading environment.

Alpaca provides the trading infrastructure.

Qodo provides an additional layer of code review and quality assurance.

The result is an AI agent that can investigate trading incidents, explain what happened, and help determine what to do next while keeping humans in control of consequential actions.

Project

GitHub:

https://github.com/siddarth709/trading-desk-sentinel

Live UI:

https://sentinel-ui-pqwl.onrender.com/

MCP Server:

https://sentinel-alpaca-mcp.onrender.com/mcp

ChronicleOps: Autonomous Chaos Engineering & Self-Healing Telemetry Engine published

Siddarth NS — Sun, 09 Aug 2026 17:55:44 +0000

What I Built

ChronicleOps is an autonomous cloud chaos engineering, real-time observability, and self-healing platform built natively on Zerops.

Instead of manually troubleshooting downtime, ChronicleOps lets developers spin up target application stacks, inject controlled fault vectors (such as process kills or container crashes), observe real-time telemetry degradation, and let the system automatically recover services while generating Gemini AI root-cause diagnoses.

The Problem & Solution

The Problem: Microservice failures in cloud environments are often chaotic and hard to diagnose. Reproducing fault states safely without breaking persistent infrastructure is difficult, and measuring exact Mean Time to Recovery (MTTR) requires heavy monitoring setups.
The Solution: ChronicleOps automates the entire chaos-to-recovery loop. With a single click, it provisions isolated Zerops environments, kills target processes, measures recovery latency down to the millisecond, and feeds real-time container log streams into vector-matched AI for automated incident reports.

Architecture & Tech Stack

ChronicleOps is structured as a multi-service architecture running directly on Zerops infrastructure:

Component	Stack	Description
Orchestrator	FastAPI (Python 3.11), Uvicorn	Primary API gateway, environment lifecycle orchestrator, and Gemini AI diagnosis engine.
Worker	Python 3.11, Zerops CLI (`zcli`)	Background polling daemon that monitors target health, triggers recovery restarts, and runs TTL resource pruning.
Ingestion Engine	FastAPI, Pub/Sub	Log collector and real-time event pipeline for telemetry streams.
Database & Cache	PostgreSQL, Valkey	Vector store for historical incident embeddings and real-time state synchronization.
Frontend UI	HTML5, Modern CSS3, Canvas, Vanilla JS	Zero-dependency Command Center featuring CRT scanline aesthetic, real-time Canvas telemetry charts, and interactive controls.

                     ┌─────────────────────────────────────────┐
                     │          ChronicleOps Dashboard         │
                     │    (Vanilla JS + HTML5 Canvas RTC)      │
                     └────────────────────┬────────────────────┘
                                          │
                                          ▼
                     ┌─────────────────────────────────────────┐
                     │          Orchestrator Service           │
                     │          (FastAPI / Python 3.11)        │
                     └───────┬─────────────────────────┬───────┘
                             │                         │
                             ▼                         ▼
┌─────────────────────────────────────────┐   ┌─────────────────────────────────┐
│              Worker Service             │   │       Ingestion & AI Engine     │
│   (Zerops CLI / Health & TTL Poller)    │   │  (PostgreSQL + Gemini Vector RAG)│
└─────────────────────────────────────────┘   └─────────────────────────────────┘

Key Features

1-Click Demo Quickstart: Instantly provisions an isolated application stack on Zerops using a demo repository without local configuration.
Controlled Chaos Injection: Injects process termination signals (SIGKILL / 502 Bad Gateway) into active containers to test fault tolerance.
Real-time Canvas Telemetry: Tracks 1.0s RTC uptime signals with visual telemetry drop and step-up curves.
Autonomous Self-Healing & MTTR: Detects service outages in real time, executes automated restarts via zcli, and calculates exact Mean Time to Recovery metrics.
Gemini AI Root-Cause Diagnosis: Correlates log tails against historical incident vector embeddings stored in PostgreSQL/Valkey to generate root-cause reports, confidence scores, and recommended fixes.
TTL Environment Pruning: Automatically cleans up expired demo environments and Zerops project resources to keep platform quotas clean.

How Zerops Powers ChronicleOps
ChronicleOps relies on Zerops for its deployment pipeline and multi-service orchestrations:

Isolated Multi-Service Deployments: Deploys orchestrator, worker, and ingestion services independently via zerops.yaml.
Dynamic Infrastructure Provisioning: Uses Zerops CLI (zcli) inside the worker service to spin up and tear down temporary user environments on demand.
Internal Service Discovery & Public Ingress: Connects microservices using internal Zerops environment variables (VALKEY_HOST, POSTGRES_URL) while exposing public ingress endpoints for target apps.

YAML
zerops:
  - setup: orchestrator
    build:
      base: python@3.11
      buildCommands:
        - pip install -r orchestrator/requirements.txt
      deployFiles:
        - ./orchestrator
    run:
      base: python@3.11
      ports:
        - port: 8080
          httpSupport: true
      start: uvicorn orchestrator.app.main:app --host 0.0.0.0 --port 8080

Links & Live Demo
Live Dashboard: https://orchestrator-2b86-8080.prg1.zerops.app
Main GitHub Repository: https://github.com/siddarth709/ChronicleOps
Demo App Target Repository: https://github.com/siddarth709/ChronicleOps_Demo

Peering Inside the Quant Black Box: How I Instrumented an 8-Stage ML/DL Pipeline Using OpenTelemetry & SigNoz

Siddarth NS — Sat, 25 Jul 2026 23:49:01 +0000

In high-frequency quantitative modeling and algorithmic trading, standard logging mechanisms fall drastically short. When an automated prediction pipeline produces an erratic trading signal or suffers from an unexpected latency spike, line-by-line terminal logs offer little help. Correlating raw financial tick data with downstream deep learning inferences quickly turns into an operational nightmare.

To solve this transparency deficit, I built AlphaTrace—a fresh, 7-stage stock forecasting engine powered by FinBERT, GARCH, Kalman Filters, LSTM, Transformer, XGBoost, and Black-Scholes pricing models. By instrumenting every layer end-to-end with OpenTelemetry (OTel) and routing metrics, traces, and logs through SigNoz, AlphaTrace eliminates the "black box" nature of quantitative ML/DL workflows.

Here is an architectural walkthrough of how full-stack observability transforms quantitative modeling from guesswork into a deterministic, real-time engineering discipline.

1. The Operational Challenge: The Algorithmic Black Box

Standard microservices process requests by reading from a database and returning a response. A quantitative prediction pipeline, by contrast, executes a series of heavy mathematical transformations where failure in an early stage silently degrades all downstream steps:

News Retrieval: Fetching live, current headlines for the given ticker directly from Yahoo Finance, filtered and deduplicated before scoring.
Data Ingestion: Fetching high-frequency price and volume data using yfinance.
Sentiment Analysis: Scoring news headlines via FinBERT and VADER.
Volatility Modeling: Fitting GARCH / EGARCH(1,1) time-series models to compute dynamic conditional volatility.
State Estimation: Removing market noise through a discrete Kalman Filter.
Ensemble Modeling: Synthesizing weighted outputs from LSTM, Transformer, and XGBoost architectures.
Options Pricing: Valuing synthetic options via Black-Scholes equations coupled with Monte Carlo Geometric Brownian Motion (GBM) simulations.
Signal Generation: Emitting a final BUY, SELL, or HOLD action alongside an overall confidence score.

When a model yields a low-confidence decision, traditional setups can't easily identify the root cause. Was the ingested pricing data stale? Did the GARCH optimization fail to converge? Did the deep learning ensemble exhibit extreme variance? Without distributed tracing, pin-pointing the bottleneck requires tedious manual debugging.

2. Infrastructure Deployment with SigNoz Foundry

To ensure local reproducibility, I deployed SigNoz using SigNoz Foundry. Foundry automates the environment provisioning in Docker and outputs configuration locks (casting.yaml and casting.yaml.lock) directly into the repository root.

# Provisioning SigNoz and its native MCP server via Foundry
Install signoz using foundry
foundryctl cast
foundry cast --lock

With the observability backend live, I constructed a reusable initialization module (otel_setup.py) to configure the OpenTelemetry TracerProvider and send OTLP gRPC payloads to the local collector at localhost:4317.

# otel_setup.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Define service identity
resource = Resource.create({"service.name": "alphatrace-pipeline"})
provider = TracerProvider(resource=resource)

# Configure batch exporter to local SigNoz collector
exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))

trace.set_tracer_provider(provider)
tracer = trace.get_tracer("alphatrace.pipeline")

3. Engineering the End-to-End Span Waterfall

Instead of emitting disconnected logs, every prediction cycle runs inside a root span (pipeline.run) containing eight individual child spans. This design renders every ticker analysis as a clear, multi-stage execution waterfall in the SigNoz UI.

Embedding Domain Attributes directly into Spans

Rather than pushing unstructured text strings into logs, key algorithmic parameters are bound directly to span attributes:

# Example: Volatility Estimation Stage with Fallback Instrumentation
with tracer.start_as_current_span("stage.volatility") as span:
    try:
        volatility = fit_garch(returns)
        span.set_attribute("model.garch_converged", True)
    except ConvergenceError:
        # Graceful fallback to historical volatility calculation
        volatility = historical_volatility(returns)
        span.set_attribute("model.garch_converged", False)
        span.add_event("garch_fallback_triggered")  # Explicit span event

    span.set_attribute("model.volatility", float(volatility))

By tagging model.garch_converged, model.finbert_score, and ensemble.confidence directly onto spans, algorithmic metrics become instantly searchable and queryable within SigNoz.

Correlating Metrics and Logs via Trace Identifiers

Alongside spans, OTel meters collect histograms of stage execution times, counter metrics for total runs, and gauge values measuring prediction drift (predicted price vs. actual close price). Furthermore, by routing application events through the OpenTelemetry logging bridge, runtime warnings and error messages automatically attach the active trace_id. Clicking an entry in SigNoz lets you jump instantly from a log line to its exact position in the execution waterfall.

4. Operational Control Panels via SigNoz Query Builder

Using SigNoz’s native Query Builder, I built three dedicated operational dashboards to track systemic and statistical performance:

Pipeline Latency & Health: Displays max execution time per stage, and it surfaced something I wouldn't have guessed by reading the code. In a real run, pipeline.run totaled 12.77 seconds — and stage.ingestion alone accounted for 11.56 of them. Every model in the pipeline — GARCH, the Kalman filter, the LSTM/Transformer/XGBoost ensemble, both pricing methods — combined took roughly a second. The bottleneck was never the machine learning. It was a single network call to Yahoo Finance, and the trace made that a one-line, provable diagnosis instead of a guess.
Model Health & Fallback Frequency: Tracks GARCH convergence rate, Kalman filter residuals, ensemble disagreement, and Black-Scholes/Monte Carlo pricing divergence — four panels that, on their own, look like routine time series. Together, they told a story none of them could tell alone: during one window, the GARCH fallback rate spiked, the Kalman residual spiked, and the pricing divergence between Black-Scholes and Monte Carlo spiked — all in the same few minutes. Three independently-coded models, none of which know the others exist, all reacted to whatever happened in the underlying price data at that moment. That correlation is invisible from any single dashboard or any one model's logs. It only shows up when you can put all three side by side and watch the timestamps line up.
Data Ingestion Quality: Tracks data.freshness_seconds and ingested row counts, surfacing network timeouts or stale exchange data before it reaches the deep learning layer.

Where the Models Disagree:
Observability isn't just for finding bottlenecks — it's also for catching the moments your own models don't agree with each other. Running AlphaTrace live on AAPL against real, current headlines ("4 Big Tech earnings reports, a Fed meeting, and $100 oil," among others), FinBERT scored the batch at a flat 0.0, while VADER read the same headlines at +0.075. Two sentiment models, same real input, meaningfully different conclusions.
Rather than averaging that disagreement away, sentiment.finbert_spread and sentiment.vader_spread are their own span attributes — so a flat blended score can never quietly hide two models pulling in different directions. I don't have a fully satisfying explanation for the gap yet. What I have, because of tracing, is visibility into exactly when and how often it happens, which is a more honest place to start than pretending it doesn't.
On the pricing side, the two independent methods agreed closely — Black-Scholes valued a sample AAPL option at $6.91, Monte Carlo (10,000 simulated paths) at $6.81, about 1.4% apart. Small, reassuring, and — because both numbers are span attributes on the same trace — instantly verifiable rather than asserted.

5. Conversational Observability with the SigNoz MCP Server

AlphaTrace also integrates the SigNoz MCP Server paired with the Claude Code agent-skills plugin. This setup allows developers and operators to inspect telemetry and configure alerts using natural language prompts directly from the CLI:

This turns SigNoz querying into a conversation instead of a dashboard hunt. A developer can ask something like "show me the latency breakdown across all pipeline stages over the last hour and list any failed GARCH convergences," and the MCP server answers from the live trace and metric data itself — no manual Query Builder construction required. In practice, this is where the ingestion bottleneck and the GARCH fallback pattern above were first noticed, before they became dashboard panels.

Key Takeaways

Rich Attributes Over Unstructured Text: Attaching quantitative metadata (model.confidence, pricing.black_scholes, data.freshness) to spans turns observability tools into real-time model auditing platforms.
Parent-Child Span Hierarchies: Wrapping sequential ML workflows in structured parent/child relationships makes identifying bottlenecks straightforward.
Automated Guardrails Keep Models Safe: Setting automated alerts on data staleness and model non-convergence prevents bad data from triggering automated trades.
Correlated Signals Beat Isolated Metrics: No single dashboard panel told the real story on its own. It was only visible by tracing enough of the pipeline, in enough detail, to watch three unrelated models react to the same event at the same time.

By combining OpenTelemetry with SigNoz, complex machine learning systems no longer need to be black boxes. You get complete visibility into the health, efficiency, and accuracy of every prediction your models make.

GitHub Repository: github.com/siddarth709/alphatrace
Tech Stack: SigNoz, OpenTelemetry, PyTorch, XGBoost, Arch, React