DEV Community: Siddarth NS

ChronicleOps: Autonomous Chaos Engineering & Self-Healing Telemetry Engine published

Siddarth NS — Sun, 09 Aug 2026 17:55:44 +0000

What I Built

ChronicleOps is an autonomous cloud chaos engineering, real-time observability, and self-healing platform built natively on Zerops.

Instead of manually troubleshooting downtime, ChronicleOps lets developers spin up target application stacks, inject controlled fault vectors (such as process kills or container crashes), observe real-time telemetry degradation, and let the system automatically recover services while generating Gemini AI root-cause diagnoses.

The Problem & Solution

The Problem: Microservice failures in cloud environments are often chaotic and hard to diagnose. Reproducing fault states safely without breaking persistent infrastructure is difficult, and measuring exact Mean Time to Recovery (MTTR) requires heavy monitoring setups.
The Solution: ChronicleOps automates the entire chaos-to-recovery loop. With a single click, it provisions isolated Zerops environments, kills target processes, measures recovery latency down to the millisecond, and feeds real-time container log streams into vector-matched AI for automated incident reports.

Architecture & Tech Stack

ChronicleOps is structured as a multi-service architecture running directly on Zerops infrastructure:

Component	Stack	Description
Orchestrator	FastAPI (Python 3.11), Uvicorn	Primary API gateway, environment lifecycle orchestrator, and Gemini AI diagnosis engine.
Worker	Python 3.11, Zerops CLI (`zcli`)	Background polling daemon that monitors target health, triggers recovery restarts, and runs TTL resource pruning.
Ingestion Engine	FastAPI, Pub/Sub	Log collector and real-time event pipeline for telemetry streams.
Database & Cache	PostgreSQL, Valkey	Vector store for historical incident embeddings and real-time state synchronization.
Frontend UI	HTML5, Modern CSS3, Canvas, Vanilla JS	Zero-dependency Command Center featuring CRT scanline aesthetic, real-time Canvas telemetry charts, and interactive controls.

                     ┌─────────────────────────────────────────┐
                     │          ChronicleOps Dashboard         │
                     │    (Vanilla JS + HTML5 Canvas RTC)      │
                     └────────────────────┬────────────────────┘
                                          │
                                          ▼
                     ┌─────────────────────────────────────────┐
                     │          Orchestrator Service           │
                     │          (FastAPI / Python 3.11)        │
                     └───────┬─────────────────────────┬───────┘
                             │                         │
                             ▼                         ▼
┌─────────────────────────────────────────┐   ┌─────────────────────────────────┐
│              Worker Service             │   │       Ingestion & AI Engine     │
│   (Zerops CLI / Health & TTL Poller)    │   │  (PostgreSQL + Gemini Vector RAG)│
└─────────────────────────────────────────┘   └─────────────────────────────────┘

Key Features

1-Click Demo Quickstart: Instantly provisions an isolated application stack on Zerops using a demo repository without local configuration.
Controlled Chaos Injection: Injects process termination signals (SIGKILL / 502 Bad Gateway) into active containers to test fault tolerance.
Real-time Canvas Telemetry: Tracks 1.0s RTC uptime signals with visual telemetry drop and step-up curves.
Autonomous Self-Healing & MTTR: Detects service outages in real time, executes automated restarts via zcli, and calculates exact Mean Time to Recovery metrics.
Gemini AI Root-Cause Diagnosis: Correlates log tails against historical incident vector embeddings stored in PostgreSQL/Valkey to generate root-cause reports, confidence scores, and recommended fixes.
TTL Environment Pruning: Automatically cleans up expired demo environments and Zerops project resources to keep platform quotas clean.

How Zerops Powers ChronicleOps
ChronicleOps relies on Zerops for its deployment pipeline and multi-service orchestrations:

Isolated Multi-Service Deployments: Deploys orchestrator, worker, and ingestion services independently via zerops.yaml.
Dynamic Infrastructure Provisioning: Uses Zerops CLI (zcli) inside the worker service to spin up and tear down temporary user environments on demand.
Internal Service Discovery & Public Ingress: Connects microservices using internal Zerops environment variables (VALKEY_HOST, POSTGRES_URL) while exposing public ingress endpoints for target apps.

YAML
zerops:
  - setup: orchestrator
    build:
      base: python@3.11
      buildCommands:
        - pip install -r orchestrator/requirements.txt
      deployFiles:
        - ./orchestrator
    run:
      base: python@3.11
      ports:
        - port: 8080
          httpSupport: true
      start: uvicorn orchestrator.app.main:app --host 0.0.0.0 --port 8080

Links & Live Demo
Live Dashboard: https://orchestrator-2b86-8080.prg1.zerops.app
Main GitHub Repository: https://github.com/siddarth709/ChronicleOps
Demo App Target Repository: https://github.com/siddarth709/ChronicleOps_Demo

Peering Inside the Quant Black Box: How I Instrumented an 8-Stage ML/DL Pipeline Using OpenTelemetry & SigNoz

Siddarth NS — Sat, 25 Jul 2026 23:49:01 +0000

In high-frequency quantitative modeling and algorithmic trading, standard logging mechanisms fall drastically short. When an automated prediction pipeline produces an erratic trading signal or suffers from an unexpected latency spike, line-by-line terminal logs offer little help. Correlating raw financial tick data with downstream deep learning inferences quickly turns into an operational nightmare.

To solve this transparency deficit, I built AlphaTrace—a fresh, 7-stage stock forecasting engine powered by FinBERT, GARCH, Kalman Filters, LSTM, Transformer, XGBoost, and Black-Scholes pricing models. By instrumenting every layer end-to-end with OpenTelemetry (OTel) and routing metrics, traces, and logs through SigNoz, AlphaTrace eliminates the "black box" nature of quantitative ML/DL workflows.

Here is an architectural walkthrough of how full-stack observability transforms quantitative modeling from guesswork into a deterministic, real-time engineering discipline.

1. The Operational Challenge: The Algorithmic Black Box

Standard microservices process requests by reading from a database and returning a response. A quantitative prediction pipeline, by contrast, executes a series of heavy mathematical transformations where failure in an early stage silently degrades all downstream steps:

News Retrieval: Fetching live, current headlines for the given ticker directly from Yahoo Finance, filtered and deduplicated before scoring.
Data Ingestion: Fetching high-frequency price and volume data using yfinance.
Sentiment Analysis: Scoring news headlines via FinBERT and VADER.
Volatility Modeling: Fitting GARCH / EGARCH(1,1) time-series models to compute dynamic conditional volatility.
State Estimation: Removing market noise through a discrete Kalman Filter.
Ensemble Modeling: Synthesizing weighted outputs from LSTM, Transformer, and XGBoost architectures.
Options Pricing: Valuing synthetic options via Black-Scholes equations coupled with Monte Carlo Geometric Brownian Motion (GBM) simulations.
Signal Generation: Emitting a final BUY, SELL, or HOLD action alongside an overall confidence score.

When a model yields a low-confidence decision, traditional setups can't easily identify the root cause. Was the ingested pricing data stale? Did the GARCH optimization fail to converge? Did the deep learning ensemble exhibit extreme variance? Without distributed tracing, pin-pointing the bottleneck requires tedious manual debugging.

2. Infrastructure Deployment with SigNoz Foundry

To ensure local reproducibility, I deployed SigNoz using SigNoz Foundry. Foundry automates the environment provisioning in Docker and outputs configuration locks (casting.yaml and casting.yaml.lock) directly into the repository root.

# Provisioning SigNoz and its native MCP server via Foundry
Install signoz using foundry
foundryctl cast
foundry cast --lock

With the observability backend live, I constructed a reusable initialization module (otel_setup.py) to configure the OpenTelemetry TracerProvider and send OTLP gRPC payloads to the local collector at localhost:4317.

# otel_setup.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Define service identity
resource = Resource.create({"service.name": "alphatrace-pipeline"})
provider = TracerProvider(resource=resource)

# Configure batch exporter to local SigNoz collector
exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))

trace.set_tracer_provider(provider)
tracer = trace.get_tracer("alphatrace.pipeline")

3. Engineering the End-to-End Span Waterfall

Instead of emitting disconnected logs, every prediction cycle runs inside a root span (pipeline.run) containing eight individual child spans. This design renders every ticker analysis as a clear, multi-stage execution waterfall in the SigNoz UI.

Embedding Domain Attributes directly into Spans

Rather than pushing unstructured text strings into logs, key algorithmic parameters are bound directly to span attributes:

# Example: Volatility Estimation Stage with Fallback Instrumentation
with tracer.start_as_current_span("stage.volatility") as span:
    try:
        volatility = fit_garch(returns)
        span.set_attribute("model.garch_converged", True)
    except ConvergenceError:
        # Graceful fallback to historical volatility calculation
        volatility = historical_volatility(returns)
        span.set_attribute("model.garch_converged", False)
        span.add_event("garch_fallback_triggered")  # Explicit span event

    span.set_attribute("model.volatility", float(volatility))

By tagging model.garch_converged, model.finbert_score, and ensemble.confidence directly onto spans, algorithmic metrics become instantly searchable and queryable within SigNoz.

Correlating Metrics and Logs via Trace Identifiers

Alongside spans, OTel meters collect histograms of stage execution times, counter metrics for total runs, and gauge values measuring prediction drift (predicted price vs. actual close price). Furthermore, by routing application events through the OpenTelemetry logging bridge, runtime warnings and error messages automatically attach the active trace_id. Clicking an entry in SigNoz lets you jump instantly from a log line to its exact position in the execution waterfall.

4. Operational Control Panels via SigNoz Query Builder

Using SigNoz’s native Query Builder, I built three dedicated operational dashboards to track systemic and statistical performance:

Pipeline Latency & Health: Displays max execution time per stage, and it surfaced something I wouldn't have guessed by reading the code. In a real run, pipeline.run totaled 12.77 seconds — and stage.ingestion alone accounted for 11.56 of them. Every model in the pipeline — GARCH, the Kalman filter, the LSTM/Transformer/XGBoost ensemble, both pricing methods — combined took roughly a second. The bottleneck was never the machine learning. It was a single network call to Yahoo Finance, and the trace made that a one-line, provable diagnosis instead of a guess.
Model Health & Fallback Frequency: Tracks GARCH convergence rate, Kalman filter residuals, ensemble disagreement, and Black-Scholes/Monte Carlo pricing divergence — four panels that, on their own, look like routine time series. Together, they told a story none of them could tell alone: during one window, the GARCH fallback rate spiked, the Kalman residual spiked, and the pricing divergence between Black-Scholes and Monte Carlo spiked — all in the same few minutes. Three independently-coded models, none of which know the others exist, all reacted to whatever happened in the underlying price data at that moment. That correlation is invisible from any single dashboard or any one model's logs. It only shows up when you can put all three side by side and watch the timestamps line up.
Data Ingestion Quality: Tracks data.freshness_seconds and ingested row counts, surfacing network timeouts or stale exchange data before it reaches the deep learning layer.

Where the Models Disagree:
Observability isn't just for finding bottlenecks — it's also for catching the moments your own models don't agree with each other. Running AlphaTrace live on AAPL against real, current headlines ("4 Big Tech earnings reports, a Fed meeting, and $100 oil," among others), FinBERT scored the batch at a flat 0.0, while VADER read the same headlines at +0.075. Two sentiment models, same real input, meaningfully different conclusions.
Rather than averaging that disagreement away, sentiment.finbert_spread and sentiment.vader_spread are their own span attributes — so a flat blended score can never quietly hide two models pulling in different directions. I don't have a fully satisfying explanation for the gap yet. What I have, because of tracing, is visibility into exactly when and how often it happens, which is a more honest place to start than pretending it doesn't.
On the pricing side, the two independent methods agreed closely — Black-Scholes valued a sample AAPL option at $6.91, Monte Carlo (10,000 simulated paths) at $6.81, about 1.4% apart. Small, reassuring, and — because both numbers are span attributes on the same trace — instantly verifiable rather than asserted.

5. Conversational Observability with the SigNoz MCP Server

AlphaTrace also integrates the SigNoz MCP Server paired with the Claude Code agent-skills plugin. This setup allows developers and operators to inspect telemetry and configure alerts using natural language prompts directly from the CLI:

This turns SigNoz querying into a conversation instead of a dashboard hunt. A developer can ask something like "show me the latency breakdown across all pipeline stages over the last hour and list any failed GARCH convergences," and the MCP server answers from the live trace and metric data itself — no manual Query Builder construction required. In practice, this is where the ingestion bottleneck and the GARCH fallback pattern above were first noticed, before they became dashboard panels.

Key Takeaways

Rich Attributes Over Unstructured Text: Attaching quantitative metadata (model.confidence, pricing.black_scholes, data.freshness) to spans turns observability tools into real-time model auditing platforms.
Parent-Child Span Hierarchies: Wrapping sequential ML workflows in structured parent/child relationships makes identifying bottlenecks straightforward.
Automated Guardrails Keep Models Safe: Setting automated alerts on data staleness and model non-convergence prevents bad data from triggering automated trades.
Correlated Signals Beat Isolated Metrics: No single dashboard panel told the real story on its own. It was only visible by tracing enough of the pipeline, in enough detail, to watch three unrelated models react to the same event at the same time.

By combining OpenTelemetry with SigNoz, complex machine learning systems no longer need to be black boxes. You get complete visibility into the health, efficiency, and accuracy of every prediction your models make.

GitHub Repository: github.com/siddarth709/alphatrace
Tech Stack: SigNoz, OpenTelemetry, PyTorch, XGBoost, Arch, React