ChronicleOps: Autonomous Chaos Engineering & Self-Healing Telemetry Engine published

#automation #cloud #devops #sre

What I Built

ChronicleOps is an autonomous cloud chaos engineering, real-time observability, and self-healing platform built natively on Zerops.

Instead of manually troubleshooting downtime, ChronicleOps lets developers spin up target application stacks, inject controlled fault vectors (such as process kills or container crashes), observe real-time telemetry degradation, and let the system automatically recover services while generating Gemini AI root-cause diagnoses.

The Problem & Solution

The Problem: Microservice failures in cloud environments are often chaotic and hard to diagnose. Reproducing fault states safely without breaking persistent infrastructure is difficult, and measuring exact Mean Time to Recovery (MTTR) requires heavy monitoring setups.
The Solution: ChronicleOps automates the entire chaos-to-recovery loop. With a single click, it provisions isolated Zerops environments, kills target processes, measures recovery latency down to the millisecond, and feeds real-time container log streams into vector-matched AI for automated incident reports.

Architecture & Tech Stack

ChronicleOps is structured as a multi-service architecture running directly on Zerops infrastructure:

Component	Stack	Description
Orchestrator	FastAPI (Python 3.11), Uvicorn	Primary API gateway, environment lifecycle orchestrator, and Gemini AI diagnosis engine.
Worker	Python 3.11, Zerops CLI (`zcli`)	Background polling daemon that monitors target health, triggers recovery restarts, and runs TTL resource pruning.
Ingestion Engine	FastAPI, Pub/Sub	Log collector and real-time event pipeline for telemetry streams.
Database & Cache	PostgreSQL, Valkey	Vector store for historical incident embeddings and real-time state synchronization.
Frontend UI	HTML5, Modern CSS3, Canvas, Vanilla JS	Zero-dependency Command Center featuring CRT scanline aesthetic, real-time Canvas telemetry charts, and interactive controls.

                     ┌─────────────────────────────────────────┐
                     │          ChronicleOps Dashboard         │
                     │    (Vanilla JS + HTML5 Canvas RTC)      │
                     └────────────────────┬────────────────────┘
                                          │
                                          ▼
                     ┌─────────────────────────────────────────┐
                     │          Orchestrator Service           │
                     │          (FastAPI / Python 3.11)        │
                     └───────┬─────────────────────────┬───────┘
                             │                         │
                             ▼                         ▼
┌─────────────────────────────────────────┐   ┌─────────────────────────────────┐
│              Worker Service             │   │       Ingestion & AI Engine     │
│   (Zerops CLI / Health & TTL Poller)    │   │  (PostgreSQL + Gemini Vector RAG)│
└─────────────────────────────────────────┘   └─────────────────────────────────┘

Key Features

1-Click Demo Quickstart: Instantly provisions an isolated application stack on Zerops using a demo repository without local configuration.
Controlled Chaos Injection: Injects process termination signals (SIGKILL / 502 Bad Gateway) into active containers to test fault tolerance.
Real-time Canvas Telemetry: Tracks 1.0s RTC uptime signals with visual telemetry drop and step-up curves.
Autonomous Self-Healing & MTTR: Detects service outages in real time, executes automated restarts via zcli, and calculates exact Mean Time to Recovery metrics.
Gemini AI Root-Cause Diagnosis: Correlates log tails against historical incident vector embeddings stored in PostgreSQL/Valkey to generate root-cause reports, confidence scores, and recommended fixes.
TTL Environment Pruning: Automatically cleans up expired demo environments and Zerops project resources to keep platform quotas clean.

How Zerops Powers ChronicleOps
ChronicleOps relies on Zerops for its deployment pipeline and multi-service orchestrations:

Isolated Multi-Service Deployments: Deploys orchestrator, worker, and ingestion services independently via zerops.yaml.
Dynamic Infrastructure Provisioning: Uses Zerops CLI (zcli) inside the worker service to spin up and tear down temporary user environments on demand.
Internal Service Discovery & Public Ingress: Connects microservices using internal Zerops environment variables (VALKEY_HOST, POSTGRES_URL) while exposing public ingress endpoints for target apps.

YAML
zerops:
  - setup: orchestrator
    build:
      base: python@3.11
      buildCommands:
        - pip install -r orchestrator/requirements.txt
      deployFiles:
        - ./orchestrator
    run:
      base: python@3.11
      ports:
        - port: 8080
          httpSupport: true
      start: uvicorn orchestrator.app.main:app --host 0.0.0.0 --port 8080