How we combined Google ADK, Gemini 3.5 Flash, A2A protocol, Go, and immutable ledgers to automate incident recovery without risking production outages.
- The 3:00 AM Problem: Speed vs. Safety At 3:00 AM, a critical data pipeline fails due to an upstream schema drift. In a traditional setup, engineers are paged out of bed, spending hours manually inspecting logs, tracking down dependencies, and executing recovery scripts.
The obvious modern reaction is: "Let an autonomous AI agent fix it!"
However, handing an autonomous LLM unconstrained write permissions to production introduces a terrifying new vulnerability: Ungoverned AI Mutations. A single prompt injection, swallowed log error, or hallucinated SQL command could execute DELETE FROM production_db or rewrite a production schema.
Prompt engineering alone cannot solve this—prompts are soft guidance, not hard security boundaries.
That's why we built Chronos.
- What is Chronos? Chronos is a governed incident-remediation control plane designed for the Fortified Enterprise Fleet track. It automatically diagnoses data pipeline failures and generates verified repair proposals, while using deterministic code policy to make unauthorized production mutations structurally impossible.
Our core design principle:
"Chronos does not trust the LLM with execution. Every proposal passes typed validation, identity checks, deterministic policy, multi-agent debate, approval gates, and a tamper-evident audit ledger."
- System Architecture: Reasoning vs. Execution Chronos strictly separates AI Reasoning (which happens in Python using Google ADK and Gemini 3.5 Flash) from Deterministic Policy Enforcement (which happens in an isolated Go microservice over A2A).
┌─────────────────────────────────────────────────────────────┐
│ INGESTION & DEFENSE │
│ Upstream Log → Pub/Sub → Model Armor (PII & Injection) │
└──────────────────────────────┬──────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ PYTHON ORCHESTRATOR (Google ADK / Gemini 3.5 Flash) │
│ 1. DetectionAgent → Classifies failure │
│ 2. DebateProposer → Proposes repair strategy │
│ 3. DebateAuditor → Attacks strategy (Max 3 rounds) │
└──────────────────────────────┬──────────────────────────────┘
▼ [A2A Protocol]
┌─────────────────────────────────────────────────────────────┐
│ GO ACTION BROKER (Zero-Trust Policy Engine) │
│ Evaluates Proposal → ALLOW_SANDBOX | APPROVAL_REQUIRED | BLOCKED │
└──────────────────────────────┬──────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ IMMUTABLE AUDIT LEDGER (Firestore) │
│ Append-only SHA-256 Hash Chain + OTel Reasoning Spans │
└─────────────────────────────────────────────────────────────┘
- The Differentiator: Structurally Unreachable T3 Actions In Chronos, remediation actions are categorized into 4 tiers:
Tier Classification Description Outcome
T0 Safe Reversible sandbox operations (e.g., cache.flush) ALLOW_SANDBOX
T1 Material Reversible operations requiring approval (e.g., queue.drain) REQUIRE_APPROVAL
T2 High Risk Non-reversible operations requiring approval + ticket REQUIRE_APPROVAL
T3 Blocked Destructive mutation (DELETE_DATA, ALTER_PRODUCTION_SCHEMA) BLOCKED
Instead of relying on prompt instructions like "please don't delete data", Tier 3 ((\text{T3})) actions are structurally unreachable by code:
Enum & Schema Rejection: DELETE_DATA and ALTER_PRODUCTION_SCHEMA are explicitly excluded from executor enums.
Policy Evaluation: The Go Action Broker's Evaluate() function evaluates proposals deterministically against an allow-list:
go
// internal/policy/policy.go
if proposal.ActionType == "DELETE_DATA" || proposal.ActionType == "ALTER_PRODUCTION_SCHEMA" {
return Blocked, "T3_PRODUCTION_MUTATION_BLOCKED"
}
Static Compiler AST Guarantee: A Go static analysis test walks the AST at build time and fails the compiler build immediately if any execution handler for destructive actions is introduced.
- Adversarial Multi-Agent Debate Before a proposal reaches the broker, it undergoes a hardened Proposer vs. Auditor debate loop managed by a deterministic Python controller:
DetectionAgent: Uses Gemini 3.5 Flash with strict Pydantic schemas to output a FailureClassification.
DebateProposer: Formulates a step-by-step repair plan with rollback strategies and success criteria.
DebateAuditor: Attacks the proposal, searching for hidden dependencies, resource limits, and edge cases.
The state machine controller caps the debate at maximum 3 rounds and guarantees that no LLM can upgrade an action's risk tier.
- Cryptographic Audit Ledger & Observability Every incident decision is committed to a Firestore-backed hash-chained ledger. Each entry stores the SHA-256 hash of the current record concatenated with the previous record's hash:
$$H_i = \text{SHA-256}(H_{i-1} \parallel \text{Actor} \parallel \text{Action} \parallel \text{Decision} \parallel \text{Seq}_i)$$
The verify_chain() function verifies sequence continuity and cryptographic hash integrity, providing tamper-evident auditing for enterprise compliance.
Additionally, OpenTelemetry spans log every step of the reasoning chain, allowing operators to visualize the agent's exact decision path.
- Key Takeaways for Developers Building Production AI Agents Never Give LLMs Direct Executor Access: Let the AI generate structured proposals, but let zero-trust code decide whether to allow execution. Validate at the Wire: Run Model Armor on telemetry before it reaches the LLM to neutralize prompt injection and redact PII. Use AST Checks: Write build-time AST tests to mathematically verify that destructive capability handlers do not exist in your codebase.
- Try Chronos Chronos is fully open-source and ready to deploy on Google Cloud Run:
💻 GitHub Repository: https://github.com/ejemi1989/chronos
🚀 Tech Stack: Gemini 3.5 Flash, Google ADK, A2A Protocol, Go 1.23, Python FastAPI, Cloud Run, Firestore Native, Pub/Sub.
Top comments (1)
The hard boundary is the most useful part here: removing destructive actions from the executor is stronger than asking the model not to choose them. One case I would still test is stale approval. If a T1 or T2 proposal is approved, then the target resource or policy version changes before execution, the broker should revalidate against the current snapshot rather than treat the old approval as a reusable permit.