The latest articles on DEV Community by Garapati Janardhan swamy (@garapati_janardhanswamy_). https://dev.to/garapati_janardhanswamy_ https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4150654%2F3f3cfc22-2ed0-454a-846f-4737fb39b12c.png https://dev.to/garapati_janardhanswamy_ en Garapati Janardhan swamy Tue, 29 Sep 2026 17:47:47 +0000 https://dev.to/garapati_janardhanswamy_/hindsight-told-my-on-call-agent-not-to-scale-replicas-4f06 https://dev.to/garapati_janardhanswamy_/hindsight-told-my-on-call-agent-not-to-scale-replicas-4f06 <p>The alert was one line:</p> <p><code>checkout-service: requests failing shortly after a release</code></p> <p>I ran it through the same LLM twice.</p> <p>Without memory, I got six plausible causes: code regression, config drift, dependency changes, schema mismatch, resource exhaustion, and traffic routing.</p> <p>With memory, I got this:</p> <p><strong>Likely root cause:</strong> a missing or misnamed environment variable.</p> <p><strong>Historical context:</strong> the same pattern appeared in a previous incident.</p> <p><strong>Warning:</strong> do not scale replicas because that mitigation failed before.</p> <p><strong>Recommended fix:</strong> roll back and correct the Helm value.</p> <p><strong>Previous resolver:</strong> Priya Nair.</p> <p>The model didn't get smarter.</p> <p><strong>It got experience.</strong></p> <h2> What I built </h2> <p>OnCall Memory is an incident triage agent for software engineering teams.</p> <p>The history belongs to a fictional payments company, Northwind Payments, with eight services across Java, Go, Python, and Node, backed by Postgres, Redis, Kafka, and Elasticsearch.</p> <p>The data is synthetic: 40 past incidents drawn from six failure families:</p> <ul> <li>exhausted database pools</li> <li>Redis eviction stampedes</li> <li>bad configuration after a release</li> <li>expired certificates</li> <li>Kafka poison messages</li> <li>full disks</li> </ul> <p>The important design choice is that the same underlying failure can appear with different symptoms on different services.</p> <p>The team's experience matters more than matching exact words.</p> <h2> Why persistent memory matters </h2> <p>An incident is more than a document.</p> <p>I wanted the agent to remember:</p> <ul> <li>which fix worked</li> <li>which fix failed</li> <li>who solved the incident</li> <li>what happened in earlier incidents</li> <li>what should be avoided next time</li> </ul> <p>That is why the project uses Hindsight as the persistent memory layer.</p> <p>The architecture is simple:</p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> text Current Alert ↓ Hindsight Recall ↓ Historical Experience ↓ LLM Reasoning ↓ Operational Recommendation ↓ Resolved Incident ↓ Post-mortem retained as new memory </code></pre> </div> ai agents hindsight devops