DEV Community: DESAR DEV RAJEEV

How I Traced Investigation Context Across Incidents With Hindsight

DESAR DEV RAJEEV — Tue, 29 Sep 2026 16:30:29 +0000

# How I Traced Investigation Context Across Incidents With Hindsight

A security alert is rarely interesting in isolation. The useful question is usually, “Have I seen this pattern before, and what happened when I did?”

That question became the design constraint behind CyberMemory AI. I built the system around a simple idea: an incident-response agent should not have to rediscover the same investigation every time an alert arrives. It should be able to retain what happened, retrieve the parts that matter to a new alert, and use that history when deciding what to investigate next.

CyberMemory
AI takes security alerts such as repeated failed logins, suspicious activity, or a potential malware detection and turns them into an investigation workflow.

The application has three important paths.

First, an alert enters the investigation layer. The agent normalizes the event, identifies the important entities and signals, and asks what context is available from previous incidents.

Second, the memory layer searches historical incidents. I use Hindsight as persistent memory rather than treating the language model's context window as the application's memory. Hindsight exposes three core operations that fit this workflow well: retain information, recall relevant memories, and reflect over those memories to produce a contextual answer. Its recall path combines multiple retrieval strategies, including semantic, keyword, graph, and temporal retrieval.

Third, the investigation result is presented as a report. The report includes a summary of the alert, related incidents, possible causes, and recommended investigation or remediation steps. When the incident is resolved, I write the outcome back to memory.

That last step is the part that changes the system over time.

The application is not simply doing retrieval-augmented generation over a static knowledge base. The incident itself becomes part of the future investigation context.

The important boundary: an alert is not a memory

One of the design decisions I cared about early was separating the current alert from historical experience.

Suppose I receive this:

{
  "type": "failed_login",
  "user": "alex",
  "source_ip": "10.20.14.8",
  "attempts": 17,
  "window": "4 minutes"
}

The alert is evidence about something happening now. It should not automatically become a conclusion.

I first turn it into an investigation query that contains the signals that matter:

query = f"""
Investigate this security alert.

Type: {alert.type}
User: {alert.user}
Source IP: {alert.source_ip}
Attempts: {alert.attempts}
Time window: {alert.window}

Find previous incidents with similar authentication behavior,
including their root causes, investigation steps, and outcomes.
"""

That distinction sounds small, but it keeps the memory layer from becoming a dumping ground for raw alerts. Hindsight's retain operation is designed to process content into structured memories rather than merely storing text verbatim, which is useful here because an incident contains facts, observations, entities, and experience that may be useful in different future investigations.

Why persistent memory changed the investigation flow

Without persistent memory, an LLM-based incident assistant has a familiar failure mode: every investigation starts from the same blank page.

The model may know general security practices. It may even produce a reasonable checklist. But it does not automatically know that our environment has already seen this exact combination of signals three times, that two cases were caused by a misconfigured identity provider, and that another was a compromised credential.

That information lives in incident history.

I use a dedicated Hindsight memory bank for the incident-response context:

from hindsight_client import Hindsight

memory = Hindsight(
    base_url=HINDSIGHT_URL,
    api_key=HINDSIGHT_API_KEY,
)

BANK_ID = "cybermemory-incidents"

For a new investigation, I can recall historical experience before asking the reasoning layer to produce its report:

memories = memory.recall(
    bank_id=BANK_ID,
    query=query,
)

context = "\n".join(
    result.text
    for result in memories.results
)

This is where Hindsight is more useful to me than simply putting embeddings into a vector database and doing one nearest-neighbor search.

Security incidents have relationships that are not purely semantic.

A previous incident may involve the same user, source IP range, hostname, malware family, authentication provider, or sequence of events. It may also matter because it happened recently, or because it represents a repeated operational pattern. Hindsight's multi-strategy recall is designed to combine semantic, keyword, graph, and temporal signals before returning a ranked result set.

For incident response, that combination is a much better fit than assuming “similar wording” means “similar incident.”

I also wanted the reasoning to stay grounded

Retrieval gives me evidence. It does not, by itself, give me an investigation.

That is where Hindsight's reflect operation becomes interesting.

Instead of manually concatenating every recalled memory into a huge prompt, the system can ask Hindsight to reason over the memory bank:

report = memory.reflect(
    bank_id=BANK_ID,
    query=f"""
    Analyze this security alert using relevant historical incidents.

    Current alert:
    {query}

    Explain:
    1. Which previous incidents are relevant
    2. What patterns are repeated
    3. What possible causes should be investigated
    4. What evidence should be collected next
    5. What remediation actions were effective previously
    """
)

return report.text

The distinction between recall and reflect is useful architecturally. Recall gives the application retrieved memory results; reflect synthesizes an answer from the memory context. Hindsight documents reflect as an agentic reasoning operation that searches memory and produces a grounded response.

I use that distinction deliberately.

When I need to show an analyst the underlying historical evidence, I want recall. When I need a coherent investigation narrative, I want reflection.

Closing the loop after remediation

The most important write happens after the alert is resolved.

An incident record contains more than the initial detection. It has the root cause, investigation path, actions taken, and outcome.

I retain that resolved experience as a single incident context:

incident_memory = f"""
Incident ID: {incident.id}
Alert type: {incident.alert_type}
Observed behavior: {incident.summary}

Root cause:
{incident.root_cause}

Investigation:
{incident.investigation_steps}

Actions taken:
{incident.actions_taken}

Outcome:
{incident.outcome}
"""

memory.retain(
    bank_id=BANK_ID,
    content=incident_memory,
    context="resolved security incident",
    metadata={
        "incident_id": incident.id,
        "alert_type": incident.alert_type,
    },
)

This is the part I consider the real feedback loop.

If a failed-login incident is eventually confirmed as an identity-provider synchronization problem, I do not want the next investigation to see only the original alert. I want it to have access to the resolution.

That makes the historical record operationally useful.

I also keep the incident identity and metadata separate from the prose itself. The prose gives Hindsight enough context to extract meaningful memories, while the metadata gives the application stable identifiers for tracing the result back to the incident record.

A concrete investigation

Imagine the first incident contains 17 failed logins from one source address. The analyst investigates the identity provider and finds that a synchronization job repeatedly retried an expired credential. The incident is resolved by fixing the service credential and restarting the affected synchronization process.

Two weeks later, another alert arrives with a similar pattern.

A stateless assistant might produce a generic list:

check whether the account is compromised
inspect source IP reputation
review authentication logs
reset credentials
check for lateral movement

Those are reasonable checks, but they are not particularly specific.

With persistent incident memory, the investigation can surface the earlier resolution and say, in effect: this pattern resembles a previously resolved authentication incident; that incident was traced to a synchronization credential rather than interactive credential abuse; inspect the corresponding identity-provider logs and service-account activity before treating this as an account compromise.

That is the behavior I was aiming for.

The memory does not replace evidence collection. It changes the order in which I look for evidence.

The dashboard is deliberately boring

The dashboard is organized around the things an incident responder actually needs to see: current alerts, investigation reports, resolved incidents, and incident history.

I did not want the UI to become another abstraction over the investigation. A responder should be able to open an alert, see the current analysis, inspect related historical incidents, and understand what was eventually recorded as the resolution.

The most useful screen is therefore not “AI output.” It is the chain connecting:

alert → investigation → historical context → resolution → retained experience

That chain is what makes the system inspectable.

It also makes mistakes easier to correct. If the agent recommends the wrong next step, I can trace whether the problem came from the alert normalization, historical retrieval, or reasoning over the retrieved context.

What I learned

1. Memory quality matters more than memory volume

My first instinct was to retain everything. That is the wrong abstraction.

An incident contains lots of operational noise: repeated log lines, temporary identifiers, routine status messages, and details that are irrelevant outside the original investigation.

The useful memory is the part that explains what happened and why.

That is why I treat a resolved incident as a structured experience with context, cause, actions, and outcome rather than blindly storing every event.

2. Retrieval should be part of the investigation, not an afterthought

It is tempting to generate the LLM report first and attach memory afterward.

I found the opposite architecture more useful.

The investigation should start by asking what the system already knows. Historical context can change which hypotheses are worth investigating, which logs deserve attention, and which remediation actions have already worked.

Memory therefore belongs before reasoning, not after it.

3. `recall` and `reflect` solve different problems

I do not treat them as interchangeable.

recall is useful when I need evidence: which previous incidents were related, what facts were stored, and what context was attached.

reflect is useful when I need synthesis: given the available experience, what should I understand about this case?

Keeping those responsibilities separate also makes the application easier to debug.

4. The resolution is often more valuable than the alert

An alert tells me what the detector noticed.

A resolved incident tells me what the organization learned.

That distinction is why the write-back path is a first-class part of CyberMemory AI rather than an optional analytics feature.

5. Persistent memory changes what an agent can be

The interesting part of an incident-response agent is not that it can summarize a log file. LLMs are already good at summarization.

The harder engineering problem is continuity.

I want the system to be able to say, “We have seen something like this before,” and then make that statement useful by pointing to what happened, what was learned, and what should be checked now.

That is the role I am using Hindsight for.

The Hindsight GitHub repository and Hindsight documentation are useful references for the underlying memory model and API. The broader idea is also captured well in Vectorize's explanation of agent memory for persistent context.

For CyberMemory AI, persistent memory is not a feature bolted onto an otherwise stateless chatbot. It is the mechanism that connects one incident to the next.

That connection is where the system becomes substantially more useful: not because it remembers everything, but because it can remember the parts of previous investigations that change how I investigate the next one.

Building Neuro-Morph: A Python Full-Stack AI App Using MongoDB

DESAR DEV RAJEEV — Thu, 30 Apr 2026 14:26:16 +0000

Written By Mahendra, Dev Rajeev, Nihal
Guided By Chanda Rajkumar

WHY PYTHON END-TO-END?

When the Neuro-Morph team chose their language, the answer was deliberate: Python end-to-end. Python sits at a rare intersection — dominant in machine learning, mature in web frameworks, and deeply integrated with container orchestration tooling. A single language across every layer means minimal glue code, zero context-switching overhead, and a unified type system from the API boundary all the way down to the RL training loop.

TECH STACK OVERVIEW

API — FastAPI
Async REST endpoints, Pydantic validation, zero-config OpenAPI docs

ML — PyTorch
DQN RL agent training & inference with dynamic computation graphs

QUEUE — Celery + Redis
Async mutation task dispatch, retry logic, result backend

OPS — Docker + K8s
Minimal multi-stage images, Python operators via kopf

OBS — Prometheus + Grafana
Metrics scraping, alerting rules, real-time dashboards

CI/CD — GitHub Actions
Lint → test → build → push in under four minutes

TEST — Pytest + Hypothesis
**

[API LAYER] FastAPI in Production

The control plane backbone is FastAPI, chosen over Flask for three non-negotiable reasons: native async/await, automatic request validation via Pydantic, and zero-effort OpenAPI documentation. For a system that dispatches mutation commands with millisecond responsiveness, synchronous WSGI was simply not an option.
Mutation Endpoint

Below is the core endpoint that receives a trigger from the RL engine and dispatches it as an async Celery task:
from fastapi import FastAPI, HTTPException

`from pydantic import BaseModel
from app.tasks import dispatch_mutation

app = FastAPI(title='Neuro-Morph Control API', version='1.0')

class MutationRequest(BaseModel):
    action: str          # e.g. 'rotate_port', 'redeploy_container'
    target_service: str
    priority: int = 1

@app.post('/mutations/dispatch')
async def trigger_mutation(req: MutationRequest):
    if req.priority not in range(1, 4):
        raise HTTPException(status_code=422, detail='Priority must be 1-3')
    task = dispatch_mutation.delay(req.action, req.target_service)
    return {'task_id': task.id, 'status': 'queued'}`

Pydantic's BaseModel validates incoming JSON, converts types, and returns structured 422 errors — all with zero boilerplate. The endpoint is a pure async function, freeing the event loop while Celery handles the heavy lifting in the background worker pool.

[ML LAYER] The PyTorch RL Agent

The Reinforcement Learning engine is Neuro-Morph's most Python-native component. Built in PyTorch with a Deep Q-Network (DQN) architecture, the agent continuously learns the optimal mutation policy from environment interactions. PyTorch was chosen over TensorFlow for its dynamic computation graph, which makes rapid policy experimentation significantly easier.

RL Module Structure

neuro_morph/rl/
  ├── agent.py         # DQN: select_action(), learn(), update_target()
  ├── environment.py   # Gym-compatible: step(), reset(), observation_space
  ├── replay_buffer.py # Experience replay: push(), sample(batch_size)
  ├── reward.py        # Shaping: attack_blocked, service_uptime
  ├── train.py         # Episode runner, checkpoint saving
  └── serve.py         # Inference server: load model, expose predict(state)

The DQN Agent Core

`import torch, torch.nn as nn

class DQNAgent(nn.Module):
    def __init__(self, state_dim: int, action_dim: int):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, 128), nn.ReLU(),
            nn.Linear(128, 128),       nn.ReLU(),
            nn.Linear(128, action_dim)
        )

    def forward(self, state: torch.Tensor) -> torch.Tensor:
        return self.net(state)   # Q-values for each action

    def select_action(self, state, epsilon=0.05):
        if torch.rand(1).item() < epsilon:
            return torch.randint(self.net[-1].out_features, (1,)).item()
        with torch.no_grad():
            return self.forward(state).argmax().item()`

The state vector encodes real-time metrics: connection anomaly scores, port scan frequency, CPU/memory deltas, and recent mutation history. The action space maps to concrete operations — port rotation, container redeploy, IP swap, API key refresh, TLS certificate cycle. Rewards balance two competing objectives: maximize attack disruption while minimizing legitimate traffic latency.

Dev Tip: Use torch.jit.script() to compile your trained DQN to TorchScript before serving — it cuts inference latency by 30–40% and removes the Python GIL bottleneck in high-throughput environments.

[TASK QUEUE] Celery + Redis

Mutation operations are never executed synchronously inside the API request lifecycle. Redeploying a container or rotating a port can take several seconds — blocking an async FastAPI handler would be wasteful and unpredictable. The solution: a Celery task queue backed by Redis, which decouples command dispatch from command execution.

` # app/tasks.py
from celery import Celery
from app.orchestrator import KubernetesOrchestrator

celery_app = Celery('neuro_morph', broker='redis://redis:6379/0')
k8s = KubernetesOrchestrator()

@celery_app.task(bind=True, max_retries=3, default_retry_delay=5)
def dispatch_mutation(self, action: str, service: str):
    try:
        if action == 'redeploy_container':
            k8s.rolling_restart(service)
        elif action == 'rotate_port':
            k8s.patch_service_port(service)
        return {'status': 'success', 'action': action}
    except Exception as exc:
        raise self.retry(exc=exc)`

Celery's bind=True pattern gives the task access to its own context for retry logic. With max_retries=3 and exponential backoff, transient Kubernetes API errors are handled gracefully without manual intervention. Redis serves double duty — broker for task messages and result backend for task status queries from the monitoring dashboard.

[INFRA] Docker, Kubernetes & Python Operators

Every Neuro-Morph service is packaged as a minimal Docker image built from python:3.11-slim. Multi-stage builds keep production images under 180 MB — the build stage installs all dependencies; the runtime stage copies only the compiled application.

`# Stage 1 — Build
FROM python:3.11-slim AS builder
WORKDIR /build
COPY requirements.txt .
RUN pip install --prefix=/install --no-cache-dir -r requirements.txt

# Stage 2 — Runtime
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /install /usr/local
COPY . .
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]`

Kubernetes mutations are applied via a custom operator written in Python using the kopf framework. The operator watches for MutationEvent custom resources and applies rolling restarts, network policy patches, and config map updates. It runs as a standard Deployment using in-cluster service account credentials — no external access required. Kopf handles leader election, retry logic, and structured logging, keeping the operator code focused purely on business logic.

[TESTING] Pytest, Fixtures & Mocking

Neuro-Morph uses Pytest as the sole test runner across all layers, with four distinct test categories:

Unit Tests — individual RL agent methods, reward functions, and API validators in isolation using mock Kubernetes clients and synthetic state tensors.
Integration Tests — spin up a test FastAPI app with TestClient, verify that POST /mutations/dispatch correctly enqueues a Celery task and returns the expected task ID.
Property-Based Tests — use Hypothesis to generate random state vectors and verify that the DQN agent always returns a valid action index.
End-to-End Tests — a Docker Compose environment with a stubbed Kubernetes API server runs full mutation cycles, verifying observable state changes within defined time bounds.

# tests/test_api.py
    from fastapi.testclient import TestClient
    from unittest.mock import patch
    from app.main import app

    client = TestClient(app)

    @patch('app.tasks.dispatch_mutation.delay')
    def test_mutation_dispatch_queues_task(mock_delay):
        mock_delay.return_value.id = 'abc-123'
        resp = client.post('/mutations/dispatch', json={
            'action': 'rotate_port',
            'target_service': 'api-gateway',
            'priority': 2
        })
        assert resp.status_code == 200
        assert resp.json()['task_id'] == 'abc-123'
        mock_delay.assert_called_once_with('rotate_port', 'api-gateway')

Coverage Target: 90% on API and task layers, 80% on the RL module. Use pytest-cov with --fail-under=80 in CI to enforce the floor and prevent regressions from sneaking into main.

Github Repository:-https://github.com/N1hal-tech/Neuro-Morph