How We Built a Cross-Company Incident Agent With Hindsight

Ashmit Saxena — Wed, 12 Aug 2026 18:39:10 +0000

When a production database connection pool hits 100% capacity at 3 a.m., every minute spent in incident response counts. On-call engineers routinely burn hours re-diagnosing failure shapes that dozens of other companies hit last week — a Kubernetes OOM-kill cascade, an AWS NAT gateway port exhaustion, or a Redis cache eviction storm.
Every engineering team rediscovers the exact same failure shapes from scratch. While internal wiki search and post-mortem tools provide isolated company memory, there has never been a clean way to leverage collective incident intelligence across companies. The barrier isn't interest; it's privacy. Raw incident logs contain database connection strings, internal service hostnames, API tokens, and customer email addresses. If you throw raw incident logs from multiple companies into a single shared vector store, cosine similarity will inevitably leak internal topology and proprietary post-mortems across organizational boundaries.
To solve this, we built SoLit, a privacy-preserving incident response agent centered around dual-tier agent memory. By pairing private corporate memory banks with an anonymized, cross-company pattern mesh, we enabled teams to benefit from industry-wide outage history without exposing sensitive internal data.

What the System Does and How It Hangs Together
SoLit relies on a fundamental separation between private corporate context and shared pattern recognition. Rather than mixing multi-tenant data with metadata filters inside a single vector index, we structured the memory layer around isolated memory banks using Vectorize agent memory concepts and the Hindsight documentation architecture patterns.
The system uses two distinct memory tiers:

Private Tier: A dedicated memory bank for each organization. It retains complete, un-redacted incident reports, including service names, internal hostnames, post-mortem summaries, and historical resolutions. When an engineer queries the system, private recall runs first to answer: “Has our team seen this exact failure before?”
Shared Tier (global-incident-mesh): A single, shared memory bank accessible across all participating organizations. Crucially, this bank never accepts raw incident logs or narrative text. It accepts only anonymized, structural fingerprints — stripped of all identifying details and converted into categorical signatures. Shared recall answers: “Have other engineering teams experienced this shape of failure, even if our company never has?”

The end-to-end execution flow follows a 5-step pipeline:

Private Recall: Query the organization's private Hindsight bank using the raw incident text.
Two-Pass Anonymization: Strip PII, hostnames, IP addresses, and proprietary codenames using deterministic regex scrubbers followed by a zero-temperature LLM redaction pass.
Structural Fingerprinting: Convert the scrubbed incident description into a key-value signature defining the component type, failure mode, trigger, resolution category, and downstream cascade pattern.
Shared Mesh Recall with k-Anonymity Gating: Query the global Hindsight bank using the fingerprint signature. Suppress results unless the pattern matches at least k ≥ 3 independent source submissions.
Dual Retain & Remediation Synthesis: Retain the raw incident into private memory, retain the anonymized fingerprint into shared memory, and synthesize an actionable response containing immediate mitigations, long-term fixes, and copyable configuration diffs.

Core Technical Story: Preventing Vector Memory Privacy Leaks
The central engineering challenge in cross-company agent memory is vector space leakage. Traditional vector retrieval works by embedding text into high-dimensional space where semantically similar items sit near each other. If Company A retains a raw log line like FATAL: db-prod-auth-01.internal.corp reached max_connections (1000) into a shared vector index, and Company B later queries auth service database connection timeout, vector search will bring Company A's internal hostname, max connection settings, and underlying infrastructure details directly into Company B's LLM context window.
Even if you run regex redaction over IP addresses and emails, narrative prose retains structural identifiers — internal project codenames, custom service names like checkout-hydra, or unique customer names in stack traces.
We solved this through a four-layer privacy defense built directly into the memory pipeline:
Layer 1: Two-Pass Redaction (anonymizer.py)
Before any incident data approaches the shared memory tier, it passes through two redaction stages. The first stage is a fast, deterministic regex pass that scrubs IPv4/IPv6 addresses, URLs, email addresses, and internal domain suffixes (*.internal, *.corp, *.prod). The second stage routes the scrubbed text through a strict LLM pass configured with zero temperature. The prompt explicitly instructs the model to replace proper nouns and custom service names with generic role labels (service-A, db-cluster-1) and preserve only the structural facts of the failure.
Layer 2: Structural Fingerprinting (fingerprint.py)
To ensure narrative text never enters the shared vector space, we converted the anonymized prose into a structured signature:
component=database; failure=connection_pool_exhaustion; trigger=traffic_spike;
resolution=increase_pool_size; cascade=api_latency -> 5xx_errors

Because Hindsight retains and recalls text embeddings, embedding this standardized string guarantees that similarity matching operates strictly on architectural failure shapes rather than vocabulary choices or sentence structures.
Layer 3: Strict Memory Bank Isolation (hindsight_memory.py)
Hindsight structures memory into logical banks. we maintained complete physical separation between private-{company_id} banks and global-incident-mesh. Private memory read/write operations require authenticated company tokens, whereas the shared mesh bank is write-only for anonymized fingerprints and read-gated by strict privacy thresholds.
Layer 4: k-Anonymity Gating (memory_agent.py)
Even anonymized fingerprints could theoretically reveal a single company's identity if the failure shape is extremely obscure (e.g., a rare proprietary queue configuration). To eliminate this risk, we enforced k-anonymity gating (k ≥ 3). The agent calculates the distinct source count of matched fingerprints in the shared mesh. If a pattern has been reported by fewer than 3 independent sources, the agent retains the fingerprint for future learning but suppresses the shared match from the final response.

Code-Backed Walkthrough
Let me walk through how this architecture is implemented in code using the Hindsight SDK on GitHub.

Bank Isolation in hindsight_memory.py We wrapped Hindsight's retain, recall, and reflect primitives inside a clean wrapper that enforces explicit bank routing. from hindsight_client import Hindsight

SHARED_BANK_ID = "global-incident-mesh"

def _private_bank_id(company_id: str) -> str:
return f"private-{company_id}"

class HindsightMemory:
"""Wraps retain/recall/reflect for both private team history and shared mesh signatures."""

def __init__(self, base_url: str = None, api_key: str = None):
    self.client = Hindsight(
        base_url=base_url or os.environ["HINDSIGHT_BASE_URL"],
        api_key=api_key or os.environ.get("HINDSIGHT_API_KEY"),
    )

def retain_private(self, company_id: str, content: str) -> None:
    """Store un-redacted incident in the company's private bank."""
    self.client.retain(bank_id=_private_bank_id(company_id), content=content)

def retain_shared_fingerprint(self, fingerprint_text: str) -> None:
    """Store ONLY anonymized fingerprints in the shared mesh bank."""
    self.client.retain(bank_id=SHARED_BANK_ID, content=fingerprint_text)

def recall_shared_fingerprint(self, fingerprint_query: str) -> List[RecallResult]:
    """Query global mesh for matching failure shapes."""
    resp = self.client.recall(bank_id=SHARED_BANK_ID, query=fingerprint_query)
    return [RecallResult(text=r.text) for r in getattr(resp, "results", [])]

Two-Pass Redaction in anonymizer.py The anonymizer guarantees that identifying strings never survive past the pipeline entry point. _EMAIL_RE = re.compile(r"[\w.-]+@[\w.-]+.\w+") _IP_RE = re.compile(r"\b(?:\d{1,3}.){3}\d{1,3}\b") _HOSTNAME_RE = re.compile(r"\b[\w-]+.(?:internal|corp|prod|local)\b", re.IGNORECASE)

def _regex_scrub(text: str) -> str:
text = _EMAIL_RE.sub("[redacted-email]", text)
text = _IP_RE.sub("[redacted-ip]", text)
return _HOSTNAME_RE.sub("[redacted-host]", text)

def anonymize_incident(raw_text: str) -> str:
scrubbed = _regex_scrub(raw_text)
client = Groq(api_key=os.environ["GROQ_API_KEY"])

completion = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "system", "content": _ANONYMIZE_SYSTEM_PROMPT},
        {"role": "user", "content": scrubbed},
    ],
    temperature=0,
)
return completion.choices[0].message.content.strip()

Converting Outages to Signatures in fingerprint.py
We mapped anonymized text to structured dataclasses that serialize into standardized text signatures for vector indexing.
@dataclass
class IncidentFingerprint:
component_type: str
failure_type: str
trigger: str
resolution_category: str
cascade_pattern: List[str] = field(default_factory=list)
recommended_actions: List[str] = field(default_factory=list)

def to_text(self) -> str:
cascade = " -> ".join(self.cascade_pattern) if self.cascade_pattern else "none"
return (
f"component={self.component_type}; failure={self.failure_type}; "
f"trigger={self.trigger}; resolution={self.resolution_category}; "
f"cascade={cascade}"
)
Orchestration & k-Anonymity Gating in memory_agent.py
The orchestration agent coordinates recall across both tiers and applies source threshold checks before returning remediation steps.
MIN_SOURCES = int(os.environ.get("MIN_SOURCES", "3"))

class IncidentMemoryAgent:
def handle_new_incident(self, company_id: str, raw_incident_text: str) -> TriageResult:
# 1. Recall private memory
private_matches = self.memory.recall_private(company_id, raw_incident_text)

    # 2. Anonymize + fingerprint
    anonymized = anonymize_incident(raw_incident_text)
    fp = generate_fingerprint(anonymized)

    # 3. Recall shared mesh
    shared_matches = self.memory.recall_shared_fingerprint(fp.to_text())
    source_count = len({m.text for m in shared_matches})

    # 4. Retain into both tiers
    self.memory.retain_private(company_id, raw_incident_text)
    self.memory.retain_shared_fingerprint(fp.to_text())

    # 5. Compose response with k-anonymity gate
    answer, solution_steps = self._compose_answer(
        private_matches, shared_matches, fp, source_count
    )
    return TriageResult(
        private_matches=private_matches,
        shared_matches=shared_matches,
        fingerprint=fp,
        shared_pattern_source_count=source_count,
        answer=answer,
        solution_steps=solution_steps,
    )

Real-World Triage Progression
The power of this two-tier memory architecture becomes obvious when observing how the agent behaves across different organizational interaction stages.
Interaction 1: Fresh System (Empty Private & Shared Memory)
An engineer reports an incident: "Postgres database connection timeouts during morning user login surge."
Because neither private memory nor shared memory contains prior data, the agent acknowledges the novel failure and suggests generic troubleshooting steps: checking active connection counts and reviewing application connection limits. It retains the raw incident into the team's private bank and writes the anonymized fingerprint (component=database; failure=connection_pool_exhaustion; trigger=traffic_spike; resolution=increase_pool_size) to the global mesh.

Interaction 2: Repeat Team Failure (Private Memory Hit)
Three weeks later, another engineer at the same company hits the exact same issue on a secondary microservice.
Private memory recall triggers immediately:
“Your team has seen this before — pulling the resolution from your own incident history: 'Updated DB_POOL_SIZE from 10 to 50 in settings.yaml and restarted PgBouncer.'”

Interaction 3: Novel Internal Failure, Shared Mesh Match
A different company encounters a Postgres connection pool exhaustion under load for the first time. Their private memory contains zero records of this failure mode. However, because three other organizations have previously retained matching fingerprints into the global mesh, shared recall succeeds and passes the k-anonymity gate (k ≥ 3).
The agent outputs:
“This looks new for your team. However, this matches a pattern seen in 4+ prior incidents across the mesh (component: database, failure: connection_pool_exhaustion, trigger: traffic_spike). Most common fix category: increase pool size.”

Along with the summary, the agent automatically provides actionable remediation steps and a ready-to-apply configuration diff:

db_config.py

DB_POOL_SIZE = 10
DB_MAX_OVERFLOW = 20
DB_POOL_SIZE = 50
DB_MAX_OVERFLOW = 100
DB_POOL_TIMEOUT = 30

pg_bouncer.ini

pool_mode = transaction
max_client_conn = 1000
default_pool_size = 50

Key Engineering Takeaways
Building a privacy-preserving cross-tenant memory system provided several reusable lessons for designing production agent memory:

Structural Strings Outperform Raw Prose for Cross-Domain Search: Free-text LLM outputs vary dramatically in vocabulary, phrasing, and verbosity. By mapping incidents to key-value fingerprint strings (component=database; failure=connection_pool_exhaustion...), vector embeddings cluster tightly around architectural failure shapes rather than writing styles.
Bank-Level Isolation Trumps Metadata Tagging: Relying on multi-tenant metadata filters within a single vector collection is error-prone and vulnerable to implementation bugs. Hard physical isolation into separate Hindsight memory banks (private-{company_id} vs global-incident-mesh) provides a robust security boundary that guarantees raw data cannot bleed between tenants.
Enforce k-Anonymity at the Memory Layer: Never surface pattern recommendations from single-source external reports. Requiring at least k ≥ 3 independent submissions prevents false confidence from one-off outliers and protects niche configurations from signature-correlation attacks.
Deterministic Pre-Scrubbing is Mandatory: Relying solely on an LLM for redaction will fail when processing unexpected formats, multi-line stack traces, or URL-encoded connection strings. Combining deterministic regex scrubbing with a zero-temperature LLM pass ensures both mechanical identifiers (IPs, hostnames) and semantic proper nouns are completely erased.

DEV Community: Ashmit Saxena

How We Built a Cross-Company Incident Agent With Hindsight

db_config.py

pg_bouncer.ini