Calling Hindsight From FastAPI Without Breaking the Event Loop

Shailendra — Sat, 07 Nov 2026 18:09:27 +0000

One of the more annoying bugs we hit while building IncidentIQ passed every isolated test and then failed intermittently once the real server was running. The code path was identical. The Hindsight bank was the same. The only thing that changed was which thread made the SDK call.

IncidentIQ is an incident investigation assistant. An engineer submits a production incident, a Groq model proposes root-cause hypotheses, and an engineer later verifies what actually happened. Only that verified outcome is retained in Hindsight, so the next related incident can recall it. This article is about the backend that makes that loop work, and about the event-loop problem that nearly derailed it. Everything here uses synthetic incidents, and the system is a prototype running locally, not something we have exposed to real production traffic.

The shape of the backend

The backend is FastAPI with SQLAlchemy on PostgreSQL, and Alembic handles migrations. Routes stay thin: api/routes.py validates input and translates errors. The real work lives in three services. InvestigationService coordinates the flow, HindsightMemoryService owns every memory operation, and GroqAIService owns the model call. PostgreSQL stores incidents, logs, deployments, investigation runs, findings, verified resolutions and an audit table of memory events. Hindsight holds only the experience we want an agent to recall later.

The SDK client and both services are created once, in the FastAPI lifespan handler, and attached to app.state:

if settings.hindsight_api_key:
    base_url, api_key = settings.require_hindsight()
    application.state.hindsight_client = Hindsight(
        base_url=base_url,
        api_key=api_key,
    )
application.state.hindsight_service = HindsightMemoryService(
    application.state.hindsight_client,
    bank_id=settings.hindsight_bank_id,
)

If the key is missing, the service still exists with a None client and every operation reports client_unavailable instead of crashing. That made local development far less brittle. The app boots, the dashboard loads, and the health endpoint tells you which dependency is missing.

One endpoint, two modes

The investigation endpoint takes a memory_mode of MEMORY_OFF or MEMORY_ON. The mode is stored on every InvestigationRun, so you can always see afterwards whether a run had historical context. The core of run_investigation is short:

historical_memories = []
if mode is MemoryMode.MEMORY_ON:
    recall_result = await self._recall(context_without_memory)
    if recall_result.status != "success":
        ...  # run is marked failed, audit event written
        raise InvestigationServiceError(f"hindsight_{category}", run_id)
    historical_memories = list(recall_result.memories)

ai_context = context_without_memory.model_copy(
    update={"historical_memories": historical_memories}
)
analysis = self._ai.analyze_incident(ai_context)

Building the context this way means MEMORY_OFF and MEMORY_ON share one code path. The only difference is whether historical_memories is empty. That keeps the comparison honest, because when we show two runs side by side, nothing else in the pipeline differs.

Inside analyze_incident, current evidence, recalled memories and engineer-verified findings are sent as separate lists, each item with a reference like historical:0. The system prompt tells the model that historical similarity is not proof of a current cause, and that it may only cite references it was given. The call asks Groq for a JSON object at a low temperature, and the result is validated against a Pydantic schema. A malformed answer becomes a categorised error instead of a half-parsed result.

The bug: threads and event loops

The Hindsight Python client has sync methods and async ones. At one point our routes were ordinary def handlers calling the sync recall. FastAPI runs those in a thread pool, while the client had been created on the main event loop during startup. Our QA notes traced the intermittent 503s to that mismatch: the SDK's internal async machinery was being driven from a different thread than the one that owned the loop. In a standalone script or a pytest run there is only one loop, so nothing ever complained.

The symptoms were maddening. Memory Explorer queries and MEMORY_ON runs failed with a generic "dependency unavailable", and the audit table recorded sdk_error, which told us nothing.

Two changes followed. The routes that touch Hindsight became async def, and the service now uses the client's async methods, arecall and aretain, so everything runs on the loop that created the client. The retain path also takes an asyncio.Lock, so two concurrent verifications of the same incident cannot race to write the same memory.

Making failures legible

That sdk_error result pointed at a second bug, this one in our own code. Our error classifier looked for status_code on the exception, but the generated Hindsight API client exposes status. Every HTTP failure fell through to the default category. The fix is one line:

status_code = getattr(error, "status_code", getattr(error, "status", None))

With it, a 401 becomes authentication_error, a 429 becomes rate_limited, and a 5xx becomes provider_error. The API returns only these stable categories to the browser, inside a {"detail": {"code": ..., "message": ...}} envelope, because raw provider messages can leak details you do not want in a UI:

if category.startswith(("hindsight_", "groq_")):
    return HTTPException(
        status_code=503,
        detail={"code": "DEPENDENCY_UNAVAILABLE",
                "message": "An investigation dependency is unavailable."},
    )

The browser gets something deliberately vague and the audit log gets something precise. That split is what we wanted.

What the behaviour looks like

Take a synthetic incident: payment-api returns HTTP 503 at checkout while database connection utilisation sits near its limit. With MEMORY_OFF, the model reasons from the logs, metrics and deployment note alone. Later an engineer verifies the real cause. In our seeded scenario, a worker-concurrency increase exhausted the PostgreSQL connection pool. That resolution is retained with the tags incident and verified-resolution.

A follow-up incident on the same service can be investigated with MEMORY_ON. In this mode, IncidentIQ requests relevant historical context from Hindsight and, when memories are successfully returned, passes them to the model as historical information rather than confirmed evidence. This distinction is important because a previous resolution can guide an investigation, but the current incident must still be evaluated using its own logs and metrics.

We have not measured accuracy or time to resolution, and we are not claiming any improvement beyond what the recorded runs show.

Lessons

Create SDK clients on the loop that will use them. If a library is async underneath, a thread-pool handler is the wrong place to call its sync wrapper.

Test through the running server. Our unit tests passed because they never crossed the thread boundary.

Classify errors from real exception objects. Check what the exception actually carries before writing a mapper for it.

Keep memory operations behind one service. With a single place for retain and recall, the audit trail and the locking were easy to add.

Give modes a single code path. MEMORY_OFF and MEMORY_ON differ by one list, which is what makes comparing them meaningful.

Limits and next steps

Today a failed recall fails the whole MEMORY_ON run instead of falling back to a clearly labelled baseline. The Groq call is still synchronous inside an async function, so it should move to a worker thread. The health endpoint performs real provider checks and should cache them. There is no authentication yet, so the typed engineer name is an audit label, not an identity.

If you are building something similar, the Hindsight documentation covers the async client, and Vectorize's explanation of agent memory is a good primer on why recalled context should sit beside the model rather than inside it.

DEV Community: Shailendra

Calling Hindsight From FastAPI Without Breaking the Event Loop

The shape of the backend

One endpoint, two modes

The bug: threads and event loops

Making failures legible

What the behaviour looks like

Lessons

Limits and next steps

ai

python

fastapi

hindsight