DEV Community

Richard Atodo
Richard Atodo

Posted on

🚨 Building an Incident Lifecycle and Timeline Engine with FastAPI and PostgreSQL

#ai #fastapi #backend #postgressql

Milestone 5 of IncidentCopilot: moving from log correlation to structured incident management.

Introduction

Collecting logs is only part of incident investigation.

A useful incident management system needs to do more than store operational events. It should provide a structured way to:

  • Create incidents
  • Track investigation progress
  • Associate supporting evidence
  • Build a timeline of what happened

In Milestone 5 of IncidentCopilot, I focused on implementing this foundation using:

  • FastAPI
  • PostgreSQL
  • SQLAlchemy
  • The deterministic correlation engine introduced in Milestone 4

One important design decision remained unchanged:

Build deterministic systems first. Introduce AI only after the evidence pipeline is reliable.

At this stage, the application reports what it has observed and what its correlation engine has inferred. It does not generate AI diagnoses or treat correlations as proven causes.

🔄 From Log Correlation to Incident Management

Previous milestones established:

✅ FastAPI backend and PostgreSQL persistence

✅ Multi-source log ingestion

✅ Deterministic log correlation

However, correlated logs alone do not provide a complete investigation workflow.

The next step was adding:

  1. Incident lifecycle management
  2. Log-to-incident association
  3. Timeline generation

Together, these capabilities allow operational evidence to be organized around a specific incident.

🏗️ Designing the Incident Lifecycle

I implemented three incident states:

OPEN
INVESTIGATING
RESOLVED
Enter fullscreen mode Exit fullscreen mode

The workflow is intentionally explicit.

An incident:

  • Starts in OPEN
  • Moves to INVESTIGATING when an operator begins investigation
  • Moves to RESOLVED when the incident is closed

Supported Transitions

Current State Action Result
OPEN Investigate INVESTIGATING
OPEN Resolve RESOLVED
INVESTIGATING Resolve RESOLVED
INVESTIGATING Investigate Again INVESTIGATING
RESOLVED Resolve Again RESOLVED
RESOLVED Investigate Rejected

A key design choice:

Attaching new evidence should not automatically change an incident's operational state.

Log association and lifecycle transitions remain separate operations.

✅ Idempotency and Lifecycle Safety

Two important behaviors were implemented.

Repeated Investigation

Calling the investigation endpoint multiple times does not create duplicate state transitions.

Repeated Resolution

When an incident is resolved, the system records its:

ended_at
Enter fullscreen mode Exit fullscreen mode

timestamp.

Subsequent resolution requests:

  • Do not overwrite the original timestamp
  • Do not create additional transitions

Attempting to move a resolved incident back into investigation returns:

409 Conflict
Enter fullscreen mode Exit fullscreen mode

This keeps lifecycle behavior predictable for both API consumers and future frontend components.

🔗 Associating Logs with Incidents

An incident becomes useful when all relevant operational evidence can be reviewed in one place.

I added an endpoint that accepts multiple log IDs and associates them with an incident.

The implementation supports:

  • Multiple logs per request
  • Duplicate detection
  • Validation of unknown log IDs
  • Preservation of existing associations
  • No automatic lifecycle transitions

The service retrieves all logs currently associated with the incident and combines them with newly attached logs.

It then re-evaluates the complete set using the correlation engine.

This is important because newly attached evidence can affect the strongest known relationships among previously associated logs.

➡️ Preserving Correlation Direction

The correlation engine models relationships directionally:

log_id          = later event
related_log_id  = earlier event
Enter fullscreen mode Exit fullscreen mode

That direction is preserved when correlation metadata is stored.

Why?

Because temporal ordering contributes to the meaning of a correlation.

A relationship between two events does not automatically establish causation.

Preserving direction helps maintain that distinction.

📅 Building the Incident Timeline

One of the most important features of this milestone is the incident timeline.

The timeline distinguishes between:

  1. Observed events
  2. Derived events

Observed Log Events

Observed events represent actual persisted logs.

observed_log
Enter fullscreen mode Exit fullscreen mode

Each event includes:

  • Timestamp
  • Log ID
  • Source
  • Severity
  • Service
  • Message

These records come directly from ingested operational evidence.

Correlation Events

Correlation events represent relationships generated by the deterministic correlation engine.

correlation
Enter fullscreen mode Exit fullscreen mode

Each event includes:

  • Later log ID
  • Earlier log ID
  • Relationship type
  • Relevance score
  • Correlation explanation

These events are derived information rather than raw observations.

Why This Distinction Matters

Imagine a payment service experiencing database connectivity issues.

The system may observe:

  1. A database connection refusal
  2. A payment API exception shortly afterward
  3. Additional service-level errors

The timeline can display:

  • Raw log evidence
  • Deterministic correlation signals

This allows operators to distinguish:

What happened

from

What the system inferred

That distinction becomes even more important as future milestones introduce AI-generated diagnoses and recommendations.

⏱️ Timeline Ordering

Timeline events are sorted chronologically.

When an observed log event and a correlation event share the same timestamp, the observed log is displayed first.

This creates a more natural investigative flow:

  1. Show the evidence
  2. Show the interpretation

Correlation events inherit the timestamp of the later event in the relationship, keeping derived information anchored to the event that completed the correlation pair.

🌐 API Endpoints

Method Endpoint Purpose
POST /api/v1/incidents Create an incident
GET /api/v1/incidents List incidents
GET /api/v1/incidents/{incident_id} Retrieve an incident
GET /api/v1/incidents/{incident_id}/events Retrieve timeline events
POST /api/v1/incidents/{incident_id}/logs Associate logs
POST /api/v1/incidents/{incident_id}/investigate Start investigation
POST /api/v1/incidents/{incident_id}/resolve Resolve incident

The API layer handles:

  • Request validation
  • HTTP concerns
  • Exception mapping

The service layer owns:

  • Lifecycle management
  • Correlation refresh logic
  • Timeline generation

This separation keeps business logic out of route handlers and improves testability.

🧪 Testing the Implementation

Testing covered both successful workflows and invalid scenarios.

The suite validates:

  • Incident creation
  • Invalid input handling
  • Title normalization
  • Incident retrieval and listing
  • Lifecycle transitions
  • Repeated lifecycle actions
  • Invalid transitions
  • Log association
  • Duplicate log detection
  • Timeline generation
  • Correlation ordering

Final Result 

60 passed, 1 warning in 2.53s
Enter fullscreen mode Exit fullscreen mode

Additional Verification 

git diff --check
Enter fullscreen mode Exit fullscreen mode

✅ No whitespace issues detected

alembic check
Enter fullscreen mode Exit fullscreen mode

✅ No new upgrade operations detected

No database migration was required because the existing schema already supported the incident and log relationships.

📚 Key Lessons

1. Keep Lifecycle Changes Explicit

Attaching evidence should never silently change incident state.

2. Separate Evidence from Interpretation

Raw logs and correlation signals represent fundamentally different types of information.

3. Keep Correlation Deterministic

The correlation engine remains independent of:

  • FastAPI
  • PostgreSQL queries
  • Vector databases
  • LLM integrations

4. Correlation Scores Are Not Probabilities

A relevance score is a heuristic.

It indicates potential importance but does not prove causation.

5. Refresh Correlations When Evidence Changes

Newly associated logs can alter the strongest known relationships among existing evidence.

6. Test State Transitions, Not Just Endpoints

A successful HTTP response is not enough.

Lifecycle correctness requires explicit testing of:

  • Idempotency
  • Invalid transitions
  • State persistence

🔮 What's Next?

Milestone 6: Qdrant Integration & Evidence Retrieval

With incident lifecycle management and timeline generation complete, the next step is introducing vector search and evidence retrieval.

This milestone will establish the retrieval foundation needed for future RAG workflows.

AI-powered diagnosis remains intentionally separate. Before asking a language model to interpret evidence, the system must first be able to retrieve and organize that evidence reliably.

🏁 Conclusion

Milestone 5 moves IncidentCopilot beyond ingestion and correlation into structured incident management.

The platform can now:

  • Create incidents
  • Manage lifecycle transitions
  • Associate operational evidence
  • Generate chronological timelines
  • Distinguish observed events from derived relationships

Most importantly, it reinforces a principle that has guided the project from the beginning:

A reliable AI incident assistant starts with a reliable engineering foundation.

Before introducing AI-generated diagnoses, the system must be able to organize evidence, preserve chronology, and clearly communicate what it knows versus what it has inferred.

That is the foundation Milestone 5 establishes.

GitHub: https://www.github.com/richardatodo/incidentcopilot

Next: Milestone 6 — Qdrant Integration & Evidence Retrieval

Top comments (0)