📌 Introduction
One of the first challenges in building an incident investigation tool is getting logs from different systems into a form the application can work with consistently.
- Nginx → access and error logs
- Kubernetes → events about workloads and containers
- Docker → container output
- Applications → structured/unstructured messages
- GitHub Actions → workflow and job failures
These sources differ in structure and terminology, but an incident investigation system needs to examine them together.
Milestone 3 of IncidentCopilot focused on building the ingestion foundation — ensuring the application can accept, validate, normalize, and persist logs reliably.
🔎 The Scope: Five Log Sources
Supported sources:
- Nginx
- Kubernetes
- Docker
- Application logs
- GitHub Actions
Each source has its own parser module. Parsers produce a shared normalized representation with fields like:
- Source
- Timestamp
- Severity
- Service
- Event type
- Message
- Metadata
- Raw message
Source-specific details remain in metadata (e.g., Nginx → HTTP status code, Kubernetes → namespace + pod info).
⚙️ Ingestion API
Endpoints:
| Method | Endpoint | Purpose |
|---|---|---|
| POST | /api/v1/logs |
Submit one log |
| POST | /api/v1/logs/batch |
Submit multiple logs |
| GET | /api/v1/logs |
List logs with pagination |
| GET | /api/v1/logs/{id} |
Retrieve a specific log |
- Batch endpoint accepts 1–100 records.
- Source-specific schemas validate incoming data.
- Invalid batch items → request rejected before ingestion.
✅ What Worked — and What Needed Fixing
- Live API checks confirmed single + batch ingestion worked (
201 Created,200 OK). - But one test failed: expected empty collection → got six records.
- Cause: test queried dev DB with leftover data from live checks.
- Fix: test environment isolation.
🧪 Isolating the Test Database
Solution:
- Created dedicated DB →
incidentcopilot_test. - Applied Alembic migrations.
- Added pytest fixture in
backend/tests/conftest.py:- Test-specific SQLAlchemy engine + session
- Overrides FastAPI DB dependency
- Transaction rollback after each test
Benefits:
- Environment isolation → tests no longer run against dev DB.
- Test isolation → records rolled back after each test.
🔍 Verification
After correction:
| Verification | Result |
|---|---|
| First run | 27 passed in 0.88s |
| Second run | 27 passed in 2.00s |
| Alembic check | No new upgrade ops |
| Git diff check | No whitespace errors |
Suite covers API, DB connectivity, health checks, ingestion validation, batch handling, unknown IDs.
🚫 What This Milestone Does Not Do
Milestone 3 = ingestion foundation.
It does not yet:
- Correlate logs into incidents
- Build investigation timelines
- Query vector DB
- Generate AI diagnoses
📚 Lessons Learned
- Different sources need different validation, but a common representation.
- Live API checks ≠ automated tests. Controlled setup matters.
- Test isolation improves engineering quality. Dedicated test DB + rollback fixtures made the suite reliable.
🔮 What’s Next?
Next milestone → log normalization + correlation.
That work will connect related events into incident context. AI diagnosis and retrieval layers will come later.
🏁 Conclusion
Milestone 3 delivered:
- Multi-source ingestion workflow
- Source-specific parsers + validation
- Normalization with raw evidence preserved
- Single + batch ingestion endpoints
- PostgreSQL persistence
- Reliable test isolation (27 passing tests)
This milestone wasn’t just about accepting logs — it was about making the foundation testable enough to trust as the project grows.
🔗 GitHub: github.com/richardatodo/incidentcopilot
➡️ Next: Milestone 4 — Normalization & Correlation Engine
Top comments (0)