Introduction
An AI-powered DevOps incident assistant needs more than an LLM.
When an application experiences an outage, the system needs access to relevant operational knowledge: troubleshooting procedures, infrastructure documentation, deployment guidance, error explanations, and historical incident information.
Without that knowledge, an LLM may generate plausible but generic suggestions that do not reflect the actual environment or established operational procedures.
For IncidentCopilot, Milestone 6 focused on building the local Retrieval-Augmented Generation (RAG) foundation that will eventually supply relevant technical knowledge to the incident diagnosis engine.
The goal was to create a pipeline capable of:
- Loading Markdown runbooks
- Processing them into meaningful chunks
- Generating local embeddings
- Storing vectors in a vector database
- Retrieving relevant knowledge using natural-language search
Importantly, this milestone deliberately stopped short of AI diagnosis.
The objective was to build and verify retrieval independently before introducing LLM reasoning.
๐๏ธ Architecture Overview
The Milestone 6 pipeline consists of five stages:
- Document loading
- Document chunking
- Embedding generation
- Vector storage
- Similarity retrieval
flowchart TD
A[Markdown Runbooks] --> B[Document Loader]
B --> C[Section-Aware Chunker]
C --> D[Embedding Provider]
D --> E[Qdrant Vector Database]
F[User Search Query] --> G[Query Embedding]
G --> H[Vector Similarity Search]
E --> H
H --> I[Retrieved Knowledge Chunks]
I --> J[API Response]
The indexing pipeline and search pipeline share the same embedding model so that both document vectors and query vectors exist within the same embedding space.
๐ค Why RAG?
Retrieval-Augmented Generation combines information retrieval with language-model generation.
Instead of relying entirely on knowledge embedded in a model's training data, a RAG system retrieves relevant external information and supplies it as context.
The process typically looks like this:
- User submits a question
- Question is converted into an embedding
- The embedding is compared against stored document embeddings
- Relevant document chunks are retrieved
- Retrieved content is supplied to an LLM
- The LLM generates a grounded response
For this milestone, I implemented steps 1โ4 only.
The API currently exposes retrieved content directly. LLM generation will be introduced in a future milestone.
This separation allows retrieval quality to be tested independently from model behavior.
๐ฆ Adding Qdrant
I selected Qdrant as the vector database for local development.
Qdrant provides:
- Vector storage
- Similarity search
- Payload metadata
- Containerized deployment
It aligns well with IncidentCopilot's local-first architecture.
Docker Configuration
Persistent storage is configured using a named volume:
volumes:
- qdrant_data:/qdrant/storage
This ensures vector data survives container recreation.
Exposed Ports
-
6333โ HTTP API -
6334โ gRPC API
The backend waits for Qdrant's health check before startup to ensure the vector database is available.
Network Configuration
Inside Docker:
http://qdrant:6333
Host machine:
http://localhost:6333
Keeping these values separate avoids confusion between container networking and host networking.
๐ง Choosing Local Embeddings
IncidentCopilot is designed to operate without paid cloud APIs.
Embeddings allow the system to compare meaning rather than relying on exact keyword matching.
For example:
- "The application is unavailable"
- "The service is down"
- "The upstream server cannot be reached"
Although worded differently, these statements describe similar concepts.
Embedding Model
The project uses:
sentence-transformers/all-MiniLM-L6-v2
via FastEmbed.
Vector size:
384 dimensions
Provider Abstraction
The embedding layer uses a provider-neutral interface:
class EmbeddingProvider(Protocol):
@property
def dimension(self) -> int:
...
def embed_documents(
self,
texts: Sequence[str],
) -> list[list[float]]:
...
def embed_query(
self,
text: str,
) -> list...
This keeps the RAG system independent from any specific embedding implementation.
Future providers can be substituted without changing retrieval logic.
Lazy Model Loading
The embedding model is loaded only when needed.
Benefits:
- Faster application startup
- Reduced memory usage
- Avoids unnecessary initialization
The model instance is cached after first use so subsequent requests do not repeatedly load model weights.
๐ Building the Document Loader
The initial knowledge base consists of Markdown runbooks.
The loader validates that:
- The runbooks directory exists
- The path is a directory
- Markdown files are not empty
- Documents begin with an H1 heading
- Titles are not empty
- Documents contain content after the title
Document Structure
@dataclass(frozen=True)
class MarkdownDocument:
document_id: str
title: str
content: str
source_path: str
Deterministic Document IDs
Document IDs are derived from the file path using SHA-256.
Example:
runbooks/nginx-502.md
The same file path always generates the same document ID.
This means document identity remains stable even when content changes.
โ๏ธ Document Chunking
Embedding an entire runbook as one vector reduces retrieval precision.
Queries often need a specific troubleshooting step rather than an entire document.
Chunking allows retrieval of relevant sections instead of full runbooks.
Why Section-Aware Chunking?
An early implementation revealed an issue:
- Chunks sometimes contained multiple sections
- Metadata represented only the final section
This reduced retrieval quality.
The revised chunker treats Markdown sections independently.
Example:
# Nginx 502 Troubleshooting
## Overview
An Nginx 502 error indicates an invalid upstream response.
## Common Causes
- Upstream service unavailable
- Incorrect upstream configuration
## Investigation Steps
- Inspect Nginx logs
- Verify upstream connectivity
Each section becomes a logical retrieval unit.
Chunk Metadata
Each chunk retains:
- Document ID
- Document title
- Source path
- Chunk index
- Section
- Content
- Chunk ID
This metadata provides context during retrieval.
Word-Boundary Preservation
A second issue emerged when chunks split words.
Example:
restartingor
This hurts:
- Readability
- Embedding quality
- Retrieval relevance
The updated chunker now:
- Splits on complete words
- Preserves overlap using whole-word boundaries
- Maintains readability while retaining context
Chunk Configuration
Chunking behavior is configurable:
- Chunk size
- Chunk overlap
Validation ensures:
- Chunk size > 0
- Overlap โฅ 0
- Overlap < chunk size
This allows future tuning without changing implementation logic.
๐๏ธ Vector Storage Design
Qdrant stores each chunk as a vector point with metadata.
Payload fields include:
chunk_id
document_id
document_type
title
content
source_path
chunk_index
section
Metadata enables explainable retrieval.
The API can return:
- Retrieved content
- Document title
- Source runbook
- Relevant section
- Similarity score
Deterministic Point IDs
Qdrant point IDs are generated using UUID5 derived from the chunk ID.
Benefits:
- Predictable indexing
- Idempotent upserts
- No accidental duplication
Re-indexing the same content updates existing points instead of creating new ones.
Replacing Obsolete Chunks
Runbooks evolve over time.
Simply upserting new chunks would leave outdated chunks searchable.
To solve this, indexing performs document-level replacement:
- Retrieve existing point IDs
- Generate IDs for new chunks
- Upsert new chunks
- Identify obsolete points
- Delete obsolete points
This ensures search results remain current.
โ๏ธ The RAG Service
The RAG service orchestrates:
- Document loading
- Chunking
- Embedding generation
- Vector storage
- Retrieval
Indexing Workflow
Load Documents
โ
Chunk Documents
โ
Generate Embeddings
โ
Validate Embedding Count
โ
Group Chunks by Document
โ
Replace Chunks in Vector Store
Embeddings are generated before modifying Qdrant, reducing the risk of partial indexing failures.
Search Workflow
Validate Query
โ
Generate Query Embedding
โ
Search Qdrant
โ
Return Ranked Results
Invalid queries and invalid limits are rejected.
๐ API Endpoints
Index Runbooks
POST /api/v1/rag/index
Example response:
{
"documents_indexed": 5,
"chunks_indexed": 30
}
Search Runbooks
POST /api/v1/rag/search
Example request:
{
"query": "Nginx returns 502 because the upstream application is unavailable",
"limit": 3
}
The response contains:
- Query
- Result count
- Retrieved chunks
- Metadata
- Similarity scores
๐ Building the Initial Knowledge Base
Five operational runbooks were created:
Nginx 502 Bad Gateway
Covers:
- Upstream availability
- Bad upstream configuration
- Docker networking
- Connection timeouts
- Log analysis
Database Connection Failure
Covers:
- Database availability
- Authentication issues
- Connection configuration
- Network restrictions
- Connection pool exhaustion
Kubernetes CrashLoopBackOff
Covers:
- Repeated container restarts
- Pod inspection
- Exit codes
- Resource limits
- Health probes
Docker Container Restart
Covers:
- Unexpected exits
- Restart policies
- OOM events
- Health check failures
- Host resource issues
GitHub Actions Workflow Failure
Covers:
- Workflow syntax
- Secrets
- Permissions
- Dependency failures
- Runner configuration
- Deployment authentication
๐งช Testing Strategy
Testing covered multiple layers.
Document Tests
Validates:
- Document loading
- Empty directories
- Empty files
- Missing titles
- Missing content
- Metadata preservation
- Chunk generation
- Deterministic IDs
Service Tests
Validates:
- Indexing
- Retrieval
- Empty retrieval
- Repeated indexing
- Chunk replacement
- Invalid queries
- Invalid limits
Vector Store Tests
Validates:
- Collection creation
- Upserts
- Metadata persistence
- Similarity ordering
- Vector validation
- Idempotent indexing
API Tests
Validates:
- Successful indexing
- Search requests
- Validation failures
- Empty results
Final Test Results
103 passed
0 failed
1 warning
โ Live Integration Verification
Infrastructure verification confirmed:
- PostgreSQL healthy
- Backend running
- Qdrant healthy
- Runbooks mounted
- Embedding model downloaded
- Live indexing successful
Indexing Result
{
"documents_indexed": 5,
"chunks_indexed": 30
}
Repeated indexing produced the same result.
Qdrant verification confirmed:
Indexed points: 30
No duplicate vectors were created.
๐ Retrieval Quality
Five live natural-language queries were executed:
| Query Category | Correct Runbook Retrieved First |
|---|---|
| Nginx 502 | โ Yes |
| Database Connectivity | โ Yes |
| Kubernetes CrashLoopBackOff | โ Yes |
| Docker Restart | โ Yes |
| GitHub Actions Failure | โ Yes |
Retrieved results included both relevant content and supporting metadata.
๐ Lessons Learned
Retrieval Quality Depends on Documentation Quality
Even a high-quality vector database cannot compensate for poor documentation.
Good retrieval requires:
- Clear runbooks
- Meaningful headings
- Logical sections
- Useful metadata
Chunking Is Critical
Poor chunking can:
- Split words
- Remove context
- Produce misleading metadata
- Harm retrieval quality
Section-aware chunking proved far more effective for operational documentation.
Deterministic IDs Simplify Indexing
Stable identifiers make repeated indexing predictable and prevent duplication.
Local-First Architecture Matters
Using FastEmbed and Qdrant provides:
- Offline-capable development
- Lower operating costs
- Better control of operational data
- Reproducible environments
Test Isolation Remains Important
The backend test suite continues to use a dedicated PostgreSQL test database.
This ensures tests never run against development data.
โ ๏ธ Current Limitations
The system retrieves operational knowledge but does not yet generate diagnoses.
It does not:
- Analyze incidents using an LLM
- Generate root-cause conclusions
- Combine retrieved knowledge with correlation data
- Index historical incidents
- Perform reranking
- Benchmark retrieval quality formally
The retrieval layer should currently be treated as an evidence source, not a diagnostic authority.
๐ฎ What Comes Next?
The next milestone introduces a local LLM through Ollama.
The diagnosis pipeline will build upon:
- Incident lifecycle management
- Log ingestion
- Deterministic correlation
- Incident timelines
- RAG retrieval
The future system will clearly distinguish:
- Observed facts
- Correlation signals
- Retrieved guidance
- AI-generated hypotheses
- Recommendations
- Missing evidence
The goal is not to make the AI sound confident.
The goal is to produce explainable, evidence-grounded incident analysis.
๐ Conclusion
Milestone 6 successfully established a local RAG infrastructure for IncidentCopilot.
The platform can now:
- Load Markdown runbooks
- Create section-aware chunks
- Generate local embeddings
- Store vectors in Qdrant
- Retrieve relevant operational knowledge through an API
All five initial runbooks were indexed and verified through live retrieval.
Repeated indexing produced the expected 30 vector points without duplication, and the full backend suite completed successfully with 103 passing tests.
This milestone establishes the knowledge retrieval layer required for the next stage:
Locally powered, evidence-grounded AI incident diagnosis.
GitHub: https://github.com/richardatodo/incidentcopilot
Next: Milestone 7 โ Ollama-Powered Incident Diagnosis
Top comments (0)