DEV Community

Richard Atodo
Richard Atodo

Posted on Edited on

๐Ÿ” Building a Local RAG Pipeline for an AI DevOps Incident Copilot

Introduction

An AI-powered DevOps incident assistant needs more than an LLM.

When an application experiences an outage, the system needs access to relevant operational knowledge: troubleshooting procedures, infrastructure documentation, deployment guidance, error explanations, and historical incident information.

Without that knowledge, an LLM may generate plausible but generic suggestions that do not reflect the actual environment or established operational procedures.

For IncidentCopilot, Milestone 6 focused on building the local Retrieval-Augmented Generation (RAG) foundation that will eventually supply relevant technical knowledge to the incident diagnosis engine.

The goal was to create a pipeline capable of:

  • Loading Markdown runbooks
  • Processing them into meaningful chunks
  • Generating local embeddings
  • Storing vectors in a vector database
  • Retrieving relevant knowledge using natural-language search

Importantly, this milestone deliberately stopped short of AI diagnosis.

The objective was to build and verify retrieval independently before introducing LLM reasoning.


๐Ÿ—๏ธ Architecture Overview

The Milestone 6 pipeline consists of five stages:

  1. Document loading
  2. Document chunking
  3. Embedding generation
  4. Vector storage
  5. Similarity retrieval
flowchart TD
    A[Markdown Runbooks] --> B[Document Loader]
    B --> C[Section-Aware Chunker]
    C --> D[Embedding Provider]
    D --> E[Qdrant Vector Database]

    F[User Search Query] --> G[Query Embedding]
    G --> H[Vector Similarity Search]
    E --> H
    H --> I[Retrieved Knowledge Chunks]
    I --> J[API Response]

The indexing pipeline and search pipeline share the same embedding model so that both document vectors and query vectors exist within the same embedding space.


๐Ÿค” Why RAG?

Retrieval-Augmented Generation combines information retrieval with language-model generation.

Instead of relying entirely on knowledge embedded in a model's training data, a RAG system retrieves relevant external information and supplies it as context.

The process typically looks like this:

  1. User submits a question
  2. Question is converted into an embedding
  3. The embedding is compared against stored document embeddings
  4. Relevant document chunks are retrieved
  5. Retrieved content is supplied to an LLM
  6. The LLM generates a grounded response

For this milestone, I implemented steps 1โ€“4 only.

The API currently exposes retrieved content directly. LLM generation will be introduced in a future milestone.

This separation allows retrieval quality to be tested independently from model behavior.


๐Ÿ“ฆ Adding Qdrant

I selected Qdrant as the vector database for local development.

Qdrant provides:

  • Vector storage
  • Similarity search
  • Payload metadata
  • Containerized deployment

It aligns well with IncidentCopilot's local-first architecture.

Docker Configuration

Persistent storage is configured using a named volume:

volumes:
  - qdrant_data:/qdrant/storage
Enter fullscreen mode Exit fullscreen mode

This ensures vector data survives container recreation.

Exposed Ports

  • 6333 โ†’ HTTP API
  • 6334 โ†’ gRPC API

The backend waits for Qdrant's health check before startup to ensure the vector database is available.

Network Configuration

Inside Docker:

http://qdrant:6333
Enter fullscreen mode Exit fullscreen mode

Host machine:

http://localhost:6333
Enter fullscreen mode Exit fullscreen mode

Keeping these values separate avoids confusion between container networking and host networking.


๐Ÿง  Choosing Local Embeddings

IncidentCopilot is designed to operate without paid cloud APIs.

Embeddings allow the system to compare meaning rather than relying on exact keyword matching.

For example:

  • "The application is unavailable"
  • "The service is down"
  • "The upstream server cannot be reached"

Although worded differently, these statements describe similar concepts.

Embedding Model

The project uses:

sentence-transformers/all-MiniLM-L6-v2
Enter fullscreen mode Exit fullscreen mode

via FastEmbed.

Vector size:

384 dimensions
Enter fullscreen mode Exit fullscreen mode

Provider Abstraction

The embedding layer uses a provider-neutral interface:

class EmbeddingProvider(Protocol):
    @property
    def dimension(self) -> int:
        ...

    def embed_documents(
        self,
        texts: Sequence[str],
    ) -> list[list[float]]:
        ...

    def embed_query(
        self,
        text: str,
    ) -> list...
Enter fullscreen mode Exit fullscreen mode

This keeps the RAG system independent from any specific embedding implementation.

Future providers can be substituted without changing retrieval logic.


Lazy Model Loading

The embedding model is loaded only when needed.

Benefits:

  • Faster application startup
  • Reduced memory usage
  • Avoids unnecessary initialization

The model instance is cached after first use so subsequent requests do not repeatedly load model weights.


๐Ÿ“„ Building the Document Loader

The initial knowledge base consists of Markdown runbooks.

The loader validates that:

  • The runbooks directory exists
  • The path is a directory
  • Markdown files are not empty
  • Documents begin with an H1 heading
  • Titles are not empty
  • Documents contain content after the title

Document Structure

@dataclass(frozen=True)
class MarkdownDocument:
    document_id: str
    title: str
    content: str
    source_path: str
Enter fullscreen mode Exit fullscreen mode

Deterministic Document IDs

Document IDs are derived from the file path using SHA-256.

Example:

runbooks/nginx-502.md
Enter fullscreen mode Exit fullscreen mode

The same file path always generates the same document ID.

This means document identity remains stable even when content changes.


โœ‚๏ธ Document Chunking

Embedding an entire runbook as one vector reduces retrieval precision.

Queries often need a specific troubleshooting step rather than an entire document.

Chunking allows retrieval of relevant sections instead of full runbooks.


Why Section-Aware Chunking?

An early implementation revealed an issue:

  • Chunks sometimes contained multiple sections
  • Metadata represented only the final section

This reduced retrieval quality.

The revised chunker treats Markdown sections independently.

Example:

# Nginx 502 Troubleshooting

## Overview

An Nginx 502 error indicates an invalid upstream response.

## Common Causes

- Upstream service unavailable
- Incorrect upstream configuration

## Investigation Steps

- Inspect Nginx logs
- Verify upstream connectivity
Enter fullscreen mode Exit fullscreen mode

Each section becomes a logical retrieval unit.


Chunk Metadata

Each chunk retains:

  • Document ID
  • Document title
  • Source path
  • Chunk index
  • Section
  • Content
  • Chunk ID

This metadata provides context during retrieval.


Word-Boundary Preservation

A second issue emerged when chunks split words.

Example:

restartingor
Enter fullscreen mode Exit fullscreen mode

This hurts:

  • Readability
  • Embedding quality
  • Retrieval relevance

The updated chunker now:

  • Splits on complete words
  • Preserves overlap using whole-word boundaries
  • Maintains readability while retaining context

Chunk Configuration

Chunking behavior is configurable:

  • Chunk size
  • Chunk overlap

Validation ensures:

  • Chunk size > 0
  • Overlap โ‰ฅ 0
  • Overlap < chunk size

This allows future tuning without changing implementation logic.


๐Ÿ—„๏ธ Vector Storage Design

Qdrant stores each chunk as a vector point with metadata.

Payload fields include:

chunk_id
document_id
document_type
title
content
source_path
chunk_index
section
Enter fullscreen mode Exit fullscreen mode

Metadata enables explainable retrieval.

The API can return:

  • Retrieved content
  • Document title
  • Source runbook
  • Relevant section
  • Similarity score

Deterministic Point IDs

Qdrant point IDs are generated using UUID5 derived from the chunk ID.

Benefits:

  • Predictable indexing
  • Idempotent upserts
  • No accidental duplication

Re-indexing the same content updates existing points instead of creating new ones.


Replacing Obsolete Chunks

Runbooks evolve over time.

Simply upserting new chunks would leave outdated chunks searchable.

To solve this, indexing performs document-level replacement:

  1. Retrieve existing point IDs
  2. Generate IDs for new chunks
  3. Upsert new chunks
  4. Identify obsolete points
  5. Delete obsolete points

This ensures search results remain current.


โš™๏ธ The RAG Service

The RAG service orchestrates:

  • Document loading
  • Chunking
  • Embedding generation
  • Vector storage
  • Retrieval

Indexing Workflow

Load Documents
      โ†“
Chunk Documents
      โ†“
Generate Embeddings
      โ†“
Validate Embedding Count
      โ†“
Group Chunks by Document
      โ†“
Replace Chunks in Vector Store
Enter fullscreen mode Exit fullscreen mode

Embeddings are generated before modifying Qdrant, reducing the risk of partial indexing failures.


Search Workflow

Validate Query
      โ†“
Generate Query Embedding
      โ†“
Search Qdrant
      โ†“
Return Ranked Results
Enter fullscreen mode Exit fullscreen mode

Invalid queries and invalid limits are rejected.


๐ŸŒ API Endpoints

Index Runbooks

POST /api/v1/rag/index
Enter fullscreen mode Exit fullscreen mode

Example response:

{
  "documents_indexed": 5,
  "chunks_indexed": 30
}
Enter fullscreen mode Exit fullscreen mode

Search Runbooks

POST /api/v1/rag/search
Enter fullscreen mode Exit fullscreen mode

Example request:

{
  "query": "Nginx returns 502 because the upstream application is unavailable",
  "limit": 3
}
Enter fullscreen mode Exit fullscreen mode

The response contains:

  • Query
  • Result count
  • Retrieved chunks
  • Metadata
  • Similarity scores

๐Ÿ“š Building the Initial Knowledge Base

Five operational runbooks were created:

Nginx 502 Bad Gateway

Covers:

  • Upstream availability
  • Bad upstream configuration
  • Docker networking
  • Connection timeouts
  • Log analysis

Database Connection Failure

Covers:

  • Database availability
  • Authentication issues
  • Connection configuration
  • Network restrictions
  • Connection pool exhaustion

Kubernetes CrashLoopBackOff

Covers:

  • Repeated container restarts
  • Pod inspection
  • Exit codes
  • Resource limits
  • Health probes

Docker Container Restart

Covers:

  • Unexpected exits
  • Restart policies
  • OOM events
  • Health check failures
  • Host resource issues

GitHub Actions Workflow Failure

Covers:

  • Workflow syntax
  • Secrets
  • Permissions
  • Dependency failures
  • Runner configuration
  • Deployment authentication

๐Ÿงช Testing Strategy

Testing covered multiple layers.

Document Tests

Validates:

  • Document loading
  • Empty directories
  • Empty files
  • Missing titles
  • Missing content
  • Metadata preservation
  • Chunk generation
  • Deterministic IDs

Service Tests

Validates:

  • Indexing
  • Retrieval
  • Empty retrieval
  • Repeated indexing
  • Chunk replacement
  • Invalid queries
  • Invalid limits

Vector Store Tests

Validates:

  • Collection creation
  • Upserts
  • Metadata persistence
  • Similarity ordering
  • Vector validation
  • Idempotent indexing

API Tests

Validates:

  • Successful indexing
  • Search requests
  • Validation failures
  • Empty results

Final Test Results

103 passed
0 failed
1 warning
Enter fullscreen mode Exit fullscreen mode

โœ… Live Integration Verification

Infrastructure verification confirmed:

  • PostgreSQL healthy
  • Backend running
  • Qdrant healthy
  • Runbooks mounted
  • Embedding model downloaded
  • Live indexing successful

Indexing Result

{
  "documents_indexed": 5,
  "chunks_indexed": 30
}
Enter fullscreen mode Exit fullscreen mode

Repeated indexing produced the same result.

Qdrant verification confirmed:

Indexed points: 30
Enter fullscreen mode Exit fullscreen mode

No duplicate vectors were created.


๐Ÿ” Retrieval Quality

Five live natural-language queries were executed:

Query Category Correct Runbook Retrieved First
Nginx 502 โœ… Yes
Database Connectivity โœ… Yes
Kubernetes CrashLoopBackOff โœ… Yes
Docker Restart โœ… Yes
GitHub Actions Failure โœ… Yes

Retrieved results included both relevant content and supporting metadata.


๐Ÿ“– Lessons Learned

Retrieval Quality Depends on Documentation Quality

Even a high-quality vector database cannot compensate for poor documentation.

Good retrieval requires:

  • Clear runbooks
  • Meaningful headings
  • Logical sections
  • Useful metadata

Chunking Is Critical

Poor chunking can:

  • Split words
  • Remove context
  • Produce misleading metadata
  • Harm retrieval quality

Section-aware chunking proved far more effective for operational documentation.

Deterministic IDs Simplify Indexing

Stable identifiers make repeated indexing predictable and prevent duplication.

Local-First Architecture Matters

Using FastEmbed and Qdrant provides:

  • Offline-capable development
  • Lower operating costs
  • Better control of operational data
  • Reproducible environments

Test Isolation Remains Important

The backend test suite continues to use a dedicated PostgreSQL test database.

This ensures tests never run against development data.


โš ๏ธ Current Limitations

The system retrieves operational knowledge but does not yet generate diagnoses.

It does not:

  • Analyze incidents using an LLM
  • Generate root-cause conclusions
  • Combine retrieved knowledge with correlation data
  • Index historical incidents
  • Perform reranking
  • Benchmark retrieval quality formally

The retrieval layer should currently be treated as an evidence source, not a diagnostic authority.


๐Ÿ”ฎ What Comes Next?

The next milestone introduces a local LLM through Ollama.

The diagnosis pipeline will build upon:

  • Incident lifecycle management
  • Log ingestion
  • Deterministic correlation
  • Incident timelines
  • RAG retrieval

The future system will clearly distinguish:

  • Observed facts
  • Correlation signals
  • Retrieved guidance
  • AI-generated hypotheses
  • Recommendations
  • Missing evidence

The goal is not to make the AI sound confident.

The goal is to produce explainable, evidence-grounded incident analysis.


๐Ÿ Conclusion

Milestone 6 successfully established a local RAG infrastructure for IncidentCopilot.

The platform can now:

  • Load Markdown runbooks
  • Create section-aware chunks
  • Generate local embeddings
  • Store vectors in Qdrant
  • Retrieve relevant operational knowledge through an API

All five initial runbooks were indexed and verified through live retrieval.

Repeated indexing produced the expected 30 vector points without duplication, and the full backend suite completed successfully with 103 passing tests.

This milestone establishes the knowledge retrieval layer required for the next stage:

Locally powered, evidence-grounded AI incident diagnosis.


GitHub: https://github.com/richardatodo/incidentcopilot

Next: Milestone 7 โ€” Ollama-Powered Incident Diagnosis

Top comments (0)