My first RAG prototype did exactly what the tutorials suggested. The API received a document, split it into chunks, generated embeddings, and inserted them into a vector database. Then it queried the database, passed the context to an LLM, and returned the answer.
It worked perfectly for a demo. It fell apart when I tried to use it for anything real.
The failure wasn't in the retrieval logic or the LLM prompting. The failure was in the ingestion pipeline. When I tested the pipeline with a 47-page PDF, the embedding API would time out after 28 seconds on page 38, returning a 504 Gateway Timeout. Because the entire process was synchronous, the API request failed. If I retried, I had to re-chunk the entire document, risking duplicate vectors and wasted compute.
I realized I was treating the vector database as the core of the system. It isn't. The vector database is just a derived index. The actual engineering challenge in a production RAG pipeline is managing the state, boundaries, and failure modes of the ingestion process.
Source of Truth vs. Derived Index
The mental model shift that fixed my architecture was separating the "source of truth" from the "read model."
In my initial design, the vector database was acting as both. If a vector was missing, the document was effectively lost. If I needed to change my chunking strategy, I had to re-upload every document from the client.
I needed to treat the raw documents as the immutable source of truth and the vector store as an eventually consistent, derived index. This meant the document upload could not be tied to the embedding process. The API needed to accept the file, store it durably, and return immediately. The processing had to happen asynchronously.
Building the Asynchronous Boundary
To implement this on AWS, I needed an event-driven pipeline that could handle slow, flaky downstream dependencies without dropping data or failing the user's request.
I designed the ingestion boundary around Amazon S3. Once the file is successfully uploaded to S3, the API returns a 200 OK to the client. The file is now durable.
From S3, I needed a way to trigger processing. I could send S3 event notifications directly to Lambda, or route the events through SQS. I chose SQS because I wanted an explicit buffering boundary between uploads and workers.
The embedding process is slow and susceptible to rate limits. If I triggered Lambda directly from S3, a spike in uploads would result in a spike in concurrent Lambda invocations, likely hitting embedding API rate limits and causing widespread failures. SQS acts as a buffer. It absorbs the S3 events and allows the worker Lambda to pull messages at a controlled rate. More importantly, SQS provides a visibility timeout. If the worker Lambda crashes while generating an embedding, the message becomes visible again and is automatically redelivered. This gives the system a natural retry mechanism, though it still requires the application to handle idempotency.
For permanent failures—malformed PDFs, embedding APIs that consistently return 400s on specific inputs—I configured a dead-letter queue. After three failed attempts, the message moves to the DLQ where it can be inspected and the document status updated to FAILED. Without this, a permanently unprocessable message would retry indefinitely and never surface as a failure in the status table.
For the vector store, I chose OpenSearch Serverless because I wanted the retrieval layer to remain an explicit subsystem without managing the underlying infrastructure. If the application were already heavily dependent on PostgreSQL, I would probably start with pgvector instead.
Idempotency in the Ingestion Path
Moving to an asynchronous, retry-heavy pipeline introduces a new problem: duplicate processing. If the worker Lambda processes a message, updates OpenSearch, but crashes before deleting the message from SQS, SQS will redeliver the message.
If the worker blindly inserts chunks into OpenSearch on every retry, the vector index will fill with duplicates, degrading retrieval quality and increasing costs.
The worker needed to be idempotent. The simplest way to achieve this in a vector database is to use a deterministic document ID.
import hashlib
def generate_chunk_id(s3_key: str, s3_version_id: str, chunk_index: int) -> str:
# Create a deterministic hash based on the exact source file and chunk position
raw_string = f"{s3_key}:{s3_version_id}:{chunk_index}"
return hashlib.sha256(raw_string.encode()).hexdigest()
def process_document(s3_key: str, s3_version_id: str, chunks: list[str]):
documents_to_index = []
for index, chunk_text in enumerate(chunks):
doc_id = generate_chunk_id(s3_key, s3_version_id, index)
documents_to_index.append({
"_id": doc_id,
"text": chunk_text,
"source": s3_key,
"version": s3_version_id
# embedding vector would be added here
})
# OpenSearch bulk API will overwrite existing documents with the same _id
opensearch_client.bulk(body=documents_to_index)
By including the S3 object key and the S3 version ID in the hash, I ensure that if the underlying document changes (creating a new version ID), it generates new chunk IDs. But if the exact same message is retried, it generates the exact same chunk IDs. When the worker sends the bulk request to OpenSearch, OpenSearch simply overwrites the existing documents with the same _id. The retry becomes a no-op at the data layer.
There's a subtle issue this approach doesn't solve: orphaned chunks. If a document gets reprocessed with a different chunking strategy, or if the content changes such that the chunk count shrinks (say from 12 chunks to 8), chunks 8–11 from the old version never get deleted. They remain in the index as stale data. Deterministic IDs solve duplicate-on-retry, but they don't solve delete-on-reduce. To handle this, I added a cleanup step at the start of processing: before indexing new chunks, the worker queries OpenSearch for all documents with the same s3_key and version_id, then deletes any chunks with an index greater than or equal to the new chunk count. Alternatively, you could track the upper-bound chunk count in your DynamoDB status record and sweep for orphans after processing completes.
One more edge case: OpenSearch's _bulk API returns per-item errors. The snippet above treats the bulk operation as all-or-nothing, but in practice, a batch where 3 of 20 chunks fail (due to mapping errors or throttling) will silently leave your index partially written. For production use, you need to inspect the bulk response and retry or report the failed items individually. I handled this by parsing the items array in the bulk response and logging any failures with their corresponding chunk indices, then retrying those specific chunks on the next invocation.
The Cost of Eventual Consistency
Decoupling the upload from the processing solved my reliability and scaling problems, but it introduced a user experience trade-off: eventual consistency.
When a user uploads a document, it is no longer immediately searchable. There is a delay while the file sits in SQS, gets pulled by Lambda, chunked, embedded, and indexed.
I had to build a status tracking mechanism. For the design, I used a simple DynamoDB table keyed by the S3 object key. The upload handler writes a PROCESSING status. The worker Lambda updates it to COMPLETED (or FAILED) when it finishes. The frontend polls this status endpoint to show a progress indicator.
This added complexity to the frontend and required managing the lifecycle of these status records. It is a direct trade-off for the reliability of the backend. If your use case requires immediate, synchronous searchability upon upload, this architecture will not work without significant compromises to the processing limits.
What I Would Change Next
Looking back at the architecture, the custom pipeline gave me complete control over the chunking logic and the retry boundaries. But it also required me to design and maintain the SQS workers, the idempotency logic, and the status tracking.
If I were starting a new project today with standard chunking requirements, I would seriously consider using Amazon Bedrock Knowledge Bases. It handles the S3 ingestion, chunking, embedding, and vector storage internally. I would only build the custom SQS/Lambda pipeline if I needed highly specialized chunking logic (like preserving complex table structures) or if I needed to integrate with a vector store not natively supported by Bedrock.
The biggest change in my design wasn't the vector database. It was treating the document and its vector representation as two different things: the document is the source of truth, while the vector index is something I can rebuild. Once I made that distinction, asynchronous ingestion, retries, idempotency, and processing status all became much easier to reason about.
Top comments (0)