DEV Community: Le Trung Hieu

What Actually Happens When You Build a Production RAG Chatbot on Amazon Bedrock

Le Trung Hieu — Fri, 23 Oct 2026 07:00:47 +0000

How we built a document Q&A system with NestJS, PostgreSQL, pgvector, Hybrid Search (RRF), Prompt Caching, and Claude on Amazon Bedrock.
Building a demo chatbot is easy.

Building one that consistently answers correctly after weeks of document updates, changing requirements, and real user traffic is where things become interesting.

Most Retrieval-Augmented Generation (RAG) failures don't look like failures.

The API returns 200.

The model responds.

Infrastructure dashboards stay green.

Yet answer quality gradually gets worse.

You ask about error code E-4021 and receive documentation for E-4012.

A recently updated document never appears in search results.

Prompt caching is enabled, but costs don't go down.

Hybrid search is implemented, but retrieval quality barely improves.

Nothing crashes.

Nothing raises an alert.

The chatbot simply becomes less useful over time.

For teams involved in application software development, the real work begins after the first successful response. Retrieval quality, chunking strategy, embedding consistency, prompt caching, document lifecycle management, and hallucination control quickly become more important than model selection itself.

This article walks through a complete production-oriented RAG implementation built with:

Amazon Bedrock
Claude
Cohere Embed Multilingual v3
NestJS
PostgreSQL + pgvector
Hybrid Search using Reciprocal Rank Fusion (RRF)

More importantly, it explains why each component exists, what problem it solves, and which production issues it prevents.

1. What is RAG

RAG (Retrieval-Augmented Generation) does not ask the LLM directly. Before each question, the system:

Finds a few internal document passages most relevant to the question.
Pastes them into the prompt together with the question.
Asks the LLM to answer based only on those passages, with citations.

The consequence of this design:

The problem "how does the LLM know my data" becomes a search problem. If retrieval is weak, the chatbot is weak, because no prompt can produce information that isn't in the context.

This distinction is easy to underestimate.

Many teams spend weeks experimenting with prompts while retrieval quality remains unchanged.

In reality, most enterprise application software development projects that incorporate AI succeed or fail based on retrieval architecture rather than prompt engineering.

That is why most of this article is about chunking documents and search, not about prompts.

Compared with the alternatives:

Stuffing all documents into the prompt: only works for a very small corpus (a few dozen pages). Beyond that you hit the context limit and pay for the tokens on every turn.
Fine-tuning: teaches style, not facts. A fine-tuned model still makes things up, just in your company's voice, which makes it harder to spot. Every data change means retraining.
RAG: updating knowledge only requires re-ingesting, and answers can cite their sources. This is the default choice for business chatbots.

2. Overall architecture

A RAG system has three stages:

Indexing (offline, re-run whenever documents change): read documents → chunk → generate embeddings → store in an index.
Retrieval (online, per question): embed the question, find the most relevant chunks.
Generation (online, per question): put the chunks in the prompt, the LLM writes the answer.

The three stages of RAG: documents are indexed ahead of time, then each question goes through retrieval and then generation

 ══ OFFLINE ══════════════════════════════════════════════════════════
   data/in/*.md|txt|json
        │
        ▼
   LoaderService ──► ChunkerService ──► Bedrock (Cohere Embed) ──► Postgres
   read files        2-level split +    input_type =               chunks table
                     header injection   search_document            (vector + tsvector)

 ══ ONLINE ═══════════════════════════════════════════════════════════
   POST /api/chat { question }
        │
        ▼
   SearchService.hybridSearch
        ├─ embedQuery (Cohere, input_type = search_query)
        ├─ DenseSearch   (HNSW index)  ─┐  run in parallel
        ├─ KeywordSearch (GIN index)   ─┘
        └─ reciprocalRankFusion ──► top 5 chunks
        │
        ▼
   PromptBuilder ── [system + few-shot: CACHED] + [context + question: not cached]
        │
        ▼
   Bedrock (Claude, InvokeModel)
        │
        ▼
   { answer, citations, usage }

Folder layout, one folder per stage:

src/
├── config/     all config, read from env
├── database/   schema.sql + pg.Pool wrapper (no ORM)
├── llm/        LlmService interface + Bedrock implementation
├── ingest/     loader → chunker → repository → ingest.service
├── search/     dense · keyword · rrf · search.service
├── prompt/     system prompt · few-shot · prompt builder
└── chat/       wires the three steps into one RAG loop + HTTP API
scripts/        db-init · ingest · search (CLI)
test/           unit tests for chunker, rrf, prompt builder

This architecture may look surprisingly simple.

That is intentional.

Every stage has a single responsibility and can be tested independently.

When retrieval quality drops, we can inspect retrieval.

When generation quality drops, we can inspect prompts.

When ingest breaks, we can inspect indexing.

The separation becomes extremely valuable once the system grows beyond a proof of concept.

3. Following the data: the two flows as they run in the code

The diagram in section 2 shows the blocks. This section slows down: it takes one concrete document and one concrete question and follows them through each function, each SQL query, and each Bedrock call. Once you have read this section, the code in the later steps is just detail.

The sample data used throughout is data/in/warranty.md:

# Warranty Policy

## Warranty period
All products are covered by a warranty of 12 months from the date of purchase.

## Returns
Products with a manufacturing defect can be exchanged within the first 7 days.

3.1 The ingest flow: from raw files to the index

This flow runs offline via npm run ingest, not over HTTP. Re-run it whenever documents change.

npm run ingest [-- <folder>]
   │
   ▼
scripts/ingest.ts
   │  NestFactory.createApplicationContext(AppModule)   (no HTTP port opened)
   ▼
IngestService.ingestDirectory(dir = INGEST_DIR)
   │
   ▼
[1] LoaderService.loadDirectory ─────────► RawDocument[]
   │    recursive scan; .md/.txt = 1 document; .json = n documents; other extensions skipped
   │
   │    for EACH RawDocument:
   ▼
[2] ChunkerService.chunk ────────────────► Chunk[]
   │    split by heading, split long sections (1200 chars, 120 overlap), prepend heading
   ▼
[3] BedrockService.embedDocuments ───────► one 1024-dim vector per chunk
   │    up to 96 texts per call, Cohere input_type = search_document
   ▼
[4] ChunkRepository.upsert (per chunk) ──► chunks table
   │    INSERT ... ON CONFLICT (chunk_id) DO UPDATE
   │    Postgres computes search_vector and updates HNSW + GIN indexes
   ▼
[5] ChunkRepository.deleteStale ─────────► delete leftover chunks of this document
   │
   ▼
IngestReport { documents, chunks, deleted }  →  printed to console, pool closed

[1] Read files. LoaderService walks the folder recursively. A markdown file becomes one RawDocument:

{
  "source": "data/in/warranty.md",
  "title": "Warranty Policy",
  "text": "# Warranty Policy\n\n## Warranty period\nAll products are covered..."
}

source is the file path, and title comes from the first H1 (or the file name if there is none). For .json files, each element with a text field becomes its own document.

[2] Chunk. splitByHeading reads line by line and starts a new section whenever a line begins with #. The sample file produces 3 sections:

section A  heading = "Warranty Policy"   text = ""                  → no chunk
section B  heading = "Warranty period"   text = "All products are..."
section C  heading = "Returns"           text = "Products with a..."

Section A has a heading but no body, so it is dropped. Sections B and C are shorter than 1200 characters, so they are kept whole. A longer section would be split further by splitLongText: paragraph boundaries first, then sentences, with 120 characters of overlap between consecutive pieces.

After prepending the heading (header injection), we get 2 chunks:

[
  {
    "chunkId": "data/in/warranty.md#0",
    "title": "Warranty period",
    "content": "Warranty period\n\nAll products are covered by a warranty of 12 months from the date of purchase.",
    "metadata": { "heading": "Warranty period" }
  },
  {
    "chunkId": "data/in/warranty.md#1",
    "title": "Returns",
    "content": "Returns\n\nProducts with a manufacturing defect can be exchanged within the first 7 days.",
    "metadata": { "heading": "Returns" }
  }
]

chunkId has the form <source>#<position>. It is the key that lets the next ingest update the existing row instead of adding a new one.

[3] Embed. Both content strings go to Cohere in a single InvokeModel call:

{ "texts": ["Warranty period\n\nAll products are...", "Returns\n\nProducts with a..."],
  "input_type": "search_document", "truncate": "END" }

The result is 2 vectors of 1024 floats each. Because the heading is part of content, chunk #0's vector carries the meaning "warranty period", not just the sentence about 12 months.

[4] Store. Each chunk is one INSERT ... ON CONFLICT (chunk_id) DO UPDATE. The application only writes content and embedding. Postgres does the rest:

The search_vector column is recomputed from title + content with to_tsvector('simple', ...). Chunk #0 yields the tokens warranty, period, all, products, are, covered, by, a, of, 12, months, from, the, date, purchase.
The HNSW index (on embedding) and the GIN index (on search_vector) are updated in the same statement.

[5] Clean up leftovers. deleteStale('data/in/warranty.md', ['...#0', '...#1']) deletes every row of this file whose chunk_id is not in the list just written. On the first run there is nothing to delete. If the file later shrinks from 5 chunks to 2, rows #2, #3 and #4 are deleted here.

At the end, the console prints a summary: number of documents, chunks, and stale chunks deleted.

Three things worth knowing about this flow:

Every run re-embeds everything, even unchanged files. Fine for a small corpus; for a large one, store a content hash in metadata and skip chunks whose hash hasn't changed.
source is the path exactly as scanned. Running npm run ingest (giving data/in/...) and later npm run ingest -- /absolute/path produces a different source, i.e. a duplicate set of chunks. Always use the same form of path.
A file that yields no chunks is skipped, and its old chunks are not cleaned up. To remove a document from the index entirely, use ChunkRepository.deleteBySource.

3.2 The question flow: from question to answer

This flow runs online, once per request. One question costs 2 Bedrock calls (1 embed, 1 generate) and 2 SQL queries run in parallel.

Client
   │  POST /api/chat  { "question": "How long is the product warranty?" }
   ▼
[1] ValidationPipe + AskDto ─────────────► 400 if empty, > 2000 chars, or unknown fields
   │
   ▼
ChatController.ask  →  ChatService.ask
   │
   ▼
[2] SearchService.hybridSearch(question)
   │
   ├─ [2a] BedrockService.embedQuery ────► question vector (Cohere, input_type = search_query)
   │
   ├─ [2b] in parallel (Promise.all):
   │      ├─ DenseSearch   : ORDER BY embedding <=> $vec LIMIT 50   → HNSW index
   │      └─ KeywordSearch : search_vector @@ websearch_to_tsquery  → GIN index
   │
   └─ [2c] reciprocalRankFusion(k = 60) ─► top 5 chunks, with denseRank / keywordRank
   │
   ▼
[3] PromptBuilder.build ─────────────────► { system: [...], user: [...] }
   │      system = SYSTEM_PROMPT + few-shot                    (cached)
   │      user   = <context>[1]..[5]</context> + <question>    (not cached)
   ▼
[4] BedrockService.generate ─────────────► InvokeModel Claude
   │      cache_control on the few-shot block; retry on throttling
   ▼
[5] ChatService assembles the result
   │      answer = all "text" blocks joined
   │      citations = the chunks that were sent, numbered to match [n]
   │      usage = input/output tokens + cache write/read tokens
   ▼
Client receives { answer, citations, usage }

[1] Guard the input. ValidationPipe runs with whitelist and forbidNonWhitelisted, so the body may contain only the question field. AskDto requires question to be a non-empty string of at most 2000 characters. Invalid requests get a 400 immediately, before a single cent is spent on Bedrock.

[2a] Embed the question. The question goes to Cohere with input_type: "search_query". This mirrors step [3] of the ingest flow: documents use search_document, questions use search_query, same model. Get either one wrong and question vectors and document vectors no longer live on the same "map".

[2b] Two search branches run at the same time.

The dense branch compares the question vector with every chunk vector by cosine; thanks to HNSW it does not scan the whole table. With our 2-chunk sample corpus:

dense:   1. ...#0  "Warranty period"   highest cosine
         2. ...#1  "Returns"

The keyword branch passes the raw question to websearch_to_tsquery('simple', ...):

'how' & 'long' & 'is' & 'the' & 'product' & 'warranty'

This function joins words with AND. Chunk #0 contains none of how, long, is, and has products rather than product (the simple dictionary does not stem), so the keyword branch returns 0 results for this question. That is the real behavior of the code, and the reason the dense branch matters: for long natural-language questions the keyword branch usually stays silent. It shines on short queries containing codes or names, like E-4021 or warranty 12 months.

[2c] Merge with RRF. Each chunk gets 1 / (60 + rank) from every branch it appears in:

...#0  rrf = 1/61 = 0.01639   denseRank = 1   keywordRank = null
...#1  rrf = 1/62 = 0.01613   denseRank = 2   keywordRank = null

If the query were warranty 12 months, chunk #0 would appear in both branches, score 1/61 + 1/61 = 0.03279, and pull far ahead of chunks found by only one branch. The list is cut to the top 5 (TOP_K).

[3] Build the prompt. PromptBuilder numbers the chunks from 1 and wraps them in tags (the repository uses Vietnamese prompt text and tag names; English equivalents are shown throughout this article, see section 12):

<context>
[1] Warranty period
Warranty period

All products are covered by a warranty of 12 months from the date of purchase.

[2] Returns
Returns

Products with a manufacturing defect can be exchanged within the first 7 days.
</context>
<question>How long is the product warranty?</question>

The heading appears twice because title is printed first while content already contains the heading from header injection. It's harmless, but you can drop the title line when rendering to save tokens.

If neither branch finds anything, the context is (no relevant documents found) and the LLM is still called. The system prompt and few-shot example #2 teach the model to say it could not find the information instead of inventing an answer.

[4] Call Claude. The body sent via InvokeModel:

{
  "anthropic_version": "bedrock-2023-05-31",
  "max_tokens": 2000,
  "system": [
    { "type": "text", "text": "You are a Q&A assistant that answers from the provided documents..." },
    { "type": "text", "text": "Examples of good answers: ...",
      "cache_control": { "type": "ephemeral" } }
  ],
  "messages": [
    { "role": "user", "content": [
      { "type": "text", "text": "<context>\n[1] Warranty period\n...\n</context>" },
      { "type": "text", "text": "<question>How long is the product warranty?</question>" }
    ] }
  ]
}

Everything before cache_control (system prompt + few-shot) is identical on every request, so from the second request on it is read from cache. messages changes with every question and is always billed at full price. If Bedrock returns ThrottlingException or ServiceUnavailableException, the request is retried after 500ms and then 1s; any other error is thrown immediately.

[5] Return the result. Claude returns a content array. The code keeps only type: "text" blocks and joins them, ignoring other block types. citations is the list of chunks put into the prompt, numbered to match [n] in the context, so the client can turn [1] in the answer into a link to source:

{
  "answer": "All products are covered by a 12-month warranty from the date of purchase [1].",
  "citations": [
    { "index": 1, "chunkId": "data/in/warranty.md#0", "source": "data/in/warranty.md", "title": "Warranty period" },
    { "index": 2, "chunkId": "data/in/warranty.md#1", "source": "data/in/warranty.md", "title": "Returns" }
  ],
  "usage": { "inputTokens": 180, "outputTokens": 25, "cacheWriteTokens": 0, "cacheReadTokens": 650 }
}

(The usage values above are illustrative.) Note that citations contains every chunk that was sent, not only the ones the model actually cited. The client should filter by the [n] numbers that appear in answer.

3.3 What makes answers accurate, and at which step

Accuracy doesn't come from one place. It accumulates across both flows:

Ingest [2], header injection: chunks carry their section heading, so questions phrased like the heading still match the body text.
Ingest [2], split by heading first, by length second: each chunk is about one idea, so its vector isn't "averaged out" across several topics.
Ingest [3] + Question [2a], correct input_type: documents and questions are projected into the same vector space.
Ingest [5], cleaning up leftovers: content deleted from a document can no longer leak into answers.
Question [2b], two branches + over-fetching 50: the dense branch catches paraphrases, the keyword branch catches codes and names; over-fetching keeps chunks that do well in both branches from falling off the list.
Question [2c], RRF: chunks found by both branches rise to the top.
Question [3], system prompt rules + a refusal few-shot: the model answers only from context, cites [n], and knows how to say "not found".

What these two flows don't do yet, covered in the "Upgrade path" section at the end: Unicode NFC normalization for both documents and questions, rewriting questions using conversation history (the API is currently stateless), a confidence threshold to skip the LLM when results are too weak, and checking in code that the [n] the model cites actually exist.

4. Tech stack and why

One design goal guided every technology choice: keep the architecture understandable.

Every component should be replaceable without rewriting the rest of the system.

NestJS 10 (TypeScript): modules + dependency injection make each layer replaceable on its own. Switching LLM providers is a one-line useClass change.
Amazon Bedrock: one API and one AWS bill for both embeddings and generation. Credentials follow your organization's existing IAM.
Cohere Embed Multilingual v3 (cohere.embed-multilingual-v3, 1024 dimensions): multilingual, which matters when documents mix a local language with English UI labels.
Claude on Bedrock: generates answers and supports prompt caching.
PostgreSQL 16 + pgvector: vector search (HNSW) and full-text search (GIN) in the same database. No extra service, no data sync between two systems.
Plain pg, no ORM: SQL lives next to the module that uses it, so reading a query tells you which index it uses.
@aws-sdk/client-bedrock-runtime: calls InvokeModel.
Jest, ESLint/Prettier, multi-stage Docker, GitHub Actions.

That's the entire runtime dependency list:

"dependencies": {
  "@aws-sdk/client-bedrock-runtime": "^3.696.0",
  "@nestjs/common": "^10.4.0",
  "@nestjs/core": "^10.4.0",
  "@nestjs/platform-express": "^10.4.0",
  "class-transformer": "^0.5.1",
  "class-validator": "^0.14.1",
  "dotenv": "^16.4.5",
  "pg": "^8.12.0",
  "reflect-metadata": "^0.2.2",
  "rxjs": "^7.8.1"
}

5. Prerequisites

5.1 Local environment

Node.js 20+
Docker + Docker Compose
An AWS account

5.2 Enable models on Amazon Bedrock

Open the Amazon Bedrock console and pick your region (this article uses ap-northeast-1).
Make sure your account has access to two models: Cohere Embed Multilingual v3 and a Claude model. The first time you use an Anthropic model, AWS may ask you to fill in a use-case form.
Open the Model catalog and note the exact model ID for Claude. Bedrock model IDs have the anthropic. prefix, for example anthropic.claude-opus-5-5. Some regions require calling through a cross-region inference profile; in that case use the profile ID the console shows.

5.3 Minimal IAM policy

The application only needs bedrock:InvokeModel:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["bedrock:InvokeModel"],
      "Resource": [
        "arn:aws:bedrock:*::foundation-model/*",
        "arn:aws:bedrock:*:*:inference-profile/*"
      ]
    }
  ]
}

In production, narrow Resource down to the exact ARNs of the two models you use.

About credentials: static access keys work, but it's better to leave them empty and let the AWS SDK pick up AWS_PROFILE, SSO, or the EC2/ECS IAM role. The code below supports both.

6. Step 1: Create the project

npm i -g @nestjs/cli
nest new rag-chatbot --package-manager npm --skip-git
cd rag-chatbot

npm i @aws-sdk/client-bedrock-runtime pg dotenv class-validator class-transformer
npm i -D @types/pg ts-node

Add these scripts to package.json:

"scripts": {
  "build": "nest build",
  "start:dev": "nest start --watch",
  "start:prod": "node dist/main",
  "db:init": "ts-node scripts/db-init.ts",
  "ingest": "ts-node scripts/ingest.ts",
  "search": "ts-node scripts/search.ts",
  "test": "jest",
  "lint": "eslint \"{src,scripts,test}/**/*.ts\""
}

7. Step 2: Config in one place

Every setting is read from env, with sensible defaults so it runs out of the box. No per-project profile files.

.env.example:

# ── App
PORT=8000

# ── PostgreSQL + pgvector
DB_HOST=127.0.0.1
DB_PORT=5432
DB_NAME=ragdb
DB_USER=postgres
DB_PASSWORD=postgres
# Must match the embedding model's dimensions and the vector(N) column in the schema.
# cohere.embed-multilingual-v3 → 1024. Changing it = re-run db:init + re-embed everything.
EMBEDDING_DIMENSIONS=1024

# ── AWS Bedrock
AWS_REGION=ap-northeast-1
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_SESSION_TOKEN=
# Leave the keys above empty to let the SDK use AWS_PROFILE / SSO / instance role.

BEDROCK_EMBEDDING_MODEL_ID=cohere.embed-multilingual-v3
BEDROCK_GENERATION_MODEL_ID=anthropic.claude-opus-5-5

# ── Ingest / chunking
INGEST_DIR=./data/in
CHUNK_MAX_CHARS=1200
CHUNK_OVERLAP_CHARS=120
EMBED_BATCH_SIZE=96

# ── Search
TOP_K=5            # number of chunks finally placed in the prompt
CANDIDATE_LIMIT=50 # candidates fetched by EACH branch before fusion
RRF_K=60           # RRF constant k, 60 is the value from the original paper

# ── Generation
MAX_TOKENS=2000
ENABLE_PROMPT_CACHE=true

src/config/config.ts (abridged; the full version has an AppConfig interface documenting each field):

const num = (v: string | undefined, fallback: number) =>
  v === undefined || v === '' ? fallback : Number(v);

export function loadConfig(): AppConfig {
  return {
    port: num(process.env.PORT, 8000),
    database: {
      host: process.env.DB_HOST || '127.0.0.1',
      port: num(process.env.DB_PORT, 5432),
      name: process.env.DB_NAME || 'ragdb',
      user: process.env.DB_USER || 'postgres',
      password: process.env.DB_PASSWORD || 'postgres',
      embeddingDimensions: num(process.env.EMBEDDING_DIMENSIONS, 1024),
    },
    bedrock: {
      region: process.env.AWS_REGION || 'ap-northeast-1',
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
      sessionToken: process.env.AWS_SESSION_TOKEN,
      embeddingModelId: process.env.BEDROCK_EMBEDDING_MODEL_ID || 'cohere.embed-multilingual-v3',
      generationModelId: process.env.BEDROCK_GENERATION_MODEL_ID || 'anthropic.claude-opus-5-5',
    },
    ingest: {
      dir: process.env.INGEST_DIR || './data/in',
      chunkMaxChars: num(process.env.CHUNK_MAX_CHARS, 1200),
      chunkOverlapChars: num(process.env.CHUNK_OVERLAP_CHARS, 120),
      embedBatchSize: num(process.env.EMBED_BATCH_SIZE, 96),
    },
    search: {
      topK: num(process.env.TOP_K, 5),
      candidateLimit: num(process.env.CANDIDATE_LIMIT, 50),
      rrfK: num(process.env.RRF_K, 60),
    },
    generation: {
      maxTokens: num(process.env.MAX_TOKENS, 2000),
      enableCache: process.env.ENABLE_PROMPT_CACHE !== 'false',
    },
  };
}

export const CONFIG = Symbol('CONFIG');

Register CONFIG as a global provider in AppConfigModule ({ provide: CONFIG, useFactory: loadConfig }) so every service can inject it with @Inject(CONFIG).

8. Step 3: PostgreSQL + pgvector

8.1 Docker Compose

name: rag-core

services:
  postgres:
    image: pgvector/pgvector:pg16   # pgvector 0.7+, needed for the HNSW index
    container_name: rag-postgres
    environment:
      POSTGRES_USER: ${DB_USER:-postgres}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres}
      POSTGRES_DB: ${DB_NAME:-ragdb}
    ports:
      - "${DB_PORT:-5432}:5432"
    volumes:
      - pgvector_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-postgres} -d ${DB_NAME:-ragdb}"]
      interval: 5s
      timeout: 5s
      retries: 10

volumes:
  pgvector_data:

name: rag-core is there for a reason: without it, Compose uses the folder name as the project name, and two checkouts with the same folder name on one machine will silently recreate each other's containers.

8.2 Schema: one table, three indexes

src/database/schema.sql:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE IF NOT EXISTS chunks (
  id           BIGSERIAL PRIMARY KEY,
  -- Logical key "<source>#<index>": re-ingesting the same file upserts the same row.
  chunk_id     TEXT NOT NULL UNIQUE,
  source       TEXT NOT NULL,
  chunk_index  INT  NOT NULL,
  title        TEXT,
  content      TEXT NOT NULL,
  metadata     JSONB NOT NULL DEFAULT '{}'::jsonb,
  embedding    vector(1024),

  -- Auto-generated full-text column, 'simple' dictionary.
  search_vector tsvector GENERATED ALWAYS AS (
    to_tsvector('simple', coalesce(title, '') || ' ' || content)
  ) STORED,

  created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Vector search: HNSW + cosine
CREATE INDEX IF NOT EXISTS chunks_embedding_idx
  ON chunks USING hnsw (embedding vector_cosine_ops);

-- Full-text search
CREATE INDEX IF NOT EXISTS chunks_fts_idx
  ON chunks USING gin (search_vector);

CREATE INDEX IF NOT EXISTS chunks_source_idx ON chunks (source);

Three decisions worth noting:

chunk_id is a unique logical key. Thanks to it, running ingest a second time updates rows (upsert) instead of duplicating them.
search_vector is a generated column. Postgres updates it whenever content changes, so application code never has to keep it in sync.
The 'simple' dictionary instead of 'english'. simple does no stemming and removes no language-specific stopwords. For languages like Vietnamese, the english config would chop word endings in nonsensical ways. simple is less sophisticated but never damages the data, and it keeps error codes, product names, and version numbers intact. Those are exactly what the keyword branch needs to catch.

8.3 Schema setup script

scripts/db-init.ts reads the SQL file, swaps in the vector dimension from EMBEDDING_DIMENSIONS, and runs it. Safe to run any number of times (every statement is IF NOT EXISTS):

const sql = await fs.readFile(path.join(__dirname, '../src/database/schema.sql'), 'utf8');
await client.connect();
await client.query(sql.replace('vector(1024)', `vector(${config.database.embeddingDimensions})`));
await client.end();

8.4 Database service: a thin wrapper around `pg.Pool`

@Injectable()
export class DatabaseService implements OnModuleDestroy {
  private readonly pool: Pool;

  constructor(@Inject(CONFIG) config: AppConfig) {
    this.pool = new Pool({
      host: config.database.host,
      port: config.database.port,
      database: config.database.name,
      user: config.database.user,
      password: config.database.password,
    });
  }

  async query<T extends QueryResultRow>(sql: string, params: unknown[] = []): Promise<T[]> {
    const result = await this.pool.query<T>(sql, params);
    return result.rows;
  }

  /** pgvector accepts a vector as the literal '[1,2,3]' cast to ::vector. */
  static toVector(values: number[]): string {
    return `[${values.join(',')}]`;
  }

  async onModuleDestroy(): Promise<void> {
    await this.pool.end();
  }
}

9. Step 4: The LLM layer, calling Bedrock

9.1 Interface first, implementation second

The rest of the codebase only knows this interface, not Bedrock:

export const LLM = Symbol('LLM');

export interface PromptBlock {
  text: string;
  /** true = block belongs to the static prefix and gets a cache_control marker. */
  cacheable?: boolean;
}

export interface GenerateRequest {
  system: PromptBlock[]; // persona, rules, few-shot: the cached part
  user: PromptBlock[];   // retrieved context + question: changes every turn
  maxTokens?: number;
}

export interface Usage {
  inputTokens: number;
  outputTokens: number;
  cacheWriteTokens: number;
  cacheReadTokens: number;
}

export interface LlmService {
  embedDocuments(texts: string[]): Promise<number[][]>;
  embedQuery(text: string): Promise<number[]>;
  generate(request: GenerateRequest): Promise<{ text: string; usage: Usage }>;
}

// src/llm/llm.module.ts: switching providers = changing this one useClass line
@Global()
@Module({
  providers: [{ provide: LLM, useClass: BedrockService }],
  exports: [LLM],
})
export class LlmModule {}

9.2 Client and retries

@Injectable()
export class BedrockService implements LlmService {
  private readonly client: BedrockRuntimeClient;

  constructor(@Inject(CONFIG) private readonly config: AppConfig) {
    const { region, accessKeyId, secretAccessKey, sessionToken } = config.bedrock;
    this.client = new BedrockRuntimeClient({
      region,
      // Don't pass credentials when there are no static keys,
      // so the SDK uses its own chain (AWS_PROFILE, SSO, instance role).
      ...(accessKeyId && secretAccessKey
        ? { credentials: { accessKeyId, secretAccessKey, sessionToken } }
        : {}),
    });
  }

  private async invoke(modelId: string, body: unknown): Promise<Record<string, any>> {
    let lastError: unknown;

    for (let attempt = 1; attempt <= 3; attempt++) {
      try {
        const response = await this.client.send(
          new InvokeModelCommand({
            modelId,
            contentType: 'application/json',
            accept: 'application/json',
            body: JSON.stringify(body),
          }),
        );
        return JSON.parse(new TextDecoder().decode(response.body));
      } catch (err) {
        lastError = err;
        const name = (err as { name?: string }).name ?? '';
        const retryable = name === 'ThrottlingException' || name === 'ServiceUnavailableException';
        if (!retryable || attempt === 3) break;
        await new Promise((r) => setTimeout(r, 2 ** attempt * 250)); // 500ms, 1s
      }
    }
    throw lastError;
  }

Only transient errors (throttling, service unavailable) are retried, with exponential backoff. A ValidationException or AccessDeniedException won't change no matter how many times you retry, so those are thrown immediately.

9.3 Embeddings with Cohere: don't forget `input_type`

  private async embed(
    texts: string[],
    inputType: 'search_document' | 'search_query',
  ): Promise<number[][]> {
    const out: number[][] = [];
    const batchSize = Math.min(this.config.ingest.embedBatchSize, 96); // Cohere: max 96 texts per call

    for (let i = 0; i < texts.length; i += batchSize) {
      const body = await this.invoke(this.config.bedrock.embeddingModelId, {
        texts: texts.slice(i, i + batchSize),
        input_type: inputType,
        truncate: 'END',
      });
      out.push(...(body.embeddings as number[][]));
    }
    return out;
  }

  async embedDocuments(texts: string[]) {
    return texts.length ? this.embed(texts, 'search_document') : [];
  }

  async embedQuery(text: string) {
    const [embedding] = await this.embed([text], 'search_query');
    return embedding;
  }

input_type directly affects results. Cohere only projects documents and queries into the same space when each side declares its role correctly: documents are search_document, questions are search_query. Using the same value for both still runs and reports no error; recall just drops a few percent, and hardly anyone notices.

9.4 Generating answers with Claude + cache markers

The body sent to Claude via InvokeModel follows the Anthropic Messages format, with anthropic_version: 'bedrock-2023-05-31':

type TextBlock = { type: 'text'; text: string; cache_control?: { type: 'ephemeral' } };

  /** Put cache_control on the LAST block of the static prefix. */
  private toBlocks(blocks: PromptBlock[]): TextBlock[] {
    const out: TextBlock[] = blocks
      .filter((b) => b.text.trim())
      .map((b) => ({ type: 'text' as const, text: b.text }));

    if (!this.config.generation.enableCache) return out;

    const lastCacheable = blocks.reduce(
      (acc, b, i) => (b.cacheable && b.text.trim() ? i : acc),
      -1,
    );
    if (lastCacheable >= 0 && out[lastCacheable]) {
      out[lastCacheable].cache_control = { type: 'ephemeral' };
    }
    return out;
  }

  async generate(request: GenerateRequest) {
    const body = await this.invoke(this.config.bedrock.generationModelId, {
      anthropic_version: 'bedrock-2023-05-31',
      max_tokens: request.maxTokens ?? this.config.generation.maxTokens,
      system: this.toBlocks(request.system),
      messages: [{ role: 'user', content: this.toBlocks(request.user) }],
    });

    // Keep only 'text' blocks. Newer Claude models may return other block
    // types (e.g. thinking); without this filter they leak into the answer.
    const text = ((body.content ?? []) as Array<{ type: string; text?: string }>)
      .filter((block) => block.type === 'text')
      .map((block) => block.text ?? '')
      .join('')
      .trim();

    const usage = body.usage ?? {};
    return {
      text,
      usage: {
        inputTokens: usage.input_tokens ?? 0,
        outputTokens: usage.output_tokens ?? 0,
        cacheWriteTokens: usage.cache_creation_input_tokens ?? 0,
        cacheReadTokens: usage.cache_read_input_tokens ?? 0,
      },
    };
  }
}

Why the cache marker goes on the last block of the static part: see Step 7.

10. Step 5: The ingest pipeline

The pipeline has four classes, each with one job:

LoaderService  ──►  ChunkerService  ──►  LlmService.embedDocuments  ──►  ChunkRepository
(read sources)      (chunking)            (batches of 96)                 (upsert + clean up)
                         ▲
                  IngestService only orchestrates, no logic of its own

10.1 Loader: the only place that knows where data comes from

export interface RawDocument {
  source: string;   // file path, URL, CMS id...
  title?: string;
  text: string;
  metadata?: Record<string, unknown>;
}

LoaderService scans the folder recursively:

.md / .txt: reads the whole file. Uses the first H1 as the title, or the file name if there is none.
.json: an array of objects with a text field (optional source, title, metadata), handy for data exported from a CMS or database.

async loadTextFile(file: string): Promise<RawDocument> {
  const text = await fs.readFile(file, 'utf8');
  return {
    source: file,
    title: /^#\s+(.+)$/m.exec(text)?.[1]?.trim() ?? path.basename(file),
    text,
  };
}

To add S3, Confluence, Notion and so on, add one method that returns RawDocument[]. Chunking and embedding stay the same.

10.2 Chunker: two-level splitting + header injection

This step has a big impact on quality but is often done carelessly.

Chunks too large: one chunk covers three topics, its embedding is "averaged out" and represents none of them.
Fixed character cuts: sentences are cut mid-way, leaving meaningless fragments.
Chunks lose context on their own: the passage "If the code cannot be read, enter the product code manually." contains no word about "inventory check" or "scanner", so a real user question will match it only weakly.

The codebase's strategy:

Level 1: split by markdown heading. Headings are natural semantic boundaries, so use them first.
Level 2: any section still longer than CHUNK_MAX_CHARS is split further by paragraph (\n\n), then by sentence, with an overlap between consecutive chunks. A hard cut only happens when a single sentence is longer than the limit.
Header injection: prepend the section heading to the chunk content before embedding. Each chunk is embedded and full-text indexed together with its context.

@Injectable()
export class ChunkerService {
  constructor(@Inject(CONFIG) private readonly config: AppConfig) {}

  chunk(doc: RawDocument): Chunk[] {
    const { chunkMaxChars, chunkOverlapChars } = this.config.ingest;
    const pieces: Array<{ heading?: string; text: string }> = [];

    for (const section of this.splitByHeading(doc.text)) {
      for (const part of this.splitLongText(section.text, chunkMaxChars, chunkOverlapChars)) {
        pieces.push({ heading: section.heading, text: part });
      }
    }

    return pieces.map((piece, index) => ({
      chunkId: `${doc.source}#${index}`,
      source: doc.source,
      chunkIndex: index,
      title: piece.heading ?? doc.title,
      // Header injection: the heading goes into the embedding and FTS with the content.
      content: piece.heading ? `${piece.heading}\n\n${piece.text}` : piece.text,
      metadata: { ...(doc.metadata ?? {}), heading: piece.heading ?? null },
    }));
  }

  private splitByHeading(text: string) {
    const sections: Array<{ heading?: string; text: string[] }> = [];
    let current: { heading?: string; text: string[] } = { text: [] };

    for (const line of text.split('\n')) {
      const heading = /^(#{1,6})\s+(.+)$/.exec(line);
      if (heading) {
        if (current.text.join('').trim() || current.heading) sections.push(current);
        current = { heading: heading[2].trim(), text: [] };
      } else {
        current.text.push(line);
      }
    }
    sections.push(current);

    return sections
      .map((s) => ({ heading: s.heading, text: s.text.join('\n').trim() }))
      .filter((s) => s.text || s.heading);
  }

  private splitLongText(text: string, maxChars: number, overlapChars: number): string[] {
    if (text.length <= maxChars) return text ? [text] : [];

    // Prefer paragraph boundaries, then sentences (Japanese punctuation included).
    const units = text
      .split(/\n{2,}/)
      .flatMap((p) => (p.length <= maxChars ? [p] : p.split(/(?<=[.!?。？！])\s+/)))
      .filter((u) => u.trim());

    const out: string[] = [];
    let buffer = '';

    const flush = () => {
      if (!buffer.trim()) return;
      out.push(buffer.trim());
      buffer = overlapChars > 0 ? buffer.slice(-overlapChars) : ''; // overlap
    };

    for (const unit of units) {
      if (buffer && buffer.length + unit.length + 2 > maxChars) flush();

      if (unit.length > maxChars) {
        // The smallest unit is still too long, no boundary left to respect.
        for (let i = 0; i < unit.length; i += maxChars - overlapChars) {
          out.push(unit.slice(i, i + maxChars).trim());
        }
        buffer = '';
        continue;
      }
      buffer = buffer ? `${buffer}\n\n${unit}` : unit;
    }

    if (buffer.trim()) out.push(buffer.trim());
    return out.filter(Boolean);
  }
}

If you can make only one chunking improvement, make it header injection: it costs almost nothing (string concatenation) and improves both the vector branch and the keyword branch.

A note on size: CHUNK_MAX_CHARS=1200 counts characters. For the same content, Vietnamese takes roughly 1.5x as many tokens as English, and Japanese roughly 3x. For Japanese documents, lower this limit so chunks stay within the embedding model's limit (anything beyond it is silently cut by truncate: 'END').

10.3 Repository: upsert and clean up old chunks

async upsert(chunk: Chunk, embedding: number[]): Promise<void> {
  await this.db.query(
    `
    INSERT INTO chunks (chunk_id, source, chunk_index, title, content, metadata, embedding)
    VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7::vector)
    ON CONFLICT (chunk_id) DO UPDATE SET
      source = EXCLUDED.source, chunk_index = EXCLUDED.chunk_index,
      title = EXCLUDED.title, content = EXCLUDED.content,
      metadata = EXCLUDED.metadata, embedding = EXCLUDED.embedding,
      updated_at = NOW()
    `,
    [chunk.chunkId, chunk.source, chunk.chunkIndex, chunk.title ?? null,
     chunk.content, JSON.stringify(chunk.metadata), DatabaseService.toVector(embedding)],
  );
}

async deleteStale(source: string, keepChunkIds: string[]): Promise<number> {
  const rows = await this.db.query<{ chunk_id: string }>(
    `DELETE FROM chunks WHERE source = $1 AND chunk_id <> ALL($2::text[]) RETURNING chunk_id`,
    [source, keepChunkIds],
  );
  return rows.length;
}

Upsert alone is not enough for safe re-ingestion. Say a document used to have 10 chunks and the new version has 6. Upsert updates the first 6 rows, but the last 4 stay there with their old embeddings and keep being retrieved. The chatbot ends up answering from content that no longer exists.

deleteStale deletes by exclusion after the upsert finishes, instead of "delete everything, then write again". That way search never sees a window where the document is missing.

10.4 IngestService: orchestration only

Abridged version (in the repo, embed + upsert live in a separate embedAndStore method):

async ingestDocuments(docs: RawDocument[]): Promise<IngestReport> {
  const report = { documents: 0, chunks: 0, deleted: 0 };

  for (const doc of docs) {
    const chunks = this.chunker.chunk(doc);
    if (!chunks.length) continue;

    // Embed in batches, then upsert each chunk
    for (let i = 0; i < chunks.length; i += this.config.ingest.embedBatchSize) {
      const batch = chunks.slice(i, i + this.config.ingest.embedBatchSize);
      const embeddings = await this.llm.embedDocuments(batch.map((c) => c.content));
      for (const [j, chunk] of batch.entries()) await this.repo.upsert(chunk, embeddings[j]);
    }

    report.deleted += await this.repo.deleteStale(doc.source, chunks.map((c) => c.chunkId));
    report.documents += 1;
    report.chunks += chunks.length;
  }
  return report;
}

The CLI script scripts/ingest.ts boots Nest as an application context (no HTTP server) and calls the service:

const app = await NestFactory.createApplicationContext(AppModule, { logger: ['log', 'warn', 'error'] });
const report = await app.get(IngestService).ingestDirectory(process.argv[2]);
console.log(`done: ${report.documents} documents, ${report.chunks} chunks, ${report.deleted} stale chunks deleted`);
await app.close();

11. Step 6: Hybrid search (vector + full-text) merged with RRF

11.1 Why you need both branches

Each branch fails exactly where the other is strong:

Dense (vector) handles synonyms and paraphrasing well. The user asks "scanner doesn't recognize the code", the doc says "barcode cannot be read", and the vectors still match. But for error codes, product codes, and proper names, vectors are noisy: E-4021 and E-4012 are almost identical in embedding space.
Keyword (full-text) is the opposite: it matches E-4021 exactly, but if no token overlaps, the right chunk never shows up.

You combine the two because they cover for each other, not because more branches are always better.

11.2 The dense branch

async search(queryVector: number[], limit: number): Promise<Candidate[]> {
  const rows = await this.db.query<any>(
    `
    SELECT chunk_id AS "chunkId", source, title, content, metadata,
           1 - (embedding <=> $1::vector) AS score
    FROM chunks
    WHERE embedding IS NOT NULL
    ORDER BY embedding <=> $1::vector
    LIMIT $2::int
    `,
    [DatabaseService.toVector(queryVector), limit],
  );
  return rows.map((r) => ({ ...r, score: Number(r.score) }));
}

<=> is pgvector's cosine distance operator, so 1 - distance is cosine similarity. The query deliberately has just ORDER BY embedding <=> $1 LIMIT n, because that is the shape the planner needs to use the HNSW index. If you add filters, check again with EXPLAIN, since it is very easy to turn this into a full table scan.

11.3 The keyword branch

async search(queryText: string, limit: number): Promise<Candidate[]> {
  const rows = await this.db.query<any>(
    `
    WITH tsq AS (SELECT websearch_to_tsquery('simple', $1) AS q)
    SELECT c.chunk_id AS "chunkId", c.source, c.title, c.content, c.metadata,
           ts_rank(c.search_vector, tsq.q) AS score
    FROM chunks c, tsq
    WHERE c.search_vector @@ tsq.q
    ORDER BY score DESC
    LIMIT $2::int
    `,
    [queryText, limit],
  );
  return rows.map((r) => ({ ...r, score: Number(r.score) }));
}

websearch_to_tsquery accepts the syntax users already know ("exact phrase", a OR b, -exclude) and never throws a syntax error. Passing it as a bind parameter is safe enough; no extra escaping needed.
ts_rank is not BM25 (Postgres has no built-in BM25). With RRF that doesn't matter, as you'll see right below.

11.4 Merging two lists: why not add the scores

The "obvious" approach is score = 0.7 × cosine + 0.3 × ts_rank. It's wrong:

Cosine lies in [0, 1], while ts_rank is typically around 0.0x. Add them directly and the keyword branch contributes almost nothing. You think hybrid search is running, but it is really just vector search.
Switch to real BM25 (scores 0-20, unbounded) and the error flips the other way: the keyword branch dominates.

Reciprocal Rank Fusion (RRF) throws away the raw scores and uses only ranks:

RRF(d) = Σ        1 / (k + rank_branch(d))        k = 60
        branch

A worked example:

Dense:   1. chunk_A   2. chunk_B   3. chunk_C
Keyword: 1. chunk_C   2. chunk_A   3. chunk_D

chunk_A: 1/61 + 1/62 = 0.03252   ← rank 1
chunk_C: 1/63 + 1/61 = 0.03226   ← rank 2
chunk_B: 1/62        = 0.01613
chunk_D: 1/63        = 0.01587

chunk_A wins without being first in either branch, because it ranks well in both. Agreement between two independent methods is a stronger signal than a high score from one.

The constant k flattens the curve: with k = 60, the gap between rank 1 and rank 2 is only about 1.6%, so appearing in several lists matters more than the exact rank.

export function reciprocalRankFusion(
  branches: { dense: Candidate[]; keyword: Candidate[] },
  k = 60,
  topK = 5,
): SearchResult[] {
  const byId = new Map<string, SearchResult>();

  const ingest = (candidates: Candidate[], branch: 'denseRank' | 'keywordRank') => {
    candidates.forEach((candidate, i) => {
      const rank = i + 1;
      const existing = byId.get(candidate.chunkId);

      if (existing) {
        existing.rrfScore += 1 / (k + rank);
        existing[branch] = rank;
        return;
      }

      const { score: _ignored, ...rest } = candidate;
      byId.set(candidate.chunkId, {
        ...rest,
        rrfScore: 1 / (k + rank),
        denseRank: null,
        keywordRank: null,
        [branch]: rank,
      } as SearchResult);
    });
  };

  ingest(branches.dense, 'denseRank');
  ingest(branches.keyword, 'keywordRank');

  return [...byId.values()].sort((a, b) => b.rrfScore - a.rrfScore).slice(0, topK);
}

The result keeps each chunk's denseRank and keywordRank. When debugging, one look tells you whether a result came from semantics, keywords, or both.

11.5 SearchService: two parallel queries, over-fetching

async hybridSearch(query: string, options: SearchOptions = {}): Promise<SearchResult[]> {
  const topK = options.topK ?? this.config.search.topK;            // 5
  const limit = options.candidateLimit ?? this.config.search.candidateLimit; // 50

  const queryVector = await this.llm.embedQuery(query);

  const [dense, keyword] = await Promise.all([
    this.dense.search(queryVector, limit),
    this.keyword.search(query, limit),
  ]);

  return reciprocalRankFusion({ dense, keyword }, this.config.search.rrfK, topK);
}

Two details decide the quality:

The two branches are two independent queries. A single SQL query that computes both cosine and ts_rank for every row and then sorts still ranks correctly, but no index is used and Postgres has to scan the whole table. Split apart, each branch uses its own index (HNSW / GIN).
Over-fetching. You need 5 final results, yet each branch still fetches 50. The best chunk after fusion is often not first in either branch; it ranks reasonably well in both. Fetch too few and it falls off the lists before it can be merged.

Adding a third branch (multi-query, or real BM25) is just one more array passed to RRF. No score normalization, no weight tuning.

12. Step 7: Designing the prompt so caching works

12.1 System prompt: the static part

The repository ships this prompt in Vietnamese (with <ngu_canh> / <cau_hoi> tags); the English equivalent is shown here.

export const SYSTEM_PROMPT = `You are a Q&A assistant that answers from the provided documents.

Answering rules:
- Answer only from the CONTEXT provided. Do not guess, and do not use knowledge
  outside the context.
- If the context is not enough to answer, say plainly that the information was
  not found. A wrong answer is more dangerous than "I don't know".
- Cite sources with [number] matching the position of the context passage you
  relied on.
- Answer in the same language as the question.
- Be concise and get straight to the point. Do not start with "Based on the documents...".`;

Three rules for this file:

No dynamic values here (dates, user names, session ids). Changing one byte invalidates the cache for everything after it.
No conditional string building (if (x) prompt += ...). Every combination of conditions is a different prefix, i.e. a different cache entry.
Keep it stable across deploys. Editing this file invalidates every existing cache entry.

The rule "answer in the same language as the question" is essential: when documents are in English and the user asks in another language, without this rule the model tends to answer in the language of the context.

12.2 Few-shot: teaching both how to answer and how to decline

export const EXAMPLES: Example[] = [
  {
    question: 'How long is the product warranty?',
    context:
      '[1] Warranty policy\nAll products are covered by a 12-month warranty from the date of purchase.\n\n' +
      '[2] Returns\nDefective products can be exchanged within the first 7 days.',
    answer: 'Products are covered by a 12-month warranty from the date of purchase [1].',
  },
  {
    // Negative example: there is context, but no passage answers the question.
    question: 'Does the company have a branch in Da Nang?',
    context: '[1] Warranty policy\nAll products are covered by a 12-month warranty from the date of purchase.',
    answer: 'The available documents do not mention any branch information.',
  },
];

Examples are the strongest signal in a prompt: the [1] citation format is easy to misread when described in words, but obvious from a single example.

The refusal example matters as much as the answer example. If every example is a successful answer, the model learns that there must always be an answer. Example #2 teaches it to decline when the context is irrelevant.

12.3 PromptBuilder: ordering by decreasing stability

Prompt caching is a prefix match: the system hashes the prompt from the start up to the cache_control marker. Changing one byte at position N invalidates the cache for everything from N onward. Consequence: content must be ordered from most stable to most volatile.

┌─ system ─────────────────────────────────── CACHED ─┐
│ [0] SYSTEM_PROMPT   persona + rules, never changes  │
│ [1] EXAMPLES        few-shot, never changes         │ ← cache_control
└─────────────────────────────────────────────────────┘
┌─ user ───────────────────────────────── NOT CACHED ─┐
│ [0] CONTEXT         retrieved chunks, per question  │
│ [1] QUESTION        the question, every turn        │
└─────────────────────────────────────────────────────┘

@Injectable()
export class PromptBuilder {
  build(question: string, results: SearchResult[]): GenerateRequest {
    const system: PromptBlock[] = [
      { text: SYSTEM_PROMPT, cacheable: true },
      { text: renderExamples(), cacheable: true },
    ];

    const user: PromptBlock[] = [
      { text: `<context>\n${this.renderContext(results)}\n</context>` },
      { text: `<question>${question}</question>` },
    ];

    return { system, user };
  }

  /** Number from 1 so the model cites [1], [2]... consistent with the few-shot. */
  private renderContext(results: SearchResult[]): string {
    if (!results.length) return '(no relevant documents found)';
    return results
      .map((r, i) => `[${i + 1}] ${r.title ? `${r.title}\n` : ''}${r.content}`)
      .join('\n\n');
  }
}

Why is the retrieved context NOT in the cached region? Each question retrieves a different set of chunks. Put it before the marker and every request creates a new cache entry that no later request can read. You end up paying the cache write premium (about 1.25x the input price) without ever getting the cache read price (about 0.1x or lower, depending on the model).

Cache economics: with a 5-minute TTL, you break even on the second request (1.25 + 0.1 = 1.35 versus 2x without caching). A chatbot with a steady stream of users is the ideal case.

Minimum length: a prefix shorter than the model's minimum cacheable length is not cached, and no error tells you so. The minimum varies by model (from a few hundred to a few thousand tokens). The system prompt + few-shot above is fairly short, so check cacheWriteTokens on the first request. If it is 0, the prefix is too short: add business rules, add few-shot examples, or accept running without cache.

Why <context> and <question> tags: XML tags mark a clear boundary between data and question, which helps the model avoid treating document content as instructions.

13. Step 8: Chat service and HTTP API

13.1 ChatService: one RAG loop

@Injectable()
export class ChatService {
  constructor(
    @Inject(LLM) private readonly llm: LlmService,
    private readonly search: SearchService,
    private readonly prompt: PromptBuilder,
  ) {}

  async ask(question: string): Promise<ChatResponse> {
    const results = await this.search.hybridSearch(question);    // 1. retrieve
    const request = this.prompt.build(question, results);        // 2. build prompt
    const { text, usage } = await this.llm.generate(request);    // 3. generate

    return {
      answer: text,
      citations: results.map((r, i) => ({
        index: i + 1,            // matches [n] in the answer
        chunkId: r.chunkId,
        source: r.source,
        title: r.title,
      })),
      usage,
    };
  }
}

13.2 Controller

class AskDto {
  @IsString()
  @IsNotEmpty()
  @MaxLength(2000)
  question!: string;
}

@Controller('api')
export class ChatController {
  constructor(
    private readonly chat: ChatService,
    private readonly search: SearchService,
    private readonly repo: ChunkRepository,
    private readonly db: DatabaseService,
  ) {}

  @Get('health')
  async health() {
    return { status: (await this.db.ping()) ? 'ok' : 'degraded' };
  }

  @Post('chat')
  ask(@Body() dto: AskDto) {
    return this.chat.ask(dto.question);
  }

  /** Inspect retrieval results directly, without the LLM. Useful when tuning. */
  @Get('search')
  search_(@Query('q') q: string, @Query('mode') mode?: string) {
    if (mode === 'dense') return this.search.denseSearch(q);
    if (mode === 'keyword') return this.search.keywordSearch(q);
    return this.search.hybridSearch(q);
  }

  @Get('stats')
  stats() {
    return this.repo.stats();
  }
}

Besides validation, @MaxLength(2000) stops users from sending huge questions that inflate token costs.

13.3 `main.ts`

import 'dotenv/config';

async function bootstrap(): Promise<void> {
  const app = await NestFactory.create(AppModule);
  const config = app.get<AppConfig>(CONFIG);

  app.useGlobalPipes(
    new ValidationPipe({ whitelist: true, forbidNonWhitelisted: true, transform: true }),
  );
  // Let the pool close cleanly on SIGTERM instead of cutting off running queries.
  app.enableShutdownHooks();

  await app.listen(config.port);
}
bootstrap();

14. Step 9: End-to-end test run

cp .env.example .env      # fill in AWS credentials (or use AWS_PROFILE)
docker compose up -d      # Postgres + pgvector
npm install
npm run db:init           # create extension, table, indexes

Put a few markdown files in data/in/, for example data/in/warranty.md:

# Warranty Policy

## Warranty period
All products are covered by a warranty of 12 months from the date of purchase.

## Returns
Products with a manufacturing defect can be exchanged within the first 7 days.

npm run ingest
npm run start:dev

Check retrieval before spending generation tokens. Make this a habit from day one when tuning RAG:

npm run search -- "warranty 12 months"            # hybrid, with per-branch ranks
npm run search -- "warranty 12 months" dense      # vector only
npm run search -- "warranty 12 months" keyword    # full-text only

Hybrid output prints rrf=... dense=1 keyword=2 <source> for each chunk, so you can see at a glance which branch a chunk came from.

Ask the chatbot:

curl -s -X POST http://localhost:8000/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"question": "How long is the product warranty?"}'

The response looks like this (illustrative values):

{
  "answer": "All products are covered by a 12-month warranty from the date of purchase [1].",
  "citations": [
    { "index": 1, "chunkId": "data/in/warranty.md#0", "source": "data/in/warranty.md", "title": "Warranty period" },
    { "index": 2, "chunkId": "data/in/warranty.md#1", "source": "data/in/warranty.md", "title": "Returns" }
  ],
  "usage": {
    "inputTokens": 180,
    "outputTokens": 25,
    "cacheWriteTokens": 0,
    "cacheReadTokens": 650
  }
}

Send the same question twice: the first request has cacheWriteTokens > 0, the second has cacheReadTokens > 0. If cacheReadTokens stays 0 across many requests, something is breaking the cache. It's usually a dynamic value that slipped into the system prompt, or a prefix below the minimum length.

Other endpoints:

GET /api/search?q=...&mode=hybrid|dense|keyword: inspect retrieval results.
GET /api/stats: chunk count per source.
GET /api/health: health check (used by Docker).

15. Step 10: Tests, production Docker, and CI

15.1 Unit tests for the pure logic

The chunker and RRF are pure functions, so their tests need no database and no AWS:

it('a chunk ranked well in both branches beats a chunk ranked first in only one', () => {
  const results = reciprocalRankFusion(
    {
      dense: [candidate('dense-only'), candidate('both')],
      keyword: [candidate('keyword-only'), candidate('both')],
    },
    60,
    3,
  );
  expect(results[0].chunkId).toBe('both');
});

it('splits by heading and prepends the heading to chunk content', () => {
  const chunks = makeChunker(1000).chunk({
    source: 'doc.md',
    text: '# Title\n\nIntro.\n\n## Part A\n\nContent A.\n\n## Part B\n\nContent B.',
  });
  expect(chunks).toHaveLength(3);
  expect(chunks[1].title).toBe('Part A');
  expect(chunks[1].content).toContain('Part A');
});

15.2 Multi-stage Dockerfile

FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY tsconfig.json tsconfig.build.json nest-cli.json ./
COPY src ./src
RUN npm run build
RUN npm prune --omit=dev

FROM node:20-alpine AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder --chown=node:node /app/node_modules ./node_modules
COPY --from=builder --chown=node:node /app/dist ./dist
COPY --chown=node:node package.json ./
USER node
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
  CMD node -e "require('http').get('http://127.0.0.1:'+(process.env.PORT||8000)+'/api/health',r=>process.exit(r.statusCode===200?0:1)).on('error',()=>process.exit(1))"
CMD ["node", "dist/main"]

The npm ci layer is separate so the Docker cache survives source edits.
The runtime runs as the node user, not root.
The healthcheck uses Node's built-in http, so there's no need to install curl.

15.3 Production Compose: fail fast on missing variables

docker-compose.prod.yml layers on top of the dev file, adds the app service, hides the Postgres port, and makes every sensitive value mandatory:

services:
  postgres:
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required}
    ports: !reset []   # Postgres not exposed outside the network

  app:
    build: .
    environment:
      DB_HOST: postgres
      DB_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required}
      BEDROCK_GENERATION_MODEL_ID: ${BEDROCK_GENERATION_MODEL_ID}
      # ...
    depends_on:
      postgres:
        condition: service_healthy

docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

The ${VAR:?message} syntax makes the deploy fail immediately when a variable is missing, instead of silently running with the default postgres password.

15.4 CI with GitHub Actions

name: CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm test
      - run: npm run build

16. Production Pitfalls That Quietly Destroy RAG Quality

None of these throw an exception. The chatbot just answers worse, and the cause is very hard to track down.

Wrong input_type when embedding. Using search_document for questions → recall drops a few percent, and no log tells you.
Mismatched vector dimensions. Switching embedding models without updating EMBEDDING_DIMENSIONS and vector(N) → inserts fail (the lucky case, at least there's an error). Switching to a model with the same dimensions without re-embedding the corpus → old and new vectors share one index, results are meaningless, and there is no error at all.
Unicode NFC/NFD in accented languages such as Vietnamese. The character ề has two valid encodings that look identical on screen but are not equal to a computer. Documents copied from PDFs or created on macOS are often NFD → full-text search doesn't match. Fix: text.normalize('NFC') at both ingest time and query time, inside one shared function.
Full-text tokenizers for Vietnamese/Japanese. Postgres splits on whitespace, so the Vietnamese word "kiểm kê" (inventory check) becomes two tokens kiểm + kê, and also matches documents containing "kê khai" (declaration). Japanese has no spaces at all → the keyword branch is nearly useless; you need word segmentation in the application layer (MeCab, Sudachi) or an extension such as PGroonga.
Adding un-normalized scores from the two branches. Hybrid "runs" but only one branch actually matters. The test: if hybrid isn't better than the best single branch on its own, the problem is the fusion step; don't go tuning weights.
Prompt cache not hitting. A dynamic value slipped into the system prompt, or the prefix is below the minimum length. Always watch cacheReadTokens.
Re-ingest without cleaning old chunks. The chatbot answers from content that has been removed from the documents.

17. From here to production: the upgrade path

By this point, the system is already capable of answering questions over a document collection.

The next challenge is not functionality.

The next challenge is reliability.

Most production RAG systems evolve through a series of small improvements rather than a single architectural rewrite.

The following upgrades are listed roughly in order of value delivered relative to implementation effort.

The codebase above is Advanced RAG at its minimum: hybrid search + RRF + header injection + prompt caching. When you put it in front of real customers, these are the next steps, ordered by benefit per unit of effort:

A golden set before any optimization. 30-50 (question, correct chunk) pairs taken from real logs, not questions you made up (made-up questions are always easier than reality). Measure Recall@5 and Recall@50. Without measurement, every "improvement" is a hunch.
Rewrite questions using conversation history. On turn 2 the user asks "what if it can't read it?". Searching with that sentence as-is finds almost nothing. One LLM call to rewrite it into a standalone search query is the biggest improvement for multi-turn chatbots.
Confidence threshold + escalation path. Retrieval always returns k results even when the corpus has no answer, and that is the most common source of hallucination. Note that the RRF score cannot be used as a threshold (it only reflects rank); keep the raw cosine separately to measure confidence.
Verify citations in code. Ask the model to return JSON with citations as a list of chunk_id, then check every id is in the set you just sent. A few lines of code that catch most blatant fabrications.
Output guardrails. A layer that runs after the LLM answers and before the customer sees it: check for PII, unknown URLs, out-of-scope topics, length. Many systems go to production without this layer.
Handle stop_reason. Before reading content, check stop_reason (max_tokens = the answer was truncated; some newer models may return refusal). Return an appropriate message instead of an empty or half-finished string.
Stream answers to the client (InvokeModelWithResponseStreamCommand) to reduce perceived waiting time.
Parent-child chunking for summary questions like "what are the steps of this process".
Reranker: only once Recall@50 is good but the top-5 ordering is still messy. This step adds the most latency.
Observability: log latency per stage (embed / search / generate), tokens, and cache hit rate per request.
Loosen the keyword branch query. websearch_to_tsquery joins every word with AND, so long questions match almost nothing (see section 3.2). Extract a few key terms (codes, names, nouns) and join them with OR before searching, and the keyword branch will contribute much more.

For multi-tenant setups (several businesses on shared infrastructure), filtering by tenant_id from the schema up is a matter of right/wrong, not quality. Without it, data leaks between customers.

18. Wrapping Up

After building several generations of RAG systems, one lesson keeps repeating itself:

RAG is fundamentally a retrieval problem.

Most production issues are not caused by the language model.

They come from:

poor chunking
weak retrieval
stale documents
missing observability
broken citations
unreliable synchronization

The architecture in this article intentionally avoids unnecessary complexity.

There is no external vector database.

No orchestration framework.

No hidden abstraction layers.

Just PostgreSQL, pgvector, Amazon Bedrock, Hybrid Search, and a clear separation between indexing, retrieval, and generation.

For teams involved in application software development, that simplicity is often an advantage rather than a limitation. A system that can be understood, measured, and debugged is significantly easier to improve than one hidden behind multiple abstraction layers.

A system that can be understood, measured, and debugged is much easier to improve than one that depends on layers of abstraction.

If you're building a RAG chatbot today, invest in retrieval first.

Everything else becomes easier afterwards.

What Happens When an OAG System Goes Through a Real Enterprise Knowledge Workflow?

Le Trung Hieu — Thu, 15 Oct 2026 07:52:49 +0000

How Gnosis applies Ontology-Augmented Generation (OAG) to transform enterprise documents into governed knowledge, evidence-backed answers, and AI outputs that can be validated instead of merely trusted.

Introduction

When we started building Gnosis during Relipa Hackathon 2026, the goal was not to build another chatbot.

The problem was more practical.

Enterprise knowledge was scattered across PDF contracts, Word specifications, Excel trackers, meeting records, and screenshots. Information existed, but answering a seemingly simple question often required searching multiple documents, understanding relationships between entities, and verifying whether the answer was actually supported by evidence.

Most Retrieval-Augmented Generation (RAG) systems improve access to information. But we wanted to explore a different question:

What happens when AI-generated outputs must be governed by ontology constraints, evidence requirements, and deterministic validation before they are allowed to influence a business decision?

That question led to Gnosis, a document intelligence project inspired by the principles of Ontology-Augmented Generation (OAG).

At Relipa, the project was not intended to become a generic AI assistant.

Instead, it was designed as an exploration of how enterprise knowledge could be transformed into a governed knowledge system where every answer remains connected to evidence, ontology constraints, and deterministic validation.

For organizations delivering AI development services, this distinction becomes increasingly important when moving from proof-of-concept demonstrations toward production systems. In many enterprise environments, trust, traceability, and governance ultimately matter as much as answer quality.

Rather than focusing only on retrieval performance, Gnosis explored how knowledge extraction, graph construction, validation, retrieval, and generation could operate within a single governed workflow.

The core principle remained simple throughout the project:

The LLM proposes; policy and deterministic code decide which proposals go forward.

This article focuses on how Gnosis applies that principle throughout the knowledge lifecycle.

Building Gnosis: From Documents to Governed Knowledge

The Starting Point

Enterprise documents rarely arrive as clean, structured data.

Contracts exist as PDFs. Requirements live in Word files. Operational data often appears in spreadsheets. Important information may only exist inside screenshots or embedded images.

Before retrieval, reasoning, or question answering can happen, the system must first transform those artifacts into a form that can be governed consistently.

Gnosis treats this as a knowledge-building problem rather than a search problem.

flowchart LR
DOC[Documents] --> EXT[Constrained Extraction]
EXT --> KG[Knowledge Graph]
KG --> RET[Retrieval]
RET --> VAL[Validation]
VAL --> GEN[Generation]
GEN --> OUT[Answer with Citations]

Each stage introduces a new control point where deterministic logic can evaluate what the model proposes.

Rather than viewing retrieval and generation as isolated capabilities, the project explores how they can operate inside a broader framework where evidence, policy, and ontology constraints influence the final outcome.

The Knowledge Layer

The ontology serves as the system's data contract.

Rather than allowing the model to invent arbitrary entity types and relationships, Gnosis defines:

which entity types exist;
which properties belong to each type;
which relationships are allowed;
which constraints apply to those relationships.

The ontology is therefore used as an active constraint rather than passive documentation.

A model may propose a new entity or relationship.

The system decides whether that proposal complies with the ontology.

Anything that violates the contract is rejected or placed into a review process instead of silently entering the graph.

This distinction proved critical when processing large collections of heterogeneous enterprise documents.

One practical challenge repeatedly emerged throughout development.

Enterprise knowledge is rarely organized according to how language models consume information.

The same business concept may appear in multiple documents under different names, formats, or levels of detail. A requirement might exist in a specification document, a meeting note, and an operational report while referring to the same underlying concept.

Without a shared semantic framework, every retrieval and reasoning step must repeatedly infer whether those references describe the same entity.

Ontology helps reduce this ambiguity.

Rather than relying solely on statistical similarity, the system evaluates extracted entities and relationships against explicit domain definitions. The result is not a replacement for language models, but a structured environment in which model-generated proposals can be interpreted consistently.

For organizations investing in AI development services, this distinction becomes increasingly important as AI systems move closer to business-critical workflows.

Provenance as a First-Class Requirement

One design decision influenced nearly every other component in the system.

Every piece of knowledge must retain a path back to its source.

For Gnosis, retrieval quality alone was not sufficient.

An answer needed to be traceable.

The system therefore maintains provenance information that links extracted knowledge back to the original document location.

Conceptually:

Answer
  ↓
Citation
  ↓
Knowledge Node
  ↓
Source Reference
  ↓
Original Document

This principle enables users to move from an answer to the supporting evidence rather than treating the answer as an independent source of truth.

Provenance also plays an important role in governance. When knowledge can be traced back to its origin, validation becomes possible. When provenance is missing, confidence becomes much more difficult to justify.

Constrained Extraction

A common failure mode in graph construction occurs when ontology is applied only after extraction.

By that point, the graph may already contain inconsistent entity types and relationships.

Gnosis instead applies ontology constraints during extraction.

flowchart TD
RAW[LLM Output] --> PARSE{Valid Structure?}
PARSE -->|No| RETRY[Retry]
PARSE -->|Yes| TYPE{Known Type?}
TYPE -->|No| REVIEW[Review Queue]
TYPE -->|Yes| ENTITY[Valid Entity]
ENTITY --> REL{Valid Relationship?}
REL -->|Yes| GRAPH[Knowledge Graph]
REL -->|No| REJECT[Reject Relationship]

The important architectural decision is that the prompt guides extraction, but the parser enforces the rules.

The model can suggest.

The system validates.

This approach helps prevent invalid entities and relationships from propagating through later stages of the pipeline.

Building Trustworthy Retrieval

Enterprise questions frequently require more than one retrieval strategy.

A contract number may require keyword matching.

A business requirement may require semantic search.

An ownership question may require traversing relationships.

For that reason, Gnosis combines multiple retrieval approaches before answer generation begins.

flowchart LR
Q[Question] --> HS[Hybrid Search]
HS --> RR[Rerank]
RR --> RG[Relevance Gate]
RG --> GX[Graph Expansion]
GX --> GV[Guardrails & Validation]
GV --> LLM[LLM]

The purpose of retrieval is not to provide the model with as much context as possible.

Its purpose is to identify context that is sufficiently relevant to qualify as evidence.

The retrieval stage therefore becomes more than a search operation.

Enterprise queries often contain multiple information needs simultaneously. A user may reference an entity, imply a relationship, and request supporting evidence in a single question.

No individual retrieval strategy is sufficient for every situation.

Keyword search helps identify explicit identifiers. Semantic retrieval helps discover conceptually related information. Knowledge graph traversal helps expand context through relationships.

The objective is not to maximize context volume.

The objective is to identify candidate evidence that deserves further validation before generation begins.

No Evidence, No Answer

One of the most important behaviors implemented in Gnosis is the ability to refuse an answer.

A retrieval system will often return something.

That does not mean it has found evidence.

The relevance gate exists to distinguish between those two situations.

If no candidate satisfies the required criteria, the system returns a “no information” response rather than asking the model to improvise.

This mechanism directly reflects the central architecture principle:

The responsibility of the model is to generate answers.

The responsibility of the system is to determine whether an answer should be generated at all.

In practice, refusing to answer can be just as important as answering correctly.

Citation Validation

Even grounded answers can drift away from their sources.

For that reason, generation is followed by a citation validation stage.

flowchart LR
A[Generated Answer] --> S[Split Sentences]
S --> C[Check Citation Tags]
C --> V{Valid Source?}
V -->|Yes| KEEP[Keep Sentence]
V -->|No| DROP[Drop Sentence]

The rule is intentionally strict.

A sentence without a valid source does not remain in the final answer.

Rather than asking the model to self-evaluate, deterministic validation determines whether the sentence is eligible to survive.

This stage reinforces the separation between generation and approval that appears throughout the system.

Why Gnosis Extends Basic OAG

Standard OAG discussions often begin with structured objects that already exist.

Gnosis starts much earlier in the lifecycle.

The system must first create trustworthy knowledge objects from unstructured documents.

That requirement introduces several additional responsibilities:

ontology-constrained extraction;
provenance tracking;
entity resolution;
conflict handling;
retrieval validation;
citation validation.

These layers are not separate from OAG.

They apply the same principle to stages that occur before object-based reasoning becomes possible.

Another important observation is that many OAG discussions assume the existence of structured knowledge objects.

In practice, enterprise environments often begin with unstructured documents rather than governed objects.

Documents may contain duplicate entities, conflicting references, inconsistent terminology, or incomplete context.

As a result, governance cannot start at answer generation.

Governance must be applied throughout the knowledge lifecycle, beginning with extraction and continuing through graph construction, retrieval, relevance validation, reasoning, and final answer generation.

This perspective influenced many of the architectural choices explored in Gnosis.

Ontology, provenance, validation, retrieval, and citation checking were not treated as isolated components. Instead, they functioned as connected control layers responsible for determining which model-generated proposals were permitted to move forward.

Seen from this perspective, Gnosis extends the philosophy of OAG beyond reasoning and into the broader process of enterprise knowledge construction.

Closing Thoughts

Building Gnosis changed how we think about enterprise AI.

The challenge was rarely generating text.

The challenge was deciding whether generated outputs deserved to move forward.

Ontology constraints, provenance data, retrieval validation, relevance gates, guardrails, and citation checks all serve the same purpose.

They separate proposal from approval.

That separation ultimately became the most important lesson from the project.

The LLM proposes; policy and deterministic code decide which proposals go forward.

For enterprise knowledge systems and organizations delivering AI development services, that distinction may matter more than any individual model, retrieval strategy, or prompting technique.

What Happens When an AI Agent Goes Through a Real Development Workflow?

Le Trung Hieu — Wed, 30 Sep 2026 08:00:00 +0000

I’ve been experimenting with an AI-Driven Development workflow at Relipa, and one thing became clear: the interesting part isn't getting an AI agent to write code. It's designing what happens before and after the code is written.

This article follows one task through the workflow we built: Requirement → Spec → Plan → Test → Code → Review.

1. Why all six stages matter
2. Stage 1 — Requirement
3. Stages 2 and 3 — Spec and Plan
4. Stage 4 — Test: why it comes before Code
5. Stage 5 — Code: a loop with a ceiling
6. Stage 6 — Review
7. Where it still leaks
8. Three lessons
9. Closing

1. Why all six stages matter

When we started building an AI-Driven Development workflow at Relipa, one thing became clear quite quickly: having an agent that can write code does not remove the need for engineering decisions.

The developer's role shifts from writing every line of code toward deciding what the agent should do, what it should not do, and how we can prove that the result is correct.

In our workflow, that means putting several stages around the coding step: defining the scope, creating an isolated workspace, deciding where human approval is required, controlling what the agent can do, and verifying the result.

Here's the pipeline we use:

flowchart LR
    R["1 . REQUIREMENT issue on the Kanban board"] --> S["2 . SPEC user stories + criteria"]
    S --> P["3 . PLAN architecture + data model"]
    P --> T["4 . TEST define CORRECT"]
    T --> C["5 . CODE agent executes"]
    C --> V["6 . REVIEW check against + exit gate"]
    V -.->|"inline comments"| C

The unusual part is that Test comes before Code.

That is deliberate. We define what should be tested before asking the agent to implement the task, so the tests can serve as an independent reference when we evaluate the code later.

That decision turned out to affect several other parts of the workflow, so I'll come back to it in more detail in the Test stage.

2. Stage 1 — Requirement: turning a line of description into a unit of work

Everything starts with a card on the Kanban board. But the distance between "a card" and "something an agent can work on" is larger than it looks.

The first thing the system does is not call AI. It allocates a place to work.

flowchart TD
    TASK["Task on the Kanban board"] --> TRIG{"Trigger"}
    TRIG -->|"drag the card into a column with auto-run enabled"| HAS
    TRIG -->|"create manually"| HAS
    HAS{"Does this task already have a workspace?"}
    HAS -->|"Yes"| RES["RESUME in the existing worktree + branch, skip duplicate creation"]
    HAS -->|"No"| B["Generate branch name: prefix/4-hex-slug, strip Vietnamese diacritics, slug under 40 chars"]
    B --> W["Create a SEPARATE worktree for EVERY repo in the task"]
    W --> OK{"Did every repo succeed?"}
    OK -->|"No"| RB["ROLLBACK: remove every worktree just created"]
    OK -->|"Yes"| CP["Copy files per repo config, copy attachments, generate CLAUDE.md + AGENTS.md"]
    CP --> SU["Run the setup script, configured at repo level"]
    SU --> M{"Mode"}
    M -->|"Sequential . default"| M1["Agent WAITS for setup to finish"]
    M -->|"Parallel"| M2["Agent starts IMMEDIATELY"]

    style RB fill:#f8d7da,stroke:#721c24,color:#111827

A few implementation details turned out to matter more than expected.

Each issue has at most one workspace. Dragging the card back does not create a new one. It resumes in the existing worktree and branch. Without that constraint, a mis-drag can create an orphan branch and make it unclear which branch is the active one later.

Rollback is all or nothing. If creating the worktree for any repository fails, the workspace is removed rather than leaving a partial state behind.

Branch naming has to handle Vietnamese text. We normalize to NFD, remove combining marks, map đ separately because it has no decomposed form, cap the slug at 40 characters, cut back to a word boundary, and append the workspace's 4-hex-character identifier.

For example, without proper normalization, "Sửa lỗi đăng nhập" can turn into "s-a-l-i-ng-nh-p", which is technically usable but almost unreadable.

By the end of this stage, the agent has its own branch, its own directory per repository, copied environment variables, and two instruction files at the workspace root: CLAUDE.md and AGENTS.md. Those files contain @<repo>/<file> references pointing to each repository's conventions.

The agent now has a place to work before it has to make any decisions.

3. Stages 2 and 3 — Spec and Plan: from prose to constraints

These two stages run together, so I'll cover them together.

Spec turns the ticket description into a structured specification: user stories with P1/P2/P3 priorities and acceptance criteria for each story.

The important part is that scope has to be declared. Ambiguity that might otherwise appear after implementation is surfaced while the cost of changing the plan is still low.

Plan defines the tech stack, directory structure, data model, and interface contracts between components. It gives the reviewer a concrete description of the implementation path before the agent starts writing code.

At both stages, the prompt points the agent to the project's mandatory conventions file. The file itself remains in the worktree rather than being pasted into every prompt.

Where the artifacts live

flowchart TB
    subgraph DISK["On disk, inside the worktree"]
        D1[".specify/memory/constitution.md, prompt points at it"]
        D2["spec-kit templates + skills, written to disk by backend"]
        D3["tasks.md, written by the agent"]
    end
    subgraph DB["In the database, plans table"]
        B1["specification"]
        B2["content . the plan body"]
        B3["tasks_markdown"]
        B4["SubTask . TestCase"]
    end
    D3 -->|"scan .md files, read checkboxes back"| B4

    style B2 fill:#fff3cd,stroke:#856404,color:#111827

There is an important asymmetry here.

spec and plan are stored in the plans table rather than as files on disk. tasks.md, on the other hand, is written to disk by the agent, and the backend scans it back into the system.

That distinction affects token usage. The agent can be told to read the constitution from its existing path, but it cannot be told to read a plan file that does not exist. When a new session is required, the plan therefore has to be included in the prompt.

Three stopping points for humans

flowchart LR
    SP["Spec"] --> G1{{"approve-spec"}} --> PL["Plan"] --> G2{{"approve"}} --> TC["Testcase + Tasks"] --> G3{{"approve-tasks"}} --> IM["Implement"]

    style G1 fill:#fff3cd,stroke:#856404,color:#111827
    style G2 fill:#fff3cd,stroke:#856404,color:#111827
    style G3 fill:#fff3cd,stroke:#856404,color:#111827

The position of these gates matters.

They sit before implementation, where a correction can still be made by editing the specification or plan. Before approval, the plan can also be edited manually; the reviewer is not limited to a binary accept/reject decision.

For an AI development company building agent-assisted workflows, this is one of the more important design questions: where should a human be required to make a decision before the cost of being wrong becomes expensive?

4. Stage 4 — Test: why it comes before Code

This is the stage I wanted to get right from the beginning.

The familiar order is code first, then test. In our AIDD workflow, the test definition comes first.

The reasoning is simple. If the agent writes the code and then derives the tests from that same code, both can reflect the same misunderstanding of the requirement. The tests may be green while still validating the wrong behavior.

So this stage has two halves, on either side of Code:

flowchart TB
    subgraph A["4a . DEFINE . before Code"]
        A1["Generate testcase.md FROM THE SPECIFICATION, grouped by user story"]
        A2["Order within each group: Contract to Unit to Integration"]
        A3["Each test records its file path + a back-link to its task"]
        A4["A HUMAN APPROVES, this is the independent source of truth"]
        A1 --> A2 --> A3 --> A4
    end
    subgraph B["4b . EXECUTE . during and after Code"]
        B1["Agent writes actual tests against the approved test cases"]
        B2["Run the tests every round"]
        B3["Sync TestCase status back to the system"]
        B1 --> B2 --> B3
    end
    A --> B

    style A4 fill:#d4edda,stroke:#155724,stroke-width:3px,color:#111827

The agent still writes the actual test code, but the test cases it implements have already been defined and approved by a human based on the specification.

That distinction is important.

Generated before implementation, testcase.md acts as a constraint. Generated after implementation, tests can become a restatement of what the code already does.

For a reviewer, the approved test cases provide a written standard to compare against. You can identify a missing business flow before reading hundreds of lines of implementation.

Once the test cases are approved, the system decomposes the plan into a T001, T002... checklist, marks [P] on work that can run in parallel, and writes SubTasks and TestCases into the database.

5. Stage 5 — Code: a loop with a ceiling

This is the stage that can spawn another process without a new human action.

flowchart TD
    START(["Round n"]) --> PICK["Take up to 2 unfinished SubTasks + EXACTLY the test cases of those 2 tasks"]
    PICK --> SESS{"Did the previous round leave an agent session?"}
    SESS -->|"Yes"| DELTA["DELTA prompt: resume the old session, does NOT repeat plan/constitution/test plan"]
    SESS -->|"No"| FULL["FULL prompt: new session, plan verbatim + a line pointing at the constitution"]
    DELTA --> RUN["Agent codes, TDD-style"]
    FULL --> RUN
    RUN --> GUARD{"Guard inspects every tool call"}
    GUARD -->|"Block"| DENY["Refused outright"]
    GUARD -->|"Ask"| WAIT["Ask a human, TimedOut after 10 hours"]
    GUARD -->|"Warn / no rule matched"| EXEC["Allowed to run"]
    WAIT --> EXEC
    EXEC --> CM["System AUTO-COMMITS after every turn"]
    CM --> SYNC["Scan .md for checkboxes, parse agent JSON, then scan .md again"]
    SYNC --> CHECK{"Any SubTask left? loop_count under max? Agent hasn't reported COMPLETE?"}
    CHECK -->|"All three true"| NEXT["SELF-SPAWN round n+1"]
    NEXT --> START
    CHECK -->|"Any one false"| STOP(["Stop, record the reason"])

    style NEXT fill:#cce5ff,stroke:#004085,color:#111827
    style STOP fill:#f8d7da,stroke:#721c24,color:#111827
    style CM fill:#d4edda,stroke:#155724,color:#111827

The design uses a 5-round ceiling, with at most 2 unfinished SubTasks handled per round.

The ceiling is deliberate. If a task still needs correction after the configured number of rounds, the workflow treats that as a signal for human intervention rather than automatically spawning another round.

Auto-committing after every agent turn became one of the most useful features for me. It turns an agent session into a chain of restore points. If a change from several turns ago needs to be recovered, the state is already committed.

Scanning the markdown files is another important fallback. The system does not rely entirely on the agent's self-reported state; it re-reads the checkboxes left in the files.

The principle is simple: don't take state only from the thing you're supervising. Check the traces it leaves behind.

The token bill: three places we optimized

This was where some of our first assumptions about AIDD turned out to be wrong.

Place one: every round used to be a completely fresh session.

The first version reloaded the entire plan, the constitution, and all test cases on every round, including test cases for tasks that were not being worked on.

That creates what we call triangular cost: each new round pays again for content from previous rounds.

We made three changes.

First, we send only the test cases for the exact 2 tasks being worked on.

Second, we stopped pasting the constitution into every prompt. The prompt points the agent to .specify/memory/constitution.md, which is already available in the worktree.

Third, when a previous turn leaves a session ID, the next round resumes that session with a delta prompt containing only the next 2 tasks and their test cases. The plan, constitution, and templates remain in the existing conversation context.

Only when there is no session ID do we build the full prompt and open a new session.

One arithmetic constraint remains: 10 SubTasks at 2 tasks per round reaches the 5-round ceiling exactly. There is no spare automated round if something goes wrong.

Place two: prompt cache invalidation.

In our implementation, prompt caching depends on the static prefix remaining byte-for-byte identical.

The Loop {n}/{max} line originally sat at the top of the prompt. Because the number changed every round, the prefix changed too, invalidating the static content behind it.

We moved the variable section lower in the prompt:

flowchart TB
    subgraph BAD["Before . cache invalidated"]
        X1["Loop 3/5, CHANGES every round"] --> X2["line pointing at the constitution"] --> X3["plan verbatim"] --> X4["the current 2 tasks"]
    end
    subgraph GOOD["After . cache preserved"]
        Y1["line pointing at the constitution"] --> Y2["plan verbatim"] --> Y3["templates + JSON instruction block"] --> Y4["Loop 3/5"] --> Y5["the 2 tasks + their test cases"]
    end
    BAD -->|"reorder"| GOOD

    style X1 fill:#f8d7da,stroke:#721c24,color:#111827
    style Y1 fill:#d4edda,stroke:#155724,color:#111827

The principle is prefix matching: if the leading part of the prompt changes, the static content after it may no longer benefit from the same cache behavior.

Place three, and the biggest: shell output.

We initially focused heavily on optimizing the prompts we generated. The larger source of context usage was often what the agent received back from its own commands: git diff, cargo test, linters, logs, and other command output.

We added a compression layer at the tool boundary.

flowchart TD
    CMD["Agent runs a shell command"] --> W{"Wrapped in the compression layer?"}
    W -->|"Command outside the allowlist, about 21 commands"| RAW1["Run as-is"]
    W -->|"Contains a pipe or shell operator"| RAW2["Run as-is"]
    W -->|"spec-kit's own commands"| RAW3["Run as-is"]
    W -->|"Everything else"| RTK["Wrap in the compression layer: filter, fold, de-duplicate"]
    RTK --> SMALL["Sample estimate: about 80% reduction, roughly 118K to 24K tokens"]

    style RAW2 fill:#f8d7da,stroke:#721c24,color:#111827
    style SMALL fill:#d4edda,stroke:#155724,color:#111827

In the sample documented for our compression experiment, shell-output tokens dropped from roughly 118K to 24K per session, an estimated 80% reduction. This was an experiment-level estimate, not a production-wide benchmark.

The scope is deliberately narrow. Only a list of commands whose output is long and repetitive gets wrapped. Commands containing pipes or shell operators are excluded, as are spec-kit's own process commands because their output is parsed verbatim.

There is one special case: the agent frequently types:

cd <worktree> && <command>

Treating the whole thing as a compound command would bypass compression almost every time. We therefore peel off the cd portion and apply the wrapper to the command itself.

The broader lesson is that agent cost does not always come from the prompt you wrote. Measure at the tool boundary as well.

6. Stage 6 — Review: check against the standard, then out the door

The final stage has two different responsibilities: human review against an approved standard, and machine checks at the point where code leaves the workspace.

The human part: review against a standard

flowchart LR
    D["Diff updated in real time"] --> H["Human checks it against the test cases approved at stage 4"]
    H --> C["Inline comments on individual lines"]
    C --> A["Sent straight to the agent as a new prompt"]
    A --> W["Agent continues, SAME branch, SAME session"]
    W --> D

    style H fill:#d4edda,stroke:#155724,color:#111827

By this point, the reviewer has a written standard they approved earlier.

The question changes from:

"Does this look okay?"

to:

"Is this what we approved?"

Inline comments are sent to the agent as a new prompt, and the agent continues in the same branch and session.

The machine part: four exits and deterministic gates

flowchart TD    
    OUT["Code wants to leave the workspace: merge, push, force-push, open PR"] --> SCAN["Scan the repo's diff"]
    SCAN -->|"scan errors"| ERR(["Error, does NOT count as a pass"])
    SCAN --> Q1{"Any blocking regex finding?"}
    Q1 -->|"Yes"| BLK["Blocked, hard stop. DETERMINISTIC"]
    Q1 -->|"No"| Q2{"Has an AI security review finished AFTER the latest commit?"}
    Q2 -->|"Yes"| PASS(["Through the gate"])
    Q2 -->|"No"| ASK["ReviewNotRun, ask once, not a hard block"]
    BLK --> OV{"security_override with a stated reason?"}
    ASK --> OV
    OV -->|"Yes"| LOG["Log it + notify, then let it through"]
    OV -->|"No"| END(["Stop"])

    style BLK fill:#f8d7da,stroke:#721c24,color:#111827
    style ASK fill:#fff3cd,stroke:#856404,color:#111827
    style ERR fill:#f8d7da,stroke:#721c24,color:#111827

The principle we settled on is:

Only deterministic checks are allowed to block.

The AI security review can identify issues that regex cannot, such as authorization logic problems or data leakage. But its conclusions do not have veto power over the merge.

What the gate checks deterministically is whether the AI security review ran to completion after the latest commit. That is established from process-log timestamps and the repository's latest commit.

If the review has not run, the gate asks for confirmation. It does not treat the model's judgement itself as a deterministic pass/fail condition.

A scan error also counts as an error, not as a pass. Otherwise a gate that cannot see the code could accidentally become a gate that lets everything through.

We scan only added lines rather than entire files. Otherwise old test fixtures or historical content could repeatedly trigger the same finding.

There is also an explicit override with a stated reason. False positives are unavoidable, and without an escape hatch, one bad block can make people disable the entire control.

The level of assurance matters too: these are regex-based checks over strings generated by the agent. They are guardrails against accidents, not a security sandbox against an adversary.

The part nobody clicks a button for

flowchart LR
    M["Merge / PR"] --> W1["Check PRs every 60 seconds"]
    W1 --> AR["No open PRs left + workspace not pinned, AUTO-ARCHIVE"]
    AR --> CL["Scheduler every 30 minutes removes the worktree from disk"]
    CL --> NOTE2["1 hour after archiving, or 72 hours if never archived"]

None of these features is particularly interesting in a product demo. Together, they determine whether the workflow leaves behind workspaces and branches that someone has to clean up manually.

7. Where it still leaks

This section is intentionally less polished than the rest of the system. These are the gaps we still have.

The three approval buttons don't exist in the UI. All three stopping points exist only at the API layer. The backend flow works end to end, but the UI does not expose those controls yet.

Looking at a Plan's status doesn't tell you which step it's at. All three approvals set the status to Approved; you need the running process type to determine the actual step.

The agent doesn't know what tasks lie ahead. The "remaining tasks as context" section of the prompt is currently dead code because the list is cut off at the data layer.

The stopping condition has an off-by-one. The round counter is read from a snapshot taken before the increment, and the "agent reported blocked" condition is not checked at the self-spawn step. The consequence is that the system can run to 6 rounds, and the last round can run against a plan already marked blocked.

There's no unblock button and no real-time notification when something is blocked. The plan can sit there until someone notices it.

The markdown scan runs twice per round and doesn't exclude node_modules, target, or dist. On a large repository, that creates a significant disk-read cost.

The runtime guard has uneven coverage. Some executors can send file-write requests without the patch content attached, which means secrets cannot always be inspected at that layer. The diff-scanning gate exists partly to cover this gap.

Direct merge uses squash, so the detailed commit history is not preserved. That's a deliberate trade-off, but it is still a trade-off.

Writing these limitations down matters because they are easier to manage when they are visible.

8. Three lessons

One: put human intervention points at the front, not the end.

All three approval gates sit before implementation. A correction at the Spec stage is cheaper than discovering the same misunderstanding after implementation.

Two: don't take state only from the thing you're supervising.

An agent reporting "done" is a claim. Commits, file checkboxes, and test results provide traces that can be checked independently.

Three: measure cost at the tool boundary, not just at the prompt.

We spent time optimizing the prompt structure while a large share of context usage was coming from shell output. The tool boundary made that visible.

9. Closing

None of the six stages above requires a smarter model. What they require are familiar engineering practices: defining scope, establishing acceptance criteria, putting human judgment at explicit points, and using deterministic checks where deterministic checks are appropriate.

The interesting engineering problem is therefore not simply getting an agent to write code. It is building a workflow that gives the agent enough context to work, constrains what it can do, verifies what it produces, and knows when to stop.

That is still an ongoing area of exploration for us at Relipa. We're continuing to look at how AI can fit into real software development workflows without losing the engineering practices that make those workflows reliable.

If you're working on AI-assisted software development and have encountered similar challenges around architecture, process, testing, or risk management, I'd be interested in comparing notes in the comments.

This article was drafted by our engineering team and edited with the help of AI writing tools.