DEV Community

Cover image for Build a rag for customer support knowledge base that answers tickets automatically
Sam Chen
Sam Chen

Posted on Originally published at getaab.com

Build a rag for customer support knowledge base that answers tickets automatically

#ai #automation #nocode #tutorial

You'll create a Retrieval-Augmented Generation (RAG) pipeline that pulls the most relevant support articles from your knowledge base, feeds them to an OpenAI LLM, and returns a ready-to-send answer to Zendesk tickets. The result is a hands-free response engine that reduces agent load while keeping answers accurate and up-to-date.

What is RAG? RAG (Retrieval-Augmented Generation) is an architecture that first retrieves relevant documents from an external source and then conditions a large language model on those passages before generating a response.

What you need

Tool Plan / Price Role
OpenAI API Pay-as-you-go (see https://openai.com) Embedding generation and LLM inference
LangChain Open-source (MIT) Glue code for retrieval, prompting, and chaining
Chroma Self-hosted, free Vector store for embeddings (local dev)
Pinecone Check Pinecone's current pricing Hosted vector DB (production scaling)
n8n Free Community edition (Docker) Workflow engine that connects Zendesk ↔ RAG pipeline
Zendesk Existing support subscription Ticket source and destination
Python 3.11 Free Runtime for LangChain code
Git Free Version control for reproducibility

Time to build: ~8 hours for a functional prototype (2 h env setup, 3 h data ingestion, 2 h integration, 1 h testing).

Step-by-step construction

1. Prepare the development environment

Create a fresh directory and initialise a Python virtual environment:

mkdir rag-support && cd rag-support
python3 -m venv .venv
source .venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

This isolates dependencies and lets you run the same code locally and in Docker later.

Install the required libraries:

pip install openai langchain chromadb tqdm
Enter fullscreen mode Exit fullscreen mode

Why: openai provides the embedding and completion endpoints, langchain offers high-level abstractions for retrieval, and chromadb (the Python client for Chroma) stores vectors efficiently on disk.

2. Pull your support articles from Zendesk

Export the knowledge base as Markdown or plain-text files. A quick way is to use Zendesk's API:

curl -s -H "Authorization: Bearer $ZENDESK_TOKEN" \
 "https://yoursubdomain.zendesk.com/api/v2/help_center/articles.json" \
 | jq -r '.articles[] | "\(.title)\n\(.body)"' > articles.txt
Enter fullscreen mode Exit fullscreen mode

Replace $ZENDESK_TOKEN with a token that has read permission on the Help Center. The output concatenates every article into articles.txt, one article after another.

3. Chunk the articles for efficient retrieval

Long documents need to be split into manageable pieces (≈ 300 tokens) so that embeddings stay within OpenAI's token limits.

from langchain.text_splitter import RecursiveCharacterTextSplitter

with open("articles.txt", "r", encoding="utf-8") as f:
 raw = f.read()

splitter = RecursiveCharacterTextSplitter(
 chunk_size=300,
 chunk_overlap=30,
 separators=["\n\n", "\n", " "]
)
chunks = splitter.split_text(raw)
print(f"Created {len(chunks)} chunks.")
Enter fullscreen mode Exit fullscreen mode

Why: Overlapping chunks preserve context across paragraph boundaries, improving retrieval relevance.

4. Generate embeddings with OpenAI

Convert each chunk into a 1536-dimensional vector using the text-embedding-ada-002 model:

import os, openai
from tqdm import tqdm

openai.api_key = os.getenv("OPENAI_API_KEY")

def embed(texts):
 # Batch up to 2048 tokens per request (OpenAI limit)
 return openai.Embedding.create(
 model="text-embedding-ada-002",
 input=texts
 )["data"]

embeddings = []
batch_size = 100
for i in tqdm(range(0, len(chunks), batch_size)):
 batch = chunks[i:i+batch_size]
 resp = embed(batch)
 embeddings.extend([r["embedding"] for r in resp])
Enter fullscreen mode Exit fullscreen mode

Each embedding costs $0.0001 per 1 000 tokens, so a 10 000-article knowledge base typically stays under $5 per month on the OpenAI pay-as-you-go tier.

5. Store embeddings in a vector database

Option A - Local development with Chroma 

import chromadb
from chromadb.utils import embedding_functions

client = chromadb.Client()
collection = client.create_collection(
 name="support-knowledge",
 embedding_function=embedding_functions.OpenAIEmbeddingFunction(
 api_key=os.getenv("OPENAI_API_KEY")
 )
)

ids = [f"doc-{i}" for i in range(len(chunks))]
collection.add(
 ids=ids,
 documents=chunks,
 embeddings=embeddings
)
print("Vectors persisted to ./chromadb")
Enter fullscreen mode Exit fullscreen mode

Option B - Production with Pinecone (hosted) 

import pinecone

pinecone.init(api_key=os.getenv("PINECONE_API_KEY"), environment="us-west1-gcp")
index = pinecone.Index("support-knowledge")
vectors = [(ids[i], embeddings[i]) for i in range(len(embeddings))]
index.upsert(vectors=vectors, namespace="support")
Enter fullscreen mode Exit fullscreen mode

Why choose Pinecone? It offers sub-millisecond latency, automatic scaling, and built-in metadata filtering - critical for high-traffic support desks.

6. Build the LangChain retrieval-augmented chain 

from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
from langchain.vectorstores import Chroma, Pinecone
from langchain.embeddings import OpenAIEmbeddings

# Choose the backend that matches step 5
if use_chroma:
 vectorstore = Chroma(
 collection_name="support-knowledge",
 embedding_function=OpenAIEmbeddings()
 )
else:
 vectorstore = Pinecone.from_existing_index(
 index_name="support-knowledge",
 embedding=OpenAIEmbeddings(),
 namespace="support"
 )

retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
qa = RetrievalQA.from_chain_type(
 llm=OpenAI(model_name="gpt-3.5-turbo"),
 chain_type="stuff",
 retriever=retriever,
 return_source_documents=True
)
Enter fullscreen mode Exit fullscreen mode

This chain fetches the four most relevant chunks, concatenates them, and prompts the LLM to answer the user's question while citing sources.

7. Expose the chain as an HTTP endpoint with n8n

  1. Run n8n (Docker recommended): 
 docker run -d --name n8n \
 -p 5678:5678 \
 -v ~/.n8n:/home/node/.n8n \
 n8nio/n8n
Enter fullscreen mode Exit fullscreen mode

  1. Create a workflow:

    • Webhook node → receives a JSON payload from Zendesk (ticket_id, question).
    • Execute Command node → runs a short Python script that calls qa.run(question) and returns answer and sources.
    • HTTP Request node → posts the answer back to the Zendesk ticket via PUT /api/v2/tickets/{ticket_id}.

  2. Python script for the Execute Command node (answer.py):

 import sys, json
 from answer_chain import qa # assumes qa defined in previous step

 payload = json.loads(sys.stdin.read())
 question = payload["question"]
 resp = qa({"query": question})
 result = {
 "answer": resp["result"],
 "sources": [doc.metadata["source"] for doc in resp["source_documents"]]
 }
 print(json.dumps(result))
Enter fullscreen mode Exit fullscreen mode

n8n pipes the incoming JSON to stdin; the script writes a JSON response to stdout which n8n captures for downstream nodes.

  1. Save and activate the workflow. Its public URL (e.g., https://n8n.mycompany.com/webhook/rag-support) becomes the endpoint you register in Zendesk's Triggers UI.

8. Wire the endpoint into Zendesk

In Zendesk, create a Trigger that fires on Ticket Created with the condition Ticket is a support request. Add an Action → Notify target → HTTP target pointing at the n8n webhook URL, passing { "ticket_id": "{{ticket.id}}", "question": "{{ticket.description}}" }.

When a ticket arrives, Zendesk calls the webhook, the RAG chain returns an answer, and the workflow updates the ticket with the response.

9. Test end-to-end

Create a dummy ticket in Zendesk:

Enter fullscreen mode Exit fullscreen mode

After a few seconds, the ticket body should contain a concise answer such as:

To reset your password, click "Forgot password?" on the login page, enter your email, and follow the link you receive. See article "Password Reset Procedure" for screenshots.

If the answer is missing, check n8n's execution log (accessible at https://n8n.mycompany.com/executions) for any runtime errors.

Where this breaks

> The most common failure is hitting OpenAI's rate limits or token quotas, which silently abort the embedding step.

Failure mode Symptom Fix
OpenAI rate limit (60 requests/min for text-embedding-ada-002) Embedding script stalls, openai.error.RateLimitError raised Add exponential back-off (time.sleep(2**retry)) and request higher limits via the OpenAI dashboard.
Vector DB cost overrun (Pinecone reads > 2 M per month) Unexpected bill spike, API returns 429 Too Many Requests Enable Pinecone's request throttling and monitor usage via the Pinecone console; switch to Chroma for bulk offline queries.
n8n webhook authentication Zendesk receives 401 Unauthorized and tickets remain unchanged Secure the webhook with a static X-API-KEY header; add the same header in the Zendesk HTTP target settings.
Chunk size too large openai.error.InvalidRequestError: This model's maximum context length is 4096 tokens Reduce chunk_size to ≤ 300 tokens or upgrade to gpt-4 (larger context).
Source document mismatch Answer cites wrong article IDs Ensure each chunk's metadata includes a source field (e.g., article URL) when adding to the vector store.
Network latency End-to-end response > 10 s, causing Zendesk timeout Deploy n8n behind a low-latency VPC, enable keep-alive connections, and consider caching the most common queries in Redis.

For a deeper technical reference, see n8n's documentation.

FAQ

How does RAG differ from a plain LLM prompt?

RAG first retrieves factual snippets from a searchable store, then conditions the LLM on those snippets. This reduces hallucinations because the model's output is anchored to concrete documentation rather than relying solely on its pre-training.

Can I use a different LLM than OpenAI's?

Yes. LangChain supports Cohere, Anthropic, and open-source models like LLaMA via the llama-cpp-python wrapper. Swap the OpenAI object with the provider's equivalent and adjust the embedding model accordingly.

Is Chroma suitable for production?

Chroma is excellent for prototyping and low-traffic environments because it runs locally and costs nothing. For high-volume SaaS or multi-region deployments, a managed vector DB such as Pinecone or Weaviate provides automatic scaling and SLA guarantees.

What if my knowledge base updates daily?

Schedule the ingestion script to run nightly (via cron or an n8n timer) and use vectorstore.delete(ids=old_ids) followed by vectorstore.add(...) to replace stale vectors. Pinecone's upsert operation automatically overwrites vectors with matching IDs.

How do I keep the system GDPR-compliant?

Never store raw personally identifiable information (PII) in the vector store. Strip or redact PII during the chunking stage, and configure the OpenAI API to disable data logging (openai.api_key = "..."; openai.api_base = "https://api.openai.com/v1"; openai.log = "none").

If you want to sell this automation to other SaaS teams, check out AI automations you can sell for pricing ideas, and grab the free guide for a walkthrough of similar workflows.

Ready to replace manual ticket replies with a reliable rag for customer support knowledge base? Deploy the steps above, monitor the metrics, and iterate on your retrieval prompts - your support agents will thank you.

Related reading

Top comments (0)