Build a RAG System with Python and OpenAI

🚀 Technical Briefing: This tutorial is part of our deep-dive series on Agentic Workflows at Gate of AI. For the full technical breakdown, interactive code sandbox, and the native Arabic translation, visit the original article here.

<span>Tutorial</span>
<span>Intermediate</span>
<span>⏱ 60 min read</span>
<span>© Gate of AI 2026-06-15</span>

In this tutorial, you will learn how to build a powerful Retrieval-Augmented Generation (RAG) system using Python and OpenAI's latest SDK. This system will enhance your language model's responses by grounding them in relevant data, with a focus on applications in the GCC region.

Prerequisites

• Python 3.10 or later
• OpenAI API key
• Intermediate Python programming skills

What We're Building

We will construct a Retrieval-Augmented Generation (RAG) system that leverages the strengths of large language models with the precision of targeted data retrieval. The system will be capable of fetching relevant information from a specified dataset and using that information to generate more accurate, contextually grounded responses. This is particularly useful in the GCC region where initiatives like Saudi Vision 2030 emphasize AI integration.

The finished project will allow you to input a query, retrieve pertinent data from your database, and then produce a response that integrates this data using OpenAI's GPT model. This setup is ideal for applications such as customer support, educational tools, or any context where accurate and informed responses are crucial.

Setup and Installation

To start building our RAG system, we need to set up our development environment with the necessary tools and libraries. This includes installing the OpenAI SDK and setting up a vector database for data retrieval.

pip install openai pinecone-client

We will also need to set up environment variables to securely store our API keys and other configuration settings. Create a .env file in your project directory with the following content:

OPENAI_API_KEY=your_openai_api_key_here

PINECONE_API_KEY=your_pinecone_api_key_here

Step 1: Setting Up the Vector Database

The vector database is central to our RAG system, as it allows us to perform efficient similarity searches. We will use Pinecone, a leading vector search engine, to store and retrieve data based on similarity to our input queries.

from pinecone import Pinecone

  
  
  Initialize Pinecone client


pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))

  
  
  Define schema for your data


index = pc.Index('document-index')

  
  
  Create schema in Pinecone


index.create_index(dimension=512)

Here we initialize a Pinecone client with a secure connection. We define an index for our documents, specifying the dimensionality of the vectors. This index is then created in our Pinecone instance, allowing us to store and query documents.

Step 2: Ingesting Data into the Vector Database

With our database schema ready, we can now ingest data into Pinecone. This involves adding documents that the system will later retrieve and use to augment its responses.

documents = [

    {"content": "OpenAI develops AI technologies and models for various applications."},

    {"content": "Pinecone is a leading vector search engine."},

    {"content": "Retrieval-Augmented Generation enhances language model outputs."}

]

  
  
  Add documents to Pinecone


for doc in documents:

    index.upsert(vectors=[(doc['content'], vector)])

This code snippet loops through a list of documents and adds each one to the Pinecone database. These documents will be used during the retrieval phase to provide contextually relevant information to our language model.

Step 3: Building the RAG System

Now that our data is stored, we can construct the core of the RAG system. This involves querying the vector database to retrieve relevant documents and using the OpenAI API to generate a response based on these documents.

from openai import OpenAI

  
  
  Initialize OpenAI client


client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

def generate_response(query):

    # Retrieve relevant documents from Pinecone

    result = index.query(query, top_k=3)

# Extract content from retrieved documents
retrieved_texts = [doc['content'] for doc in result]

# Construct a prompt for the language model
prompt = f"Using the following information, answer the query: {query}\n" + "\n".join(retrieved_texts)

# Generate a response using OpenAI's GPT model
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "system", "content": prompt}]
)

return response['choices'][0]['message']['content']



    

    





  
  
  Example usage


query = "What is RAG in AI?"

response = generate_response(query)

print(response)

In this step, we define a function generate_response that takes a user's query as input. It retrieves the top 3 most relevant documents from Pinecone and constructs a prompt that includes these documents. This prompt is then sent to the OpenAI GPT model to generate a coherent response. The function returns the generated response, which can be printed or used in your application.

⚠️ Common Mistake: Ensure your Pinecone client is correctly authenticated and your OpenAI API key is valid. Misconfiguration can lead to authentication errors.

Testing Your Implementation

To verify that your RAG system works correctly, you should test it with various queries and check that the responses are both relevant and accurate. The goal is to ensure that the retrieved documents genuinely enhance the language model's output.

  
  
  Test the system


test_queries = [

    "Explain the concept of RAG in AI.",

    "What is OpenAI known for?",

    "Describe Pinecone's functionality."

]

for query in test_queries:

    print(f"Query: {query}")

    response = generate_response(query)

    print(f"Response: {response}\n")

Run this test script to see how well your system performs. The responses should reflect the content of your stored documents and provide informative answers to the queries.

What to Build Next

Here are a few ideas for expanding the capabilities of your RAG system:

• Integrate More Data Sources: Enhance your RAG system by integrating additional data sources such as live news feeds or proprietary databases.
• Improve Retrieval Strategies: Experiment with hybrid retrieval methods that combine vector search with traditional keyword search to improve accuracy.
• Deploy to Production: Package your RAG system into a microservice and deploy it on a cloud platform for scalability and availability.