DEV Community: Tim Pap

🚀 🤖 Let's Retrieve Data and Talk: A Full-stack RAG App with Create-Llama and LlamaIndex.TS

Tim Pap — Mon, 22 Jan 2024 21:13:00 +0000

🦙 LlamaIndex is an open-source data framework that allows developers to build applications powered by large language models like GPT-3 using their own custom data.

🦾 The create-llama CLI tool makes it easy to scaffold a full LlamaIndex app with just one command. It sets up a Next.js frontend, choice of Express, Node or Python backends, data ingestion and indexing, and LLM integration out of the box so you can start building quickly. With LlamaIndex and create-llama, you can take advantage of powerful LLMs while keeping your data private and customizing the models to your specific domain. I'm excited to start experimenting with building conversational apps enhanced by LlamaIndex's data retrieval capabilities!

In this hands-on tutorial, we’ll use create-llama to scaffold a custom search assistant app. I’ll show you how I set up data ingestion pipelines, indexing, storage all using LlamaIndex’s TypeScript library.

Setup

The easiest way to leverage create-llama is by invoking its interactive mode:

npx create-llama@latest

When create-llama runs, it will prompt you to name your project and set additional configuration:

✔ What is your project named? … llamaindex-ts-first-steps
✔ Which template would you like to use? › Chat with streaming
✔ Which framework would you like to use? › NextJS
✔ Which UI would you like to use? › Just HTML
✔ Which model would you like to use? › gpt-3.5-turbo
✔ Which data source would you like to use? › Use an example PDF
✔ Would you like to use a vector database? › No, just store the data in the file system
✔ Please provide your OpenAI API key (leave blank to skip): #### Add your OpenAI key here
✔ Would you like to use ESLint? … No / Yes

Running the Example App

Navigate into the llamaindex-ts-first-steps directory:

cd llamaindex-ts-first-steps

Next, open the project in your editor - for example using VS Code:

code .

To provide custom data for LlamaIndex to ingest, first create a text file named my-file.txt within the data/ directory. Add whatever content you would like LlamaIndex to have access to - this can be any freeform text.

For example:

hey my name is Tim
the secret number is 12

Then run:

npm run generate

This preprocesses the documents and creates vector embeddings to enable semantic search.

With data indexed, start the application:

npm run dev

You can now ask natural language questions about your documents!

Note: Re-run generate whenever new data is added to update the index.

This application utilizes Vercel's AI SDK for the text generation components.

Indexing Logic

The main indexing logic resides in generate.mjs. Let's examine how it works

const documents = await new SimpleDirectoryReader().loadData({
  directoryPath: STORAGE_DIR,
});

await VectorStoreIndex.fromDocuments(documents, {
  storageContext,
  serviceContext,
});

The SimpleDirectoryReader provides a straightforward method to ingest local files into LlamaIndex. While more robust Readers from LlamaHub may be better suited for production systems, the SimpleDirectoryReader offers a simple on-ramp to start loading data and experimenting with LlamaIndex.

A Document encapsulates a data source such as a PDF, API response, or database query result. Within LlamaIndex, data is divided into discrete Node objects representing atomic semantic units. For example, a node could contain a paragraph of text or table from a document.

Nodes maintain metadata linking them to their parent Document and any related Nodes. This connectivity between nodes and back to source documents creates a rich knowledge graph for targeted information retrieval.

Vector stores play a vital role in retrieval-augmented generation by efficiently indexing vector embeddings. You'll leverage vector stores, whether directly or behind the scenes, in most LlamaIndex applications.

A vector store ingests Node objects, analyzing the data to construct an optimized search index.

The most straightforward approach for indexing data utilizes the vector store's fromDocuments method. Simply pass in your documents and the vector store handles building the index

Indexes and Embeddings

After loading data, LlamaIndex facilitates indexing to optimize retrieval. Indexing transforms the raw content into vector embeddings - numeric representations of semantic meaning. These embeddings get stored in a vector database engine specialized for efficient similarity searches.

The index may also track extra metadata like relationships between nodes. This supplementary information bolsters the relevance of fetched content.

To locate relevant context for a query, LlamaIndex first converts the search terms into an embedding vector. It then identifies stored nodes with the closest matching embeddings to the query vector. This vector similarity search allows retrieving the most contextually related data points for any natural language query.

The chat functionality is enabled by createChatEngine defined in app/api/chat/engine/index.ts.

This exports the createChatEngine factory function which constructs a ChatEngine instance for conducting conversational flows.

export async function createChatEngine(llm: LLM) {
  const index = await getDataSource(llm);
  const retriever = index.asRetriever();
  retriever.similarityTopK = 5;

  return new ContextChatEngine({
    chatModel: llm,
    retriever,
  });
}

The ContextChatEngine enables conversational interactions powered by retrievals over indexed data.

Its chat loop works as follows for each user message:

Retrieve the most relevant text passages from the index based on the message text.
Incorporate those texts as context when formulating the system prompt.
Generate a response to the user message using the context-aware prompt.
Return the reply to the user.

This simplicity makes ContextChatEngine suitable for queries directly related to the knowledge base and basic conversational flows. By basing each response on contextual retrievals, it can directly leverage the indexed data.

The full source code for this application is available at:

https://github.com/blackpr/llamaindex-ts-first-steps

Feel free to reference the repository to see the complete working example of initializing LlamaIndex, configuring indexing and retrieval, and enabling conversational interactions.

It demonstrates an end-to-end pipeline from ingesting custom data through to asking questions via a chat interface powered by vector similarity search.

🤖📚 Take Your First Steps into RAG: Building a LlamaIndex Retrieval Application using OpenAI’s gpt-3.5-turbo

Tim Pap — Sat, 20 Jan 2024 20:10:00 +0000

🔗 Retrieval-Augmented Generation (RAG) combines the knowledge look up and retrieval capacities of search engines with the fluent language generation capabilities of large language models.

🦙 LlamaIndex is an open-source data framework that enables developers to ingest, structure, and query data for use in large language model (LLM) applications. It facilitates a Retrieval Augmented Generation (RAG) approach, where relevant knowledge is retrieved from data sources before being fed to language models to generate high-quality responses.

LlamaIndex handles the complexity of connecting to data, building indexes and retrieval pipelines, and integrating LLMs. Whether your data lives in APIs, databases, or documents, LlamaIndex makes it seamless to leverage for AI applications. With intuitive APIs for querying knowledge and conversing with chatbots to customizable search functionality, LlamaIndex lets you focus on creating performant, tailored LLM experiences.

In this post, I'll provide a step-by-step tutorial for building your first RAG application with LlamaIndex and OpenAI’s gpt-3.5-turbo

If you already have a Python environment configured, you can skip the next section and start building your LlamaIndex application directly.

Alternatively, if you have Docker installed, you can leverage a VS Code development container for a ready-made environment without any additional setup.

Otherwise, the following section will guide you through installing Python and setting up a virtual environment to run LlamaIndex smoothly. The choice depends on your existing tools and preference.

(optional) setup development environment

open wsl
install build essential sudo apt-get install build-essential
install deps sudo apt-get install build-essential libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ xz-utils tk-dev libffi-dev liblzma-dev python3-openssl git
install pyenv curl [https://pyenv.run](https://pyenv.run/) | bash
install python pyenv install 3
install pipx sudo apt install pipx, pipx ensurepath
install poetry pipx install poetry
poetry config virtualenvs.in-project true
pyenv local 3
poetry new rag, cd rag
poetry shell. If you want to deactivate use deactivate

Setup

LlamaIndex utilizes OpenAI's gpt-3.5-turbo model for text generation and text-embedding-ada-002 for retrieval operations by default. To leverage these models, you need an OpenAI API key configured as the OPENAI_API_KEY environment variable.

To obtain an API key:

Log into your OpenAI account
Create a new API key

This unique key authorizes LlamaIndex to call OpenAI models on your account's behalf.

install libraries

poetry add llama-index
poetry add python-dotenv
open app in vs code code .
add a .gitignore file with the following content: https://www.toptal.com/developers/gitignore/api/python
create a .env file and add:

OPENAI_API_KEY=ADD_YOUR_KEY_HERE

To provide custom data for LlamaIndex to ingest, first create a text file named my-file.txt within the rag/data directory. Add whatever content you would like LlamaIndex to have access to - this can be any freeform text.

For example:

hey my name is Tim
the secret number is 12

Now LlamaIndex can ingest this data file and allow querying over the content using natural language

Application

To build your LlamaIndex application, first create a Python file called app.py in the rag/ directory.

This app.py will hold the code powering your application. Next we'll start adding Python logic to initialize LlamaIndex, load data, define queries, and ultimately enable asking questions over your custom knowledge.

from dotenv import load_dotenv
from llama_index import (
    VectorStoreIndex,
    SimpleDirectoryReader,
)

load_dotenv()

documents = SimpleDirectoryReader("./rag/data/").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("What is the secret number?")
print(response)

run the app python rag/app.py

By calling load_dotenv(), we populate the environment variables for the current Python process from the .env file. This allows us to store configuration, credentials, and other sensitive information in .env rather than hard-coding them in our application code. The .env file is gitignored by default so it won't be committed into source control.

documents = SimpleDirectoryReader("./rag/data/").load_data()

index = VectorStoreIndex.from_documents(documents)

A vector store ingests Node objects, analyzing the data to construct an optimized search index.

The most straightforward approach for indexing data utilizes the vector store's from_documents method. Simply pass in your documents and the vector store handles building the index

Indexes and Embeddings

The index may also track extra metadata like relationships between nodes. This supplementary information bolsters the relevance of fetched content.

query_engine = index.as_query_engine()

A query engine enables querying a knowledge base through natural language. It takes a question expressed in plain text, retrieves the most relevant supporting content from the indexed data, supplies both the question and contextual information to a language model, and returns the model's response. This end-to-end pipeline allows users to extract information from data by simply asking questions in everyday language.

Simple storage for embedings

The vector embeddings representing your indexed data reside in memory by default. You can optimize performance by persisting these embeddings to local storage instead. Add this line to save the index:

index.storage_context.persist()

The data will persist to the "storage" directory by default. To customize this, pass the desired location to the persist_dir parameter.

To leverage a persisted index, check if one exists and load it. If not found, generate a new index before persisting

import os.path
from llama_index import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
    load_index_from_storage,
)

# check if storage already exists
PERSIST_DIR = "./storage"
if not os.path.exists(PERSIST_DIR):
    # load the documents and create the index
    documents = SimpleDirectoryReader("data").load_data()
    index = VectorStoreIndex.from_documents(documents)
    # store it for later
    index.storage_context.persist(persist_dir=PERSIST_DIR)
else:
    # load the existing index
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)

Inspecting Activity with Logging

To understand everything occurring within your LlamaIndex application, configure logging to output internal events and queries. At the start of starter.py, add:

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

If you encounter rate limiting while sending requests to OpenAI, you may be using a free API key instead of a paid plan. Rate limits apply to OpenAI's free tier. Also verify you have configured LlamaIndex with a valid OpenAI API key associated with a paid subscription https://platform.openai.com/account/billing/overview

Final application

from dotenv import load_dotenv
import os.path
from llama_index import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
    load_index_from_storage,
)

load_dotenv()

# check if storage already exists
PERSIST_DIR = "./rag/storage"
if not os.path.exists(PERSIST_DIR):
    # load the documents and create the index
    documents = SimpleDirectoryReader("./rag/data").load_data()
    index = VectorStoreIndex.from_documents(documents)
    # store it for later
    index.storage_context.persist(persist_dir=PERSIST_DIR)
else:
    # load the existing index
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)

documents = SimpleDirectoryReader("./rag/data/").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("What is the secret number?")
print(response)

application tree

├── README.md
├── poetry.lock
├── pyproject.toml
├── rag
│   ├── __init__.py
│   ├── app.py
│   ├── data
│   │   └── my-file.txt
│   └── storage
│      ├── default__vector_store.json
│      ├── docstore.json
│      ├── graph_store.json
│      ├── image__vector_store.json
│      └── index_store.json
└── tests
    └── __init__.py

The full source code for this LlamaIndex example is located at:

https://github.com/blackpr/llamaindex-rag-first-steps

This repository contains a complete application showcasing core LlamaIndex concepts including:

Loading custom documents
Indexing via vector embeddings
Defining a query engine
Enabling querying

🐥 Demystifying the AI Jargon: A Beginner's Guide

Tim Pap — Fri, 19 Jan 2024 23:05:54 +0000

As AI technology advances rapidly, the terminology can be confusing. Here's a quick guide to some of the key AI jargon I came across recently:

Models - Large data sets that are trained on lots of data to understand prompts and questions. They vary in speed, price, size and quality.
LLMs (Large Language Models) - Models specifically focused on understanding and generating natural language text.
Providers - Services like OpenAI, Anthropic, Replicate, Cohere, Amazon Bedrock that provide easy APIs to test and run AI models.
Hugging Face - A hub for open source machine learning models anyone can use. Has hundreds of models for text, image and speech tasks.
Tokens - The way models count length of text prompts sent to them. More tokens used = more expensive.
Embeddings - Turning text into mathematical representations models can match for similarity.
Temperature - Setting that affects how creative/random a text generation model will be.
Top percentile (Top p) - Similar setting that tweaks how deterministically the same a model's outputs will be.
Fine tuning - Customizing a model by training it further on your own specialized data set.
Prompts - What you send the model to prime it to generate a useful response.
VectorDB - Database for searching embeddings to find similar documents.
Streaming - Displaying a model's output as it generates, rather than waiting for it to finish.
Evals - Test suites to evaluate model quality over time.