DEV Community: Hassan Mehmood

Knowledge Graph Index with NebulaGraph and OpenAI LLM

Hassan Mehmood — Wed, 06 Dec 2023 04:49:01 +0000

This tutorial contains code to set up a Knowledge Graph Index using NebulaGraph as the storage backend and OpenAI's Language Model (LLM) for query processing.

Setup

Environment Configuration

Load Environment Variables: Ensure you have a .env file set up with required environment variables. Use dotenv to load them in your Python environment.

OPENAI_API_KEY=ak**

```python
import os
from dotenv import load_dotenv

load_dotenv()
```

Logging Configuration: Configure the logging settings.

import logging
import sys

logging.basicConfig(
    stream=sys.stdout, level=logging.INFO
)  # logging.DEBUG for more verbose output

Import Dependencies: Import necessary libraries and modules.

# Import necessary modules and libraries
from llama_index import (
    KnowledgeGraphIndex,
    LLMPredictor,
    ServiceContext,
    SimpleDirectoryReader,
)
from llama_index.storage.storage_context import StorageContext
from llama_index.graph_stores import NebulaGraphStore
from llama_index.llms import OpenAI
from IPython.display import Markdown, display

NebulaGraph Setup

Ensure NebulaGraph (version 3.5.0 or newer) is set up with the following steps:

Cluster Creation: Create a NebulaGraph cluster using one of the following options:

Docker Installation (Machines with Docker Installed):
- Option 0: Run the command: curl -fsSL nebula-up.siwei.io/install.sh | bash
Desktop Installation:
- Option 1: Install NebulaGraph Docker Extension from Docker Hub.
Manual Setup via NebulaGraph Console: If the above options are not applicable, manually create the NebulaGraph space, tags, edges, and indexes using the NebulaGraph console. Refer to the provided commands in the code comments.

Environment Configuration: Set NebulaGraph environment variables.

os.environ["NEBULA_USER"] = "root"
os.environ["NEBULA_PASSWORD"] = "nebula"  # default is "nebula"
os.environ["NEBULA_ADDRESS"] = "127.0.0.1:9669"  # assumed NebulaGraph installed locally

Knowledge Graph Indexing

Define Graph Structure: Define the graph structure (space, tags, edges).

space_name = "llamaindex"
edge_types, rel_prop_names = ["relationship"], ["relationship"]
tags = ["entity"]

Initialize NebulaGraph Store and Storage Context:

graph_store = NebulaGraphStore(
    space_name=space_name,
    edge_types=edge_types,
    rel_prop_names=rel_prop_names,
    tags=tags,
)
storage_context = StorageContext.from_defaults(graph_store=graph_store)

Load Data: Load documents/data to be indexed.

documents = SimpleDirectoryReader("data").load_data()

Create Knowledge Graph Index:

kg_index = KnowledgeGraphIndex.from_documents(
    documents,
    storage_context=storage_context,
    max_triplets_per_chunk=10,
    service_context=service_context,
    space_name=space_name,
    edge_types=edge_types,
    rel_prop_names=rel_prop_names,
    tags=tags,
    include_embeddings=True,
)

Querying the Knowledge Graph

Initialize Query Engine:

from llama_index.query_engine import KnowledgeGraphQueryEngine

query_engine = KnowledgeGraphQueryEngine(
    storage_context=storage_context,
    service_context=service_context,
    llm=llm,
    verbose=True,
)

Querying:

response = query_engine.query(
    "your question on the data?",
)
display(Markdown(f"<b>{response}</b>"))

Further Customization

Adjust parameters and configurations in the code based on specific requirements or environment setup.

GITHUB REPO

https://github.com/hasanmehmood/llamaindex-knowledge-graph

Github is now free for teams!!

Hassan Mehmood — Tue, 14 Apr 2020 19:03:44 +0000

Visit the below link to get full details, Thanks.

https://github.blog/2020-04-14-github-is-now-free-for-teams/

Docker Cheat Sheet

Hassan Mehmood — Sun, 12 Apr 2020 16:00:22 +0000

CONTAINERS

Start a running container from an image
docker run -ti ubuntu bash
List running containers
docker ps
List all containers (running + stopped)
docker ps -a
List last container
docker -l
Delete a container after it finishes
docker run -ti --rm ubuntu bash
Commit a container and create a new image from it. First find the container id or its name then run the following command. Lets say 1234 is container id and my_container is the name
docker commit 1234 new_image_name
OR
docker commit my_container new_image_name
Attach an existing running container
docker attach name_of_container
Now leaving container wihtout exiting it
press and hold ctrl + p, ctrl + q
docker exec creates a new process in the existing container. Although it will die automatically if you close the original process
docker logs will print the logs of the container
docker logs container_name
docker kill will kill the running container
docker kill container_name
Removing containers
docker rm container_name
If the outside ports are not assigned then it will choose any available port. To see which port it choose, run the following command
docker port container_name

NETWORKS

For displaying list of already existing networks.
docker network ls
For creating a new network
docker network create new_network_name
for starting a container within a network
docker run -ti --net my_network_name --name my_container_name ubuntu bash

VOLUMES

2 main varieties of volumes
Persistent --> they will still exist when the contaner is stopped
Ephemeral --> they will be gone when no one (container) is using them
- Sharing a volume with the host (Persistant)
  
  docker run -ti -v /full/path/to/folder:/shared_folder ubuntu bash
- Sharing a volume b/w containers using volume-from (Ephemeral)
  docker run -ti -v /shared-data ubuntu bash
  - In above command Im creating a shared volume on the container that is not shared with the host
  - Now create another container and shared the above volume with the new continer
    
    docker run -ti --volume-from above_container_name ubuntu bash
  - Now important note: The above volume will be available as long as atleast one container is still using it. when no container is active that volume will disappear

Git Repo: https://github.com/hasanmehmood/docker-cheat-sheet