Alright, dear utterly clueless audience, gather around. It’s time to dive into the mystical world of setting up a private chat GPT with Llama 2. If you’re an expert or even moderately savvy in tech, shoo! This isn’t for you. Go compile a kernel or something.
Welcome to Chat GPT with Llama 2: A Noob’s Tale
Once upon a time in the land of “I-Don’t-Know-What-I’m-Doing,” there were brave souls who decided to embark on a quest to set up their own chat GPT with Llama 2. It was said to be a task only for the valiant, but fear not, for I shall guide thee through this enchanted forest of technology. Once you get enough confidence, you can further enhance, train & use it locally on your data with minimal hardware in a windows laptop.
Step 1: The hardware - I am using my normal work laptop with 24GB ram and an old nvidia graphics card with 4 GB DRAM.
Step 2: I am going to install CUDA in my system. Note down your version. I am using 12.2
Step 3: I am going to use prebuilt llama.cpp from ggerganov in github to avoid any complexity of environment setup. I will need lama-b2440-bin-win-cublas-cu12.2.0-x64.zip & cudart-llama-bin-win-cu12.2.0-x64.zip. Download & extract them in the same folder.
Step 4: Finding the Llama. Look for llama-2-7b-chat.Q5_K_M.gguf in Hugging face & download it into the same directory as above
Step 5: Awakening the Llama. Run below command if have reached upto here successfully. If all goes well, you’ll have a chatbot ready to converse with you about the mysteries of the universe or, more likely, the weather.
main -m llama-2-7b-chat.Q5_K_M.gguf -i --n-gpu-layers 32 -ins --color -p "Starting your language model"
Step 6: Chatting with the Llama Congratulations!
You’ve successfully set up your chat GPT with Llama 2. Now you can chat about life’s greatest questions, like “Why are we here?” or “What’s for dinner?” Just remember, the Llama is wise, but it’s not a chef.
And there you have it, a simple guide for the utterly clueless. May your chats be merry, and your Llama never spit.
Top comments (0)