How to run LLAMA 2 on your local computer

Nithin I Bhandari — Fri, 21 Jul 2023 17:58:01 +0000

Introduction

LLAMA 2 is a large language model that can generate text, translate languages, and answer your questions in an informative way. In this blog post, I will show you how to run LLAMA 2 on your local computer.

Prerequisite:

Install anaconda
Install Python 11

Steps

Step 1:

1.1: Visit to huggingface.co
Model Link: https://huggingface.co/meta-llama/Llama-2-7b-hf
1.2: Create an account on HuggingFace
1.3: Request for llama model access
It may take a day to get access.
1.4: Go to below link and request llama access
Link: https://ai.meta.com/resources/models-and-libraries/llama-downloads/
1.5: As llama 2 is private repo, login by huggingface and generate a token.
Link: https://huggingface.co/settings/tokens



pip install huggingface_hub



huggingface-cli login

Step 2: Create a conda environment and activate conda environment



conda create -n py_3_11_lamma2_run python=3.11 -y



conda activate py_3_11_lamma2_run

Step 3: Install library



pip install transformers torch accelerate

Step 4: Create a file "touch run.py"



import time

import torch

from transformers import AutoTokenizer, AutoModelForCausalLM

timeStart = time.time()

tokenizer = AutoTokenizer.from_pretrained(

    "meta-llama/Llama-2-7b-chat-hf"

)

model = AutoModelForCausalLM.from_pretrained(

    "meta-llama/Llama-2-7b-chat-hf",

    torch_dtype=torch.bfloat16,

    low_cpu_mem_usage=True,

)

print("Load model time: ", -timeStart + time.time())

while(True):

    input_str = input('Enter: ')

    input_token_length = input('Enter length: ')

if(input_str == 'exit'):
    break

timeStart = time.time()

inputs = tokenizer.encode(
    input_str,
    return_tensors="pt"
)

outputs = model.generate(
    inputs,
    max_new_tokens=int(input_token_length),
)

output_str = tokenizer.decode(outputs[0])

print(output_str)

print("Time taken: ", -timeStart + time.time())

Step 5: Run python file



python run.py

Performance:

I am using a CPU with 20 GB of RAM (4 GB + 16 GB).
It took 51 seconds to load the model and 227 seconds to generate a response for 250 tokens.
If you use a GPU, it will take significantly less time.
On Google Colab, i got 16 second for a response.

Congratulations! You have successfully run llama on local machine.

Local Shop Search - Search shop near you

Nithin I Bhandari — Tue, 11 Jan 2022 18:43:44 +0000

Local Shop Search is used to search a shop near you by a map.
Shop keeper can add a shop and list his product.
End-User can Search Products and can visit his shop to purchase the product.

Live Website: https://local-shop-search.netlify.app/search/

Overview of My Submission

Tech Stack

MongoDB and MongoDB Autocomplete
Node JS
Express JS
React JS
Leaflet JS
Netlify for frontend and Netlify serverless backend.

How search functionality works

Here i am using MongoDB autocomplete nGram which offer autocomplete search and location index to search query by location.

Aggregate query example

Code Link

MongoDB Autocomplete index