In this article, we will learn how to run Llama-3.1 model locally on our PC using Ollama and LangChain in Python
Outline
- Install Ollama
- Pull model
- Serve model
- Create a new folder, open it with a code editor
- Create and activate Virtual environment
- Install langchain-ollama
- Run Ollama with model in Python
- Conclusion
Install Ollama
Follow the instructions based on your OS type in its GitHub README to install Ollama:
https://github.com/ollama/ollama
I am on a Linux-based PC, so I am going to run the following command in my terminal:
curl -fsSL https://ollama.com/install.sh | sh
Pull model
Fetch the available LLM model via the following command:
ollama pull llama3.1
This will download the default tagged version of the model. Typically, the default points to the latest, smallest sized-parameter model. In this case, it will be llama3.1:8b model.
To download another version of the model, you can go to: https://ollama.com/library/llama3.1 and select the version to install, and then run the ollama pull command with the model and its version number. Example: ollama pull llama3.1:70b
On Mac, the models will be downloaded to ~/.ollama/models
On Linux (or WSL), the models will be stored at /usr/share/ollama/.ollama/models
Serve model
Run the following command to start ollama without running the desktop application.
ollama serve
All models are automatically served on localhost:11434
Create a new folder, open it with a code editor
Create a new folder on your computer and then open it with a code editor like VS Code.
Create and activate Virtual environment
Open the terminal. Use the following command to create a virtual environment .venv and activate it:
python3 -m venv .venv
source .venv/bin/activate
Install langchain-ollama
Run the following command to install langchain-ollama:
pip install -U langchain-ollama
The above command will install or upgrade the LangChain Ollama package in Python. This package allows users to integrate and interact with Ollama models, which are open-source large language models, within the LangChain framework. The -U flag ensures that the package is upgraded to the latest version if it is already installed.
Run Ollama with model in Python
Create a Python file for example: main.py and add the following code:
from langchain_ollama import OllamaLLM
llm = OllamaLLM(model="llama3.1")
response = llm.invoke("The first man on the moon was ...")
print(response)
The above code imports the OllamaLLM class from the LangChain library and initializes an instance of the language model "llama3.1". We pass a prompt about the first man on the moon, and store the generated response in the variable response. When we run the above code we get the following response from the model:
...Neil Armstrong!
On July 20, 1969, Neil Armstrong became the first person to set foot on the lunar surface during the Apollo 11 mission. As he stepped off the lunar module Eagle and onto the moon's surface, he famously declared: "That's one small step for man, one giant leap for mankind."
Conclusion
Thanks for reading.
You can view the Ollama documentation for more commands.
Top comments (1)
Thank you :-)