Running and Creating Your Own LLMs Locally with Node.js API using Ollama

In this guide, you'll learn how to run Large Language Models (LLMs) on your local machine and create your own LLM. We'll also cover how to create an API for your custom model using the ollama-js library in Node.js.

Step 1: Installing Ollama

Ollama is an ideal choice for running LLMs locally due to its simplicity and compatibility with non-GPU intensive machines. Start by installing Ollama from the official website:

Ollama Official Site

Step 2: Selecting Your Preferred LLM Model

After installing Ollama, you can choose from a variety of LLM models available. You can find the list of available models on their GitHub repository:

Ollama GitHub Repository

Step 3: Running the Model Locally

To run the model locally, use the following command in your terminal. Note that the first run might take longer as Ollama downloads and stores the model locally. Subsequent runs will be faster since the model is accessed locally.

ollama run {model_name}

Step 4: Creating Your Own LLM

To create your custom LLM, you need to create a model file. Below is an example of how to define your model:

FROM <name_of_your_downloaded_model>

# Define your parameters here
PARAMETER temperature 0.5

SYSTEM """
You are an English teaching assistant named Mr. Kamal Kishor. You help with note-making, solving English grammar assignments, and reading comprehensions.
"""

Save this as modelfile. To create the model from this file, run the following command in your terminal:

ollama create mrkamalkishor -f ./modelfile

After creating the model, you can interact with it locally using:

ollama run mrkamalkishor

Step 5: Creating a Node.js API for the Custom Model

For this step, we will use the ollama-js library to create an API in Node.js.

1. Install the Ollama library in your Node.js project:

npm install ollama

1. Create your API endpoint:

import express from 'express';
import ollama from 'ollama';

const app = express();
const router = express.Router();

app.use(express.json());

router.post('/ask-query', async (req, res) => {
  const { query } = req.body;

  try {
    const response = await ollama.chat({
      model: 'mrkamalkishor',
      messages: [{ role: 'user', content: query }],
    });

    res.json({ reply: response.message.content });
  } catch (error) {
    res.status(500).send({ error: 'Error interacting with the model' });
  }
});

app.use('/api', router);

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

This code sets up an Express.js server with an endpoint to interact with your custom model. When a POST request is made to /ask-query with a JSON body containing the user's query, the server responds with the model's output.

Summary

By following these steps, you can install Ollama, choose and run LLMs locally, create your custom LLM, and set up a Node.js API to interact with it. This setup allows you to leverage powerful language models on your local machine without requiring GPU-intensive hardware.

Comments

[Sedat KPunkt]

Thanks for sharing your experiences with ollama and the short well explained introduction.
Now to my question:
I came here for "creating your own LLM" which you're explaining in step 4.
But it looks rather like a typical LLM prompt to my noob eyes sparking questions like
"does it really modify/extend the existing LLM model?"
I wonder what the limits of such prompts are.
Is it possible to "train" such LLMs with any further knowledge about anything,
e.g. own write-ups, notes or a colourful story about the village of my parents?
Long story short:
Is it possible to extend those models with any knowledge which they weren't trained on before?
PS: Let's see if your other Ollama articles give me more insight!
Thanks…this article was fire

[Sedat KPunkt]

BTW…I've found the answers to all my questions and even more on Ollama's README…!
Thanks you for directing my attention to that project!