DEV Community: Zakhar Smirnoff

A better way to handle errors

Zakhar Smirnoff — Wed, 12 Feb 2025 08:38:00 +0000

Introduction

Hi, I guess! It's been a while since I have posted something in here. Recently I've been dealing a lot with libraries in TypeScript which involve lots of remote dependencies such as database connections or http requests You know how it's annoying to wrap everything in try-catch, especially when you don't even know in advance whether a certain function throws or not! Actually, I really love how Go handles errors:

f, err := os.Open("filename.ext")
if err != nil {
    log.Fatal(err)
}
// do something with the open *File f

It returns two values, and you should always check whether the error value is not nil. That's neat and disciplined. In Go, you are always aware of potential errors and you always handle them. Go does not use exceptions at all.

Yet I also love the elegance and flexibility of TypeScript and I started to think how I could achieve something similar (at least remotely) in this language.

Always check

That's how I came to these functions:

async function checkAsync<Args extends any[], Type>(
  fnToCheck: (...args: Args) => Promise<Type>,
  ...args: Args
): Promise<Type | Error> {
  try {
    const result = await fnToCheck(...args);
    return result;
  } catch (e) {
    return new Error(String(e), { cause: e });
  }
}

function check<Args extends any[], Type>(
  fnToCheck: (...args: Args) => Type,
  ...args: Args
): Type | Error {
  try {
    const result = fnToCheck(...args);
    return result;
  } catch (e) {
    return new Error(String(e), { cause: e });
  }
}

It differs from Go approach, but still does the job: instead of try-catch, you just have to check your function, and you get a Type | Error. Then you check for the error, and you are good to go! Essentially it's just a try-catch wrapper which doesn't throw but instead returns errors as values. Of course, it doesn't know whether some functions throw or not (apart from the functions defined by you), but this is still an improvement.

I have published this as an npm package, it works with all runtimes and it's type-safe.
Here is the repo link: https://github.com/zakharsmirnoff/always-check
Here is the npm link: https://www.npmjs.com/package/always-check

Feel free to discuss in the comments, maybe I'm missing something or maybe there is some functionality you might want to add?

I have optimized my work with text-to-SQL translator improved with vector search

Zakhar Smirnoff — Mon, 05 Feb 2024 15:12:35 +0000

Intro

I have always struggled with SQL syntax. Well, not exactly struggled, it's pretty simple, but I just dislike those weird joins, excessive capitalization, and those pesky ORMs that only make things worse. Wouldn't it be great if SQL syntax was as universal as data structures?

Fortunately, with the emergence of modern AI technologies, especially LLM, we can now write natural language queries and directly ask the database in plain English or even in your native language! Well, at least we can try. In this article, I will discuss the different approaches attempted by the community and share two public repositories containing my projects.

Naïve approach

Perhaps the first idea that comes to mind when trying out ChatGPT is simply prompting it to translate text into SQL. It's as simple as that:

Please translate this to SQL and output only SQL code: show all customers

However, this approach may not work in 80-90% of cases when the table name is not something that made up. Is it "Customer," "Customers," or "customer"? To solve this issue, we need to consider the

Schema

To address the problem, we can easily retrieve the schema using our preferred method. For example, here's what I do for SQLite:

SELECT name, sql
FROM sqlite_master 
WHERE type='table';

Once we have the schema, we can modify the prompt:

You should translate everything to SQL queries. For assistance, here is the schema: {schema}

Please translate this to SQL and output only SQL code: show all customers

I have experimented with this setup, and it has worked well with almost any database. About 8 out of 10 queries have been accurate. I haven't conducted any benchmarks or measurements yet, but so far it looks promising! However, generating queries might be costly, especially when every interaction with a new database begins with simple queries like "show all tables" or "show all customers." What if we could memorize some of the most frequent queries?

Semantic search

What is this and why? There is plenty of information available online for a quick introduction. For example, you can refer to this article from Pinecone. Or, since I use Weaviate in my project, here is the link to their nice blog post about vectors, embeddings and vector search. Semantic search focuses on the meaning rather than just string matching. Now, let's integrate our translator with the vector database! After generating a query, we store the question-query pair in our database:

{
"question": "show all tables",
"query": "SELECT name from FROM sqlite_master WHERE type='table';"
}

When a new query comes in, such as "all tables" or "names of tables," our application first searches the database for similar queries and questions. If none are found, it generates a new one and stores it in the database. This ensures that our configuration learns and enriches itself with new queries and questions. For example, let's say our team of 2-3 users retrieves information every day. Over time, there will be fewer and fewer new generations as there are more similar queries. I haven't conducted extensive tests to know if my hypothesis really holds in the long run, apart from the Chinook sample questions. Nonetheless, the concept appears promising!

Implementation

Based on the aforementioned idea, I have developed two projects. The first is an API service written in Go with Weaviate on the backend, while the second is a prototype built rapidly in Python with ChromaDB to demonstrate the hypothesis. The first project is likely to be supported and improved, so you're welcome to participate if you're interested! Please note that both projects are highly experimental, so don't expect a production-ready solution. However, considering that vector databases are extremely fast, the first project should perform exceptionally well with the appropriate deployment. Feel free to test it out with your databases!

Inside the repositories you will find a quick video demo

Python version: https://github.com/zakharsmirnoff/charm-query-py
Go version: https://github.com/zakharsmirnoff/charm-query

Thank you for reading this! If you like the idea, please favorite the repository on GitHub, react to this post, and leave your comments below. I would be incredibly happy to hear your thoughts on the concept.

I was tired of Langchain and created my own wrapper

Zakhar Smirnoff — Tue, 15 Aug 2023 09:46:27 +0000

Hello everyone! In this post I would like to share with you my recent development on my very own tiny wrapper for OpenAI API.

Motivation

Nowadays, every programmer who wants to build a production-ready LLM application, inevitable stumbles upon certain libraries such as LangChain, Semantic Kernel, Guidance etc. Then a lot of developers quickly grow frustrated and disappointed in their stack choice. Why? Because all these tools introduce unnecessary abstractions, add complicated logic and hide prompts they are using. As a result, the development starts to resemble a very tedious task even for simple applications.

In fact, you don't need anything to work with existing Open AI API. You don't even need any wrappers, all you need to get started is just using your own logic, reasoning and naked API calls via your preferred programming language. I created the wrapper based on my experience and my current tasks and thought it would probably be useful for a larger audience.

Concept

I had some requirements in mind which I tried to convey in code:

No hard-coded prompts. It's just a text, if you feel like your prompts are not working, there are thousands of them over the internet available for free
Conversation history should be stored by default. This is a chat model anyway, so it's a good idea for the conversation history to be stored as a dictionary for consistent experience. It's easier to implement a method to clear the history rather than constructing the history
Token count validation. Tokens should be counted and if the amount exceeds the available quota, the program should reduce the messages or at least produce a warning or an error
No hard-coded complicated logic for any chains/agents etc. It should implement only basic methods, while anything else should be done by the developer. Later below I will show that it's quite easy

Lloom

With concepts listed above, I have implemented and released "lloom" version 0.0.1: https://github.com/zakharsmirnoff/lloom. It's available via pip, the documentation for methods can be found on github. Here I would like to give some examples in comparison with Langchain.

Examples

One of the main use cases for GPT-powered applications is summarization. I studied this page in Langchain docs and implemented the same logic using my wrapper:

from lloom import LloomConfig, Lloom
from pypdf import PdfReader
import re

# I'm using PdfReader package for processing PDF docs, the pdf here is a paper available on arxiv
reader = PdfReader("virtual_injection.pdf")

config = LloomConfig(your_api_key, model="gpt-3.5-turbo-0613")
lloom = Lloom(config)

answers = []

# A minimal helper function to achieve a stuff chain. I know it's not exactly stuff chain as in Langchain documentation, but the concept is similar: putting the whole document into the prompt. I just split the PDF by pages
def one_shot(text):
    res = loom.generate(f"Write a concise summary of the following: {text}")
    answers.append(res)
    loom.clear_history()

# The code below took approximately 1.5 minutes to execute, as a result we got a list with gpt-generated summaries. I chose only 11 first pages, since the rest is examples and references
for page in reader.pages[0:11]:
    text = page.extract_text()
    one_shot(text)

# This single line of code along with the function above implements a map-reduce chain
summary = loom.generate(f"Write a short and concise summary for these pieces of text: {', '.join(answers)}")

print(summary) # it gave me this: 

'''
The papers introduce the concept of Virtual Prompt Injection (VPI) as a method to manipulate the behavior of Large Language Models (LLMs) without directly modifying the model input.
VPI allows an attacker to control the model's responses by specifying a virtual prompt, leading to biased views and potentially harmful outcomes.
The papers propose a method for performing VPI by poisoning the model's instruction tuning data and demonstrate its effectiveness in steering the LLM's behavior.
They emphasize the importance of ensuring the integrity of instruction tuning data and suggest data filtering as a defense against poisoning attacks.
The effectiveness of VPI is evaluated in various scenarios such as sentiment steering and code injection, with comparisons to baseline methods and different model scales.
The papers also discuss defense mechanisms and the need for further research in this area to develop better defense mechanisms against VPI attacks.
The limitations of the study are acknowledged, and the authors emphasize the importance of studying vulnerabilities in instruction-tuned language models to enhance security measures.
'''

You can check other examples of summarization in the repo (to be added later actually)

Let's head to the agent example. Below you can see the implementation of DnD for one player:

from lloom import LloomConfig, Lloom

protagonist_name = "Storm Ryder"
storyteller_name = "Captain Quill"
quest = '''Set sail on the "Marauder's Dream" with your unique crew, seeking the fragmented map leading to the legendary "Mythic Isles" and the coveted "Seafarer's Heart" artifact. Battle fierce rivals, sea monsters, and unravel hidden histories while forging unbreakable bonds. A thrilling quest inspired by One Piece awaits with treasure beyond gold adventure, camaraderie, and freedom.'''
protagonist_description = "Storm Ryder, a daring and enigmatic adventurer, possesses the heart of a true pirate. Guided by an unyielding sense of justice and fueled by an insatiable thirst for adventure, Storm sails the uncharted seas, leaving a legacy of courage and camaraderie in their wake."
storyteller_description = "Captain Quill, a weathered and charismatic storyteller, carries tales of ancient legends and forgotten myths in their ink-stained logbook. With a twinkle in their eye and a voice that mesmerizes, they narrate the epic quest of Storm Ryder and the fabled Mythic Isles, inspiring awe and wonder in all who listen."

# I have already taken the ready-made system message from langchain docs, but the same functionality of so-called specifying the description can be implemented in 3 lines of code if needed
player_sysmsg = f'''
Here is the topic for a Dungeons & Dragons game: {quest}.
There is one player in this game: the protagonist, {protagonist_name}.
The story is narrated by the storyteller, {storyteller_name}.
Never forget you are the protagonist, {protagonist_name}, and I am the storyteller, {storyteller_name}. 
Your character description is as follows: {protagonist_description}.
You will propose actions you plan to take and I will explain what happens when you take those actions.
Speak in the first person from the perspective of {protagonist_name}.
For describing your own body movements, wrap your description in '*'.
Do not change roles!
Do not speak from the perspective of {storyteller_name}.
Do not forget to finish speaking by saying, 'It is your turn, {storyteller_name}.'
Do not add anything else.
Remember you are the protagonist, {protagonist_name}.
Stop speaking the moment you finish speaking from your perspective.
'''

master_sysmsg = f'''
Here is the topic for a Dungeons & Dragons game: {quest}.
There is one player in this game: the protagonist, {protagonist_name}.
The story is narrated by the storyteller, {storyteller_name}.
Never forget you are the storyteller, {storyteller_name}, and I am the protagonist, {protagonist_name}. 
Your character description is as follows: {storyteller_description}.
I will propose actions I plan to take and you will explain what happens when I take those actions.
Speak in the first person from the perspective of {storyteller_name}.
For describing your own body movements, wrap your description in '*'.
Do not change roles!
Do not speak from the perspective of {protagonist_name}.
Do not forget to finish speaking by saying, 'It is your turn, {protagonist_name}.'
Do not add anything else.
Remember you are the storyteller, {storyteller_name}.
Stop speaking the moment you finish speaking from your perspective.
'''
master_config = LloomConfig(your_api_key, temperature=1.0, logging=False, model="gpt-3.5-turbo-0613",system_message=master_sysmsg)
player_config = LloomConfig(your_api_key, temperature=1.0, logging=False, model="gpt-3.5-turbo-0613", system_message=player_sysmsg)

master = Lloom(master_config)
player = Lloom(player_config)

max_iter = 3
initial_message = "I'm ready to start!"
n = 0

# as the history is stored by default, we don't need to implement any additional classes or methods
while n < max_iter:
    master_message = master.generate(initial_message)
    print(f"Master message: {master_message}")
    player_message = player.generate(master_message)
    initial_message = player_message
    print(f"Player message: {player_message}")
    n += 1

If you check the Langchain docs, you will see that their implementation is at least of the same length, but usually they need more code. Same is true for Semantic Kernel. Thank you for reading! If you like my work, please star the repository and share your use cases, would be excited to learn new ways!

P.S. In no means I'm trying to make a competition with LangChain and similar libraries, and no hate either! Langchain and others might be a good choice in some cases, I just found out that simpler implementation works best for me.

Asked ChatGPT and other models to write a simple script

Zakhar Smirnoff — Fri, 09 Jun 2023 12:43:05 +0000

Hello! I continue my journey in the world of deep learning, ai and language models.

I was once tasked to write a simple PowerShell script, that gets all groups from Azure Active Directory and extracts all the members from the groups. The trick here, is that there might be nested groups!

For an average system administrator or coder it's a trivial task: just write a recursive function, then get all the groups and process via that function. I was interested in how well our language models can accomplish this basic task.

For all models I was using default settings and the same prompt. I removed all the comments generated by model and added mine, evaluating the solution. The results are not so obvious I must say! Let me present THE PROMPT:

Please write a PowerShell script which gets all groups from Azure AD, then iterates over all of the groups and extracts all members. Some of the groups can have nested groups. Members of those should be extracted as well.

Let's now check the outputs of our models:

GPT-4

Import-Module AzureAD

# That's good, a separate function to get group members recursively is nice
Function Get-GroupMembers ($GroupId) {
    $members = Get-AzureADGroupMember -ObjectId $GroupId | Where-Object {$_.ObjectType -eq "User"} # Correct! ObjectType is the property we are looking for, though no -All $true! Bad!

    foreach($member in $members){
        Write-Host "Member DisplayName: $($member.DisplayName), UserPrincipalName: $($member.UserPrincipalName)"
    }

    $nestedGroups = Get-AzureADGroupMember -ObjectId $GroupId | Where-Object {$_.ObjectType -eq "Group"}

    foreach($nestedGroup in $nestedGroups){
        Write-Host "Nested Group: $($nestedGroup.DisplayName)"
        Get-GroupMembers -GroupId $nestedGroup.ObjectId
    }
}

# Although usually I'm already connected, it's nice that GPT-4 suggested to handle the exception. Yet it could make it better because right now it gives you this which is funny:
# Get-AzureADUser : You must call the Connect-AzureAD cmdlet before calling any other cmdlets.
try {
    if(-not (Get-AzureADUser)) { Connect-AzureAD }
} catch {
    Write-Host "Failed to connect to Azure AD" -ForegroundColor Red
    exit
}

$groups = Get-AzureADGroup # It forgot to add -All $true, without it we get only first 100. Bad!

# Iterate over each group
foreach($group in $groups){
    Write-Host "Group: $($group.DisplayName)"
    Get-GroupMembers -GroupId $group.ObjectId
}

Overall, very good! Almost a working solution from the very first attempt.

Pros: the logic and cmdlets are correct

Cons: it didn't get ALL groups and ALL members although I specified this in the prompt. Also it just printed out the info. It could be nice to have an array which I could turn into CSV later, but that's fine, didn't ask for it in the prompt. Actually, it did tell me after writing the script: This script simply writes the details to the console. If you want to save the details to a file or do something else with them, you will need to modify the script accordingly.

GPT-3.5

# Nice! That's a simplier solution comparing to one provided by gpt-4. At least it works!
Connect-AzureAD

# It specified all! Already better than gpt-4
$groups = Get-AzureADGroup -All $true

# I hate this param stuff in PS but the logic is correct
function Get-NestedGroupMembers {
    param (
        [Parameter(Mandatory=$true)]
        [string]$GroupId
    )

    $nestedGroupMembers = Get-AzureADGroupMember -ObjectId $GroupId # forgot all here

    foreach ($nestedGroupMember in $nestedGroupMembers) {
        if ($nestedGroupMember.ObjectType -eq "Group") {
            Get-NestedGroupMembers -GroupId $nestedGroupMember.ObjectId # yep, that's okay
        } else {
            $nestedGroupMember
        }
    }
}


foreach ($group in $groups) {
    Write-Host "Group Name: $($group.DisplayName)"
    Write-Host "Members:"

    $groupMembers = Get-NestedGroupMembers -GroupId $group.ObjectId

    foreach ($groupMember in $groupMembers) {
        Write-Host "- $($groupMember.UserPrincipalName)"
    }

    Write-Host
}

Disconnect-AzureAD

I must say, it's already quite better that gpt-4 for some reason. At least we got ALL groups (but not all members)

Pros: it connects you to azure ad correctly and the logic is correct

Cons: again, not writing to any array and not getting all the members

Koala 13B

# Wrong from the start. We are in cloud dude, this cmdlet is for on-prem active directory
$groups = Get-ADGroup -Filter *

foreach ($group in $groups) {
    $members = Get-ADGroupMember -Identity $group.Name

    foreach ($member in $members) {
        # why do you need to check if the user is external or not?
        if ($member.MembershipType -eq "External") {
            $email = $member.EmailAddress

            # I don't understand the logic at all. It creates an array, then add something weird and on line 15, it checks whether the array is empty or not. What?
            $isNested = @()
            $isNested += $member.GroupName -like "$($group.Name)*"
            if ($isNested) {
                $nestedGroups = Get-ADGroupMember -Identity $member.GroupName | Where-Object {$_.MembershipType -eq "External"}
                foreach ($nestedGroup in $nestedGroups) {
                    $email += ",$($nestedGroup.EmailAddress)"
                }
            }

            Write-Output "$email"
        }
    }
}

Very bad in general. I will not write pros and cons as I really don't understand, what it's trying to do.

Nous-Hermes 13B

$groups = Get-ADGroup -Filter * | Select-Object Name | Sort-Object Name 

    foreach ($group in $groups) {
        $members = Get-ADGroupMember -Identity $group.Name -Recursive | Where-Object {($_.ObjectClass -eq 'user')} | Select-Object SamAccountName

        Write-Output "$group.Name has the following members: $(($members -join ', '))"
    }

Again, Active Directory is used not Azure Active Directory. But I checked, Get-ADGroupMember has a parameter -Recursive, I can't try it out but probably it works. Anyway, not for my needs

StarChat-beta

# good, getting aaaaal the groups
$groups = Get-AzureADGroup -All $true 

foreach ($group in $groups) {
    Write-Host "Processing group: $($group.DisplayName)"

    $members = (Get-AzureADGroupMember -ObjectId $group.ObjectID).Value # I don't know what it does, the Value is empty

    foreach($member in $members){
        if(-not $processedMembers.Contains($member)){
            $processedMembers += $member # this variable was never declared nor initialized, so I assume it's just null, not sure

            $isGroupMember = ((Get-AzureADUser -ObjectId $member).memberof | Where-Object{$_ -like "*$($group.displayName)*"} ) # so it assumes that nested group should have a similar name? 
            if($isGroupMember){
                Process-NestedGroups -GroupId $member
            }else{

                $outputMembers += $member
            }
        }
    }
}


function Process-NestedGroups([string]$groupId){
    $nestedGroups = (Get-AzureADGroupMember -ObjectId $groupId ).value 

    foreach ($ngroup in $nestedGroups) {
        if (-not $processedGroups.contains($ngroup)) {
            $processedGroups += $ngroup
            Process-NestedGroups -GroupId $ngroup
        }
    }

    $nestedGroupMembes= get-azureadgroupmember -objectid $ngroup.objectId | select -ExpandProperty id
    $outputMembers+=$nestedGroupMemebrs
}

Write-Output "Total Number of Groups:" $groups.count 
Write-Output "Number of unique members:" ($outputMembers| Measure-Object -Unique).Count # the script fails here, no unique property
write-host "List of Unique Members"
$outputMembers | sort-object -unique # even if you get there, you will get emptiness

Overall, the logic seems to be almost there, but the script is overcomplicated and doesn't actually achieve what it's supposed to

Anthropic Claude

# why in the world you forget to add -all
$groups = Get-AzureADGroup

foreach ($group in $groups) {
    $members = Get-AzureADGroupMember -ObjectId $group.ObjectId

    foreach ($member in $members) {
        if ($member.ObjectType -eq "Group") {
            $nestedMembers = Get-AzureADGroupMember -ObjectId $member.ObjectId
            $members += $nestedMembers
        }
    }

    Write-Host "Group: $($group.DisplayName)"
    $members | Format-Table

    $members = $null # why?
}

A very neat solution! It worked from the very first run.

Pros: less code, easy-to-read

Cons: not all groups, it didn't write to the variable, just printed to the screen

Conclusion

First of all, commercial models are way better at this. Secondly, sometimes even smaller and less powerful models can produce a more readable and better code.

Thank you for reading! If you like this post, please check my other two posts on how to work with LLaMa-based models. See you!

Running language models on AWS

Zakhar Smirnoff — Thu, 08 Jun 2023 12:52:28 +0000

The goal

Hi all! In my previous post I wrote about my experience running LLaMa model at my personal MacBook. Soon I started to want more than this, so I decided to spin up an instance on AWS to try out more powerful models there! I will give you a complete guide which should work for everyone. It will cost something, but should fit in a very tight budget (around 10 bucks for one try). The guide assumes that you have a very basic knowledge of cloud stack or AWS. Nevertheless, I tried to make it very complete for everyone.

AWS account

If you don't have an AWS account, it's pretty straightforward to create. Just click Sign Up on that home page and follow the instructions. Then sign in to the console and you should see the home page. I advise you set up MFA for your root account immediately after creating an account. Follow the instructions provided here

Creating a EC2 instance

When you are on the home page, right on the top just search for "EC2" That's a weird (like everything else) name for "virtual machines" on AWS.

Then on the left go to Instances, then on the right find an orange button Launch Instances.

You should see something like this:

Here you should input the following details:

Name and tags: any name you prefer
Application and OS Images: select Ubuntu then in a drop-down menu select Ubuntu Server 22.04 or 20.04. Make sure you chose 64-bit(x86)
Instance type: it's one of the most important parts, where you select the CPU and RAM for your machine. I use r5d.2xlarge with 8 CPUs and 64GM ram (which should be enough even for very large models), but you can select anything you want. Just make sure that the type has enough resources to run whatever you want to run
Key pair: that's for SSH connection. You can select an existing one, or create a new one. It will download the private key to your computer, later you can use it to access the machine remotely via SSH
Network settings: For the sake of simplicity, you can leave the default settings, like this:

Of course, if you understand what you are doing, feel free to implement routing and security rules to your liking

Configure storage: you can expand your root volume to make it more spacious, or attach a new volume. I use a separate volume with 150GB. Keep in mind, volumes cost something like $0.08 GB/month. Before proceeding, please check the exact pricing for your type of volume
Advanced details: the most interesting part. Here you can check "Request Spot Instances". This is a very cheap options to rent AWS unused resources (up to 90% discount). It comes with a small disadvantage: at any time provisioned resources can be revoked, and your instance will be shutdown. In reality, it happens very rarely, so if you don't plan on running the instance 24/7, feel free to choose this option. Here you can see my setup:

Make sure, request type is set to Persistent, otherwise you won't be able to stop the instance.

Aaaand, that's it! Just click Launch an instance and in 1 or 2 minutes your VM will be ready!

We are almost ready. The initial goal was running llama.cpp compatible models, so let's get there

Configuring VM

By default, VM is pretty clean and empty hence we have to install some tools in order to work with it.

First, we need to get into the machine. There are various means to achieve it, we will use the easiest one: EC2 Instance Connect.

First, click on your newly created instance and on the right click Connect. You will see the window with different connecting methods, where you should also click connect under EC2 Instance Connect tab. That's it, it will load a terminal window with SSH session.

If you attached a volume storage (not a root volume), then you have to mount it and make it persistent across the runs:

Check the location of the NVMe drive, running this in shell:

lsblk

Format it with the following command (all data on this drive will be lost), replacing /dev/nvme1n1 with your actual volume name:

sudo mkfs -t ext4 /dev/nvme1n1

Create a directory (here it's /data) and mount the drive:

sudo mkdir /data 
sudo mount /dev/nvme1n1 /data

To make the mount persist across reboots, add the following entry to /etc/fstab: /dev/nvme1n1 /data ext4 defaults,nofail 0 0 For example:

sudo nano /etc/fstab

Then paste /dev/nvme1n1 /data ext4 defaults,nofail 0 0, press Ctrl+X and type Y. It will save and close the file.

Validate the /etc/fstab entries:

sudo mount -a

It should not produce any errors. If it does, then it's better to clear the file and troubleshoot. But usually, it should be fine.

Now you have your storage mounted at /data. By default only root user (sudo) has permissions to this storage, hence we need to make it writable for everyone:

sudo chmod 777 /data

We will need some build tools to compile llama.cpp tool therefore you need to run this:

sudo apt install build-essential

We are done now! Now you have a complete virtual machine ready to run almost any large language model. Don't forget to stop your instance when you are done working with it. You are charged only for running time and storage.

If you want to have a constant DNS and IP, you should use Elastic IP. Otherwise, DNS and IP change every time you restart the instance.

Now all you have to do is to clone llama.cpp, compile it, download the model to your VM and start chatting! To do this, you can follow the steps outlined in my previous blog post.

If you have any questions or issue, feel free to ask anything in the comment section!

How to run LLaMa at home

Zakhar Smirnoff — Mon, 05 Jun 2023 13:52:04 +0000

If you are interested in running language models on AWS, please take a look at my other post: Running language models on AWS

What is LLaMa?

You've probably heard of or even used ChatGPT. ChatGPT and similar neural networks are called LLMs (large language models). LLaMa is one such model, developed by Meta AI (formerly Facebook) for research purposes, with a license that isn't suitable for commercial use. You can read more about it here: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/

What is this post about?

Shortly after the release of LLaMa, enthusiasts started exploring this suite of models, training and creating new ones. As a result, we now have Alpaca, Koala, Vicuna, StableVicuna, and many more. In this blog post, I'll describe my experience of trying to run a LLaMa model with 7 billion parameters on my 2018 MacBook 12".

Getting weights, converting and quantizing

All these models require a GPU by default to handle all these complex computations. I don't have any PCs with this kind of GPUs so I started exploring other ways. Right now there is one agreed approach to run open-source models at home: using llama.cpp. This is a freely available software, which converts models to ggml format (just another format for neural nets, which requires CPU and RAM instead of GPU), then quantizes them to make them runnable (is that even a word?) even on lower specs. I really struggled throughout the whole process, but as they say, no pain no gain! This guide isn't a one-size-fits-all solution, but rather the approach that worked for me.

Okay, let's start:

Download the weights. Actually, the weights (also called parameters, or even for simplicity, the model itself) should be requested from Meta but it seems they no longer provide them to anyone. No weights, no fun! Luckily, we have many torrents floating around that can download the models pretty quickly. I will not share any links here (and please, refrain from sharing anything in the comment section) but if you google "How do I obtain Llama weights" probably the first result would lead you to the GitHub discussion where you can find the magnet link. Just grab it and put into your favorite Bittorrent client. Select the model which you would like to download (there are 7B, 13B, 33B and 65B, the larger the better). I used the smallest one.
Converting the weights and quantizing the model. As I said, we will use llama.cpp: https://github.com/ggerganov/llama.cpp. Just scroll down to the usage section and follow the instructions. Esentially, you should clone the repo, make (compile) the binary program, put your downloaded model to models folder, convert to ggml FP16 format, then quantize to 4-bit. You will only need section "Build" and "Prepare Data and Run". After this, you should get a .bin file in your models folder, which is the model itself! Then you can just run it and communicate with in your terminal:

./main -m ./models/7B/ggml-model-q4_0.bin -n 128

For the steps above, you should have Python installed (should already come pre-installed with macOS and most Linux distros). I recommend you use miniconda to manage different python environments.

Actually I thought this blog post would have been a lot longer. However in reality, all my struggles were solved with one or two commands, solution for which took forever to find. I decided to cover some additional points:

Are there some bindings for llama.cpp?

Yes, there are some bindings for different languages. The list of supported ones is listed in llama.cpp repository. Binding is just a convenient library to call methods against the models. For example, in Python you could run something like this:

response = llm(prompt="What is the meaning of life?")

Is there a UI available for interaction?

Yes, llama.cpp repo points out to 2 different UI solutions. Neither of them worked for me. Maybe, I'm too dumb, anyway I just skipped this and developed my own very basic UI solution with streamlit and llama-cpp-python binding. If you want me to share the process and how it works, feel free to let me know in the comments!

What is quantization?

Let's ask ChatGPT! "LLMs, such as GPT-3, have a vast number of parameters (175 billion in the case of GPT-3), making them quite resource-intensive. For instance, GPT-3 in FP16 format consumes at least 350GB of memory to store and run. Quantization addresses this problem by representing the weights and activations of the model with low-bit integers. This can significantly reduce GPU memory requirements and accelerate compute-intensive operations like matrix multiplications. For example, using INT8 (8-bit integer) quantization can halve the GPU memory usage and nearly double the throughput of matrix multiplications compared to FP16". In other words, you will get a way smaller model, but not that impressive in performance although generally it's not noticeable.

What are the tech requirements to run this?

See here: https://github.com/ggerganov/llama.cpp#memorydisk-requirements

Why do I need this if I have ChatGPT?

Just for fun! For real tasks I use GPT-4 and Claude. Yet these models are so much fun to play with! I have only tried LLaMa 7B and Koala 7B, and planning to try Alpaca, GPT4ALL, Falcon and other models! Anyway, the models are made for researching so let's do it!

I would be happy to answer any questions you have (if I know the answer, of course)