DEV Community: Ray

Locally running LLaMA2 'finish_reason': 'length'

Ray — Tue, 27 Feb 2024 20:35:48 +0000

A very small problem-solving share about running the LLama model locally!

Previously I used the library provided by llama-cpp-python directly to build the chat model for my local deployment of LLaMA and focused mostly on issues such as RAG model building, data preprocessing, and M1 Mac GPU acceleration. There was no patience to work on the quality and setup issues of the basic language-generated models.

Today I was reading the GitHub documentation for llama-cpp-python to try out the simple directed chat architecture function llm.create_chat_completion(), but suddenly I realized that on my M1 Mac, the model inference would suddenly stop as if the words were not finished.

This problem has been encountered often before when I was building the LLM backend and frontend chat interface projects.

The solution I found before was to keep changing the
n_ctx, n_batch, n_threads, n_parts, n_gpu_layers`

Even adjusting the max token to a different value such as 512 can sometimes fix the problem.

According to author abetlen and GitHub user zhangchn
The __call__ of Llama has an unfortunate default value of 16, which was changed to be consistent with the OpenAI API, but I didn't realize until later that a lot of people call the completions function without setting max_tokens. tokens.

So to avoid the situation in the image shown above, where the generation of text suddenly stops, we can override this value by setting max_tokens to -1 when we call llm for inference generation to avoid this situation.

Google Gemma first try

Ray — Thu, 22 Feb 2024 21:37:29 +0000

Kia Ora! Everyone!

Yesterday, Google released Hugging Face's Gemma Large Language Model, which is Gemini, but now developers can use Hugging Face to do more research or development, such as fine-tuning the model or calling RAGs and other ways to customize the LLM output.

Personally, I don't think Gemma is targeting ChatGPT, but more like Llama2 as the main competitor to capture the market of low configuration or thin and light office locally deployed large language modeling assistants.
Without further ado, today we're using Kaggle Note book for online deployment (since I'm currently using the M1 Mac Pro as my primary development environment, which doesn't currently have support for GPU acceleration, etc.).

First we find the Gemma homepage on Kaggle, then log in and accept the license for Gemma.
According to the information on the Kaggle homepage, we can see that Google has sent out four models, two 2B and two 7B. we directly choose 'New Notebook' on the top of the page.
After entering the Kaggle Notebook, on the right side of the page we set the parameters of the Gemma model:

After the setup is complete, we tap 'add model' to see the model information and path of Gemma2B (we will use it later). For GPU acceleration, Kaggle provides us with 30 hours of free usage per month.

If you don't have the option "ACCELERATOR", it may be that your Kaggle hasn't been verified by your cell phone, after verification, you can see this option~.
So far our Notebook setup is over, and now we have started to install the required libraries.

First of all the code in the first cell is automatically assigned to us by Kaggle, and in my experience, it's best to run it (although we don't use these functions).
These three libraries are the main ones we need to use

!pip install -U transformers accelerate bitsandbytes

After calling them, we can set the mod to use different precisions on the GPU, such as 8 or 16, 4, etc.

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

quantization_config = BitsAndBytesConfig(load_in_8bit=False) #load_in_4bit

Note that to start here, we need to set the path to the model, on the right-hand side of the page select the three dots to the right of 'Gemma V2' and then select 'copy directory path'. We assigned it as 'model_id'

model_id = "your directory path"

Because we need to use Hugging Face's Tokenizer, here we need to log into the Hub to get access()：

!huggingface-cli login --token Your_huggingface_api

Remember to accept the license on HuggingFace's Gemma page as well.

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=dtype,
    quantization_config = quantization_config
)

Take Care！

If the error is reported consistently here:

ValueError: Tokenizer class GemmaTokenizer does not exist or is not currently imported.

Then most likely Kaggle Notebook needs to be restarted once after the transformers are updated.
When you see the familiar HuggingFace download progress bar you've done it!

It's only been a day since Gemma was released, and I can already see a lot of incredibly talented developers on Kaggle making a very large number of attempts at RAG, fine-tuning, and so on. Looking forward to more LLM form changes!

YOLO V5 running on an Android phone Part 1(train model)

Ray — Fri, 26 Jan 2024 03:18:08 +0000

Kia Ora!
Recently in our AI project, we have been working on monitoring river heights using the Yolo algorithm. It sends an alert like a server when the river height exceeds a certain safe range.
Today I am bringing you a share on how to deploy the Yolo algorithm program using an Android phone

My main development environment is currently an M1 chip MAC, and I often encounter all kinds of bugs on Yolo v8 (but I already have a solution, I'll organize it and update it later), so for today's demo tutorial we're going to start with Yolo v5. I'd like to thank this YouTuber, whose video solved a very large number of problems for me:
https://www.youtube.com/watch?v=zs43IrWTzB0

I won't go over the steps to install Yolo v5, you can easily find their repository on Git Hub.
For labelling and processing the data I used: roboflow
This page is very user-friendly for all versions of Yolo support. There are also many open-source datasets available.

Use the dataset to train:



!python train.py --img 640 ---batch 16 --epochs 10 --data DataSet/data.yaml --weights yolov5s.pt

Validate using the dataset:



!python detect.py --source DataSet/valid/images --weights runs/train/exp/weights/best.pt

After completing the training, we need to export to the tflite format



!python export.py --weights runs/train/exp/weights/best.pt --include tflite

This concludes our training of the model and I can't wait to try it out:



!python detect.py --source 0 --weights runs/train/exp/weights/best.pt

This way we have our own model, and of course we are able to achieve better performance through various manipulations of the dataset during the training process.

RAG in LLM

Ray — Fri, 12 Jan 2024 09:29:27 +0000

It's a good new year! How was everyone's vacation? How was your first week back at work?

Last month we discussed separating front-end and back-end web applications in React. In this post, I'm going to share some information about RAG in the Large Language Model LLM and the impact it may have on your daily work.

When discussing the integration of RAG (Rapid Adaptation Generator) into LLM, we delve into an exciting realm within the field of Natural Language Processing.

RAG can be simply defined as an advanced language model crafted to respond to user queries more flexibly and adaptively.

Advantages of RAG

Rapid Adaptation: RAG is engineered for swift adaptation to new information. It possesses the capability to consult external (or internal) resources while posing a question, thereby delivering more precise and real-time responses.

Controlled Retrieval Capabilities: In today's rapidly evolving AI landscape, the security and privacy of information and data are paramount. Questions like "Is my chat on ChatGPT secure?" highlight this concern. While GPT exhibits robust retrieval capabilities, tapping into vast amounts of text, there is a growing demand for a secure AI usage paradigm. Responding to this need, locally-run programs like "PrivateGPT" on GitHub leverage RAGs, enabling users to furnish documentation or data for LLM to comprehend and draw more effective inferences.

Features of LLM

Powerful Performance of LlamaCpp: LLM employs modern deep learning techniques, and pre-training to generate potent representations for improved text understanding and processing.

Utilization of Embeddings: LLM utilizes Embeddings to map text to a high-dimensional vector space, enhancing its ability to comprehend and represent semantic information. In my experience with LLaMA2 7B Chat, I observed that the Embedding function integrates LLM. For instance, the 'LlamaCppEmbeddings' I used necessitates passing the model into the Embedding function for construction, subsequently passing it through various text splitter methods to structure the data.

RetrievalQA Model:

Once vector data is obtained, the RetrievalQA model acts as a 'chain', utilizing the data retrieved by Embedding LLM as a searchable database, delivering more comprehensive and accurate answers.

Impact on Everyday Work

Increased Speed of Information Access: LLM with RAG significantly enhances the speed of information retrieval in daily work. Models efficiently retrieve information from both external texts and internal team knowledge bases, providing organizations with an easier means to deploy local datasets and ensure data security.

In conclusion, the amalgamation of RAG and LLM presents a smarter, more efficient tool for navigating the information age. The ongoing development and optimization of these models will further propel the application of natural language processing technology across various domains. In our daily work, we anticipate a surge in the applications of these technologies to effectively meet the escalating demand for information.

React: Front-end and back-end combined LLaMA Artificial Intelligence Project Demo

Ray — Thu, 07 Dec 2023 05:35:01 +0000

My internship organization, BOPRC, has a large amount of water meter data, but the current method of data entry is cumbersome and inefficient, requiring employees to manually take pictures and enter the information into an Excel spreadsheet. This approach is time-consuming and error-prone, increasing labour costs.

To solve this challenge, we try to develop a new approach based on AI technology. By integrating a GPT chatbot into the web front end, employees can upload photos of water meters and get automatically recognized text information. In this way, they can more quickly enter data into the back-end database and automate the office.

This project not only improves the efficiency of data entry but also reduces BOPRC's labor costs. In the future, we hope to further optimize the recognition accuracy and expand the system's functionality to make it more intelligent and easy to use.

First of all, I made a lot of modelling attempts, but unfortunately, the OpenAI API needs to be paid for (I'm a bit shy to apply for funds from my unit), but fortunately when I tried to deploy the LLaMA2 model introduced by FaceBook, I found that the model has good language understanding and generation ability, and it can be useful in multiple language tasks, which is a versatile and highly customizable language model. model. Most importantly, it can be deployed locally on my M1 MacBook Pro.

So my idea was to build a front-end page using my self-taught knowledge of React and deploy the LLaMA2 model locally as a back-end program. Implement a simple demo first.

First of all, there are several ways to deploy LLaMA locally, in general using llama.cpp is the most efficient, it also supports M1 GPU calls.

First, we need to clone the library from Github:



git clone https://github.com/ggerganov/llama.cpp.git

Since LLaMA2 has a lot of models, if you want to run them locally (and are not looking for performance for the moment) I recommend 7B-Chat, which is about 4Gb in size.



cd llama.cpp
curl -L https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf --output ./models/llama-2-7b-chat.Q4_K_M.gguf

Take note of

LLaMA models are now in '.gguf' format, the previous '.bin' format can no longer be used, if you need other models, you can go to this URL to view and download. Please put the models in the models directory.
https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/tree/main

Once we are ready we can try LLaMA in a local deployment：



LLAMA_METAL=1 make
./server -m ./models/llama-2-7b-chat.Q4_K_M.gguf

Back end

Locally built models can be used to open the user interface in a browser using the URL displayed in the terminal, set the adjustments you want and start communicating.

You can see that the M1 GPU resources have been used to

So how do you connect an AI model running on the back-end to a front-end web page? We need a couple of tool libraries: 'FastAPI' and 'llama_cpp(Python)', the

After installing these two we can start by writing our backend Python program:

Since I'm using a Python virtual environment, the code to start the back-end program looks like this:



source ~/myenv/bin/activate    
python mian.py

Now the backend program is already waiting for the request

Front-end

Now we can start with the front-end tweaks, here I used a React page that I made myself (I wanted to use the OpenAI API before but I had to pay for it) You can find the library on my Github, I'm not going to expand too much on the front-end code as my knowledge of the front-end is very scarce:
https://github.com/KyrieRui/BOPRC_water_meter_Monitor_Demo

Note that botReply is set here because on the backend we receive the request and return the value with the 'data' flag. Once the front end is set up we can test it:

The front-end user clicks on the 'Send' button, the back end receives the correct request and then the model starts working, after which it returns 'data' to the front-end.

This Demo is just a demonstration of the experiments I have conducted for the time being, next I will do more research and try to connect the local AI model to the database to meet the specific needs of the customer, I welcome all partners to make suggestions on my Demo code, I will take the time to reply and make changes! Thanks for reading and may you all have a great day!

React Learning Note: Optional Chaining

Ray — Mon, 20 Nov 2023 08:12:59 +0000

As we can see, each book has reviews, one for GoodReads and the other for librarything. They have different review accounts.
So let's create a function that gives us the total reviews count for each book.

function getTotalReview(book) {
  const firstValue = book.reviews.goodreads.reviewsCount;
  const secondValue = book.reviews.librarything.reviewsCount;
  return firstValue + secondValue;
}

console.log(getTotalReview(book));

That is quite easy to get right, so now we go find out book id 3

Book id 3 don't have 'librarything' reviews value, if we still use the function we write up that will return Error"undefined".
To avoid that error we could read reviews value optionally

function getTotalReview(book) {
  const firstValue = book.reviews.goodreads.reviewsCount;
  const secondValue = book.reviews.librarything?.reviewsCount;
  return firstValue + secondValue;
}

console.log(getTotalReview(book));

This '?.reviewsCount' question mark is what we call "Optional Chain", it will pass the null or undefined value, even if it still shows 'NaN' as a result but we will not receive an Error message.

Remember the double question mark to set the default value?
Let's do this:

function getTotalReview(book) {
  const firstValue = book.reviews.goodreads.reviewsCount;
  const secondValue = book.reviews.librarything?.reviewsCount ?? 0;
  return firstValue + secondValue;
}

console.log(getTotalReview(book));

Now the problem is solved, we set this undefined 'secondValue' to be '0', and we can still get the total count number as a result without any error or null value.

In summary, we can use Optional Channing to let JS not read undefined values and double question marks to set a default value to make sure the function is working well all the time.

React learning note: Logical Operators

Ray — Thu, 09 Nov 2023 08:49:18 +0000

Hello, everyone! Sorry for stopping updating for a couple of weeks while I was busy with finals a while back, I'll try to get back to a weekly update as often as I can later on. In the meantime, I've been offered a summer internship at BOPRC, so if I gain any new understandings during the internship, I'll be blogging about them as well~!

Back to our React study notes. Logical Operators are used quite frequently in React programming.
The 'end Operator' and 'or Operator' have a feature called: short-circuiting, short-circuiting In logical Operators, in certain conditions the operator will return the first value.

'end Operator'

console.log(false && "hello");
console.log(true && "hello");

'hasMovieAdaptation' is the variable we defined earlier, and we can also use it as an end operator, more like 'if'.

This syntax also applies to truthy and falsy Values, eg:

'or Operator'
The 'or operator' is the inverse case and returns the first upper end as long as it is true

We can use this feature to define default values for specific data. For example:
A book with an ID of 2 has no Spanish translation

Also works for undefined values

There is also a notation: nullish coalescing operator, which works much like 'or operator', but it does also short circuit or falsy value

const count = book.reviews.librarything.rating.reviewsCount ?? "No data here";
count;

React: Template Literals & Ternaries Instead of if/else Statements

Ray — Mon, 09 Oct 2023 09:14:09 +0000

This week we continue our React study notes, and this week I've learned two simple but very useful uses: Template Literals & Ternaries Instead of if/else Statements.

Continue to use the book model used in the first two study notes. First, we create a simple summary, in defining an object we can use the Template Literals

`${js or element's own attributes}`

JavaScript can be used inside to achieve the effect we wish to print! Eg:

For example, we want the printed String to determine whether the book object has the property 'hasMovieAdaptation', where we use the judgment

${hasMovieAdaptation ? "(if true)" : "(if false)"}

See, the output changed

Now let's take this simple judgment and add a little bit of new conditions, such as for the number of pages to be judged.

pages > 1000 ? "over 1000": "less than 1000";

But every time you need to print when writing such a long judgment is always inconvenient, then we try to define this judgment as an object, like this:

You can see that our log successfully prints out the result of the desired judgment.

React: Rest and Spread Operator

Ray — Mon, 02 Oct 2023 08:06:59 +0000

Kia Ora!
Did you guys all try it yourself with the homemade VPN instruction I brought last week?
This week I continue my React learning notes and today I bring you the application of Rest and Spread Operator features in React.

Rest Operator
Continuing on from last week we discussed the code section on Destructuring Objects.

We deposited primaryGenre, and secondaryGenre into Array: genres. Now we want to extract the other genres as well. This is where we can use 'the rest operator'

After secondaryGenre we add '...(any name)', which will automatically add all the values to this Array, even if we didn't define them separately like we did for the first and second one.

**Note! **Here we can notice that all the remaining values are deposited into the same Array, except for the first and second Genre already defined, so '.... . otherGenres' can only be placed in the last bit of the const definition, and if the rest operator is placed in the middle of the first and second one, it will report an error.

Spread Operator
Now let's switch to a different scenario: create a new Array with all the genres, while adding a new genre at the end. If we add it like this, we can see that our new genre is not in the same Array as
all the other genres.

At this point, we use '...' again. Add our new genre. like the one shown below, and this time the result is what we want.
Tips: If you want to add to the top of the Array, put '...genres' after it.

Spread Operator can be used not only in Array but also in objects is very important, it can help us to add new properties and also help to update the old properties.
Creates an updatedbook object, calls an existing book object and adds a 'moviePublicationDate'.
You can see this new property at the end.

Creates an updatedbook object, calls an existing book object and adds a 'moviePublicationDate'.
You can see that this new property is not where we want it to be, it's outside of this book object.

We can add the moviePublicationDate to the book object by using the Spread Operator in the same way as we did with the Array.

You can see that at this point the page value of books is 1216, now let's try to update this data. At the end of the updatedbook object, add the change to the page property, and you can see that we have successfully updated the value from 1216 to 1220.

Note: As the Rest and Spread Operator I mentioned earlier when we put the Rest & Spread Operator in front of the book, we can copy the original data first, and then we can overwrite the updated data later, but if we put the Rest & Spread Operator in the back of the book, we can't! successfully overwrite.

Use Azure set up owen VPN

Ray — Tue, 26 Sep 2023 06:45:49 +0000

Today, I bring a simple way to build a VPN you can use yourself using Microsoft Azure and OpenVPN.
There are many ways to use a VPN, my mum is a university professor and she often needs to use Google Scholar as a tool, but it is very difficult to use Google services where she works. So I decided to learn and build a VPN for her to use.

Microsoft Azure has a student discount that gives a $100 limit to set up a virtual machine.
Firstly, search for 'OpenVPN' in Create Resource, select 'OpenVPN Access Server' and create it.

In the Administrator account choose a password as our way to connect to the server, set your username and password, and make sure you remember the password you set. Then you can just click Review + Create. Check your settings are correct then create your virtual machine and wait for Azure to do the set-up.

Note that！

Azure only provides 3 public IP addresses for Education Offer users, so if you've created more than 3 before, please don't click Create directly here, and consider whether you need to delete public IPs that you no longer use to assign to your VPN server.

After checking that there are no problems with the settings, remember the username and the set password and create that server. When Azure finishes the process go to 'resource' and you can see our static IP address, connect via ssh with the username we just used.

When the connection is successful, it will display 'Are you sure you want to continue the connection', enter yes.
After that, it will show agreements, so we will enter yes again.
Most of the subsequent settings are selected as default, later there is a setting that will ask you if you want to use 'openvpn' as the default Web UI user account, I choose the default here and set the password. You can also change it to your favourite username and password. I left the Activation key blank.

Once OpenVPN is set up we can use the Web UI username and password we just set up after connecting to the server via ssh.
https://static IP address:943/admin
Open the browser and log in
The first time we login we need to agree to the Agreement, then we remember to go into the Network setting and change the IP address to our static IP address and in the DNS setting change it to "Have clients use specific DNS servers "
I have set the Primary DNS Server to 1.1.1.1.
The second DNS Server set to 1.0.0.1

After all the settings are done you have your own VPN and can add username and password in User Management. Users only need to use https://your IP address, and you can see the OpenVPN client download interface in your browser.

Destructuring in JS

Ray — Tue, 19 Sep 2023 22:03:49 +0000

Firstly thanks for reading Ray's React Learning Notes, a React course I purchased from Udemy to document what I've learnt.
Link to the full course:
https://www.udemy.com/course/the-ultimate-react-course/learn/lecture/37350446#overview

Thanks again to Udemy for the high-quality instructional content.

In this project, I am learning how to use one basic tool in VScode: Quokka.js

This plugin can easily to found in the VScode extension store.

Quokka.js is used to quickly create JavaScript / TypeScript prototypes. It's easy to run JS files in VScode and helps to troubleshoot bugs.

to run it just sample type: >quokka start at VScode command palette. Then click the 'Start on current file'

It Very important is that he can return the results of the current code in a timely manner in the terminal on the left and below. This is very useful for me to start learning React.

Destructuring

Destructuring is getting data out of an object or Array.

A simple data structure about books is used here:

const data = [
  {
    id: 1,
    title: "The Lord of the Rings",
    publicationDate: "1954-07-29",
    author: "J. R. R. Tolkien",
    genres: [
      "fantasy",
      "high-fantasy",
      "adventure",
      "fiction",
      "novels",
      "literature",
    ],
    hasMovieAdaptation: true,
    pages: 1216,
    translations: {
      spanish: "El señor de los anillos",
      chinese: "魔戒",
      french: "Le Seigneur des anneaux",
    },
    reviews: {
      goodreads: {
        rating: 4.52,
        ratingsCount: 630994,
        reviewsCount: 13417,
      },
      librarything: {
        rating: 4.53,
        ratingsCount: 47166,
        reviewsCount: 452,
      },
    },
  },

Use getBooks() to get all books data

const books = getBooks();  
const title = book.title; 
const author = book.author;

Or use getBook(id) to get a specific data

const book = getBook(1);

There are many other elements in the book object such as author, name, etc. We can use Destructuring to create them all at once.

const { title, author, pages, publicationDate, genres, hasMovieAdaptation } = book;

this way you can get the object in the book. This puts the elements of the object together. Passed with console.log.

A similar approach can be used for elements in book, except for book:

const primaryGenre = genres[0];
const secondaryGenre = genres[1];

Or:

const [primaryGenre, secondaryGenre] = genres;

So we can use ourselves to create an array structure, in a specific location to go to the data we want, similar to the use of [1].