DEV Community: Youdiowei Eteimorde

How AI generates images from noise

Youdiowei Eteimorde — Sun, 28 Jan 2024 14:38:51 +0000

Generative AI models like DALL-E and stable diffusion generate jaw-dropping and mind-boggling images. While the results may seem magical, understanding how these models work often involves delving into mathematical concepts. In this article, I will break down the workings of these models for non-technical readers, aiming to demystify the complexities behind their magic.

Diffusion models

Models like DALL-E belong to a class of machine learning models called Diffusion models. They are currently the best performing models when it comes to generating realistic images.

Diffusion models were inspired by the physical phenomenon known as diffusion. In physics, diffusion is the process by which particles move from an area of higher concentration to an area of lower concentration.

For instance, when you spray paint on a wall, it moves from the paint container where there's a high amount of paint to the wall where there isn't any.

In machine learning, diffusion models aren't so different. Their task involves adding noise to an image incrementally at different time steps until the image becomes barely visible.

This process is called forward diffusion. In the picture above, at time step 1, the image is clean, but starting from time step 2, we begin incrementally adding noise. This is similar to spraying paint on a wall, which is a form of diffusion in physics. In our case, we are spraying the "paint" on the image.

You might be wondering how this principle helps AI generate new images. Well, forward diffusion itself doesn't, but the reverse process does.

Reverse diffusion

Reverse diffusion is the process of taking a noisy image and reverting it back to the prior time step.

The image above reverses the noise from time step 4 back to 3. So, how is this helpful in generating new images? Well, let's walk through one scenario.

A friend presents us with the image above and challenges us to restore the image to its original form. We look at the image, since parts of it are still visible, it shouldn't be too hard to infer the other missing parts. So, we accept their challenge.

Here's how we are going to fill in the gaps. We start by breaking the noise removal process into time steps, and at each time step, we decide what part of the image to fill. Let's divide the time steps into 5. The initial noisy image is at time step 5, and time step 1 is when we have removed all the noise present in the image.

In our initial noisy image, which is at time step 5, we notice the stick figure is holding something in its left arm. We can make the assumption that it is a suitcase or a box, so we can remove the noise by completing the drawing. This will result in a new image, now at time step 4.

At time step 4, we have less noise. Let's remove the noise at the stick figure's foot. Since we can't see the direction of his right foot, we can make the assumption it is facing the same direction as his other foot. So, we remove the noise and draw his new foot again based on our assumptions. This will give us the image at time step 3.

At time step 3, we have to decide how to go about drawing the stick figure's second arm. What direction should we place his arm? Let's look at the noise. From the shape of the noise, we can assume the figure's arm is raised. The figure is apparently holding a suitcase, so that means the figure might be a businessman. What do businessmen tend to do? They tend to make calls a lot. So, we can assume this figure is making a phone call with its other arm. We draw this, clean off the noise, and move on to time step 2.

At time step 2, our diagram is now pretty clear with just a few noises, which we can easily remove. This will give us our final image at time step 1 with no noise at all.

How were we able to fill in the gaps? Well, it came from our knowledge of the world. Over the course of our lives, we have seen a lot of stick figures, and we have an understanding of how stick figures should appear. We made assumptions based on our past experiences in life.

Our friend who gave us this challenge provides us with the original image and suggests that we compare it with what we had generated through noise removal.

Oops, we were wrong; the stick figure wasn't making a call with his other hand. But would you look at that? While trying to fill in the gaps in the noise, we created a totally new image. In theory, that's how diffusion models generate new data.

If we want a diffusion model to solve this task as we did, we will present it with a lot of image data of stick figures. By a lot, I mean a significant amount, preferably in the thousands or even millions.

By providing the model access to such a dataset, it will develop an understanding of how stick figures should look. So, when we give it an image with noise, it will behave similarly to us and think, based on its training data, what is most likely supposed to fill that noise.

The Need for Time Steps

While generating our image, we divided the process into time steps. Why? By breaking the task into different time steps, it enables the model to gradually think through the task, much like we did. This is what makes diffusion models proficient at generating images compared to other models. At every time step, the model only cares about the prior time step, which helps it generate the next time step image. This process is known as a Markov chain.

Image Synthesis

In our last task, we focused on restoring an already existing image that was just noisy. Now, let's explore the challenge of creating an image without any preexisting image. The process of generating images from scratch is called image synthesis. This process isn't new, several ML algorithms are capable of performing it perfectly. However, diffusion models are currently considered the best at generating realistic images.

So, how can we get our model to make images out of thin air, or should I say, out of thin noise? Before we delve into that, let's consider how humans would achieve this task.

Our friend from before presents us with a highly noisy image, asking us to clean the image once again.

We take a look at the image and tell them it is indeed very noisy. They reply and say they know, but we should examine the image and utilize the patterns in the noise to recreate the image.

Like the previous example, we will divide our noise removal process into time steps. From the noisy image above, I can see a couple of black dots. Let's try to connect these dots and see what we can come up with.

After connecting the dots, we obtain two lines for this current time step. We then move on to the next time step.

In this time step, we connect our other two dots to the currently existing lines. I can definitely see a stick figure now. We can argue that our stick figure has limbs now, but then there's still one dot to connect.

We connect our existing lines to that dot, and we have what appears to be a stick figure with no head. Now, we have to carefully consider where to place its head.

Let's place the head at the bottom of the image. Although it is unlikely for a stick figure to be upside down, it is even more unlikely for us to have a stick figure with longer arms than legs. We can say the stick figure is somersaulting.

We then remove the remaining noise and present the figure to our friend, declaring that we are done. We then ask for the original image so we can compare our generated image. However, our friend replies to us that there was never an original image, and all he gave to us is noise.

That's unexpected; I could have sworn I saw a stick figure somersaulting in that noise. Well, this is what we humans do. We sometimes see patterns where there are none, and it can lead to some of the most artistic results. Take the ancient Greeks, for instance.

They saw these dots in the night sky and envisioned a hunter holding a club and a shield. This interpretation gave birth to the Orion constellation, even though there wasn't a hunter in the sky; it was just noise in the form of stars.

Over the course of generations (time steps), various artists have reimagined the Orion constellation, making it look more realistic.

This is exactly what diffusion models do. They are initially given noise, but thanks to their vast training data, they start seeing patterns that are clearly not there. These apparent patterns are enough for them to create novel and realistic images.

You are probably asking, 'When I use DALL-E, I provide a prompt, and then it generates an image. How does this work?' Well, it is a text-guided diffusion model. That will be the subject of the next article.

If you found this article interesting or useful, please drop a ❤️. Share it with anyone who is curious about Generative AI. Feel free to drop a comment if you have any questions.

"Everything is a file" Explained

Youdiowei Eteimorde — Fri, 08 Dec 2023 09:52:51 +0000

If you have ever used Linux or any other Unix-like operating system, you have probably come across the saying 'Everything is a file.' But what does that mean? Is everything in these systems actually a file? Well, that is not the case. To understand the saying, we will walk through various definitions that try to answer the question: What is a file?

What Is a File in the Real World

In the world we live in, a file is simply a container for documents. The document contains useful information, usually in the form of pages. There are four fundamental actions you can perform on a file in the real world:

open
read
write
close

The process of working with a file involves these four operations. When given a file, you first open it, then read through it by flipping through the pages of the file. If you have the right permissions, you can write to the file. When you are done, you can then close the file.

Computer files on storage devices

In computer systems, one way to define a file is as any piece of data stored on a storage device.

The same four fundamental operations that can be performed on real-world files can also be performed on computer files. The only difference is how the data is stored. Computer files are stored as streams of data.

They are called streams because instead of storing your file in one big location, the file is subdivided into smaller chunks. These chunks are similar to pages in a real-world file, and the process of streaming through those chunks is analogous to flipping through a file.

When you open a file on a computer, the operating system goes to your storage device and fetches the location of the file. Once its location has been fetched, you can either choose to read from the stream or write to the stream or perform both actions. When you are done with the file, you close it. These operations are exactly the same as the operations performed on files in the real world.

When working with files, the only thing that ever changes is the data source. Your file might be on a hard disk, CD-ROM, or flash drive, but you will still perform the same four fundamental operations, and its content is always streams.

Everything isn't a file rather everything is file like

The heading above is what the saying should have been, but it is a mouthful and isn't catchy. In Unix-like operating systems, everything is said to be a file because you can perform the same set of operations described for files stored on storage devices on other resources. These resources are commonly referred to as file-like.

You can open a particular file-like resource, read from it, write to it, and close it when you are done. The data from that resource is also streamed. The only thing that ever changes is the source of the data.

For instance, when you are typing in your terminal, you are actually writing to a resource called standard input. When data gets displayed to your terminal using the echo command, your operating system is writing to the standard output resource.

Another example of a file-like entity is a network connection to a server. When you make a connection to a server, you are opening a line of communication. You can send network requests to the server, which is equivalent to writing, and when the server responds, you are then reading. After the communication is over, you can then close the line of communication. The data is also in the form of a stream.

This same process applies to the majority of other resources in Unix-like operating systems, like when two processes are communicating; their means of communicating is file-like.

Reason why everything is a file

Seeing everything as a file is actually a clever design decision made by the creators of Unix. This allows the same set of APIs to be used across different tasks. When you open a file-like entity, it returns a file descriptor. A file descriptor is a unique integer that represents an open file. Standard input has the file descriptor of 0, while standard output has the file descriptor of 1. Several system calls in Unix work with file descriptors.

The example below uses the write system call to write Hello, world! to standard output.



#include <unistd.h>

int main() {
    write(1, "Hello, world!\n", 14);
    return 0;
}

The write system call can also be used to write to a file on disk.



#include <fcntl.h>
#include <unistd.h>

int main() {
    // Get the file description
    int fd = open("output.txt", O_WRONLY | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR);

    // write to the file
    write(fd, "Hello, World!\n", 14);

    // close the file
    close(fd);

    return 0;
}

The program above opens a file called output.txt. Then writes hello world to it. After that it closes the file using the close system call. As you can see both writing to standard output and writing to a file use the same function.

The code snippet below shows a C function that makes a request to a server. It does this by writing to the client socket. Which is file-like because it returns a file descriptor when created.



void send_http_request(int client_socket, const char *host, const char *path){
    // Create the HTTP request
    char request[MAX_BUFFER_SIZE];
    snprintf(request, sizeof(request), "GET %s HTTP/1.1\r\nHost: %s\r\n\r\n", path, host);

    // send the HTTP request to the server
    write(client_socket, request, strlen(request));
}

The function below reads from the socket. The loop is used to receive the streams of data as they come.



void receive_http_response(int client_socket){
    // Recieve and print the HTTP response from the server
    char response[MAX_BUFFER_SIZE];
    ssize_t bytes_received;

    // Read response streams
    while ((bytes_received = read(client_socket, response, sizeof(response) -  1)) > 0 ) {
        // Add string termination
        response[bytes_received] = '\0';
        printf("%s", response);
    }
}

So to reiterate on the question I asked earlier, what is a file? A file is any resource that streams it's data content. This content can be read from or written to. A lot resources on Unix-like Operating system follow this pattern and that is why everything is a file.

How python's Multithreading differs from other languages

Youdiowei Eteimorde — Mon, 13 Nov 2023 05:14:39 +0000

Python, like most programming languages, features multithreading. However, unlike many languages, Python's multithreading isn't capable of fully harnessing the potential of the system it operates in. In this post, I will explain why this is the case and how the Global Interpreter Lock (GIL) plays a role in this limitation.

Before we delve deeper into why Python's multithreading has limitations, let's get a couple of key concepts out of the way. This will help us have a better understanding of multithreading in Python.

What is a process?

As programmers, when we write programs and execute them, they become processes running on our CPU. A process has access to its own memory and is isolated from other running processes. At any given time, only one process can be running on a CPU. The image above represents a CPU with four running processes.

So, you might be wondering: if only one process can run at a time, how is it possible for most computers to handle multiple tasks simultaneously? Well, computers have the ability to multitask, and they employ a clever trick that fools us humans into believing they are doing multiple things at the same time. This trick is known as context switching. This technique involves the operating system quickly switching between the currently running processes so fast that we humans perceive it as happening at the same time.

A process can be single-threaded or multithreaded. All the processes running in our image above are single-threaded.

Threads explained

A process usually performs one task at a time. For instance, consider the program below, which calculates the square of natural numbers from 1 to 100.



def calculate_square():
    for i in range(1, 101):
        square = i ** 2
        print(f"The square of {i} is: {square}")

if __name__ == "__main__":
    calculate_square()

When we execute this program, it becomes a process that is performing one task, so we say it is single-threaded. What if we also want to calculate the cube of the numbers at the same time?

Well, that is when we employ multithreading. It allows one process to perform multiple tasks. The program below uses Python's threading library to calculate both the square and cube of the numbers.



import threading

def calculate_square():
    for i in range(1, 101):
        square = i ** 2
        print(f"The square of {i} is: {square}")

def calculate_cube():
    for i in range(1, 101):
        cube = i ** 3
        print(f"The cube of {i} is: {cube}")

if __name__ == "__main__":
    square_thread = threading.Thread(target=calculate_square)
    cube_thread = threading.Thread(target=calculate_cube)

    square_thread.start()
    cube_thread.start()

    # Wait for both threads to complete
    square_thread.join()
    cube_thread.join()

In the picture below, we can see five running processes, but the green process occurs twice. This shows that it is utilizing threading.

One thread is performing the square task, and the other is performing the cube task. Both threads that belong to the green process can share memory and other resources.

As mentioned before, only one process can run on the CPU. Even if the green process is multi-threaded, only one of its threads can run at a time, and it is left for the operating system to select which one.

You are probably thinking it would be so cool if two threads could run at the same time. Well, that is possible if the CPU has multiple cores.

Making threads run in parallel

Modern computers have multiple cores, and each core can run its own individual process. So, a CPU with two cores can handle two tasks simultaneously. In theory, this means our multithreaded program, as written above, can run on two separate cores simultaneously. This is called parallelism, and it is one of the benefits that come with multithreading.

Now, our CPU has two cores, and our green process still has two threads, with each thread on a different core. One core will be performing the square task, and the other will be performing the cube task, all at the same time.

This is awesome; our program is now more efficient. Well, not quite. If you wrote the program in a language like Java or C++, that would be the case, but not in Python. Threads in Python can't run in parallel because of the GIL.

What exactly is the GIL?

It is the Global Interpreter Lock. Basically, what the GIL does is simple; it tells any running Python process that only one of its threads can run at a time, even if it is on a multi-core CPU.

This is how our multithreaded program will actually work in Python. When one of our green process's threads is running, even if the other thread has the opportunity to run on a different core, it would have to wait for the running thread.

This is because the running thread basically grabs the interpreter and locks it, making it impossible for the other thread in the process to have access to the interpreter. The other thread just has to wait for its chance so it can get the interpreter and lock it.

Why the GIL?

Python has the GIL because it was not designed to be thread safe. Multithreading is a very complex feat to achieve. Consider this scenario: you have two threads, thread1 and thread2, each running on separate cores. They both have access to a list, my_list = [1, 2, 3].

If thread1 grabs the list and empties it my_list = [], then thread2 tries to access the first index of an empty list, thinking it still has values within it, that will be a problem. This is an example of a race condition.

The Python list and other objects in Python weren't designed to be used in a multithreaded environment; that's what it means not to be thread-safe. That is why the Python team decided to lock everything so errors like these won't occur.

How to escape the GIL

There are several ways of bypassing the GIL. First of all, the GIL is only present in the C implementation of Python, CPython. Other implementations of Python like Jython, IronPython, and PyPy don't have the GIL. Additionally, Python provides the multiprocessing library, which allows for parallelism in your Python program.

Also PEP 703 proposes making the GIL optional in CPython during build time.

Is multithreading useless in python

So, what's the point of having multithreading in Python, you might ask. Multithreading isn't completely useless in Python because it is useful when working on I/O-bound tasks. Imagine if you have two threads: one is performing a massive computation, and another is waiting for the user's input. The thread waiting for the user's input can yield control to the computation thread while the user is providing their input. The same applies when a thread is trying to access the file system, a remote database, or a network. This is where threading shines in Python.

Running Large Language Models on the CPU

Youdiowei Eteimorde — Sun, 03 Sep 2023 23:56:54 +0000

At some point while using ChatGPT, you might encounter the error shown above. The reason behind this lies in the necessity of substantial computational resources to run a large language model like ChatGPT. These models require a significant amount of computational power, which is typically provided by specialized hardware known as Graphical Processing Units (GPUs).

Unlike regular programs, large language models cannot efficiently run on Central Processing Units (CPUs). The cost associated with operating such massive language models is substantial. It's an expense that usually only large corporations like Google and Microsoft can afford due to the substantial resources involved.

There are two computationally intensive tasks that a language model must consider:

Training
Inference

Training is the process of instructing a language model on how to perform its intended task. It stands as the more computationally demanding process between the two. Training an LLM consumes both time and monetary resources.

While Inference is the utilization of a trained large language model. Whenever you engage with ChatGPT, you're utilizing it in the inference mode. Although less computationally intensive than training, running inference on LLMs remains relatively expensive due to the substantial requirement for GPUs. Especially if you are running inference on the scale of ChatGPT.

Considering all of this, you might be pondering whether it's feasible to run Large Language Models on a CPU. The answer is yes, at least for inference. In this article, we will delve into the recent advancements, techniques, and technology that have enabled LLMs to operate using nothing more than regular CPUs.

Introducing LLaMa

LLaMa, or Large Language Model Meta AI, is a lightweight and efficient open-source Large Language Model developed by Meta. It is designed to deliver performance similar to models that are ten times its size.

An essential aspect of Large Language Models (LLMs) is their parameters, which play a crucial role in determining their performance. LLaMa introduces various versions with differing parameter counts, including a variant with 7 billion parameters and another boasting around 65 billion parameters.

Historically, enhancing LLM performance involved augmenting the number of parameters. This adheres to the scaling law, a strategy that has proven successful for models like the GPT series.

However, this strategy poses a challenge due to the inherent relationship between parameter increase and computational demands. As parameter count grows, so does the computational workload, presenting a trade-off between model complexity and processing efficiency.

Recent research has indicated a shift in the approach to enhancing LLM performance. Instead of solely increasing the number of parameters, an effective alternative involves augmenting the training data. LLaMa adopts this strategy, relying on data enrichment to achieve its improved performance.

LLaMa marked a groundbreaking achievement in the realm of LLMs. LLaMa was trained with 2048 A100 GPUs. The big breakthrough in LLaMa wasn't its training, but rather its inference step. The LLaMa model can be ran on a single GPU during inference, a distinct advantage compared to other LLMs that demand multiple GPUs for operation.

Although this single-GPU capability was remarkable, it is still a far cry from running on the CPU. To enable a lightweight LLM like LLaMa to run on the CPU, a clever technique known as quantization comes into play. But before we dive into the concept of quantization, let's first understand how LLMs store their parameters.

How are LLMs parameters stored

The parameters of a Large Language Model (LLM) are commonly stored as floating-point numbers. The majority of LLMs utilize 32-bit floating-point numbers, also known as single precision. However, certain layers of the model may employ different numerical precision levels, such as 16-bit.

Let's perform a simple mathematical calculation:

\text{ bits} = 1 \text{ byte}\quad 32 \text{ bits} = 4 \text{ bytes}

There are 8 bits in one byte, which means 32 bits equals 4 bytes. Consider a LLaMa model with 7 billion parameters, where these parameters are stored as 32-bit floats. To calculate the total memory required, we can multiply 4 bytes by 7 billion:

\, \text{bytes} \times 7 \, \text{billion} = \text{28 billion bytes}

28 billion bytes is equivalent to 28 gigabytes (GB).

This implies that utilizing the model would require a memory capacity of 28 gigabytes (GB). However, it's worth considering the limited memory capacity of consumer based CPU devices. Such devices typically do not possess memory sizes that can accommodate 28 GB. We need to reduce the memory consumption of the LLM, and this is where quantization becomes relevant.

What is Quantization?

Quantization is the process of reducing the precision of the parameters of a LLM. In this context, it involves taking the existing parameters, which are typically stored as 32-bit floating-point numbers, and converting them to 4-bit integers. This reduction in precision significantly decreases the memory consumption required to store the model. While there might be a minor degradation in the model's performance, this impact is generally quite small and often negligible or unnoticeable.

Consider an array of values in floating-point representation:

[0.333, 0.98, 0.234]

When we apply quantization, these values are converted to integers:

[2, 15, 0]

While the two sets of numbers are different, they retain their meaning. For instance, the second value remains the largest in both arrays, and the third value is still the smallest.

Quantization can be likened to expressing values in a manner similar to using percentages. For example, someone might express a need for "3200 out of the 10000 cans". This expression can be simplified by stating the need as "32 percent of the cans". Although the specific numerical values differ, the conveyed information remains identical. This analogy shows how quantization simplifies value representation while retaining their inherent meaning.

Applying 4 bit quantization to LLaMa7B

Let's apply 4-bit quantization to the LLaMa model, which has 7 billion parameters, assuming that all parameters are originally stored as 32-bit..

\text{ bits} = 1 \text{ byte}\quad 4 \text{ bits} = 0.5 \text{ bytes}

Given that 1 byte equals 8 bits, 4 bits are equivalent to 0.5 bytes. Therefore, to determine the memory consumption after applying 4-bit quantization to a Large Language Model with 7 billion parameters, we multiply 0.5 bytes by 7 billion.

\times 0.5 \, \text{bytes} = 3,500,000,000 \, \text{bytes}

After performing our calculation, we find that the memory consumption is 3 billion bytes, equivalent to 3.5 GB. This indicates that it's possible to run the LLaMa 7 billion parameter model on a device with more than 3.5 GB of available RAM space.

The GGML ecosystem

Another key technology that has contributed to running LLMs on CPUs is GGML, which stands for Georgi Gerganov Machine Learning. It is named after its creator.

GGML is a machine learning library that implements tensor operations in C. Due to its implementation in C, it can facilitate the execution of ML models on diverse platforms. Additionally, GGML offers support for model quantization.

GGML was initially used to develop Whisper.cpp, a project designed to enable the execution of OpenAI's Whisper model on compact devices such as smartphones and other small devices. This laid the foundation for a subsequent project named llama.cpp, which facilitates the execution of LLaMa models on similarly compact devices.

Furthermore, GGML serves as a file format that enables the storage of model information within a single file.

Introduction to Llama.cpp

Llama.cpp is a runtime for LLaMa-based models that enables inference to be performed on the CPU, provided that the device has sufficient memory to load the model. It is written in C++ and utilizes the GGML library to execute tensor operations and carry out quantization processes.

Despite being written in C++, there exist bindings for several programming languages. The Python binding is referred to as llama-cpp-python.

Working with llama-cpp-python

Let's put into practice everything we've learned so far. We will utilize llama-cpp-python to execute a LLaMa 2-7B model. You can run this example locally if you have approximately 6.71 GB of available memory. If you don't, you can use the link below to run the code in Google Colab.

Firstly, we need to obtain our model, convert it into GGML format, and then proceed to quantize it. However, there's no need to be concerned about this step since we can readily acquire an already quantized model from HuggingFace. A user named TheBloke has quantized numerous models.



wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q4_1.bin

I used the command-line tool wget to download the model. This specific variant of LLaMa has undergone fine-tuning similar to ChatGPT.



pip install llama-cpp-python

Next, we proceed to install llama-cpp-python. Once the installation is complete, you have the option to either create a Python file to contain your code or utilize the Python interpreter for execution.



from llama_cpp import Llama

LLM = Llama(model_path="./llama-2-7b-chat.ggmlv3.q4_1.bin")

We load our model and subsequently instantiate it. Following that, we input our prompt to the model.



prompt = "Tell me about the Python programming language? "

output = LLM(prompt)

Llama-cpp offers an API that is similar to the one provided by OpenAI. Here's the response generated by our provided prompt:



{'id': 'cmpl-4c10c54e-f6a1-4d80-87af-f63f19ce96c2',
 'object': 'text_completion',
 'created': 1692538402,
 'model': './llama-2-7b-chat.ggmlv3.q4_1.bin',
 'choices': [{'text': '\npython programming language\nThe Python programming language is a popular, high-level programming language that is used for a wide range of applications, such as web development, data analysis, artificial intelligence, scientific computing, and more. Here are some key features and benefits of Python:\n\n1. **Easy to learn**: Python has a simple syntax and is relatively easy to learn, making it a great language for beginners.\n2. **High-level language**: Python is a high-level language, meaning it abstracts away many low-level details, allowing developers to focus on the logic of their code rather than',
   'index': 0,
   'logprobs': None,
   'finish_reason': 'length'}],
 'usage': {'prompt_tokens': 10, 'completion_tokens': 128, 'total_tokens': 138}}

The text returned from the model is this:

python programming language
The Python programming language is a popular, high-level programming language that is used for a wide range of applications, such as web development, data analysis, artificial intelligence, scientific computing, and more. Here are some key features and benefits of Python:

Easy to learn: Python has a simple syntax and is relatively easy to learn, making it a great language for beginners.

High-level language: Python is a high-level language, meaning it abstracts away many low-level details, allowing developers to focus on the logic of their code rather than

The response received is actually quite satisfactory. However, it's essential to bear in mind that this particular model isn't equivalent to ChatGPT in terms of capabilities. For enhanced performance, consider utilizing a model with higher quantization, such as the 8-bit version. For even more advanced results, acquiring the 13-billion-parameter models could be beneficial. Using prompting engineering techniques can also enhance the quality of the responses obtained.

The ability to run Large Language Models on the CPU represents a significant breakthrough in the field. This advancement paves the way for various applications, benefiting small businesses, researchers, hobbyists, and individuals who prefer not to share their data with third-party organizations. This development is set for continuous growth, and in the upcoming years, if not months, we can expect more accessibility to LLMs.

Here are few useful resources to help expand your knowledge on running LLMs on CPU.

The blog post and paper that introduced LLaMa.
Check out ggml website and read their manifesto to learn more about the project's philosphy.
Watch this video to have an understanding of the hardware ChatGPT runs on.
To get a more in depth overview of quantization watch this video
To understand the GGML file format read this.

Understanding LangChain's RecursiveCharacterTextSplitter

Youdiowei Eteimorde — Sat, 12 Aug 2023 01:55:59 +0000

Large language models are powerful tools with extensive capabilities; nonetheless, they grapple with a distinct limitation known as the context window. This context window defines the boundaries within which these models can proficiently process text. Take, for example, gpt-3.5-turbo, which operates within a context length of 4,096 tokens, approximately corresponding to 3,500 words.

But what occurs when you present these models with a document that exceeds their context window? This is where a clever strategy known as "chunking" comes into play. Chunking involves dividing the document into smaller, more manageable sections that fit comfortably within the context window of the large language model.

Langchain provides users with a range of chunking techniques to choose from. However, among these options, the RecursiveCharacterTextSplitter emerges as the favored and strongly recommended method.

Quick overview

The RecursiveCharacterTextSplitter takes a large text and splits it based on a specified chunk size. It does this by using a set of characters. The default characters provided to it are ["\n\n", "\n", " ", ""].

It takes in the large text then tries to split it by the first character \n\n. If the first split by \n\n is still large then it moves to the next character which is \n and tries to split by it. If it is still larger than our specified chunk size it moves to the next character in the set until we get a split that is less than our specified chunk size.

Code implementation

What I Worked On

February 2021

Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.

The first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.

The text above is extracted from an article written by Paul Graham, titled: What I Worked On. Let's utilize the RecursiveCharacterTextSplitter to break it into small chunks, each with a maximum size of 100 characters.

First we import it from langchain:



from langchain.text_splitter import RecursiveCharacterTextSplitter

Let's load the text we wish to create chunks from into a variable called text.



text = """What I Worked On

February 2021

Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.

The first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.
"""

Next we create a RecursiveCharacterTextSplitter instance, configuring it with a chunk_size of 100 and a chunk_overlap value of zero. Our approach involves using the length function to measure each chunk based on its character count.



text_splitter = RecursiveCharacterTextSplitter(
    chunk_size = 100,
    chunk_overlap  = 0,
    length_function = len,
)

The RecursiveCharacterTextSplitter offers several methods for performing splits. In our case, we will utilize the split_text method. This method requires a string input representing the text and returns an array of strings, each representing a chunk after the splitting process.



texts = text_splitter.split_text(text)
print(len(texts)) # 11
print(texts[0]) # 'What I Worked On\n\nFebruary 2021'

Upon performing the split our text was successfully divided into a total of 11 separate chunks.

In-Depth Explanation

Just as its name suggests, the RecursiveCharacterTextSplitter employs recursion as the core mechanism to accomplish text splitting. Now, let's take a detailed journey through the process of how our earlier code was capable of achieving this feat.

For our walkthrough, we'll utilize the same text and parameters that we employed during the code implementation. This involves a segment from Paul Graham's essay, and we'll consider a chunk size of 100 characters. The characters we use for splitting will be ['\n\n', '\n', ' ', '' ].

Let's begin with our initial text. Currently presented in human-readable form, our next step involves transforming it into a format that computers can readily comprehend.

Now, the new lines have been converted to \n, which is precisely what we need in order to carry out our splitting process.

Let's select our text. This can be likened to invoking the split_text method on our text.

As mentioned earlier, the RecursiveCharacterTextSplitter attempts to initiate splits using a predefined set of characters. Its first attempt involves the \n\n character, which serves as a means to split by paragraphs. Let's now identify all occurrences of this character within our text.

Once we've located all instances of the \n\n characters, the subsequent step involves executing a split using this character as our designated separator.

Presently, we have four splits. Our next step involves assessing each split to check whether they meet the condition of being smaller than our specified chunk size, which is set at 100 characters.

The first two splits satisfy this condition, thus earning them the label of good splits. Since both segments consist of fewer than 100 characters, we can combine them to create our initial chunk.

Proceeding to the second split, we find ourselves in a situation where further reduction isn't achievable using the \n\n character. Therefore, we proceed to the next character: \n. Our objective here is to execute a split using the \n character and determine if we can achieve a reduction in the split's size.

This operation is akin to invoking the split_text on the second split text, but with the inclusion of the \n character. This is where the concept of recursion comes into play.

Upon executing the split using the \n character, we end up with two splits. The first split qualifies as a good split, given that it contains only one character. However, the second split surpasses our designated chunk size.

Consequently, we need to invoke the split_text method on this particular split once again. However, this time we'll employ a split using the next character in our character list, which happens to be the ' ' character.

Finally, we have successfully decreased the split size. Now, we proceed to iterate through each split in order to perform a merge. The guiding principle for these merges is that no resulting merged split should exceed our designated chunk size of 100 characters.

Following the merge, we end up with four chunks, each adhering to our condition that a chunk should not surpass 100 characters.

Now, let's revisit the original text splits and identify which split remains to be processed.

We still have one split that is greater than our chunk size. We repeat the same procedures again.

We initiate the split using the new line character as the separator.

We perform a split using spaces as the separators.

Next, we proceed with a merge, ensuring that no merged segments exceed the defined chunk size.

After going through the entire process, we arrive at generating eleven individual chunks. Each of these eleven chunks successfully adheres to the 100-character limit.
This outcome aligns precisely with what we achieved programmatically.

And there we have it. We've delved into the inner workings of LangChain's RecursiveCharacterTextSplitter. For those who are intrigued, you can explore the source code here. If you found this article informative, please consider showing your appreciation with a reaction: 💖 🦄 🤯 🙌 🔥

A quick introduction to language models

Youdiowei Eteimorde — Fri, 04 Aug 2023 10:20:29 +0000

A language model is a computer program capable of understanding text in the form of natural language. This understanding enables it to achieve tasks such as predicting the sentiment of the text, performing grammatical corrections, translating languages, or generating new text. The generation aspect is currently capturing the interest of everyone.

With text generation, we can still perform the other tasks. The performance of a language model depends on its size: the larger the model, the better its performance. We have always had language models; examples include autocomplete on our phones and spam filters. However, these models were basic and could only perform one task at a time.

The models that are currently astonishing the world are large, hence the name Large Language Models.

Prompting

A compiler and a language model aren't so different; they both work with text. However, a compiler takes in structured text in the form of computer programs, while a Language Model works with unstructured text in the form of human language. The text it takes in is called prompts.

The output of most language models is text. Let's work on a simple example using the following prompt:

What is the capital of Nigeria?

The Language Model will analyze the prompt and generate the response that best completes it. The output of this prompt is:

The capital of Nigeria is Abuja.

The output essentially completes the input prompt. We can demonstrate this by providing an incomplete prompt:

Nigeria is a ....

This will produce the output:

Nigeria is a country located in West Africa.

OpenAI Completion API

OpenAI offers an API endpoint that allows us to interact with their language model, appropriately named the Completion API. It is a HTTP endpoint so all we need is a HTTP client and API keys then you are set to go.

curl https://api.openai.com/v1/completions   
 -H "Content-Type: application/json"   
 -H "Authorization: Bearer $OPENAI_API_KEY"
 -d '{
    "model": "text-davinci-003",
    "prompt": "What is the capital of Nigeria?"
  }'

I utilized curl as my HTTP client and stored my API key in the environment variable $OPENAI_API_KEY. I then passed it as a Bearer token in the header. The response to this request should resemble the following:

{
  "id": "cmpl-7jjcu1cmQqUZy7qawoQ3rlUBgmrqh",
  "object": "text_completion",
  "created": 1691134504,
  "model": "text-davinci-003",
  "choices": [
    {
      "text": "\n\nThe capital of Nigeria is Abuja.",
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 7,
    "completion_tokens": 10,
    "total_tokens": 17
  }
}

The response contains several pieces of helpful information, but we are primarily interested in the choices field within our response.

"choices": [
  {
    "text": "\n\nThe capital of Nigeria is Abuja.",
    "index": 0,
    "logprobs": null,
    "finish_reason": "stop"
  }
]

The choices field contains an array of possible responses but we have only one response. We can obtain our completed text from the text field.

This was a concise introduction to language models. Their primary function is to complete the preceding text they receive. If you're keen on delving deeper into the concept of prompting, you can explore this site. Additionally, for detailed information about OpenAI's completions endpoint, take a look at their documentation.

Building a Machine Learning Model with tensorflow.js

Youdiowei Eteimorde — Thu, 15 Jun 2023 06:48:52 +0000

The most important thing in all of machine learning is the model. A model is a computer program that is trained using a dataset to recognize patterns and perform a given task. It emphasizes the training aspect, as opposed to regular programs that are manually programmed by humans. Instead, a machine learning model learns from data using an algorithm. In this article, we will be building a machine learning model with TensorFlow.js.

The Task

Before we begin building our model, we need to define the task at hand.

The image above displays a graph that plots a series of dots. Our goal is to find a straight line that can best fit these dots. This problem is known as linear regression. The line we are searching for can be considered as our model. So, how do we find this line?

We humans can easily draw a line that fits the data. So the goal is to train our computer program to do the same.

We will explore two approaches: one using statistical methods without machine learning, which utilizes mathematical equations to find the best fit line, and the other using machine learning to find the best fit line for us.

The dataset

First, we need to create a dataset. We are going to use TensorFlow.js operations to create an artificial dataset. We will use an equation to generate the data. Here's the equation:

y = m x + c

This equation represents a straight line, where m is the slope and c is the intercept. x is the input and y is the output. Let's randomly select values for m and c. For m, let's choose 2, and for c, let's choose 1. This gives us the equation:

y = 2 x + 1

However, this equation will only give us a straight line. To make the data entries in our dataset unique, we need to add noise. Let's modify the equation:

y = 2 x + 1 + n o i se

For our x values, we are going to generate one hundred values between 0 and 1. The value of y depends on x. We can use the equation to create new values of y. The equation says that 2 times x plus 1 plus random noise will give us y.

Now let's generate x values in code:



const x_data = tf.linspace(0, 1, 100);

The code above uses the linspace function to generate 100 values between 0 and 1. Let's convert the equation $y = 2 x + 1 + n o i se$ into code using TensorFlow.js:



const y_data = tf.add(tf.mul(2, x_data), 1).add(tf.randomNormal([100], 0, 0.1));

These operations multiply x by 2, add 1, and then add random noise to generate the corresponding y values for the dataset.

Ordinary Least squares

The mathematical technique we are going to use is called Ordinary least squares. This technique aims to find a line that best fits the dataset by calculating the difference between the predicted value and the expected value. Here is the equation:

y_i = \beta_0 + \beta_1x_i

$y_i$ represents our dependent variable or the y-coordinate in our graph. $x_i$ represents our independent variable or the x-coordinate in the graph. $β0\beta_0$ represents the intercept, and $β1\beta_1$ represents the slope. From our dataset, we obtain values for $x_i$ and $y_i$ . To calculate the values of $β0\beta_0$ and $β1\beta_1$ , we use the following equation:

\beta_1 = \frac{\sum((x_i - \bar{x})(y_i - \bar{y}))}{\sum((x_i - \bar{x})^2)}

\beta_0 = \bar{y} - \beta_1 \bar{x}

Let's convert the equation above into code. $xˉ\bar{x}$ and $yˉ\bar{y}$ are mean values of $x$ and $y$ . In TensorFlow.js, we can represent them as:



const xMean = tf.mean(x);
const yMean = tf.mean(y);

The numerator of the first equation $∑((xi−xˉ)(yi−yˉ))\sum((x_i - \bar{x})(y_i - \bar{y}))$ can be represented as:



const numerator = tf.sum(tf.mul(tf.sub(x, xMean), tf.sub(y, yMean)));

While the denominator $∑((xi−xˉ)2)\sum((x_i - \bar{x})^2)$ can be represented as:



const denominator = tf.sum(tf.square(tf.sub(x, xMean)));

Now we can obtain our slope( $β1\beta_1$ ) and intercept( $β0\beta_0$ ):



const slope = numerator.div(denominator);
const intercept = yMean.sub(slope.mul(xMean));

We can combine everything into a single function:



function calculateInterceptAndSlope(x, y) {

  const xMean = tf.mean(x);
  const yMean = tf.mean(y);

  const numerator = tf.sum(tf.mul(tf.sub(x, xMean), tf.sub(y, yMean)));
  const denominator = tf.sum(tf.square(tf.sub(x, xMean)));

  const slope = numerator.div(denominator);
  const intercept = yMean.sub(slope.mul(xMean));

  return { intercept: intercept.dataSync()[0], slope: slope.dataSync()[0] };
}

Let's provide the function with the data we defined earlier:



const { intercept, slope } = calculateInterceptAndSlope(x_data, y_data);

console.log('Intercept:', intercept); // Intercept: 0.980187177658081

console.log('Slope:', slope); // Slope: 2.0238144397735596

Our calculateInterceptAndSlope function calculates the intercept as 0.980187177658081 and the slope as 2.0238144397735596, which is close to the values we defined. This approach of using mathematical equations is effective, but it requires us to specify the relationship between the data variables in the form of an equation.

However, what we truly desire is a method that can automatically discern the relationship within the data without relying on a predefined equation.

Machine Learning Model

A machine learning model is a program that has the ability to identify and discover patterns within data without being manually programmed. Instead, it undergoes a training process where it learns from the data. Let's explore the training process of a machine learning model in more detail.

Training process of a model

The training process of a model involves three essential components:

The model
Loss function
Optimizer

Here's the process: The model takes an input value x and produces a prediction. This prediction is then compared to the actual value using the loss function. The loss function measures the performance of the model during training and returns a loss value. A high loss value indicates poor performance, while a low loss value indicates good performance. The optimizer is used to adjust the model's parameters in order to minimize the loss value.

This process is repeated multiple times, with each iteration referred to as an epoch. The number of epochs is determined by the creator of the model. For example, if the model is trained for 50 epochs, it means the training data is passed through the model, loss function, and optimizer 50 times.

Building a model

To create a model in TensorFlow.js, we can utilize the tf.sequential function:



const model = tf.sequential();

A model consists of layers, and these layers have sets of values called weights attached to them. The weights are what are adjusted during the training process. Upon creation of the model, the weights are randomly initialized. Data flows from the input through each corresponding layer before reaching the output.

Let's create a layer for our model:



tf.layers.dense({ units: 1, inputShape: [1] })

Here, we have defined what is known as a dense layer. There are different types of layers, but for now, we will focus on dense layers. The dense layer takes in a JavaScript object { units: 1, inputShape: [1] }, which contains two keys: units and inputShape. The units parameter defines the number of weights our layer will have, and in this case, it is set to one.

The weight in a dense layer is analogous to the slope in the ordinary least squares method. It can be any number of our choosing and doesn't have to be limited to one. Additionally, each layer has an associated bias value, which is automatically incorporated by TensorFlow.js. We can add the layer to our model using the add method provided by the defined model.

The inputShape key specifies the shape of our input data, particularly when our model takes in a scalar value.



model.add(tf.layers.dense({ units: 1, inputShape: [1] }));

With this, we have defined our model. Here is a diagram illustrating our model:

Let's experiment with different model configurations:



const model = tf.sequential();
model.add(tf.layers.dense({ units: 3, inputShape: [1] }));

The code above creates a model with one layer that takes one value as input. This layer has 3 units and 1 bias term, which is automatically added for us.

Let's go deeper and create a model with two layers.



const model = tf.sequential();
model.add(tf.layers.dense({ units: 3, inputShape: [1] }));
model.add(tf.layers.dense({ units: 2 }));

The second layer has two units and a bias attached to it. Here's the corresponding diagram:

All the units from Layer 1 have a connection to individual units of the next layer, which is why it is called a dense layer.

Let's get back to our task and place our original model configuration into a function for simplicity.



function createModel() {
  const model = tf.sequential();
  model.add(tf.layers.dense({ units: 1, inputShape: [1] }));
  return model;
}

This function returns a model. Currently, the model has randomly initialized weights and is not yet trained. The goal is to train the model to fit our data, and that's where the training process comes in. To train our model, we need two things: a loss function and an optimizer.

Loss function

The loss function, also known as the cost function, is a way for the model to measure its performance. It quantifies the difference between the predicted values and the actual values. TensorFlow.js provides several pre-defined loss functions, but for a better understanding, let's build one from scratch. We will use the mean squared error (MSE) as our loss function.

\text{MSE}(\text{predicted}, \text{actual}) = \frac{1}{n} \sum_{i=1}^{n} (\text{predicted}_i - \text{actual}_i)^2

The mean squared error (MSE) measures the difference between the predicted values and the actual values by squaring the difference to avoid negative values. The mean of these squared differences is then calculated. Here's the TensorFlow.js code equivalent for the MSE loss function:



function loss(predicted, actual) {
  return predicted.sub(actual).square().mean();
}

The optiimizer

While the loss function is used to evaluate the model's performance, the optimizer is responsible for updating the weights of the model based on the information provided by the loss function. There are several algorithms that can be used to optimize the weights, and in this case, we will use an algorithm called Stochastic Gradient Descent (SGD). TensorFlow.js provides an implementation of SGD that we can use.

The tf.train.sgd function implements SGD and takes one argument called the training step. The process of optimizing the weights of a model is a search problem.

The training step determines the size of the jumps taken during the optimization process. A higher training step value may lead to faster convergence to the optimal solution, but there's a risk of overshooting it. On the other hand, a lower training step will take longer but can provide a more thorough search for the optimal solution.



tf.train.sgd(0.1) // SGD with a training step of 0.1

The training step

Now let's put everything together and train our model. First, we need to create our model using the createModel function we defined earlier. Then, we add the loss function we created and the optimizer to our model using the compile method.



const model = createModel();
model.compile({ optimizer: tf.train.sgd(0.1), loss });

To train our model, we use the fit method. This method takes in our inputs and their corresponding outputs, which in our case are the x_data and y_data variables. Additionally, we need to provide additional information as an object.



model.fit(x_data, y_data, {
  epochs: 50,
  callbacks: {
    onEpochEnd: (epoch, logs) => {
      console.log(`Epoch ${epoch + 1}: Loss = ${logs.loss}`);
    },
  },
});

The third argument is an object that contains the epochs and callbacks keys. The epochs value determines how many times our dataset will go through the entire training process. The callbacks value is an object that defines functions to be executed at specific times during the training process. In our case, we have the onEpochEnd callback, which runs at the end of every epoch.

The code above won't work yet because model.fit is asynchronous we have to await it and place it in an asynchronous function.



async function trainModel() {
  const history = await model.fit(x_data, y_data, {
    epochs: 50,
    callbacks: {
      onEpochEnd: (epoch, logs) => {
        console.log(`Epoch ${epoch + 1}: Loss = ${logs.loss}`);
    },
  },
});

The process of using a model after training is called inference. Let's create a function for performing inference with our trained model.



async function inference(){
  // start the training process
  await trainModel();

  // Get the weights of the model
  const [ unit, bias ] = model.getWeights();

  console.log('Unit:', unit.dataSync()[0]);
  console.log('Bias:', bias.dataSync()[0])

  // predict a singular value
  const y = model.predict(tf.tensor([2]));
  console.log(y.dataSync()[0]);
}

This function calls the trainModel function to initiate the training process. After training, we obtain the trained weights from the model's getWeights method. Then, we use the model's predict method to make actual predictions on our data values.

The dataSync() method allows us to get the tensor data values in a synchronous manner. This is a blocking operation, but there's an asynchronous equivalent called data.

Let's summarize everything we have done so far:



const tf = require('@tensorflow/tfjs');

// Define a linear regression model
function createModel() {
  const model = tf.sequential();
  model.add(tf.layers.dense({ units: 1, inputShape: [1] }));
  return model;
}

// Prepare the data
const x_data = tf.linspace(0, 1, 100);
const y_data = tf.add(tf.mul(2, x_data), 1).add(tf.randomNormal([100], 0, 0.1));

// Define the loss function
function loss(predicted, actual) {
  return predicted.sub(actual).square().mean();
}

// Compile the model
const model = createModel();
model.compile({ optimizer: tf.train.sgd(0.1), loss });

// Define the function to train the model
async function trainModel() {
  const history = await model.fit(x_data, y_data, {
    epochs: 50,
    callbacks: {
      onEpochEnd: (epoch, logs) => {
        console.log(`Epoch ${epoch + 1}: Loss = ${logs.loss}`);
      },
    },
  });
}

// Define the inference function
async function inference(){
  // start the training process
  await trainModel();

  // Get the weights of the model
  const [ unit, bias ] = model.getWeights();

  console.log('Unit:', unit.dataSync()[0]);
  console.log('Bias:', bias.dataSync()[0])

  // predict a singular value
  const y = model.predict(tf.tensor([2]));
  console.log(y.dataSync()[0]);
}

// Call the inference function
inference();

The code above should work as it is, and here's the log message after the function has been called.



Epoch 1: Loss = 3.0514166355133057
Epoch 2: Loss = 0.3914150893688202
Epoch 3: Loss = 0.16473960876464844
Epoch 4: Loss = 0.12166999280452728
Epoch 5: Loss = 0.11165788769721985
Epoch 6: Loss = 0.1014869213104248
Epoch 7: Loss = 0.09044851362705231
Epoch 8: Loss = 0.0814569965004921
Epoch 9: Loss = 0.07538492977619171
Epoch 10: Loss = 0.0688215121626854
Epoch 11: Loss = 0.06399987637996674
Epoch 12: Loss = 0.05861040949821472
Epoch 13: Loss = 0.054225534200668335
Epoch 14: Loss = 0.04796575382351875
.
.
.
.
Epoch 40: Loss = 0.012772820889949799
Epoch 41: Loss = 0.01268770918250084
Epoch 42: Loss = 0.012648873031139374
Epoch 43: Loss = 0.012167338281869888
Epoch 44: Loss = 0.012036673724651337
Epoch 45: Loss = 0.011740590445697308
Epoch 46: Loss = 0.011509907431900501
Epoch 47: Loss = 0.01139025203883648
Epoch 48: Loss = 0.011547082103788853
Epoch 49: Loss = 0.011259474791586399
Epoch 50: Loss = 0.011122853495180607
Unit: 1.9403504133224487
Bias: 1.0415139198303223
4.922214508056641

From the log message, we can observe that during the first epoch, our loss value is high, but it subsequently reduces. This indicates that our model is improving with each epoch.

By using the model's getWeights method, we obtained the values of the unit and bias as 1.9403504133224487 and 1.0415139198303223 respectively. These values are approximately the same as the ones we obtained using the OLS method. When we provided the value of 2 to our model, it predicted 4.922214508056641.

Comparing both methods

The goal of linear regression is to find a straight line that bests represents our data. Our original equation was $y = 2 x + 1 + n o i se$ . The ML model came up with an equation of:



slope = 1.9403504133224487 // our unit's value from the model's weight
intercept = 1.0415139198303223 // our bias's value from the model's weight
x = 2 // input value
y = x * slope + intercept 
console.log(y) // 4.92221474647522

The solution using the Ordinary Least Squares (OLS) method can be expressed as:



slope = 2.02381443977355967
intercept = 0.980187177658081 
x = 2 // input value
y = x * slope + intercept 
console.log(y) // 5.0278160572052

Both of them are converging towards a slope of 2 and an intercept of 1.

When comparing the Ordinary Least Squares (OLS) method to the ML model, it is important to consider their generality. The OLS method provides a specific solution for linear regression and assumes a linear relationship between the variables. If the data doesn't follow a linear pattern, the OLS method may not yield accurate results, and finding a different equation or using alternative regression techniques becomes necessary.

On the other hand, the ML model offers more flexibility and generalizability. By adjusting the model's configuration, such as adding more layers, we can potentially capture more complex patterns in the data beyond simple linear relationships. This adaptability makes the ML model well-suited for handling a wider range of data patterns and potentially achieving better performance in different scenarios.

In this article, we were introduced to the concept of a machine learning model, specifically the linear model. However, it's important to note that there are other types of models available as well. One such example is the polynomial model, which is used when the data does not follow a straight line trend but exhibits a more complex pattern. Additionally, there are models specifically designed for natural language processing tasks, such as language models, which aim to understand and generate human language by capturing its underlying patterns and relationships. These different types of models allow us to tackle a wide range of problems and data patterns effectively.

All models, regardless of their specific architecture or purpose, share a common structure consisting of layers and weights. This allows them to process and learn from data in a structured manner. For instance, a language model like GPT-3 (Generative Pre-trained Transformer 3) is referred to as "large" due to its extensive scale. GPT-3 is composed of 96 layers and a staggering 175 billion weights, which enables it to capture complex patterns and relationships in language data. The size and complexity of models like GPT-3 contribute to their impressive performance on various natural language processing tasks.

Useful resources

Tutorial by TensorFlow that walks you through building a model using TensorFlow.js
A Guide that introduces the model and layers in TensorFlow.js.
Video introducing the concept of regression by the YouTube channel crash course(statistics).
Video explaining the Ordinary Least Sqaure(OLS) method by YouTuber Organic Chemistry Tutor.

Introduction to Tensorflow.js

Youdiowei Eteimorde — Sat, 18 Mar 2023 18:27:34 +0000

Tensorflow.js is a library that enables machine learning in web applications. It is a part of the Tensorflow ecosystem, which is a deep learning framework written in C++ and provides a Python API.

Tensorflow.js was written from scratch to take advantage of web technologies. It runs on WebGL and has an API that is similar to that of the actual Tensorflow library. It can also run on the server via Node.js. Additionally, Tensorflow has bindings for other languages such as Java, C++, and Rust.

Why tensorflow.js?

Tensorflow.js has the unique advantage of running directly on the web among all the ML libraries. Traditionally, ML models are trained and run directly on the server, and the client makes a request to the server when it is time to use the model. However, with Tensorflow.js, machine learning models can be run directly in the user's browser, eliminating the need for server requests and enabling real-time, interactive applications.

Tensorflow.js is not designed to train large models, but rather it is built for inference. Machine learning models trained in Python can be loaded into Tensorflow.js and used for inference in the user's browser.

Training machine learning models can require a large amount of computational resources. However, once a model has been trained, it can be reused multiple times for inference without requiring as much computation.

Setting up Tensorflow.js

To use Tensorflow.js we can either use it via the script tag:

<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@latest/dist/tf.min.js"></script>

or via npm:

npm install @tensorflow/tfjs

or via yarn:

yarn add @tensorflow/tfjs

The packages above were designed to be used in the web browser. To use Tensorflow.js on nodejs check out this package.

The `Tensor` class

The main class in Tensorflow.js is the Tensor. It represents the tensor data structure. The tensor data structure is immutable. Tensorflow.js contains various methods to work with and manipulate this data structure.

Creating Tensors in tensorflow.js

Tensorflow.js provides several functions to help create tensors.

tf.tensor

The tf.tensor function is a general function for creating tensors. It can create tensors of any shape and data type.

tf.tensor(1); // creating a scalar 
tf.tensor([1,2,3,4]); // creating a vector
tf.tensor([[1,2],[3,4]]); // creating a matrix

tf.scalar

The tf.scalar is a more specialized function. Its job is to create scalars(rank 0 tensors) and they can be of any type.

tf.scalar(2)

It's encouraged to use this rather than tf.tensor when creating scalars. It helps with code readability.

tf.tensor1d

The tf.tensor1d is used to create vectors(rank 1 tensors). Like tf.scalar it is more specialized but helps with code readability.

tf.tensor1d([1,2,3,4]); // creating a vector

tf.tensor2d

The tf.tensor2d is used to create matrices(rank 2 tensors).

tf.tensor2d([[1,2],[3,4]]); // creating a matrix

tf.tensor3d

The tf.tensor3d is used to create rank 3 tensors.

tf.tensor3d([[[1], [2]], [[3], [4]]]); // creating a rank 3 tensor

Tensorflow.js has tensor creation functions up to rank 6. Check out their documentation page.

tf.Variable

The Tensor data structure is immutable but mutability is essential to solve common problems. Tensorflow.js provides the tf.Variable class which is a mutable tensor.

The tf.variable function is used to create tf.Variable objects.

const x = tf.variable(tf.tensor([1, 2, 3]));

In other to mutate them we use the assign method of the tf.Variable class.

x.assign(tf.tensor([4, 5, 6]));

Math operations

Basic mathematical operations like addition, subtraction, multiplication, and division can be performed on tensors.
Operators like +, -, /, and * don't work with tensors rather methods are used.

const a = tf.tensor1d([1, 2, 3, 4]);
const b = tf.tensor1d([5, 6, 7, 8]);

a.add(b) // tf.tensor1d([6, 8, 10, 12]) ✅
tf.add(a, b); // This also works: tf.tensor1d([6, 8, 10, 12]) ✅

a + b // This doesn't work ❌

To perform element-wise addition between tensors, they typically need to have the same shape. However, broadcasting can be used to add together tensors of different shapes.

Broadcasting involves when a smaller tensor stretches to make its shape the same as the bigger tensor.

const s = tf.scalar(2);
const a = tf.tensor1d([1, 2, 3]);

s.add(a); // tf.tensor1d([3, 4, 5]);

In the code above s has smaller shape and it stretches to fit the shape of a. The result ends up having a shape equivalent to a.

const s = tf.scalar(3);
const a = tf.tensor2d([[1, 2, 3, 4], [5, 6, 7, 8]]);

s.add(a); // tf.tensor2d([[4, 5, 6 , 7 ], [8, 9, 10, 11]]);

The same rules of addition apply to other mathematical operations:

// substraction
const a = tf.tensor1d([1, 2, 3, 4]);
const b = tf.tensor1d([5, 6, 7, 8]);


b.sub(a) // tf.tensor1d([4, 4, 4, 4]) ✅
tf.sub(b, a); // This also works: tf.tensor1d([4, 4, 4, 4]) ✅

a - b // This doesn't work ❌

// substraction broadcasting
const s = tf.scalar(3);
const a = tf.tensor1d([1, 2, 3, 4]);

s.sub(a); // tf.tensor1d([2, 1, 0, -1])

// multiplication
const a = tf.tensor1d([1, 2, 3, 4]);
const b = tf.tensor1d([5, 6, 7, 8]);


b.mul(a) // tf.tensor1d([5, 12, 21, 32]) ✅
tf.mul(b, a); // This also works: tf.tensor1d([5, 12, 21, 32]) ✅

a * b // This doesn't work ❌

// multiplication broadcasting
const s = tf.scalar(3);
const a = tf.tensor1d([1, 2, 3, 4]);

s.mul(a); // tf.tensor1d([3, 6, 9, 12]);

// division
const a = tf.tensor1d([1, 2, 3, 4]);
const b = tf.tensor1d([5, 6, 7, 8]);


b.mul(a) // tf.tensor1d([5, 3, 2.3333335, 2]) ✅
tf.mul(b, a); // This also works: tf.tensor1d([5, 3, 2.3333335, 2]) ✅

a / b // This doesn't work ❌

// division broadcasting
const s = tf.scalar(3);
const a = tf.tensor1d([1, 2, 3, 4]);

s.div(a); // tf.tensor1d([3, 1.5, 1, 0.75]);

Tensorflow.js also supports other mathematical operations like floordiv, exponent, and modulus etc.

Javascript has an in-built Math object for working with numbers. Tensorflow.js provide equivalent methods but for tensors.

Javascript	Tensorflow.js
Math.abs	tf.abs
Math.acos	tf.acos
Math.acosh	tf.acosh
Math.ceil	tf.ceil
Math.cos	tf.cos
Math.exp	tf.exp
Math.floor	tf.floor
Math.log	tf.log
Math.sin	tf.sin
Math.tan	tf.tan

Tensorflow.js has other operations that are not present in the Math object like the relu and sigmoid.

Linear algebra

Linear algebra is a branch of mathematics that deals with the study of vectors and matrices and the transformations that can be performed on them. It is used heavily in the field of Machine Learning.Tensorflow.js has several methods for linear algebra.

The dot product

The dot product also known as the scalar product is an operation that takes two vectors and produces a scalar.
It is represented mathematically as:

a⋅b=∑i=1naibi=a1b1+a2b2+⋯+anbn\textbf{a} \cdot \textbf{b} = \sum_{i=1}^{n} {\textbf{a}_i\textbf{b}_i} = {\textbf{a}_1\textbf{b}_1} + {\textbf{a}_2\textbf{b}_2} + \cdots + {\textbf{a}_n\textbf{b}_n}

In Tensorflow.js, we have the tf.dot function for computing the dot product between two tensors.

const a = tf.tensor1d([1, 2, 3, 4]);
const b = tf.tensor1d([5, 6, 7, 8]);

tf.dot(a, b); // 70

tf.Tensor objects also have a dot method to perform dot products.

a.dot(b);

The Norm

The norm also known as the magnitude is a mathematical concept used to measure the size or length of a vector. It is expressed mathematically as:

∣x∣=x12+x22+⋯+xn2|{x}|={\sqrt {x_{1}^{2}+x_{2}^{2}+\cdots +x_{n}^{2}}}

Tensorflow.js provides the method euclideanNorm to calculate the norm of a tensor.

const x = tf.tensor1d([1, 2, 3, 4]);

x.euclideanNorm() // 5.477225303649902

Matrix multiplication

In linear algebra, there are several ways of performing multiplication each of them has its usage. One of them is matrix multiplication which defines how two matrices can be multiplied by each other.

In tensorflow.js, we can perform matrix multiplication using the matMul function.

const a = tf.tensor2d([[1, 2],[3, 4]]);
const b = tf.tensor2d([[2, 4, 6],[8, 10, 12]]);

a.matMul(b); // [[18, 24, 30], [38, 52, 66]]

Transpose

Transposing a matrix is an operation that involves reshaping it by swapping its rows with its columns.

In tensorflow.js we have the Transpose function for performing transpose on a matrix.

const a = tf.tensor2d([[1, 2, 3], [4, 5, 6]] );
tf.transpose(a); // result: [[1, 4],[2, 5],[3, 6]]

Statistics

Tensorflow.js has several functions that can be used to perform statistical operations. One important concept in statistics is distribution, which determines how the data in a dataset is arranged. Tensorflow.js has functions for working with different types of distributions.

Normal distribution

The normal distribution is a probability distribution that models the way many natural phenomena are distributed. It is characterized by a bell-shaped curve, which indicates that the mean (average) value is more likely to occur than any other value. Tensorflow.js has the mean function for checking the mean of a tensor.

const x = tf.tensor1d([1, 2, 3, 4]);

x.mean().print();  // result: 2.5

In addition to the mean, the concept of standard deviation is also important in statistics. It is a value that measures how spread out data points are relative to the mean.

For example, in a room full of people, the average height is more likely to occur than either very tall or very short heights. Tensorflow.js has the randomNormal function that can be used to generate random values that are normally distributed.

tf.randomNormal([2, 2]); // [[0.0635663, 1.2073363],[-1.681078, 0.1550148]]

The code above generates a tensor with normally distributed values of shape 2 by 2.

Uniform distrubtion

In a uniform distribution, no values have a higher tendency of occurring. Rather all values are more likely to occur.
For example, flipping a coin, there's an equal chance of getting heads or tails. It is uniformly distributed, the same applies to a dice.

The randomUniform function of tensorflow.js can be used to generate random values that are uniformly distributed.

tf.randomUniform([2, 2]); // [[0.7101336, 0.6085116],[0.5631702, 0.1648723]]

Automatic differentiation

Automatic differentiation is one of the most important features of Tensorflow.js. Differentiation is the process of finding the rate of change of a function with respect to one of its variables. The derivative of a function describes how it changes as its input variable changes over time.

For example, consider the graph above which plots two functions: the red one is the sigmoid function, while the green one is its derivative. The green function shows how the value of the sigmoid function changes over time, as its input variable changes.

The graph plots the function $x^2$ in blue. We can easily calculate its derivative by using the power rule:

ddxx2=nxn−1\frac{d}{dx}x^2 = nx^n-1

Let's convert the equation above to JavaScript

// Define the function y = x^n
function powerFunction(x, n) {
  return Math.pow(x, n);
}

// Define the derivative of y = x^n using the power rule
function powerRuleDerivative(x, n) {
  return n * Math.pow(x, n-1);
}

// Test the power rule for n = 2 and x = 3
let x = 3;
let n = 2;

let y = powerFunction(x, n);
console.log(y) // 9

// calculate its derivative 
let dydx = powerRuleDerivative(x, n);
console.log(dydx) // reusult: 6

After using the formula above we find out the derivative is $2 x$ . In other to calculate the derivative we needed to use the formula above. Tensorflow.js can calculate the derivative of any function but it doesn't use formulas rather it uses automatic differentiation.

The tf.grad is used to take the derivative of any function provided it returns a tensor.


// f(x) = x ^ 2
const f = x => x.square();

// calculate the derivative of the function
const g = tf.grad(f);

const x = tf.tensor1d([3]);
g(x); // result 6

Automatic differentiation is a key component of the backpropagation algorithm used to train neural networks. It allows for the calculation of gradients efficiently without having to manually derive them using formulas.

In this article, we got introduced to the Tensorflow.js library that brings machine learning to the web. We explored its APIs and discovered it is capable of all sorts of computations.

Tensors are great but ML practitioners rarely work with them they are a bit low level rather most ML practitioners work on a higher level. In the next article, we will build our first ML model using Tensorflow.js.

Useful resources

Check out Coding train video on Tensorflow.js operations here.
Tensorflow.js API docs.
Check out this youtube playlist on linear algebra.
Check out this youtube video on differentiation.
To learn more about distribution watch this
To learn more about the normal distribution watch this

The Tensor: The fundamental data structure of ML

Youdiowei Eteimorde — Fri, 17 Feb 2023 01:15:34 +0000

In the field of computing, data is crucial and the way we store it is vital. Data structures are a means of organizing and storing data within a program, making it possible to manipulate the data using algorithms.

As a web developer, you may be familiar with the data structure known as the array, but in the field of machine learning, a crucial data structure is the tensor. Tensors are multi-dimensional arrays that can store and manipulate large amounts of data, making them essential for deep learning algorithms like neural networks, which rely on them to perform complex mathematical operations and achieve desired results.

What is a Tensor

In many ways, a tensor is similar to an array - in fact, an array can be thought of as a type of tensor. However, a tensor is a multidimensional data structure where all values in it share the same type.

Tensors can be categorized by their ranks, which represent the number of dimensions in the tensor. Additionally, an important attribute of a tensor is its shape, which describes the size of each dimension.
Two tensors can be of the same rank but have different shapes. An analogy to understand tensors is to think of a library containing books and shelves. Let's explore the different tensor ranks.

Rank-0 Tensor

A rank-0 tensor is also known as a scalar, and it has the lowest rank of all tensors. This type of tensor is equivalent to a variable and represents a single value. Rank-0 tensors have no shape because they are just individual values, but they do have a data type.

Using the library analogy, a rank-0 tensor can be thought of as a single book taken down from a shelf.

let r0 = 10 // a variable is equivalent to a rank-zero tensor
typeof r0 // this has a type of number

Rank-1 Tensor

A rank-1 tensor is equivalent to a one-dimensional array and can be thought of as a row of books on a shelf. All values in a rank-1 tensor must be of the same data type. The shape of a rank-1 tensor is defined by its length. It is also known as a vector.

let r1 = [1, 2, 3, 4] // an array is equivalent to a rank-1 tensor
r1.length // This can be considered a rank-1 tensor with 4 scalars

Rank-2 Tensor

A tensor with a rank of 2 is equivalent to a bookshelf with multiple rows stacked upon each other. It is an array of arrays. All rank 2 tensors vary by their shape which is two-dimensional. The example below has the shape of 3 by 3 or simply (3, 3).

[[1, 2, 3],
 [4, 5, 6],
 [7, 8, 9],]

A rank 2 tensor with the shape of 2 by 3 or (2, 3):

[[1, 2],
 [3, 4],
 [5, 6]]

A rank-2 tensor is also known as a matrix.

Rank-3 Tensor

A rank-3 tensor is an array of matrices or rank-2 tensors. The shape of a rank-3 tensor is three-dimensional, and it can vary depending on the number of rows, columns, and depth.

Using the library analogy, a rank-3 tensor can be visualized as a library with several shelves of books. The shelves can represent individual matrices that have their corresponding heights and lengths representing their rows and columns.

A rank-3 tensor with the shape of (3, 3, 2) and a data type of strings would be an array of two 3 by 3 matrices, as shown in the example below:

[[['A', 'B', 'C'],
  ['D', 'E', 'F'],
  ['G', 'I', 'J'],],

 [['K', 'L', 'M'],
  ['N', 'O', 'P'],
  ['Q', 'R', 'S'],]]

Tensors can have ranks beyond rank-3. The idea is the same for higher ranks, where each rank contains an array of the previous rank.
The number of dimensions in a tensor increases with its rank, and the shape of each rank will depend on the amount of data being stored in it.

Why Tensor?

Tensors are similar to arrays in that they store data, but what makes them special is that they are optimized for mathematical operations.
While arrays are mainly designed for storage, and can only perform simple operations like pushing or popping elements, tensors are more versatile and can perform a wide variety of mathematical operations.

In JavaScript, there is no built-in tensor data structure, but we can use a library like TensorFlow.js to work with them.
TensorFlow.js also provides a set of tensor operations that enable us to perform complex mathematical computations on tensors.

For example, we can use the tf.tensor1d method to create a rank-1 tensor, and then use the add method to add it to another tensor of the same shape and type:

let a = tf.tensor1d([1, 2, 3, 4]); 
let b = tf.tensor1d([5, 6, 7, 8]);
a.add(b) // [6, 8, 10, 12]

Tensors in the real world.

Tensors can be used to represent the real-world. Tensors have a versatile structure that enables the representation of all kinds of data. They are used with algorithms like neural networks to perform tasks like Image classification, Object detection, and Natural Language Processing. Here are a few ways of using tensors to represent real-world data:

Images

Images can be represented as tensors.The smallest unit of an image is called the pixel which can be represented as a rank-0 tensor.

A greyscale image is an image that consists of a matrix(rank-2 tensor) with pixels as values. Each pixel has an intensity value that determines if it will be dark or light. its shape is determined by its width and height.

Coloured images can be represented as rank-3 tensors which consist of three sets of images called channels. The three channels represent the intensity of the colors red, green, and blue, respectively.

Videos

Videos can be represented as rank-4 tensors. Videos are made up of stacks of images called frames.

Each frame of the video is a tensor of rank 3 that contains matrices of the 3 color channels.

In this article, I have introduced the fundamental data structure of machine learning, the tensor. In the next article, I will dive deeper into the library known as TensorFlow.js, which is a machine-learning library for JavaScript.

Useful resources

Introduction to Machine Learning

Youdiowei Eteimorde — Fri, 27 Jan 2023 21:40:51 +0000

The field of Machine learning has made a lot of breakthroughs in the last couple of years and it seems like it's only the beginning. The web itself has also evolved in many ways. It is no longer just used to host websites, it can also host web applications that can rival native apps. The beauty of the web is you can share anything with just one link.

In this series of articles, I will introduce Machine learning to you the web developer who is curious about the field. Throughout the series, we will learn various concepts and implement them in code. No prior knowledge of Machine Learning or (mathematics/statistics) is needed all you need to follow up, is to be familiar with JavaScript.

Intermediate knowledge of JavaScript might be needed like ES6 concepts, promises, async/await, and basic data structures. Even if you are not familiar with most of them you can learn them as you go.

What is Machine Learning

Machine learning is the act of teaching a machine to perform a task without giving it any actual instructions. Normally as a programmer, you give instructions to your computer in the form of code but in ML you don't give any instructions rather you give it data.

Data and Datasets

Data is the most important aspect of the Machine learning process. In Machine learning a collection of data is processed into a dataset. Depending on the task the dataset can either be structured as tables or unstructured as in images. Each data point in a dataset contains the actual data and its corresponding label.

Sentence	label
I hate you	0
I love you	1
Go f*ck yourself	0
Burn in Hades	0
Have a nice day	1
This is awesome	1

The table above is a dataset that contains data for the task of sentiment analysis. Sentiment analysis involves predicting the sentiment of a sentence whether it is positive or negative. This dataset has its data(the sentence) and corresponding label that tells you if a sentence is positive (1) or negative(0).

Datasets are usually divided into two:

Training sets
Testing sets

The training set is used to train the ML model while the testing set is used to check the performance of the model after training.

Let's take our sentiment dataset and split it into training and testing sets.

Training set

Sentence	label
I hate you	0
I love you	1
Go f*ck yourself	0
This is awesome	1

Testing set

Sentence	label
Burn in Hades	0
Have a nice day	1

Datasets are usually massive and can contain millions of data points. The more data that is available the better the model will perform.

What is a Model

A model is what takes in the data and produces a desired result. The model goes through a process called training. Training is the act of showing a model of data and its corresponding labels. This training is usually performed on the training set. After supplying the model with data and labels your model will be able to draw insights from the data.

A model could be a mathematical equation or an advanced algorithm that is capable of learning. For the sake of simplicity let's create a dummy model using Javascript classes.

class Model{

  constructor(){
    // Initialize model
  }

  train(X,y){
    // train model using the training set
  }

  test(Xtest, ytest){
    // test model using the testing set
  }

  predict(x){
     // predict the result of x 
  }
}

const model = new Model()

The Model above is abstract and currently hasn't been implemented but it will help show you how actual ML models work. The train method takes in the training set as X and y. X represents the data and y represents its labels. The train method handles the training of the model. The test method takes in a testing set and uses it to evaluate the performance of the model. The training phase usually returns an accuracy score.

model.test(Xtest, ytest)

When it is time to use your model you can use the predict method. Let us say we trained our model on the sentiment dataset above and we want to see if it can properly perform well on data it has never seen.

let sent = "I love this world"
model.predict(sent)

If our model was trained properly it should predict 1. Meaning the sentence is positive.

Machine Learning subcategory

Machine Learning is majorly divided into two:

Supervised ML
Unsupervised ML

Supervised Machine Learning: This branch of ML deals with Models that require labels to perform their task. The model we built above is a supervised model because it needed a label y and its data X. It also has its sub-categories that are:

Regression: This involves trying to predict a continuous value like the price of a product given a set of inputs.
Classification: This involves predicting discrete values. Like if a sentence is positive or negative.

Here are a few supervised learning models:

2.Unsupervised Machine Learning deals with models that don't require labels to perform their task. If our model was unsupervised here's how it will work.

const model = new Model()
model.train(X)

This train method doesn't require y label for training all it needs is data.

Here are a few Unsupervised models:

This article has served as an intro to the field of ML. It is not an in-depth guide. It is more of a build-up of what is to come in the series. For a more in-depth, guide check out the following:

Crash Course AI: A youtube series that explains Machine Learning concepts in a fun and exciting way.
Chapter 1 of Deep learning with JavaScript by Shanqing Cai et al.
What is Machine Learning?: Short video by Google cloud tech.
Intro to ML by the Tensorflow youtube channel.

In the next article, we will be looking at the fundamental data structure of ML, the tensor.

Modeling a Banking system in OOP

Youdiowei Eteimorde — Wed, 11 Jan 2023 20:24:26 +0000

The beauty of Object-oriented programming is that any real-world system can easily be modeled with it. Using abstraction a complex system can be made simple and represented in code.

In this post, I will be modeling a Banking system and along the way, I will be introducing key concepts from OOP.

Abstraction

First of all, let's create an abstraction of the whole system. Banks have accounts that they manage for their clients. A client can create an account. These accounts have balances from which they can make withdrawals and deposits.

Following the Single Responsibility Principle let's create two classes one called Account that manages the balances. Then another class called Bank that manages and creates Accounts.

Implementation

Here's the UML diagram of the system. First, we have to implement an abstract base class from which all account class inherit from.

from abc import ABC, abstractmethod
from datetime import datetime


class AccountBase(ABC):
    """
    Abstract base class for account.
    """

    def __init__(self, account_name: str, bank_name: str, balance: float = 0 ) -> None:
        """
        Initialization method for AccountBase subclasses.
        """

        self._account_name = account_name
        self._bank_name = bank_name
        self._created = datetime.now()
        self._updated = None
        self._balance = balance

    def withdraw(self, amount: float) -> None:
        """
        method to make withdrawals.
        """

        if self._balance >= amount:
            self._balance -= amount
            self._created = datetime.now()
            print(f"Detail: ₦{amount} succesfully withdrawn.")
        else:
            print("Withdrawal failed")

    @abstractmethod
    def deposit(self, amount: float):
        """
        Abstract method to make deposit. To be implemented by subclasses.
        """

        pass

    def balance(self):
        """
        Method to return blance of account
        """

        return self._balance

    def info(self):
        print(f"Bank name: {self._bank_name}")
        print(f"Account name: {self._account_name}")
        print(f"Account balance: ₦{self._balance}")
        print(f"Account created at: {self._created}")
        print(f"Account updated at: {self._updated}")

The AccountBase class is abstract which means no object can be created from it. It contains five instance variables and five methods. Four of which have been implemented. The deposit method is an abstract method that should be implemented by the AccountBase subclasses.

Let's create the first subclass of the AccountBase:

class AdultAccount(AccountBase):
    """
    Adult account class
    """

    def deposit(self, amount: float):
        """
        AdultAccount implementation of deposit.
        """

        self._balance += amount
        self._created = datetime.now()
        print("Deposit succesful")

The AdultAccount class inherits from the AccountBase. It is a concrete class meaning objects can be created from it. It implemented the deposit method and allows an unlimited amount of deposits.

Let's take a look at another subclass:

class StudentAccount(AccountBase):
    """
    Student account class
    """

    ACCOUNT_LIMIT = 4E5

    def deposit(self, amount: float):
        """
        StudentAccount implementation of deposit.
        """

        if (self._balance + amount) > self.ACCOUNT_LIMIT:
            print("Account limit exceeded.")
        else:
            self._balance += amount
            self._updated = datetime.now()
            print("Deposit succesful")

This subclass has a limit to the amount that can be deposited. The ACCOUNT_LIMIT is a class variable

The fact that the two subclasses have different implementations is an example of polymorphism(many forms).

Now let's create the Bank class:

class Bank:
    """
    Factory class for Account class.
    A Bank can create Accounts.
    """

    def __init__(self, name: str):
        """
        Initialization method for bank.
        """

        self._name = name
        self._accounts: list[AccountBase] = []

    def create_adult_account(self, account_name: str, initial_deposit: float) -> AccountBase:
        """
        Creation method to create AdultAccount
        """

        acct = AdultAccount(account_name, self._name, initial_deposit)
        self._accounts.append(acct)
        return acct

    def create_student_account(self, account_name: str, initial_deposit: float) -> AccountBase:
        """
        Creation method to create StudentAccount
        """

        acct = StudentAccount(account_name, self._name, initial_deposit)
        self._accounts.append(acct)
        return acct

    def list_accounts(self) -> list[AccountBase]:
        """
        Return list of accounts in Bank.
        """

        return self._accounts

    def total_amount(self) -> float:
        """
        Return total amount in bank.
        """

        total = sum([acct.balance() for acct in self._accounts])
        return total

The bank class creates accounts. It has two creation methods that can create AdultAccount and StudentAccount respectively. This class is a factory that produces accounts. It contains two instance variables self._name and self._accounts. The self._accounts stores every account created by the object.

Client code

Here's the client code that will use the Bank system:

b = Bank(name="devbank")

acct1 = b.create_adult_account(account_name="EteimZ", initial_deposit=40000) # create an adult account with initial deposits of 40000
acct2 = b.create_student_account(account_name="John", initial_deposit=4000) # create a student account with initial deposits of 4000

for acct in b.list_accounts():
    acct.info() # displays all account's info from bank

b.total_amount() # 44000 
acct1.deposit(8000) # make deposit in acct1
b.total_amount() # 52000

First, we created an instance of the Bank class called b. Using that instance we created an Adult and Student Account. These results are instances of the AccountBase subclasses. Thanks to the creation methods we don't have to worry about creating the objects ourselves.

From each account, we can make deposits and withdrawals. The _accounts instance variable maintains a list of all accounts created from that instance. It can be accessed from the list_accounts() method of the Bank class. The total amount in the bank can also be retrieved from the total_amount() method.

The algorithm used to calculate prime numbers

Youdiowei Eteimorde — Wed, 07 Dec 2022 14:50:30 +0000

Here's a coding challenge, find all the prime numbers with a range of numbers. How would you go about this problem? Could you come up with an efficient algorithm to solve this problem? Don't bother because about two thousand years ago a mathematician named ~~Erastosenes~~ ~~Erathosense~~ Eratosthenes developed an efficient algorithm for calculating prime numbers within a given range.

In this article, I will be discussing his algorithm known as The sieve of Eratosthenes.

The algorithm

We are given the number 1 through 10 and we are asked to find all prime numbers within that range.

numbers = [1,2,3,4,5,6,7,8,9,10]

One requirement of the algorithm is that the first number in the range should be prime. We have to remove one from the list because it isn't a prime number.

numbers = [2,3,4,5,6,7,8,9,10]
           ^

Take the first prime find all multiples of it and remove them from the list.

numbers = [2,3,5,7,9]
             ^

Now we move to the next number which is 3. Find all multiples of it in the list and remove it.

numbers = [2,3,5,7]  
               ^

We move to the next prime number check for all multiples of it in the list. Since there isn't any multiple of 5 in the list we move to the next number.

numbers = [2,3,5,7]
                 ^

We are at the end of the list and there isn't any other number. All the numbers in the list are currently prime. That's why it is called a sieve because it filters all non-prime numbers.

Implementation

Here's a python implementation of the sieve of Eratosthenes.

# maximum number in the range
n = 100

# prime numbers
primes = []

# list of numbers
numbers = list(range(2, n+1))

while numbers:

    # Assume the first number is prime
    prime = numbers.pop(0)
    # Add prime numbers to the list
    primes.append(prime)

    # remove all numbers divisible by the prime number
    numbers = [ x for x in numbers if x % prime != 0 ]

print(primes)

# [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

Time complexity

The time complexity of the algorithm is O(n * log(log(n))). This means it is almost linear but also depends on the logarithm of the logarithm of n.

DEV Community: Youdiowei Eteimorde

How AI generates images from noise

Diffusion models

Reverse diffusion

The Need for Time Steps

Image Synthesis

"Everything is a file" Explained

What Is a File in the Real World

Computer files on storage devices

Everything isn't a file rather everything is file like

Reason why everything is a file

How python's Multithreading differs from other languages

What is a process?

Threads explained

Making threads run in parallel

What exactly is the GIL?

Why the GIL?

How to escape the GIL

Is multithreading useless in python

Running Large Language Models on the CPU

Introducing LLaMa

How are LLMs parameters stored

What is Quantization?

Applying 4 bit quantization to LLaMa7B

The GGML ecosystem

Introduction to Llama.cpp

Working with llama-cpp-python

Understanding LangChain's RecursiveCharacterTextSplitter

Quick overview

Code implementation

In-Depth Explanation

A quick introduction to language models

Prompting

OpenAI Completion API

Building a Machine Learning Model with tensorflow.js

The Task

The dataset

Ordinary Least squares

Machine Learning Model

Training process of a model

Building a model

Loss function

The optiimizer

The training step

Comparing both methods

Useful resources

Introduction to Tensorflow.js

Why tensorflow.js?

Setting up Tensorflow.js

The Tensor class

Creating Tensors in tensorflow.js

tf.tensor

tf.scalar

tf.tensor1d

tf.tensor2d

tf.tensor3d

tf.Variable

Math operations

Linear algebra

The dot product

The Norm

Matrix multiplication

Transpose

Statistics

Normal distribution

Uniform distrubtion

Automatic differentiation

Useful resources

The Tensor: The fundamental data structure of ML

What is a Tensor

Rank-0 Tensor

Rank-1 Tensor

Rank-2 Tensor

Rank-3 Tensor

Why Tensor?

Tensors in the real world.

Images

Videos

Useful resources

Introduction to Machine Learning

The `Tensor` class