How ChatGPT and other LLMs work - A 1000 ft view

Karan R Singh — Sun, 21 Jul 2024 14:57:02 +0000

Hi, I am Karan back with a new article on How ChatGPT and other LLMs work, this article will give you a very high level understanding of how this all works not going too deep in concepts today, because i am also learning all this and this is a way to document my learnings through out my journey, there will be series of articles in future on this topic in which I will go deep on many concepts related to AI and ML.

Let's start with basics, what actually AI is and where do ChatGPT like LLMs fall into. So, AI is short for Artificial Intelligence which is a field of computer science which focuses on creating systems to perform tasks that usually requires human intelligence. AI is not new it is there for decades with Machine Learning and Computer Vision, used in recommendations, search engines, classifications etc.

Nowadays, the hot topic in AI is GenAI or Generative AI which generates new content rather than finding or classifying existing content. GenAI contains text gen, image gen, audio gen, video gen etc. Yes, the G in GPT refers to Generative.
The AI which mainly deals with text gen is called as LLM short for Large Language Model. ChatGPT is one of the LLMs in the internet you may had used, there are also others like Google Bard, Anthropic Claude, etc. Today we will have a high level look into LLMs.
So, LLMs consist of three things, we can show it as

LLM = data + architecture + training

Let's assume we are creating a ChatGPT grade LLM named Chiti, we will go step by step

1. Data - Chiti Knowledge Source💡

first we need a massive amount of Text Data for eg. ChatGPT was Pre-trained (P in GPT) on 570GB (1gb ≈ 178m words) of text data, a good chunk of internet. Pre-trained means to initially train a model with large corpus of general data before being fine-tuned or adapted for specific tasks. Pre-training helps the model to learn language patterns, grammar, facts, and some level of reasoning.

Let's say we collected sufficient data by extracting content from the websites, articles, books, etc. and our dataset is ready. Now, comes the most important part creating a Architecture.

2. Architecture - Chiti Brain 🧠.

Here comes the concept of Neural Networks, its like neurons in a human brain. It helps LLMs to understand, learn and respond. When we talk about training a model we talk about training the NN. NN do not understand text, thus we have to convert the data, first in tokens(words or subwords) and then in Embeddings(numbers) this process is called as Encoding (It includes more things), then the encoded data is fed to NN and the response which we get is decoded and showed as response.

There are various types of NNs which have their own purpose. In Modern LLMs we use Transformers(T in GPT), It is a very core and complex concept in LLM, my next article will be on explaining transformers.

3. Training - Chiti Education 🏫.

Model training is the most crucial and expensive part of LLM development, it marks the quality in a LLMs response. In this part we splits our dataset which we have created in first step in two parts (90%-10%), one part for training and second for testing, the training data will contain input and answer both for eg. Input: Tajmahal is in and Answer: Agra. So that the Transformer NN can learn and understand patterns and assign weights and biases to each params in NN (will explain in next article).
Note: the training data is encoded before sending to Transformer NN.

After training is completed, our LLM Chiti will be tested with the test data.

This was the very sky level view of how LLMs work and created, trust me there are many concepts here which i do not mention like Tensors, back propagation, position encodings, layers, etc. I will learn and explain all this in future. If you have any questions do let me know.

Note: Neural Networks require a good maths understanding specially Matrices, so if you are in college, please don't bunk maths lecture.

Thank You,
Karan R Singh

Who & What is behind the magic of tech world

Karan R Singh — Tue, 09 Jul 2024 11:51:02 +0000

Ganpati Bappa Morya 🙏

Hi, this is me with my first article, which will explain you how the industry of the most prominent sector of the world looks like, who works behind the screens of your most used app or site or game, how the development process works and much more. Let's Start.

I am taking example of Instagram, below is the whole system design of Instagram

A system design is a design of the backend architecture of a tech product, the design should be reliable and scalable to handle projected users (unlike our government websites), usually it is done by seniour software engineers in the company, but it is a crucial part in development so everyone should know about this.

let's slowly breakdown the design. In every tech product three things are permanent and everything else are just the branches. Hence, the design can be devided into Client (Frontend), Server (Backend) and Database.

In the Instagram design you can see the first Icon named users it is called client or frontend from which we normal consumer interacts and been devotees of. Instagram have two clients one its APP and other is WEBSITE.

The people who make Apps are called as App developers or more specificaly Android developers or IOS developers and this branch of IT is called Android development.

To be a App developer here are the tech stack you will require,

Android Dev
- Kotlin or Java (recommended Kotlin)
IOS Dev
- Swift
- And a Mac device to develop IOS apps

A third type also exists in which we can make apps for both through one codebase and tech stack

Cross Platform Dev
- React Native (Insta uses this)
- Flutter (recommended if you are starting out)

Note: A App developer should know the fundamentals of server and database also in order to build app and connect them to architecture smoothly, this all three are interconnected

Now, the people who develop Websites is called as Web developers more specificaly Frontend Web Developer and the field is called as Frontend Web Development.
The Fundamentals which is required to do Frontend Web Development is

HTML
CSS
Javascript

the above is Important and should not be ignored, but this is not enough to be hired as FW dev, you need to learn the frameworks of javascript in order to make a site which is more value providing than the plain javascript

the top frameworks are for job market:

React JS (Recommended)
Vue JS
Angular JS

Here the Job market splits into respective frameworks, like React Developer, Vue Developer, Angular Developer etc. React has more demand in the market.

Note: In order to learn the framework you should have a good amount of knowledge in javascript

If you had choosen to learn react i will suggest you to must learn Nextjs which is the superset of react and it enables you to stand in the global market.

The above was the frontend world of app and web, there is a world also exists for pc apps but it is a topic for an another day.

Now after the user icon you can see various things, here the actual business logic is written down this is called server or backend.

I found a goldmine video on youtube which will explain backend like no one could, watch now 👇

Backend web development - a complete overview - YouTube

We learn every backend technology we need to build a web application. We go over backend programming languages, backend web frameworks, databases, APIs, REST...

youtube.com

Backend is really very vast.

Now, the person who knows frontend and backend and able to make product from start to finish, from development to deployment is known as Full Stack Developer and yes I am a Full Stack Developer (my portfolio), but how do i learned all this without spending a peny? the answer is youtube. But youtube is scatered, right? So, i will tell you how do i learned all this and what is the thaught process behind while learning a new tech.

Whichever thing you are learning first know the path, watch roadmap videos of the choosen topic from youtube
Watch youtube tutorials which are project building oriented not theory based for specified topic in the roadmap
Code along with the tutorial, its okay to make initial projects by watching tutorials (max 2).
Try to build something yourself, feel the feeling of being a creator.

So yes this was my approach towards learning from a scatterd platform, but believe we can do anything, build anything, be anything, if we know something that can be implemented in everything.