DEV Community: King

Rag Concept

King — Tue, 15 Oct 2024 15:31:57 +0000

Introduction to RAG

Retrieval-augmented generation (RAG) is a technique that boosts the performance of Large Language Models (LLMs) by incorporating specific datasets relevant to the task. While LLMs are pre-trained on vast amounts of general data, they may not always have access to domain-specific information necessary for niche applications. RAG addresses this limitation by integrating external datasets, improving the LLM's ability to generate relevant and accurate responses for specific queries.

At its core, RAG works by creating an index of the user-provided data, enabling the model to retrieve the most pertinent information during the query process. This indexed data, along with the user's query, forms a more accurate prompt, leading to more context-aware responses from the LLM. RAG is especially valuable for applications like chatbots or document query systems, where users need answers based on specific data sources rather than general knowledge.

Key Stages in the RAG Workflow

The RAG process can be broken down into five essential stages, each critical for the successful implementation of this approach. Let's take a look at these stages:

Data Loading

The first step involves loading your data into the processing pipeline. The data can come in various formats—PDFs, databases, web content, or APIs. Tools such as LlamaHub simplify this task by offering connectors to different data sources, making it easy to import and prepare the data for further processing.

Indexing

Indexing is the process of transforming your data into a format that is easily searchable. This typically involves generating vector embeddings—numerical representations that capture the essence of the data. These embeddings allow the system to identify contextually relevant information during the query stage. Metadata can also be attached during indexing to enhance retrieval accuracy.

Storing

After the data has been indexed, it is crucial to store the index and associated metadata. This avoids the need to re-index the data in future sessions, saving time and computing resources. Efficient storage ensures that the system can quickly access the index when a query is made.

Querying

With the data indexed and stored, the next step is querying. The RAG framework allows various querying techniques, including multi-step queries and hybrid methods. These queries leverage both the LLM’s capabilities and the indexed data, ensuring that the most relevant chunks of information are retrieved.

Evaluation

Finally, it's important to evaluate how well your RAG implementation performs. Metrics such as accuracy, speed, and relevance can help measure effectiveness. Regular evaluations can also highlight areas for improvement as you update or modify the pipeline.

Building a RAG-Based Query System with LlamaIndex

Let's walk through how to build a RAG system using LlamaIndex, which allows you to query specific data sources like PDFs. For this demonstration, we'll use data from titanic.txt

Loading Data

Access your data (in this case, titanic.txt) and load it into LlamaIndex:

Create Document file

Indexing the data

Defining query tools With the query engines set up, we create tools that allow you to interact with them:

Static Methods

King — Tue, 08 Oct 2024 15:42:25 +0000

Understanding Static Methods in TypeScript

When building applications, especially in object-oriented programming, you often hear about static methods. These methods are used differently compared to instance methods, and knowing when and how to use them can be incredibly useful in organizing your code.

What is a Static Method?

A static method is a function associated with a class rather than instances of the class. In TypeScript, static methods belong to the class itself and can be called without creating an object instance. This means that static methods are called on the class itself, rather than on objects created from the class.

When is it Ideal to Use a Static Method?

Static methods are perfect for situations where:

The function performs a task that doesn't require access to instance properties or methods.
You need utility functions (e.g., math calculations, helper functions) that don't depend on the object's state.
To enforce that certain operations are global to the class rather than tied to specific instances.

Practical Use Cases for Static Methods

Utility Methods

Static methods are frequently used to create utility functions. For example, functions that calculate something based on input values without needing any data from class instances.

Factory Methods

Static methods can also be used for object creation patterns like factory methods. A factory method returns a new instance of a class from a static method without directly using the new keyword.

Global Counters or States

You can use static methods to handle counters or maintain global states within the class.

Static Methods vs. Instance Methods

The major difference between static methods and instance methods lies in how they are invoked and what they operate on.

Static Methods are tied to the class itself, not to an instance of the class. They cannot access or modify instance properties directly.
Instance Methods, on the other hand, require you to create an instance of the class. These methods can access and modify the object's properties.

In this example, greet() is an instance method, while info() is a static method. The instance method relies on the name property of the created object, while the static method can be called directly from the class.

Conclusion

Static methods offer a powerful way to create utility functions, maintain global states, and streamline certain operations that don't rely on an instance's data. They're perfect for actions related to the class itself rather than any particular object of the class. By understanding when to use static methods, you can write cleaner, more organized, and more efficient TypeScript code.

Embracing Llamaindex

King — Tue, 01 Oct 2024 13:36:47 +0000

LlamaIndex provides a straightforward approach that allows users to input custom data into any Large Language Model (LLM) to influence the type of responses generated. This functionality makes it easier to tailor LLM outputs to specific needs or datasets.

As I continue to explore the framework, my interest in understanding its underlying mechanisms has grown. I am curious about how the interaction between the custom data and the LLM functions and the technical processes involved in shaping the results.

Given this interest, I intend to contribute to the LlamaIndex open-source project. I will focus on identifying and resolving bugs, enhancing the existing documentation for clarity and completeness, and developing integrations that can expand the project's usability. I see each contribution as an opportunity to learn more about LLMs while also improving the platform.

Additionally, I look forward to engaging with the open-source community around LlamaIndex. Collaborating with others with similar goals and exchanging knowledge and ideas as we work together to advance the project will be valuable.

Why Next.js ??

King — Wed, 26 Jan 2022 22:39:28 +0000

This article is based on personal experience. I have spent a year learning and using React and i have also spent a couple of months on Next.js. Below is my opinion on both and why i prefer Next.js

INTRO

I have enjoyed using React.js for the past year. It makes creating UI's simpler and that is what i appreciate about it. As a frontend developer, it is only right to be curious about trending tools and frameworks from time to time hence i tried my hands on Next.js and i was totally blown away.

What is Next. js ?

Nextjs is an open source React framework created by Vercel. What makes it efficient is that it enables several extra features, including pre-rendering, web app preview and fast compilation.

What is React. js ?

React is a JavaScript library developed by Facebook now Meta , that is traditionally used to build sing page web applications rendered in the client's browser with JavaScript.

Difference between React.js and Next.js

Just like many other frameworks, Next.js and React.js have their perks and drawbacks. The main differences between these two are based on the following paramters which are essential in Software development.

Documentation

For a new user of a framework, documentation reliability is a must. The best of documentations makes learning how to use the product easy, provides a limited number of user entry points, simplify the product and help cut support costs. Next.js provides a "learn-by-doing" documentation set that takes you through a number of basic features like component creation, serverside rendering and many more. React also takes a similar route with instructional writeups and videos that teaces the fundamentals.

Easiness to Learn

With the availability of a ton of resources of both React and Next.js , learning these frameworks is made easier. However, React has a relatively lower degree of difficulty in learning. Also, you really don't have any choice here since you must know React if you want to know Next.js.

Set-up

Facebook created create-react-app which comes with a set configuration for using React hence you would have to disconnect from the predetermined create-react-app setup if you want to configure React yourself. It doesn't come with great support for configuration making it difficult to make definite changes.

However, with Next.js you can manually configure files such as eslintrc , jest.config , babelrc and it also comes with added support.

Maintenance

In this category, both Next.js and React.js rollout regular stable updates which more often than not improve on existing versions. When it comes to maintenace all you have to do is keep your version up to date.

Popularity and Community

The popularity of a framework and an active community is an important metric because when you face issues when using a product you have a community to fall to who would be willing to help you out in most situations.

React has a very high popularity in the tech ecosystem and the React community has been available for a longer time, there are tons of materials in the form of videos, write ups, blogs, publications and frequently asked questions available online.

Next.js is fast rising and also has a lot of materials online for anyone who wants to learn or enquire.

Talent Pool

Popularity of a framework has a corelation with the number of developers who learn how to use that framework. React developers are in high demand and have a relatively higher talent pool.

Before you can become a Next.js developer you must understand React and know how to use it hence there is no comparison here. It is much more easier to find a React developer than to find a Next.js developer

Advantages and disadvantages

Advantages of React.js

It has a great documentation and an amazing community, anyone with experience with Javascript can easily grasp the concept and use it in building web applications.
With the continual improvements being made all the time it has become simpler building powerful web applications with React.
React addresses the issue of SEO by aiding designers in identifying various web crawlers
React enforces the use of reusable components which makes it easier to use and makes code light weight.

Disadvantages of React

React is a library and its basic functionality is for building UIs hence it requires you to use a lot of third party tools in other for it to react its full potential
The regular updates means developers always have to keep up or risk not being able to efficiently use new versions

Advantages of Next.js

Next.js unlike React, offers a lot of inbuilt tools hence there is a limited requirement for third party tools. You can develop a complex application using only inbuilt tools in Next.js
It offers enhanced User experience as compared to React.
Web applications built with Next.js is faster because of features such a server-side rendering and static site creation.
Images in Next.js are automatically scaled and adjusted to fit into varying viewports

Disadvanatges of Next.js

The community support for Next.js is relatively smaller than that of React.
Next.js has a routing system which is file based and hence are not enough for some use cases. Developers need to have proficient knowledge in Node.js if you intend creating dynamic routes in Next.js

Again, Why Next.js ??

In trying to answer this question and carefully analysing the pros and cons i realise it boils down the what you want to develop with those tools. In the end it has more to do with which you understand better and what you want to use the tools for. I am biased towards Next.js because it makes developing web applications much easier for me.

My Chingu Experience

King — Sun, 19 Dec 2021 18:56:36 +0000

CHINGU ?

I discovered Chingu on from twitter, it is basically a platform for developers and project managers at any level of their journey to gain experience working on real life projects.

How I benefitted from Chingu

I learned about Agile methodologies. It is basically the act of dividing the time between the start of your project and its delivery date into sprints of equal duration in which you'll be completing tasks.As a beginner this helped me to break down tasks into user stories (small chunks) making it easier for me to meet my daily and weekly goals bringing my closer to the end of whatever project I am working on as a whole
How to write a good ReadMe file.Before I joined Chingu I didn't know of the importance of writing a good ReadMe file and I learned that through Chingu. Writing a detailed description of how your project works and how to run it and many other details is just as important as the project itself.
Working in a team. I had the opportunity to work in team for the first time. I appreciated mastered my git cli commands more, resolving merge conflicts and understanding how the git workflow works. As a new developer, this experience equipped me with enough knowledge to continue to work with a team. It gave me a first hand feel of how work is done in teams. I learned from my colleagues and appreciated the knowledge I gained from them.I learned how to collaborate with other developers. I learned how far good and effective communication can bring out the best in people and can create a conducive environment for discourse.
Networking, In this industry competence goes hand in hand in marketing yourself. Marketing yourself is easier when you have a lot of networks. I appreciated the opportunity I had in meeting new developers from all over the world.

Special thanks to all the members of toucans-team-03
Program manager - Cathy
Developer - Ayman
Developer - Mariam
Developer - Opeoluwa
Developer - Carl (Myself)

First time with Node.js

King — Tue, 18 May 2021 16:38:57 +0000

What is node?

When we are developing a website we normally talk about code inside the frontend (browser) or the backend (server). In the past, Javascript was confined to work only in the browser (frontend) which was used to add interactivity to the website. We never used to run Javascript directly on the computer or server but in recent times Node.js has made that possible. Node.js allows us to run javascript on the server side (backend) or directly on computers.

Computers can’t directly understand Javascript or compile them down directly into machine code. However, in browsers there’s an engine called V8 engine written in C++ which compiles Javascript into machine code at runtime allowing the computer to understand Javascript within the context of the browser (browser confined).

Outside browsers, the V8 engine is also present in Node.js written in C++ hence it can also take our Javascript, run it through the V8 compiler running inside of it and compiles it into machine code making it possible to run Javascript on a computer or server and not just a browser.

However, Node is more than just a compiler. It can be used for the following;

1.Read and write files on a computer
2.Connect to a database
3.Act as a server for contents

In brief terms, the role of node in a website is basically to run Javascript in the backend and then handling requests coming in from a browser.

THE GLOBAL OBJECT

The global object is the window object equivalent of node. In node we can get access to the global object with several different methods and properties attached to it just like as do in a browser with the window object. To see these methods and properties we can run;

console.log(global);

and then we get to see those properties.

MODULES IN NODE.JS

Node comes with several different core modules that come with different methods and property. An example is the “os” module which stands for operating system. It gives us every information about the operating system node is running on. To import a module we use the require method hence;

There are several different modules in node and I will explain the use of some of them.

FILE SYSTEM CORE MODULE

The file system core module in node allows us to perform functions such as creating files, reading files, deleting files and many more.

The in-built module used in interacting with the file system is the fs module. We import it by using;

fs.readFile

It takes two arguments, the first argument is the relative path to the file we want to read and the second argument is a function that fires when the reading of file is complete. Inside the callback function we take two things, (err,data). The error “err”, if there was one and the data which is the stuff that we read

writing files

fs.writeFile

It takes three arguments, the first argument is the relative path to the file we want to write to, the second argument is the text we actually want to write to the file which replaces any existing file then finally a callback function.

creating or making a directory

fs.mkdir

Make directory takes two argument. We just need to specify where we want to create the directory and what name to give to it and then we fire a callback function when it is done. The callback function takes an err which informs us when an error has occurred.
However, if we run the make directory code more than once with the same file name we get an error message. In other to ensure that the code only runs if the filename does not exist we use the existSync method

deleting or removing a directory

fs.rdmir

it takes two arguments, relative path to the folder we want to delete and then a callback with an error message.

deleting a file

fs.unlink

It takes two arguments, the relative path to the file we want to delete and then a callback function

We have seen how node can read, create and delete files from a computer now but sometimes those files can be very very very large that it would take a long time for those activities to happen, In other to combat that we use something known as streams. With streams, we can start using a data before it is fully read.

STREAM

reading streams

fs.createReadStream

We create a variable that takes createReadStream which takes an argument of where we want to pass data from through this stream.

the variable we have created now takes in two arguments, the data and then chunk, the chunk represents a chunk of data. The “on” on our readStream I similar to an event listener but this time we are listening to a data event, anytime we get a chunk of data we fire the callback function and we get access to the new stream of data.

writing a stream

fs.createWriteStream

We create a variable that takes fs.createWriteStream and then where we want to write the stream to as an argument. And now, anytime we get a new stream we have to write the chunk to the file we want to write to

piping

This is a more direct way of sending readable data directly into the writeStream and we use the method “pipe”

CLIENTS AND SERVERS

When a user goes to the browser and types the name of a website, a request is sent to the server which also responds with the html, css and javascript used in creating that web page. There are more than a million servers out there, how does the request know which exact server to request from?

That is possible due to IP addresses and Domains.

IP Address and Domain Names

IP addresses are like addresses for computers which are connected to the internet and all computers connected to the internet have unique addresses that help in identifying them. There are some special computers known as hosts meaning they host websites on the internet. If you create and publish a website it will be hosted on a computer somewhere and that computer will have an IP address that would be used to identify it. Now, if we want to connect to that computer acting as the host server we need to know the IP address to be able to do it. We can now type that IP Address in the address bar if we wanted to connect to our server. Now, IP addresses are just a series of numbers and they would be really hard to remember hence we use domain names to mask these IP addresses and when we type these domain names into a browser it will find the IP address associated with it and then it would use that to find the computer hosting the website and it will use that to communicate with that computer. This type of request is known as the GET request. Which occurs anytime we type a domain name in the address bar or we click on a link. This type of communication is via HTTP and it stands for Hyper-Text Transfer Protocol. HTTP simply is a set of instructions that dictates how communication between the browser and the server occurs.

Creating a server

In node we actually write code to create a server and listen in on requests coming to the server.

STEP 1

We have to require the http module and store it in a constant of your naming. Here we used http

STEP 2

createServer()
we called the createServer method on the http constant.

The createServer method takes two arguments, the request object “req” which represents requests made by any user and then the response object “res” which represents the response of the server to whatever request.

As it stands, we have a server with no ability to listen to any request. In other to achieve that we have to create a listener event which takes three arguments. The port number and in this case we are using port number 3000 and then the host name and this time we are using local host and finally a callback function that fires when we start listening in for request

Localhost

A local host is like a domain name on the web however it has a specific IP address called the loopback IP address and that address is 127.0.0.1 and it takes us back to our own computer. That means, when we connect to the localhost domain in our browser, the browser is actually connecting back to our own computer which is then acting as a host for our website. So the host name localhost means to listen for request coming to our own computer.

Port number

The port number represents a specific port on our computer that a server should communicate through. For instance, when you have various communication softwares on your computer like whatsapp, skype and telegram they would all have specific port numbers that their various servers would communicate through on our computer so not to mix up information

In the long run of things, when you type localhost followed by a port number (localhost:300) the browser will know how to communicate with our computer via the particular port number which is where our server is going to be listening

Requests and Response

Requests

The request object contains information about the request that the user sends. When you log out the request object you see various methods available such as the url, the method used and so on.

Response

The response object is what we use to send a response to the browser. We have to formulate some kind of response anytime a request is made or the browser is left hanging. The first step in doing this is to formulate the response headers, the response headers gives the browser some information about what kind of response is coming back to it for example a GET or POST response, what type of data we are sending back.

Setting content type headers

We do that by first using the setHeader method on the res object. There are a lot of different types of headers but in this case we are using the content-type and this could be in either text, html, json or anything but in this instance we are using text/plain.

Now, how do we actually send that data to the browser?

We use the res object and the write method to write to the response as an argument and when we are done we have to end the response using res.end()

This occurred in three steps;

We set the header for the content type
We write whatever content we want to send to the browser
We end the response we send to the browser.

returning html pages

Step 1

We must require the file system as fs

Step 2

We must read the files like we always do

Step 3

We must write the files with res.write

Basic routing

Currently we are returning a single html regardless of the url the user requests for which is wrong, we have to find a way to send specific page url depending on what the user requests for. For instance, if the user requests for the about page we have to find a way of identifying the request and sending the corresponding url.

How do we do this?

We set up a path variable, the path variable is basically the path to the folder containing all the html pages
Then we look at the users request url using the switch. That is, did the user visit eg “/” or “about” or many more
Then we create various cases, so if the request url matches any of the cases then we add the path created to the html file and then break out. If the request doesn’t match any of the cases we then send a default 404 page

Status codes

The status codes describes the type of response being sent to the browser and how successful the response was;
200 – This means everything was okay with the response
301- Signifies a permanent redirect
404 – means the file is not found
500 – means some kind of internal server error

There are so many codes but its best to learn the ranges

100 range – information for the browser
200 range – success codes whereby everything goes to plan
300 range – codes for redirect
400 range – user or client errors
500 range – server error

We can add the statusCode method on the response object so we can get information anytime things go as planned or not

NPM (Node Package Manager)

So far we have seen how to use core modules and pacakges that are built into the node core for example the fs module to work with the file system but what if we want to use some additional packages for extra features in node that are not available in the node core? There are about 1000s of user-made packages that we can use with node and we can install them using npm. It is a tool that we can use to install, update or remove packages onto our computer directly and also into our individual projects. Most of these packages are available at www.npmjs.com.
One of the first packages we are going to install is nodemon for our development server so we don’t have to keep restarting it manually.

TO BE CONTINUED ....

Practical Machine Learning

King — Sun, 20 Dec 2020 17:31:09 +0000

PRACTICAL MACHINE LEARNING

WHO PREDICT THINGS
1.Google -> They use machine learning to show you ads you’re most likely to click based on ads you’ve clicked on in the past to increase revenue
2.Netflix -> They’d show you movies you might be interested in based on movies you’ve watched in the past to increase revenue
3.Insurance companies -> They use machine learning to predict your risk of death so they can increase revenue.

WHY PEOPLE PREDICT

1.For glory – sense of pride
2.For money- win prediction contests eg Kaggle
3.To save lives – better medical decisions.

WHAT ARE THE COMPONENTS OF A PREDICTOR

Question (a very specific and well-defined question..eg what are you trying to predict? And what are you trying to predict it with?)

Input data(you collect the best input data you can to use to predict)

Features (you either use measured characteristics that you have or you might use computation to build features that you think you can use to predict the outcomes that you want)

Algorithm (you can use the machine learning algorithms that you’ve learnt about such as random forest and decision trees)

Parameters (estimate the parameters of those algorithms and use those parameters to apply the algorithm on a new data set

Evaluate (evaluate the algorithm on that new data)

IN SAMPLE VS OUT OF SAMPLE

In sample error is the error you get on the same data you used in training your predictor and it is sometimes called resubstituition error.
Out sample error is the error you get when using your predictor on a new data sometimes called generalization error.
In every data set we have the noise and the signal. The goal of the predictor is to locate the signal and ignore the noise. The signal is the part we use to predict and the noise is the random variation we get in the data set during measurement.
In-sample error occurs because sometimes your predictor algorithm would tune itself to the noise you collected in that particular data set but with a new data set there would be a different noise hence the accuracy goes down a little bit. Hence its always best to test your predictor on new data set to get a realistic idea of how the algorithm works.

PREDICTION STUDY DESIGN

Define your error rate
Split your data set into (training , testing and validation (optional))
Training set must be created to build a model
Testing set is to evaluate the model
Validation set is to validate the model (optional)
On the training set you pick features using cross validation method (the idea is to use the training set to pick features that are most important in your model)
Use cross-validation to pick the prediction functions on the training set to estimate all the parameters you might be interested in to build a model.
If there is no validation model then we apply the best model that we have on the test set just one time. If we apply multiple models on the test set before picking the best one then in some sense we are using the test set to train the model which shouldn’t be the case.
If there is a validation set and a test set then we apply the best prediction model on the test set and refine it a little bit and then apply the result on the validation set just one time.

CROSS VALIDATION

Cross-validation is one of the most widely used tools for detecting relevant features for building models and estimating their parameters

Key ideas;

Accuracy on the training set will always be optimistic (resubstituition accuracy). We are always trying many different models and picking the best one on the training set and as a result the predictive tool could be in tune with the data set and it might not give the best accuracy when a new data set is introduced to it.
The better accuracy comes from an independent set (test set accuracy). However, if we keep using the test set to evaluate the out of sample accuracy, then in a sense the test set has become part of the training set and we still don’t have an outside independent evaluation of the predictive model.
Hence the goal is to build our model entirely on the training set and only evaluate the once on the test set. The only way to do that is by doing cross-evaluation

CROSS-EVALUATION APPROACH

Use training set
Split training set into training set and test set
Build a model on the training set
Evaluate it on the test set
Repeat and average the estimated errors

DIFFERENT WAYS PEOPLE CAN DO TRAINING AND TEST SET

1. Random sub-sampling

Imagine that every observation you are trying to predict arrayed along this axis. The first row represents only the training samples. However we take subset of the training samples and call them the test samples. In the case of the picture above it’s the light grey bars. And then we build our predictor on the dark grey samples and then we apply it to predict the light grey samples and evaluate their accuracy. We do that for several random samples. The first three rows represent 3 different random sampling and then we average the errors to know what the accuracy would be.

2. K-fold cross-evaluation

The idea here is that we break the data set into K equal sized data set. For a 3 fold data set this is what it would look like ( you take the first 1/3 of the first fold, second 1/3 of the second fold and third 1/3 on the third fold). On the first fold we would build the model on the dark grey training data and apply it on the light grey test data. We do this for all three folds. Again we would average the error for all three folds and get the estimate of our error for the predictive model we have built.

3. Leave-one out cross validation

We leave one sample, we build the predictive model on the other sample and then we predict the sample that we left out find the mean errors.

WHAT DATA SHOULD YOU USE?

After identifying the question being asked you should use the best possible data in answering the exact question you’ve been asked.

Machine learning

When you are shown images of different kinds of fishes, how are you able to tell that these are fishes? You are able to tell because the images possess all the features of a fish. If the same image is fed to a machine, how would the machine identify it to be a fish. This is where machine learning comes in, we keep on feeding the machine images of fishes with the label fish until the machine learns all of the features associated with a fish. When it has learned all the features we feed the machine new data to see how much it has learnt. In other words, raw data called training data is given to a machine so it learns all the associated features associated with that data and once the learning is done the machine is given new data or the test data to determine how well it has learned and that is the underlying concept of machine learning.

STEPS IN MACHINE LEARNING

Acquiring data from various sources
Cleaning data. You usually gain insight on the features of data when cleaning it.
Split data into training and testing set
Build your model with the training set. Teach the model all the features of the training set until it has learnt everything.
Testing the accuracy of the model on the test set.
Evaluate the model using confusion matrix or root mean square.

DIFFERENT TYPES OF MACHINE LEARNING ALGORITHMS

Supervised learning: The machine is given training data set to learn all the features and when it’s done we test its accuracy using the test set. Eg. A student being taught by a teacher and writing a test afterwards
Unsupervised learning: Unsupervised learning algorithm draws inference from data which does not have labels. Eg. A student who does not require external teaching and hence learning on their own before a test.
Reinforcement learning: This is where a machine learning algorithm placed in an environment learns the ideal behaviour in order to maximize its performance. A simple feedback is required for the machine to learn best. Eg A self-driving car that obeys the laws of the traffic light to ensure safety of the passengers.

CARET PACKAGE

It stands for Classification and Regression Training. This package was specifically designed to perform machine learning functionalities better and faster.

FUNCTIONS OF THE CARET PACKAGE

There are so many people in the R community creating multiple packages on a daily basis. Caret makes them easier for use by wrapping 100s of these packages together while providing a common interface.
It provides a lot of functionalities for data splitting and sampling
It has a simple functionality for feature selection. That is, it can automatically run through your data set and algorithm and bring out the best permutation that works for the particular problem.
Some of the best models are very difficult to operate because they have a lot of functionalities. Caret can help you simplify various models while maintaining their peak efficiency.

HOW TO CHOOSE FEATURES FOR A MODEL (FEATURE CREATION)

Features are variables that you include to a model that you combine to help you predict the outcome that you want. The best features are those that capture only the relevant information.The levels of feature creation include;

Level 1:

Taking the raw data that you have and turning it into a predictor that you can use. It often takes the form of an image, txt file or a website. It’s very difficult building a model with those forms of raw data until it is summarized into a qualitative or quantitative variable. These new variables are called features or covariates, they are variables that describe the data as much as possible while giving it some compression while making it easier to fit standard machine learning algorithms.
Example; A typical email data would be very difficult to use in your prediction models unless they are summarized into features such as (the number of times a word occurs, the number of capitalized words, the number of currency signs included and the likes)

Level 2:

Transforming tidy features. These are taking features that you’ve already created and making new features out of them. These could be functions or transformation of a feature that might be useful when building a prediction model. Example including the average number of words that occur, the average number of capitalized words and so on.

REFERENCES

https://www.coursera.org/learn/practical-machine-learning/home/welcome

Learning How To Learn

King — Mon, 14 Dec 2020 22:54:02 +0000

LEARNING HOW TO LEARN

The brain is a magnificent organ. It has amazing abilities but unfortunately it doesn’t come with an instruction manual on how to use it to its full potential. Some argue that the greatest gift our brain gives us is its ability to learn new things every day. The goal of this article is for us to gain a better understanding of how to learn so perhaps we could become better learners backed by years of research. This article give an insight on effective new techniques on how to learn and reframe how we think about learning to help reduce our frustration and to increase our understanding. Hopefully, by the end of the article we would have enough ideas to improve on our learning skills

WHAT DO YOU DO WHEN YOU JUST CAN’T FIGURE SOMETHING OUT?

This is a complex situation for human beings. However, if you can understand even the basics of how the brain works you can learn more easily and be less frustrated. Researchers have found that we have two fundamentally different modes of thing. They are; focused mode and diffused mode.

Focused mode: We are familiar with the focusing. This is when you concentrate intently on something you want to learn.

The picture illustrates a thought represented by a pin ball. With the focused mode the problem you are thinking of solving is familiar with the person hence the “pinball” follows a familiar path

Diffuse mode: This more relaxed thinking style is related to a neural resting state.

The picture here describes the diffused mode where you have not thought that thought before so you don’t know what it feels like hence the “pinball” has to develop new pathways. You’d have to form new neural connections. Until you are able to solve a problem or understand a concept.

The bottom-line is when you are learning something new and difficult, your mind needs to go back and forth between the diffused and focused mode, that’s what helps you learn effectively. An example is when you are training for a weight lifting challenge, you just don’t wait till the day before the challenge to start lifting weight. The same way you’d have to start learning little by little until you form familiar neural paths in your brain.

PROCRASTINATION

When you find yourself doing something you really don’t want to do you activate the portion of the brain responsible for pain, the brain naturally enough tries to switch off that negative response by switching to something else. This process is what we know to be procrastination.

HOW TO OVERCOME PROCRASTINATION

Pomodoro technique

25 minutes of no interruption
Focus
Reward yourself when you exceed the 25th minute

Practicing is very important in math and science since this helps in strengthening the neural connections you make while learning. From the picture below you can see neurons linking together through repeated use. The more abstract something is the more important for you to practice to bring those ideas into reality for you. Practice makes permanent.

You are supposed to slowly build neural connections by continuous practice a little everyday instead of crumping everything all in one day

INTRODUCTION TO MEMORY

When you look back on your childhood you are drawing on portions of your brain related to long term memory but when I’m trying to bring a few ideas to mind to understand them to get a concept to solve a problem I’m using my working memory.

Working memory: This has to do with what you are immediately or consciously processing in your mind. The working memory can’t store large amounts of information at a time. You’d have to consistently repeat what you are trying to learn. Repetition is necessary so you don’t forget what you are learning.

Long-term memory: This can store large amount of information at a goal. When you are trying to put information into your long term memory you have to revisit it severally to ensure that you would find that information when you need it later in the future. Long term memory is very important because it is where you store fundamental concepts and techniques that are often involved in whatever you are learning

When you encounter something new you often use your short-term memory to handle it. If you want it in your long term memory it takes a lot of time and practice. In other to achieve this you have to do spaced repetition. Repeating what you are trying to learn over a number of days. Extending practice over a number of days is the most effective way of getting things into your long-term memory.

IMPORTANCE OF SLEEPING IN LEARNING...

Sleeping is such a very important way of keeping your mind healthy and rid of toxins. Getting too little sleep over too long of a time can cause depression, diabetes and a lot of dangerous diseases. When you sleep, you allow your brain to wash away toxins. When you sleep your brain erases the less important aspects of learning and focuses on the most important aspect.

Procrastination and memory

CHUNKING

When you first look at a brand new concept it first doesn’t make much sense as you see in the picture below.

Chunking is the mental leap that allows you to unit bit of information together through meaning. The new logical whole makes the chunk easier to remember and also makes it easier to fit the chunk into the bigger picture of what you are learning..

When you are focusing your attention on something connections of ideas are being made in your brain this is however different from the random connection of the diffused mode. Focusing your attention to connect parts of the brain to tie together ideas is an important part of the focused mode of learning and it is what gets you started in creating a chunk.

The ability to male chunks is greatly reduced when you are stressed or afraid. Chunks are pieces of information that are bind to make meaning.. Its like binding the letters C-A-R-L to form the name CARL.

HOW TO MAKE A CHUNK

Every discipline is a little different. Chunking in history is different from chunking in playing a musical instrument. One is mental and the other is physical. They are both different but similar in ideas.

Focus – The first step is simply to focus your undivided attention on the information you want to chunk devoid of all forms of distractions. When you first begin to learn something, you’re making new neural paths connecting them to existing neural patterns in different part of the brain.
Understanding- The second step is to understand what you are learning. You can figure out the idea or main ideas naturally or at least you can grasp those ideas when you allow the focused mode and diffuse mode in thinking to take turns to figure out the main idea behind what you are trying to learn. Understanding is like a superglue that helps hold together the underlining memory traces together. You can create a chunk without understanding but it would be a useless chunk that won’t fit in with or relate to other material you are learning. Understanding isn’t enough to show that you are done. You have to test yourself shortly after and this creates the neural pattern on how to do it.
Gaining context – So you can see not just when but how to use the chunks. Context means going beyond the immediate problem but going more broadly so you can see not only how but when to use the chunk. This helps you see how your newly formed chunk fits into the bigger picture. Practice helps you broaden your network of neurons that are connected to your chunks, ensuring its not only firm but also accessible for many different paths

HOW TO BECOME A BETTER LEARNER

Physical exercise: As you learn new information new neurons are created in your brain. These neurons die over sometime and physical exercise is one of the best ways to sustain these new neurons created. Exercise helps other organs not just the brain.
Practice makes perfect.

LEARNING USING METAPHORS AND ANALOGIES

One of the best things to do when trying to understand concepts is to create a metaphor or analogy for them and its better if these metaphors are visual. A metaphor is just a way of realizing that one thing is somehow similar to another.
Example I remember when I was in primary one of the best use of this concept was when we were told to imagine the map of Ghana as a pregnant woman. This made it very easy for me to be able to reconstruct this map back then.

These techniques can be applied in all learning fields not just art. Metaphors give a physical understanding behind of the sensual idea behind a process or concept you are trying to understanding.

HOW CHANGING YOUR THOUGHTS CAN CHANGE YOUR LIFE

We can make significant changes in our brain by changing how we think. A key to success is perseverance and coupled with the flexible ability to change your mind and admit errors. Anyone can change their brain so even the less gifted can do marvelous things.

Approaching new information with a goal of learning it on your own can give you a new path into mastering it. Often when you are taught by a single teacher or from one textbook, it’s only when you take other materials or watch lecture videos from other teachers is when you see that relying on one teacher or text book gives you only a partial version of the full reality of the subject. Taking responsibility for your own learning is one of the most important things you can do.

No matter how good or bad you are academically there would be people who would continually criticize you. It is important to tune people out and try to be the best you can be.

THE VALUE OF TEAM WORK

The left hemisphere interprets the world for us. It can cause us to be overconfident.
The right hemisphere of the brain helps us to step back and put our work into big picture perspective. It helps us get onto the right track when tackling problems.
The brain has the potential for rigidity and dogmatism. Be aware that the brain as the ability to make you believe that a wrong way of thinking or the wrong approach to answering a question is right and hence you cling unto it.

One of the best ways of catching a blind spot scenario is to brainstorm or to work in a team because sometimes no matter how much you brainstorm it can never be compared to working in a team with people who are knowledgeable about the topic.

HOW TO PREPARE FOR A TEST

Practice solving numerous problems
Learn with a study group and always ensure that some of the members are smarter than you are.

HOW TO APPROACH A TEST

Always alternate between the hard and easy problem but make sure to start with a hard problem and switch to the easy as soon as you find yourself getting stuck.

This makes efficient use of the brain by allowing your brain by allowing different parts of the brain to work simultaneously on different thoughts.

REFERENCES

https://www.coursera.org/learn/learning-how-to-learn

R programming

King — Wed, 14 Oct 2020 18:21:02 +0000

Cluster Analysis R.

Clustering organizes things that are close together into groups.

How do we define "close"?
How do we group things?
How do we visualize the grouping?
How do we interpret the grouping?

Hierarchical clustering

Hierarchical clustering is a simple way of quickly examining and displaying multi-dimensional data. This technique is usually most useful in the early stages of analysis when you're trying to get an understanding of the data, e.g., finding some pattern or relationship between different factors or variables. As the name suggests hierarchical clustering creates a hierarchy of clusters.

In hierarchical clustering we use the agglomerative approach (bottom up approach). Here, we start with individual data points and we start lumping them together to form a cluster until eventually the whole data becomes one big cluster.

Agglomerative approach

1.Find two points in the data set which are closest together.
2.Merge them together to get a "super point"
3.Remove the initial data points and substitute them with the merged point.
4.Repeat process until you form a tree showing how close things are to each other.

This method requires;

A distance metric : How do you calculate the distance between two point?
Approach for merging points: How do you merge points that have been proven to be the closest point.

A. How do we define close? (Distance metric)

Euclidean distance : Distance between two points in a straight line

Example 1

consider these random points generated. We'll use them to demonstrate hierarchical clustering in this lesson. We'll do this in several steps, but first we have to clarify our terms and
concepts.

Hierarchical clustering is an agglomerative, or bottom-up, approach. In this method, "each observation starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy." This means that we'll find the closest two points and put them together in one cluster, then find the next closest pair in the updated picture, and so forth. We'll repeat this process
until we reach a reasonable stopping place.

Note the word "reasonable". There's a lot of flexibility in this field and how you perform your analysis depends on your problem. Again, Wikipedia tells us, "one can decide to stop clustering either when the clusters are too far apart to be merged (distance criterion) or when there is a sufficiently small number of
clusters (number criterion)."

STEP ONE

How do we define close? This is the most important step and there are several possibilities depending on the questions you're trying to answer and the data you have. Distance or similarity are usually the metrics used. In the above diagram its quite obvious that points 5,6 and 10,11 are the closest points to each other using the distance metric.

There are several ways to measure distance or similarity.
Euclidean distance and correlation similarity are continuous measures, while Manhattan distance is a binary measure. In this blog we'll briefly discuss the first and last of these. It's important that you use a measure of distance that fits your problem.

Euclidean distance

Given two points on a plane, (x1,y1) and (x2,y2), the Euclidean distance is the square root of the sums of the squares of the distances between the two x-coordinates (x1-x2) and the two y-coordinates (y1-y2). You probably recognize this as an application of the Pythagorean theorem which yields the length of the hypotenuse of a right triangle.

Euclidean distance is distance "as the crow flies". Many applications, however,can't realistically use crow-flying distance. Cars, for instance, have to follow roads.

Manhattan or City block distance :

You want to travel from the point at the lower left to the one on the top right. The shortest distance is the Euclidean (the green line "as the crow flies"), but you're limited to the grid, so you have to follow a path similar to those shown in red, blue, or
yellow. These all have the same length (12) which is the number of small gray segments covered by their paths.

Manhattan distance is basically the sum of the absolute values of the distances between each coordinate, so the distance between the points (x1,y1) and (x2,y2) is |x1-x2|+|y1-y2|. As with Euclidean distance, this too generalizes to more than 2 dimensions

Using a dataFrame to demonstrate an agglomerative (bottom-up) technique of hierarchical clustering and create a dendrogram

This is a graph which shows how the 12 points in our dataset cluster together. Two clusters (initially, these are points) that are close are connected with a line, We'll use Euclidean distance as our metric of closeness

I ran a R command dist with the argument dataFrame to compute the distances between all pairs of these points. By default dist uses Euclidean distance as its metric.

You see that the output is a triangular matrix with rows numbered from 2 to 12 and columns numbered from 1 to 11. The minimum distance between two points is denoted by 0.0815 which represents points 5 and 6.

We can put these points in a single cluster and look for another close pair of points

Looking at the original picture we can see than points 10 and 11 are the next pair of points with the shortest distances between them hence we can put them in a cluster. By putting them together we slowly begin with the formation of our dendrogram.

We can keep going like this and pair up individual points, but fortunately, R provides a simple function which you can call which creates a dendrogram for you. It's called hclust().

R's plot labeled everything. The points we saw are at the bottom of the graph, 5 and 6 are connected, as are 10 and 11. Moreover, we see that the original 3 groupings of points are closest together as leaves on the picture

Using the R command abline to draw a horizontal blue line at 1.5 on this plot. This requires 2 arguments, h=1.5 and col="blue"

We see that this blue line intersects 3 vertical lines and this tells us that using the distance 1.5 gives us 3 clusters (1 through 4), (9 through 12), and (5 through 8). We call this a "cut" of our dendrogram. So basically the number of clusters in your data depends on where you draw the line! (We said there's a lot of flexibility here)

Notice that the two original groupings, 5 through 8,
and 9 through 12, are connected with a horizontal line near the top of the graph

COMPLETE LINKAGE

There are several ways to do this. I'll just mention two. The first is called complete linkage and it says that if you're trying to measure a distance between two clusters, take the greatest distance between the pairs of points in those two clusters. Such pairs usually contain one point from each cluster.

So if we were measuring the distance between the two clusters of points (1 through 4) and (5 through 8), using complete linkage as the metric we would use the distance between points 4 and 8 as the measure since this is the largest

The distance between the two clusters of points (9 through 12) and (5 through 8), using complete linkage as the metric, is the distance between points 11 and 8 since this is the largest distance between the pairs of those groups

The distance between the two clusters of points (9 through 12) and (1 through 4), using complete linkage as the metric, is the distance between points 11 and 4

AVERAGE LINKAGE

The second way to measure a distance between two clusters that I'll just mention is called average linkage. First you compute an "average" point in each cluster (think of it as the cluster's center of gravity). You do this by computing the mean (average) x and y coordinates of the points in the cluster. Then you compute the distances between each cluster average to compute the intercluster distance.

HEAT MAP

The last method of visualizing data we'll mention in this lesson concerns heat maps. A heat map is "a graphical representation of data where the individual values contained in a matrix are represented as colors. Heat maps originated in 2D displays of the
values in a data matrix. Larger values were represented by small dark gray or black squares (pixels) and smaller values by lighter squares.

The image below is a sample of a heat map with a dendrogram on the left edge mapping the relationship between the rows. The legend at the top shows how colors relate to values.

R provides a function to produce heat maps. It's called heatmap. This function is called with 2 arguments

This is a very simple heat map - simple because the data isn't very complex. The rows and columns are grouped together as
shown by colors. The top rows (labeled 5, 6, and 7) seem to be in the same group (same colors) while 8 is next to them but colored differently. This matches the dendrogram shown on the left edge. Similarly, 9, 12, 11, and 10 are grouped together (row-wise) along with 3 and 2. These are followed by 1 and 4 which are in a separate group. Column data is treated independently of rows but is also grouped.

Example 2

See how four of the columns are all relatively small numbers and only two (disp and hp) are large? That explains the big difference in colour columns.

K-means Clustering

k-means clustering is another simple way of examining and organizing multi-dimensional data. As with hierarchical clustering, this technique is most useful in the early stages of analysis when you're trying to get an understanding of the data, e.g., finding some pattern or relationship between different factors or variables. R documentation tells us that the k-means method "aims to partition the points into k groups such that the sum of squares from points to the assigned cluster centres is minimized.

To illustrate the method, I'll use these random points generated. I will demonstrate k-means clustering in several steps, but first I'll explain the general idea of what K-means clustering is.

As said, k-means is a partitioning approach which requires that you first guess how many clusters you have (or want). Once you fix this number, you randomly create a "centroid" (a phantom point) for each cluster and assign each point or observation in your dataset to the centroid to which it is closest. Once each point is assigned a centroid, you readjust the centroid's position by making it the average of the points assigned to it.

Once you have repositioned the centroids, you must recalculate the distance of the observations to the centroids and reassign any, if necessary, to the centroid closest to them. Again, once the reassignments are done, readjust the positions of the centroids based on the new cluster membership. The process stops once you
reach an iteration in which no adjustments are made or when you've reached some predetermined maximum number of iterations.

So k-means clustering requires some distance metric (say Euclidean), a hypothesized fixed number of clusters, and an initial guess as to cluster centroids. When it's finished k-means clustering returns a final position of each cluster's centroid as well as the assignment of each data point or observation to a
cluster.

Now we'll step through this process using our random points as our data. The coordinates of these are stored in 2 vectors, x and y. We eyeball the display and guess that there are 3 clusters. We'll pick 3 positions of centroids, one for each cluster

The coordinates of these points are (1,2), (1.8,1) and (2.5,1.5). We'll add these centroids to the plot of our points

The first centroid (1,2) is in red. The second (1.8,1), to the right and below the first, is orange, and the final centroid (2.5,1.5), the furthest to the right, is purple

Now we have to recalculate our centroids so they are the average (center of gravity) of the cluster of points assigned to them. We have to do the x and y coordinates separately. We'll do the x coordinate first. Recall that the vectors x and y hold the respective coordinates of our 12 data points.

We can use the R function tapply which applies "a function over a ragged array". This means that every element of the array is assigned a factor and the function is applied to subsets of the array (identified by the factor vector).

Now that we have new x and new y coordinates for the 3 centroids we can plot them.

We see how the centroids have moved closer to their respective clusters. This is especially true of the second (orange) cluster. Now call the distance function. This will allow us to reassign
the data points to new clusters if necessary

Now that you've gone through an example step by step, good news is that R provides a command to do all this work for you. Unsurprisingly it's called kmeans and, although it has several parameters, we'll just mention four. These are x, (the numeric matrix of data), centers, iter.max, and nstart. The second of these (centers) can be either a number of clusters or a set of initial centroids.The third, iter.max, specifies the maximum number of iterations to go through, and nstart is the number of random starts you want to try if you specify centers as a number.

The program returns the information that the data clustered into 3 clusters each of size 4. It also returns the coordinates of the 3 cluster means, a vector named cluster indicating how the 12 points were partitioned into the clusters, and the sum of squares within each cluster. It also shows all the available components returned by the function

Let's plot the data points color coded according to their cluster

Dimension Reduction

Dimension reduction include two very important techniques known as the principal component analysis (PCA) and singular value decomposition (SVD). PCA and SVD are used in both the exploratory phase and the more formal modelling stage of analysis. We'll focus on the exploratory phase and briefly touch on some of the underlying theory. We'll begin with a motivating example - random data

dataMatrix has 10 columns (and hence 40 rows) of random numbers.
Let's see how the data clusters with the heatmap function.

We can see that even with the clustering that heatmap provides, permuting the rows(observations) and columns (variables) the data still looks random.

Here's the image of the altered data after the pattern has been added. The pattern is clearly visible in the columns of the matrix. The right half is yellower or hotter, indicating higher values in the matrix.

Running R command heatmap again with dataMatrix as its only argument. This will perform a hierarchical cluster analysis on the matrix

Again we see the pattern in the columns of the matrix. As shown in the dendrogram at the top of the display, these split into 2 clusters, the lower numbered columns (1 through 5) and the higher numbered ones (6 through 10).

As data scientists, we'd like to find a smaller set of multivariate variables that are uncorrelated and explain as much variance (or variability) of the data as possible. This is a statistical approach. In other words, we'd like to find the best matrix created with fewer variables (that is, a lower rank matrix) that explains the original data. This is related to data compression.

An example is having two variables U and V each have uncorrelated columns. U's columns are the left singular vectors of X and V's columns are the right singular vectors of X. D is a diagonal matrix, by which we mean that all of its entries not on the diagonal are 0. The diagonal entries of D are the singular values of X.

mat is a 2 by 3 matrix. R provides a function to perform singular value decomposition. It's called svd.

We see that the function returns 3 components, d which holds 2 diagonal elements,u, a 2 by 2 matrix, and v, a 3 by 2 matrix.

PRINCIPAL COMPONENT ANALYSIS

Now we'll talk a little about PCA, Principal Component Analysis, "a simple,non-parametric method for extracting relevant information from confusing datasets. Basically, PCA is a method to reduce a high-dimensional data set to its essential elements (not lose information) and explain the variability in the data. However, SVD and PCA are closely related.

EXAMPLE

First we have to scale mat, our data matrix. This means that we subtract the column mean from every element and divide the result by the column standard deviation. R has a command, scale,
that does this for you.

Now we run the R program prcomp on scale(mat). This will give you the principal components of mat. See if they look familiar

Notice that the principal components of the scaled matrix, shown in the Rotation component of the prcomp output, are the columns of V, the right singular values.Thus, PCA of a scaled matrix yields the V matrix (right singular vectors) of the same scaled matrix

Here's a picture showing the relationship between PCA and SVD for that bigger matrix. I've plotted 10 points (5 are squished together in the bottom left corner). The x-coordinates are the elements of the first principal component (output from prcomp), and the y-coordinates are the elements of the first column of V, the first right singular vector (gotten from running svd). We see that the points all lie on the 45 degree line represented by the equation y=x. So the first column of V IS the first principal component of our bigger data matrix.

Now i'll show you another simple example of how SVD explains variance with a 40 by 10 matrix.

You can see that the left 5 columns are all 0's and the right 5 columns are all 1's.

Here the picture on the left shows the heat map of the Matrix. You can see how the left columns differ from the right ones. The middle plot shows the values of the singular values of the matrix,
i.e., the diagonal elements. Nine of these are 0 and the first is a little above 14. The third plot shows the proportion of the
total each diagonal element represents.

PYTHON PROGRAMMING

King — Mon, 13 Jul 2020 17:24:56 +0000

WHAT IS PROGRAMMING?

Programming is simply giving a list of instructions for the computer to follow, these instructions given are what we know as algorithms. Even though computers are very reliable in following instructions, we can’t give instructions in the natural language we speak because natural languages have a lot of ambiguity, for instance; words or sentences have multiple meanings which computers won’t be able to figure out. Instead we have to give them instructions in a programming language which computers can understand because they have specific set of rules.

How to learn programming languages?

The process of learning programming is more than just learning the rules of the python programming, it’s also about how to break down and solve problems regardless of the programming language. Think of programming as a translation technique where you are trying to translate natural language to programming language. The best way to learn programming is through practice.

PYTHON

Python is a widely used programming language which is very easy read and simple to implement. It is free to use, even for commercial applications hence it is considered as open source. It is known to be a high-level programming language because of its closeness to human language than to machine language. This language is the invention of Guido van Rossum in 1991.

WHAT IS PYTHON USED FOR ?

It is used for manipulating large volumes of data and performing complex mathematics.
It is used to create web applications
It is used to create software
It is used for reading and modifying files.

Data Types

When we write code we write what is known as expressions and when we write expressions the python interpreter computes the value of the expression and every value has a type, that’s the type of data. For instance, when we write an expression such as 10, the python interpreter computes the value of the expression which is 10 and then the type of the value for this expression in this case is an integer.

WHAT ARE THE DATA TYPES IN PYTHON?

Numeric Types: int, float, complex
Sequence Types: list, tuple, string
Mapping Type: dict
Set Types: set, frozenset
Boolean Type: bool
Binary Types bytes

NUMERIC DATA TYPES IN PYTHON

Integers (int) : These are whole numbers. Examples include (1,2,3,4,5)

Float : These are numbers with a dot/point within them. Examples include (1.0,2.0,3.0,4.0,5.0)

How can we find out the type of an object in python?

we use the function type() and the number in parenthesis. When we execute that python tells us whether the number is a float or integer

eg. type(4)
class "int"

type(3.2)
class "float"

Operators and Operands

You can build complex expressions out of simpler ones using operators. Operators are special tokens that represent computations like addition, multiplication and division. The values the operator works on are called operands.

Addition +

5+5
10

Subtraction -

5-4
1

Multiplication * That’s the star symbol

5*5
25

Division / the slash symbol

Anytime we use the division symbol we get a float. For instance 10/3 would give us 3.33. There are times we would want an integer hence we use the truncated division (double slash) and then 10 // 3 would give us 3.

Modular % This gives us the remainder

So 10 % 3 would give us a remainder of 1 because 3 goes into 10 3 times perfectly and then we have a remainder of 1.
Exponential equations . So 42 would give us 16. That means 4 raised to the power 2 giving us 16.

Just like bodmas, python also follows the normal order of operations in math. Bracket eg (1 +1) -> Exponentiation eg 4**2 -> Multiplication & Division -> Addition & Subtraction.

SEQUENCE MUTATION

Sometimes we create an object and along the line we feel the need to change it. For instance, you create a list and along the time you want to keep adding items to it. There are two ways to change a list after you’ve created it;

You make a modified copy of the object
Modify the original

That process is what we call mutation. It can be very confusing however, I’m going to talk about how it works.

SEQUENCE

A sequence is an ordered collection. Basically what that means is that it has a list of items that come in first, second, third order etc and it also has a length. The three types of sequence in python are strings, lists and tuples.

Strings

A string is an ordered collection of characters of a single type. I create a string by assigning a sequence of characters in single, double or triple quotes to a variable. If I want to create a single lined string I use a single or double quote

S = “Hello world”
Print(s)

However, if I want to create a multi-lined string I use triple quotes

S = “”” Hello world
I hope you are all having a nice time
I wish you well”””

Print(s)

A string that contains no characters, often referred to as the empty string, is still considered to be a string. It is simply a sequence of zero characters and is represented by ‘’ or “” (two single or two double quotes with nothing in between).

"Carl is a good man"
'Carl is a good man'

'Carl's a good man'
syntax error <-- This happens because the single quote before the Carl is closed by the single quote before the s. In this instance we correct it by using double quotes.

"Carl's a good man"

Carl = Handsome
Carl
Handsome

If i want lets say the d and s in Handsome i can call it by using the square bracket.
Carl[3:5]
ds <--- in python the first element is always assignment the index value of 0.The first number within a square bracket is always inclusive and the last number is exclusive hence we used [3:5] to call out the 'ds' in Handsome

Methods in strings

There are numerous methods in strings and we are going to use a few.

Carl = Carl.upper
HANDSOME <-- you notice all the letters are in upper case

len(Carl)
7 <--- there are 7 letters in the word "handsome"

LIST

A list is an ordered sequence of characters of any type in a square bracket where every element is identified by an index. Lists are similar to strings, which are ordered collections of characters, except that the elements of a list can have any type and for any one list, the items can be of different types.

There are several ways to create a new list. The simplest is to enclose the elements in square brackets ( [ and ]).

Example;

[10, 20, 30, 40]

["spam", "bungee", "swallow"]

The first example is a list of four integers. The second is a list of three strings. As we said above, the elements of a list don’t have to be the same type. The following list contains a string, a float, an integer, and another list.

["hello", 2.0, 5, [10, 20]]

lets say we want to call out a particular element in the list we use the square brackets;

we can also work backwards using a negative number;

we can also use the slice method to get a range of elements in the list. Always remembering that the first member is inclusive and the last member is excluded;

We can also concatenate the several lists of elements and join them together to make one list;

we can also change an element within a list by equating that variable to another. For instance;

Tuples

A tuple is a sequence of characters of any type just like a list. It is enclosed in parentheses and separated by comma. One way to create a tuple is to write an expression, enclosed in parentheses, that consists of multiple other expressions, separated by commas.

julia = ("Julia", "Roberts", 1967, "Duplicity", 2009, "Actress", "Atlanta, Georgia")

The key difference between lists and tuples is that a tuple is immutable, meaning that its contents can’t be changed after the tuple is created.

To create a tuple with a single element we have to include the final comma, because without the final comma, Python treats the (9) below as an integer in parentheses:

S = (9,)
Print(type(S))

Also;

S = “100”
A = [100]
B = 100

Python views all three as different data types. S is a string, A is a list and B is an integer.

STANDARD INPUT
my text editor of choice is visual code and i created a python file called basic.py and i ran everything at the terminal below.

first of all i use ( cd "path to python file") to navigate into the directorate and i use (exit()) to leave a directorate.

So in python , we can ask the user of our programme questions and user can type in data, strings or numbers and we can take that data into the programme and do something with it and we do that by using the function called (input). For instance;

PYTHON CONDITIONS AND IF STATEMENTS

The if statement allows us to control the flow of our code based on certain conditions. For instance, if the condition is true and it would perform a certain function but if it is false then it performs another function. It follows a cause and effect model.

Python supports the usual logical conditions from mathematics:

Equals: a == b
Not Equals: a != b
Less than: a < b
Less than or equal to: a <= b
Greater than: a > b
Greater than or equal to: a >= b

Examples of "if, elif and else" statements;

ITERATION

Iteration is used to describe repetitive execution in computing. This process may involve sending an email to countless number of people or updating bank balance to over millions of customers every night. It involves running a piece of code for every item in a sequence be it a string, list or tuple.

FOR LOOPS

A for loop is used for iterating over a sequence (that is either a list, a tuple, a dictionary, a set, or a string).

With the for loop we can execute a set of statements, once for each item in a list, tuple, string.

The overall syntax is for in :
• Between the words for and in, there must be a variable name for the loop variable. You can’t put a whole expression there.
• A colon is required at the end of the line
• After the word in and before the colon is an expression that must evaluate to a sequence (e.g, a string or a list or a tuple). It could be a literal, or a variable name, or a more complex expression.

Lets say i only want to go through a specific portion of the list, i use the slice method.

WHILE LOOP

The while loop allows us to cycle through a certain portion of code while a specific condition is true

RANGE

A range generates a list of numbers for us which we can then use to iterate over with a loop

Examples

All the range does is to generate a list of numbers. It would loop through a range of members from 0 to 5 except 5 itself.

If we want to specify our starting point we use range( , ) the first number before the comma is the starting point and the number after the comma is the end point which is always non-inclusive in python

If we want to specify the step size we include another comma and mention a number( , , ).

FUNCTIONS

A function allows us to create a block of code which we can call out and used when we want to. To create a function we use the "def" which stands for "define"

If we want to pass data into a function , we can specify the parameters we can use to pass through the function.

We can issue defaults and then subsequent inputs can override previous inputs

Also,

THE SCOPE OF A VARIABLE

A scope refers to an area or zone in which a variable can be accessed in. There are two areas involved here, the local area and the global area.

You realize that when the global variable is defined from within the function it affects it locally and any other instance of "my_name" outside of the function.

Here is an instance where the local variable overrides the global variable in the "my_name" variable which occurs inside the function only

If we want to override the global variable from within a local function we use the "global" statement and call out the name of the variable as seen below.

DICTIONARY

Dictionary is a collection of key and value pairs. A dictionary doesn’t allow duplicate keys but the values can be duplicate. It is an ordered, indexed and mutable collection of elements. In Python, dictionaries are defined within braces {} with each item being a pair in the form key:value.

If i intend to retrieve the value of one of the keys we can use the square bracket notation with the quotes in it [''].

If i try accessing a key or value that is not present in the dictionary i get an error.

In order to avoid that error we can check the existence of a key before we try to retrieve them. We use a statement known as the "key in dict". The "key" here is the name of the key you are trying to retrieve, 'in' is a key word in python and "dict" is the name of the dictionary you want to retrieve the key from. False indicates the absence of the key and True indicates its presence.

Another way to check of the availability of the key is to call out the full list of the keys in the dictionary using "name of dictionary".keys() afterwhich we can change it to an actual list using list("name of dictionary".keys())

The same process is done when you want to retrieve the values using "name of dictionary".values() afterwhich we can change it to an actual list using list("name of dictionary".values())

We can typecast this into a list and store it into a variable so we can work with it. Example;

Now that we have stored it into a variable if we want to count how many instances of a value we can use (values.count(name of value or value number).

If we want to add a new key:value pair to an existing dictionary we can use name of dictionary['new key'] = new value

Now applying all that;

SETS and SORTING
SORTING
We are going to learn how to sort a collection of elements in numerical or alphabetical order. We do that using a method called sorted(). In the first example you'd realize the elements have been sorted according to numerical order.

If you pay attention to the next example you'd realize that it is true that it has been sorted in alphabetical order, then again you'd also realize that all the elements starting with capital letters were sorted first before those starting with small letters

SETS

In sets, duplicates aren't allowed unfortunately they do not preserve order.

Applying this in python, lets say we want to remove duplicates.

CLASSES

A class describes how an object to look and behave. Let's go on to how to create a class;

We start by creating attributes for the class

We further create a method for this method;

If we want to pass in our own parameters we can allow the initialization function accept custom values by;

METHODS AND ATTRIBUTES

Every time we see 'self' introduced to a method then it is an instant method and also everything defined in that attribute is an instant attribute. We can apply class level attributes by coming out of the init function and defining our attribute. The class level attribute is not peculiar for a particular instance but rather the same for all. We can access a class attribute (pleasantries) by an instance class (Asare) or the class itselfs (Family) and below is an example of that;

class methods

we can create a class method by using the decorator @classmethod.This method is going to be common for all instances (families) and not peculiar to a particular instance. It takes the attribute (cls) which refers to the class. It has access to class level attributes and it is defined by

static method

This is a method that doesn't have access to "self" and it also doesn't have access to "cls". It only has access to the parameters that we pass into it individually.

MODULES AND PACKAGES

A module is a single Python file, a package is a directory of Python modules containing an additional init.py file, to distinguish a package from a directory that just happens to contain a bunch of Python scripts.

When you are working on a complex python project and you have loads of different functions in them with different custom classes, tons of lines of codes. it would become very hard to update because its very long. We can split that code into logical sections and those sections are going to encompass some kind of functionality for your program. Each of these different sections will have their own python file and those files could be called upon when we need them, we can import them and use them. These files are what we call modules.

HOW TO CREATE A PACKAGE

You create a new folder 2.Inside the folder you create a new file called init.py which tells python that this is a package. We don't need to place anything in this file. Just the existence alone tells python this is a package.
We can then create other new files and begin to code.
We import files within the package by (from "package name"."name of module" import "class name")
If we want to import multiple functions or classes we could do that by separating the names of the functions by a comma.
(from "package name"."name of module" import "class name","function")

LIST COMPREHENSIONS

A list comprehension gives us a simple way to construct a list based on other collections. Example

In the list comprehension we pass in what we want to do first followed by the loop and any conditional statement followed.

For the first example there is no conditional statement

In this example there is the presence of a conditional statement

MAPS

The map() function applies a given function to each item of a list or tuple and returns a list of the results.

new_list = map(function,data)

So first of all we have the map function and then it takes in two parameters that is the function we want to pass the data through to change the data and then the data itself.

First example we using the long approach using a for loop.

Using the same data we apply the simple map function approach

Applying my knowledge on list comprehension

FILTERS

The filter() method filters the given sequence with the help of a function that tests each element in the sequence to be true or not.

For the first example we have a sample end of semester results and we attempt filtering out the grades that are not equal to F

Using the loop method

Applying list comprehension

LAMBDA

What are lambda functions in Python? In Python, an anonymous function is a function that is defined without a name. While normal functions are defined using the def keyword in Python, anonymous functions are defined using the lambda keyword. Hence, anonymous functions are also called lambda functions

lambda argument : expression

DECORATORS

A decorator is a design pattern in Python that allows a user to add new functionality to an existing object without modifying its structure. Decorators are usually called before the definition of a function you want to decorate. It starts with an @ symbol attached to any word/name of your choice. It starts with an @ symbol attached to any word/name of your choice.

In this example a decorator named Hello was created (@Hello). Anytime we called on it it printed an output above and below a function.This decorator doesn't modify the function itself in the sense that in the absence of the decorator the function still works.

READING/WRITING FILES IN PYTHON

When it comes to reading files in python we use a special command called 'open'.

open("name of file","mode i want to open the file in")

"name of file"

absolute path of the file in whatever format
relative path of the file
exact file name if it is in the same directorate.

"mode i want the file in"

'r' = this stands for read, you can only read the information in the file without being able to modify it.

'w' = write, you have the authority to write new files or change existing files with this mode.

'a' = append, you can only add new information to existing information. you cannot change or modify existing information, you can only add new information to existing one.

'r+' = you can read and write.

here is an example.