DEV Community: MdMusfikurRahmanSifar

Predictive Models

MdMusfikurRahmanSifar — Fri, 21 Jul 2023 16:38:24 +0000

Predictive models are like tools or functions.

You give inputs and it gives outputs. But the outputs are based on that function, that is the model. The way you build your function, its going to give outputs accordingly. And to build that function you need to make relation between your parameters. This depends on the datas that has been inputted previously, that is the historical data.
Let's understand this through example.

1️⃣ Suppose you inputted the total class attended of 100
students and the their results of final exam in a software.

2️⃣ Then the software found that the relation between the class
attended and the final score is, final score = 0.25*class
attended + 5.[y=0.25*x+5]

3️⃣ Now, if you ask the software randomly, if a student attended
25 classes than what would be his probable final test score?

4️⃣ The software would give the answer following the model that
is the formula which is the relation between the parameters,
final score and class attended. Just put the value of x.

The first point describes the first thing to create a model that is inputting the Historical data.
Then considering necessary features that is Feature Selection(here are only two features, class attended and exam score. You might need to consider multiple features or select the relevant features out of many according to the context).
After that, finding the relation between the features through the most suitable approach that is Model Selection(our model y=0.25*x+5 is a straight line, but to get that we need a certain approach or algorithm which depends on the given and expected data).
Then comes Model Training.
After training the model with the historical data regarding the features we find our predictive model.( Here it is the straight line formula)
But we need to check how accurate the model is with the prediction.(We found the model but we need to check that if that actually is a good one. Because straight line have slopes and constant which is the m and c in the formula y=mx+c, we need to find the best suitable m and c for our data) So we run accuracy test that is Model Evaluation.
Checking to find the best model.(Finding the right m and c for the formula. Here the best model we found is y=0.25*x+5)
When we are satisfied with the accuracy of our model, we Predict

These are the main steps for creating a predictive model. But there are many sub points and lots of forms or options in each step. The choices depend on the nature of the data, the problem being addressed, and the desired accuracy and interpretability of the predictions. For example, the numerical data and the categorical data have different approaches in creating model and predicting. The depth is unceasing...the more you gain knowledge the better you get in dealing with wide range of datas..

The goal of a predictive model is-

to analyze patterns, relationships, and trends in the available data
to make predictions about future events or outcomes.

To sum up, Predictive models learn from historical data by identifying relevant features and building a mathematical framework or algorithm that captures the underlying patterns and relationships in the data. Once trained, the model can then be used to make predictions on new, unseen data. Predictive models are widely used in various applications, such as weather forecasting, stock market prediction, customer churn prediction, fraud detection, healthcare diagnostics, and many others.

Choropleth Map using Plotly: Geographical Visualization

MdMusfikurRahmanSifar — Tue, 31 Jan 2023 11:33:32 +0000

We will discuss the followings-
1. Structure
2. Nationwise example
-Without chart
-With chart
3. Global example

Stucture:

First we need to-

import pandas as pd
import chart_studio.plotly as py
import plotly.graph_objs as go
from plotly import __version__
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
init_notebook_mode(connected=True)

It might seem really complex...so let's break it down into steps-

Step-1:

We need to begin to build our data dictionary. Easiest way to do this is to use the dict() function of the general form:

Custom colorscale
locationmode can be: USA|ISO-3|country names
location and locationmode are connected
For details check this

Step-2:

Then we create the layout nested dictionary:

The 'scope' property is an enumeration that may be specified as:

One of the following enumeration value:['africa', 'asia', 'europe', 'north america', 'south america', 'usa', 'world']

Step-3:

Then we use:

to set up the object that finally gets passed into iplot()

Step-4:

To know all the dimensions, arguments1 and editing options check this

Nationwise example:

Without chart:

Let's plot USA states-

With chart:

For this a statistical chart of nationwise states has to be present in your repository.

We will use the columns for 'location', 'text', 'z' etc.

Here we used some customization.

Global Example:

Here we will have to include a statistical chart that includes all the countries in the world

We can plot different types of maps by changing the projection

So we have lots of customization option...Don't forget to reference the documentation....
(!)Wanna see the choropleth map of Bangladesh...see you soon...let's dive😉

Important Links:

Seaborn: Python Visualization

MdMusfikurRahmanSifar — Thu, 26 Jan 2023 18:37:14 +0000

Seaborn is a python visualization library built on matplotlib. Its a great tool for statistical visualization. Let's dive....

First we need to pip install seaborn then-

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

For visualization we need data. Here we can use any stock data, csv file etc. Even input manually. More data is better for understanding the plots. Here we can use already made dataset saved in seaborn. Run sns.get_dataset_names() to get all the saved dataset names.

To get the table we gonna use .load_dataset()

Now in seaborn there are mainly three types of plots-

Distribution Plots
Categorical Plots
Matrix Plots

To be really informal-

Distribution plot: There's a data against a parameter.
Categorical plot: There's multiple sub-parameter or categories for a parameter and there's data for each one.
Matrix plot: There's a data against two parameter

For example-

We need to keep this idea in mind while choosing our plot according to the data.
Secondly, we should use shift+tab
Now let's start-

Distribution Plot:

Let's discuss some plots that allow us to visualize the distribution of a data set. These plots are:

distplot
jointplot
pairplot
rugplot
kdeplot

Distplot:

Again there's lots of option to customize. Just see the docstrings with shift+tab and play with it.

Jointplot:

Here we join two different distribution plot so we need to define x and y axis. Also we can select the plot kind.

Here's some kind options-scatter,reg,resid,kde,hex

Pairplot:

Pairplot will plot pairwise relationships across an entire dataframe (for the numerical columns) and supports a color hue argument (for categorical columns).

Again docstring...add hue(color separation according to the categorical column) and palette(a color combo)

Rugplot:

Rugplots are actually a very simple concept, they just draw a dash mark for every point on a univariate distribution. They are the building block of a KDE plot.

Kdeplot:

kdeplots are Kernel Density Estimation plots. These KDE plots replace every single observation with a Gaussian (Normal) distribution centered around that value.

Categorical Plots:

Now let's discuss using seaborn to plot categorical data! There are a few main plot types for this:

factorplot
boxplot
barplot
countplot
violinplot

Factorplot:

Factorplot is the most general form of a categorical plot. It can take in a kind parameter to adjust the plot type:

Barplot and Countplot:

These very similar plots allow you to get aggregate data off a categorical feature in your data. barplot is a general plot that allows you to aggregate the categorical data based off some function, by default the mean:

Countplot is essentially the same as barplot except the estimator is explicitly counting the number of occurrences. Which is why we only pass the x value:

Boxplot and violinplot:

Boxplot:

Boxplots and violinplots are used to shown the distribution of categorical data. A box plot (or box-and-whisker plot) shows the distribution of quantitative data in a way that facilitates comparisons between variables or across levels of a categorical variable. The box shows the quartiles of the dataset while the whiskers extend to show the rest of the distribution, except for points that are determined to be “outliers” using a method that is a function of the inter-quartile range.

Violin plot:

A violin plot plays a similar role as a box and whisker plot. It shows the distribution of quantitative data across several levels of one (or more) categorical variables such that those distributions can be compared. Unlike a box plot, in which all of the plot components correspond to actual datapoints, the violin plot features a kernel density estimation of the underlying distribution.

Matrix Plot:

Matrix plots allow you to plot data as color-encoded matrices.
Let's begin by exploring seaborn's heatmap-
In order for a heatmap to work properly, your data should already be in a matrix form, the sns.heatmap function basically just colors it in for you. Now for getting a matrix presentation we can use .pivot_table().To get a good plot let's input 'flights' dataset.

There are more customizing options and plotting dimensions. But this was the basic idea.

Notes:

Know what data goes with your plot and select those datas for the desired plot
Know the types of the plots
Syntax is the main thing
shift+tab is your best friend....get to know him and play with him.

Plotly and Cufflinks

MdMusfikurRahmanSifar — Mon, 16 Jan 2023 06:53:36 +0000

Plotly is a interactive python library and cufflinks connects plotly with pandas.

First we need to-
pip install plotly
pip install cufflinks
pip install chart_studio

Then we need to do these-

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import chart_studio.plotly as py
import cufflinks as cf
%matplotlib inline
from plotly import __version__
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
init_notebook_mode(connected=True)
cf.go_offline()

Ahh...now if you don't want to know what are they for just go forward or-
Explanation:

We will require plotly version greater than 1.9.0, if you don't have it update plotly
We need to give comman to use open source library of plotly. from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot N.B. Plotly is a company that offers online and offline facilities. Some are paid. For our usage we are going to use the offline free services. -init_notebook_mode(connected=True) connects javascript with notebook. Plotly is gonna basically use pandas and python to javascript library
cf.go_offline() will allow cufflink use offline

In Matplotlib we use .plot(), in plotly we use iplotly()
So, it is i that is the difference in a sense.
Obviously the arguments are not same.

Basically plotly and cufflinks makes different kinds of plots based on data like matplotly but plotly is interactive and has more dimensions to it. Let's see-

But first let's see for the basics which plots we are gonna discuss-

scatter
bar
box
spread
ratio
heatmap
surface
histogram
bubble

Now-

Scatter:

We have some tools in the right top corner of the plot when we hover the mouse on it. The legends are also interactive.

That's one of the advantages we get. Plus the interactive part-

Bar:

Box:

Spread:

Ratio:

Heatmap:

Surface:

We can rotate the plot

Histogram:

Bubble:

in .iplot() there's tons of arguments use shift+tab to see them. Now play with the interactive powers. Play with it to know it....

Matplotlib: Plotting in Matplotlib

MdMusfikurRahmanSifar — Fri, 13 Jan 2023 05:04:14 +0000

Matplotlib is the "grandfather" library of data visualization with Python. It is an excellent 2D and 3D graphics library for generating scientific figures.

Basics of matplotlib basically means-

How to plot
How to edit the plot

Our main focus in this will be the plotting part. Some editing will be along the way.

We will discuss-
- Plotting in two ways
- Subplot in two ways

First we need to-

import matplotlib.pyplot as plt
%matplotlib inline

%matplotlib inline is used in jupyter. It allows us to plot the graph without extra command. Otherwise on other editors to see the plot we will need to use plt.show()

Plotting in two ways:

Functional method:

Object Oriented Method:

This method actually gives us control over the figure's axes...It is really helpful. Play with it to know it!

Subplots in two ways:

Functional method:

OO method:

We can actually see the power of this method in this example.

Here .set_xlabel() .set_ylabel() .set_title() are used for editing. There's a lot that we can edit in matplotlib....go ahead and find out them. I am mentioning some-

You can use the label="label text" keyword argument when plots or other objects are added to the figure, and then using the legend method without arguments to add the legend to the figure:
To change the line width, we can use the linewidth or lw keyword argument. The line style can be selected using the linestyle or ls keyword arguments:

There is more to matplotlib. Let's keep exploring....

Pandas-Basics In Short

MdMusfikurRahmanSifar — Mon, 09 Jan 2023 13:03:53 +0000

Pandas is a python library that is used to analyse data. It is a table themed library like spreadsheet in excel unlike numpy which had a matrixlike theme. It allows us to analyse, manipulate and explore huge amount of data.

For the basics, we will discuss a few topics-

Series
DataFrame
Missing data
Groupby
Merging,joining & concatinating

To start we need to-

import numpy as np
import pandas as pd

Series:

syn=pd.Series(data,index)

Here data and index can be edited and fixed according to our need. It can be list, numpy array or even dictionary.

Here's some examples-(look up the variations)

If index not mentioned then by default it is added from 0

In dictionary the keys are the index and values are data

Series is just an idea but we won't see it most often. Its like a string in a list. It won't show a table rather a tablelike presentation. Now what we will use is dataframe which gives us our expected output.

DataFrame:

syn=pd.DataFrame(data,index,columns)
It is the fundamental topic. So we need to know about some of the usage and applications-

Selection and indexing
Conditional selection
Creating new column
Removing column-row

Selection and indexing:

Selecting a row-column:

Syntax:
Column selection: arr[column]- returns a series
Row selection: arr.loc[row]- returns a series
Row selection: arr.iloc[row number(starts from 0)]

Selecting range:

By rows-

By columns-

Selecting data:

By combining previous methods we can get a data from the dataframe

Conditional Selection:

Here we apply condition. If we just apply condition, it gives us boolean result. If we call the dataframe then it gives us values true to the condition and NaN in the false ones.

We can even combine conditions by 'and'/'or'. But here in pandas to combine conditions we use '&'/'|' instead of 'and'/'or'.

Creating columns:

syn: arr[column name]=data of the column

Removing row-column:

axis=0 -> Row
axis=1 -> Column
inplace=True is used to make the change permanent

by default axis=0

Missing value:

Adding missing value:

use np.nan in data

Removing NaN:

By default .dropna() removes row with NaN
For column use .dropna(axis=1)

We can also spare row or columns with certain number of true values

Filling missing values:

Groupby:

We can group common data in a column and work with them

After .groupby() all common data gets stored...it doesn't print other then when we work with them. Like-

try .min(), .max(), .describe(), .mean() etc.

Concatenating, Merging, Joining:

Concatenating:

to attach column-wise or row-wise: pd.concat([],axis= ) by default axis=0

Merging:

to attach regarding common column

Joining:

to attach regarding common index

Summary:

This was the basics of pandas. It is really the fundamental stuff. There are features related to file handling, data analysis, plotting etc.
Let's keep exploring...let's dive together😉

Numpy Basics-My experience

MdMusfikurRahmanSifar — Sun, 25 Dec 2022 16:21:45 +0000

Numpy is a python library.
One of the most famous ones. Numpy stands for numerical python.
It is used for working with arrays.
Array is like python list but it is more than just list. It is faster than list and also consumes less storage. It's like in math what we call 'matrix'. All that we do with matrices in math we can do with array in numpy in programming rather, using the same theme, numpy has huge amount of other features. It allows us to use data and visualise in a matrix themed way and it is really powerful.

I mentioned previously in

#ml_with_mitul #dive_with_sifar | Md Musfikur Rahman Sifar

Week-1 Experience from Basics of Python From my journey of learning python so far, I would describe my experience and understandings in three parts-Notes from…

linkedin.com

what are the basic terms and subtopics of basic python. We always need to keep in mind about those after all we are using python. Here we also have those common topics like slicing, operating etc. So we can easily relate them. So, following that here is the list of the basic topics-

Creating arrays
Array attributes and methods
Slicing
Operating

Creating Arrays

we can create arrays two ways-

Conversion from other python structures
Intrinsic numpy creation objects

Conversion from other python structures(From lists, tuple, etc)

Firstly, we need to import numpy. We can use numpy as np.
import numpy as np
We use array()to create array. The syntax is

import numpy as np
arr=np.array()

array() takes arguements where we can give input.

We can create multi-dimensional arrays by putting'[]'
and check the dimension by putting .ndim

Trick: Dimension=Number of '[' from one side

We can take multiple arguments

There's also different datatypes..I don't know much about them yet😅

Intrinsic numpy creation objects(arange,zeros,ones etc.)

.arange(start_inclusive,end_exclusive,step)-creates array within the range of number serially
.zeros((row number,column number))-returns array only with 0
.ones((row number,column number))-returns array only with 1
.identity((row-column number))-return identity matrix

We can also create random numbers:
.random.rand((row_number,column_number))-creates random number in [0,1) range
.random.randn((row_number,column_number))-Return a sample (or samples) from the "standard normal" distribution. Unlike rand which is uniform
.random.randint((start_inclusive,end_inclusive,total numbers))-random numbers within a range

Array attributes and methods

.max()-returns the maximum entry
.min()-returns the minimum entry
.argmax()-returns the index of max value
.argmin()-returns the index of min value

We can also see the shape of the array and also reshape it
Note:For reshape it must be total entries=new_row*new_column
.shape-to know the shape of the matrix
.reshape-to reshape

Slicing

Syntax:
arrayname[rowstart_inclusive:rowend_exclusive,columnstart_inclusive:columnend_exclusive]
Note:careful about inclusive and exclusive

Operating

It is like regular operation in math

There are also other methods and attributes like-

More we explore more we know about these things and play with them

Experience:

Numpy gets really easy to understand when you relate it with matrix. Know your methods and attributes. But need to be careful about syntax. It was the biggest challenge for me.

There are many other features. It will get even amazing..Still at the beginning...let's continue learning...let's dive together.