DEV Community: Ruthvik Raja M.V

How to create a DataFrame that consists of a list of DataFrames and corresponding Name using Python

Ruthvik Raja M.V — Mon, 04 Sep 2023 01:25:12 +0000

Hello polymaths,
The above-mentioned task is important to know for most of the Python Developers working in the Data Field. Imagine, you have several Excel files (or) CSV files (or) a single Excel file with multiple sheets etc. and you want to compute a logic that considers the entire data for calculation -> Obviously, you have to append each file and its corresponding Name to a separate DataFrame that consists of a list of DataFrames to obtain the Output.

Step 1
Create an empty list to append the names of each DataFrame.

Step 2
Create an empty list to append the data related to each DataFrame.

Step 3
Make use of loop concepts to iterate through each DataFrame.

Step 4
Perform Data Cleaning, Transformations etc. if necessary and finally append the data to the previously created empty lists.

Step 5
Create a new DataFrame and assign the data parameter with the above two lists.

Sample Code:-


# Import necessary Libraries
import pandas as pd;

ef=pd.ExcelFile("path/input.xlsx") # Load the Excel File

dataframes=[]; # Empty List to append the data of each File
names=[]; # Empty List to append the name of each File

# Iterate through all the sheets within the Excel object
for i in ef.sheet_names:
    df=ef.parse(i); # Store the data as a DataFrame from each sheet
    df_name=i; # Store the name of the DataFrame from each sheet

    # Perform Data Cleaning and Tranformations, if necessary #

    dataframes.append(df);
    names.append(df_name);

df_final=pd.DataFrame(data={"Name":names, "DataFrame":dataframes}); # Create the Final DataFrame

Done

How to read all the sheets from an Excel file and push into Data Lake using Azure Data Factory

Ruthvik Raja M.V — Sun, 03 Sep 2023 00:14:15 +0000

Azure Data Factory doesn't have an in-built function to read all the sheets from an Excel file but it supports reading data from an Excel file using various methods, including Azure Data Factory Mapping Data Flows. Below are the general steps to read all the sheets from an Excel file in Azure Data Factory:-

Create an Azure Data Factory Service in Microsoft Azure Portal
If you haven't already, create an Azure Data Factory instance.

Create Linked Services for Source and Destination Files
In Azure Data Factory, create two Linked Services for your source (Excel File) and destination (Azure Data Lake) files. This is the connection information needed to access and store the data.

Create Datasets for Source and Destination Files
In Azure Data Factory, create two Datasets for your source (Excel File) and destination (Azure Data Lake) files. Set the Linked Services to the one you created in the previous step.

Let us name these two Datasets as follows:-
Source -> source_ds
Destination -> destination_ds

In source_ds create a new parameter under Parameters. Name the parameter as source_sheet_names. Under Connection send the newly created parameter as an input to the Sheet name. This action can be done by adding in the dynamic content under the Sheet name. Use the following expression to add -> @dataset().source_sheet_names.

Source Dataset -> source_ds
Parameters -> New -> source_sheet_names (Type: String)
Connection -> Sheet name -> Dynamic Content -> @dataset().source_sheet_names

Thereby, the sheet names will be sent as a Parameter from the Pipeline.

In destination_ds create a new parameter under Parameters. Name the parameter as destination_file_name. Under Connection send the newly created parameter as an input to the File path. This functionality will create a new folder for each sheet name and name it with the respective sheet name. This action can be done by adding in the dynamic content under the File path. Use the following expression to add -> destination/final_file/@{dataset().destination_file_name}/@dataset().destination_file_name.

Destination Dataset -> destination_ds
Parameters -> New -> destination_file_name (Type: String)
Connection -> File path -> Dynamic Content -> destination/final_file/@{dataset().destination_file_name}/@dataset().destination_file_name

Thereby, the file names will be sent as a Parameter from the Pipeline.

Create a Pipeline
Create a new Pipeline in Azure Data Factory.

Let us name the above created Pipeline as follows:-
Pipeline -> adl_pipeline

Create a new Variable under Variables and name it as sheet_names. Select the type as Array and under Default value send the list of sheet names as a default value -> ["sheet 1", "sheet 2", "sheet 3", "sheet 4"]...

Pipeline -> adl_pipeline
Variables -> New -> sheet_names -> ["sheet 1", "sheet 2", "sheet 3", "sheet 4"]...

Thereby, the sheet names will be sent during the runtime.

Add an Activity
In your Pipeline, add a ForEach activity and send the sheet_names as input to the ForEach activity. This can be achieved by navigating to Settings -> Items -> Dynamic Content -> @variables('sheet_names').

ForEach Activity -> Settings -> Items -> Dynamic Content -> @variables('sheet_names')

Add an Activity within the ForEach Activity to copy the data from Source to Destination

In your ForEach Activity, add a Copy data Activity and map the Dataset properties with the sheet_names. This can be achieved by navigating to Source -> Name -> source_sheet_names -> @item() and Sink -> Name -> destination_file_name -> @item().

ForEach Activity -> Copy data Activity -> 
Source -> Name -> source_sheet_names -> @item() 
Sink -> Name -> destination_file_name -> @item()

Thereby, the ForEach Activity will iterate through each of the sheet and copies the data to the destination folder.

Save and Execute the Pipeline

The entire workflow in a Nutshell:-

Source Dataset -> source_ds
Parameters -> New -> source_sheet_names (Type: String)
Connection -> Sheet name -> Dynamic Content -> @dataset().source_sheet_names

Destination Dataset -> destination_ds
Parameters -> New -> destination_file_name (Type: String)
Connection -> File path -> Dynamic Content -> destination/final_file/@{dataset().destination_file_name}/@dataset().destination_file_name

Pipeline -> adl_pipeline
Variables -> New -> sheet_names -> ["sheet 1", "sheet 2", "sheet 3", "sheet 4"]...

ForEach Activity -> Settings -> Items -> Dynamic Content -> @variables('sheet_names')

ForEach Activity -> Copy data Activity -> 
Source -> Name -> source_sheet_names -> @item() 
Sink -> Name -> destination_file_name -> @item()

Done

How to read an Excel file from Azure Databricks using PySpark

Ruthvik Raja M.V — Sat, 02 Sep 2023 20:42:35 +0000

The Excel file cannot be read directly using Py-Spark in Databricks so necessary Libraries (com.crealytics.spark.excel) have to be installed in the Cluster to successfully run the Python code.

Step 1
Navigate to the Cluster that will be used to run the Python script under Compute in Databricks.

Step 2
Click on the tab Libraries -> Install new.

Step 3
Select Maven as a Library source and click on Search Packages.

Step 4
Type com.crealytics in the search bar and select Maven Central.

Step 5
Select the com.crealytics.spark.excel package version that matches with the version of Scala (Cluster -> Configuration -> Databricks Runtime Version) in your Cluster.

Step 6
Click Install

Use the following code to load the Excel file:-

// Specify the path to the Excel file
val excelFilePath = "/mnt/<your-mount-name>/path_to_your_excel_file.xlsx";  // Replace with your actual spark file path

// Read the Excel file into a Spark DataFrame
val df = spark.read
  .format("com.crealytics.spark.excel")
  .option("location", excelFilePath)
  .option("useHeader", "true")  // Use this option if your Excel file has headers
  .load();

Whereas using Pandas Library the following Python code could be used:-

import pandas 
excelFilePath = "/mnt/<your-mount-name>/path_to_your_excel_file.xlsx"; # Replace with your actual file path

ef=pandas.ExcelFile(excelFilePath); # Load the Excel file as an object

# Mention the Sheet_Name or use ef.sheet_names to iterate through each sheet data

df=ef.parse(Sheet_Name); # Load the required Excel sheet data

Done

How to impute Null values using Python

Ruthvik Raja M.V — Sun, 19 Jun 2022 23:05:17 +0000

Hello all, this blog will provide you with an insight into handling Null values using Python programming language.

Download the pre-processed and final Datasets, python code (.ipynb file) from the following links:-

https://www.kaggle.com/datasets/ruthvikrajamv/home-insurance-dataset
https://github.com/ruthvikraja/Python-Code-on-Home-Insurance-Dataset.git

This project uses the home insurance dataset from Stable Home Insurance company, and data were collected from 2007 to 2012 for residential house sales. This dataset contains 66 data measurements containing various information from each of 256,136 insurance clients. To properly extract information from the dataset, some transformations need to be made to clean the data for analysis. An important step in this process is checking the data for null values. These values occur when a particular measurement does not apply to a client and can cause issues when building computer models to analyse the data. For example, a client who does not provide their personal information would contain null values for these information columns in the dataset. To ensure the data models run smoothly, I need to either remove data points with null values or replace these Null values with a valid proxy.

It is not sufficient to simply drop all the rows that consist of Null values, so I implemented different techniques to remove the Null values without losing information. In the stable home insurance data set, there are a few features which contain little information, as they are missing more than 98% of the data. These columns – CAMPAIGN DESC, P1 PT EMP STATUS and CLERICAL were all removed from the data set. After these columns were removed, I performed some further cleaning by removing all the rows which contain information about less than half of the features. These data points represent clients that the company has very little information about and thus will not be useful for analysis.

Additionally, further data processing was performed on the columns like QUOTE DATE, RISK RATED AREA B, RISK RATED AREA C, PAYMENT FREQUENCY, MTA FAP, MTA FAP & MTA DATE as these columns contain many Null values of different proportions. Features like MTA FAP, MTA APRP and MTA DATE were removed for having above 70% Null values. Features containing this high proportion of null values can be removed without losing important information. The PAYMENT FREQUENCY feature consisted of 57% Null values and contained only one unique value for the entire column so I dropped the column since it doesn’t contribute to the response variables of the analysis. QUOTE DATE was also dropped for having 58% Null values. The remaining columns with Null values were RISK RATED AREA C and RISK RATED AREA B. For these columns, there was enough information present to impute the Null values. Imputation is the process whereby Null values are replaced with a value based on the information present in the dataset. Mean Imputation is the process of replacing Null values with the mean of the remaining data points. This technique is appropriate in situations where there are few missing data points and thus was used for RISK RATED AREA C.

The above Figure illustrates the Box plot for the feature RISK RATED AREA C and it is clear that this feature contains many outliers. Initially, I treated it using different methodologies and finally imputed the Null values with the mean value. For the feature RISK RATED AREA B, more of the data was missing, so I implemented the K Nearest Neighbors (KNN) Imputer, which is a more appropriate technique for this amount of missing data. This process involves using the value of the most similar point as determined by the nearest neighbour (k). The input values were initially scaled using MinMax Scaler and then the data was sent to the algorithm to make unbiased predictions. The final dataset consists of 189021 rows & 57 features.

Mathematical Formulae behind Optimization Algorithms for Neural Networks

Ruthvik Raja M.V — Thu, 24 Mar 2022 01:04:01 +0000

Hello,
The following topics are covered in this blog:-

Introduction
Optimization Algorithms:

Gradient Descent (GD)

Stochastic Gradient Descent (SGD)

Mini-batch SGD

SGD with Momentum

AdaGrad

AdaDelta & RMSProp

Adam

Conclusion

Download the whole blog from the following link:-
https://github.com/ruthvikraja/Optimization-Algorithms.git

Introduction

Neural networks is a subset of Machine Learning in which Neural networks adapt and learn from vast amounts of data.
The Neuron is the building block of a Neural network that takes some input, does some mathematical computation by multiplying the input values with their corresponding random weights and finally produces the output.

Each node (Hidden and Output layers) in a Neural network is composed of two functions, namely linear and activation function. In the forward propagation, the Linear function is computed by summation of multiplying previously connected nodes output and corresponding weight, bias as shown in the Figure.
After applying the Linear function, Activation functions like Sigmoid, Relu, Leaky Relu, Parametric Relu, Swish Relu, Softplus AF’s etc are implemented based on the problem type and requirement.

Role of an Optimizer

After computing the output at the output layer, the predicted value is compared with the actual value by computing Loss.
The Loss function is used to determine the error between the actual and predicted value. The Optimization algorithm is used to determine the new weight values i.e Loss w.r.t change in weights to bring the output of the next trial closer to the actual output.

Gradient Descent

The formula to compute new weights using Gradient Descent is as follows:-

The formula to compute Loss using Gradient Descent is as follows:-

Stochastic Gradient Descent

The formula to compute new weights using Stochastic Gradient Descent is as follows:-

The formula to compute Loss using Stochastic Gradient Descent is as follows:-

Mini-Batch Stochastic Gradient Descent

The formula to compute new weights using Mini-Batch Stochastic Gradient Descent is as follows:-

The formula to compute Loss using Mini-Batch Stochastic Gradient Descent is as follows:-

Overall Comparison (GD (vs) SGD (vs) Mini-Batch SGD)

Stochastic Gradient Descent with Momentum

The formula to compute new weights using Stochastic Gradient Descent with Momentum is as follows:-

The formula to compute Loss using Stochastic Gradient Descent with Momentum is as follows:-

For better illustration, consider the following scenario to calculate Exponential Weighted Average:-

Therefore, the final updated formulae to calculate new weights & bias are as follows:-

where,

Adaptive Gradient Descent

The formula to compute new weights using Adaptive Gradient Descent is as follows:-

The formula to compute Loss using Adaptive Gradient Descent is as follows:-

where,

Adaptive Learning Rate Method & Root Mean Squared Propagation

The formula to compute new weights using AdaDelta & RMSProp is as follows:-

The formula to compute Loss using AdaDelta & RMSProp is as follows:-

where,

Adaptive Moment Estimation

The formulae to compute new weights & bias using Adam are as follows:-

The formulae to compute Loss for Regression & Classification problems using Adam are as follows:-

where,

When utilising Exponential Weighted Averages, there is a process known as bias correction. Scientists have introduced Bias correction to get better results at the initial time stamps. Therefore, the formulae for Bias correction is as follows:-

The updated Weight & Bias formulae are as follows:-

Conclusion

In this Presentation, different Optimization algorithms that are available in the field of Artificial Intelligence were discussed in detail to reduce the Loss function of a Neural Network.
Overall, Adam Optimizer is comparatively better than other algorithms because it was implemented using some advanced theories.
However, there is no guarantee that the Adam optimizer will outperform all the given datasets because it depends on several other features like the type of problem, size of the input data, number of features, etc.
Gradient Descent and SGD algorithm work well for small datasets, Mini-batch SGD, SGD with Momentum & RMSProp can be tried on large datasets.

THANK YOU

Difference between Iteration and Epoch in Neural Networks

Ruthvik Raja M.V — Tue, 22 Mar 2022 23:46:43 +0000

Epoch describes the number of times the algorithm sees the entire dataset whereas an Iteration tells the number of times a batch of data passed through the algorithm.

For example, consider the input dataset consists of 100,000 records and if we train the model for 10 epochs, let the batch size = 1 i.e loading each data point every time and performing forward, backward propagation for every data point [Optimiser would be the Stochastic Gradient Descent]. Hence, the number of iterations per epoch for the above example would be 100,000 and the total number of epochs would be 10.

How to install Apache PySpark on Mac using Python?

Ruthvik Raja M.V — Tue, 08 Feb 2022 16:34:55 +0000

Hello,

Apache PySpark works with Java 8 version and not with the latest Java version so, make sure that you install the correct version to run Apache PySpark on your Machine.

Download Java 8 from the following link and install the software:
https://www.java.com/en/download/manual.jsp

If you already have the latest Java version on your Machine and want to remove the latest Java software from your Machine, then please visit the following blog:

https://dev.to/ruthvikraja_mv/how-to-uninstall-java-on-mac-104a

Now, its time to check the installed Java version on your Mac, enter the following command in the terminal to check the version:

java -version

Launch Anaconda Navigator (or) any other IDE to run python code for installing Apache PySpark on your Machine. Type the following command and hit enter:

pip install pyspark

It will take some time to install the software and once the software got installed you can check the version by entering the following command:

pyspark

Once everything is done please type the following command in the terminal to check whether we can create a new session using PySpark:

spark-shell

If the software has been successfully installed on your machine without any dependency errors then it should show as follows:

Done...

How to Uninstall Java on Mac?

Ruthvik Raja M.V — Tue, 01 Feb 2022 04:31:28 +0000

Hello everyone,

There are two methods to uninstall Java on your Mac and they are as follows:-

Method 1:-

Click on the Finder App.[Bottom left of your Mac]
Navigate to Applications Folder and type JavaAppletPlugin.plugin in the search bar.
Finally, move the plug-in to the Bin and empty the Bin.

Then, check whether the Java software was successfully uninstalled by typing the following command in the Terminal:-

java -version

It should prompt you to install Java if the software was successfully uninstalled.

Method 2:-

Consider if you are unable to find the plugin by entering JavaAppletPlugin.plugin in the search bar [Application Folder], then follow the below steps to remove the concrete version of the Java file [Ex: JDK file]:-

Check what Java versions are available by entering the following command in the Terminal:

ls /Library/Java/JavaVirtualMachines/

Remove the corresponding folder with that version:

sudo rm -fr /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk

[Ex: In the above statement I am uninstalling Java 9]

Then, check whether the Java software was successfully uninstalled by typing the following command in the Terminal:-

java -version

It should prompt you to install Java if the software was successfully uninstalled.

Done...

Prediction of Customer Churn in the Telecom Industry using Neural Networks

Ruthvik Raja M.V — Sun, 26 Dec 2021 01:43:58 +0000

Introduction:-

Customer churn is one of the major problem and very important problems in the field of Telecommunications. It directly impacts the companies revenue, particularly in the field of the Telecom Industry. Thereby, companies are trying to develop methods for predicting client attrition. Thereby, it is much important to find the factors that impact the customer churn from the company. This article provides a brief description of customer churn in the Telecom Industry, provides a churn prediction model which assists telecom companies to predict customers who are likely to churn and also provides some useful data analysis to draw insights from the data. Neural Networks, Machine Learning Algorithms & other technologies can be implemented to develop a churn prediction model that can predict with high Accuracy Score. Performance metrics like Accuracy Score, Area under Curve (AUC), Sensitivity, Specificity etc can be implemented on the model to measure the goodness of the model on test data. The Dataset consists of various parameters and multiple datasets are extracted from different Telecom companies to predict the likelihood of a customer to churn from the company so that measures can be taken by the company from being churned from the telecom company. Also, Big Data technologies could be implemented if the datasets are too big for faster calculation and easy access.

Download the Datasets and Python file(.ipynb) from the following link:-

https://github.com/ruthvikraja/Customer-Churn-Prediction.git

Dataset:-

The datasets are collected from different Telecom companies and open source databases. Each dataset consists of a different set of features because they are taken from different service providers and each provider has its own set of parameters to make a better strategy so that the customer won’t churn from the service provider. The features (or) parameters from different datasets are as follows:-

Churn - Customers who left the service provider last month.
Account Length (or) Tenure - This feature describes the number of months the customer has stayed with the company.
PhoneService, MultipleLines, InternetService, OnlineSecurity, OnlineBackup, DeviceProtection, TechSupport, StreamingTV, ScreamingMovies etc - These are categorical features that describe different services opted by each customer.
PaperlessBilling, PaymentMethod, MonthlyCharges and TotalCharges - These are the features that describe customer billing and payment methods.
Gender, Age, Dependents etc - These are the features that describe the demographic info of a customer.
Number of Calls, Minutes, Messages etc - These features describes the number of minutes a customer has spoken in a day, number of calls a customer has made in a day, number of messages a customer has sent to another person in a day etc.
InternationalPlan - This feature gives information about whether a customer has opted for an international plan (or) not.

Also, the dataset consists of some additional features like the total number of minutes a customer has spoken in the Morning, Afternoon, Evening and during Nights.
The above mentioned features are some of the important features from different datasets that impacts the output parameter Churn.

Data Analysis:-

In this study, Data Analysis played a vital role than models, which were built to predict the churn of a customer from the telecom industry because data analysis provides useful insights for the companies to concentrate more on particular parameters in which they are lacking to lose and acquire more new customers.

To perform analysis on the Datasets, Uni-variate and Bi-variate analysis was performed on both categorical and numerical typed features. Uni-variate analysis was performed on each individual feature whereas the Bi-variate analysis was performed for every input feature with respect to the output feature Churn.

Uni-Variate Analysis:-

In the Figure 1, it is clear that almost 49.6% of the customers are staying at area code 415 so, it is better to provide more services and comparatively better plans for the customers who are residing at area code 415 to increase the retention period of a customer from the company.

Also, from Figure 2 it is clear that the distribution of the number of customers who will leave the company is very low than the number of customers who stays with the telecom company. This creates bias due to imbalanced classes so, it is crucial to apply Upsampling technique on the minority class i.e on the data points with churn value “yes”. Thereby, the Synthetic Minority Oversampling Technique could be applied to the minority class to overcome the problem.

Bi-Variate Analysis:-

In Figure 3, it is clear that the customers with a tenure period of more than 50 months are less likely to churn from the telecom company whereas customers with a tenure period of fewer than 10 months are more likely to churn from the company so, the telecom company has to provide better deals and extra data for new customers to increase the retention period of a customer.

From the above Figure, it is clear that the customers who are paying very little and high monthly charges are not leaving the company whereas the customers who are paying monthly charges around $70 - $100 are having a 50% chance of leaving the company. Also, the company should concentrate more on the customers who are paying around $70 as monthly charges.

From the Figures 5 & 6, it is clear that if "The total day calls" is between 85 to 115 times then the churn rate is high and if the "Number of voice mail messages" is equal to 0, then the churn rate is very high.

Methods:-

In this Article, two different approaches have been proposed, Neural Networks based and Machine Learning based to predict the Customer Churn from a company. Many authors have proposed different algorithms that are available in Artificial Intelligence to predict the churn of a customer and found that AI technology has outperformed most of the Datasets. Also, for large datasets, Big Data technology can be implemented to store, process and retrieve the data. So, in this Article, I have chosen to build Random forest classifier and Dense Neural Networks to predict the churn of a customer.

Different weight initialisation techniques, optimisers and hidden layers are implemented to build a better predictive model.

The Neural network chosen for this study consists of 3 to 5 hidden layers for different datasets and different weight initialisation techniques, activation functions are applied on the Neural network. He Normal weight initialisation technique was implemented on neurons having activation function “Relu”, Glorot Normal (or) Xavier Normal weight initialisation technique was implemented on neurons having activation function “Sigmoid” because He Normal works well with Relu whereas Glorot Normal works well with Sigmoid activation function.

Initially, I tried implementing a Neural network with random weights but the accuracy score was not pretty good so, I tried implementing different methods on the models to achieve a better accuracy score. Also, at the output layer, the Sigmoid activation function was implemented because it will transfer the input function between 0 and 1. In the hidden layers, the Sigmoid activation function was not implemented instead Relu activation function was implemented because the Sigmoid activation function creates a Vanishing Gradient problem [The derivative of Sigmoid activation function is between 0 and 0.25 only].

Results:-

Machine Learning classification algorithms like Random Forest classifier, KNN classifier didn’t perform as expected on the first dataset i.e Telco Customer Churn, even after performing Hyperparameter tuning and Principal component analysis to capture the variability of all the features. The reason behind this not performing as expected is due to the lack of correlation between the input features and the output feature. I have implemented the Pearson correlation coefficient on the Telco customer churn dataset to find the important features that have more impact on the output feature but not even a single feature showed a high correlation value with the output parameter “Churn”. So, I tried implementing dense Neural Network to capture the input data and the results are as follows:-

The AI algorithms have performed well on the second dataset i.e Customer Churn Prediction 2020 with more than 90% accuracy score using Machine learning algorithms. Also, I have tried implementing Artificial Neural networks on the dataset to predict the churn of a customer with a different number of epochs and weight initialisation techniques. The Accuracy scores are as follows:-

Thereby, from the above table, it is clear that implementing a Neural network using different weight initialisation techniques is better than initialising the weights randomly. Therefore, the final results i.e ROC curve and Accuracy scores are mentioned in Table 3 and Figure 7. Also, I have implemented dill library to save the current python session so, that I can continue my work from where I have left.

In this Article, Artificial Intelligence was implemented to predict the Churn of a customer from a telecom company and in this study, two different datasets are analysed to extract important features using Feature Engineering and Data Analysis techniques. Overall, Deep Learning performed well to predict the churn of a customer whereas Machine Learning performed better to some extent however so many factors come into the picture when it comes to predicting the goodness of different algorithms. In future, different AI algorithms can be implemented to further decrease the error value.

################# THE END

Back Propagation in Neural Networks

Ruthvik Raja M.V — Sun, 28 Nov 2021 05:41:05 +0000

Hello all,

It is very important to know how the Back Propagation works in Neural Networks in order to find the optimal weights. So, to know more about this concept let us quickly dive into following slides:-

Download the whole document as a pdf document from the following link:-
https://www.kaggle.com/ruthvikrajamv/back-propagation

How to save the entire user session using Python?

Ruthvik Raja M.V — Sun, 28 Nov 2021 00:14:39 +0000

It is very important to know how to save the entire current session like local variables, objects etc when we are working with AI projects using Python because it is very difficult to run the entire python code every time to initialise the objects, Models, variables etc.

To overcome this problem we have pickle to take care of it but some times it fails to deserialise the pickled objects so,
dill library can be implemented to quickly store and retrieve the current session.

Here is a quick introduction to dill:-

dill extends python’s pickle module for serializing and de-serializing python objects to the majority of the built-in python types. Serialization is the process of converting an object to a byte stream, and the inverse of which is converting a byte stream back to a python object hierarchy.

dill provides the user the same interface as the pickle module, and also includes some additional features. In addition to pickling python objects, dill provides the ability to save the state of an interpreter session in a single command. Hence, it would be feasable to save an interpreter session, close the interpreter, ship the pickled file to another computer, open a new interpreter, unpickle the session and thus continue from the ‘saved’ state of the original interpreter session.

Therefore, the following code can be implemented to save the current session:-

##### User's python code #####
# pip install dill (or) conda install dill

import dill;
# Save the entire session by creating a new pickle file 
dill.dump_session('./your_bk_dill.pkl');

# Restore the entire session
dill.load_session('./your_bk_dill.pkl');

From, above it is clear that implementing dill is very easy and also it provides function like dill.detect to investigate the failed attributes inside that object.

Stock Price Prediction using Supervised Learning

Ruthvik Raja M.V — Mon, 16 Aug 2021 06:18:53 +0000

Introduction:-

The impact of numerous factors on stock prices makes stock prediction a complex and time-consuming endeavour. Predicting the price of a stock is computationally hard because of its non-stationary nature and also it depends on many factors like News Headlines, Tweets, Historical Trends, Social Media News etc. In this paper, Machine Learning Algorithms and Neural Networks are implemented on various Companies like Apple, Amazon, Pfizer, Walmart Stores etc to overcome the difficulties and to achieve better accuracy in predicting the price of a stock. Artificial Intelligence algorithms like Random Forest, XG Boost (Extreme Gradient Boosting), LSTM (Long Short Term Memory), GRU (Gated Recurrent Units) etc are developed and their RMSE (Root Mean Square Error) are compared in predicting the price of a stock. The Dataset is an open- source Time Series dataset and consists of stock prices for 88 different companies that fall under 9 different sectors for around 5 years.

About the Dataset:-

The dataset is a Time Series data and consists of stock prices of 88 different companies like Apple, Amazon, Chevron Corporation, Sanofi, Duke Energy Corporation, Visa, Alphabet etc. These companies fall under 9 different categories namely Basic Materials, Consumer Goods, Healthcare, Services, Utilities, Conglomerates, Financial, Industrial Goods and Technology. In total there are 88 files in the dataset and each file consists of features like Date, Open price of a stock when it was opened on a particular day, High and Low price of a stock within a period, Volume of the stocks and the Adjusted closing price of an individual company. The output (or) the predicted variable is the Closing price of a stock for a particular day.

Workflow:-

To train the Machine Learning algorithm like Random Forest Regressor, initially, all the CSV files are loaded and converted into Data frames, then Scaling is done on all the Data frames such that each feature is translated to a given range. MinMax Scaler can be implemented for scaling to normalise all the features because each feature in the dataset is of a different scale and it is very important to scale each feature before it is sent to the model for training. Also, the dataset consists of a date feature but this is not understood by the algorithm so, the datasets have to be pre-processed by splitting the date column into three different columns (Year, Month and Day). At last, the dataset is split for training and testing data.

Results:-

Machine Learning Algorithm like Random Forest Regressor has outperformed for 79 companies among 88 companies and the RMSE values ranges from 0 to 1 for those 79 companies. The RMSE values for the test data on 79 best performing companies are as follows:-

Nearly, for 8 companies the Random Forest Regressor didn’t perform as expected and the RMSE values for those 8 companies range from 0 to 10 whereas for 1 company the algorithm has performed poorly and the RMSE value ranges from 750 to 830. The better and worst-performing companies are shown as follows:-

Overall the RMSE values for all the companies are shown as follows:-

From the above Figure it is clear that the Machine Learning has not performed well on one company that is BRK-A so, Deep Learning was implemented on Worst performing Companies to achieve better RMSE value and the results are as follows:-

The Python code, Datasets and all other files can be found from the following Link:-

https://github.com/ruthvikraja/Stock-Market-Price-Prediction-using-Supervised-Learning.git

Conclusion:-

In this article, Artificial intelligence is used to make predictions about stock market prices. A stock market is a place where buying and selling of shares happen for companies. The data that is used in this work is a time Series dataset and consists of stock prices of 88 different companies as described in Section About the Dataset. However, while the time component adds additional information, it also makes time series problems more difficult to handle compared to many other prediction tasks. In this study, we proposed two methods, Machine Learning-based, and Deep Learning-based. As the proposed methods show in this study different AI-based algorithms together with ensemble learning are used to make the predictions and make a comparison between the results of different methodologies. It has been shown that GRU performs better than Deep Learning based methods in terms of both accuracy and processing time. Also, Machine Learning-based methods perform pretty well for most of the companies but it fails when it comes to large stock price values so, Deep Learning methods were implemented on the stocks with high price values and the results were far better using Deep Learning Methods. In future, different AI algorithms can be implemented to further decrease the error value.