This article was originally published at:
https://www.blog.duomly.com/python-course-with-building-a-fintech-investment-ai-lesson-3-how-to-build-and-train-ai
Intro
In today's episode of the Python with AI course, we will learn how to build AI and how to train AI.
Last episodes you can find here:
Lesson 2 - How to find financial data and how to use Pandas:
Python course with building a fintech investment AI – Lesson 2: Pandas and getting financial data
Lesson 1 - Prepare the AI project:
Python course with building a fintech investment AI – Lesson 2: Pandas and getting financial data
In this lesson, we can leave boring preparation and focus on the super exciting things!
Today we will build the first AI model, which will be exactly RNN (Recurrent Neural Network).
We will use some Long short-term memory and train our artificial intelligence.
Are you ready for that awesome journey?
I hope you are because I cannot wait to show you all of those powerful things!
Let's start!
If you prefer video, here is the youtube version:
Create module AI
As the first step, we will create a module named "AI".
First, you need to create a directory named "AI" at the root of the project.
Next, we need to create a file with the same name, with the ".py" extension.
Import dependencies
The second step should be to import dependencies that we will use later.
Go into the ai.py file and import these dependencies.
from datetime import date, timedelta
import numpy as np
import pandas as pd
import yfinance as yf
from pandas_datareader import data as pdr
import matplotlib.pyplot as plt
Install and import Keras
Next, we should install a library named "Keras".
Of course, if you have not installed them, you should install sklearn, pandas, NumPy, and matplotlib.
If you use conda, you probably have them already installed.
The first open terminal, and type:
pip install keras
Next, below the previous imports, you need to import "layers", and "Sequential" from Keras.
As the third line, import MinMaxScaler from sklearn.preprocessing.
from keras import layers
from keras import Sequential
from sklearn.preprocessing import MinMaxScaler
Get stock prices from API
As the fourth step, we need to focus on the prices.
We need to define a function named "getStock" with "stock" as a param.
Inside the function, we need to define today's date.
def getStock(stock):
today = date.today()
Take data for the last 5 years
The next step is to take the pricing date for the previous five years.
We should use a similar method that we use in Lesson 2.
The only change here will be to change the number of days to 1850.
monthAgo = today - timedelta(days=1850)
data = pdr.get_data_yahoo(stock, start=monthAgo, end=today)
return data
Split data
Now, we can go into some work on data that we have got.
As the first step, we need to create the function named "splitData", and take "stockPrices", as a param.
Next, we need to split data into two parts.
The first one prices from the last month(20 days).
The second part is all of the data without the last 20 days.
Both of the parts should be returned.
def splitData(stockPrices):
stockPricesLastMonth = stockPrices[-20:]
stockPricesTest = stockPrices[:-20]
return stockPricesLastMonth, stockPricesTest
Prepare data
In the next step, we can go into preparing that data.
Let's create the function named "prepareData", and take "prices", as the param.
def prepareData(prices):
stockPricesLastMonth, stockPricesTest = splitData(prices)
Get only "Open" price value as the important one
Now, we should continue with preparing data inside the "prepareData" function.
We need to take only "Open" prices from the row and assign it to the variable named "scaledData".
cleanData = stockPricesTest.iloc[:, 1:2].values
Scale data to speed up the algorithm
Next, we need to scale the data, that will speed up our calculation.
We will use "fit_transform" function on the data that we cleaned before.
All of these variables should be returned in the last line of the function "prepareData".
scaledData = scaler.fit_transform(cleanData)
return scaledData, stockPricesTest, stockPricesLastMonth
Split train and result
In this step, we need to define the function named "splitTrainTest".
Next, we should add "scaledData" as a param.
def splitTrainTest(scaledData):
Take the data for the first 6 months and real prices
Inside the function "splitTrainTest", we will take data for the last 6 months and return two variables.
One of them will be "inputs", and the second one, "realPrice".
inputs = []
realPrice = []
for i in range(120, len(scaledData)):
inputs.append(scaledData[i-120:i, 0])
realPrice.append(scaledData[i, 0])
return inputs, realPrice
Create the table of arrays and reshape to have one list
Next, we should create a 3d array and reshape the variable named "inputs".
To do that, we need to create a function named "reshapeData", put "inputs", and "realPrice" as variables.
Inside the function, we should use "np.array", and "np.reshape" methods.
As the last step, we will return inputs.
def reshapeData(inputs, realPrice):
inputs, realPrice = np.array(inputs), np.array(realPrice)
inputs = np.reshape(inputs, (inputs.shape[0], inputs.shape[1], 1))
return inputs
Create a model
Great! Now we will go into the real thing. We will create an AI model.
As the first step, we need to create the function named "createModel", and pass "inputs", as a param.
def createModel(inputs):
Create 3 LSTM layers
In the second step of creating a model, we should add long short-term memory layers, setup units, and dropout.
You can try to experiment with that because different setup will give different results.
I've found 3 LSTM layers with 30 units, and 0.2 dropout gives good results.
model.add(layers.LSTM(units = 30, return_sequences = True, input_shape = (inputs.shape[1], 1)))
model.add(layers.Dropout(0.2))
model.add(layers.LSTM(units = 30, return_sequences = True))
model.add(layers.Dropout(0.2))
model.add(layers.LSTM(units = 30))
model.add(layers.Dropout(0.2))
Add model and compile
The last part of the "createModel" function should be to add the Dense layer and compile the model.
I've used "adam", as an optimizer, and "mse" as a loss.
model.add(layers.Dense(units = 1))
model.compile(optimizer='adam', loss='mse')
Train model
Awesome!
Now we can go into the training model.
We need to create the function named "trainModel", and pass there four params.
We will add "inputs", "realPrice", "epochs", and "batch" as params.
Next, we need to add the "model.fit", that is responsible for the training.
def trainModel(inputs, realPrice, epochs, batch):
model.fit(inputs, realPrice, epochs = epochs, batch_size = batch)
Define the price, model, and scaler
All the logic is ready, in this step, we should complete that.
The first is to define variable responsible for the getting price.
Next, we need to assign "Sequential" to the "model".
And we need to add "scaler".
prices = getStock('MSFT')
model = Sequential()
scaler = MinMaxScaler(feature_range=(0,1))
Train AI
In the almost last step, we need to put all of these functions to the one named "trainAI".
Add the "prices", "epochs", and "batch" as params.
def trainAI(prices, epochs, batch):
scaled, stockPricesTest, stockPricesLastMonth = prepareData(prices)
inputs, realPrice = splitTrainTest(scaled)
reshapedInputs = reshapeData(inputs, realPrice)
createModel(reshapedInputs)
trainModel(reshapedInputs, realPrice, epochs, batch)
Call train AI
Now is the last step, and the fun begins!
Call the method "trainAI", and pass the necessary data.
About the amount of epochs, and batch size, you can experiment, I've made 50, but some people recommend even 1000, and batch size small as 4-8.
Remember about the fact, more epochs, and smaller batch size required much more iterations, and training can take many long hours.
trainAI(prices, 50, 64)
Conclusion
Congratulations!
Your project has its own and trained artificial intelligence that you can use to see what can be the next pricing trends for the stocks.
Of course, you need to remember it's only AI, it can make a lot of mistakes, and the real stock prices can be different (it depends on a few points).
Code repository for the Python course Lesson 3 is here:
https://github.com/Duomly/python-ai-investment-fintech/tree/Python-AI-course-Lesson-3
In the next lesson, we will test our AI, and we will predict the stock prices.
We will compare these prices with the real ones to know how smart is our AI.
Keep learning with us, and I cannot wait to show you the next episode, where we will see the real results, how our AI works!
Thanks for reading,
Radek from Duomly
Top comments (2)
hi i'm having an issue that you don't seem to be running into. i get the below error when running ai.py. any ideas?
Failed to find data adapter that can handle input: class 'numpy.ndarray', (class 'list' containing values of types {"class 'numpy.float64'"})
Also getting the issue mentioned by gumbo-bot