DEV Community: shady shafik

How to Backtest a Trading Bot with BackTrader.

shady shafik — Mon, 21 Mar 2022 19:15:35 +0000

Hello there and welcome to another article, today I will get into one of the most important topics in algorithmic trading which is backtesting.

Backtesting is the process of testing your strategy on historical data to get a clear vision about how this strategy would performed and how it will perform in the live trading.

also, backtesting doesn’t guarantee that the strategy or the trading bot will perform as it performed on historical data for many reasons….

but still backtesting is a precious tool every algorithmic trader and traders in general need as it can transform the performance by manipulating and testing new strategies.

In this article we will develop a simple strategy and backtest it using backtrader library and I’ll use jupyter notebook.

let’s jump into it.

Import the backtrader library.

import backtrader as bt

Download Bitcoin data from yahoo finance
starting from 2018 we’ll get the daily time frame

import yfinance as yf
data = yf.download("BTC-USD",start='2018-01-01')
data.head()

Now it’s time to think about the Strategy.

I will develop a Buy the dip strategy which is :

If prices drop three days in a row we buy
and sell after 2 days.

create dipStrategy class and pass strategy(a built in object inside backtrader.)

class dipStrategy(bt.Strategy):
    def log(self, txt, dt=None):

        dt = dt or self.datas[0].datetime.date(0)
        print('%s, %s' % (dt.isoformat(), txt))

backtrader log function *** datas[0] is the current data row.

def __init__(self):
 self.dataclose = self.datas[0].close
 self.order = None

Keep a reference to the “close” line in the data[0] dataseries in dataclose
set order to None as there is no order yet.

def notify_order(self, order):
       if order.status in [order.Submitted, order.Accepted]:
            return
        if order.status in [order.Completed]:
            if order.isbuy():
                self.log('BUY EXECUTED,%.2f' % order.executed.price)

            elif order.issell():
                self.log('SELL EXECUTED,%.2f' %order.executed.price)

            self.bar_executed = len(self)

        elif order.status in [order.Canceled,
                                       order.Margin,order.Rejected]:
            self.log('Order Canceled/Margin/Rejected')
        self.order = None

check order status if in submitted or accepted then return.
*** not completed yet but accepted by the broker.

len(self) will return day number

ex: if we bought at day 200 we sould keep track of that number as we’ll sell after 2 days at day 202

we save the order day in bar_executed

Check if an order has been completed

if buy order completed log the price isbuy() built in function in backtrader
if order in status in canceled… log this
if sell order completed log the price
set order to None as there is no pending order

def next(self):

        self.log('Close, %.2f' % self.dataclose[0])
        if self.order:
            return


        if not self.position :

            if self.dataclose[0] < self.dataclose[-1]:

                if self.dataclose[-1] < self.dataclose[-2]:

                    if self.dataclose[-2] < self.dataclose[-3]:

                        self.log('BUY CREATE, %.2f' % 
                                                 self.dataclose[0])
                        self.buy()

                        self.order = self.buy()


        else:

            if len(self) >= self.bar_executed + 2 :
                self.log('SELL CREATE, %.2f' % self.dataclose[0])
                self.order = self.sell()

Simply log the closing price of the series from the reference
Check if there is pending order if yes, return
check if we have position in the market.

we’ll check if the price drops three days in a row
if today close less than yesterday till three previous days

*** [-1] is the preious day index and so on till [-3]

Now it’s time to buy as our conditions met log price
set buy order

set order to the buy order to keep track and avoid sending second order

self.buy() return buy order object

Sell conditon

after 2 days we sell our position
and we keep track by bar_executed variable
we’ve finished coding our strategy
Now it’s show time, let’s execute.

Backtesting

cerebro = bt.Cerebro()

cerebro.broker.setcash(100000.0)

cerebro.addsizer(bt.sizers.SizerFix, stake=2)
df = bt.feeds.PandasData(dataname=data)

cerebro.adddata(df)

cerebro.addstrategy(dipStrategy)

print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue())

cerebro.run()

print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue())

cerebro is backtrader engine we’ll create an instance

set how much cash in your account is set it to 100k

set size of your order ** how much you’ll buy i set it to 2 shares (bitcoins)

data feed in our case we have pandas data dataframe

pass data feed to cerebro with adddata() method
add our strategy to cerebro with addstrategy() method

print account cash in start

run the engine start trade

print cashe at the end

Results

we get along list with all trader executed.

We started with 100k and at the end we have 141k 🔥 👌 that’s not bad isn’t it…😁

note: of course this strategy underperformed the market but the aim is to come up with your strategy, optimize and manipulate till your reach a point where you have a strategy that can outperform the market with consistent monitoring, optimization, and idea generation.

What about visualizing our trades

cerebro.plot()

what’s next

I think backtrader is a great tool for traders to test and manipulate their strategies

In my previous article we developed a crypto trading bot but without the backteting part, and today we did.

That’s it for this article I hope you enjoy.

How to build a Trading bot with Python and SQlite.

shady shafik — Sat, 26 Feb 2022 18:14:58 +0000

Hello there and welcome to another article about algorithmic trading with python.

In the previous article I gave a light introduction to algorithmic trading, and we developed a simple trading bot using Binance API to get data and execute trades.

Today we will develop another trading bot, with the use of SQlite to save data and process this data to execute trades.

We Start By connecting with Binance and Socket Manager

A WebSocket is a communication protocol which is a bidirectional connection full duplex communication channel between a client's web browser and a server.

I am working with Jupyter notebook, python and sqlite.

import pandas as pd
!pip install python-binance
from binance import Client
from binance import BinanceSocketManager

Import pandas, install python binance, import client and socket manager.

APIKEY= 'Your API Key'
SECRETKEY = 'Your Secret Key'

client = Client(APIKEY,SECRETKEY)
client.API_URL = 'https://testnet.binance.vision/api'

get an API KEY and SECRET KEY, provide the two to the client function to establish a connection.

*client.API_URL = *, here we provide the Binance Testnet url to work with test net environment.

to generate an API key and secret key go to Binance Spot Test Network sign with GitHub and generate HMAC SHA256 key.
finally, replace APIKEY and SECRETKEY variables in the code with actual keys.

from sqlalchemy import create_engine
engine = create_engine('sqlite:///SqlDb.db')

import create_engine from sqlalchemy ( Python SQL toolkit and Object Relational Mapper ), then define new database sqlDB.db.

Getting Data with Binance socket manager

I'll start by testing binance socket manager connection by print the received data.

 bsm = BinanceSocketManager(client)
 ts = bsm.trade_socket('BTCUSDT')
 await ts.__aenter__()
 msg = await ts.recv()
 print(msg)

{'e': 'trade', 'E': 1645890024494, 's': 'BTCUSDT', 't': 1271100538, 'p': '39317.92000000', 'q': '0.00051000', 'b': 9560827435, 'a': 9560827538, 'T': 1645890024493, 'm': True, 'M': True}

Now we can see what we'll receive, it's a dictionary with 's' as
a currency symbol, 'p' as price and 'E' as time ( in unix format), these three are what we care about.

Next step is getting a stream of real time data, then send these data to a function in order to produce a dataFrame, and we'll start with the latter.

def dframe(msg):

    df = pd.DataFrame([msg])
    df = df.loc[:,['s','E','p']]    
    df.columns = ['Symbol','Time','Price']
    df.Price = df.Price.astype(float)
    df.set_index('Time')

    return df

I defined a function wich will take the received dictionary, convert it to pandas dataframe slice the Symbol,Time and price.

while True:
   bsm = BinanceSocketManager(client)
   ts = bsm.trade_socket('BTCUSDT')
   await ts.__aenter__()
   msg = await ts.recv()
   frame = dframe(msg)
   frame.to_sql('BTCUSDT',engine,if_exists='append',index=False)
   print(frame)

in order to get a stream of data, I'll request data in an endless loop, after getting the msg dictionary it will be sent to *dframe * function to update the data frame.

Then to_sql() Write records stored in a DataFrame to a SQL database.

and that's this is the output of the endless loop, a dataframe updated every second with recieved data, and write records to BTCUSDT table in SQl database.

   Symbol           Time     Price
0  BTCUSDT  1645891574400  39268.11
    Symbol           Time     Price
0  BTCUSDT  1645891576796  39268.89
    Symbol           Time     Price
0  BTCUSDT  1645891578340  39262.33
    Symbol           Time     Price
0  BTCUSDT  1645891579856  39262.33
    Symbol           Time     Price
0  BTCUSDT  1645891581334  39262.32
    Symbol           Time    Price
0  BTCUSDT  1645891582679  39250.4
    Symbol           Time    Price
0  BTCUSDT  1645891584140  39250.4
    Symbol           Time     Price
0  BTCUSDT  1645891585365  39250.39
    Symbol           Time     Price
0  BTCUSDT  1645891586707  39258.25
    Symbol           Time    Price
0  BTCUSDT  1645891587986  39262.4
    Symbol           Time    Price
0  BTCUSDT  1645891589897  39262.4
    Symbol           Time     Price
0  BTCUSDT  1645891591257  39262.39
    Symbol           Time     Price
0  BTCUSDT  1645891593257  39253.68
    Symbol           Time     Price
0  BTCUSDT  1645891594699  39253.68
    Symbol           Time     Price
0  BTCUSDT  1645891596403  39253.68
    Symbol           Time     Price
0  BTCUSDT  1645891597752  39253.68
    Symbol           Time     Price
0  BTCUSDT  1645891599107  39253.69
    Symbol           Time     Price

let's make a plot to visualize btc price.

import matplotlib.pyplot as plt

df = pd.read_sql('BTCUSDT',engine)

plt.figure(figsize=(15, 8))
plt.plot(df.Price,color='red',label = "Price", linewidth = 2, marker='*', markerfacecolor='black',markersize=9)

plt.title('BTC Price', color='black',size=20)
plt.ylabel('Price', color='blue',size=16)
plt.xlabel('no of records', color='blue',size=16)
plt.legend()

Trend Following Strategy

I'll define a simple enough trend following strategy, which is Buy if the price increased with a percentage(0.1%), and Sell if certain percentage up or down (+0.2% , -0.2%) is reached.

def strategy(in_position=False):
    percen = 0.001
    periods = 100


    while True:
        df = pd.read_sql('BTCUSDT', engine)
        df1 = df.tail(periods)
        cumret = (df1.Price.pct_change() + 1).cumprod() - 1
        if not in_position:
            if cumret[cumret.last_valid_index()] > percen:
                buyOrder = client.create_order(symbol='BTCUSDT', 
                                              side='BUY', 
                                              type='MARKET' ,
                                              quantity=0.001)
                print(buyOrder)
                in_position = True
                break
  if in_position:
        while True:
            df = pd.read_sql('BTCUSDT', engine)
            orderPrice = df.loc[df.Time > buyOrder['transactTime']]


            if len(orderPrice) > 1 :
                postionRet = (orderPrice.Price.pct_change() + 1).cumprod() - 1
                lastRet = postionRet[postionRet.last_valid_index()] 

                if lastRet > 0.002 or lastRet < -0.002:
                    sellOrder = client.create_order(symbol='BTCUSDT' ,
                                                   side='SELL' ,
                                                   type='MARKET' ,
                                                   quantity=0.001 ) 

                    print(sellOrder)            
                    break

start with calculating the last no of records to calculate the cumulative return, and when the cumulative return exceeds certain percentage then Buy order will be triggered, then break the while loop.

The buy order will return a dictionary with transaction Time so, we'll slice the dataframe from this time and wait till a cumulative return of the sliced dataframe exceeds or go below sell percentage, then a sell order will be triggered.

Time To Execute a Trade...

it's time to call the strategy function.

strategy()

we get a Buy order filled at price 39166.91

Buy order

{'symbol': 'BTCUSDT', 'orderId': 10940581, 'orderListId': -1, 'clientOrderId': 'XCAVbax0xaQxRs0erpds5P', 'transactTime': 1645895549187, 'price': '0.00000000', 'origQty': '0.00100000', 'executedQty': '0.00100000', 'cummulativeQuoteQty': '39.16691000', 'status': 'FILLED', 'timeInForce': 'GTC', 'type': 'MARKET', 'side': 'BUY', 'fills': [{'price': '39166.91000000', 'qty': '0.00100000', 'commission': '0.00000000', 'commissionAsset': 'BTC', 'tradeId': 3562682}]}

we get a Sell order filled at price 39097.37

Sell order

{'symbol': 'BTCUSDT', 'orderId': 10945890, 'orderListId': -1, 'clientOrderId': 'txnRsAuc1gcMs3aRiQ6YGc', 'transactTime': 1645896773538, 'price': '0.00000000', 'origQty': '0.00100000', 'executedQty': '0.00100000', 'cummulativeQuoteQty': '39.09737000', 'status': 'FILLED', 'timeInForce': 'GTC', 'type': 'MARKET', 'side': 'SELL', 'fills': [{'price': '39097.37000000', 'qty': '0.00100000', 'commission': '0.00000000', 'commissionAsset': 'USDT', 'tradeId': 3564175}]}

We have a trade with about 0.18 loss, the strategy is working and of course if you know about trading you won't be surprised about the loss because you can't win each time you just try to have a good winning ratio and the overall will result a profit.

What's next ?!

You can optimize this bot by changing the parameters or manipulate the strategy which will make it better.

I will share more in the upcoming articles

and that's it for this article, I hope you the best.

Algorithmic Trading: A Beginners Guide series.

shady shafik — Sun, 13 Feb 2022 18:31:52 +0000

What is algorithmic Trading ?!

In simple words, algorithmic trading is the use of programs to make systematic trades based on the strategy implemented by the programming language.
algorithmic trading can be fully-automated, semi-automated, or give signals to be executed manually.

In this article, we will develop a simple semi-automated strategy using Binance API to get data and execute trades.
I am working with Jupiter notebook, using python, pandas, and NumPy

Starting with importing the libraries needed install python-binance to connect to Binance API.

I am using binance testnet API to make anyone able to go and execute the code even if you don’t have binance account.

first, you have to have an API KEY and SECRET KEY to provide to the client function to establish a connection
to generate an API key and secret key go to Binance Spot Test Network sign with GitHub and generate HMAC SHA256 key.
finally, replace APIKEY and SECRETKEY variables in the code with actual keys.


import pandas as pd
import numpy as np
!pip install python-binance
from binance import Client

APIKEY = 'Your API KEY'
SECRETKEY = 'Your Secret KEY'
client =Client(APIKEY,SECRETKEY)
client.API_URL = 'https://testnet.binance.vision/api'
acc = client.get_account()
acc

after establishing connection let’s see our TEST account balance Info with client.get_account() function, and here is the result.


---


{'makerCommission': 0,
 'takerCommission': 0,
 'buyerCommission': 0,
 'sellerCommission': 0,
 'canTrade': True,
 'canWithdraw': False,
 'canDeposit': False,
 'updateTime': 1644598822834,
 'accountType': 'SPOT',
 'balances': [{'asset': 'BNB',
   'free': '1000.00000000',
   'locked': '0.00000000'},
  {'asset': 'BTC', 'free': '1.01300000', 'locked': '0.00000000'},
  {'asset': 'BUSD', 'free': '10000.00000000', 'locked': '0.00000000'},
  {'asset': 'ETH', 'free': '100.00000000', 'locked': '0.00000000'},
  {'asset': 'LTC', 'free': '500.00000000', 'locked': '0.00000000'},
  {'asset': 'TRX', 'free': '500000.00000000', 'locked': '0.00000000'},
  {'asset': 'USDT', 'free': '9449.55739000', 'locked': '0.00000000'},
  {'asset': 'XRP', 'free': '50000.00000000', 'locked': '0.00000000'}],
 'permissions': ['SPOT']}

Now it’s time to get crypto data stream

we have more than one option about how to get data from Binance, in this article I want to make it simple enough so I am using

client.get_historical_klines() function

Parameters provided to this function are:

symbol: which is the cryptocurrency symbol you want to trade in our case its BTCUSDT

Time Frame: time frame which you’ll trade in our case I chose a 1minute time frame.

Period: How many past periods to get in our case I chose 20 minutes.


def dataorg(symbol):
    data = pd.DataFrame(client.get_historical_klines( symbol ,
                                                      '1m',                                               
                                            '20 min ago UTC'))
    df = data.iloc[:,:6]
    df.columns = ['Time', 'Open', 'High' , 'Low',
                                         'Close','Volume']
    df = df.set_index('Time')
    df.index = pd.to_datetime(df.index,unit='ms')
    df = df.astype(float)
    return df


dataorg('BTCUSDT')

I defined a function that takes a symbol argument to organize data,
and make a data frame out of it

here is the resulting data frame containing:

Time , Open Price , Low Price , Close Price and Volume.

Here we want to calculate the cumulative return of the last 20 minutes to start work on the trading strategy.

Cumulative return formula: (1+ 1min return)*(1+cumlative return of last min)-1

pct_change(): calcluate the change in price in the last period.

cumprod(): calculate the second half of the formula.

test = dataorg('BTCUSDT')
ret = (test.Open.pct_change() + 1).cumprod() -1
ret
Time
2022-02-11 16:52:00         NaN
2022-02-11 16:53:00    0.000279
2022-02-11 16:54:00    0.000974
2022-02-11 16:55:00    0.001551
2022-02-11 16:56:00    0.001645
2022-02-11 16:57:00    0.002303
2022-02-11 16:58:00    0.002782
2022-02-11 16:59:00    0.002060
2022-02-11 17:00:00    0.001818
2022-02-11 17:01:00    0.000858
2022-02-11 17:02:00    0.001216
2022-02-11 17:03:00    0.002534
2022-02-11 17:04:00    0.002666
2022-02-11 17:05:00    0.003242
2022-02-11 17:06:00    0.002548
2022-02-11 17:07:00    0.001416
2022-02-11 17:08:00    0.002460
2022-02-11 17:09:00    0.002109
2022-02-11 17:10:00    0.002550
2022-02-11 17:11:00    0.003376
Name: Open, dtype: float64

Strategy implementation

We will develop a trend following strategy which as simple as:
If the price of Bitcoin increases by more than a certain percentage (0.2%) then BUY order and after buy If price increased or decreased by certain percentage ( 0.2% , -0.2%) then SELL.

def strategy(symbol,qty,inPosition=False):
    while True:
        df = dataorg(symbol)
        cumret = (df.Open.pct_change()+1).cumprod()-1
        if not inPosition:
               if cumret[-1] > 0.002:
                    ordertime = df.index[-1]
                    order = client.create_order(symbol=symbol,  
                    side='BUY',type='MARKET', quantity=qty)

                    print(order)
                    inPosition=True
                    break
                else:
                    print('No trade excuted')
     if inPosition:
        while True:
            df=dataorg(symbol)
            afterbuy = df.loc[df.index > pd.to_datetime(
            order['transactTime'],
            unit='ms')]

            if len(afterbuy) > 0:
                afterbuyret =  (df.Open.pct_change()+1).cumprod()-1                    
                print(afterbuyret)
                if afterbuyret[-1]>0.002 or afterbuyret[-1]< -0.002:

                    sellorder = client.create_order(symbol=symbol,
                                        side='SELL',type='MARKET',
                                                 quantity=qty)
                    print(sellorder)
                    break

it’s time to see the prices in the last 20 mins in a plot.

test.Open.plot()

It seems like we have an uptrend .

Trade execution

Our final step is to call strategy() function and wait for a trade to be filled,

In this case we have a buy order filled then list of cumulative return after a trade and when the return exceed our threshold (+0.2% or -0.2%) then a sell trade been filled
and it’s a profitable trade 😀

strategy('BTCUSDT', 0.001)
{'symbol': 'BTCUSDT', 'orderId': 4982968, 'orderListId': -1, 'clientOrderId': 'oappYtNU1466jYiXCV3URd', 'transactTime': 1644599469117, 'price': '0.00000000', 'origQty': '0.00100000', 'executedQty': '0.00100000', 'cummulativeQuoteQty': '43.65218000', 'status': 'FILLED', 'timeInForce': 'GTC', 'type': 'MARKET', 'side': 'BUY', 'fills': [{'price': '43652.18000000', 'qty': '0.00100000', 'commission': '0.00000000', 'commissionAsset': 'BTC', 'tradeId': 1240296}]}
Time
2022-02-11 16:52:00         NaN
2022-02-11 16:53:00    0.000279
2022-02-11 16:54:00    0.000974
2022-02-11 16:55:00    0.001551
2022-02-11 16:56:00    0.001645
2022-02-11 16:57:00    0.002303
2022-02-11 16:58:00    0.002782
2022-02-11 16:59:00    0.002060
2022-02-11 17:00:00    0.001818
2022-02-11 17:01:00    0.000858
2022-02-11 17:02:00    0.001216
2022-02-11 17:03:00    0.002534
2022-02-11 17:04:00    0.002666
2022-02-11 17:05:00    0.003242
2022-02-11 17:06:00    0.002548
2022-02-11 17:07:00    0.001416
2022-02-11 17:08:00    0.002460
2022-02-11 17:09:00    0.002109
2022-02-11 17:10:00    0.002550
2022-02-11 17:11:00    0.003376
2022-02-11 17:12:00    0.003520
Name: Open, dtype: float64

{'symbol': 'BTCUSDT', 'orderId': 4983220, 'orderListId': -1, 'clientOrderId': 'OWLlWGmI044UlKEFaURffy', 'transactTime': 1644599521374, 'price': '0.00000000', 'origQty': '0.00100000', 'executedQty': '0.00100000', 'cummulativeQuoteQty': '43.65542000', 'status': 'FILLED', 'timeInForce': 'GTC', 'type': 'MARKET', 'side': 'SELL', 'fills': [{'price': '43655.42000000', 'qty': '0.00100000', 'commission': '0.00000000', 'commissionAsset': 'USDT', 'tradeId': 1240505}]}

what's next

Ok guys, that's it for this article of course this isn't the best trading bot to make profits and it lacks a lot,
especially data anlysis and backtesting ,
I'll provide more insights in my upcoming article,
I hope you enjoy.

Machine learning: Cost function and Gradient Descent For Linear Regression.

shady shafik — Tue, 02 Nov 2021 11:06:40 +0000

Introduction

Gradient descent is one of the most popular optimization algorithms in machine learning and deep learning,

it used to find the local minimum of a differentiable function.

In machine learning, gradient descent is used to find the values of parameters that minimize the cost function (loss function).

“A gradient measures how much the output of a function changes if you change the inputs a little bit.” — Lex Fridman

In this article, we’ll go through

What is gradient in the first place ?!

hypothesis function(st-line equation).

Cost function for linear regression.

Gradient descent for linear regression.

In the previous article, we went through Linear regression, what is regression, hypothesis function, and cost function.

What is a gradient in the first place ?!

In mathematics, a gradient is another name for the slope of the st-line or how steep is the st-line.

Gradient = slope = derivative of a function

Hypothesis function and plotting different values of theta.

let’s start by taking an example,
we have 3 sample points ( 2, 2 ), ( 4, 4 ), ( 8, 8 )

the straight-line equation (Hypothesis function)

it’s also written as: h(x)= y = θ1 + θ2 x

let’s make it simple enough and make θ1= 0.

the equation will be:

h (x) = θ2.X

now we want to draw a straight line h(x) = θ2 x to best fit the data

then predict (y) values of training data (x,y) to see how good the line fits the data by calculating the cost function ( loss function )

of course in this example, it's obvious that we want to choose θ2= 1 to best fit the line but with too many samples ( thousands or millions ) it’s not that easy

we have to use Gradient descent or other algorithms to choose the best parameters,
even if this is a very simple example but it will clear the vision about how the parameters are chosen and how gradient descent works.

I’m choosing two values for θ2 ( 1.5 and 0 )
( that’s the Gradient descent rule to choose the best parameters but we are choosing different values to understand how it goes)

so now the blue line equation will be h(x) = 1.5 x

and the red line equation h (x) = 0x = 0

so the slope = tan ( θ ) = tan( 0 ) = zero degree.

that will make the line lie on the x-axis as you see

and the last line we will draw of course will be with θ = 1
the green st-line equation will be

h(x) = x

and that will make the line pass through each point of the samples. (which is not good by the way and called overfitting but in this case let’s assume it's a good thing)

Cost function and Plotting different values for theta.
Now we have three models to choose from the best model which makes the best predictions.

h(x) = 1.5 x

h(x) = 0

h(x) = x

We want each equation of those to make predictions of output h(x) based on a given value of the input (x) by using our training examples

Then we calculate the cost function ( model accuracy ) to choose the best model of them.

We have 3 training examples of (x,y)
(2,2 ), (4,4),(8,8)

we’ll start by:

h(x) = 1.5x

for x = 2 , h(x)=1.5*2 = 3
for x = 4 , h(x)=1.5*4 = 6
for x = 8, h(x)=1.5*8 = 12

It’s time to calculate the cost function (model accuracy)

The cost function for linear regression.

where

θ: equation parameters

i: index

m: number of training examples( in our case we have 3 training examples)

h(x): predicted value

y: expected value

we’ll calculate J(1.5 ) , J(0) , J(1)

let’s substitute our predicted values of h(x)

for J(1.5)
h(2) = 3, h(4)= 6 and h(8)= 12

so, J(1.5) = 1/2m ( (3–2 )² +(6–4)² +(12–8 )²)
J(1.5) = 1/2*3 ( 1² + 2² + 4²) = 1/6 ( 21)
J(1.5) = 3.5

We will do the same for J(0)
h(x) = 0 so, all outputs will be equal 0
the cost will be as follow

J(0) = 1/2m ( (0–2 )² +(0–4)² +(0–8 )²)
J(0) = 1/6* (84) = 14

the last cost will be for θ = 1
**h(x) = x
h(2) = 2 , h(4)= 4 and h(8)= 8
J(1) = 1/2m ( (2–2 )² +(4–4)² +(8–8 )²)

J(1) = 1/6 ( 0 )
J(1) = Zero**

which means that h(1) make perfect predictions

So, we’ll choose θ= 1 for the best predictions,

what we just did in the previous steps is what the Gradient descent algorithm does.

Gradient Descent For Linear Regression (How it works).

Now we want to see the relationship between J(θ) and θ

we have 3 points to plot ( θ, J(θ) ) with J(θ) on the y-axis and θ on the x-axis

Points : (0, 14) , (1, 0) and (1.5, 3.5)

the perfect choice of parameters will make J(θ) = zero.

but, a perfect model with 100% accuracy working with a lot of data is rare if not impossible at least for now.

gradient descent algorithm starts by initializing starting point (x,y)
and then tries to find the Global minimum ( the value of θ gives minimum value to J(θ))

which in our graph the point ( 1, 0 )(Global minimum)
when θ=1, J(θ) = 0.

But how the heck the gradient descent algorithm does that ?!

Now it’s time to see the gradient descent general formula

Where

θ : parameter ( also called weight )

j = 1 or j = 2 ( number of parameters θ)

α : learning rate

∂ / ∂θ : partial derivative ( slope of straight line )

J(θ1,θ2) : cost function

In our example θ1 = 0, so we just have θ2 to deal with

gradient descent algorithm goal is to find values for equation parameters (θ2) which make J(θ2) is minimum

it starts with an initial value of θ then update this value until converge ( reach the minimum cost )

θ2 ( new value ) = θ2 (initial value ) - α( number )* ∂/∂θ2 ( slope of the st- line )

gradient descent may do this step thousands of times or more with one or more parameters.

now let’s see how the slope of the straight line work in this equation

how it converges if the slope is positive

how it converges if the slope is negative

Machine Learning From Scratch: Linear Regression step by step.

shady shafik — Sat, 23 Oct 2021 13:21:08 +0000

Introduction

If you want to get into machine learning, I can’t think of a better way than start by learning Linear regression.
In this article, you will get a clear vision on what is linear regression, hypothesis function, and cost function.
Linear Regression is one of the most straightforward algorithms.
It is easy to understand and known by all data scientists and machine learning engineers.
also, it’s a very good starting point to learn other machine learning algorithms.
yet, linear regression is a very useful and efficient algorithm,
when used in the right problem with the right dataset.
In this article, we will go step by step to understand how linear regression is working.
So let’s not waste more time and get started.

What is a regression in the first place ?!

In statistical modeling, regression is the process of forecasting a dependent variable (Y). based on one or more of an independent variable (X).
in other words, a regression model is a statistical tool used to find the relation between the output (Y). and the input (X).

Linear Regression Algorithm.

The linear regression model is the basic form of regression analysis,
it estimates that the relation between the dependent and independent variables is linear.
We know from linear algebra that the straight-line equation :
y = mx+ b

we will use this equation to predict the output variable (y) with the relation to x( feature), m ( the slope of a straight line ), and b ( intersection of st-line with y-axis ).

equation of blue line y = x + 2
equation of red line y = 2x + 0

hypothesis equation

hypothesis equation is the st-line equation used in the machine learning algorithm with optimized parameters to best fit the data.

it’s a model that maps inputs to outputs and is used to make predictions.

the basic idea here or in any machine learning problem is:
feed the model with data.

test the model by predicting values you already know.
optimize the model for better predictions, and now you have a working machine learning model that can predict outputs.
let’s change the parameters of the st-line equation m → θ2, b → θ1

Let's give different values for theta (θ1,θ2) and see how the line changes.

we see that as θ2 changes the angle (slope) of the st-line changes and as θ1 changes the line intersection with the y-axis changes.

Accuracy

when our models make predictions, it’s not 100% accurate
in classification problems where the prediction is binary ( 0 or 1 )

the ratio between right predictions and wrong predictions is the model accuracy

if our model predicts 7 out of 10 right predictions so, the accuracy of the model equal to 70%

Cost function

Now we have a model that makes predictions by the hypothesis equation and we want to know how well are those predictions
in simple words, the cost function tests how well the model performs,
if the cost function is equal to zero, then the model is 100% accurate.

different from accuracy cost function is not a percentage
but, calculated by the error difference between the predicted value (ŷ) and expected value ( True value) (y).

Mean Squared Error

MSE = Mean Squared Error
m = number of training examples ( samples ).
i = index of sample
ŷ = predicted value
y = expected value

the predicted value ŷ is the output of the hypothesis while the expected value y is the value of training examples.

let’s use this dummy plot to get an intuition about how the MSE works.

the orange line height is the difference between the predicted value and the expected value (blue dot ).

let’s predict Y for these three values of X (3,4,6)

when
x =3 the predicted value ŷ = 3.9 while the expected value y = 2
x =4 the predicted value ŷ = 4.7 while the expected value y = 6
x =6 the predicted value ŷ = 5.8 while the expected value y = 4

now we calculate the cost with the MSE formula

we just have 3 training examples m = 3

MSE = 1/ 2* 3 ( (3.9–2)² + (4.7–6)² + (5.8–4)² )

MSE = 1/6 ( (1.9)² + (-1.3)² + ( 1.8)² ) = 1.42

of course, if we have a large dataset the cost will be much higher than that number.

I hope you’ve got an intuition about how cost function works, if not then am sure that the little project in the next section will give you a better understanding of the whole process.

Weight prediction based on height.

I picked 20 samples of weights and heights from a dataset so we can have a better understanding of the linear regression algorithm

first, we’ll plot the data then draw a regression line to make our predictions and see our model working.

Regression Line

Now we have a plot with height (x-axis)(independent variable) and weight (y-axis)(dependent variable).

it’s time to draw the regression line using the hypothesis equation

y = θ1 + θ2. x

But, how do we choose the θ1 and θ2 parameters?

Ok, that’s where Gradient Descent comes to play, it’s a learning algorithm used to choose the best values for parameters to best fit the data, therefore, minimize the cost function (prediction error ).

we’ll talk about gradient descent in another article and for now, let’s say we have chosen the optimized parameters to draw the regression line.

and here comes the interesting part, let’s make a prediction,
I want our model to predict the weight of a 70 inches, let’s do it

it predicted ~135 pounds for 70 inches, if we look in our dataset table you’ll find 70.1 inches with 136.4 pounds.
so, we can say that our simple model has good accuracy.

Recap

In this article, we went through the main idea behind linear regression and how it works.

Regression is the process of finding the relation between input and output.

Linear regression algorithm estimates that the relation between the dependent and independent variables is linear.

The hypothesis function is the straight-line equation used in the machine learning model to map the input to output.

Cost function tests the accuracy of the hypothesis function by calculating the average difference between the predicted value and the actual values from the dataset.