DEV Community: Charles White

Getting Historical Stock Data Using Python

Charles White — Sun, 16 May 2021 00:26:03 +0000

There's an easy way to get stock data using pandas_datareader from yahoo finance which uses basics of pandas dataframe and also doing a moving average. It actually only takes a few lines of code and can help you in your data pipeline. This is just the quick snippet to get you going, if you want to see the full article you can scroll to the bottom.

Extracting data from yahoo with Pandas

We will be using Yahoo Finance for obtaining stock data. Firstly we will import pandas_datareader.data then send a request to yahoo finance with ticker , start and end date. The “pandas_datareader” module is a useful module that abstracts extracting web data very easily. You don’t have to bother with “requests” or “urllib” and parse the HTML data, this is all done for you! From the module in response we will get a dataframe containing our data in rows and columns . The columns are basically High , Low , Open , Close , Volume and Adj Close.

import pandas_datareader.data as web
import datetime as dt

ticker = "AAPL" # This is Apple's ticker
start = dt.datetime(2020,1,1) # Starting date in format (year,month,day)
end = dt.datetime(2020,12,1) # Ending date 

df = web.DataReader(ticker,"yahoo",start,end)
print(df.head()) # Prints first 5 columns of data
print(df.shape) # Prints the number of rows and columns

You get the output:

df = web.DataReader(ticker,"yahoo",start,end) - This stores the dataframe obtained in the variable df. The first value passed here is ticker i.e. “AAPL” , second is the source name . As we are using yahoo finance here so it should be “yahoo”. Then the end and start date (in the form of datetime). You can see other data sources here.
print(df.head()) - It prints the first 5 columns of the data obtained.
print(df.shape) - Prints the rows and columns in the dataframe . As seen in output there are 241 rows and 6 columns

There's more..

There's still want to know more you can find out all of these and more at our full article: Full article: how to get and chart historical stock data

Execute commands from the command line in a python program

Charles White — Sun, 16 May 2021 00:15:02 +0000

While working in Python you may come across cases where you want a quick way to run programs from the command line. This could be your own python scripts or any other command line program. This is where the subprocess module can come into play.

The subprocess module, present in a standard installation of Python is used to run new applications or programs through Python code by creating new processes.

It gives us the ability to:

spawn new processes
connect to their input, output and error pipes
obtain their return codes

A common use for subprocess would be to list files and folders in a directory. This makes use of the run method of subprocess.

The code will differ depending on what operating system you are on. For example, on a windows based computer this would be using the command dir

For Linux

import subprocess

subprocess.run('ls')

For Windows

import subprocess

subprocess.run("dir", shell=True)

In the code above we have:

import subprocess module
call the run module from it
pass the list directory command based on your system (ls/dir). If you are on Windows you will have to additionally pass shell=True because dir is a shell command and you need to tell the system that you want to use it.

We automatically got output in the terminal even though we did not print it ourselves. That is how the run command works – it does not capture our command output by default.

Let’s see what happens if we try to capture our command by placing it in a variable

import subprocess

result = subprocess.run("dir", shell=True)

print(result)

The output is not what you would expect. We get the output still, and a message containing the arguments that were passed and the return code.

If we wanted to capture the output we would modify out code to this:

import subprocess

result = subprocess.run(["dir"], shell=True, capture_output=True, text=True)

print(result.stdout)

We have added:

capture_output=True: to capture the output
text=True: to decode the output to a readable format since it is captured as bytes
We then print result.stdout : the result as standard output

The same can be achieved with the follow code:

import subprocess

result = subprocess.run(["ls", "-la"], stdout=subprocess.PIPE, text=True)

print(result.stdout)

The difference is stdout=subprocess.PIPE This code performs the same function as capture_output=True

In addition, we use the ls command with the argument of -la both passed as a list

There's more..

There's still more you can do with subprocess including outputting to a file, reading data from a file, and to trigger programs interactively to name a few things. Find out all of these and more at: Full Article at PythonHowToProgam

Supercharge your python code with Decorators

Charles White — Sat, 30 Jan 2021 17:34:20 +0000

What is a decorator?

A decorator is a feature in python that allows you to add extra capabilities to your function, without having to change the inner workings of your function.

Some examples of uses of decorators include: determining the execution time of a function, determine which URL call should result in this function being called, adding logging messages when a function starts/stops. There are many uses for this.

More technically, a decorator is a function that takes another function as an argument, adds some functionality then returns another function. This happens without altering the original source code of the function that was passed.

The Concept Behind Decorators

So the above should explain the power of decorators. So how does it actually work? Well it’s a combination of a few interesting features in python:

Functions are First-Class Objects: this means you can assign functions to variables, pass them as arguments, and return them as values from other functions.
Scope / Closure: this specifically refers to how a nested function that has access to variable(s) from its enclosing function.

It’s these two things that make it possible to have decorators. This is a diagram that shows what happens conceptually. The actual call to the original function is circumvented to call the decorator which encapsulates the call to the original call with some additional functionality. That’s where the magic lies

See the following diagram that helped me to conceptualise the flow which was really helpful.

To find out more about decorators including a practical example using parameters, you can see our full article Simple Guide To Decorators in Python 3 – When and How to Use Them. The articles show examples with code you can readily cut and paste.

Add Some Colour To Your Python Logs

Charles White — Mon, 04 Jan 2021 14:48:29 +0000

Logging is an essential part of any mid-large size application which will give you an idea of the inner working of your code. Once you get beyond a small application, just using print statements can be quite cumbersome and difficult to maintain.

The python logging library is already very useful to allow you to create output of logs, but then they can be quite difficult to read. It can be especially difficult when you are trying to trace through an execution to debug what happened.

This is where the colorlogs library can really help. With the library you can add some color based on types of logging messages - e.g. red for error, blue for debug etc.

See the following code snippet:

import coloredlogs, logging

mylogs = logging.getLogger(__name__)

coloredlogs.install(level=logging.DEBUG, logger=logger)
# Some examples.
mylogs.debug("This is debug")
mylogs.info("This is info")
mylogs.warning("This is warning")
mylogs.error("This is an error")
mylogs.critical("This is a critical message")

coloredlogs creates a stream handler and attaches it to the logger passed. It has its own default format and colour settings which can be customised as per interest. Lets first look at the few important parameters we can pass in coloredlogs.install()

level - An integer to denote the level. (Remember logging.DEBUG returns an integer)
logger - Name of the logger in which this stream handler has to be attached. (Performs same act as logger.add_handler() )
fmt - A string denoting the format style.
datefmt - A string denoting the format of asctime .
level_styles - A dictionary containing the data of level and their colors
field_styles - A dictionary containing the data of field and their colors

To find out more about setting up logs, you can see our full article Logging in Python 3, How To Output Logs to File and Console . We give tips on how to setup log output both to a screen and a file, as well as other great tips.

Plugin Architecture in Python

Charles White — Tue, 29 Dec 2020 16:31:53 +0000

For some of your Python applications, a plugin architecture could really help you to extend the functionality of your applications without affecting the core structure of your application. Why would you want to do this? Well, it helps you to separate the core system and allows either yourself or others to extend the functionality of the core system safely and reliably.

Some of the advantages include that when a new plugin is created, for example, you would just need to test the plugin and the whole application. The other big advantage is that your application can grow by your community making your application even more appealing. Two classic examples are the plugins for Wordpress blog and the plugins for Sublime text editor. In both cases, the plugins enhance the functionality of the core system but the core system developer did not need to create the plugin.

There are disadvantages too however with one of the main ones being that you can only extend the functionality based on the constraints that is imposed on the plugin placeholder e.g. if an app allows plugins for formatting text in a GUI, it's unlikely you can create a plugin to play videos.

There are several methods to create a plugin architecture, here we will walkthrough the approach using importlib.

The Basic Structure Of A Plugin Architecture

At its core, a plugin architecture consists of two components: a core system and plug-in modules. The main key design here is to allow adding additional features that are called plugins modules to our core system, providing extensibility, flexibility, and isolation to our application features. This will provide us with the ability to add, remove, and change the behaviour of the application with little or no effect on the core system or other plug-in modules making our code very modular and extensible .

The Core System

The core system defines how it operates and the basic business logic. It can be understood as the workflow, such as how the data flow inside the application, but, the steps involved inside that workflow is up to the plugin(s). Hence, all extending plugins will follow that generic flow providing their customised implementation, but not changing the core business logic or the application's workflow.

In addition, it also contains the common code being used (or has to be used) by multiple plugins as a way to get rid of duplicate and boilerplate code, and have one single structure.

The Plug-in Modules

On the other hand, plug-ins are stand-alone, independent components that contain, additional features, and custom code that is intended to enhance or extend the core system. The plugins however, must follow a particular set of standards or a framework imposed by the core system so that the core system and plugin must communicate effectively. A real world example would be a car engine - only certain car engines ("plugins") would fit into a Toyota Prius as they follow the specifications of the chassis/car ("core system")

The independence of each plugin is the best approach to take. It is not advisable to have plugins talk to each other, unless, the core system facilitates that communication in a standardized way so that independent plugins can talk to each other. Either way, it is simpler to keep the communication and the dependency between plug-ins as minimal as possible.

Building a Core System

As mentioned before, we will have a core system and zero or more plugins which will add features to our system, so, first of all, we are going to build our core system (we will call this file core.py) to have the basis in which our plugins are going to work. To get started we are going to create a class called "MyApplication" with a run() method which prints our workflow



#core.py
class MyApplication:
    def __init__(self, plugins:list=[]):
        pass
    # This method will print the workflow of our application
    def run(self):
        print("Starting my application")
        print("-" * 10)
        print("This is my core system")
        print("-" * 10)
        print("Ending my application")
        print()

Now we are going to create the main file, which will import our application and execute the run() method



#main.py
# This is a main file which will initialise and execute the run method of our application

# Importing our application file
from core import MyApplication

if __name__ == "__main__":
    # Initialising our application
    app = MyApplication() 
    # Running our application 
    app.run()

And finally, we are run our main file which result is the following:

Once that we have a simple application which prints it's own workflow, we are going to enhance it so we can have an application which supports plugins, in order to perform this, we are going to modify the init() and run() methods.

The importlib package

In order to achieve our next goal, we are going to use the importlib which provide us with the power of implement the import statement in our init() method so we are going to be able to dynamically import as many packages as needed. It's these packages that will form our plugins



#core.py
import importlib

class MyApplication:
    # We are going to receive a list of plugins as parameter
    def __init__(self, plugins:list=[]):
        # Checking if plugin were sent
        if plugins != []:
            # create a list of plugins
            self._plugins = [
                # Import the module and initialise it at the same time
                importlib.import_module(plugin,".").Plugin() for plugin in plugins
            ]
        else:
            # If no plugin were set we use our default
            self._plugins = [importlib.import_module('default',".") .Plugin()]


    def run(self):
        print("Starting my application")
        print("-" * 10)
        print("This is my core system")

        # We is were magic happens, and all the plugins are going to be printed
        for plugin in self._plugins:
            print(plugin)

        print("-" * 10)
        print("Ending my application")
        print()

The key line is "importlib.import_module" which imports the package specified in the first string variable with a ".py" extension under the current directory (specified by "." second argument). So for example, the file "default.py" which is present in the same directory would be imported by calling: importlib.import_module('default', '.')

The second thing to note is that we have ".Plugin()" appended to the importlib statement: importlib.import_module(plugin,".").Plugin()

This (specifically the trailing brackets Plugin() is present) to create an instance of the class and store it into _plugins internal variable.

We are now ready to create our first plugin, the default one, if we run the code at this moment it is going to raise a ModuleNotFoundError exception due to we have not created our plugin yet. So let's do it!.

Creating default plugin

Keep in mind that we are going to call all plugins in the same way, so files have to be named as carefully, in this sample, we are going to create our "default" plugin, so, first of all, we create a new file called "default.py" within the same folder than our main.py and core.py.

Once we have a file we are going to create a class called "Plugin", which contains a method called process. This can also be a static method in case you want to call the method without instantiating the calls. It's important that any new plugin class is named the same so that these can be called dynamically



#deafult.py
# Define our default class 
class Plugin:
    # Define static method, so no self parameter 
    def process(self, num1,num2):
        # Some prints to identify which plugin is been used
        print("This is my default plugin") 
        print(f"Numbers are {num1} and {num2}")

At this moment we can run our main.py file which will print only the plugin name. We should not get any error due to we have created our default.py plugin. This will print out the module (from the print statement under the MyApplicaiton.run() module) object itself to show that we have successfully imported out the plugin

Let's now modify just one line in our core.py file so we call the process() method instead of printing the module object



#core.py
import importlib

class MyApplication:
    # We are going to receive a list of plugins as parameter
    def __init__(self, plugins:list=[]):
        # Checking if plugin were sent
        if plugins != []:
            # create a list of plugins
            self._plugins = [
                importlib.import_module(plugin,".").Plugin() for plugin in plugins
            ]
        else:
            # If no plugin were set we use our default
            self._plugins = [importlib.import_module('default',".").Plugin()]


    def run(self):
        print("Starting my application")
        print("-" * 10)
        print("This is my core system")

        # Modified for in order to call process method
        for plugin in self._plugins:
            plugin.process(5,3)

        print("-" * 10)
        print("Ending my application")
        print()

Output as follows:



# Output
$ py main.py 
 Starting my application
 This is my core system   
 This is my default plugin
 Numbers are 5 and 3      
 Ending my application

We have successfully created our first plugin, and it is up and running. You can see the statement "This is my default plugin" which comes from the plugin default.py rather than the main.py program.

Next Steps

Using this framework, you can easily extend this to add more plugins easily. The key is that you don't need to change the core.py nor the main.py which helps to keep your code clean but at the same time help to extend your application.

To see a more in depth version of this article and more examples, you can see the full article at PythonHowToProgram.com

How to Parse Arguments from the Command Line using argparse in Python 3

Charles White — Sat, 26 Dec 2020 16:13:41 +0000

Having command line parameters is an essential mechanism to manage the behaviour of your application but it can get complex to validate and extract the arguments. In our previous article on arguments (How to use argv parameters in Python), we went over how to use command line parameters with sys.argv. This is a useful module, but we will come across cases where we want to do more than just that. If you code enough in Python, you are most likely going to want to let users provide values for variables at runtime.

This is where argparse comes in. In this article, we are going to look into argparse - what it is. why we use it and how we parse arguments using it.

argparse is a python module whose primary purpose is to create command line interfaces. This module was released as a replacement for the getopt and optparse modules because they lacked some important features.

How to Parse Arguments with argparse

To understand how argparse works and how we go about making our own command line interfaces, we will begin with a simple example illustrated below:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("-n", "--name", required=True)
args = parser.parse_args()
print(f'Hi {args.name} , Welcome ')

Confusing? Yes, I know. But let's break this down line by line:

We initially import the argparse module
We then initialize the ArgumentParser object as parser
We add the only argument --name, here we must specify both the shorthand and longhand versions. Either can be used when the program is run
We stipulate that this argument is required
We execute the parse_args() method
We output the parsed argument

We get the output of our print statement plus the parsed argument. But what would happen if we ran the program but didn't give any arguments? We will try that next:

We got an error message, explaining the usage of the program and the arguments that are required. But we never added that to our code, did we? Well, not explicitly. When we added arguments for the program, argparse generates the error message if those arguments ate not given.

Looking at the output once again, you will notice there is another argument, one that we did not give -h. This is an optional argument that the program accepts. It serves as a sort of help command to describe what is required by the program.

Let's parse that argument to see what output we get:

As we see, parsing the -h argument gives a bit more detailed explanation of the program. Armed with a basic understanding of argparse we can look at another example.

We are going to code our own program that simulates the same operation as ls:

# Import the argparse library
import argparse

import os
import sys

my_parser = argparse.ArgumentParser(description='List all the contents of a folder')

my_parser.add_argument('Path',
                       metavar='path',
                       type=str,
                       help='the path to list')

args = my_parser.parse_args()

input_path = args.Path

if not os.path.isdir(input_path):
    print('The path specified does not exist')
    sys.exit()

print('\n'.join(os.listdir(input_path)))

We need to import a number of modules first namely os, sys, and argparse. After that we:

create the parser and assign it the name my_parser
add the necessary arguments - in this case, path.
We also specify the type of the argument as string which tells argparse to check the type automatically
execute the parse_args() function
create a variable named input_path and assign it args.Path - calling path on the parser
run a check if the parsed directory does not exist and display an error message if it doesn't
we output the inputted directory

We are running the program on the same directory as before:

Conclusion

There are many other things you can do - the original article came from Python How To Program: Argparse library article where you can find the complete post.