DEV Community: shellyalmo

A quick guide to building a Docker container for your Python application

shellyalmo — Sun, 25 Oct 2020 12:06:41 +0000

On my previous post I explained how to build your own Data Pipeline from scratch. I mentioned that in order to use my GitHub repository you will need to have Python 3 and pip installed.

After building the pipeline, I wanted to run my software on a brand new server, where I hadn't setup Python and the libraries my program needs like pip, pandas and so on.

I was wondering how could I make sure my program would work exactly the same way on the server, or on some other developer's laptop, without having to worry about what version of Python they have. This is where Docker containers are very useful, and I thought it would be a great opportunity to combine Docker technology with my project.

There are some amazing tutorials on Docker that are great for beginners, like this video crash course, and this article that focuses on Docker in Data Science. Since I still have a lot to learn myself, my post is not going to be a complete Docker guide, and I recommend you to read more about it and get comfortable with this concept.

Let's cover the basics briefly:

Image = a blueprint of our application. Imagine you can take a snapshot of your program after clicking the "run" button. The image is built from a Dockerfile that has all the commands that are necessary to run your program, and that copies the necessary libraries and dependencies. Inside the image the application has everything it needs in order to work properly (operating system, application code, system tools etc).
Container = an instance of our image. As explained on the Docker website:

"Container images become containers at runtime...[and they] isolate software from its environment..."

This post will teach you:

How to build a Docker file that copies anything needed to run your Python project.
How to create an image from that Docker file.
How to run this image in a Docker container.

Let's get started:

All the instructions are following my previous post about the weather data pipeline that I created.
Now the only thing you have to install is Docker Desktop on your computer.

Writing the Dockerfile:

The Dockerfile is the recipe of commands that are building our image.
I created a file called "Dockerfile" without any extensions:

Let's go through it line by line:

Line 1: Base image. This preconfigured Docker image has Python and pip pre-installed and configured.

FROM python:3

Line 3: Set the working directory in the container:

WORKDIR /usr/src/app

Line 5: Copy all the dependencies from your project's requirements.txt. Keep in mind that each line of a Dockerfile creates a separate layer. Docker knows when the input to any given layer has changed, so when your application code changes but your requirements do not, Docker doesn't waste time downloading all your requirements.txt every time you build the image.

COPY requirements.txt ./

Line 6: Reduce the size of your Docker image. Disabling cache allows to avoid installing source files and installation files of pip.

RUN pip install --no-cache-dir -r requirements.txt

Line 8: Copy the project's folder into an image layer. My Python files are stored locally in a folder called src.

COPY src/ .

Line 9: Make a new directory called data_cache. This is where all the current weather data will be stored after retrieving it.

RUN mkdir data_cache

Line 11: Run main.py by default when the container is started. CMD tells Docker what commands to execute.

CMD [ "python", "main.py" ]

Building the image from the Dockerfile:

Run this to build the Docker image:

$ docker build . -t <yourDockerID>/weather

Name your local image using your Docker Hub username. In this case, weather is the name of the image.
Let's look at the output:

Running the image on a Docker container:

The following command will run your image. As I mentioned before, the API key you get from OpenWeatherMap has to remain private, and that's why I showed previously how to make it an environment variable.
Now instead of using the .env file to set the environment variable, we can use Docker's command --env.

$ docker run --env api-token=<yourapifromOpenWeatherMap> -it <yourDockerID>/weather

We can see our weather image on the Docker Desktop:

Another way to see our image is to run on the command line:

$ docker images

And we can see our container on Docker Desktop:

Now you can proudly say that you have built your own image of your Python application. I uploaded mine on DockerHub so you can just download it and use it for retrieving weather data from web API.

How to build a Data Pipeline for the first time

shellyalmo — Mon, 12 Oct 2020 09:10:05 +0000

I am fascinated by Machine Learning models and the incredible tools they offer our world, whether for making decisions, predicting trends, improving lives and even saving them. Those models are trained and tested on tremendous amounts of data that is constantly being collected and stored in databases.

I was really curious to learn how raw data gets into programs and what processes are being done to make this raw data useful, even before training a brilliant Machine Learning model.

One type of data that we use on a daily basis for predictions is weather observations. Obviously, weather forecasting is mostly based on complicated physics equations and statistical models. Still, I thought it would be fun to build my own weather database for practicing my Data Science skills.

Instead of connecting temperature sensors to my computer and opening my own meteorological station at home (which could be very cool for my next project), I decided to build a simple data pipeline to see how it's done.

My mission was to build a pipeline from scratch - starting with retrieving weather data from the OpenWeatherMap current weather web API, parsing the data using Pandas (python data analysis) library and storing it in a local SQLite database.

Here's how I did it:

Step 1: Get an API key

OpenWeatherMap offers a few paid API plans, some are monthly subscriptions with great powerful tools. I was happy to start with the free plan which offers access to current weather data from any city.

Step 2: Keep your API key private

You wouldn't want some stranger to have your ATM password, right? API keys can also be used to steal from your wallet. Especially if you are a paying customer for the service this API provides, and if you have limited API calls. Environment variables come in handy when dealing with this problem, since they are variables whose value is set outside the program (highly recommended tutorial here). By making my API key an environment variable, it is hidden outside of my program and can be used without being exposed on my public GitHub repository. I saved my API key into a new file and called it .env .This is what it looks like on the inside:

api-token = "typeyourapikeyhere"

Now we can start coding our pipeline. Before coding I took some time to design my program and eventually decided to separate my program into files by responsibility.
My program design guidelines were:

Each function does one thing.
Each function is tested on its own, in order to make sure it actually does what it's supposed to do. Also, it saves a big headache when other bugs pile up and you have to figure out which function is the problematic one. Here I chose to use doctest for simple small tests.

For the next steps, make sure that:

Python 3 and pip are installed already.
If you are using the files on my GitHub repository, you can install the dependencies by running the following command:

pip install -r requirements.txt

In the folder where you'll run my program, you need a folder called data_cache. This is where all the data will be saved before storing it in the database.

Step 3: Create a new empty Database

In a python file, I created a SQLite Database with the sqlite3 library:

Step 4: Retrieve the data and save as a json file

At this point you will be able to get the data in json format and save it as a json file in your current folder. Each json file is named after the "dt" value which stands for datetime. Please notice that the datetime format is Unix Epoch Timestamp.

Step 5: From json file to dictionary to Pandas Dataframe

This might seem a bit Sisyphean, but I preferred to break down the process into as many "baby steps" as possible. To me, it's organized, clear and helps keeping track of each step.

Step 6: ETL with Pandas

The ETL procedure enables us to extract and transform the data according to our analysis needs, and then load it to our data warehouse. In order to make my data useful for future Data Science projects, I made sure my database will contain necessary parameters for daily temperature prediction (current temperature in Celsius, minimal and maximal temperatures in Celsius, humidity, pressure and wind). Also, I chose the datetime ("dt") column to be the row index as a primary key for my database.

Step 7: Update the Database

Now that we have the current weather data saved in a dataframe, we can easily load it to our database by using the Pandas library.

For flexibility, I used the argparse library, which lets you run main.py from the command line and give it a city id as an optional argument. So even though I defined Tel Aviv city by default, the user can still run the program for any city in the world. For example, if you would like to get the weather data of Detroit, US:

python3 src/main.py --city_id "4990729"

When running main.py, steps 3-7 are executed:

Done!

And there you have it! A pipeline you can build on your own. Eventually, the program is meant to be run on a schedule to build a database over time for my next Data Science project. For now, Windows Scheduler is a great way to start, but I recommend checking out Python Scheduler as well. There are some wonderful tutorials out there, just waiting to be explored.

Edit 10/18/20 :

The program runs on a default schedule every 15 minutes by using a while loop, and you can set the frequency to any you would like. For example, in order to run it every 5 seconds, run the command:

python3 src/main.py --frequency 5

All the Python files are currently stored in a local folder called src. In the next post I will explain how to create a docker image that only copies the Python files and dependencies that are necessary for this project.