From Project to Productionized with Python

Ed Morley — Thu, 16 Jul 2020 18:20:20 +0000

We hope that you and your loved ones are staying safe from the COVID-19 pandemic. As a result of its effect on large gatherings, PyCon 2020 was ~~cancelled~~ changed to an online event. Although not being able to gather in person was disheartening for organizers, speakers, and attendees, the Python community shared virtual high-fives and hugs with PyCon US 2020 Online. We recorded our planned Heroku workshop for the event, on which this blog post is based.

Imagine that you've just spent the last two weeks pouring all your energy into an application. It's magnificent, and you're finally ready to share it on the Internet. How do you do it? In this post, we're going to walk through the hands-on process aimed at Python developers deploying their local application to Heroku.

An application running on Heroku works best as a 12-factor application. This is actually a concept that Heroku championed over 10 years ago. It's the idea that you build an application with robust redeployments in mind. Most of this workshop is actually not specific to Heroku, but rather, about taking a regular Django application and making it meet the 12 factor app methodology, which has become a standard that most cloud deployment providers not only support but recommend.

Prerequisites

Before completing this workshop, we're going to make a few assumptions about you, dear reader. First, this is not going to be a Django tutorial. If you're looking for an introduction to Django, their documentation has some excellent tutorials to follow. You will also need a little bit of Git familiarity, and have it installed on your machine.

In order to complete this workshop, you'll need a few things:

An account on Heroku. This is completely free and doesn't require any payment information.
The Heroku CLI. Once your application is on Heroku, this will make managing it much easier.
You'll need to clone the repository for this workship, and be able to open it in a text editor.

With all that sorted, it's time to begin!

Look around you

With the project cloned and available on your computer, take a moment to explore its structure. We'll be modifying the manage.py and requirements.txt files, as well as settings.py and wsgi.py in the gettingstarted folder.

Updating `.gitignore`

To begin with, we'll be updating the gitignore file. A gitignore file excludes files which you don't want to check into your repository. In order to deploy to Heroku, you don't technically need a gitignore file. You can deploy successfully without one, but it's highly recommended to always have one (and not just for Heroku). A gitignore can be essential for keeping out passwords and credentials keys, large binary files, local configurations, or anything else that you don't want to expose to the public.

Copy the following block of code and paste it into the gitignore file in the root of your project:

/venv
__pycache__
db.sqlite3 # not needed if you're using Postgres locally
gettingstarted/static/

The venv directory contains a virtual environment with the packages necessary for your local Python version. Similarly, the __pycache__ directory contains precompiled modules unique to your system. We don't want to check in our database (db.sqlite3), as we don't want to expose any local data. Last, the static files will be automatically generated for us during the build and deploy process to Heroku, so we'll exclude the gettingstarted/static/ directory.

Go ahead and run git status on your terminal to make sure that gitignore is the only file that's been modified. After that, call git add, then git commit -m "step 1 add git ignore".

Modularize your settings

Next up, we want to modularize our Django settings. To do that, add a new folder within gettingstarted called settings. Then, move the settings.py file into that directory. Since this naming scheme is a bit confusing, let's go ahead and rename that file to base.py. We'll call it that because it will serve as the base (or default) configuration that all the other configurations are going to pull from. If something like dev.py or local.py makes more sense to you, feel free to use that instead!

Local projects only have one environment to keep track of: your local machine. But once you want to deploy to different places, it's important to keep track of what settings go where. Nesting our settings files this way makes it easy for us to keep track of where those settings are, as well as take advantage of Heroku's continuous delivery tool pipelines.

By moving and renaming the settings file, our Django application now has two broken references. Let's fix them before we move on.

The first is in the wsgi.py in your gettingstarted folder. Open it up, and on line 12 you'll see that a default Django settings module is being set to gettingstarted.settings, a file which no longer exists:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "gettingstarted.settings")

To fix this, append the name of the file you just created in the settings subfolder. For example, since we called ours base.py, the line should now look like this:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "gettingstarted.settings.base")

After saving that, navigate up one directory to manage.py. On line 6, you'll see the same default being set for the Django settings module. Once again, append .base to the end of this line, then commit both of them to Git.

Continuous delivery pipelines

In an application's deployment lifecycle, there are typically four stages:

You build your app in the development stage on your local machine to make sure it works.
Next comes the review stage, where you check to see if your changes pass with the full test suite of your code base.
If that goes well, you merge your changes to staging. This is where you have conditions as close to public as possible, perhaps with some dummy data available, in order to more accurately predict how the change will impact your users.
Lastly, if all that goes well, you push to production, where the change is now live for your customers.

Continuous delivery (CD) workflows are designed to test your change in conditions progressively closer and closer to production and with more and more detail. Continuous delivery is a powerful workflow that can make all of the difference in your experience as a developer once you've productionized your application. Heroku can save you a lot of time here, as we've already built the tools for you to have a continuous delivery workflow. From your dashboard on Heroku, you can—with the mere click of a button!–set up a pipeline, add applications to staging and production, and deploy them.

If you connect your GitHub repository, pipelines can also automatically deploy and test new PRs opened on your repo. By providing the tooling and automating these processes, Heroku's continuous delivery workflow is powerful enough to help you keep up with your development cycle.

Adding new middleware to `base.py`

Modularizing your Django settings is a great way to take advantage of this continuous delivery workflow by splitting up your settings, whether you're deploying to Heroku or elsewhere, but there's one more change we have to make to base.py.

Django static assets work best when you also use the whitenoise package to manage your static assets. It's really easy to add to your project.

In your base.py file, scroll down to about line 43, and you should see an array of package names like this:

MIDDLEWARE = [
    "django.middleware.security.SecurityMiddleware",
    # Whitenoise goes here
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.middleware.common.CommonMiddleware",
    "django.middleware.csrf.CsrfViewMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "django.contrib.messages.middleware.MessageMiddleware",
    "django.middleware.clickjacking.XFrameOptionsMiddleware",
]

This is your list of Django middleware, which are sort of like plugins for your server. Django loads your middleware in the order that it's listed, so you always want your security middleware first, but it's important to add whitenoise as the second step in this base file.

Copy the following line of code and replace the line that says Whitenoise goes here with this:

"whitenoise.middleware.WhiteNoiseMiddleware",

We've loaded whitenoise as middleware, but to actually use the whitenoise compression, we need to set one more variable. Copy the following code and paste it right at the end of your base.py file:

STATICFILES_STORAGE = "whitenoise.storage.CompressedManifestStaticFilesStorage"

With that, we're done with base.py. Congratulations! Save your work and commit it to Git.

Setting up `heroku.py`

Our base settings are complete, but now we need our Heroku-specific settings. Create a new file under gettingstarted/settings called heroku.py and paste the following block of code:

"""
Production Settings for Heroku
"""

import environ

# If using in your own project, update the project namespace below
from gettingstarted.settings.base import *

env = environ.Env(
    # set casting, default value
    DEBUG=(bool, False)
)

# False if not in os.environ
DEBUG = env('DEBUG')

# Raises django's ImproperlyConfigured exception if SECRET_KEY not in os.environ
SECRET_KEY = env('SECRET_KEY')

ALLOWED_HOSTS = env.list('ALLOWED_HOSTS')

# Parse database connection url strings like psql://user:pass@127.0.0.1:8458/db
DATABASES = {
    # read os.environ['DATABASE_URL'] and raises ImproperlyConfigured exception if not found
    'default': env.db(),
}

You can see in this file the values that we're listing here are the ones that we're overriding from our base settings, so these are the settings that will be different and unique for Heroku.

To do this, we're using one of my favorite packages, Django-environ. This allows us to quickly and easily interface with the operating system environment without knowing much about it. It has built-in type conversions, and in particular it has automatic database parsing. This is all we need in order to parse our Heroku Postgres database URL that we will be given. It's just really convenient.

Heroku-specific files

That's all the work we need to do to get our application into 12 factored shape, but there are three more files we need in order to deploy to Heroku.

`requirements.txt`

In addition to the packages your project already uses, there are a few more you need to deploy to Heroku. If we take a look at the provided requirements.txt file, you can see these required packages here. We've already talked about Django, Django-environ, and whitenoise, and we've already configured those for use. But the other two are also important and needed for deployment.

The first one is called Gunicorn. This is the recommended WSGI server for Heroku. We'll take a look at configuring this in just a bit. The next one is psychopg2. This is a Postgres database adapter. You need it in your requirements.txt file to deploy, but you don't need any code changes in order to activate it.

A quick side note: we're keeping our discussion on packages simple for the purpose of this demo, but when you're ready to deploy a real project to Heroku, consider freezing your dependencies. You can do this with the pip freeze command. This will make your build a little bit more predictable by locking your exact dependency versions into your Git repo. If your dependencies aren't locked, you might find yourself deploying one version of Django one day and a new one the next.

`runtime.txt`

Heroku will install a default Python version if you don't specify one, but if you want to pick your Python version, you'll need a runtime.txt file. Create one in the root directory, next to your requirements.txt, manage.py, .gitignore and the rest. Specify your Python version with the prefix python-, followed by the major, minor, and patch version that you want your application to run on:

python-3.8.2

`Procfile`

The last file we need to add is a file specific to Heroku: the Procfile. This is what we use to specify the processes our application should run. The processes specified in this file will automatically boot on deploy to Heroku. Create a file named Procfile in the root level directory, right next to your requirements.txt and runtime.txt files. (Make sure to capitalize the P of Procfile otherwise Heroku might not recognize it!) Copy-paste the following lines into it:

release: python3 manage.py migrate
web: gunicorn gettingstarted.wsgi --preload --log-file -

The release phase of a Heroku deployment is the best place to run tasks, like migrations or updates. The command we will run during this phase is to simply run the migrate task defined in manage.py.

The other process is the web process, which is very important, if not outright essential, for any web application. This is where we pass our Gunicorn config, the same things we need when running the server locally. We pass it our WSGI file, which is located in the gettingstarted directory, and then we pass a few more flags to add it a bit more configuration. The --preload flag ensures that the app can receive requests just a little bit faster; the --logfile just specifies that the log file should get routed to Heroku.

Readying for deployment

Take a second before moving on and just double check that you've saved and committed all of your changes to Git. Remember, we need those changes in the Git repo in order for them to successfully deploy. After that, let's get ready to make an app!

Creating an app with `heroku create`

Since we have the Heroku CLI installed, we can call heroku create on the command line to have an app generated for us:

$ heroku create
Creating app... done, ⬢ mystic-wind-83
Created http://mystic-wind-83.herokuapp.com/ | git@heroku.com:mystic-wind-83.git

Your app will be assigned a random name—in this example, it's mystic-wind-83—as well as a publicly accessible URL.

Setting environment variables on Heroku

When we created our heroku.py settings file, we used Django-environ to load environment variables into our settings config. Those environment variables also need to be present in our Heroku environment, so let's set those now.

The Heroku CLI command we'll be using for this is heroku config:set. This will take in key-value pairs as arguments and set them in your Heroku runtime environment. First, let's configure our allowed hosts. Type the following line, and replace YOUR_UNIQUE_URL with the URL generated by heroku create:

$ heroku config:set ALLOWED_HOSTS=<YOUR_UNIQUE_URL>

Next, let's set our Django settings module. This is what determines what settings configuration we use on this platform. Instead of using the default of base, we want the Heroku-specific settings:

$ heroku config:set DJANGO_SETTINGS_MODULE=gettingstarted.settings.heroku

Lastly, we'll need to create a SECRET_KEY. For this demo, it doesn't matter what its value is. You can use a secure hash generator like md5, or a password manager's generator. Just be sure to keep this value secure, don't reuse it, and NEVER check it into source code! You can set it using the same CLI command:

$ heroku config:set SECRET_KEY=<gobbledygook>

Provisioning our database

Locally, Django is configured to use a SQLite database but we're productionizing. We need something a little bit more robust. Let's provision a Postgres database for production.

First, let's check if we have a database already. The heroku addons command will tell us if one exists:

$ heroku addons
No add-ons for app mystic-wind-83.

No add-ons exist for our app, which makes sense—we just created it! To add a Postgres database, we can use the addons:create command like this:

$ heroku addons:create heroku-postgresql:hobby-dev

Heroku offers several tiers of Postgres databases. hobby-dev is the free tier, so you can play around with this without paying a dime.

Going live

It is time. Your code is ready, your Heroku app is configured, you are ready to deploy. This is the easy part!

Just type out

$ git push heroku master

And we'll take care of the rest! You'll see your build logs scrolling through your terminal. This will show you what we're installing on your behalf and where you are in the build process. You'll also see the release phase as well that we specified earlier.

Scaling up

The last step is to scale up our web process. This creates new dynos, or, in other words, copies of your code on Heroku servers to handle more web traffic. You can do this using the following command:

$ heroku ps:scale web=1

To see your app online, enter heroku open on the terminal. This should pop open a web browser with the site you just built.

Debugging

If you hit some snags, don't worry, we have some tips that might help:

Are all of your changes saved and checked into Git?
Are your changes on the master branch or are they on a different branch? Make sure that whatever you're deploying, all of your changes are in that Git branch.
Did you deploy from the root directory of your project? Did you also call heroku create from the root directory of your project? If not, this could absolutely cause a trip up.
Did you remove anything from the code in the provided demo that we didn't discuss?

Logging

If you've run through this list and still have issues, take a look at your log files. In addition to your build logs—which will tell you whether your application successfully deployed or not—you have access to all logs produced by Heroku and by your application. You can get to these through a couple of different ways, but the quickest way is just to run the following command:

$ heroku logs --tail

Remote console

Another tool you have is the heroku run bash command. This provides you with direct access from your terminal to a Heroku dyno with your code deployed to it. If you type ls, you can see that this is your deployed application. It can be useful to check that what is up here matches what is locally on your machine. If not, you might see some issues.

Wrapping up

Congratulations on successfully deploying your productionized app onto Heroku!

To help you learn about Heroku, we also have a wealth of technical documentation. Our Dev Center is where you'll find most of our technical how-to and supported technologies information. If you're having a technical issue, chances are someone else has asked the same question and it's been answered on our help docs. Use these resources to solve your problems as well as to learn about best practices when deploying to Heroku.

DEV Community: Ed Morley