DEV Community: brunomichetti

How to manage large files with Heroku and Amazon S3 Buckets in Django Projects

brunomichetti — Tue, 16 Aug 2022 13:58:33 +0000

As a developer, I’ve recently worked on a Django API that processes large-sized images and videos. The system requires the admin to upload those mentioned files, and since the API is deployed in Heroku, we used to get the TIMEOUT error.

This happens because it takes some time to upload a file that is not light causing Heroku to crash. So, what can we do if we don’t want to reduce the size or the quality of the files?

In this blog post, I will provide a solution for this problem and will explain the flow and tools used. This information will hopefully help you with any working Django project that requires the management of large files.

The software architecture and flow

After discussing different approaches, my team and I decided to maintain the files in Amazon S3 buckets. These are containers where you can store objects (such as images and videos) and access them with fast performance.

As an example, if a Django model has an image attribute, let’s call it profile_picture, then we store the file in a bucket, and store the corresponding URL to that file in the database. So, if the Frontend requires the picture, the Backend returns the corresponding URL for that instance.

But, we don’t want everyone on the internet to have access to the image in the bucket, we only wish for the Frontend to do that. That’s why we have to configure the bucket as private.

Now, we need to answer: if the bucket is private, how does our Frontend access the files in it?. Well, in this case, we have to generate a pre-signed URL, that is a URL to grant temporary access to the file.

Here I will explain the proposed solution:

Store the system files in private buckets of Amazon S3.
Frontend and/or Django admin upload the files directly to a private Amazon S3 bucket.
The Backend stores the corresponding URL of each file.
When the Django admin creates/updates a file attribute for a given instance, the file is uploaded directly to the bucket, and the corresponding URL is stored in the database.

Explanatory interaction diagram:

When Frontend is about to upload a file:

It requests the Backend to get a valid upload URL to upload it to the private bucket.
The Backend generates and sends the upload URL.
The Frontend uses the upload URL to store the file in the private bucket.
After the upload, the Frontend sends the file URL to the Backend to store it for the corresponding file.

Explanatory interaction diagram:

When the Backend sends the URL of a file to the Frontend:

It uses a function to generate a pre-signed URL so the Frontend can access the file in the private bucket.

Explanatory interaction diagram:

The tools I used

Now that we have defined the solution flow, let’s talk about the tools. The first one I want to mention is django-s3direct, a library to directly upload files to the Amazon bucket from the admin panel. Also, it provides a model field that corresponds to the URL stored in the database.

On the other hand, we will use boto3 to generate the pre-signed URLs that the Backend sends to the Frontend in order to access the file. This library also generates the upload URL to allow the Frontend to upload files without the need of knowing the Amazon credentials.

In the following section, I’ll show the corresponding configurations.

Amazon S3 private bucket

I won't speak about the creation of buckets as there is plenty of documentation available. Next up, I’ll describe the configuration of the bucket to make it private and integrate it with django-s3direct.

Assuming you have already created the bucket and you have a user with access ID, access secret key, and permissions, these are the next steps:

Log in to Amazon S3 and create the bucket. Make sure you have the permissions.
Select the bucket, and go to the Permission tab.
In the bucket policy, paste the next policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ],
            "Resource": "arn:aws:s3:::<name-of-bucket>/*"
        }
    ]
}

Change by your bucket name.
Now in the same tab, block all the public access to make it private:

Finally, in the same tab, paste this CORS configuration (needed by django-s3direct):


[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD",
            "PUT",
            "POST",
            "DELETE"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [
            "ETag"
        ],
        "MaxAgeSeconds": 3000
    }
]

And that’s it, you have configured your private bucket. Keep in mind, if you want to change the policy in the future, first need to uncheck the blocking access.

Django-s3direct library

Now I will explain the use of the django-s3direct library to directly upload the files from the Django admin.

If you have a model that has a file attribute, when you change that attribute, this library will directly upload the file from the browser without sending it to the server. This is to avoid the named timeout error.

Let’s go step by step on how to get this configured:

Install django-s3direct in your project.
Add the library to your INSTALLED_APPS list in the settings:

# your settings file

INSTALLED_APPS = [ 
    …
    's3direct',
    …
]

Make sure you have the APP_DIRS configuration set as True in your TEMPLATES settings:

# your settings file

TEMPLATES = [
   {
        …
        'APP_DIRS': True,
        …
   }
]

Add django-s3direct urls to the urlpatterns list in your main urls.py file:

from django.urls import include, path

...

urlpatterns += [
    ...
    path('s3direct/', include('s3direct.urls')),
    ...
]

Add the configuration corresponding to Amazon to your settings (strongly recommended to put the access id and secret key values in environment variables):

# your settings file

# If these are set to None, the EC2 instance profile and IAM role are used.
AWS_ACCESS_KEY_ID = 'your-aws-access-key-id'
AWS_SECRET_ACCESS_KEY = 'your-aws-secret-access-key'
# Bucket name
AWS_STORAGE_BUCKET_NAME = 'your-aws-s3-bucket-name'
# The region of your bucket, more info:
# http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
AWS_S3_REGION_NAME = 'eu-west-1'
# The endpoint of your bucket, more info:
# http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
AWS_S3_ENDPOINT_URL = 'https://s3.eu-west-1.amazonaws.com'

Run collect static if needed:

python manage.py collectstatic

Now you can define in your model a file attribute corresponding to an image or video. Let’s take a look at an example:

from django.db import models
from s3direct.fields import S3DirectField

class Example(models.Model):
    image = S3DirectField(dest='example_destination')

And, what is example_destination? When you define a file attribute as an S3DirectField, you have to specify the dest parameter. There you have to put the string corresponding to the key in the S3DIRECT_DESTINATIONS dictionary in the configuration.
Let’s look at an example of the configuration to understand this better:

# your settings file

S3DIRECT_DESTINATIONS = {
   'example_destination': {
       'key': 'images/example-images/',
       'region': AWS_S3_REGION_NAME,
       'acl': 'private',
       'allow_existence_optimization': False,
   },
}

Inside each dictionary, you can configure a lot of parameters and check the library documentation to know them. - This example has the key where you define the folders where you want to store the corresponding file inside the bucket. You can test that it’s working by creating an instance of the class in the Django admin:

from django.contrib import admin

from example_app.models import Example


class ExampleAdmin(admin.ModelAdmin):
   fields = ('image',)


admin.site.register(Example, ExampleAdmin)

Now you should go to django admin and you will see something like this:

Then if you select an image, the library will directly upload the image to the private bucket and store the corresponding URL in the database. The Backend won’t receive a file, just a string URL. After the correct upload, you will see something like:

And the stored URL will have this structure: https://s3.<region-name>.amazonaws.com/<bucket-name>/<key>/<file-name>
If you click on the name of the picture, the browser will try to open it, but you won’t be able to see it because the bucket is private, and that’s ok. Also, if you click on REMOVE, the file will be removed from the instance but it will continue existing in the bucket.

Customize the name of the pictures

As highlighted, in the S3DIRECT_DESTINATIONS dictionary, you can configure in the key attribute the folders route to store the files corresponding to the given destination.

But what happens when you upload a file with the same name as another existing one in the same folder?

Well, that will overwrite the file, and I assume you don’t want that because it can be the file of another instance in your system.

Here are two tips for this:

Add a timestamp to the file name to avoid repeated file names in the system.
Use a slugify function to manage spaces and/or invalid characters in the file name.

If you do that, you can avoid a lot of future problems. Let’s look at an example:

# your settings file
import os

from django.utils import timezone
from django.utils.text import slugify


… 


EXAMPLE_DEST_KEY = 'images/example-images/'
TIMESTAMP_FORMAT = '%m-%d-%Y_%H-%M-%S-%f'

def normalize_filename(filename):
   name, extension = os.path.splitext(filename)
   time_stamp = timezone.now().strftime(TIMESTAMP_FORMAT)
   return f'{time_stamp}--{slugify(name)}{extension}'


S3DIRECT_DESTINATIONS = {
  'example_destination': {
      'key': lambda filename: f'{EXAMPLE_DEST_KEY}/{normalize_filename(filename)}',
      'region': AWS_S3_REGION_NAME,
      'acl': 'private',
      'allow_existence_optimization': False,
  },
}

In this example, for each new file, we apply a normalization of the file name. First, we normalize the filename using the slugify function from Django, and then we append at the beginning the timestamp.

If we upload a file with the name image.png, the system will store that file with the form <timestamp>--image.png.

Another example is if we have a file with the name FILE with SPACEs.png, the system will store <timestamp>--file-with-spaces.png. This is super easy to understand and solves a lot of problems.

Now, in this next section, we will see how to return presigned files to the Frontend.

It’s important to note, that this mentioned normalization will be executed only when uploading files in the Django admin. To be consistent, it would be nice to make the same normalization in the Frontend when it has to upload a file.

Boto3 and pre-signed URL

We know how to configure a private Amazon S3 bucket, and how to integrate it with django-s3direct to directly upload a file from the Django admin.

Now, we need to know how to do the following:

Send a pre-signed URL from the Backend to the Frontend so this last can access the existing file in the private bucket.
Send a pre-signed upload URL from the Backend to the Frontend so this last can upload a file.

Let’s now look at how we do this using the boto3 library.

Generate and send a view pre-signed URL

Since the bucket is private, if we take a file URL in the database and we try to access it, we won’t be able to. We would then see something like this:

This is why we need to generate a pre-signed URL in the Backend for the corresponding file to be temporarily available for the Frontend. For this I use the boto3 library, let’s look at the used functions:

# some utils file for boto3 functions

from boto3 import Session
from typing import Any, Dict

from django.conf import settings

def get_object_key_from_url(url: str) -> str:
   '''
   Returns the object key for the given url
   '''
   endpoint = settings.AWS_S3_ENDPOINT_URL
   bucket_name = settings.AWS_STORAGE_BUCKET_NAME

   return url.split(f'{endpoint}/{bucket_name}/')[1]


def get_presigned_url(object_key: str) -> str:
   '''
   Returns a presigned URL for the given object key
   '''
   session = Session(
       aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
       aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
       region_name=settings.AWS_S3_REGION_NAME,
   )
   s3client = session.client('s3')
   url = s3client.generate_presigned_url(
       ClientMethod='get_object',
       Params={
           'Bucket': settings.AWS_STORAGE_BUCKET_NAME,
           'Key': object_key,
       },
       ExpiresIn=settings.AWS_PRESIGNED_URL_EXPIRATION_TIME_MINUTES,
   )
   return url

In this example, the function get_object_key_from_url obtains the object key from the file URL. As mentioned, the object key is the route of folders and the file name in the bucket.

Why do we need the object key? We need it to generate the presigned URL of the file. That object key will be used in the get_presigned_url function for that purpose.

The next thing to do is to execute that function when the Frontend makes a GET request and needs to access the file.

There are several ways of doing that, and I’m going to show you one that I find easy to understand:

Create a property in the model, that corresponds to the pre-signed attribute:

# your models file

from django.db import models
from s3direct.fields import S3DirectField

from example_app.s3_files import get_object_key_from_url, get_presigned_url

class Example(models.Model):
   image = S3DirectField(dest='example_destination')

   @property
   def presigned_image(self):
       return get_presigned_url(get_object_key_from_url(self.image))

The presigned_image property first extracts the object key for the stored URL and then generates the pre-signed one to send it to the Frontend.
Now add the property to a model serializer:

from rest_framework import serializers

from example_app.models import Example


class ExampleSerializer(serializers.ModelSerializer):

   class Meta:
       model = Example
       fields = ('presigned_image',)

This way, if you use that serializer to send the data to the Frontend, it will return a pre-signed URL in the presigned_image field.

A pre-signed URL looks like this:

https://<bucket name>.s3.amazonaws.com/images/<object key>?<lots of needed parameters>

With that URL, the file will be available for the time period defined in your settings as in the example.

Generate an upload pre-signed URL

Last but not least, I’m going to explain how to send an upload URL from the Backend to the Frontend. Imagine we have an app where the users have profile pictures and the Frontend needs to upload the image.

The flow would then look like this:

The Frontend generates the object key of the image file (recommend adding a timestamp and normalizing the file name as in the example in the previous section).
The Frontend sends the object key in a POST request to the Backend.
The Backend takes the object key and generates a pre-signed upload URL.
The Frontend uploads the image using the pre-signed upload URL.
The Frontend sends the image URL to the backend so it can be stored in the database.

Let’s look at an example of this function:

def get_presigned_url_dict_to_upload_file(object_key: str) -> Dict[str, Any]:
   '''
   Returns a dict with the necessary data to upload a file for the given key
   '''
   session = Session(
       aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
       aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
       region_name=settings.AWS_S3_REGION_NAME,
   )
   s3client = session.client('s3')

   url = s3client.generate_presigned_post(
       Bucket=settings.AWS_STORAGE_BUCKET_NAME,
       Key=object_key,
       Fields={
           'acl': 'private',
       },
       Conditions=[
           {'acl': 'private'},
       ],
       ExpiresIn=settings.AWS_PRESIGNED_URL_EXPIRATION_TIME_MINUTES,
   )

   return url

Finally, let’s look at an example in a viewset for the action POST to receive an object key and return the necessary information to upload the file.

First, we must define the serializer like so:

from rest_framework import serializers

from example_app.models import Example


class ExampleSerializer(serializers.ModelSerializer):

   class Meta:
       model = Example
       fields = ('presigned_image',)


class ExampleUploadURLSerializer(serializers.Serializer):
   object_key = serializers.CharField()
   upload_url_dict = serializers.DictField(required=False)

Second, we must define the viewset with the corresponding action:

from rest_framework import viewsets, status
from rest_framework.decorators import action
from rest_framework.response import Response
from rest_framework.request import Request

from example_app.models import Example
from example_app.serializers import ExampleSerializer, ExampleUploadURLSerializer
from example_app.s3_files import get_presigned_url_dict_to_upload_file


class ExampleViewSet(viewsets.ModelViewSet):
   serializer_class = ExampleSerializer
   queryset = Example.objects.all()

   @action(detail=False, methods=('post',), url_path='upload-url')
   def upload_url(self, request: Request) -> Response:
       serializer = ExampleUploadURLSerializer(data=request.data)
       serializer.is_valid(raise_exception=True)

       object_key = serializer.validated_data['object_key']

       upload_url_dict = get_presigned_url_dict_to_upload_file(object_key)

       return_serializer = ExampleUploadURLSerializer(
           data={'object_key': object_key, 'upload_url_dict': upload_url_dict,}
       )
       return_serializer.is_valid(raise_exception=True)

       return Response(return_serializer.validated_data, status=status.HTTP_200_OK)

So, after this, we are sending the necessary information to upload the file. And that’s it! We have finished the proposed solution.

I have created a public repo where you can see this little example project to get further insight. If you’d like, you can download it and give it a try for yourself.

Summary

Throughout this blog, I presented a solution for a common problem for developers - managing big files in Django projects that are deployed using Heroku.

I detailed in-depth the defined flow and tools I used, as well as providing some useful tips and examples to help you create a good solution.

However, this doesn’t mean that the workaround we found is the only possible one. But it could very well be the most efficient.

I hope you enjoyed reading and if you have another process that you think is useful, feel free to let us know in the comments section. Thanks for reading!

Testing in Django & Django REST – Useful Tools & Best Practices

brunomichetti — Mon, 06 Jun 2022 19:44:10 +0000

If you are a developer, you already know how important testing is in any software project. In particular, automatic testing, as it can help you to corroborate your coding, and quickly see what your program does what you want it to do. It also helps to corroborate any new changes in your code that didn’t break any previous functionalities.

In saying that, does this mean that if you have automatic tests then your project won’t have any errors? It does not. As computer scientist Edsger W. Dijkstra once said:

“Program testing can be used to show the presence of bugs, but never to show their absence”.

This phrase helps us to understand that we can’t be sure that a program is perfect, but, testing is fundamental to help us discover errors and make fixes and improvements. As a person that works in software development, I can say that it’s a lot better when you discover an error rather than your client or a user in production. In this article, I’ll talk about basic knowledge in automatic testing, useful tools, and good practices in Django projects, with a focus on API’s.

Testing in Django & Django REST Framework

Django has very nice documentation about testing, and Django REST Framework too. So, in this blog, I’ll talk about the main tools in those two frameworks, and what you can do and use to improve your testing.

Main Testing Classes

Django provides several classes for testing. The one I’ll talk about here is TestCase, which is very useful if your application uses databases.
As the name states, to create a test case in your Django project, you will define a class that inherits from TestCase. By doing this, you can use all the methods and properties of the named class that will help you to create and execute your tests. Then you will define different functions that will correspond to each unit test inside the test case. Let’s see an example of the structure of a test case:

# your test file
from django.test import TestCase


# class to define a test case for login
class UserLoginTestCase(TestCase):

    ...
    # some setup here, explained later

    def test_correct_login(self):
        # unit test
        # Corroborate the expected scenario
        ...

    def test_if_password_incorrect_then_cant_login(self):
        # unit test
        # Corroborate that user's password needs to be only the correct one
        ...

    def test_if_user_not_registered_cant_login(self):
        # unit test
        # Corroborate that user's are able to login only if they're registered

    ...

Also, you can use the Client class of Django which simulates a dummy browser so you can make HTTP requests and test how your Django API responds.

Since we are focusing on API testing, I want to talk about APITestCase, a class of Django REST framework that is a mirror of TestCase but uses a different client class: APIClient. This client extends the TestClient, meaning that it has the same functionalities and adds others such as the credentials function. This function is very useful to overwrite authentication headers, for example, using OAuth1, OAuth2, or any simple token authentication scheme.
So, if you are working on a project with the Django REST framework, you can change the previous example in this way:

# your test file
from rest_framework.test import APITestCase


# class to define a test case of login
class UserLoginTestCase(APITestCase):

    ...

Main Testing Functions

The aforementioned classes each have a set of methods that will help you through the testing. With these functions, you are able to corroborate that your software works as expected and make necessary test fixtures.
A test fixture is an environment that you can create to run your tests using consistency. Using fixtures you can make sure that certain conditions are met before the execution. For example, to have a determined set of data in your testing database, or to create needed objects. Here is a list of the main ones:

setUp and tearDown: These are functions to be executed before (setUp) and after (tearDown) in each unit test. These are very useful for fixtures.
setUpClass and tearDownClass: In an analogous way of setUp and tearDown, these functions are executed before (setUpClass) and after (tearDownClass) during the whole test case. This means that it is executed only once for a test case. Since they’re used by Django to make important configurations, if you overwrite these methods in your test case class, don’t forget to call the super implementation inside the functions.
setUpTestData: this function can be used to have a class-level atomic block to define data for the whole test case. That means that this function automatically rollbacks the changes in the database after the finalization of all the unit tests in the test case. This is mainly used to load data to your test database.
Assert methods: Django has the set of assert methods from unittest, and has another ones created for the framework. These methods, such as assertEqual, assertIsNone, assertTrue, etc, can be used in the unit test to check the different conditions that have to be met. For example, in the unit test of login, you can assert that the response has a status code of 200 because you want to make sure that if the data is sent correctly, then your application has to return the response with that code.

You can avoid using these functions, or use any combination of them. They’re not needed, but they’re very useful and will help you a lot.

Test Discovery & Databases

Where do you have to put your test classes? By default, Django recognizes any file that fulfills the pattern test*.py under the current working directory. That is, any file under the directory that starts with a test, and of course, has the .py extension. Inside it, Django will execute any function in the test case class starting with test.
When you run your tests, you can pass a parameter indicating the desired pattern just in case you want to use a different one. With that behavior, the place to put the test cases can be anywhere, but I recommend dividing the tests into the several Django apps that you may have in your project. For example, if you have a Django app called users that has the models and functionalities related to the users in your project, you can define in there a test folder as a module (adding the init.py file) with all the tests that cover that important part: tests for login, for sign-up, for user data, account deletion, etc. Something similar to this:

my_project
   |__django_apps
       |__users
          |__test
             |__ __init__.py
             |__ test_login.py
             |__ test_signup.py
             |__ test_delete_user.py
             ...
        ...
    ...
...

Regarding the databases, when you run your tests by default, Django creates one and then destroys it after the finalization. This is to avoid conflicts between your production and/or development databases, as well as your testing database. You can define and customize a database for tests, inside the DATABASES dictionary, adding another one named TEST. Something like:

DATABASES = {
    'default': {
        'ENGINE': '<engine>',
        'NAME': '<database name>',
        'USER': '<database user>',
        ...
    },
    ...
    'TEST': {
        # testing database customization here
            'NAME': '<your testing database name>',
            # some other customization, for example the user user, password, etc
        },
}

Take a look at the available keys for the TEST dictionary. If you don’t define it, Django will create a database naming it equal to your database in the default settings, appending the _test suffix.

Useful Tools

In this section, I will talk about two external tools that are very helpful for fixtures: Faker and Factory Boy.

Faker

Faker is a python package used to generate fake but realistic data. You can use its functionalities to load data into your testing database, generate data for the requests that you want to test, generate data for the models, etc. Faker has a huge and diverse set of possibilities. You can generate execution time, for example, first names, last names, phone numbers, dates, passwords, emails, etc.

Additionally, you can pass parameters to the functions to generate data with different constraints, for example, you can obtain a password specifying if you want to use special characters or not, the desired length, and if you want to have an upper case or not, etc. Take a look at the Faker providers to see all the different fake but realistic data that you can generate for your tests. Instead of using for example user_test@mail.com for your test cases, you can have data that is closer to reality and improve the quality.

Factory Boy

If you use factories for tests, you define in a class the data structure that you want to have in your testing database. Then in your tests, it will be easier to have instances of your models, and loaded data with the desired structure. Factory Boy is a tool to replace static, hard-to-maintain fixtures using factories. Also, you can combine it with Faker to have factories with fake but realistic data. Let’s see an example:

In this case, we can see a static fixture of Django, to load data to the testing database in a model called Person, which has a first name and last name. This JSON file defines two instances:

[
  {
    "model": "myapp.person",
    "pk": 1,
    "fields": {
      "first_name": "John",
      "last_name": "Lennon"
    }
  },
  {
    "model": "myapp.person",
    "pk": 2,
    "fields": {
      "first_name": "Paul",
      "last_name": "McCartney"
    }
  }
]

Now let’s see the analogous definition using Factory Boy and Faker:

from factory import django, Faker
from myapp import models 

class PersonFactory(factory.django.DjangoModelFactory):

    class Meta:
        model = models.Person

    first_name = Faker('first_name')
    last_name = Faker('last_name')

With this, you can use functions that Factory Boy provides, for example, PersonFactory.create_batch(2) to create instantly 2 instances in your testing database of Persons with different and realistic first and last names. In the JSON case, if you want to have for example 10 instances, you will need to add them and define new first and last names explicitly. And if you want to have huge datasets, the JSON file will be enormous and hard to maintain, while in the factory case you only need to call a function. Also, what if the model Person changes? In the JSON case, you have to update one by one each defined instance. In the factory case, you only need to change a class definition. Factory Boy documentation has a section called common recipes, where you can find useful tools, practices, and tips from the library. Here is a list of interesting functionalities:

You can create factories of models including the model relationships using the RelatedFactory or SubFactory class.
Use the create function to create a new instance of the model in the factory and save it into the testing database. If you want to have instances but without saving them into the database, use build instead.
In an analogous way, you can create and save into the database N instances of the model in the factory using create_batch(N), and build_batch(N) if you don’t want to save them.
You can define an attribute in a factory that selects a choice of a set of choices just like a Django model choice field, using random_element of Faker. In the next section, we will see an example of this.

Test Example & Good Practices

In this section, I want to wrap up and show a little example. This is the tiny reality: In the project, we have a Django app called users where we have the user model that has a username, phone_number, and a category in the system, that could be admin, common user, and guest. The user can sign-up and login into the application.

Model & Factory Definition

# model.py in the Django app called users
from django.db import models
from django.contrib.auth.models import AbstractUser


# class to define choices
class Category(models.TextChoices):
    GUEST = 'G', 'Guest user'
    COMMON_USER = 'C', 'Common user'
    ADMIN = 'A', 'Admin user'


# user model that inherits from AbstractUser
# there is no need to define a username field
# because is defined in the parent class
class User(AbstractUser):
    phone_number = models.CharField(max_length=50, blank=True, null=True)
    category = models.CharField(
        max_length=1,
        choices=Category.choices,
        default='G',
    )

Now, let’s see the implementation of the user factory. This file is inside a test folder in the users Django app:

# factory.py inside users/test
from faker import Faker as FakerClass
from typing import Any, Sequence
from factory import django, Faker, post_generation

from users.models import User, Category


CATEGORIES_VALUES = [x[0] for x in Category.choices]


class UserFactory(django.DjangoModelFactory):

    class Meta:
        model = User

    username = Faker('user_name')
    phone_number = Faker('phone_number')
    category = Faker('random_element', elements=CATEGORIES_VALUES)

    @post_generation
    def password(self, create: bool, extracted: Sequence[Any], **kwargs):
        password = (
            extracted
            if extracted
            else FakerClass().password(
                length=30,
                special_chars=True,
                digits=True,
                upper_case=True,
                lower_case=True,
            )
        )
        self.set_password(password)

A couple of things about this:

In the Meta class inside the factory, we have to indicate the corresponding model.
In the category attribute, we use random_element of Faker to randomly select a choice of the Category choice list. We’ve defined CATEGORIES_VALUES to specify which part of the choice we want to take. That is because each element in the choices list is a tuple, where the first part is the value, and the second one the explanatory text. So, that CATEGORIES_VALUES set has only that first part for each element.
Through the post_generator decorator and the password function, we indicate that at the moment of the creation of a user instance, we can pass a desired password to be used, or else we generate one using the Faker functionality.

Django Test Case

Now, let’s see an example of testing for the user sign-up functionality. The implementation of the authentication part for this example project has been made using dj-rest-auth. I won’t talk about that, but if you want to know more, please take a look at this blog. This is a test_singup.py file inside the test folder of the user Django app:

from django.urls import reverse
from rest_framework import status
from rest_framework.test import APITestCase, APIClient

from users.test.factory import UserFactory
from users.models import User, Category
from faker import Faker


class UserSignUpTestCase(APITestCase):

    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        cls.user_object = UserFactory.build()
        cls.user_saved = UserFactory.create()
        cls.client = APIClient()
        cls.signup_url = reverse('rest_register')
        cls.faker_obj = Faker()

    def test_if_data_is_correct_then_signup(self):
        # Prepare data
        signup_dict = {
            'username': self.user_object.username,
            'password1': 'test_Pass',
            'password2': 'test_Pass',
            'phone_number': self.user_object.phone_number,
            'category': self.user_object.category,
        }
        # Make request
        response = self.client.post(self.signup_url, signup_dict)
        # Check status response
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        self.assertEqual(User.objects.count(), 2)
        # Check database
        new_user = User.objects.get(username=self.user_object.username)
        self.assertEqual(
            new_user.category,
            self.user_object.category,
        )
        self.assertEqual(
            new_user.phone_number,
            self.user_object.phone_number,
        )

    def test_if_username_already_exists_dont_signup(self):
        # Prepare data with already saved user
        signup_dict = {
            'username': self.user_saved.username,
            'password1': 'test_Pass',
            'password2': 'test_Pass',
            'phone_number': self.user_saved.phone_number,
            'category': self.user_saved.category,
        }
        # Make request
        response = self.client.post(self.signup_url, signup_dict)
        # Check status response
        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
        self.assertEqual(
            str(response.data['username'][0]),
            'A user with that username already exists.',
        )
        # Check database
        # Check that there is only one user with the saved username
        username_query = User.objects.filter(username=self.user_saved.username)
        self.assertEqual(username_query.count(), 1)

Well, in this file we have a test case, defined as UserSignUpTestCase, and two unit tests inside, one to test the expected case, and the other to test an error case. I want to list a couple of main points to discuss good practices:

Take a look at the unit tests names: You can name them as you want (always starting with test unless you change the pattern). A good practice is to make a name that describes it. This will help you when testing and debugging, for example, if a test fails and you want to trace the error.
In the setUpClass method, we call the super() function, remembering that it’s important for Django. Also, we set up the needed instances, such as the API client, a user, and a faker instance. Besides we load a user to the database with the create function.
Notice that the sign_up URL defined in the setUpClass method, uses reverse from Django. You can put directly the URL string if you want. By using reverse, you obtain an URL entering the related name. In the dj-rest-auth package, the related name is rest_register. I recommend you use reverse in your project to improve maintainability. When you define URLs, give them a name, and then in the tests use reverse. After that, if a URL changes, you won’t need to change anything in the tests, because reverse solves this.
The general structure chosen for the unit test is:
- Prepare data: You can use the instance generated with build to obtain the data for the request.
- Make the request: Use the APIClient to make the request.
- Check the response: Check the response data (if there is data) and status. When working with Django REST framework, I recommend using status. With this, you have the HTTP status codes in a cleaner way. If you don’t want to, you can just put the number directly.
- Check the database: If the request generates changes in the database, please corroborate it to make sure that the functionality modifies it correctly. If you have the case that doesn’t modify the database, it’s a good practice to test that the database remains without changes (just as in the second unit test).

This was an example of a Test Case with two unit tests. You can add more unit tests, for example, to test if some data is incorrect in the request or the user can’t sign-up. Besides you can add in that file (or another file) more test case classes, for example, to test the login functionality of the app.

Running Your Tests

Once you have coded a couple of tests, you can execute them with this command in the main folder of the project (the one that has the manage.py file):

python manage.py test

This will search for all of your tests files under that folder and then execute them, showing the error or success of each unit test.

Customize the Execution of the Tests

You can customize the test execution, for example, you can indicate to test:

Only a Django app, specifying <Django app name>:

python manage.py test users

Only a file in a given Django app, specifying <Django app name>.path.to.file:

python manage.py test users.test.test_signup

Only a test case class, specifying: <Django app name>.path.to.file.ClassName:

python manage.py test users.test.test_signup.UserSignUpTestCase

Only a unit test function, specifying <Django app name>.path.to.file.ClassName.function_name:

python manage.py test users.test.test_signup.UserSignUpTestCase.test_if_data_is_correct_then_signup

You can also add some flags to the command to customize the execution, for example:

--keepdb to avoid erasing the database between executions. This can improve the speed.
--parallel to run the tests in parallel improving also the speed on multi-core hardware. If you do this make sure they’re well isolated.

Summary

Throughout this blog, I’ve presented some basics of testing with Django and Django REST framework. Also, I showcased some useful tools and good practices that are more focused on API’s. I highlighted some examples, and how to run and customize your tests with the named framework. I hope you enjoyed reading and I hope that this blog can help you to test and improve your own Django API projects.

Read this article and more content in the Rootstrap blog: https://www.rootstrap.com/blog

How to Manage your Python projects with Pipenv & Pyenv

brunomichetti — Mon, 06 Jun 2022 16:17:09 +0000

A Python virtual environment is an important tool for developers when separating project dependencies in isolated environments. If working locally on several Python projects, developers may need to use different Python versions on each.

By doing this, they will be able to install different packages and have different versions of the named packages for each environment.

However, this can generate a lot of compatibility issues unless you manage them correctly using virtual environments. There are many ways to do this and I will highlight two tools to do so: Pyenv and Pipenv.

Python Virtual Environments

In the following image, you can see an abstract example of different Python projects containing the different combinations of packages that you can install:

Each big colored box represents a Python virtual environment. It’s common for certain package versions to be supported only by determined Python versions. Also, you may need a determined version of a package in one project and another version for others. As well as this, a determined version of a package can generate conflicts with packages or Python versions.

Many combinations of compatibility issues can be found but by having those isolated environments you can avoid them. You will be able to install, uninstall, update, etc in one virtual environment, without affecting the rest of them. As mentioned, there are many tools to do this, and I will talk about the ones I find easy to understand and use.

Pyenv

Pyenv is a tool used to manage different Python versions. With this tool you can:

Install several Python versions.
Set/change the global(default) Python version in your computer.
Set/change a Python version locally for a project.
Create and manage virtual environments.

Installation

On macOS, you can install this tool using Homebrew:

brew install pyenv

To install it on other platforms, you can watch this section in the GitHub docs. After the installation, edit your used shell startup script (.bashrc, .zshrc, .bash_profile, …) adding the following line:

if command -v pyenv 1>/dev/null 2>&1; then
  eval "$(pyenv init -)"
fi

Then reload your shell startup script. An example if you have .bashrc:

cd
source .bashrc

That’s it! You have installed Pyenv successfully.

Managing Python Versions

Now let’s take a look at the possible Python versions that you can install through Pyenv, executing:

pyenv install --list

You will see a large list of Python versions. You can pick one to install it, let’s say 3.9.1:

pyenv install 3.9.1

You can run that command with any version that you want to have installed on your computer. To see all the installed versions, just run:

pyenv versions

Also, you can set the global version that you want to use in your system, this will be the default one. For example, if you have already installed 3.8.8, you can set it as the global version running:

pyenv global 3.8.8

You can change that, of course, by executing the command again with the desired installed version. If you want to check your global version, just run:

python -V

With Pyenv, you can have a global and a local version for each project. To set a local version, go to the project folder in the terminal:

cd ~/path/to/the/project/folder

And set the local version, for example, 3.9.1:

pyenv local 3.9.1

This will create a .python-version file in the folder indicating the current local Python version for the project. Also, if you run python -V in that folder, you will see the local version, and not the global one.

So, to sum up, you have the following useful commands at your disposal:

pyenv install --list to see the available Python versions you can install.
pyenv versions to see the installed Python versions.
pyenv global <Python version> to set an installed Python version as global.
pyenv local <Python version> to set an installed Python version for a given project folder.
pyenv uninstall <Python version> to uninstall an already installed Python version.
Pyenv also allows us to manage environments with a plugin called pyenv-virtualenv. However, I prefer to use Pipenv. Let’s take a look.

Pipenv

Pipenv is a tool used to manage the required packages in a Python project. With this tool you can:

Automatically create or delete a virtual environment.
Install, uninstall and update packages maintaining the desired versions.
Set up very quickly a working Python environment. Have a list of the installed packages and the corresponding version.
If you use Pipenv in your projects, you will have two special files in the root folder, both generated and changed automatically. Those are:
Pipfile: A file that specifies the installed packages in your virtual environment for development and execution.
Pipfile.lock: A file that specifies which versions of the installed packages (detailed in the Pipfile) has to be used.
Also has another important metadata.

With Pipenv, is very easy to work on projects with other developers using the same packages and versions as it provides the files containing all that information. Developers must simply run a command to have the same environment in their own computer.

Installation

On macOS, you can install this tool using Homebrew:

brew install pipenv

And that’s it. You’re ready to start! To install it on other platforms, check out this section in the pipenv docs.

Creating Virtual Environments

Now, let’s look at how to create a virtual environment for a Python project. First, go to the project folder:

cd ~/path/to/the/project/folder

And then execute:

pipenv install

This will create a virtual environment and:

If you already have a Pipfile and a Pipfile.lock: it will also install all the specified packages on them.
If you don’t have a Pipfile and a Pipfile.lock: it will generate them for that environment. This is how a Pipfile recently created looks like:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.7"

Under the [packages] section, you will see the installed packages for execution – the ones that your Python program will need to be executed correctly.

Under the [dev-packages] section, you will see the packages that are needed only for development. Also, you can see in that example that the used Python version is 3.7.

If you use Pyenv, the pipenv install command will use the global Python version of your system. But as you know, you can specify a local Python version in your project. For example:

pyenv local 3.8.8

And, if you want to use that local Python version, you can indicate that to Pipenv by executing:

pipenv install --python 3.8.

Now, what if you want to delete the created virtual environment to create another one? You can do so by running:

pipenv --rm

Another important thing to remember is to activate the created virtual environment, which you do by running:

pipenv shell

With this command, you “have entered” your virtual environment. In your terminal, you can see if you have already activated the virtual environment, if you see the name between brackets at the beginning of the line, as follows:

There you can see that after the pipenv shell command has been executed, the name of the virtual environment appeared between brackets at the beginning of the line. To deactivate the virtual environment, just run:

exit

Managing Packages

Ok, so you now know how to create, delete, activate and deactivate a virtual environment. Now let’s see how to manage the packages inside them. First, don’t forget to activate the virtual environment. To install a package, for example django-drip-campaings, you can execute:

pipenv install django-drip-campaigns

This command will:

Install the latest available version of that package.
Automatically update the Pipfile file adding the package.
Automatically update the Pipfile.lock file adding the package version and some other important metadata. Now the Pipfile file looks like this:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
django-drip-campaigns = "*"

[dev-packages]

[requires]
python_version = "3.8"

As you can see, the package was installed under the [packages] section. As previously mentioned, in the [dev-packages] section you only install what’s needed for development.

For example, you want to have flake8 in your project, which is only required by the development process. To install it under the [dev-packages] section, execute:

pipenv install flake8 --dev

After that, you will see something like this in the Pipfile file:

[packages]
django-drip-campaigns = "*"

[dev-packages]
flake8 = "*"

Also, what if you need a specific version of a package? For example – Django, not the latest but the 2.2 version. Then run:

pipenv install django==2.2

Now the Pipfile is updated and will resemble something like:

[packages]
django-drip-campaigns = "*"
django = "==2.2"

[dev-packages]
flake8 = "*"

In the Django package line, you don’t see the “*”, you instead see the specific version indicated in the command. If you need to uninstall a package, for example django-drip-campaigns, just run:

pipenv uninstall django-drip-campaigns

That command will uninstall the package and remove the information from the Pipfile and Pipfile.lock files. Now suppose you need the latest version of Django in your project (not the 2.2 version). To update a package to the latest version, you have to:

Change the django = "==2.2" line in the Pipfile by django = "*".

Then run:

pipenv update django

Alternatively, you can uninstall Django and then install it again without specifying the version. You can repeat those steps to downgrade a package version instead of upgrading it.

You can do a lot of things with Pipenv, but the previously mentioned throughout are the main features. Pretty easy, right? Now let’s see a summary of the mentioned commands provided by Pipenv:

pipenv install to create a virtual environment.
pipenv install --python <Python version> to create a virtual environment indicating the desired Python version (that you have installed using Pyenv).
pipenv --rm to delete the current virtual environment.
pipenv shell to activate the created virtual environment.
exit to deactivate an already activated virtual environment.

Now let’s take a look at the summary of the commands after the virtual environment has been activated:

pipenv install <package name> to install the latest version of the package under the [packages] section.
pipenv install <package name>==<package version> to install a specified version of a package, under the [packages] section.
pipenv update <package name> to update a version package (update or downgrade) to the one that you have previously specified in the Pipfile.
Adding --dev to the previous commands, Pipenv will do the same but under the [dev-packages] section.
pipenv uninstall <package name> to uninstall a package.

Summary

As highlighted throughout, I have presented two different tools for managing Python projects using virtual environments. There are many ways to do this, but I showcased the two I believe to be the most effective.

As tools, Pyenv and Pipenv are tidy and maintainable as well as easy to understand and use. Pyenv is used to manage different Python versions, whereas Pipenv is used to manage Python packages.

I really like using both and I hope you are encouraged to try these tools for Python development. Thank you for reading and stay tuned for more useful content.

Read this article and more content in the Rootstrap blog: https://www.rootstrap.com/blog

Registration and Authentication in Django apps with dj-rest-auth

brunomichetti — Mon, 15 Mar 2021 21:10:53 +0000

A huge amount of existing applications have registration and authentication for users. Maybe every developer in the world has implemented something related to this in their work or while they learned. After the creation of the Django REST framework, Django developers started to implement more and more app-level REST API endpoints.

As a result, an open-source package called dj-rest-auth has been created to provide a set of REST API endpoints to handle user registration and authentication tasks. In this article I will not only talk about this particular package but will also give you some tips to avoid some of the most common problems faced when configuring it.

The dj-rest-auth package

If you are a Django developer and you are coding a REST API with authentication, you will find the dj-rest-auth package very useful. This project is a fork from django-rest-auth that is no longer maintained.

As said, dj-rest-auth provides a set of REST API endpoints to manage user registration and authentication. After an easy installation and configuration, you will have endpoints for:

User registration with optional activation.
Login and logout.
Retrieve and update the Django User model.
Password change.
Password reset via e-mail.
Social media authentication.

You don't need to do too much work to have those functionalities in your app. Besides, given that dj-rest-auth is an open source package, it has the advantage of being used by lots of programmers that can find errors, propose and add improvements, and so on. So, if you use dj-rest-auth you can be sure that you are using code created and maintained by an open source community.

Browsable endpoints

Thanks to the Django REST framework, most of the endpoints that dj-rest-auth provides are browsables. This means that if you have added the package to your Django app, you can test those endpoints through the browser. As an example, if you open the browser and put the URL of the login endpoint, you will see something like this:

In the image above you can enter a username, email, and password to test the login functionality. The message of Method GET not allowed is not an error. It means that your endpoint doesn't allow the GET method but you can test it doing a POST. That method is possible because the corresponding blue button is shown.

As you can see, it's very useful to very quickly have a nice interface to use while you are coding.

Configuration

The documented installation and minimal configuration of dj-rest-auth is pretty clear and I won't talk about that in this blog post. The main idea of this section is to tell you about the big set of possibilities that you can have in your Django app just by adjusting a couple of parameters. This package uses django-allauth behind so you can take advantage of all the available settings in your own app. For example, you can define these parameters in your app configuration:

ACCOUNT_EMAIL_REQUIRED (false by default):: If true, the user is required to provide an e-mail address when signing up.
ACCOUNT_LOGOUT_ON_PASSWORD_CHANGE (false by default): If true, users will automatically be logged out once they have reset their password.
ACCOUNT_USER_MODEL_EMAIL_FIELD ('email' by default): The name of the field in the user model containing the email.
ACCOUNT_USERNAME_REQUIRED (true by default): If true, the user is required to enter a username when signing up.
ACCOUNT_EMAIL_VERIFICATION ('optional' by default): Determines the e-mail verification method during signup. The possibilities are one of mandatory, optional, or none.

And many other parameters. As a conclusion, we can say that adjusting a couple of parameters in your Django app settings, you get lots of different behaviors for the registration and authentication.

Note:

To better understand the examples in the next sections:

You need to follow the dj-rest-auth installation steps.
Have installed django-allauth.
Make sure you have the authentication backends in your settings:

AUTHENTICATION_BACKENDS = [
    # allauth specific authentication methods, such as login by e-mail
    'allauth.account.auth_backends.AuthenticationBackend',
    # Needed to login by username in Django admin, regardless of allauth
    'django.contrib.auth.backends.ModelBackend',
]

Customization

If you are coding a REST API with Django and Django REST framework, you may need to customize the functionalities related to the registration and authentication. With those frameworks and dj-rest-auth it's quite simple. You can customize the Django User model, the views, the serializers, the main functionalities, etc.

For example, let's define a custom User model with a unique email, with no username, with a gender and a phone number attribute:

# models.py in the users Django app
from django.db import models
from django.contrib.auth.models import AbstractUser


GENDER_SELECTION = [
    ('M', 'Male'),
    ('F', 'Female'),
    ('NS', 'Not Specified'),
]


class CustomUser(AbstractUser):
    # We don't need to define the email attribute because is inherited from AbstractUser
    gender = models.CharField(max_length=20, choices=GENDER_SELECTION)
    phone_number = models.CharField(max_length=30)

In our settings, we have to use the Django parameter called AUTH_USER_MODEL to specify our User model. If the CustomUser model is defined in an app called users, we must add:

AUTH_USER_MODEL = 'users.CustomUser'

Besides, we have to add in our settings these parameters to use the email as the User identifier in our app:

ACCOUNT_AUTHENTICATION_METHOD = 'email'
ACCOUNT_USERNAME_REQUIRED = False
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_UNIQUE_EMAIL = True

Also, let's configure a signup without email verification just for now:

ACCOUNT_EMAIL_VERIFICATION = 'none'

I'll talk about account verification later.

Now we need to add our custom serializer to define the two added attributes and use them in the registration. The custom register serializer needs to be a child class of the one named RegisterSerializer that dj-rest-auth provides. Also we have to overwrite the save method to set those values that came in the request:

# serializers.py in the users Django app
from django.db import transaction
from rest_framework import serializers
from dj_rest_auth.registration.serializers import RegisterSerializer

from users.models import GENDER_SELECTION


class CustomRegisterSerializer(RegisterSerializer):
    gender = serializers.ChoiceField(choices=GENDER_SELECTION)
    phone_number = serializers.CharField(max_length=30)

    # Define transaction.atomic to rollback the save operation in case of error
    @transaction.atomic
    def save(self, request):
        user = super().save(request)
        user.gender = self.data.get('gender')
        user.phone_number = self.data.get('phone_number')
        user.save()
        return user

Now we can't forget to tell dj-rest-auth to use the recently created serializer. Let's add the CustomRegisterSerializer to the settings:

REST_AUTH_REGISTER_SERIALIZERS = {
    'REGISTER_SERIALIZER': 'users.serializers.CustomRegisterSerializer',
}

After this, if we go to the registration endpoint, we would see something like this:

The username appears in the browsable endpoint, but thanks to the configuration you can leave it empty.

And that's it! We have added a customized registration endpoint to our Django app. Now we can create users with email, gender, and phone number, and make some tests around the login functionality too.

Use of JWT

When an app manages authentication, a good and standard implementation of authentication by token authorization is using JWT (JSON web tokens). JWTs are a good way of securely transmitting information between parties because they can be signed, which means you can be sure that the senders are who they say they are. We can easily configure our Django app to use JWT. First we need to install djangorestframework-simplejwt:

pip install djangorestframework-simplejwt

Then add to the settings:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'dj_rest_auth.jwt_auth.JWTCookieAuthentication',
    ),
}

REST_USE_JWT = True
JWT_AUTH_COOKIE = 'my-app-auth' # The cookie key name can be the one you want

And we are good to go! We have an app that manages the authentication with JWT.

The User endpoint

If you need to modify or just retrieve the information of an authenticated user; you can use an endpoint provided by dj-rest-auth: dj-rest-auth/user/. It's a very useful endpoint in the case you want the user to be able to modify its own information.

What happens if you have customized the Django user model? Well, in that case you need to overwrite the user serializer to be able to see and update those added attributes. Let's continue with the example of a customized model with a gender and a phone number. In that case we need to tell dj-rest-auth to use the customized serializer already created (CustomRegisterSerializer) as the user's detail serializer, adding this to the settings:

REST_AUTH_SERIALIZERS = {
    'USER_DETAILS_SERIALIZER': 'users.serializers.CustomRegisterSerializer',
}

With that definition, we can retrieve and update the gender and phone_number of an authenticated user with the named endpoint. Let's say now, that our app only allows the gender to be updated after registration. In the CustomRegisterSerializer the phone number can be modified of course, because we want the user to add that value at the moment of the registration. We can define then another serializer that doesn't allow the phone number to be modified. Let's call it CustomUserDetailSerializer and add it in the same file that is CustomRegisterSerializer:

# serializers.py in the users Django app
...

from users.models import CustomUser

...

class CustomUserDetailsSerializer(serializers.ModelSerializer):

    class Meta:
        model = CustomUser
        fields = (
            'pk',
            'email',
            'phone_number',
            'gender',
        )
        read_only_fields = ('pk', 'email', 'phone_number',)

Then we have to add this as the correct user details serializer in our settings, replacing CustomRegisterSerializer with the new one:

REST_AUTH_SERIALIZERS = {
    'USER_DETAILS_SERIALIZER': 'users.serializers.CustomUserDetailsSerializer',
}

And this is what you will see in the endpoint of the user:

As you can see in the picture, the only value that can be modified is the gender.

Sign-up with email verification

It's possible to configure email verification at the moment that a user registers. This means that a successfully registered user has to check the inbox and confirm through the received email that he really is the owner of the email account. To do this, go to your settings and turn on email verification:

# Your settings file
...

ACCOUNT_EMAIL_VERIFICATION = 'mandatory'

With this, after a correct sign-up your app will send a verification email to the entered account. But you didn't finish: as dj-rest-auth documentation says, after enabling the email verification, you need to add a path to your urls that will be used by the verification view:

# Your urls file
...
from dj_rest_auth.registration.views import VerifyEmailView

...

urlpatterns = [
    ...
    path(
        'dj-rest-auth/account-confirm-email/',
        VerifyEmailView.as_view(),
        name='account_email_verification_sent'
    ),
    ...
]

If you don't do that, you will get a NoReverseMatch error.

Also, let's use a django-allauth configuration to activate the email account after the user clicks on the link received in the email. Add this to your settings:

# Your settings file
...
ACCOUNT_CONFIRM_EMAIL_ON_GET = True

And add this path to your urls file before the definition of the dj-rest-auth registration path:

# Your urls file
...
from dj_rest_auth.registration.views import VerifyEmailView, ConfirmEmailView

...

urlpatterns = [
    ...
    path(
        'dj-rest-auth/registration/account-confirm-email/<str:key>/',
        ConfirmEmailView.as_view(),
    ), # Needs to be defined before the registration path
    path('dj-rest-auth/registration/', include('dj_rest_auth.registration.urls')),
    path('dj-rest-auth/account-confirm-email/', VerifyEmailView.as_view(), name='account_email_verification_sent'),
    ...
]

That path needs to be there to avoid a template error in dj-rest-auth. As mentioned, dj-rest-auth it's an open source software and sometimes we can find some errors. The good thing of this is that anyone can address the discovered bugs.

Having this configuration, the user will be redirected to the login page after clicking in the received link. Don't forget to set the LOGIN_URL parameter in your settings, otherwise you can get an error if the default value is not a valid URL of your app. During development, you can set the LOGIN_URL as the login endpoint; remember that dj-rest-auth has browsable endpoints thanks to Django REST framework. So you can test sign-up flow by setting for example:

# Your settings file
...
LOGIN_URL = 'http://localhost:8000/dj-rest-auth/login'

Finally, you need to specify to Django the email backend that is in charge of sending the emails of your app.

Email backend

The Django email backend is the component that handles the sending of an email. You can choose among different possibilities that Django provides. The most common are SMTP and console backend.

You can configure an SMTP server by adding:

# Your settings file
...

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'your.email.host'
EMAIL_USE_TLS = True
EMAIL_PORT = 587
EMAIL_HOST_USER = 'your email host user'
EMAIL_HOST_PASSWORD = 'your email host password'

You have to choose the EMAIL_HOST, for example 'smtp.gmail.com'. In the EMAIL_HOST_USER and EMAIL_HOST_PASSWORD parameters put the information of the account that will be the email sender. If you are using Gmail as a mail server you will need to allow less secure apps and display unlock captcha. After this, your Django app will send verification emails for all the new users. I recommend you to use environment variables to keep sensitive information in your settings such as keys, the email host account, and its password, etc.

In real applications the best way to do this is integrating with an email service such as SendGrid.

In the second option, you can locally test the sign-up feature during development, by setting:

# Your settings file
...
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

In this case the email is sent and received by the console. So after a successful registration you will see in the console something like this:

If you open your browser and go to the link shown in the printed email, the user's account will be activated and you will be redirected to the LOGIN_URL. Now you are able to login with the entered email and password.

And there it is! You have configured sign-up with email verification. Let's see how to customize the emails that your application sends.

Email templates

Django allauth provides a few templates that are used by dj-rest-auth at the moment of sending the emails. As shown in previous section, the verification email printed in console has a template defined by the django-allauth package. It's possible to customize those email templates by overwriting a couple of files. First of all, let's create a template folder inside your main Django app folder.

From now on, let's assume that the name of your main Django app folder is myapp; so you need to add this in your TEMPLATES settings:

# Your settings file
...
import os
...

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            os.path.join(BASE_DIR, 'myapp/templates'), # Add this inside the DIRS list
        ],
        'APP_DIRS': True,
        ...
    },
]

Now you have to overwrite the corresponding template files for the desired customization. In this case, the path used by dj-rest-auth will be myapp/templates/account/email/. In there you can add these files to overwrite the ones used by default:

email_confirmation_message.txt
email_confirmation_signup_message.txt
email_confirmation_signup_subject.txt
email_confirmation_subject.txt

For example, to customize the email confirmation message, you have to create an email_confirmation_message.txt file in myapp/templates/account/email/. You can even reuse some of the content of the default file that you can find in the github repository of django-allauth. Besides, you can set a value to the ACCOUNT_EMAIL_SUBJECT_PREFIX parameter in your settings, to add a prefix to the subjects in the emails of your app.

It's important to say that there are some variables defined in the email templates that are needed and you have to keep them in case you overwrite the files. For example, this is the file of email_confirmation_message.txt:

{% load account %}{% user_display user as user_display %}{% load i18n %}{%
autoescape off %}{% blocktrans with site_name=current_site.name
site_domain=current_site.domain %}Hello from {{ site_name }}! You're receiving
this e-mail because user {{ user_display }} has given your e-mail address to
register an account on {{ site_domain }}. To confirm this is correct, go to {{
activate_url }} {% endblocktrans %} {% blocktrans with
site_name=current_site.name site_domain=current_site.domain %}Thank you from {{
site_name }}! {{ site_domain }}{% endblocktrans %} {% endautoescape %}

There you have activate_url, that is the link for the user confirmation. So you can remove for example the site_name if you want (or change it to show a different one) but you definitely have to keep the activate_url value. Otherwise, registered users won't be able to activate their accounts. Summarizing this section, if you decide to overwrite the email_confirmation_message.txt, your project structure will look like this:

root-project-folder
    |__myapp
    |    |__templates
    |        |__account
    |            |__email
    |                |__email_confirmation_message.txt
    |
    ...

Now let's see about the reset password functionality and how to customize it.

Reset password

If you want to have a reset password feature in your app, you can do it very easily with dj-rest-auth. The first thing to say is that by default you can't test it as a browsable endpoint because it will show a NoReverseMatch error. So you need to add this path to your urls file:

# Your urls file
...
from dj_rest_auth.views import PasswordResetConfirmView
...

urlpatterns = [
    ...
    path(
        'rest-auth/password/reset/confirm/<slug:uidb64>/<slug:token>/',
        PasswordResetConfirmView.as_view(), name='password_reset_confirm'
    ),
]

And now you are good to go! You can test the browsable endpoint which receives the email of the registered user, and dj-rest-auth sends the instructions to reset the password. You need of course to have configured the email backend.

You are able to modify the email template for the reset password functionality too. Assuming that you have already configured the TEMPLATES folder as in the previous section example, you can create a file in myapp/templates/registration/ called password_reset_email.html to overwrite the one used by default. Why do you have to define it there? Because is the default path used by Django. You can even reuse some code of the default file, that looks like this:

{% load i18n %}{% autoescape off %} {% blocktranslate %}You're receiving this
email because you requested a password reset for your user account at {{
site_name }}.{% endblocktranslate %} {% translate "Please go to the following
page and choose a new password:" %} {% block reset_link %} {{ protocol }}://{{
domain }}{% url 'password_reset_confirm' uidb64=uid token=token %} {% endblock
%} {% translate 'Your username, in case you’ve forgotten:' %} {{
user.get_username }} {% translate "Thanks for using our site!" %} {%
blocktranslate %}The {{ site_name }} team{% endblocktranslate %} {%
endautoescape %}

As said in the email validation section, remember to keep some of the needed variables in the template. In this case the most important is the url with the uid and the token that are required to update the user's password.

After these steps, you have finished the reset password functionality customization! Taking the previous section into account, your project folder would be now like this one:

root-project-folder
    |__myapp
    |    |__templates
    |        |__account
    |        |   |__email
    |        |        |__email_confirmation_message.txt
    |        |__registration
    |            |__password_reset_email.html
    |
    ...

Let's see in the last section how we can modify some of the default variables used in the templates.

Customize variables used in the templates

Let's think about this example: What if you want to modify the URL sent by default in the reset password email template? Well, Django allows you to define variables to be used in the templates by creating your own template tags. The first thing you need to do is define the custom value desired for the URL in your settings:

# Your settings file
...
CUSTOM_PASSWORD_RESET_CONFIRM = 'desired URL'

Then you have to create two files under myapp/templatetags/: One has to be an empty __init__.py file, and the other one registers and gets the tags. Let's call it password_reset_template_load.py:

from django import template
from django.conf import settings

register = template.Library()


@register.simple_tag
def get_settings_var(name):
    return getattr(settings, name)

Finally, go to your customized email template that you have created to overwrite the default one, and load the defined value by adding {% load password_reset_template_load %} at the top of the file, and after that add the line that has the custom value (in this particular case don't forget the uid and token):

// Your template {% load password_reset_template_load %} ... {% get_settings_var
'CUSTOM_PASSWORD_RESET_CONFIRM' %}?uidb64={{ uid }}&token={{ token }}

With these settings you have defined your custom template tags. I'm going to show you the final project structure with all the customization that I talked about:

root-project-folder
    |__myapp
    |    |__templates
    |    |   |__account
    |    |   |   |__email
    |    |   |        |__email_confirmation_message.txt
    |    |   |__registration
    |    |       |__password_reset_email.html
    |    |__templatetags
    |        |__password_reset_template_load.py
    ...

From now on, the email for password reset will show a customized URL. Remember that this was an example, you can create any template tag you want and use it in any template you have.

Notes

If while trying this you have an error that shows Site matching query does not exist., you can solve it by adding SITE_ID = 1 to your settings.
When you create your own templatetags make sure that the main app is added in the INSTALLED_APPS list in your settings.

Summary

I have presented a very useful package to handle registration and authentication with a set of REST API endpoints.
With minimal configuration, you can add endpoints to your Django app that already have those features implemented. We reviewed some advantages of the package, how easy it is to customize it, and code snippet examples.

In addition to this, I have shown you how to implement email validation at sign-up, and how to customize the emails that are sent by your app. We learned about the reset password functionality, and finally how to customize the templates with your own desired values. I recommend using the presented package because it solves lots of common problems and provides a set of features that are very important for most applications. Maybe in the future, you can not only use it but also contribute to it.

Original blog on Rootstrap

Registration and Authentication in your Django app with dj-rest-auth

brunomichetti — Thu, 12 Nov 2020 16:07:44 +0000

The dj-rest-auth package

User registration with optional activation.
Login and logout.
Retrieve and update the Django User model.
Password change.
Password reset via e-mail.
Social media authentication.

Browsable endpoints

Configuration

ACCOUNT_EMAIL_REQUIRED (false by default):: If true, the user is required to provide an e-mail address when signing up.
ACCOUNT_LOGOUT_ON_PASSWORD_CHANGE (false by default): If true, users will automatically be logged out once they have reset their password.
ACCOUNT_USER_MODEL_EMAIL_FIELD ('email' by default): The name of the field in the user model containing the email.
ACCOUNT_USERNAME_REQUIRED (true by default): If true, the user is required to enter a username when signing up.
ACCOUNT_EMAIL_VERIFICATION ('optional' by default): Determines the e-mail verification method during signup. The possibilities are one of mandatory, optional, or none.

And many other parameters. As a conclusion, we can say that adjusting a couple of parameters in your Django app settings, you get lots of different behaviors for the registration and authentication.

Note:

To better understand the examples in the next sections:

You need to follow the dj-rest-auth installation steps.
Have installed django-allauth.
Make sure you have the authentication backends in your settings:

AUTHENTICATION_BACKENDS = [
    # allauth specific authentication methods, such as login by e-mail
    'allauth.account.auth_backends.AuthenticationBackend',
    # Needed to login by username in Django admin, regardless of allauth
    'django.contrib.auth.backends.ModelBackend',
]

Customization

# models.py in the users Django app
from django.db import models
from django.contrib.auth.models import AbstractUser


GENDER_SELECTION = [
    ('M', 'Male'),
    ('F', 'Female'),
    ('NS', 'Not Specified'),
]


class CustomUser(AbstractUser):
    # We don't need to define the email attribute because is inherited from AbstractUser
    gender = models.CharField(max_length=20, choices=GENDER_SELECTION)
    phone_number = models.CharField(max_length=30)

In our settings, we have to use the Django parameter called AUTH_USER_MODEL to specify our User model. If the CustomUser model is defined in an app called users, we must add:

AUTH_USER_MODEL = 'users.CustomUser'

Besides, we have to add in our settings these parameters to use the email as the User identifier in our app:

ACCOUNT_AUTHENTICATION_METHOD = 'email'
ACCOUNT_USERNAME_REQUIRED = False
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_UNIQUE_EMAIL = True

Also, let's configure a signup without email verification just for now:

ACCOUNT_EMAIL_VERIFICATION = 'none'

# serializers.py in the users Django app
from django.db import transaction
from rest_framework import serializers
from dj_rest_auth.registration.serializers import RegisterSerializer

from users.models import GENDER_SELECTION


class CustomRegisterSerializer(RegisterSerializer):
    gender = serializers.ChoiceField(choices=GENDER_SELECTION)
    phone_number = serializers.CharField(max_length=30)

    # Define transaction.atomic to rollback the save operation in case of error
    @transaction.atomic
    def save(self, request):
        user = super().save(request)
        user.gender = self.data.get('gender')
        user.phone_number = self.data.get('phone_number')
        user.save()
        return user

Now we can't forget to tell dj-rest-auth to use the recently created serializer. Let's add the CustomRegisterSerializer to the settings:

REST_AUTH_REGISTER_SERIALIZERS = {
    'REGISTER_SERIALIZER': 'users.serializers.CustomRegisterSerializer',
}

After this, if we go to the registration endpoint, we would see something like this:

Use of JWT

pip install djangorestframework-simplejwt

Then add to the settings:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'dj_rest_auth.jwt_auth.JWTCookieAuthentication',
    ),
}

REST_USE_JWT = True
JWT_AUTH_COOKIE = 'my-app-auth' # The cookie key name can be the one you want

And we are good to go! We have an app that manages the authentication with JWT.

The User endpoint

REST_AUTH_SERIALIZERS = {
    'USER_DETAILS_SERIALIZER': 'users.serializers.CustomRegisterSerializer',
}

# serializers.py in the users Django app
...

from users.models import CustomUser

...

class CustomUserDetailsSerializer(serializers.ModelSerializer):

    class Meta:
        model = CustomUser
        fields = (
            'pk',
            'email',
            'phone_number',
            'gender',
        )
        read_only_fields = ('pk', 'email', 'phone_number',)

Then we have to add this as the correct user details serializer in our settings, replacing CustomRegisterSerializer with the new one:

REST_AUTH_SERIALIZERS = {
    'USER_DETAILS_SERIALIZER': 'users.serializers.CustomUserDetailsSerializer',
}

And this is what you will see in the endpoint of the user:

As you can see in the picture, the only value that can be modified is the gender.

Sign-up with email verification

# Your settings file
...

ACCOUNT_EMAIL_VERIFICATION = 'mandatory'

# Your urls file
...
from dj_rest_auth.registration.views import VerifyEmailView

...

urlpatterns = [
    ...
    path(
        'dj-rest-auth/account-confirm-email/',
        VerifyEmailView.as_view(),
        name='account_email_verification_sent'
    ),
    ...
]

# Your settings file
...
ACCOUNT_CONFIRM_EMAIL_ON_GET = True

And add this path to your urls file before the definition of the dj-rest-auth registration path:

# Your urls file
...
from dj_rest_auth.registration.views import VerifyEmailView, ConfirmEmailView

...

urlpatterns = [
    ...
    path(
        'dj-rest-auth/registration/account-confirm-email/<str:key>/',
        ConfirmEmailView.as_view(),
    ), # Needs to be defined before the registration path
    path('dj-rest-auth/registration/', include('dj_rest_auth.registration.urls')),
    path('dj-rest-auth/account-confirm-email/', VerifyEmailView.as_view(), name='account_email_verification_sent'),
    ...
]

# Your settings file
...
LOGIN_URL = 'http://localhost:8000/dj-rest-auth/login'

Finally, you need to specify to Django the email backend that is in charge of sending the emails of your app.

Email backend

The Django email backend is the component that handles the sending of an email. You can choose among different possibilities that Django provides. The most common are SMTP and console backend.

You can configure an SMTP server by adding:

# Your settings file
...

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'your.email.host'
EMAIL_USE_TLS = True
EMAIL_PORT = 587
EMAIL_HOST_USER = 'your email host user'
EMAIL_HOST_PASSWORD = 'your email host password'

In the second option, you can locally test the sign-up feature during development, by setting:

# Your settings file
...
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

In this case the email is sent and received by the console. So after a successful registration you will see in the console something like this:

Email templates

# Your settings file
...
import os
...

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            os.path.join(BASE_DIR, 'myapp/templates'), # Add this inside the DIRS list
        ],
        'APP_DIRS': True,
        ...
    },
]

email_confirmation_message.txt
email_confirmation_signup_message.txt
email_confirmation_signup_subject.txt
email_confirmation_subject.txt

{% load account %}{% user_display user as user_display %}{% load i18n %}{%
autoescape off %}{% blocktrans with site_name=current_site.name
site_domain=current_site.domain %}Hello from {{ site_name }}! You're receiving
this e-mail because user {{ user_display }} has given your e-mail address to
register an account on {{ site_domain }}. To confirm this is correct, go to {{
activate_url }} {% endblocktrans %} {% blocktrans with
site_name=current_site.name site_domain=current_site.domain %}Thank you from {{
site_name }}! {{ site_domain }}{% endblocktrans %} {% endautoescape %}

root-project-folder
    |__myapp
    |    |__templates
    |        |__account
    |            |__email
    |                |__email_confirmation_message.txt
    |
    ...

Now let's see about the reset password functionality and how to customize it.

Reset password

# Your urls file
...
from dj_rest_auth.views import PasswordResetConfirmView
...

urlpatterns = [
    ...
    path(
        'rest-auth/password/reset/confirm/<slug:uidb64>/<slug:token>/',
        PasswordResetConfirmView.as_view(), name='password_reset_confirm'
    ),
]

{% load i18n %}{% autoescape off %} {% blocktranslate %}You're receiving this
email because you requested a password reset for your user account at {{
site_name }}.{% endblocktranslate %} {% translate "Please go to the following
page and choose a new password:" %} {% block reset_link %} {{ protocol }}://{{
domain }}{% url 'password_reset_confirm' uidb64=uid token=token %} {% endblock
%} {% translate 'Your username, in case you’ve forgotten:' %} {{
user.get_username }} {% translate "Thanks for using our site!" %} {%
blocktranslate %}The {{ site_name }} team{% endblocktranslate %} {%
endautoescape %}

root-project-folder
    |__myapp
    |    |__templates
    |        |__account
    |        |   |__email
    |        |        |__email_confirmation_message.txt
    |        |__registration
    |            |__password_reset_email.html
    |
    ...

Let's see in the last section how we can modify some of the default variables used in the templates.

Customize variables used in the templates

# Your settings file
...
CUSTOM_PASSWORD_RESET_CONFIRM = 'desired URL'

from django import template
from django.conf import settings

register = template.Library()


@register.simple_tag
def get_settings_var(name):
    return getattr(settings, name)

// Your template {% load password_reset_template_load %} ... {% get_settings_var
'CUSTOM_PASSWORD_RESET_CONFIRM' %}?uidb64={{ uid }}&token={{ token }}

With these settings you have defined your custom template tags. I'm going to show you the final project structure with all the customization that I talked about:

root-project-folder
    |__myapp
    |    |__templates
    |    |   |__account
    |    |   |   |__email
    |    |   |        |__email_confirmation_message.txt
    |    |   |__registration
    |    |       |__password_reset_email.html
    |    |__templatetags
    |        |__password_reset_template_load.py
    ...

From now on, the email for password reset will show a customized URL. Remember that this was an example, you can create any template tag you want and use it in any template you have.

Notes

If while trying this you have an error that shows Site matching query does not exist., you can solve it by adding SITE_ID = 1 to your settings.
When you create your own templatetags make sure that the main app is added in the INSTALLED_APPS list in your settings.

Summary

Managing Drip Campaigns with Rootstrap Django Drip Library

brunomichetti — Wed, 28 Oct 2020 19:26:06 +0000

At Rootstrap we work with many initiatives, one of them is the Open Source one. The main goal in this case, is to analyze, maintain and develop open source software (OSS). In this blog post I'm going to talk about a particular project named Django Drip Campaigns and how was the process of publishing it as a pypi package.

Pypi (that stands for Python Package Index) is a repository of software for the Python programming language that helps us find and install software developed by the Python community. One of our contributions to the OSS community is to publish and maintain the mentioned project and I want to share that experience in this article.

The Django Drip Campaigns Project

Django Drip Campaigns Project is an open source app to create drip campaigns for email using Django’s admin interface and a User queryset. This means that using this app, you are able to easily schedule and send emails to the registered users, creating querysets in a friendly interface.

For example, you can very quickly configure your app to send a periodic email to the users that haven't logged in in a day or more, send a custom email to the users that registered more than a week ago, etc. To do this, you only need to define one or more querysets in the admin page.
If you want to install it and work with it, you can follow the simple steps documented in the github page of Django Drip Campaigns.

We didn't create this project, we just made a fork of the one written by Zapier that is no longer maintained. After this, we worked on fixing some bugs, doing a couple of enhancements, creating more documentation and publishing the app as a pypi package.

Since it's published, any programmer that wants to use this project in his django app is able to quickly download it with a simple command described in the mentioned docs.

Features

After a successful installation and configuration, you are able to easily create drip campaigns for email using the admin interface of Django. If you don't have an admin user, create one with the django createsuperuser command. After this, login into Django's admin page and click on Drips to manage them. There you are able to:

View created and sent drips.
Create a new drip.
Select and delete drips.

Creating a new drip

Click on the ADD DRIP + button to create a new drip. You will see a page like this:

In the next section I will talk about the queryset rules that you can build in an easy way to create very useful drips.

Queryset rules

When you create a drip, you need at least one queryset so the users in it will be the ones that receive the defined email. In the METHOD TYPE menu you are able to choose to filter or to exclude the users in the defined queryset.

On the other hand, when you click in the FIELD NAME OF USER input, you will see the fields of your User model, and the fields of your user model in the models related to it:

In the previous image, for example, last_login is the field in the User model, and groups__user__id is the user id from the Groups model that is related to it. So you can enter the name of the field or select it from the list that you see when you click on the input.

After the selection of the field name, you have to choose the type of lookup that you want to do over the field. The possibilities are exactly, exactly (case insensitive), contains, contains (case insensitive), greater than, greater than or equal to, less than, etc. This lookup type will be done over the user field and the FIELD VALUE that you enter.

The FIELD VALUE input can be a string, a number, or a regular expression. The correctness of the queryset rule will depend on the type of the user field, the lookup type, and the field value.

When you enter a user field that has a date type, Django Drip Campaigns allows you to enter a date value in natural language combining the current time and some operation with seconds, hours, days, etc. For example, if you have selected the field last_login that has a date type, and you want to create a drip to send emails to the users who logged in exactly one week ago; you can enter:

now-1 week

now- 1 w

Possible operations and values:

Add (+) or subtract (-) dates.
On the left side of the operation, write the current datetime value: now.
On the right side of the operation:
- seconds or s.
- minutes or m.
- hours or h.
- days or d.
- weeks or w.
- If you enter the number 1, you can write second, minute, etc.
- Don't enter a space between now and the operation symbol. Optionally you can add (or not) a space around the number value.

Let's see some examples of the date values that you can enter:

now-1 day
now+ 8days
now+ 1 h
now-4hours
now- 3 weeks
now-1 weeks

To finalize this subsection, I would like to show you two examples to clarify the queryset rules creation:

Send a recordatory email to the users that haven't logged in for a week or more:
- method type: filter
- user field: last_login
- lookup type: less than or equal to
- field value: now- 1 week
Send an email to the users that have a gmail account, excluding the ones that registered yesterday:
- Rule 1:
- method type: filter
- user field: email
- lookup type: contains
- field value: gmail
- Rule 2:
- method type: exclude
- user field: date_joined
- lookup type: exactly
- field value: now- 1 d

As you can see, the queryset rules creation is very powerful and for each drip you can add as many as you want.

View the timeline of a drip

In the django admin, you can select a drip and then click on the VIEW TIMELINE button to view the emails expected to be sent with the corresponding receivers:

Message class

By default, Django Drip creates and sends messages that are instances of Django’s EmailMultiAlternatives class. If you want to customize in any way the message that is created and sent, you can do that by creating a subclass of EmailMessage and overriding any method that you want to behave differently. For example:

from django.core.mail import EmailMessage
from drip.drips import DripMessage

class PlainDripEmail(DripMessage):

    @property
    def message(self):
        if not self._message:
            email = EmailMessage(self.subject, self.plain, self.from_email, [self.user.email])
            self._message = email
        return self._message

In that example, PlainDripEmail overrides the message property of the base DripMessage class to create a simple EmailMessage instance instead of an EmailMultiAlternatives instance.

In order to be able to specify that your custom message class should be used for a drip, you need to configure it in the DRIP_MESSAGE_CLASSES setting:

DRIP_MESSAGE_CLASSES = {
    'plain': 'myproj.email.PlainDripEmail',
}

This will allow you to choose in the admin, for each drip, whether the default (DripMessage) or plain message class should be used for generating and sending the messages to users.

Send drips

To send the created and enabled drips, run the command:

python manage.py send_drips

You can use cron to schedule the drips.

What we have been doing in the project

As I said, we didn't create this project. We forked it and started to make some fixes and improvements. The first thing we did was to upgrade the used Django version. Now it has django 3.0.7.

Besides we started to create the documentation with Sphinx to have this project in Read the Docs in the future.

Another enhancement was to add flake8 to enforce the use of style guides in the code. Finally we added an integration with travis-ci to have a continuous integration service on any change in the project.

We registered and fixed some issues such as the view of the User fields, and configuration problems to run the app locally. There are some issues to address and discover yet so if you like OSS you are more than welcome to work with Django Drip Campaigns.

In the open source initiative we wanted to publish this app as a pypi package, so I'll explain this a little more in the next section.

Publish a pypi package

Despite Django Drip Campaigns is still a work in progress, we already published the package in PyPI. It's nothing fancy, but it's a must to know a few things before starting, so in this section, I'll be talking about the main steps for publishing an app.

Project structure: Your project needs a couple of files in a determined place to be published correctly. An appropriate structure could be:

root_project_folder/
  |
  |__ docs/    # Documentation files
  |__ project_folder/  # Code of the app
  |__ AUTHORS.md
  |__ LICENSE.txt
  |__ MANIFEST.in
  |__ README.md
  |__ setup.py

AUTHORS: Specify the author's information of your project in this file.
LICENSE: You have to specify the license of the project. That is a document that provides legally binding guidelines for the use and distribution of software. In OSS the most known are BSD and MIT. We used an MIT license in this case.
MANIFEST: Here you can specify the files that aren't programming files but you want to include anyway in the build of your project. For example the authors file, the docs, etc.
README.md: A file where you introduce and explain your project. It's very important because otherwise you can have a very good and useful project but if new people see it and don't understand what it does nor how it works, then it won't be used.
setup.py: This is a fundamental file. It contains a global setup function that is in charge of the building and configuration of your project at the moment of the installation.
docs folder: Here is where the files of the documentation are.

You can see examples of the described files in the django-drip-campaigns github page. Of course your project can have other files, the ones mentioned are the most important to publish a pypi package.

Having all set and ready to be published, you need now to register in the pypi.org page. After you are registered, go inside the terminal to your project folder and follow these steps:

Install setuptools and wheel. They are used to generate the distribution package: a group of files needed to upload your app.

python3 -m pip install --user --upgrade setuptools wheel

Generate the distribution package. The next command will create all the necessary files under a dist/ folder:

python3 setup.py sdist bdist_wheel

Install twine, that a allows you to upload your app as a package:

python3 -m pip install --user --upgrade twine

Once installed, run twine to upload the archives under dist/. This will ask for a user and a password, that are the ones you used on the registration at pypi:

python3 -m twine upload --repository pypi dist/*

If everything went well, you have published your project as a pypi package. Now you anyone is able to download it with:

pip3 install <project-name>

Summary

In this blog post I talked about the Django Drip Campaigns project, its main features and what we did with it at Rootstrap. We love to make contributions to OSS because it's a wonderful way of learning, of sharing knowledge, and creating useful programs to be used by other programmers.

Besides I talked about the experience and how to publish a project as a pypi package so anyone who wants to give it a try can download it with a simple command.

OSS gives a great opportunity not only to see how some other group of people solves a problem addressed by the program, but also to purpose and contribute with fixes and improvements. Finally, I hope you have enjoyed this blog and feel motivated to use and work with Django Drip Campaigns and to contribute with OSS.

A prediction experiment with Machine Learning

brunomichetti — Mon, 28 Sep 2020 21:38:29 +0000

I recently participated in a Machine Learning workshop at Rootstrap, where my coworkers and I learned about the basics of data science, did some research, and created interesting experiments. We had the opportunity to choose among the studied Machine Learning algorithms and work with them. So, I decided to do an experiment where a mathematical model predicts the life expectancy of a country. That is, given some data of a given country, we can make a prediction of its life expectancy in a determined year. The experiment was made in a jupyter notebook, using the python programming language, and the scikit learn library. In this blog, I’m going to talk about my experience and explain a little bit about the work I did and the new things that I have learned.

Prediction in Machine Learning

The word prediction in machine learning refers to the output of a trained model, representing the most likely value that will be obtained for a given input. The model is trained with historical data, and then predicts a selected property of the data for new inputs.
Prediction is used in lots of different areas, since it allows us to make highly accurate guesses about many things, such as predicting what the stock markets will do on any given day, predict results in sports, or even help the medical industry predict diseases.
The algorithms for prediction are classified as supervised learning algorithms since they need a training dataset with correct examples to learn from them. This means that the first thing I had to do to start the experiment was finding a dataset, which contains information about countries, including, of course, their life expectancy.

The dataset

I used a public dataset of Life expectancy from kaggle to train the model. It comes from a statistical analysis of the factors that might influence the life expectancy of a country. The dataset is a big table containing information about each country, and several factors over several years. The years are from 2000 to 2015, and some of the factors are:

Adult Mortality: Adult mortality rates of both sexes (probability of dying between 15 and 60 years per 1000 population).
infant deaths: Number of infant deaths per 1000 population. Alcohol: Alcohol, recorded per capita (15+) consumption (in liters of pure alcohol).
Hepatitis B: Hepatitis B immunization coverage among 1-year-olds (%).
Under-five deaths: Number of under-five deaths per 1000 population.
Polio: Polio (Pol3) immunization coverage among 1-year-olds (%).
Total expenditure: General government expenditure on health as a percentage of total government expenditure (%).
Population: Population of the country.
Income composition of resources: Human development index in terms of income composition of resources (index ranging from 0 to 1).
Schooling: Number of years of schooling (years).

This dataset has 22 columns and 2,938 rows of data. I needed to preprocess the data before training the model. Why did I need that? because when we work in machine learning, we can hardly use the data in the way we get it. First, we need to solve consistency problems, prepare the data for the format and type that the model expects, and remove useless information, etc. If we don’t address those problems, we can’t affirm that the result of the algorithm achieves the main goal. For example, some of the problems could be:

Missing data.
Work with different types of data.
Work with different formats of data.
Properties of the data with null values.
The data format is different from the one that the algorithm expects.

In the next section, I talk about the problems I had to face, and how they can be solved.

Preprocessing the dataset

The process of applying the necessary transformations to the data in order to prepare it for the model is called data cleaning. There are several kinds of problems addressed in data cleaning, the ones that I faced are:

Null values: some of the rows in the dataset had null values, and given that machine learning models are mathematical models, they can’t work with that kind of input.
Unnormalized data: some machine learning models don't work well if they have variables that have many different ranges of values.
Problems with types: I needed to solve how to pass to a mathematical model, values that weren’t numbered, for example, a categorical variable.

In this section, I talk about some approaches to solve those problems.

Null values

When a row has empty spaces, that is null values in the dataset, we can delete that row, or fill those empty spaces with data. I didn’t choose to delete the rows, because I would lose a big part of the dataset, and models don’t work well if they are trained with small datasets or I might be removing relevant information. So I decided to fill the null values. To do this, there are many techniques. For example, choosing a default value, or “simulating a distribution of the data” and selecting a value of that simulation. I chose the second example because using the simulation I’m able to generate numbers similar to the real values. I’ll explain a little bit the method in the next section.

Simulate a probability distribution

A random variable is a quantity that is produced by a random process. Besides, a probability distribution is a summary of probabilities for the values of a random variable. Knowing this, we can select a column of the dataset that has null values, and simulate its probability distribution. After this, we can randomly select a value in order to fill the null values affecting our dataset. So for each column in the dataset that has some null values, I executed these steps:

Plot a histogram of the column using the hist function in python.
Compare the histogram with the graph of known distributions: normal, truncated normal, uniform, exponential, etc.
Select the probability distribution that has the most similar graph to the histogram.
Simulate the selected probability distribution.
For each null value in the column, fill it randomly choosing a value from the simulation.

Here we can see a comparison with a histogram of a column in the dataset, and the selected probability distribution simulation.

In this case, the histogram from the original data is similar to an exponential distribution. Thus, the simulation was done using an exponential distribution with the scale and size parameters taken from the original data. Another example:

In this second example, the histogram of the original data is very similar to a normal distribution. In an analogous way, I have simulated a normal distribution with parameters taken from the original data.

In the case that the histogram does not resemble any probability distribution, we can divide the mentioned histogram into intervals (looking at the graph) and approximate a probability for each one. After this, we can create a function that returns a random value of an interval, selecting it with the probabilities set. Here is an example of this case:

And the definition of the probabilities set is:

# Simulate ranges
ranges_list = [
    range(1,10),
    range(10, 40),
    range(40, 60),
    range(60, 70),
    range(70, 80),
    range(80, 90),
    range(90, 100)
]
probs = [0.05, 0.02, 0.03, 0.05, 0.1, 0.2, 0.55]

After those definitions, I created a function that gets one interval from the ranges_list according to the probabilities set and returns a randomly selected value within that interval. That value is the one that fills the empty space.

One hot-encoding

As I said, mathematical models need to work with numbers. In the dataset, I had some data that took values from a set of words, such as the country column. I needed to map each country's name to a number. But if I did that, it could have happened that the model gave more importance to a country for having a larger number. The solution to this problem is to use other techniques such as one hot-encoding: generate a new column for each country name, and put 1 if the country row has that name, and 0 if not. Here is a simple example:

In the end, I could have deleted one column of the countries, knowing that if a row has 0 in all the countries’ columns, that means that the row has the missing country name. It’s a good way to optimize the amount of data that I work with. I didn't do that because I wanted to have all the countries for better understanding of the results.

Split and normalize the dataset

Supervised learning algorithms need a train dataset to learn from it. Besides, it needs another dataset different from the train dataset, to check if the model has learned correctly. The second one is named test dataset. I only had one dataset, so I had to split it. The scikit learn library provides us with the split function, called train_test_split. With this, we can get two datasets from the original one. Generally, the split is 80% train and 20% test.
On the other hand, when the features in the dataset have different range values, we need to normalize them. If we have a column in the dataset that has values between 0 and 1, and another column that has values between -1000 and 100000, the model can lose information or represent our data in an incorrect way. There are different scalers in scikit learn library, I used the MinMaxScaler that scales all the values in a range between two values. With that class, I created two kinds of scalers: one for the target variable (the life expectancy) and other for the rest of the features. Doing this separation of scalers, I am able to use the first scaler to convert the results to the original range, and compare that with the original values from the testing data.

The models

In statistics, there is a kind of process named regression analysis. The processes of this kind are used to estimate the relationship of one variable (often called the target variable), with one or more other variables (often called features, or predictors). This is what I did in the experiment, considering the life expectancy column in the dataset as the target variable, and the rest of the columns as the predictors.
The scikit learn library provides us with lots of machine learning models already implemented, and among them, the regressors. In this section, I’m going to give a general idea of the used models in the experiment.

Linear Regression

In linear regression, the relationship between the target variable and the predictors is modeled using a linear function:

The main goal is to find the unknown coefficients, which are estimated from the data. That’s why prediction is a supervised learning algorithm. I used a linear regression with two regularization methods: Lasso and Ridge Regression.
On one hand, the LASSO (Least Absolute Shrinkage and Selection Operator) regression is a process that uses shrinkage. Shrinkage is where data values are shrunk towards a central point, like the mean. Lasso regression performs L1 regularization, which adds a penalty equal to the absolute value of the magnitude of coefficients. Some coefficients can become zero and have been eliminated from the model, something very useful when we have lots of parameters. As I said, I used the implementation of Lasso from scikit learn. With this python class, we can use the fit function to train the model with the preprocessed and scaled dataset, and then use the predict function with new inputs to obtain the prediction of the life expectancy.
On the other hand, the Ridge regression performs L2 regularization, that is, a penalty using the Euclidean norm. Generally, this algorithm reduces the coefficients but doesn't eliminate them. It’s good to use this type of model when we have overfitting problems. As the Lasso regression, scikit learn provides also a python class for Ridge regression, with the fit and predict functions.

Decision trees

In regression, a decision tree is a type of model that builds a regression in the form of a tree structure. It breaks down the dataset into smaller subsets while at the same time develops an associated decision tree. In that tree, the leaf nodes are results, and the other nodes are decisions.
In the experiment, I used the Random Forest model, which builds several decision trees (that is why it is called forest) and then ensembles them. Each tree takes only a random subset of the features and makes the decisions from it. What does “ensembles” mean in machine learning? Well, ensemble methods use multiple learning algorithms to obtain better predictive performance than the one that could be obtained from any of the constituent learning algorithms alone. Random forest it’s also implemented in scikit learn and has the fit and predict functions.

Metrics and results

How do we know if the regression model works well? We can use some metrics to analyze that. As I mentioned, it’s a common practice to split the dataset into training and testing datasets. The training dataset it’s used by the model to learn, and the testing dataset it’s used to evaluate the correctness of the resultant model.
How do we do this evaluation? By the moment of the evaluation, we have a trained model, and a testing dataset that has two main parts: a set of instances of the data, let’s say X_test, and a set of the corresponding correct result for each instance in X_test, let’s call it Y_test. A good way to measure how well our model works is by applying the predict function on X_test, and comparing that result with the correct one (Y_test). To make the comparison we have a couple of metrics that we can use:

As you can figure out, the closer to 0, the better results.
In the experiment I used R-squared, Mean squared, and Root mean squared error. For each trained model, I did a measurement of those errors, and also, I used the created scalers for the target variable to scale back the result and compare the values in the original range. In these tables you can see the results of each model:

Results with scaled data

Results in the original range

The conclusion is that Ridge and Random Forest regression work better than Lasso for this dataset, given that the errors are closer to 0. Besides, scaling back and comparing in the original range, we saw that the error in those cases are both approximately 2 years, a good threshold to predict life expectancy.

Summary

In this blog I talked about a prediction experiment with machine learning. It was very interesting to me and I learned a lot. I hope that at this point you enjoyed your reading, and understood the main ideas of the general process of prediction. Also, I mentioned important concepts of machine learning such as supervised learning, preprocessing the data, training and testing the models, etc. We saw some of the most used models in regression, and a way to evaluate them in order to see how well they are working. Finally, after reading this blog I hope that you feel inspired to work with machine learning because it’s fun and helps us solve lots of amazing and interesting problems.

A quick classification experiment using machine learning

brunomichetti — Mon, 28 Sep 2020 21:27:00 +0000

My team recently faced a brand new challenge: developing a way to classify job positions written in natural language by lots of different people. It sounds simple, but there are a few factors that made this problem hard to solve. Job positions can be ambiguous depending on language usage and new job positions appear all the time.

The set of possible positions we needed to classify is extremely wide. A few quick examples: Digital Marketing, Public Relations, Founder and Inventor of X. Creating a program with enough "if" statements to accurately analyze job positions and classify them based on the words they contain would be almost impossible. Besides that, the appearance of new job positions not yet entered into our program would cause the classifier to fail frequently.

But what if it was possible to train a program about the nature of different job positions to face new job titles and classify them accordingly? Well, it is. And that’s where machine learning comes into play.

The classification problem

The problem we faced is easy to explain: classify job positions by areas and levels. Given a set of classes, we needed to build a classifier that has a job position as the input, and the corresponding class as the output. Because we need to use two different criteria for accurate classification of job positions, we needed to create two classifiers: one for the area and one for level. For example, if an area classifier receives "CEO and Founder," it has to return the “Business” class, and the level classifier needs to return the “C-Suite” class.

The problem is simple, but the solution is not. We knew machine learning was frequently used for applications like this and started searching for the tools we’d need to create our program.

Applying machine learning

Machine learning focuses on the development of computer programs that can use data to learn on their own. The difference between this and regular programs that execute prespecified instructions is that machine learning algorithms can use previous experience to face new incoming data and act accordingly.

This means that machine learning programs learn using the data we provide them. A real-world example of this can be seen in this excerpt taken from "Understanding Machine Learning: From Theory to Algorithms."

Rats Learning to Avoid Poisonous Baits

"When rats encounter food items with novel look or smell, they will first eat very small amounts, and subsequent feeding will depend on the flavor of the food and its physiological effect. If the food produces an ill effect, the novel food will often be associated with the illness, and subsequently, the rats will not eat it. Clearly, there is a learning mechanism in play here – the animal used past experience with some food to acquire expertise in detecting the safety of this food. If past experience with the food was negatively labeled, the animal predicts that it will also have a negative effect when encountered in the future."

Supervised learning

Supervised learning is a subset of machine learning where we provide the algorithms with "labeled data." This means a dataset filled with correct examples is used to construct the mathematical model that the algorithm uses to classify job titles. In this particular case, the algorithm needs a set with elements in the form:

<example job position>, <corresponding classification>

But what does learning mean in this scenario? The main concept here is that machine learning algorithms represent their knowledge of a problem using a mathematical model. They choose initial values for the model’s parameters, process the dataset, compare it with the correct results, and then go back to further refine the parameters.

This process is repeated until the algorithms obtain the parameters that best fit the correct dataset.

Scikit learn

Scikit-learn is a great resource to work with for any machine learning project. It’s an open-source library coded in Python with lots of tools for predictive data analysis. It has far too many functions to cover in a single article, but we recommend clicking over and having a look if this sort of thing piques your interest. "Working With Text Data" was one of the examples we found particularly valuable for our machine learning application.

Selected classes

Our team tested two classes from Scikit learn: SGDClassifier and MLPClassifier. Classification algorithms try to minimize the cost of errors from a loss function during training.

The SGDClassifier class uses linear classifiers, minimizing the loss function using the Stochastic Gradient Descent training method. That’s where this class gets its name, and the methodology behind it is to iterate through data using a gradient of the function to achieve the minimum cost of errors.

The MLPClassifier is an abbreviation of a neural network called Multi-Layer Perceptron. Neural networks are quite complex and out of this article’s scope. What you need to know is that both classes represent supervised learning algorithms and follow the same overall construction process.

General working process

Scikit-learn allows us to build classifiers and measure their results. With the tools provided by the Scikit library, it’s possible to solve this problem in a few ways. Here’s the overall framework we followed:

Create the dataset
Split the dataset
Represent documents as vectors of real numbers
Fit and tune the classifier

Many different learning algorithms can be used as classifiers. The implementation might be slightly different depending on the tool, but this general process is the same for most classification problems.

Create the dataset

Supervised learning requires a dataset with correctly classified examples. The first step is creating this dataset. It’s made by humans, and there’s no way around it. It’s boring but fundamental. Since the dataset will be used by the algorithm to train and adjust the mathematical model, it’s important to make sure all of the examples are correctly classified.

The size of the example dataset required to train the model depends on the problem. It’s vital to have a representative dataset for each possible classification. For example, if we wanted to create a classifier that receives an image of a pet and classifies that as a dog, cat, or other, we would need a representative dataset containing several examples with a wide variety of cat images, dog images, and other pet images.

Split the dataset

After constructing the classifier, we need to validate that it’s working properly. To do this, a testing set is required. Note that you can’t use the same dataset for training and testing. The testing process has to be conducted using a set unknown to the classifier. That ensures the classifier works as expected with new, incoming data.

We need to split the dataset into two sets: a training set used for the learning process, and a test set to test how the classifier is working. A general guide is to keep 80% for training and 20% for testing, but there are many different methods to generate this ratio.

Represent documents as vectors of real numbers

Because a classifier builds a mathematical model to represent its knowledge and it’s easier to compute numbers than strings, we have to transform each sentence from a string into vectors of real numbers. This is known as text feature extraction. The goal is to extract important information from each document to represent the training and testing sets as a collection of vectors.

Scikit-learn provides a few functions and classes that can help us with this. The CountVectorizer class is used for tokenization and occurrence counting. The tokenization consists of separating the text into a set of its containing words according to a selected separator and the right criteria. Occurrence counting is a technique that creates a vector filled with the number of times each containing word occurs.

TfidfTransformer is another useful class that creates a vector based on the frequency of each word in a document. The main idea is to minimize the impact of frequently occurring words in the English language (the, a, is, etc.) that aren’t important during classification.

In summary, we need to transform the collection of documents into a set of vectors that we can use to train the classifier’s learning algorithm. There are many ways to go about this, and if you’re interested in learning more, check out the feature extraction tutorial Scikit has available.

Fit and tune the classifier

This is where the algorithm starts to learn. A supervised learning algorithm has parameters and hyperparameters. The parameters represent the algorithms knowledge and they are adjusted by the algorithm using the training set. An example of a parameter would be the weight selected for a determined input feature.

The hyperparameters are related to the training process and impact the way the algorithm learns. Some examples of hyperparameters are the maximum number of iterations, the fault tolerance, the number of hidden layers in a neural network, etc.

The SGDClassifier and MLPClassifier both have a function named fit that chooses the best parameters to fit the training set. However, the hyperparameters are defined by default. We can vary and combine them to optimize our results, but since there are so many possible combinations of these hyperparameters, this process can be quite tedious. Scikit-learn helps us find the most optimal combination of hyperparameters with the GridSearchCV class. This class uses a technique called cross-validation while trying all of the provided combinations of hyperparameters and measures the results from part of the training set. This part of the set is named the validation set, and it’s where the class stores the hyperparameter combination that achieves the best results.

This is the process of tuning a classifier. After this part, we have a classifier instance and we can use the function predict to get the classification of new incoming job positions.

How do we know if the classifier is working as expected?

If we want to check on how our classifier is working, it’s time to use the testing set. Several metrics are used in classification problems and Scikit-learn has a few functions to work with them. Our team used a function called metrics. This function compares the classifier’s prediction of the testing set with the correct results and returns metrics that help us define whether or not the classifier is working as expected. We have a prediction for each category of classification, and we focused on some specific metrics returned by the function to gauge the accuracy of our classifier. These metrics are:

Where tp means true positives, fp = false positives, and fn = false negatives. Precision is the number of elements classified as true positive over the total number of elements classified as positive.

The recall is the number of correctly classified elements over the total number of truly positive elements. These metrics are taken by the function for each classification category, and they will all output a number between zero and one. The closer to one, the better. We focused on the F score, a harmonic measure between precision and recall. If we get good precision but bad recall, the classifier isn’t working as expected. So, if the F score is close to one, we know our classifier is working accurately.

Overfitting and underfitting

There are two common problems in the classification process: overfitting and underfitting. To explain this, let’s use the pet classifier example from above. The goal is to create a classifier that receives an image of a pet and classifies that as a dog, cat, or other.

If we create a training set using only images of big dogs like a Labrador or a Saint Bernard, then the classifier won’t classify a Chihuahua as a dog. This is an example of overfitting. Our classifier fits particular cases of the training set, so when new input comes, the classifier doesn’t respond as expected. The problem here is that our training set is too specific.

On the other hand, if we create the training set using only images of white dogs, then the classifier will classify a white cat as a dog, and a white hampster as a dog, etc. This is underfitting. Our training set is much too broad. Here’s an image that illustrates these problems in a way that’s a little easier to visualize:

How can we avoid these common problems?

Create representative training and test sets with good proportions of each category
Do not overload the training set with overly specific cases
Tune the classifier by changing the hyperparameters
Use cross-validation
Use early stopping techniques (stop the training before overfitting occurs)

Our results

After many mistakes, tests, and iterations, we were able to create classifiers that work exceedingly well. These are the results we achieved for level classification:

MLPClassifier

                precision    recall  f1-score   support

     C-Suite       0.90      0.93      0.92        29
    Director       0.97      0.92      0.94        37
     Manager       0.90      0.95      0.92        19
       Other       0.93      0.95      0.94        41
          VP       1.00      0.96      0.98        24

    accuracy                           0.94       150
   macro avg       0.94      0.94      0.94       150
weighted avg       0.94      0.94      0.94       150

SGDClassifier

               precision    recall  f1-score   support

     C-Suite       0.97      0.97      0.97        29
    Director       0.97      0.92      0.94        37
     Manager       1.00      1.00      1.00        19
       Other       0.95      1.00      0.98        41
          VP       1.00      1.00      1.00        24

    accuracy                           0.97       150
   macro avg       0.98      0.98      0.98       150
weighted avg       0.97      0.97      0.97       150

The f1-score (F score) is close to one, so it’s safe to say the classifier is working as we expected. With perfect precision and recall in more than one category, you can see the SGDClassifier works slightly better than the MLPClassifier.

Summary

In this article, we talked about using machine learning to solve a real-world classification problem our team recently encountered. It’s also a great little demo of our work. The Scikit-learn library has a ton of different machine learning tools anybody can use to tackle a variety of problems. We went over the general process used to build a classifier and a few common hiccups you may encounter. We know other techniques can be used to improve our work and that’s why the machine learning universe is so amazing. You’re always able to learn and discover more, and as an engineering team, that keeps us motivated and happy to continue working.

Django: Tips and good practices

brunomichetti — Mon, 28 Sep 2020 21:16:56 +0000

Abstract: The Django framework save time and effort and help to create apps and REST APIs with maintainable code. This blog gives some practical examples of what to do and what not to do when you code in Python using Django.

Summary: This blog explains why good code structure and configuration is important in any environment. It describes Django’s main concepts and some easy ways to implement web app functionalities with existing tools that solve general problems. I also recommend good practices for efficiency and APIs and packages for session management and testing. Finally, I describe some things that, from my experience, are best to avoid.

Start with good code

Like all software developers, I want to create maintainable code. It’s not just for me but also for my partners. It feels good to know that the programs you use or maintain have good code quality and what you construct is easy to change, reuse, update, review, and so on.

I recently learned to code in Python programming language by using the Django and Django REST frameworks. Now I want to share some tips and good practices for developing REST APIs with those technologies. When I face a programming problem, I first try to solve it. Then I look for an even better way to reach the solution. I write that information down and store it, so I can share what I learn with others. This blog is compiled from the notes I took when I learned Django. My goal is to help others learn Django too. It’s also a good place to start for people who just want to learn code.

The Django and Django REST frameworks

Django is a solid framework for developing web apps with Python language. It’s fast, secure, scalable, and well documented. If you want to build REST APIs, you can combine Django with the Django REST framework to generate a base project in just a few seconds. These webpages tell you more:

Tip: Don’t reinvent the wheel.

Any teacher of computer science might say to students: Don't reinvent the wheel. For example, you might run into a problem when you’re coding that already has a documented solution. It's better and faster to take advantage of a known solution than to spend time creating a new one. There are many open source projects that perform common tasks to solve general problems. You can use them for free on your apps, and you can also make contributions and propose improvements!
So if you want to create a REST API, there’s no need to reinvent the wheel. Use the Django frameworks to design that kind of app, and use the maintained tools to attack the common problems you encounter.

Scaffolding and configuration structure

As I said, anyone can quickly create a base project with Django frameworks. Detailed commands for that are in Django’s documentation. It’s a good practice to build a structure that increases maintainability. Then we can easily create a different configuration for each environment we use. Some examples are development, production, and testing.
By default, Django generates a settings.py file for all the configurations of a project. That means several if's separate each part for each environment. This way is better:

Keep the general configuration, which is common to all environments, in a base file.
Then create different files that correspond to each of the different environments.

In this simple example, I change the file settings.py in a project named example_project for a settings folder containing the next files:

__init__.py
base.py
development.py
production.py

The __init__.py file defines which configuration to use, depending on the environment:

import os
import importlib

# by default use development
ENV_ROLE = os.getenv('ENV_ROLE', 'development')

env_settings = importlib.import_module(f'example_project.settings.{ENV_ROLE}')

globals().update(vars(env_settings))
    
# import local settings if present
try:
    from .local import *  # noqa
except ImportError:
    pass

With the __init__.py file, I define an environment variable named ENV_ROLE for each environment. The value is the name of the configuration file that corresponds to that environment. For example, in my production environment, I create the ENV_ROLE variable with the production value. Then I use the production.py file to set the specific configuration in the production environment.

By creating environment variables, you can separate the configurations of the different environments. Examples are different databases for development and production or different allowed hosts. In the base.py file, don’t forget to set the common configurations for all environments like INSTALLED_APPS, MIDDLEWARE, and TEMPLATES.

One project, multiple apps

Several applications might exist in a single Django project. In Django context, an app is a set of related functionalities and models. Some more good practices are to build scaffolding that shows where the apps are in order to define standard locations for general functions and classes. In short, we want an orderly and easy-to-maintain structure.

This code creates a suggested scaffolding:

project_name
|
|__project_name (Put all the settings for the app here.)
|    |__ __init__.py
|    |__settings
|          |__ __init__.py
|          |__ base.py
|          |__ development.py
|          |__ staging.py
|          |__ production.py
|          ...
|    |__wsgi.py
|    |__asgi.py(optional)
|    |__urls.py
|    ...
|
|__templates (These general templates are loaded last in the application.)
|
|__api
|  |__ __init__.py
|  ...
|  |__ urls.py (Use this file to route the apps located under the applications folder.)
|
|__applications
|   |__ __init__.py
|   |__app1
|    ... |__models.py (Split the folder and import models in __init__ if necessary.)
|        |__views.py
|        |__serializers.py
|        |__urls.py
|        |__test
|        |   ...  
|        |__templates (Optional and not present if the project is API only.)
|        |
|        |__api.py (Optional and not present if the project is API only.)
|        ... (Add more files if they’re needed.)
|
|__utils
|   |__util_1.py
|   |__util_2.py
...

This structure isn’t mandatory, but it’s a good way to increase maintainability. In my suggested scaffolding, there’s a special app, named api. In api, I define the routing of the other apps in the applications folder. In the utils folder, I also define the project’s general functions and utilities.

Main concepts of Django and Django REST

To work with Django and Django REST, it’s important to understand these concepts:

Model classes provide an object-relational map (ORM) for the underlying database. A model is mapped to a table in the database. You can query the databases without any SQL programming. With models, it’s easy to define tables and relationships between them.
Views are in charge of the process of requests. They function as controllers. You can implement them in many ways, such as functions or classes.
Serializer classes provide control of data types and structures of requests and responses. I like to define them as interfaces for the backend of the web app.
Templates are files with static and dynamic content. They’re made up of some static code and other elements that depend on the context.

There are many ways to implement the functionalities of a REST API by using the Django frameworks. There are also many ways to implement views and serializers. For general design and any bugs you might run into, I recommend the generic tested and maintained solutions, such as Generic views.

Generic views

The Django REST framework has generic views that perform common tasks related to the model instances. Some examples are retrieve information, create, destroy, list, and update. With Django’s generic views, you only need to define the class view as a child of the generic view, depending on your needs. The rest is solved by the framework.
To learn how to implement the generic view functionality to generate a list of instances for a given model, let’s look at these classes:

from django.db import models

class Person(models.Model):
    name = models.CharField(max_length=50)
    age = models.IntegerField()

Now we create a serializer to see the details of an instance of that model:

from rest_framework import serializers
from applications.people.models import Person

class PersonDetailSerializer(serializers.ModelSerializer):
    class Meta:
        model = Person
        fields = ('name', 'age')

Next, we define a view that lists all the instances:

from rest_framework import generics
from applications.people.serializers import PersonDetailSerializer

class PeopleListView(generics.ListCreateAPIView):
    queryset = Person.objects.all()
    serializer_class = PersonDetailSerializer

And that's it! It only takes a few lines to create generic view functionality. In the queryset attribute, we tell the view which instances we want to list. In serializer_class, we select the data structure. Everything else is already solved.

Customize functions

The many generic views in the Django documentation meet a wide range of functionality needs. Some examples are CreateAPIView, ListAPIView, RetrieveAPIView, and RetrieveDestroyAPIView. But what if you want to customize the functions and change the default behavior? You can redefine get, post, put, delete, and other methods. You can also use the mixins to redefine the methods and specific functions like create, update, destroy, and others. The generic views and serializers aren’t required, but they can simplify a lot work. Serializers also cover some issues related to requests and responses, so we don't have to worry about them.

Data validation

The next good practice I want to share is how to locate the validation of incoming data. For example, you might have values for the fields of serializers in your web app. You want to run some checks on them when a new request comes into the REST API. In the PersonDetailSerializer, you can do a field-level validation that just defines a function named validate_<field_name>.

Let's add a function in the serializer:

pythonclass PersonDetailSerializer(serializers.ModelSerializer):

    def validate_age(self, age):
        if age < 0:
            raise serializers.ValidationError('Age must be a positive number.')
        return age    

    class Meta:
        ...

Complex validations

Sometimes you need to define a more complex validation inside the serializer class. An example is validating constraints between fields. A good way to do that is with the validate function: validate(self, data).
In summary, serializers are a good way to validate incoming data. In the previous example, PersonDetailSerializer is attached to the Person model. So by default, it does basic validation related to the model. For example, if the incoming name length is bigger than 50 characters, then the serializer raises an exception, because the maximum length was defined in the model. You don’t need to make that explicit in the serializer. With Django, you can also define serializers that aren’t attached to models, nest serializers and relationships between them, and more. Serializers are very powerful.

Templates

Finally, let’s talk about templates. As I mentioned, they’re great for generating dynamic code in Django. For example, template files can be used by the project frontend or to send emails.

Tip: Be aware of the order of apps defined in the project configuration, INSTALLED_APPS list.

It’s important to take order into account because it’s common for several apps to provide different versions of the same resource like a template or static file. If that happens, Django’s default behavior is to first check in the templates folder of the first app in the list. But that can cause problems. Order matters. By knowing this, we can change the order of the INSTALLED_APPS according to our needs. Or we can just explicitly define the desired paths for specific resources. For a configuration example, read the Django docs.

Session management

Most web applications have users and authentication methods. In the spirit of don't reinvent the wheel, I recommend using an existing customizable solution for session management: django-rest-auth. You can easily integrate this set of REST API endpoints with your project. It resolves user registration, sign in, and sign out, as well as social account integration. Just use the provided endpoints and customize the models and functions to adapt them to your needs. For more information, see the Django REST authentication docs.

Create efficient programs

No matter what technology you use, it’s always a good practice to create efficient programs.

Django has a database-abstraction API to makes queries easily. As an example, let’s take another look at our Person model. First, we query the database for instances of people who are over 18 years old with this code:

over_18_people = Person.objects.filter(age__gte=18)

As its name suggests, the filter function filters the query according to our criteria. __gte means greater than or equal to.

Now imagine that we also have people’s pets in our database:

from django.db import models
from applications.people.models import Person

class Pet(models.Model):
    owner = models.ForeignKey(Person, related_name='pets', on_delete=models.CASCADE)
    name = models.CharField(max_length=30)

ForeignKey defines a relationship between Person and Pet. A pet can belong to one person, and one person can have many pets. We might want to print the name of each person and the information about each person’s pets for everyone in the system. This example shows an intuitive way to do that:

for person in Person.objects.all():
    print(person.name)
    print('pets info:')
    for pet in person.pets.all():
        print(pet.name)
    print('---')

This solution works well, but how many times do the queries have to run? The answer is N + 1 times: once to get all the people and N times to get each person’s pets, assuming N is the number of people in the database. If the value of N is very large, that makes a lot of queries.
Let’s look at another solution to the same problem:

for person in Person.objects.all().prefetch_related('pets'):
    print(person.name)
    print('pets info:')
    for pet in person.pets:
        print(pet.name)
    print('---')

This code also solves the problem, but the number of times it accesses the database is 1. That’s a big difference.

Tip: Minimize the number of queries.

Finally, let’s imagine your database is in another server, and a single query takes 1 ms to run. If there are 10,000 people in it, then the solution in the first example solution would take about 10 seconds to get things done. That’s much longer than the 1 ms it would take with second example.

To save time and resources, write code that minimizes the number of accesses to the database. The Django API has lots of functions that do the exact same thing in lots of different ways. Choose the ones that minimize the number of queries because it’s always faster to work in memory. Learn to write efficient code by exploring Django’s documentation about attributes and functions.

The importance of testing

I won’t say that testing is a good practice because it’s way more than that. Testing is essential for every project. Every application needs testing, and I take that for granted.

Tip: Use Factory boy and Faker.

In most of the cases, you’ll need to generate initial data for testing. So a programmer might run tests that have an “initial system” with previously loaded instances and relationships between them. This improves the quality, maintainability, and velocity of testing.
You can load initial data with Django, using fixtures, migrations, and other functionalities. Now let’s look at a JSON fixture that takes the Person class:

[
  {
    "model": "applications.person",
    "pk": 1,
    "fields": {
      "name": "Rick Sanchez",
      "age": 70
    }
  },
  {
    "model": "applications.person",
    "pk": 2,
    "fields": {
      "name": "Summer Smith",
      "age": 17
    }
  }
]

By defining this fixture, we provide two instances at the beginning of the tests, and we generate new test cases with this as starting point. What if we want to test a bigger list of people? It would be cumbersome to enter info for each person manually, and the fixture file would need a huge number of lines. Fortunately, we don’t have to do that because the named tool Factory boy gives us a better way.

Factory boy

With Factory boy, you can generate test data with a just few lines, which improves maintainability. You can also define classes attached to the model classes. So at the same time that you run tests, you can easily create instances in memory or in the database and also replace any static hard-to-maintain fixtures. This example is related to the Person class:

import factory
from applications.people.models import Person

class PersonFactory(factory.DjangoModelFactory):
    name = factory.Sequence(lambda n: 'John Doe{0}'.format(n))
    age = 25

    class Meta:
        model = Person

To generate 100 instances of people for my tests with this definition, I run this command:

PersonFactory.create_batch(100)

As you can see, it’s very simple. Whit the name defined as Sequence(lambda n: 'John Doe{0}'.format(n)), the tool generates users with the names John Doe0, John Doe1, John Doe2, and so on.
Faker
The Faker tool is a package that generates fake random and realistic data. When you integrate it with Factory boy, it generates more significant tests with more realistic initial data. To see this integration in action, let’s change the factory class:

import factory
from applications.people.models import Person

class PersonFactory(factory.Factory):
    name = factory.Faker('name')
    age = 25

    class Meta:
        model = Person

Now, if we execute the create_batch(100) function, we generate 100 people with realistic random names. Faker has a pyint function that generates random integers for age. To learn more about this package, read the Faker docs.

Tip: Use tools to replace static fixtures.

Define good initial data for testing and keep it maintainable. For more ways to integrate these tools with Django, take a look at the common recipes in the Factory boy documentation.

What to avoid

Excessive use of signals

Django and Django REST have many tools and methods to attack several problems in a lot of ways. Signals are a good tool but they can cause problems if they’re used the wrong way.

Signals implement the observer design pattern, and it’s important to understand these concepts:

A Signal is an element that corresponds to an event. A signal can send notifications about the related event to it. In the observer pattern, the signal corresponds to the subject.
Receivers are the callables connected to a given signal. In the observer pattern, they correspond to the observer. After the associated event occurs and triggers the signal, each receiver is notified by that signal.

With the signals tool, you can send a notification after some predefined event. Some examples are when a new object in a model is created or destroyed, or a change is made in a relationship between models. To explore all the functionalities of signals, read the Django signal documentation.

Tip: Be careful with signals.

Overuse of signals often generates mysterious side effects that degrade maintainability. These problems happen because signals are triggered after selected events. Programmers who forget that or simply don’t know it sometimes run unwanted code in the wrong places.

Whenever you change something about the events that trigger a signal, you might need to make changes in the corresponding signals. It’s easy to forget to check that because these signals are defined in other files.

To save time and effort, avoid the overuse of signals. For example, you might have two apps A and B, where A needs to trigger a function in B. If app A already knows app B, then don’t use signals. In this case, app A will import the needed functions from app B and make the calls directly.

When signals are useful

A good use for signals is for breaking circular dependencies between Django apps. For example, you might have app A that depends on another app B, where B also needs things from A. A signal can break that circular dependency. Another example is if you have a model of third-party apps that you have no control over, and you want to trigger a process after some event of the model. Signals are also helpful in that scenario.

url versus path

Tip: Don't define urls with regular expressions unless is neccesary.

If you search internet for help to code in Django, you’ll probably find example code with the url function. And you might try to use it to solve your problem.

For an example of how the url function works, let’s define a URL with this general form:

api/v1/people/<token>

Where <token> can be a string with any character. Whit the named function would be:

url(r'^api/v1/people/(?P<token>[\w\-]+)/$', …)

It’s much easier to define the URL by using a Django dispatcher named path:

path('api/v1/people/<str:token>', …)

The difference between these methods is that the first one needs regular expressions. The second one has defined types that already cover the most common cases. At times, you might need to use regular expressions anyway. There’s a function for that named re_path. You can learn more about URL dispatcher in the Django documentations.

Learning Django

In this blog, I shared many of the lessons I learned by programming with the Django frameworks. I also recommended good practices for developing web apps, structures, functionalities, and general programming. This information can help you improve the quality and maintainability of your code. To make things easier for you, I described some of the problems I found with these tools and provided solutions for them.

I talked first about the importance of a good structure and configuration for several environments. Then main concepts of Django related to easy ways of implementing functionalities, taking advantage of tools already implemented to solve general problems. After that, I recommended some practices for efficiency, and then APIs and packages for session management and testing. Finally I talked about things that are better to avoid, because I already faced those problems before and I would like other people could skip them while learning these tools. I still have a lot to learn about Django and Django REST framework, but I hope what I’ve learned so far and shared with you has been useful and interesting.