Efficient API development with Python3 and Django REST framework

Tanner Burns — Wed, 14 Oct 2020 16:47:55 +0000

The goal of this tutorial is to build an api to create and list subscribers.
In this tutorial we will inspect the data, create models, serializers, and views, time the difference between create and bulk create, and learn how to filter a queryset.
Well-known libraries we will use include Django, and Django REST framework.

Installing libraries
Creating a Django project and app
Project structure
Understanding the data
Creating models
Creating serializers
Creating views
Creating routes (urls)
Adding a command for test data
Creating a bulk serializer
Queryset filtering

Installing Django and Django REST framework

pip3 install django djangorestframework

Creating a Django Project and Django App

Create a working directory for the django project

mkdir django_api_tutorial && cd django_api_tutorial

Create the django project
```
django-admin startproject api .
```
Create the django app for subscribers
```
python3 manage.py startapp subscribers
```

Connect subscribers and rest_framework app to the API settings in api/settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',         # new (djangorestframework)
    'subscribers'             # new
]

Add the following to the bottom of the settings for paged list results

REST_FRAMEWORK = {
   'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination',
   'PAGE_SIZE': 32
}

Project structure

./django_api_tutorial
│   manage.py
│   README.md
│
├───api
│       asgi.py
│       settings.py
│       urls.py
│       wsgi.py
│       __init__.py
│
├───data
│       fake_users.csv
│
└───subscribers
    │   admin.py
    │   apps.py
    │   models.py
    │   serializers.py
    │   tests.py
    │   urls.py
    │   views.py
    │   __init__.py
    │
    ├───management
    │   └───commands
    │           bulktestdata.py
    │           testdata.py
    │
    └───migrations
            __init__.py

Learn more about the django project structure here.

Understanding the data

Here are the first 10 rows of the fake data for the tutorial. This data was created using Mockaroo.

first_name	last_name	email	gender	city	state
Mohammed	Poad	mpoad0@cisco.com	Male	Watertown	Massachusetts
Briana	Liddall	bliddall1@odnoklassniki.ru	Female	Indianapolis	Indiana
Jodie	Pattington	jpattington2@telegraph.co.uk	Male	Brockton	Massachusetts
Cari	Worcs	cworcs3@youku.com	Female	Richmond	Virginia
Shane	Pickford	spickford4@arstechnica.com	Male	Newark	New Jersey
Bethany	McColm	bmccolm5@comsenz.com	Female	Evansville	Indiana
Elsi	Wyrill	ewyrill6@opera.com	Female	Lexington	Kentucky
Kylynn	Hartill	khartill7@webeden.co.uk	Female	Columbia	South Carolina
Lonnie	Elliot	lelliot8@msn.com	Male	Joliet	Illinois
Kellyann	Kelso	kkelso9@sbwire.com	Female	San Bernardino	California

Now we will break down each one of these columns to their corresponding data type.

first_name      VARCHAR(64)
last_name       VARCHAR(64)
email           TEXT
gender          VARCHAR(8)
city            VARCHAR(256)
state           VARCHAR(24)

Knowing how we want to handle each of these variables is going to enable us to easily create models and serializers.

Creating models

The model definitions can be found in subscribers/models.py.

from django.db import models

class Location(models.Model):
    city = models.CharField(null=False, max_length=256)
    state = models.CharField(null=False, max_length=64)

class Subscriber(models.Model):
    first_name = models.CharField(null=False, max_length=64)
    last_name = models.CharField(null=False, max_length=64)
    email = models.TextField()
    gender = models.CharField(null=False, max_length=8)
    location = models.ForeignKey(Location, related_name='subscriber_location', on_delete=models.DO_NOTHING)

Above, we defined the models for both a Subscriber and Location.
The Subscriber model is a Many to One relationship with the Location model since we can have many subscribers in one location.

Creating serializers

We will now define serializers for the models. This allows us to easily convert the model objects into json objects for API responses.
When we create serializers, we can add a lot of functionality to the API easily with Django REST framework.

The serializers are defined in subscribers/serializers.py. Notice that in the SubscriberSerializer we have to overwrite the create method.
This is normal when you are using relational models.

from rest_framework import serializers

class LocationSerializer(serializers.Serializer):
    city = serializers.CharField(required=True, max_length=256)
    state = serializers.CharField(required=True, max_length=64)

    class Meta:
        fields = ['city', 'state']

class SubscriberSerializer(serializers.Serializer):
    id = serializers.IntegerField(required=False)
    created = serializers.DateTimeField(required=False)
    first_name = serializers.CharField(required=True, max_length=64)
    last_name = serializers.CharField(required=True, max_length=64)
    email = serializers.CharField(required=False)
    gender = serializers.CharField(required=True, max_length=8)
    location = LocationSerializer(required=True)

    class Meta:
        fields = ['first_name', 'last_name', 'email', 'gender', 'location']
        read_only_fields = ['id', 'created']

    def create(self, validated_data):
        # remove location from serialized data and add model object
        location = validated_data.pop('location')
        city = location.get('city', None)
        state = location.get('state', None)

        if not city and not state:
            raise serializers.ValidationError('No location input found')

        # call get or create to reuse location objects
        location_obj = Location.objects.get_or_create(city=city, state=state)[0]
        # add location back to validated data
        validated_data.update({'location': location_obj})

        # unpack validated_data to create a new Subscriber object
        return Subscriber.objects.create(**validated_data)

We have now defined the serializers so that we can easily do things like create, read, update, delete and list objects from the models.
We will see this in action when we implement the Views.

Creating views

The views are defined in subscribers/views.py and contain the functionality that will be available to users of the API.
In this tutorial we will focus on being able to create and list subscribers, but along with adding data we will also likely need a way to easily remove data.
I will give an example how to easily add a delete operation to the API using Django REST framework mixins below.

from rest_framework import viewsets, mixins

from .models import Subscriber
from .serializers import SubscriberSerializer

class SubscriberView(viewsets.GenericViewSet, mixins.ListModelMixin, mixins.CreateModelMixin):
    queryset = Subscriber.objects.all()
    serializer_class = SubscriberSerializer

This simple view above provides a generic API interface with list and create functionality for Subscribers.
To easily add the delete method and functionality it would look like the following:

from rest_framework import viewsets, mixins

from .models import Subscriber
from .serializers import SubscriberSerializer

class SubscriberView(viewsets.GenericViewSet, mixins.ListModelMixin, mixins.CreateModelMixin,
                     mixins.DestroyModelMixin):
    queryset = Subscriber.objects.all()
    serializer_class = SubscriberSerializer

Notice the only new change was the addition of mixins.DestroyModelMixin in the class definition. More info about mixins can be found on the Django REST framework docs.

Creating routes

Now to be able to navigate to the API we will need to add the urls. The urls are defined in subscribers/urls.py.
We will also want to tell the base API where to find the routes from subscribers and this is defined in api/urls.py.

# api/urls.py
from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('tutorial/', include('subscribers.urls'))
]

# subscribers/urls.py
from rest_framework import routers

from .views import SubscriberView

router = routers.DefaultRouter(trailing_slash=False)
router.register(r'subscribers', SubscriberView, basename='subscribers')
urlpatterns = router.urls

In the subscribers urls we use a router from Django REST framework to easily add all of the functionality that we defined in the views.
The router with the viewset enabled the following routes for the API.

GET     /tutorial/subscribers       List view of subscribers
POST    /tutorial/subscribers       Create a new subscriber

Adding a command for test data

Now we will add a command to django to easily allow us to add test data to the API. The commands are defined in subscribers/management/commands and the first command is named testdata.py.

import csv

from time import time
from django.core.management.base import BaseCommand, CommandError

from subscribers.serializers import SubscriberSerializer

class Command(BaseCommand):
    help = 'Adds the fake test data to the API'

    def handle(self, *args, **options):
        try:
            with open('data/fake_users.csv', 'r') as fin:
                csvreader = csv.reader(fin)
                headers = next(csvreader)
                data = [{'first_name': row[0],
                         'last_name': row[1],
                         'email': row[2],
                         'gender': row[3],
                         'location': {'city': row[4], 'state': row[5]}
                         } for row in csvreader
                        ]
                # time how fast it takes to add all records 1 by 1
                start = time()
                for item in data:
                    serializer = SubscriberSerializer(data=item)
                    if serializer.is_valid():
                        serializer.create(item)
                stop = time()
                print(f'{len(data)} items added in {stop-start} seconds')
        except FileExistsError:
            raise CommandError('No testdata found')

This command will add test records to the API. It will also track how many records and how quickly they were added.
We will run the command and see what the output is.

python3 manage.py testdata

Output: 6000 items added in 31.6553955078125

Next we will implement a bulk serializer and add a new bulk command to see if we can speed up creating records.

Creating a bulk serializer

Instead of having to create objects one by one we will create a bulk serializer to create many at a time.

class BulkSubscriberSerializer(serializers.Serializer):
    subscribers = SubscriberSerializer(many=True)

    class Meta:
        fields = ['subscribers']

    def create(self, validated_data):
        # store the Subscriber objects to be created in bulk
        create_objects_list = []
        # iterate over the validated_data and add Subscriber objects to a list to be created
        for data in validated_data:
            # notice the same functionality from the regular serializer
            location = data.pop('location')
            city = location.get('city', None)
            state = location.get('state', None)
            location_obj = Location.objects.get_or_create(city=city, state=state)[0]
            # combine data and {'location': location_obj} and unpack to the Subscriber model
            create_objects_list.append(Subscriber(**{**data, **{'location': location_obj}}))
        return Subscriber.objects.bulk_create(create_objects_list)

We will also create a new command called bulktestdata that is defined in subscribers/management/commands/bulktestdata.py.
This will use the bulk serializer to add the records and track how long it takes.

import csv

from time import time
from django.core.management.base import BaseCommand, CommandError

from subscribers.serializers import BulkSubscriberSerializer

class Command(BaseCommand):
    help = 'Adds the fake test data to the API'

    def handle(self, *args, **options):
        try:
            with open('data/fake_users.csv', 'r') as fin:
                csvreader = csv.reader(fin)
                headers = next(csvreader)
                data = [{'first_name': row[0],
                         'last_name': row[1],
                         'email': row[2],
                         'gender': row[3],
                         'location': {'city': row[4], 'state': row[5]}
                         } for row in csvreader
                        ]
                # time how fast it takes to add records in bulk
                start = time()
                bulk_serializer = BulkSubscriberSerializer(data={'subscribers': data})
                if bulk_serializer.is_valid():
                    bulk_serializer.create(data)
                stop = time()
                print(f'{len(data)} items added in {stop-start} seconds')
        except FileExistsError:
            raise CommandError('No testdata found')

Now when we run the new command lets see how fast all of the records get added.

python3 manage.py bulktestdata

Output: 6000 items added in 5.3229029178619385 seconds

Lastly, we will update the views to use the regular or bulk serializer based on the data sent to the route.

from rest_framework import viewsets, mixins
from rest_framework.response import Response

from .models import Subscriber
from .serializers import SubscriberSerializer, BulkSubscriberSerializer

class SubscriberView(viewsets.GenericViewSet, mixins.ListModelMixin, mixins.CreateModelMixin,
                     mixins.DestroyModelMixin):
    queryset = Subscriber.objects.all()
    serializer_class = SubscriberSerializer

    def create(self, request, *args, **kwargs):
        # if the data is a dictionary, use parent create that relies on serializer_class
        if isinstance(request.data, dict):
            return super(SubscriberView, self).create(request, *args, **kwargs)
        # if the data is a list, send to the bulk serializer to handle creation
        elif isinstance(request.data, list):
            serializer = BulkSubscriberSerializer(data={'subscribers': request.data})
            if serializer.is_valid():
                serializer.create(request.data)
                return Response(serializer.data, status=201)
            else:
                return Response(serializer.errors, status=400)
        else:
            return Response('Invalid data received', status=400)

At this point, the API can now create one to many records at a time and allow users to browse the current subscribers.
Here is a snippet of a response from the API for a GET request to http://127.0.0.1:8000/tutorial/subscribers.

{
    "count": 6000,
    "next": "http://127.0.0.1:8000/tutorial/subscribers?page=2",
    "previous": null,
    "results": [
        {
            "id": 1,
            "created": "2020-10-13T15:51:50.850563Z",
            "first_name": "Mohammed",
            "last_name": "Poad",
            "email": "mpoad0@cisco.com",
            "gender": "Male",
            "location": {
                "city": "Watertown",
                "state": "Massachusetts"
            }
        },
        {
            "id": 2,
            "created": "2020-10-13T15:51:50.862560Z",
            "first_name": "Briana",
            "last_name": "Liddall",
            "email": "bliddall1@odnoklassniki.ru",
            "gender": "Female",
            "location": {
                "city": "Indianapolis",
                "state": "Indiana"
            }
        },
        ...
    ]
}

Queryset filtering

We can easily list all of the subscribers but what if we only want to see subscribers from a specific state?
Currently, as a user, we would have to pull all of the subscribers from the API and filter our own results.
This is where queryset filtering from Django can help give the users more control.
The user can send a query parameter in the request and we can use it to filter the results. The new view will look like the following.

from rest_framework import viewsets, mixins
from rest_framework.response import Response

from .models import Subscriber
from .serializers import SubscriberSerializer, BulkSubscriberSerializer

class SubscriberView(viewsets.GenericViewSet, mixins.ListModelMixin, mixins.CreateModelMixin,
                     mixins.DestroyModelMixin):
    serializer_class = SubscriberSerializer

    def get_queryset(self):
        queryset = Subscriber.objects
        if 'state' in self.request.query_params:
            queryset = queryset.filter(location__state__icontains=self.request.query_params['state'])
        return queryset.order_by('created')


    def create(self, request, *args, **kwargs):
        # if the data is a dictionary, use parent create that relies on serializer class
        if isinstance(request.data, dict):
            return super(SubscriberView, self).create(request, *args, **kwargs)
        # if the data is a list, send to the bulk serializer to handle creation
        elif isinstance(request.data, list):
            serializer = BulkSubscriberSerializer(data={'subscribers': request.data})
            if serializer.is_valid():
                serializer.create(request.data)
                return Response(serializer.data, status=201)
            else:
                return Response(serializer.errors, status=400)
        else:
            return Response('Invalid data received', status=400)

We have added the get_queryset method and can now send state as a query parameter on a GET request.
For example if we send a GET request to http://127.0.0.1:8000/tutorial/subscribers?state=Texas we can see that we have less total results.

{
    "count": 629,
    "next": "http://127.0.0.1:8000/tutorial/subscribers?page=2&state=Texas",
    "previous": null,
    "results": [
        {
            "id": 13,
            "created": "2020-10-13T19:51:29.461522Z",
            "first_name": "Laure",
            "last_name": "Chitter",
            "email": "lchitterc@t-online.de",
            "gender": "Female",
            "location": {
                "city": "Corpus Christi",
                "state": "Texas"
            }
        },
        ...
    ]
}

We now have an API that can create one to many subscribers based on the payload, list all subscribers, and list subscribers from a certain state.
Hope you enjoyed the tutorial, all the code can be found here.

How to speed up python3 code in one line

Tanner Burns — Sat, 03 Oct 2020 17:24:16 +0000

In this short article, we will use asnycio to speed up our code. We will use a function from the modutils library to show the effectiveness and ease of adding asnyc to your code.

Here is example code we might want to speed up. This is a basic example that is just meant to prove the point of asyncio. It will send a requests to google and return the response. We are going to do this 32 times and see how long it takes.

import requests

from time import time

def task(url):
    """ a simple task to return the response of a given url

    :param url: {str} -- url to send requests

    :return: Response object from requests
    """
    return requests.get(url, headers={'User-Agent':'Chrome; Python'})

# timing execution
start = time()
# create list to store Response objects
responses = []
for _ in range(0, 32):
    # add Response to list
    responses.append(task('https://www.google.com'))
print(f'{time()-start} to get all responses')

Output:

2.498640775680542 to get all responses

Now we will update this example to use asyncio to retrieve 16 responses at a time instead of 1. This is determined by the max_async_pool variable which we set to 16. This can be increased or decreased depending on desired speed and system resources.

import requests

from time import time
from modutils import aioloop


def task(url: str):
    """ a simple task to return the response of a given url

    :param url: {str} -- url to send requests

    :return: Response object from requests
    """
    return requests.get(url, headers={'User-Agent': 'Chrome; Python'})

# timing execution
start = time()
# create a list of arguments for the task function
args = [['https://www.google.com'] for _ in range(0, 32)]
# sending 16 requests at one time by setting max_async_pool to 16
# responses is a list of Response objects
responses = aioloop(task, args, max_async_pool=16)
print(f'{time()-start} to get all responses')

Output:

0.2680017948150635 to get all responses

Using the aioloop function from modutils we were able to speed up our code and make what took two seconds almost instant.

Note: For the aioloop function we created an args list that unpacks each inner item to the given function. We can also add named arguments with a dictionary in the list of args. Example below:

import requests

from time import time
from modutils import aioloop


def task(url: str, params: dict=None):
    """ a simple task to return the response of a given url

    :param url: {str} -- url to send requests
    :param params: {dict} -- optional named argument for requests parameters

    :return: Response object from requests
    """
    return requests.get(url, headers={'User-Agent': 'Chrome; Python'}, params=params)

# timing execution
start = time()
# create a list of arguments for the task function
args = [['https://www.google.com', {'params':{'q':'testing'}}] for _ in range(0, 32)]
# sending 16 requests at one time by setting max_async_pool to 16
# responses is a list of Response objects
responses = aioloop(task, args, max_async_pool=16)
print(f'{time()-start} to get all responses')

Learn more about modutils here.

DEV Community: Tanner Burns