DEV Community: Michal Dobrzycki

Create proper REST API with Django, DRF, JWT and OpenApi - testing

Michal Dobrzycki — Sat, 26 Sep 2020 15:42:18 +0000

This is a continuation of previous post

Summary of this post.

Install pytest and configure it for Django project
Write a test that checks Profile and User count (it should be the same)
Write the test that checks if the User can edit his Profile
Write the test that checks if the User cannot edit other Profile than his own.
Fix bugs found by our tests
Create Github Actions to validate every pull request to master branch
Play with the builds in Github Actions

Step #1 - install pytest and configure it

I'll assume for this article, that you have a good knowledge of pytest. If not, please read first this article

Let's start with writing tests for our REST API. First, we need to install pytest-django (wrapper for pytest, which will help us to write more complex tests much easier), install it with the command:

pip install pytest-django

Then we need to create pytest.ini file which will be used by pytest as a configuration to run tests in our application

[pytest]
DJANGO_SETTINGS_MODULE = restapi_article.settings
python_files = tests.py test_*.py *_tests.py

Step 2 - Write a test that checks Profile and User count

Now we can start writing tests. Because in the previous article we used signals to create Profile along with User, let's write a test that will check the number of Users and Profiles in DB is the same (we want to create a new user and then see that amount of profiles is the same).

@pytest.mark.django_db decorator will allow us to use clean DB (right now we are using default SQLite) in our tests. So running pytest will use our settings.py to use configured
DB, and after the test, everything will be reverted back to the state before tests were executed. So first easy test will look like this

import pytest

from django.contrib.auth.models import User
from .models import Profile


@pytest.mark.django_db
def test_user_create_creates_profile():
    User.objects.create_user('michal', 'test@scvconsultants.com', 'michalpassword')
    assert Profile.objects.count() == 1
    assert User.objects.count() == 1

Step 3 - Write the test that checks if the User can edit his Profile

Now we should test that the user can update his (and only his) profile. So let's write a test (I'll extend the previous test to test Profile). The modified test case will check that:

Creating User creates also Profile in DB
User field in created Profile will link to the correct User URL
Next we'll obtain the token for the created user and save it in the script to authorize the next API call.
With this token we'll populate Profile with some data.
And check that response from PUT API call contains the same data that was sent in the previous request.

First, we need to add a fixture to use ApiClient from rest_framework.test. Let's add this to tests.py

@pytest.fixture
def api_client():
    from rest_framework.test import APIClient
    return APIClient()

And now we can use this fixture and extend the previous test. Please notice that i've changed test name, but it still checks the number of users:

@pytest.mark.django_db
def test_user_can_update_his_profile(api_client):
    user = User.objects.create_user('michal', 'test@scvconsultants.com', "michalpassword")
    assert Profile.objects.count() == 1
    assert User.objects.count() == 1

    profile_url = reverse('profile-detail', args=[user.id])
    user_url = reverse('user-detail', args=[user.id])

    # check that profile was created for created user
    response = api_client.get(profile_url)
    assert response.status_code == 200
    assert response.data['user'].endswith(user_url)

    #create bio
    bio_data = {
        "user": response.data['user'],
        "bio": "This is test user",
        "location": "Wroclaw"
    }
    #create login data as user.password contains now encrypted string
    login_data = {
        "username": user.username,
        "password": "michalpassword"
    }

    # get token
    token_url = reverse('token_obtain_pair')
    token = api_client.post(token_url, login_data, format='json')
    # check that access token was sent in response
    assert token.data['access'] is not None
    # add http authorization header with Bearer prefix
    api_client.credentials(HTTP_AUTHORIZATION='Bearer ' + token.data['access'])
    # update profile
    response = api_client.put(profile_url, bio_data, format='json')
    # validate response
    assert response.status_code == 200
    assert response.data['bio'] == bio_data['bio']
    assert response.data['location'] == bio_data['location']

Now we can run tests by typing pytest in our project root in a terminal. Our test should pass without any errors.

Step 4 - Write the test that checks if the User cannot edit other Profile than his own.

Until now we don't see any issues with our application. But let's try to populate other profile than our user's in the next test. We will expect to see error 403 when updating other profile with the wrong JWT Token. Changing someone else Profile shouldn't be possible, but...

@pytest.mark.django_db
def test_user_should_not_be_able_to_update_other_profile(api_client):
    first_user = User.objects.create_user('michal', 'test@scvconsultants.com', "michalpassword")
    second_user = User.objects.create_user('michal2', 'test2@scvconsultants.com', "michalpassword2")
    assert Profile.objects.count() == User.objects.count()

    #get token for first_user
    token_url = reverse('token_obtain_pair')
    login_data = {
        "username": first_user.username,
        "password": "michalpassword"
    }
    token = api_client.post(token_url, login_data, format='json')
    api_client.credentials(HTTP_AUTHORIZATION='Bearer ' + token.data['access'])
    # now update second_user Profile with first_user token
    profile_url = reverse('profile-detail', args=[second_user.id])
    response = api_client.get(profile_url)
    bio_data = {
        "user": response.data['user'],
        "bio": "This is test user",
        "location": "Wroclaw"
    }
    response = api_client.put(profile_url, bio_data, format='json')
    assert response.status_code == 403

Let's run those tests again with pytest and now we should see that the second test fails because we are able to update other Profile than ours!

FAILED restapi/tests.py::test_user_should_not_be_able_to_update_other_profile - assert 200 == 403

Step 5 - Fix bugs found by our tests

We need to fix this bug in our code (and it's a security bug). First, let's add permissions.py file and create proper permission logic there:

from rest_framework import permissions


class IsProperUserOrReadOnly(permissions.BasePermission):
    """
    Custom permission to only allow owners of an object to edit it.
    """
    def has_object_permission(self, request, view, obj):
        # Read permissions are allowed to any request,
        # so we'll always allow GET, HEAD or OPTIONS requests.
        if request.method in permissions.SAFE_METHODS:
            return True

        # Write permissions are only allowed to the User linked with the Profile.
        return obj.user == request.user

Then we can change ProfileViewSet in our views.py to include this logic. Don't forget to import newly created permission:

from .permissions import IsProperUserOrReadOnly
# some code here
permission_classes = [permissions.IsAuthenticatedOrReadOnly, IsProperUserOrReadOnly]

Now we can run again tests with pytest and we should have 2 passing tests! It's that easy :)

Step 6 - Create Github Actions to validate every pull request to `master` branch

Once the tests are passing, it's a good time to set up GitHub and GitHub actions. Let's block our repository from pushing anything straight to the master branch.

We can go to settings -> branches and add branch policy for starters. Type "master" into branch name pattern and select "Require pull request reviews before merging".

Then we go to Actions in our Github repo, and add Python application which should be suggested on the first page.

It will add yaml file python-app.yml in directory .github/workflows/. Our file should look like this:

# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: Python application

on:
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.8
      uses: actions/setup-python@v2
      with:
        python-version: 3.8
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install flake8 pytest
        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
    - name: Lint with flake8
      run: |
        # stop the build if there are Python syntax errors or undefined names
        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
    - name: Test with pytest
      run: |
        pytest

Step 8 - Play with the builds in Github Actions

Let's break our tests on a locally created branch, so do the following:

git pull
git checkout -b feature/broken-test

And change last line in tests.py from 403 to 200

assert response.status_code == 200

Now we can add, commit, and push the code and create a pull request to trigger Github Action. After pushing this code to branch, create Pull Request from the Github repository page.

And you should see in your GitHub actions Error on step "Test with pytest"

And the log should show something like this:

>       assert response.status_code == 200
E       assert 403 == 200
E        +  where 403 = <Response status_code=403, "application/json">.status_code

restapi/tests.py:79: AssertionError
------------------------------ Captured log call -------------------------------
WARNING  django.request:log.py:224 Forbidden: /api/v1/profile/2/

Those error will populate to Pull Request view and show you "All checks failed". This is a pretty good warning for you not to merge changes to master branch.

Let's fix the last line in tests.py to expect status 403 again and push the code again. And we should see now that the tests are passing in Actions and now we can safely merge this to master (pull request will also update it's view).

Coming next:

Deployment to Heroku
Dockerization

Create proper REST API with Django, DRF, JWT and OpenApi

Michal Dobrzycki — Sun, 06 Sep 2020 18:26:13 +0000

Summary of this post.

In this post I’ll introduce following concepts, feel free to skip to the next post, if you are familiar with them already:

Create Django project with Django Rest Framework
Add OpenApi specs to generate dynamically API documentation
Use JWT (JSON Web Token) for authentication and authorization

Step #1 - create new project

Start a new project (either with PyCharm -> new Django project or django-admin startproject restapi_article). I won't cover basics, because You can always go to Django Tutorial and learn from official documentation (which I find the best at the time of writing this article).

Once we have Django project set up we need to install python libraries below (for best experience You should use virtual environment):

Django
djangorestframework
drf-yasg
djangorestframework-simplejwt

It's always good to conform to standards, so we should create requirements.txt file with all libraries. You can create it by using this command in the project root folder pip freeze > requirements.txt. The file should look like this:

asgiref==3.2.10
certifi==2020.6.20
chardet==3.0.4
coreapi==2.3.3
coreschema==0.0.4
Django==3.1.1
djangorestframework==3.11.1
djangorestframework-simplejwt==4.4.0
drf-yasg==1.17.1
idna==2.10
inflection==0.5.1
itypes==1.2.0
Jinja2==2.11.2
MarkupSafe==1.1.1
packaging==20.4
PyJWT==1.7.1
pyparsing==2.4.7
pytz==2020.1
requests==2.24.0
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
six==1.15.0
sqlparse==0.3.1
uritemplate==3.0.1
urllib3==1.25.10

Next create an application with command python3 manage.py startapp restapi

Your file structure after creation should look like this:

. 
├── restapi
│   ├── migrations
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── restapi_article
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── venv
├── db.sqlite3
├── manage.py
└── requirements.txt

And add to our restapi_article\settings.py those three apps

INSTALLED_APPS = [
    ...
    'rest_framework',
    'drf_yasg',
    'restapi',
]

Edit urls.py to use root restapi.urls as main API application urls, this way we will have localhost:8000/admin to play with Django admin panel, and /api/v1 endpoint as prefix for all urls (which is a really good way to version your API, as it might easily change in the future)

from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/v1/', include('restapi.urls')),
]

Step 2 - create data models

I want to have a Drinking Diary application, where I can select a day and then post a note for myself. Also, I want to have a user profile with Bio field, location and date of Birth.

Profile and Day will be linked as Many-To-Many relationship through the DrinkingDay model where I can post a note for a given day (what I drank and with whom for example).

TODO (future): create an Event model, where you can ask some people to join you in drinking. It will have to combine multiple Profiles/Users with one Date

We will also use signals to create hooks for User creation, so we need only to create or update the Profile model, and User will be created/updated automatically with one request.

models.py

from django.contrib.auth.models import User
from django.db import models
from django.db.models.signals import post_save
from django.dispatch import receiver


class Day(models.Model):
    day = models.DateField()


class Profile(models.Model):
    user = models.OneToOneField(User, on_delete=models.CASCADE)
    bio = models.TextField(max_length=500, blank=True)
    location = models.CharField(max_length=30, blank=True)
    birth_date = models.DateField(null=True, blank=True)
    days = models.ManyToManyField(Day, through='DrinkingDay')


class DrinkingDay(models.Model):
    profile = models.ForeignKey(Profile, on_delete=models.CASCADE)
    day = models.ForeignKey(Day, on_delete=models.CASCADE)
    note = models.TextField(max_length=1000, blank=True)

# Use signals so our Profile model will be automatically created
# or updated when we create or update User instance
# https://docs.djangoproject.com/en/3.1/topics/signals/
@receiver(post_save, sender=User)
def create_user_profile(sender, instance, created, **kwargs):
    if created:
        Profile.objects.create(user=instance)


@receiver(post_save, sender=User)
def save_user_profile(sender, instance, **kwargs):
    instance.profile.save()

Now let's create and apply migrations for DB and then create superuser to be able to login to our admin panel. You need to run those three commands one by one (you can change email and username of course)

python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser --email admin@example.com --username admin

Step 3 - create serializers

Now we need to add serializers. Serializer in shortcut is a way to create JSON from an object.
Let's start writing serializers for User and Profile model (we need to do them both, as it won't work without User):

serializers.py

from django.contrib.auth.models import User
from rest_framework import serializers

from restapi.models import Profile


class UserSerializer(serializers.HyperlinkedModelSerializer):
    class Meta:
        model = User
        fields = ['url', 'username', 'email', 'groups']


class ProfileSerializer(serializers.HyperlinkedModelSerializer):
    class Meta:
        model = Profile
        fields = ['user', 'bio', 'location', 'birth_date', 'days']

Step 4 - create viewsets

Now that we have serializers in place, we can create Views in Django. So at every url we will see what we want. Let's do that for both UserView and ProfileView (although here we could omit User view, as it's not required.

Change views.py (in restapi folder):

from django.contrib.auth.models import User
from rest_framework import viewsets

from restapi.models import Profile
from restapi.serializers import ProfileSerializer, UserSerializer


class UserViewSet(viewsets.ModelViewSet):
    """
    API endpoint that allows users to be viewed or edited.
    """
    queryset = User.objects.all().order_by('-date_joined')
    serializer_class = UserSerializer


class ProfileViewSet(viewsets.ModelViewSet):
    """
    API endpoint that allows profiles to be viewed or edited.
    """
    queryset = Profile.objects.all()
    serializer_class = ProfileSerializer

Step 5 - register router for Viewsets

DRF allows us to use something called routers. This will handle automatic URL routing to Django for us.

Change urls.py (in restapi folder):

from django.urls import path, include
from rest_framework import routers

from restapi import views

router = routers.DefaultRouter()
router.register(r'profile', views.ProfileViewSet)
router.register(r'user', views.UserViewSet)

urlpatterns = [
    path('api-auth/', include('rest_framework.urls', namespace='rest_framework')),
    path('', include(router.urls)),
]

Now we can run our app with python manage.py runserver and go to http://127.0.0.1:8000/api/v1/ to see that our application is working :)

If you created a superuser, that should automatically create Profile for him, so going to http://127.0.0.1:8000/api/v1/profile/1/ should allow you to update your superuser Profile.

Step 6 - add Swagger documentation

Everything is already good to go, we just need to create URL for showing Swagger documentation. Also, it's a good practice to create schema_view to describe our API for other developers.

Update urls.py (in restapi folder):

from django.urls import path, include, re_path
from drf_yasg import openapi
from drf_yasg.views import get_schema_view
from rest_framework import routers, permissions

from restapi import views

router = routers.DefaultRouter()
router.register(r'profile', views.ProfileViewSet)
router.register(r'user', views.UserViewSet)

schema_view = get_schema_view(
   openapi.Info(
      title="Drinking Day API",
      default_version='v1',
      description="This API allows us to keep a diary of our daily drinking",
      terms_of_service="https://www.scvconsultants.com",
      contact=openapi.Contact(email="michal@scvconsultants.com"),
      license=openapi.License(name="MIT License"),
   ),
   public=True,
   permission_classes=(permissions.AllowAny,),
)

urlpatterns = [
    path('api-auth/', include('rest_framework.urls', namespace='rest_framework')),
    path('', include(router.urls)),
    re_path(r'^swagger(?P<format>\.json|\.yaml)$', schema_view.without_ui(cache_timeout=0), name='schema-json'),
    path('swagger/', schema_view.with_ui('swagger', cache_timeout=0), name='schema-swagger-ui'),
]

Now we should be able to see Swagger UI at:
http://127.0.0.1:8000/api/v1/swagger/
and .json + .yaml schema at links below:
http://127.0.0.1:8000/api/v1/swagger.json
http://127.0.0.1:8000/api/v1/swagger.yaml

Step 7 - Add JWT tokens to limit access to Profile model.

First, we need to update settings.py to add JWT auth classes. I also add pagination as this is a great practice to use default page size for collections (we don't need to send everything at once, right?).

# REST FRAMEWORK config
REST_FRAMEWORK = {
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination',
    'PAGE_SIZE': 25,
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework_simplejwt.authentication.JWTAuthentication',
    )
}

Then we need to create an endpoint for grabbing and refreshing Token (as tokens should expire due to security reasons). You can set up it yourself with the official documentation.

Let's add it in urls.py

from rest_framework_simplejwt.views import (
    TokenObtainPairView,
    TokenRefreshView,
)

urlpatterns = [
    ...
    path('api/token/', TokenObtainPairView.as_view(), name='token_obtain_pair'),
    path('api/token/refresh/', TokenRefreshView.as_view(), name='token_refresh'),
    ...
]

Now you can go to: http://127.0.0.1:8000/api/v1/token/ and send your superuser credentials as a POST and you should get a beautiful working token, which you can decode and verify here and it should decode to user_id that corresponds with your credentials (user_id will be probably 1).

Adding now permissions to our views is really easy, first import in views.py

from rest_framework import permissions

And then in our specific view (let's use ProfileViewSet for this) add:

class ProfileViewSet(viewsets.ModelViewSet, mixins.UpdateModelMixin):
    ...
    permission_classes = [permissions.IsAuthenticatedOrReadOnly]

Now we can easily add users, but we can't update our Profile unless we are providing proper JWT Token. The whole permissions list is available in Rest Framework documentation

Coming next:

Creating CI/CD with usage of Github and Heroku.
Dockerization
Integration tests for API with pytest

DEV Community: Michal Dobrzycki

Create proper REST API with Django, DRF, JWT and OpenApi - testing

Summary of this post.

Step #1 - install pytest and configure it

Step 2 - Write a test that checks Profile and User count

Step 3 - Write the test that checks if the User can edit his Profile

Step 4 - Write the test that checks if the User cannot edit other Profile than his own.

Step 5 - Fix bugs found by our tests

Step 6 - Create Github Actions to validate every pull request to master branch

Step 8 - Play with the builds in Github Actions

Coming next:

Create proper REST API with Django, DRF, JWT and OpenApi

Summary of this post.

Step #1 - create new project

Step 2 - create data models

Step 3 - create serializers

Step 4 - create viewsets

Step 5 - register router for Viewsets

Step 6 - add Swagger documentation

Step 7 - Add JWT tokens to limit access to Profile model.

Coming next:

Step 6 - Create Github Actions to validate every pull request to `master` branch