Idempotence and POST Requests in Django

#webdev #django

I have a confession to make. I should have listened to all of the articles and StackOverflow posts. I shouldn't have been so naive. But, there I was, staring at my app's poorly formatted logs trying to figure out how my users were causing my app to behave unexpectedly. I confess, that I should have taken to heart the advice that if there is a way for your users to break your app, they will.

Things to know

I have been developing web apps professionally for just over a year and have been going through the self-taught developer journey for about three years. I decided to learn Python and chose to work with the Django framework. At my current role, I create web apps to help with operational tasks. The web app who is the main subject of this article was built to help ease the new employee onboarding and set up process. Specifically, the problem was coming from the 'new employee form.'

The problem

The problem I ran into was ultimately a succinct one, thankfully. It came to my attention when users were confused why there were duplicates of the same object, in my case, duplicate employee profiles. When entering Jane Smith into the app, for example, her information was being duplicated and two instances of Jane were created in the database.

After looking through the logs and running a few tests I found out that the problem was POST requests were being sent twice, likely due to users double clicking the submit button. It is something that I considered would happen, but I decided to move forward with developing the rest of the app and would come back to later once the initial product was released.

In hindsight, it was a lack of understanding of POST requests that lead to my choice to shelf that concern. I was actually more concerned that users would try to navigate back to a previously submitted form to make edits than I was about what I assumed were harmless double clicks. Silly boy...

Considerations

After googling around the internet, I learned more about a concept called idempotence and why it is important to consider when your app is making POST requests.

At the time of working through this problem, I hadn't even begun to sniff JavaScript, let alone try and write a script that would keep a user from double clicking a submit button (I just learned that two techniques I could have deployed are called debouncing and throttling. Neat!). Another JS option I considered was giving the user some indication that the post request had successfully been initiated, like a loading spinner. Currently, the user doesn't get any help from the app letting them know that the POST request is working.

Solution pt.1

Since I needed to deploy a solution quicky and I couldn't use JS, my remaining option was to develop a solution within the 'view' of my MVC app. There didn't seem to be any obvious native Django or third-party solutions, so I would have to build something myself. Rock and Roll!

After searching for how other developers implemented idempotent POST requests, my initial idea was to send a unique identifier with each post request. This UID would be stored in a new table in my database. Each POST request from the new employee form would include a UID that was created during the POST request, and it would be checked against the database. If the UID was already present, the POST request wouldn't be allowed to finish and the user would be redirected to a different page. In the situation of a double click, each request would contain the same information, including matching UIDs. The second, errant request wouldn't be allowed to finish since a matching UID had just been added to the DB. That was the theory at least.

The solution worked, but it was far from elegant and seemed kinda hacky. I planned to setup a cron job to clean the UID table periodically. But in the meantime, I didn't want to accidently create duplicate UIDs. Random numbers wouldn't work for this solution as there remained a small chance false positives could happen. I decided to use Python 's built-in hashlib library to create a hash that was based on the employee's information, like their name and location. But what happens if the one location hires two people with the same name? That would create a false positive for my idempotent check. So, I also added the date and time to the string to be hashed. It worked. But it was gross.

Solution pt.2

While my duct taped solution was in place, I quickly got to work on a better patch. Immediately, I saw that I had multiple places where my POST requests were not idempotent. Any good solution should be able to quickly be deployed across my entire app. I take advantage of Django's class-based views and also write custom function-based views. My solution needs to work for both.

I decided to best place to start was to see how Django's maintainers approached similar problems and started digging through the documentation and code base for their built-in CSRF (SEA SURF!!!) protection.

Using that as a jumping off point, I decided to implement the following:

Add a token, similar to Django' CSRF token, to each POST request.
Grab token from the request and check if it is in the current session
If not, proceed with processing the request and also add the token to the session.
If the token is present in the session, redirect the user*.

*Deciding where to redirect the user was a hard decision to make. Where do I send the user if the post is not idempotent? Do I give them an error message? Do I act like nothing happened? If they double clicked the submit button an error message may not make sense, because technically the post request could have worked fine. It could cause confusion. I ultimately decided to make this an error that handles itself silently. Do you have other suggestions? Let me hear 'em!

(I'll explain the code in small pieces as I go along. The full code in one chunk is below.)

In more specific terms, both my custom mix-in for class-based views and my custom class for function-based views do the same thing:

Custom Middleware checks to see if the session contains the idempotent token key. If not, it creates one and sets the value to an empty list. This happens on every request/response.

# middleware.py
from django.shortcuts import render
from django.contrib.sessions.models import Session
import random

def idempotent_post_middleware(get_response):

    def middleware(request):
        if not request.session.__contains__('idempo_token'):
            request.session.__setitem__('idempo_token', [])

        response = get_response(request)

        return response

    return middleware

Next, I needed to create and pass a token to the template. I use of Django's built-in template engine. Since everything is session based, I can use a random number for my UID. A random number is assigned to the new employee form on a GET request.

# class-based view
# IdempotentMixin
class IdempotentMixin:
    idempo_redirect = None

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        idempo_token_list = self.request.session.__getitem__('idempo_token')
        token = random.randrange(999999)
        while token in idempo_token_list:
            token = random.randrange(999999)
        context['idempo_token'] = token
        return context

# function-based view
# idempotent_helper.py
def create_token(self):

        idempo_token_list = self.request.session.__getitem__('idempo_token')
        token = random.randrange(999999)
        while token in idempo_token_list:
            token = random.randrange(999999)
        self.token = token

Next I needed to pass this token to the template. To be honest, I really wanted to match how Django built their CSRF solution. All the developer needs to do is add the template tag. Even for function-based views, you don't have to explicitly pass the CSRF token to the view's context, it is just made available. I wasn't able to fully understand how they were doing this, so within function-based views the idempotent token must be explicitly passed. Class-based views, though, only need to extend the IdempotentMixin.

# class-based view example
class YourViewClass(IdempotentMixin, CreateView):
# Tell the mixin where you want to redirect users to on duplicate requests
idempo-redirect='some_other_page'

Function based views tripped me up a little and show me how much I still have to learn about python and OOP. Everything works well, but I am not sure if my implementation is correct or best practice. Maybe you have some suggestions?

# function-based view example
def your_function_view(request, pk):

    idempo = IdempotentHelper(request, os.environ, redirect_to='some_other_page')
    if idempo.bad_request:
        return idempo.redirect
...
# more of your functions code

return(request, 'a_very_good_template.html',{
# your other context items
'idempo_token':idempo.token
}

I built a custom template tag to easily add the token as a hidden field underneath the form's CSRF tag.

# custom_template_tags.py
@register.simple_tag(takes_context=True)
def idempotent_token(context):
    token = context['idempo_token']
    html = format_html('<input type="hidden" name="idempo_token" value="{}">', token)
    return html

# a_very_good_template.html
<form method="post">
    {% csrf_token %}
    {% idempotent_token %}
    ...
</form>

The project's full code:

# custom template tag
@register.simple_tag(takes_context=True)
def idempotent_token(context):
    token = context['idempo_token']
    html = format_html('<input type="hidden" name="idempo_token" value="{}">', token)
    return html

###

# custom middleware
from django.shortcuts import render
from django.contrib.sessions.models import Session
import random

def idempotent_post_middleware(get_response):

    def middleware(request):
        if not request.session.__contains__('idempo_token'):
            request.session.__setitem__('idempo_token', [])

        response = get_response(request)

        return response

    return middleware

###

# custom mixin for class-based views
from django.http import HttpResponseRedirect
from django.urls import reverse
import os

import random

class IdempotentMixin:
    idempo_redirect = None

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        idempo_token_list = self.request.session.__getitem__('idempo_token')
        token = random.randrange(999999)
        while token in idempo_token_list:
            token = random.randrange(999999)
        context['idempo_token'] = token
        return context

    def post(self, request, *args, **kwargs):       
        if os.environ.get('test') == 'true':
            return super().post(self, request, *args, **kwargs)
        else:

            idempo_token_list = self.request.session.__getitem__('idempo_token')
            token = self.request.POST['idempo_token']
            if str(token) in idempo_token_list:
                return HttpResponseRedirect(reverse(self.get_idempo_redirect()))
            else:
                idempo_token_list.append(token)
                self.request.session.__setitem__('idempo_token', idempo_token_list)
            return super().post(self, request, *args, **kwargs)

    def get_idempo_redirect(self):
        if not self.idempo_redirect:
            return 'home'
        else:
            return self.idempo_redirect

###

# Class for use with function-based views
import random
from django.http import HttpResponseRedirect
from django.urls import reverse
from django.shortcuts import render

class IdempotentHelper:

    def __init__(self, request, environment, redirect_to=None):
        self.token = None
        self.request = request
        self.environment = environment
        self.test_name = 'test'
        self.bad_request = False
        self.redirect = None
        self.redirect_to = redirect_to

        self.create_token()
        self.idempotent_check()

    def create_token(self):

        idempo_token_list = self.request.session.__getitem__('idempo_token')
        token = random.randrange(999999)
        while token in idempo_token_list:
            token = random.randrange(999999)
        self.token = token

    def idempotent_check(self):
        if self.request.method == 'POST':            
            if self.environment.get(self.test_name) == 'true':
                pass
            else:
                idempo_token_list = self.request.session.__getitem__('idempo_token')
                token = self.request.POST['idempo_token']
                if not token:
                    self.bad_request = True
                    exception = 'There appears to be an issue with the Idempotent token. Please try your request again. If the problem persists contact easyHUB admin.'
                    self.redirect = render(self.request, '403.html',{'exception': exception}, status=403)
                if str(token) in idempo_token_list:
                    self.bad_request = True
                    self.redirect = HttpResponseRedirect(reverse(self.get_redirect_url()))
                else:
                    idempo_token_list.append(token)
                    self.request.session.__setitem__('idempo_token', idempo_token_list)

    def get_redirect_url(self):
        if not self.redirect_to:
            return 'home'
        else:
            return self.redirect_to

Conclusion

I like the final solutyion I came up with, but it does have a ton of room for improvement.

I like if it acted even closer to how Django's native CSRF protection worked, especially for function-based views.
I would like to explore finding a better way to either log that the a post request didn't pass the idempotent check or alert the user that an error happened, but their post still worked.
I wonder if the custom middleware was needed. Maybe the token key within the session could be created within both the mixin and the class.
Maybe the solution for function-based views could be refactored into a decorator. I am still working on understanding and creating decorators, but that might be a more elegant solution.

What do you think needs to be fixed here? I am far from an expert in Django, Python, web development and basically anything else, so I welcome all comments!

Either way, the solution works and it keeps my users from creating multiple instances. Let's see what else they find to break!