DEV Community: spikelantern

Simple infinite scroll in Django

spikelantern — Sun, 26 Jul 2020 00:00:00 +0000

If you've used Twitter, Instagram, or Facebook, you would have used something called "infinite scroll", sometimes also called "infinite loading" or "endless pagination".

Basically, it means that once you scroll down near the bottom of a list of items, the page fetches new items automatically and adds them to the page. That makes it a smoother experience compared to traditional pagination in some cases.

You may have wondered how to do that in Django. I'm going to show you a very simple way to do this using no JavaScript libraries.

Note: The solution here is inefficient for lists with a large number of items (e.g. thousands of items). Efficient infinite scroll uses windowing, and removes items not in view from the DOM. However, if you are only dealing with a few hundred items, this is a very simple way to achieve this, with no dependencies.

Don't worry about the specific details of the code snippets here, just make sure you understand the basic concepts. At the end of the article, I will link to some example code.

Detecting when we've scrolled to the bottom

In the frontend, we will need a way to detect when you've scrolled to the bottom.

This used to be pretty difficult to do, but there is a new browser API or feature called Intersection Observer.

We can think of it as having two components, the scrollable element, and a "sentinel":

Basic concept

The idea is that when the user scrolls down to where the sentinel element is visible, we fetch new items from the Django backend.

First, let's look at how we can detect when this happens using the Intersection Observer API.

First, the Django template, as you can see it's just a list of items and then a sentinel element:

<div id="scrollable-element">
    {% for post in posts %}
        {% include "_post.html" with post=post %}
    {% endfor %}
</div>

<div id="sentinel"></div>

Now the JavaScript

document.addEventListener("DOMContentLoaded", () => {
  let sentinel = document.getElementById("sentinel");

  let observer = new IntersectionObserver((entries) => {
    entry = entries[0];
    if (entry.intersectionRatio > 0) {
        alert("This happened");
    }
  })
  observer.observe(sentinel);
})

When you scroll to the bottom where the sentinel is visible, you should get a popup alert.

That's great! Now we can replace that popup alert with an AJAX request to our backend. But let's add that functionality to our backend first.

We can choose to have our backend return JSON, and render it client-side (e.g. using a templating library), but in this tutorial I have opted to have the backend return HTML, and appending this to the scrollable element through innerHTML. This is a pretty old-fashioned AJAX technique that you sometimes still see in websites like GitHub. If you do this, you need to be very careful with XSS, but we'll discuss this later.

Django pagination

You might be familiar with Django's pagination, if not check out the documentation on that topic here.

Here's how it works. Let's say you have a simple list view like this:

from django.shortcuts import render
from django.views.decorators.http import require_GET, require_POST
from .models import Post

@require_GET
def post_list(request):
    posts = Post.objects.order_by('-created_at').all()
    context = {'posts': posts}
    return render(request, 'post_list.html', context)

You can paginate this by changing it like this:

from django.shortcuts import render
from django.core.paginator import Paginator
from django.http import Http404
from django.views.decorators.http import require_GET, require_POST
from .models import Post

@require_GET
def post_list(request):
    all_posts = Post.objects.order_by('-created_at').all()
    paginator = Paginator(all_posts, per_page=10)
    page_num = int(request.GET.get("page", 1))
    if page_num > paginator.num_pages:
        raise Http404
    posts = paginator.page(page_num)
    context = {'posts': posts}
    return render(request, 'post_list.html', context)

However, we want to have our backend return a short HTML snippet, rather than the full HTML page. So what we want to do is a bit of conditional processing.

First we add another partial template like this, let's call it _posts.html:

    {% for post in posts %}
        {% include "_post.html" with post=post %}
    {% endfor %}

And include that in our list view:

<div id="scrollable-element">
    {% include "_posts.html" with posts=posts %}
</div>

<div id="sentinel"></div>

Then we need to conditionally change the response when the request is an AJAX request. We used to be able to do this using request.is_ajax(), but starting from version 3.1 this is deprecated.

Luckily, it's easy to replicate that functionality:

def is_ajax(request):
    """
    This utility function is used, as `request.is_ajax()` is deprecated.

    This implements the previous functionality. Note that you need to
    attach this header manually if using fetch.
    """
    return request.META.get("HTTP_X_REQUESTED_WITH") == "XMLHttpRequest"

Then, we simply change the above paginated view to do this:

@require_GET
def post_list(request):
    """
    List view for posts.
    """
    all_posts = Post.objects.order_by('-created_at').all()
    paginator = Paginator(all_posts, per_page=10)
    page_num = int(request.GET.get("page", 1))
    if page_num > paginator.num_pages:
        raise Http404
    posts = paginator.page(page_num)
    if is_ajax(request):
        return render(request, '_posts.html', {'posts': posts})
    return render(request, 'post_list.html', {'posts': posts})

That's all we need for the backend.

Making AJAX requests

Now, we can update our JavaScript code to fetch the data from the backend when we scroll down to the sentinel.

Note that we are using fetch in this example, and it doesn't add the X-Requested-With header by default (which is actually why it request.is_ajax() was deprecated). So we need to manually add that header.

const fetchPage = async (url) => {
  let headers = new Headers()
  headers.append("X-Requested-With", "XMLHttpRequest")
  return fetch(url, { headers })
}

document.addEventListener("DOMContentLoaded", () => {
  let sentinel = document.getElementById("sentinel");
  let scrollElement = document.getElementById("scroll-element");
  let counter = 2;
  let end = false;

  let observer = new IntersectionObserver(async (entries) => {
    entry = entries[0];
    if (entry.intersectionRatio > 0) {
        let url = `/posts/?page=${counter}`;
        let req = await fetchPage(url);
        if (req.ok) {
            let body = await req.text();
            // Be careful of XSS if you do this. Make sure
            // you remove all possible sources of XSS.
            scrollElement.innerHTML += body;
        } else {
            // If it returns a 404, stop requesting new items
            end = true;
        }
    }
  })
  observer.observe(sentinel);
})

A note about `innerHTML` and XSS

In this example, we appended HTML from the server's response to innerHTML. This is a technique that used to be pretty common, and we still see in websites like GitHub. Try opening your "Network" tab in dev tools the next time you're on GitHub, and see the responses when you interact with the website!

As mentioned, if you do this, you need to be very careful to remove sources of XSS in your HTTP response from the backend, otherwise an attacker can inject malicious JavaScript that runs on a user's browser.

If you use Django templates, your template is escaped by default. However, if you use the safe template filter (e.g. you're allowing user to input some HTML) this method is unsafe.

First, you should re-evaluate whether you should be allowing users to enter untrusted HTML that you will display. In many cases you shouldn't need to.

If you can't get around this, you will need to sanitise the HTML. You can do it in the backend using a library like bleach, or a frontend library like DOMPurify.

Alternatively, you can return a JSON response from the backend, and render the HTML client-side. This is a more common way to do this today, as there are frontend frameworks and libraries to do exactly this. This is beyond the scope of this tutorial.

Example code

If you want to see a full working example, I have pushed some sample code to my repository here:

https://github.com/spikelantern/simple-infinite-scroll.git

Summary

In this tutorial, we covered a very simple infinite scroll implementation that doesn't use any special libraries. This uses the new Intersection Observer API.

As mentioned, this technique isn't ideal for very large lists. For an efficient solution, it's necessary to delete DOM elements to prevent the DOM from growing too much. However, for a few hundred items, this should work just fine.

We also discussed the use of innerHTML and its security implications, and recommended some possible mitigations and alternatives.

Hopefully, this post helped you. If you liked this article, make sure to subscribe. I plan to post more articles about using JavaScript with Django, so make sure you subscribe to get notified!

How much Python do I need to learn for Django?

spikelantern — Wed, 15 Jul 2020 00:00:00 +0000

Recently, someone on Reddit posted a question asking how much Python one needs to know before moving on to Django.

That's a great question.

Learning Python can take a long time! You wouldn't want to spend years "learning" Python before moving to the fun stuff.

You want results much sooner than that!

In fact, I would say one of the best ways to get better at Python (but not necessarily learning from scratch) is through Django. People always recommend getting better at Python through doing. Using Django to build web applications is one way of "doing"!

That being said, you do need to have some fundamentals in order to use Django at all, otherwise a lot of things won't make sense.

But then how do you know when you've learned "enough"?

The bare minimum

I will tell you what I recommended to do Django at a basic level.

If this seems like a lot, don't worry. If you work at it, you will learn everything here and more.

You should ask yourself the following questions. If you say "yes" to all (or most of them), you are well equipped to move on to Django.

If not, I suggest working at them in this specific order.

1. Basic program flow

Can you write simple scripts that take basic input and output?

Do you know how to write if statements?

2. Working with collections

Do you know the differences between a list, a tuple, a dict and a set? Do you know why each one is used?

Do you know how to iterate through them using for loops and while loops?

Do you know how to write basic list and generator comprehensions?

3. Working with strings

Do you know that a string is also something you can iterate over, like a list? Do you know how to do that?

Can you do basic string formatting? For example, can you take a variable and create a string that contains that variable?

Do you know that a string is also an object with its own methods?

Do you know how to join strings together, and split them based on a delimiter?

4. Writing and using functions

Can you write a function?

Do you know how to specify arguments and keyword arguments in your function, and how they work?

5. Using decorators

Do you know what a decorator is, and how to use it?

(Knowing how to write one is optional)

6. Modules, packages, and libraries

Do you know what Python modules and packages are?

Do you know how to import something from a module or a package? Do you know that you can import variables, functions, classes, as well as other modules and packages?

Do you know at least one way to install a Python library?

7. Object-oriented programming

Do you know what a class is?

Do you know what an object is?

Do you know how to write a class?

Do you know what __init__ () does?

Do you know the difference between a function and a method?

Do you know how to instantiate a class to create an object?

Do you know what inheritance is, and how to write a class that inherits from another class?

Do you know what a mixin is and how to use it? (Optional)

That's all!

That's the minimum, I believe, that you need to know about Python to work with Django at a basic level.

I say "minimum", but don't let that fool you! Most of the time, everything contained in the list above is more than sufficient to write a Django application.

In fact a lot of the production code I've seen rarely deviates from the list above.

So, there's a lot of things there.

If you already know everything there, congratulations!

Otherwise, I recommend working on it step-by-step, as mentioned, in the same order as above.

Wrapping up

I'm planning to make cheat sheets of useful tips like this. If you found this helpful, make sure to sign up for my mailing list to receive them!

Options for public facing IDs in Django

spikelantern — Fri, 26 Jun 2020 00:00:00 +0000

Django uses auto-incrementing integer columns as primary keys for models by default. If you've done some basic tutorials you may have used these integer IDs as identifiers in URLs or HTTP responses.

For example, things like /posts/23/ where 23 is the primary key of a Post.

At some point you may have read or received advice that it's probably not that good to use these auto-incrementing IDs publicly. One reason is that it leaks information about the size of database table, which may be undesirable for a few reasons (like leaking information to competitors, or even a potential attacker). Additionally, it's easy for a potential attacker to guess, and if you don't secure your endpoints properly you may be more vulnerable to some IDOR-type situations.

Have you wondered what you can use instead? Luckily, you have a few good alternatives.

Maybe just use another unique field as an identifier

Using another unique field is the most straightforward approach, assuming it's URL-safe. For example, for a profile page in a social networking platform, you could simply use the username.

Both Twitter and Instagram does this, as well as most social networking sites I'm aware of, e.g.:

https://twitter.com/spikelanterncom

Slugs

In many cases it may be appropriate to use something called a "slug". Django has a built-in field for this called SlugField, and in their documentation they have a good definition for what a slug is:

Slug is a newspaper term. A slug is a short label for something, containing only letters, numbers, underscores or hyphens. They’re generally used in URLs.

So, if you have something that can naturally fits well to slugs, such as articles, blog posts, and anything with a unique title, slugs might be a decent option.

UUIDs

Another option is to use Django's built-in UUIDField.

What does a UUID look like? You can generate one yourself by firing up the Python REPL and running this code:

>>> import uuid
>>> my_uuid = uuid.uuid4()
>>> print(my_uuid)
f22db40f-3e1e-4ac2-9c10-62ecea301f5e

These are URL-safe, that means you can safely put them in URLs.

So instead of /products/23/ you'll see /products/f22db40f-3e1e-4ac2-9c10-62ecea301f5e/

It's kind of ugly, but not too bad. To make it shorter, you can run something like a base62 or base58 encoding on the UUID to produce a more compact string. The django-extensions library does this in ShortUUIDField (more on that later).

Note that if you do this you should keep using UUIDField, and not use CharField, see this blog post for a good discussion.

Base64 encode a few bytes from `/dev/urandom`

You could also grab a few bytes from /dev/urandom and do a url-safe base64 encoding.

How many bytes are enough? I recently found this fantastic article from Neil Madden, titled "Moving away from UUIDs" which compared this method to using UUIDs, and using some math, determined that using 160 bits (20 bytes) is a good option for general random strings like access tokens.

For an identifier, you could probably use something much shorter, but I found that 160 bits works okay for URLs anyway, and is still shorter than the standard representation of UUIDs. UUIDs can also be shorter if you also encode them, as they are also just strings representing 128 bits. We'll see options for that later in this article.

Here's some code to generate such a random string:

import os
from django.utils.http import urlsafe_base64_encode as b64encode

def generate_random_field(bytes=20):
    return b64encode(os.urandom(bytes))

Try running that on your Python shell:

>>> generate_random_field()
'OVXxfipWi2VdPC8GGCKmlR6oDhc'
>>> generate_random_field()
'8ApcI0mvC_a_MTJN8Hj4um_DAsQ'
>>> generate_random_field()
'W9myCGsv93zo0vk5x9rLyd9cwI0'

The result is a 27 character string, which you could just put in a CharField, but you probably want to write a custom field like this:

import os
from django.db.models import CharField
from django.utils.http import urlsafe_base64_encode as b64encode

def generate_random_field():
    return b64encode(os.urandom(20))

class RandomIDField(CharField):
    description = "A random field meant for URL-safe identifiers"

    def __init__ (self, **kwargs):
        kwargs['max_length'] = 27

        super(). __init__ (**kwargs)

    def deconstruct(self):
        name, path, args, kwargs = super().deconstruct()
        del kwargs["max_length"]

        return name, path, args, kwargs

    def pre_save(self, model_instance, add):
        """
        This is used to ensure that we auto-set values if required.
        See CharField.pre_save
        """
        value = super().pre_save(model_instance, add)
        if not value:
            value = generate_random_field()
            setattr(model_instance, self.attname, value)
        return value

Note that the above is for the specific case of 20 bytes, and I didn't bother including any configuration options. If you want to configure this field by passing in the number of bytes you want, you might want to modify the code.

I would recommend just copying that code to your project directly and modifying as needed, but if for some reason you want it in a package, I have published it to PyPI. You can see the code here: https://github.com/spikelantern/django-randomcharfield

The resulting string can have hyphens and underscores, which you might be okay with, or not. I've personally found that this is a very simple method that is acceptable for most of my needs.

Use `django-extensions` fields

The popular package django-extensions has a few options for such fields.

If you look at the documentation for field extensions you will find several options you can use.

Particularly they have options for ShortUUIDField and RandomCharField (and AutoSlugField if you want to automate the generation of slug fields).

Their implementation of RandomCharField chooses from a limited set of characters randomly, and thus can be configured to not use any hyphens or underscores.

Tip: You don't need to replace your primary key

By the way, even if you choose any of these options, you don't need to replace your default primary key. An auto-incrementing integer as your primary key has some benefits, like being easy to sort (e.g. if you don't have a timestamp for when the entry is added). Also it's compact, and easy to work with in, say, the Django admin.

What you can do is add a second field to your model which you use for anything external, and keep using AutoField for your internal operations.

Tip: Look at your responses too

Make sure you remove the auto-incrementing IDs from HTTP responses as well. It's pointless to just change your URLs to hide the IDs, only to leak them somewhere else!

Takeaways

You have a number of options if you are concerned about revealing auto-incrementing IDs publicly (e.g. in URLs).

If you already have an existing unique field that can be public (e.g. a username), then that is normally a good option to use. Slugs can be good options for when you have a resource that fits well such as articles or blog posts. Both of these options will give you nice, human-readable URLs.

Some other alternatives discussed were UUIDs and random character fields, by either writing your own custom field, or using a library like django-extensions.

Hope you found this post helpful! If you enjoyed this, please subscribe to my mailing list and receive some very occasional emails about Django.

How to choose your next Django project

spikelantern — Sun, 14 Jun 2020 00:00:00 +0000

Have you done a couple of Django tutorials? Maybe you've already done Django's official Polls tutorial, as well as the DjangoGirls tutorial. You're now wondering what to do next to get better.

So you go on the internet, and ask people what to do. More likely than not, they'll answer with some variation of "start doing projects" and "learn by doing".

But if you're a fairly new to web development or even programming, it may be difficult to come up with project ideas.

For example, you may be worried if a project idea may be beyond your abilities at this moment, and you might be worried that if it's too difficult you might just give up. Worse, you might give up forever.

When you're new to web development, it's hard to gauge if a project idea is too difficult or not. In fact, even more experienced developers often underestimate the difficulty of some projects.

So what should you do? Wouldn't it be great if there was a way to get better through "learning by doing" but not pick a project that may overwhelm you, potentially turning you away from web development?

Your goal is to get better

The first thing is to examine your goals. Is your goal to finish a project, or to get better at Django and web development?

There is a lot of value in completing a project. You get a sense of achievement, and can tell people "hey, look at this cool thing I built". It's definitely rewarding. If you're able to complete something, you definitely deserve to be applauded.

But ultimately, that's just one project. It may be rewarding, but at some point you want to be able to build more than one type of web application.

Think of a piano player who is only able to play one tune. Sure, they'd be able to impress people when they need to perform, and that's pretty cool. But at some point they will need to move beyond that. In order to learn other tunes and play them well, they will need to get better at playing the piano in general.

People have known this for centuries, which is why many composers wrote didactic pieces – pieces of music meant to make you practice and get better. If you have any experience with the piano, you may have heard of compositions from Hanon and Czerny, and also etudes from many composers like Chopin.

Each Hanon piece, for example, aims to train your hands to play a certain type of movement or phrase. Like arpeggios, or scales, or parallel thirds.

It's helpful to learn how to play arpeggios, for instance, because many compositions have arpeggios.

Nobody would want to be performing a Hanon piece at a concert. They're not very nice to listen to, and they're things you mostly play at home to practice. Once you get better at the areas they're meant to improve, they've served their purpose, and then you can move on to expanding your repertoire.

Maybe you should do some Hanon

The equivalent for web development and Django, in my opinion, is to do some things I like to call "microprojects".

They're extremely small projects, rather than a full web application, where the aim is to learn about a specific area, rather than to produce something useful.

So rather than "build a Twitter clone", you can build parts or components of things you see on Twitter or other websites.

Let me give you an example. You may have noticed many websites with "infinite scroll". That means when you scroll down to the end of a feed, or a list, you see that the website automatically fetches new items.

No idea how to do that? Well, that's a great idea for your next project.

Start a new Django project, with one model (e.g. a Tweet model).

Use Django Admin or the shell to create 50 or so tweets. Figure out how to display 5 tweets at first. Then figure out how to make it progressively load more tweets when you scroll to the bottom, or when you click "load more".

Don't worry about authentication, user permissions, any of that other crap. Your goal is to get infinite scroll working. You can worry about that other stuff in other projects.

If you've never built something with infinite scroll before, that may be just challenging enough for you, but with the advantage that it's small and focused, so that you never get overwhelmed.

Now, consider this: once you're able to build this yourself, and understand how it works, you're now capable of doing infinite scroll in any web application. That's extremely valuable. You're now able to play arpeggios in other compositions.

Does it matter if it's not a polished project? Nope, at this point it has served its purpose, and you can just shelve your project away for your own reference without ever having to show anyone.

That also means the bar can be as low as you want. Is your design and CSS shitty? Who cares? You don't have to show anyone. If you just keep it hidden, there is no opportunity for embarrassment.

Examples of microprojects

So here's a few more microproject ideas apart from infinite scroll:

AJAX forms
Multi-step forms or wizards
An "add to cart" feature
Generating a PDF invoice
Sending an email in an automated fashion
Searching for an item based on some keyword or criteria
An auto-updating news feed
An application and approval workflow
Accepting payments via Stripe or Braintree for a single product
A simple dashboard with relevant statistics (graphs, pivot table, etc.)
Auto-processing a user-uploaded image

That's just off the top of my head. By the way, these are all components of real projects I've done for money. I'd go out on a limb and predict that you will encounter at least one of the tasks in that list in your career as a software developer.

But if you run out of ideas, you could always look for inspiration in any website or web application you use.

Did you see a cool feature on Twitter, Instagram, or even JIRA? Try replicating it in a microproject!

You don't have to replicate it fully, just try to do the simplest possible approximation of that feature.

When you should try microprojects

Ultimately, everyone's learning style is different. It may be that you actually learn better and are able to keep your motivation better when you're doing a full project.

If that's you, then please don't let this blog post deter you. By all means do a full project.

But if you get stuck, you could always do some explorations with microprojects in the thing you're stuck on. I've found that helpful.

Maybe you've already attempted doing full projects a couple of times but never completed them because you always found the experience overwhelming. I know when I started of as a programmer, that happened a lot, and was the source of a significant amount of personal frustration. Microprojects helped me get better.

Ultimately, doing microprojects is just a tool I've used in the past to learn about new concepts in a safe manner. It's a tool to consider, if you find it useful. If you don't find it useful, by all means do what works better for you.

Takeaways

So to summarise:

Doing a full blown project can be overwhelming if there are too many unfamiliar components
Microprojects, or small focused projects aimed to increase your understanding of a certain area (e.g. infinite scroll) can be helpful
For inspiration, you could always look at other web applications
Micoprojects are helpful when you get stuck doing full projects too

DEV Community: spikelantern

Simple infinite scroll in Django

Detecting when we've scrolled to the bottom

Django pagination

Making AJAX requests

A note about innerHTML and XSS

Example code

Summary

How much Python do I need to learn for Django?

The bare minimum

That's all!

Wrapping up

Options for public facing IDs in Django

Maybe just use another unique field as an identifier

Slugs

UUIDs

Base64 encode a few bytes from /dev/urandom

Use django-extensions fields

Tip: You don't need to replace your primary key

Tip: Look at your responses too

Takeaways

How to choose your next Django project

Your goal is to get better

Maybe you should do some Hanon

Examples of microprojects

When you should try microprojects

Takeaways

A note about `innerHTML` and XSS

Base64 encode a few bytes from `/dev/urandom`

Use `django-extensions` fields