DEV Community

Cover image for Handling Big Data with Django and AgGrid
Zach
Zach

Posted on • Originally published at circumeo.io

Handling Big Data with Django and AgGrid

We're going to build an app that can handle filtering and sorting large amounts of data with AgGrid.

Sending all of your data to the browser is a quick way to get started with AgGrid, but what if you have a million rows? That's when server-side filtering and sorting come to the rescue.

This guide includes:

  • using AgGrid in infinite scroll mode so that the server is responsible for paginating, filtering, and sorting
  • implementing filtering, sorting, and pagination for AgGrid on top of a Django ListView class
  • creating a Django management command to populate test data

But first, here's a quick peek at what we're building today:

Image description

Previewing the project

Want to see a live version of the app? You can create a copy of this project now and try it out.

Create a fork of the AgGridDemo project on Circumeo.

Setting up the Django app

Install packages and create the Django application.

pip install --upgrade django faker
django-admin startproject bigdata .
python3 manage.py startapp core
Enter fullscreen mode Exit fullscreen mode

Add core to the INSTALLED_APPS list.

# settings.py
INSTALLED_APPS = [
  "core",
    ...
]
Enter fullscreen mode Exit fullscreen mode

Adding the templates

  • Create a directory named templates within the core app.
  • Create a file named index.html within the templates directory.
<!DOCTYPE html>
<html>
  <head>
    <script src="https://unpkg.com/ag-grid/dist/ag-grid.min.noStyle.js"></script>
    <link rel="stylesheet" href="https://unpkg.com/ag-grid/dist/styles/ag-grid.css">
    <link rel="stylesheet" href="https://unpkg.com/ag-grid/dist/styles/ag-theme-balham.css">
    <style>
      #container {
        max-width: 1250px;
        width: 100%;
        height: 100%;
        margin: 0px auto;
        margin-top: 30px;
      }
    </style>
  </head>
  <body>
    <div id="container">
      <div id="data-grid" style="height: 600px; width: 100%" class="ag-theme-balham"></div>
    </div>
    <script type="text/javascript" charset="utf-8">
      const numberFilterParams = {
        filterParams: {
          filterOptions: ["equals", "notEqual", "lessThan", "greaterThan"],
          suppressAndOrCondition: true,
        },
      };

      var columnDefs = [
        { headerName: "Name", field: "name", filter: "agTextColumnFilter" },
        {
          headerName: "Description",
          field: "description",
          filter: "agTextColumnFilter",
        },
        { headerName: "Category", field: "category", filter: "agTextColumnFilter" },
        {
          headerName: "Price",
          field: "price",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
        {
          headerName: "Stock Quantity",
          field: "stock_quantity",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
        {
          headerName: "Rating",
          field: "rating",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
      ];

      var gridOptions = {
        columnDefs: columnDefs,
        defaultColDef: {
          filterParams: {
            suppressAndOrCondition: true,
          },
        },
        enableServerSideSorting: true,
        enableServerSideFilter: true,
        rowModelType: "infinite",
        cacheBlockSize: 100,
        maxBlocksInCache: 10,
      };

      var dataSource = {
        getRows: function (params) {
          var filtering = encodeURIComponent(JSON.stringify(params.filterModel));
          var sorting = encodeURIComponent(JSON.stringify(params.sortModel));

          var startRow = params.startRow;
          var endRow = params.endRow;

          var url = `/products?startRow=${startRow}&endRow=${endRow}&filter=${filtering}&sort=${sorting}`;

          fetch(url)
            .then((response) => response.json())
            .then((data) => {
            params.successCallback(data.rows, data.totalRows);
          })
            .catch((err) => {
            params.failCallback();
          });
        },
      };

      var gridDiv = document.querySelector("#data-grid");

      new agGrid.Grid(gridDiv, gridOptions);
      gridOptions.api.setDatasource(dataSource);
      gridOptions.api.sizeColumnsToFit();
    </script>
  </body>
</html>
Enter fullscreen mode Exit fullscreen mode

Adding the views

  • Remove the existing views.py file.
  • Create a new directory named views within the core folder.
  • Create a new file named index.py within the views directory.
  • Copy and paste the following into index.py within the views directory.
from django.shortcuts import render

def index_view(request):
    return render(request, "core/index.html")
Enter fullscreen mode Exit fullscreen mode
  • Create another file named products.py within the same directory.
  • Copy and paste the following into the products.py file.
import json

from django.views.generic.list import ListView
from django.http import JsonResponse
from django.db.models import Q

from core.models import Product


class ProductListView(ListView):
    model = Product

    def get_queryset(self):
        """
        Convert AgGrid filter and sort objects into a Django query.
        An example filter:
            {"category": {"type": "contains", "filter": "electronics"}}
        """
        queryset = super().get_queryset()

        filter_params = self.request.GET.get("filter", None)
        if filter_params:
            filters = json.loads(filter_params)
            q_objects = Q()

            for key, filter_info in filters.items():
                filter_type = filter_info.get("type")
                filter_value = filter_info.get("filter")

                if filter_type == "contains":
                    lookup = f"{key}__icontains"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "equals":
                    lookup = f"{key}__exact"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "notEqual":
                    lookup = f"{key}__exact"
                    q_objects &= ~Q(**{lookup: filter_value})
                elif filter_type == "greaterThan":
                    lookup = f"{key}__gt"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "lessThan":
                    lookup = f"{key}__lt"
                    q_objects &= Q(**{lookup: filter_value})

            queryset = queryset.filter(q_objects)

        sort_params = self.request.GET.get("sort", None)
        if sort_params:
            sort_objects = json.loads(sort_params)
            sort_fields = []
            for sort_object in sort_objects:
                col_id = sort_object["colId"]
                sort_order = sort_object["sort"]
                if sort_order == "asc":
                    sort_fields.append(col_id)
                elif sort_order == "desc":
                    sort_fields.append(f"-{col_id}")

            if sort_fields:
                queryset = queryset.order_by(*sort_fields)
        return queryset

    def get(self, request, *args, **kwargs):
        start_row = int(request.GET.get("startRow", 0))
        end_row = int(request.GET.get("endRow", 100))

        queryset = self.get_queryset()

        total_rows = queryset.count()
        queryset = queryset[start_row:end_row]

        products = list(
            queryset.values(
                "name", "description", "category", "price", "stock_quantity", "rating"
            )
        )

        return JsonResponse({"rows": products, "totalRows": total_rows})
Enter fullscreen mode Exit fullscreen mode

Updating URLs

  • Create urls.py in the core directory.
from django.urls import path

from core.views.index import index_view
from core.views.products import ProductListView

urlpatterns = [
    path("", index_view, name="index"),
    path("products", ProductListView.as_view(), name="products"),
]
Enter fullscreen mode Exit fullscreen mode
  • Update the existing urls.py within the project bigdata directory.
from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", include("core.urls")),
]
Enter fullscreen mode Exit fullscreen mode

Adding the database models

Overwrite the existing models.py with the following:

from django.db import models

class Product(models.Model):
    name = models.CharField(max_length=255)
    description = models.TextField(blank=True)
    category = models.CharField(max_length=255)
    price = models.DecimalField(max_digits=10, decimal_places=2)
    stock_quantity = models.IntegerField()
    rating = models.FloatField(blank=True, null=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)
Enter fullscreen mode Exit fullscreen mode

Adding a Django Management Command for Test Data

  • Create the directory structure management/commands within the core folder.
  • Open a file named populate_products.py within the new directory and enter the following.
from faker import Faker
import random

from django.core.management.base import BaseCommand
from core.models import Product

class Command(BaseCommand):
    def handle(self, *args, **kwargs):
        faker = Faker()

        categories = [
            'Electronics',
            'Books',
            'Clothing',
            'Home & Garden',
            'Toys & Games',
            'Sports & Outdoors',
            'Health & Beauty',
            'Automotive',
            'Groceries',
            'Pet Supplies'
        ]

        for _ in range(1000):
            Product.objects.create(
                name=faker.text(max_nb_chars=20).capitalize(),
                description=faker.text(),
                category=random.choice(categories),
                price=round(random.uniform(5.0, 500.0), 2),
                stock_quantity=random.randint(0, 100),
                rating=round(random.uniform(1.0, 5.0), 1)
            )

        self.stdout.write(self.style.SUCCESS('Successfully populated the database with products.'))
Enter fullscreen mode Exit fullscreen mode

Open a shell session in order to run the Django management command.

python3 manage.py populate_products
Enter fullscreen mode Exit fullscreen mode

Up and Running with Django and AgGrid

You're now ready to handle millions of rows without crashing browser tabs! Add database indexes where needed, and you can scale this approach pretty far.

Top comments (0)