DEV Community

Cover image for How to Use GraphQL in Django with Elasticsearch
Aidas Bendoraitis
Aidas Bendoraitis

Posted on • Originally published at djangotricks.com

How to Use GraphQL in Django with Elasticsearch

When you need to use GraphQL with Django, a common practice is to use Graphene-Django, which allows you to query Django models. However, directly querying Django models might be too slow, especially when you have many complex relations. To speed that up, you can add one more layer of abstraction and index your Django models to an Elasticsearch server. In this article, I will show you how to create a GraphQL interface for querying posts from the Elasticsearch server.

Elasticsearch Server

The easiest way to install the Elasticsearch server locally is using EVM - Elasticsearch Version Manager:

$ sudo curl -o /usr/local/bin/evm https://raw.githubusercontent.com/duydo/evm/master/evm
$ sudo chmod +x /usr/local/bin/evm
Enter fullscreen mode Exit fullscreen mode

Then you can install Elasticsearch 7.17.27 (or other version) with EVM, as follows:

$ evm install 7.17.27
Enter fullscreen mode Exit fullscreen mode

By default, the Elasticsearch server will raise a warning if you don't have HTTPS or basic authentication. You can change one setting to turn off those warnings for localhost. To do so, add a line to ~/.evm/elasticsearch-7.17.27/config/elasticsearch.yml:

xpack.security.enabled: false
Enter fullscreen mode Exit fullscreen mode

Now, you can start the server with:

$ evm start
Enter fullscreen mode Exit fullscreen mode

If you need more Elasticsearch versions on the same computer, you can easily install and switch between them.

Python Packages

Install the latest packages of Django Elasticsearch DSL for connecting to Elasticsearch server, and Graphene-Django for GraphQL implementation:

(venv)$ pip install Django
(venv)$ pip install django-elasticsearch-dsl
(venv)$ pip install graphene-django
Enter fullscreen mode Exit fullscreen mode

Django Settings

Add the installed apps to INSTALLED_APPS in the myproject/settings.py. Also, add a few more settings that we will need in this exercise:

INSTALLED_APPS = [
    # Contributed Django apps
    # ...

    # Third-party apps
    "django_elasticsearch_dsl",
    "graphene_django",

    # Local apps
    "posts",
]

PROJECT_NAME = "myproject"
ENVIRONMENT = "dev"

ELASTICSEARCH_DSL = {
    "default": {
        "hosts": "http://localhost:9200",
    }
}

GRAPHENE = {
    "SCHEMA": "elasticsearch_graphql.schema.schema",
}
Enter fullscreen mode Exit fullscreen mode

Models and Administration for the Posts App

Let's create a posts app with two models: Category and Post:

The posts/models.py file will look like this:

from django.db import models


class Category(models.Model):
    title = models.CharField("Title", max_length=255)
    slug = models.SlugField("Slug", max_length=255)

    class Meta:
        verbose_name = "Category"
        verbose_name_plural = "Categories"
        ordering = ["title"]

    def __str__(self):
        return self.title


class Post(models.Model):
    title = models.CharField("Title", max_length=255)
    slug = models.SlugField("Slug", max_length=255)
    content = models.TextField("Content")
    categories = models.ManyToManyField(Category)

    class Meta:
        verbose_name = "Post"
        verbose_name_plural = "Posts"
        ordering = ["title"]

    def __str__(self):
        return self.title
Enter fullscreen mode Exit fullscreen mode

Also let's create administration posts/admin.py to be able to add some entries easily:

from django.contrib import admin
from .models import Category, Post


@admin.register(Category)
class CategoryAdmin(admin.ModelAdmin):
    list_display = ["title", "slug"]
    prepopulated_fields = {"slug": ["title"]}


@admin.register(Post)
class Post(admin.ModelAdmin):
    list_display = ["title", "slug"]
    list_filter = ["categories"]
    filter_horizontal = ["categories"]
    prepopulated_fields = {"slug": ["title"]}

Enter fullscreen mode Exit fullscreen mode

Create the database schema and the superuser, then run the local development server with the following Django management commands:

(venv)$ python makemigrations
(venv)$ python migrate
(venv)$ python createsuperuser
(venv)$ python runserver
Enter fullscreen mode Exit fullscreen mode

Add some posts and categories in the administration:

http://localhost:8000/admin/

Elasticsearch Documents for the Posts

Let's create PostDocument definition for the Elasticsearch index of the Post model and its categories at posts/documents.py:

import graphene

from django.conf import settings
from django_elasticsearch_dsl import Document
from django_elasticsearch_dsl.registries import registry
from django_elasticsearch_dsl import fields
from .models import Post


@registry.register_document
class PostDocument(Document):
    title = fields.TextField()
    slug = fields.KeywordField()
    content = fields.TextField()
    categories = fields.NestedField(
        properties={
            "title": fields.TextField(),
            "slug": fields.KeywordField(),
        }
    )

    class Index:
        name = f"{settings.PROJECT_NAME}_{settings.ENVIRONMENT}_posts"
        settings = {"number_of_shards": 1, "number_of_replicas": 0}

    class Django:
        model = Post

        ignore_signals = False
        auto_refresh = True
        queryset_pagination = 5000

    def prepare(self, instance):
        data = super().prepare(instance)

        # Prepare categories field
        data["categories"] = [
            {
                "title": category.title,
                "slug": category.slug,
            }
            for category in instance.categories.all()
        ]
        return data
Enter fullscreen mode Exit fullscreen mode

To index the posts on the Elasticsearch server, run the following:

(venv)$ python manage.py search_index --rebuild
Enter fullscreen mode Exit fullscreen mode

Let's see if the entries were indexed and can be retrieved.

At first, run the Django shell:

(venv)$ python manage.py shell
Enter fullscreen mode Exit fullscreen mode

Then try to list out items from the Elasticsearch index:

>>> from posts.documents import PostDocument
>>> for doc in PostDocument.search():
...    print(doc.title)
Enter fullscreen mode Exit fullscreen mode

It will list up to 10 posts (that's the default pagination) from the index:

Creating AI-based Summaries in a Django Website
Creating Open Graph Images in Django for Improved Social Media Sharing
HTTPS for Django Development Environment
Enter fullscreen mode Exit fullscreen mode

GraphQL Types and Queries for the Posts App

Now, create the GraphQL schema posts/schema.py for the categories and posts, as follows:

import operator
from functools import reduce

import graphene
from .documents import PostDocument


class ElasticsearchObjectType(graphene.ObjectType):
    @classmethod
    def from_elasticsearch(cls, elasticsearch_document):
        instance = cls()
        instance.elasticsearch_document = elasticsearch_document
        return instance


class ElasticsearchCategoryType(ElasticsearchObjectType):
    title = graphene.String()
    slug = graphene.String()

    def resolve_title(self, info):
        return self.elasticsearch_document.title

    def resolve_slug(self, info):
        return self.elasticsearch_document.slug


class ElasticsearchPostType(ElasticsearchObjectType):
    title = graphene.String()
    slug = graphene.String()
    content = graphene.String()
    categories = graphene.List(of_type=ElasticsearchCategoryType)

    def resolve_title(self, info):
        return self.elasticsearch_document.title

    def resolve_slug(self, info):
        return self.elasticsearch_document.slug

    def resolve_content(self, info):
        return self.elasticsearch_document.content

    def resolve_categories(self, info):
        return [
            ElasticsearchCategoryType.from_elasticsearch(
                elasticsearch_document=category
            )
            for category in self.elasticsearch_document.categories or []
        ]


class Query(object):
    elasticsearch_posts = graphene.List(
        ElasticsearchPostType,
        limit=graphene.Int(),
        offset=graphene.Int(),
        search=graphene.String(),
        categories=graphene.List(of_type=graphene.String),
    )

    def resolve_elasticsearch_posts(
        self,
        info,
        limit=10,
        offset=0,
        search=None,
        categories=None,
    ):
        from elasticsearch_dsl.search import Q

        search_obj = PostDocument.search()

        queries = []
        if search:
            queries.append(
                Q(
                    "multi_match",
                    query=search,
                    fields=[
                        "title^5",
                        "slug^4",
                        "content^3",
                    ],
                    type="best_fields",
                    tie_breaker=0.3,
                    fuzziness="AUTO",
                    fuzzy_transpositions=True,
                    lenient=True,
                )
            )

        if categories:
            queries.append(
                Q(
                    "nested",
                    path="categories",
                    query=Q("terms", categories__slug=categories),
                )
            )

        if queries:
            # All filters AND-ed
            search_obj = search_obj.query(reduce(operator.iand, queries))

        search_obj = search_obj[offset : offset + limit if limit else None]

        return [
            ElasticsearchPostType.from_elasticsearch(elasticsearch_document=result)
            for result in search_obj.execute()
        ]
Enter fullscreen mode Exit fullscreen mode

Here, we have an ElasticsearchObjectType that provides a custom method from_elasticsearch to initiate GraphQL objects from Elasticsearch instead of Django models.

The query for elasticsearch_posts has two filters: by a search string and by a list of OR-ed category slugs. The search query looks into title, slug, and content fields with different boosting weights and searches for the best match. The fuzziness set to "AUTO" allows mistakes in the search string by two letters.

At the project level, create Graphene Relay - a schema class that uses a Query class inheriting from all the Query classes from all Django apps. The file can be located at myproject/schema.py and have this content:

import graphene
import posts.schema

class Query(posts.schema.Query, graphene.ObjectType):
    pass

schema = graphene.Schema(query=Query)
Enter fullscreen mode Exit fullscreen mode

Finally, enable the GraphQL endpoint in the myproject/urls.py:

from django.contrib import admin
from django.urls import path
from django.views.decorators.csrf import csrf_exempt
from graphene_django.views import GraphQLView

urlpatterns = [
    path("graphql/", csrf_exempt(GraphQLView.as_view(graphiql=True))),
    path("admin/", admin.site.urls),
]
Enter fullscreen mode Exit fullscreen mode

Checking the GraphQL Queries

Navigate to http://localhost:8000/graphql/

Graphene-Django comes with a GraphiQL user interface, which, when accessed from a browser, allows the user to query GraphQL and see results in the same window:

GraphiQL user interface

A search query for "AI" with categories "Intermediate" or "Advanced" can look like:

query FilteredPosts {
  elasticsearchPosts(
    search: "AI"
    categories: ["intermediate", "advanced"], 
    limit: 10, 
    offset: 0,
  ) {
    title
    slug
    categories {
      title
      slug
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

It will produce results in JSON just like this:

{
  "data": {
    "elasticsearchPosts": [
      {
        "title": "Creating AI-based Summaries in a Django Website",
        "slug": "creating-ai-based-summaries-in-a-django-website",
        "categories": [
          {
            "title": "Basics",
            "slug": "basics"
          },
          {
            "title": "Django",
            "slug": "django"
          },
          {
            "title": "Gemini",
            "slug": "gemini"
          },
          {
            "title": "Intermediate",
            "slug": "intermediate"
          },
          {
            "title": "Large Language Models",
            "slug": "large-language-models"
          },
          {
            "title": "PyBazaar",
            "slug": "pybazaar"
          },
          {
            "title": "Simplemind",
            "slug": "simplemind"
          }
        ]
      },
      {
        "title": "HTTPS for Django Development Environment",
        "slug": "https-for-django-development-environment",
        "categories": [
          {
            "title": "DevOps",
            "slug": "devops"
          },
          {
            "title": "Development",
            "slug": "development"
          },
          {
            "title": "Django",
            "slug": "django"
          },
          {
            "title": "HTTPS",
            "slug": "https"
          },
          {
            "title": "Intermediate",
            "slug": "intermediate"
          },
          {
            "title": "SSL",
            "slug": "ssl"
          },
          {
            "title": "TLS",
            "slug": "tls"
          }
        ]
      }
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

Conclusion

Using Elasticsearch with GraphQL is not trivial, but it provides excellent performance, especially for complex data structures. The best way to implement that in Django that I know of is combining Django Elasticsearch DSL and Graphene-Django.


Cover photo Angela Roma

Billboard image

The fastest way to detect downtimes

Join Vercel, CrowdStrike, and thousands of other teams that trust Checkly to streamline monitoring.

Get started now

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

👋 Kindness is contagious

Immerse yourself in a wealth of knowledge with this piece, supported by the inclusive DEV Community—every developer, no matter where they are in their journey, is invited to contribute to our collective wisdom.

A simple “thank you” goes a long way—express your gratitude below in the comments!

Gathering insights enriches our journey on DEV and fortifies our community ties. Did you find this article valuable? Taking a moment to thank the author can have a significant impact.

Okay