DEV Community: Hesbon

Apache Kafka with Python

Hesbon — Wed, 25 Jan 2023 14:15:50 +0000

In this article, we will cover the following

Introduction to Kafka and its use cases
Setting up a Kafka server
- Manual setup
- Docker setup
Installing the Python Kafka library (e.g. Kafka-python)
Producing messages to a Kafka topic
Consuming messages from a Kafka topic
Advanced Kafka features (e.g. custom serialization/deserialization, message keys, etc.)
Error handling and troubleshooting
Conclusion and resources for further learning

1. Introduction to Kafka and its use cases

Apache Kafka is a distributed streaming platform that allows you to publish and subscribe to streams of records, similar to a message queue or enterprise messaging system. One of its key features is its ability to handle a large number of concurrent reads and writes, making it well-suited for handling high volumes of data from multiple sources.

Some common use cases for Kafka include:

Real-time data pipelines: Collecting and processing data from various sources in real-time, such as log data, sensor data, and social media feeds.
Stream processing: Analyzing and processing data streams as they are generated, such as detecting patterns or anomalies in data.
Event-driven architectures: Building systems that respond to specific events, such as sending a message or triggering a workflow.

How do the Kafka brokers and clients keep track of all the Kafka brokers if there is more than one? The Kafka team decided to use Zookeeper for this purpose.

Zookeeper is used for metadata management in the Kafka world. For example:

Zookeeper keeps track of which brokers are part of the Kafka cluster
Zookeeper is used by Kafka brokers to determine which broker is the leader of a given partition and topic and perform leader elections
Zookeeper stores configurations for topics and permissions
Zookeeper sends notifications to Kafka in case of changes (e.g. new topic, broker dies, broker comes up, delete topics, etc.…)

In this tutorial, you'll learn how to use the Kafka-python library to interact with a Kafka cluster. We'll start by setting up a Kafka cluster, then move on to producing and consuming messages using python code.

2.0. Setting up a Kafka server (Option 1)

Setting up a Kafka server can be a bit involved, but once set up, it can be run on any machine that has a Java Runtime Environment (JRE) installed. Here are the general steps to set up a Kafka server:

There are two options of Kafka: One by Apache foundation and other by Confluent as a package. For this tutorial, I will go with the one provided by Apache foundation.

Download the latest version of Kafka from the Apache Kafka.
Extract the downloaded file to a directory on your machine
Navigate to the extracted folder directory ```bash

cd /kafka_

4. Start the zookeeper server by running the following command from the Kafka directory:
    ```bash


bin/zookeeper-server-start.sh config/zookeeper.properties

Start the Kafka server by running the following command from the Kafka directory: ```bash

bin/kafka-server-start.sh config/server.properties

By default, the server will listen on port `9092` and the zookeeper server on port `2181`. You can change these settings by modifying the server.properties and zookeeper.properties file respectively.

If you want to create a multi-node Kafka cluster, you'll need to set up and configure additional Kafka brokers on separate machines. Each broker will need its own unique broker ID, and you'll need to configure the cluster so that the brokers can communicate with each other.

You can also run Kafka on cloud providers like AWS, GCP, or Azure. You can use their managed Kafka service or launch kafka clusters on their virtual machines.

It's worth noting that running a kafka server in production requires a lot of configuration and monitoring, so it's recommended to use a managed service or use Confluent Platform which is a more complete distribution of Apache Kafka that includes additional features and management tools.

---

## 2.1. Setting up a Kafka server using Docker (Option 2)

Setting up a Kafka server using Docker can be a convenient way to quickly spin up a Kafka cluster for testing or development purposes. Here are the general steps to set up a Kafka server using Docker:

1. Install [Docker](https://docs.docker.com/engine/install/) on your machine if it's not already installed.
2. Install [docker-compose](https://docs.docker.com/compose/)
3. Create a `docker-compose.yml` file and add the following `zookeeper` and `kafkga` config
    ```yaml


version: '3.7'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
    restart: on-failure
  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    restart: on-failure

Start a zookeeper & kafka container by running the following command: ```bash

docker-compose up -d

    This command will start a new container named "kafka" and map port `29092` of the host machine to port '29092' of the container. It also links the "kafka" container to the "zookeeper" container, so that the Kafka container can connect to the zookeeper container.

5. Check the logs to see the zookeeper container has booted up successfully
    ```bash


 docker logs zookeeper
## Output -> 
## ...
## 2023-01-25 13:22:48 [2023-01-25 10:22:48,634] INFO binding to port 0.0.0.0/0.0.0.0:32181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
## ...

Check the logs to see the kafka server has booted up successfully ```bash

docker logs zookeeper

Output ->

## ....
## [2023-01-25 13:00,295] INFO [Kafka Server 1], started (kafka.server.KafkaServer)
## [2023-01-25 13:31:00,295] INFO [Kafka Server 1], started (kafka.server.KafkaServer)
## ...


> It's worth noting that the above setup is for development and testing purposes, running a Kafka server in production requires a lot of configuration and monitoring, so it's recommended to use a managed service like AWS MSK, GCP, or Azure or use Confluent Platform for more advanced features and management tools.

---

## Simple Python Project
---

**_NOTE:_** Ensure you get the right Kafka version for the step below. You can get this bu using the following command
```bash


docker exec kafka kafka-topics --version # with docker setup
bin/kafka-topics.sh --version # If you used the manual setup

The output should be something like this:



7.3.1-ccs (Commit:8628b0341c3c46766f141043367cc0052f75b090)

3.1. Installing the Python Kafka library (e.g. Kafka-python)

Make sure that you have the kafka-python library installed, you can install it via pip:



pip install kafka-python

3.2. Producing messages to a Kafka topic

Create a new file named producer.py and add the following code:



import os
import time
import random
import json
from kafka import KafkaProducer
KAFKA_BOOTSTRAP_SERVERS = os.environ.get("KAFKA_BOOTSTRAP_SERVERS", "localhost:29092")
KAFKA_TOPIC_TEST = os.environ.get("KAFKA_TOPIC_TEST", "test")
KAFKA_API_VERSION = os.environ.get("KAFKA_API_VERSION", "7.3.1")
producer = KafkaProducer(
    bootstrap_servers=[KAFKA_BOOTSTRAP_SERVERS],
    api_version=KAFKA_API_VERSION,
)
i = 0
while i <= 30:
    producer.send(
        KAFKA_TOPIC_TEST,
        json.dumps({"message": f"Hello, Kafka! - test {i}"}).encode("utf-8"),
    )
    i += 1
    time.sleep(random.randint(1, 5))
producer.flush()

This code creates a new Kafka producer, which is connected to the Kafka cluster specified by the "bootstrap_servers" parameter. The code then sends a message "Hello, Kafka!" to the topic "test". We simulate events being published by taking take a random time break between 1 to 5 on every iteration.

3.3. Consuming messages from a Kafka topic

Create a new file named consumer.py and add the following code:



import os
from kafka import KafkaConsumer
KAFKA_BOOTSTRAP_SERVERS = os.environ.get("KAFKA_BOOTSTRAP_SERVERS", "localhost:29092")
KAFKA_TOPIC_TEST = os.environ.get("KAFKA_TOPIC_TEST", "test")
KAFKA_API_VERSION = os.environ.get("KAFKA_API_VERSION", "7.3.1")
consumer = KafkaConsumer(
    KAFKA_TOPIC_TEST,
    bootstrap_servers=[KAFKA_BOOTSTRAP_SERVERS],
    api_version=KAFKA_API_VERSION,
    auto_offset_reset="earliest",
    enable_auto_commit=True,
)
for message in consumer:
    print(message.value.decode("utf-8"))

This code creates a new Kafka consumer, which is connected to the Kafka cluster specified by the "bootstrap_servers" parameter. The code then subscribes to the topic "test" and continuously polls for new messages. Each message received is printed to the console

Run the producer.py script to send the message to the kafka topic ```bash

python producer.py

2. Run the consumer.py script to consume the message from the kafka topic
    ```bash


python consumer.py

This will start the consumer, which will continuously poll for new messages from the "test" topic. Each time a message is received, it will be printed to the console.

6. Advanced Kafka features

Kafka provides many advanced features that you can use in your application, including:

Compression: you can compress the messages before sending to kafka to save on bandwidth
Message keys: you can specify a key for each message which can be used for partitioning the messages
Consumer Groups: allows you to have multiple consumers reading from the same topic, enabling parallel processing and load balancing
Fault Tolerance: kafka is designed to be fault-tolerant and can handle failures of individual nodes without losing messages or affecting performance

7. Error handling and troubleshooting

As with any distributed system, errors can occur while working with Kafka. Some common errors you may encounter include:

Connection errors: These occur when the producer or consumer is unable to connect to the Kafka cluster.
Leader not available: This error occurs when the leader broker for a partition is not available
To troubleshoot these errors, you can check the logs of the Kafka broker and consumer, and check for error messages. The Kafka-python library also provides several exception classes that you can catch and handle in your code, such as KafkaError and KafkaTimeoutError.

8. Conclusion and resources for further learning

In this tutorial, we've covered the basics of using the Kafka-python library to interact with a Kafka cluster. We've shown how to set up a Kafka cluster, produce and consume messages, and use some of the advanced features provided by Kafka.

To continue learning about Kafka, you can check out the official Kafka documentation, as well as the documentation for the Kafka-python library. There are also many tutorials and blog posts available online that cover different aspects of using Kafka.

I hope this tutorial has been helpful in getting you started with using Kafka and the Kafka-python library. Let me know if there is anything else that you would like me to include or explain in more detail.

PS: Here's a GitHub link to the final project

Hesbon5600 / kafka-python

Apache Kafka with Python

Screen.Recording.2023-01-25.at.16.55.49.mov

Tutorial Link (Apache Kafka with Python)

How to set up the project

Features

python 3.10
poetry as dependency manager

PROJECT SETUP

clone the repository

git clone https://github.com/Hesbon5600/kafka-python.git

cd into the directory

cd kafka-python

create environment variables

On Unix or MacOS, run:

cp .env.example .env

You can edit whatever values you like in there.

Note: There is no space next to '='

On terminal

source .env

VIRTUAL ENVIRONMENT

To Create:

make env

Installing dependencies:

make install

View on GitHub

Customizing mozilla-django-oidc

Hesbon — Fri, 30 Dec 2022 12:07:27 +0000

Prerequisite:

In order to implement OIDC authentication using Django and mozilla-django-oidc, I would recommend reading this article: [How to implement OIDC authentication with Django and Okta].

How to implement OIDC authentication with Django and Okta

Hesbon ・ Dec 13 '22

#ai #programming

The mozilla-django-oidc package provides a Django authentication backend that allows you to use OIDC as the authentication mechanism for your Django application. It handles the details of verifying the user's identity and obtaining the necessary information from the OIDC provider. The backend can be used to authenticate users in Django's built-in authentication system, or you can use it to add OIDC support to your own custom authentication backend.
Ensure you have an Okta developer account and configure an application. Follow the steps detailed in the article above.

Note: Here's a link to the final project on Github.
Hesbon5600 / oidc-connect
OIDC Oauth2 authentication using Django and mozilla-django-oidc with Okta

Tutorial 1: How to implement OIDC authentication with Django and Okta

Tutorial 2: Customizing mozilla-django-oidc

How to set up the project

Features

python 3.10

poetry as dependency manager

PROJECT SETUP

clone the repository
git clone https://github.com/Hesbon5600/oidc-connect.git
cd into the directory
cd oidc-connect
create environment variables

On Unix or MacOS, run:
cp .env.example .env
You can edit whatever values you like in there.

Note: There is no space next to '='

On terminal
source .env
VIRTUAL ENVIRONMENT

To Create:
make env
To Activate:
source ./env/bin/activate
Installing dependencies:
make install
MIGRATIONS - DATABASE

Make migrations
make makemigrations
THE APPLICATION

run application
make run
View on GitHub
In the Okta application you created, ensure you have enabled token refresh as shown below.

Introduction

In this article, I am going to address three main areas:

Storing the user's OIDC access_token, id_token, and refresh_token. By default the package only allows us to store the access token and id token. Since we want to implement session refresh and logout, we will need to store the "access" and "refresh" tokens.
OIDC Logout. mozilla_django_oidc logout only terminates the existing session in our Django app. We want to also terminate the session in the Okta authorization server.
Implementing our own session refresh middleware based on the mozilla_django_oidc.middleware.SessionRefresh middleware. By default, the package forces a user to login when their token expires. We might want to automatically refresh the user's access token as long as the refresh token is still valid. We will also rotate the refresh token after each use.

When using the Okta authorization server, the lifetime of the JWT tokens is hard-coded to the following values: ID Token: 60 minutes. Access Token: 60 minutes. Refresh Token: 100 days.

Part 1: Storing access and refresh tokens

For this, we need to create our own custom authentication backend that subclasses mozilla_django_oidc.auth.OIDCAuthenticationBackend and override the authenticate method.

# oidc_app/core/backends.py

import logging

from django.core.exceptions import SuspiciousOperation
from django.urls import reverse
from mozilla_django_oidc.auth import OIDCAuthenticationBackend
from mozilla_django_oidc.utils import absolutify


LOGGER = logging.getLogger(__name__)


class CustomOIDCAuthenticationBackend(OIDCAuthenticationBackend):
    """Custom OIDC authentication backend."""

    def authenticate(self, request, **kwargs):
        """Authenticates a user based on the OIDC code flow."""

        self.request = request
        if not self.request:
            return None

        state = self.request.GET.get("state")
        code = self.request.GET.get("code")
        nonce = kwargs.pop("nonce", None)

        if not code or not state:
            return None

        reverse_url = self.get_settings(
            "OIDC_AUTHENTICATION_CALLBACK_URL", "oidc_authentication_callback"
        )

        token_payload = {
            "client_id": self.OIDC_RP_CLIENT_ID,
            "client_secret": self.OIDC_RP_CLIENT_SECRET,
            "grant_type": "authorization_code",
            "code": code,
            "redirect_uri": absolutify(self.request, reverse(reverse_url)),
        }

        # Get the token
        token_info = self.get_token(token_payload)
        id_token = token_info.get("id_token")
        access_token = token_info.get("access_token")
        refresh_token = token_info.get("refresh_token")

        # Validate the token
        payload = self.verify_token(id_token, nonce=nonce)

        if payload:
            self.store_tokens(access_token, id_token, refresh_token) # <--- HERE: store tokens
            try:
                return self.get_or_create_user(access_token, id_token, payload)
            except SuspiciousOperation as exc:
                LOGGER.warning("failed to get or create user: %s", exc)
                return None

        return None

    def store_tokens(self, access_token, id_token, refresh_token): # <--- HERE: store tokens
        """Store OIDC tokens."""
        session = self.request.session

        if self.get_settings("OIDC_STORE_ACCESS_TOKEN", True):
            session["oidc_access_token"] = access_token

        if self.get_settings("OIDC_STORE_ID_TOKEN", False):
            session["oidc_id_token"] = id_token

        if self.get_settings("OIDC_STORE_REFRESH_TOKEN", True): # <--- HERE: Add refresh token option
            session["oidc_refresh_token"] = refresh_token

Update the settings file to add the option of storing the tokens and add the new authentication backend. We also need to explicitly tell "mozilla_django_oidc" to set the token expiry time to 1 hour from the authentication time. Furthermore, since Okta does not automatically issue a refresh token, you need to add offline_access scope to your OIDC_RP_SCOPES value as follows:

# oidc_app/settings.py

...
AUTHENTICATION_BACKENDS = (
    "oidc_app.core.backends.CustomOIDCAuthenticationBackend",
    "django.contrib.auth.backends.ModelBackend",
)
...
OIDC_RENEW_ID_TOKEN_EXPIRY_SECONDS = 60 * 60  # 1 hour
OIDC_STORE_ACCESS_TOKEN = os.environ.get("OIDC_STORE_ACCESS_TOKEN", True) # Store the access token in the OIDC backend
OIDC_STORE_ID_TOKEN = os.environ.get("OIDC_STORE_ID_TOKEN", True) # Store the ID token in the OIDC backend
OIDC_STORE_REFRESH_TOKEN = os.environ.get("OIDC_STORE_REFRESH_TOKEN", True) # Store the refresh token in the OIDC backend
OIDC_RP_SCOPES = os.environ.get("OIDC_RP_SCOPES", "openid profile email offline_access")  # The OIDC scopes to request

...

That's it. The token sill be stored in the user session whenever the user authenticates.

Part 2: OIDC Logout

Although both refresh tokens and access tokens have an expiration time, it is highly advised to revoke these tokens once they aren’t needed (e.g. in case the user logs out of your application).

Create a logout view that handles; django logout and oidc logout.

# oidc_app/authentication/views.py
...
from django.contrib.auth import logout
from django.http import HttpResponseRedirect
from django.contrib.auth.views import LogoutView
from django.utils.decorators import method_decorator
from django.views.decorators.cache import never_cache
from oidc_app.core.logout import oidc_logout # to be implemented
...
...
class LogoutViewSet(LogoutView):
    """
    API endpoint that handles user logout
    """
    @method_decorator(never_cache)
    def dispatch(self, request, *args, **kwargs):
        """
        Override the dispatch method to add the okta logout functionality
        """

        oidc_logout(request) # This is the function that does the oidc logout
        logout(request) # This is the django logout
        redirect_to = self.get_success_url()
        if redirect_to != request.get_full_path():
            # Redirect to target page once the session has been cleared.
            return HttpResponseRedirect(redirect_to)
        return super().dispatch(request, *args, **kwargs)

Update the urls file to include the new logout viewset

# oidc_app/urls.py
...
from oidc_app.authentication import views as auth_views

urlpatterns = [
   ...
    path("logout", auth_views.LogoutViewSet.as_view(), name="logout"),

Okta Logout

Add the token revoke endpoint to the settings file as follows.

# oidc_app/settings.py
OIDC_OP_TOKEN_REVOKE_ENDPOINT = (
    f"https://{OKTA_DOMAIN}/oauth2/default/v1/revoke"  # The OIDC token revocation endpoint
)

With all the configurations in place, we need to make a request to Okta and revoking the access and refresh tokens

# oidc_app/core/oidc_logout.py

import requests
import logging
from django.conf import settings

LOGGER = logging.getLogger(__name__)


def revoke_token(token_type, token):
    """Revoke an OIDC token."""

    token_revoke_payload = {
        "client_id": settings.OIDC_RP_CLIENT_ID,
        "client_secret": settings.OIDC_RP_CLIENT_SECRET,
        "token": token,
        "token_type_hint": token_type,
    }

    try:
        response = requests.post(
            settings.OIDC_OP_TOKEN_REVOKE_ENDPOINT, data=token_revoke_payload
        )
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        LOGGER.error("Failed to revoke token: %s", e)


def oidc_logout(request):
    """Logout the user."""

    token_types = {
        "refresh_token": request.session.get("oidc_refresh_token"),
        "access_token": request.session.get("oidc_access_token"),
    }

    for token_type, token in token_types.items():
        if token:
            LOGGER.info("Revoking token of type %s", token_type)
            revoke_token(token_type, token)

That's it!! When your user logs out, the access and refresh tokens will be revoked by Okta.

Part 3: Session Refresh middleware.

Access and ID tokens are JSON web tokens that are valid for a specific number of seconds. Typically, a user needs a new access token when they attempt to access a resource for the first time or after the previous access token that was granted to them expires. A refresh token is a special token that is used to obtain additional access tokens. This allows you to have short-lived access tokens without having to collect credentials every time one expires. You request a refresh token alongside the access and/or ID tokens as part of a user's initial authentication and authorization flow. Applications must then securely store refresh tokens since they allow users to remain authenticated.

Ideally, you would want to check the user session validity status whenever they make a request. Hence, we are going to subclass mozilla_django_oidc.middleware.SessionRefresh and handle the session refresh if the access token has expired.

# oidc_app/core/middleware.py

import logging
import time
from re import Pattern as re_Pattern

import requests
from django.urls import reverse
from django.utils.functional import cached_property
from mozilla_django_oidc.middleware import SessionRefresh as OIDCSessionRefresh

LOGGER = logging.getLogger(__name__)


class OIDCSessionRefreshMiddleware(OIDCSessionRefresh):
    @cached_property
    def exempt_urls(self):
        """Generate and return a set of url paths to exempt from SessionRefresh

        This takes the value of ``settings.OIDC_EXEMPT_URLS`` and appends three
        urls that mozilla-django-oidc uses. These values can be view names or
        absolute url paths.

        :returns: list of url paths (for example "/oidc/callback/")

        """
        exempt_urls = []
        for url in self.OIDC_EXEMPT_URLS:
            if not isinstance(url, re_Pattern):
                exempt_urls.append(url)
        return set(
            [url if url.startswith("/") else reverse(url) for url in exempt_urls]
        )

    def refresh_session(self, request):
        """Refresh the session with new data from the request session store."""

        refresh_token = request.session.get("oidc_refresh_token", None)

        token_refresh_payload = {
            "refresh_token": refresh_token,
            "client_id": self.get_settings("OIDC_RP_CLIENT_ID"),
            "client_secret": self.get_settings("OIDC_RP_CLIENT_SECRET"),
            "grant_type": "refresh_token",
        }

        try:
            response = requests.post(
                self.get_settings("OIDC_OP_TOKEN_ENDPOINT"), data=token_refresh_payload
            )
            response.raise_for_status()
        except requests.exceptions.RequestException as e:
            LOGGER.error("Failed to refresh session: %s", e)
            return False
        data = response.json()
        request.session.update(
            {
                "oidc_access_token": data.get("access_token"),
                "oidc_id_token_expiration": time.time() + data.get("expires_in"),
                "oidc_refresh_token": data.get("refresh_token"),
            }
        )
        return True

    def process_request(self, request):
        if not self.is_refreshable_url(request):
            LOGGER.debug("request is not refreshable")
            return

        expiration = request.session.get("oidc_id_token_expiration", 0)
        now = time.time()
        if expiration > now:
            # The id_token is still valid, so we don't have to do anything.
            LOGGER.debug("id token is still valid (%s > %s)", expiration, now)
            return

        LOGGER.debug("id token has expired")
        if not self.refresh_session(request):
            # If we can't refresh the session, then we need to reauthenticate the user.
            # As per the default OIDCSessionRefresh implementation.
            return super().process_request(request)

        LOGGER.debug("session refreshed")

We need to update the MIDDLEWARE settings to include the new middleware we just created. We also have a couple of URLs that needs to be skipped when doing the session refresh: oidc_authentication_callback, oidc_authentication_callback, and logout. Update your settings file with the list of these urls.

# oidc_app/settings.py

MIDDLEWARE = [
    ...
   "django.contrib.auth.middleware.AuthenticationMiddleware",
    "oidc_app.core.middleware.OIDCSessionRefreshMiddleware",
    ...
]

OIDC_EXEMPT_URLS = [
    "oidc_authentication_init",
    "oidc_authentication_callback",
    "logout",
]

Thats it! You now have a custom backend that stores the refresh and access token, a middleware that handles fetching a new refresh and access token as well as a logout view that invalidates the access and refresh tokens.

Next Up: Implement the Authorization Code with PKCE flow in Okta. Stay tunned.

Feel free to leave a comment or suggestion. Thank you!

How to implement OIDC authentication with Django and Okta

Hesbon — Tue, 13 Dec 2022 10:38:26 +0000

This article details how to implement OIDC authentication using Django and mozilla-django-oidc with Okta as our identity provider

Introduction

OpenID Connect (OIDC) is a protocol that allows a user to authenticate with a third-party service and then use that authentication to sign in to other services. OIDC is built on top of the OAuth2 protocol and adds an additional layer of authentication on top of it. This allows a user to not only grant permission for a service to access their data, but also to verify their identity.
Django is a popular web framework for building web applications in Python. It includes a robust authentication system that can be easily configured to support OIDC.
Mozilla-django-oidc is an open-source library that adds OIDC support to Django, making it easy to authenticate users with an OIDC provider.
Okta is an identity and access management platform that enables organizations to securely connect users to technology. It supports the OIDC protocol, which allows users to be authenticated and receive information about their identity and access rights across different applications.

Note: Here's a link to the final project on Github.

Hesbon5600 / oidc-connect

OIDC Oauth2 authentication using Django and mozilla-django-oidc with Okta

Tutorial 1: How to implement OIDC authentication with Django and Okta

Tutorial 2: Customizing mozilla-django-oidc

How to set up the project

Features

python 3.10

poetry as dependency manager

PROJECT SETUP

clone the repository

git clone https://github.com/Hesbon5600/oidc-connect.git

cd into the directory

cd oidc-connect

create environment variables

On Unix or MacOS, run:

cp .env.example .env

You can edit whatever values you like in there.

Note: There is no space next to '='

On terminal

source .env

VIRTUAL ENVIRONMENT

To Create:

make env

To Activate:

source ./env/bin/activate

Installing dependencies:

make install

MIGRATIONS - DATABASE

Make migrations

make makemigrations

THE APPLICATION

run application

make run

View on GitHub

Part 1 : Setting up an Okta Account

To set up an Okta account for our OIDC account, follow these steps:

Go to the Okta developer website and create an account. Use your gmail account in order to get a free developer account.
After you have created your Okta account, log in to the Okta dashboard.
Click on the Applications tab in the left menu and then click on the Create App Integration button.
Choose OIDC - OpenID Connect Sign-in method and Select an Application type of Web Application, then click Next
Enter an App integration name (e.g oidc-connect).
Enter the Sign-in redirect URIs for local development, such as http://localhost:8080/authorization-code/callback.
Optionally Enter the Sign-out redirect URIs for both local development, such as http://localhost:8080/signout/callback.
In the Assignments section, define the type of Controlled access for your app. Select the Everyone group for now. For more information, see the Assign app integrations topic in the Okta product documentation.
Click Save to create the app integration. The configuration pane for the integration opens after it's saved.

Note: Keep this pane open as you copy some values when configuring your app.

Part 2.0 : Setting up the Django application

To set up a Django application, follow these steps:

Install Django by running the following command:
```
pip install django
```
Create a new Django project by running the following command:
```
django-admin startproject oidc_app
```
This will create a new directory called "oidc_app" with the basic structure for a Django project. Change into the new directory by running the following command:
```
cd oidc_app
```
Apply the default database migrations using the command.
```
python manage.py migrate
```
Run the app on port 8080 using the following command.
```
python manage.py runserver 8080
```
That's it! Your Django application is now set up and you are ready to move on to the next step: configuring Django to use OIDC.

Part 2.2 : Creating a dummy login page and home screen

Create a superuser with the following command: bash python manage.py createsuperuser

Create a new app in your project directory by running the command:

python manage.py startapp authentication

Add authentication to the INSTALLED_APPS in settings.py as follows:

#oidc_app/settings.py
INSTALLED_APPS = [
...
"authentication",
]

Let's make our login page! By default, Django will look for auth templates within a templates folder. To create a login page, create a directory called templates, and within it a directory called registration.

Inside the registration directory, create a file called login.html. Add the following piece of code.

<!-- oidc_app/templates/registration/login.html -->
<h2>Log In</h2>
<form method="post">
{% csrf_token %}
{{ form.as_p }}
<button type="submit">Log In</button>
</form>

Update the settings.py file to tell Django to look for a templates folder inside the oidc_app directory. Update the DIRS setting within TEMPLATES as follows.
```
# django_project/settings.py
TEMPLATES = [
{
    ...
    'DIRS': [BASE_DIR.joinpath("oidc_app/templates")],
    ...
},
]
```
If you start the app and navigate to: http://127.0.0.1:8080/accounts/login/, You should see/test the login page.

For the home page, we need to make a file called home.html located in the templates folder. The home page will display a different message to logged out and logged in users.

<!-- oidc_app/templates/base.html -->
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>{% block title %}Django OIDC{% endblock %}</title>
</head>
<body>
<main>
    {% if user.is_authenticated %}
    Hi {{ user.username }} - {{ user.email }}!
    <p><a href="{% url 'logout' %}">Log Out</a></p>
    {% else %}
    <p>You are not logged in</p>
    <a href="{% url 'login' %}">Log In</a>
    {% endif %}
</main>
</body>
</html>

We now update the urls.py file in order to display the home page.

from django.contrib import admin
from django.urls import path, include
from django.views.generic.base import TemplateView
urlpatterns = [
path("admin/", admin.site.urls),
path("accounts/", include("django.contrib.auth.urls")),
path('', TemplateView.as_view(template_name='home.html'), name='index')
]

Now that we have a homepage view we should use that instead of the default setup. We update the settings file as follows:
```
#oidc_app/settings.py
LOGIN_REDIRECT_URL = "index"
LOGOUT_REDIRECT_URL = "index"
```
Let's create a superuser that we can use to test the authentication. Type in the command:
```
python manage.py createsuperuser
```
Enter your desired username, email address and password. You should now have a superuser created for your Django project.

Logged Out	Logged In

Note: The above steps are a very basic example of how to set up a Django application. In a real-world scenario, you would likely want to do more advanced configuration and set up additional features such as a database and static file serving. For more information on how to do this, please see the Django documentation.

Part 3.0 : Configuring Django to use OIDC.

Note: We will be using the OpenID Connect Authorization Code Flow. Refer to the Curity documentation for more details.

The Authorization Code Flow is the most advanced flow in OpenID Connect. It is also the most flexible, that allows both mobile and web clients to obtain tokens securely. It is split into two parts, the authorization flow that runs in the browser where the client redirects to the OpenID Provider (OP) and the OP redirects back when done, and the token flow which is a back-channel call from the client to the token endpoint of the OP.

mozilla-django-oidc package has abstracted all of the code flow steps required to enable OIDC authentication.

Once you’ve created and configured your Django application, you can start configuring mozilla-django-oidc. First, install mozilla-django-oidc using pip:
```
pip install mozilla-django-oidc
```

Make the following changes to your settings.py file:

# oidc_app/settings.py
# Add 'mozilla_django_oidc' to INSTALLED_APPS
INSTALLED_APPS = (
# ...
'django.contrib.auth',
'mozilla_django_oidc',  # Load after auth
# ...
)
# Add 'mozilla_django_oidc' authentication backend
AUTHENTICATION_BACKENDS = (
'mozilla_django_oidc.auth.OIDCAuthenticationBackend',
# ...
)
OKTA_DOMAIN =  "[Your Okta domain]"
OIDC_RP_CLIENT_ID = "[Your Okta application’s client ID]"
OIDC_RP_CLIENT_SECRET = "[Your Okta application’s client secret]"
OIDC_RP_SIGN_ALGO = "RS256"
OIDC_OP_AUTHORIZATION_ENDPOINT = f"https://{OKTA_DOMAIN}/oauth2/default/v1/authorize" # The OIDC authorization endpoint
OIDC_RP_TOKEN_ENDPOINT = f"https://{OKTA_DOMAIN}/oauth2/default/v1/token" # The OIDC token endpoint
OIDC_OP_USER_ENDPOINT = f"https://{OKTA_DOMAIN}/oauth2/default/v1/userinfo" # The OIDC userinfo endpoint
OIDC_OP_TOKEN_ENDPOINT = f"https://{OKTA_DOMAIN}/oauth2/default/v1/token" # The OIDC token endpoint
OIDC_OP_JWKS_ENDPOINT = f"https://{OKTA_DOMAIN}/oauth2/default/v1/keys" # The OIDC JWKS endpoint

Next, edit your urls.py and add the following:

#oidc_app/settings.py
from mozilla_django_oidc import views as oidc_views
urlpatterns = [
# ...
path("authorization-code/authenticate/", oidc_views.OIDCAuthenticationRequestView.as_view(), name="oidc_authentication_init"),
path("authorization-code/callback/", oidc_views.OIDCAuthenticationCallbackView.as_view(), name="oidc_authentication_callback"),
# ...
]

We need to add Login with Okta option to our login page. Edit the login.html file as follows:

<!-- oidc_app/templates/registration/login.html -->
<h2>Log In</h2>
<form method="post">
{% csrf_token %}
{{ form.as_p }}
<button type="submit">Log In</button>
<br />
or
<br />
<a href="{% url 'oidc_authentication_init' %}">
    <button type="button">Log In with Ota</button>
</a>
</form>

Note:

You can get your Okta Domain by clicking on your profile section on the top right side of the page.
The Okta client ID and client secret are fund in the application settings (oidc-connect app we created)

Part 3.1 Testing the Okta integration

Your login page should now have a 'Login with Okta' option
When you click the 'Login with Okta' button, you should be redirected to the okta domain for authentication.
Enter your username and password or sign in with Google.
After successful authentication, you should be redirected to the home page. Note: The Username is a base64 encoded sha224 of the email address. More information can be found here

That's it! You now have a Django app that authenticated with Okta.

The mozilla-django-oidc package can be further customized to better suit your application needs.

Next Up:

Customizing mozilla-django-oidc

Hesbon ・ Dec 30 '22

#python #django #okta #tutorial

Feel free to leave a comment or suggestion. Thank you!