DEV Community: Muhammad Rahiman

Developing REST API Using Django REST Framework (DRF)

Muhammad Rahiman — Sun, 11 Jul 2021 14:03:01 +0000

Introduction

This is a sample project guide for exploring the development of REST APIs using Django REST Framework (DRF). GitHub repository link.

Setting Up Django

1. Create new Django Project

To create your Django project from scratch (make sure to have Django installed):

$ django-admin startproject project_name

2. Create new Virtual Environment

Next, navigate into the newly created project folder. Then, create a new virtual environment.

Using virtualenv:

$ cd project_name
$ virtualenv ENV_NAME
$ source ENV_NAME/bin/activate

Using Python venv:

$ cd project_name
$ python3 -m venv
$ source ENV_NAME/bin/activate

3. Install PIP Dependencies

Run the following command preferably inside your virtual environment to install all relevant dependencies:

Using pipenv:

$ pipenv install -r requirements.txt # (Python 2)
$ pipenv3 install -r requirements.txt # (Python 3)

Using venv:

$ pip install -r requirements.txt # (Python 2)
$ pip3 install -r requirements.txt # (Python 3)

4. Start New Django App

After that, start a new Django app. We will also run migrations and start up the server:

$ python manage.py startapp app_name
$ python manage.py migrate
$ python manage.py runserver

If everything works well, we should see an instance of a Django application running on this address — http://localhost:8000

5. Configure `settings.py`

Once you’ve created the app, you need to install it in your project. In project_name/settings.py, add the following line of code under INSTALLED_APPS:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'app_name.apps.App_name',
]

That line of code means that your project now knows that the app you just created exists.

Add templates folder directory in TEMPLATES (project_name/settings.py)

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': ['templates/'], # HERE
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

Add static and media folder directory in STATIC_ROOT (project_name/settings.py)

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

5. Configure `urls.py`

Add desired URL for the app (project_name/urls.py)

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('INSERT_URL', include('APP_NAME.urls')),
]

Finally, create new urls.py for the app (app_name/urls.py).

Setting Up Django REST Framework

1. Install Django REST Framework

$ pip install djangorestframework

2. Configure `settings.py`

Configure INSTALLED_APPS as follows. Then, remigrate using the command python manage.py migrate.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
    # To include support for DEFAULT_AUTHENTICATION_CLASSES, must include this and migrate!
    'rest_framework.authtoken',
    'app_name.apps.AppName',
]

Next, we need to add REST_FRAMEWORK settings as follows. Descriptions of the settings as included for your quick reference.

# ...

# Disable the Browsable HTML API
DEFAULT_RENDERER_CLASSES = (
    'rest_framework.renderers.JSONRenderer',
)

# Only enable the browseable HTML API in dev (DEBUG=True)
if DEBUG:
    DEFAULT_RENDERER_CLASSES = DEFAULT_RENDERER_CLASSES + (
        'rest_framework.renderers.BrowsableAPIRenderer',
    )

REST_FRAMEWORK = {
    # Disable the Browsable HTML API UI when in production (DEBUG=False)
    'DEFAULT_RENDERER_CLASSES': DEFAULT_RENDERER_CLASSES,

    # Pagination allows you to control how many objects per page are returned
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination',
    'PAGE_SIZE': 5,

    # The default permission policy may be set globally, using the DEFAULT_PERMISSION_CLASSES setting
    # To be explained in detail later in the guide
    # 'DEFAULT_PERMISSION_CLASSES': [
    # 'rest_framework.permissions.AllowAny',
    # 'rest_framework.permissions.IsAuthenticated',
    # 'rest_framework.permissions.IsAuthenticatedOrReadOnly',
    # 'rest_framework.permissions.DjangoModelPermissions',
    # 'rest_framework.permissions.DjangoModelPermissionsOrAnonReadOnly',
    # 'rest_framework.permissions.DjangoObjectPermissions',
    # 'rest_framework.permissions.TokenHasReadWriteScope',
    # ],

    # A list or tuple of authentication classes, that determines the default set of authenticators used when accessing the request.user or request.auth properties.
    # The default authentication schemes may be set globally, using the DEFAULT_AUTHENTICATION_CLASSES setting
    # To be explained in detail later in the guide

    'DEFAULT_AUTHENTICATION_CLASSES': (
        # 'rest_framework.authentication.BasicAuthentication',
        # 'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication',
    ),

# ...

Browserable HTML API is the GUI that is included in DRF which allows users to interact with the API and for the API to return a fully web-browsable HTML representation. While this is useful, this should not be accessible during production as we do not want any user to access the API without authentication (more on authentication later in the guide).

Hence, we need to override the DEFAULT_RENDERER_CLASSES value from 'rest_framework.renderers.BrowsableAPIRenderer' to 'rest_framework.renderers.JSONRenderer' depending on which environment the web app is running at.

3. Create a Model

In this guide, we will be creating a model for a simple Todo list. However, feel free to make a model of your own preference.

Go to app_name/models.py and create your model similar to the following:

from django.db import models

class Todo(models.Model):
    title = models.CharField(max_length=120)
    description = models.TextField()
    completed = models.BooleanField(default=False)

    def __str__(self):
        return self.title + " (" + str(self.id) + ")"

Refer to this guide for more information on Django Models.

4. Configure Serializers

Serializers allow complex data such as querysets and model instances to be converted to native Python datatypes that can then be easily rendered into JSON, XML or other content types.

Serializers also provide deserialization, allowing parsed data to be converted back into complex types, after first validating the incoming data.

To create our serializers, create a file called serializers.py inside app_name:

$ touch serializers.py

There are 2 main types of serializers:

HyperlinkedModelSerializer

The HyperlinkedModelSerializer class is similar to the ModelSerializer class except that it uses hyperlinks
to represent relationships, rather than primary keys.

ModelSerializer

The ModelSerializer class provides a shortcut that lets you automatically create a Serializer class with
fields that correspond to the Model fields.

In this guide, we will be using ModelSerializer instead:

from rest_framework import serializers
from django.contrib.auth.models import User, Group
from .models import *

class TodoSerializer(serializers.ModelSerializer):
    class Meta:
        model = Todo
        # fields = ('id', 'title', 'description', 'completed')
        # Shortcut for getting all fields
        fields = '__all__'

5. Configure Views (`views.py`)

There are various ways to configure the views for the APIs. We can categorise them into 2 types:

Function-based views (FBVs)
Class-based views (CBVs)

APIView
ModelViewSet
Generics

Regardless of which type of view you choose, we typically need to develop 2 views:

List (for listing all objects/instances of the model)
Detail (for viewing a specific object/instance of the model)

Let's take a look at each type in details:

Function-based views (@api_view decorator)

Function-based views or FBVs are popular for Django newcomers as it is verbose and easy to understand. REST framework provides the @api_view decorator for working with function based views. The wrapper providse a few bits of functionality such as making sure you receive Request instances in your view, and adding context to Response objects so that content negotiation can be performed.

The following is an example of the FBV for our Todo app:

    from rest_framework import status
    from rest_framework.decorators import api_view
    from rest_framework.response import Response
    from .models import *
    from .serializers import *

    # FBV for list of all Todo objects
    @api_view(['GET', 'POST'])

        def todo_list(request):
        """
        List all code todos, or create a new todo.
        """
        if request.method == 'GET':
            todos = Todo.objects.all()
            serializer = TodoSerializer(todos, many=True)
            return Response(serializer.data)

        elif request.method == 'POST':
            serializer = TodoSerializer(data=request.data)
            if serializer.is_valid():
                serializer.save()
                return Response(serializer.data, status=status.HTTP_201_CREATED)
            return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

        @api_view(['GET', 'PUT', 'DELETE'])
        def todo_detail(request, pk):
            """
            Retrieve, update or delete a code todo.
            """
            try:
                todo = Todo.objects.get(pk=pk)
            except todo.DoesNotExist:
                return Response(status=status.HTTP_404_NOT_FOUND)

            if request.method == 'GET':
                serializer = TodoSerializer(todo)
                return Response(serializer.data)

            elif request.method == 'PUT':
                serializer = TodoSerializer(todo, data=request.data)
                if serializer.is_valid():
                    serializer.save()
                    return Response(serializer.data)
                return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

            elif request.method == 'DELETE':
                todo.delete()
                return Response(status=status.HTTP_204_NO_CONTENT)

Class-based views (APIView)

We can also write our API views using class-based views, rather than function based views. As we'll see this is a powerful pattern that allows us to reuse common functionality, and helps us keep our code DRY (Don't Repeat Yourself). The APIView class is the wrapper used for working with class-based views.

   from rest_framework import status
   from rest_framework.decorators import api_view
   from rest_framework.response import Response
   from .models import *
   from .serializers import *

   class TodoList(APIView):
       """
       List all todos, or create a new todo.
       """
       def get(self, request, format=None):
           todos = Todo.objects.all()
           serializer = TodoSerializer(todos, many=True)
           return Response(serializer.data)

       def post(self, request, format=None):
           serializer = TodoSerializer(data=request.data)
           if serializer.is_valid():
               serializer.save()
               return Response(serializer.data, status=status.HTTP_201_CREATED)
           return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

   class TodoDetail(APIView):
       """
       Retrieve, update or delete a todo instance.
       """
       def get_object(self, pk):
           try:
               return Todo.objects.get(pk=pk)
           except Todo.DoesNotExist:
               raise Http404

       def get(self, request, pk, format=None):
           todo = self.get_object(pk)
           serializer = TodoSerializer(todo)
           return Response(serializer.data)

       def put(self, request, pk, format=None):
           todo = self.get_object(pk)
           serializer = TodoSerializer(todo, data=request.data)
           if serializer.is_valid():
               serializer.save()
               return Response(serializer.data)
           return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

       def delete(self, request, pk, format=None):
           todo = self.get_object(pk)
           todo.delete()
           return Response(status=status.HTTP_204_NO_CONTENT)

Class-based views (ModelViewSets)

ModelViewSet inherits from GenericAPIView and includes implementations for various actions. In other words, you don't need implement basic actions as list, retrieve, create, update or destroy.

In comparison with APIView, APIView allow us to define functions that match standard HTTP methods like GET, POST, PUT, PATCH, etc. Meanwhile, because CRUD is so common, DRF provides ModelViewSet which allow us to define functions that match to common API object actions like LIST, CREATE, RETRIEVE, UPDATE, etc.

When compared with GenericViewSet (the 4th type), to create CRUD, GenericViewSet needs two classes(ListCreateAPIView and RetrieveUpdateDestroyAPIView).
But ModelViewSet needs only one class(ModelViewSet)

The biggest difference between ModelViewSet and other types of views is that ModelViewSet support creating url pattern automatically with DRF router while others require you to configure the API endpoint manually yourself in urls.py.

Here's an example of what a ModelViewSet views look like:

   from .models import *
   from .serializers import *
   from django.shortcuts import render
   from django.contrib.auth.models import User, Group
   from rest_framework import permissions, viewsets

   class UserViewSet(viewsets.ModelViewSet):
       """
       API endpoint that allows users to be viewed or edited.
       """
       queryset = User.objects.all().order_by('-date_joined')
       serializer_class = UserSerializer

   class GroupViewSet(viewsets.ModelViewSet):
       """
       API endpoint that allows groups to be viewed or edited.
       """
       queryset = Group.objects.all()
       serializer_class = GroupSerializer

Class-based views (GenericViewSet)

REST framework provides a set of already mixed-in generic class-based views that we can use to trim down our views.py module even more.

GenericViewGenericsSet inherits from GenericAPIView but does not provide any implementations of basic actions. Just only get_object, get_queryset.

To create CRUD, Generics needs two classes(ListCreateAPIView and RetrieveUpdateDestroyAPIView).

This is the simplest way for creating a view for an API in DRF (literally only with 2 lines of code!) that balances between simplicity and customization (for the API endpoint/URL).

The following is an example of how a GenericViewSet for our Todo app API:

   from .models import *
   from .serializers import *
   from django.shortcuts import render
   from rest_framework.response import Response
   from rest_framework import generics, permissions, status

   class TodoList(generics.ListCreateAPIView):
       queryset = Todo.objects.all()
       serializer_class = TodoSerializer


   class TodoDetail(generics.RetrieveUpdateDestroyAPIView):
       queryset = Todo.objects.all()
       serializer_class = TodoSerializer

6. Configure API Endpoints (`urls.py`)

Earlier, we have created a new urls.py inside our app. All we have to do next is to link the views we configured just now inside the file.

The following is an example of the url patterns for each type of view that we discussed:

# todo/urls.py
from . import views
from rest_framework import routers
from django.urls import path, include
# Rename views to avoid conflict with app views
from rest_framework.authtoken import views as rest_views

"""
When using viewsets instead of views, we can automatically generate the URL conf for our API, by simply registering the viewsets with a router class.

If we need more control over the API URLs we can simply drop down to using regular class-based views (APIViews), and writing the URL conf explicitly.
"""

router = routers.DefaultRouter()
router.register(r'users', views.UserViewSet)
router.register(r'groups', views.GroupViewSet)

# Rename views to avoid conflict with app views
from rest_framework.authtoken import views as rest_views

urlpatterns = [
    # URLs for class-based views (Generics, APIViews)
    # http://localhost:8000/todo/
    # http://localhost:8000/todo/<int:pk>
    path('todo/', views.TodoList.as_view(), name='todo_list'),
    path('todo/<int:pk>', views.TodoDetail.as_view(), name='todo_detail'),

    # URLs for class-based views (ModelViewSets)
    # http://localhost:8000/general/users/
    # http://localhost:8000/general/groups/
    path('general/', include(router.urls)),

    # Include default login and logout views for use with the browsable API. 
    # Optional, but useful if your API requires authentication and you want to use the browsable API.
    path('api-auth/', include('rest_framework.urls', namespace='rest_framework')),
    # API to generate auth token from user. Note that the URL part of the pattern can be whatever you want to use.
    path('api-token-auth/', rest_views.obtain_auth_token, name='api-token-auth'),

    # URLs for function-based views
    # http://localhost:8000/todo/
    # http://localhost:8000/todo/<int:pk>
    # path('todo/', views.todo_list),
    # path('todo/<int:pk>', views.todo_detail),
]

Note that regardless of which type of CBV you use, the method of writing the URL pattern is the same. The exact URL pattern is not fixed and can be customised to your preference.

7. Test API Connection

At this point, we pretty much have completed the basic configuration of the REST API. Time to test and see if it works!

Run Django server using python manage.py runserver command then navigate to the API endpoint based on the URL pattern that you configured. For this project, it is http://localhost:8000/todo/ or http://127.0.0.1:8000/todo/

IMPORTANT!: Always remember that API endpoints MUST end with a slash '/' or else it will not work!

You should be seeing something similar like this:

The HTML you are seeing is the Browsable HTML API that we talked about earlier. You can click the 'GET' button to retrieve all Todo instances in our model as well filling in the 'POST' form and click the 'POST' button to submit a new Todo instance.

Another alternate and more popular way of testing APIs is via the command line interface (CLI):

Using HTTPie (recommended):
```
$ http GET localhost:8000/todo/
```
Using curl:
```
$ curl -X GET localhost:8000/todo/
```

For GET request, you should be seeing something like this:

Using HTTPie (recommended):

HTTP/1.1 200 OK
Allow: GET, POST, HEAD, OPTIONS
Content-Length: 153
Content-Type: application/json
Date: Sat, 10 Jul 2021 03:56:59 GMT
Referrer-Policy: same-origin
Server: WSGIServer/0.2 CPython/3.9.5
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "completed": false,
            "description": "FYP2 task for Aug",
            "id": 7,
            "title": "Research on Django-React stack"
        }
    ]
}

Using curl:

{"count":1,"next":null,"previous":null,"results":[{"id":7,"title":"Research on Django-React stack","description":"FYP2 task for Aug","completed":false}]}%

Same goes for POST requests:

Using HTTPie (recommended):

http POST localhost:8000/todo/ title="Research on RF for EDM" description="FYP2 task for Aug"

HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Content-Length: 94
Content-Type: application/json
Date: Sat, 10 Jul 2021 04:01:48 GMT
Referrer-Policy: same-origin
Server: WSGIServer/0.2 CPython/3.9.5
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "completed": false,
    "description": "FYP2 task for Aug",
    "id": 10,
    "title": "Research on RF for EDM"
}

Using curl:

curl -X POST -d 'title=Research on SVM for EDM&description=FYP2 task for Sept' localhost:8000/todo/
{"id":11,"title":"Research on SVM for EDM","description":"FYP2 task for Sept","completed":false}%

Configure Permissions and Authentication

So far, we've successfully configured our REST API using the default permissions and authentication settings. In this basic configuration, DRF defaults to AllowAny permission class which will allow unrestricted access, regardless of if the request was authenticated or unauthenticated. This is certainly dangerous and not ideal for our web app especially during production. Hence, we need to configure the permissions for restricted access on our REST APIs.

1. Permissions

There are several types of permission classes available:

AllowAny

The AllowAny permission class will allow unrestricted access, regardless of if the request was authenticated or unauthenticated.

This permission is not strictly required, since you can achieve the same result by using an empty list or tuple for the permissions setting, but you may find it useful to specify this class because it makes the intention explicit.

IsAuthenticated

The IsAuthenticated permission class will deny permission to any unauthenticated user, and allow permission otherwise.

This permission is suitable if you want your API to only be accessible to registered users.

IsAdminUser

The IsAdminUser permission class will deny permission to any user, unless user.is_staff is True in which case permission will be allowed.

This permission is suitable is you want your API to only be accessible to a subset of trusted administrators.

IsAuthenticatedOrReadOnly

The IsAuthenticatedOrReadOnly will allow authenticated users to perform any request. Requests for unauthorised users will only be permitted if the request method is one of the "safe" methods; GET, HEAD or OPTIONS.

This permission is suitable if you want to your API to allow read permissions to anonymous users, and only allow write permissions to authenticated users.

DjangoModelPermissions

This permission class ties into Django's standard django.contrib.auth model permissions. This permission must only be applied to views that has a .queryset property set.

Authorization will only be granted if the user is authenticated and has the relevant model permissions assigned.

DjangoModelPermissionsOrAnonReadOnly

Similar to DjangoModelPermissions, but also allows unauthenticated users to have read-only access to the API.

DjangoObjectPermissions

This permission class ties into Django's standard object permissions framework that allows per-object permissions on models. In order to use this permission class,
you'll also need to add a permission backend that supports object-level permissions, such as django-guardian.

TokenHasReadWriteScope

This permission class is intended for use with either of the OAuthAuthentication and OAuth2Authentication classes, and ties into the scoping that their backends provide. More info about this here.

The permissions setting can be configured either at

global-level (at settings.py)

   REST_FRAMEWORK = {

    #...

    # The default permission policy may be set globally, using the DEFAULT_PERMISSION_CLASSES setting:
    # 'DEFAULT_PERMISSION_CLASSES': [
    # 'rest_framework.permissions.AllowAny',
    # 'rest_framework.permissions.IsAuthenticated',
    # 'rest_framework.permissions.IsAuthenticatedOrReadOnly',
    # 'rest_framework.permissions.DjangoModelPermissions',
    # 'rest_framework.permissions.DjangoModelPermissionsOrAnonReadOnly',
    # 'rest_framework.permissions.DjangoObjectPermissions',
    # 'rest_framework.permissions.TokenHasReadWriteScope',
    # ],

view-level (at views.py)

   class TodoList(generics.ListCreateAPIView):
       queryset = Todo.objects.all()
       # View-level permissions
       # IsAuthenticatedOrReadOnly, which will ensure that authenticated requests get read-write access,
       # and unauthenticated requests get read-only access.
       permission_classes = [permissions.IsAuthenticatedOrReadOnly]
       serializer_class = TodoSerializer


   class TodoDetail(generics.RetrieveUpdateDestroyAPIView):
       queryset = Todo.objects.all()
       # View-level permissions
       # IsAuthenticatedOrReadOnly, which will ensure that authenticated requests get read-write access,
       # and unauthenticated requests get read-only access.
       permission_classes = [permissions.IsAuthenticatedOrReadOnly]
       serializer_class = TodoSerializer

Once the permissions have been set, if you try to send a request (GET and/or POST, depending on permission class set), it will be blocked:

http POST localhost:8000/todo/ title="Research on Django-React stack" description="FYP2 task for Aug" 

HTTP/1.1 403 Forbidden
Allow: GET, POST, HEAD, OPTIONS
Content-Length: 58
Content-Type: application/json
Date: Sat, 10 Jul 2021 04:01:28 GMT
Referrer-Policy: same-origin
Server: WSGIServer/0.2 CPython/3.9.5
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "detail": "Authentication credentials were not provided."
}

2. Authentication

Authentication is the mechanism of associating an incoming request with a set of identifying credentials, such as the user the request came from, or the token that it was signed with. The permission and throttling policies can then use those credentials to determine if the request should be permitted.

REST framework provides a number of authentication schemes out of the box, and also allows you to implement custom schemes:

BasicAuthentication

This authentication scheme uses HTTP Basic Authentication, signed against a user's username and password. Basic authentication is generally only appropriate for testing.

SessionAuthentication

This authentication scheme uses Django's default session backend for authentication. Session authentication is appropriate for AJAX clients that are running in the same session context as your website.

TokenAuthentication

This authentication scheme uses a simple token-based HTTP Authentication scheme. Token authentication is appropriate for client-server setups, such as native desktop and mobile clients.

To use the TokenAuthentication scheme you'll need to configure the authentication classes to include TokenAuthentication, and additionally include rest_framework.authtoken in your INSTALLED_APPS setting inside settings.py. More on this later in the guide.

Note: Don't forget that authentication by itself won't allow or disallow an incoming request, it simply identifies the credentials that the request was made with.

Authentication can be configured globally in settings.py:

REST_FRAMEWORK = {
#...

    # The default permission policy may be set globally, using the DEFAULT_AUTHENTICATION_CLASSES setting:
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.BasicAuthentication',
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication',
    ),
}

One of the most common ways of authentication is via token authentication. We can easily do so via the following steps:

Set 'DEFAULT_AUTHENTICATION_CLASSES' to Token Authentication

    REST_FRAMEWORK = {
    #...

        # The default permission policy may be set globally, using the DEFAULT_AUTHENTICATION_CLASSES setting:
        'DEFAULT_AUTHENTICATION_CLASSES': (
            'rest_framework.authentication.TokenAuthentication',
        ),
    }

Configure app_name/urls.py

   from . import views
   from rest_framework import routers
   from django.urls import path, include
   # Rename views to avoid conflict with app views
   from rest_framework.authtoken import views as rest_views

   urlpatterns = [
       #...
       # Include default login and logout views for use with the browsable API. 
       # Optional, but useful if your API requires authentication and you want to use the browsable API.
       path('api-auth/', include('rest_framework.urls', namespace='rest_framework')),

       # API to generate auth token from user. Note that the URL part of the pattern can be whatever you want to use.
       path('api-token-auth/', rest_views.obtain_auth_token, name='api-token-auth'),
   ]

Generate token for user

- Using Django manage.py command:

    ```shell
    $ python manage.py drf_create_token USERNAME
    ```

- Using CLI

    ```shell
    $ http POST API_ENDPOINT username=USERNAME password=PASSWORD
    ```

- Using `views.py`

    ```python
    from rest_framework.authtoken.models import Token
    token = Token.objects.create(user=...)
    print(token.key)
    ```

Use token in accessing API

   $ http POST API_ENDPOINT model_fields 'Authorization: Token TOKEN'

Where API_ENDPOINT refers to the url pattern of the API e.g. localhost:8000/todo/

API Documentation using `drf_spectacular`

Among the best practices and requirements when developing REST APIs for real software projects is providing in-depth documentation. drf_spectacular allows you to:

Extract as much schema information from DRF as possible.
Provide flexibility to make the schema usable in the real world (not only toy examples).
Generate a schema that works well with the most popular client generators.

To install:

Install using pip

   $ pip install drf-spectacular

Add drf-spectacular to INSTALLED_APPS in settings.py

   INSTALLED_APPS = [
       # ...
       'drf_spectacular',
   ]

   REST_FRAMEWORK = {
       # YOUR SETTINGS
       'DEFAULT_SCHEMA_CLASS': 'drf_spectacular.openapi.AutoSchema',
   }

Configure URL patterns inside project_name/urls.py

   from drf_spectacular.views import SpectacularAPIView, SpectacularRedocView, SpectacularSwaggerView

   urlpatterns = [
       # YOUR PATTERNS
       path('api/schema/', SpectacularAPIView.as_view(), name='schema'),
       # Optional UI:
       path('api/schema/swagger-ui/', SpectacularSwaggerView.as_view(url_name='schema'), name='swagger-ui'),
       path('api/schema/redoc/', SpectacularRedocView.as_view(url_name='schema'), name='redoc'),
   ]

Note that the URL paths are customisable.

Try to access the URLs upon running the Django server and you should see something similar as shown below:

References

Deploying Django Web App Using Heroku (Updated)

Muhammad Rahiman — Fri, 02 Jul 2021 03:47:46 +0000

Heroku is one of the most popular hosting platforms for web applications. In this article, I will provide a step-by-step guide on how to deploy a Django web application in Heroku.

Signup for Heroku if you don't have an existing account.
Install the Heroku CLI. For MacOS, use $ brew install heroku.
Log in to your Heroku account by entering your credentials using $ heroku login or $ heroku login -i if you faced IP address mismatch issue like the one shown below:
```
$ heroku login
Enter your Heroku credentials:
Email: your_email@emailprovider.com
Password: *********
Logged in as jameseze.ca@gmail.com
```
Create a new Heroku app either via Heroku CLI ($ heroku create APP_NAME) or directly in the Heroku dashboard:
Add the Heroku remote via $ heroku git:remote -a your-heroku-app
Configure the Heroku buildpacks. For Django, we just need 1 buildpack for Python: $ heroku buildpacks:set heroku/python. Run $ heroku buildpacks to see the configured buildpack. For your information, if you have multiple buildpacks, the last buildpack on the list determines the process type of the app.
Configure PostgreSQL Heroku addon

* During production, Heroku will **not be using SQLite database**. Instead, we need to use **PostgreSQL** by configuring the addon to our app using `$ heroku addons:create heroku-postgresql:hobby-dev`
* You can check whether this is successful by running `$ heroku config`:

```Shell
    $ === APP_NAME Config Vars
    DATABASE_URL: postgres://[DATABASE_INFO_HERE]
```

* The database info from the code snippet above refers to the URL containing your database’s location and access credentials all in one. Anyone with this URL can access your database, so be careful with it.
* You will notice that Heroku saves it as an **environment variable** called `DATABASE_URL`. This URL can and does change, so you should never hard code it. Instead, we’ll use the variable `DATABASE_URL` in  Django.

Please note that APP_NAME mentioned here onwards refers to the Django project name, NOT the individual apps within the Django project!

Configure Heroku config variables

* According to Heroku, **config variables** are environment variables that can change the way your app behaves. In addition to creating your own, some add-ons come with their own.
* There are several environment variables that need to be set:

```Shell
    $ heroku config:set ALLOWED_HOSTS=APP_NAME.herokuapp.com
    $ heroku config:set ALLOWED_HOSTS=APP_NAME.herokuapp.com
    $ heroku config:set SECRET_KEY=DJANGO_SECRET_KEY
    $ heroku config:set WEB_CONCURRENCY=1
```

Install the following essential Python libraries using pip install:

$ pip install django-heroku gunicorn python-dotenv dj-database-url whitenoise psycopg2

django-heroku is a Django library for Heroku applications that ensures a more seamless deployment and development experience.
- This library provides:
  - Settings configuration (Static files / WhiteNoise)
  - Logging configuration
  - Test runner (important for Heroku CI)
dj-database-url allows the app to automatically use the default database depending on the environment the app is in. For example, if the app is run locally, it will use SQLite3 whereas in Heroku, it will default to PostgreSQL.
python-dotenv is useful for setting up and load environment variables.
gunicorn is a Python web server that is commonly used for deployment.
whitenoise allows your web app to serve its own static files, making it a self-contained unit that can be deployed anywhere without relying on nginx, Amazon S3 or any other external service (especially useful on Heroku and other PaaS providers).
psycopg2 is the most popular PostgreSQL database adapter for the Python programming language. This is essential to allow the Django web app to use external PostgreSQL database.

Create a .env file containing DATABASE_URL=sqlite:///db.sqlite3. This is to tell Django to use SQLite when running locally. We don’t want .env to make it to Heroku, because .env is the part of our app that points to SQLite and not PostgreSQL. Hence, we need git to ignore .env when pushing to Heroku.

        $ echo 'DATABASE_URL=sqlite:///db.sqlite3' > .env
        $ echo '.env' >> .gitignore

Configure installed libraries in settings.py

First, import the following packages at the top of the file:

import django_heroku
import dj_database_url
import dotenv

Load the environment variables:

from dotenv import load_dotenv
load_dotenv()
# or
dotenv_file = os.path.join(BASE_DIR, ".env")
if os.path.isfile(dotenv_file):
    dotenv.load_dotenv(dotenv_file)

Next, we configure the DEBUG, SECRET_KEY and ALLOWED_HOSTS. DEBUG must be set to False during production. ALLOWED_HOSTS should also only point to the Heroku address once deployed (which is when DEBUG is set to False).

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '' # Change to empty string
DEBUG = False
# ALLOWED_HOSTS = ['*']
if DEBUG:
    ALLOWED_HOSTS = ["localhost", "127.0.0.1"]
else:
    ALLOWED_HOSTS = ['https://APP_NAME.herokuapp.com/']

Then, we configure whitenoise middleware, static and media files settings. Django security middleware should already be the first thing on the list. Never load any middleware before Django security.

 MIDDLEWARE = [
        'django.middleware.security.SecurityMiddleware',
        'whitenoise.middleware.WhiteNoiseMiddleware',
        # ...
 ]
# ...
STATIC_URL = '/static/'
STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
X_FRAME_OPTIONS = 'SAMEORIGIN'

Afterwards, update the DATABASES using dj-database-url. The idea here is to clear the DATABASES variable and then set the 'default' key using the dj_database_url module. This module uses Heroku’s DATABASE_URL variable that we set up previously if it’s on Heroku, or it uses the DATABASE_URL we set in the .env file if we’re working locally.

# Database
# https://docs.djangoproject.com/en/3.1/ref/settings/#databases
# DATABASES = {
#     'default': {
#         'ENGINE': 'django.db.backends.sqlite3',
#         'NAME': BASE_DIR / 'db.sqlite3',
#     }
# }
DATABASES = {}
DATABASES['default'] = dj_database_url.config(conn_max_age=600)

Lastly, configure django-heroku and SSL issue workaround at the very bottom of the file. If you ran the Django application, you might get an error when working locally because the dj_database_url module wants to log in with SSL. Heroku PostgreSQL requires SSL, but SQLite doesn’t need or expect it. So, we’ll use a hacky workaround to get dj_database_url to forget about SSL at the last second:

django_heroku.settings(locals())
options = DATABASES['default'].get('OPTIONS', {})
options.pop('sslmode', None)

Set up Heroku-specific files

A. runtime.txt

Heroku will install a default Python version if you don't specify one, but if you want to pick your Python version, you'll need a runtime.txt file.

Create one in the root directory, next to your requirements.txt, manage.py, .gitignore and the rest. Specify your Python version with the prefix python- that you want your application to run on. Heroku usually recommends running on the latest stable version of Python:
```
    python-3.9.5
```
B. requirements.txt

When deploying the web app, Heroku will need to install all the required dependencies for the web app to run by referring to the requirements.txt file.

To ensure that all dependencies are included, consider freezing your dependencies using the command $ pip freeze > requirements.txt. This will make your build a little bit more predictable by locking your exact dependency versions into your Git repository. If your dependencies aren't locked, you might find yourself deploying one version of Django one day and a new one the next.

C. Procfile

Heroku apps include a Heroku-specific Procfile that specifies the processes our application should run. The processes specified in this file will automatically boot on deploy to Heroku.

Create a file named Procfile in the root level directory using $ touch Procfile command, right next to your requirements.txt and runtime.txt files. (Make sure to capitalize the P of Procfile otherwise Heroku might not recognise it!):

Then, fill in the codes below:
```
    release: python manage.py migrate
    web: gunicorn DJANGO_PROJECT_NAME.wsgi --log-file -
```
Deploy the Web App

Once all the previous steps are completed, we need to collect all the static files using python manage.py collectstatic command. A staticfiles folder should be created.

After that, we are ready to finally commit and push all changes:
```
    $ git add .
    $ git commit -m "blah blah blah"
    $ git push heroku master
```
After the build is done and your app has been released, visit YOUR-APP-NAME.herokuapp.com

IMPORTANT NOTE:

When deploying to Heroku, make sure that your migrations folder are not included inside .gitignore! Heroku will need the migration files to update the PostgreSQL database.
If you encounter 500 Server Error issues in only the following cases:

* Debug=True && DB=local => Runs fine
* Debug=True && DB=production => Runs fine
* Debug=False && DB=local => Runs fine
* **Debug=False && DB=Production => 500 Server Error**

First, try including the following code in settings.py to display logging messages:

import logging
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console'],
            'level': os.getenv('DJANGO_LOG_LEVEL', 'DEBUG'),
        },
    },
}

Then, try refreshing the site. Once the same error occurs, run $ heroku logs --tail inside terminal, the log will tell you the error message caused. This is because this problem is commonly caused by missing static files. By running said command, we can see what static files cannot be found.

Web Scraping using Django and Selenium

Muhammad Rahiman — Thu, 25 Feb 2021 11:55:22 +0000

This is a mini side project to tinker around with Django and Selenium by web scraping FSKTM course timetable from MAYA UM as part of my self-learning prior to FYP.

Please note that this is my first ever attempt at Selenium done in < 1 day and my solutions may or may not be the best approach. Feel free to let me know for any improvements!

GitHub repository link: https://github.com/mdrhmn/dj-selenium

Getting Started with Django

1. Set up your Django project

If you are cloning this repo, run the following command preferably inside your virtual environment to install all dependencies:

Using venv:

$ pip install -r requirements.txt # (Python 2)
$ pip3 install -r requirements.txt # (Python 3)

Else, to create your Django project from scratch (make sure to have Django installed):

$ django-admin startproject project_name

And then create a virtual environment (highly recommended):

To create virtual environment:

$ python3 -m venv env_name # using Python's venv
# or
$ virtualenv env_name # using virtualenv

To activate virtual environment (Linux/Mac OS):
```
$ source env_name/bin/activate
```

Install all dependencies:

$ pip install -r requirements.txt # (Python 2)
$ pip3 install -r requirements.txt # (Python 3)

Next, navigate into the newly created project folder. Then, start a new Django app. We will also run migrations and start up the server:

$ cd project_name
$ python manage.py startapp app_name
$ python manage.py migrate
$ python manage.py runserver

If everything works well, we should see an instance of a Django application running on this address — http://localhost:8000

2. Configure project settings

Add app inside INSTALLED_APPS (settings.py)

Once you’ve created the app, you need to install it in your project. In project_name/settings.py, add the following line of code under INSTALLED_APPS:
```
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'app_name',
]
```
That line of code means that your project now knows that the app you just created exists.

Add templates folder directory in TEMPLATES (project_name/settings.py)

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': ['templates/'], # HERE
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

Add static and media folder directory in STATIC_ROOT (project_name/settings.py)

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

Add desired URL for the app (project_name/urls.py)

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('(INSERT_URL)', include('APP_NAME.urls')),
]

Create new urls.py for the app (app_name/urls.py)

3. Configure app settings

Create new template (app_name/templates/)
- You need to create the HTML template to display to the user after creating a view function.
- Create a directory named templates and subsequently a file named app_name.html inside it:
```
# Create directory (if haven't)
mkdir app_name/templates/

# Create HTML template
touch app_name/templates/app_name.html
```
Create view function (FBV/CBV) in app's views.py
- Views in Django are a collection of functions or classes inside the views.py file in your app directory.
- Each function or class handles the logic that gets processed each time a different URL is visited.
```
from django.shortcuts import render

def view_name(request):
    return render(request, 'template_name.html', {})
```

- The function defined is called a **view function**. When this function is called, it will render an HTML file called `app_name.html`.

Add URL to app's urls.py
- The final step is to hook up your URLs so that you can visit the page you’ve just created.
- Your project has a module called urls.py in which you need to include a URL configuration for the app. Inside app_name/urls.py, add the following:
```
from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('', views.view_name, name="view_name"),
]
```
  - This looks for a module called urls.py inside the application and registers any URLs defined there.
  - Whenever you visit the root path of your URL (localhost:8000), the application’s URLs will be registered.

Getting Started with Selenium

What is Web Scraping?

Web scraping is a technique for extracting information from the internet automatically using a software that simulates human web surfing.

What is Selenium?

Selenium is a free (open-source) automated testing framework used to validate web applications across different browsers and platforms. It can be used for automating web browsers to do a number of tasks such as web-scraping.

Installing Selenium

To install Selenium:

    $ pip install selenium # (Python 2)
    $ pip3 install selenium # (Python 3)

Installing Webdrivers

Selenium requires a driver to interface with the chosen browser. Firefox, for example, requires geckodriver, which needs to be installed before the below examples can be run. Note that the webdriver must be located in your PATH, e. g., place it in /usr/bin or /usr/local/bin.

Other supported browsers will have their own drivers available. Links to some of the more popular browser drivers are as follows:

Chrome: https://sites.google.com/a/chromium.org/chromedriver/downloads
Edge: https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/
Firefox: https://github.com/mozilla/geckodriver/releases
Safari: https://webkit.org/blog/6900/webdriver-support-in-safari-10/

For this project, I am using Chrome's webdriver called Chromedriver. There are multiple ways to install Chromedriver:

Using webdriver-manager (recommended)

- Install package:

    ```Shell
    $ pip install webdriver-manager # (Python 2)
    $ pip3 install webdriver-manager # (Python 3)
    ```

- Load package:

    ```python
    from selenium import webdriver
    from webdriver_manager.chrome import ChromeDriverManager

    driver = webdriver.Chrome(ChromeDriverManager().install())
    ```

Manual download from Chrome's website

- Load package:

    ```python
    from selenium import webdriver

    driver = webdriver.Chrome(executable_path='/path/to/chromedriver)
    ```

Set Up Selenium in Django

Depending on the use case, you can set up Selenium codes inside views.py for direct use or pair with Celery/Django-Crontab (a discussion for another time).

The following code snippet is set up inside views.py:

# Django
from django.http import HttpResponseRedirect
from django.shortcuts import render, redirect
import os

# Selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def scrap(request):
    options = Options()
    options.headless = True
    options.add_argument("--window-size=1920,1080")

    driver = webdriver.Chrome(options=options)

I defined a function called scrap() to contain the Selenium initialisation codes. selenium.webdriver.chrome.options allows us to specify the Selenium webdriver settings such as the following:

# Add single argument (method 1)
options.add_argument("--window-size=1920,1080")

# Add single argument (method 2)
options.headless = True

# Add many arguments
options.AddArguments("--headless", "--window-size=1920,1080", "--disable-gpu", "--disable-extensions", "--no-sandbox", "--incognito")

The important option setting to highlight here is headless, which allows you to launch the browser without creating a visual browser window. This way, you can run tests faster and with fewer resources, and most importantly, it will allow you to run tests on systems without a graphical component.

Web-Scrapping MAYA UM using Selenium

1. Going through Authentication

The first hurdle that I encountered when scraping MAYA is going through the authentication. I did some research and luckily I found a working solution from StackOverflow that allows for auto-login:

def autologin(driver, url, username, password):

    driver.get(url)
    password_input = driver.find_element_by_xpath("//input[@type='password']")
    password_input.send_keys(password)
    username_input = password_input.find_element_by_xpath(
        ".//preceding::input[not(@type='hidden')]")
    username_input.send_keys(username)
    form_element = password_input.find_element_by_xpath(".//ancestor::form")
    submit_button = form_element.find_element_by_xpath(
        ".//*[@type='submit']").click()
    return driver


def scrap(request):

    options = Options()
    options.headless = True
    options.add_argument("--window-size=1920,1080")

    driver = webdriver.Chrome(options=options)

    USERNAME = os.getenv("USERNAME")
    PASSWORD = os.getenv("PASSWORD")

    autologin(driver, 'https://maya.um.edu.my/sitsvision/wrd/siw_lgn',
              USERNAME, PASSWORD)

First, I declared constants USERNAME and PASSWORD to store the SiswaMail and password environment variables set within the .env file. If you fork/clone this repository, remember to rename .settings.env as .env and fill in environment variables in the file.

USERNAME = os.getenv("USERNAME")
PASSWORD = os.getenv("PASSWORD")

Setting Up Environment Variables

For environment variables storage, I use python-dotenv package. There are many other Python alternatives for adding .env support to your Django/Flask apps in development and deployments.

To install python-dotenv:

pip install -U python-dotenv

Then, add the following code to settings.py:

# settings.py
from dotenv import load_dotenv
load_dotenv()

# OR, the same with increased verbosity
load_dotenv(verbose=True)

# OR, explicitly providing path to '.env'
from pathlib import Path  # Python 3.6+ only
env_path = Path('.') / '.env'
load_dotenv(dotenv_path=env_path)

At this point, parsed key/value from the .env file is now present as system environment variable and they can be conveniently accessed via os.getenv():

# settings.py
import os

SECRET_KEY = os.getenv("EMAIL")
DATABASE_PASSWORD = os.getenv("DATABASE_PASSWORD")

autologin(driver, 'https://maya.um.edu.my/sitsvision/wrd/siw_lgn',
            USERNAME, PASSWORD)

Next, I defined a function called autologin() that accepts the webdriver, site URL, username and password for authentication.

def autologin(driver, url, username, password):

    driver.get(url)
    password_input = driver.find_element_by_xpath("//input[@type='password']")
    password_input.send_keys(password)
    username_input = password_input.find_element_by_xpath(
        ".//preceding::input[not(@type='hidden')]")
    username_input.send_keys(username)
    form_element = password_input.find_element_by_xpath(".//ancestor::form")
    submit_button = form_element.find_element_by_xpath(
        ".//*[@type='submit']").click()
    return driver

Extracting XPath

In order to extract the information that you’re looking to scrape, you need to locate the element’s XPath. An XPath is a syntax used for finding any element on a webpage.

To locate the element’s XPath, right click and select Inspect. This opens up the developer tools. Highlight the portion of the site that you want to scrape and right click on the code. Select Copy -> Copy XPath.

Finding Elements

find_element_by_xpath() function is used to find an element that matches the XPath given. There are many selectors that you can use to find the right element(s) which you can refer in the official documentation.

send_keys() types a key sequence in DOM element which in this case, is the Username and Password input fields.

2. Redirecting to Search Timetable page

One of the important tips of webscraping is to understand the structure of the website. This can be done by using the browser's developer tools as previously mentioned (for Chrome, it's Inspect/Inspect Element).

My next goal is to redirect to MAYA's Search Timetable page, which allows me to filter and access the course timetable. This can be done by first clicking the 'Timetable' button which pops up a modal, followed by clicking the 'Search Timetable' button.

Normally, most websites would have a simple HTML structure where most elements will contain a unique but straightforward attributes such as id and class which you can manipulate for finding elements. However, this is unfortunately not the case for MAYA.

If you can see clearly, much of the elements' attributes such as href for anchor tag are somewhat encrypted/hashed and linked to the user's session. Obviously this is done for security purposes.

<a role="button" href="../wrd/SIW_POD.start_url?CA0D2CBA7E7A40B4Prj-9fG3lZZiN6_h3HOcPlM90_2TQ3CTYLeGYXH4ryqqi-Hfrfz936h2qegofnf16s4AW2Mb7gwV5EcqOYjofk8-FCvH6NU2XVmZYA7qt4UAGxjNyG_jk4swwW5d2cLnDCtCLr4Ubr2uZXF9wjeiVK0oYm8HiigicuumeEvKanxW0hltEIwXfdjFZiYCIIwi4uTZavngKUUk38jmd2tS6b6npTNQELrCKzSkGHu4Ea8" class="sv-btn sv-btn-block sv-btn-danger">
    <h1><span class="glyphicon glyphicon-search"></span></h1> Search Timetable
</a>

However, this makes it much, much harder for me to mimic the interaction properly.

Ideally, you should use XPath that utilises the element's id and class. Instead, due to the aforementioned case, I had to resort by directly extracting the XPath from the browser itself (based on element hierarchy):

timetable_popup = driver.find_element_by_xpath(
   "//a[@href='javascript:timetable_popup();']").click()

search_timetable = driver.find_element_by_xpath(
   "//*[@id='sits_dialog']/center/div/div/div[3]/a").click()

While this works fine, it is not the best approach as it is fragile solution and will not work if the developer changes the structure in the future. In other words, it's not futureproof.

Here, I utilised the click() method to mimic cursor-clicking inside the browser.

3. Filling Up Search Timetable Form

This next stage is hands down the hardest part of the whole project and it clearly demonstrates the aforementioned case of obscuring ID's and classes. It involves filling up the 'Search Timetable' form shown below:

<label for="POP_UDEF.F1C9C66D61D74A2087AA32FF9EE17931.POP.MENSYS.1-1" class="sv-col-sm-3 sv-control-label">Academic Year*</label>
<span id="POP_UDEF_F1C9C66D61D74A2087AA32FF9EE17931_POP_MENSYS_1_1_chosenspan">2009/2010</span>

Upon close inspection, I realised that the HTML structure for this page involves even more encryption or obscuring of IDs and classes. They change every time you refresh the page, so it confirms my suspicion that it is based on the user's session.

I spent hours being stuck at this phase and was on the point of giving up. However, that's when I had an eureka moment.

When I reinspected and reanalysed the HTML structure once again, I realised that there exists a pattern with the way the IDs and classes are configured. Although slightly differing in format (e.g. some '.'s are replaced with '_'), they all share the same 'encryption' code, so to speak.

After a bit of brainstorming, I drafted the following algorithm to test out:

Get encrypted ID from one of the elements (preferably standalone and unnested) using hierarchy-based XPath
Modify the encrypted ID based on the identified pattern/format
Fill in the form using Selenium DOM (select the dropdown, and then select the desired option)

Get encrypted ID

For this part, I utilised one of the elements that I find easiest to extract. I copied its XPath and used the get_attribute() function to retrieve the encrypted ID.

<label for="POP_UDEF.F1C9C66D61D74A2087AA32FF9EE17931.POP.MENSYS.1-1" class="sv-col-sm-3 sv-control-label">Academic Year*</label>

login_id = driver.find_element_by_xpath('//*[@id="poddatasection"]/div[2]/div[2]/div/div/fieldset/div[2]/label').get_attribute("for")
login_id_modified = login_id.removesuffix('.1-1').replace(".", "_")
login_id_truncated = login_id.removesuffix('.1-1')

Then, I declared two variables to store the encrypted ID that was modified in different ways, based on my key observation of the format.

My observations:

The select dropdown input fields use encrypted IDs that are slightly modified (replacing '.' with '_') and appended with a suffix ('_#_1', where # = index of select dropdown input fields)

- Example:

```html
<div class="chosen-container chosen-container-single" style="width: 100%;" title="" id="POP_UDEF_F1C9C66D61D74A2087AA32FF9EE17931_POP_MENSYS_1_1_chosen" tabindex="-1">
...
</div>
```

The options for each select dropdown input field use encrypted IDs that are appended with a different suffix ('_#_%', where # = index of parent select dropdown input field and % = index of option to be selected)

- Example:

```html
<li class="active-result result-selected" data-option-array-index="11" style="" id="POP_UDEF.F1C9C66D61D74A2087AA32FF9EE17931.POP.MENSYS.1-111" role="option" tabindex="-1">2020/2021</li>
```

For some unknown reason, the first three select dropdown input fields do not function like a typical select dropdown. Instead of setting the selected option as selected, the selection process is done elsewhere in another section of the code. This is most probably due to the system using some sort of CSS/JS package.

For the 'Campus Location' select dropdown input field, since it functions normally unlike the others, I utilised from selenium.webdriver.support.ui import Select module to help click the select dropdown input field and select the desired option using select_by_visible_text() function.

With all these things in mind, I successfully managed to fill in and submit the form using the following code:

login_id = driver.find_element_by_xpath('//*[@id="poddatasection"]/div[2]/div[2]/div/div/fieldset/div[2]/label').get_attribute("for")
login_id_modified = login_id.removesuffix('.1-1').replace(".", "_")
login_id_truncated = login_id.removesuffix('.1-1')

select_year_dropdown = driver.find_element_by_xpath("//*[@id='" + login_id_modified + "_1_1_chosen']/a").click()
select_year = driver.find_element_by_xpath("//*[@id='" + login_id_truncated + ".1-111']").click()

select_semester_dropdown = driver.find_element_by_xpath("//*[@id='" + login_id_modified + "_2_1_chosen']/a").click()
select_semester = driver.find_element_by_xpath("//*[@id='" + login_id_truncated + ".2-120']").click()

select_faculty_dropdown = driver.find_element_by_xpath("//*[@id='" + login_id_modified + "_3_1_chosen']/a").click()
select_faculty = driver.find_element_by_xpath("//*[@id='" + login_id_truncated + ".3-121']").click()

select_campus = Select(driver.find_element_by_id(login_id.removesuffix('.1-1') + '.5-1'))
select_campus.select_by_visible_text('UNIVERSITI MALAYA KUALA LUMPUR')

submit_timetable = driver.find_element_by_xpath("//*[@id='poddatasection']/div[2]/div[3]/div/input[3]").click()

4. Extracting Course Timetables from DataTables

Once we have submitted the 'Search Timetable' form, we finally arrived at the desired page which is the 'Teaching Timetable' page, where all the course schedules (called 'module occurrences') are displayed.

This final task involves extracting all the data displayed in the table, which I identified as in DataTables format. For those who don't know, DataTables is a plug-in for the jQuery Javascript library. It is a highly flexible tool, built upon the foundations of progressive enhancement, that adds all of these advanced features to any HTML table.

This poses a few challenges:

Extracting the data stored in each row (including the header)
Iterating step 1 for each page in the DataTable (how to know what is the last page?)

I managed to solve these two challenges as follows:

Get last page number

last_page = driver.find_element_by_xpath("//*[@id='DataTables_Table_0_last']/a")
driver.execute_script("arguments[0].click();", last_page)
last_page_num = driver.find_element_by_xpath("//*[@id='DataTables_Table_0_paginate']/ul/li[7]/a").get_attribute('text')
first_page = driver.find_element_by_xpath("//*[@id='DataTables_Table_0_first']/a")
driver.execute_script("arguments[0].click();", first_page)

- Firstly, I need to find a way to **get the last page number** to be used as terminating condition for the loop to be used for iterating through the pages.
- Because the DataTable pagination also includes a handy **'First' and 'Last' button** to indicate skipping to the first and last page of the table, I utilised them by first clicking the 'Last' button and **get its page number** using the `get_attribute('text')` method. Then, I **reverted back to the 'First' page** by clicking the 'First' button.

Extract whole table

table_element = driver.find_element_by_xpath("//*[@id='DataTables_Table_0_wrapper']")

- Next, I need to **extract the entire DataTable**. This can easily be done by using the **XPath** of the **table's wrapper**.

Iterate through table pages and rows and extract data

if not os.path.exists("maya.txt"):
    f = open("maya.txt", "a")
else:
    f = open("maya.txt", "a")

- The last step is the extraction process. I need to s**tore all the scrapped data** inside a **text file** by using a **simple file I/O**. I also captured the case where the text file does not exist.
- Note that the file has to be set in 'append' ('a') setting since configuring it to 'write' ('w') will overrite the content each time the loop repeats.
<br>

```python
for i in range (int(last_page_num)):
    for tr in table_element.find_elements_by_tag_name('tr'):

        if tr.get_attribute('class') == '' or tr.get_attribute('class') == None:
            f.write("PAGE " + str(i + 1) + "\n\n")
            f.write(tr.text)
        else:
            f.write(tr.text)

        f.write("\n\n")

    next_page = driver.find_element_by_xpath("//*[@id='DataTables_Table_0_next']/a")
    driver.execute_script("arguments[0].click();", next_page)

f.close()
```

- For the extraction, I used a **nested for-loop**. The **first for-loop iterates through the pages** with the last page number as the terminating condition. The **second for-loop iterates through each row** in the table page.
- I also did some styling such as writing the page number as well as writing newlines to separate between pages.
- Once all table rows in a particular page has been extracted, Selenium will click the 'Next' button to proceed to the next page. The cycle repeats.
- When everything is done, we close the I/O with `f.close()`.

And voila! All the course schedules will be extracted and written inside the maya.txt text file!

References

Django + Firebase Cloud Storage

Muhammad Rahiman — Mon, 15 Feb 2021 19:52:10 +0000

This is a sample project to test Firebase storage integration with Django using Pyrebase package.

Repository link: https://github.com/mdrhmn/dj-firebase

Getting Started

1. Set up your Django project

If you are cloning this repo, run the following command preferably inside your virtual environment and skip to step 5:

Using pipenv:

$ pipenv install -r requirements.txt # (Python 2)
$ pipenv3 install -r requirements.txt # (Python 3)

Using venv:

$ pip install -r requirements.txt # (Python 2)
$ pip3 install -r requirements.txt # (Python 3)

Else, to create your Django project from scratch (make sure to have Django installed):

$ django-admin startproject project_name

Next, navigate into the newly created project folder. Then, start a new Django app. We will also run migrations and start up the server:

$ cd project_name
$ python manage.py startapp app_name
$ python manage.py migrate
$ python manage.py runserver

If everything works well, we should see an instance of a Django application running on this address — http://localhost:8000

2. Configure project settings

Add app inside INSTALLED_APPS (settings.py)

Once you’ve created the app, you need to install it in your project. In project_name/settings.py, add the following line of code under INSTALLED_APPS:
```
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'app_name',
]
```
That line of code means that your project now knows that the app you just created exists.

Add templates folder directory in TEMPLATES (project_name/settings.py)

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': ['templates/'], # HERE
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

Add static and media folder directory in STATIC_ROOT (project_name/settings.py)

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

Add desired URL for the app (project_name/urls.py)

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('(INSERT_URL)', include('APP_NAME.urls')),
]

Create new urls.py for the app (app_name/urls.py)

3. Configure app settings

Create new template (app_name/templates/)
- You need to create the HTML template to display to the user after creating a view function.
- Create a directory named templates and subsequently a file named app_name.html inside it:
```
# Create directory (if haven't)
mkdir app_name/templates/

# Create HTML template
touch app_name/templates/app_name.html
```
Create view function (FBV/CBV) in app's views.py
- Views in Django are a collection of functions or classes inside the views.py file in your app directory.
- Each function or class handles the logic that gets processed each time a different URL is visited.
```
from django.shortcuts import render

def view_name(request):
    return render(request, 'template_name.html', {})
```

- The function defined is called a **view function**. When this function is called, it will render an HTML file called `app_name.html`.

Add URL to app's urls.py
- The final step is to hook up your URLs so that you can visit the page you’ve just created.
- Your project has a module called urls.py in which you need to include a URL configuration for the app. Inside app_name/urls.py, add the following:
```
from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('', views.view_name, name="view_name"),
]
```
  - This looks for a module called urls.py inside the hello_world application and registers any URLs defined there.
  - Whenever you visit the root path of your URL (localhost:8000), the application’s URLs will be registered.

4. Install Pyrebase package

Pyrebase is a simple python wrapper for the Firebase API.

To install Pyrebase:

$ pip install pyrebase

5. Create a new Firebase project

In order to use Firebase, you need to be logged into your Google account. Go to Firebase and click Get started. You will be redirected to Firebase Console.

Next, click Add project and fill up the necessary fields.

6. Create Firebase app

Once the project has been successfully created, click the '</>' button to create a new Firebase web app.

Go through the 'Add Firebase to your web app' creation process as shown below.

When you reach the 'Add Firebase SDK' part, copy the web app's Firebase configuration as highlighted in the image below. You will need this for your Pyrebase config inside Django later.

7. Set up Django app views

The next step is to initialise our Firebase connection in Python (Django) by first importing the pyrebase package and creating a dictionary called config based on the Firebase configuration that you copied previously. You will need to amend the codes a bit by encapsulating the keys as strings.

You also need to add another key-value pair inside the dictionary called databaseURL. This is because in order to connect with Firebase, you need to declare the databaseURL environment variable (which in this guide is not configured since we will be focusing more on Firebase Cloud Storage).

# app_name/views.py
import pyrebase
import os

config = {
    "apiKey": "KEY",
    "authDomain": "project_name-id.firebaseapp.com",
    "projectId": "project_name-id",
    "storageBucket": "project_name-id.appspot.com",
    "messagingSenderId": "msg_sender_id",
    "appId": "1:app_id",
    "measurementId": "G-measurement_id",
    "databaseURL": ""
}

Then, to initialise the Firebase connection and storage, write the following codes:

# app_name/views.py

firebase = pyrebase.initialize_app(config)
storage = firebase.storage()

8. Create new Firebase Cloud Storage

To create a Cloud Storage, click 'Storage' on the left sidebar and then click 'Get Started' button.

Follow the two steps prompted. Keep in mind that the Cloud Storage location should be in your region (in our case, southeastasia2).

Once the Cloud Storage has been successfully created, go to the Rules tab and edit the rules to the following (IMPORTANT: only for development purposes!)

rules_version = '2';
service firebase.storage {
  match /b/{bucket}/o {
    match /{allPaths=**} {
      allow read, write
    }
  }
}

9. Configure file upload to Firebase Cloud Storage

Going back to app_name/views.py, to upload files inside the Firebase Cloud Storage, we need to use the storage.child() and storage.put() functions as follows:

# app_name/views.py

storage = firebase.storage()
storage.child(PATH/DIRECTORY_ON_CLOUD).put(PATH_TO_LOCAL_IMAGE  )

# Example (same directory, same file name)
storage.child("images/example.jpg").put("example.jpg")

# Example (different directory, different file name)
storage.child("images/renamed_img.jpg").put("media/original_img.jpg")

The child() method is used to build paths to your data with the Storage service. You can specify the file directory and even the uploaded file name (renaming). The put() method on the other hand takes the path to the local file that you want to upload on Cloud Storage.

10. Implement file upload logic (optional)

This is an additional and optional step on my part, where I created a simple file upload logic from the template form:

# app_name/views.py

from django.shortcuts import render, redirect
from django.shortcuts import render
from django.conf import settings
from django.core.files.storage import default_storage
from django.contrib import messages
import pyrebase
import os

config = {
    "apiKey": "AIzaSyA-99K3f5TK26Z7n5szV4wLq_fl6Gf9kcs",
    "authDomain": "dj-firebase-6f10d.firebaseapp.com",
    "projectId": "dj-firebase-6f10d",
    "storageBucket": "dj-firebase-6f10d.appspot.com",
    "messagingSenderId": "719069501836",
    "appId": "1:719069501836:web:822c7843a3d428d32f9ea9",
    "measurementId": "G-VLZXVGDCZF",
    "databaseURL": ""
}

firebase = pyrebase.initialize_app(config)
storage = firebase.storage()

def main(request):
    if request.method == 'POST':
        file = request.FILES['file']
        file_save = default_storage.save(file.name, file)
        storage.child("files/" + file.name).put("media/" + file.name)
        delete = default_storage.delete(file.name)
        messages.success(request, "File upload in Firebase Storage successful")
        return redirect('main')
    else:
        return render(request, 'main.html')

What this essentially does is that the user is able to upload any file via the form created using TailwindCSS. The function-based view will then retrieve the uploaded file, save it locally (MEDIA_ROOT directory as specified in settings.py and using Django's default_storage library), upload the file in Firebase Cloud Storage and then delete the local file (since I only cared about uploading in cloud instead of local).

References

Hackathon: A Newbie's Experience

Muhammad Rahiman — Sun, 14 Feb 2021 20:08:18 +0000

Hey, everyone! Ray here back with another update at my DEV.to!

It's been a while since I last posted anything here as I was mostly preoccupied with internship matters. Now that I'm on my semester break, I have a few writings planned out to make full use of my free time.

For this post, I would like to share with you guys a detailed look into arguably one of the wildest roller-coaster rides of my entire student life: my first-time experience in joining hackathon!

What is Hackathon?

A hackathon is an event, usually hosted by a tech company or organisation, where programmers get together for a short period of time to collaborate on a project and solve problems. The participants work rapidly and often work without sleep to achieve their task, as the events generally only last 24 hours or take place over a weekend.

I personally first came to know about the existence of hackathons back when I was a first year in first semester. There was this huge fuss over this hackathon competition called A.I.ckathon **which was organised by Universiti Malaya. Our faculty students and lecturers became part of the organising committee. It was a huge competition at nationwide scale, and it was held at Johor. I remember my peers actually managed to win first runner-up and got RM5,000 prize money as a result. Keep in mind, these people are my peers, first years!**

I personally always felt somewhat intimidated at the prospect of hackathons in general. The competition seems exclusive and is out of my league. Given my skill sets then, I deemed myself to be unworthy and unqualified to even participate in it, let alone to win. It takes a lot of guts, expertise and perseverance to be able to code for 24-48 hours straight. Juggling between sleep deprivation and stress. It doesn't even help the fact that the imposter-syndrome-ishpart of me is making me feel less confident in my own ability.

Why Now?

So, why did I decide to join hackathon now?

A few reasons:

1. Training for Final Year Project (FYP)

In case you don't know, I'll be taking my final year project (FYP) starting next semester alongside my dear friend Faidz Hazirah a.k.a. my partner in the Dell Virtual Hack2Hire hackathon. We had history of working together in group assignments since we were in first year. But because we went into separate internship programmes for the past 6 months, I figured the two of us need to rendezvous on the new skills we learnt and have a little warm-up session before our FYP commences. We thought that hackathon would be a good exposure for us to test our teamwork and problem-solving skills. After all, team dynamics is extremely important for FYP and you will see how it plays a crucial role in our hackathon journey.

2. Test New Skills

During the course of my 6-month internship at Infront, I was given the golden opportunity to learn and apply Django framework extensively by directly working with actual projects (one internal, the other external). This skill in particular prove extremely beneficial and useful for not only FYP, but also my journey in becoming a full-stack developer. Nevertheless, it occurred to me that rather than directly putting this newly acquired skill to the test during my FYP, why not do so earlier by joining a hackathon?

3. Final Chance

Since I'm doing my FYP soon, that also means that I'm left with only 2 semesters for my undergraduate study. Writing that honestly makes me feel so old. But what that means is that I won't have much chance to involve myself in a hackathon once I start my FYP. I desperately want to tick that box off the bucket list before I graduate. It's really now or never.

How It Began

The story began sometime at the end of last year, when a friend of mine from UNIMY tweeted that he got a job offer from Dell Technologies Malaysia after joining their Hack2Hire programme.

We came a long way since our foundation days in PASUM as both of us were programming tutors then. So of course, I was so proud of his achievement. Seeing a friend of mine being able to secure a job at one of the very few well-known Silicon Valley companies having branches here in Malaysia motivates me to join one in the future, in hopes of securing one for myself.

At the time, I didn't know when's the next Hack2Hire programme will be organised. I thought it will be months to a year before they hold another one of this hackathon. Little did I know that it was way sooner than expected.

Earlier this year, I was browsing through my SiswaMail student mail as part of my daily routine. That was went my eyes stumbled upon one email in particular sent by my faculty's internship coordinator:

Words cannot express how excited I was when I saw the email! I immediately approached my friend Faidz and asked her if she's busy during the competition dates. After all, we're nearing the end of our internship which is packed with final report and logbook submissions as well as project handover and knowledge transfer.

After deliberations, both of us confirmed that we can make it for the competition dates and will be registering as a pair under the team name Raidz, a compilation of both of our nicknames (Ray + Idz). There was an option in the registration form for us to allow other team members to join our team in order to have a full 4-member team. However, as explained previously, we decided to challenge ourselves by registering as a 2-member team instead. Incredibly bold move considering that it's our first hackathon ever!

The Prep

A few weeks after we submitted our registration and a week before D-day, we attended the briefing, Q&A and problem statement release session via Zoom by the organiser. I remember asking for clarification from Dell on whether the two of us qualifies for the grand prize which is the permanent job offer should we win (bold of me then to assume that I have a chance of winning but hey, no harm in asking am I right?). Thankfully, because our last semester ends around Jan/Feb 2022 which is within the 1 year time period that they are willing to consider, we qualified for the permanent job offer.

The organiser also stressed that we must not prepare any coded prototypes beforehand and that it's imperative for us to turn on our webcams and share our screen throughout the duration of the competition to ensure that we are the ones coding the prototypes on the spot and not copy it elsewhere.

Unfortunately, I don't think I'm allowed to share the problem statement for the hackathon here. But basically, we need to develop an event management system for Dell's induction program for newly-hired fresh graduates known as IT Development Programs (ITDP).

Right after the briefing, I set up a Discord server as well as Trello for our prep discussions and task management. The latter in particular proved most useful and actually was one of the key highlights and advantage that our team has over others (will explain further during the presentation part). Both of us had past experiences in using Trello for our group assignments, so we were no stranger in how it works.

We had a total of 7 near-consecutive meetings after the briefing and before competition day, mainly to discuss regarding the modules/components that we want to implement, the database structure or ERD, the front-end design ideas as well as teaching Faidz some Django basics in order for her to integrate her front-end design to the project. We also took the liberty of drafting all the possible questions to direct to our mentors (each team will be assigned at least 1 mentor/coach to help you out). Because there's only two of us in the team, we split our roles through the ends of the stack. Faidz will be in charge of front-end design and development while I'll be in charge of back-end i.e. database design and server-side functions configuration.

I cannot emphasise how important it is to have preliminary meetings with your teammate(s) before the actual competition. Even though you cannot code anything in advance, that doesn't mean that you cannot test out your ideas. We only have < 24 hours to code a working prototype, so any form of preparation will make a night and day difference in terms of your performance during D-day.

D-Day 1

Morning

Fast-forward to competition day, and to say that I was nervous is a huge understatement. I can barely sleep the night before out of sheer anxiety. I remember during our final prep meeting the night before, Faidz expressed her concerns that we might screw things up. I told her upfront that we will screw things up regardless, it's just a matter of how fast we solve them. Especially when considering the fact that we are newbies.

The day started with a roll-call at 8:30AM, before leading to a briefing on the flow and tentative for both days of the competition (Day 1 is coding, Day 2 is presentation).

Around a quarter to 10:00AM, everyone splits into their designated break rooms. I had a surge of confidence and ready to start everything as swiftly as possible. After all, the initial setup of the project has to be done fast so that we can start progressing on the modules.

However, that's where we began to face our first set of hurdles.

When our mentors start coming in, we introduced ourselves and outlined our plans for the project by showing him our Trello. They were actually shocked that there's only two of us in the team, even more when they knew we were first-timers. Upon hearing our plans, one of our mentors called Mufhim immediately expressed his concerns that we need to strategise properly in order to make sure that we can complete the working prototype in time since we are severely lacking in manpower.

Soon after, more and more mentors begin to join the break room. Some of these people represent the personas or stakeholders involved in the problem statement. We are supposed to elicit requirements and confirm the functionalities that we planned with them. The list of questions that we prepared beforehand helps a lot in ensuring we have a checklist of things to validate.

It was during this phase where things began heading south for our team. One of the unexpected difficulties that I personally faced was trying to juggle between communicating our thoughts and ideas with the mentors and stakeholders and at the same time focusing on your codes. It takes a lot of skill to multitask. I remember screwing up my GitLab setup due to the state of confusion I was in.

For your information, you can't simply ignore your mentors or stakeholders as they also serve as judges for the competition, especially for the grand prize which is the permanent job offer. They will observe and monitor everything, from the way you communicate your ideas and respond to questions to teamwork. The grand prize is awarded individually. In other words, you could be in the top 3 team but still not qualify for the job or internship offer if you fail to exhibit a good impression to the judges throughout the competition, vice versa.

It doesn't help the fact that both Faidz and I began having internet connection issues around this time. I sort off expected this earlier on since I'm fully relying on my mobile data connection as internet source (I don't have WiFi at my place). I even warned Faidz to standby in case my internet goes haywire. But, what I didn't expect was that Faidz would face the same issue as well. This hindered the requirements gathering process as we occasionally got disconnected and thrown out of the Zoom call. It was honestly embarrassing as the mentors and stakeholders had to wait for us to come back and there was a constant back-and-forth between us saying "Hello, can you hear me?" and repeating the questions again and again. I had to apologise for disabling my webcam and screen-sharing due to poor internet connections, which was a huge no-no. Nevertheless, I was grateful that the stakeholders were willing to emphatise in our situation albeit it giving a bad impression on the team.

Then, sh*t hits the fan. While working on the front-end, Faidz's laptop suddenly crashed. This was certainly unpredictable and I was honestly at loss of words. Faidz went MIA for quite some time before coming back using her back-up laptop (her sister's). I asked her if she's all right and whether she can recover the work she has done. There was a bit of work that's lost but fortunately for us, much of the progress can be recovered from our GitLab repo and Google Drive (all hail version control systems!). However, the series of unfortunate events that led to this moment caused us to be severely unmotivated. We were behind schedule and I remember feeling like giving up halfway. The mood of the team became somber (and mind you, **our mentors are still in the break room watching everything that happened**).

Afternoon

After making a bit of progress in the aftermath of the morning disaster, we decided to have a 30-minute lunch and prayer break. During this time, I actually contacted an ex-colleague of mine during my internship who happened to stay at the same condo as mine, and asked him if I can temporarily stay at his place and use his WiFi. To my relief, he was fine with it and I immediately brought along my MacBook and all the essentials to his place upstairs.

When we came back at 1:45PM, things finally began to improve thanks to my colleague's fast internet connection. I was able to turn on my webcam and share my screen with ease. At least we have one issue solved, I thought to myself. Even though Faidz still struggles with her internet, we can breathe a sigh of relief knowing that one of us is able to share our screens.

The afternoon session was generally quiet at first, since I explicitly told my mentor Mufhim that we would want to focus on coding and that we will contact him personally should we need any assistance.

However, that was when the unexpected 'interview' sessions began.

While focusing on my codes, suddenly one of the mentors/judges unmuted himself and asked us upfront on why we were making little to no progress based on what I'm currently showing him on my screen. He explained how he's been observing us since morning and yet hours later, it seems like we haven't done much headway in terms of our prototype development and that at this rate, we might not make it to the end. I told him truthfully that it was mostly due to the issues we faced previously which significantly delayed our progress, and reassured him that we are picking up speed as we speak. It was partially my fault as well as I was only sharing the home/landing page that we did (I chose to share selective windows instead of the whole screen), even though we're already proceeding to developing other modules.

Somewhat satisfied with the answer, he then pointed out how throughout the day, it was mostly me doing all the talking, pinpointing how Faidz is being unreasonably quiet and uncommunicative. Knowing that Faidz is still shell-shocked and recovering over her laptop issue, I defended her on her behalf by again explaining what happened and how in normal circumstances, she is far more talkative. I also apologised for being too dominant in the team and that I should give chances for Faidz to speak for herself.

I thought that my answers ruined everything and that was the end of it for my hopes in getting the job offer. To my huge surprise, the interviewer laughed and explained how he wasn't actually reprimanding me. Rather, he praised me for being able to confidently and actively communicate our progress and problems faced. On the issue of being dominant, he gave his advice on how it depends on the situation I'm in, and how the circumstances we were facing entails me to step up on behalf of the team which is commendable.

Hearing what he said made me so relieved and reassured. Soon after, we focused back to our tasks at hand. Just when I thought we're done with the impromptu interview sessions, the competition had another surprise in store for us.

Out of nowhere, we heard a guy entering our break room and asking us on how we're doing. To our surprise, it was the head judge himself! He immediately asked us to demo what we've done so far. At that moment, part of me just wanted to jump off a cliff honestly. I did as instructed, praying that everything we've done so far worked. Thankfully, the event registration module that we finished worked flawlessly. He was especially impressed by the front-end design that Faidz made which closely resembles Dell's existing website and how much we're able to achieve given the fact that we're among the smallest team there is. Considering that there's only the two of us in the team, he advised us to focus on one use case that covers all the personas, mainly the event management functionality. Before leaving us, he teased by saying that he expects to be surprised further during our presentation the next day.

Knowing that the head judge himself was impressed with our progress gave us just the right surge of motivation. We took full advantage of that adrenaline rush to complete more functionalities and fix several bugs/errors.

As we're about to wrap up for the day at 5:00PM, we had gotten to know our turn for tomorrow's presentation. We were extremely glad to know that we'll be presenting later in the afternoon, giving us slightly more time to prepare for our presentation.

Our mentor Mufhim also came over at the last minute to check on our progress. We showed him what we've done so far and told him what happened during the afternoon. Echoing the head judge's comments, he praised our UI and even gave us some useful tips and advices for us to consider ahead of our presentation tomorrow. He even suggested that Faidz will lead the demo of the prototype during the presentation as a way to 'redeem' herself and prove her soft-skills, while I will handle the Q&A session.

Night

I told Faidz that we'll meet up again at 9:00PM sharp. I remember feeling so exhausted and went to sleep for 2 hours before our meeting. When we're both ready and freshen up, I went through what we achieved during the day and the remaining deliverables to be completed during the night. We had to restrategise a bit by dropping certain modules and replacing them with new ones, following the advices given by both Mufhim and the head judge as well as feedbacks obtained from the stakeholders. This part of the hackathon really tests your flexibility and adaptability to changes. Not everything that we planned beforehand were steps in the right direction. Hence, you have to ensure that you leave enough room in your planning in case there are changes to be made.

We spent the whole night, from 9:00PM until 7:00AM straight doing nothing but code, code and code. I'm an all-nighter myself, but even that doesn't prepare me on the sheer levels of fatigue that I experienced that night. There were times where I almost dozed off, but I forced myself to carry on knowing the finishing line is near. I lost count on how many times Faidz and I exchanged yawns. On top of completing all the necessary functionalities, we also had to juggle with prepping for our presentation slides. It was stressful for sure, having to work while being sleep-deprived as you'll experience brain fog. I can recall times where I literally forgot what I wanted to do in the middle of coding. But, it is certainly far more manageable since it's just the two of us (no interference from others) and our internet connections proved more reliable late at night. More importantly, it proved to be worth the effort as we managed to complete all the required features by dawn!

D-Day 2

Morning

After Subuh, the two of us immediately went asleep, albeit for 2 hours only as we need to wake up at 9:00AM to witness the other teams' presentations. Only 10 minutes is allocated for presentation and demo, while another 5 minutes is reserved for Q&A. The judges were extremely strict with the timing. Once the 10 minutes is over, you will be cut off immediately. In terms of the questions asked by the judges, they are mostly tailored and grilled towards the functionalities implemented. We also noticed how certain groups spent too much time in the PowerPoint presentations rather than their prototype demonstrations.

Keeping these observations in mind, we met up in our break room and outlined the flow of our presentation. I will lead the PowerPoint presentation, followed by Faidz taking over the demonstration. Finally, the Q&A will be handled by the two of us.

Mufhim was also kind enough to contact Faidz and give some final pointers for our presentation. Among his advices were:

Make sure we don't spend more than like 2 mins on the PowerPoint presentation in order to allocate as much time as possible for the prototype demonstration.
Try to avoid switching between screen-sharing in order to not waste time. Have someone standby to lead the demo while the other commentates.
Do at least 3 practice runs! Make sure you're familiar enough that you don't need to think about what to say.

We did exactly as he advised. After doing several rehearsals, we managed to keep our presentation well within the 10-minute mark. We felt quite confident after knowing the flow and what needs to be said.

Fast-forward to 1:15PM, the two of us finally went into the main room while patiently waiting for our turn. There was a huge timer on the screen, counting down to the next presentation slot at 1:30PM. I remember wishing Faidz good luck via WhatsApp a few minutes before.

Once the time is up, the emcee welcomed us and asked us to start whenever we are ready. I shared my screen, asking if everyone can see it and whether I can start. Once we were given the green-light, with the utterance of bismillah;

Houston, we have lift off!

And that, my friends, was honestly the longest 15 minutes of my life! Everything went smoothly and way better than expected! From the presentation to demonstration and finally the Q&A. I was able to answer all of the judges' questions, which centred mostly on the functionalities as predicted. They were also impressed as I walked through our planning stages using Trello, something other groups didn't do. Both our mentors even praised us for our performance.

Pro-tip during presentation/pitching session: If the judges grilled you as to why your prototype is missing certain features (trust me, you will as it's difficult to build a full-fledged prototype even with a team of four, let alone a duo like us), just tell them upfront that you don't have enough time (no bullsh*tting), but don't just stop there. Instead, outline clearly how in an ideal situation, the ways you intend to scale or improve some of the features. For instance, one of the judges questioned the lack of dashboard feature for one of our personas. I answered him by saying we couldn't implement the feature on time, however I explained to him what are the plans we had for said feature should there be no time constraint.

You can tell from our reactions in the WhatsApp conversation below on the aftermath on just how relieved and grateful we both were:

It was during this time that part of me legit thought that we might actually have a shot at winning!

UPDATE (16/2/2021): The organiser just sent me the Zoom video recording of our team's presentation. You can have a look at my YouTube channel!

Closing Ceremony

Following the excitement I got post-presentation, I can hardly contain my anxiety ahead of the closing ceremony. Despite being sleep-deprived, I wasn't even able to take a nap due to the eagerness and impatience.

When the clock stroke 5:30PM, I immediately went into the main room. We first had a photo-taking (or should I say, screenshot) session with all the participants, mentors and judges.

Then, the time has come. The moment that all of us are waiting for. At that time, there was a hodgepodge of feelings happening within me. On one hand, I'm praying that we'll nab a spot among the top 3 teams. On the other hand, part of me is concerned that I'm putting my hopes up too high and just leading my way to disappointment. After all, Surely, they won't award the win to a bunch of newbies, right? I mean, what are the odds?

At first, they announced the third placing. From there, I knew it's either we make it or we don't. Then, much to everyone's surprise, we actually won! Not only did we win, but we also tied as champion; a first in the competition's history. At that time, I can only afford to gasp and smile in shock. I was so fatigued that I can hardly process everything that's happening.

But that's not all. Deep down, we were all waiting for the grand prize: the permanent job and internship offer. When the head judge began calling out names, Faidz's was the first to be announced. I unmuted myself and cheering her up. As the head judge went through the list of names, I await anxiously. Finally, when my name was announced, I breathe a sigh of relief. I covered and shook my face in total disbelief. I remember the head judge giving a remark saying everyone was waiting to see my reaction.

Conclusion

It took me days to finally absorb everything that happened. In fact, I'm still sort of trying to grapple with the truth to this very moment. The idea that I** won a hackathon on first try, with merely a team of two and gotten a job offer at a prestigious company such as Dell felt like a distant dream.** Try and tell that to the first-year version of myself and he would think that you're a big fat liar.

This was undeniably a hallmark of my student life. A personal milestone. Looking back, what I'm most proud of is not the win and the job offer, contrary to everyone's opinion. Instead, it's the fact that I witnessed a stunning growth in myself. From that pessimistic, self-doubting freshman who believes he's incapable of anything but mediocrity, to who I am today. That is what I'm most proud of.

Several takeaways to share to everyone especially my juniors:

Never be afraid to try something new! Making that leap of faith can be tough at first, but trust me you'll be damn thankful to your past self once you did so. Your student life days are numbered, so make the most out of it by getting out of your comfort zones and explore new things. In most cases, ask yourself: what's there to lose?
Great things take time. I didn't get to where I am today by miracle means. It is a culmination of years of hard work and trial and error. As cliche as it sounds, be consistent, be patient and never give up. Trust me, your hard work will pay off in the end!
Team dynamics is key! If I was asked to single out 1 main factor that contributes to our success, it would be the sense of camaraderie that I had with Faidz. The high-octane environment of hackathon will really test your team's synergy. How you manage crises along the way is dependent on teamwork. So if you intend to join hackathon in the future (which you should), team up with those that you can work well with.
Start networking! If you are observant enough, you will notice that my journey towards this achievement wouldn't have been possible without the help of my friends. Building trust and connection is key. It might not be so apparent now, but it will bring to fruition in the future. As the old adage goes:

If you want to go fast, go alone. If you want to go far, go with others.

Deploying React-Django App using Heroku

Muhammad Rahiman — Sun, 06 Dec 2020 10:14:09 +0000

React-Django-Heroku Web App Deployment

Introduction

This is a simple Todo web application built using React.js (front-end) and Django (back-end).

There are mainly two ways you can deploy this kind of web app:

Separating Back-end and Front-end:
In this method, you serve your back-end and front-end apps separately and they connect to each other with their respective URIs. One major overead of this approach is you have to configure CORS yourself. More about CORS will be explained later in the guide.
Serving from the same host:
In this method you will be serving the app from the same URI so it removes the CORS overhead. Also, it makes it easier to maintain smaller-medium sized apps. You don't want to create two separate repositories for some simple sites like a blog, to-do etc.

This method requires building the React app first (npm run build) and update both the Django views (views.py) and urls (urls.py) to render the index.html template directly.

For this guide, we will be demonstrating this longer, more complex but more flexible approach for a simple todo app. I will try out this second, easier method in the future.

Technology Stack

Front-end

react@17.0.1
- react-dom@17.0.1
axios@0.21.0
bootstrap@4.5.3
reactstrap@8.7.1
Back-end
Django==3.1.3
- djangorestframework==3.12.2
- django-cors-headers==3.5.0
- dj-database-url==0.5.0
- django-heroku==0.3.1
gunicorn==20.0.4
whitenoise==5.2.0
python-dotenv==0.15.0
psycopg2==2.8.6
psycopg2-binary==2.8.6

Prerequisites

1. Install Node.js

Download Node.js and Node Package Manager (NPM) here.

2. Set up Virtual Environment

a. Using venv (preferred)

To create virtual environment:
```
$ python3 -m venv venv
```
To activate virtual environment (Linux/Mac OS):
```
$ source venv/bin/activate
```

b. Using pipenv

Install pipenv:

$ pip install pipenv # Python 2
$ pip3 install pipenv # Python 3

Enter virtual environment shell:
```
$ pipenv shell
```

Deployed Web Application

Click here to view the deployed todo web application!

Click here to visit the GitHub repository.

Method 1: Separating Back-end and Front-end

Setting up the Backend

1. Create new project directory

$ mkdir (DIR_NAME)
$ cd (DIR_NAME)

2. Install dependencies

Run the following command inside your virtual environment:

Using pipenv:

$ pipenv install -r requirements.txt # (Python 2)
$ pipenv3 install -r requirements.txt # (Python 3)

Using venv:

$ pip install -r requirements.txt # (Python 2)
$ pip3 install -r requirements.txt # (Python 3)

3. Create new Django project

Run the following command inside your virtual environment:

$ django-admin startproject backend

4. Extract all Django project files to root directory (recommended)

The directory should look as follows:

.
├── README.md
├── backend
│   ├── __init__.py
│   ├── __pycache__
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── db.sqlite3
├── manage.py

5. Create new Django app

First, navigate into the newly created backend folder. Then, start a new Django app, in this case called todo since we want to create a todo app. We will also run migrations and start up the server:

$ cd backend
$ python manage.py startapp todo
$ python manage.py migrate
$ python manage.py runserver

If everything works well, we should see an instance of a Django application running on this address — http://localhost:8000

6. Register new Django app

Open the backend/settings.py file and update the INSTALLED_APPS section as so:

# backend/settings.py

# Application definition
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'todo' # Add this
]

7. Set up the APIs

In order to allow for API CRUD operations between both front and back end, we need to utilise the Django REST Framework as well as Django CORS Headers.

Django REST framework is a powerful and flexible toolkit for building Web APIs, while Django CORS Headers is a Django app for handling the server headers required for Cross-Origin Resource Sharing (CORS).

The image below essentially sums up the application architecture of a React-Django stack web app:

Django exports REST APIs using Django REST Framework and interacts with Database using Django Model.
React Client sends HTTP Requests and retrieve HTTP Responses using axios, shows data on the components.

After installing all the necessary requirements in Step 2, we need to add rest_framework and corsheaders to the list of installed applications, so open the backend/settings.py file and update the INSTALLED_APPS and MIDDLEWARE sections accordingly:

# backend/settings.py

# Application definition
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'corsheaders',                              # add this
    'rest_framework',                           # add this
    'todo',
  ]

MIDDLEWARE = [
    'corsheaders.middleware.CorsMiddleware',    # add this
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

At the bottom of the backend/settings.py file, you need to choose to add either the following:

# # Whitelist localhost:3000 because that's where frontend will be served

# Option 1
CORS_ORIGIN_WHITELIST = [
    'https://localhost:3000',
]

# Option 2
CORS_ORIGIN_ALLOW_ALL = True

CORS_ORIGIN_ALLOW_ALL: If True, all origins will be accepted (not use the whitelist below). Defaults to False.
CORS_ORIGIN_WHITELIST: List of origins that are authorised to make cross-site HTTP requests. Defaults to [].

8. Creating serializers for the Django model and views

We need serializers to convert model instances to JSON so that the frontend can work with the received data easily. We will create a todo/serializers.py file:

touch todo/serializers.py

Open the todo/serializers.py file and update it with the following code:

# todo/serializers.py
from rest_framework import serializers # This is important
from .models import Todo

class TodoSerializer(serializers.ModelSerializer):
    class Meta:
        model = Todo
        fields = ('id', 'title', 'description', 'completed')

For any type of Django app, you can configure its own model as long as serializers are imported and serializers.ModelSerializer model class parameter is defined.

For the views in todo/views.py:

# todo/views.py

from django.shortcuts import render
from rest_framework import viewsets          # add this
from .serializers import TodoSerializer      # add this
from .models import Todo                     # add this

class TodoView(viewsets.ModelViewSet):       # add this
    serializer_class = TodoSerializer          # add this
    queryset = Todo.objects.all()              # add this

The viewsets base class provides the implementation for CRUD operations by default, what we had to do was specify the serializer class and the query set.

Finally, we need to update the backend/urls.py:

# backend/urls.py

from django.contrib import admin
from django.urls import path, include                   # add this
from rest_framework import routers                      # add this
from todo import views                                  # add this

router = routers.DefaultRouter()                        # add this
router.register(r'todos', views.TodoView, 'todo')       # add this

urlpatterns = [
    path('admin/', admin.site.urls),         
    path('api/', include(router.urls))                   # add this
]

The router class allows us to make the following queries:

/todos/ - This returns a list of all the Todo items (Create and Read operations can be done here).
/todos/id - this returns a single Todo item using the id primary key (Update and Delete operations can be done here).

Setting up the Frontend

1. Create a new React application

Once Node.js is installed, you can quick start creating your first React app by using the following commands:

$ npx create-react-app frontend
$ cd my-app
$ npm start

In this case, the app we are creating is called frontend to compliment the Django's backend project name. This will take a few minutes to complete.

When everything is done, a new folder will be created with the following directory tree:

.
├── README.md
├── backend
│   ├── __init__.py
│   ├── __pycache__
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── db.sqlite3
├── manage.py
└── frontend
    ├── README.md
    ├── node_modules
    ├── package.json
    ├── .gitignore
    ├── public
    │   ├── favicon.ico
    │   ├── index.html
    │   └── manifest.json
    └── src
        ├── App.css
        ├── App.js
        ├── App.test.js
        ├── index.css
        ├── index.js
        ├── logo.svg
        └── reportWebVitals.js
        └── setupTests.js

2. Extract all React project files to root directory (recommended)

In order to make things much easier later in the process, you are advised to move all the files inside


 to the **root directory** (same level as

  ```backend/```

 folder) as follows:



```shell
.
├── README.md
├── backend
│   ├── __init__.py
│   ├── __pycache__
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── db.sqlite3
├── manage.py
├── node_modules
├── package-lock.json
├── package.json
├── public
├── requirements.txt
├── runtime.txt
├── src
└── todo

3. Develop your React application

For this step, it is entirely up to you on how you are going to develop your React application. You can refer to the React codes included inside my GitHub repository (react-dj-todoapp) as reference.

4. Connecting Django with React using axios

For us to make requests to the API endpoints on the Django back-end server, we will need install a JavaScript library called axios.

axios is a popular, promise-based HTTP client that sports an easy-to-use API and can be used in both the browser and Node.js. For more information regarding axios, you can read them here.

First, we need to install axios using NPM:

$ npm install axios

Once axios is successfully installed, head over to the package.json file and add a proxy like so:

// package.json

[...]       
"name": "frontend",
"version": "0.1.0",
"private": true,
"proxy": "http://localhost:8000",
"dependencies": {
    "axios": "^0.18.0",
    "bootstrap": "^4.1.3",
    "react": "^16.5.2",
    "react-dom": "^16.5.2",
    "react-scripts": "2.0.5",
    "reactstrap": "^6.5.0"
},
[...]

The proxy will help in tunnelling API requests to http://localhost:8000 where the Django application will handle them, so we can simplify writing the requests like this in React:

axios.get("/api/todos/")

Instead of this:

axios.get("http://localhost:8000/api/todos/")

After that, you need to modify the React codes (App.js)

Here's a snippet on how to use axios:

// src/App.js

handleSubmit = item => {
    this.toggle();
    if (item.id) {
    axios
        .put(`http://localhost:8000/api/todos/${item.id}/`, item)
        .then(res => this.refreshList());
    return;
    }
    axios
      .post("http://localhost:8000/api/todos/", item)
      .then(res => this.refreshList());
};

handleDelete = item => {
    axios
        .delete(`http://localhost:8000/api/todos/${item.id}`)
        .then(res => this.refreshList());
};

For further information and example, please refer to this Digital Ocean guide.

5. Test the Web Application

You can check whether everything is working by running both React and Django concurrently at your local development server:

$ npm start
$ python manage.py runserver

Test whether both the UI and API requests are working or not.

Deploying the Web App using Heroku

1. Set up Heroku account and CLI

Here is an outline following Heroku's from-product-to-productionized instructions for a Django deployment to Heroku:

Signup for Heroku if you don't have an existing account
Install the Heroku CLI. For MacOS, use $ brew tap heroku/brew && brew install heroku:
Log in to your Heroku account by entering your credentials using $ heroku login:
Create a n*ew Heroku app* either via Heroku CLI ($ heroku create APP_NAME) or directly in the Heroku dashboard:
Add the Heroku remote via $ heroku git:remote -a your-heroku-app.
Configure the Heroku buildpacks

* The React app build process depends on NPM, so we need Node.js. We also need Python to run Django.

* Heroku uses **buildpacks** to **transform deployed code into slugs** which can be executed by Dynos (server instances on Heroku). We’ll be needing two buildpacks. One for Node and another for Python.

* Our app would run on a Python server, even though we will use Node/NPM to build/bundle the React frontend. So the Python buildpack will be the main one in our config. The main buildpack determines the process type of the Heroku app. You can read about multiple buildpacks to understand how they work.

* You can add buildpacks via the Heroku CLI. Head back to your terminal and run the following to set/add the buildpacks we need:

```shell
$ heroku buildpacks:add --index 1 heroku/nodejs
$ heroku buildpacks:add --index 2 heroku/python
```

* Note that the buildpacks **must be added in that order**. We can see the buildpacks we’ve added by running `$ heroku buildpacks`. The **last buildpack** on the list **determines the process type** of the app.

![alt text](https://alphacoder.xyz/images/dply-dj/buildpacks.png)

Configure PostgreSQL Heroku addon

* During production, Heroku will **not be using SQLite database**. Instead, we need to use **PostgreSQL** by configuring the addon to our app using `$ heroku addons:create heroku-postgresql:hobby-dev`
* You can check whether this is successful by running `$ heroku config`:

 ```shell
$ === APP_NAME Config Vars
DATABASE_URL: postgres://[DATABASE_INFO_HERE]
```

* The database info from the code snippet above refers to the URL containing your database’s location and access credentials all in one. Anyone with this URL can access your database, so be careful with it.
* You will notice that Heroku saves it as an **environment variable** called `DATABASE_URL` . This URL can and does change, so you should never hard code it. Instead, we’ll use the variable `DATABASE_URL` in  Django.

Configure Heroku config variables

* According to Heroku, **config variables** are environment variables that can change the way your app behaves. In addition to creating your own, some add-ons come with their own.
* There are several environment variables that need to be set:

```shell
$ heroku config:set ALLOWED_HOSTS=APP_NAME.herokuapp.com
$ heroku config:set ALLOWED_HOSTS=APP_NAME.herokuapp.com
$ heroku config:set SECRET_KEY=DJANGO_SECRET_KEY
$ heroku config:set WEB_CONCURRENCY=1
```

Import django-heroku inside settings.py

* `django-heroku` is a **Django library** for Heroku applications that ensures a more seamless deployment and development experience.
* This library provides:
    * **Settings configuration** (Static files / WhiteNoise)
    * **Logging configuration**
    * **Test runner** (important for Heroku CI)
* In `settings.py`, include the following at the very bottom:

```python
# backend/settings.py

# Configure Django App for Heroku.
import django_heroku
django_heroku.settings(locals())
```

2. Configure the Django back-end side

Database Configuration

A. Create .env

As mentioned above, the local version of the Django app is using db.sqlite3 as its database. However, when we visit the Heroku version, APP_NAME.herokuapp.com, Heroku will need to use a PostgreSQL database instead.

What we want to do is to get our app running with SQLite whenever we’re working on it locally, and with Postgres whenever it’s in production. This can be done using the installed python-dotenv library.

We will then use a file called .env to tell Django to use SQLite when running locally. To create .env and have it point Django to your SQLite database:

    $ echo 'DATABASE_URL=sqlite:///db.sqlite3' > .env

Include the .env file inside our .gitignore when pushing to Heroku by running the following command:

    $ echo '.env' >> .gitignore

B. Update settings.py

First, import the necessary libraries for deployment purposes:

# backend/settings.py

import django_heroku
import dotenv
import dj_database_url

Next, we need to set up the database configuration:

# backend/settings.py

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().root.root

dotenv_file = os.path.join(BASE_DIR, ".env")
if os.path.isfile(dotenv_file):
    dotenv.load_dotenv(dotenv_file)

Since .env won’t exist on Heroku, dotenv.load_dotenv(dotenv_file) will never get called on Heroku and Heroku will proceed to try to find its own database — PostgreSQL.

We also need to configure the DATABASES setting as shown below:

# backend/settings.py

DATABASES = {}
DATABASES['default'] = dj_database_url.config(conn_max_age=600)

The idea here is to clear the DATABASES variable and then set the 'default' key using the dj_database_url module. This module uses Heroku’s DATABASE_URL variable if it’s on Heroku, or it uses the DATABASE_URL we set in the .env file if we’re working locally.

If you ran the Django application as specified above, you might get an error when working locally because the dj_database_url module wants to log in with SSL. Heroku Postgres requires SSL, but SQLite doesn’t need or expect it. Here's how to fix that:

# backend/settings.py

# This should already be in your settings.py
django_heroku.settings(locals())

# Add these at the very last line of settings.py
options = DATABASES['default'].get('OPTIONS', {})
options.pop('sslmode', None)

Test everything out by running the local Django server using python3 manage.py runserver.

Static files serve (WhiteNoise)

A. WhiteNoise settings

WhiteNoise allows your web app to serve its own static files, making it a self-contained unit that can be deployed anywhere without relying on nginx, Amazon S3 or any other external service. (Especially useful on Heroku, OpenShift and other PaaS providers.)

Since this is already installed from the requirements.txt file earlier on, we need to update settings.py:

# backend/settings.py

MIDDLEWARE = [
  'django.middleware.security.SecurityMiddleware',
  'whitenoise.middleware.WhiteNoiseMiddleware',
  # ...
]

STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

B. Static files and template settings

In order to correctly serve the static files from both Django and React, we need to update the following in settings.py:

# backend/settings.py

STATIC_URL = '/static/'

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'build/static')
]

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

STATIC_ROOT points to the directory containing all the static files, while STATICFILES_DIRS refers to other directories where Django will collect the static files as well. In this case, it is pointing to React's 'build/static' directory which contains the static files for frontend when Heroku builds the React app using npm run build during deployment.

Therefore, when Heroku runs python3 manage.py collectstatic during deployment, it will automatically compile all the static files from both React and Django.

Meanwhile for Django's template directory, we need to include the React build directory which stores the index.html inside TEMPLATES as follows:

# backend/settings.py

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'build')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

C. Static files MIME Type issue

Upon deploying the web app in Heroku, one of the common issues that occur is the static files failing to load due to MIME type limitations. The particular MIME type (text/html) problem is related to your Django configuration.

The views.py in your React frontend needs a content_type argument in the HttpResponse.
Heroku needs to know where the static files are.

The "refused to execute script ... MIME type ('text/html')" problem stems from Django's default content_type setting for an HttpResponse, which is text/html.

This can be fixed by including a content_type='application/javascript' argument in the return statement of a new class-based view called Assets(View) inside views.py like so:

# todo/views.py

from django.shortcuts import render
from rest_framework import viewsets
from .serializers import TodoSerializer
from .models import Todo
from django.views import View
from django.http import HttpResponse, HttpResponseNotFound
import os

# The viewsets base class provides the implementation for CRUD operations by default,
# what we had to do was specify the serializer class and the query set.
class TodoView(viewsets.ModelViewSet):
    serializer_class = TodoSerializer
    queryset = Todo.objects.all()

# Add this CBV
class Assets(View):

    def get(self, _request, filename):
        path = os.path.join(os.path.dirname(__file__), 'static', filename)

        if os.path.isfile(path):
            with open(path, 'rb') as file:
                return HttpResponse(file.read(), content_type='application/javascript')
        else:
            return HttpResponseNotFound()

3. Debug and Access

During production, it is strongly recommended that DEBUG is set to False.

For ALLOWED_HOSTS, if you set CORS_ORIGIN_ALLOW_ALL to True:

# backend/settings.py

ALLOWED_HOSTS = ['*'] # Set to open for all access

Else, you need to specify the URL access permissions as follows:

# backend/settings.py

ALLOWED_HOSTS = ['react-dj-todoapp.herokuapp.com', '127.0.0.1:8000', 'localhost']

4. Configure the React front-end side

A. Configure package.json

For React deployment using Heroku, there are a few changes needed to be made inside the package.json:

// package.json

// Add homepage and engine settings

{   
    "name": "frontend",
    "homepage": ".",
    "version": "0.1.0",
    "private": true,
    "proxy": "http://localhost:8000",
    "dependencies": {
        "axios": "^0.18.0",
        "bootstrap": "^4.1.3",
        "react": "^16.5.2",
        "react-dom": "^16.5.2",
        "react-scripts": "2.0.5",
        "reactstrap": "^6.5.0"
    },
    "scripts": {
        "start": "react-scripts start",
        "build": "react-scripts build",
        "test": "react-scripts test",
        "eject": "react-scripts eject"
    },
    "engines": {
        "node": "14.6.0",
        "npm": "6.14.9"
    },
    [...]

The engines part refers to the preferred version of Node and NPM.

B. Fix Django's CSRF token verification conflict

The built-in CSRF protection provided by Django is very useful to protect your server from malicious websites that can exploit your visitor browser to attack you. However, when using modern JavaScript libraries you will need to handle CSRF differently.

Referring back to the first part of the README.md, because React and Django apps use different URL and port number, Django's CSRF token verification mechanism will prevent React from executing any API requests to Django during production.

To solve this, we need to tell axios to handle CSRF tokens in React. For axios client you have three options:

Manually attach the CSRF token in the header of each Axios call
Use the Axios xsrfHeaderName for each call
Use a default xsrfHeaderName (axios.defaults.xsrfHeaderName = "X-CSRFToken")

Here is how you can simply use the CSRF token with Axios without any further configuration:

// src/App.js

import axios from 'axios';

axios.defaults.xsrfCookieName = 'csrftoken'
axios.defaults.xsrfHeaderName = 'X-CSRFToken'

5. Set up Heroku-specific files

A. runtime.txt

Heroku will install a default Python version if you don't specify one, but if you want to pick your Python version, you'll need a runtime.txt file.

Create one in the root directory, next to your requirements.txt, manage.py, .gitignore and the rest. Specify your Python version with the prefix python- that you want your application to run on:

python-3.9.0

B. requirements.txt

When deploying the web app, Heroku will need to install all the required dependencies for the web app to run by referring to the requirements.txt file.

To ensure that all dependencies are included, consider freezing your dependencies using the command $ pip freeze > requirements.txt. This will make your build a little bit more predictable by locking your exact dependency versions into your Git repo. If your dependencies aren't locked, you might find yourself deploying one version of Django one day and a new one the next.

C. Procfile

Heroku apps include a Heroku-specific Procfile that specifies the processes our application should run. The processes specified in this file will automatically boot on deploy to Heroku.

Create a file named Procfile in the root level directory using $ touch Procfile command, right next to your requirements.txt and runtime.txt files. (Make sure to capitalize the P of Procfile otherwise Heroku might not recognize it!):

Then, fill in the codes below:

release: python manage.py migrate
web: gunicorn backend.wsgi --log-file -

6. Commit and Push

Once all the previous steps are completed, we are ready to finally commit and push all changes:

$ git add .
$ git commit -m "blah blah blah"
$ git push heroku master

After the build is done and your app has been released, visit YOUR-APP-NAME.herokuapp.com.

7. Database Syncing

Upon successful deployment, one of the expected problems that may arise is the 500 Server Error or the following error:

ProgrammingError at /
relation "TABLE_NAME" does not exist

This error can be resolved by running $ python3 manage.py migrate --run-syncdb using Heroku run bash:

$ heroku run python3 manage.py migrate --run-syncdb

And if everything goes well up to this point, congratulations! You have successfully deployed your first React-Django web app!

8. Debugging

If everything did not go well, you can use the $ heroku logs --tail to investigate any errors.

DEV Community: Muhammad Rahiman

Developing REST API Using Django REST Framework (DRF)

Introduction

Setting Up Django

1. Create new Django Project

2. Create new Virtual Environment

3. Install PIP Dependencies

4. Start New Django App

5. Configure settings.py

5. Configure urls.py

Setting Up Django REST Framework

1. Install Django REST Framework

2. Configure settings.py

3. Create a Model

4. Configure Serializers

5. Configure Views (views.py)

6. Configure API Endpoints (urls.py)

7. Test API Connection

Configure Permissions and Authentication

1. Permissions

2. Authentication

API Documentation using drf_spectacular

References

Deploying Django Web App Using Heroku (Updated)

A. runtime.txt

B. requirements.txt

C. Procfile

IMPORTANT NOTE:

Web Scraping using Django and Selenium

Getting Started with Django

1. Set up your Django project

2. Configure project settings

3. Configure app settings

Getting Started with Selenium

What is Web Scraping?

What is Selenium?

Installing Selenium

Installing Webdrivers

Set Up Selenium in Django

Web-Scrapping MAYA UM using Selenium

1. Going through Authentication

Setting Up Environment Variables

Extracting XPath

Finding Elements

2. Redirecting to Search Timetable page

3. Filling Up Search Timetable Form

Get encrypted ID

4. Extracting Course Timetables from DataTables

References

Django + Firebase Cloud Storage

Getting Started

1. Set up your Django project

2. Configure project settings

3. Configure app settings

4. Install Pyrebase package

5. Create a new Firebase project

6. Create Firebase app

7. Set up Django app views

8. Create new Firebase Cloud Storage

9. Configure file upload to Firebase Cloud Storage

10. Implement file upload logic (optional)

References

Hackathon: A Newbie's Experience

What is Hackathon?

Why Now?

How It Began

The Prep

D-Day 1

Morning

Afternoon

Night

D-Day 2

Morning

Closing Ceremony

Conclusion

Deploying React-Django App using Heroku

React-Django-Heroku Web App Deployment

Introduction

Technology Stack

Front-end

5. Configure `settings.py`

5. Configure `urls.py`

2. Configure `settings.py`

5. Configure Views (`views.py`)

6. Configure API Endpoints (`urls.py`)

API Documentation using `drf_spectacular`