Understanding celery and creating simple task in django

#python #django

Aim of this post

Understand what is celery
Create a simple task in Django project (celery)

What is celery?

Celery is an asynchronous processing and scheduling tool in Django (not limited to Django).

So what is asynchronous?
When a client sends a request to the server, normally client cannot do anything until it receives a response from the server. That said, when request is time-consuming work, I mean more than 1 minute to handle,
user will start clicking the refresh button couple of times or click on the back button.

To prevent this situation, we process APIs asynchronously. Asynchronous processing allows clients to send requests and do other tasks immediately. When the response returns, the processing of the response continues.

This asynchronous processing is done by celery.

What about scheduling?
You can make a task which runs at 8 o'clock every morning. Or you might also want to send e-mails to users every Friday at 4:00 p.m. This is possible with celery-beat.

How does celery work?

The unit of work that Celery has to deal with is called a task. Producer (celery) creates a task and the task is added to the Task Queue before it is executed. The broker then forwards the task in the task queue to the appropriate worker node which is the consumer (celery) where the task is executed. Celery can be seen as a larger concept, including producer, broker, and consumer.

Let's create a simple task!

Prepare a django project (any project is fine - you can just use https://docs.djangoproject.com/en/3.1/intro/tutorial01/ if you don't know where to start)
Install and run rabbitmq. (I used homebrew)
Try https://www.rabbitmq.com/install-homebrew.html if you don't have rabbitmq yet.
brew services start rabbitmq to start up rabbitmq
Install celery in virtual environment
pip install celery
create new celery app
python manage.py startapp djangocelery <- you can name your app anything. I called mine djangocelery
Settings and more
5-1. create celery.py djangocelery app(directory) and add following code

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'betweak.settings') 
app = Celery('djangocelery') 

# You can add celery settings in settings.py starting with CELERY_ 
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(lambda : settings.INSTALLED_APPS)

5-2. Edit djangocelery > init.py file

from .celery import app as celery_app
# celery app is automatically imported when django starts
__all__ = ['celery_app']

5-3. Edit settings.py

INSTALLED_APPS = [

   ...
   'celery'
   'djangocelery' # newly created app
]

# set CELERY settings with CELERY_{}
CELERY_BROKER_URL = amqp://guest:guest@localhost:5672/ # rabbitmq setting
CELERY_TIMEZONE = 'Asia/Seoul'
CELERY_ENABLE_UTC=False
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']

6 . Migrate database
python mange.py migrate

7 . Create task

Create task.py file in djangocelery and add following code

from djangocelery.celery import app

@app.task(name="mod", bind=True, default_retry_delay=10, max_retries=5)
def mod(self, x, y):
    try:
        z = x % y
        print(f'{x} % {y} = {x%y}')
        return z
    except :
        mod.retry()
        print(f'Error with mod')

8 . Add views so we can access task with API

djangocelery > views.py

from django.http import HttpResponse
from rest_framework.decorators import api_view
from djangocelery.tasks import mod

@api_view(['GET'])
def get_test_add(request):
    x, y = int(request.query_params.get('x')), int(request.query_params.get('y'))
    mod.apply_async(args=(x, y), countdown=20)
    return HttpResponse(f'Success !')

djangocelery > urls.py

from django.urls import path

from djangocelery import views

urlpatterns = [
    path('mod', views.get_test_add)
]

betweak > urls.py (project urls)

urlpatterns = [
    path('celery/', include('djangocelery.urls')),
    path('admin/', admin.site.urls),
]

Ok so settings are ready. Let's test it out!

Run celery

celery -A djangocelery worker loglevel-info

Run django (in another terminal)

python manage.py runserver

Let's check celery log to see what's happening.

We can see that at 22:32:33 the task is received. Approximately 20 seconds later at 22:32:54, the task succeeded.

If we see djangocelery > views.py again

ffrom djangocelery.tasks import mod
@api_view(['GET'])
def get_test_add(request):

    # so we get the x and y values from the url
    x, y = int(request.query_params.get('x')), int(request.query_params.get('y'))

    # Here countdown=20 means executes in 20 seconds from now 
    mod.apply_async(args=(x, y), countdown=20)
    return HttpResponse(f'Success ! ')

Well that was easy?

Let's try to make an error. We will pass 0 value for y. Python raises when we try to use 0 for modulo.

Let's check the log out

There's a lot going on there. We see there was a ZeroDivisionError, and the task retried 5 times!

djangocelery > task.py

# Here are the settings for retries!!
@app.task(name="mod", bind=True, default_retry_delay=10, max_retries=5)
def mod(self, x, y):
    try:
        z = x % y
        print(f'{x} % {y} = {x%y}')
        return z
    except :
        mod.retry()
        print(f'Error with add')

So that's it for getting started with celery in django :)