DEV Community

loading...

Understanding celery and creating simple task in django

daeenchoi profile image Daeen Choi ・4 min read

Aim of this post

  1. Understand what is celery
  2. Create a simple task in Django project (celery)

What is celery?

Celery is an asynchronous processing and scheduling tool in Django (not limited to Django).

So what is asynchronous?
When a client sends a request to the server, normally client cannot do anything until it receives a response from the server. That said, when request is time-consuming work, I mean more than 1 minute to handle,
user will start clicking the refresh button couple of times or click on the back button.

To prevent this situation, we process APIs asynchronously. Asynchronous processing allows clients to send requests and do other tasks immediately. When the response returns, the processing of the response continues.

This asynchronous processing is done by celery.

What about scheduling?
You can make a task which runs at 8 o'clock every morning. Or you might also want to send e-mails to users every Friday at 4:00 p.m. This is possible with celery-beat.

How does celery work?
Alt Text
The unit of work that Celery has to deal with is called a task. Producer (celery) creates a task and the task is added to the Task Queue before it is executed. The broker then forwards the task in the task queue to the appropriate worker node which is the consumer (celery) where the task is executed. Celery can be seen as a larger concept, including producer, broker, and consumer.

Let's create a simple task!

  1. Prepare a django project (any project is fine - you can just use https://docs.djangoproject.com/en/3.1/intro/tutorial01/ if you don't know where to start)

  2. Install and run rabbitmq. (I used homebrew)
    Try https://www.rabbitmq.com/install-homebrew.html if you don't have rabbitmq yet.
    brew services start rabbitmq to start up rabbitmq

  3. Install celery in virtual environment
    pip install celery

  4. create new celery app
    python manage.py startapp djangocelery <- you can name your app anything. I called mine djangocelery

  5. Settings and more
    5-1. create celery.py djangocelery app(directory) and add following code

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'betweak.settings') 
app = Celery('djangocelery') 

# You can add celery settings in settings.py starting with CELERY_ 
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(lambda : settings.INSTALLED_APPS)
Enter fullscreen mode Exit fullscreen mode

5-2. Edit djangocelery > init.py file

from .celery import app as celery_app
# celery app is automatically imported when django starts
__all__ = ['celery_app']
Enter fullscreen mode Exit fullscreen mode

5-3. Edit settings.py

INSTALLED_APPS = [

   ...
   'celery'
   'djangocelery' # newly created app
]

# set CELERY settings with CELERY_{}
CELERY_BROKER_URL = amqp://guest:guest@localhost:5672/ # rabbitmq setting
CELERY_TIMEZONE = 'Asia/Seoul'
CELERY_ENABLE_UTC=False
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']

Enter fullscreen mode Exit fullscreen mode

6 . Migrate database
python mange.py migrate

7 . Create task

Create task.py file in djangocelery and add following code

from djangocelery.celery import app

@app.task(name="mod", bind=True, default_retry_delay=10, max_retries=5)
def mod(self, x, y):
    try:
        z = x % y
        print(f'{x} % {y} = {x%y}')
        return z
    except :
        mod.retry()
        print(f'Error with mod')

Enter fullscreen mode Exit fullscreen mode

8 . Add views so we can access task with API

djangocelery > views.py

from django.http import HttpResponse
from rest_framework.decorators import api_view
from djangocelery.tasks import mod

@api_view(['GET'])
def get_test_add(request):
    x, y = int(request.query_params.get('x')), int(request.query_params.get('y'))
    mod.apply_async(args=(x, y), countdown=20)
    return HttpResponse(f'Success !')
Enter fullscreen mode Exit fullscreen mode

djangocelery > urls.py

from django.urls import path

from djangocelery import views

urlpatterns = [
    path('mod', views.get_test_add)
]

Enter fullscreen mode Exit fullscreen mode

betweak > urls.py (project urls)

urlpatterns = [
    path('celery/', include('djangocelery.urls')),
    path('admin/', admin.site.urls),
]

Enter fullscreen mode Exit fullscreen mode

Ok so settings are ready. Let's test it out!

Run celery

celery -A djangocelery worker loglevel-info

Run django (in another terminal)

python manage.py runserver

Alt Text

Let's check celery log to see what's happening.
Alt Text

We can see that at 22:32:33 the task is received. Approximately 20 seconds later at 22:32:54, the task succeeded.

If we see djangocelery > views.py again

ffrom djangocelery.tasks import mod
@api_view(['GET'])
def get_test_add(request):

    # so we get the x and y values from the url
    x, y = int(request.query_params.get('x')), int(request.query_params.get('y'))

    # Here countdown=20 means executes in 20 seconds from now 
    mod.apply_async(args=(x, y), countdown=20)
    return HttpResponse(f'Success ! ')

Enter fullscreen mode Exit fullscreen mode

Well that was easy?

Let's try to make an error. We will pass 0 value for y. Python raises when we try to use 0 for modulo.
Alt Text

Let's check the log out
Alt Text

There's a lot going on there. We see there was a ZeroDivisionError, and the task retried 5 times!

djangocelery > task.py

# Here are the settings for retries!!
@app.task(name="mod", bind=True, default_retry_delay=10, max_retries=5)
def mod(self, x, y):
    try:
        z = x % y
        print(f'{x} % {y} = {x%y}')
        return z
    except :
        mod.retry()
        print(f'Error with add')

Enter fullscreen mode Exit fullscreen mode

So that's it for getting started with celery in django :)

Discussion (0)

pic
Editor guide