Aim of this post
- Understand what is celery
- Create a simple task in Django project (celery)
What is celery?
Celery is an asynchronous
processing and scheduling
tool in Django (not limited to Django).
So what is asynchronous?
When a client sends a request to the server, normally client cannot do anything until it receives a response from the server. That said, when request is time-consuming work, I mean more than 1 minute to handle,
user will start clicking the refresh button couple of times or click on the back button.
To prevent this situation, we process APIs asynchronously. Asynchronous processing allows clients to send requests and do other tasks immediately. When the response returns, the processing of the response continues.
This asynchronous processing is done by celery.
What about scheduling?
You can make a task which runs at 8 o'clock every morning. Or you might also want to send e-mails to users every Friday at 4:00 p.m. This is possible with celery-beat.
How does celery work?
The unit of work that Celery has to deal with is called a task. Producer (celery) creates a task and the task is added to the Task Queue before it is executed. The broker then forwards the task in the task queue to the appropriate worker node which is the consumer (celery) where the task is executed. Celery can be seen as a larger concept, including producer, broker, and consumer.
Let's create a simple task!
Prepare a django project (any project is fine - you can just use https://docs.djangoproject.com/en/3.1/intro/tutorial01/ if you don't know where to start)
Install and run rabbitmq. (I used homebrew)
Try https://www.rabbitmq.com/install-homebrew.html if you don't have rabbitmq yet.
brew services start rabbitmq
to start up rabbitmqInstall celery in virtual environment
pip install celery
create new celery app
python manage.py startapp djangocelery
<- you can name your app anything. I called minedjangocelery
Settings and more
5-1. createcelery.py
djangocelery
app(directory) and add following code
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'betweak.settings')
app = Celery('djangocelery')
# You can add celery settings in settings.py starting with CELERY_
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(lambda : settings.INSTALLED_APPS)
5-2. Edit djangocelery > init.py file
from .celery import app as celery_app
# celery app is automatically imported when django starts
__all__ = ['celery_app']
5-3. Edit settings.py
INSTALLED_APPS = [
...
'celery'
'djangocelery' # newly created app
]
# set CELERY settings with CELERY_{}
CELERY_BROKER_URL = amqp://guest:guest@localhost:5672/ # rabbitmq setting
CELERY_TIMEZONE = 'Asia/Seoul'
CELERY_ENABLE_UTC=False
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']
6 . Migrate database
python mange.py migrate
7 . Create task
Create task.py file in djangocelery and add following code
from djangocelery.celery import app
@app.task(name="mod", bind=True, default_retry_delay=10, max_retries=5)
def mod(self, x, y):
try:
z = x % y
print(f'{x} % {y} = {x%y}')
return z
except :
mod.retry()
print(f'Error with mod')
8 . Add views so we can access task with API
djangocelery > views.py
from django.http import HttpResponse
from rest_framework.decorators import api_view
from djangocelery.tasks import mod
@api_view(['GET'])
def get_test_add(request):
x, y = int(request.query_params.get('x')), int(request.query_params.get('y'))
mod.apply_async(args=(x, y), countdown=20)
return HttpResponse(f'Success !')
djangocelery > urls.py
from django.urls import path
from djangocelery import views
urlpatterns = [
path('mod', views.get_test_add)
]
betweak > urls.py (project urls)
urlpatterns = [
path('celery/', include('djangocelery.urls')),
path('admin/', admin.site.urls),
]
Ok so settings are ready. Let's test it out!
Run celery
celery -A djangocelery worker loglevel-info
Run django (in another terminal)
python manage.py runserver
Let's check celery log to see what's happening.
We can see that at 22:32:33 the task is received. Approximately 20 seconds later at 22:32:54, the task succeeded.
If we see djangocelery > views.py again
ffrom djangocelery.tasks import mod
@api_view(['GET'])
def get_test_add(request):
# so we get the x and y values from the url
x, y = int(request.query_params.get('x')), int(request.query_params.get('y'))
# Here countdown=20 means executes in 20 seconds from now
mod.apply_async(args=(x, y), countdown=20)
return HttpResponse(f'Success ! ')
```
Well that was easy?
Let's try to make an error. We will pass 0 value for y. Python raises when we try to use 0 for modulo.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/jrsp78p46hagpqt0rh26.png)
Let's check the log out
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/0dekbl3tza3atqtta6u8.png)
There's a lot going on there. We see there was a ZeroDivisionError, and the task retried 5 times!
djangocelery > task.py
```
# Here are the settings for retries!!
@app.task(name="mod", bind=True, default_retry_delay=10, max_retries=5)
def mod(self, x, y):
try:
z = x % y
print(f'{x} % {y} = {x%y}')
return z
except :
mod.retry()
print(f'Error with add')
```
So that's it for getting started with celery in django :)
Top comments (1)
i just got "[2021-12-28 11:40:50,478: INFO/MainProcess] Task mod[ee4acdff-062b-4871-a7ca-92f66681b370] received"