DEV Community: Daeen Choi

Database Scheduler with celery beat and manage schedule in Django admin

Daeen Choi — Sat, 20 Feb 2021 14:29:03 +0000

Must read previous post first before reading this one :D
Previous Post about celery

Aim of this post

Install celery beat in django project and understand how it is used
Settings for sending emails in django project using gmail
Create a task that sends email
Create schedule using django admin and integrate it with email task Eg) Send notification email to users every Fridays at 4 a.m.

What is celery beat?

celery beat is a scheduler. When it's time to run the task, it delivers the entry to the worker node. (Periodic task execution)

Install celery beat
pip install django-celery-beat

Migrate - Tables will be created to store tasks and schedules
python manage.py migrate

settings.py



INSTALLED_APPS = (
    ...,
    'django_celery_beat',
)

Settings for emails

1 . Click Manage your Google Account

2 . Make sure 2 step verification is on in Security > 2 step verification

3 . Create app password. App passwords > select app > other > Name it whatever!

4 . Add settings.py



# e-mail settings
EMAIL_USE_TLS = True
EMAIL_PORT = 587
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_HOST_USER = {YOUR EMAIL ADDRESS}
EMAIL_HOST_PASSWORD = {YOUR PASSWORD FROM NO.3}
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

Email setting is finished!

Where does celery beat store tasks and schedules?

1. Default entries are taken from the `beat_schedule` setting

task.py



app.conf.beat_schedule = {
  'add-every-30-seconds': {
        'task': 'tasks.add',
        'schedule': 30.0,
        'args': (16, 16)
    },
}
app.conf.timezone = 'UTC'

Or if you want more control over your schedules use crontab



from celery.schedules import crontab

app.conf.beat_schedule = {
    # Executes every Monday morning at 7:30 a.m.
    'add-every-monday-morning': {
        'task': 'tasks.add',
        'schedule': crontab(hour=7, minute=30, day_of_week=1),
        'args': (16, 16),
    },
}

Start celery beat service
celery -A <project name> beat

2 . Use custom stores - I will store them in database

Let's create a task that sends email to users ```

@app.task(name="send_notification", bind=True, default_retry_delay=300, max_retries=5)
def send_notification(self, subject, message):
from django.core.mail import send_mail as sm

# Fetch all users except superuser
users = User.objects.exclude(is_superuser=True).all()
user_emails = [user.email for user in users]

# try sending email
try:
    res = sm(
        subject=subject,
        html_message=message,
        from_email=EMAIL_HOST_USER,
        recipient_list=user_emails,
        fail_silently=False,
        message=None)
    print(f'Email send to {len(user_emails)} users')
except Exception:

    # retry when fail
    send_notification.retry()

So here I created a task called `send_notification`.
`default_retry_delay` : retry after 300 seconds when fail
`max_retries` : only retry 5 more times then STOP.
Here I am also getting `subject` and `message` as arguments. This means I can pass `subject` and `message` from django admin. I will show you this later.


2. Let's go to django admin
`localhost:8080/admin`.  

**You will see new `Periodic Tasks` Menu**
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3dd9pno3z8miartnckan.png)


**Let's make a `crontab` Crontab > Add crontab**
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p9plewa00ay3yf9tz2so.png)



**Let's create a crontab .Here I created one.  `Every Saturday @ 10:47 p.m.`**
 If you are not sure about crontab, check this out [Crontab examples] (https://crontab.guru/examples.html)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bojci16hsrh89fnko09z.png)

**Let's create a periodic task**
Click `Periodic tasks` > `Add`. Create a new name of your choice. Find `send_notification` task that we have created up there
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r7mppp6d2owprmlrarnk.png)


`Schedule` > `Crontab Schedule` > Find the `crontab` you just created.
Add the `Start Datetime`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zpac15g3tsefqgvh8csp.png)


This is where you pass the arguments `subject` and `message`
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gy2gy4z6m5gb3r8tv4t2.png)



So we are all set and ready to go.

Start django server, start rabbitmq

Start celery
`celery -A djangocelery(app name) worker --loglevel=info`

Start celery beat in different terminal
`celery -A djangocelery(app name) beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler`


Let's check out the terminal of celery beat

@10:47 p.m. celery beat scheduler is sending `send_notification` task that we have created!! to celery!!
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h47zq65i4rujz2wx5or6.png)



Check celery terminal. Celery has received the task and email is sent.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mm193y1n7beyaaixsoud.png)


And I have received the email at 10:47 p.m. on Saturday with the title and content that I have set up in django admin!
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mqip5axzexn9e157q3xt.png)



###Conclusion
So `celery beat` is a `scheduler` and `celery` is the one that executes the `task`. `Database scheduler` is one of the method of celery beat where tasks and schedules are stored in database which means you can manage them in `django admin`.

<br/>
That's it for django celery beat!
The end.

Understanding celery and creating simple task in django

Daeen Choi — Sat, 30 Jan 2021 14:55:11 +0000

Aim of this post

Understand what is celery
Create a simple task in Django project (celery)

What is celery?

Celery is an asynchronous processing and scheduling tool in Django (not limited to Django).

So what is asynchronous?
When a client sends a request to the server, normally client cannot do anything until it receives a response from the server. That said, when request is time-consuming work, I mean more than 1 minute to handle,
user will start clicking the refresh button couple of times or click on the back button.

To prevent this situation, we process APIs asynchronously. Asynchronous processing allows clients to send requests and do other tasks immediately. When the response returns, the processing of the response continues.

This asynchronous processing is done by celery.

What about scheduling?
You can make a task which runs at 8 o'clock every morning. Or you might also want to send e-mails to users every Friday at 4:00 p.m. This is possible with celery-beat.

How does celery work?

The unit of work that Celery has to deal with is called a task. Producer (celery) creates a task and the task is added to the Task Queue before it is executed. The broker then forwards the task in the task queue to the appropriate worker node which is the consumer (celery) where the task is executed. Celery can be seen as a larger concept, including producer, broker, and consumer.

Let's create a simple task!

Prepare a django project (any project is fine - you can just use https://docs.djangoproject.com/en/3.1/intro/tutorial01/ if you don't know where to start)
Install and run rabbitmq. (I used homebrew)
Try https://www.rabbitmq.com/install-homebrew.html if you don't have rabbitmq yet.
brew services start rabbitmq to start up rabbitmq
Install celery in virtual environment
pip install celery
create new celery app
python manage.py startapp djangocelery <- you can name your app anything. I called mine djangocelery
Settings and more
5-1. create celery.py djangocelery app(directory) and add following code

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'betweak.settings') 
app = Celery('djangocelery') 

# You can add celery settings in settings.py starting with CELERY_ 
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(lambda : settings.INSTALLED_APPS)

5-2. Edit djangocelery > init.py file

from .celery import app as celery_app
# celery app is automatically imported when django starts
__all__ = ['celery_app']

5-3. Edit settings.py

INSTALLED_APPS = [

   ...
   'celery'
   'djangocelery' # newly created app
]

# set CELERY settings with CELERY_{}
CELERY_BROKER_URL = amqp://guest:guest@localhost:5672/ # rabbitmq setting
CELERY_TIMEZONE = 'Asia/Seoul'
CELERY_ENABLE_UTC=False
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']

6 . Migrate database
python mange.py migrate

7 . Create task

Create task.py file in djangocelery and add following code

from djangocelery.celery import app

@app.task(name="mod", bind=True, default_retry_delay=10, max_retries=5)
def mod(self, x, y):
    try:
        z = x % y
        print(f'{x} % {y} = {x%y}')
        return z
    except :
        mod.retry()
        print(f'Error with mod')

8 . Add views so we can access task with API

djangocelery > views.py

from django.http import HttpResponse
from rest_framework.decorators import api_view
from djangocelery.tasks import mod

@api_view(['GET'])
def get_test_add(request):
    x, y = int(request.query_params.get('x')), int(request.query_params.get('y'))
    mod.apply_async(args=(x, y), countdown=20)
    return HttpResponse(f'Success !')

djangocelery > urls.py

from django.urls import path

from djangocelery import views

urlpatterns = [
    path('mod', views.get_test_add)
]

betweak > urls.py (project urls)

urlpatterns = [
    path('celery/', include('djangocelery.urls')),
    path('admin/', admin.site.urls),
]

Ok so settings are ready. Let's test it out!

Run celery

celery -A djangocelery worker loglevel-info

Run django (in another terminal)

python manage.py runserver

Let's check celery log to see what's happening.

We can see that at 22:32:33 the task is received. Approximately 20 seconds later at 22:32:54, the task succeeded.

If we see djangocelery > views.py again

ffrom djangocelery.tasks import mod
@api_view(['GET'])
def get_test_add(request):

    # so we get the x and y values from the url
    x, y = int(request.query_params.get('x')), int(request.query_params.get('y'))

    # Here countdown=20 means executes in 20 seconds from now 
    mod.apply_async(args=(x, y), countdown=20)
    return HttpResponse(f'Success ! ')

```



Well that was easy?

Let's try to make an error. We will pass 0 value for y. Python raises when we try to use 0 for modulo.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/jrsp78p46hagpqt0rh26.png)


Let's check the log out
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/0dekbl3tza3atqtta6u8.png)

There's a lot going on there. We see there was a ZeroDivisionError, and the task retried 5 times!

djangocelery > task.py


```
# Here are the settings for retries!!
@app.task(name="mod", bind=True, default_retry_delay=10, max_retries=5)
def mod(self, x, y):
    try:
        z = x % y
        print(f'{x} % {y} = {x%y}')
        return z
    except :
        mod.retry()
        print(f'Error with add')

```




So that's it for getting started with celery in django :)

DEV Community: Daeen Choi

Database Scheduler with celery beat and manage schedule in Django admin

Aim of this post

What is celery beat?

Settings for emails

Where does celery beat store tasks and schedules?

1. Default entries are taken from the beat_schedule setting

2 . Use custom stores - I will store them in database

Understanding celery and creating simple task in django

Aim of this post

What is celery?

Let's create a simple task!

Run celery

Run django (in another terminal)

1. Default entries are taken from the `beat_schedule` setting