Hey folks! So, I had a task of scheduling a Dashboard execution daily at 12.00 pm and 4.00 pm and was supposed to use DigDag Job Scheduler. But, after spending quite some time on DigDag’s documentation, I couldn’t find anything relevant as per my requirement. Hence, I decided to try a workaround.
"Instead of scheduling the Dashboard using a single workflow (the *.dig file), I can import/include the workflow into another workflow(s) and schedule it as per my need."
For example:
Note: The below example is demonstrated on Ubuntu 18.04 (Linux). Please refer DigDag’s documentation for other platform-specific commands.
Lets generate a sample digdag project schedule-demo using command:
$ digdag init schedule-demo
The above command will create a workflow schedule-demo.dig in schedule-demo directory/project with contents:
timezone: UTC +setup: echo>: start ${session_time} +disp_current_date: echo>: ${moment(session_time).utc().format(‘YYYY-MM-DD HH:mm:ss Z’)} +repeat: for_each>: order: [first, second, third] animal: [dog, cat] _do: echo>: ${order} ${animal} _parallel: true +teardown: echo>: finish ${session_time}
Let's schedule the above workflow for every one and two minutes on same day using file one_minute.dig and two_minutes.dig workflow.
Create workflows one_minute.dig and two_minutes.dig in the project schedule-demo with below-mentioned contents:
- Workflow one_minute.dig:
!include : ‘schedule-demo.dig’ schedule: minutes_interval>: 1
- Workflow two_minute.dig:
!include : ‘schedule-demo.dig’ schedule: minutes_interval>: 2
Now change your current working directory to schedule-demo using the command:
$ cd {current_path}/schedule-demo/
then push the project schedule-demo to digdag server using the command:
$ digdag push schedule-demo
(Note: On local Ubuntu setup, to start digdag server execute command ‘digdag server’ on the terminal)
If you are outside the project directory and want to push project use command:
$ digdag push schedule-demo --project /{path}/schedule-demo --revision schedule-demo-v1
{path} => path to the schedule-demo project
Then, launch a browser (any) and add URL ‘http://localhost:65432/’ and under Workflow tab you will see the result similar to below image (you will have to wait for at least 5–6 minutes to get similar output as of mine):
If you observe the image, the output with ID 2 and ID 5 has 2 minutes difference (refer column 'Last Attempt' ) and the value of Workflow is two-minute. You will find similar result for the one-minute workflow (i.e. the execution time difference between every present and next workflow attempt will be 1 minute).
P.S. There could be different ways to achieve the result demonstrated in the above example. This is just one of those :)
If you enjoyed this blog, share it with your friends!
Thank-You for reading!
Sources:
https://www.digdag.io/
https://docs.digdag.io/getting_started.html
https://docs.digdag.io/workflow_definition.html?highlight=include#include-another-file
Top comments (2)
Maybe, you can use
cron>
syntaxScheduling workflow
Yeah, that works too.