DEV Community

Cover image for Self-Scheduling Recurring Cloud Tasks (with Terraform + Python code)
Charlotte Towell
Charlotte Towell

Posted on

Self-Scheduling Recurring Cloud Tasks (with Terraform + Python code)

#googlecloud #terraform #python #serverless

Using a fully serverless cloud set-up, sometimes the easy questions like: "how can I run this function on a recurring basis?" are not so easy to answer. CRON job? nope.

Instead, this is how I achieve it utilising scheduled Cloud Tasks for a fully serverless approach.

Let's say we have a Cloud Run Function we want to call on a daily, or perhaps weekly, basis. The process we'll follow is:

  • Set up a Cloud Tasks queue
  • Set up the function to reschedule itself
  • Create our initial task to set off the chain

Service Account

First, we create the service account used to run our cloud function

resource "google_service_account" "<my_sa>" {
  account_id   = "<my_sa>"
  display_name = "My Service Account"
  description  = "Service account used for my function"
  project      = var.project_id
}
Enter fullscreen mode Exit fullscreen mode

Then we add all required roles, specifically those for Cloud Tasks & invoking Cloud Run

resource "google_project_iam_member" "<my_sa>" {
  for_each = toset([
    "roles/run.invoker",
    "roles/cloudtasks.enqueuer"
  ])

  project = var.project_id
  role    = each.value
  member  = "serviceAccount:${google_service_account.<my_sa>.email}"

  depends_on = [google_service_account.<my_sa>]
}
Enter fullscreen mode Exit fullscreen mode

and finally, we also need to allow the service account to impersonate itself for queuing the next cloud task:

resource "google_service_account_iam_member" "<my_sa_self_impersonation>" {
  service_account_id = google_service_account.<my_sa>.name
  role               = "roles/iam.serviceAccountUser"
  member             = "serviceAccount:${google_service_account.<my_sa>.email}"
}
Enter fullscreen mode Exit fullscreen mode

Cloud Task Queue

Next, create a cloud task queue.

resource "google_cloud_tasks_queue" "<my_task_queue_nane>" {
  name     = "<my_task_queue_name>"
  location = var.region

  http_target {
    http_method = "POST"
    oidc_token {
      service_account_email = <my_sa_email>
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Queue Task from Function

Then, we can add a final bit of code to the end of our function to allow scheduling the next task. This is where we will set any rules like how long to schedule (1 day ahead, 1 week, etc), and what time of day to run at for example. Any shared data to be passed along must be passed as the HTTP body, and the function entry set up to parse this accordingly.


def cloud_task_scheduler(days_increment: int = 1):
    try:
        client = tasks_v2.CloudTasksClient()
        parent = client.queue_path(GCP_PROJECT_ID, LOCATION, CLOUD_TASK_QUEUE_ID)
        FUNCTION_URL="<my_cloud_function_url>"

        # define next task
        payload = {
            "schedule_next_cloud_task": True
        }
        body_bytes = json.dumps(payload).encode("utf-8")

        scheduled_date = (datetime.now(timezone.utc) + timedelta(days=days_increment)).replace(
            hour=16, minute=0, second=0, microsecond=0
        )
        ts = timestamp_pb2.Timestamp()
        ts.FromDatetime(scheduled_date)

        task: tasks_v2.CreateTaskRequest = {
            "http_request": {
                "http_method": tasks_v2.HttpMethod.POST,
                "url": f"{FUNCTION_URL}",
                "headers": {"Content-Type": "application/json"},
                "body": body_bytes
            },
            "schedule_time": ts
        }

        # add task to queue
        client.create_task(parent=parent, task=task)

    except Exception as e:
        raise Exception(f"Error scheduling next Cloud Task: {e}")
Enter fullscreen mode Exit fullscreen mode

Schedule the First Task

To set the chain off running, we must schedule the first cloud task. Simplest way is via the command line with Google Cloud CLI.

gcloud tasks create-http-task my-first-task \
  --queue=my-queue \
  --url="my-function-url" \
  --method=POST \
  --body-content='{"schedule_next_cloud_task": true}' \
  --header="Content-Type":"application/json" \
  --oidc-service-account-email="my-sa-email" \
  --location="region" \
  --schedule-time="YYYY-MM-DDTHH:MM:SS+00:00"
Enter fullscreen mode Exit fullscreen mode

And just like you should see your first task sitting in the queue waiting to run! Of course, I reccomend testing first to ensure that your function can run fully and then create irs following task successfully.

This approach can be used for a daily schedule like I've outlined above or even to split up long-running tasks that may exceed timeouts by either passing shared data through the body of the task request or writing to a database

The benefit is because the cost of Cloud Tasks is basically nothing, or literally free if you are below certain usage tiers, then this method is extremely efficient as we only pay for compute while our function is running and it can effectively scale to zero while waiting for the next cloud task to run.

Hopefully this helps anyone else after a simple approach of implementing a self-sustainaing schedule with serverless compute efficiently!

Top comments (0)