Cloud Run Jobs vs. Cloud Functions for Batch Security Scans: What I Actually Run in Production

I run periodic security scans as part of my incident-readiness platform: dependency checks, config drift detection, TLS posture checks against customer-facing assets. All of it is batch work on a schedule, not request-driven. When I first designed the pipeline, the obvious GCP answer was Cloud Functions — trigger on a Pub/Sub message from Cloud Scheduler, do the scan, write results to a database, done. It worked. It was also the wrong choice, and it took me a month of fighting timeouts and half-finished scans to figure out why.

This post is the decision framework I wish I had: when to use Cloud Functions and when to use Cloud Run Jobs for security batch work, with the actual code and configs that run in production today.

The workload

Let me be specific about what the scans look like, because the choice depends entirely on workload shape:

• Dependency scan: pull a lockfile manifest from a repo, compare pinned versions against OSV/known advisory feeds, emit findings. Runtime: 2–8 minutes.
• Config drift check: pull the last-known-good config from Cloud Storage, diff against live config fetched via API, emit drift findings. Runtime: 1–3 minutes.
• TLS posture sweep: handshake against a list of hostnames, record certificate expiry and protocol versions. Runtime: 5–15 minutes depending on list size.

None of these are fast enough to ignore timeout limits. That is the entire story, but the details matter.

Why Cloud Functions stopped fitting

Cloud Functions (2nd gen runs on Cloud Run infrastructure, so the underlying tech is the same) has a hard ceiling: 60-minute max timeout for HTTP functions, 9 minutes for event-driven (1st gen). My TLS sweep occasionally crossed 15 minutes for larger host lists, and that was fine — but the dependency scan had a nasty failure mode. When a container registry was slow, image pulls inside the function stalled, and I could not control the timeout granularity per stage. More importantly, Cloud Functions gives you a request lifecycle: your work is tied to a request context, and long-running work fights that model. You end up writing continuation logic — publish a Pub/Sub message to re-trigger yourself, persist partial state, hope the chain completes. It works until it does not, usually at 3 AM.

The other issue: concurrency semantics. A Cloud Function instance handles requests with configured concurrency, which is great for webhooks and terrible for scans. I do not want two scans of the same asset running concurrently because Scheduler fired twice. With Cloud Functions I had to build idempotency locks myself (a row in Cloud SQL with a lease). With Jobs, the execution model handles this natively.

What Cloud Run Jobs gives you instead

A Cloud Run Job is a container that runs to completion. You define the task count and parallelism, Scheduler or Pub/Sub triggers an execution, and the job runs until it exits. The timeout ceiling is 24 hours. You get:

• One container, one task. No request context, no concurrency confusion. The container starts, does the work, exits with a code.
• Parallelism control. A TLS sweep over 200 hosts splits into N task replicas, each handling a shard. You get horizontal fan-out without writing a fan-out layer.
• Retries that make sense. Failed executions retry the task, not a chained request. You can set max retries per execution.
• Direct Cloud Scheduler targeting. Scheduler invokes the job directly via the Cloud Run Admin API — no intermediate HTTP endpoint, no Pub/Sub hop required (though Pub/Sub-triggered execution is supported).

Here is the actual pattern I run. One container image per scan type, defined as a Job, triggered by Cloud Scheduler:

# cloud-run-job.yaml
apiVersion: run.googleapis.com/v1
kind: Job
metadata:
  name: tls-posture-scan
  namespace: "project-number"
spec:
  template:
    spec:
      template:
        spec:
          containers:
            - image: gcr.io/my-project/tls-scan:v1.4.2
              resources:
                limits:
                  cpu: "2"
                  memory: 2Gi
              env:
                - name: HOST_LIST_GCS
                  value: "gs://scan-configs/hosts.txt"
                - name: DB_CONNECTION_NAME
                  valueFrom:
                    secretKeyRef:
                      name: db-connection
                      key: connection-string
          taskCount: 8
          parallelism: 4
          maxRetries: 2
          timeoutSeconds: 3600

Deploy and schedule:

gcloud run jobs create tls-posture-scan \
  --image gcr.io/my-project/tls-scan:v1.4.2 \
  --tasks 8 --max-retries 2 --task-timeout 3600 \
  --region us-east1

gcloud scheduler jobs create http nightly-tls-scan \
  --schedule "0 3 * * *" \
  --uri "https://us-east1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/PROJECT_NUMBER/jobs/tls-posture-scan:run" \
  --http-method POST \
  --oauth-service-account-email [e-mail] \
  --time-zone "America/New_York"

The Scheduler job invokes the Run Admin API directly. No public HTTP endpoint on the scan container, no Pub/Sub topic to manage. The service account scheduler-runner needs exactly one role: roles/run.developer on the job. That is the entire IAM surface.

Inside the container: sharding work across tasks

Cloud Run Jobs exposes CLOUD_RUN_TASK_INDEX and CLOUD_RUN_TASK_COUNT as environment variables. Your code shards on those:

import os

task_index = int(os.environ["CLOUD_RUN_TASK_INDEX"])
task_count = int(os.environ["CLOUD_RUN_TASK_COUNT"])

def main():
    hosts = load_hosts_from_gcs()          # full list, every task reads it
    shard = [h for i, h in enumerate(hosts) if i % task_count == task_index]
    results = [scan_host(h) for h in shard]
    write_findings(results)                # idempotent upsert keyed on (host, check, date)

The upsert must be idempotent keyed on natural keys. Retries re-run the shard, and a retried shard must not double-count findings. That is the one piece of discipline the model requires of you.

What I still use Cloud Functions for

This is not a "Jobs won" post. Cloud Functions still handles everything event-driven in my stack:

• A Pub/Sub message arrives when a new container image is pushed → a Function runs the image-layer vulnerability check. Runtime: under 2 minutes. This is webhook-shaped work and Functions fits it perfectly.
• A Security Command Center finding lands → a Function enriches it and writes a Slack-ready summary. Seconds of runtime, triggered by an event, no scheduling involved.

The rule I now apply: if the work starts because something happened, use Functions. If the work starts because it is time, use Jobs.

The trade-offs, honestly

Cloud Run Jobs is not free of friction:

• Cold starts on schedules. A job that runs nightly cold-starts every night. My scan containers pull in 20–40 seconds. I accept it because the scan itself takes minutes. If your batch work takes 45 seconds, a cold-started job is a bad trade — use a Function or keep a service warm.
• No built-in progress visibility. A job execution is either running, succeeded, or failed. There is no built-in "47% done" channel. I emit structured logs per shard and watch them in Cloud Logging. Fine, but it is manual.
• Logging is your debugger. Failed executions give you exit codes and logs, nothing more. Structured JSON logging from the container is not optional — it is the only observability you get without building more.

On cost: both are pay-per-use. Jobs bill for the vCPU/memory seconds of the execution. My nightly scans cost single-digit dollars a month at this scale. The billing model did not decide anything for me; the timeout and execution model did.

The migration checklist

If you are moving batch work off Cloud Functions onto Jobs, the concrete steps:

1. Containerize the scan. You probably already have a Dockerfile if you are on 2nd-gen Functions — extract the logic into a main() with a clean exit code contract.
2. Replace request handlers with task-index sharding.
3. Make result writes idempotent (natural-key upserts) — retries will re-run shards.
4. Deploy as a Job, trigger from Cloud Scheduler via the Run Admin API, one service account with roles/run.developer.
5. Emit structured logs per shard. You will need them.

The takeaway

Cloud Functions and Cloud Run Jobs solve different halves of the automation problem. Functions owns the event-driven half — webhooks, Pub/Sub, triggers. Jobs owns the scheduled batch half — scans, sweeps, reports. The mistake I made was treating them as interchangeable because they share infrastructure. They are not. Pick by trigger shape and runtime, and your security automation stops fighting the platform.

Comments

[Devyani Nandanwar]

Wow what an interesting read! definitely something to think about.