Cron Does Not Care About Environment

Recently I have faced a case while deploying a laravel application with scheduler on kubernetes. The jobs in scheduler needed to access the database but the logs were indicating that the scheduler jobs running with cronjob could not connect to the database. The following error log was showing up.

“production.ERROR: SQLSTATE[HY000] [2002] Connection refused”

But the application was perfectly running without any problem to connect to the database. The debugging process unmasked that cron was not getting the environment variables i.e. the db credentials for the application.

Kubernetes Cronjob Resource

A solution is to use Cronjob resource natively supported by Kubernetes. As described by the official documentation of Kubernetes,

CronJob is meant for performing regular scheduled actions such as backups, report generation, and so on. One CronJob object is like one line of a crontab (cron table) file on a Unix system. It runs a job periodically on a given schedule, written in Cron format.

So, we can create a cronjob resource named cron-test which will run the job every minute.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cron-test
spec:
  schedule: "* * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cron-test
            image: busybox:1.28
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - echo $(date) >> /tmp/cron-test.txt
          restartPolicy: OnFailure

This Cronjob will execute the job every minute like this.

But in our case, we need application context. The application is heavy and we cannot guarantee a graceful execution of the job. So, we decided to create a deployment which will execute the jobs with help of cronjob.

Deployment with cronjob

To test this, we will first create a configmap named env which contains our environment variables.

kubectl create cm env --from-literal=NAME=dip --from-literal=AGE=25 -n env-cron-tests

Then, we will create a configmap for cronjobs and a deployment to run the cron.

apiVersion: v1
kind: ConfigMap
metadata:
  name: scheduler-config
data:
  root: |
    * * * * * env > /tmp/env.txt
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: scheduler
  name: scheduler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: scheduler
  template:
    metadata:
      labels:
        app: scheduler
    spec:
      containers:
      - envFrom:
        - configMapRef:
            name: env
        image: debian
        imagePullPolicy: IfNotPresent
        name: scheduler
        command: ["/bin/sh","-c"]
        args: ["apt-get update && apt-get install -y cron && crontab /tmp/cron/* && cron -f"]
        volumeMounts:
        - mountPath: /tmp/cron
          name: cronjob
      volumes:
      - name: cronjob
        configMap:
          name: scheduler-config

cron is usually run as a daemon so on startup cron forks, which means it creates a running copy of itself and then terminates the original process.

Which is normally what you want, but when running inside a container docker will stop your container when your main process exits. to instruct cron to start in foreground mode, we have used cron -f command in the entrypoint. Notice the environment of the deployment comes from our previously configured configmap env.

You can see that in the pod, if I get the environment variables, I can get the variables NAME and AGE. But in /tmp/env.txt, where cron is writing the env variables, there the env variables NAME and AGE are not available. Cron is deliberately ignoring the environment here. So, we came to a solution that if we get the environment vairables in /etc/environment, which is a system-wide configuration file, then cron will use our desired environment. The command will be changed like this.

command: ["/bin/sh","-c"]
args: ["apt-get update && apt-get install -y cron && env >> /etc/environment && crontab /tmp/cron/* && cron -f"]

Now cron gets our desired env variables NAME and AGE. Therefore, they are in /tmp/env.txt file. It can be sometimes risky to expose the env variables in a system-wide configuration file. We can also choose specific environment variables or grep a set of environment variables for /etc/environment.