Most "automate this with cron" walkthroughs leave out the parts where things actually break. The PATH is wrong, the virtual environment is not activated, the logs go nowhere, and the script that ran perfectly in the terminal silently fails at 2:00 AM. This guide is the version you wish you had read the first time.
The steps assume a Linux host (or WSL on Windows). The same shape applies to macOS with one small note at the end.
Photo by Pixabay on Pexels
Step 1: write the script so it runs cleanly outside its directory
The most common silent failure is a script that uses relative paths. It works when you run it from its own folder and breaks the moment cron runs it from /.
Fix this once and forever by writing your script as if it could be called from anywhere. Two changes are enough.
#!/usr/bin/env python3
import os
import sys
from pathlib import Path
HERE = Path(__file__).resolve().parent
LOG_DIR = HERE / "logs"
LOG_DIR.mkdir(exist_ok=True)
def main():
# do the work
print("hello from", HERE)
if __name__ == "__main__":
main()
The shebang line lets you call the script directly without typing python3 script.py. The HERE constant gives every file path in the script an absolute anchor, so the script does not care what the current working directory is when cron invokes it.
Make the script executable with chmod +x script.py. Confirm it runs by calling it with its absolute path from your home directory: /full/path/to/script.py. If it works there, it will work under cron.
Step 2: pin the Python interpreter
If your script depends on packages installed in a virtual environment, the shebang line above is not enough. cron will use the system Python, not your venv's Python, and the imports will fail.
You have two clean options. The first is to point the shebang directly at the venv's interpreter:
#!/full/path/to/venv/bin/python3
The second is to wrap the script in a tiny shell script that activates the venv before running Python:
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")"
source venv/bin/activate
exec ./script.py "$@"
The wrapper approach scales better if you have multiple Python scripts that share a venv. It also makes it easy to add environment variables, source a profile, or set umask before the Python process starts.
For projects that use poetry or pipenv, the wrapper becomes exec poetry run python script.py or exec pipenv run python script.py. The Python documentation lists the tooling options if you want to compare them.
Step 3: write the cron line
Open your crontab with crontab -e. You will get an editor (usually vi or whatever $EDITOR is set to) with whatever lines are already in your crontab.
Add a line with the schedule, the command, and an output redirect. For a daily 9 AM job:
0 9 * * * /full/path/to/script.py >> /var/log/myjob.log 2>&1
The five fields before the command are the schedule. The redirect at the end captures both stdout and stderr to a log file you control. Without that redirect, your output goes to the local mail spool (and almost certainly nowhere).
Before you save, test the expression. The five fields are minute, hour, day-of-month, month, day-of-week, in that order. Hours are 24-hour. Day-of-week zero is Sunday. If you are not 100% sure your expression matches the schedule you described in plain English, paste it into the Cron Expression Builder and confirm the next ten execution times. The longer how to read cron expressions step by step guide covers the field-by-field syntax if you want to go deeper before committing.
Photo by Walls.io on Pexels
Step 4: handle the cron environment, not the login environment
This is the step everybody skips and everybody pays for. cron does not inherit your interactive shell's environment. It has a minimal PATH, often just /usr/bin:/bin. It does not source your .bashrc. It does not know about Homebrew, pyenv shims, or anything you set up in your terminal startup files.
Two habits keep this from biting you.
Use absolute paths everywhere in the command. Not python3 but /usr/bin/python3. Not aws but /usr/local/bin/aws. The few extra characters at write time save you an hour of debugging the first time something fails because cron could not find the binary.
Set the environment explicitly at the top of the crontab if you need it. You can add PATH=/usr/local/bin:/usr/bin:/bin as a line before any cron schedules and it applies to every line after. You can also add SHELL=/bin/bash if your default crontab shell is /bin/sh and you wrote your wrapper as bash.
For environment variables your script needs (API keys, database URLs), load them inside the script from a file you trust, not from the shell. Putting secrets in a crontab is a recipe for accidental exposure.
"Every time we have debugged a 'cron is broken' ticket, the bug was in the environment, not the schedule. PATH, working directory, missing env vars. The cron daemon is doing exactly what it advertised." - Dennis Traina, founder of 137Foundry
Step 5: confirm the job actually ran
After you save the crontab, wait for the next scheduled time and check.
tail -f /var/log/myjob.log will show you the script's output as it runs, assuming you put the redirect in place. If the log is empty after the scheduled time has passed, the cron daemon did not run the command. Check /var/log/syslog (Debian/Ubuntu) or journalctl -u cron (systemd hosts) for the daemon's view of the world. It will tell you whether the job fired and whether it exited with a non-zero status.
If the log has output but the script did not do what you expected, the failure is in the script, not the schedule. Reproduce it interactively with the same command line cron used and you will see the error.
For long-running production jobs, do not rely on tailing the log. Add a heartbeat: at the end of the script, touch a file or hit a monitoring endpoint (a service like Healthchecks.io will page you if the heartbeat does not arrive on schedule). This converts "I hope it ran" into "I know it ran or I have been told it did not."
Step 6: harden for edge cases
A few small habits separate a flaky cron job from one you can trust for years.
Make the script idempotent. If the same job runs twice (which can happen during a daylight-saving transition, or if a clever operator runs it manually after it has already fired), the second run should be safe. Record a last-run timestamp and exit early if you have already done the work.
Add a lock. If the job takes longer than its schedule (a job scheduled every five minutes that occasionally takes seven), you can end up with overlapping runs. The flock utility solves this in one line:
*/5 * * * * /usr/bin/flock -n /var/lock/myjob.lock /full/path/to/script.py >> /var/log/myjob.log 2>&1
The -n means "fail fast if the lock is held," which is usually the safe default. The GNU coreutils documentation has more on the locking primitives if you need finer control.
Rotate the log. A daily job that writes a few hundred bytes per run will fill a small disk in a year. logrotate is the standard answer; a 30-day rotation is fine for most jobs.
On macOS, one small note
macOS still has cron, and the steps above work, but Apple's preferred scheduler is launchd. For personal automation on your laptop, the cron route is faster to set up. For anything you will run on a Mac you actually care about (a CI runner, a Mac mini doing builds), launchd's plist files give you better integration with login sessions and energy management. The Linux man pages for crontab cover the syntax that macOS cron mostly shares.
What good looks like
A production-grade scheduled Python job has all of the following in place: absolute paths everywhere, an explicit interpreter (venv-aware), output redirected to a rotated log file, a heartbeat to an external monitor, idempotency in the script body, and a lock to prevent overlapping runs. None of those individually is hard. Together they convert a scheduled script from a recurring source of late-night incidents into a thing you stop thinking about.
The schedule itself is the easy part. The discipline around the schedule is what makes the difference, and an extra ten minutes per job at setup pays off the first time the host reboots in the middle of the night.
Top comments (0)