Burned by PEP 668 on Ubuntu 24.04 — How I Fixed the Monthly Automation Script

TL;DR

Ubuntu 24.04 blocks pip install with the externally-managed-environment error. Using a virtual environment is the correct fix — but in my case, the real root cause was an environment mismatch: the check phase ran on a different server than the execution phase. I solved it with an SSH pivot.

---

Background

I have a month-end automation script that handles timesheet entry, expense submission, and invoice sending. It uses Playwright + openpyxl + Google APIs to auto-login to multiple vendor systems and fill out forms.

This script is triggered by a cron job on joe (192.168.x.x, Ubuntu 24.04).

---

What Happened

Around 5 PM on March 25th, the month-end automation failed.

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, either use apt or
    use a virtual environment.

pip install playwright was blocked. PEP 668.

PEP 668 is a specification introduced in Python 3.11+ that prohibits installing packages directly into the system Python environment. Ubuntu 24.04 enforces this strictly. You can force it with --break-system-packages, but that's not something I wanted to do.

---

The Real Problem: Environment Mismatch

Here's what I realized.

Phase 0 (pre-flight checks) ran on infra (Debian-based server). infra had all the required libraries: Playwright, openpyxl, Google API client.

The actual execution script was triggered from joe (Ubuntu 24.04). joe did not have these libraries in its system Python.

So Phase 0 checks passing on infra gave a false sense of security — "all checks passed!" — when the actual runtime environment on joe was completely different. The check environment and execution environment were mismatched, and that was the root cause.

---

Solution: SSH Pivot

I had a few options:

1. Create a venv on joe and install all dependencies → The correct approach, but infra also needed Chrome/Xvfb configuration that's heavy to replicate on joe
2. Dockerize everything → Overkill for this use case
3. Route Python invocations via SSH to infra → Leverages existing setup

I went with option 3. I modified monthly-endofmonth.sh to replace direct Python calls with SSH-based execution:

# Before
python3 /mnt/shared/scripts/make_timesheet.py

# After
ssh -o StrictHostKeyChecking=no linou@192.168.x.x \
  "python3 /mnt/shared/scripts/make_timesheet.py"

SSH keys were already configured for passwordless login, so cron runs work unattended. /mnt/shared/ is NFS-mounted with the same path on all nodes, so no path rewrites were needed.

---

Results

The 6 PM cron re-run confirmed:

Task	Result
HajimariWORKS time entry	✅ All 21 days saved
Pasture NOB DATA Next PJ	✅ All days entered
Pasture Optage	✅ Entered (manual submission check needed)
Timesheet xlsx generation	❌ SameFileError (leftover from 5 PM run)

The SameFileError was because a file generated at 5 PM was still there — a logic bug. Since this runs once a month, I'll fix it before the next run.

---

Lesson Learned

A pre-flight check is only meaningful when the check environment matches the execution environment.

"Phase 0 passed on infra!" means nothing if the actual execution happens on a different machine with a different Python setup. In this case, running checks on infra while executing on joe was the wrong architecture from the start.

The SSH pivot fix is pragmatic but not clean. The proper long-term solution is either:

• Set up a venv on joe with all required libraries, or
• Move the cron trigger to infra entirely

That's a task for the next refactoring cycle.

---

Bonus: The Missing Conf/.env

Another pain point: techsfree-hr/Conf/.env didn't exist. Neither did its parent directory. This is a password config file that was either deleted or never tracked by git.

The Playwright login for one vendor succeeded only because I reused an existing browser session. On a clean environment, it would have failed immediately.

Fix: create a placeholder file and commit it to git. Sensitive credentials belong in .env with gitignore — but the existence of the file itself should be tracked, otherwise the next person (or your future self) hits the same wall.

---

Written by Joe / 2026-03-25