DEV Community

Alex Fler
Alex Fler

Posted on • Originally published at theopsmechanic.com on

Where Did That Environment Variable Come From?

#bash #cli #devops #linux

“It works in my terminal but not in cron.”

If you’ve been doing this long enough, that sentence alone is enough to make your eye twitch. You know exactly what’s coming: an hour of grepping through dotfiles, manually reading shell startup chains in your head, and eventually finding that some vendor install script shoved a PATH override into /etc/profile.d/java-vendor-garbage.sh three years ago and nobody noticed because interactive shells got the right value from ~/.zshrc anyway.

I’ve done this dance hundreds of times. Literally hundreds. And every single time it’s the same tedious archaeology: which files does this shell context actually read, in what order, and which one of them is clobbering my variable?

So I finally wrote a tool to do it for me.

The Shell Startup Problem

If you already know how shell startup files work, skip ahead. If you don’t, buckle up, because this is where most env var bugs come from and nobody teaches it properly.

When you open a terminal on macOS, zsh reads seven files in a specific order:

/etc/zshenv → /etc/zprofile → ~/.zshenv → ~/.zprofile → /etc/zshrc → ~/.zshrc → ~/.zlogin
Enter fullscreen mode Exit fullscreen mode

Bash on Linux does its own thing:

/etc/profile → /etc/profile.d/*.sh → ~/.bash_profile → ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

That’s just for login shells. An interactive non-login shell? Different chain. A cron job? Skips almost everything—you might only get /etc/environment and whatever the crontab itself sets. systemd services? They have environment.d drop-in directories. And macOS launchd agents live in their own plist-based dimension that has nothing to do with your shell at all.

Every one of these contexts can set, override, append to, or unset the same variable. A variable can enter ~/.zshenv as /usr/bin, get /usr/local/bin prepended in /etc/zprofile, pick up /opt/homebrew/bin in ~/.zshrc, and then get completely overwritten by some ancient ~/.zlogin that hardcodes a value from 2021.

The “it works on my machine” of environment variables is really “it works in my specific shell context.”

envtrace

envtrace walks the actual startup file chain for your platform and context, and shows you every file that touches a given variable. That’s it. No magic.

cargo install envtrace


$ envtrace PATH
Enter fullscreen mode Exit fullscreen mode

You get a chronological trace: file, line number, operation (set, export, append, prepend, unset, conditional), and the resulting value after each step. Every file in the chain that doesn’t touch your variable gets skipped, unless you pass --verbose and want to see the full list of files it checked.

The “Works in Terminal, Breaks in Cron” Fix

Here’s the scenario that made me actually write this thing.

A Java developer files a ticket: “JAVA_HOME is wrong in the CI runner.” I SSH into the box. echo $JAVA_HOME gives me /usr/lib/jvm/java-17. Correct. I trigger the CI job manually. It picks up Java 11. What?

The answer, as always, is context. My interactive login shell reads ~/.zshrc, which sources an internal java-env.sh that exports the Java 17 path. The CI runner spawns a non-interactive non-login shell. It never touches ~/.zshrc. It gets its JAVA_HOME from /etc/profile, which still points to Java 11 because nobody updated it after the migration six months ago.

This is not a bug. This is how Unix shells have always worked. But figuring this out by hand means you have to know which files each context reads, then manually read each one, then mentally track the value as it mutates through the chain. Every time.

Or:

$ envtrace -C login,cron JAVA_HOME
Enter fullscreen mode Exit fullscreen mode

The -C flag compares a variable across multiple contexts side by side. You see the full trace for each context and exactly where the values diverge. This one flag alone would have saved me more hours than I’m comfortable admitting.

Tracing Functions

Variables aren’t the only casualties. Shell functions get sourced, overridden, and unset -f’d through the same chain. If you’ve ever wondered why nvm is suddenly not a function after some dotfile change, this is for you:

$ envtrace -F nvm
Enter fullscreen mode Exit fullscreen mode

Picks up POSIX function syntax, bash function keyword syntax, zsh autoload declarations, and unset -f removals. Same chronological trace, same file chain awareness.

Finding Things When You Have No Idea

Sometimes you don’t even know which context is relevant. You just know something somewhere is setting LD_LIBRARY_PATH to a directory that was decommissioned in the Obama administration and you need to find it.

$ envtrace --find LD_LIBRARY_PATH
Enter fullscreen mode Exit fullscreen mode

Ignores context entirely, searches every config file it knows about, and shows you every mention. Broad net.

PATH Health Checks

While I was in there, I added a sanity checker:

$ envtrace --check
Enter fullscreen mode Exit fullscreen mode

Scans your PATH for the usual sins: directories that don’t exist, duplicates, empty entries from stray :: separators. On macOS, it also compares what your shell sees against what launchd provides to GUI apps—which is the root cause of every “it works in Terminal but not in VS Code” bug report you’ve ever gotten.

Under the Hood

I want to be upfront about what this is and isn’t. envtrace is not a shell parser. Fully parsing bash is a Lovecraftian horror that I am not interested in. What it does is pattern-match against the forms people actually use in dotfiles:

  • export VAR=value and VAR=value
  • VAR="$VAR:/new/path" (append) and VAR="/new/path:$VAR" (prepend)
  • unset VAR
  • [-f /something] && export VAR=value (conditional)
  • source and . directives (followed recursively, with circular-include detection so your weird dotfile setup doesn’t send it into an infinite loop)

This covers the vast majority of what you’ll find in real startup files. If you have some baroque eval $(generate_env_dynamically.py) situation, envtrace won’t help you and frankly nothing will.

It spits out JSON too (--format json) if you want to pipe it into jq or build something on top of it.

Getting It 

cargo install envtrace
Enter fullscreen mode Exit fullscreen mode

Or grab a binary from the releases page. Needs Rust 1.85+ if you’re building from source.

It runs on macOS (zsh) and Linux (bash). Those are the two platforms I care about and the two platforms where this problem actually exists.

The next time you’re staring at a broken PATH at 2 AM, trying to remember whether /etc/profile runs before or after ~/.bash_profile (it does), and whether cron even reads either of them (it doesn’t)—just trace it.

Top comments (0)