The program starts, prints Configuration error, and exits. No file name, no reason. It works on your laptop and fails on the server. You could add logging, rebuild and redeploy. Or, on Linux, you could ask the operating system what the program actually tried to do. That is what strace is for.
Every time a program opens a file, connects to another machine, reads a variable or starts a process, it asks the kernel through a system call. strace prints those calls as they happen, with their arguments and results. You do not need the source code, and you do not need to change anything.
Start simply. Run strace followed by the command you normally run. The output is long, so add -o trace.txt to write it to a file, and -f to follow any child processes or threads the program starts.
Then search the file for the word ENOENT, which means no such file or directory, and EACCES, which means permission denied. Very often the answer is a line like this one. openat for /etc/myapp/config.yml returned -1 ENOENT. The program was looking in a different place than you thought, perhaps because an environment variable was missing on the server. Many lookups are expected to fail, because programs try several paths in turn, so look at the last failures before the exit.
A few other options are worth knowing. With -e trace=file you see only calls involving file names, and -e trace=network shows only networking. -T shows how long each call took, which helps when a program hangs and you want to see which connect or read it is waiting on. -s 200 shows longer strings, because by default they are cut short. And -p followed by a process id attaches to something already running. That may need sudo, and inside a container it may need extra permission.
Be careful in production. strace can slow the traced program down a great deal, so attach briefly and detach.
On a Mac, dtruss is the rough equivalent, and on Windows, Process Monitor shows file and registry activity in the same spirit.
Try it today on something harmless, such as strace -e trace=file ls. Once you have seen a program's real conversation with the system, an unhelpful error message stops being a dead end.
– Asael Shinder
Top comments (0)