You know the situation. A service starts, sits there, and produces exactly zero log lines before eventually timing out. You've checked the config, the permissions, the env vars. Nothing. This is where strace earns its place.
The scenario
A daemon process refuses to start cleanly under systemd but works fine when you run it by hand. No errors, no logs, just silence. Classic case of something happening before your application gets to log anything.
\`bash
Find the PID
systemctl status your-daemon
Attach strace to it
strace -p -f -e trace=write
`\
The -f flag follows forked child processes — essential for daemons that spawn workers. -e trace=write filters to only show write syscalls, which is usually what you're interested in when looking for output the process is trying to produce.
What you actually find
In one case, strace showed the process repeatedly attempting to open a file in /var/run/ that didn't exist and wasn't being created by the init script. The application was calling getcwd() and dying silently when it couldn't resolve a path it expected to be there.
\
openat(AT_FDCWD, "/var/run/daemon/workdir", O_RDONLY) = -1 ENOENT
write(2, "failed to chdir", 15) = 15
\\
The write(2) was going to stderr, which systemd swallows unless you have StandardOutput=journal set. The fix was a one-liner in the init script, but you'd never have found it without strace.
The useful flags, distilled
\`bash
Tail syscalls as they happen
strace -p -f
Filter by syscall type
strace -p -e trace=open,openat,read,write
Timestamp each line so you can see timing
strace -p -f -t
Print relative timestamps from attach moment
strace -p -f -r
Save to file for later analysis
strace -p -f -o /tmp/trace.log
`\
Worth knowing
strace adds overhead. Don't run it on a high-throughput production process permanently — use it in a maintenance window or against a test instance. The -c flag gives you a summary of syscall counts after detach, which is useful for profiling without the noise.
\`bash
strace -p -c -f
... let it run for a bit, Ctrl+C
Summary prints syscall counts
`\
The output format is SYSCALL(arg1, arg2) = RESULT. If you're used to reading it the first time, it takes a few minutes to adjust but it's consistent and the manual is decent.
When logs have failed you
strace isn't magic. It's just direct access to what the kernel is telling the process. When application-level logging has nothing, that's often exactly when you need to go one layer lower.
Top comments (0)