The Setup
A program works on your laptop. You copy it to another Linux machine, run the same command, and it fails immediately, before any of your own code has executed. This article follows one small imaginary program, orders, from source code to exit, so you can place any startup failure at the right step. orders reads a list of orders and prints a total. It is illustrative: nothing here was measured, and no outputs are shown that were not produced.
Behavior described is for Linux with the glibc dynamic linker unless stated. Other systems and C libraries can differ.
Source, machine code and the executable file
These are three different things:
- Source code is text for people.
- Machine code is what the processor runs.
- The executable file packages the machine code with what the OS needs to load it. On Linux this is usually an ELF file. It records where each part goes in memory, which dynamic linker to use, and which shared libraries the program needs.
Not every language works this way. CPython compiles source to bytecode and runs it on a virtual machine; bytecode is not machine code, and the Python documentation says it is an implementation detail that can change between versions. Java source compiles to class files with bytecode; the JVM loads, links and initializes classes. So the useful question is not "compiled or interpreted?" but "what exists on the machine at the moment the program starts?" The C version arrives as an executable. The Python version needs an interpreter installed. The Java version needs a Java runtime.
From the shell to a process
A shell starts a command in two separate steps:
-
forkcreates a new process by duplicating the caller. The child has its own process ID and a memory space with the same content as the parent's at that moment. On Linux, pages are copy-on-write. - In the child,
execvereplaces the program being run with a new one, with fresh stack, heap and data. It does not return on success because the old image is gone. The process ID stays the same.
Fork alone gives you copies of the shell. Exec alone would replace the shell with your program. Starting a command needs both, in that order.
What survives exec:
| Thing | After execve |
|---|---|
| Open file descriptors | stay open, unless marked close-on-exec |
| Environment | passed to the new program |
| Signal handlers | reset to default (ignored signals stay ignored) |
| Memory mappings | not preserved; fresh layout |
Scripts start through exec too: if the file is executable and its first line is #! followed by an interpreter, the kernel runs that interpreter with the script path.
What the process looks like inside
Each process has its own virtual address space. Regions: code (text), data (globals and statics), heap (memory requested at run time), stack (function-call frames), and mappings including shared libraries. On a real Linux system, /proc/<pid>/maps lists each mapped region with its address range, permissions, whether it is shared or private, and the backing file if any.
# Read the map of the current shell process (standard Linux command).
cat /proc/self/maps
If a program touches memory it is not allowed to use, the kernel sends SIGSEGV, and the default action is to terminate the process. Reaching the stack size limit also generates SIGSEGV.
How the dynamic linker finds shared libraries
If orders uses a shared library, that file must exist on the machine where it runs. The ELF file names a program interpreter (PT_INTERP), which on glibc systems is the dynamic linker ld.so. It runs before main.
For a dependency name without a slash, the glibc linker searches in this order:
| Order | Source |
|---|---|
| 1 |
DT_RPATH in the executable, only if there is no DT_RUNPATH
|
| 2 | The LD_LIBRARY_PATH environment variable |
| 3 |
DT_RUNPATH in the executable |
| 4 | The cache file /etc/ld.so.cache
|
| 5 | Default directories (/lib, /usr/lib; lib64 on some 64-bit systems) |
DT_RUNPATH applies to direct dependencies; DT_RPATH also applies to the dependencies of those. In secure-execution mode (for example set-user-ID programs), LD_LIBRARY_PATH is ignored, so a user-controlled variable cannot redirect a privileged program.
If the library is in none of those places, the linker cannot finish preparing the program and the process ends before main. No file was opened, no log line was written, and none of your tests could have caught it. The program is fine; the machine lacks what the program declared. A container image that omits a needed library behaves the same way.
To inspect what an executable declares (standard binutils commands; outputs vary by binary):
# Which program interpreter does this ELF file name?
readelf -l ./orders | grep -i interpreter
# Which shared libraries does it list as needed?
readelf -d ./orders | grep -i NEEDED
Interpreted languages ask the same question at a different time: Python fails when an import runs for a package that is not installed, and Java fails when the runtime cannot find a class.
The startup stages and where each one fails
| Stage | What happens | Typical failure |
|---|---|---|
| Shell + kernel | find the file, check permission, build the image | not found (127), not executable (126) |
| Dynamic linker | find and load shared libraries | missing or incompatible library |
| Language runtime | C library setup; interpreter or JVM starts, imports or loads classes, runs initialization | missing config or module, import error |
| Your code | first line of your own logic | bugs in your logic |
A service that never listens on its port might have failed in the linker, the runtime, or its own configuration code. Asking which handoff it reached tells you where to look.
How programs end, and what the numbers mean
There are two different ways for a process to end.
Normal exit. The program returns from main or calls exit with a number. The parent calls wait, and the status can be decoded: one macro reports normal termination, another returns the exit status (the least significant eight bits, so 0 to 255). By convention 0 is success; non-zero is failure.
Termination by a signal. SIGKILL and SIGSTOP cannot be caught, blocked or ignored. The default action for SIGSEGV is to terminate the process and dump core, if the core file size limit is not zero. The parent can tell this apart from a normal exit using a different wait macro.
Bash then folds both into one number. When a command dies on a fatal signal numbered N, Bash reports 128 + N:
# Documented Bash convention: killed by SIGKILL (signal 9 on x86 and ARM) -> 128 + 9.
# Shown as a pattern, not as captured output.
sleep 60 &
kill -9 $!
wait $!
echo $? # expected by the convention: 137
| Status | Meaning |
|---|---|
| 127 | Bash could not find the command |
| 126 | found, but not executable |
| 128+N | terminated by signal N (137 = SIGKILL) |
The Bash manual also warns that the shell may use values above 125 specially, so statuses in that range deserve a second look. Signal numbers vary on some architectures.
137 does not prove an out-of-memory kill. The kernel's out-of-memory killer is one possible cause: a control group that reaches its memory limit and cannot reduce usage has the OOM killer invoked inside that group, which is how a memory-limited container can lose its main process. But kill -9, a supervisor, or another program sending SIGKILL gives the same 137. To claim an OOM kill you need more evidence, such as the kernel's report of the tasks involved or the OOM kill counter in the cgroup memory events file. The number is a lead, not a verdict.
Runtimes add their own conventions. In Node.js, an uncaught exception by default prints the stack trace to standard error and exits with code 1.
What to take away
- Source, machine code and the executable file are different things; what exists at startup differs by language.
- Starting a command is
forkthenexecve: create a process, then replace what it runs. - The dynamic linker runs before
mainand searches a fixed order; a missing library ends the process before your code starts. - 127 is not found, 126 is not executable, 128+N is a signal, and 137 is SIGKILL, not proof of memory exhaustion.
Limits of this article
orders is imaginary. Commands above are standard Linux commands shown as patterns; they were not run for this article, so outputs are not reproduced. The linker order is glibc on Linux. Python bytecode and JVM/HotSpot internals vary by version.
Sources
- fork(2), execve(2), wait(2), signal(7), getrlimit(2), ld.so(8), proc_pid_maps(5): https://man7.org/linux/man-pages/
- Bash Reference Manual, Exit Status: https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html
- Kernel cgroup v2 and vm sysctl documentation: https://docs.kernel.org/admin-guide/cgroup-v2.html
- Python glossary (bytecode): https://docs.python.org/3/glossary.html
- JVM Specification SE 21, chapter 5: https://docs.oracle.com/javase/specs/jvms/se21/html/jvms-5.html
- Node.js process documentation: https://nodejs.org/api/process.html
Video version with animations: [https://youtu.be/FFZaY7Di66c]. Next in the series: CPU, memory and storage for developers.
Top comments (1)
Great technical breakdown. One subtle case worth adding: if the dynamic linker (ld.so) cannot resolve a symbol at startup, the process exits before main() too -- often visible only via dmesg or strace on the execve syscall, not in the program own logs. Did you find pre-main failures more often caused by missing shared libraries, or by environment/loader issues?