DEV Community: Zsolt Kacsándi

System calls: how programs talk to the Linux kernel

Zsolt Kacsándi — Thu, 18 Dec 2025 08:25:10 +0000

In the previous post we took a look at the kernel, what it is and what it’s responsible for.
It is a single file, and among other things it "gives us APIs to interact with the hardware".

We also built a simple distribution that consisted of the kernel and our init program written in Go.

But if the kernel makes hardware interactions possible for programs, how was the our init able to print on the screen? The screen is a hardware device too.

And where is that API that makes these interactions possible?

Let's find out.

Observing our init program

First, let's install a tool called strace that lets us observe what programs are doing:

~$ sudo apt -y install strace

Now let's enter the directory where we built our Go program and run our init from the previous post, but this time with strace:

~$ cd linux-inside-out/init
~/linux-inside-out/init$ strace -f -e trace=execve,getpid,write ./init

Note: the -f flag instructs strace to follow any other process that our program might start. The -e trace=execve,getpid,write part filters the output to only show these three system calls. Without this filter, strace output can be very verbose and overwhelming. We will focus on these for now.

You will see something like this:

execve("./init", ["./init"], 0x7ffdd61f6f88 /* 25 vars */) = 0
strace: Process 25165 attached
strace: Process 25166 attached
strace: Process 25167 attached
strace: Process 25168 attached
[pid 25164] write(1, "Hello from Go init!\n", 20Hello from Go init!
) = 20
[pid 25164] getpid()                    = 25164
[pid 25164] write(1, "PID: 25164\n", 11PID: 25164
) = 11
[pid 25164] write(1, "tick 0\n", 7tick 0
)     = 7
[pid 25165] getpid()                    = 25164
[pid 25164] write(1, "tick 1\n", 7)     = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
[pid 25164] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=25164, si_uid=1000} ---
[pid 25164] write(1, "tick 1\n", 7tick 1
)     = 7
[pid 25164] write(1, "tick 2\n", 7tick 2
)     = 7
[pid 25165] getpid( <unfinished ...>
[pid 25164] write(1, "tick 3\n", 7 <unfinished ...>
[pid 25165] <... getpid resumed>)       = 25164
tick 3
[pid 25164] <... write resumed>)        = 7
[pid 25164] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=25164, si_uid=1000} ---
[pid 25165] getpid( <unfinished ...>
[pid 25164] write(1, "tick 4\n", 7 <unfinished ...>
[pid 25165] <... getpid resumed>)       = 25164
tick 4
[pid 25164] <... write resumed>)        = 7
[pid 25164] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=25164, si_uid=1000} ---
^Cstrace: Process 25164 detached
strace: Process 25167 detached
strace: Process 25168 detached
strace: Process 25166 detached
strace: Process 25165 detached

Press Ctrl + C to exit from the process.

Let's go step by step through what we see here:

execve("./init", ["./init"], 0x7ffdd61f6f88 /* 25 vars */) = 0

The ./init might be familiar. This is the name of our program. With execve we ask the kernel to execute a program.

[pid 25164] write(1, "Hello from Go init!\n", 20Hello from Go init!
) = 20

This is what we print at the beginning of our program (fmt.Println("Hello from Go init!")).

[pid 25164] getpid()                    = 25164
[pid 25164] write(1, "PID: 25164\n", 11PID: 25164

We ask the kernel the process ID of our running program that is 25164 (getpid() = 25164).
Then we print it (write(1, "PID: 25164\n", 11PID: 25164).
That is exactly what we did in our Go code: fmt.Println("PID:", os.Getpid()) // printing the PID (process ID).

Then we wait two seconds and print a tick [number] in a loop:

[pid 25164] write(1, "tick 1\n", 7tick 1
)     = 7
...
[pid 25164] write(1, "tick 2\n", 7tick 2
)     = 7
[pid 25164] write(1, "tick 3\n", 7 <unfinished ...>
...

fmt.Println("tick", i)
time.Sleep(2 * time.Second)

So what is the execve, getpid, write that we see here?

System calls

These are system calls, the kernel's API for userspace programs.

When your Go program calls fmt.Println(), somewhere deep in the Go runtime, it calls write() to ask the kernel to output text.

Every time a program needs to interact with the outside world, whether it's a hardware device, another program, a service on the network, or data on the disk, it makes a system call.

Your program runs in a sandbox environment. It can only access memory the kernel assigned to it, and it can only follow control flow and do math on the CPU.

The kernel owns all of the resources in the system. If a program needs access to any of them, it needs to ask the kernel.

This design provides us multiple benefits:

Security: The kernel can check permissions during system calls
Isolation: A misbehaving program can't crash other programs or the computer
Observability: Because everything goes through system calls, you can trace and audit what any program is doing

All programs use the same system call interface

Now let's trace another program. echo, a simple utility written in C that you might be familiar with.

~$ strace echo "Hello from Go init!"

execve("/usr/bin/echo", ["echo", "Hello from Go init!"], 0x7ffea82ea1d8 /* 25 vars */) = 0
...
write(1, "Hello from Go init!\n", 20Hello from Go init!
)   = 20
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

write(1, "Hello from Go init!\n", 20Hello from Go init!
)   = 20

This line is exactly the same as we saw above:

[pid 25164] write(1, "Hello from Go init!\n", 20Hello from Go init!
) = 20

Our Go program and the C program make the same system calls. The language doesn't matter, they both talk to the kernel using the same API.

Every tool and command that you see in a book or a tutorial is a userspace program that performs a set of system calls to ask the kernel to do things.

How system calls actually work

Now we know that system calls are the kernel's API for userspace programs.

Web APIs use the HTTP protocol as their communication channel. \
But what is the channel of communication for system calls? How do programs perform these calls?

The answer lies in the hardware. Modern CPUs have different execution modes:

User mode (restricted): Where your programs run. They have limited access to memory and some CPU instructions. It is a sandboxed mode.
Kernel mode (privileged): Where the kernel runs. It has total access to all memory and all hardware instructions.

When a program wants to perform an operation that needs elevated privileges, it puts the system call number and arguments into the CPU registers (these are small memory banks built into the processor) and
executes a built-in CPU instruction (on x86_64 architecture this is SYSCALL).

This instruction causes the CPU to instantly switch from user mode to kernel mode. The program stops here and control is handed over to the kernel.

The kernel looks at the registers, sees what the program wants to perform, checks if it is allowed to, and then performs the action.

The kernel puts the result back into a register and switches the CPU back to user mode. The program resumes its normal operation.

Note: the name of the modes and the syscall instruction vary by CPU architecture, but the concept is pretty much the same.

Your Program (User Mode)            Linux Kernel (Kernel Mode)
────────────────────────            ──────────────────────────

     │
     ├─ Put syscall number in register
     ├─ Put arguments in registers
     │
     └─ Execute: SYSCALL ───────────────────┐
                                            │
                                       [CPU MODE SWITCH]
                                            │
                                            ▼
                                     Kernel receives request
                                            │
                                            ├─ Check permissions
                                            ├─ Perform operation
                                            │
                                            └─ Put result in register
                                            │
                                       [CPU MODE SWITCH]
                                            │
result ◄────────────────────────────────────┘
     │
continue program...

You can even find this instruction in the source code of Go if you start tracking down what happens when you print something: src/internal/runtime/syscall/linux/asm_linux_amd64.s.

Troubleshooting with strace

Since all resource access goes through system calls, strace (and similar tracing tools) give you a troubleshooting superpower. \

It works on any program, written in any language, without access to source code. You're observing the kernel's API.

You can use it to find:

Files a web application tries to load but does not find

[pid 764640] stat("/app/assets/assets/app-5pEwFF_.js", 0x7ffdfef9af00) = -1 ENOENT (No such file or directory)
[pid 764640] stat("/app/assets/assets/bootstrap-xCO4u8H.js", 0x7ffdfef9af00) = -1 ENOENT (No such file or directory)
[pid 764640] stat("/app/assets/assets/bootstrap.js", 0x7ffdfef9af00) = -1 ENOENT (No such file or directory)
[pid 764640] stat("/app/assets/assets/styles/app-17aMgWE.css", 0x7ffdfef9af00) = -1 ENOENT (No such file or directory)
[pid 764640] stat("/app/assets/assets/styles/app.css", 0x7ffdfef9af00) = -1 ENOENT (No such file or directory)

What your application communicates with other services

[pid 764640] sendto(7, "l\0\0\0\3SELECT sess_data, sess_lifetime FROM sessions WHERE sess_id = '3666497cf012d41afe736f4eaf869e71' FOR UPDATE", 112, MSG_DONTWAIT, NULL, 0) = 112
[pid 764640] poll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, 86400000) = 1 ([{fd=7, revents=POLLIN}])
[pid 764640] recvfrom(7, "\1\0\0\1\2A\0\0\2\3def\twebapp\10sessions\10sessions\tsess_data\tsess_data\f?\0\377\377\377\377\374\221\20\0\0\0I\0\0\3\3def\tangolklub\10sessions\10sessions\rsess_lifetime\rsess_lifetime\f?\0\n\0\0\0\3)P\0\0\0\5\0\0\4\376\0\0\3\0\5\0\0\5\376\0\0\3\0", 32768, MSG_DONTWAIT, NULL, NULL) = 169

An application tries to connect to an API but it cannot

connect(4, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("10.255.255.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
getsockname(4, {sa_family=AF_INET, sin_port=htons(59202), sin_addr=inet_addr("65.108.213.19")}, [128 => 16]) = 0
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fb3514ccdf0}, NULL, 8) = 0
poll([{fd=4, events=POLLOUT}, {fd=3, events=POLLIN}], 2, 1000) = 0 (Timeout)
rt_sigaction(SIGPIPE, NULL, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fb3514ccdf0}, 8) = 0
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fb3514ccdf0}, NULL, 8) = 0
poll([{fd=4, events=POLLPRI|POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)

Note: strace slows down programs, and you can see sensitive information like secrets and personal data. Always take this into consideration when you use it.

The output of strace can be overwhelming at first.
Start with simple commands like ls, cp that you are familiar with, and focus on these common system calls:

They might have variants like openat for open, but the concept is the same. \
You can look up any system call with man 2 <name> (you may need to install manpages-dev first):

~$ sudo apt -y install manpages-dev
~$ man 2 write

What we learned

We expanded our mental model that the kernel is a program that has special privileges
The kernel runs in kernel mode, regular programs run in a sandboxed environment called user mode
System calls are the kernel’s API for userspace programs
When a program interacts with the outside world, it does it via system calls
You can trace these system calls with tools like strace

What’s next

If you’d like to follow the rest of the Linux Inside Out series, you can find the index here:

https://serversfor.dev/linux-inside-out/

The Linux kernel is just a program

Zsolt Kacsándi — Wed, 03 Dec 2025 19:25:24 +0000

Most books and courses introduce Linux through shell commands, leaving the kernel as a mysterious black box doing magic behind the scenes.

In this post, we will run some experiments to demystify it: the Linux kernel is just a binary that you can build and run.

The experiments are designed so you can follow along if you have a Linux PC.
But this is completely optional, the goal is to build a mental model about how Linux works, seeing how components of the system fit together.

But first let's talk about what a kernel is.

What is a kernel?

Computers are built from CPUs, memory, and other devices, like video cards, network cards, keyboards,
displays, and a lot of other stuff.

These devices can be manufactured by different companies, have different capabilities, and can be programmed differently.

An operating system kernel provides an abstraction to use these devices and resources conveniently and securely. Without one, writing programs would be much more difficult. We would need to write the low-level code to use every device that our program needs, and it's likely that it wouldn't work on other computers.

A kernel

gives us APIs to interact with the hardware over a unified interface
manages how programs can use the computer's CPU, memory and other resources
provides access control over what resources can a program access
provides additional features like firewalls, file systems, mechanisms for programs to communicate, etc.

The closest analogy from the software development world is that the kernel is a runtime for our computer.

Where is the kernel?

On most Linux distributions we will find the kernel under the /boot directory.
Let's enter the directory and list its contents:

~$ cd /boot
/boot$ ls -1
System.map-6.12.43+deb13-amd64
System.map-6.12.48+deb13-amd64
config-6.12.43+deb13-amd64
config-6.12.48+deb13-amd64
efi
grub
initrd.img-6.12.43+deb13-amd64
initrd.img-6.12.48+deb13-amd64
vmlinuz-6.12.43+deb13-amd64
vmlinuz-6.12.48+deb13-amd64

We see a few files here, but the one we are looking for is vmlinuz-6.12.48+deb13-amd64. This single file is the kernel.

If you ever wondered what this name means:

vmlinuz: vm for virtual memory, linux, and z indicating compression
6.12.48+deb13: this is the kernel version, and the distribution (Debian 13)
amd64: this is the architecture of our system

Note: Different distributions may use slightly different naming conventions. vmlinuz is commonly the bootable compressed kernel image.

Let's start the kernel

In our first experiment we will copy this kernel into another directory and run it.

First, let's create a directory and copy the kernel there.

Note: Your kernel version might differ, remember to check it before the cp command.

/boot$ cd
~$ mkdir linux-inside-out
~$ cd linux-inside-out/

~/linux-inside-out$ cp /boot/vmlinuz-6.12.48+deb13-amd64 .
~/linux-inside-out$ ls -lh
total 12M
-rw-r--r-- 1 zsoltkacsandi zsoltkacsandi 12M Dec  1 09:44 vmlinuz-6.12.48+deb13-amd64

Then install some tools that are needed for this experiment.

We will use QEMU, a virtual machine emulator, because our kernel needs something that works like a computer, and because we do not want to mess up our original operating system.

~$ sudo apt update
~$ sudo apt install -y qemu-system-x86 qemu-utils

Then start a virtual machine with our kernel:

~/linux-inside-out$ qemu-system-x86_64 \
  -m 256M \
  -kernel vmlinuz-6.12.48+deb13-amd64 \
  -append "console=ttyS0" \
  -nographic

The output should be something like this:

SeaBIOS (version 1.16.3-debian-1.16.3-2)
iPXE (https://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+06FC6D30+06F06D30 CA00
Booting from ROM...
Probing EDD (edd=off to disable)... o
[    0.000000] Linux version 6.12.48+deb13-amd64 (debian-kernel@lists.debian.org) (x86_64-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, )
[    0.000000] Command line: console=ttyS0
...
[    2.055627] RAS: Correctable Errors collector initialized.
[    2.161843] clk: Disabling unused clocks
[    2.162218] PM: genpd: Disabling unused power domains
[    2.179652] /dev/root: Can't open blockdev
[    2.180871] VFS: Cannot open root device "" or unknown-block(0,0): error -6
[    2.181038] Please append a correct "root=" boot option; here are the available partitions:
[    2.181368] List of all bdev filesystems:
[    2.181477]  fuseblk
[    2.181516]
[    2.181875] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    2.182495] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.12.48+deb13-amd64 #1  Debian 6.12.48-1
[    2.182802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
...
[    2.186426] Kernel Offset: 0x30e00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    2.186949] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

You can exit by pressing Ctrl + A then X.

So, we've just started the same kernel that is running on our computer. It took 2 seconds, it printed a lot
of log messages, then panicked.

This panic is not a bug, actually this is expected - once our kernel initializes itself, it tries to mount
the root filesystem, and hand over control to a program called init.

So let's give one to it.

We will write a simple program that we will use as an init program.

We will use Golang for two reasons:

it has an easy to learn syntax, that readers coming from different backgrounds can pick up and understand quickly
it can build a statically-linked binary with no C dependencies, making it portable and perfect for our minimal experiment

First let's install Golang, and create a new project called init:

~/linux-inside-out$ sudo apt -y install golang
~/linux-inside-out$ mkdir init
~/linux-inside-out$ cd init
~/linux-inside-out/init$
~/linux-inside-out/init$ go mod init init
go: creating new go.mod: module init
go: to add module requirements and sums:
go mod tidy

Create a new file, called main.go:

package main

import (
    "fmt"
    "os"
    "time"
)

func main() {
    fmt.Println("Hello from Go init!")
    fmt.Println("PID:", os.Getpid()) // printing the PID (process ID)

    for i := 0; ; i++ { // every two seconds printing the text "tick {tick number}"
        fmt.Println("tick", i)
        time.Sleep(2 * time.Second)
    }
}

Then build the program and run it:

~/linux-inside-out/init$ CGO_ENABLED=0 go build -o init .
~/linux-inside-out/init$ ./init
Hello from Go init!
PID: 3086
tick 0
tick 1

Press Ctrl + C to stop it.

As you can see, this is a regular program that got the PID 3086 and prints some text to the output.
There is nothing special about it.

Note: If you use another programming language for this experiment, you will need to compile a statically-linked binary.
Without that the following parts of the experiment will not work.

Now we create a simple initramfs filesystem.
When the kernel starts it does not have all of the parts loaded that are needed to access the disks in the computer, so it needs a filesystem loaded into the memory called initramfs (Initial RAM filesystem).

~/linux-inside-out$ mkdir -p rootfs/{proc,sys,dev}
~/linux-inside-out$ cp ./init/init rootfs/init
~/linux-inside-out$ sudo mknod rootfs/dev/console c 5 1
~/linux-inside-out$ sudo mknod rootfs/dev/null c 1 3

The cp and mkdir commands might be familiar. The mknod command creates special files that programs use to communicate with hardware devices.

Our root filesystem directory structure looks like this:

|-- dev             # dev (devices) directory
|   |-- console     # console device
|   `-- null        # null device
|-- init            # our Golang program
|-- proc            # a directory called proc
`-- sys             # a directory called sys

Now let's package the files into an archive file, called initramfs.img.

( cd rootfs && find . | cpio -H newc -o ) > initramfs.img

Then start a virtual machine again, with the kernel and initramfs:

qemu-system-x86_64 \
  -m 256M \
  -kernel vmlinuz-6.12.48+deb13-amd64 \
  -initrd initramfs.img \
  -append "console=ttyS0 rdinit=/init" \
  -nographic

SeaBIOS (version 1.16.3-debian-1.16.3-2)
iPXE (https://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+0EFC6D30+0EF06D30 CA00
Booting from ROM...
Probing EDD (edd=off to disable)... o
[    0.000000] Linux version 6.12.48+deb13-amd64 (debian-kernel@lists.debian.org) (x86_64-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PR)
[    0.000000] Command line: console=ttyS0 rdinit=/init
...
[    1.922229] RAS: Correctable Errors collector initialized.
[    2.158525] clk: Disabling unused clocks
[    2.158865] PM: genpd: Disabling unused power domains
[    2.264545] Freeing unused decrypted memory: 2028K
[    2.327128] Freeing unused kernel image (initmem) memory: 4148K
[    2.406015] Write protecting the kernel read-only data: 28672k
[    2.407968] Freeing unused kernel image (rodata/data gap) memory: 488K
[    2.555150] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    2.557822] tsc: Refined TSC clocksource calibration: 2903.977 MHz
[    2.558399] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x29dbf0142be, max_idle_ns: 440795300983 ns
[    2.565700] clocksource: Switched to clocksource tsc
[    2.672446] Run /init as init process
Hello from Go init!
PID: 1
tick 0
tick 1
tick 2

Our kernel booted normally, then it started our Go program, the init process.
A program that is running is called process.

There are a few important points to note here:

Our Go program got the process ID 1 (PID: 1). PID 1 is the first process to start, it is called the init process. The purpose of the init process is to start the other programs that need to be running for the operating system.
Up until the Run /init as init process line we are in the kernel space. With the init process starting we are entering into the user space.
We have just built a (rather simple) Linux distribution. Two files, that's it. A Linux distribution is really just a Linux kernel, a bunch of programs and config files packaged together.

What we learned

We have already learnt quite a few important concepts that are essential to understand Linux systems:

The Linux kernel is a single, few megabytes file, sitting on your disk
A Linux distribution is just a kernel and a set of other programs and config files
A process is a program that is under execution
PID is the process ID
What the init process is
We familiarized ourselves with the concepts of kernel space and user space

What’s next

If you’d like to follow the rest of the Linux Inside Out series, you can find the index here:

https://serversfor.dev/linux-inside-out/

The next post will be about understanding system calls.