DEV Community: Peng Zhang

Who killed my service: collecting kernel kill logs with OTEL

Peng Zhang — Tue, 31 Mar 2026 05:56:35 +0000

We run a container platform. For privacy and security reasons, we do not collect kernel logs because customer workloads use the same kernel as the host and kernel messages can contain sensitive customer data, such as command-line arguments surfaced in audit logs. However, we recently hit a blind spot: foo.service was killed with no trace in its own logs or systemd logs. No error, no panic, no graceful shutdown message. The process was just gone. Our suspicion pointed to the kernel, but we had no evidence. This post shows how to close the gap by collecting selected kernel kill logs.

When Does the Kernel Kill a Process?

The kernel protects the system and it will kill user space processes that either violated a safety rule or the system has run out of a physical resource needed to stay stable. Here are some common causes of a kernel kill.

Out of Memory (OOM) Killer: When RAM and swap are fully exhausted, the kernel picks a victim using an OOM score and kills it to prevent a total system hang. This is the most common cause of mysterious process deaths.

Log signal: Out of memory: Killed process ...

Memory cgroup limit: In containerized environments, each container typically has a memory cgroup limit. When a container exceeds its limit, the kernel's cgroup OOM killer fires even if the host has plenty of free memory.

Log signal: Memory cgroup out of memory: Killed process ...

Segmentation Fault: A process accesses memory it doesn't own, or writes to a read-only region. The CPU raises a hardware exception, the kernel catches it, and the process is terminated with SIGSEGV. Note that the Go runtime captures SIGSEGV and turns it into a panic. See Default behavior of signals in Go programs. JVM also does similar: catch SIGSEGV and throw NPE, see Handle Signals and Exceptions.

Note that the Go runtime does not catch the SIGSEGV if CGO is enabled and the segfault occurs in C code. Similar to JVM, segfault in JNI are not caught either.

Log signal: segfault at ... ip ... sp ... error 4

Illegal Instruction: A process executes a CPU instruction it doesn't understand, often from a binary compiled for the wrong architecture, or a corrupted executable.

Log signal: traps: invalid opcode ...

Kernel Oops via a Driver: A system call triggers a bug in a kernel driver. The kernel tries to stay alive but kills the offending process to stabilize the system.

The common thread: none of these show up in the application's own logs. The process is gone before it can write anything. The evidence lives exclusively in the kernel ring buffer, accessible via dmesg or journalctl -k.

The Config Change

You can use OpenTelemetry Collector journald receiver to collect kernel logs. However, the receiver needs to be separate from the service receiver. For example, in the following, there is one receiver for systemd service (journalctl -u) and another receiver for kernel log (journalctl -k).

receivers:
  journald/service:
    directory: /var/log/journal
    units:
      - foo
    priority: info
    storage: file_storage
    start_at: beginning

  journald/kernel:
    directory: /var/log/journal
    matches:
      - _TRANSPORT: kernel
    grep: '(?i)(out of memory|killed process|memory cgroup out of memory|segfault at|traps: invalid opcode|general protection)'
    priority: debug
    start_at: beginning

A few things worth explaining.

Why `priority: debug` for the kernel receiver?

Kernel messages don't follow the same severity conventions as application logs. Critical events like OOM kills are often emitted at debug or info priority in journald's view, not err or crit. If you set priority: info or higher, you'll silently drop the exact messages you're looking for. Setting priority: debug ensures the receiver reads all kernel messages and lets the grep filter do the real work of keeping only what's relevant.

Why a separate receiver?

The units filter and matches filter in the journald receiver are ANDed together when both are specified. That means combining units: [foo] with matches: [{_TRANSPORT: kernel}] would look for kernel messages that also belong to the foo unit, which is contradictory. Kernel messages don't have a unit. There's also no OR logic at the receiver level. Multiple entries within matches are ANDed as well. So if you want "service logs OR kernel logs", you need two receivers. The only alternative would be to drop all filtering and read everything from journald, then use a filter processor downstream, but that means ingesting far more data than you need, which defeats the purpose.

Each journald receiver is a separate journalctl process

OTEL doesn't read journald logs directly; it spawns a journalctl process per receiver. This also means, to use the grep filter in the receiver, systemd v239+ is required.

bash-5.3# systemctl status otelcol
● otelcol.service - Executes open telemetry collector to send logs.
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/otelcol.service; static)
     Active: active (running) since Sat 2026-03-07 00:07:51 UTC; 3 days ago
   Main PID: 2341 (otelcol)
      Tasks: 12 (limit: 9256)
     Memory: 25.5M
        CPU: 24min 5.574s
     CGroup: /system.slice/otelcol.service
             ├─2341 /usr/sbin/otelcol 
             ├─2348 journalctl --utc --output=json --follow --no-tail --unit foo --priority info --directory /var/log/journal --after-cursor xxxxx
             └─2349 journalctl --utc --output=json --follow --no-tail --priority debug --directory /var/log/journal _TRANSPORT=kernel --grep=xxxx

Appendix: Examples that trigger kernel kill

Trigger OOM (scenario-oom.service)

[Unit]
Description=Demo Scenario - OOM Kill
After=network.target

[Service]
Type=simple
MemoryMax=32M
# tail /dev/zero reads zeros endlessly into memory — hits the limit in seconds
ExecStart=/bin/bash -c 'echo "Eating memory..."; tail /dev/zero'
Restart=no

[Install]
WantedBy=multi-user.target

Trigger segfault

Go version

package main

/*
// Dereference NULL in C — bypasses Go's runtime signal handler,
// so the CPU fault reaches the kernel and appears in journalctl -k:
//   "segfault at 0 ip ... error 4 in segfault[...]"
#include <stdlib.h>
void trigger_segfault() {
    volatile int *p = NULL;
    *p = 1;
}
*/
import "C"

func main() {
    C.trigger_segfault()
}

Python version

import ctypes
print("Triggering segfault via null pointer dereference...")
ctypes.string_at(0)

Trigger sigill

Go version

package main

/*
// Execute UD2 in C — bypasses Go's runtime signal handler,
// so the CPU trap reaches the kernel and appears in journalctl -k:
//   "traps: sigill[PID] trap invalid opcode ..."
void trigger_sigill() {
    __asm__("ud2");
}
*/
import "C"

func main() {
    C.trigger_sigill()
}

Python version

import ctypes, mmap
print("Executing UD2 (undefined instruction) to trigger SIGILL...")
buf = mmap.mmap(-1, mmap.PAGESIZE, prot=mmap.PROT_READ|mmap.PROT_WRITE|mmap.PROT_EXEC)
buf.write(b'\x0f\x0b')  # UD2 opcode
ftype = ctypes.CFUNCTYPE(None)
fptr = ctypes.cast((ctypes.c_char * 2).from_buffer(buf), ctypes.c_void_p)
ftype(fptr.value)()

You probably want to disable cgo: Go's stdlib has pure-Go implementations

Peng Zhang — Tue, 31 Mar 2026 05:48:14 +0000

CGO_ENABLED defaults to 1. That means a standard go build produces a binary that links against C libraries (e.g., glibc) at runtime. For many parts of the Go standard library, there is a C-backed implementation and a pure-Go implementation. CGO_ENABLED selects which one gets compiled in. Pure-Go alternatives also exist for third-party libraries, so it is likely you can turn off cgo by setting CGO_ENABLED=0.

Whether to use C libraries or not is a build-time decision, not a runtime fallback like "foo.so doesn't exist, fall back to pure-Go". If the required .so is missing when the cgo binary runs, the dynamic linker fails immediately with an error like:

./dns-with-cgo: error while loading shared libraries: libresolv.so.2: cannot open shared object file: No such file or directory

Example: DNS resolution with net.LookupHost

With cgo enabled, net.LookupHost delegates to glibc's getaddrinfo via libresolv.so.2. With cgo disabled, the same call uses Go's built-in resolver that reads /etc/resolv.conf directly, with no C library involved. Same API either way.

package main

import (
    "fmt"
    "net"
)

func main() {
    addrs, err := net.LookupHost("amazon.com")
    if err != nil {
        fmt.Println("error:", err)
        return
    }
    fmt.Println("resolved:", addrs)
}

Build it both ways:

# cgo on (default)
go build -o dns-with-cgo .

# cgo off
CGO_ENABLED=0 go build -o dns-no-cgo .

ldd shows the difference

% file dns-with-cgo
dns-with-cgo: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),     
dynamically linked (uses shared libs), BuildID[sha1]=4d51b80894921ca4be742c64a7dd76c4c7205697, not stripped

% ldd dns-with-cgo 
        linux-vdso.so.1 (0x00007ffd8efb9000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f5bc44b8000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5bc429a000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f5bc3eed000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5bc46ce000)

% file dns-no-cgo 
dns-no-cgo: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), 
statically linked, BuildID[sha1]=e8c17c6cb53d034c052336acd4b83a09527bd22c, not stripped

% ldd dns-no-cgo 
        not a dynamic executable

The cgo binary dynamically links against libresolv.so.2, libc.so.6, and other glibc libraries. The CGO_ENABLED=0 binary is fully static, no shared libraries at all.

strace confirms it at runtime

# cgo on: loads libresolv.so.2, then delegates to glibc
% strace -e trace=openat -f ./dns-with-cgo 2>&1 | grep -E "(resolv|\.conf)"
openat(AT_FDCWD, "/lib64/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3 # <-- glibc dependency
[pid  1553] openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 5
[pid  1553] openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 5
resolved: [98.82.161.185 98.87.170.74 98.87.170.71]

# cgo off: Go reads config directly, no .so loaded
strace -e trace=openat -f ./dns-no-cgo 2>&1 | grep -E "(resolv|\.conf)"
[pid  2326] openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 5
[pid  2326] openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 5
resolved: [98.82.161.185 98.87.170.74 98.87.170.71]

With cgo, the first thing that happens is loading the C library. Without cgo, Go skips straight to reading the resolver config itself.

Why CGO_ENABLED=0 matters

A cgo binary built on one Linux may depend on libc.so.6 being present and ABI-compatible on the target. That's usually fine, but it's an implicit runtime dependency you may not notice until deployment fails on a minimal OS, or an OS with incompatible libc versions, or inside a scratch container image with no libc. CGO_ENABLED=0 removes that dependency entirely. The binary becomes fully static and runs on any Linux system regardless of what libc is installed.

Conclusion

Go's pure-Go implementations are not always full replacements. The pure-Go DNS resolver doesn't support all NSS plugins, so environments relying on custom NSS modules for service discovery or LDAP may see different behavior. For standard DNS over /etc/resolv.conf, it works fine.

Wherever the stdlib has both a cgo and a pure-Go path, CGO_ENABLED=0 at build time opts into the pure-Go one. You get a simpler, more portable binary at the cost of potentially narrower feature coverage in those specific areas. Just remember the choice is made when you run go build, not when the binary runs on the target machine.