DEV Community

Cover image for Linux File System Hunting: What Your OS Is Hiding in Plain Sight
Janmejai Singh
Janmejai Singh

Posted on

Linux File System Hunting: What Your OS Is Hiding in Plain Sight

#security #devops #linux #sysadmin

Stop running commands. Start reading the OS like a story.

Most engineers learn Linux by memorizing commands. But the real insight comes when you stop asking "what command does X?" and start asking "why does this file exist at all?"

This post is a field report from a deep dive into the Linux file system — not a command tutorial, but an investigation. Here's what I found, why it matters, and what it reveals about how Linux actually thinks.

1. /etc/resolv.conf — The DNS Brain of Your Machine

When your system needs to resolve google.com, it doesn't magically know where to look. It opens /etc/resolv.conf.

cat /etc/resolv.conf
Enter fullscreen mode Exit fullscreen mode 
nameserver 127.0.0.53
options edns0 trust-ad
search home
Enter fullscreen mode Exit fullscreen mode

What's actually happening here?

On modern Ubuntu/Debian systems, the nameserver points to 127.0.0.53 — a loopback address handled by systemd-resolved, not a real external DNS server. This is a stub resolver. Your queries hit systemd-resolved first, which checks its cache, applies policies, and only then forwards upstream.

Why this matters for engineers: If you're debugging DNS failures inside a container or a VPN, this file is the first place to look — but remember, editing it directly on systemd-resolved systems often gets overwritten. The real config lives in /etc/systemd/resolved.conf.

Insight: Linux separates "who handles DNS" from "where DNS queries go". That layering enables caching, split-horizon DNS, and link-specific resolvers. Many DNS bugs in containerized apps trace back to not understanding this indirection.

2. /proc/net/route — Your Routing Table, Unfiltered

Everyone uses ip route to see routes. But that command is just reading from a file:

cat /proc/net/route
Enter fullscreen mode Exit fullscreen mode 
Iface   Destination Gateway  Flags RefCnt Use Metric Mask     MTU Window IRTT
eth0    00000000    0101A8C0  0003  0      0   100    00000000 0   0      0
eth0    0001A8C0    00000000  0001  0      0   100    00FFFFFF 0   0      0
Enter fullscreen mode Exit fullscreen mode

The values are in little-endian hexadecimal. Decoding 0101A8C0: reverse the byte pairs → C0.A8.01.01 → 192.168.1.1. That's your default gateway.

Why this exists: The kernel exposes routing decisions through /proc as virtual files — no database, no daemon needed. Tools like ip, netstat, and route are essentially pretty-printers for this raw data.

Insight: The kernel routing table isn't stored anywhere on disk — it's computed in memory and exposed live through /proc. You're always reading the present state of the kernel's mind.

3. /etc/passwd and /etc/shadow — The Auth Split That Saved Linux Security 

cat /etc/passwd | head -3
Enter fullscreen mode Exit fullscreen mode 
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
Enter fullscreen mode Exit fullscreen mode

Notice the x in the password field? That x is a pointer — it means "go look in /etc/shadow".

sudo cat /etc/shadow | head -1
Enter fullscreen mode Exit fullscreen mode 
root:$6$rounds=5000$xyz...:19200:0:99999:7:::
Enter fullscreen mode Exit fullscreen mode

The hash prefix $6$ means SHA-512. The rounds= value controls iteration count — the computational cost of cracking.

Why the split exists: Originally, /etc/passwd held password hashes directly — but it had to be world-readable so programs could look up usernames. This meant anyone on the system could grab hashes and crack them offline. The fix: move hashes to /etc/shadow, readable only by root.

Insight: This is a textbook least-privilege design. Readable metadata in one file, sensitive secrets in another, permissions separating them. Vault, AWS SSM, and Kubernetes Secrets all follow the exact same principle.

4. /proc/PID/fd/ — Every File Your Process Has Open

Pick any running process and inspect it live:

ls -la /proc/$$/fd
Enter fullscreen mode Exit fullscreen mode 
lrwxrwxrwx 1 user user 64 Apr 20 10:01 0 -> /dev/pts/0
lrwxrwxrwx 1 user user 64 Apr 20 10:01 1 -> /dev/pts/0
lrwxrwxrwx 1 user user 64 Apr 20 10:01 2 -> /dev/pts/0
Enter fullscreen mode Exit fullscreen mode

File descriptors 0, 1, 2 are stdin, stdout, and stderr — symlinks to the terminal device.

Now try it on a running service:

ls -la /proc/$(pgrep nginx | head -1)/fd
Enter fullscreen mode Exit fullscreen mode

You'll see open sockets, config files, log files, and PID files — everything that process is currently touching.

Why this matters: This is exactly how lsof and strace work under the hood. In incident response, checking /proc/PID/fd can reveal if a process is holding onto a deleted file (keeping disk space locked) or has unexpected outbound connections open.

5. /etc/hosts and /etc/nsswitch.conf — DNS That Predates DNS 

cat /etc/hosts
Enter fullscreen mode Exit fullscreen mode 
127.0.0.1   localhost
127.0.1.1   myhostname
::1         localhost ip6-localhost ip6-loopback
Enter fullscreen mode Exit fullscreen mode

This file is a relic from the ARPANET era — before DNS existed, machines shared a central HOSTS.TXT file. Today it's still consulted before DNS, governed by /etc/nsswitch.conf:

grep hosts /etc/nsswitch.conf
# hosts: files mdns4_minimal [NOTFOUND=return] dns
Enter fullscreen mode Exit fullscreen mode

files comes first — meaning /etc/hosts wins over any DNS server.

Why engineers should care: Kubernetes, Docker, and many CI environments inject hostnames via this file. Understanding the lookup order explains why a DNS change doesn't take effect but a hosts file entry does — and why service resolution differs inside and outside a container.

6. /etc/systemd/system/ — Where Services Are Born 

cat /lib/systemd/system/ssh.service
Enter fullscreen mode Exit fullscreen mode 
[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

After= defines boot ordering. Restart=on-failure means systemd auto-restarts on crash. WantedBy=multi-user.target determines which boot stage activates the service.

The key insight: Everything in systemd is a declarative dependency graph — not a sequential boot script. The OS calculates start order at runtime from After= and Wants= relationships across all unit files. This is fundamentally different from old SysV init where order was hardcoded line by line.

7. /var/log/auth.log — Your Machine's Security Diary 

sudo grep "Failed password" /var/log/auth.log | tail -5
Enter fullscreen mode Exit fullscreen mode 
Apr 20 03:14:22 host sshd[4821]: Failed password for root from 185.220.101.x port 52344 ssh2
Apr 20 03:14:25 host sshd[4821]: Failed password for root from 185.220.101.x port 52391 ssh2
Enter fullscreen mode Exit fullscreen mode

If your machine is internet-facing, you almost certainly have hundreds of these every day — automated credential stuffing from bots scanning the entire IPv4 space.

What else lives here: Sudo usage, su attempts, PAM authentication events, and SSH key fingerprints from successful logins. Cross-referencing with /var/log/syslog gives you a full timeline of system behavior around any suspicious event.

Insight: Logs are only useful if you know what normal looks like. On a healthy server, SSH access comes from a handful of known IPs. Once you understand the baseline, anomalies are immediately obvious.

8. /dev/null, /dev/zero, /dev/urandom — The Kernel's Built-In Utilities

These aren't real files. They're character devices — kernel interfaces that behave like files.

# Silence all output
command 2>/dev/null

# Generate a 1MB file of zeros
dd if=/dev/zero of=zeros.bin bs=1M count=1

# Generate cryptographically secure random bytes
head -c 32 /dev/urandom | base64
Enter fullscreen mode Exit fullscreen mode
  • /dev/null — a black hole. Writes disappear. Reads return EOF immediately.
  • /dev/zero — infinite stream of zero bytes. Used for zeroing disks and memory.
  • /dev/urandom — reads entropy from the kernel's CSPRNG, seeded by hardware interrupts, disk I/O timing, and network events.

Why /dev/urandom matters more than you think: When your Node.js app calls crypto.randomBytes() or Python calls secrets.token_hex(), they ultimately read from /dev/urandom. It's the root of all cryptographic randomness on Linux — every TLS handshake, every UUID, every session token traces back here.

9. /proc/meminfo — Why "Free Memory" Is a Lie 

cat /proc/meminfo | grep -E "MemTotal|MemFree|Buffers|Cached|MemAvailable"
Enter fullscreen mode Exit fullscreen mode 
MemTotal:       16237568 kB
MemFree:          423108 kB
Buffers:          312440 kB
Cached:          5621700 kB
MemAvailable:    7840200 kB
Enter fullscreen mode Exit fullscreen mode

MemFree is nearly always the wrong number to watch. The kernel holds Buffers + Cached memory for disk I/O caching — it shows as "used" but is immediately reclaimable. MemAvailable is the real number.

This is why a freshly booted server shows "high memory usage" — the kernel is aggressively using idle RAM for caching, which is the correct behavior.

Also worth checking:

cat /proc/cpuinfo | grep "model name" | uniq
Enter fullscreen mode Exit fullscreen mode

On cloud VMs, CPU flags like vmx (Intel VT-x) or svm (AMD-V) reveal whether the machine supports hardware virtualization — useful for detecting nested virtualization environments.

10. /boot/grub/grub.cfg — The First Thing That Actually Runs

Before the kernel, before init, before systemd — GRUB runs. Its config:

sudo grep -A5 "menuentry" /boot/grub/grub.cfg | head -10
Enter fullscreen mode Exit fullscreen mode 
menuentry 'Ubuntu' {
    linux /boot/vmlinuz-6.5.0-generic root=UUID=xxxx ro quiet splash
    initrd /boot/initrd.img-6.5.0-generic
}
Enter fullscreen mode Exit fullscreen mode

The linux line passes parameters directly to the kernel. quiet splash suppresses boot messages. ro mounts root read-only initially (remounted read-write after filesystem checks). root=UUID= tells the kernel which block device is the root filesystem.

Why this matters: Kernel parameters like init=/bin/bash (drop to shell), nomodeset (fix GPU issues), or systemd.unit=rescue.target (recovery mode) are configured here. Understanding GRUB is what separates engineers who can recover a broken system from those who reinstall it.

The Pattern Behind All of This

After this investigation, one theme is unmistakable: Linux exposes almost everything as a file.

What you want to inspect Where it lives
Running processes /proc/PID/
Hardware devices /dev/
Live kernel settings /sys/
Network & system config /etc/
Service state /run/systemd/

This isn't just a design quirk — it's a philosophy. If it's a file, you can read it, watch it with inotify, pipe it through grep, version-control it with git, and audit it with standard Unix tools. No proprietary APIs, no special SDKs, no vendor lock-in.

The Linux file system isn't just where your files live. It's the kernel's window into itself.

Going Deeper

If this sparked your curiosity, explore further:

  • /sys/class/net/ — live network interface stats including dropped packets per interface
  • /etc/cron.d/ — scheduled tasks; a favourite persistence location for attackers
  • /proc/net/tcp — all active TCP connections in raw hex format
  • /etc/ld.so.conf — controls which directories the dynamic linker searches for shared libraries
  • /proc/sys/kernel/ — live-tunable kernel parameters (also writable via sysctl)

The more you read the filesystem, the more the OS reveals itself — not as a black box, but as a transparent, well-documented machine that's been explaining itself all along. You just have to know where to look.

Part of the Web Dev Cohort 2026 — Linux File System Hunting.

your-hashnode-username.hashnode.dev

Top comments (0)