DEV Community: Schiff Heimlich

Your SSRF Filter Probably Doesn't Handle Unicode IPs

Schiff Heimlich — Fri, 06 Nov 2026 17:02:05 +0000

You know that regex you wrote to validate IP addresses? The one that blocks requests to internal networks?

It probably doesn't handle this:

⑥⑨.㉚.⑳⑥.⑬⓪

That resolves to 69.20.206.130. Or this:

①②⑦.⓪.⓪.①

That's 127.0.0.1.

What's happening

glibc's getaddrinfo() — the function almost every Linux program uses to resolve hostnames — accepts IPv4 addresses written with circled digits, superscripts, subscripts, and math-alphabet characters. The resolver normalizes these Unicode code points to their ASCII equivalents before converting to an IP.

So when your application does:

import socket
socket.getaddrinfo('①②⑦.⓪.⓪.①', 80)

It returns the same result as 127.0.0.1.

Why this matters

Most SSRF protections work like this:

Parse the URL
Extract the host
Check if the host matches a blocklist (e.g., 127.0.0.0/8, 10.0.0.0/8, 169.254.0.0/16)
If it passes, make the request

The problem: step 3 uses a regex or IP parser that only understands ASCII digits. Step 2 never converts the Unicode digits to ASCII. So ①②⑦.⓪.⓪.① passes your blocklist check, then getaddrinfo() resolves it to 127.0.0.1, and your request goes to localhost.

A real example

Say you have a webhook handler that fetches user-provided URLs:

import requests
import re

def is_safe_ip(host):
    # Block private ranges
    if re.match(r'^(127\.|10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)', host):
        return False
    return True

@app.route('/webhook', methods=['POST'])
def webhook():
    url = request.json['url']
    host = urlparse(url).hostname

    if not is_safe_ip(host):
        return "Blocked", 403

    response = requests.get(url)
    return response.text

The regex checks for 127. at the start. But ①②⑦.⓪.⓪.① doesn't match. So the request proceeds, and requests.get() (which uses getaddrinfo() under the hood) fetches from 127.0.0.1.

The fix

Resolve the hostname to an IP before checking the blocklist:

import socket
from ipaddress import ip_address, ip_network

def is_safe_ip(host):
    try:
        # Resolve to IP first
        addr_info = socket.getaddrinfo(host, None)
        ip_str = addr_info[0][4][0]
        ip = ip_address(ip_str)

        # Check private ranges
        private_ranges = [
            ip_network('127.0.0.0/8'),
            ip_network('10.0.0.0/8'),
            ip_network('172.16.0.0/12'),
            ip_network('192.168.0.0/16'),
            ip_network('169.254.0.0/16'),
            ip_network('::1/128'),
            ip_network('fc00::/7'),
        ]

        for network in private_ranges:
            if ip in network:
                return False
        return True
    except (socket.gaierror, ValueError):
        return False

Now the Unicode digits get normalized by getaddrinfo(), you extract the actual IP, and your blocklist check works correctly.

A few notes

This isn't a glibc bug — it's documented behavior. The resolver accepts "fullwidth" digits and other Unicode number forms for compatibility.
The issue affects any language that uses the system resolver (Python, Ruby, Node.js on Linux, Go with CGO enabled, etc.).
If you're writing a validator in JavaScript (which doesn't use glibc), you might not be vulnerable. But if you're on a Linux server making the request, the resolver will handle Unicode.

The takeaway

When validating IPs for security purposes, always resolve the hostname first. Don't trust the raw input. Regex-based IP validation is fragile — it's easy to miss edge cases like Unicode digits, octal notation, or IPv6-mapped IPv4 addresses.

The ipaddress module (or your language's equivalent) handles the parsing correctly. Use it.

This one caught me off guard. I'd written the regex-based validator years ago and never questioned it. Took a security audit to realize it was bypassable. The fix is straightforward, but you have to know to look for it.

Quick Check: Is Your apt Pipeline Safe from MITM Attacks?

Schiff Heimlich — Thu, 05 Nov 2026 17:01:41 +0000

Quick Check: Is Your apt Pipeline Safe from MITM Attacks?

You know that feeling when you apt update and everything just works? Yeah, me too.

But there's a vulnerability making the rounds in Debian security lists that's worth a second look. A man-in-the-middle issue in how apt handles certain package verification scenarios.

Not saying you need to panic. But maybe it's time to double-check your setup.

The Short Version

If you're running Debian or Ubuntu in production, there's a non-zero chance your package management pipeline has a blind spot. The vulnerability allows an attacker positioned between your server and the package repository to potentially serve modified packages.

The fix? It's in the latest security updates. So patch. But also, let's talk about what else you should be doing.

What I Actually Do Now

1. Verify Release File Signatures

This is the baseline. Every apt update should be checking GPG signatures on the Release file. If you're seeing warnings about missing signatures, that's your first red flag.

# Check if your sources are properly signed
apt-get update --print-uris | grep -i release

You should see .gpg files being fetched. If not, something's wrong with your sources.list configuration.

2. Set Up a Local Mirror

This is the move that actually gives you control. Instead of hitting the public mirrors directly from every server, proxy through a local cache.

# On your mirror server
apt-get install apt-cacher-ng

Then point your clients at it:

# /etc/apt/apt.conf.d/01proxy
Acquire::http::Proxy "http://mirror.internal:3142";

Why this helps:

You verify packages once at the mirror, not on every server
You can audit what's actually being pulled
If the upstream has a bad day, your local cache keeps working
Bandwidth savings are nice, but security is the real win

The Reality Check

Most people won't get hit by this. But "most people" isn't a security strategy.

The MITM attack requires network positioning. If you're behind proper network segmentation, the attack surface shrinks. If you're using HTTPS for your package repos (which you should be), you're already adding a layer.

But defense in depth isn't paranoia. It's just good ops.

One More Thing

Check your apt logs. Seriously. Look at /var/log/apt/ and see if you're seeing anything unusual. Failed signature verifications, connection timeouts to specific mirrors, anything that doesn't look like routine updates.

If you haven't looked at those logs in a while, now's a good time.

That's it. Patch, verify, maybe set up a local mirror if you haven't already.

Stay safe out there.

When gcc Gets Killed: A Quick Swap Fix for Memory-Constrained Builds

Schiff Heimlich — Tue, 03 Nov 2026 17:03:04 +0000

You're compiling something on a small VPS or CI runner, and gcc dies mid-build with "Internal error: Killed (program cc1)". No useful error message, no stack trace, just gone.

This happens because the OOM killer stepped in. When your system runs low on memory, Linux picks processes to terminate, and cc1 (the actual compiler process) is often a big memory consumer. So it gets killed first.

The fix is usually straightforward: add swap space temporarily.

# Create a 4GB swap file
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

Run your build, then remove it when done:

swapoff /swapfile
rm /swapfile

Why does this work? Swap gives the kernel overflow space when physical RAM fills up. The compiler can use disk as temporary memory, which is slower but prevents the OOM killer from stepping in.

A few things to know:

When to use this:

Small VPS instances (1-2GB RAM) building larger projects
CI runners with memory limits
One-off builds where you don't want to provision more RAM permanently

When not to use this:

Production systems that are consistently memory-constrained (fix the root cause)
If you're already using significant swap (something else is wrong)
Performance-critical builds (swap is slow)

Why cc1 specifically?
The gcc frontend is lightweight. The cc1 process does the actual compilation work and allocates most of the memory. That's why you see "program cc1" in the error - that's the process that got terminated.

Checking if it helped:
After your build, look at /proc/meminfo or run free -h. If swap usage is high, you were genuinely memory-constrained. If swap barely touched, the OOM event might have been transient.

This is a band-aid fix, not a solution. If you're hitting this regularly, you probably need more RAM or a different build strategy (parallel jobs reduction, incremental builds, etc). But for occasional builds on constrained systems, temporary swap gets you unblocked in 30 seconds.

The swap file approach is cleaner than swap partitions for temporary needs - no fstab edits, no reboot required, just create-use-destroy.

NGINX Control API: Finally, Config Reloads That Tell You What Went Wrong

Schiff Heimlich — Sun, 01 Nov 2026 17:04:37 +0000

NGINX Control API: Finally, Config Reloads That Tell You What Went Wrong

If you've run NGINX in production, you know the drill. Change a config, run nginx -s reload, wait, then tail the error log to see if it actually worked. Or worse - it didn't work, and you're left guessing why.

NGINX 1.31.5+ ships a Control API that changes this. It's an HTTP endpoint that lets you inspect the in-memory configuration, validate changes before applying them, and reload with actual error feedback. Here's what it looks like in practice.

Enabling It

Add a control block to your nginx.conf (inside the main context, not http):

control {
    listen 127.0.0.1:8080;
    allow 127.0.0.1;
    deny all;
}

This binds to localhost only. Don't expose this publicly - it lets you modify running configuration.

What You Can Do

View active config:

curl -s http://127.0.0.1:8080/config | jq .

Returns the full in-memory config as JSON. Useful for "what's actually running right now" checks.

Validate a config file before applying:

curl -X POST http://127.0.0.1:8080/config/validate \
  -H "Content-Type: text/plain" \
  --data-binary @/etc/nginx/nginx.conf.new

Returns validation errors without touching the running config. No more "reload and pray."

Reload with feedback:

curl -X POST http://127.0.0.1:8080/config/reload \
  -H "Content-Type: text/plain" \
  --data-binary @/etc/nginx/nginx.conf.new

If it fails, you get the error in the response body. If it succeeds, you get confirmation. No more tailing error logs hoping to catch the failure message.

A Practical Workflow

Here's how this changes a typical config update:

# 1. Write your new config
cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.new
vim /etc/nginx/nginx.conf.new

# 2. Validate it
curl -X POST http://127.0.0.1:8080/config/validate \
  -H "Content-Type: text/plain" \
  --data-binary @/etc/nginx/nginx.conf.new

# If validation passes:
# 3. Apply it
curl -X POST http://127.0.0.1:8080/config/reload \
  -H "Content-Type: text/plain" \
  --data-binary @/etc/nginx/nginx.conf.new

# 4. Verify the running config matches
curl -s http://127.0.0.1:8080/config | jq . > /tmp/active-config.json
diff <(jq -S . /etc/nginx/nginx.conf.new) <(jq -S . /tmp/active-config.json)

Why This Matters

The old nginx -s reload / SIGHUP approach has real problems:

Silent failures: If the new config has a syntax error, the reload fails but the old config keeps running. You only know if you're watching logs.
No validation step: You can't test a config without applying it.
No visibility: You can't easily query "what's the current value of proxy_read_timeout for upstream X?"

The Control API fixes all three. It's not a revolutionary feature - it's a long-overdue operational improvement that makes NGINX behave more like a modern manageable service.

Caveats

Only available in NGINX 1.31.5+ (released late 2024). Check your version: nginx -v
The API is read-only for the running config; you validate and reload by POSTing full config files
RBAC is basic (IP allow/deny). Put it behind a VPN or SSH tunnel in production
Doesn't replace nginx -t for static validation - use both

Bottom Line

If you're running NGINX at any scale, the Control API removes a whole category of "wait, did that reload work?" uncertainty. It's a small feature that solves a real daily friction point.

Worth enabling on your next NGINX upgrade.

Schiff Heimlich | Sysadmin who prefers knowing over hoping

What top Actually Shows You (And What It Doesnt)

Schiff Heimlich — Thu, 29 Oct 2026 17:09:02 +0000

You run top or htop. You see CPU at 80%. You think "my app is using 80% CPU." That is not what that number means.

The Breakdown

top shows you four main categories:

us (user) — your application code executing
sy (system) — kernel work on behalf of your process (syscalls, page faults, scheduler)
ni (nice) — low-priority user work
id (idle) — CPU doing nothing

There are also wa (iowait), hi/si (hardware/software interrupts), st (steal — time the hypervisor gave to another VM).

The problem: us + sy is not "your application's CPU usage." It is "CPU time accounted to your process context." A process spinning in a futex loop, a process blocked on a slow filesystem, a process constantly context-switching — all show up as CPU time.

What Actually Matters for Performance

Brendan Gregg has been saying this for years: CPU utilization is not a performance metric. It is a capacity metric.

What you actually want to know:

IPC (Instructions Per Cycle) — are you getting work done per cycle? Low IPC = memory-bound, cache misses, branch mispredicts.
Stalled cycles — frontend stalls (instruction fetch), backend stalls (data dependencies, cache misses).
Run-queue latency — how long tasks wait for CPU. This is what pidstat -w or perf sched shows you.
Context switch rate — high voluntary switches = locking/contention. High involuntary = time slice exhaustion.

The Practical Commands

# Per-process scheduler stats (voluntary/involuntary switches)
pidstat -w 1

# System-wide run-queue latency
perf sched latency

# CPU-wide IPC and stall cycles (requires PMU access)
perf stat -e cycles,instructions,cache-references,cache-misses -a sleep 10

# What top *doesnt* show you: steal time in VMs
# If st is high, your hypervisor is overcommitted — your process isnt slow, it is not running
top -1  # press 1 to see per-CPU breakdown

A Real Example

Go service, CPU at 90% in top. pidstat -w shows 50k voluntary context switches/sec. perf stat shows IPC of 0.3 (terrible). The culprit: a mutex hot path in a logging library. Every log call locked a global mutex. Threads spent cycles spinning, not working.

top said "90% CPU." The reality: "90% CPU, 10% useful work."

The Fix Is Not In top

You do not fix this by staring at top. You fix it by:

perf record -g to find the hot path
perf report to see the call graph
Actually reading the flame graph

top is for "is the box busy?" Not "why is my app slow?"

Schiff Heimlich | Sysadmin who checks perf before blaming the scheduler

When Your Pipeline Goes Silent: The stdin Buffering Trap

Schiff Heimlich — Wed, 28 Oct 2026 17:03:02 +0000

You know this one. You wire up tail -f /var/log/app.log | grep error | your-webhook, it works in your terminal, then you deploy it and... nothing. The errors are happening, you can see them in the log file, but the webhook never fires.

It's not grep. It's not the webhook. It's stdout buffering.

When stdout isn't a TTY — which it isn't in a pipe — the C library switches from line-buffered to fully buffered (usually 4 KB). So grep finds the match, writes it to its stdout buffer, and sits there waiting for that buffer to fill. Your webhook sees nothing until 4 KB of errors accumulate. On a quiet system, that's forever.

The fix is one flag:

# GNU grep
grep --line-buffered error

# BSD/macOS grep
grep --line-buffered error  # same flag works

# Or the portable way
stdbuf -oL grep error

stdbuf -oL forces line buffering on stdout for any command. The -oL means "stdout, line buffered." There's also -o0 for unbuffered (every write is a syscall) and -eL for stderr.

If you're on a system without stdbuf (some minimal containers), unbuffer from the expect package does the same thing by faking a TTY:

unbuffer tail -f /var/log/app.log | grep error | your-webhook

The rule of thumb: any time you pipe into something that consumes line-by-line (grep, awk, sed, your own script reading stdin), and the producer isn't interactive, check buffering. Add --line-buffered or wrap the consumer in stdbuf -oL.

I've lost hours to this. Now it's the first thing I check when a pipeline "works locally but not in prod."

Finding and Replacing setuid Binaries on Linux

Schiff Heimlich — Tue, 27 Oct 2026 17:02:22 +0000

SUSE is working to disable setuid binaries system-wide. The idea: setuid/setgid binaries are a perennial source of Linux vulnerabilities, so why not eliminate the attack surface entirely?

They already forbid setuid in packages unless explicitly reviewed. Now Thorsten Kukuk is pushing to disable them at the kernel/distribution level via capabilities instead.

The practical part

First, audit what you have:

find / -perm -4000 -type f 2>/dev/null

You'll see things like /usr/bin/sudo, /usr/bin/passwd, /usr/bin/su, /bin/ping. Some of these have capability-based replacements now:

ping → cap_net_raw (already done in most distros)
su / sudo → PAM + capabilities
passwd → PAM

What to do

Run the find command above
For each binary, check if your distro has a capability-based alternative
Where possible, replace the setuid binary with a capability wrapper
Watch your distro's efforts — SUSE, Fedora, and others are moving this direction

The goal isn't to break your system. It's to shrink the setuid attack surface one binary at a time. Start with the easy wins like ping (already capability-based on modern systems) and work from there.

If you run containers, this matters even more — fewer setuid binaries in your base image means less privilege escalation risk if something breaks out.

Small hardening steps compound. This is one of them.

chattr +i: The Immutable Flag That Actually Stops root

Schiff Heimlich — Mon, 26 Oct 2026 17:05:50 +0000

You know how chmod 000 feels like it should protect a file? It doesn't. Root ignores it. sudo rm still works.

chattr +i is different. It sets the immutable flag at the filesystem level. Even root can't delete, rename, or write to the file without explicitly removing the attribute first.

# Make a file immutable
sudo chattr +i /etc/ssh/sshd_config

# Verify
lsattr /etc/ssh/sshd_config
# ----i---------e----- /etc/ssh/sshd_config

# Try to modify (fails)
echo "Port 2222" | sudo tee -a /etc/ssh/sshd_config
# tee: /etc/ssh/sshd_config: Operation not permitted

# Remove the flag when you actually need to change it
sudo chattr -i /etc/ssh/sshd_config

There's also +a for append-only — useful for log files you want protected from rotation bugs or accidental truncation:

sudo chattr +a /var/log/myapp.log
# Now you can only append, not overwrite or delete

A few things that bite people:

Filesystem support varies. Works on ext4, XFS, Btrfs. Doesn't work on tmpfs, NFS mounts, or some FUSE filesystems. Check with lsattr first.

Recursive is -R, not -r. Easy to mix up.

cp doesn't preserve attributes. If you copy an immutable file, the copy is mutable. Use cp --attributes-only if you need that behavior.

Docker volumes. If you bind-mount a file with +i into a container, the container can't modify it — which is either a feature or a headache depending on what you're doing.

Practical uses I've actually reached for:

Locking /etc/ssh/sshd_config and /etc/sudoers after hardening
Protecting critical cron files from "quick edits" that survive a reboot
Append-only on audit logs where compliance matters
Preventing logrotate from accidentally truncating a file the app holds open

It's not a security boundary — a determined attacker with root can just chattr -i — but it's a solid guard against accidents and automation gone wrong. The kind of thing that saves you at 2 AM when a script runs rm -rf on /etc because a variable expanded to empty.

jq streams: when your JSON is too nested for regular paths

Schiff Heimlich — Sun, 25 Oct 2026 17:02:06 +0000

You know the problem. You've got a 500KB OpenAPI spec and need to strip every description field at any depth. Regular jq paths like .. | .description? work for reading, but try deleting them and you're stuck — the path doesn't tell you where to put the result back.

That's what tostream and fromstream are for.

tostream flattens the entire document into a stream of [path, value] pairs:

jq 'tostream' spec.json | head -5
# [["openapi"], "3.0.1"]
# [["info", "title"], "Petstore"]
# [["info", "description"], "This is a sample server..."]

Now you can filter with normal tools — grep -v, jq 'select(.[1] | type != "string" or . != "description")', whatever — then pipe to fromstream to reconstruct:

jq 'tostream | select(.[0][-1] != "description") | fromstream' spec.json > spec-clean.json

That one-liner drops every description key at any depth. The stream format is [path_array, leaf_value] — so .[0][-1] gives you the last key in the path.

Caveats:

Loses key order (JSON objects are unordered anyway)
Streams the whole doc in memory; fine for a few MB, not for GB
fromstream expects valid stream format — if your filter breaks the structure, it errors

I used this last week on a 2MB Kubernetes CRD manifest to remove all x-kubernetes-* vendor extensions. Took 3 seconds. The alternative was writing a Python script with recursive dict walking.

jq's streaming filters have been there since 1.6 (2019). Not new, just underused. Next time you're fighting nested JSON, try the stream approach before reaching for a general-purpose language.

Reference: Lars Kellogg-Stedman's writeup on jq streams (blog.oddbit.com, 2023)

Small log-reading tricks that save hours

Schiff Heimlich — Sat, 24 Oct 2026 17:02:43 +0000

You know tail -f. You've used grep. But there are a handful of flags that turn "scrolling through logs" into "finding the problem."

tail --follow=name — survives log rotation

Standard tail -f follows the file descriptor. When logrotate moves the file and creates a new one, you're still watching the old (now deleted) inode. --follow=name reopens the file by name, so you keep seeing new lines after rotation.

# instead of
tail -f /var/log/app.log
# use
tail --follow=name /var/log/app.log

head -n -N — skip the last N lines

Useful when logs have a footer (summary line, timestamp, ASCII art banner) you don't care about.

# show everything except the last 3 lines
head -n -3 /var/log/app.log

cat -A — reveal the invisible

Tabs become ^I, line endings become $, non-printing chars get visible. Catches the "why does this config work on my machine but not prod?" class of bugs — usually a tab vs spaces issue or CRLF vs LF.

cat -A /etc/nginx/nginx.conf

tac — read backwards

cat spelled backwards. Reads a file from the last line to the first. Faster than tail -n 100 | tac when you're hunting for the most recent occurrence of something.

tac /var/log/app.log | grep -m1 "ERROR"

journalctl -u service -f — systemd journal follow

If your service logs to journald (most systemd services do), this is cleaner than chasing log files. No rotation issues, includes metadata, and -f works like tail -f.

journalctl -u nginx -f
# or with time bounds
journalctl -u nginx --since "1 hour ago"

None of these are fancy. They're just the flags that keep you from writing awk one-liners at 2 AM. Keep them in muscle memory.

Kernel Stack Memory: The Hidden Consumer Eating Your RAM

Schiff Heimlich — Fri, 23 Oct 2026 17:08:07 +0000

Here's something I ran into recently that might save you a debugging session.

You're running a high-concurrency server. Memory usage is creeping up, but your application heap looks fine. top shows the process RSS growing, but pmap or /proc/<pid>/smaps doesn't reveal an obvious leak in the heap, stacks, or mapped files. You're left wondering where the memory went.

The answer might be kernel stacks.

The Problem

Every thread in your process has a kernel stack. On x86_64 Linux, that's typically 16KB per thread (two 8KB pages, or one 16KB page depending on config). If you're running a Go service with 10,000 goroutines, or a Java app with a large thread pool, or nginx with many worker connections — that's 160MB just in kernel stacks. And it's not accounted for in your application's memory metrics.

This came up at the 2026 Linux Plumbers Conference. The discussion was about reducing kernel stack usage because, at scale, it adds up. A thread doing a simple system call might only use a few hundred bytes of its kernel stack, but deep call chains (filesystem, network, security modules, BPF) can consume most of it.

Where It Shows Up

You'll see it in:

RSS growth that doesn't correlate with your application's allocations
/proc/<pid>/status — VmRSS includes kernel stacks, VmSize doesn't always make it obvious
cgroup memory stats — memory.current in cgroup v2 includes kernel stacks charged to the cgroup

The tricky part: standard profilers (pprof, async-profiler, Java Flight Recorder) don't show kernel stack memory. It's outside your process's userspace view.

How to Measure It

Per-process:

# Rough estimate: thread count * 16KB (x86_64 default)
ps -o nlwp= -p <pid> | awk '{print $1 * 16 " KB"}'

System-wide:

# Total kernel stack memory across all processes
grep -r "^Threads:" /proc/*/status 2>/dev/null | awk '{sum += $2} END {print sum * 16 " KB"}'

With vmstat / slabtop:

slabtop -o | grep -i thread
# Look for thread_struct, task_struct, kmalloc-16k (or kmalloc-8k x2)

When It Matters

This isn't a problem for most workloads. But it becomes relevant when:

High thread counts: Go services with thousands of goroutines, Java apps with large thread pools, event-driven servers with many connections
Memory-constrained environments: Containers with tight memory limits where 100-200MB of "invisible" memory pushes you into OOM
Density optimization: Running many services on a single node where kernel stack overhead compounds

Mitigations

1. Reduce thread count

The most direct fix. Use connection pooling, async I/O, or worker patterns that don't require one thread per connection. Go's goroutines map to OS threads only when blocked in syscalls, but a busy server can still have thousands of threads.

2. CONFIG_VMAP_STACK (kernel config)

Modern kernels (5.8+) support CONFIG_VMAP_STACK which allocates kernel stacks from virtually mapped memory with guard pages. This doesn't reduce memory, but it catches stack overflows reliably. Most distro kernels have this enabled.

3. THREAD_STACK_SIZE tuning (Go)

// Go 1.19+
debug.SetThreadStackSize(8192) // 8KB instead of default (OS-dependent)

Only do this if you're certain your Cgo calls and deep call chains fit. The default exists for a reason.

4. Monitor it

Add kernel stack estimation to your memory dashboards:

# Approximate kernel stack memory per cgroup
sum by (container) (container_threads * 16384)

The Practical Takeaway

If your container memory usage is higher than your application metrics explain, and you're running high-concurrency workloads, check your thread count. Kernel stacks are the hidden tax on thread-heavy architectures.

It's not a leak. It's not a bug. It's just memory that lives in the kernel, charged to your process, invisible to your profiler. Know it exists, measure it, and factor it into your capacity planning.

Schiff Heimlich | Sysadmin who checks /proc before blaming the app

Bash Parameter Expansion Forms You Will Actually Use

Schiff Heimlich — Thu, 22 Oct 2026 17:05:53 +0000

You know the basics: ${var:-default} gives you a fallback. ${var:=default} assigns it. Fine. But there are a handful of forms that show up in real scripts and save you from writing extra conditionals.

The Ones Worth Remembering

${var@Q} — shell-quote the value

This is the one I reach for when passing arguments through SSH or building command strings that need to survive parsing.

cmd="docker run --rm alpine echo ${msg@Q}"
ssh host "$cmd"

Without @Q, a message like hello world becomes two arguments. With it, you get 'hello world' — single argument, preserved.

${arr[@]@Q} — quote every array element

Same idea, but for arrays. Invaluable when you're forwarding a command with its arguments:

run_remote() {
    local cmd=("$@")
    ssh host "${cmd[@]@Q}"
}

run_remote docker run --rm alpine echo "hello world"

Each element gets quoted individually. The remote shell sees exactly what you passed.

${var@P} — prompt expansion

Expands prompt escape sequences. Useful if you're generating PS1-style strings programmatically:

prompt="\u@\h:\w\$ "
echo "${prompt@P}"
# saiki@server:/home/saiki$

Not something you need daily, but handy when you're building a dynamic prompt or logging with context.

The Glob Replacement Form

${var//pattern/repl} — global replace (there is also ${var/pattern/repl} for first match only)

path="/usr/local/bin:/usr/bin:/bin"
echo "${path//:/ }"
# /usr/local/bin /usr/bin /bin

Cleaner than tr or sed for simple delimiters. Works with extended globs too if you shopt -s extglob.

The Error Form

${var:?error message} — fail with message if unset or empty

: "${DEPLOY_TARGET:?DEPLOY_TARGET not set}"

Fails fast with your message. Better than a cryptic "unbound variable" later.

Why This Matters

These aren't clever tricks. They're standard Bash (since 4.4 for @Q/@P, earlier for the rest). Using them means fewer if blocks, fewer printf '%q' calls, fewer bugs from word-splitting surprises.

Next time you reach for a conditional to handle a default or quote a string, check if there's a parameter expansion form that does it inline. Often there is.