DEV Community: Schiff Heimlich

Linux 7.2 cut pipe mutex contention — why your shell pipelines just got faster

Schiff Heimlich — Wed, 01 Jul 2026 17:04:58 +0000

A Meta engineer profiling caching code found something worth fixing in the kernel's pipe write path. The fix landed in Linux 7.2: anon_pipe_write now pre-allocates up to 8 pages before grabbing the lock, cutting the critical section down significantly.

What actually changed

Pipes use a ring buffer backed by anonymous pages. On write, the kernel would allocate pages under the same mutex used for the actual data copy — meaning every write had to wait for allocation, and allocation held the lock. Under load this caused measurable mutex contention.

The fix separates allocation from the critical section. If 8 pages are already pre-allocated and available, the write just copies data and updates the ring buffer pointer — no allocation, no lock held during the slow path.

Numbers

The gains depend on your workload. Meta's testing showed meaningful improvements under memory pressure — the kind of situation where page allocation itself becomes expensive. Under lighter load the difference is smaller, but still present since you're avoiding the allocation path entirely when the pre-allocated pages cover the write.

Why this matters for pipeline work

If you run anything that pipes data through shell utilities — log processing, build artifact transforms, text munging, any sort | uniq | awk chain — you benefit from this. Pipes are the fundamental I/O primitive underneath all of it. Reducing contention at this layer makes the whole chain a bit more predictable under concurrent load.

You don't need to do anything. This lands in your kernel update. But it's worth knowing why those cat bigfile | sort | head runs feel a touch snappier on a recent kernel — it's not just compiler optimizations, it's a genuine kernel path improvement.

Kernel 7.2 or later required. Check with uname -r.

The Microsoft UEFI CA from 2011 expired last week. Here's what to check.

Schiff Heimlich — Tue, 30 Jun 2026 17:03:02 +0000

The Microsoft UEFI CA 2011 quietly expired on June 27, 2026. If you're running anything with Secure Boot enabled, this is worth five minutes of your time.

What actually breaks

Third-party binaries that were signed only by that CA — things like option ROMs, older third-party bootloaders, or hardware firmware blobs — can fail Secure Boot validation. The machine may refuse to boot, or just silently skip what it can't verify.

The good news: most major Linux distros pushed dual-signed shim binaries in time. Debian, Ubuntu, Fedora — they're all covered. If you've been keeping up with updates, you're probably fine.

Who probably needs to act

Self-hosted bare metal with custom partitions or hardware RAID controllers that carry their own Option ROMs
Edge devices that haven't been touched in a while — routers, appliances, IoT gateways running older firmware
VMs on older hypervisors where the firmware blob hasn't been updated
Anything running Secure Boot on hardware that predates 2022

How to check

On a Linux system with Secure Boot enabled:

\`bash

Check what UEFI vars are loaded

cat /sys/firmware/efi/efivars/SecureBoot*

See what keys are enrolled

ls /sys/firmware/efi/efivars/ | grep -i db
`\

If you're seeing boot failures that coincide with June 27 or later, that's your culprit.

What to do

Update firmware on affected hardware — most vendors have pushed updated Option ROMs
Update your bootloader shim if you're on an older distro that missed the window — grab a recent signed shim from your distro's repos
Check the shim-review repo if you're a hardware vendor or maintain a custom bootloader: the new dual-signed shims are all documented there

The shim-review team processed 21 reviews in a few weeks to handle this. Most distros got it done. The outliers are the devices nobody has touched in three years.

Worth a quick audit if you manage anything outside the normal update cycle.

Your CI ran code it should not have, and you cannot prove it did not

Schiff Heimlich — Tue, 30 Jun 2026 04:55:09 +0000

A few weeks ago someone force-pushed 75 of 76 version tags in aquasecurity/trivy-action. Pipelines that had pinned to a tag — the thing we all tell people to do — pulled credential-stealing code on their next run. It read /proc//environ and sent secrets to a typosquat domain.

A few days later, two litellm releases on PyPI shipped a stealer in a .pth file. Python runs .pth files on startup. You did not have to import the package. If it touched the machine, the code already ran.

Both attacks had the same shape: CI ran code it had no reason to trust, with credentials it had no reason to hold, and afterward nobody could prove what actually executed.

The problem with "read the workflow file"

You can audit your CI workflow YAML. You can review the action source before you pin a version. But when the tag gets retagged after you pin, or when a dependency runs arbitrary code at import time, your workflow file does not show you what actually executed. There is no signed record.

This is the gap CI/Lock (cilock) tries to close. It wraps a CI command, traces what it actually does using ptrace (or eBPF if you prefer), and signs the evidence as an in-toto/DSSE attestation. The output is a signed document that says: here is what ran, here is what it read, here is what it produced.

cilock run -- go build -o app ./...
cilock verify ./app -p release.policy.signed -k policy.pub

The policy is signed by a human. The build agent is separate. This is the part that matters: an AI coding agent can run the build, gather the evidence, and draft a release — but it cannot sign the policy, so "the agent did it" is not provenance.

How it works in practice

You install cilock in your CI environment
You define a policy: what is allowed to ship, what is allowed to read
You sign that policy with your key (or use GitHub Actions OIDC for keyless signing)
You run your build through cilock run -- <command>
On release, anyone can verify: cilock verify <artifact> -p <policy> -k <pubkey>

The attestation records every file the process opens, the environment at build time, and the artifacts produced. If something later turns out to be malicious, you can compare what you think ran against what actually ran.

Why this is not just for big orgs

If you ship software — open source or internal — you are a potential supply chain target. The tooling here is not exotic. The in-toto framework has been around in the CNCF for years. CI/Lock is the practical layer that makes it usable in a normal CI pipeline without standing up a full PKI.

You do not need to trust that your CI environment is clean. You need a signed record of what happened, so you can verify rather than trust.

Source: https://news.ycombinator.com/item?id=48728596

The jq newline that silently corrupted my URLs

Schiff Heimlich — Thu, 25 Jun 2026 17:04:34 +0000

Ran into a fun one last week. Was debugging why some shortened URLs I was sharing kept returning 404 — turned out the links all ended with a stray "j" character.

Here's what happened. I have a small shell script that shortens URLs via an API and copies the result to my clipboard. Simplified version:

!/bin/sh

shirt=$(curl -s -X POST "$API" ...)
echo "$shirt" | jq -r '.short_url' | wl-copy

This worked fine for months. Then one evening people started telling me the links were broken. After staring at the messages long enough, I noticed every URL ended with "j".

Root cause: jq adds a newline after each output by default. When that gets pasted into an ANSI terminal, the newline character gets interpreted as "j". In my terminal it displayed fine — the clipboard had the actual link — but in Signal messages, the paste behavior was different and the "j" got embedded in the URL.

The fix is one flag:

echo "$shirt" | jq -j -r '.short_url' | wl-copy

jq -j (or --join-output) suppresses the trailing newline. That's it.

Two things worth remembering here:

jq outputs a newline by default — this is almost always what you want for human-readable output. When you're piping to another tool or storing the result programmatically, you usually want -j.
Clipboard tools and terminals have different paste semantics — what looks like a clean URL in your clipboard might carry extra characters depending on how it's pasted into different contexts.

This one's been in jq forever and I've hit it before. Still surprises me every few years.

Making HTTP requests from a container with no curl

Schiff Heimlich — Wed, 24 Jun 2026 17:02:41 +0000

You SSH into a minimal container image to debug something, and curl isn’t there. Neither is wget. You need to check if an endpoint is reachable or poke at an API, and you are stuck.

Bash can actually do TCP directly. It’s been able to since Bash 2.0-something.

The trick

exec 3<>/dev/tcp/example.com/80
echo -e "GET / HTTP/1.0\r\nHost: example.com\r\n\r\n" >&3
cat <&3

/dev/tcp/ is a bash-specific pseudo-device. Opening it sets up a TCP connection to the host and port you specify. The first line opens file descriptor 3 for read-write on that socket. You write your HTTP request to it, then read the response back.

For HTTPS you will need openssl s_client or similar, but for internal HTTP endpoints this gets you surprisingly far.

A reusable function

If you do this often enough, wrap it:

http_get() {
  local host="$1"
  local path="${2:-/}"
  exec 3<>/dev/tcp/${host}/80
  printf "GET %s HTTP/1.0\\r\\nHost: %s\\r\\n\\r\\n" "$path" "$host" >&3
  cat <&3
  exec 3<&-
}

http_get example.com /

When this matters

Minimal images for security scanning, single-purpose sidecars, or base images you are building from scratch — these often strip out anything not explicitly needed. If you are debugging a network issue inside one of those containers, you do not have to install curl just to test connectivity. Bash is usually there.

This is not a replacement for proper HTTP clients. But for the moment you need to check if something is listening, it saves a round-trip to install a tool.

You might not need Redis: a DB table and your filesystem do more than you think

Schiff Heimlich — Tue, 23 Jun 2026 17:04:24 +0000

Had yet another conversation yesterday with a team burning themselves on a Redis cache misconfiguration. Hot keys, thundering herd, memory policy oversights — the usual scars. One of the replies on the HN thread mentioned something that keeps showing up in these discussions: they just use a database table + filesystem, no Redis, no Memcached.

Sounded like a lazy joke. It isn't.

The setup

You already have a database. You already have a filesystem. You can build a perfectly functional caching layer with both.

The database table looks something like:

CREATE TABLE cache_entries (
  cache_key   TEXT PRIMARY KEY,
  expires_at  TIMESTAMPTZ,
  computing   BOOLEAN DEFAULT FALSE
);

The filesystem holds the actual cached values — call them /var/cache/myapp/<key>. Reads check the filesystem first; if the file is stale or missing, the DB is consulted.

What you actually get

Thundering herd protection. This is the part teams reach for Redis for, and it's the easiest win. Use a database row lock — SELECT FOR UPDATE — so that when a cache miss hits, only one process computes the result while others wait. Postgres handles this fine. The filesystem doesn't need to know about it.

Coordinated expiration. The DB holds the TTL. Workers can query expires_at before even touching the filesystem, so you don't serve stale content.

No extra daemon. No Redis process to babysit, no persistence configuration to get wrong, no memory limit to tune.

Persistence you actually control. The filesystem is on disk. If you want to survive restarts, you already have that. You don't have to configure appendfsync always and pray.

The part that bites

This is not Redis. If your cache layer needs to be shared across many application servers, you're back to a central store — the DB becomes that bottleneck and you haven't actually solved anything. This pattern works best on a single server or when the DB is already the shared coordinator anyway.

For high-TPS workloads (thousands of cache reads per second from many nodes) you'll still want something designed for the job. But most internal tooling, sidekiq-style job caching, and slow-query caching isn't that workload.

When it's worth considering

You're already running a database
Your cache is per-server or your DB is already shared
You want to reduce operational dependencies
You're building something that doesn't need Redis' data structures

The comment that stuck with me from the thread: "Using a db table as a k/v store + the FS can do so much before even considering paying the price of setting up a dedicated caching store."

He's right. The boring choice is boring for a reason.

IPv6 zone identifiers do not belong in URLs

Schiff Heimlich — Mon, 22 Jun 2026 17:02:58 +0000

Every now and then you run into something that breaks in a way that is hard to google because the error messages never mention the actual cause. IPv6 zone identifiers are one of those things.

What a zone identifier looks like

IPv6 link-local addresses are scoped to an interface. The same address can exist on multiple interfaces on the same machine, so you need to say which interface you mean. That is what the zone identifier does:

fe80::1%eth0
fe80::1%wlan0

The %eth0 part is the zone identifier. It is a local concept — it never leaves the machine.

The problem with URLs

RFC 3986 is explicit: the zone identifier is not part of a URL. When you write a URL, you stop at the address. Browsers know this and strip the zone identifier before doing anything with the URL.

But parsers, proxies, load balancers, and DNS tools do not always do the same thing.

Some libraries will decode %eth0 as part of the URL path and happily try to resolve a hostname that looks like fe80::1eth0. Others will pass it through unchanged and cause downstream failures in systems that do not expect it. A load balancer parsing an X-Forwarded-For header or a URL from a log file will occasionally hit something that looks like a zone identifier and behave unexpectedly.

Where this shows up in practice

If you are working with IPv6 link-local addresses — for example, accessing a service on a local machine via [fe80::1%eth0]:8080 — the browser handles it fine. But if you are writing a tool that processes URLs, or if you are debugging something that touches raw URL strings, zone identifiers are one of those edge cases that can silently produce wrong results.

The practical takeaway is simple: when you are writing URL parsing or HTTP request handling code, strip or normalize zone identifiers before processing. And if you ever see a lookup failure that does not make sense, checking for an unexpected % in what should be a clean address is worth 30 seconds of your time.

It is a niche problem, but it is the kind of niche that surfaces at 2am when you are trying to figure out why a health check is failing.

IPv6 zone identifiers in URLs are a quiet parser gotcha

Schiff Heimlich — Sun, 21 Jun 2026 17:03:51 +0000

You're debugging a URL parsing issue in your load balancer logs. Someone pasted a link like http://[fe80::1%eth0]:8080/ and your proxy choked on it.

That's the zone identifier — the %eth0 part. IPv6 link-local addresses need it to specify which interface to use. The problem is, zone identifiers don't actually belong in URLs, and different tools handle them very differently.

What zone identifiers are for

Link-local IPv6 addresses like fe80::1 are only usable on a single network segment. When you have multiple interfaces, the OS needs to know which one to send the packet out of. So you append %eth0 (or %1, %en0, whatever the interface name is) to disambiguate.

In code, this looks like:

import socket

# This is valid - bind to a specific interface
sock = socket.socket(socket.AF_INET6)
sock.bind(('fe80::1%eth0', 8080, 0, 0))

Why URLs are different

RFC 3986 is explicit: the zone identifier is not part of the URL syntax. It's a parsing artifact that only makes sense in address literals used by the local system.

Browsers know this. Chrome and Firefox strip zone identifiers before sending the URL to a server. But if you're writing a proxy, a URL parser, a DNS tool, or anything that processes URLs programmatically, you might be getting raw user input with zone identifiers still attached.

Where it breaks

URL parsers: stdlib urllib.parse in Python will happily parse [fe80::1%eth0] as the host, but whether it keeps or strips the zone depends on the library and version
Load balancers: HAProxy, nginx, Envoy all handle this differently in their URL parsing layers
DNS tools: Some dig or nslookup wrappers that accept URLs will fail or misinterpret the zone
Log parsers: If you're ingesting access logs and someone hotlinks a zone-scoped IPv6 address, your parser might silently drop the zone or fail to match the address

What to do about it

Normalize on input: Strip zone identifiers from IPv6 addresses in URLs before processing, unless your code is actually doing local interface binding
Be consistent: If you're storing or comparing addresses, normalize them the same way every time
Check your tools: Review how your proxy, load balancer, or any URL-processing code handles link-local IPv6 addresses

The short version

Zone identifiers are for local interface routing, not for URLs. Browsers strip them. Your parsers might not. If you're building anything that touches IPv6 URLs, this is one of those details that will bite you once before you remember to handle it.

Not groundbreaking, but if you've ever spent an hour wondering why a link-local address won't bind or why a URL won't parse, this is probably why.

The queue probability drops faster than you think when you add servers

Schiff Heimlich — Sat, 20 Jun 2026 17:03:41 +0000

A load-balanced system with more servers gives you better latency not just because you have more capacity, but because the queuing math has a shape that surprises most people.

I ran into this recently when someone asked me to walk through an M/M/c queue model. The setup: c servers, each processing one request at a time, with a load balancer holding an infinite queue. If you're offering c * 0.8 requests per second (80% utilization per server), what happens to mean request latency as you increase c?

Most people expect latency to improve slowly, or maybe linearly. The actual result is that latency drops faster than your intuition suggests, especially in the tail.

Here's the specific finding that stuck with me. At 5 servers and half-saturation (offering load equal to half what the servers can handle), about 13% of requests end up in queue. Double the servers to 10, keep the per-server load constant at 80%, and that queuing probability drops to 3.6%. Double again to 20 servers, and you're effectively at zero queuing at half load.

That's not a linear relationship. The queuing probability is a function of the Erlang C formula, and its shape is convex in a way that favors scale.

The practical takeaway isn't "add more servers." It's that load balancer latency is often not the right thing to optimize when you have headroom. If your services are sitting at 60-70% utilization, adding capacity will reduce queuing more than you'd predict from simple capacity math.

The flip side: if you're already running lean (85%+ utilization), adding a couple servers won't move the needle much. You're in the steep part of the curve where queuing probability climbs quickly.

This is a reminder that queue theory has real teeth in distributed systems design. The counterintuitive behavior is worth knowing when you're sizing clusters or debugging latency spikes that seem out of proportion with load.

Source: Marcus Brooks (brooker.co.za) on M/M/c queuing systems and Erlang's C formula.

Homebrew 6.0 sandbox: what the systemd confinement actually does

Schiff Heimlich — Fri, 19 Jun 2026 17:04:28 +0000

Homebrew 6.0 shipped a Linux sandbox. Here's what that actually means in practice.

The short version

The sandbox isn't containers. It's systemd sleep confinement applied per-formula at install/run time. When a formula runs, systemd places it in a cgroup slice with restricted access to filesystem paths, syscall capabilities, and device nodes. If the formula tries to write somewhere it shouldn't, the kernel enforces it at the cgroup level — not at the container boundary.

Why this matters for dev environments

On a dev workstation or shared Linux build box, Homebrew installs run under the same user context as everything else. A buggy or malicious formula can overwrite your dotfiles, read SSH keys if the agent is running, or trash /usr/local if permissions allow. The sandbox doesn't eliminate that risk, but it limits the blast radius.

Specifically, the confinement:

Denies access to paths outside the Homebrew prefix unless explicitly allowlisted
Drops capabilities like CAP_SYS_ADMIN that aren't needed for most formula builds
Restricts device access so the formula can't probe /dev

What doesn't change

This isn't containerisation. There's no namespace isolation, no separate mount table, no seccomp filter applied by default. The sandbox constrains what the process can do through cgroup v2 and capability bounding, but a determined formula running as your user can still cause plenty of damage inside those limits.

Also worth noting: this only applies on Linux. macOS Homebrew still relies on SIP and the normal Unix permission model.

Checking if a formula is sandboxed

You can inspect the cgroup a formula is running in after install:

# Find the PID of a running formula
pgrep -f <formula-name>

# Check its cgroup slice
cat /proc/<PID>/cgroup

If Homebrew's sandbox is active, the process will be under a homebrew.sandbox slice rather than the default user slice.

If you need to opt out

Some formulae legitimately need wider access — building kernel modules, probing hardware, etc. You can bypass the sandbox per-install:

HOMEBREW_NO_SANDBOX=1 brew install <formula>

Just be deliberate about when you do this. The sandbox exists precisely because a formula you didn't write is executing arbitrary code on your system.

The practical takeaway

6.0 moves Homebrew's threat model in the right direction for shared Linux environments. It's not a substitute for isolation tools when you need hard boundaries, but it's a sensible default that raises the bar without the operational overhead of containers. If you're managing multi-user Linux build machines, test it against your common formulae before rolling it out broadly — some build systems make assumptions about what they can access that conflict with the new restrictions.

Batch-converting documents to markdown with Microsoft's markitdown

Schiff Heimlich — Thu, 18 Jun 2026 17:01:26 +0000

Here's a quick tool that landed in my queue recently: microsoft/markitdown

It's a Python CLI that converts PDFs, Word docs, PowerPoint, and Excel files to Markdown. Not groundbreaking, but if you've ever had to process a folder of legacy documentation for a static site, you know the value of not doing it manually.

Two things I found useful:

Batch conversion with piping

markitdown --input document.docx --output converted/

You can point it at a directory and it processes everything in one shot. Combine with standard Unix tools:

find ./legacy-docs -name '*.docx' | xargs -I{} sh -c 'markitdown --input {} --output ./md/'

stdout output for scripting

markitdown document.pdf

Dumps the markdown to stdout, which makes it easy to pipe into other text processing or redirect to specific filenames based on the input.

It's on PyPI (pip install markitdown), so it'll drop into a CI pipeline without much friction. If you've got a documentation migration on your plate and you're tired of manual conversions, it's worth a look.

https://github.com/microsoft/markitdown

Systemd timer units: two things cron still cant do

Schiff Heimlich — Wed, 17 Jun 2026 17:02:53 +0000

Every time I see a cron tab I wonder why nobody reached for systemd timers. Cron works fine until it doesnt, and by then youre already in a hole.

Here are the two things that always bite us.

1. Your cron PATH is a coin flip

Cron runs everything with a stripped down environment. PATH is usually /usr/bin:/bin. So when you write a cron job that calls vault or python3 or anything not in that short list, it silently fails or runs the wrong binary.

Systemd services inherit the full environment from the service manager. If it works in your shell it works in your timer.

With cron:

0 2 * * * backup.sh  # fails because backup.sh calls vault not /usr/bin/vault

With a systemd timer, your PATH is what you expect.

2. Cron has no concept of a run completing

You set a schedule. Cron fires the job. If the job is already running, cron fires another one anyway. You end up with five backup scripts running simultaneously because the previous one was slow.

Systemd timers have AccuracySec= and you can set Unit=backup.service with RefuseManualStop=no and the service itself just handles one execution at a time. Or you use Persistent=true to catch up on missed runs after a reboot.

A minimal working example

# /etc/systemd/system/nightly-backup.timer
[Timer]
OnCalendar=2026-01-01 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/nightly-backup.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

Enable with systemctl enable --now nightly-backup.timer. Check next run with systemctl list-timers.

Logs go straight to journald. No more hunting for cron output in mail.

Cron is fine for simple stuff. But when your scheduled job touches production systems, the systemd approach gives you control that cron simply cant match.