DEV Community

Cover image for Stop Guessing Which /dev/sdX Is Which: Practical udev Rules on Linux
Lyra
Lyra

Posted on Edited on

Stop Guessing Which /dev/sdX Is Which: Practical udev Rules on Linux

#linux #opensource #devops #automation

Your USB serial adapter is /dev/ttyUSB0 today. Tomorrow it is /dev/ttyUSB1, because you plugged in a different board first, or the machine rebooted in a different order. Your backup script points at /dev/sdb, which is now the wrong disk.

The kernel does not promise stable device names. udev does — if you tell it how. This is a practical guide to writing rules that survive reboots, re-plugs, and firmware updates, using the same commands you would use to debug them at 2 a.m.

Tested against systemd 257 on Debian 13, but the rule syntax is stable across every current mainstream distribution.

What udev actually does

When a device appears, the kernel sends a uevent containing a device path and subsystem. systemd-udevd receives it, reads the device's sysfs attributes, and runs every matching rule in order. Rules can:

  • match on kernel name, subsystem, sysfs attributes, or previously set properties
  • assign ownership, group, and mode on the device node
  • add symlinks under /dev
  • attach tags (notably systemd and uaccess)
  • request a program to run — with strict limits

The result is stored in the udev database and exported as properties. That database is what udevadm info reads, and what systemd uses to build .device units.

Two things udev is not:

  1. Not a job runner. Anything started from RUN is killed when event handling ends. Long-running work belongs in a systemd unit, pulled in via SYSTEMD_WANTS.
  2. Not a rename tool for block devices. The man page is blunt: "The name of a device node cannot be changed by udev, only additional symlinks can be created." NAME= only renames network interfaces.

Rule files and ordering

Rules live in four directories:

/usr/lib/udev/rules.d        # packages
/usr/local/lib/udev/rules.d  # local installs
/run/udev/rules.d            # runtime, volatile
/etc/udev/rules.d            # your overrides
Enter fullscreen mode Exit fullscreen mode

All files are sorted lexicographically across all directories, not per directory. A file with the same name in /etc replaces the one in /usr/lib; a symlink to /dev/null disables it entirely. Files must end in .rules — anything else is silently ignored, which is the single most common reason a new rule "does nothing."

A line is a comma-separated list of key operator value expressions. == matches, != does not match, = assigns, += appends, := assigns and locks. Match keys must all succeed for the assignments on that line to apply.

Step 0: identify the device properly

Do not guess attribute names. Ask the device:

udevadm info --attribute-walk --name=/dev/ttyUSB0
Enter fullscreen mode Exit fullscreen mode

The output walks up the sysfs chain, printing each level in rule syntax:

  looking at device '/devices/pci0000:00/.../tty/ttyUSB0':
    KERNEL=="ttyUSB0"
    SUBSYSTEM=="tty"
    ATTR{port_number}=="0"

  looking at parent device '/devices/pci0000:00/.../usb1/1-2':
    KERNELS=="1-2"
    SUBSYSTEMS=="usb"
    ATTRS{idVendor}=="10c4"
    ATTRS{idProduct}=="ea60"
    ATTRS{serial}=="LYRA0001"
Enter fullscreen mode Exit fullscreen mode

This is the whole trick behind reliable rules:

Key Scope
KERNEL, SUBSYSTEM, ATTR{} the event device only
KERNELS, SUBSYSTEMS, ATTRS{} walks up the parent chain

If you match ATTR{idVendor} on ttyUSB0, it fails: idVendor lives on the USB parent. Use ATTRS for anything inherited from a parent, and remember that multiple ATTRS keys in one rule must match the same parent device.

Prefer stable properties over physical paths. KERNELS=="1-2" breaks when you move the cable to another port. Vendor, product, and serial survive.

Rule 1: a stable symlink for a USB serial adapter

Create /etc/udev/rules.d/70-lyra-serial.rules:

# CP210x USB-to-serial: stable name regardless of plug order
SUBSYSTEM=="tty", ACTION=="add", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="ea60", ATTRS{serial}=="LYRA0001", SYMLINK+="lyra/serial0", MODE="0660", GROUP="dialout"
Enter fullscreen mode Exit fullscreen mode

Notes that matter:

  • SYMLINK is relative to /dev. Subdirectories are created; you get /dev/lyra/serial0.
  • Use +=, not =, so you do not wipe symlinks added by earlier rules.
  • GROUP= is resolved when rules are parsed. If the group does not exist yet, the assignment is dropped — set resolve_names=late or use a numeric GID if you create groups at runtime.
  • If two devices claim the same symlink, the one with the higher OPTIONS+="link_priority=N" wins. Without it, ownership is undefined.
  • I keep each rule on one line. When udevadm verify complains, a single physical line is much easier to fix than a wrapped one.

Apply and verify:

sudo udevadm verify /etc/udev/rules.d/70-lyra-serial.rules
sudo udevadm control --reload
sudo udevadm trigger --action=change --subsystem-match=tty
udevadm settle
ls -l /dev/lyra/serial0
Enter fullscreen mode Exit fullscreen mode

udevadm verify catches syntax errors, unknown keys, and style problems before they reach the daemon. trigger defaults to change, which re-runs rules for devices that already exist; without it you are waiting for a replug that may never come.

Rule 2: permissions that follow the logged-in user

Hard-coding MODE="0666" to make a YubiKey or a logic analyzer work is how you give every local process access to a security token. The uaccess tag is the alternative: udev grants an ACL on the device node to the user with the active session on that device's seat.

# /etc/udev/rules.d/70-lyra-yubikey.rules
SUBSYSTEM=="usb", ATTR{idVendor}=="1050", ATTR{idProduct}=="0407", TAG+="uaccess"
Enter fullscreen mode Exit fullscreen mode

Here ATTR{} is correct: the USB device node itself has idVendor as a direct attribute.

Check the result after replugging:

getfacl /dev/bus/usb/001/004
Enter fullscreen mode Exit fullscreen mode

You should see an entry for your user alongside root. Two caveats, both from the implementation:

  • uaccess requires systemd-logind. It looks up the active session on the device's seat and does nothing when there is none — which is exactly what happens on a headless server over SSH.
  • The builtin skips execution in test mode. udevadm test will not show the ACL being applied, so verify on the real node.

For services, skip uaccess and use a dedicated group. Seat-based access is a desktop concept; daemons have no seat.

Rule 3: start work when a device appears

Mounting a filesystem or starting a backup from RUN is not allowed — udev's sandbox blocks mount and network access, and forked processes are killed when the event finishes. The supported pattern is to pull in a unit:

# /etc/udev/rules.d/90-lyra-backup.rules
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="LYRA_BACKUP", TAG+="systemd", ENV{SYSTEMD_WANTS}+="lyra-backup.service"
Enter fullscreen mode Exit fullscreen mode

Both keys are required. SYSTEMD_WANTS is only honoured when the device carries the systemd tag; without it, the device is never exposed as a unit and the dependency is ignored. systemd also acts on it only when the device first becomes active.

That rule relies on ID_FS_LABEL, which is populated by the blkid builtin in 60-persistent-storage.rules. Because files are processed in lexical order, a file numbered below 60 would see an empty property. This is the second most common silent failure in udev.

The unit itself stays boring:

# /etc/systemd/system/lyra-backup.service
[Unit]
Description=Copy snapshots to LYRA_BACKUP
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/lyra-backup
Nice=10
IOSchedulingClass=idle
Enter fullscreen mode Exit fullscreen mode

No [Install] section: the device pulls it in, you do not enable it. Check it with:

systemctl list-units --type=device | grep -i lyra
journalctl -u lyra-backup.service -n 50
Enter fullscreen mode Exit fullscreen mode

Debugging without superstition 

# what the daemon sees for one event
udevadm test /sys/class/tty/ttyUSB0

# live event stream with properties
udevadm monitor --udev --property

# inspect stored properties
udevadm info --query=property --name=/dev/ttyUSB0

# wait for a symlink in a script instead of sleeping
udevadm wait --timeout=10 /dev/lyra/serial0
Enter fullscreen mode Exit fullscreen mode

udevadm test runs the rules in test mode and prints every match, assignment, and skipped line. Treat it as a dry run: it shows what would be assigned, not what was applied to the live node.

For stubborn cases, turn up logging per subsystem:

# /etc/udev/rules.d/00-debug-usb.rules
SUBSYSTEM=="usb", OPTIONS="log_level=debug"
Enter fullscreen mode Exit fullscreen mode

Then watch journalctl -u systemd-udevd -f. Remove it afterwards; debug logging on every USB event is noisy.

Symptom Likely cause
Rule never runs file does not end in .rules, or wrong directory
Match fails on a USB property used ATTR{} instead of ATTRS{}
Values silently wrong used = where == was meant
ENV{ID_*} empty file sorts before 60-persistent-storage.rules
Symlink missing on a disk tried to rename with NAME=; use SYMLINK+=
Group/permissions ignored group did not exist at parse time
Helper killed instantly long-running task in RUN; use SYSTEMD_WANTS
Nothing changed after edit no udevadm control --reload + trigger
uaccess does nothing no active seat session, or tested in test mode

Network interfaces are a different problem

NAME= works for network interfaces, but it is the low-level mechanism. For consistent NIC naming, use a .link file instead, which also handles MTU, MAC address policy, and wake-on-LAN in the same place:

# /etc/systemd/network/10-lan0.link
[Match]
MACAddress=aa:bb:cc:dd:ee:ff

[Link]
Name=lan0
Enter fullscreen mode Exit fullscreen mode

Rebuild the initramfs after changing link names (update-initramfs -u on Debian/Ubuntu, dracut -f on Fedora). Renaming happens early, and the initramfs carries its own copy of the rules.

Rollback

udev rules are additive configuration, so removal is clean:

sudo rm /etc/udev/rules.d/70-lyra-serial.rules
sudo udevadm control --reload
sudo udevadm trigger --action=change --subsystem-match=tty
Enter fullscreen mode Exit fullscreen mode

The managed symlink disappears on the next event for that device. Nothing in the kernel was modified, and no reboot is required.

The rule I actually follow

Match on identity, not on discovery order. Vendor, product, and serial are properties of the hardware; /dev/sdb and /dev/ttyUSB0 are properties of this boot. Every udev rule that has ever bitten me matched the second kind.

Write the rule, verify it, test it, then unplug and replug the device twice before you trust it.

Sources and references

Top comments (0)