DEV Community

Super Funicular
Super Funicular

Posted on

du -sh | sort -rn Picked an 884K Camera Folder Over a 12M One: sort -n Reads the Digits, Not the Unit

A cleanup job for a phone camera needs to know which folder of clips is using the most space, so it can prune that one first. The usual one-liner is du -sh piped through sort -rn and head -n 1. I built four camera folders of known size and ran it. It named the second-smallest folder.

The setup

Four folders, one clip each. The bytes are random so nothing compresses, and the sizes are chosen so du -sh has to print both K and M:

#!/bin/bash
# Four camera folders, one clip each, random bytes so nothing compresses or dedupes.
set -eu
rm -rf clips && mkdir clips && cd clips
mkdir driveway porch garage nursery
head -c 1500000  /dev/urandom > driveway/clip_0001.mp4
head -c 900000   /dev/urandom > porch/clip_0001.mp4
head -c 12000000 /dev/urandom > garage/clip_0001.mp4
head -c 300000   /dev/urandom > nursery/clip_0001.mp4
Enter fullscreen mode Exit fullscreen mode

The script under test is the one-liner, wrapped so a cleanup job could call it:

#!/bin/bash
# Name the camera folder using the most space, so a cleanup job knows where to prune.
set -euo pipefail
cd clips
du -sh -- * | sort -rn | head -n 1
Enter fullscreen mode Exit fullscreen mode

What it said

#!/bin/bash
./mk.sh
./biggest.sh
echo "--- every folder, du -sh"
(cd clips && du -sh -- *)
Enter fullscreen mode Exit fullscreen mode
884K    porch
--- every folder, du -sh
1.5M    driveway
12M garage
300K    nursery
884K    porch
Enter fullscreen mode Exit fullscreen mode

The garage folder holds the largest clip by far, and du -sh reports it correctly as 12M. The one-liner still picked porch at 884K. The sizes are right. The ranking is wrong.

Why sort -n reads 884K as bigger than 12M

sort -n compares a number at the start of each line, and the GNU manual spells out what that number is: "The number begins each line and consists of optional blanks, an optional ‘-’ sign, and zero or more digits possibly separated by thousands separators, optionally followed by a decimal-point character and zero or more digits."

There is no letter in that list. The K and the M are not part of the number, so sort -n stops reading right before them. It ranks 884 above 300, above 12, above 1.5, with no way to know that one of those means megabytes.

The manual's own advice for a surprising result is "try the --debug option to see what happened." It underlines the part of each line that the sort key used:

#!/bin/bash
cd clips
du -sh -- * | sort -n --debug
Enter fullscreen mode Exit fullscreen mode
sort: text ordering performed using ‘C.UTF-8’ sorting rules
1.5M>driveway
___
_____________
12M>garage
__
__________
300K>nursery
___
____________
884K>porch
___
__________
Enter fullscreen mode Exit fullscreen mode

The first underline on each line is the key, and it stops before the unit every time: three characters for 1.5, two for 12, three for 300 and 884. The second underline is the whole line, which sort falls back on only when two keys tie.

Nothing errors here. du exits 0, sort exits 0, head exits 0, and pipefail has nothing to report. A job that trusts the first line would prune the porch camera and leave the garage folder to keep growing.

sort -h fixes this case

GNU sort has an option made for this output. -h (--human-numeric-sort) sorts "first by numeric sign (negative, zero, or positive); then by SI suffix", and only after that by the number. A value with M outranks any value with K, whatever the digits say:

#!/bin/bash
cd clips
echo "--- sort -h"
du -sh -- * | sort -h
echo "--- sort -rh | head -n 1"
du -sh -- * | sort -rh | head -n 1
Enter fullscreen mode Exit fullscreen mode
--- sort -h
300K    nursery
884K    porch
1.5M    driveway
12M garage
--- sort -rh | head -n 1
12M garage
Enter fullscreen mode Exit fullscreen mode

That is the correct order, and sort -rh | head -n 1 names the garage folder. For a single du -sh call, this is the whole fix.

Where sort -h stops being enough

The manual is specific about when -h works. It sorts values that are "consistently scaled to the nearest suffix", which is why it handles "the output of any single invocation of the df, du, or ls commands" run with their human-readable options.

Those conditions matter once size lines come from more than one place, for example a log that collects sizes over time from a mix of commands. du -s -BK prints every size in kilobytes, however large, so its 11724K is not scaled to the nearest suffix. Here is one log with three lines from du -sh and the garage line from du -s -BK:

#!/bin/bash
cd clips
# Three folders from du -sh, one from du -s -BK, appended to the same log.
{ du -sh -- driveway porch nursery; du -s -BK -- garage; } > ../sizes.log
cat ../sizes.log
echo "--- sort -h"
sort -h ../sizes.log
Enter fullscreen mode Exit fullscreen mode
1.5M    driveway
884K    porch
300K    nursery
11724K  garage
--- sort -h
300K    nursery
884K    porch
11724K  garage
1.5M    driveway
Enter fullscreen mode Exit fullscreen mode

The garage folder is the largest, but it sorts below 1.5M. Every K value ranks under every M value, so sort -h does not get as far as comparing 11724 with 1.5. That is the documented rule working as written: a value expressed in a smaller unit than its size calls for lands in the wrong place.

The same thing would happen with sizes written by a script that prints every size in kilobytes, or a hand-edited line. The manual also notes that "the SI suffix must immediately follow the number", so a line written as 12 M with a space is not read as megabytes either. When I piped a 12 M line and a 900K line through sort -h, the 12 M line came first.

Sort the bytes, then make them readable

The manual's own suggestion sits at the end of the -h entry: "To sort more accurately, you can use the numfmt command to reformat numbers to human format after the sort."

So the order of operations flips. Ask du for plain byte counts, which sort -n compares exactly, and only then turn them into K and M for a person to read:

#!/bin/bash
cd clips
# Sort on exact byte counts, then turn the numbers into K/M/G for people.
du -s -B1 -- * | sort -rn | numfmt --to=iec | head -n 1
echo "--- full list"
du -s -B1 -- * | sort -n | numfmt --to=iec
Enter fullscreen mode Exit fullscreen mode
12M garage
--- full list
300K nursery
884K porch
1.5M driveway
12M garage
Enter fullscreen mode Exit fullscreen mode

-B1 makes du report disk usage in one-byte blocks, so every line is a plain integer with no suffix, and sort -n has nothing to misread. numfmt --to=iec rewrites the first field into the familiar short form after the order is already settled. The top line names the garage folder, and the list matches what sort -h gave for the clean case.

A cleanup job that acts on the result should keep the byte count rather than the human form. A plain integer can be compared against a limit in a script. A string like 12M has to be parsed again, and that is the step that went wrong here.

What to check in your own scripts

  • Search for sort -n or sort -rn downstream of du -h, du -sh, df -h or ls -lh. Each of those pairings ranks by the digits alone.
  • If you keep the human-readable command, switch to sort -h and keep the input from one command, one invocation, one set of flags.
  • If sizes are collected over time or from more than one tool, store bytes (du -s -B1) and format with numfmt only at the point of display.
  • When an order looks wrong, sort --debug shows exactly which characters it compared. It is faster than guessing.

Things I did not test

  • Other sort implementations. I ran GNU coreutils 8.32 on Linux. The quotes come from the current online GNU manual. BSD and macOS sort are separate programs, and I did not check how their -h handles mixed input.
  • Locales. My runs used the locale shown in the --debug line. sort -n takes its decimal point from the locale, so a locale that uses a comma could read 1.5M differently. I did not try one.
  • Huge folders. The folders here hold one small clip each. The ordering rules do not depend on size, but I did not time any of this on a large tree.
  • Apparent size. du reports space used on disk, not the length of the files. For this question that is usually what you want, but sparse or compressed files can make the two differ, and I did not test either.

Sources. Quotes are from the GNU Coreutils manual page sort invocation. Outputs are pasted unedited from the scripts above.

If you want an old Android phone recording in the background so there are clip folders worth cleaning up, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

sort -n ranks the digits and ignores the unit, so sort the bytes first and add the K, M and G last.

Top comments (0)