DEV Community

Shubham Chaudhary
Shubham Chaudhary

Posted on

Linux uniq Command: Why Duplicates Survive (and the Fix)

#linux #cybersecurity #devops #bash

The bug: you run uniq on a file and the duplicates are still there.

printf 'alice\nalice\nbob\nalice\n' | uniq
# alice
# bob
# alice
Enter fullscreen mode Exit fullscreen mode

uniq only compares each line with the one directly before it, so scattered duplicates slip through. Sort first and the picture changes:

printf 'alice\nalice\nbob\nalice\n' | sort | uniq -c | sort -nr
#       3 alice
#       1 bob
Enter fullscreen mode Exit fullscreen mode

That pattern is a fast triage tool for logs. On a web server access log, it ranks client IPs (the first field in common Apache/Nginx formats):

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -10
Enter fullscreen mode Exit fullscreen mode

One address dominating the list is worth a closer look, but CDNs and corporate proxies can also produce high counts, so treat it as a lead, not proof.

The full guide covers -d, -u, -i, -f, -s, and -w, plus SSH brute-force ranking, an /etc/passwd UID 0 audit, rare-event hunting, and how to save output without overwriting your source file:
https://www.xpert4cyber.com/2026/09/linux-uniq-command-log-analysis.html

What's your go-to one-liner for log triage?

Top comments (0)