uniq command in Linux: remove duplicate lines, count with -c, and sort -u

Short answer: uniq removes duplicate lines, but only when the copies sit next to each other. Sort first: sort file | uniq prints every distinct line once, sort file | uniq -c counts them, and sort file | uniq -c | sort -nr | head ranks them. -d shows only repeated lines, -u only lines that occur once.

sort names.txt | uniq                    # each distinct line once
sort names.txt | uniq -c                 # with a count in front
sort names.txt | uniq -c | sort -nr      # most frequent first
sort -u names.txt                        # same lines as sort | uniq, one command

Every output on this page was produced in the C.UTF-8 locale on Debian 12.15 with GNU coreutils 9.1 (uniq, sort, cut) and GNU Awk 5.2.1, in a scratch directory on 21 and 22 October 2026, against small sample files including a 16-line nginx access log with documentation IP addresses. Ubuntu, Rocky Linux and AlmaLinux ship the same GNU uniq; the BSD uniq on macOS shares the core options but not every GNU long option, such as --group.

uniq cheat sheet

Why does uniq not remove all duplicates?

Because it never looks further back than the previous line. uniq reads its input once, compares each line with the one before it, and drops it when they match. That makes it fast and needs almost no memory on any file size, but a duplicate three lines further down survives. The man page says so in its last note: repeated lines are not detected unless they are adjacent.

$ cat fruits.txt
apple
apple
banana
apple
cherry
cherry
banana
kiwi
$ uniq fruits.txt
apple
banana
apple
cherry
banana
kiwi
$ sort fruits.txt | uniq
apple
banana
cherry
kiwi

Plain uniq only merged the two runs (apple apple and cherry cherry). After sort puts equal lines next to each other, every duplicate goes. Skip the sort only when you want exactly that run behaviour, for example to collapse a log message that repeats many times in a row.

How do I count duplicate lines with uniq -c?

-c puts the number of occurrences in front of each line, right-aligned in a seven-character column. Pipe that into sort -nr (numeric, reversed) for a frequency list, and head to keep the top entries:

$ sort fruits.txt | uniq -c
      3 apple
      2 banana
      2 cherry
      1 kiwi
$ sort fruits.txt | uniq -c | sort -nr
      3 apple
      2 cherry
      2 banana
      1 kiwi
$ sort fruits.txt | uniq -c | sort -k1,1nr -k2
      3 apple
      2 banana
      2 cherry
      1 kiwi

Note the tie: with plain sort -nr, lines with the same count come out in reverse alphabetical order, because -r also reverses the whole-line comparison sort falls back on. sort -k1,1nr -k2 sorts by count, highest first, and breaks ties alphabetically. Without the first sort, uniq -c counts runs, not totals: uniq -c fruits.txt reports apple twice, once with 2 and once with 1.

Show only repeated lines (-d, -D) or only unique ones (-u)

$ sort fruits.txt | uniq -d
apple
banana
cherry
$ sort fruits.txt | uniq -u
kiwi
$ sort fruits.txt | uniq -cd
      3 apple
      2 banana
      2 cherry
$ sort fruits.txt | uniq -D
apple
apple
apple
banana
banana
cherry
cherry

• -d prints one copy of each line that appears more than once. Combined with -c it lists only the duplicates with their counts.
• -u is the opposite: only lines that appear exactly once. It does not mean “unique values” in the database sense; for that you want plain sort | uniq.
• -D prints every copy of the repeated lines, and --all-repeated=separate does the same with a blank line between groups. --group shows all lines, unique ones included, with a blank line between groups.

Ignore case with uniq -i (and why it needs sort -f)

-i treats Apple and apple as equal, but they still have to be adjacent. In the C and C.UTF-8 locales sort puts every capital letter before every small one, so the variants end up apart and uniq -i misses them. Sort with -f (fold case) so they line up:

$ cat case.txt
apple
Banana
APPLE
banana
Apple
$ sort case.txt | uniq -i
APPLE
Banana
apple
banana
$ sort -f case.txt | uniq -ic
      3 APPLE
      2 Banana
$ sort -fu case.txt
apple
Banana

uniq keeps the first line of each group, so which spelling survives depends on the order sort produced. sort -fu gives the same set in one step, keeping whichever spelling came first in the input (here apple rather than APPLE).

Compare only part of a line: -f, -s and -w

Three options narrow what uniq compares. The printed line is always the whole first line of the group:

• -f N skips the first N fields. A field is a run of blanks followed by non-blank characters, so uniq has no option for another separator such as a comma.
• -s N skips the first N characters (after any skipped fields).
• -w N compares no more than N characters.

$ cat app.log
09:15:01 disk /var 91% full
09:15:31 disk /var 91% full
09:16:01 disk /var 91% full
09:16:31 backup finished
09:17:01 disk /var 92% full
$ uniq -c -f 1 app.log
      3 09:15:01 disk /var 91% full
      1 09:16:31 backup finished
      1 09:17:01 disk /var 92% full
$ uniq -c -w 10 days.log
      2 2026-10-20T23:58:11 GET /
      3 2026-10-21T00:00:02 GET /

-f 1 ignored the timestamp and collapsed the repeated disk warning; uniq -s 9 gives the same result here, because the timestamp is exactly nine characters with its space. -w 10 compared only the date, which counts lines per day, but prints the first full line of each day. To print only the key, cut it out first: cut -c1-10 days.log | uniq -c. For a comma-separated file, pick the column with the cut command (cut -d, -f1) or deduplicate on a key with sort -t, -k1,1 -u, which keeps one whole line per key.

sort -u vs uniq: which one should I use?

On a sample of one million random numbers, sort -u and sort | uniq both returned the same 50,000 lines. The awk command version keeps every distinct line in memory, which is fine for log keys and a poor idea for a multi-gigabyte file of mostly distinct lines; sort spills to temporary files instead.

Real example: top client IPs in an nginx access log

This is where uniq earns its place on a server. In nginx’s default combined log format, field 1 is the client address, field 7 the path and field 9 the status code. Take the field with awk, sort, count, sort by count:

$ awk '{print $1}' access.log | sort | uniq -c | sort -nr | head
      8 203.0.113.7
      4 198.51.100.23
      2 2001:db8::42
      2 192.0.2.10
$ awk '{print $9}' access.log | sort | uniq -c | sort -nr
      8 200
      7 404
      1 502
$ awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -nr | head -n 3
      2 /wp-login.php
      1 /xmlrpc.php
      1 /config.php

In this sample one address sent half the requests, seven of them 404s for paths such as /wp-login.php, /.env and /backup.zip: that is a scanner, not a visitor. On a live server the log is root-only, so put sudo in front of the awk and point it at /var/log/nginx/access.log. Narrow it to a time window or a path with the grep command before counting, and for the number of distinct visitors rather than a list, end with wc -l: awk '{print $1}' access.log | sort -u | wc -l printed 4 here. The same pipeline shape works on any list: remote addresses from ss, repeated messages from journalctl -o cat, or the commands in your shell history.

Common problems and error messages

Two more things catch people in scripts. A second file name is the output file, not another input: uniq in.txt out.txt writes to out.txt. And uniq exits with 0 whether or not it found duplicates, so test the output, not the exit status: [ -n "$(sort list | uniq -d)" ] && echo duplicates. For the other text tools in the same pipelines, the Linux commands cheat sheet has them on one page.

Frequently asked