uniq Command: Tutorial & Examples
Remove duplicate lines from a file
uniq is a command used to filter out duplicate lines from a sorted file. It compares adjacent lines in the input file and removes duplicate lines, leaving only one copy
of each unique line.
The basic syntax of the uniq command is:
uniq [options] [input-file]
For example, if you have a file named
file.txt that contains duplicate lines, you can use
uniq file.txt to remove the duplicates and display the unique lines.
uniq command only considers consecutive duplicated lines, so if you want to use it you should
sort the file first, for example:
sort file.txt | uniq
Some of the options that can be used with the uniq command include:
-c: prefixes each line with the number of occurrences
-d: only print duplicate lines
-u: only print unique lines
-i: ignore case distinctions
It's important to note that the
uniq command only works with sorted input. If the input file is not sorted, the command will not work as expected.