uniq Command: Tutorial & Examples
Remove duplicate lines from a file
In Linux, uniq
is a command used to filter out duplicate lines from a sorted file. It compares adjacent lines in the input file and removes duplicate lines, leaving only one copy
of each unique line.
The basic syntax of the uniq command is:
uniq [options] [input-file]
For example, if you have a file named file.txt
that contains duplicate lines, you can use uniq file.txt
to remove the duplicates and display the unique lines.
The uniq
command only considers consecutive duplicated lines, so if you want to use it you should sort
the file first, for example:
sort file.txt | uniq
Some of the options that can be used with the uniq command include:
-c
: prefixes each line with the number of occurrences-d
: only print duplicate lines-u
: only print unique lines-i
: ignore case distinctions
The uniq
command is a simple yet powerful tool for removing duplicates from a sorted file. It's commonly used in combination with other commands
like sort
and grep
to filter and process data.
It's important to note that the uniq
command only works with sorted input. If the input file is not sorted, the command will not work as expected.
Further Reading
- Bash Cookbook by Carl Albing, J.P. Vossen (partner link)
- Wicked Cool Shell Scripts by Dave Taylor, Brandon Perry (partner link)
- Black Hat Bash by Nick Aleks, Dolev Farhi (partner link)
- Bash Pocket Reference by Arnold Robbins (partner link)
- The Linux Command Line by William Shotts (partner link)
- Learning the Bash Shell by Cameron Newham (partner link)
- Mastering Linux Shell Scripting by Mokhtar Ebrahim, Andrew Mallett (partner link)
- Linux Command Line and Shell Scripting Bible by Richard Blum, Christine Bresnahan (partner link)
- Shell Scripting by Jason Cannon (partner link)
As an Amazon Associate, I earn from qualifying purchases.