uniq Command: Tutorial & Examples

Remove duplicate lines from a file

In Linux, uniq is a command used to filter out duplicate lines from a sorted file. It compares adjacent lines in the input file and removes duplicate lines, leaving only one copy of each unique line.

The basic syntax of the uniq command is:

uniq [options] [input-file]

For example, if you have a file named file.txt that contains duplicate lines, you can use uniq file.txt to remove the duplicates and display the unique lines.

The uniq command only considers consecutive duplicated lines, so if you want to use it you should sort the file first, for example:

sort file.txt | uniq

Some of the options that can be used with the uniq command include:

• -c: prefixes each line with the number of occurrences
• -d: only print duplicate lines
• -u: only print unique lines
• -i: ignore case distinctions

The uniq command is a simple yet powerful tool for removing duplicates from a sorted file. It's commonly used in combination with other commands like sort and grep to filter and process data.

It's important to note that the uniq command only works with sorted input. If the input file is not sorted, the command will not work as expected.

Further Reading

• Bash Cookbook by Carl Albing, J.P. Vossen
• Wicked Cool Shell Scripts by Dave Taylor, Brandon Perry
• Black Hat Bash by Nick Aleks, Dolev Farhi
• Bash Pocket Reference by Arnold Robbins
• The Linux Command Line by William Shotts
• Learning the Bash Shell by Cameron Newham
• Mastering Linux Shell Scripting by Mokhtar Ebrahim, Andrew Mallett
• Linux Command Line and Shell Scripting Bible by Richard Blum, Christine Bresnahan
• Shell Scripting by Jason Cannon