Bash - uniq

uniq writes the unique lines in the given standard input

uniq [option]… [input [output]]

  1. Find and then remove duplicate elements - In uniq -d is used to find duplicates and -u is used to find unique records.
uniq -d movies.csv
  1. Remove duplicate records - In uniq -d is used to find duplicates and -u is used to find unique records.
uniq -u movies.csv > uniq_movie.csv
  1. Crosscheck the new file for duplicates
uniq -d uniq_movie.csv

-c, --count - prefix lines by the number of occurrences

-d, --repeated - only print duplicate lines, one for each group

-D   -  print all duplicate lines

-u, --unique - only print unique lines