Get Duplicate lines & Count for all files in given directory

cat>/home/prabhucloudxlab/prabhu/input.txt

Hai this is hari

I am looking for an ETL / Hadoop / Big Data /DWH Testing profile job..

Please let me know if you have any openings regarding ETL / Hadoop / Big Data / DWH Tester.

cat>/home/prabhucloudxlab/prabhu/input2.txt

Hai this is prabhu

Hai this is manyam

Hai this is prabhu

Hai this is taraka

I am looking for an ETL / Hadoop / Big Data /DWH Testing profile job..

Please let me know if you have any openings regarding ETL / Hadoop / Big Data / DWH Tester.

#Script

#! bin/bash

ls /home/prabhucloudxlab/prabhu/*.txt | awk '{print $1}'

file_name="$(<file_names.txt)"

sleep 2

echo " "

#-------------------------------->>1st method(Sort , Uniq)

for file1 in $file_name;

echo "duplicate lines in $file1 are:"

sort $file1 | uniq -d

echo " "

echo "Count of duplicate lines in $file1:"

sort $file1 | uniq -c

echo " "

echo "================================================================="

done

#-------------------------------->>2nd method(Sort, AWK)

for file2 in $file_name;

echo "duplicate lines in $file2 are:"

sort $file2 | awk 'dup[$0]++ == 1'

echo " "

done

Note:

#sort---->>SORT command is used to sort a file, arranging the records in a particular order.

#uniq---->>The uniq command can count and print the number of repeated lines. Just like duplicate lines, we can filter unique lines (non-duplicate lines) as well and can also ignore case sensitivity.

#sort -r---->> sort and print the lines in reverese order.

#uniq -d ---->>only prints one instance of lines that have duplicates

#uniq -c ---->>to count the number of duplicates in the file

#Output:

[prabhucloudxlab@cxln4 ~]$ sh sh3.sh

/home/prabhucloudxlab/prabhu/input2.txt

/home/prabhucloudxlab/prabhu/input.txt

duplicate lines in /home/prabhucloudxlab/prabhu/input2.txt are:

Hai this is prabhu

Count of duplicate lines in /home/prabhucloudxlab/prabhu/input2.txt:

1 Hai this is manyam

2 Hai this is prabhu

1 Hai this is taraka

1 I am looking for an ETL / Hadoop / Big Data /DWH Testing profile job..

1 Please let me know if you have any openings regarding ETL / Hadoop / Big Data / DWH Tester.

=========================================================================

duplicate lines in /home/prabhucloudxlab/prabhu/input.txt are:

Hai this is hari

Count of duplicate lines in /home/prabhucloudxlab/prabhu/input.txt:

4 Hai this is hari

1 I am looking for an ETL / Hadoop / Big Data /DWH Testing profile job..

1 Please let me know if you have any openings regarding ETL / Hadoop / Big Data / DWH Tester.

=========================================================================

duplicate lines in /home/prabhucloudxlab/prabhu/input2.txt are:

Hai this is prabhu

duplicate lines in /home/prabhucloudxlab/prabhu/input.txt are:

Hai this is hari

Q A - Z O N E

Pages

Get Duplicate lines & Count for all files in given directory