- Welcome to the third video of section three:
Finding and Deleting Duplicate Files.
In the previous video, we've learned how intersect
and set difference on text files.
In this video, we're going to see how to find
duplicate files and perform operations
based on the result.
Duplicate files are copies of the same files.
In some circumstances, we may need to remove
duplicate files and keep a single copy of them.
Identification of a duplicate file
by looking at the file content
is an interesting task.
It can be done using a combination of shell utilities.
We can identify the duplicate files
by comparing file content.
Checksums are ideal for this task,
since files with exactly the same content
will produce the same checksum values.
We can use this fact to remove duplicate files.
Generate some test files as follows:
test copy one and test copy two are copy of test.
The code for the script to remove the duplicate files
is as follows:
run it as...
Let's see how it works.
The proceeding commands will find the copies
of the same file in the directory
and remove all except one copy of the file.
Let's go through the code and see how it works.
LS minus LS will list the details of the file
sorted by file size in the current directory.
Dash dash time style equals long dash I-so
tells LS to print the data in the I-so format.
Awk will read the output of LS dash LS
and perform comparisons on columns and rows
of the input text to find the duplicate files.
The logic behind the code is as follows:
we list the files sorted by size
so that similarly sized files will be grouped together.
The files having the same file size
are identified as a first step
to finding files that are the same.
Next, we calculate the checksum of the files.
If the checksums match, the file's a duplicate
and one set of duplicates are removed.
To begin, curly brace block of awk is executed first
before the lines are read from the file.
Reading lines takes place in the curly brace block,
and after the end of the reading and processing all lines
the end curly brace block statements are executed.
The output of LS minus LS is
the output of the first line tells us
the total number of files,
which in this case is not useful.
We use getline to read the first line and then dump it.
We need to compare each of the lines
and the next line for sizes.
For that we read the first line explicitly
using getline and store the name and size
which are the eighth and fifth columns.
Hence, a line is read ahead using getline.
Now an awk enters the curly brace block
in which the rest of the lines are read.
That block is executed for every read or a line.
It compares the size obtained from the current line,
and the previously stored size kept in the size variable.
If they're equal, it means,
two files are duplicates by size.
Hence they have to be further checked by nd5sum.
We've played some tricks on the way to the solution.
The external command output could be read inside awk as.
Then we receive the output in line $0.
Each column output can be received in $1, $2, $n and so on.
Here we read the nd5sum checksum of the files
in csum1 and csum2 variables.
Variables name1 and name2 are used to store
consecutive filenames.
If the checksums of the two files are the same,
they're confirmed to be duplicates and are printed.
We need to find a file from each group of duplicates
so we can remove all other duplicates.
We calculate the nd5sum value of the duplicates
and print one file from each group of duplicates
by finding unique lines.
Comparing nd5sums only from each line using the w32.
The first 32 characters in the nd5sum output.
Usually, the nd5sum output consists of a 32 character hash
followed by the filename.
Therefore, one sample from each group of duplicates
is written in duplicate_sample.
Now we need to remove all the files listed
in duplicate_files excluding the files
listed in duplicate_sample.
The come command prints files in duplicate_files,
but not in duplicate_sample.
For that we use a set difference operation.
Refer to the recipes on intersection
difference and set difference.
Come always accepts files that are sorted,
therefore sort -u is used to filter before redirecting
to duplicate_files and duplicate_sample.
Here the tee command is used to perform a trick
so it can pass filenames to the rm command as well as print.
The tee command writes lines that appears
as standard in to a file and sends them to a standard out.
We can also print text to the terminal by redirecting
to standard error.
/dev/stderr is the device corresponding
to stder, standard error.
By redirecting to a standard error device file,
text appears through standard in will be printed in terminal
as standard error.
Awesome! In this video, we've successfully learned
how to find and delete duplicate files.
In the next video,
we'll see how to work with file permissions,
ownership, and the sticky bit.