Finding duplicate files - how do I do this?

[stockhausen]

Over the years I have run (and eventually disposed of) a number of systems and as a result have hoarded a significant number of disk drives of increasing capacity.

Along with many other things, these disks have included a vast number of .MP3 and .JPG files and a number of other types such as .ISOs, .PDFs, etc.

I would like to rationalise these disks by identifying and eliminating duplicate files, copying them to a clean drive and disposing of the old disks - many of them use the IDE (PATA) interface and are not even SATA drives!

I can't guarantee that the duplicate files have the same name and/or creation date - I would work on the assumption that all the duplicates have the same file size!

Is there any software that will identify and report duplicate files?

Ideally this would be by comparing hashes, preferably based on file size. I would even be interested in a LINUX application that can work with NTFS & FAT file types.

I am not unduly concerned about run time in that I could leave this running while I am at work - having said that, I have a feeling that I need to disable the "Power saving" hibernation function on Windows 7.

 

[stockhausen]

CCleaner => Tools => Duplicate Finder appears to do something (very fast) but it only appears to be able to find files where the name is identical - which doesn't really help

 

[bledd]

Treesize

https://youtu.be/OWYt4plEOQE

 

[Semple]

I'll see if i can dig out the name of the duplicate photo finder i used a while back.

It was quite an impressive tool in that it scans in pictures and then throws up any that are very similar - ones where you've taken two separate photo's of the same thing and may have just moved slightly.

 

[COYS]

Nothing like a good new year sort out of all your porn

 

[Scougar]

I am in similar situation. I have multi drive, backed up drives/directories/files that are clearly duplicated sometimes many times. Whilst back ups are important... I would also like to rationalize then do a clean backup.

Interested to see what is out there as well.

 

[jpaul]

Below python script looks interesting

https://www.pythoncentral.io/finding-duplicate-files-with-python/

python dupFinder.py /folder1 ./folder2

but it will not produce a consolidated directory hierarchy ...
I would probably modify it to copy unique version of each file into folders specific to extension type ie pdf, mp3, etc

 

[stockhausen]

Treesize

https://youtu.be/OWYt4plEOQE

Treesize Pro (LINK) looks like a real possibility - additionally it is available on a free 30-day trial


The Python solution offered later may well be even better but I don't feel that keen on debugging it - sorry, but thanks for the suggestion