Merging drives / directories

[stockhausen]

I have been asked to merge the contents of three drives that have been used over a number of years in a series of systems onto a single drive to be used in a new system.

The problem is that there are many duplicated documents and spreadsheets. I know this because I have used Ccleaner => Tools => Duplicate Finder to try and locate and tidy them up - there are a lot!

I believe that there is/was a way in UNIX to spot such files using MD5 hashes and automatically to keep the most recent. I have long since forgotten how to do this and wondered whether there is any way of doing the same thing in Windows?

I can't rely on filenames or dates and need to compare the content of the actual files in some (automated) way. Is there anything in Windows that can do this?

 

[Mcnumpty2323]

I believe windows has a md5 verification option
But it's not an automated process so no good for more than a
Few files
Have a look at auslogic duplicate finder
Used it a while ago and think it had an option to compare
By checksum etc
And an option to backup the duplicate files just to be safe

It's free though just watch it doesn't offer extra stuff during installation
Edited
I have 3 copies of everything
I used some other software to sync drive e to f then to g
And to remove duplicate files so each drive ended up identical
But without duplicates
Annoyingly can't seem to sync multiple drives at once
But you can just set 2 tasks to achieve it
It had a compare files option in it
But can't remember what software it was right now
It will probably come to me when not trying to remember it
It might have been from easeus though not
Totally sure

Further edit
If you Google
Find and delete duplicate files using powershell
It may be possible to do that which could automate
Windows md5 checking

 

[Donnie Fisher]

I believe that there is/was a way in UNIX to spot such files using MD5 hashes and automatically to keep the most recent. I have long since forgotten how to do this and wondered whether there is any way of doing the same thing in Windows?

I can't rely on filenames or dates and need to compare the content of the actual files in some (automated) way. Is there anything in Windows that can do this?

This doesn't match up. When you find 2 files the same, your still relying on the date alone to decide ?

 

[whitecrook]

Use beyond compare

thank me later

 

[Mcnumpty2323]

Use beyond compare

thank me later

not heard of that one thanks
though i forgot to ask if he wanted a totally free
or payed for solution
whitecrook wow that brings back memories lol

 

[whitecrook]

not heard of that one thanks
though i forgot to ask if he wanted a totally free
or payed for solution
whitecrook wow that brings back memories lol

I know but beyond compare gives a trial but i think youll find that its actually very much worth paying for. Nothing touches it cant praise it enough. You have to try it to see how well it works. Comparing md5 hashes manually is caveman stuff compared to this


Edit whitecrook as in Clydebank ayyeeee?!

 

[Mcnumpty2323]

I know but beyond compare gives a trial but i think youll find that its actually very much worth paying for. Nothing touches it cant praise it enough. You have to try it to see how well it works. Comparing md5 hashes manually is caveman stuff compared to this

oh i wasnt knocking it for not being free
just didnt know if the op wanted free or not
since he asked for something
that was in windows
but using powershell was about the only option
i could find that might be any use within windows
though they will need to do a bit of googling
to find the correct commands to make the built in md5 checker
do a load of files automatically
then delete or move the duplicates
so the cost of the software you mentioned may well be worth
the time saved trying to find how to powershell it
especially if he can charge the person hes doing it for
what the software costs

 

[d_brennen]

What OS?

I just use file dedupe within Windows Server (2016 and above iirc) across all our user storage shares

If you're moving the shares to another volume/server, you'll need a volume big enough to take all the duplicated data before the dedupe can do it's thing. I've had to extend and shrink volumes to make stuff fit across our estate as it wasn't enabled before I took over. Space savings are immense with ~1500 users all making copies of the same nonsense