Advice on Data copy

katana6434

katana6434

katana6434

Associate
Joined
6 Oct 2006
Posts
375
Location
Luton
Hi All


I have scenario where I need to copy data from one drive to another drive on the same server. Both of these drives are LUNs on different SANs.

The constraint that I have is that data can only be copied out of hours. It can start at 6pm and needs to be finshed by 6am the next morning. Ordinarily this wouldn't be a problem, but the data that I am copying is around 770GB and consists of around 16.5 million files. It would take around 2-3 days to copy it all in one go.

Ideally I need some software which could sync it from one drive to another only between the times above. So it owuld start the copy at 6pm, stop cpying at 6am and than start of again from where it left off at 6pm again.

It this possible. Anybody know any software that could do this.

Thanks
 
You should check out robocopy. an MS support tool, now included in 2008.

You can set it to sync everynight so that your switchover night has the last nights delta is as little as possible.

Great tool, suprised you havent heard of it.
 
Can you remove the files element and just do the copy at disk image level? You'd only need to achieve 18M/s to do all that data in 12 hours.
 
Nutbusta said:
You should check out robocopy. an MS support tool, now included in 2008.

You can set it to sync everynight so that your switchover night has the last nights delta is as little as possible.

Great tool, suprised you havent heard of it.
Click to expand...

i have heard of Robocopy and have used it in the past. I don't think it would be suitable in this case as the copy would not finish in the desired timeframe and then there would be no way to 'pause' the copy during the working day, unless I am mistaken.
 
Katana, you are right. Robocopy is only really controllable programatically. Not practical in this situation.
Another problem would be that any files changed during this period which had already been copied would not be updated.

I know DFS has bandwidth restrictions depending on the time period, but I'm not sure if you can set up a replication between two shares on the same server.
If you can set DFS to replicate between two shares on the same server, then this is definitely the best way forward.
//Server1/ShareA to //Server1/ShareB for example.
 
If you kicked off Robocopy doing a /MIR at 6pm and then 6am, stop it.
Next day, start it again. It'll scan from the start and skip anything done the previous night and update anythign copied the night before and carry on from pretty much where you stopped it.


Does mean either being in work at 6am or being up to VPN in to stop it.
 
PistolPete said:
If you kicked off Robocopy doing a /MIR at 6pm and then 6am, stop it.
Next day, start it again. It'll scan from the start and skip anything done the previous night and update anything copied the night before and carry on from pretty much where you stopped it.


Does mean either being in work at 6am or being up to VPN in to stop it.
Click to expand...

+1 robocopy should be fine.
 
Set robocopy to only copy files that have changed. Because of the number of files you would probably need multiple instances of robocopy targeting different directories. Also regarding run times blurb from manual below

"use of /RH is to limit Robocopy’s copy activity to a time slot that is outside of normal business hours (e.g. /RH:2200-0600)"

"You should use the /PF switch if you would like the Run Hours check to be performed before each individual file copy is started."
 
Thanks for those. Looks like I will investigate robocopy a bit more. I knew about the /mir command, alhtough I am a bit dubious about how long it could take to scan the directory. This could significantly eat into the copy time available.

I will look into the /RH and /PF switches.

Thanks
 
katana6434 said:
Thanks for those. Looks like I will investigate robocopy a bit more. I knew about the /mir command, alhtough I am a bit dubious about how long it could take to scan the directory. This could significantly eat into the copy time available.

I will look into the /RH and /PF switches.

Thanks
Click to expand...

I backup my important data from my RAID array to USB2 external disk.
If I've not changed anything it'll scan and quit with /MIR in about a minute with 854GB of data.
 
beware using the MIR, if files are removed from the source tree, then robocopy will remove them on the destination as well... So you dont want to go 'ahh all those files are copied, I'll just remove them..' and then forget to delete the robocopy schedule task.
 
whitecrook said:
beware using the MIR, if files are removed from the source tree, then robocopy will remove them on the destination as well... So you dont want to go 'ahh all those files are copied, I'll just remove them..' and then forget to delete the robocopy schedule task.
Click to expand...


Just don't run a scheduled task unless you want an ongoing mirror.
 
You must log in or register to reply here.
Back
Top Bottom