RSYNC - Can it copy files that are constantly being added to the source?

Ol!ver · 25 Sep 2011 at 10:42

Bit of an odd question this, but I'm going to try and make it make sense!

I'm copying my iPhoto Library to a Ubuntu machine which I use as a server. I'm using rsync to get the iPhoto folder/file/package off the Macbook, and it's copying to the server fine. At 16GB over a WiFi connection, it's never going to be quick.

I then want to copy the same files from the server, to a remote machine at a mate's house. I've got all our wedding photos and usual gubbins on there, and want it all safe. Plus, I'm playing with this stuff so it makes sense for now.

The question is thus: what happens if I start the transfer to the remote server while the files are still being copied to the local one? Will rsync stop when it's copied what was there when I ran the command, or will it just keep going?

Yes, I know I could have moved them straight from the Mac to the remote server, but I need all three copies at some point, and I can leave the server on whereas the Mac needs to be portable.

Not entirely sure all that makes sense, but there you go

Cheers!

memyselfandi · 25 Sep 2011 at 11:39

Rsync creates a list of files to be copied at the start of it's run and then acts on that list. Any files added after that list has been created will not be acted upon until the next time the command is run.

Ol!ver · 25 Sep 2011 at 11:46

Morning.

Since I wrote that, I decided to give it a shot to see what happened. It appears that, despite what we all thought, the files are continuing to be copied across, even though they were not present when I initially ran the command.

Could it be something to do with running rsync to send the files to the local server?

JonJ678 · 25 Sep 2011 at 19:27

Rsync does some weird things with recursion. In order to copy files across in less time, it makes a list of files while copying other files. It would be possible to add files after rsync had started running, but before the files in the directory had been listed (indexed might be a better term).

I think (that is, I haven't bothered to check the man pages) the term to make it build a complete list beforehand is --no-inc-recursive. I needed this when copying a large amount of data with many hard links within, the normal behaviour is to copy data and patch up hard links as and when they're found. This wastes some bandwidth, and in my case meant the external hard drive filled up. --no-inc-recursive means it spends ages building a list of files to begin with, but didn't have any trouble with hard links.