Block level replication in Linux

SMN · 5 Dec 2010 at 14:15

Hi Chaps,

Bit of a replication question (i think I already know the answer roughly just looking for thoughts).

I have 2 storage pools, both 3TB's in volume each (1.9TB used).

Storage pool A (herein known as "ACTIVE") and Storage pool B (herein "BACKUP").

Now I have all my files, day to day stuff, etc on my ACTIVE pool, and I've recently created BACKUP to do a week by week backup on Sunday evenings. I'm trying to copy only files that have been modified, or NEW files added.

Now looking at rsync's MAN page:

"# -u Update. Don't copy file if file on destination is newer."

I suspect this is the option I need to use, however just wondering if anyone else has a similar setup like this and is using anything different?

ATB,
Sam

zetec452 · 5 Dec 2010 at 15:21

IIRC by default Rsync will automatcially compare files and only transfer changes.

something like:

rsync -avz foo:src/bar /data/tmp

Would work. See: http://www.samba.org/ftp/rsync/rsync.html

It may be worth doing an initial copy (cp command) as rsync will be a lot slower for a massive copy.

memyselfandi · 5 Dec 2010 at 16:15

SMN said:
"# -u Update. Don't copy file if file on destination is newer."

You would use this option if you were changing files on the BACKUP copy so that they were newer than those on the ACTIVE copy. As you indicated that you are just making a backup then of the ACTIVE copy then you probably don't need this.

What JonRohan has put is likely correct. I.e. if ACTIVE is on box1 and BACKUP is on box2 ....

box1:/ # rsync -avvz /active/ box2:/backup/

(I tend to throw a --progress in there as well so I can see what is being transferred and at what rate but it's not necessary)

zetec452 · 5 Dec 2010 at 16:17

If you want to get a little posh with the setup you can take a look at:

http://www.cyberciti.biz/faq/linux-rsnapshot-backup-howto/

BigglesPiP · 13 Dec 2010 at 18:55

rsync would be file, not block. It's probably the best way to to file level replication for your application (getting a representation of all the files once a week or something).

The best way to do block level replication would be with snapshots. And the best way to do that would be with ZFS or LVM.

With LVM you would:
-Have 2 Physical Volumes (your drives).
-Make 2 Logical Volume Groups (each containing all the extents from one drive, both the same extent size)
-Make 1 Logical volume in the 1st logical volume group.

-Every n time, create a snapshot of the 1st logical volume in the second logical volume group, deleting the existing one 1st. A copy on write snapshot would get you lots of copies, in the space and remove the window where you have no copy, but would need both drives to remain online, don't know if you're going for redundancy or just accident recovery.