Global DeDupe & Replication

ubern00b · 5 Nov 2014 at 10:02

Hi All

I have 2 Broadberry servers that we use for our backup devices which exist in different locations connected via 1GB WAN link. SQL backups are done in location 1 and deduplicated and replicated to location 2.

Sounds great until you discover that DFS-R re-hydrates deduplicated data before replicating it. Trying to replicated that amount of data (multiple TB's) is simply not working, we have over 10K items in backlog.

My question is, does anyone know of software that will perform a global dedupe and replication? I am sure ArcServe used to provide something but I cannot find the product. ArcServe is preferable only because that is our current vendor. We will of course look at any solution as we are currently exposed due to not having replicated backups for core systems!

Thanks in advance.

anything I don't mind · 5 Nov 2014 at 12:51

What storage is it on? isn't there any storage replication solutions you could consider as that would most definitely include dedup before.

Can't you do a offline data transfer first and then just replicate changes going forward. By transporting a hard drive with the data to the other side for example?

I use arcserve, they do have replicating products but i have not used them. Try CA new product, UDP, it is web based but it is their latest offering.

ubern00b · 6 Nov 2014 at 09:49

The Broadberry systems are just "cheap storage" so to speak. 36 drives connected with a MegaRAID controller, so there is no storage replication like, say the P4800's we have.

We backup about 100 databases a day totalling roughly 1TB and that is excluding log shipping. DFS-R simply cannot cope with the speed necessary to replicate this amount of data, in my testing anyway.

I upgraded to UDP a couple months back. Fantastic product (In comparison to Backup 16.5) but has it's limitations. Unable to traverse mount points is a big one for me (volumes mounted as folders).

I will contact my ArcServe rep and ask about the replication product. Actually, I think it's called ArcServe D2D, it just came to me!

ASE001 · 6 Nov 2014 at 10:20

Enable SQL native backup compression potentially saving 60%plus in backup file size, 60% less to go over the WAN? It may buy you sometime to find a better solution? It has a down side, it will take longer to complete backups and use more (CPU/Disk I/O etc) resources during backups! It won't save space on the storage device itself since it is using dedup.

Deduplication doesn't work at the file level, its much lower looking for binary patterns that are duplicated across the whole filesystem, so that's why it appears to reconstruct(re-hydrates) the file. The best type of replication for dedup storage, is replication that replicates at the binary level and only replicates binary blocks of data that have altered. Enterprise SANs typical use this type of replication and you are talking mega bucks.

Perhaps a WAN accelerator, take look at wanos.org if for nothing else it may give you some ideas?

The size of you purse strings will constrain the solutions available to you?