VMware Backup Challenge

[Arthalen]

Ok, here's an interesting once for you.

I have a VMware ESXi deployment consisting of 10x ESXi 5.1 Physical Hosts with back-end Dell iSCSI MD3600 Storage. It currently runs VCentre 5.1.

One of my colleagues, in his infinite wisdom, at one time in ages past dedicated an entire SAN shelf (20TB, to be precise) to a VM via a Physical RDM mapping of the storage to the VM.

We have in place a Physical Backup solution in the form of Backup Exec 2012 using 4x LTO6 tape drives. However, Backup Exec does not support RDM or indeed independant disks in VMware when performing VMDK-level backup jobs.

I've looked at a possible Windows Agent style backup, but have had consistent errors with Backups of very large storage volumes when going over a two-day backup window.

The complication of a repeated 2-day backup to capture all the files is that the storage volume pretty much contains 19TB of 256KB Files, without any kind of folder structure. It would be a nightmare to keep deselecting the backed up files from the selection list.

I've looked at VEEAM, and that has the same limitation around RDMs that Backup Exec has.

So, any thoughts on how to get a backup?

 

[Arthalen]

If you upgrade to Esxi 5.5 you will be able to use large VMDK's. Then your backup becomes trivial with something like Veeam or Unitrends.

Regardless of VMDK size limitations (2TB limit under 5.1 has never been a problem to manage or backup so far, despite having a 100TB+ SAN deployment), I need a solution to deal with a 20TB Physical Raw Disk Mapping (RDM).

Once it's backed up, the server and the RDM will be getting permanently removed so the space can be properly presented through a LUN and then managed with individual VMDK files at the correct sizes.

But I need the backup of the server and the attached LUN first

 

[Arthalen]

Just checked the latest Veeam FAQ, it still doesn't support Physical RDM backups either.

 

[Arthalen]

As mentioned, its in physical compatibility mode (Physical RDM), which blows holes in all sorts of 'solutions' I've researched so far.

At the moment, I'm thinking my best bet is BE2012 Windows Agent using Checkpoint restart and File Deletions so I can get a straight copy of the data over multiple attempts.



Some more, interesting, background:

Using RDM's would be the last thing I would ever consider using (since we're invested in Backup Exec 2012 for the time being), but it was a necessity at the time to facilite doing a P2V conversion.

The P2V conversion was of an old BE 2010 DeDupe server that had a (dangerously full) 22TB Single Volume. A hacked together solution of a 20TB Physical RDM and a couple of 1TB VMDK's was used to virtualise the server into so that it could be given some more disk space.

Needless to say, the attempted ressurection of the system proved fruitless.

It was at the point where I came in a built a new BE2012 system to perform direct-to-tape backups using a TL4000 Library with 4x LT06 Drives and a 10Gb SAN environment. Needless to say, this has been working without major issues for the last 12 months.

Edit: I have no intention of rebuilding the server post backup. It's going to get terminated with extreme prejudice so I can have my 20TB of 1Gb iSCSI storage back.

 

[Arthalen]

Genuine question: Why would you use an RDM mapping to storage over mapping it inside the OS, or is it a desire to keep iSCSI traffic on the storage network and VM network on the VM network?

Neither - It was simply the best solution at the time to Virtualise (P2V) a physical server that had a single 23TB Volume, when running under ESX 5.0, by using the largest available SAN Raid Group (20TB) as an RDM we had with a couple of 1TB VMK files. On the original server in windows, there was a single 20TB Disk from a DAE enclosure with a single 2TB/12-Disk RAID5 Group that had been dynamically expanded with a further 1TB and 2TB RAID5 Groups.

After that detour, the lack of ability for VMware backup products to get independent disks is a limitation of VMware I believe. You're either going to have to use a backup product with an agent like BackupExec or snapshot/replicate it at the SAN level. The MD arrays are pretty basic though so I have no idea what sort of snapshot capabilities they have or how many they can retain.

Your mostly right - It's down to the fact that VMware cannot snapshot these disks, so the backup software cannot snapshot them for backups at the VMDK/VMware level.

Which forces me down in the 'inside Windows' Agent route, with the problems illustrated above.

 

[Arthalen]

Have a look at PHD Virtual, I'm not sure if it supports RDM but it's a worth a look. We currently use it to backup our VMware environment, it's pretty good.

Nice try

From a quick glance I couldn't spot anything that prevents this product from accessing Physical RDMs, but it doesn't seem to support direct Tape Drive backups either and I don't have the 20TB+ of backup storage to play with to do this. Since the original server was a BE2010 DeDupe Backup sevrer in the first place, I wouldn't expect any further DeDupe to be especially effective.

 

[Arthalen]

As long as it isn't used in a cluster there isn't anything stopping you from changing it to a virtual RDM if you think that'll help.
<Link>

Ooh, this could work! Nice suggestion.

It's either that or you have to treat it like a physical server and do an agent level backup.

This is where I'm at, for now. I've devised and ran overnight a little batch script to actually seperate out the huge 19TB Folder so that I can use the Windows Agent to backup the volume in 2TB chunks. That should make it a helluva lot easier for BE2012 to swallow and get me a one-time, to Tape backup of the whole server.

I'll try this over the next few days and see where I can get to. It's a bit messier than what I wanted, but Any backup is better than none right now!

 

[Arthalen]

Thanks for all the suggestions Gents, I'll report back when done.

 

[Arthalen]

This is where I'm at, for now. I've devised and ran overnight a little batch script to actually seperate out the huge 19TB Folder so that I can use the Windows Agent to backup the volume in 2TB chunks. That should make it a helluva lot easier for BE2012 to swallow and get me a one-time, to Tape, backup of the whole server.

I think trying to backup the entire 20TB volume via some sort of whole-volume snapshot will be fruitless anyway -- even if you were successful, if you ever needed to recover something from it, what do you reckon the chances are that you'd be able to dig into such a large image, especially one with so many files? Potentially you would need to restore the entire image just to restore a file.

Here are a few suggestions:

<snip>

Already there

I've completed some test backups today. I was able to get the first 2TB folder backed up sucessfully using the Windows Agent, so I think I'm pretty much golden.

I was able to do that 2TB in around 4 Hours using local disk storage on the BE2012 Backup server itself, so I can then duplciate that backup set to Tape at my leisure (during the day when the sevrer is sat waiting for out-of-hours operations).