VMware Backup Challenge

Arthalen

Arthalen

Arthalen

Associate
Joined
20 Apr 2006
Posts
2,037
Location
Leeds, UK
Ok, here's an interesting once for you.

I have a VMware ESXi deployment consisting of 10x ESXi 5.1 Physical Hosts with back-end Dell iSCSI MD3600 Storage. It currently runs VCentre 5.1.

One of my colleagues, in his infinite wisdom, at one time in ages past dedicated an entire SAN shelf (20TB, to be precise) to a VM via a Physical RDM mapping of the storage to the VM.

We have in place a Physical Backup solution in the form of Backup Exec 2012 using 4x LTO6 tape drives. However, Backup Exec does not support RDM or indeed independant disks in VMware when performing VMDK-level backup jobs.

I've looked at a possible Windows Agent style backup, but have had consistent errors with Backups of very large storage volumes when going over a two-day backup window.

The complication of a repeated 2-day backup to capture all the files is that the storage volume pretty much contains 19TB of 256KB Files, without any kind of folder structure. It would be a nightmare to keep deselecting the backed up files from the selection list.

I've looked at VEEAM, and that has the same limitation around RDMs that Backup Exec has.

So, any thoughts on how to get a backup?
 
If you upgrade to Esxi 5.5 you will be able to use large VMDK's. Then your backup becomes trivial with something like Veeam or Unitrends.
 
tbz_ck said:
If you upgrade to Esxi 5.5 you will be able to use large VMDK's. Then your backup becomes trivial with something like Veeam or Unitrends.
Click to expand...

Regardless of VMDK size limitations (2TB limit under 5.1 has never been a problem to manage or backup so far, despite having a 100TB+ SAN deployment), I need a solution to deal with a 20TB Physical Raw Disk Mapping (RDM).

Once it's backed up, the server and the RDM will be getting permanently removed so the space can be properly presented through a LUN and then managed with individual VMDK files at the correct sizes.

But I need the backup of the server and the attached LUN first :)
 
Is the RDM mapping in virtual or physical compatibility mode?

if you storage vmotion a vm with a RDM in physical compatibility mode, the mapping file is migrated. If this is for a RDM in virtual compatibility mode, the RDM is converted to a vmdk, providing you're selecting same as source.

http://searchvirtualstorage.techtarget.com/definition/Raw-device-mapping-RDM

Virtual mode provides benefits found in VMFS, such as advanced file locking and snapshots. Physical mode provides access to most hardware functions of the storage system that is mapped.

Not that experienced with the RDM mappings but i think i would try and storage vmotion it to a vmfs5 datastore in standard vmdk format. This will allow for normal backups with backup exec.

Im intrigued as to what type of data you would have on a RDM mapping on a san like that, why not just put the data directly on the san in the first place ?

What OS is it ? Why not just backup the data directly and then rebuild the server from scratch?
 
As mentioned, its in physical compatibility mode (Physical RDM), which blows holes in all sorts of 'solutions' I've researched so far.

At the moment, I'm thinking my best bet is BE2012 Windows Agent using Checkpoint restart and File Deletions so I can get a straight copy of the data over multiple attempts.



Some more, interesting, background:

Using RDM's would be the last thing I would ever consider using (since we're invested in Backup Exec 2012 for the time being), but it was a necessity at the time to facilite doing a P2V conversion.

The P2V conversion was of an old BE 2010 DeDupe server that had a (dangerously full) 22TB Single Volume. A hacked together solution of a 20TB Physical RDM and a couple of 1TB VMDK's was used to virtualise the server into so that it could be given some more disk space.

Needless to say, the attempted ressurection of the system proved fruitless.

It was at the point where I came in a built a new BE2012 system to perform direct-to-tape backups using a TL4000 Library with 4x LT06 Drives and a 10Gb SAN environment. Needless to say, this has been working without major issues for the last 12 months.

Edit: I have no intention of rebuilding the server post backup. It's going to get terminated with extreme prejudice so I can have my 20TB of 1Gb iSCSI storage back.
 
Genuine question: Why would you use an RDM mapping to storage over mapping it inside the OS, or is it a desire to keep iSCSI traffic on the storage network and VM network on the VM network?

After that detour, the lack of ability for VMware backup products to get independent disks is a limitation of VMware I believe. You're either going to have to use a backup product with an agent like BackupExec or snapshot/replicate it at the SAN level. The MD arrays are pretty basic though so I have no idea what sort of snapshot capabilities they have or how many they can retain.
 
Have a look at PHD Virtual, I'm not sure if it supports RDM but it's a worth a look. We currently use it to backup our VMware environment, it's pretty good.
 
Arthalen said:
As mentioned, its in physical compatibility mode (Physical RDM), which blows holes in all sorts of 'solutions' I've researched so far.
Click to expand...

As long as it isn't used in a cluster there isn't anything stopping you from changing it to a virtual RDM if you think that'll help.
<Link>

It's either that or you have to treat it like a physical server and do an agent level backup.
 
Caged said:
Genuine question: Why would you use an RDM mapping to storage over mapping it inside the OS, or is it a desire to keep iSCSI traffic on the storage network and VM network on the VM network?
Click to expand...

Neither - It was simply the best solution at the time to Virtualise (P2V) a physical server that had a single 23TB Volume, when running under ESX 5.0, by using the largest available SAN Raid Group (20TB) as an RDM we had with a couple of 1TB VMK files. On the original server in windows, there was a single 20TB Disk from a DAE enclosure with a single 2TB/12-Disk RAID5 Group that had been dynamically expanded with a further 1TB and 2TB RAID5 Groups.

Caged said:
After that detour, the lack of ability for VMware backup products to get independent disks is a limitation of VMware I believe. You're either going to have to use a backup product with an agent like BackupExec or snapshot/replicate it at the SAN level. The MD arrays are pretty basic though so I have no idea what sort of snapshot capabilities they have or how many they can retain.
Click to expand...

Your mostly right - It's down to the fact that VMware cannot snapshot these disks, so the backup software cannot snapshot them for backups at the VMDK/VMware level.

Which forces me down in the 'inside Windows' Agent route, with the problems illustrated above.
 
MartynB85 said:
Have a look at PHD Virtual, I'm not sure if it supports RDM but it's a worth a look. We currently use it to backup our VMware environment, it's pretty good.
Click to expand...

Nice try :)

From a quick glance I couldn't spot anything that prevents this product from accessing Physical RDMs, but it doesn't seem to support direct Tape Drive backups either and I don't have the 20TB+ of backup storage to play with to do this. Since the original server was a BE2010 DeDupe Backup sevrer in the first place, I wouldn't expect any further DeDupe to be especially effective.
 
Little_Crow said:
As long as it isn't used in a cluster there isn't anything stopping you from changing it to a virtual RDM if you think that'll help.
<Link>
Click to expand...

Ooh, this could work! Nice suggestion.

Little_Crow said:
It's either that or you have to treat it like a physical server and do an agent level backup.
Click to expand...

This is where I'm at, for now. I've devised and ran overnight a little batch script to actually seperate out the huge 19TB Folder so that I can use the Windows Agent to backup the volume in 2TB chunks. That should make it a helluva lot easier for BE2012 to swallow and get me a one-time, to Tape backup of the whole server.

I'll try this over the next few days and see where I can get to. It's a bit messier than what I wanted, but Any backup is better than none right now!
 
Last edited:
I think trying to backup the entire 20TB volume via some sort of whole-volume snapshot will be fruitless anyway -- even if you were successful, if you ever needed to recover something from it, what do you reckon the chances are that you'd be able to dig into such a large image, especially one with so many files? Potentially you would need to restore the entire image just to restore a file.

Here are a few suggestions:

- Batch script that sets the Archive bit on every file on the entire drive
- Incremental (not full) agent backup of the drive, but specifying that you only want to backup files "newer than" X (whatever a reasonable X is that backs up a % of the files that completes in 24h, but not all of them). The backup must clear the archive bit. If you re-run the exact same job, it shouldn't backup anything, because the archive bits have been cleared on those files.
- Repeat the job each time, dialling X backwards in time, until you've backed the entire drive up.

- Another possibility could be start with all files a*.*, let that run, then modify the job for b*.*, etc. And then a complete one to mop up any leftovers.

- Another option would be to use something like TAR (or 7-zip with compression turned off or set to a very low value) to group files into a smaller number of large files, which are much quicker to backup. You would have to have some logical boundary (e.g. one TAR file per month of files). Backing up one 100MB file will be MUCH faster than backing 100MB worth of 256kB files (400 files).
 
Arthalen said:
This is where I'm at, for now. I've devised and ran overnight a little batch script to actually seperate out the huge 19TB Folder so that I can use the Windows Agent to backup the volume in 2TB chunks. That should make it a helluva lot easier for BE2012 to swallow and get me a one-time, to Tape, backup of the whole server.
Click to expand...

rotor said:
I think trying to backup the entire 20TB volume via some sort of whole-volume snapshot will be fruitless anyway -- even if you were successful, if you ever needed to recover something from it, what do you reckon the chances are that you'd be able to dig into such a large image, especially one with so many files? Potentially you would need to restore the entire image just to restore a file.

Here are a few suggestions:

<snip>
Click to expand...

Already there :)

I've completed some test backups today. I was able to get the first 2TB folder backed up sucessfully using the Windows Agent, so I think I'm pretty much golden.

I was able to do that 2TB in around 4 Hours using local disk storage on the BE2012 Backup server itself, so I can then duplciate that backup set to Tape at my leisure (during the day when the sevrer is sat waiting for out-of-hours operations).
 
You must log in or register to reply here.
Back
Top Bottom