Raid 5 write performance in proliant range

[ginjaninja]

i am looking to spec a server for reverse incremental backup jobs in veeam.
Currrently on a 5 year old server, the 2 TB daily backup job takes 10 hours to process the reverse incremental (storage subsystem limited)

am looking for next proliant server to be better specced.

in the proliant range, What sort of raid controller(s) do i need to be looking to buy to run some standard class 2TB sas drives (7,2k) at their max write rate for raid 5 over 4 disks (ie so the controller is not a bottleneck)..would sata be just as good for this application? What sort of write speed can i expect to get.

thanks for feedback / ideas.

 

[Avalon]

Before we go any further, if you are doing this commercially what’s the logic behind R5 on mechanical disks when you’ve already stated it’s an IO limited scenario? R5 is the red headed step child of the family, it’s been shunned from the family photo’s for several years now.

 

[skyripper]

Raid5 made sense when disks were measured in £/MB or £/GB. With the cost of storage these days in £/TB and the time required to recalculate a failed array you have to look at 0+1 or non-raid alternatives like ZFS.

Also a server whose purpose is receiving incremental backups should be optimised for write-performance, which 5 really isn't.

For actual performance in a current HPE server with 4 HPE disks in an array - ask the vendor! That's what they're there for.

 

[ginjaninja]

Before we go any further, if you are doing this commercially what’s the logic behind R5 on mechanical disks when you’ve already stated it’s an IO limited scenario? R5 is the red headed step child of the family, it’s been shunned from the family photo’s for several years now.

thanks both for feedback,

is raid 5 not the most cost effective way of achieving the disk redundancy coupled with the write performance?
maybe if we went raid 0+1 ...could we save of the disk controller , put that into the disks and achieve better io for the same costs? and still have redundancy...that sounds like a good suggestion..

Would the standard raid controller with a write cache be as good as it needs to be for 0+1?

(ie no need to spend on a better controller)

 

[skyripper]

Having just read what a Reverse Incremental Backup in Veeam actually is, it appears to be..

You start with a full backup.
You run another backup job, in this case an incremental.
Veeam them magically turns that into a brand new full backup (by merging the incremental changes with the old full backup).
Veeam then even more magically turns the old full backup into a delta from the latest full backup.
And it does this every time you do an incremental.
And presumably its doing a ****-ton of reads and writes all from the same disks, rather than between two different sets of disks (one for the incrementals and old full backup, one for the creation of the latest full backup).

Rather than go to all the effort of creating a new server, disk array etc. - two questions;

• Do you really need to always have the latest backup as a full backup, or could you have A full backup and some incrementals?
• Can you tell Veeam to do its stuff across different sets of disks (so array A for source, array B for target) massively reducing the amount of simultaneous Read+writes to the same disks at the same time?

Can you use some SSDs ( a mirrored pair ) as scratch disks for all the reverse gibberypokery until you get the end result a Latest Full Backup which you then move to mechanical drives?

PS I've never used Veeam.

 

[ginjaninja]

for backing up and replicating and restore virtual environments nothing ive found comes close to veeam, a truly excellent bit of software.

we use reverse incremental so that, even though we have a massive 30 day chain of backup files, there is one file which represents the last backup, which we can copy offsite.
as you say it is very io intensive.

any gibberypokery would have to be in the os/hardware layer, veeam does have this type of capability. am hoping REFS and windows 2016 might introduce some performance gains, more like wishful thinking.

so it looks 7.2k RAID 10 is chepaer than 10K raid 5 disk space for disk space, and by all accounts 7.2K raid 10 will be significantly faster, so thanks for that suggestion..i am going with it...

can i just go with the cheapest raid controller with a write cache for raid 10? will that be as good as anything?

 

[skyripper]

Can i just go with the cheapest raid controller with a write cache for raid 10? will that be as good as anything?

Don't see why not, as long as its supported.
Edit: See Armageus post below

Raid 0+1 is, two (or more) RAID 1 arrays (each disk is a simple exact clones of the other), with a RAID 0 (stripes) spread across the top of them to make a single array.
Raid 1+0 (aka 10) is two (or more) RAID 0 arrays which are simple stripes, which are then merged using a RAID 1 (mirror) across the top of them to make a single array.
better explanation

Raid5 involves striping across 3 or more disks, and calculating parity of those 3 stripes and writing that too. Which takes computing power to calculate.
Cheap controllers offload this to the host server/PC. Expensive ones do it for you.
When you're doing a ton load of Writes, you don't really want the overhead of either.
If you're just reading (and all the disks are fine) then the parity calculations don't happen.

Which is why Raid-5 is fine for data which doesnt change much (think traditional fileservers) vs data which changes a lot (think transactional databases or servers hosting backups).

If you can, set up a write-cache in memory and see if that improves the throughput.
Most controllers will be set to be write-thru caching by default - ie. they will cache reads, but not tell the OS "I'm ready for the next bit" until the disks have completed the write.
If you're doing a ton of writes, it could be beneficial to let those be cached too.

This is where ZFS (and other modern disk file systems) with the ability to add in-memory caches, dedicated cache SSDs etc. really shines over oldskool Raid controllers.

 

[Armageus]

in the proliant range, What sort of raid controller(s) do i need to be looking to buy to run some standard class 2TB sas drives (7,2k) at their max write rate for raid 5 over 4 disks (ie so the controller is not a bottleneck)..would sata be just as good for this application?

Any SmartArray controller will be able to do this

can i just go with the cheapest raid controller with a write cache for raid 10? will that be as good as anything?

No - the cheapest option on Gen10 Servers is Software raid, and the next option is a Cache-less E series SmartArray. The cheapest you should go for would be a P408i.

https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=a00047736enw

 

[bledd]

Raid10 (4 disk)

4x read, 2x write (50% capacity, 1 disk fail)

.

Raid5 (4 disk)

3x read, 1x write (75% capacity, 1 disk fail)


no one should use raid5.

Probably quicker to not pick reverse incremental. But to stagger the full backup (eg if you have 14 servers, do 2x full every night and the rest incremental)

 

[ginjaninja]

thanks again for feedback.

yes...was thinking that ....with enough disk space on the backup server, i would just do forward incremental for 30 points and a separate backup job that just did a separate full backup every day with 1 point.... the two jobs will probably be faster than reverse incremental as you say..

 

[Semple]

How much data are you backing up? It sounds like you're trying to build a server to host your backups on. If you're anything from a small SME upwards you should really be looking at enterprise data protection solutions for a number of reasons.

 

[blueboy2001]

Reverse Incremental is never fast because you're doing 3x the random I/O for every changed block:

Read changed block from backup
Write changed block to Incremental file
Delete changed block from full file
Write new block to full file

You would be better to do forever forward Incremental and use a backup copy job to copy off site.

 

[ginjaninja]

yes backup copy job is ideal, but the reason we discounted this, was after a conversation with Veeam, we came to the understanding that the gui was too restrictive on scheduling.

we wanted to create the backup copy job to run on tuesday, every 28 days,

we could perhaps create the copy job every day and only use it every 28 days...to save on read io on the production storage, but as production is all flash vsan its not really limited....in fact reading from production is probably faster than creating a backup copy job...

we could perhaps investigate using the command line to schedule the backup copy job..

 

[Armageus]

we wanted to create the backup copy job to run on tuesday, every 28 day

Every Fourth Tuesday not good enough?

 

[ginjaninja]

thanks, we are considering changing our process to every x day of the month.

i understand the backup copy job repository needs to keep two points on top of the one we really want.. eg fourth tuesday of month
the reason we go to the trouble of reverse incremental is we have a single vbk to copy offsite - so thats not ideal.

and i wonder if the vbk we want inside the backup copy job is going to have a predictable name, so we can easily script its copy.

anyways have implemented suggestion in test and will see how it goes.

upon reflection, it maybe that a scripted and scheduled (via windows task scheduler) standalone full backup job may be offer the best of all worlds, albeit with greater load on production storage, in that it can be safely pointed at a removable disk (whereas i understand a repository wont like that so much) AND scheduled for every 28 days..will try that next..

 

[Liquidfox]

upon reflection, it maybe that a scripted and scheduled (via windows task scheduler) standalone full backup job may be offer the best of all worlds, albeit with greater load on production storage, in that it can be safely pointed at a removable disk (whereas i understand a repository wont like that so much) AND scheduled for every 28 days..will try that next..

It'll handle it fine as long as you tick the option for "This repository is backed by rotated hard drives" in the advanced options part.

 

[ginjaninja]

thanks thats a good setting. so looks like i can force the full backup to a repository on a removable drive every 28 days via a powershell script and windows scheduler. and get rid of reverse incremental on the main job. best of all worlds. will try tomorrow.

 

[Hellsmk2]

Have you looked at EMC Data Domains instead? With a license that includes ddboost, you'll probably be able to strip those backup times down to an hour or two each day.

You could add a second data domain off site and use dd replication, or a backup copy job, all while taking advantage of the dd's dedupe ability to cut down on data hitting your network.

 

[ginjaninja]

thanks thats a good setting. so looks like i can force the full backup to a repository on a removable drive every 28 days via a powershell script and windows scheduler. and get rid of reverse incremental on the main job. best of all worlds. will try tomorrow.

this is working well, veeam automatically recreates repository folder structure on external drive, and i think it even has logic to guess the removable drive letter if it changes.

emc data domains..will take a look..although we need a full self contained backup (so no deduplication accross different media) and the limiting factor is the the single sata disk in a caddy.
were getting 200MB/s (400MB/s uncompressed), so thats pretty good.
annoys me that HP and DELL still dont provide an esata card in their range, have to hook up a cable/riser card to the motherboard to get esata support.

 

[Armageus]

annoys me that HP and DELL still dont provide an esata card in their range, have to hook up a cable/riser card to the motherboard to get esata support.

External SAS HBA (e.g. HP SC08GE) and SAS SFF8088 to eSATA breakout cable?