File System & RAID Recommendations

stringsonfire · 4 Mar 2015 at 20:20

I'm planning to build a server with 3 large hard drives in an array. I was planning on a RAID 5 array, expanding it with new drives as they get cheaper. However, given it'll have a Xeon E3-1245 v3, perhaps a Hardware RAID Card isn't the best option. As I haven't looked into this in a few years, there's also a good chance that RAID5 isn't the best solution anymore. Would anyone mind bringing me up to date?

I may be running a Host OS built for virtual environments, with various Windows and Linux systems running on it - I'm not sure if this is a factor.

Let me know if this should sit in the servers category...

Thanks!

Justintime · 5 Mar 2015 at 11:42

Would do 4 Disk RAID 10 with ZFS like with Freenas etc.., RAID 5 on large disks is asking for trouble.

stringsonfire · 5 Mar 2015 at 19:46

RAID 10 seems like a waste of capacity, given the price of large drives. Is there something better than RAID5 with similar features?

Frozennova · 5 Mar 2015 at 20:24

Personally using large disks (Larger than 2TB) then unless you are using RAID1 or RAID10 then using RAID IMO is pointless, the risks of a second drive erroring are far too likely on consumer grade drives.

Secondly, as you will be running virtual machines from these drives again RAID10 would be recommended due to the performance benefits

stringsonfire · 5 Mar 2015 at 21:16

Sorry, I should have clarified - the VMs will all sit on an SSD. The hard drives will hold backups of the VMs and other computers as well as various media files for use with Plex (on one of the VMs). RAID 10 is a bit too expensive. I can't find much on ReFS to show it's really ready for use, and I'm not sure on ZFS given the huge RAM requirements, so still very much undecided

Frozennova · 5 Mar 2015 at 22:55

Going from your other thread I assume you'll be using ESXI?

If so how are you planning on setting up your drives?

SSD as a datastore for VMs then a VM setup as a NAS which will act as your backup server?

If RAID10 with large drives is too expensive then I think you need to reconsider your requirements and budget.

Realisticly your options are as follows.

Downsize the storage/TB per drive requirement and use RAID5 as smaller disks will mitigate the issues presented with RAID5 and larger drives.

Increase budget and use RAID10

Increase budget and use Enterprise grade drives (Western Digital SE or similar) as these again will mitigate the uncorrectable read error issue when using RAID5 with large disks.

1GB RAM per TB of raw storage is recommended for ZFS for a requirement, I haven't personally attempted to see what the performance difference is as I'm currently on 12GB ram 8TB of storage

Finally if you did opt for ZFS you could implement the ZFS equivalent of RAID10 incrementally by "pooling" pairs of drives in mirrors

mrbazmondo · 6 Mar 2015 at 01:58

You could use RAID6. Its like RAID5 but can cope with 2 disk failures rather than 1 like RAID5 does. But then you'd need more drives for it to make sense.

Working out parity is slow unless you have an expensive RAID card with non volatile/battery backed cache. It'd be cheaper to buy more disks and do RAID10.

stringsonfire · 6 Mar 2015 at 10:05

A RAID Card would be great if it'll work - I can get a RAID5 8-drive card with battery for £250, pretty good against the cost of 6TB drives, around £200. However, I've read that RAID5, 5+1 and 6 are no good for 4TB+ drives.

"SSD as a datastore for VMs then a VM setup as a NAS which will act as your backup server?" - Exactly, yes.

I'm not sure there'd be any point in going for ZFS if I'm going to use mirroring, as I may as well use RAID10?

The RE and SE drives are too expensive, and probably overkill for a home media/backup/storage/web dev server.

With RAID10, I assume it can be expanded, but you'd always have to add drives in pairs? How does that work? If I fill 5TB on a mirrored pair, then add a new pair, would it be reallocated to store 2.5TB on each mirrored pair?

Frozennova · 6 Mar 2015 at 10:56

With RAID10, I assume it can be expanded, but you'd always have to add drives in pairs? How does that work? If I fill 5TB on a mirrored pair, then add a new pair, would it be reallocated to store 2.5TB on each mirrored pair?[/QUOTE]

A RAID Card would be great if it'll work - I can get a RAID5 8-drive card with battery for £250, pretty good against the cost of 6TB drives, around £200. However, I've read that RAID5, 5+1 and 6 are no good for 4TB+ drives.

Correct, due to Uncorrectable Read Error rates its more than likely that during a rebuild another drive in the array will fail

I'm not sure there'd be any point in going for ZFS if I'm going to use mirroring, as I may as well use RAID10?

ZFS has a lot of benefits over a hardware RAID setup for both data integrity and performance

With RAID10, I assume it can be expanded, but you'd always have to add drives in pairs? How does that work? If I fill 5TB on a mirrored pair, then add a new pair, would it be reallocated to store 2.5TB on each mirrored pair?

I would asssume a RAID10 array on a hardware based RAID controller can be expanded but yes you would always need to add drives in pairs.

Again due to my lack of familiarity with the usage of hardware RAID I would assume it wouldnt redistribute your data across the drives should you add more to the array. The same applies for ZFS so you have to do this manually by moving data off the array and back on

And heres a screenshot showing the disk performance of a Windows 7 Virtual Machine running from a ZFS pool. This is 4 disks in a JBOD pool (No Parity or Mirroring)

as you can see that far exceeds what mechanical disks are capable of, I could also improve performance further with more RAM, and adding a pair of SSDs for the L2ARC cache and ZIL

EDIT:

and here is the performance of the same Virtual Machine running on a Kingston V300 60GB SSD

stringsonfire · 6 Mar 2015 at 17:51

Ok. If I go for software RAID, any recommendations on what SATA cards to get? Given the TS440 uses mini-SAS, I think an LSI 9211-8i should do the job well?

Frozennova · 6 Mar 2015 at 17:57

The LSI 9211-8i is perfect.

You also have the options of rebadged LSI cards such as the IBM M1015 crossflashed with the LSI "IT" Firmware.

I'm doing the same albeit with and older generation DELL SAS6/ir

OhEsEcks · 6 Mar 2015 at 18:06

I have a 12x2TB RAID 6 array formatted as XFS with Fedora server. Also, Nova, who designated the hostname of that machine? I name all my stuff after Greek titans and Gods. So my server is Hyperion (Titan) and my desktop is Helios (God) and my laptop is Helia (one of the Heliades, daughters of Helios).

coupe69 · 6 Mar 2015 at 18:18

OhEsEcks said:
I have a 12x2TB RAID 6 array formatted as XFS with Fedora server. Also, Nova, who designated the hostname of that machine? I name all my stuff after Greek titans and Gods. So my server is Hyperion (Titan) and my desktop is Helios (God) and my laptop is Helia (one of the Heliades, daughters of Helios).

I'm not going to even ask why you need that much storage space

keyser · 6 Mar 2015 at 20:09

He's got a backup of the internet.

I assume you guys on RAID have 2 systems - one for backup? Because as we all know, RAID isn't a backup..

Whats wrong with drive A, backup drive A, drive B, backup drive B?

Avalon · 6 Mar 2015 at 20:17

coupe69 said:
I'm not going to even ask why you need that much storage space

When micro servers first became cheap 6x2TB was quite a popular build spec. They're surprisingly easy to fill up if you for example you want to back up your software/media collection/photo's. Of course the last two could easily be a euphuism for pr0n.

Frozennova · 6 Mar 2015 at 21:01

OhEsEcks said:
I have a 12x2TB RAID 6 array formatted as XFS with Fedora server. Also, Nova, who designated the hostname of that machine? I name all my stuff after Greek titans and Gods. So my server is Hyperion (Titan) and my desktop is Helios (God) and my laptop is Helia (one of the Heliades, daughters of Helios).

Hostname of my own choice, I use Greek Gods and Titans for naming too.

keyser said:
He's got a backup of the internet.

I assume you guys on RAID have 2 systems - one for backup? Because as we all know, RAID isn't a backup..

Whats wrong with drive A, backup drive A, drive B, backup drive B?

Technically single system, no parity/redundancy in my RAID array, the NAS is local storage/backup.

All important data such as Photos etc are backup midnightly offsite via rsync

stringsonfire · 21 Mar 2015 at 20:14

I think I'll make do with my existing array for as long as I can, maybe I can hold out for a year and the release of the new Windows Server (and a more refined ReFS). Either way, I'm pretty sold on RAID10, given I don't want a second, mirror device.

nutcase · 21 Mar 2015 at 21:09

RAID 5 is all but dead these days from what I've learned recently.

OBR10 (One big RAID 10) is pretty much all that's recommended.

stringsonfire · 21 Mar 2015 at 22:13

nutcase_1uk said:
RAID 5 is all but dead these days from what I've learned recently.

OBR10 (One big RAID 10) is pretty much all that's recommended.

Thanks, I hadn't seen that before. I'll still be using RAID10 as I have a separate SSD spare I may as well use for the OS drive, unless I could somehow use that for caching of some kind instead?