Best way to store DVD/Blu-ray rips on NAS/RAID

MarcLister · 24 Aug 2014 at 12:55

I have a lot of DVDs and Blu-rays in my room and I've come to the conclusion that I would like the space back so I've decided to start to rip my collection to disk to allow me to store my collection in the loft.

Would I be right in assuming that I need a maximum of 9GB per DVD and 50GB per Bluray worth of space on my NAS? I don't have an accurate number of disks, only the number of films etc. My plan is to get two WD Red drives to begin with and RAID them, as I won't be able to rip my entire collection in a weekend! I'm not yet decided on whether to get the 4TB, 5TB or 6TB. Cost wise per GB the 4TBs look best but in terms of most space the 5TB or 6TB make more sense. Could I start off with two WD Reds in say RAID 1 and then add another two WD Reds later to the same RAID 1 array? Or would I need to have one RAID array per pair of WD Reds?

I will be using my current server that is running WHS 2011 with an Intel Celeron G550 and a Gigabyte Z86AP-D3. Can I run enough RAID'd disks off the motherboard or should I buy a dedicated RAID card? If so, can someone please suggest some decent RAID cards?

My big conundrum is how much space to ultimately end up with. I keep my collection details in EMDB and I have 703 DVD titles and 299 Blu-ray titles. As I said, I'm not sure how many discs that actually is as some of the titles will be TV series etc so potentially 3-4 discs. I calculated that 9GB per DVD x 703 titles is just over 6TB but that is a maximum as not every DVD disc will be ripped, such as special feature disks and so on, and not every ripped disc will have 9GB of video to be stored. Worryingly 300 Blu-ray discs at 50GB equals 15TB!! So just for my current DVD/Blu-ray collection I potentially need a maximum of 21TB of space. Does that sound right? I half think I've miscalculated that.

If I bought 4 pairs of WD Red 6TBs I'd have 24TB of storage to be RAID'd to RAID 1. Can I run 8 x WD Reds in a single RAID 1 array or would I have to set up 4 RAID arrays, one for each pair of drives? If this is all possible, would my server cope with 8, or more, WD Reds in RAID array(s)? Would I need to upgrade the server to cope with the number of hard drives, such as perhaps getting a dedicated RAID card to lighten the load on the RAID ports on the motherboard?

I'm still planning this so I'm open to ideas, suggestions and constructive criticism of my plan. I've not done this before hence my asking for help; I'd rather have my plans and calculalations checked over by more knowledgeable people than dive into this head first on my own and get it wrong or waste hundreds of pounds on hard drives that won't work.

Frozennova · 24 Aug 2014 at 13:13

Dedicated raid card everytime.

It's far easier to replace a dedicated controller with the same model than it is a motherboard, especially a few years down the line. It will also give much better speeds than an on board controller especially with an array of that size

Nothing wrong with having 8x6TB in a single RAID1 array

nutcase · 24 Aug 2014 at 13:20

Try and find something with decent dedupe - saves quite a bit of space.

MarcLister · 24 Aug 2014 at 14:47

Frozennova said:
Dedicated raid card everytime.

Cool. Can you suggest any decent cards for me to look up reviews?

Frozennova said:
It's far easier to replace a dedicated controller with the same model than it is a motherboard, especially a few years down the line. It will also give much better speeds than an on board controller especially with an array of that size

OK, makes sense.

Frozennova said:
Nothing wrong with having 8x6TB in a single RAID1 array

Nice. Can I start off with 2x6TB and add 2 more at a time?

nutcase_1uk said:
Try and find something with decent dedupe - saves quite a bit of space.

Dedupe? That's data deduplication? Would it be the RAID card that handles dedupe?

Frozennova · 24 Aug 2014 at 14:58

The Dell H700 is recommended quote often and is reasonably priced too.

I've never personally dabbled into hardware raid that much to say if you can add drives or not. I currently use ZFS for my main storage and MDADM for iscsi.

MarcLister · 24 Aug 2014 at 15:31

Frozennova said:
The Dell H700 is recommended quote often and is reasonably priced too.

Looks to be around £130 on eBay from what I can see. Does that sound right? I've found some threads on OcUK where it seems that the H700 only works with Dell drives or something? Am I misreading stuff?

Frozennova · 24 Aug 2014 at 17:44

MarcLister said:
Looks to be around £130 on eBay from what I can see. Does that sound right? I've found some threads on OcUK where it seems that the H700 only works with Dell drives or something? Am I misreading stuff?

Sounds about right. Haven't seen it mentioned myself but the Dell cards are all rebranded LSI cards so they can be cross flashed with the LSI firmware of they did have trouble with anything other than dell drives.

I have an earlier sas 6/ir which works great but they're fairly old now and only support maximum of 2tb drives. Unfortunately I never even plugged any drives into it while it still had the Dell firmware as I flashed it as soon as I got it.

The other option would be a brand new LSI card but they're around £250

nutcase · 24 Aug 2014 at 17:53

MarcLister said:
Dedupe? That's data deduplication? Would it be the RAID card that handles dedupe?

I think it's either at OS level or hardware level. My preference is for OS level as if the hardware fails the backup is still readable by the OS when it's restored, even if new hardware of a different type is used. I user server 2012 R2 which has it native but obviously that's not cheap.

MarcLister · 24 Aug 2014 at 18:15

Cheers. Will the Dell H700 have any problems with 4TB+ drives? From my research of the H700 it has two ports to which 4 hard drives can be connected to both? Is that right?

Still need to find out if I can start off with 2 drives in an array and add 2 more at a time as space requirements demand it. Once I'm happy that I can do that then I'll look at buying the hardware I need to install into my server.

Frozennova · 24 Aug 2014 at 19:25

Support for 4+tb drives is there out of the box

Correct you need an SFF-8087 to 4 sata forward breakout cable iirc

It needs to a forward break out as reverse breakout is for connecting the card to a backplane rather than direct to the drives.

MarcLister · 24 Aug 2014 at 22:29

Frozennova said:
Support for 4+tb drives is there out of the box

Correct you need an SFF-8087 to 4 sata forward breakout cable iirc

It needs to a forward break out as reverse breakout is for connecting the card to a backplane rather than direct to the drives.

So would need two of the SFF-8087 cables? Look to be about £15-£20 each.

Frozennova · 24 Aug 2014 at 23:21

MarcLister said:
So would need two of the SFF-8087 cables? Look to be about £15-£20 each.

Yea, one for each of the two ports on the card. So you can buy one and run with 4 drives then later down the line buy another and connect another 4 drives

Ch3m1c4L · 27 Aug 2014 at 09:32

A lot of modern raid cards will support online expansion of an array, iirc my LSI 9261-8i does, the H700 is an LSI 9260-8i so I would assume it does, but it would be worth reading the doco that comes with it (LSI site).

You could look at RAID 6 instead of RAID 1 if you need more space in 8 drives, rather than just ending up adding another 8 drives. Since your current collection takes up that much space, you will want more space *fairly* soon.

Case: I went with a Fractal Design XL USB 3, due to anti vibration on hdds and how many can fit in it, and its super silent compared to my tj-07 at holding HDD's. That many make a lot of vibrations, its worth keeping in mind for case choice, and drive choice as it makes them more likely to fail. (hence enterprise drives which you are looking at in the reds).

Backblaze have published numbers from there thousands of consumer drives in raid setups, Seagate fail the most, wd next and hitachi are the most reliable.

Yes you need two cables, they can be hard cheaper than £15-20. I bought a couple of cheap ones from random place online and they work fine.

You could always save space by ripping at slightly lower than maximum bitrate. Due to the number of your titles, just saving 1GB per dvd and ~5-10GB per blu ray would save a huge amount of space for you, and probably wouldnt be visually noticeable.

MarcLister · 27 Aug 2014 at 11:11

Ch3m1c4L said:
A lot of modern raid cards will support online expansion of an array, iirc my LSI 9261-8i does, the H700 is an LSI 9260-8i so I would assume it does, but it would be worth reading the doco that comes with it (LSI site).

Cheers. I will look up the LSI 9261-8i specifications and see what I'll need to do when I add more hard drives.

Ch3m1c4L said:
You could look at RAID 6 instead of RAID 1 if you need more space in 8 drives, rather than just ending up adding another 8 drives. Since your current collection takes up that much space, you will want more space *fairly* soon.

I'm not sure how RAID 6 would benefit me more than RAID 1. I've had a quick look at RAID 6 and to me it just seems that I would need more disks to achieve the same amount of storage as I would get with RAID 1. I'm by no means a RAID expert so I might be reading things wrong but for the moment RAID 1 seems to be OK for me.

Ch3m1c4L said:
Case: I went with a Fractal Design XL USB 3, due to anti vibration on hdds and how many can fit in it, and its super silent compared to my tj-07 at holding HDD's. That many make a lot of vibrations, its worth keeping in mind for case choice, and drive choice as it makes them more likely to fail. (hence enterprise drives which you are looking at in the reds).

My server is currently in an Antec 300 case. My current PC case in an Antec 1100. I have plans to build a new PC in the near future so I could move my server into my Antec 1100 case for the extra space for the hard drives.

Ch3m1c4L said:
Backblaze have published numbers from there thousands of consumer drives in raid setups, Seagate fail the most, wd next and hitachi are the most reliable.

Yeah I've seen that report and for my purposes I shall be using the WD Reds.

Ch3m1c4L said:
Yes you need two cables, they can be hard cheaper than £15-20. I bought a couple of cheap ones from random place online and they work fine.

Hmm, OK. I'll do some googling and see what I can find.

Ch3m1c4L said:
You could always save space by ripping at slightly lower than maximum bitrate. Due to the number of your titles, just saving 1GB per dvd and ~5-10GB per blu ray would save a huge amount of space for you, and probably wouldnt be visually noticeable.

I've ripped a few Blu-rays to my PC just as a test of the speed of my Blu-ray drive. I've got a newer one being delivered in the next 90 minutes which is hopefully going to be 2-3 times faster, which will be nice! :cool:

I've only ripped Blu-rays so far but I have ripped 13 Blu-rays and they are taking up 326GB which is giving me an average of just under 22GB per disk. The 50GB per Blu-ray and 9GB per DVD was me calculating the absolute maximum of storage I would need. I plan to run the rips through Handbrake to reduce the file size a bit and see if I can do that without losing too much visual quality.

Ch3m1c4L · 27 Aug 2014 at 13:03

Sounds good. The RAID 6 vs Raid 1 is a case of in RAID 6 you lose 2 drives of space, in raid 1 you lose 50% of drives. so, below 4 drives it makes no difference, but when you get to 8 drives, it means 2 drives more space, while being able to take any two drive failures before the 3rd failure taking it out.

Raid 1 you can take up to 50% failures, but if two of the drives that fail happen to be the ones mirroring each other, you lose that data.

MarcLister · 27 Aug 2014 at 13:19

OK. I'll research in more detail RAID 6. Sounds like a better way to do things.

accod · 28 Aug 2014 at 21:34

+1 on the RAID 6, gives me more peace of mind once a drive has failed knowing that the array can definitely handle another failure.

Before you set your mind on getting this much storage, are you sure you need it? If you transcode the media, you'll save a lot of space and lose little quality (assuming you do it right).

MarcLister · 28 Aug 2014 at 22:30

accod said:
+1 on the RAID 6, gives me more peace of mind once a drive has failed knowing that the array can definitely handle another failure.

Cool. Guess I'll go RAID 6 then.

accod said:
Before you set your mind on getting this much storage, are you sure you need it? If you transcode the media, you'll save a lot of space and lose little quality (assuming you do it right).

My plan was always to calculate the absolute maximum I would need for my current media library. I also planned to run everything through Handbrake to reduce the file size to a decent level to help even more with space usage. I calculated 24TB as my absolute maximum and considering I'm averaging around 25GB per Blu-ray film at the moment, half of my maximum, I could get away with 16TB of storage, to give me space to add to my collection as I buy more films.

Oh, and I won't getting 24TB in one go, the plan was to start off with 2 x 6TB drives, RAID them up and fill them up. As I got close to filling that up, get another pair of 6TB drives and carry on until I'm finished, so an incremental increase in storage available, rather than assume I need 24TB and get that much storage in one go.

accod · 29 Aug 2014 at 08:08

MarcLister said:
Oh, and I won't getting 24TB in one go, the plan was to start off with 2 x 6TB drives, RAID them up and fill them up. As I got close to filling that up, get another pair of 6TB drives and carry on until I'm finished, so an incremental increase in storage available, rather than assume I need 24TB and get that much storage in one go.

Ah ok, cool.

Consider if you want a form of backup too. Raid is not a replacement for backups.

I tend to use old drives for backup - when I need more space, I replace the live drives with larger and use the old for backups.

Frozennova · 29 Aug 2014 at 09:21

IMHO RAID6 with drives of that capacity is a waste of time.

You've got somewhere in the region of a 30% chance of getting a URE on a drive during a rebuild with 6Tb drives.

They're a far too high a capacity to be considering anything other than RAID1 personally.