Best way to store DVD/Blu-ray rips on NAS/RAID

[accod]

IMHO RAID6 with drives of that capacity is a waste of time.

You've got somewhere in the region of a 30% chance of getting a URE on a drive during a rebuild with 6Tb drives.

They're a far too high a capacity to be considering anything other than RAID1 personally.

I'd agree with you had you said Raid 5. With Raid 6 the read error will almost certainly be recoverable due to the extra parity drive.

What's to stop an URE on a Raid 10 array?

 

[Frozennova]

I'd agree with you had you said Raid 5. With Raid 6 the read error will almost certainly be recoverable due to the extra parity drive.

What's to stop an URE on a Raid 10 array?

Nothing to stop a URE on a Raid 1 array. But given 6TB drives I'd personally feel a lot more comfortable with the additional redundancy of RAID1 over Raid6

 

[MarcLister]

Consider if you want a form of backup too. Raid is not a replacement for backups.

I tend to use old drives for backup - when I need more space, I replace the live drives with larger and use the old for backups.

Hang on! How much money do you think I have?!

So you think I should get a second server or NAS device to backup my media library onto? If I get 20TB of storage say, you reckon I need a second 20TB to back it up?

 

[Grrrrr]

In an ideal world you'd have your main storage (with redundancy), on site backup (with redundancy) and off site backup.

So you'll be needing 3 NAS units and about 15 hard drives

 

[Ch3m1c4L]

Frozennova, ure is a problem with raid 5, on raid 6 the chance of it being unrecoverable is so small its not worth considering. From experience on raid 5, the array doesnt fail either, it just loses that little bit of data.
On raid 1, you lose a drive, therefore then only have one copy of that data, and your saying 30% chance, which would apply to that drive which is now not mirrored, so just as likely to lose some data.
Raid 6 having dual parity along with data means in theory URE shouldn't be an issue at all, unless losing multiple drives at the same time.

 

[MarcLister]

In an ideal world you'd have your main storage (with redundancy), on site backup (with redundancy) and off site backup.

So you'll be needing 3 NAS units and about 15 hard drives

*gulp*

Google's cloud storage is $200 a month for 20TB.

Would a server in a shed at the bottom of the (not very big) garden count as sufficiently off-site?

 

[accod]

*gulp*
Would a server in a shed at the bottom of the (not very big) garden count as sufficiently off-site?

Nope, one lightning strike and the lot is toast.

Depends how much you value the time required to rerip. You've got the DVDs in the loft so you're unlikely to lose the data but you'll be ****** if you have to rerip everything. I'd settle for a RAID 6 array with a few hdd's kept offline (and disconnected!) containing your most important data - ideally stored at friends/parents.

 

[nutcase]

Frozennova, ure is a problem with raid 5, on raid 6 the chance of it being unrecoverable is so small its not worth considering. From experience on raid 5, the array doesnt fail either, it just loses that little bit of data.
On raid 1, you lose a drive, therefore then only have one copy of that data, and your saying 30% chance, which would apply to that drive which is now not mirrored, so just as likely to lose some data.
Raid 6 having dual parity along with data means in theory URE shouldn't be an issue at all, unless losing multiple drives at the same time.

Most likely time for a second drive to fail is during a rebuild after a first one failed. On large drives the rebuild time on a RAID 5 array is measured in days.

 

[Tomsk]

I have ~300 titles of DVD and Blu-ray. I guess about 10% of those will be box sets.
I'm about halfway through ripping everything through Handbrake.
A DVD movie takes about 1 - 1.5GB (1.3Mbps) and a Blu-ray movie 5 - 7GB (6Mbps)
My server has ~17TB and runs Windows 7 and StableBit.
I've enabled folder and file duplication so if one disk fails I don't lose anything.
Also with Stablebit I can add HDs of different sizes at any time.

 

[MarcLister]

I have ~300 titles of DVD and Blu-ray. I guess about 10% of those will be box sets.
I'm about halfway through ripping everything through Handbrake.
A DVD movie takes about 1 - 1.5GB (1.3Mbps) and a Blu-ray movie 5 - 7GB (6Mbps)
My server has ~17TB and runs Windows 7 and StableBit.
I've enabled folder and file duplication so if one disk fails I don't lose anything.
Also with Stablebit I can add HDs of different sizes at any time.

Hi Tomsk. Did you use MakeMKV at all?

What settings/options are you using with Handbrake? I've never tinkered with the options and could be missing out in terms of smaller file size with still decent visual quality.

Also have you RAID'd your 17TB of storage? If so, which RAID did you go for and why? Are you using WD Reds?

 

[Tomsk]

I've only ever used Handbrake.

Handbrake settings are pretty much left on default.
Container mp4, large file size enabled.
Display size 640x480, 720x576 or 1920x1080 depending on source.
Anamorphic strict, custom cropping to zero.
H.264 codec, 1333 or 6000 kbps avg bitrate, 2 pass with turbo first pass, frame rate same as source.

I'm not that fussed on quality. I believe if you're pixel-peeping at the the movie (or whatever) you're watching then it must be pretty boring so why bother watching it.

Don't use RAID. It's not needed with StableBit. StableBit implements disk pooling like Windows Home Servers Drive extender. The advantage is that files on each HD can be accessed as normal (after they're unhidden).

https://stablebit.com/ it's only $20

I don't know what my HDs are. They're a mix of 1, 2 and 3TB drives. Some were B-grades form OcUK. The OS is on a 320GB.

 

[MarcLister]

I've only ever used Handbrake.

Handbrake settings are pretty much left on default.
Container mp4, large file size enabled.
Display size 640x480, 720x576 or 1920x1080 depending on source.
Anamorphic strict, custom cropping to zero.
H.264 codec, 1333 or 6000 kbps avg bitrate, 2 pass with turbo first pass, frame rate same as source.

What I'm doing with MakeMKV is probably to get a 100% rip of a DVD/Blu-ray and then use Handbrake to reduce the file size.

I'm not that fussed on quality. I believe if you're pixel-peeping at the the movie (or whatever) you're watching then it must be pretty boring so why bother watching it.

True.

Don't use RAID. It's not needed with StableBit. StableBit implements disk pooling like Windows Home Servers Drive extender. The advantage is that files on each HD can be accessed as normal (after they're unhidden).

https://stablebit.com/ it's only $20

Hmm, heard of StableBit a while ago but wasn't aware it could help me out. I'll definitely check it out. If it means I wouldn't need to buy two hard drives at a time for a RAID implementation, it could halve my projected hard drive costs.

I don't know what my HDs are. They're a mix of 1, 2 and 3TB drives. Some were B-grades form OcUK. The OS is on a 320GB.

So you must have quite a few disks in your server? My thinking behind getting either 5 or 6TB drives is that you get more capicity per disk and so can get more into a machine. Of course you run the risk of losing more data per disk if one ever fails!

 

[Skilty]

What about using FreeNAS/FreeBSD and RAIDZ2?

I am currently running 12 x 3TB 3.5" WD Red drives for 36TB raw storage for MKV and iTunes rips (means I can Handbrake again if I switch from AppleTV ) I have two pools, each with six drives and has been rock solid.

As for offsite backups I have a second slow server at my father-in-laws (it has one pool and is striped as it is for backups only) which I did an initial ZFS sync on and now use an incremental ZFS sync over SSH...

 

[FrankJH]

A lot of modern raid cards will support online expansion of an array, iirc my LSI 9261-8i does, the H700 is an LSI 9260-8i so I would assume it does, but it would be worth reading the doco that comes with it (LSI site).

You could look at RAID 6 instead of RAID 1 if you need more space in 8 drives, rather than just ending up adding another 8 drives. Since your current collection takes up that much space, you will want more space *fairly* soon.

Case: I went with a Fractal Design XL USB 3, due to anti vibration on hdds and how many can fit in it, and its super silent compared to my tj-07 at holding HDD's. That many make a lot of vibrations, its worth keeping in mind for case choice, and drive choice as it makes them more likely to fail. (hence enterprise drives which you are looking at in the reds).

Backblaze have published numbers from there thousands of consumer drives in raid setups, Seagate fail the most, wd next and hitachi are the most reliable.

Yes you need two cables, they can be hard cheaper than £15-20. I bought a couple of cheap ones from random place online and they work fine.

You could always save space by ripping at slightly lower than maximum bitrate. Due to the number of your titles, just saving 1GB per dvd and ~5-10GB per blu ray would save a huge amount of space for you, and probably wouldnt be visually noticeable.

Don't buy the vibration excuse at all - maybe Im lucky but Ive been running a 8 or 9 disk unraid server for 3 years and only lost one 2tb disk to wear and tear (power cut with no ups at the time)

These are all desktop drives too.

If one can (and price differential isn't too bad) then no point not going for it but imo there is no point choosing a drive model because of "anti vibration"

A DVD movie takes about 1 - 1.5GB (1.3Mbps) and a Blu-ray movie 5 - 7GB (6Mbps)
.

You must be absolutely obliterating those blu's as well as only interested in stereo audio (HD audio usually takes up a lot of the space)

Out of the 500+ BR's Ive ripped at full quality a few of the cheaper/ less quality orientated releases are around 10-15GB vs top quality for something like Aliens (Dir cut) is more like 40GB - average over the 500 is approx. 25GB or 40 films per TB

I believe if you're pixel-peeping at the the movie (or whatever) you're watching then it must be pretty boring so why bother watching it.
.

Absolutely not true at all - just a decent quality and size of TV will immediately tell anyone even slightly observant the difference between a 7GB file and one 3 times the size. Obviously on a 32-40" tv these things will pass most by, but imo if someone is buying 200 BR's they are surely interested in quality (otherwise why not just get the DVD instead)

Also contemplate what your rips will look like after your next tv upgrade, whether that's in 2 or 5 years time (which will normally include a screen size improvement as well as quality)..........

 

[MarcLister]

What about using FreeNAS/FreeBSD and RAIDZ2?

What about it? Would it be a cheaper option?

I am currently running 12 x 3TB 3.5" WD Red drives for 36TB raw storage for MKV and iTunes rips (means I can Handbrake again if I switch from AppleTV ) I have two pools, each with six drives and has been rock solid.

If a hard drive failed, what would you lose data wise?

As for offsite backups I have a second slow server at my father-in-laws (it has one pool and is striped as it is for backups only) which I did an initial ZFS sync on and now use an incremental ZFS sync over SSH...

Haha. I don't think I have the bandwidth or allowance to transfer that much data and keep it up-to-date. In any case I don't know anyone far away from my current home to be sure the server would be isolated from any lightning strikes I might get here yet close enough to be accessible.

Out of the 500+ BR's Ive ripped at full quality a few of the cheaper/ less quality orientated releases are around 10-15GB vs top quality for something like Aliens (Dir cut) is more like 40GB - average over the 500 is approx. 25GB or 40 films per TB.

25GB is my average so far as well. This is just from raw MKVs that I've not yet passed through Handbrake.

imo if someone is buying 200 BR's they are surely interested in quality (otherwise why not just get the DVD instead)

If that was aimed at me, it is correct. I find that most films/TV series I buy on Blu-ray are similar in terms of price to the DVD equivalent, if not, understandably, a bit more expensive but justiably so I find. I tend to research the quality of a Blu-ray film before I buy it, perhaps not so much with modern films, but with older films I do research to see if the Blu-ray 'upgrade' is worth it over the DVD.

 

[FrankJH]

25GB is my average so far as well. This is just from raw MKVs that I've not yet passed through Handbrake.

If that was aimed at me, it is correct. I find that most films/TV series I buy on Blu-ray are similar in terms of price to the DVD equivalent, if not, understandably, a bit more expensive but justiably so I find. I tend to research the quality of a Blu-ray film before I buy it, perhaps not so much with modern films, but with older films I do research to see if the Blu-ray 'upgrade' is worth it over the DVD.

It was a general comment that it seems a little strange to buy a huge amount of BR's and then obliterate majority of them down to 6-7GB per file.(I think Tomsk said it originally)

(Im sure there are some files I have that could be got down to that size, but Im pretty confident its such a small number of the 500~ total it wouldn't make any difference

I take your point about researching the quality of older films, but as I have few DVD's these days it makes little difference. I rarely buy top price releases these days anyway , and to me a cost of a br is comparable to a cinema ticket for the most part so its a reasonable price to be able to watch it in very good quality several times (if I want to) or sell it on for a "small profit" if I don't like it

 

[wengole]

Just thought I'd share what I'm doing.

Got a "server" in the loft with 4 2TB Seagate barracudas in RAID-Z running on Manjaro (arch distro).

I'm also running Zabbix to monitor the health and alert me by email of any drive failures so I can replace them as quickly as possible.

Granted I don't value my data on there as super critical, but regardless this has been running 24/7 for almost 2 years without issue. In fact there was one drive failure that was caught (and within warranty, so I sent it back) and I've also done an in place upgrade from 640GB drives to 2TB. Using StarTech.com 4 Drive 3.5in Trayless Hot Swap SATA Mobile Rack Backplane made both processes pretty painless.

I guess the best part of this setup, is that excluding the drives, it cost abuot £150.

 

[img]

Surprised you feel you need raid 6 for movie rips. Not end of the world if a drives goes. Have you considered flexraid as this is what I have on my array for movies. No worries about corrupt array data. Important documents I backup to second location.

 

[Dieseldes]

You have the perfect requirement for a unraid server. I have the same sorta setup and run unraid on a hp microserver. Google unraid. IT'S a bit like raid 5 but if you loose 2 drives you loose the complete raid 5 array but in unraid you only loose the data on those 2 drives everything else is still there. You can add any size drive at any time. Just your largest drive must be your parity drive. You can upgrade your parity drive though.

seriously Google it and read on the lime technology forum.

 

[FrankJH]

Surprised you feel you need raid 6 for movie rips. Not end of the world if a drives goes. Have you considered flexraid as this is what I have on my array for movies. No worries about corrupt array data. Important documents I backup to second location.

Depends how long it takes to rip a movie......or 40~ per TB when a disk fails

You have the perfect requirement for a unraid server. I have the same sorta setup and run unraid on a hp microserver. Google unraid. IT'S a bit like raid 5 but if you loose 2 drives you loose the complete raid 5 array but in unraid you only loose the data on those 2 drives everything else is still there. You can add any size drive at any time. Just your largest drive must be your parity drive. You can upgrade your parity drive though.

seriously Google it and read on the lime technology forum.

Unraid is fantastic, have been using it for years - but its only great upto a certain point. Im getting upto about 20TB and thinking that having only 1 parity disk (or more than one disk loss) isn't enough protection when I go for more storage.

Of course that's just personal opinion, I know some who have 50TB on Unraid and have for a year plus, but this is too much of a risk for me