Massive storage options for an OCD File hoarder

[FatRakoon]

Ok, I have bee na right weirdo in the last few years, and I am a huge file hoarder.
Everything that I do on my PC, I make backups... This should I think be a fairly normal thing, and most will agree.
The thing is, that I have a family of 5 ( me, the boss, and 3 kids ) and I also have backups for all of their PCs and theirs, and my laptops too!

I also have a nice collection of ISOS of many many many variations of Linux ( To be honest I dont know why when it only takes 5 minutes these days to re-download them, but hey ho ) plus I have a great collection of Music and TV and Movies, yes, all bought and paid for, and fully legit, plus I also have ISOs of my games, and anyway, most of my games these days are downloads anyway, much are STEAM Based, but again, I have these all backed up too, add to that, my kids have their games and media ,and I have all of them backed up also... It all makes for a fairly large collection of files.

I have 4 NAS Boxes, 2 DLINK and 2 NETGEAR the DLINK have 2 Drives with 2x2TB each and the NETGEARS are 4x3TB and 4x4TB and I have recently rebuilt a new server, that has 4x8TB and I was using Windows but I recently went SuSE and Im using LVM as I have learned that is a very useful way to do things, plus, the NAS Drives use ext?fs and so with me setting them up JBOD Im able to simply take teh disk out to put into my Server to copy files over quickly, rather than over the LAN.

I also found that ( and this is a win10 issue I think ) but one of the NETGEARS ,while fully visible, and accessibler, is I enter the net name in the location bar, it wont let me mount as a network drive, nor can I open it with explorer normally... The DLINKS have both doen the same.

It did this on the other, now fully working NAS too, but I copied the fiels over ( by physically adding the disk as I said earlier ) and when I re-formatted the thing as blank etc, I was abale to access it again... I can probably do that to the other now non working NAS but thats got 4 fairly full 4TB drives in. Lots of data if it does not work.

So, anyway, I have a 1200w PSU in the server that is fairly much pushed to its limit. I have backed off and its running fine now, but adding more drives is not really an option at this time.

I was instead, considering buying a massive NAS System.

I find that NAS boxes always seem to have some limit or other.

The DLinks have a 2TB Limit on the drives,m the NETGEAR has 4TB, the Server is weird in that in IDE MODE, I can access large 8TB Drives, but in AHCI Mode, it wont, I have a few SATA Cards and some access 8TB just fine, some dont, and irritatingly, when I went from my ASUS 2 port to my new 5 port with multiplier, it would not access the data from the disks off the ASUS? - I had to buy new disks and copy before I removed the ASUS card?

Anyway, I am thinking then, of a large NAS Box, that takes 10 to 16 Drives and these drives can be 8TB or even bigger.

my curent setup contains

4x2TB + 4x3TB + 4x4TB + 4x8TB on and also 1x4TB but thats a simple file for file copy of my master PC

the 4x8TB are almost brand new, only been used a couple of months but the 4x2TB in the DLinks are fairly old now and will be given to my mate but the 4x4TB and the 4x8TB are for the bigger NAS and the 4x3TB if they can go in.

The idea is to setup a proper MIRRORED array with hotswap that when a drive does go iffy, its a simple matter of pulling the duff disk and ploopping in a new one... If possible with full automatic rebuilding, and even if I swap its mirrored drive, and if Ihave upgraded to a biger drive, the extra room will also be accounted for an the extra space will be used too ( When the second disk is added of course ).

So, money is no object ( Within reason, Im happy to pay but Im not an idiot )

Anyone have any suggestions?

God why do I waffle so much? Thats the OCD in me.

 

[Danny091]

If you haven't already have a look at the datahoarder subreddit.

You need to decide what kind of system your going to run. Linux, Windows, FreeNAS, Unraid. (I'm Currently running unraid as like being able to add individual disks rather than whole arrays, it also supports VM's and Docker which is nice.)

When you say Mirrored do you actually mean mirrored? 1 Data Disk and 1 Backup disk kept in sync essentially halving your storage capacity. Or do you mean some sort of RAID where you set 1 or 2 disks as parity disks and have the storage area of all your disks - 2. (Raid 6)

Looking at your assorted disks honestly id stay away from FreeNAS or ZFS it likes disks all the same size and type. Unraid would work quite well, Each disk can be different sizes, types ext. The only requirement is the parity disks are larger or the same size as the largest data disk. You can run Unraid in raid 5 or 6. Raid 5 gives 1 disk headroom so if one drive fails you can swap in a replacement and your data will be rebuilt. Raid 6 is the same but you can survive 2 disks dying. Raid 5 requires 1 Parity disk (which you lose the space of in the array) and Raid 6 requires 2 Parity disks (which you lose the space of). With your amount of drives stay away from RAID 5. Its likely that another disk could die during a rebuild which would make the data unrecoverable, raid 6 gives you that extra headroom.

A plus about unraid is as the whole file is stored on one disk if you did lose more than 2 disks at one you only lost the data on them disks, unlike traditional raid where bits of the file are spread over all disks so once you go over the threshold of failure you lose everything.

The downside about unraid is it needs empty disks, if you went this route id take your biggest drives which are the 8TB's from what i can see. move the data off them to the other drives if possible so you have atleast 3 empty. Make 2 parity and 1 data after the array is built, move data from 2 of your 4 TB to the array then wipe the 4TB and add them to the array then just keep repeating till your done.

For the actual system i cant see you have any hardware except for PSU, your 1200W psu should support that many drives fine i dont understand how its pushed to the limit unless your running dual CPU's and graphics cards?

My current unraid system is an I5 3570K and 32GB ram which i find works great for some VM's and dockers too. If your not interested in dockers or VM's lower CPU's and less ram would be needed. Something like maybe an I3 and 8GB??

You need someway to power and data connect that many drives, you can look for cases that support 16 disks maybe but you will struggle. You most likely be best looking at storage server cases something like Logic Case SC-4324

For data connections you can likely use some SATA connectors on your motherboard if your motherboard has 6 connecters then you need 10 more. Id look at the LSI 9211-8i which can be gotten on the bay and you want to flash them to IT mode. Each one supports 8 drives so you may need 2.

Hopefully this has given you things to think about, if you want any more help let me know.

 

[bledd]

4x NAS?

Just get one huge one, then maybe a second for backups of vital files

 

[Minstadave]

Storinator:

https://www.45drives.com/products/storage/

 

[ANDARIAL]

Same here as op
TOO many drives and systems,must to over 100TB
I am of the mind that more systems + hdds the better though.If one pc gets fubar you won`t lose all your ****
Obviously not all systems are on at once and finding that one file/folder can be a pita
How about Drivepool?
Got this on 2 of my pcs,you have the option of however many copies of files or folders you want

 

[Rannoch]

Have to admit I tend to hoard files. I have two NAS drives and several 8TB desktop external units, and some 4 & 5TB portable drives.

I use Macrium Reflect and schedule backups to the NAS drives. I also have file syncing software. If I didnt automate the backups I think I would go mad lol

I also use Drivepool

 

[233]

just a thought OP would you not be better looking at cloud storage solutions? rather than arranging offsite backups etc. I imagine your data must be valuable to you if your going to these lengths to keep it

 

[Avalon]

£6.60/m for unlimited storage from google, they are quite happy to allow a few hundred TB of Linux ISO’s without issue as long as you don’t do anything stupid, like publically share them. As someone who ran large disk arrays via Unraid, this is easier and a LOT cheaper.

 

[andy_mk3]

Storinator:

https://www.45drives.com/products/storage/

This for sure if you want to keep it all local.

 

[chroniclard]

£6.60/m for unlimited storage from google, they are quite happy to allow a few hundred TB of Linux ISO’s without issue as long as you don’t do anything stupid, like publically share them. As someone who ran large disk arrays via Unraid, this is easier and a LOT cheaper.

Which product is this as drive is way more expensive?

 

[Avalon]

Which product is this as drive is way more expensive?

Google drive for business, you’ll then read the T&C’s and tell me it has a minimum five user requirement, to save you the trouble, it’s never been enforced, if it ever is, then several other ways round it exist, just like the 750GB upload limit per day

 

[chroniclard]

Seems like I need to get a domain to use it.

 

[Avalon]

Seems like I need to get a domain to use it.

Considering how cheap a domain is, the matter of minutes required to register one and the number of providers who will happily provide said service including google themselves, I really don’t see that as being a problem

 

[Stoner81]

Google drive for business, you’ll then read the T&C’s and tell me it has a minimum five user requirement, to save you the trouble, it’s never been enforced, if it ever is, then several other ways round it exist, just like the 750GB upload limit per day

"accounts with fewer than 5 users get 1 TB per user"

Pulled directly from their website. Is it still not being enforced?

Stoner81.

 

[Avalon]

"accounts with fewer than 5 users get 1 TB per user"

Pulled directly from their website. Is it still not being enforced?

Stoner81.

Google drive for business, you’ll then read the T&C’s and tell me it has a minimum five user requirement, to save you the trouble, it’s never been enforced, if it ever is, then several other ways round it exist, just like the 750GB upload limit per day

 

[Deleted member 77746]

Why don't you take a few hours out of your day to sort through everything. You don't need to keep data that's old. There is no need to save hundreds of TB's what so ever.

 

[blueacid]

First of all, yes, reddit.com/r/datahoarder is a good place to begin!

Second of all, backups. Really, seriously, backups. The 3-2-1 rule definitely applies here; that means you need Three copies of your data, Two which are local and (ideally) on different formats, and One which is remote. For my data, the most important of it is stored both here, on a NAS at a friend's house and also in Online's C14 cold storage; it's a bit more of a faff than AWS Glacier, but the cheap price of storage is what lured me in.

Of course, your backup strategy might therefore involve you relocating some of your NAS devices to other locations; so once you have figured that out, time to start working out what you need.
Do you need rapid access to any of the files? Or do you need them to be available if you connect the right enclosure? Do you need high speed access, or does just 'access' do the trick?

A lot of these questions will get you into the right frame of mind - and will help you flesh out the ultimate solution to your file storage needs.

 

[Sp00n]

Hoarding data and waffling have nothing to do with OCD.

 

[FatRakoon]

Hoarding data and waffling have nothing to do with OCD.

I suppose not, but I do have a few rather irritating issues that do show my mind is simply NOT what it used to be.

I now have Dysexecutive Syndrome and one of the side effects, is that I am almost incapable of lying... Not entirely true, as thats a lie in itself, but if I get asked a question out of the blue, I answer it completely honestly, and there are times when thats not what you want!

I also hoard and thats getting way out of hand. For example, if anyone has a better phone trhan me, I buy the same phone or better, and just have them and never use them, same with PCs, I have extended my old house 3 times just for all the PCs and now I have moved to a bigger house purely because I have so many PCs... They have been dropped to a manageable number now, but I still have too many.

And waffling I do really because I type as I am thinking and if I stop, I forget what I am doing and then I either post half a post or a long winded one full of gibberish.

 

[bledd]

Get one big NAS, simple network and one good pc. Sell the rest, fresh start