Need a LOT (5+TB) of storage

DaveF

DaveF

DaveF

Associate
Joined
17 Oct 2005
Posts
311
I've got a huge amount of satellite imagery (6-7 TB) to process; disk IO is looking to be a major bottleneck, so sticking a bunch of 1TB USB drives in the PC isn't really ideal.

I don't know much about what the possibilities are; at the low end, I'm thinking of buying a machine with 8 SATA connectors and sticking 6 or 7 1TB drives in there; looking at external raid drives seems to be 3 or 4 times more expensive than that and I'm not sure I can really justify it.

The source imagery I've got is on USB drives, so whatever I do, I can't read the data faster than that. So although I think it's worth a SATA raid setup, I don't think there's much point going more extreme than that.

Any thoughts/suggestions?
 
usb2 is limited to around 30mb/s transfer

Samsung F1 drives are about 100mb/s, you need need to use sata or esata
 
Well you'd want at least RAID5. This also means you'll only get around 7TB (unformatted) space (or 6TB for RAID6) with an 8-port RAID controller. This probably isn't going to leave you with enough storage space so you got 2 options....

1) Get an 8-port controller and wait for the 1.5TB drives to be released.
2) Get a 12/16-port controller and get 1TB drives.

As for RAID controller cards, the new dual-core Adaptecs are incredibly fast. The high-end Areca's are also good but have driver issues with certain OS's (Vista x64 Ultimate for one) and support is a bit.... shoddy tbh.
 
Last edited:
Or firewire 800.. although your options for eSATA are probably better.

How about a fibre attached storage - such as MSA or EVA ?
 
Oi, you lot! :( It's basically a 2 million x 1 million uncompressed image. Which is only 15m resolution, so you can't even make out individual people, let alone anything else.

I don't know what the budget is, but it's not looking like a lot. So it's looking like I'll have to compress it to JPEG as a first step (I was hoping to avoid that until the final processing stage to avoid repeated uncompress/recompress lossage); it doesn't look like I can justify the expense of enough storage to do everything uncompressed. In which case a couple of 1TB drives will do and it stops becoming anything special really.
 
Do you actually have a budget? you could always get a small NAS box or something...But obviously it depends on your budget regarding what you can get.
 
Delvis said:
Do you actually have a budget? you could always get a small NAS box or something...But obviously it depends on your budget regarding what you can get.
Click to expand...
It doesn't look to me like that's cost effective; biggest I see is 4TB for £900 or so; that's not really enough for me to work uncompressed and I could get a cheap PC with more storage for less.

(I thought I'd have more budget when I made the first post, but apparently not, hence the need to rethink and the fact this thread is probably pointless now).
 
lopkinfop said:
each 1Tb drive is about £90 so for 8Tb it would be over £700, how is it stored now?
Click to expand...
I've got 5 1TB external USB drives with 1 or 2 more on the way.

Problem is, I need to put it into a more readily accessible format (and mip-map it(*)), so I'm that position of "The original data fits on the drives I've got, and the final data will fit on the drives I've got, but there's a point in the processing where I really need both sets of data". (For that matter, even after processing, removing the original data isn't a step I'd take lightly; no going back after you do that!).

(*) The mip-mapping is a pain because it's essentially a "global" operation; the top mip-map relies on all 5TB on satellite imagery. There are ways around it, but any "working out a new algorithm" on this is tricky, because the data is so large that you can spend a week doing the wrong thing before you find out.
 
lopkinfop said:
could it be done bit by bit, how big is each image
Click to expand...
There are lots of things I could do if I have to; none of them are really ideal. Pretty much any "bit by bit" solution suffers from being a "1-way" operation; get to the end and find you should have done somethinig else and you're a bit stuffed, because you've lost the original data. I am basically really really uncomfortable with any solution that involves getting rid of the original data until I know I've got a completely valid processed version of it. You don't want to get to the end and find there's a problem at the +/-180 degree lat/long line and you need the original data to fix it (yes, this has happened in the past with smaller data sets).

So I think I'm going to end up compressing to JPEG as an early step - I can easily get enough extra space to hold JPEG versions of all the data. But it means I will need to work out how to layer the processing so that I do as little uncompressing and recompressing as possible. (Whereas we have an existing and working mechanism for processing uncompressed data).

I've also said as much (probably more) about the data as I probably should (although it's all commercially available satellite imagery).
 
You must log in or register to reply here.
Back
Top Bottom