How many Hdd's do i need to saturate 10gbit?

jonneymendoza

jonneymendoza

jonneymendoza

Caporegime
Joined
7 Apr 2008
Posts
25,858
Location
Lorville - Hurston
Hello all.

I'm looking for some advice on setting up a NAS with a RAID 5 configuration using my old PC. Here are the specs of the system:

  • AMD Ryzen CPU
  • 64GB RAM
  • 10GbE NIC
  • Iron Wolf 8TB drives (I'm particularly interested in these)
My use case involves storing and accessing RAW images, video footages, sensitive documents, and entertainment media. I also plan on running Ubuntu Server with Samba file sharing and multiple Docker containers such as Jellyfish and game servers.

Here are some questions I have:

  • How many HDDs would I need in RAID 5 to sustain near 10GbE read and write speeds? If my calculator is correct, I think I'd need 10 drives.
  • Would using a separate PCI SATA adaptor with additional ports work for this setup?
  • Are there any other considerations or best practices that I should keep in mind when setting up this NAS?

Just to clarify, my use case is primarily focused on storing large amounts of RAW images and video footages, as well as sensitive documents and entertainment media. I will also be using Samba file sharing and running Ubuntu Server with multiple Docker containers such as Jellyfish and game servers.

I'm curious about the following:

  • How many HDDs would I need in RAID 5 to sustain near 10GbE read and write speeds? If my calculator is correct, I think I'd need 10 drives.
  • Would using a separate PCI SATA adaptor with additional ports work for this setup?
  • Are there any other considerations or best practices that I should keep in mind when setting up this NAS?
Thanks again for your time and expertise!
 
Will you be using a cache drive? And I suggest RAID 6 or RAID Z2 rather than RAID 5 to mitigate against either the failure of two drivers or the failure of a second drive while the array is rebuilding.

FWIW I have 5x 20 TB HDDs in RAID Z2 for 60 TB total storage with a NVME cache drive.
 
Quartz said:
Will you be using a cache drive? And I suggest RAID 6 or RAID Z2 rather than RAID 5 to mitigate against either the failure of two drivers or the failure of a second drive while the array is rebuilding.

FWIW I have 5x 20 TB HDDs in RAID Z2 for 60 TB total storage with a NVME cache drive.
Click to expand...
I am open to using a cached drive.

The ubuntu server OS and the docker containers will be running in a nvme and ssd drives anyway
 
Given most Spindle drives cap out around 150MBs for average speeds on the higher end models, you're probably looking at close to 7 to 8 drives to consistently cap a 10g network connection for Reads in RAID5, with a few more for Writes for the "less than optimum" times when you still want to saturate the connection.

Using a separate SATA card so long as it is supported and shouldn't be an issue so long as the lanes are all not competing with something else and end up cutting your bandwidth down from the expected amount to something lower. I ran into this situation a while back when I stuck my 10g card into a PCIe3 x 4 slot (which was plenty in terms of bandwidth), but because the storage connections in that machine were on the same bus region (chipset; DMI2.0), it meant it capped out at around 6-7gbps instead of 10gbps (due to everything there sharing that 2GBs bandwidth from the chipset). So moving the card into the top PCIe slot (which is connected to the CPU) permits the system to pull data from the SATA drives from the chipset (which using DMI2.0 only has 2GB bandwidth for everything) and then chuck the data back out via the CPU path to the top PCIe slot, so it can go at full 10gbps speeds.

Personally, I've been migrating my systems over to SSDs rather than stay on Spindle drives (except for long term archive and external access to family outside the household to pick up and drop files that need correcting/editing). It's nowhere near as cost efficienct per GB storage, but it's certainly easier to saturate a 10g connection with them vs Spindle drives.
 
For Read speeds - two SSDs in RAID0 (and thus 3 in RAID5 roughly with parity drive) will definitely saturate a 10g link (I've done this previously with the RAID0)

Write speeds depends on the RAID controller and the SSD themselves; DRAM cache determines how long it can write for before stalling to slower speeds, the MX500 for example can sustain 550MB writes until the cache runs out which is usually good for around 50GB of writes per drive in one go on the 2TB model; roughly 1GB DRAM cache nets you around 25GB write transfers at full speed, 2GB for 50GB and 4GB for near 100GB - so if you remain within that limit for each drive per transfer, you'll be golden for speeds throughout.

So if we say you had similar drives you're looking at 6 to 7 SSDs in RAID5 to saturate the 10g link for writes (and only writes, not a mixture), but this is guaranteed at all times of the day vs the spindle drive which would need to remain at optimum (test conditions) at all times with the same number of drives (so my previous estimate would really need to go up for the spindle drives which assumes they can sustain a more realistic 100MBs writes for all file types and loads vs the 170MB burst I put forth previously, which sets it nearer 10 at a minimum in theory, but realistically more than 20+ drives for RAID5).
 
Might not be practical for all uses especially those with more intensive usage but I changed my NAS setup away from a bunch of HDDs and now use an NVME frontend doing real-time replication to HDD - the difference working with media files where you are doing a lot of seeking, etc. and opening folders with lots of image and video thumbnails and so on is dramatically different.

As Meddling-Monk has said actual throughput and efficiency can depend a lot on shared buses and actual IO performance of the connected devices and even quality of the cabling/environment EMI conditions, etc. when talking 10Gbe.
 
Last edited:
If you want the max speed possible then look at getting a used enterprise hardware raid controller with SSD caching and battery backed RAM cache. Usually has SAS12 port, usually 2 or 4 port. Then use a SAS to SATA fan out cable and needS 8x PCI slot.
 
Rroff said:
Might not be practical for all uses especially those with more intensive usage but I changed my NAS setup away from a bunch of HDDs and now use an NVME frontend doing real-time replication to HDD - the difference working with media files where you are doing a lot of seeking, etc. and opening folders with lots of image and video thumbnails and so on is dramatically different.

As Meddling-Monk has said actual throughput and efficiency can depend a lot on shared buses and actual IO performance of the connected devices and even quality of the cabling/environment EMI conditions, etc. when talking 10Gbe.
Click to expand...
So using a nas that uses nvme is as if the nvme drive itself is connected to your computer itself!

That's is what I'm trying to achieve. A nas that is fast enough where I can't tell that I'm working on a storage device that's over 10ft away from my computer
 
Snapshot said:
Like @Rroff, I use an NVMe frontend but only replicating to HDD overnight. Two NVMe PCIe3x1 SSDs in RAID1 will saturate 10Gbps when copying to/from a PCIe4 SSD in a PC.
Click to expand...
I've actually wrote a python script that copies files from one fast drive to a slower one on whatever schedule I like and it runs on docker
 
Snapshot said:
Like @Rroff, I use an NVMe frontend but only replicating to HDD overnight. Two NVMe PCIe3x1 SSDs in RAID1 will saturate 10Gbps when copying to/from a PCIe4 SSD in a PC.
Click to expand...

Probably the best option if doing more intensive write stuff - my setup works pretty well for reads, but under intensive write scenarios you can get some slowdown at times due to the secondary writing to slower HDDs - albeit it is still fast slowdown there being relative.
 
Last edited:
Rroff said:
Probably the best option if doing more intensive write stuff - my setup works pretty well for reads, but under intensive write scenarios you can get some slowdown at times due to the secondary writing to slower HDDs - albeit it is still fast slowdown there being relative.
Click to expand...
Yes I'm thinking of a nice hefy nvme like 2tb or more and once every 6 months, dump the oldest files into the slow mechanical hdd
 
jonneymendoza said:
So using a nas that uses nvme is as if the nvme drive itself is connected to your computer itself!

That's is what I'm trying to achieve. A nas that is fast enough where I can't tell that I'm working on a storage device that's over 10ft away from my computer
Click to expand...
NAS won't be as quick as NVMe which runs at 7GB per second, your 10Gb Ethernet is under 1GB per second once you take Ethernet overhead into consideration.

Anyway, its should be quick enough for your needs.

Other thing to remember is not just transfer rate but iOPS and latency would.be significantly quicker with SSD or NVMe.

Our designers at work cache their working copy on local disk and then replicate to NAS overnight, they work fine with data that is 100's GB in size.
 
Oooo didn't realise you could do this. So if you have 2 x NVMe slots in a NAS, one could be a cache drive and one storage front end which then could copy over to the HDD (say setup in RAID5 if there were 4) at night? Or would you not need a cache drive if you already have a fast front end storage with the other NVMe?
 
doodah said:
Oooo didn't realise you could do this. So if you have 2 x NVMe slots in a NAS, one could be a cache drive and one storage front end which then could copy over to the HDD (say setup in RAID5 if there were 4) at night? Or would you not need a cache drive if you already have a fast front end storage with the other NVMe?
Click to expand...

Yeah you can do that, but unless you’re dealing with huge data transfers it’s pretty pointless.

I’ve not had experience with those iron wolf drives, but a pair of dual LUN segate HDDs or 4 single LUN will just about tap out a 10 gigabit connection to a single client +/- 5-10%

A 12 gigabit HBA card in IT mode will simply present all drives connected to operating system. From that point the software can do what’s required. The only other consideration I would have would be if to use ECC or not.
 
You must log in or register to reply here.
Back
Top Bottom