10Gbps home network, on the cheap. Sort of.

BeelzebubUK said:
I'm theorising how to make my setup work the way I want it.

I will have a 6 drive hardware RAID10 array which I want all clients in the house to access the same data contained on it.

Main PC connected directly by 10Gbit using Infiniband
Laptop, Media Player, Squeezebox connecting via switched Gbit Ethernet

So the NAS will have an iSCSI target consisting of the entire array space.
The main PC will attach to the target and create an iSCSi drive.
Then the NAS will also attach to itself (via loopback address), create an iSCSI drive and then share the folders out to the rest of the network (SMB or NFS)

Will that work ? No issues with multiple initiators writing to the same file as its just me using any of the clients.
Click to expand...

Multiple initiators (your main PC and your NAS) connected to a single target (your NAS) is going to be a problem (as far as I can tell). However, there's nothing to stop you creating an SMB share on the iSCSI device on your main PC and have your other devices access that rather than the NAS. I appreciate that's probably not ideal though ;)

The other issue is that with the free Windows iSCSI target software that I've seen you would need to store your array data in a single image, if you want to provision all of the space at once. That would make me feel quite nervous, depending on the amount of data involved. I wouldn't trust say, a 4Tb image file, not to go wrong... could be wrong but my gut doesn't like that idea!
 
Last edited:
Can't use the main PC as the point of the NAS was to not have the main PC on 24/7 :)

I'd also not trust to have my data on a single image file either. Although it will be backed up. Need some way to have direct access to the array from my main PC over infiniband but still be able to access the same data via normal file sharing. Looks like more investigation is needed :)
 
Surely, if you want lots of users accessing the same content you should be using a protocol that is suited to this mechanism such as FTP / SMB, you may not get the exact same numbers, but who really cares in the reall world when you'll see 0.00x% network usage...
 
BeelzebubUK said:
Can't use the main PC as the point of the NAS was to not have the main PC on 24/7 :)

I'd also not trust to have my data on a single image file either. Although it will be backed up. Need some way to have direct access to the array from my main PC over infiniband but still be able to access the same data via normal file sharing. Looks like more investigation is needed :)
Click to expand...

For Windows, at the moment, I think SMB shares are your only option. I'm not sure if the Linux SRP target allows multiple initiators to connect to a LUN, or if it will support direct access to a partition allowing for a share to be created at the same time.
 
Well I was pleasantly surprised. Got the cable today and just connected it up, ran subnet manager on the server and assigned the IP's and had connectivity. Then ran a quick couple of tests copying some large files from the NAS to the PC using normal windows SMB and was averaging 250MB/s which would be the theoretical maximum of 2x 1Gb ports :)

Server was running about 85% CPU on both cores (its only a HP N40L microserver) during the transfer and I was just running off the onboard RAID with default settings on the infiniband adapters so might be able to increase the throughput with a beefier CPU and using the HP RAID card I have.

Very happy so far. Would recommend it to anyone whos after extra peer to peer bandwidth.
 
Last edited:
That's some pretty good results there, nice one! I'll have to re-test an SMB share since I'm fairly sure I wasn't getting transfers that high :)

Do you know what your RAID array will max out at?
 
Only tested the array once and was getting about 400MB/s in a Windows stripe (ie RAID0).

When I've got a new case to house the RAID and Infiniband cards i'll be running RAID10 off the raid card so not sure what speeds that will give me.

I'll do some more testing tonight as the 250MB/s was just what windows was showing in the transfer window so it wasn't very scientific. I updated the firmware on the cards to the latest 5.3 from Mellanox which seemed to make the transfer a lot smoother than before. I'll try some FTP's which should show what the max transfer is.
 
RAID10 should get you similar read speeds, but around half the write speed of a 4 disk RAID0.

Are you using the OFED package or the one from Mellanox? Have you tested the latency and bandwidth using the command line tools? Sounds like it's all working spot on to be fair. Good shout on FTP testing, I'll give that a shot too.
 
I'm using the OFED package. I'll put iperf on both the machines tonight to get a proper bandwidth test. Are the command line tools some of the ones in the OFED folder ? Only used the ones for flashing the firmware in the Mellanox directory so far (flint and mst)
 
GodAtum said:
Sorry for being a noob what what would this be useful for? You cant network anything unless you buy a fibre switch?
Click to expand...

Read the original post and the link that Shad included.

My only requirement was more bandwidth between the NAS and my main PC so theres no need for a switch at all. Other connectivity (laptop/media player/internet) is done across the normal Gbit switched interfaces.

You'd need a switch only if you needed to connect multiple devices at 10Gbit
 
Ah OK. Even though this is very cool unfortunately I dont really have a need for it as I need to have fast streams to multiple PCs across the house. I mean I could have it to my PC but that would unfair :p
 
Yeah the switches are still very expensive and then add cabling on top of that its not very cost effective for a home setup. I reckon 10Gbit ethernet or thunderbolt networking will become cheaper fairly soon.
 
Nothing to stop you daisy chaining machines together though, and/or sticking 4+ dual port cards in a single server to create a 'DIY switch' on the cheap.

BeelzebubUK, yes the OFED tools are in the install directory. There's a manual.htm which explains how to use them, good for diagnostics :)

I had dire results from iperf, just a few hundred KB/s. Didn't worry too much as the reality was much better than that, but pot back what yours shows!
 
What performance speed did you get in Mbits/s using Passmark Performace Test?

ATM my external hdd is connected by eSATA to my main PC which then shares it to the network. I could put that into a NAS and fibre the two up but is that faster?

Could the fibre card fit into an HP proliant microserver?
 
Last edited:
I don't use Passmark
eSATA runs max 3Gb/s which is approx. 375MB/s so depends on how you implement infiniband. iSCSI would be faster, SMB probably wouldn't. Have you tested what speed your eSATA runs at max ?

Yes the card fits in a microserver. I have it in mine right now although my card was a full height so I had to remove the backplate from the card. I'm moving it to a different case soon so wasn't bothered. Not sure if they come in half height flavours.
 
You must log in or register to reply here.
Back
Top Bottom