10Gbps home network, on the cheap. Sort of.

[Shad]

Inspired by the series of articles by David Hunt I too decided that I fancied having a crack at a super fast home network without spending a fortune. 10G Ethernet cards are still £200-300 a pop on eBay which makes even a simple PC-to-PC connection quite expensive. On the other hand, older generation SDR and DDR (10Gbps and 20Gbps respectively) InfiniBand adapters are available for a fraction of the cost. The downside is the extra hassle to set it up and the restrictions with cabling (very short and a bit pricey or useful length and very pricey).

So to whet your appetite here's a benchmark I just took:

To achieve this I've used:

• 2x Mellanox InfiniHost III MHEA28-XT dual port 10Gbps (£45 each)
• 1x Zarlink ZLynx 30m CX4-CX4 20Gbps fibre optic cable (£170 )
• 1x Dell PE2950 (2x E5160, 12GB) running Windows Server 2008 R2 and Microsoft iSCSI Target, and a 3.5GB RAM disk
• 1x Phenom II 965 workstation running Windows 7 x64 and Microsoft iSCSI Initiator

The cost of the cable I had to buy was a killer, but cheaper cables are available! The trouble I have is the distance between the server and the workstation. Although it's only about 13 metres and some 15m cables are available, the cost difference between 20m and 30m is so little it makes more sense to get the longer one and have some options down the road. There is effectively zero signal degradation over fibre (so the extra length doesn't matter) whereas the short copper cables suffer quite quickly and it's rare to find them even as 'long' as 10m.

As David explains in his articles, InfiniBand works using RDMA which basically means the adapters read/write directly to the system memory in each connected machine, completely bypassing any protocols that slow things down. However, there is currently no way to achieve this in a Windows only environment (such as mine) and you need a Linux system to create the SRP target to enable the true high speed RDMA connectivity. Us Windows folk are stuck with IP over InfiniBand (IPoIB) which emulates TCP/IP and allows all the normal networking features. The downside is that it's all handled by the host CPU so a lot of the benefit of the 10Gb adapters is lost - for now.

So at the moment I'm quite happy to have 'only' 400MB/s sequential read as well as some other pretty good numbers. In the future I will probably create a Linux system, but at least there is headroom there if I find that the current speed isn't sufficient. My next step is to buy some disks for a RAID array on the server as the RAM disk is only for testing and proof of concept.

Also worth noting, Windows Server 2012 and SMB3 include support for RDMA on Mellanox's latest HCAs, but although these run at 40Gbps+ they are the same cost if not more than 10G Ethernet adapters. Microsoft and Mellanox look unlikely to add support for older (cheaper) IB adapters unfortunately, so Linux is the way to go to get the best speed possible at the moment.

Hope that's interesting and/or useful

 

[Shad]

You actually only need the run Linux at the target end, i.e. the end with the storage. Unfortunately my PE2950 is a rev. 2 which doesn't support Directed I/O which means I can't pass through the InfiniBand adapter to a VM (which rules out running ESXi). Since I need to use the system as a Windows primary domain control and a few other things, I can't install Linux onto the base metal either.

What I am considering though is recycling my workstation kit which is AMD but does support pass through (using IOMMU), but I need to do a bit more testing with ESXi 5.1 to see how the onboard RAID is presented and whether or not it can be passed through or if individual disks can be passed through and RAID'ed in software with Linux. It would be a nice excuse to upgrade this system to an i7...

What I have to keep reminding myself at the moment is that what I have now is well in excess of 4x faster than 1Gbe speeds that I was getting before

 

[Shad]

Some new numbers using StarWind iSCSI target:

Quite a bit faster than Microsoft iSCSI target, this is probably as fast as I'll be able to go with IPoIB. LAN Speed Test reported 850MB/s write and 930MB/s read but real world file transfers over SMB capped out at around 130-140MB/s. Anyone know what LAN Speed Test is really measuring?

 

[Shad]

I've got 1TB of SSDs arriving tomorrow so we'll see how they get on in RAID0 - should be enough to saturate a 10Gb line

By the way, when I switched to the StarWind initiator as well as target, the numbers... well they increased a bit

 

[Shad]

Good man

Well after my findings I'd say IPoIB is not anywhere as limiting as I thought it was going to be, and clearly it makes a big difference what software is running on top of it. That said, I'm not really clued up on how NetDirect and Winsock Direct come into play because if SMB shares don't take advantage of those layers and StarWind does, that would go a long way to explaining the performance even with IPoIB.

Talking of such layers, make sure you follow the instructions in the OFED driver manual to get them installed.

I'm using the free StarWind target and initiator, just head over to here and here respectively. You'll need to register for a free licence for the target. They have an active forum so it's worth posting there if something doesn't work. My workstation is Win 7 x64 so I can confirm that end works, but I believe you should be ok with Home Server as well - they seem to pride themselves on the variety of systems you can run it on, and I'm 99.9% sure Server 2003 would be fine.

Let us know how you get on

 

[Shad]

Thought I'd try Starwind across my normal 1Gb link just to see how it works. Might be my lack of experience but I can't see how to setup a disk or folder as a target. RAW disks aren't supported in the free version. Could only get a RAMDisk working.

Is there a way to do this with the free version ?

With the free licence you are only able to create devices based on image files (.img), images that support snapshots and CDP, and RAM disks. I don't know if there is a performance penalty for using images verses raw disk/partition mapping.

LACP using 2 ports and 20GBps? [I am of course, totally kidding].

It's not out of the question

 

[Shad]

I'm theorising how to make my setup work the way I want it.

I will have a 6 drive hardware RAID10 array which I want all clients in the house to access the same data contained on it.

Main PC connected directly by 10Gbit using Infiniband
Laptop, Media Player, Squeezebox connecting via switched Gbit Ethernet

So the NAS will have an iSCSI target consisting of the entire array space.
The main PC will attach to the target and create an iSCSi drive.
Then the NAS will also attach to itself (via loopback address), create an iSCSI drive and then share the folders out to the rest of the network (SMB or NFS)

Will that work ? No issues with multiple initiators writing to the same file as its just me using any of the clients.

Multiple initiators (your main PC and your NAS) connected to a single target (your NAS) is going to be a problem (as far as I can tell). However, there's nothing to stop you creating an SMB share on the iSCSI device on your main PC and have your other devices access that rather than the NAS. I appreciate that's probably not ideal though

The other issue is that with the free Windows iSCSI target software that I've seen you would need to store your array data in a single image, if you want to provision all of the space at once. That would make me feel quite nervous, depending on the amount of data involved. I wouldn't trust say, a 4Tb image file, not to go wrong... could be wrong but my gut doesn't like that idea!

 

[Shad]

Can't use the main PC as the point of the NAS was to not have the main PC on 24/7

I'd also not trust to have my data on a single image file either. Although it will be backed up. Need some way to have direct access to the array from my main PC over infiniband but still be able to access the same data via normal file sharing. Looks like more investigation is needed

For Windows, at the moment, I think SMB shares are your only option. I'm not sure if the Linux SRP target allows multiple initiators to connect to a LUN, or if it will support direct access to a partition allowing for a share to be created at the same time.

 

[Shad]

That's some pretty good results there, nice one! I'll have to re-test an SMB share since I'm fairly sure I wasn't getting transfers that high

Do you know what your RAID array will max out at?

 

[Shad]

RAID10 should get you similar read speeds, but around half the write speed of a 4 disk RAID0.

Are you using the OFED package or the one from Mellanox? Have you tested the latency and bandwidth using the command line tools? Sounds like it's all working spot on to be fair. Good shout on FTP testing, I'll give that a shot too.

 

[Shad]

Nothing to stop you daisy chaining machines together though, and/or sticking 4+ dual port cards in a single server to create a 'DIY switch' on the cheap.

BeelzebubUK, yes the OFED tools are in the install directory. There's a manual.htm which explains how to use them, good for diagnostics

I had dire results from iperf, just a few hundred KB/s. Didn't worry too much as the reality was much better than that, but pot back what yours shows!

 

[Shad]

The XT is a mem-free card so uses system memory. The XTC cards have onboard memory but they are generally more expensive unless you get lucky. No idea which would perform better.

Nope. The C denotes that the card is RoHS-R5 compliant. There is no other difference between MHEA28-XT and MHEA28-XTC. The EA28-XT cards run at SDR (10Gbps) and the GA28-XT cards run at DDR (20Gbps).

This list has some information on the different HCAs Mellanox have produced.

With regards to the VM question, your best bet is to use ESXi on a platform that supports Directed I/O (Intel) or IOMMU (AMD) and use pass through to give the VM direct access to the adapter. Although most modern CPUs support this there are unfortunately quite a limited number of consumer motherboards that have the feature enabled. I've tested it on my ASUS Crosshair IV Formula and it works ok.

sunama - yes you'll need an InfiniBand adapter in each client, such as the MHEA28-XT.

 

[Shad]

Not for the cards I've used - you need CX4 connectors, passive for copper and active for fibre. Those StarTech cables are for SFP/QSFP modules used on the newer IB cards and 10G networking gear, AFAIK.

 

[Shad]

That's the right type of cable - but about twice as long as I'd consider with copper. Even 15m is a push from what I've heard. This is quite an interesting read.

 

[Shad]

No mine's fibre with active connectors - http://www.zarlink.com/zarlink/hs/82_ZL60615.htm

 

[Shad]

Yup that's the one I bought

 

[Shad]

Is this just for the fun of it? If so cool.

For me? Yes, largely. But I also want some storage I can put video material on and use as scratch during editing. I could use a couple of SSDs in my workstation but... this approach appeals to my inner geek

I've finished rebuilding one of my servers into an ESXi host so I can run an Ubuntu VM with the Mellanox adapter passed through, with the aim of setting up an SRP target. Just need some storage behind it and I'll be away, but should have another ram disk benchy up soon...

 

[Shad]

Interesting find!

I'm quite impressed you've gotten even as high as 250MB/s with SMB and IPoIB. The best I can manage in an all-Windows setup with SMB shares is around 140MB/s.

Truth be told... I've got some proper 10GbE adapters, transceiver modules and fibre on the way. Anyone want to buy my Infiniband kit?