Best OS with ZFS and Infiniband SRP / RDMA support.

RimBlock

RimBlock

RimBlock

Associate
Joined
10 Nov 2004
Posts
2,237
Location
Expat in Singapore
I have a Dell C6100 4 nodes server that I am planning to turn in to a cluster (3x ESXi & 1x Storage server using ZFS).

Storage infrastructure will be DDR (20GB/s) Infiniband via a dedicated DDR Infiniband switch.

The node to be used as a storage server has a CONNEXTX-2 dual port 40Gb/s MCQH29-XDR mez card in it.

I am looking for an OS that supports ZFS, the Connextx-2 card and SRP / RDMA (rather than the slower IPoIB). A pointer to a how-to or config guide would also be most welcome. I also understand there are now Infiniband SRP drivers for ESXi 5.1 now available so will be trying these out.

Any suggestions ?.

Thanks
RB
 
When I was looking at IB I found that support for Ubuntu was quite well documented, although that was with Infinihost adapters. The Mellanox guys on Twitter (@mellanoxtech) are very helpful if you tweet them a question, they can probably recommend a distro at least.

I'm still new to ZFS but that also seems to be supported on Ubuntu. Whether or not something like napp-it would work for SRP targets though I don't know; you might be stuck managing LUNs etc. via cli than a GUI.
 
Yeah I have been given a link for setting up on Debian from a Mellanox guy who offered to help from another forum. I will probably tough base with him again as well.

My understanding is that ZFS is not near the same level as it is on the old Open Solaris sucessors though.

If it was Linux then CentOS would be my distro of choice but would porbably prefer to go with a Solaris based OS / Appliance.

CLI is fine but a decent overview / guide to the tools available would be most helpful (rather than just trying to work it out from the on-line Oracle manuals).

Thanks
RB
 
FreeBSD & ZFS is solid, but I don't know whether it supports infiniband.

Linux support and ZFS has been nearly there for ages, but one wonders whether the standard for "stable" is lower in linux than it is in Solaris or *BSD.
 
Have you looked at Open Indiana / Solaris 11.1 / NexentaStor these should all have native IB support through Comstar, plus native ZFS?

The downside to Solaris thought, is you have no updates unless you have a support contract which isn't cheap.

You can also use IB, I believe by using the SCST stack which can be used on both CentOS and Ubuntu but its not a simple installation as you need to recompile the kernel.

This maybe a helpful link > http://scst.sourceforge.net/targets.html
 
Last edited:
My Infiniband adaptors arrived today so I ended up installing Solaris 11.1.

It detected the card straight away and even confirmed a link though my switch to another machine. The problem is that there appears to be no subnet manager for Solaris from what I have read.

I have also installed the vib on ESXi 5.1 from Mellanox (newly released) which supports SRP but again, no subnet manager.

I am using a switch but have no access to the management interface at this time.

I have also installed a card in my Win Server 2012 DC but that seems to be having issues (Win Explorer crash restarting now and then, Power shell not allowing me to enter any text etc). As the Mellanox tools seem to be all Windows, without Power shell I am sort of stuck. May need to fdo a reinstall of 2012 or go back to SBS 2011 which I much preferred.

I know I can get the OpenSM manager with Linux but the 4 nodes were intended to be 3x ESXi and 1x SAN so I would rather not loose a ESXi node to Linux just to have a SM running.

RB
 
The switch does not have a subnet manager.

Part of the prerequisite for the management tools for the switch is a running subnet manager :).

I believe the Windows 2012 issues I am seeing may be related to a power cut we had a few days ago. The system is just running very strangely so a reinstall seems like a reasonable step. Unfortuately trying Win Server 2012 Essentials resulted in a 'An error occured in the install - click to shutdown' message, Windows 7 does not like the BR110i driver (the machine is a ML110 G7) so I am either back to Server 2012 Standard or SBS 2011, both of which were running fine previously.

RB
 
You must log in or register to reply here.
Back
Top Bottom