Best OS with ZFS and Infiniband SRP / RDMA support.

RimBlock · 20 Mar 2013 at 04:07

I have a Dell C6100 4 nodes server that I am planning to turn in to a cluster (3x ESXi & 1x Storage server using ZFS).

Storage infrastructure will be DDR (20GB/s) Infiniband via a dedicated DDR Infiniband switch.

The node to be used as a storage server has a CONNEXTX-2 dual port 40Gb/s MCQH29-XDR mez card in it.

I am looking for an OS that supports ZFS, the Connextx-2 card and SRP / RDMA (rather than the slower IPoIB). A pointer to a how-to or config guide would also be most welcome. I also understand there are now Infiniband SRP drivers for ESXi 5.1 now available so will be trying these out.

Any suggestions ?.

Thanks
RB

RimBlock · 20 Mar 2013 at 08:31

Yeah I have been given a link for setting up on Debian from a Mellanox guy who offered to help from another forum. I will probably tough base with him again as well.

My understanding is that ZFS is not near the same level as it is on the old Open Solaris sucessors though.

If it was Linux then CentOS would be my distro of choice but would porbably prefer to go with a Solaris based OS / Appliance.

CLI is fine but a decent overview / guide to the tools available would be most helpful (rather than just trying to work it out from the on-line Oracle manuals).

Thanks
RB

RimBlock · 22 Mar 2013 at 16:14

My Infiniband adaptors arrived today so I ended up installing Solaris 11.1.

It detected the card straight away and even confirmed a link though my switch to another machine. The problem is that there appears to be no subnet manager for Solaris from what I have read.

I have also installed the vib on ESXi 5.1 from Mellanox (newly released) which supports SRP but again, no subnet manager.

I am using a switch but have no access to the management interface at this time.

I have also installed a card in my Win Server 2012 DC but that seems to be having issues (Win Explorer crash restarting now and then, Power shell not allowing me to enter any text etc). As the Mellanox tools seem to be all Windows, without Power shell I am sort of stuck. May need to fdo a reinstall of 2012 or go back to SBS 2011 which I much preferred.

I know I can get the OpenSM manager with Linux but the 4 nodes were intended to be 3x ESXi and 1x SAN so I would rather not loose a ESXi node to Linux just to have a SM running.

RB

RimBlock · 25 Mar 2013 at 01:07

The switch does not have a subnet manager.

Part of the prerequisite for the management tools for the switch is a running subnet manager

.

I believe the Windows 2012 issues I am seeing may be related to a power cut we had a few days ago. The system is just running very strangely so a reinstall seems like a reasonable step. Unfortuately trying Win Server 2012 Essentials resulted in a 'An error occured in the install - click to shutdown' message, Windows 7 does not like the BR110i driver (the machine is a ML110 G7) so I am either back to Server 2012 Standard or SBS 2011, both of which were running fine previously.

RB