Infiniband setup issues

GodAtum · 23 Nov 2012 at 16:48

Hi all,

I have fitted my Mellanox card into the PCIe 8x slot and installed Win 7 but it does not seem to detect it even after installing the OFED drivers. When i run ibstat it says "winmad.dll cannot be found".

However on my other Win 7 PC it works fine (this has an Asus P6T mb). On that PC the card is listed in Device manager under Infiniband Channel Adpaters however that group does not even appear on my server.

Is this an issue with my GB g3 Sniper not supporting 8 bus cards?

GodAtum · 26 Nov 2012 at 12:46

I swapped the cards from my PC and it seems to detect it now, very odd!

GodAtum · 30 Nov 2012 at 23:11

Hi all,

Ive been following David Hunt's guide but seem to be stuck at the first hurdle. Apparently I should have an ib0 interface. But when ifconfig I only have eth0 and lo.

This is what Ive done so far;

Edit /etc/modules and add the following modules:

ib_sa
ib_cm
ib_umad
ib_addr
ib_uverbs
ib_ipoib
ib_ipath
ib_qib

Next,

apt-get install opensm

I have passed through the Mellanox card in ESXi to the VM and it detects it in lspci. I am using Ubuntu Server 12.04.

GodAtum · 1 Dec 2012 at 17:30

Here are some results of the commands people have advised me to run:

"more /proc/net/dev" lists lo and eth0.

"ibstat" lists the 2 ports, with 1 as down and 2 as initializing.

"ibstat -l" lists mthca0

"ibverbs-utils" shows No IB devices found.

GodAtum · 3 Dec 2012 at 22:06

When i run sudo opensm it says:

SUBNET UP

The result of cat /sys/class/infiniband/mthca0/ports/2/state
is ACTIVE

GodAtum · 4 Dec 2012 at 19:37

I have run through the diags, here are my results:

Code:

shell > lsmod | grep ib_
ib_mthca
ib_qid
ib_ipath
ib_uverbs
ib_addr
ib_umad
ib_cm
ib_sa
ib_mad
ib_core

Code:

shell > ibstat
Port 1: Down
Port 2: Active

This is what i expected to see as only one of the ports is plugged in.

However when I do:

Code:

shell > ifconfig -a

My infiniband interface is not showing.

When i do:

Code:

shell > ibchecknet

There is an error on port 2:

Code:

/usr/sbin/ibcheckerrs:222:exit:illegal number -1

GodAtum · 5 Dec 2012 at 06:29

RimBlock said:
Still awaiting a reponse on whether the other IB card is working correctly in the other host and if so how have you verified ?.
RB

The firmware on my Windows client is OFED 3.1 win7 x64. On my Ubuntu server it is OFED 1.5.4.1

GodAtum · 5 Dec 2012 at 12:54

I've just seen a page for ESXI drivers (http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=36&menu_section=34#tab-two). I might give those a try if they are the right ones for my card 9 a Mellanox MHEA28-XT)?

In the install guide it says:

Code:

Log into the ESXi5.0 server with root permissions.
esxcli software vib install -d <bundle_file>

But how do i get the bundle file onto ESXI?

GodAtum · 5 Dec 2012 at 13:54

Shad said:
Once you've enabled the SSH service in ESX you can use WinSCP (or anything else similar) to transfer the .vib onto the host.

I'm not sure if installing it will achieve what you need though. I'm guessing it will either allow ESX to use the adapter for iSCSI LUNs or it will expose it as a networking device that you can use on a vswitch, but I'd have thought that is likely to put you back into IPoIB hell.

Ah Ok. i was hoping using the adapter as a networking device and attach it to a VM rather then a passthrough PCI device might solve my issues!

GodAtum · 5 Dec 2012 at 15:19

Mmm

I'll try updating the firmware if needed and report back.

GodAtum · 6 Dec 2012 at 10:02

I've tried to install the MFT tools onto my Ubuntu VM. Howver when I run

shell > sudo ./install.sh there is an error

./install.sh: 32 ./install.sh: Syntax error: "(" unexpected