ESXI and Infiniband issues

GodAtum · 17 Nov 2012 at 12:38

I have a Mellanox MHEA28-XT. I have enabled passthrough on my mbb. I have enabled it in ESXI passthrough and rebooted the server. I have added it to the VM devices list. However when I try to start the VM my EXRI looses connection. On the server itself there is a purple error screen. Anyone else got this to work?

Shad · 17 Nov 2012 at 13:45

During testing I saw that error screen when I tried to pass through the onboard AMD RAID controller. It might me a compatibility issue between the board, device and pass through.

GodAtum · 17 Nov 2012 at 14:02

Did you find a way to get the RAID controller to work?

Shad · 17 Nov 2012 at 17:08

Nope. Didn't need it, was just experimenting. It's usually easier not to fight these things

GodAtum · 17 Nov 2012 at 18:38

Darn! I think I have to get rid of ESXI as I need to use the infiniband card

Not sure what to do! Install Ubuntu as the host with Vmware Workstation ontop or Win 7 as host.

Shad · 17 Nov 2012 at 22:41

Go Ubuntu, the IB card will be next to useless in Windows unless it's just for an SRP initiator.

GodAtum · 17 Nov 2012 at 23:22

Been following the guide here: http://davidhunt.ie/wp/?p=2291

It looked promising but unfortunately nothing works. I got up to the reboot stage after installing opensm and configuring /etc/network/interfaces and suddenly all my network connections are gone. I am also unable to ping the machine.

Shad · 18 Nov 2012 at 09:31

Hmm I followed his guide with my Ubuntu VM and it worked a treat. Unfortunately I'm a complete Linux n00b so don't know what to suggest other than starting again just in case you missed a step?

GodAtum · 18 Nov 2012 at 10:48

Unfortunately I think I will have to abandon this project and sell all my hardware. The PERC does not work, the Mellanox card does not work and now there is no output to my monitor. Basically everything I bought is now a pile of scrap!

RSR · 18 Nov 2012 at 11:12

Is this ESXi 5.1? As ESXi 5.1 has known problems with pass through which ESXi 5.0 didn't... :confused:

GodAtum · 18 Nov 2012 at 11:17

Yep I reverted back to 5.0 but it still had issues so just tried installing Ubuntu.

GodAtum · 18 Nov 2012 at 18:30

Shad said:
Hmm I followed his guide with my Ubuntu VM and it worked a treat. Unfortunately I'm a complete Linux n00b so don't know what to suggest other than starting again just in case you missed a step?

Just redid all the steps. When i try to bring up the network interfaces it says it cannot find the ib0 device.

The kernel recognises it:

05:00.0 Memory controller: Mellanox Technologies MT25208 [InfiniHost III Ex HCA Flash Recovery] (rev a0)

But in ifcong the device ib0 is not listed.

RimBlock · 19 Nov 2012 at 02:49

ifconfig lists active devices (i.e. configured, up and running).

What is listed in /proc/net/dev (asuming Ubuntu uses the same proc hierarchy as RHEL / CentOS).

RB

GodAtum · 23 Nov 2012 at 11:45

I used lspci and my PERC is listed:

05:00.0 Memory controller: Mellanox Technologies MT25208 [InfiniHost III Ex HCA Flash Recovery] (rev a0)

There is a light on the infiniband connector.

RimBlock · 26 Nov 2012 at 01:31

GodAtum said:
I used lspci and my PERC is listed:

There is a light on the infiniband connector.

What is listed in /proc/net/dev ??

I believe the question is why your Infiniband controller is not appearing as a network device.

Giving a listing of the above requested file with show what the system currently believes is configured as a network device regardless of its state (up / down etc).

RB