The (un)Official VMWare ESXi thread.

[RimBlock]

Come one, come all and welcome the (un)Official ESXi OCUK thread.

This thread is dedicated to VMWares ESXi Hypervisor virtulisation software.

So what is it ?.
ESXi is an operating system designed for sitting on hardware and providing multiple virtual environments or containers in which you can install other operating systems and make them think they are on standalone machines.

VMWare, of course, have a full overview here if you wish to take a look.

Documentation.
ESXi 5, the current version, documentation can be found here.

What will it run on ?.
Checkout the VMWare ESXi Hardware Compatibility List (HCL). ESXi can be pretty fussy over hardware with network chipsets being a common issue. There are also community efforts to expand on the drivers available so have a google if you don't find your chipset on the HCL. Note also that if it is not on the HCL, it does not mean it wont work, just that it has not been tested. Quite a few people run Supermicro motherboards in their ESXi servers without an issue or any patching and none are on the HCL.

Where is the free version ?.
Down load the free version of ESXi from here. You will need to register and you will get a license code required to turn the free trial version in to the free licensed version. You loose some features in the free version which are added in paid versions but currently you can run a machine with one CPU & 32GB ram on it. The free license currently has no expiry date.

I will add more as more information builds up in the thread.

vSphere 5.1 downloads;
vSphere 5.1 is available for download here (60 day trial or paid versions).

vSphere 5.1 Hypervisor (the free one) available herehttps://my.vmware.com/group/vmware/evalcenter?p=free-esxi5&lp=default (click the bottom left download link, sign in then download).

 

[RimBlock]

So what does it look like ?.
These shots are taken from my home ESXi 5 server.

The vSphere management application server summery panel.

This is the main vSphere Windows client management application which is used to control the server and the virtual machines (vCenter is the enterprise version which can control multiple ESXi servers rather than just one at a time).

From the summery screen we can see the server makeup including CPU, Ram, network ports, datastores (hard drives) and network adaptors.

Direct Path I/O (VT-d or passthrough)

If you have the right hardware then you can use direct path I/O which enables you to pass a device directly to a virtual machine and have it control it as if it were part of the virtual machines own hardware. Devices passed to a virtual machine in this way are exclusive to that machine and as such are not accessible tot he other virtual machines or the servers virtual resources.

Current Direct Path I/O compatibility
Intel
Processors in the i5/i7 and Xeon ranges support VT-d which is Intels version but you also need a motherboard with a Q67/Z68 or server/workstation chipset in order to use it. I will update on Ivy bridge at some time later.

AMD
If someone can confirm then I will also add details for AMD processors as I am not so familiar.

I will add some more screens as I get round to doing them.

 

[RimBlock]

Nice,

I saw you have a few others up on youtube .

RB

 

[RimBlock]

As requested, some kit I have used with ESXi;

I have separated out the 'supported for ESXi management network' and 'for ESXi networking' as some nics can be used from within ESXi even though they are not supported for the management network. At least one nic with ESXi management network support is required for ESXi to install. I am not including info about nics that can have their drives patched using various techniques as I have not done this.

If I list anything as 'believed' then it is strongly suspected fro evidence on other comparable hardware and information available on the internet from fairly reputable sources (Intel forums for example) but I have not personally confirmed by testing myself.

Processors

Intel

Celerons (LGA1155)
Supported on most (possibly all) LGA1155 boards.
ECC ram support unconfirmed.
Supports VT-x for virtulization

Pentiums (LGA1155)
Supported on most (possibly all) LGA1155 boards.
ECC ram support unconfirmed.
Supports VT-x for virtulization

i3 Gen2
Supported on most (possibly all) LGA1155 boards.
ECC ram support verified by myself.
Supports VT-x for virtulization

i5 Gen2
Supported on most (possibly all) LGA1155 boards.
ECC ram support unknown as no ECC boards available that take an i5 TMK.
Supports VT-x for virtulization
Supports VT-d for Direct path I/O

i7 Gen2
Supported on most (possibly all) LGA1155 boards.
ECC ram support unknown as no ECC boards available that take an i7 TMK.
Supports VT-x for virtulization
Supports VT-d for Direct path I/O

E3 Xeon 1200
Supported on Intel C202/C204/C206 motherboards.
Suspected supported on most (possibly all) LGA1155 boards but not tested.
ECC ram supported.
E3-12X0 have no onchip video (see chipset notes)
E3-12X5 have onchip video (see chipset notes)
Supports VT-x for virtulization
Supports VT-d for Direct path I/O
​

Motherboards

Intel

Intel DQ67EP mITX board
VT-d supported.
Onboard nic not natively supported for management network.
Onboard nic supported for ESXi networking.

Intel S1200KP mITX board
Intel C206 workstation chipset
Board believed to have VT-d disabled in Bios by Intel
Board ECC (unbuffered) memory or non-ECC memory
One nic supported by ESXi for management network.
One nic supported within ESXi for networking.

Intel S1200BTL ATX board
Can take up to 32GB ram with Kingston KVR1333D3E9S/8G sticks.
Intel C204 chipset
ESXi Certified
VT-d supported
One nic supported by ESXi for management network.
One nic supported within ESXi for networking.
​

Supermicro

Supermicro X9SCi-LN4F
Can take up to 32GB ram with Kingston KVR1333D3E9S/8G sticks.
Intel C204 chipset
VT-d supported
Four nics supported by ESXi for management network.

Supermicro X9SCM-F
Can take up to 32GB ram with Kingston KVR1333D3E9S/8G sticks.
Intel C204 chipset
VT-d supported
One nic supported by ESXi for management network.
One nic supported within ESXi for networking.

​

Chipsets

Intel

C202 - Entry server, motherboards usually have separate video built-in, processors without onchip video recommended.
C204 - Entry server, motherboards usually have separate video built-in, processors without onchip video recommended.
C206 - Entry workstation, recommend processors with onchip video or add-on video card.

Q67/Z68 - VT-d support possible but check manufacturers specs to confirm it is enabled in Bios.
​

Note: I am in no way against AMD processors or boards but have no experience with them and ESXi. If someone has the infor on good boards, processors or combos then please post away.

 

[RimBlock]

I have a similar setup but went for the Intel Xeon 1230 - less over head, no inbuilt graphics though so you will require a PCI-E graphics card spare.

Less power usage though.

Nice to know the E3s work on the Q67 boards. Any particular reason you didn't go for a E3-1235 Xeon though with inbuilt graphics (P3000) and keep the PCI-e free ?.

RB

 

[RimBlock]

I thought the DQ67SW has inbuilt graphics, but it requires a CPU with built in Intel HD graphics which the i5 2500 does.

DQ67SW - Tech Specs

i5 2500 - Tech Specs

Should work fine as I have been running a DQ67Ep with a i5-2400 until recently. Needed to add an Intel CT nic though as the 82579LMchipset is not supported out of the box with ESXi for the management network although you can use it from inside ESXi for passthrough or standard networking. Alternatively you could patch the drivers but when I followed the on-line instructions it either didn't work or was pretty unstable. The Intel ET dual port is great. I had one running and it was very good and passthrough worked like a charm.

RB

 

[RimBlock]

The DQ67SW has no graphics, hence why you require a chip with graphics to use it - you are correct, the i5 2500 does - but that wasn't the point of my post.

Also makes the AMT tool not work so you can't use IP KVM.

Don't talk to me about AMT . I still cant get it installed on my Win 7 home premium machine as it keeps telling me I am not an administrator (my account has administration rights). The server sits behind me so I haven't bothered to give the Intel support contact a call to sort it out.

RB

 

[RimBlock]

I only used the PCI-E graphics card for installation, then removed it.

I was already way over budget by buying the 1230 E3 in the first instance, as I was initially spec'ing an i5 2300 - lol.

Fair enough . My home build is a tad over budget as well. Probably getting 2x8GB ECC mem sticks was a bit cheeky .

It's pretty simple. I used VNC Viewer Plus+ - just ensure TLS encryption is turned off and you have set an IP in the AMT settings in the BIOS.

I will give that a go. Trying to install Intels software that came with the BTL board and it fails at the precheck and there is no way around it as far as I have found. Thanks

RB

 

[RimBlock]

I'm running ESXi 5 on my HP Microserver and it's a solid little box.

Simply add some more RAM and disks.

I've got mine running from a USB drive plugged into the back with the images on the HD's

They are sooooo cheap after the £100 cash back that I've ordered a second just as a backup.

He has that and is looking to upgrade .

Finally an ESXi thread! Anyone got any recommendations for a mini ITX server running ESXi?

Eyeing up the Fractal Design R3 or R2 line case to fit a mini-ITX board. I need one for an office to host a DC/Exchange/File Server/print server & whatever else I can squeeze on to it! Need something that has low consumption, maybe an E3/i3 with about 12+ GB RAM, won't serve more than 20 users max but may need room for expansion. Probably will use an LSI/Dell Perc RAID card and use the 6 3.5" drive bays in it for RAID 10 storage!

The Intel S1200KP board really appealed to me until I found out VT-d was disabled! Would like to passthrough a RAID card so I can manage it from one of the VM's in case it goes **** up!

I'm planning a similar set up for home so if this is too overkill for a home server I'd like another recommendation if anyone can help me out. I currently have an Adaptec 5405 RAID card + storage that I'd like to move to a more powerful home server which is currently in a HP MicroServer (which I'd like to keep as a home lab) which runs ESXi. So if the server for the office isn't too overkill (keeping in mind power consumption and house bills, etc!) I'd be happy for the same spec.
Oh and I would need the spec to be supported by ESXi's HCL!
Appreciate any help!

Bottom line is that there is nothing, TMK, mITX that is on the ESXi HCL.

There is also nothing I have found that supports VT-d and has a ESXi management network supported nic out of the box. The Intel DQ67EP is a great board and supports VT-d but the slot needs to be used for a nic for the ESXi management network. The S1200KP is an almost perfect ESXi board apart from the VT-d being, allegedly as I have not been able to test myself, disabled. The next best thing is the S1200BTS (MATX) or S1200BTL (ATX) boards but they are of course a bit bigger than you were looking for. They are however on the HCL but require ECC ram.

I an in the process of selling off my DQ67EP based setup to someone as I have moved to the S1200BTL for this exact reason. I guess you may be able to pass the two e-sata ports to a VM on the DQ67EP board and attach external e-sata das boxes for the drives but you are limited by the two e-sata connections so may get limited with more than two mechanical drives on each. Maybe a solution for home but not for the office. The S1200BTS/BTL boards (pdf) also come as ready built bare bones in 1U rack or pedestal configs. The added advantage, especially for work is the ability to adjust and add more cards for storage attachment / networking / san in the future.

As an aside, the Fractal mini array is a nice case, have one of these, although with the one fan the area above the drive cage can get a bit hot. The Lian Li PC-Q25, Q18 and Q08 are also reasonable contenders.

RB

 

[RimBlock]

If the DQ67EP is anything like the DQ67SW then the eSATA is on the same controller as the SATA. I have 2 internal drives looped through the back plugged into the eSATA.

Therefore you need a cheap SATA card (SIL3114 works great) for system drive and you can then pass the 6 Intel ports to the VM.

Quite possibly . The problem witht he DQ67Ep then is that it only has the one PCIe slot which is needed for a nic compatible with the ESXi management network. No space for an extra storage controller. I am fairly sure USB drives would not be preferred .

RB

 

[RimBlock]

ESXi 5 doesn't allow USB datastore, it has to be on a proper SATA controller.

So yes, looks like the EP is out of the question if you ask me. Definitely need to have more than one expansion slot. I am planning on using my x16 for a SATA card at some point, and my 2 PCI-E slots for NIC's.

My only PCI slot is used for my SIL3114 for datastores.

I was actually suggesting connecting the USB drives to the controller which is then passed to the VM, although to be honest, it was only a jokey suggestion .

RB

 

[RimBlock]

Might just go with a mATX motherboard then and maybe a Silverstone Fortress case

I currently have two X9SCM-F boards in machines I am building for clients but they are not ESXi server builds. They do work fine with ESXi though. Great little boards. The SCL I have not tried.

The main difference between those two models from Supermicro and the Intel board is that the Intel is consumer level and the Supermicros are server grade. The X9 boards will support VT-d with an E3 but not a i3, the Intel will support VT-d with an i5 or above. The Supermicros only support ECC ram (according to specs and I have not tried non-ECC TBH). The Intel board only supports non-ECC ram. An E3-1230 should be around the midpoint between i5 and i7 so it should not be too much of a price difference between the two. Personally I would go Supermicro over consumer Intel but if you compared like for like with the Intel S1200BTx series then it would be a bit more difficult.

RB

Oh and on the Fortress case, I have just collected a Fortress mini for another client and it is pretty nice, the outside panels are rugged and tough and look great, the inside brackets are really thin though and easily bendable by hand. Not sure if the big Fortress is the same but thought I would mention it.

 

[RimBlock]

Bringing this thread back from the depths.

vSphere 5.1 is due out 11th September.

It has a number of interesting features like;
No vRam tax - Versions priced by socket only with no individual vRam version specific restrictions.
New SSD Monitoring – New “smartd” module provides Wear Leveling and other SMART details for SAS and SATA SSD.
Enhanced iSCSI support – Jumbo frames supported on all iSCSI adapters with full UI support.
Improved vSphere Distributed Switch – Greater scalability and more efficient administration with configuration rollback, configuration backup and restore, and enhanced network traffic monitoring.
Improved Graphics for Desktop VMs – Run hardware intensive 3D graphics applications on Desktop VMs with Virtual Shared Accelerated Graphics, now supporting the acceleration of VDI workloads using physical GPU resources. With this new capability it is now possible to virtualize physical GPU resources, sharing them across virtual desktops.
Official LACP support (for network link bonding).

List of vSphere 5.1 new features here.

Article on The Register here.

RB

 

[RimBlock]

vSphere 5.1 is available for download here (60 day trial or paid versions).

vSphere 5.1 Hypervisor (the free one) available herehttps://my.vmware.com/group/vmware/evalcenter?p=free-esxi5&lp=default (click the bottom left download link, sign in then download).

RB

 

[RimBlock]

After a quick download and upgrade I have to say the resulting server was not as I had hoped. There were a number of issues and I decided to to a complete new install. To be fair, a fresh install is the default option.

I have a 4 port Intel ET network card and a HP 1810-24G switch and setting up the LACP was very easy and seems to be working fine. I have not load tested to confirm speeds but all 4 light are flashing on the switch.

Upgrade of the VMware tools was same as usual for Linux and Windows. My CentOS installs went smoothly but my WHS-2011 install does not seem to like the VMXNET3 adaptor. Changing it back to a ET1000 sorted any issues out. I will have to dig a bit deeper in to that one.

Copying a 1.5GB test file hit transferred between 109MB/s and 122MB/s from my PC (Hitachi HDD, Broadcom Netlink (TM) integrated network, Win 7 64bit) to my server on the 4x1GbE trunk (Intel Quad ET, Stablebit Drivepool storage, WHS 2011).

The vSphere client feels a fair bit faster. A few extra storage metrics predefined.

Only a very quick install and browse so I am sure more info will come to light as more people get their hands on it and I have a bit more of a play around.

RB

 

[RimBlock]

Well i'm going to look at the upgrade, just need to download it, and maybe figure out if I can do an upgrade like applying patches or else I need to go and move stuff round in the garage to get the server monitor connected!!

WHS2011 doesn't have drivers for the VMXNET3, if you can install them when it says no nic detected it works fine, I had to extract the drivers from a 2008r2 machine with vmware tools installed.

Found an post on the VMware Communities regarding the issues I am seeing. Previously the WHS box was running with a VT-d passthrough NIC but as I am now using teaming (LACP) I though I would not bother. The NIC is picked up as a VMXNET3 adaptor and installing the updated VMTools does not fix the problem for me. According to the post this affects WHS 2011 / Win Server 2008r2 with remote access turned on and something is manipulating packets for traffic flow (QoS etc). I did try removing the network device in device manager and then refreshing and it found the adaptor and installed the VMXNET3 driver but still no network.

The install disc (ISO download) allows an upgrade (ie. patch) or fresh install. The upgrade didn't seem to go so well for me. VM auto-startup screen was acting strangely with turning off of auto-startup for VMs resulting in multiple instances of all VMs appearing in the auto-startup pane of that window. There were a couple of extra issues as well but I cannot think what they were at the moment. Bottom line is that it didn't instil me with confidence so I installed from scratch which seems to be working as expected.

RB

 

[RimBlock]

For those who are using the free vSphere hypervisor and were happy, as I was, that VMware have removed the vRam entitlement in it, please be aware there is a hard limit of 32GB ram per machine (physical) with the free version now.

I also had a rather worrying encounter ove rthe last day or so with vSphere. I had just setup a raid 1 array with a couple of Seagate Barracuda 1.5TB drives and after importing as a data store I started to copy my WHS-2011 Vm from my current store to this new raid protected one. The copy estimated over 5 days to copy the 200GB vmdk file.

I left it to run overnight but it was still reporting days to complete in the following evening so I cancelled the move. The vSphere client still reported the copy percentage and no acknowledgement of the cancellation in the bottom notification bar but the copy window closed.

I then started to see pauses in the vSphere client. The performance logging listed an average write of 9MB/s to the array during the copy but was showing gaps in the performance charts and was continuing to show more gaps as the chart was updated.

Logging on to the servers console I was also experiencing pauses of, estimated, 10 seconds or above where I could not do anything.

I rebooted the server and went in to the raid controllers BIOS and found the raid 1 array was in degraded status and unsynced. I let it sync and rebooted. I will see how it goes copying tonight.

My concern is that a degraded array had such a big impact on the vSphere 5.1 server as a whole.

I also now have an issue where after copying back the files that were moved to the array I get an error trying to copy back the WHS-2011-flat.vmdk file saying it already exists in the destination directory. The file does not exist in the destination but a WHS-2011.vmdk does.

The VM will not start and errors with a message about the VM missing a disk file that snapshot is reliant on. I have converted the only vmdk in to an independent disk but no joy. I cannot start this VM and cannot copy this -flat file over. I can stand to loose this VM if need be but this is also a bit of a concern if it is so easy to trash a VM on a failed move.

RB

 

[RimBlock]

jfish has kindly made an VM auditing script available here. Linking to it in this thread so it doesn't just disappear off the end of the forum over time.

RB

 

[RimBlock]

Unmasked link for VMware communities posters here.

Look to be very helpful.

I have just put vSphere 5.1 Essentials on a client machine, setup the vCenter appliance and am now sorting out the vMA for use with GhettoVCBg2 although I notice vSphere 5.x is not listed as being supported where as GhettoVCB states that it is.

Anyone able to confirm GhettoVCBg2 does work ok with vSphere 5.x (pref 5.1 if possible).

Thanks.
RB

 

[RimBlock]

Hi all,

I've added a HP NC380T to my ESXI ITX server as the built in Realtek 8168 stopped working in ESXI 5.1. The HP NC380T I've added is working a treat but I've noticed that the adapter is listed as a Broadcom NetXtreme 5706 Gigabit Ethernet.

What should I do get it recognised as the correct adapter? Assuming its anything like windows surely running it without the correct driver will limit the performance?

In the HP manual for the card, available here (pdf), under specifications it is listed as using a Broadcom 5706C chipset.

Regards
RB