ESXI experts - opinion 10GB network copy speed 90MB/s

Soldato
Joined
17 Jul 2008
Posts
7,367
enterprise class storage
HP G8s blades
testing in out prod and preproduction (so unused) environments gives the same results

inter vm (windows 2008 r2) on the same host gives me a copy speed of 90MB/s with a CPU usage of pushing 100% (on 1 core) - VMs are on the same virtual switch

we have the vmnet3 drivers installed

any thoughts do other people get the same sort of results?
 
Associate
Joined
9 Aug 2004
Posts
2,061
Location
Sea of Dirac
chuck more ram and cores at it and see what happens.

after that it's checking on the host for disk write/read issues and provisioning. making sure you have driver's for the disk controller and not broken old vibs installed.
 
Soldato
Joined
18 Oct 2002
Posts
4,898
What ESXi version and build are you running?

I've come across this but I can't remember what fixed it. I think it might have been an ESXi bug fixed in a later build, or a HP driver bug

I would:

Set all Power Management settings to High Performance in the BIOS and inside the VM's.
Update the blade server firmware with the latest SPP.
Update the blade enclosure firmware.
Install the latest build for the version you need from a HP Custom Image.
 
Associate
Joined
4 Dec 2002
Posts
316
Location
Chelmsford
Check the power management in the BIOS as mentioned, I had a real issue with a particular VM running MS SQL that was dog slow, turned out to be power management.

Check MTU sizes.

Could be CPU related as there is no TCP offload in VM's so more reliant on CPU, throw another vcpu at it and see if that helps.

90MB/s sounds like something is configured at 1GB/s - Make sure none of the NICs have negotiated at 1000mb/s
 
Caporegime
Joined
18 Oct 2002
Posts
26,053
Don't write "enterprise storage" and then not expand on that. What is the storage? What relevant statistics are being shown when doing this? What happens if you create two new Server 2016 guests and try again?

If you storage vmotion the guests to a local datastore (assuming one exists) then what happens?
 
Soldato
Joined
26 Nov 2002
Posts
6,852
Location
Romford
I did the same test, and you get nowhere near 10GBe speeds, but 300-400MB's should be expected.

Using ATTO or iometer from inside a VM, what sort of performance do you see against your `enterprise storage`?
 
Soldato
OP
Joined
17 Jul 2008
Posts
7,367
Check the power management in the BIOS as mentioned, I had a real issue with a particular VM running MS SQL that was dog slow, turned out to be power management.

Check MTU sizes.

Could be CPU related as there is no TCP offload in VM's so more reliant on CPU, throw another vcpu at it and see if that helps.

90MB/s sounds like something is configured at 1GB/s - Make sure none of the NICs have negotiated at 1000mb/s

Jumbo frames get me 200 - 250 MBS copy speed but our physical switches don't have that so it only helps vm to vm on the same host (so not configurable)

CPU seems to be able to max out 1 and only 1 core - dual / quad makes no difference - I was amazed to see 100% core usage! (xeon e5-2660 v4)

We built the G9s with the latest HP esxi 6 ISO..

ok someone has a point I've just spotted one of our environments can easily break the 1GB speed barrier, I think we have something to fix here!

Good to see people confirming way over 1GB speeds (lanz I see pushing those speeds in one of our environments)

Thanks for the replies ill pick up this "anomaly"
 
Man of Honour
Joined
30 Oct 2003
Posts
13,229
Location
Essex
enterprise class storage
HP G8s blades
testing in out prod and preproduction (so unused) environments gives the same results

inter vm (windows 2008 r2) on the same host gives me a copy speed of 90MB/s with a CPU usage of pushing 100% (on 1 core) - VMs are on the same virtual switch

we have the vmnet3 drivers installed

any thoughts do other people get the same sort of results?

When you say ent class storage what is it? Are we talking EVA/3PAR? How many disks? What config on the disks? How is it connected? So many questions. I can do a quick throughput test on a very similar environment:

DL380 G7 Hosts
HP 6300EVA FC 10GB Dual Controllers
Also using vmxnet3.

Ill give it a go now see what we got.

Edit: Im seeing speeds in the 100's of MB/s on small files. I did start a bigger file and was seeing speeds approaching the gb/s between datastores.
 
Last edited:
Soldato
OP
Joined
17 Jul 2008
Posts
7,367
When you say ent class storage what is it? Are we talking EVA/3PAR? How many disks? What config on the disks? How is it connected? So many questions. I can do a quick throughput test on a very similar environment:

DL380 G7 Hosts
HP 6300EVA FC 10GB Dual Controllers
Also using vmxnet3.

Ill give it a go now see what we got.

Edit: Im seeing speeds in the 100's of MB/s on small files. I did start a bigger file and was seeing speeds approaching the gb/s between datastores.

VMAX with £100k+ of SSDs + 100TBish of HDDs
8GB fibre connections
VMNet3 HP BL460 G8's

I think I have the important information which is - I should be seeing much better than 1GB speeds
 
Soldato
Joined
18 Oct 2002
Posts
4,521
First thing I would do is remove storage as a bottleneck in your testing. I use LANbench, which should give you accurate throughput without the overhead of storage r/w.

Have done countless 10gb deployments, and usually anywhere between 5-8Gbps means it is working as intended with VMXNET3. The e1000e nic can also be used to test, as it will normally hit 2.3 to 3Gbps, which would confirm you are at least utilising a 10gb network. That would help you identify whether it was the vmxnet3 nic at fault.

I'd also be wondering if you are missing something simple if your speeds are consistent with a 1gb network. We did a recent deployment where our client left their vm gateways as a 1gb switch, which meant traffic left the 10gb network and was instantly gimped. Obviously easily resolved by making the 10gb switches the gateway.
 
Last edited:
Back
Top Bottom