1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ESXI experts - opinion 10GB network copy speed 90MB/s

Discussion in 'Servers and Enterprise Solutions' started by edscdk, Sep 29, 2017.

  1. edscdk


    Joined: Jul 17, 2008

    Posts: 6,163

    enterprise class storage
    HP G8s blades
    testing in out prod and preproduction (so unused) environments gives the same results

    inter vm (windows 2008 r2) on the same host gives me a copy speed of 90MB/s with a CPU usage of pushing 100% (on 1 core) - VMs are on the same virtual switch

    we have the vmnet3 drivers installed

    any thoughts do other people get the same sort of results?
  2. crinkleshoes


    Joined: Jun 9, 2009

    Posts: 9,809

    Location: Ongar

    Sounds like CPU limit...
  3. Sparky__H

    Wise Guy

    Joined: Aug 9, 2004

    Posts: 1,968

    Location: Sea of Dirac

    chuck more ram and cores at it and see what happens.

    after that it's checking on the host for disk write/read issues and provisioning. making sure you have driver's for the disk controller and not broken old vibs installed.
  4. blueboy2001


    Joined: Oct 18, 2002

    Posts: 4,762

    What ESXi version and build are you running?

    I've come across this but I can't remember what fixed it. I think it might have been an ESXi bug fixed in a later build, or a HP driver bug

    I would:

    Set all Power Management settings to High Performance in the BIOS and inside the VM's.
    Update the blade server firmware with the latest SPP.
    Update the blade enclosure firmware.
    Install the latest build for the version you need from a HP Custom Image.
  5. Syngress


    Joined: Dec 4, 2002

    Posts: 312

    Location: Chelmsford

    Check the power management in the BIOS as mentioned, I had a real issue with a particular VM running MS SQL that was dog slow, turned out to be power management.

    Check MTU sizes.

    Could be CPU related as there is no TCP offload in VM's so more reliant on CPU, throw another vcpu at it and see if that helps.

    90MB/s sounds like something is configured at 1GB/s - Make sure none of the NICs have negotiated at 1000mb/s
  6. Caged


    Joined: Oct 18, 2002

    Posts: 22,364

    Don't write "enterprise storage" and then not expand on that. What is the storage? What relevant statistics are being shown when doing this? What happens if you create two new Server 2016 guests and try again?

    If you storage vmotion the guests to a local datastore (assuming one exists) then what happens?
  7. Lanz


    Joined: Nov 26, 2002

    Posts: 6,815

    Location: Romford

    I did the same test, and you get nowhere near 10GBe speeds, but 300-400MB's should be expected.

    Using ATTO or iometer from inside a VM, what sort of performance do you see against your `enterprise storage`?
  8. edscdk


    Joined: Jul 17, 2008

    Posts: 6,163

    Jumbo frames get me 200 - 250 MBS copy speed but our physical switches don't have that so it only helps vm to vm on the same host (so not configurable)

    CPU seems to be able to max out 1 and only 1 core - dual / quad makes no difference - I was amazed to see 100% core usage! (xeon e5-2660 v4)

    We built the G9s with the latest HP esxi 6 ISO..

    ok someone has a point I've just spotted one of our environments can easily break the 1GB speed barrier, I think we have something to fix here!

    Good to see people confirming way over 1GB speeds (lanz I see pushing those speeds in one of our environments)

    Thanks for the replies ill pick up this "anomaly"
  9. Vince


    Joined: Oct 30, 2003

    Posts: 5,253

    Location: Essex

    When you say ent class storage what is it? Are we talking EVA/3PAR? How many disks? What config on the disks? How is it connected? So many questions. I can do a quick throughput test on a very similar environment:

    DL380 G7 Hosts
    HP 6300EVA FC 10GB Dual Controllers
    Also using vmxnet3.

    Ill give it a go now see what we got.

    Edit: Im seeing speeds in the 100's of MB/s on small files. I did start a bigger file and was seeing speeds approaching the gb/s between datastores.
    Last edited: Oct 12, 2017
  10. edscdk


    Joined: Jul 17, 2008

    Posts: 6,163

    VMAX with £100k+ of SSDs + 100TBish of HDDs
    8GB fibre connections
    VMNet3 HP BL460 G8's

    I think I have the important information which is - I should be seeing much better than 1GB speeds
  11. DustyMiller

    Wise Guy

    Joined: Oct 18, 2002

    Posts: 1,657

    Location: Swindon

  12. Hellsmk2


    Joined: Oct 18, 2002

    Posts: 4,153

    First thing I would do is remove storage as a bottleneck in your testing. I use LANbench, which should give you accurate throughput without the overhead of storage r/w.

    Have done countless 10gb deployments, and usually anywhere between 5-8Gbps means it is working as intended with VMXNET3. The e1000e nic can also be used to test, as it will normally hit 2.3 to 3Gbps, which would confirm you are at least utilising a 10gb network. That would help you identify whether it was the vmxnet3 nic at fault.

    I'd also be wondering if you are missing something simple if your speeds are consistent with a 1gb network. We did a recent deployment where our client left their vm gateways as a 1gb switch, which meant traffic left the 10gb network and was instantly gimped. Obviously easily resolved by making the 10gb switches the gateway.
    Last edited: Oct 14, 2017

Share This Page