10gbit home network speed issues

Kei · 23 Nov 2019 at 14:28

I've never fully seen 10Gbit on mine and it is always on windows I see speed issues. Changing to jumbo frames is not an ideal option as it will require packet fragmentation in order to access the internet and any other devices on your network which don't support it.

A loopback test will show what the cpu can handle as you are testing the network stack.

This is what I could get on linux.

Same test and hardware on windows could barely edge out 10Gbit. (Yes I tried NTTcp and multiple instances, all came out the same)

This is the best I could get out of mine during testing which is a shade over 7Gbit from a local NVME ssd to my server running 7x western digital SE hard disks in RAID 5. (Which is the limiting factor here)

Some good info can be found here for checks that you can do in powershell. Get-NetAdapterHardwareInfo will tell you the pcie version and link width that it is actually getting, which is quite useful.

Quartz · 23 Nov 2019 at 14:47

jonneymendoza said:
The desktop pc is connected to the negtear 10gb switch using a cat 8 cable thats 2m long and the ubuntu server desktop is also connected to the same switch using a CAT8 cable.

What is Cat 8 cable? You want Cat 6 cable that is NOT (edit) copper and copper-clad aluminium (CCA)

You should also try connecting the two 10 Gb NICs directly - you'll need to manually assign IP addresses & stuff - and measuring the speed you get that way.

spoffle · 23 Nov 2019 at 14:59

Quartz said:
What is Cat 8 cable? You want Cat 6 cable that is copper and copper-clad aluminium (CCA)

You should also try connecting the two 10 Gb NICs directly - you'll need to manually assign IP addresses & stuff - and measuring the speed you get that way.

You definitely don't "want" CCA cable.

Quartz · 23 Nov 2019 at 15:02

spoffle said:
You definitely don't "want" CCA cable.

Whoops! I meant to write ISNT. I'll change it.

DragonQ · 23 Nov 2019 at 16:33

I only get 6.4 Gb/s from my desktop to server (i.e. server running iperf -s, desktop running iperf -c...I assume that's the right way around) and 5.8 Gb/s the other way. Considering the server's NIC is restricted to 8 Gb/s raw that's not bad I guess.

jonneymendoza · 25 Nov 2019 at 09:47

Quartz said:
What is Cat 8 cable? You want Cat 6 cable that is NOT (edit) copper and copper-clad aluminium (CCA)

You should also try connecting the two 10 Gb NICs directly - you'll need to manually assign IP addresses & stuff - and measuring the speed you get that way.

I have some new cat 6a cables and i only get around 2gbit speeds still when performing that Loopback test between my windows desktop as a client and my ubuntu server .

i also checked the PCI E port slot on my ubunut server and it should not bottleneck the NIC.

i will double check the PCI E slot on my main desktop but should be ok as that motherboard is a Asus x99 Board

ChrisD. · 25 Nov 2019 at 09:48

I'll test my Windows desktop to my QNAP and also Ubuntu VM later on today and see if I get the same issue.

ChrisD. · 26 Nov 2019 at 12:21

It's a Windows thing (shock horror):

My desktop (Win 10) to/from Ubuntu (VM) = 1.5 Gbps
My desktop to/from QNAP = 3.4 Gbps
Server 2019 (VM) to/from Ubuntu = 4.6 Gbps
Server 2019 to/from QNAP = 4.5 Gbps
Ubuntu to/from QNAP = 9.4 Gbps
Server 2019 to/from Server 2019 (both VMs, same vSphere environment so internal networking) = 3.3 Gbps

I'll do some more testing but the only way I can get full speed is between Linux type boxes.

Above done with default iperf settings, I haven't changed window size.

bremen1874 · 26 Nov 2019 at 12:32

SMB version issues?

ChrisD. · 26 Nov 2019 at 12:49

bremen1874 said:
SMB version issues?

Not sure, if I do a test to the loopback interface on my desktop I get 15 Gbps.

jonneymendoza · 26 Nov 2019 at 13:01

Umm I may try install Ubuntu on my desktop. Duel boot into it and retest the speeds

ChrisD. said:
Not sure, if I do a test to the loopback interface on my desktop I get 15 Gbps.

bremen1874 · 26 Nov 2019 at 13:29

jonneymendoza said:
Umm I may try install Ubuntu on my desktop. Duel boot into it and retest the speeds

For testing, you should be able to get away with a live version running from a DVD or USB stick.

jonneymendoza · 26 Nov 2019 at 14:08

bremen1874 said:
For testing, you should be able to get away with a live version running from a DVD or USB stick.

Wouldnt the USB interface/controller affect the speed though?

bremen1874 · 26 Nov 2019 at 14:12

jonneymendoza said:
Wouldnt the USB interface/controller affect the speed though?

Not unless your testing included copying data to/from the USB drive.

jonneymendoza · 26 Nov 2019 at 15:24

bremen1874 said:
Not unless your testing included copying data to/from the USB drive.

So just running iperf via usb bootable live cd would do the job?

bremen1874 · 26 Nov 2019 at 15:30

I can't see any reason why not. It'd be simple and quick to try it.

bledd · 26 Nov 2019 at 16:08

A live os often totally loads into ram.

jonneymendoza · 26 Nov 2019 at 16:30

bledd said:
A live os often totally loads into ram.

o i did not know that.

very interesting. i will give this a go later

RSR · 30 Nov 2019 at 15:52

I've never really had any problems with mine maxing out a 10Gb connection moving files about, I use CAT6 cabling end to end with Intel or Aquantia NICs which seem to work well.

jonneymendoza · 30 Nov 2019 at 17:58

Have not tried using Linux yet but hopefully will soon and see the speed I get on iperf and @RSR what hdd are you using at either ends of that connection?