MTU / Jumbo packet problem...

[faxfan]

In an interested is squeezing as much "line" speed out of my network as possible I attempted to enable Jumbo packets on my network.

Elements involved -

Windows 10, Asus X99 deluxe motherboard with Intel 1218-v NIC
Cisco WS-C3560CX-12PC-S
ESXi 5.5, Broadcom NetXtreme NIC (E1000E driver)
Ubuntu 14.4 (under ESXi) SMB share

Baseline file transfer speed as tested 100MB/s

Steps -

Enabled Jumbo frames on Windows to via the intel interface (latest drivers) to 9088 (preset value)
Enabled MTU on the Cisco switch to 9000
Increased MTU on ESXi switch to 9000
Increased MTU on Ubuntu to 9000
(rebooted all elements)

Tested using MTUroute on windows to Ubuntu which confirmed I 9000 MTU, tested from Ubuntu to Windows ping <host> -M do -s 8972 this also confirmed 9000.

All good and well, right?

So baseline 100MB/s - new speed 85MB/s ?!?
Decreased the MTU on the Windows client to 4088 (preset value) - new speed 106MB/s.

So MTU is kinda working and the problem "appears" to be on windows or a compatibility issue between Windows and Ubuntu.

Anyone played with MTU before?

 

[joeyjojo]

So you've got gigabit ethernet on every machine and you're disappointed with 100 Mb?

My first thought wouldn't be MTU - default settings will be fine for a gigabit link, 1500 MTU should go up to several thousand Mbps before being the limiting factor.

You need to track down the bottleneck first, I would suspect one of the devices on the end (read/write speeds?) or your testing method before the ethernet link.

 

[faxfan]

I'm not disappointed at all, just playing around and seeing what is possible. All native read / write speed are more than fine.

I've just loaded a Windows 2012 server on ESXi and boom! 115MB/s - line speed (ok so 125MB/s is line speed but 115MB/s is probably the highest expectation). So it does like a Ubuntu / Windows client / SMB issue... hmmm

 

[joeyjojo]

I mean what are your expectations? Saturating read/write speeds presumably? Because from a quick glance it looks like you have gigabit all the way along so obviously 100 Mb is a bit slow.

What are you using for the file transfer? I'd start with something very lightweight if I were you, maybe plain FTP. If scp you're probably using WinSCP and should try a different client e.g. FileZilla.

 

[faxfan]

This is all using SMB, pretty much proved it's a Linux / Linux SMB issue -

All elements set to 9000 MTU and proved with basic MTU testing.

SMB file transfer speeds from Windows 10 client to Linux SMB share -
1500 MTU (on Windows) = ~100MB/s
4000 MTU (on Windows) = ~106MB/s
9000 MTU (on Windows) = ~85MB/s

SMB file transfer speeds from Windows 10 client to Windows server SMB share -
1500 MTU (on Windows) = ~100MB/s
4000 MTU (on Windows) = ~106MB/s
9000 MTU (on Windows) = ~115MB/s <=== This is expected speed

Ftp maxes out at around 40MB/s

 

[Steveocee]

SMB is a problem full stop. A certain well publicised youtube reviewer lately has had problems with a huge file server that should be doing GB transfers and was only capable of about 300MB due to SMB.

NFS would probably work better.

 

[joeyjojo]

I have no idea what speeds you're getting because you keep mixing up bytes and bits. Are you getting 115 Mbps or 115 MBps (920 Mbps)?

Is that FTP 40 MBps or 40 Mbps?

What's the sustained read/write of the storage at either end?

 

[faxfan]

all should be MB/s as per Windows shows on the file copy. Sustained read / write are well in excess of the copy speeds so that is not the bottleneck. Anyway pretty sure it's a Linux / Linux SMB problem as Windows to Windows gives the expected results.

 

[faxfan]

SMB is a problem full stop. A certain well publicised youtube reviewer lately has had problems with a huge file server that should be doing GB transfers and was only capable of about 300MB due to SMB.

NFS would probably work better.

NFS is my next stop on Linux (given that Windows to Windows is fine).