ISL Speed Issues

m4cc45 · 7 Aug 2013 at 20:14

Hi,

This is a weird one.

We have a 10Gb inter-site link with around 4ms of latency.

Now we are experiencing issues that the fastest transfer from either end is 20MB/s. If we have 4 servers they all run at 20MB/s and so on.

Initially the network support team suggests that this should be the fastest we can get - initially they said per TCP stream and now it's per server. I believe it should be per TCP stream and that we should be seeing far faster speeds over the ISL.

Opinions?

M.

m4cc45 · 8 Aug 2013 at 06:30

It's a nightmare getting any information out of them. So the below is what I know to be correct apart from the connection.

MPLS connection (assuming fibre)
Checkpoint firewalls at each end
Target devices are servers (ranging from DL380 G5s onwards)

M.

m4cc45 · 8 Aug 2013 at 19:24

Super paranoid is right - it actually terminates onto a BT owned unit then firewall but you get the idea.

I think it's QoS / bad routing / bad config or bad cable but I don't think it's bad windows which is what it seems to be blamed on (tcp stack at the moment)

M.

m4cc45 · 10 Aug 2013 at 10:32

When we go above 64k it doesn't really make any difference - between source and destination I can see 160Mb (roughly cant remember exactly) connections available (giving us the 20MB/s) but this gets magnified out (i.e. from a single source I can do 3 / 4 connections (then disk contention comes in) and get 20MB/s to each destination. I can do 3/4 simulatenous iperfs and get the same results to different destinations.

This is all over the same ISL.

Just trying to think of something that would allow me to have 80MB/s bandwidth (though divided so it's 4 x 20MB/s on a 1Gb link - I know I can't access all 10Gb of it due to the 1Gb link).

Changing the window size doesn't really matter and I believe SMB 2.1 (Windows 7 / 8) uses multiple TCP streams so wouldn't this already max it out?

Basically what I'm saying is I can see bandwidth available but can't access even though multiple connections from the source to multiple destinations allows me more.

M.

m4cc45 · 12 Aug 2013 at 18:00

Apologies. Link is 10Gb. The equipment supports 1Gb inbetween (firewall to switches) in some places this has several uplinks but I'm not sure between the MTU and the first point of contact. Even with a 1Gb link shouldn't I be looking at, in an ideal world around 126MB/s transfer rates? I know I would never achieve that but something higher than 20MB/s would be better. When you factor in that the link is 10 times faster shouldn't it be performing more than that with a theoretical 1.26GB/s? (again I know that the hardware either side will not get near that at the moment but again I'm expecting more than 20MB/s).

A single iperf from one host to one destination over the 10Gb link shows a rate of 150 to 160Mb/s.

M.

m4cc45 · 12 Aug 2013 at 18:05

P.S. I should also say that this happens on Linux and hardware based devices.

M.

m4cc45 · 12 Aug 2013 at 20:01

I'll paste tomorrow when I'm in work but when we changed it before (up to 512 KB) it didn't make any odds.

We aren't seeing speeds of anywhere near 1Gb. The speed of 160Mb is in Mbs. So again nowhere near even 1Gb. The latency varies between 3 and 4ms. Windows 7 I believe has auto-tuning enabled by default so should be able to grow beyond this. We noticed very little difference between a 96KB window size and everything after it - it seemed to make no odds. Again I'll paste tomorrow so you can get a clear idea.

Just to be clear I'm not after 10Gb. I'm just after all services to be able to have 1Gb available to them - if the service is busy then that's fine but to be able to use as much bandwidth as available is the key.

M.

m4cc45 · 13 Aug 2013 at 09:06

------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 64 KByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-20.0 sec 411 MBytes 172 Mbits/sec

------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 256 KByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-20.0 sec 342 MBytes 143 Mbits/sec

------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 512 KByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-20.0 sec 364 MBytes 152 Mbits/sec

------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 2.00 MByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-20.1 sec 364 MBytes 152 Mbits/sec

------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 20.0 MByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-20.8 sec 440 MBytes 177 Mbits/sec

------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 200.0 MByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-32.1 sec 400 Mbytes 105 Mbits/sec

As you can see from the above changing the window size doesn't really change the speed at all. These are both Windows 2008 R2 boxes and the other end was changed in parallel so both ends match.

Standard:

------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 8.00 KByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[248] 0.0-20.0 sec 184 MBytes 43.7 Mbits/sec

(This allows me to believe that SMB must be doing something to get the 20MB/s that we currently see as that would give us a transfer speed of around 4MB/s)