ISL Speed Issues

m4cc45

m4cc45

m4cc45

Soldato
Joined
10 Mar 2003
Posts
6,790
Hi,

This is a weird one.

We have a 10Gb inter-site link with around 4ms of latency.

Now we are experiencing issues that the fastest transfer from either end is 20MB/s. If we have 4 servers they all run at 20MB/s and so on.

Initially the network support team suggests that this should be the fastest we can get - initially they said per TCP stream and now it's per server. I believe it should be per TCP stream and that we should be seeing far faster speeds over the ISL.

Opinions?



M.
 
Much more information required! What kit on each end, what technology in the middle (copper, fibre, Ethernet, dwdm, distance covered), source and destination hardware etc etc would probably be needed before you can get halfway to a decent answer.
 
It's a nightmare getting any information out of them. So the below is what I know to be correct apart from the connection.

MPLS connection (assuming fibre)
Checkpoint firewalls at each end
Target devices are servers (ranging from DL380 G5s onwards)



M.
 
It would be strange to firewall an MPLS circuit unless you were super super paranoid...

It could well be some errant QoS policy although there would have to have been some dedication to setting that up. More likely there is an errant link config somewhere...
 
Super paranoid is right - it actually terminates onto a BT owned unit then firewall but you get the idea.

I think it's QoS / bad routing / bad config or bad cable but I don't think it's bad windows which is what it seems to be blamed on (tcp stack at the moment)



M.
 
Download iperf and understand what the parameters are. The key thing is the TCP Window size, as it determines how many packets are allowed to be "in transit" without acknowledgement packets. The higher the latency link, the larger the window needs to be, since more packets can fit in the longer pipe.

Using iperf means you get fine control over the parameters, and you can rule out most other variables. This should reveal the true nature of your ISL. Try the following:

Server (this will just sit there listening for inbound connections from the client)
iperf -s -f m -w 256K -l 256K

Client (first send, then receive)
iperf -c servername -f m -t 20 -w 256K -l 256K -r

Client (send and receive simultaneously - test the duplex capability of the link)
iperf -c servername -f m -t 20 -w 256K -l 256K -d
 
When we go above 64k it doesn't really make any difference - between source and destination I can see 160Mb (roughly cant remember exactly) connections available (giving us the 20MB/s) but this gets magnified out (i.e. from a single source I can do 3 / 4 connections (then disk contention comes in) and get 20MB/s to each destination. I can do 3/4 simulatenous iperfs and get the same results to different destinations.

This is all over the same ISL.

Just trying to think of something that would allow me to have 80MB/s bandwidth (though divided so it's 4 x 20MB/s on a 1Gb link - I know I can't access all 10Gb of it due to the 1Gb link).

Changing the window size doesn't really matter and I believe SMB 2.1 (Windows 7 / 8) uses multiple TCP streams so wouldn't this already max it out?

Basically what I'm saying is I can see bandwidth available but can't access even though multiple connections from the source to multiple destinations allows me more.

M.
 
Mate, your wording is very confusing. Bottom line, if it is a 1 Gbps ISL, and every piece of equipment end-to-end supports 1 Gbps, you should be able to have a single iperf running at very close to 1 Gbps.

SMB 2.1 (in Windows 7 and 2008 R2) doesn't have multiple streams. SMB 3.0 (in Windows 8 and Windows 2012) introduces SMB Multichannel.
 
Apologies. Link is 10Gb. The equipment supports 1Gb inbetween (firewall to switches) in some places this has several uplinks but I'm not sure between the MTU and the first point of contact. Even with a 1Gb link shouldn't I be looking at, in an ideal world around 126MB/s transfer rates? I know I would never achieve that but something higher than 20MB/s would be better. When you factor in that the link is 10 times faster shouldn't it be performing more than that with a theoretical 1.26GB/s? (again I know that the hardware either side will not get near that at the moment but again I'm expecting more than 20MB/s).

A single iperf from one host to one destination over the 10Gb link shows a rate of 150 to 160Mb/s.


M.
 
I still don't understand how this is setup, and how you are testing.

You mentioned in the original post that you had 4 ms latency -- this is KEY.

Required TCP buffer to reach 1000 Mbps with round-trip-time of 4.0 ms >= 512.0 KB (half a MByte)
Maximum throughput with a TCP window of 64 KByte and round-trip-time of 4.0 ms = 128.00 Mbit/sec

These figures are for a single stream. So if you aren't enjoying the benefit of TCP auto-tuning (where the buffer size increases automatically to cater for faster but higher-latency links), then any given TCP stream will max out at 128 Mbps. This is why I suggested iperf testing, because it allows you to set any arbitrary TCP window size. On a 10 Gbps link you would need 10 Gig cards on the iperf source and destination hosts, and obviously 10 Gbps links all the way through.

You state that a single iperf gives you 150-160 Mbps, which is close-ish to the quoted 128 above. This may be because your RTT is actually less than 4 ms. Try iperf with a bigger TCP window (512 kB should get you very close to 1 Gbps).

Copy and paste the results so we can see.


* The terms "window" and "buffer" are the same thing
 
I'll paste tomorrow when I'm in work but when we changed it before (up to 512 KB) it didn't make any odds.

We aren't seeing speeds of anywhere near 1Gb. The speed of 160Mb is in Mbs. So again nowhere near even 1Gb. The latency varies between 3 and 4ms. Windows 7 I believe has auto-tuning enabled by default so should be able to grow beyond this. We noticed very little difference between a 96KB window size and everything after it - it seemed to make no odds. Again I'll paste tomorrow so you can get a clear idea.

Just to be clear I'm not after 10Gb. I'm just after all services to be able to have 1Gb available to them - if the service is busy then that's fine but to be able to use as much bandwidth as available is the key.



M.
 
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 64 KByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-20.0 sec 411 MBytes 172 Mbits/sec

------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 256 KByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-20.0 sec 342 MBytes 143 Mbits/sec


------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 512 KByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-20.0 sec 364 MBytes 152 Mbits/sec


------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 2.00 MByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-20.1 sec 364 MBytes 152 Mbits/sec


------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 20.0 MByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-20.8 sec 440 MBytes 177 Mbits/sec



------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 200.0 MByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-32.1 sec 400 Mbytes 105 Mbits/sec


As you can see from the above changing the window size doesn't really change the speed at all. These are both Windows 2008 R2 boxes and the other end was changed in parallel so both ends match.

Standard:

------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 8.00 KByte
------------------------------------------------------------
[180] local 192.168.0.1 port 52956 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[248] 0.0-20.0 sec 184 MBytes 43.7 Mbits/sec

(This allows me to believe that SMB must be doing something to get the 20MB/s that we currently see as that would give us a transfer speed of around 4MB/s)
 
There must be a device in the chain that is throttling, i.e. it just can't handle the throughput.

If SMB is giving you 20 MB/s (megabytes per second), is that not very close to what you are getting with iperf (you managed 177 Mbps)? 20 MBps = 160 Mbps (roughly, without accounting for overhead).

Step 1: Connect the two Windows 2008 R2 servers together via a crossover cable and run iperf. You should get extremely close to 1000 Mbps.

Step 2: Connect the same two Windows 2008 R2 servers via a common GigE switch, and then run the exact same iperf test. You should get results very similar to those in step 1.

Step 3: Repeat, adding each new piece of networking in the chain, until you determine where the problem is.
 
I'm inclined to agree with rotor here but something doesn't quite tally with a device being out of horsepower and that is that the rate is capped seemingly per-flow. You would expect a CPU-bound router or firewall to be operating at its maximum on an individual session to not have the grunt to establish and maintain additional sessions at the same throughput.

That's why I wondered if there was any kind of WAN accelerator type device in the middle as they often have the power and control to influence per-connection optimisation, throttling, QoS etc.
 
You must log in or register to reply here.
Back
Top Bottom