Switch Topology

[anything I don't mind]

I am currently trying to setup a new switch topology at one of our clients. This is the first time I have done any topology design with switches and was looking for some tips. We have:

6x HP 2810-48g (2x 1 gigabit uplink gbic)
2x HP 24 port gigabit with 6x gbic ports didn't take down model number (existing core switches)
2x fiber switches

The storage is all configured and was easy enough to set up. Although we did not have to set up any round robin configuration in vmware, we tested by turning off one of the switches and the hp p2000 continued to work ok even with one switch. We only just got the kit so we are just in testing phase at the moment. The switches are stackable but it is not real stackable as they don't have stack ports on the back like cisco or juniper, basically all the stacking does is allow it to be manageable from one IP. They still have to be connected to each other using gigabit uplink or standard ports.

My initial plan is to have 2x core switches and then 2 switch stacks, one on the ground floor and one on the fifth floor.

2x core switches will be 2x 24 port gigabit switches.
4x 48 port switches in one stack in basement
2x 48 port switch in fifth floor

Connect two core switches together using 2x 1 gbit ports and then run a 2x 1 gigabit from each core switch to each basement switch (using lacp?) then for resilience run a 1x gbit link from each switch in the stack to the other two switches in the stack.

Using a total of 8 ports in each core switch and 5 ports in each 48 port switch.

Then run 2x fiber gbic down from commander switch in fifth floor stack to each core switch. Then run a 2x 1 gigabit from the second switch to the main switch in the fifth floor stack.

-Question I have is how can i improve this topology or configuration.
-Where would i use lacp and is that only for use in stackable switches or could i use that for the 2x 1gigabit connections to each switch.
-Should I run a third gbic down from the second switch for resiliance and if so where to connect it, one of the core switches?

 

[oli356]

This thread is missing topology diagrams.

 

[kerrgreg]

Diagram would be handy.


I would avoid connected your basement stack and 5th floor stack, if you have an issue with either it could take down the other where you should be able to implement security on your core so ports can be shutdown to prevent it killing from one point.

Are we to presume your fiber switches are FC fiber and not switching fiber? I presume you know why I ask you this?

 

[anything I don't mind]

Ok I am trying to make a diagram at the moment but visio 2010 is a nightmare, the connectors are retarded.

The FC swich is a HP San switch 24/8

FC switch:
http://h18006.www1.hp.com/products/storageworks/sanswitch824/specs.html
hi res image of fc switch
http://core4solutions.com/media/cat...ab33525d08d6e5fb8d27136e95/a/m/am868a_1_2.jpg

5x 2810-48G switch:
http://h10010.www1.hp.com/wwpc/ca/e...2136298-12136326-12702378-77904477.html?dnr=1

1x ProCurve Switch 2848:
http://h10010.www1.hp.com/wwpc/ca/e...2136298-12136316-12136322-29584733.html?dnr=1

2x ProCurve Switch 2824:
http://h10010.www1.hp.com/wwpc/ca/e...2136298-12136316-12136322-29584735.html?dnr=1

Due to the specs on the 24 port (as it has less bandwidth) I am thinking of using 2x 2810-48g as the core switches and just add the 24 port to the basement stack.

 

[unleashed]

Ok I am trying to make a diagram at the moment but visio 2010 is a nightmare, the connectors are retarded.

Pencil and paper

 

[anything I don't mind]

This is what I think i will be going with, any opinions would be appreciated.

It will be 24 port ESXi and 12 port uplink + 4 port gbic uplink leaving a lot of free ports out of the 96 ports available.

We got about 600 write and read iops on exchange 2010 2 hour performance test in raid 10. We have 300gb 15k sas disk.

Still think it seems crazy to use such disks when we have 1000+ iops SSD that can be bought for relatively low price.

 

[DRZ]

I'll start off by saying that I'm not overly familiar with HP kit but...

What?!

Unless you have the ability to do VSS/MEC-type things, you are creating loops all over the place here and (presumably) relying on Spanning Tree to fix it for you, which can result in sub-optimal traffic flows and other hassle.

From a resiliency point of view it would be much better to either put something proper in as the core of the network or use L3 links instead of L2 links.

Unfortunately, those switches appear to be Layer 2 only, so you completely defeat the point of having a "core" switch in the traditional sense. Do you plan on having different VLANs for your servers/access cabs/voice? If so, how are you planning to route between them? I don't see any L3 devices in that diagram.

Basically, this is a terrible, terrible network design.

What have your EX2010 IOPS reqs got to do with this, btw?

 

[!Sargent_Mango!]

DRZ,

Are you suggesting that if I had 4 core switches, and Trunks of 4Gb links between them that they have to be linked in a straight line rather than having them all connected with the last having a link back to the first to create one large ring ?

 

[jake108]

Agree with DRZ here; albeit a bit more diplomatically. You are creating network loops which is pretty bad in any sense unless you have proper kit (these HP series are L2 as mentioned - so no real control either).

To be more constructive what I recommend you look at is to try applying the KISS principle here as it seems your new to the more core network setup. From there you can "experiment" with more advanced setup topologies.

Without knowing the full set of requirements the best advice I can give you is to have both fibre switches acting the backbone of the design and have the rest of the switches close to where they need to be (5th floor, ground floor) in a spanning tree design from the two core switches (choice of which is yours). If you are connecting clients to switches (seems from your diagram you are) I would recommend either having them acting as one switch (from the specs they do support this) or alternatively having one switch act as the branch and the other two switches plugged into it in seperate ports (rather than creating a loop). Again keeping it simple.

 

[DRZ]

Agree with DRZ here; albeit a bit more diplomatically. You are creating network loops which is pretty bad in any sense unless you have proper kit (these HP series are L2 as mentioned - so no real control either).

To be more constructive what I recommend you look at is to try applying the KISS principle here as it seems your new to the more core network setup. From there you can "experiment" with more advanced setup topologies.

Without knowing the full set of requirements the best advice I can give you is to have both fibre switches acting the backbone of the design and have the rest of the switches close to where they need to be (5th floor, ground floor) in a spanning tree design from the two core switches (choice of which is yours). If you are connecting clients to switches (seems from your diagram you are) I would recommend either having them acting as one switch (from the specs they do support this) or alternatively having one switch act as the branch and the other two switches plugged into it in seperate ports (rather than creating a loop). Again keeping it simple.

Diplomacy is lost on groen

The problem with having those fibre switches as the backbone is that they are FC, not Ethernet switches. Essentailly, they are nothing at all to do with the LAN.

I'll ask again where the L3 functions are being done and what the proposed VLAN topology looks like because that is quite important to the overall design.

If the design is constrained to the hardware listed in this diagram I would probably be forced into using STP to provide redundant links to a pair of switches that are linked together. I don't know how much scope the HP switches have for managing the STP domain but you're going to need to make sure that the STP root is that pair of switches so you have a predictable topology.

I wouldn't bother linking the "access" switches together unless they do actual stacking and can do LACP across the switches. If not, you're probably better off linking each switch to each "core" switch and allowing STP to shut down one of the links. Use LACP if you have the cable count back to the cores because it'll converge faster than STP for a single link failure (so you have two links per LACP port channel, two port channels per switch).

Clearly this uses up a fair number of your ports but I'm guessing these switches aren't full/busy or they would be having some pretty noticeable issues by now...

Best practice would have course be to relegate all of those switches to just the access layer, put a stackable pair of L3 switches in (even a couple of 24-port 3750s with IP Base would perform adequately) and have single 2-port LACP trunks from each switch. Keep the broadcast domains down by using one VLAN per switch and reduce the IP usage by using /26 (or /25 if you have switches off switches...) per switch to avoid flooding the uplinks with chatty windows traffic.

DRZ,

Are you suggesting that if I had 4 core switches, and Trunks of 4Gb links between them that they have to be linked in a straight line rather than having them all connected with the last having a link back to the first to create one large ring ?

By "in a line" do you mean effectively daisy-chaining them? That's going to be sub-optimal from a contention point of view as things go up the line. Having a loop is "ok" providing you know what is going on with Spanning Tree but the design is certainly sub-optimal overall. Where possible, you should try and eliminate reliance on STP because doing so often ends up with a better topology in almost all cases.

For example, a stack of 2960s or 3750s will allow you to do Multi-Chassis Etherchannel across the stack - so you can have one LACP trunk with a link in each stack member going back to your core switching. For me that's a VDC on a Nexus 7000 acting as an aggregation layer but it could just as easily be a pair of 3750s or 45/6500s with VSS etc depending on the scale of your site (and collapsing core into the aggregation layer if that's applicable). The advantages of that are subsecond convergence in the event of any single switch failure and no chance of STP ruining your day (or burning CPU cycles recalculating the tree). Stacking like that also reduces points of management to one per cabinet rather than one per stack and reduces the number of high-bandwidth ports required in your core. Disadvantages would be higher contention on your uplinks but you can either scale out your LACP links or scale up the bandwidth. We have 2x10GbE uplinks from each of our stacks which is more than adequate.

Ignoring the STP stuff hiding in the LACP links etc, you've effectively pushed Spanning Tree all the way out to the access ports, decreased your uplink port requirement and in your case you'd decrease the contention ratio of your clients and make that ratio flat across all ports in your switching estate rather than it being a lottery.

 

[mrbios]

The OPs name is the same as the noise i made when i saw the diagram.....

 

[anything I don't mind]

The fiber channel switches are not the core switches. The FC switches are only for storage traffic between the esxi and the p2000 san.

Where exactly is the loop? Yes i am new to network design but I have asked several people from different locations and no one has said that this design is terrible and will create loops. Sure if we had another 5k to spend we could get some proper core switches to replace the 2x 2810 which are definitely not core switches.

What is the other alterantive exactly, i should have labelled the switches.

The reason I linked (daisy chained) the client switches in the basement with 1gigabit like that was because someone suggested that they should be connected together as well as in to the core. Initially i just had 4xgigabit going to the 2810 core. They do not do real stacking like cisco, ie no shared bandwidth across them, but you can still stack them for management, the reason why the person suggested linking them together, for resilience.

The only vlans in use will be the voip, management/vmotion and dmz.

The only alternative i can think of is to have one stack in the basement, that would be from the image 2x the core and the 3x client switches in one stack and then just connect each switch to every other switch or daisy chain them. But then if one switch goes down it can affect the whole network.

I designed it this way so that we could lose a "core" switch and everything will still work.

What have your EX2010 IOPS reqs got to do with this, btw?

Nothing i just put that in for information purposes.

I wouldn't bother linking the "access" switches together unless they do actual stacking and can do LACP across the switches. If not, you're probably better off linking each switch to each "core" switch and allowing STP to shut down one of the links. Use LACP if you have the cable count back to the cores because it'll converge faster than STP for a single link failure (so you have two links per LACP port channel, two port channels per switch).

Clearly this uses up a fair number of your ports but I'm guessing these switches aren't full/busy or they would be having some pretty noticeable issues by now...

Best practice would have course be to relegate all of those switches to just the access layer, put a stackable pair of L3 switches in (even a couple of 24-port 3750s with IP Base would perform adequately) and have single 2-port LACP trunks from each switch. Keep the broadcast domains down by using one VLAN per switch and reduce the IP usage by using /26 (or /25 if you have switches off switches...) per switch to avoid flooding the uplinks with chatty windows traffic.

Ok thanks I will see if we can get some cisco 3750. I will remove the 1gigabit link between the client switches and have 1x 2gigabit lacp to each core switch. I will take a look at seeing if i can specify the STP root on the 2810.


I was talking to some vmware guys and they suggested that i should do 2x port per core switch for servers and then 1x per core switch for management and vmotion on a vlan. I will see if our license can do a vds but i think it will have to be a vswitch. Instead of having 6 nic for servers and a separate physical switch for vmotion.

 

[DRZ]

If you can draw a path from one switch back to the same switch via another switch via any path, you have a loop.

There are a lot of things that are fundamentally lacking from your understanding which is making this tougher for you than it might otherwise be. Can I respectfully suggest that you have a look at some CCNA-level literature on this sort of thing? It will help you immensely to understand why your initial proposal was so flawed

 

[bigredshark]

If you can draw a path from one switch back to the same switch via another switch via any path, you have a loop.

There are a lot of things that are fundamentally lacking from your understanding which is making this tougher for you than it might otherwise be. Can I respectfully suggest that you have a look at some CCNA-level literature on this sort of thing? It will help you immensely to understand why your initial proposal was so flawed

+1

I'd suggest if you can't see how there are loops on that diagram and don't have a basic understanding of what spanning tree is and does (as a protocol it's an outdated and horrible bit of junk but for that kit and that topology it's all you've got) then you should seriously consider paying some money to somebody who does to at least advise you on why this won't work.

But as a primer - you cannot form topologies where traffic can go around in a circle because broadcast traffic will go round in a circle very fast until all your switches melt and your network ceases working.

You need a protocol to shutdown one of the links so there isn't a loop (which means spanning tree on cheap kit, awful as spanning tree is, or something like Cisco flexlinks or your vendors equivalent on more expensive gear) or turn the loop into another topology which still offers redundancy (ie LACP into a *proper* stack - that is Cisco 2960S/3750 Juniper EX4200 etc - or MCLAG if you have kit which supports it).

Or joining the 21st century and stopping trying to make everything layer 2, routing really isn't expensive any more. But that a long shot, nobody really seems to get how much easier it would make life...

 

[anything I don't mind]

I didn't buy the switches, my boss did and he is not very good in my opinion. But now we have five of those hp 2810. I spoke to another guy at work from a different site, who i think knows his switching/storage/vmware quite well. He said that the switch topology won't work too well for various reasons. He suggested just making one stack with the 2810 and put everything in to that, with standard up links.

He also said that i had to use lacp on the ports on the switches that are used by esxi or the esxi networking won't perform. But from what I am aware you can't do lacp without a vds and a enterprise plus license. I also spoke to some vmware people about it and they denied that having to use lacp for more than one physical nic on a vswitch was a requirement. But he seemed pretty sure of himself.

I think I will suggest to my boss that we get a network specialist in just to make sure it is right first time, to be honest it is realy him who should be doing the switching. I don't have the luxury of being able to focus just on switches or just on x. I am expected to do everything from desktops to all servers technologies. But I thought i would make a diagram and see what i would learn and I have learned a fair amount.

 

[nutcase]

I'm learning quite a bit from this thread too

Like you, it shouldn't be me that is sorting out parts of the network (much smaller than yours) but I still seem to get volunteered!

We are going for the kiss approach.

 

[DRZ]

I didn't buy the switches, my boss did and he is not very good in my opinion. But now we have five of those hp 2810. I spoke to another guy at work from a different site, who i think knows his switching/storage/vmware quite well. He said that the switch topology won't work too well for various reasons. He suggested just making one stack with the 2810 and put everything in to that, with standard up links.

Well, that is an option although they don't stack like "proper" switches. Be careful!

He also said that i had to use lacp on the ports on the switches that are used by esxi or the esxi networking won't perform. But from what I am aware you can't do lacp without a vds and a enterprise plus license. I also spoke to some vmware people about it and they denied that having to use lacp for more than one physical nic on a vswitch was a requirement. But he seemed pretty sure of himself.

LACP is not natively supported in VMWare. A Distributed vSwitch (dvs) won't help you either. You need Nexus 1000v to do LACP.

What you can do is EtherChannel which is a similar idea. To do this, you'll need to set all the nics to be active and then turn on IP Hashing in the vSwitch configuration. Be careful if your management interface is on this vSwitch, by default it doesn't inherit the vSwitch configuration, so be sure to go in and untick all the boxes first!

You can have active/standby with the NICs as well, but why would you do that if you can have active/active?

I think I will suggest to my boss that we get a network specialist in just to make sure it is right first time, to be honest it is realy him who should be doing the switching. I don't have the luxury of being able to focus just on switches or just on x. I am expected to do everything from desktops to all servers technologies. But I thought i would make a diagram and see what i would learn and I have learned a fair amount.

Sounds like a very poor outfit to me.

 

[anything I don't mind]

You can do lacp on a vmware distributed switch as i confirmed with vmware guys and in one of my train signal vmware courses that covers vds they do say that lacp is supported on it. But that is irrelevant as we don't have the license for it anyway.

I will take a look at the etherchannel.


Poor outfit? I blame my boss because he should be a senior technical manager but i don't think he is technical enough. He used to do my job (my boss) and then the guy who was the senior technical manager left and he took that job but i don't think he is technical enough. I am fairly new to the third line role and it does not help that my technical boss is not very technical. But i wouldn't class it as a poor outfit, we do well at many other areas, but when it comes to switching that should be the senior technical manager responsibility but he is not up to it. But irrelevant politics but thought i would add it, for clarity.

While i'm in the position i thought i would give it a go (the switching) but after hearing a lot of feedback from various sources, I think it is bit of out scope of my ability at current and will try and pass it on my boss who should be doing it anyway or get a third party in to do it.

Thanks for your help though.