Aaron's Worthless Words

I was doing some STP labs tonight and found something that caught me off guard a bit.  I had been meddling with some EtherChannels between a pair of 3750s earlier today, and I forgot to reset the configs before starting on the STP stuff.  One my secondary root switch, I ran a show spanning-tree vlan 1 to see what status the ports were in, and I noticed the root path cost.

VLAN0001
  Spanning tree enabled protocol ieee
  Root ID    Priority    24577
             Address     001b.d4fa.bb00
             Cost        12

This switch is directly connected to the root bridge via a pair of EtherChanneled FastEthernets, so I just assumed I’d get a cost of 19.  I surely didn’t expect a cost of 12.  I added a third interface to the channel-group and wound up with this.

VLAN0001
  Spanning tree enabled protocol ieee
  Root ID    Priority    24577
             Address     001b.d4fa.bb00
             Cost        9

Obviously there’s some internal math going on with the EtherChannel and STP.  Guess what happens when I add a fourth link?

VLAN0001
  Spanning tree enabled protocol ieee
  Root ID    Priority    24577
             Address     001b.d4fa.bb00
             Cost        8

It’s interesting to see how the path cost changes in a way to seems disproportionate to the bandwidth.

Send any new math formulae comments this way.

EtherChannel lets you aggregate links into one logical connection, but the distribution of traffic is not uniform.  It does not use per-packet load-balancing or the like to determine what interface in the bundle to use.  Instead, it uses a XOR function on packet information to generate a hash that is used to determine what interface to use.

By default, the switch will use both the source and destination IP addresses to generate the hash, but there are lots of others.

• src-ip:  Just the source IP
• dst-ip:  Just the destination IP
• src-dst-ip:  Both the source and destination IPs
• src-mac:  Just the source MAC
• dst-mac:  Just the destination MAC
• src-dst-mac:  The source and destination MACs
• src-port:  Just the source TCP/UDP port
• dst-port:  Just the destination TCP/UPD port
• src-dst-port:  The source and destination TCP/UDP ports

You change the method by giving the port-channel load-balance method directive in global config mode.  Notice that this is a system command, so any change affects all channel groups.

To generate the hash, the switch takes the binary representation of the address(es) and does a XOR (that is, are they different? yes/no) on the last few bits of each to get a value.  If you have 8 interfaces, it uses the last 3 bits (2^3 = 8).  If you have 4 interfaces, it uses 2 bits.  Two interfaces means a single bit.

What we wind up with is an index of the interface to use.  If you do a show etherchannel detail on your switch, you’ll see each interface is assigned an index that starts with 0.

Let’s go through an example.  You have a switch that has F0/1(index 0) and F0/2 (index 1) in Po10.  You also have left the load-balancing method to the default of src-dst-ip.  The switch needs to forward a packet that is sourced from 10.0.0.1 and destined to 10.0.0.2 over Po10.  Let’s step through it.

10.0.0.1 = 00001010 00000000 00000000 00000001
10.0.0.2 = 00001010 00000000 00000000 00000010

Since we have two interfaces in the channel group, we look at the last bit of each address and XOR them to get an index of 1 (1 XOR 0 = 1).  That’s F0/2, so that interface will be used to send the frame over.

If we add f0/3 and f0/4 to the channel group, we would calculate a XOR on the last 2 bits of each address, which would give us 11, which is 3 in binary.  The interface with an index of 3 (probably f0/4) would get the traffic.

What if I’m switching IPX packets?  Any non-IP packet will default to using the MAC address.  Can someone answer exactly what method it will be used?

What if we then add f0/5?  That’s a good question, but I’m not exactly sure how the switch handles a number of interfaces not a power of 2.  Can someone help on this, too?

Send any obvious corrections and questions my way.

Have you even noticed that your new servers all have 2 NICs on the board?  At least all of them that I’ve seen in the last 3 years have.  A lot of server admin actually use them in a NIC teaming scenario where both NICs are used as one logical device — much the same as Etherchannel on a switch.  This provides some fault tolerance and availability in case of failure, which is good idea in most cases.

There are a few different ways to configure teaming on the box (usually called bonding in Linux), and each has its own advantages and disadvantages.  The network dude(tte) may have to do some things on the switch side for some of them to work, though.  If you’re want to run in link aggregation mode (mode 4), for example, the switch ports need to be in the same channel group to work appropriately.

Let’s look at mode 4 a little closer to see what we need to do.  The scenario is that you have eth0 plugged into F0/15 of a 2950 and eth1 is in F0/16.  You’ve seen the configuration for channelling between switches before, so you know the basics.  Put the ports in the same channel-group and configure the proper Port-channel interface to do the work.  In this case, we’re just configuring the ports to house a host instead of being trunks.

int F0/15
 channel-group 1

int F0/16
 channel-group 1

int Port-channel 1
 speed 100
 duplex full
 switchport
 switchport mode access

I detect at least one problem with our setup, though.  Both NICs are plugged into the same switch; what happens when the switch goes down?  The server goes away.  Logic should tell you, then, to put the NICs in different switches to fix that, but you can’t do Ethernchannel on two different switches.   The ports have to be in the same device for the aggregation to work.  What’s the fix?

You can look at getting a nice chassis switch and putting each NIC in different modules.  Modern IOS versions allow etherchanneling across modules, so, if one module fails, you still have that other.  That would do it, but I’m sure you don’t have the money for a 4500 in the budget, right?

Another solution is to use a couple 3760s which, when connected using the StackWise cable, are one logical device.  That gives you two separate switches that you can configure with the same channel group.  An upgrade to this solution is to use a pair of 6500s with VSS 1440 modules in them so that you have a stack of 6500s!  I’m sure that’s not expensive at all, though.

Send any white shoes questions my way.