Aaron’s Worthless Words

EtherChannel lets you aggregate links into one logical connection, but the distribution of traffic is not uniform.  It does not use per-packet load-balancing or the like to determine what interface in the bundle to use.  Instead, it uses a XOR function on packet information to generate a hash that is used to determine what interface to use.

By default, the switch will use both the source and destination IP addresses to generate the hash, but there are lots of others.

• src-ip:  Just the source IP
• dst-ip:  Just the destination IP
• src-dst-ip:  Both the source and destination IPs
• src-mac:  Just the source MAC
• dst-mac:  Just the destination MAC
• src-dst-mac:  The source and destination MACs
• src-port:  Just the source TCP/UDP port
• dst-port:  Just the destination TCP/UPD port
• src-dst-port:  The source and destination TCP/UDP ports

You change the method by giving the port-channel load-balance method directive in global config mode.  Notice that this is a system command, so any change affects all channel groups.

To generate the hash, the switch takes the binary representation of the address(es) and does a XOR (that is, are they different? yes/no) on the last few bits of each to get a value.  If you have 8 interfaces, it uses the last 3 bits (2^3 = 8).  If you have 4 interfaces, it uses 2 bits.  Two interfaces means a single bit.

What we wind up with is an index of the interface to use.  If you do a show etherchannel detail on your switch, you’ll see each interface is assigned an index that starts with 0.

Let’s go through an example.  You have a switch that has F0/1(index 0) and F0/2 (index 1) in Po10.  You also have left the load-balancing method to the default of src-dst-ip.  The switch needs to forward a packet that is sourced from 10.0.0.1 and destined to 10.0.0.2 over Po10.  Let’s step through it.

10.0.0.1 = 00001010 00000000 00000000 00000001
10.0.0.2 = 00001010 00000000 00000000 00000010

Since we have two interfaces in the channel group, we look at the last bit of each address and XOR them to get an index of 1 (1 XOR 0 = 1).  That’s F0/2, so that interface will be used to send the frame over.

If we add f0/3 and f0/4 to the channel group, we would calculate a XOR on the last 2 bits of each address, which would give us 11, which is 3 in binary.  The interface with an index of 3 (probably f0/4) would get the traffic.

What if I’m switching IPX packets?  Any non-IP packet will default to using the MAC address.  Can someone answer exactly what method it will be used?

What if we then add f0/5?  That’s a good question, but I’m not exactly sure how the switch handles a number of interfaces not a power of 2.  Can someone help on this, too?

Send any obvious corrections and questions my way.

I promise I’m still here.  It’s just vacation time, and I’ve been slacking.  On top of that, I’m doing some work-travelling this week, and that really puts a damper on your schedule.

I’ll try my darndest to get some new stuff up this week.  I know there are people rolling around on the floor in the fetal position waiting for a new post, so (get some mental help and) keep your head up for just a little while longer.

I’ve decided to take on the CCNP certification, so I’m going to wind up with a few posts will be more my own notes than anything. 

A switch port on a 2960 comes up with a default configuration on VLAN 1.  What happens from the perspective of spanning-tree?

• First, the port comes up on blocking mode.  This is to make sure that loops aren’t created without first listening to the network to see what’s going on.
• Next, if the port may be a root or designated port, the port is moved to the listening state.  In this state, the port can send and receives BPDUs only.  It can’t send traffic, but it can discover the other switches participating in STP.
• After the forwarding delay, the port goes into the learning state.   In this state, the port can send and receive BPDUs as in listening, but it can now receive traffic.  It can’t yet send any.
• After the forwarding delay again, the port goes into the forwarding state.  The port can now send and receive data.

If the port is configured with spanning-tree portfast, the mode goes from blocking directly to forwarding without going through these steps.  Obviously you don’t want a switch plugged into a port configured for portfast since you may wind up with a loop.

Here’s the debug spanning-tree events output from one of my labs.  F0/3 is configured for portfast.  I shut/no shut it to see what happens.

*Mar  8 18:09:51.163: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to down
sw01#
*Mar  8 18:09:51.747: set portid: VLAN0007 Fa0/3: new port id 8003
*Mar  8 18:09:51.747: STP: VLAN0007 Fa0/3 ->jump to forwarding from blocking
sw01#
*Mar  8 18:09:53.739: %LINK-3-UPDOWN: Interface FastEthernet0/3, changed state to up
*Mar  8 18:09:54.739: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to up

Notice the “jump to forwarding from blocking”.

Here’s the same output when the port is not in portfast mode.  Notice the timestamps.  It takes about 30 seconds (2 x default foward delay) to go from blocking to listening to learning to forwarding.

*Mar  8 18:13:05.313: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to down
sw01#
*Mar  8 18:13:06.013: set portid: VLAN0007 Fa0/3: new port id 8003
*Mar  8 18:13:06.013: STP: VLAN0007 Fa0/3 -> listening
sw01#
*Mar  8 18:13:06.381: %LINK-3-UPDOWN: Interface FastEthernet0/3, changed state to up
*Mar  8 18:13:07.381: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to up
sw01#
*Mar  8 18:13:21.013: STP: VLAN0007 Fa0/3 -> learning
sw01#
*Mar  8 18:13:36.013: STP: VLAN0007 Fa0/3 -> forwarding

Send any obvious corrections and questions my way.

I’ve got a non-technical one for you today.  If you’re paying attention to stuff around you, you’ll probably end up with a little paranoia after reading this.

We’re having another circuit installed, and the LEC came out to do their end-to-end testing.  The tech, Dan, calls me up on the phone and tells me who he was and what he needed to do; I agree to meet him in the lobby to escort him on his way.  Now, I’ve never met Dan and can’t really vouch for him.  He had the polo shirt and khakis that we all come to expect.  He had a pile of generic-looking badges on his belt with his picture and name on them.  He had a satchel full of fulls and equipment.  He looked the part, but how hard is it to get a shirt, print up & laminate a few badges, and put some tools in a bag?  Was Dan really who he said he was?  Should I really have let Dan in the telco room?

In this case, I would say Dan was legitimate; he called the right phone number and mentioned the correct circuit we were installing, but I cannot say beyond a shadow of a doubt that he was supposed to be messing with that equipment.

My wife’s in retail, and I asked her if she has any similar stories.  She had quite a few, actually, usually involving the building’s security.  Her store has security guards come in and out from time to time, and it’s always a different person.  They never identify themselves to anyone in the store, but their decked out in the shirt we all come to expect.  Around here, it’s illegal identify yourself as the police if you’re not, and that includes patches and badges.  You can, however, go to the local store and buy security patches and maybe even a badge — now the outfit is complete.  How can employees in the store be sure that the guy with the security patches is really who he says he is?  Will people even question his being there?

People are known to be trusting.  That’s just how people are, and there’s nothing you can do about it.  We assume that people who say they are an authority are that authority, which is a bad thing if you’re trying to be secure.  A coworker on the secuirty side loves to tell the story of the KFC in Manchester, New Hampshire, where a highly-skilled social engineer phoned in, told the employees that he was with the corporate office, and had them doing all sorts of things.  Let’s just say it ended with them all naked in the snow in the parking lot, urinating on each other, and lighting their clothes on fire — all just because someone on the phone told them to do so.  What would people do if you actually showed up looking the part?

The next time a vendor shows up, I think I’ll ask him to prove who he is just to see how he or she reacts.

SSH is more than just a shell.  You can copy files from and to a server or piece of network gear with it.  You can use it to tunnel traffic.  Possibly my favorite, though, is to use SSH to run a command on a remote box without interacting with a shell.

One of my biggest pet peeves with IOS (or pretty much any Cisco OS) is the lack of complex filtering.  Let’s say I want to look at all the downed ports and interfaces on modules 3 and 6 of my 6509.  I can’t easily do that with command from the IOS, but, on my Linux box, I can use multiple grep commands to get exactly what I want really easily.  Let’s work through the example, shall we?

To start with, let’s just do a show ip int brief without getting a shell on the switch.

ssh my.switch.com "show ip int brief"

When you run this and give your password, you see the output we’ve all learned to love, and, now that you’ve got it in STDOUT on your Linux box, you can start filtering. Now, let’s use grep to find the downed ports and interfaces on modules 3 and 6.

ssh my.switch.com "show ip int brief" | grep down | grep Ethernet[36]

How about downed ports and interfaces on modules 3 and 6 that not administratively down?

ssh my.switch.com "show ip int brief" | grep down | grep Ethernet[36] | grep -v admin

I’ll stop there, but it can go on and on.  Read up on regular expression and/or grep if you don’t know what we’re doing here.

What’s really happening is that we’re taking the output of the command “ssh ….” and piping it (with |) to the command grep.  We can send it to whatever command we want, though, so don’t be shy.  I’ve actually written several scripts that take output of commands like show int description on a router to generate some reports.  When I want to run one of those, I do something like this.

ssh my.switch.com "show int desc" | parseOutput.pl

There’s always a gotcha or two to watch for, isn’t there?  I’ve found a couple.

First, your command runs at your privilege level, so, if your user is priv 1, you’re not going to be able to do a show run or reload.  You could just ignore security for a bit and set your privilege to 15, but I don’t recommend doing anything like that.  Before you say it, you’ll probably have a hard time with enabling as well.  You can only run one command at a time, so you would just enable yourself and get kicked off.  Not very helpful.

Another problem I see is the lack of public/private key pair support on Cisco devices.  On a Linux box, you can copy your keys around, and those are presented in lieu of a password.  Since (most) Cisco devices don’t have home directories, there’s no place to drop the keys, and we’re left with just using passwords.  Support for this would be nice, but the security problems associated with keep SSH keys and user home directories are probably too much to even think about.

What else?  Oh, yeah.  The PIX/FWSM/ASA family supports SSH, but it acts differently from the IOS guys.  When you run a command through SSH, you actually get an interactive shell with the command already on the CLI for you. This is probably by design; the only thing you can really do from a non-priv prompt is to enable.

Anyway, send any grilling tips questions my way.

This is a great one that I have to share.

A couple of coworkers walk in today and ask for some help on an issue.  It seems that a business unit was having latency problems with a web app, and, after research by the product team and sysadmins, nothing wrong could be found.  Lots of sites use the product, and only this one was having issues.  Also, the site was having no problems getting to other web sites and apps like Yahoo! or Google.

I informed the guys that their Internet access went over a T1 to an ISP, but, since the application was housed at our place, access to it was actually over the WAN circuit.  The guys told me that the unit had just called to complain about it again, so we checked out their WAN circuit.  Eureka!  The circuit was at 98% utilization.  There’s your cause.  The app is slow because the circuit is full.  Looking back through history, we see weeks of high utilization that explains why the users are having issues with the app.

Of course, the next logical step is to find out what’s talking so much.  I pulled up our netflow box to see what conversations were the big talkers.  I found a single IP at the site talking to a single IP at our home office via HTTP and HTTPs that accounted for 40% or so of the total bandwidth usage in the last 24 hours.  And the previous 24 hours.  And the previous 24 hours.  Hmmmm.

I didn’t recognize the IPs, so I asked the systems guys what they were.  They had no clue, and neither IP was reserved in our IP management system.  Quite strange.  I ran NMap against both boxes; that fine app told me that they were both printers.  Printers?  That makes no sense.  I noticed that HTTP was open (we saw that in netflow, too), so we pointed a browser at the one at the home office to see what turns up.  It’s the security system.  That makes even less sense.  Pointing a browser at the other IP, we find that it’s a security camera that’s looking at some server racks.  Great!  Somebody’s streaming video over the WAN.

We march upstairs to talk to the security guys.  They’re not streaming directly but pull up their app and notice that the camera in question has been sending 10 seconds of video every few minutes for the past 6 weeks.  The cameras only send video if their detect movement in the room, so what’s going on?  You can expect several movement events in a “computer room” as they call it, but people go home at night, and you shouldn’t see movies coming across for weeks at a time.  We pull up a few clips to see what the camera saw.

We see people going in.  We see people going out.  We see the rack doors open.  We see the usual stuff, but most of the videos are nothing.  Ten seconds of no change.  Finally, though, we see it.  Something stupid.  Someone had taped a piece of paper to one of the racks, and that guy was moving ever so slightly.  I couldn’t even really detect the paper itself moving; I could only see the text on the page move just a tad.  The camera saw it, though, and was doing its job and recording back to the home office.  A piece of paper took up 40% of the WAN circuit?  It’s always something new, eh?

I really hope they get our message to take down the paper.

Have you even noticed that your new servers all have 2 NICs on the board?  At least all of them that I’ve seen in the last 3 years have.  A lot of server admin actually use them in a NIC teaming scenario where both NICs are used as one logical device — much the same as Etherchannel on a switch.  This provides some fault tolerance and availability in case of failure, which is good idea in most cases.

There are a few different ways to configure teaming on the box (usually called bonding in Linux), and each has its own advantages and disadvantages.  The network dude(tte) may have to do some things on the switch side for some of them to work, though.  If you’re want to run in link aggregation mode (mode 4), for example, the switch ports need to be in the same channel group to work appropriately.

Let’s look at mode 4 a little closer to see what we need to do.  The scenario is that you have eth0 plugged into F0/15 of a 2950 and eth1 is in F0/16.  You’ve seen the configuration for channelling between switches before, so you know the basics.  Put the ports in the same channel-group and configure the proper Port-channel interface to do the work.  In this case, we’re just configuring the ports to house a host instead of being trunks.

int F0/15
 channel-group 1

int F0/16
 channel-group 1

int Port-channel 1
 speed 100
 duplex full
 switchport
 switchport mode access

I detect at least one problem with our setup, though.  Both NICs are plugged into the same switch; what happens when the switch goes down?  The server goes away.  Logic should tell you, then, to put the NICs in different switches to fix that, but you can’t do Ethernchannel on two different switches.   The ports have to be in the same device for the aggregation to work.  What’s the fix?

You can look at getting a nice chassis switch and putting each NIC in different modules.  Modern IOS versions allow etherchanneling across modules, so, if one module fails, you still have that other.  That would do it, but I’m sure you don’t have the money for a 4500 in the budget, right?

Another solution is to use a couple 3760s which, when connected using the StackWise cable, are one logical device.  That gives you two separate switches that you can configure with the same channel group.  An upgrade to this solution is to use a pair of 6500s with VSS 1440 modules in them so that you have a stack of 6500s!  I’m sure that’s not expensive at all, though.

Send any white shoes questions my way.

We talked about running multiple data centers on a stick back in August, which is where you have multiple logical pairs of client and server VLANs on a single CSM for different tiers or functions.  The big point of the article was that you had to do some fancy forwarding to get a server-initiated connection from one server VLAN to appear out the appropriate client VLAN.  Well, we ran into an interesting issue with the given solution.

Let’s set up a scenario.  Check the diagram for an overview.  We have many pairs of client and server VLANs each with a firewall interface as the gateway into the DCOAS.  Let’s just focus on just one, though — client VLAN 101 and server VLAN 102.  In VLAN 101 is ServerA (not pictured) with an IP of 1.1.101.45; in VLAN 102 is our web farm that needs to connect to ServerA to drop off some data.  We add a static route on ServerA pointing traffic for 1.1.102.0/24 back through the CSM.

When you try to connect from the web farm, though, it just times out.  Why is that?

Remember that weird forwarding vserver that we had to use to get traffic to come out of the right client VLAN?  Well, that’s stabbing you in the eye right now.  When the web server initiates a connection, it sends traffic to the server VLAN IP of the CSM.  The forwarding vserver grabs the new connection and load balances it to its only RIP, which is the IP of the firewall.  What happens when any good firewall accepts traffic destined on an interface destined for a host out of the same interface?  It drops the packet, and, eventually, the server times out.

What’s the fix, then?  There are a few that come to mind.  The first may be to just move ServerA to another network segment.  Another may be to change the process around a bit by having ServerA pull the data instead of it being pushed since client-initiated connections will work like a champ.

A really outrageous one would be to set up another forwarding vserver that has only ServerA as it’s serverfarm.  You would then add a static route in the web servers pointing to ServerA through that VIP, which would foward it over.

On the CSM, you’d do something like this.

serverfarm SERVERA-SF
 no nat server
 no nat client
 real 1.1.101.45  <--- ServerA
  inservice

vserver SERVERA-VS
 virtual 1.1.102.5 any
 vlan 102
 serverfarm SERVERA-SF
 inservice

On the server, you would add a static route to ServerA through 1.1.102.5. If you’re using some brand of Linux, you’d do this.

route add 1.1.101.45 gw 1.1.102.5

Don’t forget the static route on ServerA.

Send any Peeps questions my way.

We discussed SPANs earlier, but let’s talk about RSPANs for a bit.

Can anyone guess what the “R” means?  You guessed it — “Remote”.  An RSPAN is a way to get traffic from a SPAN source on one switch to a SPAN destination on another switch that’s connected via a trunk.

The basic premise is that a special VLAN is created on all the switches and allowed to traverse the trunks.  You then set up a SPAN session that copies your traffic to this special VLAN.  This VLAN then gets the traffic to the other switches through some voodoo magic to be used as source for a SPAN on another switch.

Let’s work through the steps.  In our example, we want to copy traffic from G2/18 on SwitchA to G3/38 on SwitchB.

First, on both switches, we need to create the new RSPAN VLAN.  We’ll assume you’ve already got it set up to allow this VLAN over your trunks.

vlan 2000
 name RSPAN
 remote-span

Notice the nice keyword remote-span.  This designates the VLAN to be used in an RSPAN.  Easy so far.

Now, let’s create the session to copy traffic to the RSPAN.  The source port is G2/18, and the destination is the RSPAN VLAN.

switchA(config)# monitor session 1 source interface Gi 2/18
switchA(config)# monitor session 1 destination remote vlan 2000

Now the traffic is being copied to the RSPAN, so let’s copy that traffic from the RSPAN to the sniffer.  In this case, the source is the RSPAN, and the destination is the sniffer’s port.  Let’s use session 8 to avoid confusion.

switchB(config)# monitor session 8 source remote vlan 2000
switchB(config)# monitor session 8 destination interface Gi 3/38

There are always things to look out for, aren’t there?  The first that comes to mind is the fact that you’re copying traffic from a port onto one or more trunks.  If the port is sending enough traffic and your trunk is close to capacity, you may wind up crushing the trunk link.  That would suck.

If you have a fully-meshed switch environment, you’ll see the additional traffic across all your trunks if you’re set up that way.  If you have four trunks that transport all VLANs, you may have four copies of the data coming out of the switch.  Let’s say the box being monitored is compromised and sending out 600Mbps of data.  That means that every switch will have to deal with that much traffic.  This sounds to me like a CPU/memory issue waiting to happen.

Don’t expect RSPANs to work on your 2950 like this.  On the lower-end switches, you have to use a reflector port to copy the traffic to the RSPAN.  I don’t get into that here, but Google is your friend.

Send any Cadbury Creme Eggs questions my way.

I can’t believe I haven’t blogged on this yet.  SPANs are one of my favorite things in the world.

The switched port analyzer (SPAN) is a mechanism on Cisco switches that allows you to take traffic on one port and copy it to another.  It’s generally used to get traffic to a sniffer or IDS for analysis, but it’s a great tool to use to sample traffic from a host for troubleshooting.

Let’s use a real-world example.  You’ve told your roommate to quit illegally downloading songs off the Internet, but you suspect he’s still doing it.  Instead of sneaking into his room at night and checking his machine, you can use a SPAN to copy his traffic to another switch interface where you can use tcpdump to record what’s happening.

Let’s say you have a 2950, and the roomie is on F0/1.  You have a Linux box plugged into F0/24 ready to capture the traffic.  Here’s what you do.

monitor session 1 source interface F0/1 both
monitor session 1 destination interface F0/24

This will create a new monitor session (that is, a SPAN session) that copies traffic from port F0/1 in both directions to port F0/24.  Now, when you run tcpdump on your Linux box, you see all the traffic coming in and going out of your roommate’s port.

That’s pretty easy, right?  You can have multiple sources ports by just adding more source lines or using ranges of ports.  You can also just copy received or transmitted traffic from a source.  Check out the contextual help for a little more info.

To see what’s going on, you can do a show monitor or a show monitor session 1 (depending on the IOS version).  You’ll see something like this.

switch#sh monitor
Session 1
---------
Type              : Local Session
Source Ports      :
Both          : Fa0/1
Destination Ports : Fa0/24
Encapsulation : Native
Ingress : Disabled

If you take a look at the destination port when the SPAN is running, you’ll see it’s in a state of up/down (monitoring).  I think you can figure out that this means we’re monitoring some traffic to this port.  Here’s what you’ll see if you look at the port.

switch#sh int f0/24
FastEthernet0/24 is up, line protocol is down (monitoring)
...

There are two big things to keep in mind when doing SPANs.  The first is that monitoring a port can drive CPU utilization way up (depending on the platform and traffic volume), so you may run into problems if you have a bunch of SPANs going at the same time.  Related to this is the fact that, if your switch has to decide between switching and copying traffic, it will stop copying until there’s enough CPU headroom to do that safely, and you’ll lose packets in the meantime.  It’s a switch — not a copier.

The second thing to keep in mind involves those little voices in your head called ethics.  What if you see a VOIP phone call from your boss to the HR department?  How about if you find someone in upper management copying a spreadsheet of people to be fired tomorrow?  How about if you find an engineer’s telnet password to a key system?  These are things that you probably shouldn’t see, so be careful when looking at the packets.  I would suggest you tell someone in your security when you’re going to do a packet capture to make sure someone knows you’re not up to no good.

Send shamrocks questions my way.