Posts tagged switch
BCMSN Notes — STP States
May 21st
I’ve decided to take on the CCNP certification, so I’m going to wind up with a few posts will be more my own notes than anything. 
A switch port on a 2960 comes up with a default configuration on VLAN 1. What happens from the perspective of spanning-tree?
- First, the port comes up on blocking mode. This is to make sure that loops aren’t created without first listening to the network to see what’s going on.
- Next, if the port may be a root or designated port, the port is moved to the listening state. In this state, the port can send and receives BPDUs only. It can’t send traffic, but it can discover the other switches participating in STP.
- After the forwarding delay, the port goes into the learning state. In this state, the port can send and receive BPDUs as in listening, but it can now receive traffic. It can’t yet send any.
- After the forwarding delay again, the port goes into the forwarding state. The port can now send and receive data.
If the port is configured with spanning-tree portfast, the mode goes from blocking directly to forwarding without going through these steps. Obviously you don’t want a switch plugged into a port configured for portfast since you may wind up with a loop.
Here’s the debug spanning-tree events output from one of my labs. F0/3 is configured for portfast. I shut/no shut it to see what happens.
*Mar 8 18:09:51.163: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to down sw01# *Mar 8 18:09:51.747: set portid: VLAN0007 Fa0/3: new port id 8003 *Mar 8 18:09:51.747: STP: VLAN0007 Fa0/3 ->jump to forwarding from blocking sw01# *Mar 8 18:09:53.739: %LINK-3-UPDOWN: Interface FastEthernet0/3, changed state to up *Mar 8 18:09:54.739: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to up
Notice the “jump to forwarding from blocking”.
Here’s the same output when the port is not in portfast mode. Notice the timestamps. It takes about 30 seconds (2 x default foward delay) to go from blocking to listening to learning to forwarding.
*Mar 8 18:13:05.313: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to down sw01# *Mar 8 18:13:06.013: set portid: VLAN0007 Fa0/3: new port id 8003 *Mar 8 18:13:06.013: STP: VLAN0007 Fa0/3 -> listening sw01# *Mar 8 18:13:06.381: %LINK-3-UPDOWN: Interface FastEthernet0/3, changed state to up *Mar 8 18:13:07.381: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to up sw01# *Mar 8 18:13:21.013: STP: VLAN0007 Fa0/3 -> learning sw01# *Mar 8 18:13:36.013: STP: VLAN0007 Fa0/3 -> forwarding
Send any obvious corrections and questions my way.
Using SSH to Run Commands on a Router or Switch
Apr 30th
SSH is more than just a shell. You can copy files from and to a server or piece of network gear with it. You can use it to tunnel traffic. Possibly my favorite, though, is to use SSH to run a command on a remote box without interacting with a shell.
One of my biggest pet peeves with IOS (or pretty much any Cisco OS) is the lack of complex filtering. Let’s say I want to look at all the downed ports and interfaces on modules 3 and 6 of my 6509. I can’t easily do that with command from the IOS, but, on my Linux box, I can use multiple grep commands to get exactly what I want really easily. Let’s work through the example, shall we?
To start with, let’s just do a show ip int brief without getting a shell on the switch.
ssh my.switch.com "show ip int brief"
When you run this and give your password, you see the output we’ve all learned to love, and, now that you’ve got it in STDOUT on your Linux box, you can start filtering. Now, let’s use grep to find the downed ports and interfaces on modules 3 and 6.
ssh my.switch.com "show ip int brief" | grep down | grep Ethernet[36]
How about downed ports and interfaces on modules 3 and 6 that not administratively down?
ssh my.switch.com "show ip int brief" | grep down | grep Ethernet[36] | grep -v admin
I’ll stop there, but it can go on and on. Read up on regular expression and/or grep if you don’t know what we’re doing here.
What’s really happening is that we’re taking the output of the command “ssh ….” and piping it (with |) to the command grep. We can send it to whatever command we want, though, so don’t be shy. I’ve actually written several scripts that take output of commands like show int description on a router to generate some reports. When I want to run one of those, I do something like this.
ssh my.switch.com "show int desc" | parseOutput.pl
There’s always a gotcha or two to watch for, isn’t there? I’ve found a couple.
First, your command runs at your privilege level, so, if your user is priv 1, you’re not going to be able to do a show run or reload. You could just ignore security for a bit and set your privilege to 15, but I don’t recommend doing anything like that. Before you say it, you’ll probably have a hard time with enabling as well. You can only run one command at a time, so you would just enable yourself and get kicked off. Not very helpful.
Another problem I see is the lack of public/private key pair support on Cisco devices. On a Linux box, you can copy your keys around, and those are presented in lieu of a password. Since (most) Cisco devices don’t have home directories, there’s no place to drop the keys, and we’re left with just using passwords. Support for this would be nice, but the security problems associated with keep SSH keys and user home directories are probably too much to even think about.
What else? Oh, yeah. The PIX/FWSM/ASA family supports SSH, but it acts differently from the IOS guys. When you run a command through SSH, you actually get an interactive shell with the command already on the CLI for you. This is probably by design; the only thing you can really do from a non-priv prompt is to enable.
Anyway, send any grilling tips questions my way.
Server NIC Aggregation to a Cisco Switch
Apr 14th
Have you even noticed that your new servers all have 2 NICs on the board? At least all of them that I’ve seen in the last 3 years have. A lot of server admin actually use them in a NIC teaming scenario where both NICs are used as one logical device — much the same as Etherchannel on a switch. This provides some fault tolerance and availability in case of failure, which is good idea in most cases.
There are a few different ways to configure teaming on the box (usually called bonding in Linux), and each has its own advantages and disadvantages. The network dude(tte) may have to do some things on the switch side for some of them to work, though. If you’re want to run in link aggregation mode (mode 4), for example, the switch ports need to be in the same channel group to work appropriately.
Let’s look at mode 4 a little closer to see what we need to do. The scenario is that you have eth0 plugged into F0/15 of a 2950 and eth1 is in F0/16. You’ve seen the configuration for channelling between switches before, so you know the basics. Put the ports in the same channel-group and configure the proper Port-channel interface to do the work. In this case, we’re just configuring the ports to house a host instead of being trunks.
int F0/15 channel-group 1 int F0/16 channel-group 1 int Port-channel 1 speed 100 duplex full switchport switchport mode access
I detect at least one problem with our setup, though. Both NICs are plugged into the same switch; what happens when the switch goes down? The server goes away. Logic should tell you, then, to put the NICs in different switches to fix that, but you can’t do Ethernchannel on two different switches. The ports have to be in the same device for the aggregation to work. What’s the fix?
You can look at getting a nice chassis switch and putting each NIC in different modules. Modern IOS versions allow etherchanneling across modules, so, if one module fails, you still have that other. That would do it, but I’m sure you don’t have the money for a 4500 in the budget, right?
Another solution is to use a couple 3760s which, when connected using the StackWise cable, are one logical device. That gives you two separate switches that you can configure with the same channel group. An upgrade to this solution is to use a pair of 6500s with VSS 1440 modules in them so that you have a stack of 6500s! I’m sure that’s not expensive at all, though.
Send any white shoes questions my way.
RSPANs on Cisco Switches
Mar 18th
We discussed SPANs earlier, but let’s talk about RSPANs for a bit.
Can anyone guess what the “R” means? You guessed it — “Remote”. An RSPAN is a way to get traffic from a SPAN source on one switch to a SPAN destination on another switch that’s connected via a trunk.
The basic premise is that a special VLAN is created on all the switches and allowed to traverse the trunks. You then set up a SPAN session that copies your traffic to this special VLAN. This VLAN then gets the traffic to the other switches through some voodoo magic to be used as source for a SPAN on another switch.
Let’s work through the steps. In our example, we want to copy traffic from G2/18 on SwitchA to G3/38 on SwitchB.
First, on both switches, we need to create the new RSPAN VLAN. We’ll assume you’ve already got it set up to allow this VLAN over your trunks.
vlan 2000 name RSPAN remote-span
Notice the nice keyword remote-span. This designates the VLAN to be used in an RSPAN. Easy so far.
Now, let’s create the session to copy traffic to the RSPAN. The source port is G2/18, and the destination is the RSPAN VLAN.
switchA(config)# monitor session 1 source interface Gi 2/18 switchA(config)# monitor session 1 destination remote vlan 2000
Now the traffic is being copied to the RSPAN, so let’s copy that traffic from the RSPAN to the sniffer. In this case, the source is the RSPAN, and the destination is the sniffer’s port. Let’s use session 8 to avoid confusion.
switchB(config)# monitor session 8 source remote vlan 2000 switchB(config)# monitor session 8 destination interface Gi 3/38
There are always things to look out for, aren’t there? The first that comes to mind is the fact that you’re copying traffic from a port onto one or more trunks. If the port is sending enough traffic and your trunk is close to capacity, you may wind up crushing the trunk link. That would suck.
If you have a fully-meshed switch environment, you’ll see the additional traffic across all your trunks if you’re set up that way. If you have four trunks that transport all VLANs, you may have four copies of the data coming out of the switch. Let’s say the box being monitored is compromised and sending out 600Mbps of data. That means that every switch will have to deal with that much traffic. This sounds to me like a CPU/memory issue waiting to happen.
Don’t expect RSPANs to work on your 2950 like this. On the lower-end switches, you have to use a reflector port to copy the traffic to the RSPAN. I don’t get into that here, but Google is your friend.
Send any Cadbury Creme Eggs questions my way.
SPANs on Cisco Switches
Mar 13th
I can’t believe I haven’t blogged on this yet. SPANs are one of my favorite things in the world.
The switched port analyzer (SPAN) is a mechanism on Cisco switches that allows you to take traffic on one port and copy it to another. It’s generally used to get traffic to a sniffer or IDS for analysis, but it’s a great tool to use to sample traffic from a host for troubleshooting.
Let’s use a real-world example. You’ve told your roommate to quit illegally downloading songs off the Internet, but you suspect he’s still doing it. Instead of sneaking into his room at night and checking his machine, you can use a SPAN to copy his traffic to another switch interface where you can use tcpdump to record what’s happening.
Let’s say you have a 2950, and the roomie is on F0/1. You have a Linux box plugged into F0/24 ready to capture the traffic. Here’s what you do.
monitor session 1 source interface F0/1 both monitor session 1 destination interface F0/24
This will create a new monitor session (that is, a SPAN session) that copies traffic from port F0/1 in both directions to port F0/24. Now, when you run tcpdump on your Linux box, you see all the traffic coming in and going out of your roommate’s port.
That’s pretty easy, right? You can have multiple sources ports by just adding more source lines or using ranges of ports. You can also just copy received or transmitted traffic from a source. Check out the contextual help for a little more info.
To see what’s going on, you can do a show monitor or a show monitor session 1 (depending on the IOS version). You’ll see something like this.
switch#sh monitor Session 1 --------- Type : Local Session Source Ports : Both : Fa0/1 Destination Ports : Fa0/24 Encapsulation : Native Ingress : Disabled
If you take a look at the destination port when the SPAN is running, you’ll see it’s in a state of up/down (monitoring). I think you can figure out that this means we’re monitoring some traffic to this port. Here’s what you’ll see if you look at the port.
switch#sh int f0/24 FastEthernet0/24 is up, line protocol is down (monitoring) ...
There are two big things to keep in mind when doing SPANs. The first is that monitoring a port can drive CPU utilization way up (depending on the platform and traffic volume), so you may run into problems if you have a bunch of SPANs going at the same time. Related to this is the fact that, if your switch has to decide between switching and copying traffic, it will stop copying until there’s enough CPU headroom to do that safely, and you’ll lose packets in the meantime. It’s a switch — not a copier.
The second thing to keep in mind involves those little voices in your head called ethics. What if you see a VOIP phone call from your boss to the HR department? How about if you find someone in upper management copying a spreadsheet of people to be fired tomorrow? How about if you find an engineer’s telnet password to a key system? These are things that you probably shouldn’t see, so be careful when looking at the packets. I would suggest you tell someone in your security when you’re going to do a packet capture to make sure someone knows you’re not up to no good.
Send shamrocks questions my way.
VLANs on Linux
Feb 19th
My home network has a Linux box running IPTables as it’s center point, and, since there are four networks, it has 4 NICs and 4 cables into the switch. I kept running into problems with the NICs (they would reorder depending on what flavor of Linux was installed), so I wanted to consolidate the NICs down to 2 — one for the Internet link and one for the LAN segments with 802.1q tagging.
Disclosure: I have only labbed this stuff out, and it seems to work, but I have yet to implement it. Use at your own risk in the wild.
Configuring VLAN tagging on Linux is pretty simple, actually. One way to do it is to use the vconfig command to add and remove VLANs from interfaces. As a demonstration, say you want to run VLANs 20 and 30 on eth0. You would just do something like this. Note that the interface you mention here has to be in an UP state, so do an ifconfig eth0 up if you need to get it into a good state.
vconfig add eth0 20
vconfig add eth0 30
Now, whenever eth0 comes up, you’ll have the interfaces eth0.20 and eth0.30. You can give them IP addresses through the command line with ifconfig.
ifconfig eth0.20 192.168.20.1 netmask 255.255.255.0 up
ifconfig eth0.30 192.168.30.1 netmask 255.255.255.0 up
I didn’t expect them to do so, but the IP addresses actually stay across network restarts; as long on the physical interface comes up, the VLANs come up with IPs and everything. Speaking of network restarts, the “downfall” with using vconfig is that VLAN interfaces don’t show as coming up or going down during network restarts; I don’t like that at all.
Another way to configure the VLANs is through the old-fashioned network-scripts directory. Copy your interface config (ifcfg-eth0) to the same name but with the VLAN extension (ifcfg-eth0.20) and edit it. Change the device field appropriately along with the IP address subnet mask info. For the final touch, at the end of the file, add this line.
VLAN=yes
Personally, this is the way I would do it. It lets you change configurations through the configuration files just like physical interfaces instead of trusting the configuration that resides out in the ether that is the Linux kernel. Also, when you restart network, the interface itself actually goes up and down, so you can see what’s going on with it. If you need some help with this, check out Redhat’s manual on it. Let me know if you’re still having problems with it.
Remember to set the box’s switch port to a matching 802.1q trunk. You’ve seen that before, but here’s a refresher, assuming the Linux box is plugged into f0/1 on the switch.
int f0/1
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
To check the status of your VLANs, look in /proc/net/vlan. You’ll see the config file, which lists all your VLANs. You’ll also see a device file (like eth0.20) with the statistics for that VLAN device (interface).
Let’s talk security, though. First of all, I could argue that a Linux box shouldn’t be participating in any trunking at all. There will be exceptions, but, in my experience, a Linux box (read: server) should only be on one network at a time and shouldn’t straddle networks. Do you really trust the Linux guys to keep their boxes from doing bad things on more than one network? I don’t. Heh.
If, however, you need to use VLANs on a Linux box, you’ll need to make sure you have only the proper VLANs running across this port (like we did with the CSM VLAN). If a box were to be compromised, the bad guy could simply start adding VLANs to the server and suddenly get around your routers and firewalls. Awesome, right? Make sure you put in the switchport trunked allowed vlan x directive so the server only has access to those VLANs.
As always, send me any four-leaf clovers questions you have.
P.S.: For the record, since I haven’t tried this in the field yet, I can’t tell you how well it works with IPTables, but, from what I’ve been reading, it should work fine. Good luck.
For the record, I’ve got this working at my house connected to a Cisco Catalyst 2950 trunk port. I’m happy to report that it works like a champ with IPTables and everything.
Mixed-platform LANs and Spanning Tree
Aug 9th
We just an HP C-class blade chassis which included two GbE2c network modules. These modules are Nortel switches running AlteonOS that connect the blades to the rest of your network. When I turned these guys up the other day, every VLAN stopped working, so I ran down to the data center and unplugged the uplink. I called HP and soon found out that the GbE2c doesn’t play nice with Cisco switches out-of-the-box. Since we have a Cisco network (not now, I guess), we can into some problems.
By default, Cisco runs Per-VLAN Spanning Tree (PVST). That means that every VLAN on a Cisco switch has its own Spanning Tree (STP) instance. The Nortels, however, run a single STP instance for all VLANs, so, when I turned up the uplinks, the single STP started talking to all of the PVSTs on the Cisco switches. The result was not good. Every VLAN converged over and over simultaneously, totally locking up the network.
When we got that fixed and the uplinks were working, I found another problem — every switch in the network decided that the Nortels were the root bridges. While this is not a show-stopper, it can prove to be a problem since the root bridge is used by STP to decide where the center of the network is. It turns out that the GbE2c has a default priority of 32768 (which is the 802.1d standard) and advertise themselves as that. Cisco switches actually add the VLAN ID to the priority on the PVST, so the Nortels always wind up with the lowest priority. These things are absolute pieces of crap, so that’s not a good thing. I had to take an outage and reduce the priority of two of our core Cisco switches to make everything right.
Moral of the story: Be careful when you have a mixed-platform LAN environment since competing problems don’t always (read: never) play nice.
