A good example came up a few months ago. It seems that, at some time in the past, one site needed some more LAN ports, but, instead of calling us and having us send them another switch, one of the “technical people” there brought in a hub from home. It really irks me to see a hub on the switched LAN, but we really have no control over those decisions. They plugged the hub into one of the existing drops somewhere in the building and plugged everyone in. It worked…until somebody moved one of the machines. The machine was at a desk near the hub, and the network cable was just unplugged and left lying there. A good Samaritan came by, saw that the hub was not plugged into the network, and plugged it back in for us — providing a nice second link from the hub to the switch stack in the closet. Take one switch stack, add a hub, insert a switching loop, bake at 350F for a few milliseconds, and you have a broadcast storm. If you don’t know already, broadcast storms are bad and eat switch CPU like the yummy cookies we baked. In this case, several 3750s were taken completely down.

How does one prevent such from happening again? Well, the first thing to do is to get the CTO to tell everyone that they can’t plug hubs into the network. That works about 0% of the time, though, so we had to find a solution that was enforceable. One of my coworkers found the traffic storm control mechanism built into Cisco switches. This mechanism allows you to set thresholds based on broadcast, multicast, and unicast traffic and take action when those are reached.

Here are the gory details. I need to mention, though, that storm-control is configured very differently across platforms and IOS versions. I would say your mileage may vary, but it’s probably more accurate to say that this won’t work on your switch. A 6500 is configured differently than a 4500. A 2900XL is different from a 2950. This will get you going, but you’re going to have to do some research on your own to find out what works on your platform.

interface FastEthernet 0/1
storm-control broadcast level 50
storm-control action shutdown

What just happened? Good question to ask. If broadcast traffic on F0/1 utilizes 50% of available bandwidth, the port is shutdown. That means that if broadcast traffic takes up 50Mbps of bandwidth on this port, the port is admined down just as if you did a shutdown on it.. You should probably do the same for multicast or unicast as well to make sure you don’t get bitten by those. If you don’t want to shut down the port, you can also use the trap action to just send an SNMP trap with the port and information, but that doesn’t prevent very much; the storm will probably wreak havoc before an email for the trap lands on your Crackberry.

Here’s another big disclaimer. Finding a good level for a port can be very, very difficult. A linux box is going to have very different broadcast/multicast/unicast traffic than a Windows box which is different than a Mac. You may have to spend a lot of time analyzing SNMP counters to find out what a good level is.  God help you if you have a hub like we did with mixed computer platforms on it.

The setup had two networks with 192.168.0.0/24 on the customer’s local network and 192.168.1.0/24 on the “DMZ” (Yes, I know it’s not really a DMZ). We want to NAT 192.168.1.100 to 192.168.0.100.

interface FastEthernet0/0
ip address 192.168.0.10 255.255.255.0
ip nat inside
!
interface FastEthernet0/1
ip address 192.168.1.10 255.255.255.0
ip nat outside
!
ip nat inside source static 192.168.1.100 192.168.0.100

This works great, but then the customer asked for some redundancy. He wanted a second router inline that could take over in case of failure on the original. “Piece of cake,” I thought and went about setting up the HSRP stuff on both the original and new second routers.

interface FastEthernet0/0
standby ip 192.168.0.1
standby preempt
standby name MYHSRP

But what about that NAT thing? If the primary router goes down, the NAT goes down with it. If I configure the secondary router for the NAT, we get an IP conflict. I looked around for a while and found a feature introduced in IOS version 12.2(4) that allows a NAT to follow an HSRP or VRRP active member. If a router is the active member of an HSRP or VRRP cluster, the NAT is with that box; if it fails over, the standby IP and the NAT move. Isn’t that cool? The only caveat is that it keys off the name/description of the HSRP/VRRP cluster instead of the ID, so you have to have that configured as we do above.

ip nat inside source static 192.168.1.100 192.168.0.100 redundancy MYHSRP

Let’s test to be sure it works. On the active member (a router called NAT0), we can look at the ARP table and see that this router is speaking for the NAT address of 192.168.0.100 as well as the HSRP IP of 192.168.0.1. The secondary member (NAT1) only has ARP entries for its interface IPs. Looks good so far.

NAT0#show standby brief
Interface Grp Prio P State Active Standby Virtual IP
Fa0/0 0 100 P Active local 192.168.0.11 192.168.0.1
NAT0#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.0.100 - c800.12bc.0000 ARPA FastEthernet0/0
Internet 192.168.0.10 - c800.12bc.0000 ARPA FastEthernet0/0
Internet 192.168.1.10 - c800.12bc.0001 ARPA FastEthernet0/1
Internet 192.168.0.1 - 0000.0c07.ac00 ARPA FastEthernet0/0
...
NAT1#show standby brief
Interface Grp Prio P State Active Standby Virtual IP
Fa0/0 0 100 P Standby 192.168.0.10 local 192.168.0.1
NAT1#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.1.11 - c801.12bc.0001 ARPA FastEthernet0/1
Internet 192.168.0.11 - c801.12bc.0000 ARPA FastEthernet0/0


When I shut down F0/0 on NAT0, the HSRP and NAT both roll over to NAT1.

AT0(config)#int f0/0
NAT0(config-if)#shut
*Mar 1 00:30:25.248: %HSRP-5-STATECHANGE: FastEthernet0/0 Grp 0 state Active -> Init
*Mar 1 00:30:27.263: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Mar 1 00:30:28.265: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
NAT0#
NAT0#sh stand b
Interface Grp Prio P State Active Standby Virtual IP
Fa0/0 0 100 P Init unknown unknown 192.168.0.1
NAT0#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.1.10 - c800.12bc.0001 ARPA FastEthernet0/1
...
NAT1#sh stand b
Interface Grp Prio P State Active Standby Virtual IP
Fa0/0 0 100 P Active local unknown 192.168.0.1
NAT1#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.0.100 - c801.12bc.0000 ARPA FastEthernet0/0
Internet 192.168.1.11 - c801.12bc.0001 ARPA FastEthernet0/1
Internet 192.168.0.11 - c801.12bc.0000 ARPA FastEthernet0/0
Internet 192.168.0.1 - 0000.0c07.ac00 ARPA FastEthernet0/0

Now we have redundancy and NAT on the same cluster. Sweet. I also did the same lab for VRRP, but I’ll spare you the innards. When you configure VRRP, you can give it a description, which is the same as the name in HSRP. You use that string as the redundancy name to have a NAT move with VRRP. Yes, you can do both an HSRP- and VRRP-based NAT at the same time.

*I labbed this out on a Dyanmips/Dynagen instance of two 2651XM routers running 12.4(19) Advanced Enterprise. Like all the configs here, your mileage may vary if you have different versions or feature sets on your routers.

Edit:  I wanted to make a not on GLBP and NAT.  Since all the nodes in a GLBP cluster actually route traffic and are routed to, you can’t use this type of NAT solution with it.  I don’t know what the best practice would be, but I imagine it would involve running HSRP on a few routers as we talked about above.

Anyway, we won’t be going over any configs this time, but we can get some in the future. I just wanted to go over some stuff that you may need to know to get an FWSM working.

The FWSM does not have any physical ports on it. It uses the backplane of the switch to get a foot in the networks that you need it to. What does that give you? Well, it’s ultrafast on the 256Gbps backplane, which is a lot better than the 1Gbps on a physical interface. It also lets you have more interfaces than you could ever have on a firewall appliance (I think you can have 128 interfaces…someone correct me). I don’t know any firewall appliance that has more than 20 or so interfaces. It also is good for your cabling guy since there are no cables. I love it since I’m the cabling guy, too.

The FWSm can be set up with several security contexts. That means that you can chop up the resources of the module into “virtual firewalls” that operate separately from each other. This is pretty cool, actually, since you can, say, make a firewall for each of your hosting customers so that configurations on one don’t affect the others. You can also set the thing up in single context mode so it acts just like a single firewall. You’ll have to do some research to figure out what you want to do, and there are a lot of possibilities.

You can also set each context up in either routed or transparent mode. Routed mode is the “traditional” way to set up a firewall, where each interface is a different IP network with packets being routed through the FWSM. Transparent mode is where the FWSM has two interfaces that bridge the same IP network. I’ve always used routed mode everywhere, but you may have some use for transparent mode.

You also have to do some stuff on the 6500 to get the FWSM working. You make a vlan-group (a list of VLANs) to present to the FWSM and then assign that to the module. If you have more than one FWSM, you can have multiple vlan-groups; different FWSMs can control access in and out of different networks. Remember the context thing? If you have an FWSM in multiple contexts, you present the module with all the VLANs on all the contexts and split them up when you get to the context config part.

No configs. This is weird, but you’ll need to get all this information straightened out before you can get an FWSM configured properly.

Logical diagrams show layer-3 (the IP networks) and how those are interconnected. From that, the diagram inherits the data paths as well — how does the packet get from network A to network B and back. You can’t see that with physical diagrams in a lot of cases.

Here’s a physical diagram of a single Internet router and a 6509 with an FWSM. It literally shows a router and a switch. How many IP networks do I have? How many firewall interfaces do I have? How many layer-3 interfaces on the 6509 am I using? This logical diagram, however, shows the same network from layer-3. A big difference, eh? Didn’t know I had 8 networks, did you?

Don’t go replacing all your physical diagrams with logical ones, though. You still have to know where things are plugged in, so keep the physical…just add a logical as well.

Tip of the day: Use Visio tabs for logical and physical diagrams on the same document.

• Corporate traffic had to go across the frame relay link during normal operations.
• Internet traffic had to go across the DSL line during normal operations.
• If the DSL circuit went down, Internet traffic should be moved over to the frame relay circuit to use the corporate Internet link.
• If the frame went down, traffic should be sent out the VPN tunnel for access to corporate stuff.

We set the default routes of the machines (via DHCP) to the frame relay router. That router’s default route sent traffic to the DSL router, which, of course, had a default route towards the provider. Both routers were participating in EIGRP with the rest of the corporate network, so they all knew where to route traffic destined for corporate traffic. If there was a frame outage, the default routes kicked in and sent traffic to the DSL router, which had the VPN tunnels. The problem came when there was a DSL outage.

At first, we were just monitoring the DSL IP and manually changing default routes when there was an outage, but you know DSL. We were lucky to have only 3 or 4 a day go down, so it was taking up a lot of our time just moving default routes around. We had to go to an automated solution, so we looked at doing object tracking but came up just short; it just didn’t do what we wanted. We had to go one more step and discovered reliable static routing (RSR).

In normal object tracking, you can track an interface or a route to an IP. With RSR, you can track reachability to an IP via ICMP (or a whole list of other things). This is just what we needed. We figured Yahoo was a site that would always be up, so we went about tracking one of Yahoo’s IPs from the frame routers, and, if it went down, we could send traffic back across the frame relay cloud.

To set up RSR, you have to set up an IP SLA entry, build a tracking object, then have the default route track the object. First, the IP SLA entry that we made.

ip sla monitor 1
type echo protocol ipIcmpEcho 66.94.234.13
frequency 10


This built IP SLA entry 1 to ping 66.94.234.13 every 10 seconds. With IP SLA, you also have to set a schedule for it to run. We wanted it to start ASAP and run forever, so we just did one of these.

ip sla monitor schedule 1 start-time now

This schedules entry 1 to start now, so every time the box came up, the IP SLA process started pinging away. You then combine the entry with a tracking object like this.

track 100 rtr 1 reachability

We built object 100 to monitor the reachability of SLA entry 1, so now we have an object to track if a host is unreachable. Sweet. How about the static part of the route? We set the default route of the frame router as the LAN IP of the DSL router tracking object 100. We also set a weighted default route pointing out the frame relay circuit to use when the object fails. Assume 10.0.0.0/24 for the frame cloud and 192.168.0.0/30 for the LAN.

ip route 0.0.0.0 0.0.0.0 192.168.0.1 track 100
ip route 0.0.0.0 0.0.0.0 10.0.0.254 250

In this setup, the default route is the DSL router, and, if the tracking object fails, it rolls to the frame cloud. Can you spot the flaw, though? When the route changes over, the object has recovered since it can reach Yahoo through the corporate Internet, and the default route get moved back to the DSL router…which then fails again…and rolls back…and…ack! An easy fix is to set a static route for the tracked IP to the DSL router.

ip route 66.94.234.13 255.255.255.255 192.168.0.1

This route also lets the frame monitor monitor the DSL service, so, when it comes back up, the tracking object recovers. Of couse, if someone at the site tried to get to Yahoo on that IP during a DSL outage, they would time out (shouldn’t they be working?), but, when the DSL recovered, the router would know.

The old adage of your chain only being as strong as your weakest length holds true in this case. Here, the weakest link is actually the most-congested trunk, though. Let’s step through to see. A 2950 is a 10/100 switch, so a single trunk can handle 100Mbps of traffic. We have 10 of these guys, Switch1 to Switch10, all trunked to the one above and below. If a server in the center of the stack on Switch5 is sending a lot of data to the Internet routers on Switch1, the trunks off of Switch5 will start to get saturated. Switch4 has a few hosts doing the same thing, so traffic from both Switch4 and Switch5 heads towards Switch1, further filling the trunks. Same for Switch3. Same for Switch2. Next thing you know, there’s 184Mbps or so trying to go across a 100Mbps link.

I fixed the problem using EtherChannel. EtherChannel, sometimes called trunking (not Cisco trunks…don’t get confused) or bonding, takes up to 8 interfaces of a switch and binds them logically as one Port-channel interface. The switch then load-balances (not really, but can be pretty close) the traffic across all the links, so, in essence, if you have 2 100Mbps ports in an EtherChannel, you get 200Mbps of bandwidth. If you put in 8 ports, you’ll get 800Mbps. Or, at least, close to it.

There are, as usual, stipulations for using it.

• All ports on the EtherChannel must terminate to the same device on both ends. You can’t have one port go to one switch and another to another switch.
• The ports must use identical media. You can’t have a copper port and a fiber port in the same group.
• The ports must have the same capabilities. You can’t put a 100Mbps port and a 1Gbps port in the same group.
• The ports must be configured identically or it won’t work. This is actually pretty easy to maintain since configuring the Port-channel actually sets the config on all participating ports.
• Both ends of the ports must be EtherChannel. You can’t run it on one switch but not the other.

Enough of that. Let’s configure. The scenario is that you have SwitchA and you want to bind F0/1 and F0/2 into an EtherChannel. Of course, you want to carry traffic from all VLANs, so let’s make it a trunk.

int F0/1
channel-group 1

int F0/2
channel-group 1

int Port-channel 1
speed 100
duplex full
switchport trunk encapsulation dot1q
switchport mode trunk

Easy as pie. Do the same thing on the other end and you have a 200Mbps, bonded trunk.

That sucks, right? One of several fixes for this is the route-reflector-client directive under the BGP neighbor configuration. A route-reflector literally reflects the routes from one client to the others just as if you’ve got a fully meshed network. Here’s a sample config for the reflector, R0.

router bgp 65000
neighbor 10.0.0.11 remote-as 65000
neighbor 10.0.0.11 route-reflector-client
neighbor 10.0.0.12 remote-as 65000
neighbor 10.0.0.12 route-reflector-client

There’s actually no additional config on R1 and R2 at all; the router reflection is transparent to them. They actually see the route just as though they got the update directly from the originating router, so any routes received from the reflector appear as though they came from the originator. In that same breath, you have to realize that R1 and R2 must have routes to each other since the untouched BGP routes will have a next-hop address as the originating router.

Edit: Here’s a crayon drawing I did to show an example of where you would use route-reflectors. R0 is connected to R1 and R2 via different WANs, but R1 and R2 aren’t connected at all.

VTP works by having designated VTP servers (not real servers like your Linux box, but a switch) tell the rest of the switches in the network with what VLANs they should be configured. All the designated VTP clients say “OK” and configure themselves with those VLANs. When you take a VLAN out of the server, all the clients take it out; when you add a new VLAN, all the clients add it as well. The server and client designation is known as the VTP mode, and there’s one more to mention. When a switch is in VTP transparent mode, he will see VTP from the servers but will ignore them and pass them on to the next switch as if nothing ever happened.

To configure VTP, you have to define the VTP domain, which is a name for a group of switches that exchange VTP information. In general, in an enterprise, all your switches will probably be in the same VTP domain, but there are situations where you have multiple VTP domains on a LAN (like production and test networks). Next, you set the mode on the switch — server, client, transparent. I would suggest you configure a password as well to at least appear as though you’re practicing good security practices (I’m not sure it’s actually secure or not since every switch has the same password and the hash is sent in every packet…somebody enlighten me, please).

Get on your configuring gloves while I give you some more detail. VTP only works over 802.1q or ISL trunks. Also, VTP packets are Ethernet frames with the VTP stuff shoved into the data section. A switch can only be in one VTP domain at a time and can only be in one mode at a time. You can have more than one VTP server for a domain if you’d like (via a revisioning mechnism where the latest update always wins).

Here’s your scenario. You have SwitchA’s port G1/1 plugged into SwitchB’s G2/1. You want SwitchA to be the VTP server and SwitchB to be the client. Let’s get the config in place.

SwitchA(config)#interface G1/1
SwitchA(config-if)#switchport trunk encapsulation dot1q
SwitchA(config-if)#switchport mode trunk
SwitchA(config-if)#exit
SwitchA(config)#vtp mode server
SwitchA(config)#vtp domain VTP-DOMAIN
SwitchA(config)#vtp password VTP.PASS

SwitchB(config)#interface G2/1
SwitchB(config-if)#switchport trunk encapsulation dot1q
SwitchB(config-if)#switchport mode trunk
SwitchB(config-if)#exit
SwitchB(config)#vtp mode client
SwitchB(config)#vtp domain VTP-DOMAIN
SwitchB(config)#vtp password VTP.PASS

Both switches are now in VTP domain “VTP-DOMAIN” and share the password “VTP.PASS”. When you add a VLAN to SwitchA, it automatically gets added to SwitchB. When you add a VLAN to SwitchB, it actually errors out and says something like “Dude, I’m a VTP client. You can’t update me.”

At work, we have a large number of ports but a small number of switches. We also make very few changes to VLANs, so the work needed to maintain the VLANs is very low. Because of that, we actually do use transparent mode on all the switches. I’m also a bit of a purist, and configuring VLANs is something I like to do by hand; in my opinion, some things, especially mechanisms that can take whole networks down, are better left to the human. That’s just me, though.

The most useful function is the include directive. This is the equivalent of just plain grep on Linux, and will show you only lines that match a string that you give it. Say that you want to find what ports on your switch are down, but don’t want to grind through all the lines of a show ip interface brief. If you just pipe it to the include command followed by the word “down”, you’ll see something like this.

Switch#show ip interface brief | include down
GigabitEthernet0/4 unassigned YES unset down down
GigabitEthernet0/7 unassigned YES unset down down
GigabitEthernet0/17 unassigned YES unset down down
GigabitEthernet0/18 unassigned YES unset down down
GigabitEthernet0/19 unassigned YES unset down down
GigabitEthernet0/20 unassigned YES unset down down

You can also use the exclude directive, which is the same as a grep -v on Linux. I hope you figured out that this gives you all lines that don’t match the word, so, let’s use the exclude directive to found out what ports are down. How about we just ignore the lines that are up.

Switch#show ip interface brief | exclude up
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/4 unassigned YES unset down down
GigabitEthernet0/7 unassigned YES unset down down
GigabitEthernet0/17 unassigned YES unset down down
GigabitEthernet0/18 unassigned YES unset down down
GigabitEthernet0/19 unassigned YES unset down down
GigabitEthernet0/20 unassigned YES unset down down

Well, this won’t exactly give you all the ports that are down. What if the port is up/down?

What else can you use with the pipe? What if you want to look at the configurations of all the ports or interfaces on a box but don’t want to go through the config hitting the spacebar over and over. If you use the begin command, you’ll see the output beginning from the first match, so, using the string interface will show you the config starting at the first interface.

Switch#show running-config | begin interface
interface GigabitEthernet0/1
description Server
switchport access vlan 4
switchport mode access
spanning-tree portfast

…

Another good one is the section command. It’s usually used on the output of show running-config and shows you sections of the config that match your string. Huh? If you want to see the BGP section of the configuration, you can do something like thing just to see that part of the configuration.

Router#show running-config | section bgp
router bgp 1
neighbor 1.1.1.1 remote-as 65000
neighbor 1.1.1.1 version 4

There are a few other commands for use with the pipe, so explore on your own. You might also want to check out regular expressions on the Cisco IOS if you want to match more than just simple text.