Archive for the ‘ios’ tag
Stubby Post – Cisco IOS Petition
Greg Ferro has brought back the petition for Cisco to provide an emulator to the community for learning. Since our current and only family of emulators is well on its way to oblivion, I ask that we all take the time and sign this petition. To use a cliché, we need to act now before it's too late.
Another Blow to Dynamips/Dynagen/GNS3
It looks like Cisco is trying to crack down on illegal distribution of their software. I can't really blame them since it's their property. Read the rest of this entry »
Syncing IOS Versions on a 3750 Stack
For those that don’t know, when I say “stack”, I mean a group of 3750s connected together using the StackWise technology. When you use a very expensive and very proprietary cable, your individual switches are combined into a single logical device. This means you configure one device to control potentially many switches.
To the point. I’ve spent the last few weeks replacing a mess of 3750s in stacks. These guys are very easy to replace, but the big problem I find is getting the IOS version in sync. When the RMA comes, it’s inevitably got a different version on it, and you’ll see something like this.
Switch# Role Mac Address Priority State -------------------------------------------------------- 1 Member 0023.33ad.a500 1 Version Mismatch *2 Master 0023.5eac.e900 15 Ready
In this case, switch 2, running 12.2(25)SEE3 is the master, and switch 1, running 12.2(35)SEB, is the new member.
A switch in the stack needs to have the same version of IOS as the master to be brought into the fold, so you’ve got to do get the code on the switch somehow. You can use traditional methods to get the right code on the box (like TFTPing one up), but there are easier ways.
If the code versions are close (I’m not sure as to the rules about how close), then you’ll be able to check out the flash of all the switches in the stack through the master. If you do a dir ?, you’ll see flash1:, flash2:, etc. Those are the individual flash drives for each switch in the stack. The numbering is based on the stack number, so all you have to do is copy the IOS image over from the master. Well, maybe.
I found that a lot (maybe all) of my 3750s actually have a directory structure to include the image and HTML files for the device manager. You don’t need the HTML files for the switch to function in the stack, but having the IOS image in different places will force you to change the boot variable for each switch. I like consistency, so I put everything in the same place when I can, so that means creating the directory by hand.
That method works fine, but there’s an easier way using the archive command. You can tell your switch to tar, copy, and extract all of the files from one switch to the others by giving it one of these.
archive copy-sw [ /destination-system X ] Y
I used this command to copy the image from switch 2 (the master) to switch 1 (the replaced member) and got a whole bunch of output. The whole process took about 3 minutes.
Switch#archive copy-sw /destination-system 1 2
System software to be uploaded:
System Type: 0x00000000
archiving c3750-ipbasek9-mz.122-25.SEE3 (directory)
archiving c3750-ipbasek9-mz.122-25.SEE3/c3750-ipbasek9-mz.122-25.SEE3.bin (7206732 bytes)
archiving c3750-ipbasek9-mz.122-25.SEE3/html (directory)
archiving c3750-ipbasek9-mz.122-25.SEE3/html/toolbar.js (7084 bytes)
archiving c3750-ipbasek9-mz.122-25.SEE3/html/title.js (577 bytes)
...
extracting c3750-ipbasek9-mz.122-25.SEE3/html/images/141280.gif (3053 bytes)
extracting c3750-ipbasek9-mz.122-25.SEE3/html/images/meter_yellow.gif (59 bytes)
extracting c3750-ipbasek9-mz.122-25.SEE3/html/images/legend_off.gif (1158 bytes)
extracting c3750-ipbasek9-mz.122-25.SEE3/info (682 bytes)
extracting info (108 bytes)
Installing (renaming): `flash1:update/c3750-ipbasek9-mz.122-25.SEE3' ->
`flash1:c3750-ipbasek9-mz.122-25.SEE3'
New software image installed in flash1:c3750-ipbasek9-mz.122-25.SEE3
All software images installed.
I left out a large chunk, but you get the idea. If you’ll notice, all the HTML files are copied along with the IOS image, so you get exactly the same structure on all your switches. It beats doing it all manually.
Send any RMA packages questions my way.
Getting Temperature Data from a 6500 via SNMP
I apologize to my adoring fans (both of you) for the lack of posting. I’m in the middle of moving, buying a new house, selling my current house, getting a mortgage, etc. I’ve up until 11:30 nearly every night filling out forms and going through red tape. Don’t get me started on getting money from a 401k! Anyway…
I got in this morning, and a coworker was telling me that the data center’s HVAC was crippled due to an oil leak, and it was 90F in there. D’oh! It wasn’t quite that high, but it was warm. Luckily, all of our network gear is on the end of the rows with AC, so we’re safe, but it got me thinking about monitoring temperature of our 6500s via SNMP. I’ve done it via Cacti, but I never really looked how to do it manually.
I read this article on Cisco’s site which made it clear as mud. Basically, you have to query the entPhysicalIndex OID branch to get an integer that represents the physical sensor, then you query the entSensorValue to get the temperature. I made the lazy choice to use grep on entPhysicalDesc OID to find the module I wanted and use that to grep the entSensorValue for the temp.
Let’s say I’m looking for the outlet temperature (the temperature of the air on the way out) on module 3. First, let’s find out what th entPhysicalIndex is by walking the entPhysicalDesc OID and grepping out “module 3″.
[myserver]$ snmpwalk -c <COMM> -v 2c <HOST> 1.3.6.1.2.1.47.1.1.1.1.2 | grep "module 3" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3001 = STRING: "module 3 power-output-fail Sensor" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3002 = STRING: "module 3 insufficient cooling Sensor" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3003 = STRING: "module 3 outlet temperature Sensor" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3004 = STRING: "module 3 inlet temperature Sensor" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3005 = STRING: "module 3 device-1 temperature Sensor" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3006 = STRING: "module 3 device-2 temperature Sensor"
Since we’re looking for the outlet temperature, we can see the index in question is 3003. Now, we use that to query the entSensorValue.
[myserver]$ snmpwalk -c <COMM> -v 2c <HOST> 1.3.6.1.4.1.9.9.91.1.1.1.1.4.3003 SNMPv2-SMI::enterprises.9.9.91.1.1.1.1.4.3003 = INTEGER: 19
Ah…a cool 19C.
This is not a very graceful way to do it, but it gets the job done in a pinch. I haven’t tested it, but I would imagine this technique would work on 7600s as well. I know it doesn’t work on 2950s and the like.
Send any house keys questions my way.
Filtering Out the Noise on the Edge
There’s a lot of noise on the Internet. I’m not talking about certain news sites, either; I’m talking about stuff like port scans or attempts on weak services from all sorts of bad people on the Internet. A large chunk of that noise can be filtered by the edge routers, taking some of the load off of the network and firewalls.
Here are a few things that we filter inbound on our Internet links. Your mileage will vary.
- Packets from RFC 1918 space — You should never see a packet from 10/8, 172.16/12, or 192.168/16.
- Packets from your IP space — Why would you receive packets from yourself from the Internet?
- SSH, telnet, cmd, rlogin, RDP, etc. – You should be doing all your admin stuff from the internal network or from a VPN, right?
- Windows ports — For God’s sake, drop these at the edge.
- Packets to your network services subnets — If you use public addresses for things like your FWSM or CSM sync networks, no one should ever talk to those subnets.
- SNMP, SNMPTrap — No monitoring from the Internet!
- SMTP to non-MX hosts — If you have a lot of hosts, you probably have email run amongst them. Only the MX hosts should accept connections from the Internet.
- TCP/UDP small services — whois, finger, chargen, etc., are just waiting to be used for something bad.
- DNS, RNDC — You may have some name caching servers or hidden masters somewhere that shouldn’t be reachable from the Desolate Plains of the Internet™.
- Syslog — No logging from the Internet. Use a VPN tunnel or something if you really need it.
- NTP — You’re not a time service, are you?
That should cut out a significant amont of noise for you. Remember to allow stuff, too. You may want to end your ACLs with an old-fashioned permit ip any any log to see what else is coming through and maybe block some of that noise, too.
Configuring Dedicated Trunks for the CSM
Did you catch the article on setting up fault tolerance on the CSM? In that article, I mentioned that Cisco recommends a dedicated trunk for the FT VLAN if you have two HA CSMs in two chassis. Discuss amongst yourselves while I drone on.
Why should you set up a dedicated trunk for this stuff? The most obvious reason is to be sure that normal traffic doesn’t step on the syncing traffic. Since we’re syncing state information as well as configuration, the frames need to arrive in a timely manner. Any errors could potentially disrupt the FT process, which is bad. You surely don’t want the primary to fail only to find out that the standby doesn’t have the complete or current config.
Another reason is to keep the syncing traffic from stepping on normal traffic. The CSM is a pretty robust box and can handle a pretty good chunk of data. If you had a 100Mbps trunk between your chassis, there is the potential for the link to get flooded if the CSM ever starts sending some real data. All things being equal, though, your trunks are probably sized properly for your network, and the addition of the syncing traffic probably won’t affect much.
Let’s review our configuration from the other article.
vlan 83 name CSM-Sync ! module csm 3 ft group 1 vlan 83 priority 100 alt 90 preempt
This snippet creates VLAN 83 and tells the CSM to use it for syncing, but how do we dedicate a trunk for that VLAN? We use the switchport trunk allowed vlan directive. We’ll assume that G1/1 on your primary switch is connected to G1/1 on your standby.
interface GigabitEthernet1/1 description CSM Syncing switchport switchport trunk encapsulation dot1q switchport trunk allowed vlan 83 switchport mode trunk
This sets G1/1 up to only allow VLAN 83 across it. If you do a show int G1/1 trunk, you’ll see that this VLAN is the only one allowed, the only one active, and the only one one forwarding on that link. Of course, you’ll need to do the same on the other side to keep traffic flow sane, but it’s fairly easy.
What if G1/1 goes down, though? You’d lose sync, so you probably want to look at a solution for that little problem. You could put in multiple links and let Spanning Tree do the work. You could even turn those links into an EtherChannel for redundancy and throughput. If you have more than two chassis, you could full mesh them with trunks dedicated to VLAN 83. There are a number of ways around the problem. Be creative.
Be sure to send turkey questions my way.
Using Probes on the CSM
There are three different ways that a CSM checks for the health of the servers — active probes, inband health checking, and inband HTTP monitoring. Let’s talk about active probes.
Active probes (or just probes) typically send traffic to one of the RIPs of a serverfarm, do some stuff, and give a pass or fail grade. If the probe fails a certain number of times in a row, that server is considered sick and taken out of the pool for use. The CSM keeps checking the unhealthy until it passes a number of times in a row, at which point it is placed back in the pool for use. Almost everything is configurable, of course, so let’s look at some of those settings.
These all have their defaults, so you don’t need to actually configure them, but it’s important to know they’re there to tweak later. There are other parameters to the more specific types of probes as well. You’ll have to venture forth on your own to figure those out, though I’ll be glad to help.
- interval: The time between probes when the server is healthy.
- retries: The number of consecutive failures before a healthy server is considered failed.
- failed: The time between probes when the server is failed.
- recover: The number of consecutive successes before a failed server is considered health.
I always said that the CSM only speaks HTTP, but it knows how to order a ham and cheese sandwich in a few other protocols, including doing some decent stuff like watching for SMTP banners, looking for ICMP reachability, or getting a response from a Tcl script. Here’s a list of the probes with some boring commentary. Depending on IOS versions, you may have more or less available to you.
- tcp: Establishes a connection to a TCP port. If the port is open, we pass.
- udp: Same as TCP, but in UDP. Duh.
- icmp: Ping-a-ling. Do I need to explain this one?
- http: Requests a URL from a webserver and looks for HTTP return codes.
- dns, smtp, telnet: These guys just attach to the service and look for a proper response header. It doesn’t do any transactions or anything but simply makes sure that DNS, SMTP, or telnet is running on the port.
- script: These probes run a Tcl script (that you write) and look for the return code of the script. Very powerful!
- kal-ap-tcp, kal-ap-udp: Admittedly, I have no earthly idea what those are. Most references I’ve seen to it involve the Cisco ACE, but I have no clue. Can someone fill in for me here?
Shall we try one or two? How about a TCP probe that makes sure your custom app is still running on TCP/8839? Since it’s a custom app, we can use a TCP probe on that port to make sure something is listening (If you need something more, you’ll need to check out script probes.).
probe MYAPP tcp
description My app on TCP/8839
port 8839
Now we apply the probe to the serverfarm.
serverfarm MYFARM
probe MYAPP
real 1.1.1.1
inservice
real 1.1.1.2
inservice
Easy enough. How about another? I want to configure an HTTP probe that gets the URL /test.php and looks for the status code 200 and apply it to a serverfarm.
probe LOOKFOR200 http
request url /test.php
expect status 200serverfarm YOURFARM
probe LOOKFOR200
real 2.2.2.1
inservice
real 2.2.2.2
inservice
By the way, you don’t want to use this in production at all. You’ll probably want to elect to look for ranges of status codes instead of a single value. Google up HTTP return codes and you’ll see why.
I will mention again, though, that the script probe is very interesting. If you know Tcl or have done any development, check these things out. You can do whatever you want with them instead of using the canned probes that come with your CSM. I suggest taking a look at Ivan Pepelnjak’s page for some insight into Tcl on Cisco devices.
I think you can take it from here. As always, comments are welcome.
Configuring Fault Tolerance on the CSM
Like (nearly) everything in the Cisco world, you can set up your CSM to fail over to another module when the primary dies a horrible death. You can have two in the same chassis or even have them in separate chassis — the process is the same no matter how you have it set up. Either way, you have a primary and a secondary module in fault tolerance (FT) mode.
First, we’ll establish a VLAN that the CSM will use to do its configuration and state syncing over. This is just an ordinary VLAN; there’s nothing special about it, really, but it should be dedicated for the CSM to use for syncing. Let’s randomly choose VLAN 83.
vlan 83
name CSM-Sync
You will, of course, have to do this on every switch that holds a CSM, so, if you’re using them in two different chassis, you’ll put the same VLAN on each making sure they can see each other through a trunk. Cisco recommends that you dedicate a trunk between the two switches for the sync VLAN in order to remove the chance of other traffic stepping on the sync packets, but I’m not convinced that’s necessary. Use your judgement on that one.
Back to it. Next, you need to decide on a FT group ID. This is similar to a HSRP group and lets you run multiple FT groups on the same VLAN. The group ID needs to be in the range of 1 to 256, so, since this is the first one, let’s just use 1. Get into config mode for the CSM that you want to be the primary and do this.
ft group 1 vlan 83
This takes you to the config-slb-ft prompt. Just like HSRP, we need to set priorities for each device and whether or not it should preempt, so let’s configure. Yes, we want to preempt, right? Let’s set the priorities to 100 and 90, too.
priority 100 alt 90
preempt
This sets the primary CSM to priority 100 and the secondary to 90; both will preempt.
What about configuring the secondary for FT? That’s easy. Go into CSM config mode on the secondary and enter the ft group 1 vlan 83 command. That’s it. The two CSMs will do a little arguing and come back as the primary and secondary. After that, all configuration is done on the primary, which is synced over to the secondary just like an ASA. Pretty cool, eh?
When configuring things like IP addresses, though, you’ll need to make provisions for the secondary with the alt directive (remember that one from the priority). I won’t go into much, but you’ll need it mostly when settings IPs to VLANs. Here’s an example of setting an IP address on client VLAN 100 for both the primary and secondary.
vlan 100 client
ip address 192.168.0.11 255.255.255.0 alt 192.168.0.12 255.255.255.0
Alright…one more thing. The configurations don’t sync automagically (at least not on my old version of code). If you make a change to the primary CSM, you’ll see an out-of-sync message when you look at the FT status.
Switch#sh mod csm X ft
FT group 1, vlan 83
This box is active
Configuration is out-of-sync
priority 100, heartbeat 1, failover 3, preemption is on
alternate priority 90
If the primary goes down now and the secondary takes over, the changes you just made won’t be reflected on the secondary. You fix this with the hw-module contentSwitchingModule X standby config-sync command (where X is the module slot in the chassis). Alternatively, you can just type hw c X s c as a shortcut. It’ll take a few minutes depending on your configuration, so check your logs for when it’s finished. Note that the secondary does not save the new configuration to its startup-config; you’ll have to log in and save that manually (or automatically through CiscoWorks or something) to save changes there.
Let me know if you have any questions and check out my page on getting output from Cisco’s fine mid-tier load balancer. :)
Setting Up System Logging on an IOS Device
I like logging on an IOS device. I like to look at the buffer and tell you that your interface went down 30 seconds ago. I like to look on the box and see that BGP with my Internet provider has been flapping since 02:13ET. I like to look and see that one of the other guys has been making changes to the gear all morning. I could go on and on.
There are lots of ways to monitor a Cisco box — SNMP polling, SNMP traps, show commands, etc. — but there’s nothing so handy as the log buffer. A show logging can provide you all sorts of information on things you do and don’t care about, so it’s important to know the destinations and levels when setting up logging.
There are four logging destinations.
- Console — logs to the console ports of the device
- Monitor — logs to any device or pseudo-device that’s in monitoring mode. The most common application is when you do a terminal monitor to see output of debugs.
- Buffer — logs to a memory device that let’s you see the log messages on demand. It has a finite size and scrolls old messages out after X bytes have been written.
- Host or Trap — logs to an external syslog server.
What’s the most important destination? There’s not one. I personally thing the syslog host is the most important since it allows you to log messages to disk on a server somewhere. The buffer is also important since it lets anyone with access see what’s going on with the device. Your mileage will vary depending on what you have set up.
There are eight logging levels as well.
- Debug – level 7
- Informational – level 6
- Notifications – level 5
- Warnings – level 4
- Errors – level 3
- Critical – level 2
- Alerts – level 1
- Emergencies – level 0
Wow. There’s some numbers. What does it mean? Every logging message comes with a level. If my CSM loses a RIP, it generates a level 6 message (%CSM_SLB-6-RSERVERSTATE) telling me. If I configure the box, I get a level 5 message (%SYS-5-CONFIG_I) saying that I’ve done so. If my router is on fire in the rack, I’ll get a level 0 message telling me. Now, when you configure a destination, you have to give a logging level, and every message at that level or below will be logged. If I set my logging buffer to 5, I’ll see the configuration message and the “Oh, the humanity!” message, but not the RIP failures. If I set it to 0, I only see the emergencies. If I set it to 7, I see everything.
Let’s do the configuration, then. After hours or research, you’ve decided to use a remote syslog server at 1.2.3.4 for warnings and the buffer for informational. Here’s what you’d do.
logging host 1.2.3.4
logging trap warnings
logging buffer information
It’s not that hard. You can even use the number instead of the words for the logging level if you would like. The same procedure holds true for the console and monitor mechanisms — logging <mechanism> <level>. Easy.
If you care, here’s what I usually run.
- Console — off. I’ve seen a console rendered unusable because the console was being obliterated with syslog messages. Not only is it an issue with being able to see what you’re typing when stuff is scrolling, but some older devices wind up using 100% CPU because they’re sending messages to the console.
- Monitor — debug. It doesn’t really log anything unless you do a term mon or something, and, in that case, I want to see my debugs.
- Buffer — informational, but it depends on the device. It lets me see all the messages except for debugs, which is probably just right for most routers and switches. If you’re switch is in a closet somewhere with users plugged directly into it, you may be flooded with up/down messages, so keep an eye out for stuff like that.
- Host or Trap — informational. Debug’s a little too much for the corporate environment, but, depending on how much disk space you have, you may be able to handle it.
There’s a lot more to syslog and log messages, so see this nifty Cisco page.
Using the Pipe in IOS
A lot of IOS commands give you a lot of information. Most of the time, though, it’s way too much information, and it sure would be nice to do some grep-like stuff on the output, right? Well, just like on Linux, you can use the pipe (|) to do such. That’s not a butt cheek, by the way.
The most useful function is the include directive. This is the equivalent of just plain grep on Linux, and will show you only lines that match a string that you give it. Say that you want to find what ports on your switch are down, but don’t want to grind through all the lines of a show ip interface brief. If you just pipe it to the include command followed by the word “down”, you’ll see something like this.
Switch#show ip interface brief | include down
GigabitEthernet0/4 unassigned YES unset down down
GigabitEthernet0/7 unassigned YES unset down down
GigabitEthernet0/17 unassigned YES unset down down
GigabitEthernet0/18 unassigned YES unset down down
GigabitEthernet0/19 unassigned YES unset down down
GigabitEthernet0/20 unassigned YES unset down down
You can also use the exclude directive, which is the same as a grep -v on Linux. I hope you figured out that this gives you all lines that don’t match the word, so, let’s use the exclude directive to found out what ports are down. How about we just ignore the lines that are up.
Switch#show ip interface brief | exclude up
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/4 unassigned YES unset down down
GigabitEthernet0/7 unassigned YES unset down down
GigabitEthernet0/17 unassigned YES unset down down
GigabitEthernet0/18 unassigned YES unset down down
GigabitEthernet0/19 unassigned YES unset down down
GigabitEthernet0/20 unassigned YES unset down down
Well, this won’t exactly give you all the ports that are down. What if the port is up/down?
What else can you use with the pipe? What if you want to look at the configurations of all the ports or interfaces on a box but don’t want to go through the config hitting the spacebar over and over. If you use the begin command, you’ll see the output beginning from the first match, so, using the string interface will show you the config starting at the first interface.
Switch#show running-config | begin interface
interface GigabitEthernet0/1
description Server
switchport access vlan 4
switchport mode access
spanning-tree portfast…
Another good one is the section command. It’s usually used on the output of show running-config and shows you sections of the config that match your string. Huh? If you want to see the BGP section of the configuration, you can do something like thing just to see that part of the configuration.
Router#show running-config | section bgp
router bgp 1
neighbor 1.1.1.1 remote-as 65000
neighbor 1.1.1.1 version 4
There are a few other commands for use with the pipe, so explore on your own. You might also want to check out regular expressions on the Cisco IOS if you want to match more than just simple text.