Aaron’s Worthless Words

On the CSM, you can configure a vserver to use a main and backup serverfarm which is used if a serverfarm is toast.  If all the RIPs in the main farm are out-of-service, the CSM will start to treat the backup farm just as if it’s configured to be the main one.  Once one or more of the main farm RIPs have recovered, the CSM reverts back and uses those again.  "Give me an example when I’d use it!," you say?  Since the CSM is made for HTTP connections, we’ll assume that you are using it for such. 

The example I’ve found over and over again involves HTTP redirect servers.  If the whole main serverfarm is dead, you can send requests to a set of redirect servers.  Once connections land there, you can send the user to another site, such as your west coast center, your European center, your main corporate page, or wherever.

You can also send traffic to a server that does nothing but show an error page.  Let’s say you set up a group of servers that only display a "We’re sorry, but we suck and can’t handle your request" page.  If your main serverfarm is unavailable, the user won’t get the page they’re looking for, but it’s better than just timing out, right?  (Remind me to do this for some customers.)

How about if you’re moving your serverfarm to another rack and you have to do it all at once (for some unknown reason)?  You can use the backup serverfarm to move traffic to another set of server (say, the dev boxes) until you get the main serverfarm back online.

These backup farms are quite easy to configure.  When you assign the serverfarm to the vserver, you simply give it a backup like this.

vserver TEST-VS
serverfarm MAIN-SF backup BACKUP-SF

You can also use a backup on a policy.

policy TEST-POL
serverfarm POL-SF backukp BACKUP-SF

Bon appetit.

• Leave a comment... (2)

Here’s a quick one for you. In Dynagen, if you want to load a configuration when you first fire up the router instance, you can use the cnfg tag in your NET file like this.

cnfg = /home/jac/labs/cfg/R0.cfg

If you put that in your dynagen NET file under a router, the contents of that file will be loaded into the router configuration when it’s brought up. This is great if you already have a configuration to use in another lab or if you want to load a basic configuration on startup. Please be warned, though; if you make changes to your router instance via the CLI and restart dyangen, the configuration changes you made will be gone.  Be sure to remove that line from the NET before you restart dynagen.

• Leave a comment... (3)

The CSM is pretty bad little box.  It not only watches layer 4 items like TCP connections, but also talks HTTP, which you can use to do some custom, or policy-based, load balancing.

Policies are the objects that make custom balancing work.  Like everything else (it seems) on the CSM, a policy is an object made up of other objects — maps and serverfarms.  A map matches patterns based on a number of things including the URL and HTTP header values, while the serverfarm directive tells where to send traffic that matches the map.  If, for example, you want to send all requests with “/admin” in the URL to a management server instead of the regular web servers, you can do it with a policy.

Let’s try one.  We first build our serverfarm and vserver for normal traffic.  This configuration will simply serve HTTP on 1.1.1.1 via two servers in a serverfarm.

serverfarm TEST-FARM
 real 192.168.0.101
  inservice
 real 192.168.0.102
  inservice

vserver TEST-VS
 virtual 1.1.1.1 tcp http
 vlan 1
 serverfarm TEST-FARM
 inservice

Now let’s set up our policy-based load balancing for this vserver.  Let’s say that we want to send all traffic with “/admin” in the URL to go to the management server at 192.168.0.150, so we first create a serverfarm with the management server in it.

serverfarm TEST-MGMTFARM
 real 192.168.0.150
  inservice

Next, we create a map that matches the URL.

map TEST-MAP url
 match protocol http url */admin*

Notice the pretty wildcards?  If you don’t include those, the URL would have to match exactly for the map to apply.  Since most apps don’t actually go to the same URL over and over, we have to use wildcards to make sure that everything matches.  Obviously, in practice, this can get complicated depending on what you’re trying to match.

Next, we create the policy itself.  Basically, we just combine the map and the serverfarm into a new object.

policy TEST-POL
 url-map TEST-MAP
 serverfarm TEST-MGMTFARM

With me so far?  Good.  There’s only one thing left to do — apply the policy to the vserver.

vserver TEST-VS
 slb-policy TEST-POL

Everytime the CSM gets a request to the virtual IP of 1.1.1.1 on HTTP, it will check the URL for the string “/admin”, and, if it’s in the request, it will send it over to the management server.  If it doesn’t match, it simply goes to the main serverfarm just as if the policy wasn’t even applied.  If you do a “show mod csm X vservers” now, you’ll see the policy listed under the vserver we just made along with stats on the number of packets that were matched.

There are caveats.  The CSM can only support a limited number of policies for each vserver.  According to a TAC case I opened a while back, you can only have 10 policies configured per vserver, so keep that in mind when designing everything out.  There’s also a 30k memory limit for policies on a serverfarm; I have no clue how to calculate how much memory a policy is using, but using wildcards definitely adds to the memory footprint.

Send me any questions since I still haven’t found anyone who uses the CSM. 

• Leave a comment... (1)

Here’s a handy list of ACL entries to allow your devices to speak routing protocols, availability protocols, and some other stuff. We’ll assume you have ACL 101 applied to your Ethernet inbound; your Ethernet has an IP of 192.168.0.1.

• BGP : Runs on TCP/179 between the neighbors

access-list 101 permit tcp any host 192.168.0.1 eq 179

• EIGRP : Runs on its own protocol number from the source interface IP to the multicast address of 224.0.0.10

access-list 101 permit eigrp any host 224.0.0.10

• OSPF : Runs on its own protocol number from the source interface IP to the multicast address of 224.0.0.5; also talks to 224.0.0.6 for DR/BDR routers

access-list 101 permit ospf any host 224.0.0.5
access-list 101 permit ospf any host 224.0.0.6

• HSRP : Runs on UDP from the source interface IP to the multicast address of 224.0.0.2. I’ve seen in the past that it runs on UDP/1985, but I didn’t find any evidence of that in a quick Google for it. Can someone verify?

access-list 101 permit udp any host 224.0.0.2

• RIP : Runs on UDP/520 from the source interface IP to the multicast address of 224.0.0.9

access-list 101 permit udp any host 224.0.0.9 eq 520

• VRRP : Runs on its own protocol number from the source interface IP to the multicast address of 224.0.0.18

access-list 101 permit 112 any host 224.0.0.18

• GLBP : Runs on UDP from the source interface IP to the multicast address of 224.0.0.102

access-list 101 permit udp any host 224.0.0.102

• DHCPD (or bootps) : Runs on UDP/67 from 0.0.0.0 (since the client doesn’t have an address yet) to 255.255.255.255 (the broadcast).

access-list 101 permit udp any host 255.255.255.255 eq 67
If anyone else has one to add, do so in the comments.

• Leave a comment... (9)

My buddy told me that my site is the only place on the web with documentation on the Cisco Content Switching Module (CSM). I also noticed a few months ago that every TAC case I’ve opened on the CSM has been handled by the same guy. I seriously think that the only people in the world that really know about these things are me and him. Cool. I better get some more content up.

The CSM is configured and controlled by the IOS running on the 6500. Unlike the FWSM, it is not independent from the switch’s operation, which is sometimes good. One good thing out of that is the fact that you can pull stats and stuff from the IOS without having to session or SSH to another module. The bad part of that, though, is that the commands wind up being long.

To start with, all show and clear commands start like this.

show|clear module contentSwitchingModule <SLOT> <COMMAND>

Do you see how long that command is? And we didn’t even tell it what we wanted to do yet. You can use the auto-complete, though, f you’re lazy like I am.

sho|cl mod csm <SLOT> <COMMAND>

Yes, it takes csm instead of contentSwitchingModule. That’s not in the contextual help. Heh.

SLOT is where the module is in the chassis, so, if your CSM is in slot 8, you…can figure it out. COMMAND is what you want to do, right? Yeah…I won’t insult your intelligence.

What are some show commands? There’s a lot of them, but here’s some I use every day. My production CSMs are in slot 3, so I’ll just use that for the slot.

• show mod csm 3 arp : Shows you the ARP table (duh!). This is good to see if the CSM can contact the real servers or the gateways properly.
• show mod csm 3 conns : Shows the current connections through the CSM. This shows you what client IP is connected to what virtual IP and on what real IP that connection lands.
• show mod csm 3 ft : Fault tolerance. This shows what your FT VLAN and status is. It also shows if your secondary configs are out-of-sync with the primary.
• show mod csm 3 reals : Shows all the real servers involved in all serverfarms along with the weight, state, and current number of connections. This shows you a lot of information that could be helpful in troubleshooting a problem. Look for FAILED, PROBE_FAILED, or OUTOFSERVICE; these are bad.
• show mod csm 3 serverfarms : Shows your setup and status of the serverfarms. Also great for troubleshooting.
• show mod csm 3 vservers : Shows the IP, VLAN, and state of your vservers along with the current number of connections.
• show mod csm 3 vlan : Shows the VLAN ID, subnet and mask, and VLAN type (server, client, FT)

Be sure to use your contextual help for more detail on these commands; most of them actually can get very specific. For example, ft shows your status, but ft detail shows your message counts, resets, and other good and bad things that deal with the fault tolerance.

How about something to clear?

• clear mod csm 3 connections : Clears the connections through the CSM. This is probably very close to clear conns on a PIX, FWSM, or ASA, so be careful not to kick everybody off.
• clear mod csm 3 arp-cache : Clears ARP
• clear mod csm 3 ft active : Forces the primary to fail over.
• clear mod csm 3 counters : Do you have to ask?

Again, these guys need to be explored with the question mark to get the full effect.

Since I’ve established myself as the long authority in the world on the CSM [sic], drop a comment with any questions.

• Leave a comment... (3)

If you’ve never used TCPDump before, you’re missing out on one of the best parts of being a network guy — pointing fingers at everyone else.

TCPDump is an open-source app that copies packets on a machine’s NIC to screen or to file. TCPDump is typically a Linux/Unix app; in the Windows world, TCPDump is replaced by WinDump or Ethereal, now known as Wireshark. It’s a must-know for network dude(tte)s since it lets you capture the packets that a machine is generating. An app may be documented to work one way, but I’ve seen many times where the documentation is out-of-date or just wrong, and I’ve had to look at captures to see what it was actualy doing. I used it one time way back when a developer told me the switch was changing his HTTP POST to an HTTP GET; I captured the packets he was sending, pointed to the GET, and never answered a phone call from him ever again.

Am I angry today?

Here’s a more down-to-earth example. How many times have you been asked to open an ACL for a host, but the requester didn’t know the destination IP or the service port? For me, this happens as least twice a week, and I use TCPDump to figure out what the requester is trying to do. Here’s a typical conversation.

Requester: I’ve installed an app on my Linux server to help me zip up my pants, but it’s not working.
Me: Well, I’ve told you on at least 6 occasions that the firewall is going to block connections unless you tell us to open it up.
R: Oh, yeah. I’m still used to the way we did things in 1988. Can you open that up for me?
M: Can you put in a ticket like I always ask you to do?
R: Sure. What do you need it to say?
M: The same thing the last 6 tickets said — source IP, protocol, port, destination IP.
R: I don’t know all that. I just know it tries to connect to my home IP via HTTP, but I don’t know what that IP is.
M: Then how are you going to zip up your pants?
R: I guess I won’t. Can you help me find that information?

Wow. I’ve got some penned up aggression, don’t I?

To find out what’s used to help this user zip up his pants, we can just run TCPDump (as root since you need access to the NIC) to capture some packets.

sudo /usr/sbin/tcpdump

Hmmm…I bet you get a lot of stuff. You’re looking at all the packets that are flying in and out of the box, including all sorts of stuff like DNS requests and your SSH session. This is where TCPDump tries to shine, though; it’s got a very powerful capture filtering system, which can get complicated at times. You can do all sorts of filtering on the capture, but the most common are the host and port filters, which, like all the other filters, can be strung together to make great huge chains of filters.  Since we know it’s trying to connect to an  HTTP server, so we can start by showing only traffic on port 80.

sudo /usr/sbin/tcpdump port 80

Much better. Here’s some output.

[jac@finland ~]$ sudo /usr/sbin/tcpdump port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
18:54:36.690198 IP 192.168.70.129.8736> yourmama.com.http: S 1864432324:1864432324(0) win 5840 <mss 1460,sackOK,timestamp 383978802 0,nop,wscale 3>
18:54:36.694046 IP yourmama.com.http > 192.168.70.129.8736: S 1931495854:1931495854(0) ack 1864432325 win 64240 <mss 1460>
18:54:36.694110 IP 192.168.70.129.8736> yourmama.com.http: . ack 1 win 5840
18:54:38.172982 IP 192.168.70.129.v> yourmama.com.http: P 1:16(15) ack 1 win 5840
18:54:38.173947 IP yourmama.com.http > 192.168.70.129.8736: . ack 16 win 64240

It looks like it’s using the hostname yourmama.com. Now we can just look up what that IP is and generate the firewall rule for it (or, if you’re like me, just give is a -nn flag to not look up IPs or service ports).
What else can you do with TCPDump? You can check out the man pages for how to do this stuff or just ask me nicely.

• Capture the packet headers to file for review later
• Capture the whole packet to screen or file
• Read in a packet capture from file
• Listen on a particular interface (like eth3 instead of eth0)
• Capture X number of packets
• Filter based on all sorts of stuff including IP address, port, protocol, MAC address, IPv6 multicast address, VLAN from 802.1Q packet, and all sorts of other good stuff

• Leave a comment... (0)