Aaron’s Worthless Words

That’s an awesome title, eh?  I’ve mentioned a router-on-a-stick before but not a data-center-on-a-stick (DCOAS).  This is one of those Cisco terms I ran across a while ago and is a group of servers sort of sticking out on their own behind a load balancer and/or firewall.  Connections to and from the server group go through a single spoke — kinda like stubby routing.  Here’s a pretty picture.

To configure this type of setup on the CSM, assuming you’re running in router mode, you just define a server and client VLAN pair along with an alias IP on the server VLAN.  The servers on that VLAN point to the alias as their gateway for return traffic, and the CSM handles the VIP/RIP conversion.

What if you have more than one server/client VLAN pair, though?

You set up your VLANs and your server VLAN alias for both sets and all is well, right?  Not really.  The CSM isn’t a router and doesn’t do a very good job at routing.  If you were to send traffic from “Other Stuff” to a VIP on VLAN101, everything works great.  If, however, one of those servers initiates a connection, the traffic will come out of the client VLAN with the lowest VLAN ID.  If you have VLANs 1 and 2 for the top pair, traffic from all servers will come out VLAN 1.  Very big problem if you have a firewall that’s expecting traffic on another interface.

How do you get around it, then?  You have to trick the CSM by generating new catch-all vservers on each server VLAN to “load balance” the gateway for each VLAN.  Let’s look at the configs.

Here are the serverfarms.

serverfarm VLAN1-SF
no nat server
no nat client
real 1.1.1.1
inservice

serverfarm VLAN101-SF
no nat server
no nat client
real 1.1.101.1
inservice

And the vservers.

vserver VLAN1-VS
virtual 0.0.0.0 0.0.0.0 any
vlan 2
serverfarm VLAN1-SF
inservice

vserver VLAN101-VS
virtual 0.0.0.0 0.0.0.0 any
vlan 102
serverfarm VLAN101-SF
inservice

When a server makes a new connection to a database or whatnot, it sends the traffic to the alias you already configured on the server VLAN.  When the CSM receives that traffic through the alias, this new vserver takes over since it’s set for any protocol and port on any IP.  This vserver uses a serverfarm that contains the gateway for the appropriate client VLAN — in our case, the IP of the firewall — so now every connection from the server will exit an appropriate VLAN instead of out the lowest ID.

Confusing?  Yes.  Should be fixed?  Yes.  Is already fixed?  I have no idea, but it’s not as of 12.2 and some change.

• Leave a comment... (0)

On the CSM, you can configure a vserver to use a main and backup serverfarm which is used if a serverfarm is toast.  If all the RIPs in the main farm are out-of-service, the CSM will start to treat the backup farm just as if it’s configured to be the main one.  Once one or more of the main farm RIPs have recovered, the CSM reverts back and uses those again.  "Give me an example when I’d use it!," you say?  Since the CSM is made for HTTP connections, we’ll assume that you are using it for such. 

The example I’ve found over and over again involves HTTP redirect servers.  If the whole main serverfarm is dead, you can send requests to a set of redirect servers.  Once connections land there, you can send the user to another site, such as your west coast center, your European center, your main corporate page, or wherever.

You can also send traffic to a server that does nothing but show an error page.  Let’s say you set up a group of servers that only display a "We’re sorry, but we suck and can’t handle your request" page.  If your main serverfarm is unavailable, the user won’t get the page they’re looking for, but it’s better than just timing out, right?  (Remind me to do this for some customers.)

How about if you’re moving your serverfarm to another rack and you have to do it all at once (for some unknown reason)?  You can use the backup serverfarm to move traffic to another set of server (say, the dev boxes) until you get the main serverfarm back online.

These backup farms are quite easy to configure.  When you assign the serverfarm to the vserver, you simply give it a backup like this.

vserver TEST-VS
serverfarm MAIN-SF backup BACKUP-SF

You can also use a backup on a policy.

policy TEST-POL
serverfarm POL-SF backukp BACKUP-SF

Bon appetit.

• Leave a comment... (2)

The CSM is pretty bad little box.  It not only watches layer 4 items like TCP connections, but also talks HTTP, which you can use to do some custom, or policy-based, load balancing.

Policies are the objects that make custom balancing work.  Like everything else (it seems) on the CSM, a policy is an object made up of other objects — maps and serverfarms.  A map matches patterns based on a number of things including the URL and HTTP header values, while the serverfarm directive tells where to send traffic that matches the map.  If, for example, you want to send all requests with “/admin” in the URL to a management server instead of the regular web servers, you can do it with a policy.

Let’s try one.  We first build our serverfarm and vserver for normal traffic.  This configuration will simply serve HTTP on 1.1.1.1 via two servers in a serverfarm.

serverfarm TEST-FARM
 real 192.168.0.101
  inservice
 real 192.168.0.102
  inservice

vserver TEST-VS
 virtual 1.1.1.1 tcp http
 vlan 1
 serverfarm TEST-FARM
 inservice

Now let’s set up our policy-based load balancing for this vserver.  Let’s say that we want to send all traffic with “/admin” in the URL to go to the management server at 192.168.0.150, so we first create a serverfarm with the management server in it.

serverfarm TEST-MGMTFARM
 real 192.168.0.150
  inservice

Next, we create a map that matches the URL.

map TEST-MAP url
 match protocol http url */admin*

Notice the pretty wildcards?  If you don’t include those, the URL would have to match exactly for the map to apply.  Since most apps don’t actually go to the same URL over and over, we have to use wildcards to make sure that everything matches.  Obviously, in practice, this can get complicated depending on what you’re trying to match.

Next, we create the policy itself.  Basically, we just combine the map and the serverfarm into a new object.

policy TEST-POL
 url-map TEST-MAP
 serverfarm TEST-MGMTFARM

With me so far?  Good.  There’s only one thing left to do — apply the policy to the vserver.

vserver TEST-VS
 slb-policy TEST-POL

Everytime the CSM gets a request to the virtual IP of 1.1.1.1 on HTTP, it will check the URL for the string “/admin”, and, if it’s in the request, it will send it over to the management server.  If it doesn’t match, it simply goes to the main serverfarm just as if the policy wasn’t even applied.  If you do a “show mod csm X vservers” now, you’ll see the policy listed under the vserver we just made along with stats on the number of packets that were matched.

There are caveats.  The CSM can only support a limited number of policies for each vserver.  According to a TAC case I opened a while back, you can only have 10 policies configured per vserver, so keep that in mind when designing everything out.  There’s also a 30k memory limit for policies on a serverfarm; I have no clue how to calculate how much memory a policy is using, but using wildcards definitely adds to the memory footprint.

Send me any questions since I still haven’t found anyone who uses the CSM. 

• Leave a comment... (1)

My buddy told me that my site is the only place on the web with documentation on the Cisco Content Switching Module (CSM). I also noticed a few months ago that every TAC case I’ve opened on the CSM has been handled by the same guy. I seriously think that the only people in the world that really know about these things are me and him. Cool. I better get some more content up.

The CSM is configured and controlled by the IOS running on the 6500. Unlike the FWSM, it is not independent from the switch’s operation, which is sometimes good. One good thing out of that is the fact that you can pull stats and stuff from the IOS without having to session or SSH to another module. The bad part of that, though, is that the commands wind up being long.

To start with, all show and clear commands start like this.

show|clear module contentSwitchingModule <SLOT> <COMMAND>

Do you see how long that command is? And we didn’t even tell it what we wanted to do yet. You can use the auto-complete, though, f you’re lazy like I am.

sho|cl mod csm <SLOT> <COMMAND>

Yes, it takes csm instead of contentSwitchingModule. That’s not in the contextual help. Heh.

SLOT is where the module is in the chassis, so, if your CSM is in slot 8, you…can figure it out. COMMAND is what you want to do, right? Yeah…I won’t insult your intelligence.

What are some show commands? There’s a lot of them, but here’s some I use every day. My production CSMs are in slot 3, so I’ll just use that for the slot.

• show mod csm 3 arp : Shows you the ARP table (duh!). This is good to see if the CSM can contact the real servers or the gateways properly.
• show mod csm 3 conns : Shows the current connections through the CSM. This shows you what client IP is connected to what virtual IP and on what real IP that connection lands.
• show mod csm 3 ft : Fault tolerance. This shows what your FT VLAN and status is. It also shows if your secondary configs are out-of-sync with the primary.
• show mod csm 3 reals : Shows all the real servers involved in all serverfarms along with the weight, state, and current number of connections. This shows you a lot of information that could be helpful in troubleshooting a problem. Look for FAILED, PROBE_FAILED, or OUTOFSERVICE; these are bad.
• show mod csm 3 serverfarms : Shows your setup and status of the serverfarms. Also great for troubleshooting.
• show mod csm 3 vservers : Shows the IP, VLAN, and state of your vservers along with the current number of connections.
• show mod csm 3 vlan : Shows the VLAN ID, subnet and mask, and VLAN type (server, client, FT)

Be sure to use your contextual help for more detail on these commands; most of them actually can get very specific. For example, ft shows your status, but ft detail shows your message counts, resets, and other good and bad things that deal with the fault tolerance.

How about something to clear?

• clear mod csm 3 connections : Clears the connections through the CSM. This is probably very close to clear conns on a PIX, FWSM, or ASA, so be careful not to kick everybody off.
• clear mod csm 3 arp-cache : Clears ARP
• clear mod csm 3 ft active : Forces the primary to fail over.
• clear mod csm 3 counters : Do you have to ask?

Again, these guys need to be explored with the question mark to get the full effect.

Since I’ve established myself as the long authority in the world on the CSM [sic], drop a comment with any questions.

• Leave a comment... (3)

I had an article a few weeks ago about the [tag]Cisco[/tag] [tag]CSM[/tag], which is a load-balancer module for the 6500 series switches. This thing is a pretty good device, but monitoring the connections to each VIP and RIP is not very straightforward. If you have an [tag]SNMP[/tag] monitoring system like Cacti or MRTG, you need to know the [tag]OID[/tag] to [tag]monitor[/tag], but it doesn’t work like anything else in the world.

Let’s start with the OID for the vserver. First, there’s the base OIDs that you can look up on CCO. This is just standard SNMP stuff that Cisco defined long ago. The slot number that the CSM is in is added to that base OID. You then have to add the length of the vserver name — don’t ask me…I don’t know. Next comes the really stupid part — you have to take the names of the vserver and get the ASCII values of every character in the name and add each to the end to get the full OID. Yes, it’s that stupid.

<BASE VSERVER CONN OID>.<SLOT>.<LENGTH>.<VSERVER NAME>

The serverfarm OID is pretty close — the base OID and slot. You then add the length of the serverfarm name along with the ASCII values of the serverfarm name (quite like we did for the vserver). Finally, the only part that might make sense, you take the IP of the real server and add it to the end with the instance of “0″. Again, I have no clue why it’s like this or what Cisco was trying to do. It’s terrible.

<BASE SF CONN OID>.<SLOT>.<LENGTH>.<SF NAME>.<REAL IP>.0

How about an example. Let’s say you have a vserver called VSERVER1 that you want to monitor. This vserver is configured to use the serverfarm FARM1, which has two reals of 192.168.1.101 and 192.168.1.102 that you also want to monitor. You also know that the CSM is in slot 3. The base OID for vserver connections is .1.3.6.1.4.1.9.9.161.1.4.1.1.17; the base OID for serverfarm connections is .1.3.6.1.4.1.9.9.161.1.3.1.1.5. This all gives you this:

VSERVER: .1.3.6.1.4.1.9.9.161.1.4.1.1.17.3.8.86.83.69.82.86.69.82.49
RIP1: .1.3.6.1.4.1.9.9.161.1.3.1.1.5.3.5.70.65.82.77.49.192.168.1.101.0
RIP2: .1.3.6.1.4.1.9.9.161.1.3.1.1.5.3.5.70.65.82.77.49.192.168.1.102.0

Did I mention it’s overly-complicated and terrible? It’s actually so bad that I just wrote a Perl script to do it for me because, for God’s sake, I’m not doing that by hand. Let me know if you need any help with it.

• (0)

Cisco’s Content Switching Module (CSM) is an application accelerator. Or is it an application networking service module? I hate those fancy buzzwords — it’s a load balancer. It’s a module for the 6500 series switches that lets you load balance services in any VLAN and can also be set up for high-availability. I could go on for a while about the features, but let’s keep it simple for now. A short tutorial, if you will.

Configurations to the CSM are made in the IOS itself. It’s a little different than other modules where you have to session over or SSH to the module. The only trick is that you go into CSM mode like this.

Switch#conf t
Switch#(conf)module csm 3
Switch#(conf-csm)

So, what’s first? The first thing is setting up the VLANs. I’ll assume you’ve already got the VLANs set up on the switch part (like you always do to get everything talking). I’m also assuming that you want to use routed processing mode (don’t ask).

The VLANs have to be configured as either server or client — the servers themselves sit on the server VLAN, and the virtual IP addresses sit on the client VLAN (this is where the client comes in to access the service). Let’s look at the client VLAN first. You obviously need to have the VLAN ID. You also need to configure the IP address for that VLAN to put a layer-3 address and a gateway, which is what the CSM uses to send return packets back to the client from the server. It’ll make sense if you think about it for a bit.

Switch#(conf-csm) vlan 1 client
Switch#(conf-csm-vlan-client) description My first client VLAN
Switch#(conf-csm-vlan-client) ip address 10.0.0.1 255.255.255.0
Switch#(conf-csm-vlan-client) gateway 10.0.0.254

Ok, then. Let’s do the server VLAN. You need the VLAN ID and the IP address the same as you did for the client VLAN. The difference here is that you don’t configure a gateway. Whereas the IP address on the client VLAN is just to have a leg into that network, the IP on the server VLAN is generally used as the default gateway for the server on that network. By the way, you don’t have to have the servers on that VLAN directly; the servers simply have to be accessed from that VLAN, so you can put servers a few hops away.

Switch#(conf-csm) vlan 2 server
Switch#(conf-csm-vlan-server) description My first server VLAN
Switch#(conf-csm-vlan-server) ip address 10.0.1.1 255.255.255.0

No problems so far, eh? I hope not because here’s the complicated part. Now we’re going to configure a new load-balanced IP that uses a couple of web servers to server pages. Let’s start with configuring the serverfarm, which is a group of servers that serve the same function.

Switch#(conf-csm) [tag]serverfarm[/tag] WEBFARM
Switch#(conf-csm-sfarm) real 10.0.1.11
Switch#(conf-csm-real) inservice
Switch#(conf-csm-real) real 10.0.1.12
Switch#(conf-csm-real) inservice

I think you can figure out what that config does. Yes, the “inservice” is required. The reals are often referred to as real IPs, or RIPs.

Now we need to configure the vserver, which is an object that includes the virtual IP (VIP), protocol, port, serverfarm, VLAN, etc. This is the main guy and, if you do some more advanced stuff, you’ll have a vserver config longer than the iPhone line. Look first, then I’ll tell you what we’re doing.

Switch#(conf-csm) vserver VIRTSERVER
Switch#(conf-csm-vserver) [tag]virtual[/tag] 10.0.0.11 tcp 80
Switch#(conf-csm-vserver) serverfarm WEBFARM
Switch#(conf-csm-vserver) vlan 1
Switch#(conf-csm-vserver) inservice

It’s pretty easy to see what we’re doing. We’re creating a VIP of 10.0.0.11 that serves traffic sent to TCP/80. When traffic is sent to this VIP on that port, the CSM will round-robin load-balance using serverfarm WEBFARM. We also have told the CSM to only answer requests to this VIP if they come from VLAN 1 (If you don’t include the VLAN line, the CSM will answer on all VLANs that it’s configured to use. In my opinion, this is a design problem and it’s fixed in the ACE.).

Let’s do a quick packet trace to be sure we’re all clear.

You open your browser and ask for the page at http://10.0.0.11. Your packets go through the network and eventually land on VLAN 1. The CSM knows that it needs to answer for it, so it accepts the packet. Since there is an existing vserver on TCP/80, it send the packet to its SLB (server load balancing) algorithms. Through the SLB magic, the CSM picks what server in the serverfarm to use and changes the destination address of the packet to the RIP and passes it on.

On the way back, the RIP sends the packets back to the server VLAN address. The CSM knows that we’re accessing the VIP, so it changes the source address to the VIP and passes it on to the gateway for VLAN 1. The packet eventually lands back on your machine, and you never knew anything happened to the packet along the way.

That’s the basics. We went through and configured the VLANs and a simple vserver. The CSM can do dozens of more things, but that’s for another time.

• Read comments... (2)