Getting Something Out of the CSM
My buddy told me that my site is the only place on the web with documentation on the Cisco Content Switching Module (CSM). I also noticed a few months ago that every TAC case I’ve opened on the CSM has been handled by the same guy. I seriously think that the only people in the world that really know about these things are me and him. Cool. I better get some more content up.
The CSM is configured and controlled by the IOS running on the 6500. Unlike the FWSM, it is not independent from the switch’s operation, which is sometimes good. One good thing out of that is the fact that you can pull stats and stuff from the IOS without having to session or SSH to another module. The bad part of that, though, is that the commands wind up being long.
To start with, all show and clear commands start like this.
show|clear module contentSwitchingModule <SLOT> <COMMAND>
Do you see how long that command is? And we didn’t even tell it what we wanted to do yet. You can use the auto-complete, though, f you’re lazy like I am.
sho|cl mod csm <SLOT> <COMMAND>
Yes, it takes csm instead of contentSwitchingModule. That’s not in the contextual help. Heh.
SLOT is where the module is in the chassis, so, if your CSM is in slot 8, you…can figure it out. COMMAND is what you want to do, right? Yeah…I won’t insult your intelligence.
What are some show commands? There’s a lot of them, but here’s some I use every day. My production CSMs are in slot 3, so I’ll just use that for the slot.
- show mod csm 3 arp : Shows you the ARP table (duh!). This is good to see if the CSM can contact the real servers or the gateways properly.
- show mod csm 3 conns : Shows the current connections through the CSM. This shows you what client IP is connected to what virtual IP and on what real IP that connection lands.
- show mod csm 3 ft : Fault tolerance. This shows what your FT VLAN and status is. It also shows if your secondary configs are out-of-sync with the primary.
- show mod csm 3 reals : Shows all the real servers involved in all serverfarms along with the weight, state, and current number of connections. This shows you a lot of information that could be helpful in troubleshooting a problem. Look for FAILED, PROBE_FAILED, or OUTOFSERVICE; these are bad.
- show mod csm 3 serverfarms : Shows your setup and status of the serverfarms. Also great for troubleshooting.
- show mod csm 3 vservers : Shows the IP, VLAN, and state of your vservers along with the current number of connections.
- show mod csm 3 vlan : Shows the VLAN ID, subnet and mask, and VLAN type (server, client, FT)
Be sure to use your contextual help for more detail on these commands; most of them actually can get very specific. For example, ft shows your status, but ft detail shows your message counts, resets, and other good and bad things that deal with the fault tolerance.
How about something to clear?
- clear mod csm 3 connections : Clears the connections through the CSM. This is probably very close to clear conns on a PIX, FWSM, or ASA, so be careful not to kick everybody off.
- clear mod csm 3 arp-cache : Clears ARP
- clear mod csm 3 ft active : Forces the primary to fail over.
- clear mod csm 3 counters : Do you have to ask?
Again, these guys need to be explored with the question mark to get the full effect.
Since I’ve established myself as the long authority in the world on the CSM [sic], drop a comment with any questions.

about 1 year ago
Said in the voices from the guys in the Guinness commercials… “BRILLIANT!” This is good stuff, keep it coming!
about 1 year ago
Thanks, Clint. I’m trying to get back on track.
about 1 year ago
Hey Aaron,
Thanks for teh great info. I don’t know if this is the right place to post this but here goes.
I understand the basic functions of the CSM but when traffic comes into a VIP and then to a farm, how do you get the CSM to pass the source IP to the destination? Wouldn’t that info be of some value?
about 1 year ago
Hi, Droudpeyma. Thanks for commenting.
In the serverfarm configuration, there are two lines that deal with NAT — “nat client” and “nat server”. By default, client (source) NATting is off, and server (destination) NATting is on, so the source IP of the connection inbound to the serverfarm does not get changed before it’s passed on to the servers in the farm.
Here’s what the source and destination of the packets are at various steps in the process. Let’s assume the source IP is 12.34.56.78, the VIP is 1.1.1.1, and the serverfarm only has one server with the IP of 2.2.2.2. Assume that no other NATting is done in our little setup.
The packet leaves the client:
S: 12.34.56.78, D: 1.1.1.1
The CSM receives the packet and passes it on to the RIP:
S: 12.34.56.78, D: 2.2.2.2
The packet lands on the server:
S: 12.34.56.78, D: 2.2.2.2
The server generates a return packet:
S: 2.2.2.2, D: 12.34.56.78
The CSM gets the packet and sends it back to the client:
S: 1.1.1.1, D: 12.34.56.78
The client receives the packet:
S: 1.1.1.1, D: 12.34.56.78
Hope that helps. Let me know if it doesn’t.
about 1 year ago
Aaron,
Thanks so much for the info. I understand now why our farms are showing the source IP to be of the VIP. At the moment the farms are assigned to a NATPOOL on the client side. When that is removed the connection to the site breaks. Is there something else that needs to be done on the (client) Nat in order to restore the connection?
Thanks,
David
about 1 year ago
Hey, David.
I think if you turn off cleint NAT, you should be alright. Just go into your serverfarm for the VIP and do a “no nat client”. After such, you should see the clients appear unNATted.
about 1 year ago
one more very (at least for me) useful command run from switch# level
“show run mod X”, where X is slot number where CSM resides
it shows only configuration block for the csm
works with other modules as well
about 8 months ago
Hi Aaron, I did find useful information hear about CSM. Thanks very much.
My question is that doing this command ’show mod csm 3 ft’, the output shows this line
‘Configuration is out-of-sync’
I wonder what does it mean, how can I get configuration “in sync” at the active and standby CSM.
Regards,
D’Halmar.
about 8 months ago
Thanks for reading, Dhalmar.
The out-of-sync status means that your secondary CSM doesn’t have the same version of the configuration on the primary. I should have put this in my article on configuring fault tolerance, but you can just do a
I think that’s the command, but I don’t have a CSM in front of me to verify. I usually just do a
This command is required to sync the configs between CSMs and forces the current active configuration to replicate to the secondary. That should clear that right up.
about 3 months ago
Hello Aaron,
excellant website, alot of very useful info, i’ve been using it on and off now for a few years, I was wondering if you had came accross of an issue we are having. We have various server farms and need on occasion to clear the connections on a real server in the farm which can be done by using the command “clear mod csm * connections real *.*.*.*.
When we do this it seems to clear the connections for the whole server farm as opposed to just the real server. This is a problem because it is running in a production environment and can involve clearing quite a number of thousand connections. We are running version 3.2(1) in a 6500 running in native mode. If you need anymore info let me know.
Thanks in advance Aaron, keep up the good work.
Eamon
about 3 months ago
Thanks for reading, Eamon. I always appreciate input from the community.
I did some experimentation on a CSM running 4.2(4), and I was not able to recreate what you described. I selected several serverfarms, picked a real in each, and ran the “clear mod csm X conn real A.A.A.A” command on them. The connections to the chosen real cleared, and the rest of the farm didn’t notice anything. I looked through the bugs for the CSM and couldn’t find anything of interest.
Does it happen to any real you try to clear or just a handful?
about 2 months ago
It must be something to do with the version were running “3.2(1)”. We rebooted the csm and the problem disappeared, we were able to reset connections to one real server for a while after the reboot but it has started happening again, we are now back to if we try to reset connections on a particular real server it resets connections to the whole farm.
The issue happens all, does not seem to matter which server you select and it clears all connections to the serverfarm.
Thanks for your help Aaron
about 2 months ago
Sorry Aaron,
The issue happens with all reals, does not matter which server you select
about 2 months ago
I don’t know what to tell you, Eamon. It obviously shouldn’t do that, and I can’t find a bug that matches what you’re seeing. There is an upgrade to 4.3(3) on Cisco’s website, but I would open a ticket with the TAC before going down that path.
Let me know how it goes.
about 2 months ago
Hi Aaron,
To monitor ft state of 2 CSMs on 2 chassis via SNMP is it required that SNMP trap for CSM ft state should be enabled on the Cisco 6509 switch ?
about 2 months ago
Hi, Kashi. Thanks for reading.
I don’t believe you have to enable traps to be able to monitor the state of the CSMs; you can simply query the correct OID to get the information you want. If you want the switch to send a trap when the FT state changes, you would then enable the traps.
about 2 months ago
Thanks for the response Aaron, i really appreciate.
Like everyone else, i’m also equally in awe by seeing the ease and clarity with which you respond to queries. Its just such a nice feeling of confidence when talking to people of your calibre.
Coming to my next query, i think for switch to send trap for CSM FT state change, the cmd is
snmp-server enable traps slb ft
but i did not find this cmd on our cisco 6509 sw running 12.2(18)SXF7 IOS but found traps only for vserver, reals and csrp.
Can advice on this.
about 2 months ago
Wow…thanks for the kind words, Kashi; that might be the nicest thing anyone has ever said to me over the Internet.
I’m always happy to hear someone is getting something out of the blog.
I’ve never tried to enable SNMP traps for the CSM, so I’m just going off of information from the CCO. This page says that the command snmp enable traps slb ft is actually entered in CSM config mode instead of global config mode. I run an older CSM version, so I can’t tell you if it actually works or not.
Did that help?
about 2 months ago
Thanks for the reply Aaron.
It did not help…as i found this
6509(config-module-csm)#s?
script serverfarm static sticky
and you can see there’s no snmp cmd even in CSM mode. We r running 4.2(10) version CSM.
BTW, without snmp traps for CSM, how did you manage to monitor CSM ft status ?
about 2 months ago
Yeah. I couldn’t find that command either, Kashi. Even looking at the 4.2.x command reference shows that it should be available, but it’s obviously not. If I get a chance, I’ll drop a note to the TAC to see what’s up with that.
We monitor the CSM status changes with syslog, actually. Every time there’s a state change with probes, RIPs, VIPs, etc., a syslog message is generated and sent to a collector. The collector parses the message and generates an email or an alert in the monitoring system.
about 2 months ago
I will try to monitor CSM Ft change with syslog too and see how it goes.
Coming to my next question:
1) Is there a mechanism to monitor capacity of CSM virtual gig etherchannel with the backplane ? Can it be monitored instead of doing manually on CLI ?
Let’s say i ve 2 webservers being loadbalanced under single VIP, suppose http application fails in one of the webservers,
2) Can traffic be redirected to another webserver in the sfarm, using probes or any other cmds or mechanism, while 1st webserver is being worked ?
3)Does executing the cmd, hw-module csm X standby config-sync, only way to sync config between CSMs or can it be done automatically also ?
about 1 month ago
They, Kashi. Sorry to take so long to get back with you.
1) On your switch, you should see four GEs on the same slot as your CSM; these are those virtual GEs you mentioned. You should also see a high-number port-channel interface (from 257 to 282 from a quick scan of documentation); this is the port channel group for those virtual GEs. If you want to monitor the traffic to and from the CSM, you would monitor those interfaces.
2) You can use the serverfarm failaction directive to send traffic destined for a failed server to another. I’ve got an article on it here.
3) That’s the only way I know how to do it. There may be an SNMP set command to do it, but I don’t know what that OID would be. Tools like CiscoWorks can send commands to your switch on a schedule. CiscoWorks is expensive, though, butKiwi Cattools can probably do that for you well. Another solution may be the kron directive. I’ve never done it this way, but I’ve found this page that looks to be pretty good.
I hope that helps. Ask more questions as you need them. I love answering questions for people.
about 1 month ago
Thanks for the reply Aaron.
1) But i was hoping that you would tell how i can montior the CSM virtual gig channel (CLI cmd i know) in any other way.
2) Does the cmd failaction help when active connection traffic is on one real server and it suddenly fails and then this cmd helps to send all subsequent traffic to 2nd real server in sfarm ?
3) Does executing the cmd, hw-module csm X standby config-sync, has any impact ? Does it have to be done every time config is changed on the primary CSM ? Is there no way that config on primary automatically gets updated to Standby CSM ? Is there any version in which thie auto config-sync feature has been introduced ?
Othrwsie, doing manual sync is awfully difficult everytime.
about 1 month ago
In our scenario, if the real server with the active connection fails, connections are not being reassigned to 2nd real server in the sfarm. Is this bcoz, we ve following cmd under the particular sfarm:
serverfarm
failaction purge
instead of this i guess, it should be
serverfarm
failaction reassign
then i think, the active connection would be directed to 2nd real server. Is this correct ?
about 1 month ago
Hello again, Kashi.
1) You can monitor those interfaces via SNMP with any SNMP-based tool such as MRTG or Cacti. We use Cacti, and it works great!
2) The failaction tells what to do if a server with an active connection goes out of service. If a server fails and you have a failaction of purge, the CSM sends a RST to the client. An action of reassign will just rebalance that connection without the client knowing anything happened. See my blog article about failaction for some more reading.
3) The hw-module csm X standby config-sync will take up resources, but it is very low-impact. I would have no problem running it every time I make a change to the CSM. You could do a weekly or daily sync if you would like, but you run the risk of failing over to a non-current config if something happens to your primary. As far as I know, there is no auto-sync feature. There may be an SSO-type of config for the whole box, but that may not be what you’re looking to do. You’ve piqued my interest now, and I’ll keep an eye out for an auto-config feature.
As for your second comment, you’re right; the purge is resetting the client connection instead of reassigning it to the other server.
Does that cover your questions?
about 1 month ago
Thanks Aaron.
But instead of the server going down, say, if IIS service goes down on RIP1,will failaction cmd reassign next active http connections to RIP2 without client knowing anything.
With probes is it possible to monitor tcp port 80 on both RIP1 & RIP2, and should port 80 go down on anyone, will that RIP be assumed as down by CSM followed by all active connections being sent to another server ?
about 1 month ago
Hey, Kashi.
The CSM checks the health of the servers in a few different ways. First, it makes sure the RIPs are reachable (that is, do they answer ARP requests). The CSM also monitors the connections for any problems such as RSTs coming from the RIPs. The probes, however, are used to do a lot more in-depth health checking. If you had a probe that checked for TCP/80 connectivity on the server, the CSM would periodically connect to that port and use the information it gathers to determine the health of the server. That way, if IIS died on you and stopped answering requests, the CSM could take that server out of service.
I’ve written an article on CSM probes.
about 1 month ago
Gr8! !!! Thanks Aaron for the quick reply.
I will come back with few more questions.
about 1 month ago
Hi Aaron,
Do you answer queries on FWSM as well ?
about 1 month ago
I can field some questions on the FWSM if you would like. How about we take this offline, though, since this is a CSM article. Check the “About” page to find an email addresses where you can reach me.