Getting Something Out of the CSM
My buddy told me that my site is the only place on the web with documentation on the Cisco Content Switching Module (CSM). I also noticed a few months ago that every TAC case I’ve opened on the CSM has been handled by the same guy. I seriously think that the only people in the world that really know about these things are me and him. Cool. I better get some more content up.
The CSM is configured and controlled by the IOS running on the 6500. Unlike the FWSM, it is not independent from the switch’s operation, which is sometimes good. One good thing out of that is the fact that you can pull stats and stuff from the IOS without having to session or SSH to another module. The bad part of that, though, is that the commands wind up being long.
To start with, all show and clear commands start like this.
show|clear module contentSwitchingModule <SLOT> <COMMAND>
Do you see how long that command is? And we didn’t even tell it what we wanted to do yet. You can use the auto-complete, though, f you’re lazy like I am.
sho|cl mod csm <SLOT> <COMMAND>
Yes, it takes csm instead of contentSwitchingModule. That’s not in the contextual help. Heh.
SLOT is where the module is in the chassis, so, if your CSM is in slot 8, you…can figure it out. COMMAND is what you want to do, right? Yeah…I won’t insult your intelligence.
What are some show commands? There’s a lot of them, but here’s some I use every day. My production CSMs are in slot 3, so I’ll just use that for the slot.
- show mod csm 3 arp : Shows you the ARP table (duh!). This is good to see if the CSM can contact the real servers or the gateways properly.
- show mod csm 3 conns : Shows the current connections through the CSM. This shows you what client IP is connected to what virtual IP and on what real IP that connection lands.
- show mod csm 3 ft : Fault tolerance. This shows what your FT VLAN and status is. It also shows if your secondary configs are out-of-sync with the primary.
- show mod csm 3 reals : Shows all the real servers involved in all serverfarms along with the weight, state, and current number of connections. This shows you a lot of information that could be helpful in troubleshooting a problem. Look for FAILED, PROBE_FAILED, or OUTOFSERVICE; these are bad.
- show mod csm 3 serverfarms : Shows your setup and status of the serverfarms. Also great for troubleshooting.
- show mod csm 3 vservers : Shows the IP, VLAN, and state of your vservers along with the current number of connections.
- show mod csm 3 vlan : Shows the VLAN ID, subnet and mask, and VLAN type (server, client, FT)
Be sure to use your contextual help for more detail on these commands; most of them actually can get very specific. For example, ft shows your status, but ft detail shows your message counts, resets, and other good and bad things that deal with the fault tolerance.
How about something to clear?
- clear mod csm 3 connections : Clears the connections through the CSM. This is probably very close to clear conns on a PIX, FWSM, or ASA, so be careful not to kick everybody off.
- clear mod csm 3 arp-cache : Clears ARP
- clear mod csm 3 ft active : Forces the primary to fail over.
- clear mod csm 3 counters : Do you have to ask?
Again, these guys need to be explored with the question mark to get the full effect.
Since I’ve established myself as the long authority in the world on the CSM [sic], drop a comment with any questions.
https://masto.ai/@aconaway
- Generating Network Diagrams from Netbox with Pynetbox - August 23, 2023
- Out-of-band Management – Useful Beyond Catastrophe - July 13, 2023
- Overlay Management - July 12, 2023
Said in the voices from the guys in the Guinness commercials… “BRILLIANT!” This is good stuff, keep it coming! 🙂
Thanks, Clint. I’m trying to get back on track.
[…] Getting Something Out of the CSM […]
Hey Aaron,
Thanks for teh great info. I don’t know if this is the right place to post this but here goes.
I understand the basic functions of the CSM but when traffic comes into a VIP and then to a farm, how do you get the CSM to pass the source IP to the destination? Wouldn’t that info be of some value?
Hi, Droudpeyma. Thanks for commenting.
In the serverfarm configuration, there are two lines that deal with NAT — “nat client” and “nat server”. By default, client (source) NATting is off, and server (destination) NATting is on, so the source IP of the connection inbound to the serverfarm does not get changed before it’s passed on to the servers in the farm.
Here’s what the source and destination of the packets are at various steps in the process. Let’s assume the source IP is 12.34.56.78, the VIP is 1.1.1.1, and the serverfarm only has one server with the IP of 2.2.2.2. Assume that no other NATting is done in our little setup.
The packet leaves the client:
S: 12.34.56.78, D: 1.1.1.1
The CSM receives the packet and passes it on to the RIP:
S: 12.34.56.78, D: 2.2.2.2
The packet lands on the server:
S: 12.34.56.78, D: 2.2.2.2
The server generates a return packet:
S: 2.2.2.2, D: 12.34.56.78
The CSM gets the packet and sends it back to the client:
S: 1.1.1.1, D: 12.34.56.78
The client receives the packet:
S: 1.1.1.1, D: 12.34.56.78
Hope that helps. Let me know if it doesn’t.
Aaron,
Thanks so much for the info. I understand now why our farms are showing the source IP to be of the VIP. At the moment the farms are assigned to a NATPOOL on the client side. When that is removed the connection to the site breaks. Is there something else that needs to be done on the (client) Nat in order to restore the connection?
Thanks,
David
Hey, David.
I think if you turn off cleint NAT, you should be alright. Just go into your serverfarm for the VIP and do a “no nat client”. After such, you should see the clients appear unNATted.
one more very (at least for me) useful command run from switch# level
“show run mod X”, where X is slot number where CSM resides
it shows only configuration block for the csm
works with other modules as well 🙂
[…] me know if you have any questions and check out my page on getting output from Cisco’s fine mid-tier load […]
Hi Aaron, I did find useful information hear about CSM. Thanks very much.
My question is that doing this command ‘show mod csm 3 ft’, the output shows this line
‘Configuration is out-of-sync’
I wonder what does it mean, how can I get configuration “in sync” at the active and standby CSM.
Regards,
D’Halmar.
Thanks for reading, Dhalmar.
The out-of-sync status means that your secondary CSM doesn’t have the same version of the configuration on the primary. I should have put this in my article on configuring fault tolerance, but you can just do a
I think that’s the command, but I don’t have a CSM in front of me to verify. I usually just do a
This command is required to sync the configs between CSMs and forces the current active configuration to replicate to the secondary. That should clear that right up.
Hello Aaron,
excellant website, alot of very useful info, i’ve been using it on and off now for a few years, I was wondering if you had came accross of an issue we are having. We have various server farms and need on occasion to clear the connections on a real server in the farm which can be done by using the command “clear mod csm * connections real *.*.*.*.
When we do this it seems to clear the connections for the whole server farm as opposed to just the real server. This is a problem because it is running in a production environment and can involve clearing quite a number of thousand connections. We are running version 3.2(1) in a 6500 running in native mode. If you need anymore info let me know.
Thanks in advance Aaron, keep up the good work.
Eamon
Thanks for reading, Eamon. I always appreciate input from the community.
I did some experimentation on a CSM running 4.2(4), and I was not able to recreate what you described. I selected several serverfarms, picked a real in each, and ran the “clear mod csm X conn real A.A.A.A” command on them. The connections to the chosen real cleared, and the rest of the farm didn’t notice anything. I looked through the bugs for the CSM and couldn’t find anything of interest.
Does it happen to any real you try to clear or just a handful?
It must be something to do with the version were running “3.2(1)”. We rebooted the csm and the problem disappeared, we were able to reset connections to one real server for a while after the reboot but it has started happening again, we are now back to if we try to reset connections on a particular real server it resets connections to the whole farm.
The issue happens all, does not seem to matter which server you select and it clears all connections to the serverfarm.
Thanks for your help Aaron
Sorry Aaron,
The issue happens with all reals, does not matter which server you select
I don’t know what to tell you, Eamon. It obviously shouldn’t do that, and I can’t find a bug that matches what you’re seeing. There is an upgrade to 4.3(3) on Cisco’s website, but I would open a ticket with the TAC before going down that path.
Let me know how it goes.
Hi Aaron,
To monitor ft state of 2 CSMs on 2 chassis via SNMP is it required that SNMP trap for CSM ft state should be enabled on the Cisco 6509 switch ?
Hi, Kashi. Thanks for reading.
I don’t believe you have to enable traps to be able to monitor the state of the CSMs; you can simply query the correct OID to get the information you want. If you want the switch to send a trap when the FT state changes, you would then enable the traps.
Thanks for the response Aaron, i really appreciate.
Like everyone else, i’m also equally in awe by seeing the ease and clarity with which you respond to queries. Its just such a nice feeling of confidence when talking to people of your calibre.
Coming to my next query, i think for switch to send trap for CSM FT state change, the cmd is
snmp-server enable traps slb ft
but i did not find this cmd on our cisco 6509 sw running 12.2(18)SXF7 IOS but found traps only for vserver, reals and csrp.
Can advice on this.
Wow…thanks for the kind words, Kashi; that might be the nicest thing anyone has ever said to me over the Internet. 🙂 I’m always happy to hear someone is getting something out of the blog.
I’ve never tried to enable SNMP traps for the CSM, so I’m just going off of information from the CCO. This page says that the command snmp enable traps slb ft is actually entered in CSM config mode instead of global config mode. I run an older CSM version, so I can’t tell you if it actually works or not.
Did that help?
Thanks for the reply Aaron.
It did not help…as i found this
6509(config-module-csm)#s?
script serverfarm static sticky
and you can see there’s no snmp cmd even in CSM mode. We r running 4.2(10) version CSM.
BTW, without snmp traps for CSM, how did you manage to monitor CSM ft status ?
Yeah. I couldn’t find that command either, Kashi. Even looking at the 4.2.x command reference shows that it should be available, but it’s obviously not. If I get a chance, I’ll drop a note to the TAC to see what’s up with that.
We monitor the CSM status changes with syslog, actually. Every time there’s a state change with probes, RIPs, VIPs, etc., a syslog message is generated and sent to a collector. The collector parses the message and generates an email or an alert in the monitoring system.
I will try to monitor CSM Ft change with syslog too and see how it goes.
Coming to my next question:
1) Is there a mechanism to monitor capacity of CSM virtual gig etherchannel with the backplane ? Can it be monitored instead of doing manually on CLI ?
Let’s say i ve 2 webservers being loadbalanced under single VIP, suppose http application fails in one of the webservers,
2) Can traffic be redirected to another webserver in the sfarm, using probes or any other cmds or mechanism, while 1st webserver is being worked ?
3)Does executing the cmd, hw-module csm X standby config-sync, only way to sync config between CSMs or can it be done automatically also ?
They, Kashi. Sorry to take so long to get back with you.
1) On your switch, you should see four GEs on the same slot as your CSM; these are those virtual GEs you mentioned. You should also see a high-number port-channel interface (from 257 to 282 from a quick scan of documentation); this is the port channel group for those virtual GEs. If you want to monitor the traffic to and from the CSM, you would monitor those interfaces.
2) You can use the serverfarm failaction directive to send traffic destined for a failed server to another. I’ve got an article on it here.
3) That’s the only way I know how to do it. There may be an SNMP set command to do it, but I don’t know what that OID would be. Tools like CiscoWorks can send commands to your switch on a schedule. CiscoWorks is expensive, though, butKiwi Cattools can probably do that for you well. Another solution may be the kron directive. I’ve never done it this way, but I’ve found this page that looks to be pretty good.
I hope that helps. Ask more questions as you need them. I love answering questions for people.
Thanks for the reply Aaron.
1) But i was hoping that you would tell how i can montior the CSM virtual gig channel (CLI cmd i know) in any other way.
2) Does the cmd failaction help when active connection traffic is on one real server and it suddenly fails and then this cmd helps to send all subsequent traffic to 2nd real server in sfarm ?
3) Does executing the cmd, hw-module csm X standby config-sync, has any impact ? Does it have to be done every time config is changed on the primary CSM ? Is there no way that config on primary automatically gets updated to Standby CSM ? Is there any version in which thie auto config-sync feature has been introduced ?
Othrwsie, doing manual sync is awfully difficult everytime.
In our scenario, if the real server with the active connection fails, connections are not being reassigned to 2nd real server in the sfarm. Is this bcoz, we ve following cmd under the particular sfarm:
serverfarm
failaction purge
instead of this i guess, it should be
serverfarm
failaction reassign
then i think, the active connection would be directed to 2nd real server. Is this correct ?
Hello again, Kashi.
1) You can monitor those interfaces via SNMP with any SNMP-based tool such as MRTG or Cacti. We use Cacti, and it works great!
2) The failaction tells what to do if a server with an active connection goes out of service. If a server fails and you have a failaction of purge, the CSM sends a RST to the client. An action of reassign will just rebalance that connection without the client knowing anything happened. See my blog article about failaction for some more reading.
3) The hw-module csm X standby config-sync will take up resources, but it is very low-impact. I would have no problem running it every time I make a change to the CSM. You could do a weekly or daily sync if you would like, but you run the risk of failing over to a non-current config if something happens to your primary. As far as I know, there is no auto-sync feature. There may be an SSO-type of config for the whole box, but that may not be what you’re looking to do. You’ve piqued my interest now, and I’ll keep an eye out for an auto-config feature.
As for your second comment, you’re right; the purge is resetting the client connection instead of reassigning it to the other server.
Does that cover your questions?
Thanks Aaron.
But instead of the server going down, say, if IIS service goes down on RIP1,will failaction cmd reassign next active http connections to RIP2 without client knowing anything.
With probes is it possible to monitor tcp port 80 on both RIP1 & RIP2, and should port 80 go down on anyone, will that RIP be assumed as down by CSM followed by all active connections being sent to another server ?
Hey, Kashi.
The CSM checks the health of the servers in a few different ways. First, it makes sure the RIPs are reachable (that is, do they answer ARP requests). The CSM also monitors the connections for any problems such as RSTs coming from the RIPs. The probes, however, are used to do a lot more in-depth health checking. If you had a probe that checked for TCP/80 connectivity on the server, the CSM would periodically connect to that port and use the information it gathers to determine the health of the server. That way, if IIS died on you and stopped answering requests, the CSM could take that server out of service.
I’ve written an article on CSM probes.
Gr8! !!! Thanks Aaron for the quick reply.
I will come back with few more questions.
Hi Aaron,
Do you answer queries on FWSM as well ?
I can field some questions on the FWSM if you would like. How about we take this offline, though, since this is a CSM article. Check the “About” page to find an email addresses where you can reach me.
Hi Aaron,
We have a URL under a vserver with a serverfarm having 2 RIPs. User connections to the URL are load balanced between the RIPs.
Now is there a way to monitor the URL availability from CSM and also send an alert ?
Hey, Kashi. I’ve got an article on polling the CSM with SNMP; is that what you’re trying to do?
http://aconaway.com/2007/10/23/49/
Aaron, i ve seen your article no polling CSM with SNMP. But i believe that is for monitoring the connections on VIP and RIPs.
In what i’m trying to understand is – VIP is mapped to a host name, say , http://xyz.com , which is being accessed publicly. Now say, if VIP and RIP all are fine, but the URL goes down or does not respond for some reason, can this be monitored from CSM point of view.
Kashi: If I’m understanding you correctly, you can write a script probe that will get the URL from each RIP and fail if the URL is not available. Is that what you were looking for?
Yes, right. How can a script probe be written ? Could you guide plz..and also how to execute ? Would it need to be executed on the server or on the CSM ?
I am not a Tcl scripter, I’m afraid. I’ve read over how to do it in the past, but I’ve never done it myself.
In short, though, you load a script into the CSM by pasting it in or through a file upload, and then create a probe object that references the script. You apply the probe to a serverfarm just as you would any other probe, and that script is run against the RIPs as a health check. In your case, you would have a script that did an HTTP get for a full URL (http://xyz.com) and returned certain codes depending on the result.
Here’s a link I found. http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/mod_icn/csm/csm_4_1/icn/scriptg.htm
Hope that helps, Kashi.
Thanks Aaron. One more.
I know that from CSM we can find out no of connections/hits coming on to a RIP. But is it possible to capture this data say every 1 hr using a TCL script ?
Another good question, Kashi, and, once again, it’s something I know so little about. If you can get a Tcl script loaded and running from CLI, you should be able to use either the EEM scheduler or Kron to run them at regular intervals. You’ll have to do all the research on your own, though, as I have never used either. Personally, I would just use SNMP to poll the connections to the RIP and graph or alert based on the value.
Hi Aaron,
I want to understand, what does the following mean in the out of the command, sh mod csm x vservers name
conns = 2, total conns = 1587
Would appreciate if you could explain elaborately.
Hey, Kashi. That command means that you currently have two active connections (conns) and 1587 connections over the life of the vserver (total conns).
Hi Aaron,
Excellent site. I'm just wondering if you have an idea on these alarms which I get every minute. Router1 currently has active CSM and Router 2 is the Active HSRP (vlan800). 172.17.1.77 is a VIP in the CSM and is complaining about this arp frame coming from the mac-address. I found out that it is the virtual mac of the active HSRP in vlan800.
Is there some hosts/server in vlan800 trying to arp for the VIP and the router2(having the active HSRP for vlan800) is broadcasting a response that it has the VIP 172.17.1.77 and since Router1 CSM already has arp entry for 172.17.1.77, it gets these errors below?
Sep 11 22:27:58.337 GMT: %CSM_SLB-4-TOPOLOGY: Module 3 warning: IP address conflict: ARP frame from 172.17.1.77 with MAC 00:00:0c:07:ac:02 received on VLAN 800
Thanks for stopping by, Peter. If your CSM is configured with the same IP as your HSRP routers, then you have a standard IP conflict. Do the CSM and HSRP routers both use the same IP?
It's not the same IP. Cisco is unable to provide me explanation on this. I saw this article below which could be related to what is happening to me. Do you know of any debug commands in csm that could capture the source sending this arp frame for 172.17.1.77?
http://www.cisco.com/en/US/products/hw/modules/ps2706/products_tech_note09186a00803937a7.shtml
Router1 Router2
vlan 800 client vlan 800 client
ip address 172.17.1.5 255.255.224.0 ip address 172.17.1.4 255.255.224.0
gateway 172.17.1.1 gateway 172.17.1.1
alias 172.17.1.6 255.255.224.0 alias 172.17.1.6 255.255.224.0
vlan 348 server vlan 348 server
ip address 172.18.1.67 255.255.255.192 ip address 172.18.1.66 255.255.255.19
alias 172.18.1.65 255.255.255.192 alias 172.18.1.65 255.255.255.192
interface Vlan800 interface Vlan800
ip address 172.17.1.3 255.255.224.0 ip address 172.17.1.2 255.255.224.0
no ip redirects no ip redirects
arp timeout 240 arp timeout 240
standby 2 ip 172.17.1.1 standby 2 ip 172.17.1.1
standby 2 preempt standby 2 priority 150
standby 2 track Vlan800 20 standby 2 preempt
Hi, Peter.
Can I ask why you don't have your CSMs in an FT configuration? It looks like you're using the same IPs on both of them through the alias command, which may cause a problem. It doesn't explain the duplicate IP from 172.17.1.77, though.
You could try the debug module csm 3 arp command to see what that says. It may be pretty verbose, so I suggest capturing your terminal buffer or using syslog.
Let me know if that helps at all.
Hi Aaron,
Thank you for all your posts, they are very helpful. I have a I challenge that we have 2 servers in severfarm, they have http, https probes and some custom TCP probes. My question, is there any way to configure the CSM if HTTP probe on server1 is down, CSM will forward HTTP traffic to server2 and continue passing another traffic to server1 and not mark server1 is down. Do you have any post about Inband Health Monitor?
Hi, Vincent. Thanks for reading.
When a probe fails for a host in a serverfarm, that host is no longer available for any vserver that uses that serverfarm. The fix is to configure multiple serverfarms – one per vserver containing the same real IPs – so that a single probe failure doesn’t affect all services that use that host. For example, you would create one serverfarm for HTTP and one for HTTPS and assign each to an appropriate server object. You can also apply your custom probes to each new serverfarm as needed.
That should fix you right up.
Hello
quick question as i am looking at my first CSM that has been running for a long time and need to fail over to the standby
i see both boxes in FT group 1 vlan 55 one active and one standby with priority 10 alt 15 so alls good there but it says out of sync
now i have come across this wording about the sync command
“You can synchronize the configurations between the active and standby CSMs in a single chassis or in separate chassis.
Enter the hw-module csm standby config-sync command after you have configured both the active and standby CSMs for synchronization.
Enter this command every time you want to synchronize the configuration
Synchronization happens over the fault-tolerant VLAN. Since traffic over the fault-tolerant VLAN uses broadcast packets, we recommend that you remove all devices, other than those necessary for communication between the active and standby CSMs, from the fault-tolerant VLAN.
If you do not enter the alt standby_ip_address command on the active CSM before you synchronize the configuration, the VLAN IP addresses on the backup CSM will be removed.”
how do i check if a alt standby ip address is present before i try this command as i have looed at the config but dont see anything that matches ?
also if i want to fail over to the standby box while we take down the active and move it do we just change the priority while we do this and then change back afterwards?
Hey, jon.
I’ve got an article at http://aconaway.com/2008/10/10/configuring-fault-tolerance-on-the-csm/ that shows how to configure the fault tolerance on the CSM. You can see in that article exactly what the config is for FT, including priorities and the FT VLAN.
To force a CSM to fail over, you do a “clear module csm X”. That’s a hard reset, though, and causes the CSM to send a RST to all current connections. I wish I could say I’ve used that command but I never have. And I don’t have a CSM on which to experiment. 🙁
Hi Aaron, Is there a way to send a msg to VIP so it returns the list of Server farms and real servers behind.
Very similar to nslookup for DNS to VIP resolution.
good document