Configuring Fault Tolerance on the CSM

Like (nearly) everything in the Cisco world, you can set up your CSM to fail over to another module when the primary dies a horrible death.  You can have two in the same chassis or even have them in separate chassis — the process is the same no matter how you have it set up.  Either way, you have a primary and a secondary module in fault tolerance (FT) mode.

First, we’ll establish a VLAN that the CSM will use to do its configuration and state syncing over.  This is just an ordinary VLAN; there’s nothing special about it, really, but it should be dedicated for the CSM to use for syncing.  Let’s randomly choose VLAN 83.

vlan 83
name CSM-Sync

You will, of course, have to do this on every switch that holds a CSM, so, if you’re using them in two different chassis, you’ll put the same VLAN on each making sure they can see each other through a trunk.  Cisco recommends that you dedicate a trunk between the two switches for the sync VLAN in order to remove the chance of other traffic stepping on the sync packets, but I’m not convinced that’s necessary.  Use your judgement on that one.

Back to it.  Next, you need to decide on a FT group ID.  This is similar to a HSRP group and lets you run multiple FT groups on the same VLAN.  The group ID needs to be in the range of 1 to 256, so, since this is the first one, let’s just use 1.  Get into config mode for the CSM that you want to be the primary and do this.

ft group 1 vlan 83

This takes you to the config-slb-ft prompt.  Just like HSRP, we need to set priorities for each device and whether or not it should preempt, so let’s configure.  Yes, we want to preempt, right?  Let’s set the priorities to 100 and 90, too.

priority 100 alt 90
preempt

This sets the primary CSM to priority 100 and the secondary to 90; both will preempt.

What about configuring the secondary for FT? That’s easy.  Go into CSM config mode on the secondary and enter the ft group 1 vlan 83 command.  That’s it. The two CSMs will do a little arguing and come back as the primary and secondary.  After that, all configuration is done on the primary, which is synced over to the secondary just like an ASA.  Pretty cool, eh?

When configuring things like IP addresses, though, you’ll need to make provisions for the secondary with the alt directive (remember that one from the priority).  I won’t go into much, but you’ll need it mostly when settings IPs to VLANs.  Here’s an example of setting an IP address on client VLAN 100 for both the primary and secondary.

vlan 100 client
ip address 192.168.0.11 255.255.255.0 alt 192.168.0.12 255.255.255.0

Alright…one more thing.  The configurations don’t sync automagically (at least not on my old version of code).  If you make a change to the primary CSM, you’ll see an out-of-sync message when you look at the FT status.

Switch#sh mod csm X ft
FT group 1, vlan 83
This box is active
Configuration is out-of-sync
priority 100, heartbeat 1, failover 3, preemption is on
alternate priority 90

If the primary goes down now and the secondary takes over, the changes you just made won’t be reflected on the secondary.  You fix this with the hw-module contentSwitchingModule X standby config-sync command (where X is the module slot in the chassis).  Alternatively, you can just type hw c X s c as a shortcut.  It’ll take a few minutes depending on your configuration, so check your logs for when it’s finished.  Note that the secondary does not save the new configuration to its startup-config; you’ll have to log in and save that manually (or automatically through CiscoWorks or something) to save changes there.

Let me know if you have any questions and check out my page on getting output from Cisco’s fine mid-tier load balancer.  🙂

Aaron Conaway

I shake my head around sometimes and see what falls out. That's what lands on these pages.

More Posts

Follow Me:
Twitter

14 comments for “Configuring Fault Tolerance on the CSM

  1. daniel
    April 12, 2010 at 2:10 am

    hey,

    thanks a lot.. thats a real easy an useable guide 🙂

    just two little questions about it:

    * can i configure this in a live-setup where i have a full configured csm and want to add a new/empty one?

    * is there any downtime when i activate the second one?

    thanks a lot

    daniel

  2. April 12, 2010 at 10:54 am

    Those are great questions, Daniel.

    If you already have a stand-alone CSM configured, configuration changes will be required; you would have to configure the alt addresses and FT VLAN. I imagine that would cause an outage. If you had the primary already configured for FT, however, then adding a second CSM shouldn’t take anything down at all.

    Of course, this is all in theory and hasn’t been labbed out, so take my opinion at your own risk. 🙂

  3. daniel
    April 13, 2010 at 2:06 am

    okay..

    here’s my report:

    * i just configured the alt-adresses and the FT stuff on the primary – no impact

    * i configured the FT on the secondary

    * i synced the config – no impact

    sometimes its the best way to just try it.. my dear little module.. i like you 🙂

    so bad that the end for the csm is near 🙁

  4. April 13, 2010 at 7:55 am

    Good info, Daniel. Thanks for sharing. I totally expected a hiccup.

  5. daniel
    April 13, 2010 at 9:33 am

    hmmm..

    now the big question for me.. how does failover work.. did you ever seen a failover? how long does it take? is it really hitless? (cisco told me that there is no downtime when switching to the backup-csm [if you have connection and sticky replication enabeld])

  6. April 13, 2010 at 9:53 am

    When I was doing our current deployment, I had several hosts connect to a few vservers and just wait. I then went down to the data center and pulled power to the 6500, and I didn’t lose any connections. I also did some ping testing through the CSM and to the alias on the back-end networks with the same result. I’m pretty confident that you’ll see a seemless failover if all the replication is successful.

    Just remember to do the hw-modules “csm X standby config-sync” when changes are made. The CSM can replicate everything but the config, which makes no sense to me.

  7. daniel
    April 14, 2010 at 2:17 am

    nice.. 🙂

    in three weeks i have a downtime to make a failover-test.. i will report back after that..

  8. Kashi
    July 22, 2010 at 12:37 am

    Aaron,

    Is it possible to setup 2 CSM mods in 2 different chassis in Active-Active mode ?

  9. July 25, 2010 at 9:40 pm

    Somehow I missed this, Kashi. Sorry about that. I believe that CSMs can only be configured as active-passive. Looking through configuration notes, I only see commands that relate to failing over; I don’t see any related to modes or having two modules be active at the same time.

  10. Edgar
    April 2, 2011 at 8:28 am

    Hello Aaron, hope you still support this post, I just installed a second CSM on the same chassis, configured FT.

    The test were successful, both CSM works when there are active.

    I noticed that even I run on the active CSM, “hw-module csm 4 standby config-sync”, I still see Configuration is out-of-sync.

    This is the event I got when run hw c 4 s c,

    ———–
    Apr 2 08:18:22.209: %CSM_SLB-6-REDUNDANCY_INFO: Module 4 FT info: Active: Bulk sync started
    Apr 2 08:18:22.209: %CSM_SLB-6-REDUNDANCY_INFO: Module 7 FT info: Standby: Bulk sync started
    Apr 2 08:18:22.217: %CSM_SLB-6-REDUNDANCY_INFO: Module 7 FT info: STANDBY:Configuration is being received, This may take several minutes!
    Apr 2 08:18:22.217: %CSM_SLB-6-REDUNDANCY_INFO: Module 4 FT info: Active: Sending configurations to Standby CSM, this may take several minutes!
    Apr 2 08:18:31.541: %CSM_SLB-6-REDUNDANCY_INFO: Module 7 FT info: Standby: Started clearing configuration
    Apr 2 08:18:31.545: %CSM_SLB-6-REDUNDANCY_INFO: Module 7 FT info: Standby: Completed clearing configuration
    Apr 2 08:18:31.549: %CSM_SLB-4-REDUNDANCY_WARN: Module 7 FT warning: Standby: Config Sync does not save running-config to startup-config
    Apr 2 08:18:31.549: %CSM_SLB-6-REDUNDANCY_INFO: Module 7 FT info: Standby: Previous configuration are being deleted from supervisor
    Apr 2 08:18:31.553: %CSM_SLB-6-REDUNDANCY_INFO: Module 7 FT info: Standby: Previous configuration being deleted on Standby CSM
    Apr 2 08:18:33.649: %CSM_SLB-6-REDUNDANCY_INFO: Module 7 FT info: Standby: New configuration are being configured
    Apr 2 08:18:36.345: %CSM_SLB-6-INFO: Module 7 info: SLB-NETMGT: vserver V-FARM0 inservice
    Apr 2 08:18:36.345: %CSM_SLB-6-INFO: Module 7 info: SLB-NETMGT: vserver V-FARM1 inservice
    Apr 2 08:18:36.345: %CSM_SLB-6-INFO: Module 7 info: SLB-NETMGT: vserver V-FARM2 inservice
    Apr 2 08:18:36.537: %CSM_SLB-6-REDUNDANCY_INFO: Module 4 FT info: Active: Manual bulk sync completed
    ————
    According to the above logs, the sync works fine, but I still get this.
    C6500#sh module csm 4 ft
    FT group 1, vlan 121
    This box is active
    Configuration is out-of-sync
    priority 50, heartbeat 1, failover 3, preemption is on

    C6500#sh module csm 7 ft
    FT group 1, vlan 121
    This box is in standby state
    Configuration is out-of-sync
    priority 10, heartbeat 1, failover 3, preemption is on

    Both CSM running the folling version 4.3(4)

    I´ll really appreciate your help on this.

    Regards

  11. April 14, 2011 at 10:33 am

    Hey, Edgar.

    I’ve seen that type of thing in the past with 2 CSMs in different chassis. The solution for me was actually just wait; it sometimes took 10 or 15 minutes to sync the configs over. Since you’re asking here, I can only assume that they’re never syncing, though.

    I can only suggest opening a TAC case. Sorry that I can’t be any more help.

  12. Edgar
    May 26, 2011 at 6:17 am

    Hello Aaron

    Sorry for not answering asap.

    Thanks you for you answer.

    Regards,
    Edgar

  13. December 27, 2012 at 5:21 pm

    Hello Aaron

    Good to hear about you.

    I hope you can help me to figure out this question, I have a CSM module with more than 10 serverfarms, all of them working fine, and all of them are on different vlans. We are using route mode for all of them.

    For example:

    ServerFarm01-> Vlan10
    Client_Side01-> Vlan11

    ServerFarm02-> Vlan20
    Client_Side02-> Vlan21

    ServerFarm03-> Vlan30
    Client_Side03-> Vlan31

    I noticed something, When I generate outbound traffic from a real server, it does not matter it belong to SeverFarm01, 02 or 03; the packet leaves the CSM using alway the vlan31.

    Can you please help to determine what’s going on?

    Actually, we want realservers from ServerFarm01 to sent traffic to internet through the CSM, and those traffic should be seen on vlan11.

    Thankds and Regards
    Edgar

Leave a Reply

Your email address will not be published. Required fields are marked *