Syncing IOS Versions on a 3750 Stack
For those that don’t know, when I say “stack”, I mean a group of 3750s connected together using the StackWise technology. When you use a very expensive and very proprietary cable, your individual switches are combined into a single logical device. This means you configure one device to control potentially many switches.
To the point. I’ve spent the last few weeks replacing a mess of 3750s in stacks. These guys are very easy to replace, but the big problem I find is getting the IOS version in sync. When the RMA comes, it’s inevitably got a different version on it, and you’ll see something like this.
Switch# Role Mac Address Priority State -------------------------------------------------------- 1 Member 0023.33ad.a500 1 Version Mismatch *2 Master 0023.5eac.e900 15 Ready
In this case, switch 2, running 12.2(25)SEE3 is the master, and switch 1, running 12.2(35)SEB, is the new member.
A switch in the stack needs to have the same version of IOS as the master to be brought into the fold, so you’ve got to do get the code on the switch somehow. You can use traditional methods to get the right code on the box (like TFTPing one up), but there are easier ways.
If the code versions are close (I’m not sure as to the rules about how close), then you’ll be able to check out the flash of all the switches in the stack through the master. If you do a dir ?, you’ll see flash1:, flash2:, etc. Those are the individual flash drives for each switch in the stack. The numbering is based on the stack number, so all you have to do is copy the IOS image over from the master. Well, maybe.
I found that a lot (maybe all) of my 3750s actually have a directory structure to include the image and HTML files for the device manager. You don’t need the HTML files for the switch to function in the stack, but having the IOS image in different places will force you to change the boot variable for each switch. I like consistency, so I put everything in the same place when I can, so that means creating the directory by hand.
That method works fine, but there’s an easier way using the archive command. You can tell your switch to tar, copy, and extract all of the files from one switch to the others by giving it one of these.
archive copy-sw [ /destination-system X ] Y
I used this command to copy the image from switch 2 (the master) to switch 1 (the replaced member) and got a whole bunch of output. The whole process took about 3 minutes.
Switch#archive copy-sw /destination-system 1 2
System software to be uploaded:
System Type: 0x00000000
archiving c3750-ipbasek9-mz.122-25.SEE3 (directory)
archiving c3750-ipbasek9-mz.122-25.SEE3/c3750-ipbasek9-mz.122-25.SEE3.bin (7206732 bytes)
archiving c3750-ipbasek9-mz.122-25.SEE3/html (directory)
archiving c3750-ipbasek9-mz.122-25.SEE3/html/toolbar.js (7084 bytes)
archiving c3750-ipbasek9-mz.122-25.SEE3/html/title.js (577 bytes)
...
extracting c3750-ipbasek9-mz.122-25.SEE3/html/images/141280.gif (3053 bytes)
extracting c3750-ipbasek9-mz.122-25.SEE3/html/images/meter_yellow.gif (59 bytes)
extracting c3750-ipbasek9-mz.122-25.SEE3/html/images/legend_off.gif (1158 bytes)
extracting c3750-ipbasek9-mz.122-25.SEE3/info (682 bytes)
extracting info (108 bytes)
Installing (renaming): `flash1:update/c3750-ipbasek9-mz.122-25.SEE3' ->
`flash1:c3750-ipbasek9-mz.122-25.SEE3'
New software image installed in flash1:c3750-ipbasek9-mz.122-25.SEE3
All software images installed.
I left out a large chunk, but you get the idea. If you’ll notice, all the HTML files are copied along with the IOS image, so you get exactly the same structure on all your switches. It beats doing it all manually.
Send any RMA packages questions my way.
https://masto.ai/@aconaway
- Generating Network Diagrams from Netbox with Pynetbox - August 23, 2023
- Out-of-band Management – Useful Beyond Catastrophe - July 13, 2023
- Overlay Management - July 12, 2023
Hi Aaron,
Nice write-up. Thanks for the info! I have a question, though, since I don’t have nearly as much experience as you do with these products. Wouldn’t it be more worthwhile to go for the offerings in the Cisco 4500 series platform over stacking a bunch of 3750’s and expensive proprietary cabling?
Thanks,
Paul
That’s a great question, Paul. I was going to bring up cost, but, by the time you get a stack of 3750s together, you wind up paying about the same as you would for a 4506-E with a sup and appropriate modules. One advantage the 3750s have, though, is in the fact that they don’t have a single point of failure to take out the whole stack. On a 4500, if you lose a supervisor, the whole switch is dead. If you lose the master on a 3750 stack, an election is held to find a new master; as long as you’re not single-homed to the failed switch, you probably won’t even notice the difference.
I’ve found the reliability of the 3750s to be absolutely piss-poor. I would choose a 4500 over a stack of 3750s any day just to keep from having to keep replacing them. We have one location with 12 3750s and have done 15 replacements. We’ve replaced all of them except for one, and four of them have been replaced twice. This is getting old. 🙂
Thanks for the follow-up. That’s exactly what I was thinking when it comes close to cost of a 4506. From your track record with the 3750’s, I would take the chance on losing a sup engine. There are other redundancies in the 4500 chassis such as fans and power supplies that win, imho. And only a blade would have to be replaced, not the whole box.
Still strikes me kind of surprising that you would have to swap out that many of the 3750’s. Any extreme operating environments? Not sure how close you are to the ocean in SC, wondering if humidity and salinity in the air is playing a factor? Been to Myrtle Beach a few times, and I remember seeing a lot of oxidation/rust on things there such as elevators, etc.. Salt and humidity can be tough on everything!
The failing 3750s are in closets all over the US — from Alaska to California, to Kansas to Georgia, to Connecticut. Our home office is about 200 miles inland and has seen a lot of failures. In those cases the closets were in the high-80F range before we got extra cooling brought in. Even when the temperatures were around 80F, we still lose switches. The big hitter office is closer to the coast, but it’s still 20 miles inland and not exposed to the ocean environment. Those closets are just above 80F. Every 3750 we have in data centers with controlled cooling have run like champs; there have no been problems with them at all.
Moral of the story: Stay away from 3750s.
We have more than two hundreds of 3750 PoE’s and non-PoE but we rarely replace them and we still keep on buying 3750’s anyway. Maybe it just happen that we have a good and controlled cooling system.
Perhaps, Ben, but we have 2950s, 2960s, 3550s, and 4500s in the same closets that run great. I hate those guys. 🙂
On the other hand, at my previous job, we swapped out a 6509 chassis every week all over the country. At my current job, we have a fair number of them that have never had an issue. It’s all a roll of the dice, I guess.
Does anyone know if the archive command is available in the boot loader? We had the situation where a stack was rebooted and one of the switches DID NOT HAVE an image file. Well, in this situation (especially as important as this stack is) you don’t have time to ask why. We ended up xmodem-ing up the file which I prefer and recommend over tftp. You just have to set the baud to an acceptable level 115200. This method has a couple of advantages.
First, you don’t need a tftp server!
Second, the little extra time it takes to load the image gives you a chance to stretch your legs.
Anyways, we got it all back working but it was odd that the switch somehow lost its image file.
Hey, Dano. The switch has to come up and negotiate membership in the stack, so I would be greatly surprised if you could do it from the boot loader. I would have just taken the switch out of the stack, plugged it into the network somewhere, and transferred the file over the network. After that, you shove it back into the stack and you’re set.
Glad you got the switch back. X-modem still gives me the chills even if it is at 115200. 🙂
You just made me think of another advantage of Xmodem: You don’t have to take the switch out of the stack and plug it in to the network somewhere else.
Well, it’s funny to me that Xmodem is supported and even Ymodem, but not Zmodem which supports windowing. I’m from the BBS era and we used Zmodem all the time. It was great to transfer files and watch the window grow and grow up to 16K I believe was the max. Anyway, those days are gone, but I’m glad Xmodem isn’t.
[…] 3750 platform. I believe Ethan Banks (@ecbanks) has posted on this in the past and more recently Aaron Conaway (@aconaway). I thought I’d post my experiences over the past 4-5 years managing ~40x 3750 […]
Hi Aaron…love the comments and tips
I am a CCNA student and is being asked to set up a wireless network with two 3750s stacked to provide the POE link to the APs (x2). The 3750s is to be trunked to two clustered asa 5510s to filter traffic going in and out.
I have very little experience with the 3750 so could you please help with the configs to get the 3750s communication with the asa 5510s.
Thanks for your help
Taniela Kailahi
Thanks for reading, Taniela. The configs for the ASA ports should be the same as the AP ports. Simply configure the trunk encapsulation (switchport trunk encapsulation dot1q) and enable the trunk (switch mode trunk). That should do it. If you need any more help on that, let me know.
I prefer to use stacks of 3750 switches at the access layer for the maintenance cost savings. The switches come with a limited lifetime hardware warranty. Provided you have 1 or 2 spares to last the week or so until the RMA arrives you don't need to buy maintenance. That's a reasonably significant saving as the sites become larger.
Hi Aaron,
This is an old thread but i am trying to find something on the web regarding a problem i encountered with a 3750 stack and i struggle. I hope someone here can help.
A brief overview of the topology is that we have some port-channels (LACP) connecting to some Blade Center switches (Nortels – sad but true!).
When i reloaded the slot that was the Master on the stack, the other member needed almost 100 secs until it started forwarding through the interfaces that were members of the port-channels. From what i know this should have been transparent. Am i wrong? Does LACP need to re-negotiate because the Master has changed?
– Georgios
Hi Aaron
I have similar problem in production environment we have 3 x 3750 Catalyst stack now due to capacity issue we plan to add another 3750 when we checked found IOS not matched with the existing so can I use same procedure which you mentioned above
Correct me if I am wrong from master (SW-1) #archive copy-sw /destination-system 1 4
Thanks
Muhammad