Archive for the ‘switch’ tag
A Little Story on Switch Configuration
Here’s another story from the late night. I’ve changed the details to protect the innocent, but you’ll get the idea.
I think most of you know that I started a new job late last year, and I’ve spent my waking hours getting caught up on how the new company works, how everything fits together, and all that jazz. One of the big reasons that I (and a number of others) were brought in was to fix the biggest problem; the company doesn’t have a real central control over customer-facing technologies. There’s a group that does central IT for the company (Exchange, SharePoint, Oracle apps, etc.), but there are dozens and dozens of applications out there. That means there are dozens of “network teams” around the world doing their own thing.
One of those groups gave me a call the other day for some help. Their stack of old 2950s was having some issues, and they needed my help to figure out what was going on. Among the symptoms were flapping interfaces on the firewall and – the best of all – every command was greeted with an memory error. Want to see the config? Too bad. Want to see the memory utilization? Too bad. How about configure the thing? Too bad. The only command that I could actually get to work was a show version, but that’s pretty pointless when trying to troubleshoot issues.
So, what did I find? Nothing that could help with the problem, but plenty of stuff to fix. Bascially, the switch has VLAN 1, it’s layer-3 interface, and a single username configured. Nothing else. The configuration items that I consider to be basic just weren’t there thanks to the group’s network guy being a jack of all trades and master of none. Does putting every host on VLAN1 work? Sure it does. Would you just turn on your switch and not configure anything? I hope not. Does someone who does networking part-time thing it’s a problem? Obviously not.
So, what was missing? For starters, there was no syslog server configured (or even existed on the network at all). That’s a problem since the only way that I could see the logs was to reboot the switch and try again. What did the logs say when I finally rebooted the guy? Nothing since the buffer is empty, but the logs for the boot messages started with “1h3m” by the time I got back to it. That means the service timestamps commands for logging were missing. That lead me to ask what the time was on the box. Did you guess it was March 1, 1993? Yeah – no NTP server set, either. Without these basic configuration items, the odds of doing any troubleshooting are just about zero. Actually, they are zero in this case. The basics were missing, and now we have no idea what the real cause of our problems was.
I found a whole mess of other issues, too. The second switch was connected over an access port. No password encryption service. Both switches were unconfigured VTP servers. Not a single interface description. My OCD definitely kicked in that day.
I guess I’ll have to work for my paycheck this week.
News – Cisco Announces EOL Schedule for 6500s
This is a surprise, but Cisco has announced the end of life of the 6500 switches that we all know and love. Usually Cisco gives a platform a few more years after they decide to retire it, but the schedule only gives the 6500s one more year of service. I’m sure this goes back the success and recent expansion of the Nexus line of switches.
Here’s the lowdown from Cisco.
End of Sale – September 30, 2011
Last Ship Date – October 30, 2011
End of Routine Failure Analysis – November 31, 2011
End of Service Contract Date – January 31, 2012
End of Support – April 1, 2012
That’s a very aggressive schedule and doesn’t really make very much sense. They just announced an ASA Service Module for it, so I’m really not sure what this is all about. I hope details will become more apparent in the press release to be published tomorrow.
Send any mischief questions to me.
DHCP ACK Error on Avaya Phones
We're an Avaya voice shop (for now if I have my way) and have Avaya systems of various sizes and shapes all around the Enterprise. I was at one of our remote locations a few weeks back and helped the guys there replace a non-PoE switch so they could get the old power injector panel out of their rack. When we moved stuff around, the phones didn't come back and had the dreaded DHCP Ack Error. Read the rest of this entry »
Lessons Learned from a Bad Day
I had a really, really bad day this past Tuesday. I mean, a really bad day. I guess I should have seen it coming since the last #stabbytuesday was uneventful. Here's what said cosmos had in for me and the lessons I took away. Most of these are things we've all lived before, but, for various reasons, I got blindsided. I expected more from myself. Read the rest of this entry »
IIUC Notes – Getting Phones on the LAN
More study notes. Correct if wrong, though I hope I get some of it right since I already since I'm an R&S guy. :$
Switchport Configuration
- switchport mode access: This config makes the port an access port that carries the primary and voice VLAN traffic
- switchport mode trunk: This config akes the port a trunk unconditionally, but it will still send DTP messages
- switchport nonegotiate: This config keeps the port from sending DTP messages.
- switchport mode dynamic auto: If the port receives DTP messages, it will become a trunk. If not, it will be an access port.
- switchport mode dynamic desirable: The port actively sends DTP messages trying to become a trunk. This is the default configuration on a Cisco switch.
Cisco IP Phone Boot Process
- Phone connects to an Ethernet switch and gets power if needed
- Switch tells the phone the correct voice VLAN through CDP
- Phone sends DHCP request for its voice VLAN
- DHCP offer includes the TFTP server from which to download the config
- Phone downloads the config from the TFTP server
- Phone contacts the call processing server as dictated in the config file
DHCP Settings on a Cisco Router or L3 Switch
R1(config)#ip dhcp pool MYPOOL
R1(dhcp-config)#network 192.168.0.0 255.255.255.0
R1(dhcp-config)#default-router 192.168.0.1
R1(dhcp-config)#dns-server 192.168.0.10
R1(dhcp-config)#option 150 ip 192.168.0.20 <– Tells the phone to download the config from this TFTP server
R1(dhcp-config)#exit
R1(config)#ip dhcp excluded-address 192.168.0.1 192.168.0.100 <– Don't use these IPs when handing out DHCP
NTP
Why should you use NTP for a CME setup?
- Phones display correct time
- Voicemails have the correct time
- CDRs are timestamped accurately
- Router logs are timestamped accurately
- Time-based access worked predictably
R1(config)#ntp server 1.1.1.1
R1(config)#clock timezone MYTZ -5 <– Sets the timezone to a zone called MYTZ that's 5 hours behind UTC
Stubby Post – What’s an IDB?
I posed the philosophical question on Twitter the other day asking if single trunk links should be in an EtherChannel bundle just in case you need to expand later. I didn’t really expect an answer, but the ever-verbose @WannabeCCIE pointed out (in not so many words) that you should watch your IDBs. What is that?
That’s an interface descriptor block. I admit that I’m not intimately familiar with them, bu they’re data structs in IOS used to keep track of the interfaces on that device. They come in two flavors – hardware and software. HWIDBs usually represent a physical interface but they also represent tunnels, SVIs, PortChannels, subinterfaces, and any other virtual interface that you can configure. The SWIDBs represent the layer-2 encapsulation of each HWIDB, so you’ll see entries talking about Ethernet, HDLC, PPP, etc. That means that every interface you have on a router consumes two IDBs (there are always exceptions). That’s important because each platform and IOS version combination has a limit to the number IDBs that device supports.
If you check out one of Cisco’s pages on IDBs, you’ll see a pretty table showing the limits. The 3640 running 12.4(25b) that I run in my GNS3 lab has a limit of 800 IDBs. That means that I can have 400 interfaces configured at most. That little 800 series router running 12.1T that you still have running at the VP’s house has an IDB limit of 300 or 150 interfaces. The 7200 in the data center running 12.3 can handle 20,000 IDBS or 10,000 interfaces!
If you guessed that you can see your IDBs by typing show idb, then you guessed right. That will show you the IDB limit, how many are being used, a summary table, and a list of all the IDBs with their details. Remember that there may be more interfaces on your device that just physical. You may have an SVI, loopback interface, or even a null or two. These all count towards the limit.
Before you get freaked out and start checking the IDB limits on all your devices, take a breath. I’ve never run into the IDB limit on any device and I’ve never heard of anyone who has. I’m sure someone has, but I don’t remember hearing about any. Think about it for a second. If I took my 3640 and filled it with 4 NM-16ESWs, I’d only have 128 IDBs used (16 ports * 4 modules * 2 IDBs for each port). Don’t forget the null interface and VLAN 1 SVI by default (VLANs take 1; VLAN SVIs take 2 each). That brings the count to 133. Let’s add 100 more VLANs and SVIs on this guy. Now we’re up to 433. How about we put each interface into a channel group of its own. That adds another 128, which is 561. Only 239 more to go.
Unless you’re doing something out of the ordinary, I don’t think the IDB limit will be a problem. Of course, that depends on your definition of “ordinary”.
Send any sort indexes questions my way.
Catalyst 3750s – Bad Luck with a Cisco Logo
Last week, @fletcherjoyce posted an article on his blog about his positive experiences with Cisco’s 3750 switches. If you follow my complaints tweets, you know that I’ve had quite the opposite experience with them. I would never pick on anyone, but I had to throw in my 2 cents.
I’m guessing here, but we have about 50 3750 stacks in the enterprise. Most of them are pairs, you wind up with roughly 120 switches. Since we’ve done about 20 replacements over the last 5 years, that means we have a 17% failure rate. That’s pretty horrible, isn’t it?
For the most part and with few (if any) exception, we use the 3750s as aggregation points for our access switches. We don’t do QoS on them. We don’t do any access control on them. We don’t even do routing on them. They’re simply used to connect all the access switches in the closet to the core, so they’re not doing anything funky or burdensome. The CPU and memory are always well within normal operating parameters. They just fail and fail repeatedly.
The flies started dropping in closets at our corporate headquarters a few years ago. It was the middle of summer, and the temperatures kept rising to over 90F (32C) until the we lost 3 switches in 3 weeks. If you could stand to make it into the closet, you could feel that the sheet metal of the switches was hot enough to make you pull your hand back! When the facilities team added more cooling, the temperatures dropped to around 82F there (28C), but we continued losing switches. I figured the newly-failed switches were feeling the effects of the earlier heat wave and were just getting around to giving up the ghost. Surely the heat was the culprit.
A few months after our headquarters meltdown, a tech for a satellite office called and asked if we could help with some latency issues. He showed me the switch stacks throughout the building, and I noticed that only one of the 10 switches actually had a label. The tech said that he never got around to relabeling them after they were replaced. Some, he said, had been replaced multiple times. The closets were running about 76F (24C), so heat didn’t seem to be the problem at this location. The closets were clean as a whistle, and everything in the racks was on building UPS. I couldn’t find a pattern at all. For the record, all their latency issues were related to two unrelated 3750s. Two RMAs later, and their problems were gone.
I’ve been trying to find patterns for the failures, but I can’t think of any. If it’s heat, humidity, power, dust, etc., then why are we not replacing 2950s as well? There are 4-10 of them for every 3750s stack we have. We’re replacing them, but it’s a rate of less than 1%. If it is environment, then the 2950s are English hooligans compared to the 3750s being French aristocracy. Maybe it’s sabotage. I still don’t know after years of watching RMA after RMA come in.
I have noticed one pattern, though. The only deployments of 3750s that have never had a problem are in data centers. They seem to love any room that has an ambient temperature of 62F (16C) with less than 40% humidity and large volumes of air flow. If only we could install micro-data centers in all our closets, then I would be a happy network dude.
Send any wooden shoes questions my way.
Edit: I went back and checked our TAC cases to see what switches we actually replaced. It turns out that we’ve done 19 replacements, and they’ve all been 3750G-12S-S switches.
Stubby Post – VTP Clients Send Updates
VTP clients send VLAN updates. Did you know that?
I had a VTP server and client in the same VTP domain, and, when I cabled up the trunk, the client overwrote the VLAN database on the server.
The moral of the story is that the best revision number will win no matter what the operating mode of the switch.
SWITCH – Epic Regression
Just because I like giving more money to Pearson Vue, I took the BCMSN test today to see how I would do. I passed with no problem.
In my mind, the CCNP is a technical certification, so I expect to be tested on technical topics. Are there topics beyond technology that P-levels should know? Of course there are, but I really don’t think whole chunks of the test should be about a preparation plan and rollback procedures. The BCMSN had a lot more technical questions at a much higher level of expertise; it seems much better suited to the CCNP track than the SWITCH test did.
I was really surprised at how many questions today were repeats from the SWITCH test last week. Of the three lab exercises I worked, two of them were exactly the same as last week. I would venture to guess that there were also 8 to 10 repeated multiple choice questions. It seems that this is going against my argument of being more technical, though, doesn’t it? If you mix in the remaining questions that were at a much higher technical level, you wind up with a pretty darn good test.
I’ve really got nothing more to say about the BCMSN. It’s a good test with an appropriate level of technical (and paper-pushing) detail. I’m very glad I was able to take it before the 31 July 2010 deadline, and I advise anyone who needs the SWITCH test to try and do the same.
The next stop is ROUTE (642-902) for me. I’m taking a class on that one soon, so I’m confident I can pass it in the next 11 weeks we have left until the deadline.
Audio commentary
Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.
SWITCH – Epic Fail
I did my standard 2ish-hour drive to the closest testing center today to take the SWTCH test (642-813). Utter failure. That’s 3 for those scoring at home.
The test was the absolute worst I’ve ever taken. I know that I complain a lot, but this is totally justified in my eyes. My 4th grade spelling tests were better than this. I’ve seen kindergarten plays with better production value.
First of all, it was poorly written. Whoever wrote those questions has a few pieces of information about English sentence structure missing from their skill set. A sentence needs a verb, right? Well, a lot of the sentences were missing those. It’s kind of important to know what the whole point of the sentence is, or is that too much to ask? The “drag this over here” exercise questions all started with the same 13-word phrase that left the question so long that it was unreadable. A couple of commas would have been nice in some. Others I just had to infer from the answers what they were trying to ask.
There were lots of spelling errors as well. Most of them were just stupid stuff like switched letters or missing characters, but, at one point, I had to figure out that I needed to look at the “router” instead of the “route”. That’s not really cool. The misspellings were so bad that they were actually misspelling the hostnames on the diagrams provided. Does anyone even try any more?
Let’s talk about the technical level of the test. If I didn’t know any better, I would swear I was taking a CCNA test. The technical material was so elementary that it bordered on comical. If I recall correctly (which I never do), there were about 3 questions on trunking which were so easy that my wife could answer them. There were about 4 FHRP questions that were out of the “Cisco for Dummies” book. I could go on, but I have better things about which to complain.
“So,” you might ask, “why did you fail it if it was so easy?” That’s a great question. I failed it because the name of the test is misleading. When Cisco says “Implementing Cisco IP Switched Networks”, they really mean “Collecting Documentation About VLANs.” There were at least four questions on this test that asked what information you need to collect before implementing some unknown step of a project involving VLANs. Sometimes, the reference was to rollback plans. Sometimes it discussed IP assignments. Sometimes it even talked about collecting user requirements. It seemed that nearly half of the questions on the test discussed planning for making changes or preparing change documentation. There was very little “implementing.”
To top it all off, too, one of my labs froze. I entered a command into a router, and it didn’t come back. I couldn’t change to the other lab windows, either (the “Scenario” or “Topology” windows included), but my timer kept ticking. I could click around in the testing software, but the lab itself was toast. I got the administrator who helped me out a bit after the machine was rebooted. I didn’t run out of time or anything, but getting up to find help to troubleshoot a problem really throws you off.
How about some closing words? First of all, I have given up on the Cisco Press books and other materials. Each time I use them they have little to no coverage about topics on the test itself. The ISCW was that way, and we all know about my problems with the ONT. I figured that those were just aged text, but SWITCH is only a month or two old, isn’t it? That means the test hasn’t had that much time to change, but the materials are totally different already.
I actually have an example of the books leading the reader directly away from the test materials. I’m reading from the “CCNP SWITCH 642-813 Quick Reference” book by Donohue. On page 8, it discusses the PPDIOO lifecycle approach.
Network engineers at the CCNP level will likely be involved at the implementation and following phases. They can also participate in the design phase.
That doesn’t make any sense, does it? Didn’t I just say that there were a good number of questions on preparation (the first P) and planning (the second P). Both of those come before the design phase.
Somebody help me out here. What am I missing? Is there some magical book series that has the answers?
I should have bought testing vouchers in bulk when they were $150.
Audio commentary
Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.
UPDATE: It seems that the idea of seeing topics on the exam that aren’t are the test go beyond just me. I’m getting in touch with as many people related to the SWITCH book as I can to let them know that this is a serious problem. I’m sure I’ll have a post or two on the outcome of that effort.