Aaron's Worthless Words

Last week, @fletcherjoyce posted an article on his blog about his positive experiences with Cisco’s 3750 switches.  If you follow my complaints tweets, you know that I’ve had quite the opposite experience with them.  I would never pick on anyone, but I had to throw in my 2 cents.

I’m guessing here, but we have about 50 3750 stacks in the enterprise.  Most of them are pairs, you wind up with roughly 120 switches.  Since we’ve done about 20 replacements over the last 5 years, that means we have a 17% failure rate.  That’s pretty horrible, isn’t it?

For the most part and with few (if any) exception, we use the 3750s as aggregation points for our access switches.  We don’t do QoS on them.  We don’t do any access control on them.  We don’t even do routing on them.  They’re simply used to connect all the access switches in the closet to the core, so they’re not doing anything funky or burdensome.  The CPU and memory are always well within normal operating parameters.  They just fail and fail repeatedly.

The flies started dropping in closets at our corporate headquarters a few years ago.  It was the middle of summer, and the temperatures kept rising to over 90F (32C) until the we lost 3 switches in 3 weeks.  If you could stand to make it into the closet, you could feel that the sheet metal of the switches was hot enough to make you pull your hand back!  When the facilities team added more cooling, the temperatures dropped to around 82F there (28C), but we continued losing switches.  I figured the newly-failed switches were feeling the effects of the earlier heat wave and were just getting around to giving up the ghost.  Surely the heat was the culprit.

A few months after our headquarters meltdown, a tech for a satellite office called and asked if we could help with some latency issues.  He showed me the switch stacks throughout the building, and I noticed that only one of the 10 switches actually had a label.  The tech said that he never got around to relabeling them after they were replaced.  Some, he said, had been replaced multiple times.  The closets were running about 76F (24C), so heat didn’t seem to be the problem at this location.  The closets were clean as a whistle, and everything in the racks was on building UPS.  I couldn’t find a pattern at all.  For the record, all their latency issues were related to two unrelated 3750s.  Two RMAs later, and their problems were gone.

I’ve been trying to find patterns for the failures, but I can’t think of any.  If it’s heat, humidity, power, dust, etc., then why are we not replacing 2950s as well?  There are 4-10 of them for every 3750s stack we have.  We’re replacing them, but it’s a rate of less than 1%.  If it is environment, then the 2950s are English hooligans compared to the 3750s being French aristocracy.  Maybe it’s sabotage.  I still don’t know after years of watching RMA after RMA come in.

I have noticed one pattern, though.  The only deployments of 3750s that have never had a problem are in data centers.  They seem to love any room that has an ambient temperature of 62F (16C) with less than 40% humidity and large volumes of air flow.  If only we could install micro-data centers in all our closets, then I would be a happy network dude.

Send any wooden shoes questions my way.

Edit:  I went back and checked our TAC cases to see what switches we actually replaced.  It turns out that we’ve done 19 replacements, and they’ve all been 3750G-12S-S switches.

Another year of Aaron’s Worthless Words has come and gone.  This month marks the third full year of blog posts for me, and things sure have changed since the beginning.

At first, this blog was just for my personal rants, but no one cares about that stuff (thus the title), so I looked to move on to something else.  I decided that I would go into the non-technical side of the network field, so I started talking about the Principle of Least Privilege and about cabling standards.  That got a bit boring, so I started puking out information on the Content Switching Module from Cisco since I couldn’t find anything worth a cracker outside of the documentation.  That was a hit, and the topics started expanding and expanding until we got to where we are now.  Today, the articles are published in online magazine and are being translated into other languages around the world.  Quite a change from complaining about drivers stopping in the crosswalk. 

Here’s to another year of good times and good information.  Salud!

We ran into an issue the other day where an application was setting the DF bit in IP packets to 1.  We thought it may be causing problems, so we looked at setting up a route-map to set the DF bit to 0.  It turned out to be a different application problem, but it was a good exercise in looking at what you can do with route-maps and policies.

I set up a lab in GNS3 to replicate and do some captures.  It’s a really simple setup.  R1 connected to R2 connected to R3.

Beyond the routing, here’s the config we put on R2.  You configure it just like PBR, but, instead of setting the next hop or interface, you just set the DF bit to 0.

route-map E0/0-IN permit 10
 set ip df 0
!
interface Ethernet0/0
 description To R1
 ip address 10.0.0.2 255.255.255.0
 ip policy route-map E0/0-IN

Really simple.  We’ve all done this many times, but I’ve never cared enough about the DF bit to have to set it on a router.  That’s usually what the applications does, and, to tell the truth, I imagine that setting it only shows up in 0.009% of networks.

To test the new policy, I did a packet capture between R1 and R2 and another between R2 and R3.  R1-R2 shows the DF bit set to 1 on a ping I did from R1 to R3.  For those that care, I did a ping 10.0.1.3 repeat 100 df-bit to generate this traffic.

The capture from R2-R3 shows the DF bit on the same packet set to 0.

For those that don’t know, when I say “stack”, I mean a group of 3750s connected together using the StackWise technology.  When you use a very expensive and very proprietary cable, your individual switches are combined into a single logical device.  This means you configure one device to control potentially many switches.

To the point.  I’ve spent the last few weeks replacing a mess of 3750s in stacks.  These guys are very easy to replace, but the big problem I find is getting the IOS version in sync.  When the RMA comes, it’s inevitably got a different version on it, and you’ll see something like this.

Switch#  Role      Mac Address     Priority     State
--------------------------------------------------------
 1       Member    0023.33ad.a500     1         Version Mismatch
*2       Master    0023.5eac.e900     15        Ready

In this case, switch 2, running 12.2(25)SEE3 is the master, and switch 1, running 12.2(35)SEB, is the new member.

A switch in the stack needs to have the same version of IOS as the master to be brought into the fold, so you’ve got to do get the code on the switch somehow.  You can use traditional methods to get the right code on the box (like TFTPing one up), but there are easier ways.

If the code versions are close (I’m not sure as to the rules about how close), then you’ll be able to check out the flash of all the switches in the stack through the master.  If you do a dir ?, you’ll see flash1:, flash2:, etc.  Those are the individual flash drives for each switch in the stack.  The numbering is based on the stack number, so all you have to do is copy the IOS image over from the master.  Well, maybe.

I found that a lot (maybe all) of my 3750s actually have a directory structure to include the image and HTML files for the device manager.  You don’t need the HTML files for the switch to function in the stack, but having the IOS image in different places will force you to change the boot variable for each switch.  I like consistency, so I put everything in the same place when I can, so that means creating the directory by hand.

That method works fine, but there’s an easier way using the archive command.  You can tell your switch to tar, copy, and extract all of the files from one switch to the others by giving it one of these.

archive copy-sw [ /destination-system X ] Y

I used this command to copy the image from switch 2 (the master) to switch 1 (the replaced member) and got a whole bunch of output.  The whole process took about 3 minutes.

Switch#archive copy-sw /destination-system 1 2
System software to be uploaded:
System Type:             0x00000000
archiving c3750-ipbasek9-mz.122-25.SEE3 (directory)
archiving c3750-ipbasek9-mz.122-25.SEE3/c3750-ipbasek9-mz.122-25.SEE3.bin (7206732 bytes)
archiving c3750-ipbasek9-mz.122-25.SEE3/html (directory)
archiving c3750-ipbasek9-mz.122-25.SEE3/html/toolbar.js (7084 bytes)
archiving c3750-ipbasek9-mz.122-25.SEE3/html/title.js (577 bytes)
...
extracting c3750-ipbasek9-mz.122-25.SEE3/html/images/141280.gif (3053 bytes)
extracting c3750-ipbasek9-mz.122-25.SEE3/html/images/meter_yellow.gif (59 bytes)
extracting c3750-ipbasek9-mz.122-25.SEE3/html/images/legend_off.gif (1158 bytes)
extracting c3750-ipbasek9-mz.122-25.SEE3/info (682 bytes)
extracting info (108 bytes)

Installing (renaming): `flash1:update/c3750-ipbasek9-mz.122-25.SEE3' ->
                                       `flash1:c3750-ipbasek9-mz.122-25.SEE3'
New software image installed in flash1:c3750-ipbasek9-mz.122-25.SEE3

All software images installed.

I left out a large chunk, but you get the idea.  If you’ll notice, all the HTML files are copied along with the IOS image, so you get exactly the same structure on all your switches.  It beats doing it all manually.

Send any RMA packages questions my way.

If you follow the blog, you know I’ve had quite an adventure getting my CCNP.  Finally, this past Monday, after what seemed liked years of struggling, I finished up my ROUTE test and got the email telling me I’d made it.  I’ve learned a lot over the course, but, more than the technical details, I learned more about how to prepare for the exams.  It’s too bad I hit the moment of enlightenment after I reached the end of the line.  Well, at least this line; there will be others very soon.

Here’s a list of some of the things I’ve learned about preparing for Cisco exams.  I’m not the authority on test taking, mind you, but I’ve done my fair share of test taking.  I hope someone can use the advice and not have to struggle through things like I did.

• Create a blog and update at least weekly.

A blog  makes you check your facts beyond a shadow of a doubt.  No one wants to get embarrassed by posting inaccurate information, and, by forcing yourself to go public with your studies, you’ll find yourself going that extra step.  One of my big problems was that I would read a topic, think about it for a second, then move on thinking I knew it thoroughly.  Usually I was wrong, and my blog actually forced me to go through a topic over and over again and focus on the details.

• Get on Twitter.

I try to avoid using buzzwords, but the collaboration is just astounding in the Twitterverse.  There are some great people on Twitter who are more than willing to help you out and provide motivation.  I’ve found that CCIEs are helping CCENT candidates, and CCNAs are all exchanging study notes.  If you don’t understand a topic, use your 140 characters to ask, and someone will usually help.

• Read more than one book.  Read more than 2 books.

The latest CCNP exams (ROUTE and SWITCH at least) go beyond any one book.  In order to understand a topic, I had to get the same information from several different places to understand what was really going on.  My ONT experiences taught me that.  I used only one book to study for that test, and that was a huge mistake.  The three chapters on wireless weren’t enough to cover the shear number of wireless questions on the exam.  I finally passed when I got other books involved, and, without them, I may still be scheduling that exam weekly trying to pass it. 

• Don’t rely on the official certification guides.

The official certification guides are notorious for being incomplete.  I had purchased the library a few years back, so I used those to study for the ISCW,  ONT, and BCMSN.  In retrospect, I can see that they are a good desk reference, but they fail to cover a lot of topics and situations that appear on the exams.  See “Read more than one book.”

• Use the blueprint.

The blueprint is Cisco’s official word on the topics on the exam, so make sure you understand each and every bullet thoroughly.

• Read blogs.

There are lots of network blogs out there, and they’re all worth your time.  Add them all to your favorite RSS reader and check them out daily.  Even if the author is speaking above your technical level, keep reading.  You will catch up quickly and learn a lot of valuable information.

• Read blog comments.

Make sure you look through the comments of a blog post you find interesting.  Someone may have left some information that will put you over the top of the understanding mountain.

• Study above the expected level of knowledge.

I’m not saying you should read CCIE books for your CCENT exam, but answer any questions you think of when studying.  If you’re reading about OSPF and read about type-3 and type-5 LSAs, natural curiosity should lead you to ask what type-4s do, right?  Well, what’s wrong with finding out and adding that to your notes?  You’re going to need that information later anyway.

• Learn the terminology.

One thing that got me was not knowing what terms were defined to mean.  I knew the general definitions of them, but, when presented outside of certain contexts, I had to scratch the old bean to figure out what the question was really asking.  If you encounter a term, define it in your head fully.  Don’t just picture it and move on.  A complete definition is the way to go here.

• Read the glossary.

After you are confident that you know a topic, go through your books’ glossaries and define everything from beginning to end.  Yes, it’s very daunting and boring, but it’s a great test of the terminology and reinforces how different terms are related.  You may know the definition of a term, but you may not realize how it’s related to a topic or line on the blueprint.  As an added benefit, the glossary also tells you where to look if you don’t know the definition.

Send any CCIE study guides questions my way.

Woohoo!  I passed the ROUTE test this morning.  That means I’m done with the CCNP track! 

If you remember, I took it over a week ago and had some bad luck on it.  Alright, bad luck is the wrong phrase.  I didn’t study enough and failed it.  This time, though, I had a special weapon on my side – the ROUTE Foundations book.  I haven’t used the Foundations books before, but, I saw some tweets about this one, so I picked it up off of Safari.  In just a couple pages, I realized that I was reading the answers to several questions directly out of the book.  It was amazing.  I only studied my weak points and wound up with 144 more points than I did last time.  I can’t say that was entirely because of the book, but I must say it was a big reason.

The test, like last time, was actually really good.  The questions were well-written and clear for the most part.  There were, of course, some that were confusing, but there weren’t any traps like you usually see in the other tests.  A couple asked you to do contradictory things.  There were a couple that just blasted you with information, but, if you read the question and know the material, the answer just pops right out at you.  Overall, another great test.  That makes 2 I’ve taken…and they’re both the 642-902. 

I’m quite excited about finishing up.  I’ve had a lot of failures along the way, but the support from the online community has been tremendous.  Thanks to everyone who kept pushing me and telling me I could do it.  I’m also happy to report that I kept the testing costs below the cost of the CCIE lab (barely) and that I may hold the record for number of P-level tests (8 P-levels tests and 10 overall) and overall Cisco test questions answered (535 questions).  What an honor.  *denotes sarcasm*

So, what’s next?  I think I’m going to take a month or two off from networking to study up for a ham radio license.  We get a lot of hurricanes down here, and having a good radio around will help us and the community out if such a disaster happens.  I’m calling it prep for CCIE-Wireless.  Heh.  When I get done with that, I either plan on hitting up the CCIE-R&S or going down the CCDP track.  I’m not really sure, but we’ll see when we get there.

Don’t worry.  The blog will stay network-related.  With the feedback from the study questions format, I think I’ll start a problem question and answer format.  I’m also thinking of generating scenarios to work through.  Again, we’ll see when we get there.

Send any trips to Delaware questions my way.

Audio Commentary:

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

As always, corrections are requested.

Study Questions

• I’ve got IGRP and EIGRP both configured with the same AS number.  What’s special about this configuration?

If both use the same AS number, then they automatically redistribute their routes into each other without using the redistribute command.

• When redistributing one IGP into another, where’s a good place to filter routes?

There’s no one good place, but at the router(s) that’s doing the redistribution is a good start.  There’s no need to send an IGP a bunch of routes it doesn’t need.

• When redistributing one IGP into another, where’s a good place to summarize routes?

There’s no one good place, but that may be best done at the router just inside the redistributing router.  If the redistributing router only sees the summary route, that’s what it will pass to the other IGP.

• What’s the default metric of RIP?

That’s infinity, so it’s unreachable with an explicit metric.

• I’ve redistributed OSPF into RIP, but I don’t see my subnets there.  What gives?

RIP automatically summarized routes, so look for summaries instead of specific subnets.

• How can you limit the number of routes redistributed into EIGRP or OSPF?

Use the redistribute maximum-prefix X directive under the routing protocol, where X is the maximum number of routes.

• What are the metrics of connected routes when redistributed into EIGRP?

Those routes take the metric of the associated interface instead of using the metric you gave to the redistribution.  [This seems fishy at best.  Can anyone help clarify, please?]

• I have 845734928 interfaces on my router, but I only want to use 3 of them for EIGRP and only want to configure a single network statement.  What’s the easiest way to do that?

Set all the interfaces as passive with the passive-interface default router subcommand.  Next, make all your interesting interfaces non-passive with the no passive-interface X subcommand.  Now you can configure network 0.0.0.0 255.255.255.255 to match all the interfaces, but only the interesting interfaces will participate.

• What is the term for the rank of trustworthiness a routing protocol provides?

Administrative distance

• How can I change the AD of external EIGRP routes to 201 while keeping the default AD for internal EIGRP routes?

Router1(config-router)#distance eigrp 90 201
You have to set both, so you’ll have to remember that EIGRP has an AD of 90 for internal routes by default.

• How can I change the AD of OSPF routes to 192.168.0.0/24 to 202?

Router1(config)#access-list 88 permit 192.168.0.0 0.0.0.255
Router1(config)#router ospf X
Router1(config)#distance 202 0.0.0.0 255.255.255.255 88

• Is it possible to set the AD of different OSPF routes types like intra-area and interarea?

Yes.  You can give it the old distance ospf inter-area X to change the AD.  It also works for intra-area and external routes.

• Is it possible to set the AD of an external OSPF route to 192.168.100.0/24 to 202 without changing the others?

I would have though you could use a route-map for that, but I can’t find a proper set command in a route-map.  [A little help, please.]

I didn’t do so well on IGP redistribution the last time out, so here’s some more stuff to study.  As always, feel free to correct.

Study Questions

• What three things are needed to be able to redistribute one routing protocol into another?

1. One or more links into each routing protocol
2. A proper, working config for each protocol
3. The addition of the redistribute command to one or more of the protocols

• You just configured OSPF to redistribute EIGRP routes, but EIGRP, with the network statement of network 0.0.0.0 0.0.0.0, is configured with a passive interface.  Does this interface’s connected network get redistributed?

Yes, it does.  Even if it’s not participating in routing, it’s still an interface that EIGRP is configured to use, so it goes along for a ride on the redistribution train.

• Name three ways to set the metric of redistributed routes in EIGRP.

1.  default-metric …
2.  redistribute X metric …
3.  redistribute X route-map …

• How can I set the metric for all OSPF routes redistributed into EIGRP?

Use the redistribute ospf X metric command.

• You are redistributing OSPF into EIGRP and want to set the metric of one particular route to another set of metric values (BW, delay, etc.).  How do you do that?

Use a route-map to match the single route and to set the new values.

• Routes from what routing protocol need a metric set when redistributing into EIGRP?  Routes from what protocols don’t?

Routes from another EIGRP instance have their metrics copied over; all others need to have it set.

• What’s the default metric of a BGP route when redistributed into OSPF?  EIGRP?

BGP has a metric of 1 in OSPF.  There is not default metric in EIGRP without some configuration.

• You left out the subnet keyword when redistributing EIGRP into OSPF.  What is the result?

Only classful routes will be redistributed and only if EIGRP has a classful route to redistribute.

• You left out the subnet keyword when redistributing OSPF into EIGRP .  What is the result?

There is no subnet keyword for redistribution under EIGRP.

• Routes from what routing protocol need a metric set when redistributing into OSPF?  Routes from what protocols don’t?

OSPF set metrics automatically.  If the route came from another OSPF process, the metric is copied over.  If the route came from BGP, the metric is set to 1; if it came from any other routing protocol, the metric is set to 20.

• What are three ways to manipulate the metric of redistributed routes in OSPF?

1.  default-metric …
2.  redistribute X metric …
3.  redistribute X route-map …

• My ASBR is advertising static routes into area 0, but I’m not seeing any type-5 LSAs in area 1.  What’s gives?

Assuming everything else is configured correctly and no filtering is done, area 1 is probably a stub area of some kind.

• My ASBR is advertising static routes into area 1, but I’m not seeing any type-5 LSAs in that area.  What gives?

Area 1 is probably an NSSA or totally NSSA area, so any external routes are flooded as type-7s – note type-5s.

• If I look at the OSPF database on my router, I see that a whole bunch of type-5 LSAs advertised from the router with the ID of 1.1.1.1.  What does that say about that router?

Among other things, that router is an ASBR and is redistributing external routes into that area.

• I see several routes in the OSPF database with a cost of 20.  What metric type are those routes?

More likely than not, they are type-2 routes (O E2).

• I have two type-5 LSAs for the same network through two different ABRs; both are of the type-2 metric.  How does the router decide which one to use?

Since both routes are E2, they will have a metric of 20 (unless manipulated somehow), so looking at the intra-area cost results in a tie.  The router will then look at the type-4 LSAs which contain the cost from the ABR to the ASBR.  Since each ABR floods these type-4s, the router knows which ABR is closer to the ASBR advertising the route.  The lower metric in the type-4 LSAs wins.

• I have two type-5 LSAs for the same network through two different ABRs; both are of the type-1 metric.  How does the router decide which one to use?

Since both routes are E1, the costs to the ABR are first compared since they may be different.  If tied, the type-4 LSA’s cost to the ASBR is compared.  If still tied, the external (type-5 LSAs) cost is compared.

• I have two type-5 LSAs for the same network through two different ABRs; one is type-1 and the other is type-2 ?  How does the router decide which one to use?

E1 routes are always preferred over E2 routes.

• Why can’t you redistribute static routes into a stubby network?  How can you make it work?

Stub networks do not flood type-5 LSAs, so the routes cannot be advertised into the area.  You can change it to a regular area to make it work.  You can also make it an NSSA or totally NSSA area.

• How do OSPF routes that come from type-7 LSAs appear in the routing table?

They appear as “O N1″ or “O N2″ depending on the metric type.

Hey, guys.  Head over to the Packet Pushers Podcast site and check out the latest episode.  You’ll hear my soothing, wonderful voice discuss some of the rantings of the week including career development.  It was fun hanging out with Greg, Dan, and Ethan.  They have a great podcast over there; be sure to subscribe and review!

Packet Pushers Podcast Episode 11: If You Can’t Be Replaced, You Can’t Be Promoted

I took the ROUTE test today and failed like I usually do.  That makes me 3-4 on these P-level tests if you’re scoring at home.  Don’t worry, though.  I’m not giving up. 

In atypical fashion, I must say that the ROUTE test was a good test.  Let me say that again.  The ROUTE test was a good test.  I said good, though…not great.  There were a few problems with it that I’ll get to, but, overall, this is the best test I’ve ever taken for a Cisco cert.  The questions were very well-written and there were no obvious omissions or wrong details.  I failed this test because I simply didn’t put in enough work.

It wouldn’t be a complete test experience without a workstation crash, though, and I had one right away.  The test guy logged me in, and I started the test.  Like all of the Cisco tests, this one started with the same tutorial that walks you through how to use the interface and whatnot; I’ve gone this enough times, so I didn’t need to look at that again.  I clicked “End Tutorial” and was asked to confirm.  When I clicked “Yes”, the desktop (sans icons) was showing unexpectedly, and nothing was happening.  I waited for 3 minutes before asking for help.  This center hadn’t ever given a Cisco test (they just got the stuff to give Vue tests), so they called up Vue’s support for some answers.  It seems that this is a known problem (with this test?), and a hard reboot would fix it.  It did, and I found myself back in the game.

Like I said before, the test was good.  I might even give it a very good if not for the lockup.  Every question was clearly written by what seemed to be an English-speaking author.  There were no difficult phrases that I had to read over and over again to interpret.  When I read the question, it was obvious what I was being asked to do.  I didn’t have to guess like a lot of the questions on the SWITCH test, and, let me tell you, having a verb in every sentence makes things easier.  I also only had one or two questions where I had to infer a piece of missing information.  It usually helps if you have all the information, and this test did a really good job of doing such in comparison to the last 8589248 tests I’ve taken.

There was one particular question that I found annoying, though, and I let someone have it in the comment section.  A diagram showed a about 10 routes all interconnected in various patterns with the Internet and data center connected into the mesh.  Each link had a number on it with no description of what it was…it was just a number.  The question asked how many possible paths there were from the data center to the Internet if a certain setting change was made (I don’t want to violate NDA here).  Well, the setting had a lot to do with routes submitted to the RTM, and question didn’t ask about routes.  It specifically said paths.  I counted 18 different paths from one end of the network to the other.  Too bad the answers were in the low single digits.  I just guessed at that one since I had no idea what they were asking.

As you figured out by now, this is yet another testing center.  I think this makes the 7th one I’ve tried over my testing career, and it was a duesy.  This one is only 30 minutes away, so it’s much more convenient than all the others, so it’s got that going for it.  The building it’s in is pretty old, but the room is well insulated from the outside.  On top of that, the town it’s in is a sleepy little fishing/tourist village, so all the traffic is at the other end of the road which makes for a very quiet facility.  The people were great, too.  They were helpful and apologetic about the lockup (even though I assured them it was a Vue problem like I’d seen in the past).  Plus, do you know any testing center with a view like this?    I’ll definitely go back there again for my tests.

Moral of the Story:  Cisco put out a good test, but I’m too lazy to pass it.

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.