Aaron's Worthless Words

Blog

  • CCIE R&S Written – Epic Fail (Again)

    Yes, I failed.  I think it’s pretty typical when you’re at Cisco Live, you stay out drinking and smoking cigars until 01:00, then you sit the exam at 08:00 the next morning.  Considering the situation I put myself in, I wasn’t very optimistic about passing, but I figured I had maybe a 40% chance to pass since I didn’t really even study.  Are you sensing a theme of ill-preparedness and self-sabotage?  Yeah, me, too.

    The big surprise of the morning was the quality of the exam.  For the very first time, I took a Cisco exam and didn’t feel like throwing the monitor across the room.  There were no blurry images.  The questions were in proper English with verbs and everything.  There were no spelling errors.  There were no questions which referred to routers as switches.  There were no questions outside the scope of the blueprint.  As a matter of fact, I can only remember one question that I even thought about questioning.  A VAST, VAST improvement over the last time I tried that exam.  It seems that Cisco’s exam quality is finally approaching that of Juniper; that really helps me stay motivated to take it again.

    What’s next?  Well, I obviously have more to actually study before taking it again.  I’m going to go back through the cert guide and my Boson practice exams to get back on track.  No fancy strategy.

  • Cisco Live 2013 Insights – Catalyst 3850

    Cisco Live is obviously the biggest networking event of the year, and Cisco likes to use all the attention to show off some of their new gear.  I must say I was impressed with some of the Enterprise offerings including the 6807-XL, the 6880-X, the 4451-X, and the Sup 8-E for the 4500-E (check out the Nexus 7700, too, even though they aren’t Enterprise class).  Those boxes definitely gave me a bit of a tingle when I was checking them out, but my eyes opened up when I saw the 3850 in one of my sessions and on the show floor.

    The 3850 is a new stackable switch platform with some really cool features.  Think 3750-X on steroids.  Here are some of the features.

    • Built-in wireless controller – Your wired and wireless clients now all terminate on the same fabric.
    • Unified access – Since the switch sees real traffic instead of just CAPWAP, then the same policies can be applied to both wired and wireless clients.  QoS, VACLs, etc.
    • StackWise-480 – Guess what the 480 means?  Yep…480 Gbps of stack bandwidth.
    • NetFlow – Get NetFlow stats off of every port without an additional module.
    • RSPAN – Guess what we should be able to monitor if the wireless clients are terminated on the switch?
    • Modules – 4 x 10G uplinks!
    • HA framework – SSO and NSF support
    • IOS-XE – No more monolithic IOS.  Now we have multiple cores and multiple threads on the switch.

    The best feature of all, though, is the price.  The retail price of the different 3850 SKUs is the same as the equivalent 3750-X.  No more 3750s, I guess.

    Send any trade-up vouchers questions to me.

  • Cisco Live 2013 Insights – Cisco Tactical Operations

    While walking through the World of Solutions, we ran across a big black truck with lots of antennas all over it.  It was obviously an emergency communications vehicle of some kind, but I was really surprised to see it was a Cisco truck.  It turns out that Cisco has a Tactical Operations group (Twitter) that was formed to provide disaster responders with much-needed communications for EMAs, fire, police, medical, etc.

    The big truck was the NERV – the Network Emergency Response Vehicle (PDF link).  It’s full of traditional HF, VHF, and UHF radios that the ham radio operators usually bring to these disasters.  This is a necessity when all phones, cell, and Internet are down.  It could be the only way fire fighters are able to call for reinforcements or the only way a hospital can call for more supplies.  The NERV, though, takes it to the next level.  On top of the radio gear, it is equipped with satellite uplinks for Internet access, wifi, and digital voice and video through UCS Express, IP phones, and Telepresence.  Analog voice is always the first method of communications restored via battery- or generator-powered gear, but an area will eventually need a network with voice and video.  That’s where the NERV comes in.

    TacOps also provides plans for small bug-out kits for smaller deployments.  The idea is that there are only a small number of NERVs, but there are dozens of situations every year where a hastily-formed network (HFN) is needed.  With these kits, emergency operations centers (EOCs) can have a way to set up a real IP-based network to support disaster relief.  If you’ve ever seen an EOC van or trailer, you know there are plenty of gadgets for monitoring and communicating, but all of them I’ve ever seen don’t really have a lot of network capability.  These kits would make a great addition to the traditional EOC van to allow data exchange on top of the analog voice.

    Do you know what the best part of TacOps is?  As was mentioned in the BRKSEC-1000 session, Cisco doesn’t give away much for free, but they’ve never charged a dime.  Amazing stuff indeed.

    Send any volunteer examiner exams questions to me.

     

  • Cisco Live 2013 Insights – Cisco Active Advisor

    Yes, I went to Cisco Live and survived.  It was the social event of the year, but the main focus is learning about the cool, new stuff.  One of the booths I visited was a demonstration of Cisco Active Advisor.

    This is a cloud-based (BINGO!) application that keeps an eye on the lifecycles of your IOS devices.  Using the web interface, you can scan a range of IP addresses from your machine and have your gear automatically added to the service.  Once in there, you can see, among other things, the warranty and support contract information for your device.  If your contracts is about to expire, it’ll let you know via email.   It also tracks any vulnerabilities that may apply and emails you if any are detected.  This beats trusting your reseller to send you renewals or watching an RSS feed for PSIRTs and field notices.

    The best part of CAA, though, is the price.  It’s FREE!  Definitely the cheapest thing in the world with a Cisco logo on it.

    Like anything that’s new and anything that’s free, there are some issues.  The first is that it only supports IOS devices.  There’s no support for your ASA or your Nexus switches.  Another problem is the fact that it runs on Java.  We all absolutely love Java, don’t we?  Ick.  The biggest problem to me, though, is the fact that you can’t manually add devices; you have to use the scanner or Cisco Network Assistant.  This will be problematic for networks where you use jump hosts to access certain networks with  no direct access.  It’s also not an inventory system; it doesn’t keep track of everything.  You won’t get a report of IP addresses or interfaces here.  That’s for your spreadsheet or real tool.

    CAA has some nice functionality, but I would like to see some more features.  Interfaces and networks would be good.  Maybe some configuration management (but I’m not putting my config online like that).  I’m still going to use it to track support contracts.

    Send any bullshit bingo cards questions to me.

  • A Simple Firewall Upgrade – A True Story

    I just got through a big weekend.  We upgraded our main production firewall, but the process had a few twists.

    The old firewalls, a pair of ASA 5520s, were running at about 80% CPU during the day.  That’s high enough that even I cringe when I saw the utilization in ASDM.  It was obviously time to upgrade to something with more beef, but we also wanted something that will last for years.  After looking around and getting some quotes (that made me jump back in my seat), we finally decided to go with a pair of 5555Xs.  These guys give about 10 times the throughput of the 5520 with about 8 times the memory.  Seems to match the requirements.  Now for the complications we had to work through.

    The old firewalls were running 8.2 and some change, and, as we all know, there is a huge jump in config from 8.2 to 8.3.  I won’t go into the details of the differences, but I dreaded having to go over that hump.  We thought that maybe we could just do a platform move as-is, but the 5555Xs actually don’t run anything below 8.6; that settled that.  Since the jump over 8.2 is the one to worry about, we decided to go with the latest-and-greatest 9.1(2) code.  We’ve already got a pair of ASAs running 9.1(1), so it’s not like it’s something foreign to us.

    We wanted to have a nice safe bosom to snuggle up to if something went wrong, so we decided to leave the old firewalls in place just as they were.  We talked about upgrading to 8.4 in one window then move to the new 5555Xs later, but we didn’t want to take two outages.  The plan was to install the 5555Xs, swing over to them, then move back when if something went wrong.  That means we’re left with taking an old config, converting it over, putting it on the new platform, and swapping them over.  A big ordeal for one window, so we had to get right into the preparation.

    After trying to figure out how to actually do the conversion, we figured that we just had to load the config up and let the firewall do its thing.  I’m surely glad, though, that we are all paranoid and wanted to do the conversion well before the window.  The first time was at about 3pm on a Monday.  When I got back into the office the next day, it was still running.  I figured out where in the config it was and did some math to see how much longer it would take.  Carry the 1…plus 4…divide by pi…12 days.  Yes, 12 days.  Let me say that again.  It was going to take 12 days!  One more time…12 days!  What was happening?

    Two of the big changes from 8.3+ were hurting us.  The first is that the NAT syntax is different; a big chunk of the NAT commands had to be converted to object-based NATs.  The conversion process had to find each static and convert it over to objects.  We didn’t have that many NATs (I think I counted 53), so that wasn’t what was causing the problem.  The big culprit was the conversion of the ACLs.  In 8.2 you use the NAT address (the inside global?) in the ACL.  In 8.3+, you use the real address (inside local?), so each ACE has to be expanded and changed to their real IP address.  All sources.  All destinations.  All lines.  All ACLs.  That’s usually no big deal, but the company I’m working for has a set of whitelists that are pretty big.  That’s not true.  They’re f&$^#*ing huge.  The main ACL expands out to 521,000 ACEs.  Yes, over half of a million lines that each need to be converted.  Guess how long that takes.  I would say about 12 days.

    The fix was easy.  We just had to find out what the huge ACEs were, take them out, and convert them by hand.  So, how do you do that?  Looking at the config doesn’t show which lines are any bigger than the others.  Doing a show access-list only shows how many elements are in the entire ACL and not the individual lines.  I wound up writing a terribly-constructed Perl script that goes through the output of show access-list WHATEVER and telling you how many elements are in each line. You can check it out here as long as you don’t laugh at my development techniques.

    The output was kind of scary, actually. We were able to see several lines above 100k elements and several more in the hundreds.  We picked a target size of 300 elements and checked out all the lines longer than that.  Some luck landed on our laps this time; most of the nested object-groups that made the lines so big were actually customer IP addresses and the rest were our own real IPs.  Of all the big hitter lines, only one referred to a NAT address, so we only had to convert one single object-group to reals.  Phew!

    Since the new firewalls are much beefier than our old ones, we decided to go with an EtherChannel so that the links wouldn’t be any sort of bottleneck.  If we changed the interface config after the conversion, there were issues.  Anything that references the nameifs would be removed from the config as soon as I blew away the original interface.  Makes sense, but that means NATs, access groups, failover monitors, AAA servers, etc., would up being unassociated or even removed.  The fix there was to change it before the conversion.  Pretty easy.

    Another struggle over, so we tried the conversion again.  We loaded up the config, copied it to start, and rebooted with our fingers crossed.  Thirty-nine minutes later, the config was running on 8.6.  I would say that qualifies as a decent improvement, don’t you?  After putting the big hitters back in (along with a few minor changes), we saved the config and rebooted to 9.1(2) without incident.  Woot!

    It was quite a few weeks discovering issues, fixing them, then waiting an hour to see what happened.  The end result, though, was a brand new pair of 5555Xs running 9.1(2) that run at 18% CPU during peak.  I feel better already.

    Send any Cisco Live meetup schedules questions to me.

  • My Schedule for Cisco Live 2013

    I’m all set up to go to Cisco Live in Orlando this year.  Good thing, too, since I couldn’t make it to San Diego last time. It’ll be a great and fun time as usual, and I’m quite excited.

    As it turns out, ARRL Field Day happens to be the weekend leading up to the festivities.  I’ve been in contact with the local Orlando club, and they say the attendees are more than welcome to join them.  They are meeting at the City of Orlando Emergency Operations Center, which is about 20 minutes away from the Convention Center.

    Anyway, here’s my schedule for the week.

    Saturday, 22 June

14:00 : Field Day begins and runs for 24 hours.  
17:00 : I'll make my way over to Field Day for a few hours.

Sunday, 23 June

Open

Monday, 24 June

08:00 : CCIE R&S Written (again)
10:00 : BRKARC-3437 - Cisco Catalyst 3750 / 3560 and 2960 Series Switching Architecture
13:00 : BRKARC-2013 - Cisco Nexus 3548 Switch Architecture
15:30 : GENSK-1294 - Enterprise Network Keynote: Getting You Where You Want to Go

Tuesday, 25 June

08:00 : BRKDCT-2218 - Scalable Midsize Data Center Designs
10:00 : GENKEY-1295 - KEYNOTE: Tomorrow Starts Here
12:30 : BRKSEC-3021 - Maximizing Firewall Performance
15:00 : BRKSEC-2020 - Firewall Deployment

Wednesday, 26 June

08:00 : BRKSEC-1050 - Are you choosing the right VPN technology for your network?
10:00 : GENKEY-1296 - KEYNOTE: Unlocking the Value of Innovation
13:30 : BRKIPM-2264 - Multicast Troubleshooting
16:00 : BRKRST-2041 - WAN Architectures and Design Principles

Thursday, 27 June

08:00 : BRKRST-2513 - QoS Design For IPSec VPNs
10:00 : BRKSEC-2691 - Identity Based Networking: IEEE 802.1X and Beyond
12:30 : BRKCRS-3090 - Implementing Network Automation
14:30 : GENKEY-1297 - Celebrity Closing Keynote
16:00 : BRKSEC-2014 - Identifying and Mitigating Network Threats

Friday, 28 June

Unknown : Leave for home

    Does this mean I’m actually going to attend over a dozen classes?  Probably not.  I’ve got to save some time for socializing and exploring the World of Solutions.

    Send any Game of Thrones premiers questions to me.

  • JNCIS – Epic Win (Again)

    I spent the last of my Juniper exam vouchers on the JNCIS-SEC exam and passed by the skin of my teeth today.  Since I took a new job last month that’s 100% Cisco, this is the last Juniper exam I’ll take for the foreseeable future.  Too bad, too.  I really like the Juniper exams.

    At my previous job, we were 90% Juniper with a whole mess of SRX firewalls around the world.  Since this exam is really about that platform, it was pretty logical that I should do alright on it.  Of course, a large part of the blueprint was on IDS and UTM, and I have no experience there.  For my entire career, those type of devices have been handled by other groups, so I had some studying to do.  That’s where I ran into problems.  I have absolutely no interest in IDS.  I have no interest in UTM.  There’s nothing about content scanning and analysis that interests me at all.  I promise you all that I tried my best to read up on these topics, but I was asleep after 10 words every time I tried.  After rescheduling the exam twice to try and study a bit more, I finally decided it wasn’t worth the trouble and just took the exam…and passed.

    The exam was typical Juniper with clearly-worded questions and perfectly-clear exhibits all around.  A near-perfect exam yet again from Juniper.  I was disappointed by three questions, though.   The problem wasn’t with the technical details; they were just worded terribly.  I’m definitely not shy about commenting on questions during the exam, so hopefully the exam team can use my comments to improve those bad apples.  I’ll miss these exams; Cisco surely doesn’t produce any exam of this quality.

    Send any Final Four tickets questions my way.

  • Junos – Logical Tunnel Interfaces with Virtual Routers

    There are a few ways to leak routes in and out of virtual routers in Junos. On the list is a cool feature called the logical tunnel interface.

    So, what am I talking about?  One way to separate traffic on a router is to use virtual routers (VRs) so that you wind up with multiple routing tables on the same router.  This separate traffic, but you will usually (read: always) have a demand to get traffic from one VR to another.  There are a few different way to do that (see rib-group, instance-import, next-table, et al.), but one really cool way to do it is through logical tunnel interfaces.

    The logical tunnel (lt-0/0/0) interface is a special little guy that allows you to connect its units to each other.  The result is similar to connecting an Ethernet cable from one physical interface to another. With a little configuration, these guys provide a point-to-point interface that you can include in your routing setup.  Let’s look at an example.

    set interfaces lt-0/0/0 unit 100 encapsulation ethernet
set interfaces lt-0/0/0 unit 100 peer-unit 200
set interfaces lt-0/0/0 unit 100 family inet address 192.168.0.100/24

set interfaces lt-0/0/0 unit 200 encapsulation ethernet
set interfaces lt-0/0/0 unit 200 peer-unit 100
set interfaces lt-0/0/0 unit 200 family inet address 192.168.0.200/24

    The encapsulation lines are pretty straightforward. In this case, I want the link to appear as an Ethernet interface, but you can choose frame-relay, vlan, bridging, and others.

    The peer unit lines tell what unit is connected to what other unit. Each lt-0/0/0 unit is a point-to-point link, so you have to tell the router what’s on the other end of the link (sorry…no multiaccess here). In this case, I want unit 100 to connect to unit 200, so I configure both units with the appropriate peer unit.

    Of course, we can all figure out that we’re using IPv4 on these new units as well. In this case, I’ve put both interfaces on the 192.168.0.0/24 network (how original!).

    Now that we have our interfaces configured, we need to put these interfaces into the correct VR. Don’t forget the security zone, too, if you’re running in flow mode. That’s beyond the scope here, though.

    set routing-instances VR100 instance-type virtual-router
set routing-instances VR100 interface lt-0/0/0.100
set routing-instances VR100 interface lo0.100

set routing-instances VR200 instance-type virtual-router
set routing-instances VR200 interface lt-0/0/0.200
set routing-instances VR200 interface lo0.200

    I put lo0.100 and lo0.200 in there to have something to advertise. You’ll see that in a second.

    Now, let’s see if we can ping across.

    root@TestSRX# run ping 192.168.0.100 routing-instance VR200
PING 192.168.0.100 (192.168.0.100): 56 data bytes
64 bytes from 192.168.0.100: icmp_seq=0 ttl=64 time=1.879 ms
64 bytes from 192.168.0.100: icmp_seq=1 ttl=64 time=3.480 ms
<SNIP>

    Woot! It works. Now we can treat these new interaces as if they are regular ole Ethernet. Since I’m not ready to try and blog about IS-IS, let’s just use the standard OSPF. I’m not going to go through the steps to configure OSPF, but here’s the routing table after all the interfaces are included.

    root@TestSRX# run show route                  

VR100.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.100/32      *[Direct/0] 00:41:33
                    > via lo0.100
10.0.0.200/32      *[OSPF/10] 00:00:28, metric 1
                    > to 192.168.0.200 via lt-0/0/0.100
192.168.0.0/24     *[Direct/0] 00:38:21
                    > via lt-0/0/0.100
192.168.0.100/32   *[Local/0] 00:38:21
                      Local via lt-0/0/0.100
224.0.0.5/32       *[OSPF/10] 00:01:18, metric 1
                      MultiRecv

VR200.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.0.0.100/32      *[OSPF/10] 00:00:28, metric 1
                    > to 192.168.0.100 via lt-0/0/0.200
10.0.0.200/32      *[Direct/0] 00:41:33
                    > via lo0.200
192.168.0.0/24     *[Direct/0] 00:38:21
                    > via lt-0/0/0.200  
192.168.0.200/32   *[Local/0] 00:38:21
                      Local via lt-0/0/0.200
224.0.0.5/32       *[OSPF/10] 00:01:18, metric 1
                      MultiRecv

    Look! OSPF routes! Sweet. Just to keep my OCD at bay, let’s ping from the loopback of one VR to the loopback of the other.

    root@TestSRX# ping source 10.0.0.100 routing-instance VR100 10.0.0.200
PING 10.0.0.200 (10.0.0.200): 56 data bytes
64 bytes from 10.0.0.200: icmp_seq=0 ttl=64 time=1.463 ms
64 bytes from 10.0.0.200: icmp_seq=1 ttl=64 time=1.443 ms
<SNIP>

    Well, look at that.  It works again.

    Let’s look back at the topic we’re discussing, though.  If we use OSPF between the VRs, we need to make sure our routing design allows us to filter routes between the VRs; the risk is that you may wind up having all the routes from each VR advertised to the other.  Kind of defeats the purpose, eh?  Running BGP between the VRs might be an option that allows you to control what routes go in and out.  Statics might be the answer, as well.  As long as you can filter the advertisements, you wind up with a pretty elegant solution for sharing routes between VRs.

    Send any Marshmallow Peeps questions my way.

  • JNCIS – Epic Win!

    I quit my job…by design.  I start a new gig on Tuesday and am getting back to the world of Cisco.  As a last nod to Juniper, I decided to use an exam voucher I had and take the JNCIS-ENT exam.  Easy pass.

    The content was right along with the exam objectives, so there were no surprises.  Most of the topics are things I’ve done a thousand times on the job.  There were some things, though, that were beyond my experience.  IS-IS was the big one.  The very first question I got was about IS-IS metrics, and I had absolutely no clue what the answer was.  Nor did I have any clue about the other IS-IS questions.  I went 0-for-3 on those guys.  The only other problematic topic was HA, which didn’t really surprised me.  I was able to answer the VRRP questions, but  I’ve never done any GRES, ISSUe, RTG, etc., at any point in my career.  It wasn’t surprising that I didn’t do too well on those.  Everything else was cake, and I only missed 6 questions in my comfort zone.

    The exam was yet another top-notch effort from Liz and the group, but there was one questions that didn’t meet the standard set by the others.  It was a VRRP question, but it used some awkward wording that that I read over and over.  I just used the context of the questions to give an answer and moved on.

    There was really nothing else to report.  It was a great exam, so don’t be afraid to take it if it’s next on your list.

    Send any Cisco refresher courses questions my way.

  • Goals for the New Year

    Yes, I know I’m late.  Just remember I’m lazy, and it all makes sense.

    This year I’ve decided to go a little more practical with my goals.  Instead of “get this cert” or “learn about that”, I’ve decided to take some steps to help myself.  That is, in order to learn and advance, I need make sure I give myself the opportunities to do so.  Damn, that sounded like some crap from a marketing department, so let me use my own words.

    • Find a place to study.  When I took my current job, we moved to another city and to a very small apartment.  Now I don’t have the spare bedroom as an office and a quiet place away from everything.  Of course, my study schedule and quality has suffered because the only place to study I have now is less than 10 feet from the wife and the TV.  I’ve got to find a place that’s quiet.
    • Find a place to lab.  Right now, I’ve got a pile of Cisco and Juniper gear sitting on a shelf at that same study location..  I’ve either got to find a way to access these remotely (somewhere quiet) or find a service to employ to get hands-on experience.
    • Go to Cisco Live US.  After missing last year, I won’t let myself miss it this year.  It’s within driving distance, and, more importantly, the hotels will be cheap again in Orlando.  There’s no excuse not to go.

    If I can accomplish these goals, then I’ll be back on track for getting some new certs or finishing out some old goals.  And I seriously do miss studying and learning.

    See you guys at Cisco Live.  Yes, I’m going to wear my Juniper Ambassador shirt.