Archive for the ‘Uncategorized’ Category
Here’s To Another Year
Here we are at the end of another year. It’s been a pretty good one for me – the wife got a new job, we have a new house, we’re in good health. I hope that you, too, have had a great year.
Here are some highlights from 2009 to read over while your boss is out on vacation. I know that I’m a little wiser thanks to the many, many network-related blogs out there, and I hope that I was able to give someone an answer or just some new knowledge in 2009.
- Filtering Out the Noise on the Edge – Knocking down network noise on your edge routers
- Fail Actions on CSM Serverfarms – Telling the CSM what to do with connections bound for failed servers
- An Interesting Problem with Mulple DCs on a Stick – Making sure traffic originating from a serverfarm member goes out the right VLAN
- The Most Random Things Can Hurt a Network – Yes, a piece of paper is hazardous to the network
- ASA and Proxy ARP – Proxy ARP on the ASA may be different than the old PIX or FWSM
- Object Groups in the ASA/FWSM/PIX – Object groups can save a lot of time
- ISCW Notes – Access List Resequencing – Getting your ACL sequence numbers back in order
I wish you and yours good luck in the new year!
Send any New Year’s party invitations questions my way. See you next year.
ISCW Down, Three To Go
I took and passed the ISCW test today. I was super-nervous going into it, which is weird for me, but I finally calmed down after the first few questions. Here’s my take. I don’t want to get into any trouble so I’m not going to include very much detail.
The testing center wasn’t very good at all. It’s in an old building on the busiest road in town, and the noise from the street was barely dampened by the 1960s building materials. I can tell you that there are three different pipes in the walls since their vibrations resonated through the room every time somebody flushed or brewed some coffee. There was also a little foot traffic, which can be expected anywhere; they were working through some software problems on another testing station and were very respectful, so it wasn’t too bad. The worst part of the whole ordeal, though, was the Microsoft class I sat through while taking the test. They were across the hall, but it sounded like they were in the room with me. Usually, you hear the instructor yelling at the top of his lungs so the whole class can hear, but I could hear questions being asked and papers being moved. I think I can go pass a test of AD replication, though. I certainly won’t be using that facility for any more tests.
The test itself was fair and pretty close to where it should be. The questions were well rounded and covered the book from front to back. I missed a few due to my ADD kicking in and not letting me finish reading all the answers. At least twice, I saw a more appropriate answer just as I released the mouse from the Next button. :(
There were lots of interactive items- a lot more than I thought there would be or that there should be. I can understand a few do-this-do-that things, but there were at least ten interactive questions, whether they be “put these in order”, “match the definition”, or “tell me what’s going on”. Some of these had multiple parts that I had to click back and forth to get all the answers. One of them in particular could have been more easily presented as an exhibit at the top of a question than a question that really zaps your time. There were a few SDM questions, but I made it through those by clicking around until I found the info.
There were two simulations that were very straightforward and very easy. The sim would present the scenario and tell you what the end result should be along with any details. I found that some details had to be configured in the order the details were presented to finish the lab. Not all of them, mind you, but enough of them to get annoying; I really expected something a little more robust. The contextual help and autocomplete worked, though, so that’s a plus.
I had a big issue with time, and, if that happens to me, it can happen to anyone. The test started with a multi-part interactive question that took me a long time to figure out through the nerves and discord. I would guess that I got a simulation or interactive question in 8 of the first 11 questions, and, at one point, I looked at the clock to see I had 40 minutes and 38 questions left, so I started picking up the pace. Luckily, after question 41, the testing gods showed mercy and ended the suffering.
Overall, I give the test an 8 out of 10. It was very honest and frank with none of the nonesense of trying to trick me. All of the problems I had were either from my lack of knowledge or my being so easily distracted this morning. As Cisco goes, it’s not a bad test at all.
Send any ear plugs questions my way.
Convenience versus Security
I coworker sent over a link today that got me thinking about an old adage that I’ve been sharing for years. The link actually has nothing to do with the philosophy but did trigger a random spewing of words from my brain.
Here’s what I tell everyone. When I deliver these lines, I usually picture myself as Socrates talking to a bunch of Greeks in togas.
There’s a line. On one end of the line is security; on the other end is convenience. You have to figure out where the best place for your users/application/system/etc. to sit on the line to be both secure and convenient enough to function.
I usually follow that up with an extreme example.
What’s the most convenient configuration for a public webserver? One solution would be to have it cabled to an Internet switch in front of a firewall with every network service enabled and all security software disabled in case it interferes with operation. Quite convenient, but not very secure.
What the most secure configuration? The server is powered down, disassembled, all parts shredded to bits, and the bits put into a dozen different boxes that are shipped to the ends of the world. Nobody’s going to get unauthorized access to that, but it’s not very convenient, is it?
In both cases, being too far to one side actually interferes with functionality. How long will it be before the convenient server get owned by a script kiddie and no longer functions? How long before someone wants to access the secure server and finds it doesn’t function at all? We should probably make a compromise, right?
This is nothing new. We’ve all been saying this for years, right?
What’s my point? I don’t think I have one, really. I guess I just wanted to refresh this in everyone’s mind today.
ISCW Notes – Access List Resequencing
I don’t know if this really pertains to the ISCW test per se, but this is something I learned in my class last week. I’m sure I should have learned this years ago, but, alas, I didn’t.
Access lists get messy. You build one, apply it to an interface, and think all is well. Then, ask for more access, so you may have to insert new entries between existing lines. Your security team may ask you to deny access from a host while allowing it from others. The next thing you know, you ACL looks something like this.
Router#sh access-lists Extended IP access list MyACL 5 deny tcp host 192.168.0.38 any eq www 6 deny tcp host 192.168.0.39 any eq www 10 permit tcp 192.168.0.0 0.0.0.255 any eq www 15 deny tcp host 192.168.0.39 any eq 443 17 deny tcp host 192.168.0.85 any eq 443 20 permit tcp 192.168.0.0 0.0.0.255 any eq 443 30 deny ip any any log
That looks horrible, doesn’t it? The sequence numbers are all out of whack, and you may run out of head room if you have to insert more lines. To quickly clean up your ACL, you can run the ip access-list resequence command.
Router(config)#ip access-list resequence MyACL 10 10
This command will take our example ACL and resequence it starting at 10 and incrementing 10 for each line. You can start at any number you want (within reason) and increment the same (within reason again). Using 10 and 10 seems pretty universal, so, once you run that command, your ACL looks like this.
Router#sh access-list Extended IP access list MyACL 10 deny tcp host 192.168.0.38 any eq www 20 deny tcp host 192.168.0.39 any eq www 30 permit tcp 192.168.0.0 0.0.0.255 any eq www 40 deny tcp host 192.168.0.39 any eq 443 50 deny tcp host 192.168.0.85 any eq 443 60 permit tcp 192.168.0.0 0.0.0.255 any eq 443 70 deny ip any any log
Cool, eh? I think I’ll spend the week doing this to all our routers at work.
Send any holiday turkeys questions my way.
ISCW Notes – Role-based Views
I’m at training for the ISCW test this week, and this topic came up yesterday. Since it came up last week at the office, I figure it was a sign from $deity that it was time for a blog entry.
An admin in another business unit was trying to set up command access for some of his techs. He was going through a couple of routers and assigning commands to privilege levels so that his techs could access them. He was having a boat load of problems, though, and couldn’t get it to work
He was trying to allow his guys to run a show ip route, but they also wanted to run show ip route x.x.x.x. He was assigning commands to privilege level 7 then giving his tech’s user accounts the same privilege.
Router(config)#privilege exec all level 7 show ip route Router(config)#username user1 privilege 7 secret his.password
For some reason, this wasn’t working, though. The user could log into the router, but they couldn’t get authorized to run the subcommands as expected. I blamed it on his non-standard 7600 running a non-standard IOS version (sorry, I can’t give any more detail without revealing too much about the company), but I came across a much easier way to do it today in class with role-based views.
A view is a set of commands that can be assigned to users, and, to give a user access to those commands, you make them a member of that view. You’ll see that in a second. You also have a superview, which is a set of views, so a user can be a member of multiple views.
There are some prerequisites to using views. First of all, you have to have the enable secret set. You should already have that on a production router, but, if you’re working in a lab or something, you may have issues. You also need to have AAA enabled. That’s beyond the scope here, but I’m sure you can figure it out.
To configure a view, you must first be in the root view. How do you do that? Just enable to it.
Router#enable view
You’ll enter the enable secret, and nothing special will happen, but now you can use the parser view command to create a new view. This takes you into the view submode which is where you list what commands you want to let users run. You also set a secret (password) so you can call up the view later.
Let’s create a view called “TechView” for my guy. We’ll give members of that view access to the “show ip route” commands to include all the subcommands. We’ll put the user “tech1″ in that view, too.
Router(config)#parser view TechView Router(config-view)#secret view.pass Router(config-view)#command exec include all show ip route Router(config)#username tech1 view TechView secret tech.pass
Every time that “tech1″ logs in, that user will have access to all the show ip route commands. If you have a user who is not in that view but wants access to it, they can run the enable view TechView command and enter the secret. On the console, you’ll see a message saying that user has switched to the view. If the user does a show parser view, they can see what view they’re in.
Router#enable view TechView Password: Router# *Mar 1 00:09:04.047: %PARSER-6-VIEW_SWITCH: successfully set to view 'TechView'. Router#sh parser view Current view is 'TechView'
Send any test vouchers questions my way.
I’ve Been Hacked
It looks like one of those Russian b*%*#rds got me some time last week. I don’t know how long the site was down for sure, but I would guess that he first got access on Thursday, 22 October. Since we’re talking about WordPress here, I just restored back to 15 October to be safe, and it looks like we’re back in business.
As a precaution, I’ve reset some passwords and deleted a whole mess of accounts. I tried to leave the ones that look familiar to me like Blindhog and LBSources, but, if I killed your account, I apologize. I’m afraid you’ll have to sign up again for the sake of security.
Thanks to Drew at GoDaddy for walking me through the restore process! An unsolicited hazzah to those guys for having such great tools so they don’t have to do any more work than they have to do!
Send any stagnant accounts questions my way.
Using SPF Records To Build Objects
My biggest complain about modern firewalls is their lack of the ability to create rules based on URLs or HTTP streams; you have to open access between IP addresses. Yes, I know there are other means to do that, but I want my ASA/PIX/FWSM to do it without making me do so much work.
Anyway, the fact that you have to use IPs brings up some interesting problems. Let’s say you have a server in a DMZ that needs to query Google for some content. Since you’re a hard-ass network guy like I am, you tell the admin that they have provide the data flow they want to use — source IP, destination IP, protocol, port. They come back and tell you that they need their server to connect via HTTP to 74.125.45.100. You put in the rules as given, but the IP has suddenly changed on you.
Google (and lots of other big sites) uses some tricks to keep the load down on their servers and to help with availability, and one such trick is to use round robin DNS, which rotates the A record so everyone doesn’t slam the same boxes. You can query google.com once and get an address, but, when you query it again, you may get a different address. That means that when your new rules don’t work, you have to check the logs, see what got denied, open that up, rinse, and repeat. That sucks.
An easier way might be to create an object-group that includes IPs as you discover them. You put in rules based on an object-group, then, when it fails, you just add to the object-group so you don’t have to put in any more rules. The problem is that you’ll spend a lot of time building up a good baseline. If only there were a way to get a list of IP addresses that Google uses. Hmmm. *segue*
Have you ever heard of SPF netblock records? SPF is an email security mechanism that allows an email server to verify that an email message is coming from an authorized email source. In other words, when a mail server receives mail, it can check to see if the sending server is actually allowed to send mail on behalf of the source domain. It supposed to cut down on spam and whatnot, but I don’t follow it closely enough to know if it’s working. The moral of the story is that is involves a list of IP addresses that an organization maintains; Google happens to be a participant in SPF.
If you query for the TXT record _netblocks.google.com, you get back a text record that looks like this.
[jac@holland ~]$ dig +short txt _netblocks.google.com
“v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ip4:173.194.0.0/16 ?all”
This record includes all IP addresses that Google says is authorized to send email from google.com. That’s a lot of IP addresses, isn’t it? It might make sense that this list might also be the definitive list of Google production IPs.
My company has used this TXT record in the past to open access to Google. We had an app that needed to query Google maps, and one of our engineers was tired of nickel and diming it to death, so he found the SPF block and put them all in. Works like a champ.
There are always dangers when you rely on information from somebody else, though, right? Google’s usually pretty good about stuff like this, but what if you did the same for another company who only half-heartedly kept their records up-to-date? You may only have half of their IPs in your object-gropu. You might even wind up opening access to or from a cable modem system or from another company who bought the IP addresses.
I’ll also note that there aren’t that many domains using this technique, so finding SPF netblock records may be a challenge. It’s worth the time to do a simply query, though; it might save you some time.
Send any carved pumpkins questions my way.
Object Groups in the ASA/FWSM/PIX
I can’t believe I haven’t talked about object-groups yet. I had a whole other blog entry written up, and, when I went to link things over, I realized I couldn’t find an intro to it. Here it goes.
Welcome to the modern world. A world of wonder. A world of quickly-advancing technology. A world where clusters of machines sit behind load balancers for scalability and availability. A world where those clusters need access to other clusters. A world where your firewall rulebase gets so big that it’s unreadable without some help.
Enough with the drama already. I would say I hate the cheesy stuff, but I think my whole blog is nothing but cheesy stuff, right? To the point. Enterprise firewall configurations can get quite large with ACLs applied in different directions to different interfaces. Our ACL entries number in the 6000 range, but the firewall we’re running says we’re only at 5% utilization in the ACLE memory space. That means that our not-top-of-the-line firewall is designed to handle 120k lines of ACLs. That can be quite a handful to configure by hand. There may be an easier-to-maintain solution, though.
Let’s say you have a cluster of servers behind your CSM that all need to access a database. Since there’s a nice ASA, FWSM, or PIX between the servers and database (as there should be), you have to open up access for this connection. Let’s say that you have four servers with the IPs of 192.168.100.101-104 that need access to 10.10.10.1 on the mySQL port (TCP/3306).
access-list LIST1 permit tcp host 192.168.100.101 host 10.10.10.1 eq 3306 access-list LIST1 permit tcp host 192.168.100.102 host 10.10.10.1 eq 3306 access-list LIST1 permit tcp host 192.168.100.103 host 10.10.10.1 eq 3306 access-list LIST1 permit tcp host 192.168.100.104 host 10.10.10.1 eq 3306
Where are your remarks? Why don’t you document something for once in your life?
Anyway, that’s easy, right. Four configuation lines isn’t so bad, but some of the server admins come to you one day and tell you that the company actually marketed the new web app and that tey are adding 37 more servers to the cluster. Now the 37 new servers need the same rules, right? The server dudes also tell you that, since the app has grown so much, the DBAs have set up a split-read-write scenario where the current database handles the reads and a new database handles the writes. That’s 78 new rules (37 to the old and 41 for the new). That’s a lot of rules.
Object-groups to the rescue. An object-group is a logical group of objects (duh!) that you can use to create ACLEs. You can create a group of hosts, a group of network, or a group of ports. For our example, let’s create an object-group that includes all the hosts in the new huge cluster.
object-group network CLUSTER1 description The Huge Cluster (that's what she said) network-object host 192.168.100.101 network-object host 192.168.100.102 ... network-object host 192.168.100.141
What do we do with it, though? You treat it (almost) just like it was a host in an ACL. Remember we wanted to open access to the old database on TCP/3306, right?
access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 eq 3306
If you do a show access-list LIST1 now, you’ll see that a new rules has been added for each object in the object-group. It should look something like this.
access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 eq 3306 access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 3306 (hitcnt=0) access-list LIST extended permit tcp host 192.168.100.102 host 10.10.10.1 eq 3306 (hitcnt=0) ... access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 3306 (hitcnt=0)
Notice that the firewall created 41 rules for you out of your one configured line, but now the rules are indented. The indention means that the rules is generated automagically instead of by hand. Since you can only take out rules that you put in by hand, so you can’t take out the line allowing 192.168.100.123 access; it’s an all-or-nothing scenario. Be aware of that.
You can use object-group for ports, too. Let’s add to our example and say that the cluster will need to access the memcached instance on the database server as well. Those processes run on TCP ports 15000 – 15100.
First we build an object-group for the ports we need.
object-group service DBPORTS description mySQL and memcached ports service-object tcp eq 3306 service-object tcp range 15000 15100
Now let’s apply it to the ACL.
access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 object-group DBPORT
What does the ACL look like now? Well, it’s a Duesenberg.
access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 object-group DBPORTS access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 3306 (hitcnt=0) access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 15000 (hitcnt=0) ... access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 15099 (hitcnt=0) access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 15100 (hitcnt=0) ... access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 3306 (hitcnt=0) access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 15000 (hitcnt=0) ... access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 15099 (hitcnt=0) access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 15100 (hitcnt=0)
That’s a lot of ACL entries for one configuration line, isn’t it? Let’s see. 102 ports times 41 servers is 4182 lines in the ACL. You can see how might be to your advantage to use object-groups at times.
Send any candy corn questions my way.
ASA and Proxy ARP
Wow. A new entry. Everyone sit down before you pass out.
I’ve got a real-world example for you today. We have an ASA 5540 installed at a business unit with interfaces in multiple networks, including one containing the production servers and another containing the accounting servers. The production network sits on a 7600 that’s not ours, so, to avoid IP conflicts, we are statically NATting connections into that network. The 7600 has with many, many VLANs, and, since the firewall production servers are on different VLANs, there’s an interface VLAN between us. Sounds pretty straightforward, but it just wasn’t working when we try to connect between the interfaces.
When we tried to connect from the accounting servers to the production gear, the firewall saw the SYN, built the outbound connection, sent the packet on, and waited. Nothing back. SYN timeout. The vendor on the production side checked the routing. Fine. Checked the ACLs. None installed. When the (other) vendor ran TCPDump on the production servers, they saw the SYN landing and the SYN-ACK leaving, but it never got to the ASA. We even looked at the inline IDS and still didn’t see the SYS-ACK hitting the firewall. It was simply not getting passed on.
I got tired of walking people through stuff over the phone, so I drove up there to see what I could find. When I checked the ARP table on the 7600, I noticed that the statically NATted IP we were serving was conveniently incomplete. For those who don’t know, that means that the 7600 was ARPing for the address, but nothing was answering for it. Obviously, our ASA should be answering, right? To make the situation a little more dire, I did a debug arp (or something close) on the firewall and generated an ARP request; the firewall saw the request but just ignored it. Ugh!
If you couldn’t tell by the title, it turns out that the solution was to enable proxy ARP. It’s off by default for good reason, but here’s how to enable it.
no sysopt noproxyarp PRODUCTIONINTERFACE
Enabling proxy ARP, however, could be a security issue. Any time you use the word “proxy”, there is a potential to spoof addresses, and, in this case, an attacker could (potentially) use the firewall to discover hosts that are on the other side of it. That wouldn’t be good.
A more-secure solution is to use static ARP entries. In our case, we added a static ARP entry on the 7600 that points our NATted IP to the MAC address of the firewall. Now, when you ping the IP, the 7600 doesn’t ARP; it already has the MAC in the ARP table, so it just sends the packet on. Since we only have one static translation in this case, it’s no big deal, but, if we had a whole class-C of addresses to NAT, there would be a management problem.
A part of me wants to do the simple thing and enable proxy ARP, but the vast majority of article, blogs, forums, lists, etc., that I’ve ready say to turn it off for security and efficiency purposes. The more I think about it, though, the more Iwonder why proxy ARP needs to be enabled to make staic NATs work. I looked back at an old PIX running 6.x, and proxy ARP is on by default. The same holds true for an FWSM running 2.x. I’m going to have to ask Cisco what’s up with that.
Send any misconfigured subnet masks questions my way.
Getting Temperature Data from a 6500 via SNMP
I apologize to my adoring fans (both of you) for the lack of posting. I’m in the middle of moving, buying a new house, selling my current house, getting a mortgage, etc. I’ve up until 11:30 nearly every night filling out forms and going through red tape. Don’t get me started on getting money from a 401k! Anyway…
I got in this morning, and a coworker was telling me that the data center’s HVAC was crippled due to an oil leak, and it was 90F in there. D’oh! It wasn’t quite that high, but it was warm. Luckily, all of our network gear is on the end of the rows with AC, so we’re safe, but it got me thinking about monitoring temperature of our 6500s via SNMP. I’ve done it via Cacti, but I never really looked how to do it manually.
I read this article on Cisco’s site which made it clear as mud. Basically, you have to query the entPhysicalIndex OID branch to get an integer that represents the physical sensor, then you query the entSensorValue to get the temperature. I made the lazy choice to use grep on entPhysicalDesc OID to find the module I wanted and use that to grep the entSensorValue for the temp.
Let’s say I’m looking for the outlet temperature (the temperature of the air on the way out) on module 3. First, let’s find out what th entPhysicalIndex is by walking the entPhysicalDesc OID and grepping out “module 3″.
[myserver]$ snmpwalk -c <COMM> -v 2c <HOST> 1.3.6.1.2.1.47.1.1.1.1.2 | grep "module 3" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3001 = STRING: "module 3 power-output-fail Sensor" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3002 = STRING: "module 3 insufficient cooling Sensor" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3003 = STRING: "module 3 outlet temperature Sensor" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3004 = STRING: "module 3 inlet temperature Sensor" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3005 = STRING: "module 3 device-1 temperature Sensor" SNMPv2-SMI::mib-2.47.1.1.1.1.2.3006 = STRING: "module 3 device-2 temperature Sensor"
Since we’re looking for the outlet temperature, we can see the index in question is 3003. Now, we use that to query the entSensorValue.
[myserver]$ snmpwalk -c <COMM> -v 2c <HOST> 1.3.6.1.4.1.9.9.91.1.1.1.1.4.3003 SNMPv2-SMI::enterprises.9.9.91.1.1.1.1.4.3003 = INTEGER: 19
Ah…a cool 19C.
This is not a very graceful way to do it, but it gets the job done in a pinch. I haven’t tested it, but I would imagine this technique would work on 7600s as well. I know it doesn’t work on 2950s and the like.
Send any house keys questions my way.