Archive for the ‘pix’ tag
A buddy asked for some help on configuring a pair of ASAs in active/passive mode, and, by pure coincidence, my newest project is to set up the same. I've done it many time, but it's one of those things that you don't really do every day (unless you're a VAR or something). These things always get covered in rust very quickly in my head, but, once I get one or two details back to the surface, it all comes flooding back. I better take the time to jot down the details. Read the rest of this entry »
My biggest complain about modern firewalls is their lack of the ability to create rules based on URLs or HTTP streams; you have to open access between IP addresses. Yes, I know there are other means to do that, but I want my ASA/PIX/FWSM to do it without making me do so much work.
Anyway, the fact that you have to use IPs brings up some interesting problems. Let’s say you have a server in a DMZ that needs to query Google for some content. Since you’re a hard-ass network guy like I am, you tell the admin that they have provide the data flow they want to use — source IP, destination IP, protocol, port. They come back and tell you that they need their server to connect via HTTP to 184.108.40.206. You put in the rules as given, but the IP has suddenly changed on you.
Google (and lots of other big sites) uses some tricks to keep the load down on their servers and to help with availability, and one such trick is to use round robin DNS, which rotates the A record so everyone doesn’t slam the same boxes. You can query google.com once and get an address, but, when you query it again, you may get a different address. That means that when your new rules don’t work, you have to check the logs, see what got denied, open that up, rinse, and repeat. That sucks.
An easier way might be to create an object-group that includes IPs as you discover them. You put in rules based on an object-group, then, when it fails, you just add to the object-group so you don’t have to put in any more rules. The problem is that you’ll spend a lot of time building up a good baseline. If only there were a way to get a list of IP addresses that Google uses. Hmmm. *segue*
Have you ever heard of SPF netblock records? SPF is an email security mechanism that allows an email server to verify that an email message is coming from an authorized email source. In other words, when a mail server receives mail, it can check to see if the sending server is actually allowed to send mail on behalf of the source domain. It supposed to cut down on spam and whatnot, but I don’t follow it closely enough to know if it’s working. The moral of the story is that is involves a list of IP addresses that an organization maintains; Google happens to be a participant in SPF.
If you query for the TXT record _netblocks.google.com, you get back a text record that looks like this.
[jac@holland ~]$ dig +short txt _netblocks.google.com
“v=spf1 ip4:220.127.116.11/19 ip4:18.104.22.168/19 ip4:22.214.171.124/20 ip4:126.96.36.199/18 ip4:188.8.131.52/17 ip4:184.108.40.206/20 ip4:220.127.116.11/16 ip4:18.104.22.168/20 ip4:22.214.171.124/20 ip4:126.96.36.199/16 ?all”
This record includes all IP addresses that Google says is authorized to send email from google.com. That’s a lot of IP addresses, isn’t it? It might make sense that this list might also be the definitive list of Google production IPs.
My company has used this TXT record in the past to open access to Google. We had an app that needed to query Google maps, and one of our engineers was tired of nickel and diming it to death, so he found the SPF block and put them all in. Works like a champ.
There are always dangers when you rely on information from somebody else, though, right? Google’s usually pretty good about stuff like this, but what if you did the same for another company who only half-heartedly kept their records up-to-date? You may only have half of their IPs in your object-gropu. You might even wind up opening access to or from a cable modem system or from another company who bought the IP addresses.
I’ll also note that there aren’t that many domains using this technique, so finding SPF netblock records may be a challenge. It’s worth the time to do a simply query, though; it might save you some time.
Send any carved pumpkins questions my way.
I can’t believe I haven’t talked about object-groups yet. I had a whole other blog entry written up, and, when I went to link things over, I realized I couldn’t find an intro to it. Here it goes.
Welcome to the modern world. A world of wonder. A world of quickly-advancing technology. A world where clusters of machines sit behind load balancers for scalability and availability. A world where those clusters need access to other clusters. A world where your firewall rulebase gets so big that it’s unreadable without some help.
Enough with the drama already. I would say I hate the cheesy stuff, but I think my whole blog is nothing but cheesy stuff, right? To the point. Enterprise firewall configurations can get quite large with ACLs applied in different directions to different interfaces. Our ACL entries number in the 6000 range, but the firewall we’re running says we’re only at 5% utilization in the ACLE memory space. That means that our not-top-of-the-line firewall is designed to handle 120k lines of ACLs. That can be quite a handful to configure by hand. There may be an easier-to-maintain solution, though.
Let’s say you have a cluster of servers behind your CSM that all need to access a database. Since there’s a nice ASA, FWSM, or PIX between the servers and database (as there should be), you have to open up access for this connection. Let’s say that you have four servers with the IPs of 192.168.100.101-104 that need access to 10.10.10.1 on the mySQL port (TCP/3306).
access-list LIST1 permit tcp host 192.168.100.101 host 10.10.10.1 eq 3306 access-list LIST1 permit tcp host 192.168.100.102 host 10.10.10.1 eq 3306 access-list LIST1 permit tcp host 192.168.100.103 host 10.10.10.1 eq 3306 access-list LIST1 permit tcp host 192.168.100.104 host 10.10.10.1 eq 3306
Where are your remarks? Why don’t you document something for once in your life?
Anyway, that’s easy, right. Four configuation lines isn’t so bad, but some of the server admins come to you one day and tell you that the company actually marketed the new web app and that tey are adding 37 more servers to the cluster. Now the 37 new servers need the same rules, right? The server dudes also tell you that, since the app has grown so much, the DBAs have set up a split-read-write scenario where the current database handles the reads and a new database handles the writes. That’s 78 new rules (37 to the old and 41 for the new). That’s a lot of rules.
Object-groups to the rescue. An object-group is a logical group of objects (duh!) that you can use to create ACLEs. You can create a group of hosts, a group of network, or a group of ports. For our example, let’s create an object-group that includes all the hosts in the new huge cluster.
object-group network CLUSTER1 description The Huge Cluster (that's what she said) network-object host 192.168.100.101 network-object host 192.168.100.102 ... network-object host 192.168.100.141
What do we do with it, though? You treat it (almost) just like it was a host in an ACL. Remember we wanted to open access to the old database on TCP/3306, right?
access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 eq 3306
If you do a show access-list LIST1 now, you’ll see that a new rules has been added for each object in the object-group. It should look something like this.
access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 eq 3306 access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 3306 (hitcnt=0) access-list LIST extended permit tcp host 192.168.100.102 host 10.10.10.1 eq 3306 (hitcnt=0) ... access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 3306 (hitcnt=0)
Notice that the firewall created 41 rules for you out of your one configured line, but now the rules are indented. The indention means that the rules is generated automagically instead of by hand. Since you can only take out rules that you put in by hand, so you can’t take out the line allowing 192.168.100.123 access; it’s an all-or-nothing scenario. Be aware of that.
You can use object-group for ports, too. Let’s add to our example and say that the cluster will need to access the memcached instance on the database server as well. Those processes run on TCP ports 15000 – 15100.
First we build an object-group for the ports we need.
object-group service DBPORTS description mySQL and memcached ports service-object tcp eq 3306 service-object tcp range 15000 15100
Now let’s apply it to the ACL.
access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 object-group DBPORT
What does the ACL look like now? Well, it’s a Duesenberg.
access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 object-group DBPORTS access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 3306 (hitcnt=0) access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 15000 (hitcnt=0) ... access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 15099 (hitcnt=0) access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 15100 (hitcnt=0) ... access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 3306 (hitcnt=0) access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 15000 (hitcnt=0) ... access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 15099 (hitcnt=0) access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 15100 (hitcnt=0)
That’s a lot of ACL entries for one configuration line, isn’t it? Let’s see. 102 ports times 41 servers is 4182 lines in the ACL. You can see how might be to your advantage to use object-groups at times.
Send any candy corn questions my way.
Wow. A new entry. Everyone sit down before you pass out.
I’ve got a real-world example for you today. We have an ASA 5540 installed at a business unit with interfaces in multiple networks, including one containing the production servers and another containing the accounting servers. The production network sits on a 7600 that’s not ours, so, to avoid IP conflicts, we are statically NATting connections into that network. The 7600 has with many, many VLANs, and, since the firewall production servers are on different VLANs, there’s an interface VLAN between us. Sounds pretty straightforward, but it just wasn’t working when we try to connect between the interfaces.
When we tried to connect from the accounting servers to the production gear, the firewall saw the SYN, built the outbound connection, sent the packet on, and waited. Nothing back. SYN timeout. The vendor on the production side checked the routing. Fine. Checked the ACLs. None installed. When the (other) vendor ran TCPDump on the production servers, they saw the SYN landing and the SYN-ACK leaving, but it never got to the ASA. We even looked at the inline IDS and still didn’t see the SYS-ACK hitting the firewall. It was simply not getting passed on.
I got tired of walking people through stuff over the phone, so I drove up there to see what I could find. When I checked the ARP table on the 7600, I noticed that the statically NATted IP we were serving was conveniently incomplete. For those who don’t know, that means that the 7600 was ARPing for the address, but nothing was answering for it. Obviously, our ASA should be answering, right? To make the situation a little more dire, I did a debug arp (or something close) on the firewall and generated an ARP request; the firewall saw the request but just ignored it. Ugh!
If you couldn’t tell by the title, it turns out that the solution was to enable proxy ARP. It’s off by default for good reason, but here’s how to enable it.
no sysopt noproxyarp PRODUCTIONINTERFACE
Enabling proxy ARP, however, could be a security issue. Any time you use the word “proxy”, there is a potential to spoof addresses, and, in this case, an attacker could (potentially) use the firewall to discover hosts that are on the other side of it. That wouldn’t be good.
A more-secure solution is to use static ARP entries. In our case, we added a static ARP entry on the 7600 that points our NATted IP to the MAC address of the firewall. Now, when you ping the IP, the 7600 doesn’t ARP; it already has the MAC in the ARP table, so it just sends the packet on. Since we only have one static translation in this case, it’s no big deal, but, if we had a whole class-C of addresses to NAT, there would be a management problem.
A part of me wants to do the simple thing and enable proxy ARP, but the vast majority of article, blogs, forums, lists, etc., that I’ve ready say to turn it off for security and efficiency purposes. The more I think about it, though, the more Iwonder why proxy ARP needs to be enabled to make staic NATs work. I looked back at an old PIX running 6.x, and proxy ARP is on by default. The same holds true for an FWSM running 2.x. I’m going to have to ask Cisco what’s up with that.
Send any misconfigured subnet masks questions my way.
Here’s a simple one since I haven’t updated in a while. I have my ASA 5505 at home and want to forward TCP/80 traffic to my public IP to my webserver at 10.10.10.10. There are two steps here — forward the port and open the ACL.
To forward the port, I would use the static directive, but there are two ways to do that. I can either set up a one-to-one NAT or a port redirection. In the one-to-one NAT, you have a outside address that’s mapped directly to an inside address, and any traffic to that IP is passed to the inside host (if it passes ACLS, of course). One of the limitation, though, of using this setup is that you can’t use that IP as your PAT address, and, since I only have one IP, no other inside hosts would have a outside address to which to be NATted. The other method — port redirection — is a much better solution. In this setup, I actually forward a protocol/port on a outside address to a protocol/port on an inside address. Since there are other ports available on that outside address, the address is still available for other hosts to use as a NAT address.
In an enterprise, I would probably use an address out of my pool for the port forwarding, but, since I only have one address at home, I’ve got another decision to make. I can configure the static statement with an IP address or I can use the reserved word interface to indicate the IP that is on an interface. This is a great feature, actually, since my outside IP could potentially change without notice. I’m going to use that feature, too.
static (inside,outside) tcp interface 80 10.10.10.10 80
This is pretty simple, but I’ll explain. The ASA will take any request that comes in on TCP/80 (HTTP) on its outside interface’s IP and forward it to TCP/80 of 10.10.10.10. If my webserver ran on TCP/81 on my box, I could just change the last 80 to 81 to make it work.
The port is redirecting, but I still need to open the ACL. When that’s done, everything should work as expected.
NATting sucks and can be confusing. I’m sure everyone agrees to that, but you have to use it at some times. In a PIX/ASA, it’s easy to configure a simple setup, but can be super-complicated in larger networks. In a simple lab, we have set up an ASA with inside and outside interfaces, with the inside as your internal and outside as the Internet.
The NAT setup here is easy.
nat (inside) 1 0.0.0.0 0.0.0.0
global (outside) 1 interface
This NATs everyone on the inside (0.0.0.0 with a mask of 0.0.0.0, or 0/0) to the IP of the outside interface (overload in the IOS world). The nat command says who gets NATted; the global command says what they get NATted to. Notice the number “1″ in both commands; this is the NAT group and allows you to have some flexibility in your NAT strategy. In essence, if you match a nat line with a “1″ in it, you’ll be matched to a “1″ on the global list.
What if you add a DMZ interface and don’t want to NAT when your inside network talks to it? That, my friend, is a little more complicated. We’ll assume your internal network is 10.0.0.0/24 and your DMZ is 192.168.0.0/24.
access-list NONAT permit ip 10.0.0.0 255.255.255.0 192.168.0.0 255.255.255.0
nat (inside) 0 access-list NONAT
nat (inside) 1 0.0.0.0 0.0.0.0
nat (dmz) 1 0.0.0.0 0.0.0.0
global (outside) 1 interface
That was painful, but what did it do? That’s a very good question.
We have multiple nat lines on the inside, so the firewall starts at the top and works its way down (there are exceptions). The first nat line has a group of 0, which is very special. If you match group 0, you are not NATted at all, and your connection is passed as-is with no changes. In our second example, you match if the ACL matches, so, if you’re going from the inside network to the DMZ, you won’t be NATted. If your connection didn’t match this line (like you’re downloading porn from the Internet), the firewall goes to the next line, which says to NAT everyone to group 1 just as we did in the first example.
Another twist here is the “nat (dmz) 1 0.0.0.0 0.0.0.0″ line. This says that anything from the DMZ is NATted to group 1 just as the inside traffic is.
So, if the inside network connects to the DMZ, it doesn’t get NATted. If the inside goes to the Internet, it gets NATted to the outside IP of the firewall. If the DMZ connects to the Internet, it gets NATted to the outside IP as well, but what if the DMZ connects to the inside? That’s another story. :)
There’s a very-overlooked feature of access-lists — the remark. Yes, this is very basic, but it’s worth mentioning, as it has saved me anguish time and time again.
I use remarks to document each line of an ACL (on IOS, PIX, FWSM, ASA, etc.) so that when I go back later, I actually know what I did. They’re simple to use, and, I promise you, you’ll thank yourself for using it when the CTO asks why access to TCP/80 is open from the Internet to the development server.
Easy to use.
access-list 101 remark This line allows access from the Internet to the development server. See ticket 1234
access-list 101 permit tcp any host 188.8.131.52 eq 80
Now, when you get asked the inevitable question, you can look at the line and know to check ticket 1234 for more information. The remark is just a string, so you can put what you want. I like to put source and destination hostnames, protocol/port, ticket number, and date/time the line was entered for reference like this.
access-list 2001 remark *** I’net — HTTP -> dev.example.com, Ticket 1234, 12Mar2008-0853 ***
It works with the ip access-list command as well.
ip access-list extended INBOUND
remark *** I’net — HTTP -> dev.example.com, Ticket 1234, 12Mar2008-0853 ***
It might be a good idea to use a remark to document what the ACL itself does. For example, on a firewall with 28974 interfaces, you might want to do something like this.
access-list DMZ1_OUT remark This ACL allows traffic out of the DMZ interface
access-list DMZ1_OUT remark *** ….
access-list DMZ1_OUT permit …
I deployed a Cisco ASA at a location and couldn’t get logged in via SSH. I would get prompted, but, no matter what username/password I put in, it would just reject me. After some digging, it turns out that I forgot this command.
aaa authentication ssh console LOCAL
When I put this in, it let me right in as expected. I have no clue what the deal was. I guess I assumed that the ASA would use the local userbase if a AAA service wasn’t configured. I guessed wrong.
I’m sure this will apply to telnet sessions as well. I’d also bet money that equivalent PIX OS versions do that same, so keep an eye out.