Aaron's Worthless Words

We talked about running multiple data centers on a stick back in August, which is where you have multiple logical pairs of client and server VLANs on a single CSM for different tiers or functions.  The big point of the article was that you had to do some fancy forwarding to get a server-initiated connection from one server VLAN to appear out the appropriate client VLAN.  Well, we ran into an interesting issue with the given solution.

Let’s set up a scenario.  Check the diagram for an overview.  We have many pairs of client and server VLANs each with a firewall interface as the gateway into the DCOAS.  Let’s just focus on just one, though — client VLAN 101 and server VLAN 102.  In VLAN 101 is ServerA (not pictured) with an IP of 1.1.101.45; in VLAN 102 is our web farm that needs to connect to ServerA to drop off some data.  We add a static route on ServerA pointing traffic for 1.1.102.0/24 back through the CSM.

When you try to connect from the web farm, though, it just times out.  Why is that?

Remember that weird forwarding vserver that we had to use to get traffic to come out of the right client VLAN?  Well, that’s stabbing you in the eye right now.  When the web server initiates a connection, it sends traffic to the server VLAN IP of the CSM.  The forwarding vserver grabs the new connection and load balances it to its only RIP, which is the IP of the firewall.  What happens when any good firewall accepts traffic destined on an interface destined for a host out of the same interface?  It drops the packet, and, eventually, the server times out.

What’s the fix, then?  There are a few that come to mind.  The first may be to just move ServerA to another network segment.  Another may be to change the process around a bit by having ServerA pull the data instead of it being pushed since client-initiated connections will work like a champ.

A really outrageous one would be to set up another forwarding vserver that has only ServerA as it’s serverfarm.  You would then add a static route in the web servers pointing to ServerA through that VIP, which would foward it over.

On the CSM, you’d do something like this.

serverfarm SERVERA-SF
 no nat server
 no nat client
 real 1.1.101.45  <--- ServerA
  inservice

vserver SERVERA-VS
 virtual 1.1.102.5 any
 vlan 102
 serverfarm SERVERA-SF
 inservice

On the server, you would add a static route to ServerA through 1.1.102.5. If you’re using some brand of Linux, you’d do this.

route add 1.1.101.45 gw 1.1.102.5

Don’t forget the static route on ServerA.

Send any Peeps questions my way.

We discussed SPANs earlier, but let’s talk about RSPANs for a bit.

Can anyone guess what the “R” means?  You guessed it — “Remote”.  An RSPAN is a way to get traffic from a SPAN source on one switch to a SPAN destination on another switch that’s connected via a trunk.

The basic premise is that a special VLAN is created on all the switches and allowed to traverse the trunks.  You then set up a SPAN session that copies your traffic to this special VLAN.  This VLAN then gets the traffic to the other switches through some voodoo magic to be used as source for a SPAN on another switch.

Let’s work through the steps.  In our example, we want to copy traffic from G2/18 on SwitchA to G3/38 on SwitchB.

First, on both switches, we need to create the new RSPAN VLAN.  We’ll assume you’ve already got it set up to allow this VLAN over your trunks.

vlan 2000
 name RSPAN
 remote-span

Notice the nice keyword remote-span.  This designates the VLAN to be used in an RSPAN.  Easy so far.

Now, let’s create the session to copy traffic to the RSPAN.  The source port is G2/18, and the destination is the RSPAN VLAN.

switchA(config)# monitor session 1 source interface Gi 2/18
switchA(config)# monitor session 1 destination remote vlan 2000

Now the traffic is being copied to the RSPAN, so let’s copy that traffic from the RSPAN to the sniffer.  In this case, the source is the RSPAN, and the destination is the sniffer’s port.  Let’s use session 8 to avoid confusion.

switchB(config)# monitor session 8 source remote vlan 2000
switchB(config)# monitor session 8 destination interface Gi 3/38

There are always things to look out for, aren’t there?  The first that comes to mind is the fact that you’re copying traffic from a port onto one or more trunks.  If the port is sending enough traffic and your trunk is close to capacity, you may wind up crushing the trunk link.  That would suck.

If you have a fully-meshed switch environment, you’ll see the additional traffic across all your trunks if you’re set up that way.  If you have four trunks that transport all VLANs, you may have four copies of the data coming out of the switch.  Let’s say the box being monitored is compromised and sending out 600Mbps of data.  That means that every switch will have to deal with that much traffic.  This sounds to me like a CPU/memory issue waiting to happen.

Don’t expect RSPANs to work on your 2950 like this.  On the lower-end switches, you have to use a reflector port to copy the traffic to the RSPAN.  I don’t get into that here, but Google is your friend.

Send any Cadbury Creme Eggs questions my way.

I can’t believe I haven’t blogged on this yet.  SPANs are one of my favorite things in the world.

The switched port analyzer (SPAN) is a mechanism on Cisco switches that allows you to take traffic on one port and copy it to another.  It’s generally used to get traffic to a sniffer or IDS for analysis, but it’s a great tool to use to sample traffic from a host for troubleshooting.

Let’s use a real-world example.  You’ve told your roommate to quit illegally downloading songs off the Internet, but you suspect he’s still doing it.  Instead of sneaking into his room at night and checking his machine, you can use a SPAN to copy his traffic to another switch interface where you can use tcpdump to record what’s happening.

Let’s say you have a 2950, and the roomie is on F0/1.  You have a Linux box plugged into F0/24 ready to capture the traffic.  Here’s what you do.

monitor session 1 source interface F0/1 both
monitor session 1 destination interface F0/24

This will create a new monitor session (that is, a SPAN session) that copies traffic from port F0/1 in both directions to port F0/24.  Now, when you run tcpdump on your Linux box, you see all the traffic coming in and going out of your roommate’s port.

That’s pretty easy, right?  You can have multiple sources ports by just adding more source lines or using ranges of ports.  You can also just copy received or transmitted traffic from a source.  Check out the contextual help for a little more info.

To see what’s going on, you can do a show monitor or a show monitor session 1 (depending on the IOS version).  You’ll see something like this.

switch#sh monitor
Session 1
---------
Type              : Local Session
Source Ports      :
Both          : Fa0/1
Destination Ports : Fa0/24
Encapsulation : Native
Ingress : Disabled

If you take a look at the destination port when the SPAN is running, you’ll see it’s in a state of up/down (monitoring).  I think you can figure out that this means we’re monitoring some traffic to this port.  Here’s what you’ll see if you look at the port.

switch#sh int f0/24
FastEthernet0/24 is up, line protocol is down (monitoring)
...

There are two big things to keep in mind when doing SPANs.  The first is that monitoring a port can drive CPU utilization way up (depending on the platform and traffic volume), so you may run into problems if you have a bunch of SPANs going at the same time.  Related to this is the fact that, if your switch has to decide between switching and copying traffic, it will stop copying until there’s enough CPU headroom to do that safely, and you’ll lose packets in the meantime.  It’s a switch — not a copier.

The second thing to keep in mind involves those little voices in your head called ethics.  What if you see a VOIP phone call from your boss to the HR department?  How about if you find someone in upper management copying a spreadsheet of people to be fired tomorrow?  How about if you find an engineer’s telnet password to a key system?  These are things that you probably shouldn’t see, so be careful when looking at the packets.  I would suggest you tell someone in your security when you’re going to do a packet capture to make sure someone knows you’re not up to no good.

Send shamrocks questions my way.