Server NIC Aggregation to a Cisco Switch
Have you even noticed that your new servers all have 2 NICs on the board? At least all of them that I’ve seen in the last 3 years have. A lot of server admin actually use them in a NIC teaming scenario where both NICs are used as one logical device — much the same as Etherchannel on a switch. This provides some fault tolerance and availability in case of failure, which is good idea in most cases.
There are a few different ways to configure teaming on the box (usually called bonding in Linux), and each has its own advantages and disadvantages. The network dude(tte) may have to do some things on the switch side for some of them to work, though. If you’re want to run in link aggregation mode (mode 4), for example, the switch ports need to be in the same channel group to work appropriately.
Let’s look at mode 4 a little closer to see what we need to do. The scenario is that you have eth0 plugged into F0/15 of a 2950 and eth1 is in F0/16. You’ve seen the configuration for channelling between switches before, so you know the basics. Put the ports in the same channel-group and configure the proper Port-channel interface to do the work. In this case, we’re just configuring the ports to house a host instead of being trunks.
int F0/15 channel-group 1 int F0/16 channel-group 1 int Port-channel 1 speed 100 duplex full switchport switchport mode access
I detect at least one problem with our setup, though. Both NICs are plugged into the same switch; what happens when the switch goes down? The server goes away. Logic should tell you, then, to put the NICs in different switches to fix that, but you can’t do Ethernchannel on two different switches. The ports have to be in the same device for the aggregation to work. What’s the fix?
You can look at getting a nice chassis switch and putting each NIC in different modules. Modern IOS versions allow etherchanneling across modules, so, if one module fails, you still have that other. That would do it, but I’m sure you don’t have the money for a 4500 in the budget, right?
Another solution is to use a couple 3760s which, when connected using the StackWise cable, are one logical device. That gives you two separate switches that you can configure with the same channel group. An upgrade to this solution is to use a pair of 6500s with VSS 1440 modules in them so that you have a stack of 6500s! I’m sure that’s not expensive at all, though.
Send any white shoes questions my way.
https://masto.ai/@aconaway
- Generating Network Diagrams from Netbox with Pynetbox - August 23, 2023
- Out-of-band Management – Useful Beyond Catastrophe - July 13, 2023
- Overlay Management - July 12, 2023
So is this required if the server’s NICs are in active-passive mode where only one NIC is active at a time?
Good question. Mode 1, which is active-backup, has a primary NIC running as normal, and, in case of link failure, the backup NIC takes over. From the switch’s perspective, it just looks like the server was recabled to another port. No additional switch configuration is required; just make sure both ports are in the same VLAN.
thanks, Aaron..
What if you have a need to trunk multiple VLANs to an ESX server in the following environment? We recently had a supervisor die in one switch and didn’t have NIC’s connected to the other switch.
– Have 2 4500’s etherchanneled together.
– Have 3 line cards in each switch
– Switches are running GLBP
– 4 NIC’s in each ESX servers.
How do I use multiple NICs to truck multiple VLAN to servers from each switch? Will spanning tree play a role in this?
Thanks for the question, Kolars.
I’m not sure what the best practice for 4 NICs in an ESX server is, so let’s simplify it for me. Let’s say you have 2 NICs in the ESX server doing VLAN tagging. You would cable these to two different modules of the 4500 (or the same if you’re adventurous) and configure those ports as an EtherChannel like in the example above. The difference is that you would configure the Port-Channel as a trunk the same way you do between your 4500s.
Since the 4500s don’t support StackWise, you can’t put all the NICs across both switches and have them in the same channel-group. Can you bond two pairs of NICs into two separate logical interfaces to use? If so, you could have a pair into one 4500 on a channel-group and the other pair on the other 4500 in another channel-group.
Good, I’m glad you suggested the 2nd option. My thought was to attempt the “2 pairs – one to each switch” solution but since the switches have an etherchannel between them I was curious how that would work since that would basically create a loop. My instinct would tell me that spanning tree would block one logical pair to break the loop but I’m not entirely sure if it works the same with all the bonding and channeling that would be happening. Any further thoughts?
The bonded interfaces on the server shouldn’t switch traffic, so there’s no danger of creating a switching loop. I wouldn’t expect any problems having multiple bonded pairs from the server to two different switches that are EtherChannelled together.
I hope that works for you. Let me know the result.