Nexus 1000v and Cisco Support
After writing my previous posts about my love/hate relationship with Nexus 1000v, I received a phone call from the Cisco Nexus 1000v Product Manager. I can only guess that he tracked me down because I posted a Bug ID in there. Regardless, he was very interested in making sure that my issues were resolved, and he pulled some resources together to help me out.
I needed the help because my Secondary VSM had started into a reboot loop. Even deploying a fresh VSM would do the same thing after the Config Sync happened. While pulling some debugs off of the busted VSM, somehow 6 of our VEMs (Hosts) unregistered with the Primary VSM. My TAC Engineer was out of the office, but an Engineer from the 1000v Escalation Team got on the phone with me, and started digging around. What he found was this: a 3750 switch, home to several Development ESX Hosts, using a port-channel connected via vPC to our Nexus 7000 switches. The only traffic allowed across this port-channel was the Control/Packet/Management VLANs. Something on the 3750 was causing a broadcast loop in the network. After the Engineer pointed out the issue, shutting down the ports connected to the 3750 instantly resolved the connectivity issues - looks like I need to revisit my spanning-tree and port-channel configurations to see where I went astray.
During the course of all of this, we were trying to add the Dell Chassis that I wrote about previously. The stupid VEMs (hosts/blades) just wouldn't register with the Nexus 1000v VSMs. Why? After spending some more time on the phone with my TAC Engineer - who I'm sure was quite sick of hearing from me - more wonderful news was discovered: The firmware version on the Emulex OneConnect Mezzanine cards (2.702.200.17) apparently has an issue processing tagged multicast/broadcast traffic. Excellent news! Thanks, Dell, for shipping me cards in brand new servers that have firmware old enough to have major issues that are at least somewhat well known. I lucked out by having a TAC Engineer who has supported Nexus 5000 switches as well, and had ran into this exact issue before. I don't even want to think about how long it may have taken to resolve this problem had she not been familiar with it. And sure enough, after upgrading the firmware (which was another fun process - I'll write about it later), the Dell VEMs were happily registered to the VSM.
To be fair, I should have caught the issue with the Emulex cards. We bought PCI Express versions with the same chipset about 6 months before. Inside each of the boxes were big warning messages stating not use the cards without first updating the firmware. Apparently it's too much to ask Emulex to do that before sending them out of the factory. It's probably because there is one customer who is still using that original firmware version; maybe it's not possible to downgrade the cards that far once they have been upgraded. Regardless, it's substantially annoying.
Oh, yeah, then there was that other Nexus 5000 issue - I'll get to that in another post.
I needed the help because my Secondary VSM had started into a reboot loop. Even deploying a fresh VSM would do the same thing after the Config Sync happened. While pulling some debugs off of the busted VSM, somehow 6 of our VEMs (Hosts) unregistered with the Primary VSM. My TAC Engineer was out of the office, but an Engineer from the 1000v Escalation Team got on the phone with me, and started digging around. What he found was this: a 3750 switch, home to several Development ESX Hosts, using a port-channel connected via vPC to our Nexus 7000 switches. The only traffic allowed across this port-channel was the Control/Packet/Management VLANs. Something on the 3750 was causing a broadcast loop in the network. After the Engineer pointed out the issue, shutting down the ports connected to the 3750 instantly resolved the connectivity issues - looks like I need to revisit my spanning-tree and port-channel configurations to see where I went astray.
During the course of all of this, we were trying to add the Dell Chassis that I wrote about previously. The stupid VEMs (hosts/blades) just wouldn't register with the Nexus 1000v VSMs. Why? After spending some more time on the phone with my TAC Engineer - who I'm sure was quite sick of hearing from me - more wonderful news was discovered: The firmware version on the Emulex OneConnect Mezzanine cards (2.702.200.17) apparently has an issue processing tagged multicast/broadcast traffic. Excellent news! Thanks, Dell, for shipping me cards in brand new servers that have firmware old enough to have major issues that are at least somewhat well known. I lucked out by having a TAC Engineer who has supported Nexus 5000 switches as well, and had ran into this exact issue before. I don't even want to think about how long it may have taken to resolve this problem had she not been familiar with it. And sure enough, after upgrading the firmware (which was another fun process - I'll write about it later), the Dell VEMs were happily registered to the VSM.
To be fair, I should have caught the issue with the Emulex cards. We bought PCI Express versions with the same chipset about 6 months before. Inside each of the boxes were big warning messages stating not use the cards without first updating the firmware. Apparently it's too much to ask Emulex to do that before sending them out of the factory. It's probably because there is one customer who is still using that original firmware version; maybe it's not possible to downgrade the cards that far once they have been upgraded. Regardless, it's substantially annoying.
Oh, yeah, then there was that other Nexus 5000 issue - I'll get to that in another post.
Comments