Undocumented Sanity

In a previous post, I mentioned problems we were having with one of our Nexus 5000 switches. During all of the Nexus 1000v issues, it was throwing these messages continually: 2011 Mar 29 05:22:13 N5K-2 %FWM-2-STM_LEARNING_RE_ENABLE: Re enabling dynamic learning on all interfaces 2011 Mar 29 05:22:20 N5K-2 %FWM-2-STM_LOOP_DETECT: Loops detected in the network among ports Eth1/10 and Eth1/2 vlan 801 - Disabling dynamic learn notifications for 180 seconds I couldn't tell if it was actually affecting anything, since VLAN 801 was being used as a FCoE VLAN. Looking at MAC addresses bound to VLAN 801 would reveal one MAC address in particular that would move around: N5K-2(config)# sho mac add vlan 801 Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+-----------...

I previously commented about issues we had with one of the pass through I/O modules with our M1000e chassis. After opening a case with support, they had us do some things such as remove the blades, remove the modules, etc, and it started working. Still not a particularly promising sign. After building out our ESX servers and trying to put VMs on them, we had all kinds of unusual issues with trying to run FCoE Active/Active on them. We were getting errors such as: Apr 23 15:02:04 host vmkernel: 0:00:39:45.582 cpu0:4284)WARNING: LinNet: netdev_watchdog: NETDEV WATCHDOG: vmnic2: transmit timed out Apr 23 15:02:04 host vmkernel: 0:00:39:45.854 cpu3:4260)NMP: nmp_DeviceUpdatePathStates: Activated path "NULL" for NMP device "naa.60060060060060060060060060060060". Apr 23 15:02:04 host vmkernel: 0:00:39:45.854 cpu4:4258)NMP: nmp_DeviceUpdatePathStates: Activated path "NULL" for NMP device "naa.60060060060060060060060060060060". Apr 23 15:02:04 host vmker...

After writing my previous posts about my love/hate relationship with Nexus 1000v, I received a phone call from the Cisco Nexus 1000v Product Manager. I can only guess that he tracked me down because I posted a Bug ID in there. Regardless, he was very interested in making sure that my issues were resolved, and he pulled some resources together to help me out. I needed the help because my Secondary VSM had started into a reboot loop. Even deploying a fresh VSM would do the same thing after the Config Sync happened. While pulling some debugs off of the busted VSM, somehow 6 of our VEMs (Hosts) unregistered with the Primary VSM. My TAC Engineer was out of the office, but an Engineer from the 1000v Escalation Team got on the phone with me, and started digging around. What he found was this: a 3750 switch, home to several Development ESX Hosts, using a port-channel connected via vPC to our Nexus 7000 switches. The only traffic allowed across this port-channel was the Control/Packet/Managemen...

We recently installed a Dell PowerEdge M1000e chassis with a couple of blades in it. Since we wanted to run FCoE to the blade, our only option (at least that was available at the time) was the 10 Gig pass-through blade. After hooking everything up and installing ESX, we found that only one of the 10 Gig links would come up for each host - not a promising start. We also had a score of annoying little problems across the CMC management interface and the KVM interface. So far, I'm not exceedingly impressed by Dell's blade offering.

Continuing on with the hate part of my relationship, we recently ran into an issue where our Primary VSM died on us. It would boot up to this: Loader Loading stage 1.5. Loader loading, please wait... User Access Verification KCN1K login: admin Password: No directory, logging in with HOME=/ Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php System coming up. Please wait... Couldn't open shared segment for cli server System is not ready. Please retry when...

Over a year ago, we deployed Cisco Nexus 1000v virtual switching into our VMware cluster. I love some of the features it offers, but the problems we have run across still haunt me. The first time I deployed into our environment, it was into a virtualized vCenter instance. We were running it on a separate SQL server VM in the cluster. This was all fine and dandy - until we had a major power outage that took the entire cluster offline. The problem with this scenario is that it leads to recursive and cascading failures if not carefully designed. Even if it is carefully designed, it can still make for an unpleasant scenario. Think this through with me. In the event of a complete power outage, what is the recovery process with a typical virtualized vCenter installation? I believe it looks something like this: 1) Power the hosts back up 2) Power up AD server (physical) 3) Locate and power up SQL server (unless this is on-box with vCenter) 4) Locate the vCenter VM and boot it up What about wi...

Concept After successfully installing Mandrake Linux on my Cisco CE560, I decided the next "logical" step was to attempt to install CallManager on my Cisco Content Router 4430 (CR4430). The hardware appears to be identical except for slightly different memory configuration and different hard drive configuration. Hardware The Cisco CR4430 has the following hardware characteristics : Intel Pentium III 600 MHz 4x256 MB SDRAM Symbios sym53c8xx SCSI controller 1x18GB 80-pin Ultra3 LVD SCSI Hard Drive (my hard drive was bad, so I ended up using one of the 36GB drives from the CE560) Asiliant 69000 Video Adapter with 16-pin output header (pinout unknown) (Win2K sees this as a Chips and Technologies 69000) 2xIntel 82559 Ethernet Adapters (Windows sees these as Intel Pro/100B network adapters) RJ-45 Serial Console port at 9600,8,N,1 Note: Can only boot from floppy or SCSI (or Flash)! Preparation In order to pull this off, I used the following items: Cisco Console cable A laptop with ...