Cisco 7609 Bugs Galore
The Cisco IOS is, I read somewhere, not unlike women :-). Just when you think that you know her, you see a new side that you hadn’t seen before. Oh Yeah! 🙂
The Nayatel Core team has been able to create two new Bug IDs in the Cisco 7609s IOS. The first one is CSCsh60112. This bug is a real nasty one. Imagine this: all internet traffic entering the Autonomous System stops. You get a call telling you that the Internet is down. You login and check the BGP neighbor relationship and find that all neighbors are up and reachable. Next, you check the IGP. Everything seems to be in order. ISIS is working perfectly. Hmmm. You dialup and reach the internet through a 2nd provider and check the routes on the Internet. Voila! BGP is not advertising the subnets anymore!.
A quick show ip route on the Internet gateway confirms that the nailed-down /24s which were being redistributed into the IGP have disappeared. You login to the PE router on which the redistribution was taking place and check the routes using the show running-config command. Well, the routes ARE shown. But they are not being redistributed anymore!!!!.
Solution: You decide to remove the static routes to Null0 and put them back again. This solves the problem and since the Gateway now learns the /24s from the IGP, BGP starts advertising those routes again. When 622Mbps of Internet connectivity goes down, keeping one’s cool is the best thing to do :-). And that reminds me of the decade old Sysadmin rule: Rule #1. Don’t Panic. Rule #2. Don’t Panic. Rule #3. Don’t Panic…. Words to Live By!
Here’s the official Bug Detail: After SSO failover, static Null0 routes are not seen in the RIB but are still seen in the running config!
As if this wasn’t enough along came Bug CSCsm32555. The Bug description is “Unable to Route traffic across MPLS VRF to GRE peer”. The Customer had multiple sites connected via Layer 3 MPLS VPNs. A remote site did not have MPLS-enabled ISP so I decided to bring the traffic via a GRE tunnel and make it a part of the Customers VPN. Easier said than done. When the GRE tunnel was made part of the VRF, connectivity to the remote site (from where the GRE tunnel originated) was broken.
I terminated the tunnel on a 7206-VXR to ensure that this was an IOS bug. On the 7206, things were fine and it was proven that this was a 7609 IOS bug.
This bug is now resolved in 12.2(33.0.6)SRC!.