PCCW Pins Youtube Outage Responsiblity on PTCL’s Internet Exchange (PIE)
A few weeks ago, a few of our subnets were found ‘lurking’ in the PCCW Network even though we were not advertising them to PTCL ITI. 22.214.171.124/24 and 126.96.36.199/24 were in the PCCW routers routing table. When the users complained of severely degraded performance, we investigated and found that PCCW routers were somehow ‘remembering’ these subnets!
A PCCW engineer then ‘refreshed’ the BGP4 routing table and that solved the problem.
In this backdrop, it is difficult for me to believe PCCW after the above skirmish with them a few weeks ago. Firstly, addresses that do not belong to a service provider are filtered by the routers. So if PIE advertised the Youtube.com subnet, it should have been filtered by PCCW in the first place. Some service provider use BGP communities to tag certain subnets so that they be blocked. This is usually used as a denial-of-service protection mechanism. So, even if this was used, this could not have blackholed the entire youtube traffic.
I’m checking with PTCL and will post the ‘actual cause’ once we arrive at a logical conclusion!.
Ok. Here is how it happened. The PTCL guru advertised a /24 prefix for the Youtube.com address space. From Google, a /22 was advertised. So the longest match rule forced the traffic to go to PCCW which in turn went to PTCL which was then dropped because of the null route.
Lesson for PTCL: Be careful where things relate to the Global Routing Table
Lesson for PCCW: Don’t trust the downstream. Routes must be filtered.
Lesson for the rest of us: Don’t trust anyone 🙂