An apparent prefix leak from an errant router misconfiguration caused Google to lose control of several million of its IP addresses for more than an hour on Monday.
During the event, Internet traffic was misrouted to China and Russia from Nigeria. The incident initially sparked concerns that it might have been a malicious hijacking attempt.
The mishap made Google’s search and other services unavailable to many users intermittently. It caused problems for Spotify, Google cloud customers, G-Suite users and Youtube viewers, among others.
The problem started when the MainOne Cable Company in Lagos, Nigeria, improperly updated tables in the Internet’s global routing system to declare that its autonomous system was the proper path to reach 212 IP prefixes belonging to Google. China Telecom shortly thereafter improperly accepted the route and announced it worldwide.
That move, in turn, caused Russia-based Transtelecom and other large service providers to follow the route. The misdirected traffic led to China Telecom, the Chinese government-owned provider that recently was caught improperly routing Western carriers’ traffic through mainland China.
“We’re aware that a portion of Internet traffic was affected by incorrect routing of IP addresses, and access to some Google services was impacted. The root cause of the issue was external to Google, and there was no compromise of Google services,” a Google spokesperson told TechNewsWorld via company rep Lindsay Hart.
Questionable Explanation
Google is adamant that the mishap resulted from a prefix leak in configuring BGP, the Internet’s main routing protocol, rather than a hijack. Each Internet Service Provider advertises to all others a list of Internet Protocols it owns. A prefix leak occurs when an ISP advertises a range of IPs it does not own, according to the Google spokesperson.
BGP is a decades’ old technology that is not cryptographically secure, enabling these types of mistakes by third parties, which is what this incident most likely was, said Rick Moy, chief marketing officer at Acalvio.
“There have certainly been nefarious BGP hijackings in the past, and I am sure they will continue because they enable traffic hijacking and even cryptojacking,” he told TechNewsWorld. “Also, unfortunately, there is no quick fix.”
These types of issues are typically due to hacking, rather than a mistake that was made, noted Chris Rivers, vice president of Web development at MGH.
However, in this case, the incident seems to have been caused by an error that occurred during planned network maintenance.
“It is interesting that the traffic was rerouted to countries already known for ‘big brother’ uses of technology to spy on citizens,” Rivers told TechNewsWorld. “There was definitely a vulnerability via mistake that Google is denying.”
Looking at the bigger picture, this type of situation caused a massive denial of service to the G Suite. Attacking a vulnerability like this would be designed to disrupt service to its intended audience, he added.
No Harm, No Foul?
Still, Google claims that a Nigerian ISP caused the problem with no malicious intent. This issue only affected network traffic.
Since nearly all Internet traffic to Google services is encrypted, there was no increased risk of data exposure as a result of this leak, according to Google.
Google maintains that nothing indicates this was an attack or a breach. Google’s internal analysis is consistent with Mainone’s claim that the situation was caused by a misconfiguration.
“Given the time to resolve this issue, it is highly likely that this was an honest mistake by a core Internet provider,” said Brian Chappell, senior director for enterprise and solutions architecture at BeyondTrust.
“The mechanisms for managing the routing of traffic across the Internet have been an area of concern for some time, as there is no real authentication for the information. It is a trust-based approach,” he told TechNewsWorld.
Regardless of an intentional attack or mistake, the implications can range from denial of service and slow response of service to the compromise of data in transit, said BeyondTrust CTO Morey Haber. If there had been an intention to target an ISP, this could have been a serious incident.
“While [data compromise] is much less likely due to all Google traffic being encrypted, there are scenarios from man-in-the-middle attacks to compromised keys that could be utilized in a blended attack to decrypt the traffic,” Haber told TechNewsWorld.
What Comes Next?
Viewed as an accident, this incident will drive attention and activity toward a more robust solution, suggested Chappell. The organization responsible for the mistake very likely will implement more stringent processes to avoid such an event happening again.
“Assuming that the systems in question are accessed through a secure solution, such as a privileged password management solution, it is likely there were session recordings that could be searched to find the event and allow for rapid remediation,” he said. “If not, that is definitely the first step that organizations should be taking.”
Viewed as a malicious action, it highlights the inherent insecurity of routing protocols. While core providers are likely to have significant controls around the manipulation of protocols and tables within their organization, that does not eliminate the possibility of malfeasance by internal and external parties. Either way, we can expect to see renewed activity in this space, according to Chappell.
Whether accidental or deliberate, there are implications that need fixing, noted Haber. The rerouting of traffic out of a geographic region due to pure ISP hygiene is unacceptable. If it had occurred in other regions — like Europe, the Middle East and Africa — it could have been perceived as an EU General Data Protection Regulation violation.
Attack or Accident: Same Impact
This type of attack or accident can have real financial impact for companies doing business online, warned Chappell. Being able to redirect traffic away from legitimate sites, either to interrupt services or worse, to present fake sites, undoubtedly would lead to immediate financial and secondary reputational loss for organizations.
“While it didn’t actually stop [Google’s] platform working, it may have impacted many sites which rely on their services. The final tally will become apparent in time,” he said.
This type of incident is a reminder of the dependencies all cloud users face. Entities in far regions of the world can affect traffic and cause an outage in services users rely on every day, added Haber.
“Businesses operating online need to be reminded that their dependencies on cloud services should have contractual requirements in the form of SLAs,” he said, “and that operational backup plans should be developed in case incidents like this materialize as full-blown attacks.”
Loading ...
