Cloudflare last week announced that it has fixed the Cloudbleed software bug responsible for a buffer overrun problem that caused its edge servers to return private information in response to some HTTP requests.
That private information included HTTP cookies, authentication tokens and HTTP POST bodies. However, SSL private keys weren’t leaked, said Cloudflare CTO John Graham-Cumming in an online post.
“This happened in response to a very small number of requests in the Cloudflare system — about 1 in 3.3 million,” a Cloudflare spokesperson said in a statement provided to TechNewsWorld by company rep Katie Warmuth.
Some of that data had been cached by search engines.
Cloudflare reviewed the available related cached information and “took comprehensive steps to clean up any residual material found in storage caches,” the spokesperson noted.
Cloudflare found that data for about 150 of its 6 million customers had been impacted.
The company has reached out to “a number of search engines to review and remediate the information in their caches,” the spokesperson said.
All identified episodes have been cleaned, and Cloudflare continues to work to confirm whether other residue persists.
There are at least 16 other search engines on the Web apart from Google, including Bing and Duck Duck Go.
What Happened
Tavis Ormandy, a vulnerability researcher with Google’s Project Zero, notified Cloudflare about the problem on Feb. 17. The memory leak occurred from September to Feb. 18, with the greatest period of impact being from Feb. 13-18.
A bug in Cloudflare’s Ragel-based parser was the cause. It had been dormant for years, but came alive last year, when Cloudflare began replacing the Ragel-based parser with a new one it wrote, named “cf-html.”
The switchover subtly changed the buffering, which enabled the leakage.
The problem lay with Cloudflare’s implementation of the Ragel-based parser it was using, and not with the parser itself or with cf-html.
When it learned of the problem, Cloudflare turned off three features — email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites — that used the parser chain causing the leakage.
The Email Obfuscation feature, which was changed on Feb. 13, was the primary cause of the leaked memory, Cloudflare’s Graham-Cumming said.
Cloudflare worked with Google and other search engines to remove any cached HTTP responses.
The initial mitigation took 47 minutes, and the team completed global mitigation in less than seven hours. The industry standard is usually three months, Graham-Cumming noted.
Cloudflare “responded incredibly swiftly and effectively to identify and remediate the bug, and work with search engines around the world to purge any sensitive data cached by their crawlers before it could be exposed to the public,” Tripwire Principal Security Researcher Craig Young told TechNewsWorld.
The Gravity of the Problem
“We realize that this was a very serious bug and that we dodged a bullet in that [it] did not lead to more problems than it did,” the Cloudflare spokesperson remarked.
Cloudflare hasn’t discovered any evidence of malicious exploits of the bug or other reports of its existence.
That “is not the same as saying [the bug] was not exploited,” remarked James Scott, senior fellow at the Institute for Critical Infrastructure.
“It just means that no exploitation was detected,” he told TechNewsWorld.
That said, “the effectual security impact would have been limited unless an adversary consistently collected information for a prolonged period of time,” Scott added, “because the captured information would be a virtual grab-bag.”
That would be a “really inefficient and cumbersome” approach, he said.
Customer Precautions
Potential victims can “review and take steps to roll out things such as long lasting cookies, API keys or other persistent secrets,” the Cloudflare spokesperson suggested. Sites should “err on the side of caution.”
Users should change to complex credentials on the affected sites — including Uber, OKCupid and Fitbit — and should enable multifactor authentication where possible, ICI’s Scott said.
Also, they shouldn’t use the same credentials on multiple sites, he cautioned, and they should report any suspicious activity immediately.
Loading ...
