Lessons Learned From the SolarWinds Supply Chain Hack

In a recent Linux Foundation blog post titled “Preventing Supply Chain Attacks like SolarWinds,” the foundation’s Director of Open Source Supply Chain Security, David A. Wheeler, adamantly pushed the need for software developers to embrace the LF’s security recommendations to prevent even worse assaults on government and corporate data security in the wake of the rampant data breach.

Wheeler’s post is timely and filled with information to make it harder for hackers to exploit the future systems we all depend on. He includes 11 Linux Foundation recommendations including how organizations can harden their build environments against attackers, the need to begin shifting towards implementing and then requiring verified reproducible builds, and the practice of changing tools and interfaces so unintentional vulnerabilities are less likely.

According to Wheeler, SolarWinds met some of the foundation’s defensive measures. None of them prevented the successful SolarWinds attack, he said. More software hardening is needed.

The SolarWinds Orion software product is proprietary. So how can open-source coding methods help create better security?

SolarWinds followed some poor practices, such as using the insecure FTP protocol and publicly revealing passwords, which may have made these attacks especially easy, Wheeler offered in his Linux Foundation blog.

“The SolarWinds breach did not provide IT pros with any new technical insights, but it did provide a new urgency for countering that kind of attack,” he told LinuxInsider.

Cyberattacks typically exploit unintentional vulnerabilities in code. Most other attacks, at least in open-source software, involve a tactic called typosquatting. This approach creates malicious code with an intentionally similar name to a real program, he explained.

The SolarWinds breach did something different. It subverted a build environment, which up to this point has been a less common kind of attack, he noted.

“Fewer security pros have focused on countering this kind of attack. That may change in the future, especially since almost all typical security measures do not counter this kind of attack,” he said.

The Blow in SolarWinds’ Assault

Numerous U.S. government agencies and many private organizations that use SolarWinds Orion software were severely compromised. This was a very dangerous set of supply chain compromises that the information technology community and the open-source community must learn from and take action on, according to the Linux Foundation.

The federal Cybersecurity and Infrastructure Security Agency (CISA) issued Emergency Directive 21-01 declaring Orion was being exploited, had a high potential of compromise, and was a grave impact on entire organizations when compromised. The more people look, the worse stuff they find. Wheeler believes that a second and third malware compromise was identified in Orion.

The Orion platform is a scalable infrastructure monitoring and management platform. It helps IT departments simplify administration for on-premises, hybrid, and software-as-a-service (SaaS) environments.

Investigators found malware called Sunspot that watched the build server for build commands. When it found such commands, the malware silently replaced source code files inside the Orion app with files that loaded the Sunburst malware.

Sunspot’s compromise of SolarWinds Orion is not the first example of these kinds of attacks. Still, it demonstrated just how dangerous they can be when they compromise widely-used software, noted Wheeler.

In-Depth Analysis

Given the magnitude of the SolarWinds hack, LinuxInsider asked Wheeler to dive deeper into how supply chain security standards might benefit from the Linux Foundation’s latest recommendations.

LinuxInsider: Would the SolarWinds breach have been less possible if the software was open source?

David A. Wheeler: The closed source nature probably made the breach harder to detect, but all software is vulnerable to this kind of attack. Software developers modify source code to maintain software. Software users usually install software packages that were generated from source code. Converting source code into an executable package is called “building,” and building runs on some “build environment.”

In this case, an attacker subverted the build environment, so the source code seen by developers was fine, but the final installed software package was unknowingly changed.

OSS is much easier to re-run a build that can detect subversions. Close source code has added technical and legal challenges to detecting them. OSS has a potential advantage, but developers have to act to take advantage of that potential.

What could have prevented the intrusion?

Wheeler: The best way is something called a verified reproducible build or deterministic build. This is a process that produces exactly the same results from identical inputs, even when run by different organizations. It has been verified by independent organizations. It makes code subversion much harder because an attacker then has to subvert multiple independent organizations, and even if that happens later detection is much easier. Other techniques are much weaker.

These attackers appear to have been well-resourced. It is dangerous to depend on an attacker never succeeding. Examining built packages can in theory find problems, but the scale of real-world programs makes such analysis expensive, and problems will often be missed. The problem was eventually found by monitoring, but in this case, it caused extensive damage before detection.

A verified reproducible build is similar to a financial audit where a financial auditor determines if a result is correct. The essential problem with SolarWinds was that no independent process verified the build result was correct.

How practical is it for the software industry to adopt this LF recommendation?

Wheeler: Some projects already have reproducible builds, so it is possible to do. The reproducible builds project has created a modified version of Debian GNU/Linux (specifically of bullseye) where over 90 percent of the packages are reproducible. However, in practice it will take time for many OSS projects and even longer for many closed source projects.

Historically no one checked if builds were reproducible, so projects have accumulated many constructs that make builds irreproducible. No fundamental technical hurdles exist; just a large number of little things must be found and changed. The combination of all those little changes takes significant effort in bigger projects.

Closed source software has additional challenges, both technical and legal. Unlike OSS, closed source software is normally not designed to be rebuilt by others. Closed source software developers will need to invest significant effort just so others can rebuild it. Plus, their business models typically depend on legal restrictions on who has access to the source code.

What might be needed are special contractual agreements to share code not done before. But while it is harder to do this with closed source software, these challenges are surmountable.

What will its adoption take?

Wheeler: Customer demand! As long as customers blandly accept black boxes and products without verified reproducible builds, developers have no reason to change.

A slow move away from true black boxes is under way. Customers often say they do not need to know how something works, but true black boxes mean that the customers are taking on an unknown amount of risk. Many closed source software suppliers (like Microsoft) now have mechanisms to provide at least some visibility to source code to help customers better manage their risks. Open-source software, of course, allows anyone to see the code.

We are at an interesting point for reproducible builds. So far, some projects have worked on it, even without obvious demand from customers. Add that demand and a rapid increase in its availability will occur.

How much impact did the open-source practice of reusing code have?

Wheeler: It is not clear to the public exactly how SolarWinds’ build environment was breached. We know it was a Windows system. In a grand sense it does not matter. Defenses can be very good, but it is unwise to assume a system cannot ever be breached. Good security involves not only good prevention but also detection and recovery.

Future build environments will also be breached. We should try to harden build environments against attack, but we should also develop detection and recovery mechanisms so that any breach will not lead to the damage this breach caused.

How viable is instituting a software bill of materials (SBOM) in preventing typosquatting as the LF suggested?

Wheeler: SBOMs can help counter typosquatting. It is easy for developers to look at a name and read what they expect it to say, not what it actually says. SBOMs provide visibility to others, including customers, of what is contained in a component, just like food ingredient lists explain what is in our food. With a list, others can look for suspicious components, including names that are similar to but not identical expected names.

As Associate Supreme Court Justice Louis Brandeis said, “Publicity is justly commended as a remedy for social and industrial diseases. Sunlight is said to be the best of disinfectants…”