During the Senate hearing on the software supply chain attack that corrupted SolarWinds and its ~17,000 Orion customers, there were several salient themes and many fascinating details. Enterprises, government entities, and software suppliers can take note.
1) Relegating this to a “SolarWinds attack” gives a false sense of security and shuts down addressing the underlying, global software supply chain problem we’ll soon face again.
2) The attack was on the supplier’s software update process, not just the source code. This is novel and more sophisticated than seen before, although the indiscriminate nature of its global spread has been seen in other software supply chain attacks, like NotPetya.
3) Without mandatory reporting procedures and punitive actions for state-sponsored attacks, they will inevitably continue.
4) Attacks on software update procedures should be treated as seriously as an illegal, physical attack on a hospital, because of the unintended consequences that reach far beyond an intended target.
The panel included CEOs from FireEye, SolarWinds, and Crowdstrike, as well as Microsoft President and longtime legal chief, Brad Smith. Some of the fascinating points are summarized here.
Anatomy of the Attack within FireEye
Kevin Mandia, Fireeye’s CEO, was detailed and concise. FireEye was the first “Stage 2” victim, infiltrated vis-à-vis the “Stage 1” supplier’s implant (SolarWinds).
Two things stood out as unique to FireEye:
- The adversary didn’t just modify the source code, they modified the software build process. Senator Rubio later questioned SolarWinds’ CEO about TeamCity, a DevOps platform used by SolarWinds. Sudhakar Ramakrishna said they hadn’t ruled it out as the entry into SolarWinds. He went on to say they are still investigating three potential vectors. Mandia pointed out that getting down into the software build process makes this a much more “portable attack” than just SolarWinds. Infiltrating at this stage inserts malware at the very last step before the code is put into production for unknowing customers.
- The adversary executed a “dry run” with harmless code” into a SolarWinds update in October of 2019, just to ensure their code made it into SolarWinds’ production environment. They patiently waited to put their first malicious implant into Orion code in March 2020. The malignant code persisted through an update in June, 2020, and wasn’t detected until FireEye disclosed in December, 2020.
FireEye knew something was amiss, but only found the Orion breach after exhausting every other lead. They’d looked “everywhere else” before finally decompiling SolarWinds code (18000 files, 3500 executable files, and 1M lines of assembly code) to find the implant. After thousands of hours of looking at every other place, Mandia described the implant as “The last place, not the first place, you’d look.”
Sophistication Points to Russia
Later, Mandia elaborated on the sophisticated stealthiness of the attack. After the malware was installed, it was silent for 11 days, so that once it started communicating with the adversary’s command/control server, it wasn’t immediately traceable to the Solarwinds upgrade that occurred 11 days earlier.
This demonstrated “Superb tradecraft,” according to George Kurtz, Crowdstrike’s CEO, who was careful not to name a perpetrator. Microsoft’s Brad Smith was not so subtle, indicating the Russian Foreign Intelligence Agency as the only evidence. Microsoft loosely estimates that “1000 very skilled, capable engineers” were needed to build this sophisticated of an attack. Mandia agreed that through process of elimination from 17 years of forensics at FireEye, the attack was “Most consistent with Russia.”
Amazon Conspicuously Absent
Many US Senators took the opportunity to point out the absence of Amazon Web Services at the hearing. It was in AWS’s US data centers where the adversaries set up their command/control servers. This provided better cover than routing to IP addresses, globally, which could’ve triggered alerts.
The malware went through keys and tokens (stealing the organization’s identity architecture) to create a backdoor. Using legit authentication keys made it harder to detect because they attackers could log in and look like a legit employee. The attackers were able to bypass MFA within FireEye. Once into the target organization’s systems, the adversary targeted emails and documents of specific victims. In FireEye’s case, they attempted to steal source code of the software tools used by a Red Team.
The Problem Runs Much Deeper than SolarWinds
Because Solarwinds’ process to develop and release code is common across the industry, Senator Warner (D-VA) agreed this attack is in a “Different category than what we have seen.” He and Smith agreed that the world relies on patching of software and highlighted the danger of this level of intrusion to software update processes. Smith mentioned the world, including hospitals and critical infrastructure, relies on software updates, and to tamper with that process is in effect the digital equivalent of an attack on the public health system.
While it’s believed the adversary conceived of the plan to specifically create backdoors into some of Orion’s customers’ networks, Brad Smith, Microsoft’s Chief Council, then explained how the attack indiscriminately went to 17,000 organizations (Stage 2 victims), including at least nine US government organizations. Smith went on about the danger of an attack with such a widespread, in fact, global reach. In other words, the adversary didn’t just target one victim, but by infiltrating SolarWinds, created up to 17,000 victims. Smith used an analogy to a burglar breaking into a single apartment, but then turning off the alarm for every residence in the city. This is clearly happening, other adversaries, not just the Russian Intelligence Agency, are attacking.
Smith called for the modernization of IT. Microsoft only saw the attack among their customers when it got to their use of cloud services like Office 365, but the initial attacks all occurred on premises. With limited telemetry of those initial compromises on premises, Microsoft and others are operating with less visibility than a collective group of services in the cloud. Smith and Kurtz outlined 5-6 best practices, many involving their own security products. The takeaway was to not only protect against breaches, but better detect and contain them before a disaster occurs.
Crowdstrike CEO Kurtz suggested “In addition to secure coding practices and adequeate code review, organizations must protect their development platforms and code repositories at least as well as their enterprise environment.”
Legal and Disclosure Recommendations
The executives and Senators called for accountability and a set of laws that (at least some) countries will abide by. Mandia stated that “Without risk and repercussions for foreign actors who attack, we’re all fighting a losing battle over time. It’s like we’re all playing goalie taking slap shots from Wayne Gretzky. Sometimes a puck is going to get through.”
Each vendor agreed that there should be a standard practice for organizations to disclose breaches. Mandia said that while FireEye had no strict legal obligation to report on the public nor private side, their duty required disclosure. He called for a disclosure model similar to the credit card industry, where they have an obligation to alert that something is going on, even before they know exactly what’s going on.
It Could Have Been (and will get) Worse
When asked about the risk to public infrastructure in the future, panelists acknowledged that the attackers exemplified “Focused, disciplined data theft” (Mandia). By focusing on data, rather than malicious intent like ransomware, the adversaries actually did more work than they would’ve had to have done to be destructive. In other words, damage to our critical public infrastructure could be imminent.
Smith agreed this is the “Largest and most sophisticated operation of this sort that we’ve seen.”
The transcript and video of the hearing is posted at Senate Intelligence Hearing on SolarWinds Hacking | C-SPAN.org (c-span.org)