“A Different category than what we have seen.”
The Solarwinds and Codecov attacks have shown that typical security mechanisms won't protect against stealthy attacks from well-funded adversaries. The Senate hearings led by Senator Warren (D-VA) (quoted above) highlighted why detection and response are now more critical than ever.
Defenders must expect copycats in the future. Relegating this to a “SolarWinds attack” or a “Codecov problem” gives a false sense of security without addressing the underlying, global software supply chain problem we’ll soon face again. See a previous blog for experts' take on the gravity of Solorigate.
Respond. Then Detect and Respond Again
Even before SolarWinds and Codecov, cautious security practitioners would “assume breach” and tend towards zero trust. But the novel and dangerous aspect of Solorigate and Codecov is that the vector is within vendor code, not malware, not phishing, not port scanning, not DDoS. These supply chain attacks are so stealthy only security firms who have been hacked themselves are discovering them (like FireEye in the SolarWinds case).
Make no mistake, a response doesn’t simply mean patching the exposed Orion servers or rotating the Codecov keys. Patching is an absolute first step. Assuming that a breach has occurred, it’s then time to search for the backdoors set up by the adversaries in the meantime.
The “dwell time” left open between the attacker's foothold and when they’re found was 6+ months for SolarWinds, two months for Codecov. During that time, Solarwinds’ 18,000 customers and Codecov’s 29,000 were potential downstream casualties. During that time, the originators of the attack and countless “affiliates” - parasitic bad actors - had time to deploy backdoors that could still be open. This leads experts to believe we’ll be seeing fallout for many years.
This post highlights three technologies within the Microsoft 365 and Azure stack to detect and respond to issues spawned from software corrupted in its supply chain. Why Microsoft? Sure, there are third party services to detect and respond against issues, but Microsoft knows better than all others the typical behavior of its critical directory and OS services. When that behavior deviates from the norm, Microsoft’s tools are closest to the source and most likely to detect subtle indicators of compromises.
Three Key Technologies to Detect and Respond
In order of simplicity vs reward, here are three technologies uniquely suited to detect and remediate stealthy supply chain attacks, and how.
- Defender for Identity
- Azure Sentinel
- Defender for Endpoint
1. Defender for Identity
No matter what the attackers intended target, it’s almost certain they’ll be performing a reconnaissance scan in the environment to determine where its crown jewels are. And the moment they scan an Active Directory Domain Controller, the moment an anomalous behavior alert may be flagged for investigation by Defender for Identity. In many attacks, especially when the goal is extortion via ransomware, Active Directory is the crown jewel.
Here’s an analogy to outline what Defender for Identity does. Imagine a busy highway mainly travelled by cars, SUVs and trucks. That’s a traditional domain controller, receiving and responding to netlogons, LDAP queries, and replications. Now imagine a B-52 airplane rolling down the highway. Of course, that’d be enough to alert a concerned driver to call the police. Well, such anomalous traffic, if occurring on the Domain Controller, would raise a flag in Defender for Identity, alerting a help desk or Security Operations Center.
Only instead of the anomaly being an aeronautical outlier, Defender for Identity can notice the stealthy traffic disguised as normal SMB, LDAP, or netlogon traffic. You can think of Defender for Identity as a specialized Intrusion Detection System for Domain Controllers.
Keeping track of the naming changes of this product/service may be more difficult than enabling it. For owners of an EMS E5 license, Defender for Identity can be used (while the other features of EMS, such as Microsoft Endpoint Manager, or Azure AD Premium 1, are EMS E3 features).
My colleague, Mark Brezicky, wrote on Defender for Identity (under its old name), so check out the technical detail here Secure and Monitor Domain Controllers with Azure ATP (enablingtechcorp.com).
Microsoft noted significant downstream affects from the backdoor being opened by Solorigate, with identity systems like AD Federation Services at risk of anomalies in SAML tokens and in Microsoft 365 API access patterns. Microsoft published identity-specific indicators of compromise to evaluate Azure AD, including:
- Modified application and service principal credentials/authentication methods
- Modified federation settings
- New permissions granted to service principals
- Directory role and group membership updates for service principals
In summary, deploy Defender for Identity on all Domain Controllers. Assume breach and search for indicators of compromise using MSFT guidance. Don’t delay, the dwell time is already high by the time you’ll be made aware.
2. Azure Sentinel
Don’t let the name of the service fool you. Sentinel is a full-service SIEM (Security Information Event Management) that happens to run out of Azure. Sentinel can ingest logs from on-premises systems like firewalls and proxies. In most organizations, sifting through such logs is done rarely if ever. Sentinel’s automation can help make that task easier.
How would a SIEM help in these supply chain attacks? If the firewall logs were ingested, they might have exposed the slightly anomalous command and control traffic Orion servers sent to adversaries’ servers. But it’s doubtful.
FireEye only found the Orion breach after exhausting every other lead. They’d looked everywhere else before finally decompiling SolarWinds’ code (18000 files, 3500 executable files, and 1M lines of assembly code) to find the implant. After thousands of hours of looking at every other place, FireEye’s CEO described the implant as “The last place, not the first place, you’d look” to ID the benign looking traffic.
Solorigate’s Command and Control (C2) traffic was nearly untraceable. Remember the dangers of a long dwell time? The adversary actually executed a dry run with harmless code into a SolarWinds update in October of 2019, just to ensure their code made it into SolarWinds’ production environment. They patiently waited to put their first malicious implant into Orion code in March 2020. The malignant code persisted through an update in June, 2020, and wasn’t detected until FireEye disclosed in December, 2020.
Sentinel would be more likely to expose the adversaries’ behavior once they started doing reconnaissance using their C2 protocols. Using logs from Defender for Identity, firewalls, and Intrusion Detection Systems, the SIEM could aggregate and collate information to highlight anomalies.
Another helpful use of a SIEM like Sentinel is to occasionally hunt for anomalies (especially when risk of breach is high, like when dwell times are 2-6 months). Microsoft published Indicators of Compromise and threat hunting scripts for organizations to use to determine if backdoors had been set up. Azure Sentinel Hunting (enablingtechcorp.com) shows a bit about hunting in general, and SolarWinds Post-Compromise Hunting with Azure Sentinel - Microsoft Tech Community shows the specific scripts Microsoft published.
3. Defender for Endpoint
Microsoft Defender for Endpoint (MDE) is an extremely powerful tool to detect odd behavior. Here, it was listed as tool #3 simply because it’s not trivial to immediately rip/replace a current Anti-Virus or Endpoint Detection and Response system. Steps 1 and 2 are quicker wins.
Defender for Endpoint has several mechanisms to Protect endpoints (not just detect and respond). Microsoft Defender for Endpoints - Endpoint Detection and Response Management (enablingtechcorp.com). Doing so can minimize the surface area, and protecting the devices from secondary attacks by parasitic actors.
Once the surface area is reduced, Defender for Endpoint detects anomalies on the desktop, and blocks known or potentially risky behavior. For instance, if an app or a macro is installed by a malicious C2 interaction, Defender blocks the installation and alerts an administrator. Reports are generated and even highlight specific Indicators of Compromise.
The reports are published in Microsoft 365 security center, available to all Microsoft Defender for Endpoint customers. In addition to detailed descriptions of the attack, and indicators of compromise (IoCs), the reports provide real-time data aggregated from signals across Microsoft 365 Defender, indicating the all-up impact of the threat to the organization, as well as details about relevant incidents and alerts to initiate investigation on. These reports continue to be updated as additional information becomes available.
And drilling down, Defender provides specific instructions for specific machines to protect or remediate them. These capabilities are rich, and found in the Microsoft Defender for Endpoints Threat and Vulnerability Management (enablingtechcorp.com) portal.
Key to this process is to hunt for issues like this and ensure the system auto-remediates, and if not, manually intervene. Admins can remove malware by entering an infected machine from the Defender for Endpoint portal Microsoft Defender for Endpoints Live Response (enablingtechcorp.com).
Best Practices for Protection
So far, this article discussed tools to detect and respond against the inevitable, stealthy supply chain attacks. The fact is, putting up an extra barrier or two of protection in your environment will not stave off a motivated adversary. Microsoft loosely estimates that “1000 very skilled, capable engineers” were needed to build the sophisticated Solorigate attack.
But, organizations who implement all available protections are better off than those who don’t. The greater the cost for an adversary to attack, the more likely they are to move on toward an easier target.
To prioritize the to-do list, use the recommendations provided by Microsoft’s Secure Score if you have no other guiding policy to protect your identities, devices, data, apps, and infrastructure.
Some highlights of common checklists include:
- Protect privileged identities with Privileged Access Management and/or Privileged Access Workstations.
- Protect all identities with Multifactor Authentication, especially on accounts eligible to upload code to CI/CD systems.
- Patch systems. This includes ensure device and app compliance and health with Microsoft Endpoint Manager/Intune. Half of Government Security Incidents Caused by Missing Patches
- Secure your backups to respond to Human Operated Ransomware | Microsoft Docs
- Educate users to avoid phishing and identity compromise. Test and train using Microsoft’s recently updated Attack Simulator service.
- When developing, do not store credentials and secrets in environment variables instead storing them outside of code in a secrets management system like Azure Key Vault.
No one can predict the next stealthy supply chain or nation-state attack, but knowing it's coming, prepare with these steps:
- Assume breach.
- Deploy Defender for Identity on all Domain Controllers.
- Import logs from other systems services into Sentinel, and use its automation to hunt for Indicators of Compromise.
- Consider Defender for Endpoint as your next-gen EDR tool. It’s an upper right leader in the Magic Quadrant, and customers are happy, for multiple reasons.
- Don’t delay, the dwell time is already high by the time you’re likely made aware of these stealthy attacks.