Working together to mitigate the threat of future malware attacks
KPMG and SolarWinds continue to work together to determine the origin of cyber attack and enable SolarWinds to move forward with confidence
Client
SolarWinds
Industry
Technology
Project
eDiscovery, Cybersecurity Defense and Application Security
Here, we describe some of the specific work KPMG has done, and continues to do, in collaboration with SolarWinds and DLA Piper to provide perspective so others can mitigate the threat of future attacks. As a trusted advisor, our overarching objective is to help SolarWinds be as open and transparent as possible and help allay clients’ security concerns.
Success in this ongoing exercise will help enable SolarWinds to maintain the trust of its various stakeholders—from customers and employees to regulators and activist investors—thus creating a foundation for responsible growth, confident decision-making, bold innovation and sustainable advances in performance and efficiency.
KPMG speaks SolarWinds' language
KPMG was initially retained for two primary purposes. The first was purely investigative, while the second focused on eDiscovery. This work focused on determining how bad actors were able to insert malicious code into SolarWinds’ flagship product, Orion, through a seemingly innocuous software update sent to thousands of customers.
Over the course of this engagement it became clear that we not only understood SolarWinds’ business and technology but could provide differentiated application security development guidance and lead an extended investigation and eDiscovery program. Mutual trust was a must and was established quickly.
Ask the right questions and get to work
When we started working with SolarWinds the attack had been broadly reported. From an investigation perspective we dug into how the SolarWinds build environment was compromised and tampered with. We immediately started working with the SolarWinds application teams. Our experience and knowledge of application development leading practices led us to ask the right questions around code security and quickly understand how SolarWinds works, their application-build processes, what systems are involved, and the most likely entry points for a breach.
A contingent of core KPMG professionals with software development experience and an understanding of what's involved in building these components at a more granular level began with SolarWinds’ source code control system (SCCS), which is where attackers typically go to modify code. Looking back over two years, we found the software code itself appeared not to have been maliciously modified by this attack. Next, we examined how the Orion software is compiled, which happens within SolarWinds third party software orchestration system, TeamCity. A close look at the TeamCity server also revealed nothing out of the ordinary.
The needle was in a non-working haystack
Ultimately, we discovered the attacker(s) had targeted a machine that was compiling the code itself. From a forensics standpoint, as detailed in SolarWinds’ blog post, KPMG discovered malware – referred to as SUNSPOT – that was deployed for the purpose of covertly inserting a backdoor into the SolarWinds Orion Platform during the software build process. It was determined the malware was designed by the threat actor(s) to target within the SolarWinds environment only the SolarWinds’ Orion Platform.
The SUNSPOT malware ran in the background on SolarWinds’ Orion Platform software build servers watching for a new build to take place. At build time, SUNSPOT would insert a backdoor (referred to as SUNBURST) contained in a temporary source code file used by the compiler. At the conclusion of the software build process, SUNSPOT would clean up the temporary source code file to circumvent detection. Hence, the codebase remained clean, while the compiled code was signed with the valid SolarWinds software certificate and shipped with the SUNBURST backdoor.
A related component of the investigation is the validation of the limited scope of the attack. Reviewing the malware indicated that only one SolarWinds product, the Orion Platform, was targeted. The other SolarWinds’ products did not appear to have been targeted by this malware.
eDiscovery to satisfy regulatory reporting
As for eDiscovery, we also worked with SolarWinds to develop a coordinated process to compile information to assist SolarWinds in expeditiously responding to existing and anticipated regulatory inquiries. KPMG evidence and discovery management professionals experienced in evidence and discovery management helped SolarWinds identify, preserve, process, review, and analyze relevant electronic information.
By implementing processes that enabled the rapid collection and review of millions of electronic records—some of which were complex file types and technologies—KPMG helped SolarWinds meet its post-event inquiries and demands, which was critical to assist SolarWinds in mitigating further financial and reputational harm. In all, our eDiscovery team downloaded terabytes of data to support SolarWinds response to inquiries including hearings before Congress and other governmental bodies such as the SEC, as well as legal proceedings involving shareholders.
Ramping up the AppSec
With so many security events happening at the application level, solidifying the application security (AppSec) risk profile is critical. KPMG is working with SolarWinds on three distinct AppSec streams. The first is pipeline security. It is likely that compile and build processes at many software companies would allow for similar exploitation by the attacker.
SolarWinds implemented additional security steps and detection points to help find and respond to various issues that materialize within the build process, at every stage, from initial code check in and compile to customer distribution. SolarWinds has rearchitected its build pipeline and is adding future forward telemetry into the process to safeguard its software development process, which is intended to give customers confidence in the code they receive.
The other two AppSec streams focus on credential and password hunting. When a network is breached, and the attacker(s) gain access to the network the typical response is to reset user and service accounts. Unfortunately, simply resetting accounts is not enough. The KPMG team helped SolarWinds identify how many different passwords or secret keys the attacker(s) had found and could potentially still use. Searching through millions of lines of source code, the AppSec team took inventory of the possible authentication credentials that had been exposed. SolarWinds then reset the credentials.
Beyond this, the AppSec team also took action to start the identification and remediation of legacy passwords as well as legacy operating systems and applications.
On the cloud
After validating the build environment, we turned our attention to the SolarWinds cloud.
There are three major components to KPMG cloud investigations. The first is Microsoft 365, which is SolarWinds’ email host. We pored over the unified access logs, and determined what was impacted, what was being accessed, what the attacker(s) were doing and when they were active in the environment.
The second is Microsoft Azure. The KPMG team is currently working on helping SolarWinds with privileged access management (PAM) and identity access management (IAM) by assessing the current state, understanding where there's room for improvement, identifying weaknesses and vulnerabilities, and facilitating the establishment of a zero-trust framework.
In connection with Azure, it was also important to confirm there were no additional surface principal names (SPN). One of the ways the attacker(s) maintained access to SolarWinds via the cloud was by placing a credential into this service account, thus allowing for remote access. We were able to confirm there were no other SPNs for the attacker(s) to retain access.
We've also been working with SolarWinds to validate its PAM and IAM in the Amazon Web Services (AWS) environment, as well as validate the current state of security, to determine whether there are known active vulnerabilities attacker(s) could potentially leverage in the future.
Regarding the cloud in general, SolarWinds’ focus is on building an industry leading secure cloud infrastructure, including rearchitecting security based on its learnings from the cyber incident.
Looking ahead
Detection is just one aspect of cybersecurity, albeit a critical one. The ability to isolate an intruder within a breached system is also essential.
Working with SolarWinds to strengthen and unify access management is the key goal of our collaborative post-attack work. The KPMG team continues to help SolarWinds in its efforts achieve a highly secure PAM model with tight oversight of accounts and the authority to read data and control systems. This PAM model will reduce SolarWinds’s attack surface and mitigate damage from compromised accounts.
Similarly, unifying IAM across disparate cloud and on-premises environments will greatly improve SolarWinds’ ability to neutralize a hacker’s effectiveness within the network. This reframed approach will enable SolarWinds to monitor its environments with one view and more rapidly lock down each of its environments in the event of a future breach.
Related Insights
Insight
SolarWinds explainer
An overview of the software supply chain attack used against SolarWinds
Insight
Security monitoring for software build pipelines
First steps to increase confidence in your build environment's integrity
Meet our team
David Nides
Principal, Advisory, KPMG US
David Nides
Principal, Advisory, KPMG US