FireEye & SolarWinds incidents: What happened?

A report that FireEye had been breached by nation-state hackers made the headlines. If you’re not familiar, FireEye provides hardware, software, and services to investigate cybersecurity attacks, protect against malicious software, and analyze IT security risks. For part of their offering, you can think of them as Red Teamers, where they can be hired by organizations to test their defenses.

Doing this means that FireEye creates custom tools to try and breach their client’s security. By becoming compromised, some of those tools were stolen by the attackers, which means they could have been used by those threat actors to access FireEye’s very own customer’s systems, which include government agencies. Obviously, that sounded some major alarms.

Then, a few days later, the name SolarWinds became involved in the incident. In fact, CISA issued an emergency directive about SolarWind’s Orion products. So what happened, and what’s going on?

What’s going on?

Until recently, we didn’t really know how the attack had been carried out, or why it was claimed to be a highly sophisticated and nation-state backed attack. More recently, details have emerged that are shining more light on the situation. This article is a higher-level explanation. If you’re interested in the technical details, I highly recommend that you read this report by FireEye and this report by Microsoft.

SolarWinds, a widely used IT management suite of products, offers a platform called Orion Platform, which is used for IT monitoring. Essentially, you download their software onto your systems, and it collects data from your entire IT stack all the way from infrastructure to applications, to then feed it back to a central location.

So this is like going in and downloading regular software onto your computer, phone, servers, etc…

As with most software, over time, you receive updates. These updates can include new features, bug patches, or general modifications. In most organizations, those downloads are verified, approved, and performed by system administrators. They take a look at what’s available for download, how it will affect the systems, and how to perform the update with minimal disruption.

All part of typical and healthy IT operations.

Little did they know at the time that this routine operation would turn out to be a nightmare.

You see, this routine update turned out to contain a maliciously modified version of SolarWind’s Orion Platform. While it’s currently unclear how this happened, the result was that SysAdmins downloading and installing the update pushed from SolarWind themselves contained malware called a trojan horse.

More specifically, it contains a backdoor trojan, which can create a “backdoor” on the system it has infected, letting an attacker access that system and control it. Data can be accessed, modified, and downloaded. Additional malware can be created, added, or downloaded, etc…

The malware is very well hidden, evading detection systems not only at FireEye, but at other organizations compromised by this same attack (including government organizations).

Once on the system, the backdoor trojan goes to sleep for about 2 weeks, at which point it wakes up and retrieves and executes commands called Jobs, which can make HTTP requests to other servers, transfer files, execute files, profile the system, reboot the machine, and disable system devices.

Once again, the malware uses advanced techniques to hide itself, such as masking its network traffic to look like it comes from Orion itself, by using the Orion Improvement Program (OIP) protocol. It even stores reconnaissance results within legitimate plugin configuration files that make it blend in with legitimate SolarWinds activity.

In all, it’s obvious that the threat agents spent a lot of time examining and understanding how SolarWind’s Orion program works. Reverse engineering it, studying its patterns, and working hard to create malware that would mimic this behavior closely, evading detection as much as possible.

On top of that, the malware contains ‘blocklists’ used to identify whether the system is running any forensic or anti-virus tools that could detect it, adjusting its behavior based on any detections.

A lot of malware or attackers can end up being quite destructive to systems by making modifications to settings and behaviors. In this case, however, the attackers tried their best to hide their actions, by only making temporary modifications when possible. They would replace legitimate utilities with theirs, execute their payload, and then restore the legitimate original file. They would remove their own tools, even the original backdoor once they had gained remote access.

As mentioned in the FireEye research report:

The attacker used a temporary file replacement technique to remotely execute utilities: they replaced a legitimate utility with theirs, executed their payload, and then restored the legitimate original file. They similarly manipulated scheduled tasks by updating an existing legitimate task to execute their tools and then returning the scheduled task to its original configuration. They routinely removed their tools, including removing backdoors once legitimate remote access was achieved.

FireEye research report

Once the malware infiltrated the system and woke up, the threat actors used a number of different techniques to:

1. Gather information about the system and other systems
2. Exfiltrate information
3. Steal or forge credentials
4. Move laterally using those credentials

The techniques used are explained in greater technical detail in the original FireEye report, and they include:

• TEARDROP and BEACON Malware used
• Attacker hostnames matching victim environment
• IP addresses located in victim’s country
• Lateral movement using different credentials
• Temporary file replacement and temporary task modification

In addition to an explanation of the technique used, FireEye also includes potential opportunities to use those techniques in order to detect malicious actions.

The report also includes an in-depth analysis of the malware contained in the SolarWinds.Orion.Core.BusinessLayer.dll

What can we learn from this?

All in all, while this is a really sticky situation, now that it has happened, we can use it as a fantastic learning opportunity. This attack reminds us that:

• Just because you don’t see anything going on does not mean your systems aren’t compromised — this is probably the scariest thought of all, at least from my perspective
• Using outside libraries, frameworks, and tools always introduces new potential attack vectors
• Clearly, the previous point also applies to closed-source products, not just open-source
• Well-crafted malware can evade good detection, but there are still tell-tale signs
• I’ve often heard people say “our systems are completely isolated from the open internet, so we don’t need to focus security as much there…someone would need to get through multiple layers to reach those systems” … sure, unless they just hijack a package you’re using (as done here), in which case they can then easily laterally move across your infrastructure since you haven’t locked things down 🙂

I’m glad FireEye is taking this seriously and appears to be transparent about things. We’ll see how SolarWind handles it and what additional details emerge.