MSPs Beware: Attackers Targeting MSP Infrastructure to Install Ransomware

Corey Nachreiner

5 years ago

In the past two weeks, sophisticated threat actors have targeted managed service providers (MSPs) and Cloud service providers (CSPs), intending to install ransomware within their infrastructure and customer base. Often, these attacks specifically target products and services MSPs use, such as ConnectWise/Kaseya software, the Webroot Management Console, RDP services and more. Though we’ve seen MSP attack campaigns begin much earlier this year, this malicious activity recently progressed with a new batch of attacks that have affected a large number of MSPs, large and small. Whilst these attacks have nothing to do with WatchGuard or our technology, we take the security of our partners very seriously. As such, we’re releasing this alert to fully inform you of these advanced attacks so you can take the actions necessary to protect your organization and your customers. We recommend you take these targeted attacks very seriously.

Latest MSP Attacks Deliver Sodinokibi Ransomware

Over the past few weeks, we’ve seen fresh reports of advanced threat actors compromising MSP organizations to install ransomware on all of their customer networks, and their computers as well. More importantly, we’ve heard directly from one of our partners who was affected by this attack, and have analyzed some script samples and malware from that incident. At a high-level, the MSP threat actors are leveraging a combination of weaknesses in MSP authentication practices, unauthorized access to MSP management tools, and general lack of certain security controls to install the Sodinokibi ransomware on as many computers as possible. Reports suggest the attackers can even exploit remote management and monitoring (RMM) platforms against MSP businesses to distribute the ransomware to customer endpoints directly.

We’ll cover these recent attacks in more detail shortly, but it’s important to know this isn’t the first campaign we’ve seen targeting MSPs this year. These attacks began in earnest February 2019, and the U.S. government even warned about them earlier. To better understand this class of MSP attacks you should be aware of all the earlier incarnations. With that in mind, let’s look at a quick timeline.

2019 MSP Attack Campaign Timeline

October 3^rd, 2018: NCCIC Warns of MSP Targeted Attacks – During Q4 2018, the National Cybersecurity and Communications Integration Center (NCCIC) published an advisory (TA18-276B) warning that advanced persistent threat (APT) actors were trying to infiltrate the network of global MSPs. Though the alert didn’t share any specific impact, other than general network compromise, it did theorize that the threat actors wanted to exploit the privileged trust relationship that MSPs have in their customers’ networks. It warned the attackers could exploit this IT supply-chain relationship to target companies that fall under U.S. critical infrastructure sectors, including IT, energy, healthcare, communications, and critical manufacturing. The advisory also included sparse details about the attackers’ tactics, techniques, and procedures (TTPs). For instance, it warned the attackers used legitimate credentials and trusted applications in the attacks, and that they leveraged malicious PowerShell and Visual Basic scripts, as well as system commands like netsh.On the same day, the NCCIC also publish a related advisory (TA2018-276A) on “rigorous credential control,” detailing some best practices around authentication security and privileged access management, as a way to mitigate these MSP-focused trusted network exploitations.
February 5^th, 2019: MSPs Compromised via ConnectWise Plug-n Vulnerability – Four months later, multiple reports surfaced about MSPs getting compromised through a specific remote management and monitoring (RMM) tool. These new reports included more technical details about the MSP attack and its impact. Specifically, the threat actors targeted an older SQL Injection vulnerability (CVE-2017-18362) in the ConnectWise ManagedITSync plug-in for the Kaseya VSA RMM. In short, the critical vulnerability allowed an unauthenticated remote attacker to execute arbitrary SQL commands on a Kaseya VSA database, which essentially allows them to do anything on the RMM, from creating administrative users to deploying software on any endpoints under management. The attackers leveraged this power to deploy GandCrab – a popular ransomware variant – throughout the MSP and customer networks. According to reports from Huntress Labs (@HuntressLabs), at least four MSPs had all their client endpoints encrypted by this attack. The first MSP affected had over 2,000 managed computers encrypted and held for $2.44 million in ransom. They ended up paying some of the ransom to recover their systems. At the time, there was a patch for the ConnectWise plug-ins, but Kaseya found at least 126 MSPs remained unpatched and vulnerable.
April 30th, 2019: New GandCrab-affiliated Ransomware Exploits Zero Day – A few months later, the researchers at Talos Intelligence Group posted details about a new ransomware variant called Sodinokibi. At the time, cyber attackers were exploiting a zero day vulnerability in Oracle’s WebLogic application server to infect victims with this new variant. Though Talos’s post doesn’t reference the MSP attacks at all, it does suggest the Sodinokibi variant might be affiliated with the Gandcrab ransomware and its creators. Other research organizations seem to concur. You’ll soon learn how Sodinokibi relates to the MSP attacks, and considering the connection between these two ransomware families, it’s possible the same threat actor was responsible for the WebLogic attacks as well.
June 20^th, 2019: MSP Attackers Install Sodinokibi via Legitimate MSP Tools – That brings us to today. Over the past few weeks, details have surfaced about recent MSP network compromises that leverage stolen privileged credentials and legitimate MSP tools. Specifically, attackers have hijacked at least three MSP networks and used legitimate RMM tools like the Webroot Management Console to forcefully deploy the Sodinokibi ransomware to the MSP customers’ endpoints, and within the MSP itself. As mentioned, we are aware of at least one WatchGuard partner who was affected by this recent campaign and have analyzed some of the malicious samples they shared with us. With that in mind, let’s explore some of the technical details and commonalities around these MSP Attacks.

Sodinokibi MSP Attack – Technical Details

At the time of writing, the community hasn’t identified a common root cause or initial entry vector for these MSP attacks. As with many targeted attacks, the sophisticated threat actors behind them often leverage different, customized tactics to compromise different victims. Some advanced tactics include spear-phishing emails, network service exploitation, and credential theft.

Though we don’t know the initial entry vector, these MSP attacks do share many technical details in common, including some that might suggest the entry vector. Furthermore, understanding the technical details can help you identify and protect yourself from the attack. Let’s explore these technical commonalities below.

The attackers target RDP. The Remote Desktop Protocol (RDP) is a Microsoft network standard that provides remote desktop management capability in Windows. Many MSPs commonly leverage RDP to manage their clients and their own desktops, and often expose the RDP services externally. Some reports suggest these threat actors gained initial access to MSPs through RDP services. We imagine two likely scenarios. One common tactic is RDP brute-forcing, where attackers automate many login attempts using common credentials, hoping one hits. The second involves exploiting a software vulnerability to gain control of an RDP server. For instance, attackers could exploit the recently patched RDP vulnerability called BlueKeep (CVE-2019-0708) to gain complete control of an MSP’s unpatched RDP servers. In any case, once an attacker owns an RDP server, it makes an ideal position to steal or acquire other privileged credentials.
The attackers obtain your privileged credentials. Whether via the RDP vector mentioned before, or some other initial attack, the first thing these attackers do is try to obtain as many privileged credentials as possible. Once they have access to legitimate MSP user accounts, they can use your tools as though they were those users.In WatchGuard’s quarterly Internet Security Report, a malware tool called Mimikatz has remained the top volume threat for over a year. Mimikatz is a tool attackers use specifically to steal credentials once they’ve compromised one of your systems. While it hasn’t been reported in these attacks, it offers one example of how attackers might obtain privileged credentials once they’ve hijacked an MSP system.
The attackers leverage common MSP tools to deploy Sodinokibi. These sophisticated attackers have studied MSPs and understand the tools they use. In fact, they even use those tools against you to deploy their malicious payloads. As previously mentioned, the earlier MSP attacks literally exploited a software vulnerability in a common ConnectWise/Kaseya plug-in used by many MSPs. That particular attack succeeded without any user interaction or stolen credentials.However, the more recent attacks are different. Once the attacker obtains one of your privileged MSP accounts, they can legitimately leverage your RMM, or other MSP tools, directly; without relying on a vulnerability. Most of these tools have features that allow you to deploy software or run commands on all the endpoints you manage. Researchers have specifically seen these threat actors leverage the ConnectWise Control and Kaseya VSA RMMs in this way. This isn’t due to any flaw in the RMM itself, but simply the attackers’ access to your privileged credentials.
Delivering Sodinokibi via Webroot’s management console. This detail is directly related to the point above, but we believe it deserves individual emphasis as a relatively new development. In the most recent MSP attacks, the attackers specifically leveraged the Webroot management console to deploy their malicious scripts and ransomware to managed endpoints. Like above, this isn’t due to any flaw in the Webroot system but simply the attacker’s access to privileged credentials. The attackers specifically leveraged this privileged access to launch malicious PowerShell commands on all an MSP’s managed endpoints at once, as seen in the screenshot below.
Figure 1: Logs from the Webroot Management Console, which was used to launch a malicious PowerShell command. Image credit to @chrisbisnett from @HuntressLabs
They disable your security controls. The attackers can also exploit their elevated access to your RMM and security management consoles to disable the security controls you so rigorously implemented. For instance, they might use the Webroot Management Console to disable some of the endpoint security features before installing ransomware. Reports suggest the attackers disable antivirus clients, and in some cases, they also disabled the Veeam endpoint backup agent to prevent data recovery. In short, if an attacker has one of your administrative management credentials, they can use your legitimate RMM solutions to remove the security you so carefully added.
Heavy use of PowerShell and the PowerSploit framework. Like past MSP attacks, the recent ones use PowerShell to download staged payloads, and likely to help their malware evade legacy security controls. The attacker specifically uses functionality from a well-known PowerShell penetration testing framework called PowerSploit, which is commonly used by cyber criminals. Specifically, they use a PowerSploit script that reflectively injects a DLL or EXE into another process, or the Powershell process itself. This effectively loads the executable in memory only, as fileless malware. As an aside, we’ve analyzed the samples we acquired from this attack and can confirm WatchGuard’s anti-malware services (like Gateway AntiVirus and APT Blocker) detect the malicious PowerShell script samples associated with this attack.
Installs the Sodinokibi ransomware on as many endpoints as possible. Simply put, the whole point of this attack is for the threat actors to load the Sodinokibi ransomware on as many of your, and your customers’ endpoints as possible, in hopes of extorting you for a large sum of money. The sample we acquired from one of these attacks looks like a pretty common variant of Sodinokibi, which all three of WatchGuard’s anti-malware services catch when sent over a network connection. However, know that the attackers’ use of PowerShell to stage their payload delivery could help the malware evade security controls.In the end, Sodinokibi is pretty typical ransomware. It strongly encrypts your files and then points you to a Dark Web (Tor onion) site to learn how to pay the ransom and recover your files. The ransomware does have some evasive capability that might help it elude some anti-malware sandboxes, but WatchGuard’s APT Blocker does catch those behaviors. If you’d like to know more about Sodinokibi, we recommend you read this post.

Now that you understand some of the common technical details about these attacks, it might help to take a more detailed look at the anatomy of an attack.

Anatomy of Sodinokibi MSP Attack

Gain access to an MSP’s privileged credentials. This is the only unclear step in the attack, as the community hasn’t yet identified any shared root cause for the initial infection vector. As in most targeted attacks, the adversary often uses many different techniques to compromise different victims depending on their individual weaknesses. That said, open RDP services seems to be a common weak point. It’s quite possible the attackers are brute-forcing credentials from an open RDP server, or leveraging a potential RDP flaw (like one targeting the BlueKeep vulnerability) to gain control of your RDP server and steal credentials using other tools. It’s also possible that attackers still use the old ConnectWise/Kaseya plug-in vulnerability to gain access, though we suspect most MSPs patched that in February. They may even use simple spear-phishing emails to trick your users out of their credentials. In any case, we know that the attacker gains enough access to an MSP’s infrastructure to obtain some privileged credentials.
Leverage MSP management tools. As mentioned above, the reason these attackers obtain your privileged credentials is to abuse them on your RMM system. Once they have credentials, the threat actors log into RMMs like ConnectWise Control, Kaseya VSA, or Webroot’s Management Console and directly manipulate your managed endpoints. They often start by disabling security features and software, like the aforementioned Webroot AV, ESET AV, and Veeam backup agents.

Launch a batch script or PowerShell from your RMM or management consoles. Once they have access to your RMM or a management console, they can directly deploy commands or software to all managed endpoints. Reports suggest they have used the Webroot Management console in some of these attacks. Using your MSP tools, they can directly launch their first PowerShell command either as a batch script (1488.bat) or a command (as seen in Figure 1 above). Here’s an example of the command they use:

In a nutshell, that command uses the Windows command line utility (cmd.exe) to start PowerShell in a hidden mode, with fewer local restrictions. The actual contents of the PowerShell are base64 encoded. Here is the decoded script:

That script checks whether the endpoint is a 64-bit system or not and sets environmental attributes accordingly, then it downloads the contents of a Pastebin page as a string, which it runs with PowerShell as the second stage of the attack. At the time of writing, the malicious script had been removed from Pastebin. However, we obtained a sample from a victim, and have shared a very small excerpt of the very large script below.

Our analysis of the full script found that this is a slightly altered version of a script from the PowerSploit collection called Invoke-ReflectivePEInjection. It’s designed to directly inject a DLL or EXE into a running process, or the PowerShell process itself. This effectively loads that executable in memory only, as a form of fileless malware. In this case, the script specifically contains the embedded portable executable (PE) file for the Sodinokibi malware, which encrypts the victim computer as expected. As an aside, our Gateway AntiVirus (GAV) service does recognize both the initial and secondary PowerShell scripts as malicious content.

To summarize, these targeted attackers leverage different tactics to gain privileged MSP account credentials. Using those credentials, the attacker can directly leverage your RMM or endpoint management systems to deliver malicious PowerShell script to all managed endpoints. The staged PowerShell script loads the Sodinokibi ransomware directly into memory as fileless malware. Leveraging your own tools, the attackers can infect all the computers you manage at once.

What Can MSPs Do to Protect Themselves?

First, there is no silver bullet defense, especially where sophisticated attackers are concerned. These advanced threat actors leverage many techniques to evade traditional defenses, so you need different layers of security to combat different attack methods. That said, there are some key defenses that really stand out in this case.

Use multi-factor authentication (MFA) throughout your enterprise. While the community has not established a single shared root cause for these MSP network compromises, we do know the attackers eventually gain access to your privileged credentials and leverage them to abuse your management tools. MFA is the only thing that really protects you against this sort of credential theft and abuse. Even if an attacker was able to learn one of your RMM admin passwords, MFA solutions could prevent those attackers from being able to log in with that password. We highly recommend you implement MFA throughout your organization, including your enterprise login, RDP sessions, VPN, internal management systems, and SaaS applications. Solutions like WatchGuard’s AuthPoint offer MFA for all these use cases, and we recommend you use it, or at least other MFA products like it. If, for whatever reason, you can’t yet implement enterprise-wide MFA, we at least recommend you setup MFA in all your critical applications that support it—at the very least your RMM solution. Products like Webroot Management Console and Kaseya VSA do support MFA, and you need to use it. MFA alone could have significantly mitigated this attack.
Patch public-facing software aggressively. The older MSP attack from February exploited an old and critical flaw a ConnectWise plug-in for Kaseya VSA. ConnectWise had fixed the flaw in 2017, yet some MSPs remained vulnerable last February. Learn from them by making sure you keep your critical software up to date, especially the powerful management tools you use to access your clients’ endpoints and network appliances. We actually don’t believe the ConnectWise plug-in flaw is associated with these newer attacks, nor do we think they are exploiting some new flaw in your RMM or endpoint management tools. Rather, they are simply accessing your tools with stolen credentials. Nonetheless you should still make sure to keep your MSP software patched just to be safe. We also suggest you check your Windows and RDP patch levels at your and your customers’ sites. Microsoft recently fixed a very critical flaw in RDP, which could be one of the attack vectors used in these incidents. Make sure you’ve patched BlueKeep.
Place stronger ACLs on remote management and use VPN. As an MSP, there are likely a number of network services that you have to expose publicly, both from your customer network and your own, in order to provide remote management services. For instance, you’ve likely exposed RDP from a number of sites so your techs can manage desktops. You’ve likely exposed your RMM management tools, so that reps can log in to them from wherever they happen to be at the time of need. You also probably have to open various network services to allow endpoint management solutions and other products to work. As you are allowing for these management capabilities, consider their security as well. Apply the principle of least privilege and try to limit access to these network services to as few IPs or users as possible. For instance, don’t just open RDP access to the world if you can instead limit access from a few IPs. Better yet, require VPN for all remote management services. WatchGuard Fireboxes allow you to make very granular, user-centric policies and offer multiple remote VPN solutions.
Use advanced anti-malware services on your network and endpoints. Even run-of-the-mill malware has become much more evasive and sophisticated lately. This attack in particular uses PowerShell to stage its malware delivery, which can sometimes bypass older network and endpoint controls. It uses a PowerSploit function to load the ransomware directly into memory, making it fileless and thus able to skirt file-centric protections. Even the ransomware executable itself has some malware sandbox evading capabilities. If you mostly rely on traditional signature-based anti-malware solutions to protect your company and clients, it will likely miss many aspects of this and other attacks.Nowadays, you need to implement different types of anti-malware on both your network and endpoints. We recommend you use more modern anti-malware solutions that leverage behavioral analysis and machine learning to detect new malware variants that signatures might miss. You should also implement some sort of endpoint detection and response solution that roots out malware that does make it onto one of your endpoints.If you are a WatchGuard customer, our Total Security services include four anti-malware services that provide very rich coverage. They include Gateway AntiVirus (GAV), IntelligentAV (IAV), APT Blocker, and Threat Detection and Response (TDR). Both our GAV and APT Blocker services detect the Sodinokibi ransomware, and the PowerShell scripts used in this attack, when they are passed over the network gateway. However, realize that attackers can use other delivery methods that might evade network detection. You should pair these services with endpoint protection as well, such as our TDR service or other endpoint protection products.
Backup your customers’ and your data regularly. While obvious, maintaining regular and rigorous on and offline backups of your and your customers’ data, can make it much easier to recover from these sorts of attacks. However, these sophisticated actors sometimes target your backups as well, and have been seen to remove the Veeam backup agent. We recommend you maintain a few sources of backup and keep offline copies as well.

As technically complex as some parts of this MSP attack are, at its core it’s essentially a credential theft. Authentication is the cornerstone of all security. If an attacker can masquerade as you, they can do anything you’re able to. The best way to secure authentication is MFA. You should deploy it internally and at your customer sites.