By now most should be familiar with computer infections and understanding what that means, despite the technical details being foreign as to how the infection works. What I mean is if you see the familiar “Your files have been encrypted” or have been getting undesired notifications claiming you’ve been infected, then we should be on the same page. But not all malware blatantly tells users they’ve been infected, some manipulate your computer in ways you may or may not recognize.
What if we set aside elaborate graphical messages and assume malware authors did some degree of masking malicious infections, what would you look for? What are some common signs of infection?
This post will cover just that – and based on a real-world example that I’ve been informed about personally!
Fair warning, this is a real-life scenario I am writing about. Upon learning of this breach, I felt this would be a great example to use to help raise awareness of detecting infections. That said, I am not going to go into too much detail for privacy reasons, but I want to share the steps taken throughout this entire process.
How the Infection Was Identified
Pertaining to this post’s specific example, this infection was fairly easy to detect once events were triggered and an analysis was performed. In hindsight, the main giveaway was the network bandwidth being saturated and another was a spike in CPU usage. The process of identifying that malware was the culprit wasn’t simply straightforward and there are more ways to go about this than what is detailed below.
The situation started when network congestion was detected and reported. All servers on the network were experiencing latency and packets were being lost. Rebooting the switch seemed most appropriate; sometimes a reboot fixes many things as is!
However, the reboot didn’t fix the issue. Swapping the switch out for another in stock was the next step; same thing, even with a third switch. Were all switches broken? Can’t be, so let’s take a closer look at the network nodes and see what’s going on.
In the case of the servers, fortunately they offered integrated monitoring tools. Most commonly, users have an overview of CPU, Memory, and Network usage statistics. If these tools are not available, using external tools or changing solutions is recommended. High-level monitoring is crucial and helps save time when investigating such matters.
After consulting the management interface of the servers, it became clear where the issue was coming from. A single host’s CPU and network bandwidth were maxed out! With the CPU and network running in overtime, the amount of traffic flowing through the switch was somewhat sustainable, as each switchport offers full gigabit traffic. However, the single uplink to the core router was being inundated with traffic from this problem host and essentially blocking other connections. Alas, the culprit was identified and then focus was shifted to just that server.
To bring closure to this scenario, the infected server was shut down and destroyed. This may be drastic, depending on what all was running on that server, but it’s better to be safe than sorry and have a dormant malicious script laying around. In the case of this server, it was actually just a test server anyway, so no harm in completely doing away with it.
Intrusion prevention is nice, that’s a given. It’s better to keep potentially unwanted programs (PUPs) out and block them at the gateway. However, if we stick to just this mentality, what does one do when they’ve been infected? Intrusion detection is equally as important; both play crucial roles in overall network security.
Ensuring adequate logging is a great starting point. Many devices allow for enabling audit logs; examples are traffic-dependent and authentication logging. Both of these offer tremendous insight. There are other auditing options available as well.
For instance, let’s look at some benefits of logging when traffic is sent from a device on one network to a different device on another network. This could be useful if a certain timeframe is allotted for backups or some other network activity. On WatchGuard firewalls, a network administrator is able to configure such a specific policy, let’s say from 1800-2000 (6pm – 8pm). If requests are sent before or after that, this could be a trigger and a starting point of further investigations.
Authentication logging, as the name entails, logs whenever there’s an authentication request to a server. This is actually how the source of the above infection scenario came to light. The source IP was logged along with the attempted login credentials.
There are many other logging options that I highly recommend are enabled. Here’s a resource to help with Linux CentOS 7 auditing. I’d recommend an online query similar to, “how to configure auditing for <insert OS name>”.
Other takeaways include starting off the investigation with a clear mind. Of course, panicking is expected but staying calm and clear-minded is more important than reacting to something. Next, start with what you have. As in the case of this story’s scenario, network congestion was the reported issue. After digging, it came to light that the issue was a server with outlandish resource usage. Looking closer into that server revealed that the server was indeed compromised with malware.
The content in this post can be used as a guide on how to investigate a potential infection. As with anything else, there is no “one size fits all” solution and this needs to be adapted to your specific use case. One thing to note is that automatically jumping to a hardware issue can lead to wasted time in troubleshooting in this matter. This wasted time could have led to other network nodes being compromised. However, the process of elimination is a sure way to check things off the list.
As for how the example server was infected. Well, good old-fashioned poor password practices were present. Threat actors are continuously scanning Internet-facing servers. Many employ brute force attacks in attempts of gaining access. Once in, this compromised server may allow for lateral movements. It’s important to have a strong password for Internet-accessible servers, otherwise you may face a similar situation.