If you follow security news, then you’ve surely heard about the recent drama between “Anonymous” and the HBGary security firm (more on who Anonymous is below), which took place over the past few weeks. While I’ve personally followed this fiasco with great interest, I’ve avoided commenting about it here, since most of our customers and readers are network administrators who are more concerned with practical business solutions than melodramatic cyber-quarrels. However, recently I read a fantastic article sharing the technical details of the HBGary breach, which I believe is a must-read for any computer security practitioner.
I’ll come back to that article in a minute, but first let me summarize the Anonymous/HBGary saga for those that may not have heard about it (if you have heard about it, feel free to skip to “Learning from Others’ Mistakes”).
I assume most of you are aware of the Wikileaks Cablegate story, since it’s made worldwide news. You know, that incident where Wikileaks — a non-profit organization that shares private or classified information with the press — publicly released some embarrassing U.S. diplomatic cables and royally peeved off the U.S. government. I don’t really want to recap the whole Wikileaks incident. I only bring it up to remind you that some camps oppose Wikileaks’ mission of “outing” sensitive information, while others camps fully endorse it.
Enter the mysterious Internet entity called “Anonymous.”
Who are Anonymous?
If you follow technology news, you’ve probably heard the “Anonymous” name in headlines before. They’re a group attributed for a wide-range of Internet incidents; from attacking Scientology to YouTube porn day (and many in between). However, in my talks with peers, I’ve found that many IT folks don’t really know who or what “Anonymous” is. Some may imagine “Anonymous” as a specific group of attackers, but that’s not really the case. In a nutshell, Anonymous is a random group of users tied strongly with popular image forums, like 4chan.
Occasionally, this group of random anonymous users decides to take on some fight, and loosely organizes what is essentially a virtual flash mob. For example, someone might post that they dislike Scientology, and ask other users to start figuring out ways to mess with Scientology, online. From there, chaos ensues. This means, the “Anonymous” group is not a specific group of people; rather it is a random group of users that happens to rally behind one cause or another; in other words, “hacktivists.”
Over the last few weeks, Wikileaks was one of those causes. I won’t pretend to speak for Anonymous, but I think it’s pretty safe to say that most 4chan users are pro-Wikileaks. Once the U.S. government started going after Wikileaks and its founder, an “Anonymous” group formed to start “fighting back” in Operation Avenge Assange. Using some very basic attack tools (Low Orbit Ion Canon), Anonymous began launching fairly successful Distributed Denial of Service (DDoS) attacks against various high-profile targets like Visa and MasterCard.
What’s this have to do with HBGary?
That’s the background story, but you might now be wondering where HBGary comes into the situation. HBGary is a security company that provides various security services to customers. It also has an offshoot company called HBGary Federal, which provides those services to the government.
Early on in the Wikileaks battle, HBGary threw its gauntlet into the fight, going after Wikileaks donors. Furthermore, the COO of HBGary Federal, Aaron Barr, thought it would be interesting to infiltrate the Anonymous group and try to find who its leaders were (assuming it has any). He seems to have attempted this by lurking on forums and IRC. Eventually, Barr started sharing his findings with the press, and intended to present them at a security conference. This, of course, set Anonymous off. They had a new target and cause… take out HBGary.
And that they did! Not to mince words, but Anonymous pretty much decimated HBGary’s defenses one brick at a time. By leveraging some very basic security issues in a number of systems, Anonymous was able to deface HBGary’s web site, delete 1 TB of backups, and steal tens of thousands of critical and sensitive emails (including some very embarrassing ones). I’ll get into more detail on how Anonymous did this below, but suffice to say a company, especially one focused on security, couldn’t suffer a more embarrassing public breach. In fact, HBGary was so affected by this attack that they even pulled out of the RSA conference.
So now you’re up-to-date with HBGary internet soap-opera, but why should you care? Well, this incident leads to a fairly obvious question. How did a respected security firm get hacked so quickly and easily?
Learning from Others’ Mistakes
That’s brings us full circle, to the article I mentioned at the beginning of this post. Last week, Ars Technica published an article detailing exactly how Anonymous broke into HBGary’s network (which they learned by talking to those who participated in the attack). This real world incident is a perfect example of how seemingly small chinks in different parts of your defenses can add up to gaping holes that totally compromise your system. Furthermore, it illustrates how not following some of the most basic best practices could land you in a heap of trouble. If you haven’t read the article, I highly recommend you go do so now. I’ll wait…
Ok, you’re back?
As you read, HBGary surprisingly fell victim to some of the most basic security mistakes one could make. To accomplish all of the mayhem I mentioned earlier, Anonymous’ attack included the following components:
- A SQL injection on a badly coded custom CMS
- A cracking attack (using rainbow tables) on badly encrypted passwords
- The discovery of some embarrassingly weak passwords used by high value targets
- The discovery of rampant password reuse (again, by high value targets)
- An elevation of privilege attack due to an very unpatched system
- …and some basic social engineering
None of those attacks are new, nor particularly extraordinary or complex. In fact, some are as old as hacking itself. All security professionals know basic security best practices to safeguard against them. That’s why this incident should wave a big red flag in the security community. How could such a well-respected security firm, who knows the right things to do, fall victim to such basic attacks? In his article, Peter Bright offers a potential answer to that question. He suggests that, “the standard advice isn’t good enough.”
I don’t think Bright means that the industry’s best practices are wrong; especially considering he also says standard advice would have protected HBGary. Rather, I believe Bright means that if our standard advice is too hard or time consuming for normal people to follow, they will ignore it. I agree with this sentiment. Few will follow technically sound best practices if they are impractical.
Let’s take the whole password reuse issue. Every security practitioner knows you should not reuse the same password at multiple sites. If you do reuse your password and an attacker gains access to it via one insecure site, then the attacker has the keys to your entire kingdom. Obviously, you should use different passwords everywhere, which is the industry best practice. However, following this best practice isn’t easy. At the very least, it takes extra time and thought. Most normal users don’t know about the password vault or keychain software that might help them manage multiple passwords, Even when they do, users don’t always use them because they adds extra steps, or roadblocks to their daily processes. As a result, many people reuse passwords.
This is the crux of the problem; the industry’s technically correct advice may not be “good enough” if normal people find it impractical. Security experts, in their white towers, often forget that security is not the core mission of most businesses. Many administrators consider security a necessary chore; something they have to do, but don’t really want to spend time on. The average user cares even less. No one like roadblocks that make doing their job harder, and users often see security controls as roadblocks.
Unfortunately, there is no easy answer to this dilemma. In order to secure things, you have to put access controls in front of them. However, I see Bright’s comment as a call to arms for the security industry. The best security mechanism in the world won’t do a thing if your users turn it off. So we need to design our security controls with ease of use in mind, which is something WatchGuard is focused on. We need to protect networks, while still facilitating business.
My second takeaway seems obvious in its simplicity, yet many people don’t really do it. That is, “Do what you know.” Over the last few years, my cohorts and I have ended many of our security presentations sharing a statistic we learned from a study done by the Verizon RISK team. Over the years, the RISK team has researched real world security breaches to study why the breach happened, and how it could have been prevented. They found that in almost 90% of the cases, the victim organization had the proper policies and technologies to have prevented the breach; they just didn’t follow their own policies, or configure their technology properly. This is what happened with HBGary. They obviously know how to prevent the simple attacks that succeeded against them; they just didn’t.
I’m not pointing fingers at HBGary. As the Verizon RISK team found, it seems like most organizations don’t follow through with best security practice. However, if we want to avoid security incidents, this is something we need to improve. When I was a kid, I remember fondly watching the G.I. Joe cartoon that always ended by saying, “and knowing is half the battle.” We need to remember that doing what you know is the other, arguably more important, half of that battle.
Learn from HBGary, and do what you know.
Telling companies to “do what you know” is basically good advice. That said, I would argue that people have to keep their minds open, and try as hard as they can to avoid tunnel vision. There is an old saying that goes: “When all you have is a hammer, everything looks like a nail.”
The hammer in HBGary’s case, is malware detection software. If you read through the emails, you will find at least one where HBGary founder Greg Hoglund tells how he would secure the company’s email communications. He goes on to describe how employees would be forced to use locked-down virtual machines to access the corporate VPN, and that people would be unhappy because these would allow no java, flash, etc. It was clear from reading his email that he was literally obsessed with the possibility of malware entering his company’s computer network.
I’m not denying that malware is a threat, but their fundamental mistake was to focus almost exclusively on this one vector, and to ignore the others. One of the reasons this became more of a factor for HBGary was that they operated 3 separate offices — their head office in California, another in Maryland, and a third in Colorado. Given the geographic distance between them, it is only natural that they used email for the bulk of their communications.
One potential solution that did not even appear to enter Mr. Hoglund’s mind was to public-key encrypt all of the company’s email traffic. According to media reports, HBGary consisted of approximately 50 employees, so having each person use a copy of PGP (or its open source equivalent, Gnu Privacy Guard (GPG)) would not have been overly difficult. In fact, there are commercial, off-the-shelf solutions e.g. by PGP Corporation, or Astaro, (among others) that make appliances to automatically secure email traffic using public-key technology.
Suppose Anonymous had broken into the company’s email server, and they found that most, if not all, of the email was PGP-encrypted. If that had been the case, we wouldn’t be discussing this right now.
Similarly, if public-key crypto had been used, it would not have been possible for an intruder to successfully impersonate Mr. Hoglund and thereby convince the hapless rootkit.com administrator to change the password and disable the firewall.
I absolutely agree that using some sort of email encryption would have been great, especially considering (as you point out) how geographically diverse their offices are. However, I also think that PGP encryption is one of those best practices that is a pain in the butt to implement.
As I mentioned, the problem with a lot of the best practices aren’t that they don’t work, but that they are too much of a “roadblock” or inconvenience for the average user. You and I understand how to use PGP from practice. We get that you sign a message with your public key and encrypt with the recipient’s public key. We know about guarding our private key, and using fingerprints to verify a public key really belongs to who it should. But a normal user doesn’t know all this stuff, to them PGP is hard to use. That, by the way, is why I believe so few ppl use email encryption, even though it is a long overdue and important solution.
Anyway, all that said, HBGary don’t qualify as “most users,” so surely they do know how to use email encryption, and likely should have used it. That said, I still think it’s important to note that security advice needs to try and cater to the lowest-common denominator. In general, people will more likely implement advice that doesn’t put as many roadblocks into their processes. The sad truth is, I’m not sure that simple advice always exists. Sometimes, you simply need to suck it up and follow the hard advice. Nonethelesss, when I think about advice I give, I’m going to try to think about it under the lens of, “is this easy enough that ppl will actually do it.”
BTW, while I personally love PGP (and using gnupg for free), I do want to point out that much easier solutions exist now that will give email the encryption it deserves, without being too hard for most users. For instance, WatchGuard’s XCS appliance supports built in email encryption, using a web-based process. Essentially, all the PKI infrastructure that’s need to support good email encryption is abstracted for the user by our web service. All they have to do is register for an account, and then the service takes care of managing, and properly applying the keys and encryption. Makes it a lot easier for the normal users in your organization to use.
Anyway, thanks you for writing in. I totally agree with your points, and really look forward to other readers in the audience sharing their expertise.
Cheers,
Corey
Very interesting post, Corey.
Thank you for the excellent advice and the thinking-throught-it processes you have described. Much appreciated.