Many companies are suffering data breaches because attackers gain access to data in AWS S3 buckets. I don’t want to repeat all the news articles outlining all the S3 data breaches. A Google search will give many examples, and it seems like by the time I write this another one will be in the news. Instead, I’d like to jump to why these S3 bucket breaches are happening and how to securely store data in an S3 bucket.
What Is an S3 Bucket?
An S3 bucket is a storage location for data on the AWS cloud platform. Companies store files in buckets, though technically Amazon calls the items in buckets “objects.” AWS gurus might quibble over technical definitions, but most people can think of an S3 bucket like a virtual directory in the sky where you can put files and retrieve them later. It’s a convenient storage option that doesn’t require setting up servers or specifying exactly how much data you need to store in advance. Customers can just continue to add data and pay for what they use. Amazon allows customers to configure bucket access however they want. Some may store images for websites that anyone on the Internet can access. Other buckets may hold sensitive data the company intends to keep private.
Why Are Attackers Able to Steal Data in an S3 Bucket?
It’s easy to say, “the bucket configuration was wrong,” but based on my experience helping multiple companies move to the cloud, the problem is bigger than configuration alone. I learned an approach to get to the root of a problem called “The 5 Whys” in a project management course. Don’t accept the first answer. Ask four more times. Here is how I would expect this conversation to go in most companies with many developers and DevOps people.
Why are attackers able to access S3 buckets? The bucket configuration is not correct. The permissions on the bucket are too broad, allowing anyone to access the data.
But why weren’t the permissions correct? Well, the developer who created the bucket might not have known how to secure the bucket. However, there is a good chance the original bucket developer created the bucket correctly, but someone else changed it later. Maybe the QA team tested the application but did not review the S3 bucket configuration.
Why did someone change the bucket configuration? That person was troubleshooting a problem that caused the application to fail. The person wanted to fix the problem and suspected the S3 bucket access was to blame, so the person tweaked the S3 bucket configuration. The application started working again, and the person patted himself or herself on the back for making it work and moved on.
Why didn’t the person take the time to configure the bucket correctly? Chances are the person was under a lot of pressure or trying to make other people happy by solving the problem. The person may have also just changed the settings to fix a problem and forgot to put the settings back. The person might have been unaware of the consequences that change. The person may have known what could happen but thought the risk was low and just needed things to work. The person could just be lazy. In the worst case, the person or some malware did it intentionally to allow personal or third-party access to steal the data.
Why don’t companies prevent developers from making these types of changes? That is the fifth question. The answer to this question is the most important one that leads to a real solution for this S3 bucket problem: Companies don’t have processes that prevent insecure software deployments.
Companies need to invest in security controls that monitor and make it hard for developers or malicious software to deploy insecure configurations. |
Steps to Securing an S3 Bucket
Understanding how someone would get access to create an insecure bucket configuration and why they would do it can help create a process to prevent that problem in the future. Cloud security goes beyond understanding best practices for S3 bucket configurations. How would a company with over 10,000 developers, such as one where I worked on the cloud engineering and operations teams, train every single developer, and ensure none of them ever makes a mistake when configuring an S3 bucket? Could they just let a handful of security people make changes for everyone else? That would become a significant bottleneck when trying to deploy new applications. Here are some other options, which I explore in greater detail in a white paper I wrote about security automation on AWS.
- Create an automated deployment pipeline. How do bad things get into an environment? Someone installs or configures something. If the only way to do that is via your automated deployment pipeline using code from a source control system, you will be able to monitor every single piece of software changed in the environment. The source control system and deployment system need to log who performed every action and what they did. Detailed logging of deployment actions taken by individual users requires customization of tools such as Jenkins and Github, but it is possible (speaking from personal experience). At the point of deployment, build in security checks and block things that do not match security policies. For configuration of AWS services like S3 buckets, I explain how AWS CloudFormation can help improve security in a related Secplicity blog post.
- Implement Segregation of Duties. Split responsibilities between multiple teams so that one person might have a chance to catch another person’s mistake. Ensure a user does not have permission to change security controls and elevate personal access. Have a team that configures IAM and separate from those that use the IAM permissions. AWS Organizations, which allows creating policies across multiple accounts, may help. Separate people that manage encryption keys from those who own the data. A different ops team can provide access to fix production incidents and monitor actions taken by individuals fixing the problem.
- Use Bucket Policies, not ACLs. An S3 bucket has a few buttons in the UI someone can click to change part of a bucket settings called an ACL. First, prevent anyone from clicking buttons and deploy everything via software as noted in #1. Second, don’t use ACLs for configuration because the options are too limited. Leave the ACL in the default setting state. Create bucket policies which allow for very fine-grained permissions for the bucket.
- Create Application Roles. When accessing buckets from an application follow the IAM best practice of creating a role for EC2 instances, Lambda functions and other S3 services that need to access buckets.
- Use S3 Endpoints for Private Buckets. Traffic traverses the Internet to access to S3 buckets secured only with ACLs and bucket policies, even when an application in your VPC is accessing a bucket in your own account. An AWS S3 Endpoint gives a customer more control over what network traffic and AWS roles can reach an S3 bucket. An S3 Endpoint does not keep traffic internal to your VPC, but it stays within the Amazon network in a single region, preventing access from the Internet. The defaults are broad – change the configuration to limit access to and from only what is necessary. Be aware that any access of S3 data from another region would not stay within the Amazon network. I investigated and asked AWS support for more detailed information about where S3 traffic flows and how S3 endpoints work if you want to know more.
- Security professionals should manage S3 Endpoints and Network Configuration. Developers are usually not security experts. In fact, some network engineers are not security experts. I believe almost any developer can become a security professional with proper training – but learning to secure networks doesn’t entail learning how to implement rules in a route table or security group. It means understanding the cyber kill chain, APTs, and how proxies or shells are used to infiltrate a network, how data is exfiltrated using packet manipulations, and how network configuration could have stopped or mitigated recent cyber attacks. Limit permission to change networking and S3 endpoint configurations to people who understand how to design secure networks.
- Assign responsibility for IAM policies to people with AWS security expertise. IAM policies give access to everything in the AWS account. Ensure the people creating IAM policies are AWS security experts or taking direction from someone who is. Improper IAM policies will allow too much access via features of the different services. The person creating the policies needs to understand how each service works and what capabilities each line in the policy enables. For example, read-only access to a whole account lets someone read unencrypted sensitive data in S3 buckets and other data stores. If a policy includes ec2.* the person or virtual machine assigned that role can do a lot more than spin up ec2 instances. A user with iam.* privileges can elevate personal permissions and grant access to the account in other ways.
- Force MFA. Bucket policies allow you to force MFA when you need to give an individual access to a bucket.
- Force Encryption. Bucket policies allow you to enforce encryption. Note that this will not help if an attacker compromises an account via a role that has access to view encrypted files in the bucket.
- Monitor Buckets. S3 access logging offers details about bucket access requests. Amazon now allows tracking S3 API calls in CloudTrail which is a good choice if you want all the logging in one place. Configure permissions correctly to ensure someone cannot inadvertently or maliciously change the log configuration or delete the logs. Monitor deployment of S3 buckets with an automated deployment system. Amazon Macie can automate the classification of data in your S3 buckets. AWS Config can monitor for unexpected changes to bucket configurations.
Rather than stopping with the first answer to the question, “Why have so many S3 buckets been breached?” and blaming the configuration alone, consider how privileges, processes, and deployment systems impact the data stored in S3 buckets over time. This article presents a few steps companies can take to ensure the deployment process for creating an S3 bucket is secure, as well as making sure future modifications continue to protect the data. Watch for more specific information on how to secure S3 buckets with bucket policies, S3 endpoints, and IAM roles in future Secplicity blog posts. — Teri Radichel (@teriradichel)