Today we will discuss something that’s pure excitement: finding hidden directories in Amazon’s famous S3 buckets.
But, before we dive deep into this subject, an important note: we do everything here in an educational manner, okay? If you mess up and things go south, it’s not on me. Always ask for permission before scanning, brute-forcing, or exploring someone else’s systems. Got it? 😉
Amazon S3
Amazon S3 is a highly scalable and durable object storage service offered by Amazon Web Services (AWS). It’s designed to store, manage, and retrieve large amounts of data reliably and securely in the cloud. S3 is widely used by companies of all sizes to store a variety of data types, such as files, images, videos, and backups, making it a popular choice for cloud storage solutions. Its scalability, high availability, and security make it an essential tool for various applications, from hosting websites to data analysis and critical information backup.
S3 Security Settings and Risks
Amazon S3 is an incredibly versatile platform, serving a wide range of needs, from storing backups and website data to running Big Data analytics and file management. However, when dealing with such a vast and diverse volume of data in one place, legitimate security concerns arise.
So, what are these security concerns, and what security settings can an administrator implement to properly protect a “bucket” and its files in Amazon S3?
First and foremost, it’s crucial to understand that Amazon S3 is considered a “publicly accessible platform.” This means that, with the appropriate URL and permissions, any “bucket” can be accessed from anywhere through HTTP requests, similar to what a web browser does to access a common website.
The accessibility of a “bucket” via its URL depends on the enabled security measures. However, it’s important to highlight that Amazon S3 is not a hidden resource available only after passing through multiple layers of authentication. On the contrary, it’s a resource that can be accessed through AWS endpoints from anywhere on the web, representing the primary security risk. Any S3 “bucket” and its associated data have the potential to be accessible.
Regarding Amazon S3 security, the checks implemented by the service to authorize access to an S3 resource, whether a “bucket” or an object, are robust. S3 evaluates permissions at the user level, through “bucket” policies, “bucket” access control lists (ACLs), and object ACLs. These security layers are essential to ensure that only authorized users can access and manipulate the data stored in S3.
How to Find Open Amazon S3 Buckets
In recent years, we have witnessed alarming data leaks in large companies due to the exposure of “buckets” on Amazon S3. If you want to check a list of the most notorious leaks, you can find it here. Through these security flaws, a wide range of confidential information was inadvertently made accessible to the public. We’re talking about Social Security numbers, personal photos, sales records, usernames and passwords, medical histories, and credit reports, among other things.
S3 Bucket Search Tools
If you’re interested in exploring publicly available “buckets” in Amazon S3, you need to use a tool that performs tests to check the existence of these “buckets.” When a “bucket” name does not exist, the service returns an error code called “NoSuchBucket.” Fortunately, there are several tools available for this purpose.
Initially, some of the early S3 “bucket” search tools included options like Bucket Lazy S3, bucket_finder, AWS Cred Scanner, sandcastle, Mass3, Dumpster Diver, S3 Bucket Finder, gobuster, and S3Scanner. These tools played a significant role in checking keywords in the names of publicly accessible “buckets,” although they had some limitations. The results often included irrelevant “buckets” and displayed only the first thousand files from their content.
What is Gobuster
Developed in the Go language, Gobuster is a high-performance scanner that proves to be a valuable tool for finding hidden directories, URLs, subdomains, and S3 buckets.
The question that often arises is, “What about Ffuf?” Although Ffuf is a notable web fuzzer, Gobuster emerges as an even faster and more versatile alternative. Additionally, Gobuster offers support for extensions that expand its capabilities, which is a differentiator. The program can also scale its operations using multiple threads and perform parallel scans to speed up results.
Now, as for installing Gobuster, the process is quite simple. If you’re using Kali Linux or Parrot OS, Gobuster is pre-installed for you.
On the other hand, if you’re using Ubuntu or a Debian-based system, you can easily install Gobuster using apt:
Linux
apt install gobuster
MacOs
brew install gobuster
After the installation is complete, you’ll need a wordlist. This list can include passwords, usernames, subdomains, and so on. There are various sources where you can obtain useful word lists.
My recommendation is to use Seclists. Seclists is a comprehensive repository of various lists used in security assessments. This includes collections of usernames, passwords, URLs, and more. If you’re using Kali Linux, you can find Seclists in the /usr/share/wordlists directory.
To try Gobuster in real-time, you have the option to use your own website as a target or, conveniently, employ a web application called Damn Vulnerable Web Application (DVWA). DVWA is a deliberately misconfigured web application created to be intentionally vulnerable. It’s a tool often used by penetration testing professionals to practice and improve their web application attack skills.
DNS Module
DNS, which stands for Domain Name System, is a cornerstone of the Internet that translates user-friendly domain names into IP addresses used by computers to identify resources on the network. It acts as a kind of “phone book” for the Internet, allowing browsers and applications to locate servers and websites corresponding to the domain names entered by users. DNS plays a crucial role in facilitating web navigation and online communication.
The DNS mode is used to discover hidden subdomains within a target domain. For example, if you have a domain called mydomain.com, you can use Gobuster to find subdomains like admin.mydomain.com, support.mydomain.com, and so on.
Here’s the Gobuster command for DNS enumeration:
┌──(higordiego㉿host)-[~]
└─$ gobuster dns -h
Uses DNS subdomain enumeration mode
Usage:
gobuster dns [flags]
Flags:
-d, --domain string The target domain
-h, --help help for dns
--no-fqdn Do not automatically add a trailing dot to the domain, so the resolver uses the DNS search domain
-r, --resolver string Use custom DNS server (format server.com or server.com:port)
-c, --show-cname Show CNAME records (cannot be used with '-i' option)
-i, --show-ips Show IP addresses
--timeout duration DNS resolver timeout (default 1s)
--wildcard Force continued operation when wildcard found
Global Flags:
--debug Enable debug output
--delay duration Time each thread waits between requests (e.g. 1500ms)
--no-color Disable color output
--no-error Don't display errors
-z, --no-progress Don't display progress
-o, --output string Output file to write results to (defaults to stdout)
-p, --pattern string File containing replacement patterns
-q, --quiet Don't print the banner and other noise
-t, --threads int Number of concurrent threads (default 10)
-v, --verbose Verbose output (errors)
-w, --wordlist string Path to the wordlist. Set to - to use STDIN.
--wordlist-offset int Resume from a given position in the wordlist (defau
We will perform a DNS enumeration. You can execute it with the following command:
┌──(higordiego㉿host)-[~]
└─$ gobuster dns -d test.io -w /usr/share/wordlists/dirb/common.txt
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Domain: test.io
[+] Threads: 10
[+] Timeout: 1s
[+] Wordlist: /usr/share/wordlists/dirb/common.txt
===============================================================
Starting gobuster in DNS enumeration mode
===============================================================
Found: api.test.io
Progress: 4614 / 4615 (99.98%)
===============================================================
Finished
===============================================================
AWS S3 Module
Amazon S3 is widely adopted by a large number of companies for distributing content, ranging from public material like website images to private and confidential files.
Now, let’s explore the help command to identify the various parameters offered by Gobuster in its S3 mode:
┌──(higordiego㉿host)-[~]
└─$ gobuster s3 -h
Uses aws bucket enumeration mode
Usage:
gobuster s3 [flags]
Flags:
--client-cert-p12 string a p12 file to use for options TLS client certificates
--client-cert-p12-password string the password to the p12 file
--client-cert-pem string public key in PEM format for optional TLS client certificates
--client-cert-pem-key string private key in PEM format for optional TLS client certificates (this key needs to have no password)
-h, --help help for s3
-m, --maxfiles int max files to list when listing buckets (only shown in verbose mode) (default 5)
-k, --no-tls-validation Skip TLS certificate verification
--proxy string Proxy to use for requests [http(s)://host:port] or [socks5://host:port]
--random-agent Use a random User-Agent string
--retry Should retry on request timeout
--retry-attempts int Times to retry on request timeout (default 3)
--timeout duration HTTP Timeout (default 10s)
-a, --useragent string Set the User-Agent string (default "gobuster/3.6")
Global Flags:
--debug Enable debug output
--delay duration Time each thread waits between requests (e.g. 1500ms)
--no-color Disable color output
--no-error Don't display errors
-z, --no-progress Don't display progress
-o, --output string Output file to write results to (defaults to stdout)
-p, --pattern string File containing replacement patterns
-q, --quiet Don't print the banner and other noise
-t, --threads int Number of concurrent threads (default 10)
-v, --verbose Verbose output (errors)
-w, --wordlist string Path to the wordlist. Set to - to use STDIN.
--wordlist-offset int Resume from a given position in the wordlist (defaults to 0)
We will create a file named ‘bucket_list.txt’ and add a list of names that can be used as bucket names in Amazon S3. After creating the file, we will proceed with the following command:
┌──(higordiego㉿host)-[~]
└─$ gobuster s3 -w bucket_list.txt
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Threads: 10
[+] Wordlist: bucket_list.txt
[+] User Agent: gobuster/3.6
[+] Timeout: 10s
[+] Maximum files to list: 5
===============================================================
Starting gobuster in S3 bucket enumeration mode
===============================================================
[ERROR] Get "https://dev.app.test.io.com.br.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not dev.app.test.io.com.br.s3.amazonaws.com
[ERROR] Get "https://beta.test.io.com.br.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not beta.test.io.com.br.s3.amazonaws.com
[ERROR] Get "https://echo.test.io.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not echo.test.io.s3.amazonaws.com
[ERROR] Get "https://test.io.com.br.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not test.io.com.br.s3.amazonaws.com
[ERROR] Get "https://test.com.br.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not test.com.br.s3.amazonaws.com
[ERROR] Get "https://app.test.io.com.br.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not app.test.io.com.br.s3.amazonaws.com
http://prod_test.s3.amazonaws.com/
http://test-images.s3.amazonaws.com/
Progress: 8 / 9 (88.89%)
===============================================================
Finished
===============================================================
Directory Module
The directory scanning function in Gobuster plays a crucial role in searching for hidden files and URL paths. This encompasses a wide variety of resources, ranging from images and script files to virtually any type of file available on the Internet.
Here’s the command to use the dir module on the previously found DNS:
┌──(higordiego㉿host)-[~/Documentos/artigo-s3]
└─$ gobuster dir -w files_list.txt -u http://test-images.s3.amazonaws.com/
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url: http://test-images.s3.amazonaws.com/
[+] Method: GET
[+] Threads: 10
[+] Wordlist: files_list.txt
[+] Negative Status codes: 404
[+] Exclude Length: 243
[+] User Agent: gobuster/3.6
[+] Timeout: 10s
===============================================================
Starting gobuster in directory enumeration mode
===============================================================
/image/b26086426c7e483b2924e775485b3215.jpg (Status: 200) [Size: 629968]
/image/contrato.jpg (Status: 200) [Size: 2830252]
===============================================================
Finished
===============================================================
How to Prevent Gobuster Enumeration Attacks
If you are concerned about Amazon S3 security and want to prevent bucket enumeration, here are some measures you can take:
Access Control: Use strict access control policies and permissions in Amazon S3 to ensure that only authorized individuals can list, access, or modify resources. Monitor Activities: Implement activity monitoring in your Amazon S3 to detect unauthorized access attempts or bucket enumeration. Restrict Access by Referrer: Use conditional referencing to restrict bucket access based on the “Referer” header of the HTTP request. Block Directory Access: Configure directories within the bucket not to allow listing, making it harder to enumerate subdirectories. Protect Your Credentials: Keep your credentials secure and avoid sharing them improperly. Regular Audits: Conduct regular audits of your Amazon S3 configuration to ensure that your security measures are up to date and effective. Use Bot Protection Solutions: Bot protection services like Cloudflare will prevent any brute force attacks, making it incredibly difficult to attack your web application.
Conclusion
Remember that Amazon S3 security is crucial because unauthorized access to your buckets can lead to data leaks and other vulnerabilities.
Be sure to follow AWS’s best security practices to properly protect your resources in S3.
Referencies
https://medium.com/quiknapp/fuzz-faster-with-ffuf-c18c031fc480
https://medium.com/@janijay007/s3-bucket-misconfiguration-from-basics-to-pawn-6893776d1007
https://takahiro-oda.medium.com/project-perform-s3-bucket-enumeration-using-various-s3-bucket-enumeration-tools-f4f63923b28
https://medium.com/stealthsecurity/finding-hidden-directories-sub-domains-and-s3-buckets-using-gobuster-5c2e3c41ff96
Top comments (0)