DEV Community

Cover image for AWS EFS Analyzer: Optimizing Storage Costs with Amazon Q Developer
Shankar Somasundaram
Shankar Somasundaram Subscriber

Posted on

AWS EFS Analyzer: Optimizing Storage Costs with Amazon Q Developer

#devchallenge #awschallenge #ai #webdev

This is a submission for the Amazon Q Developer "Quack The Code" Challenge: Crushing the Command Line

What I Built

I created the AWS EFS Analyzer, a command-line tool designed to help AWS users optimize their Elastic File System (EFS) storage costs.

The tool identifies cost-saving opportunities by analyzing file access patterns and recommending transitions to more cost-effective storage tiers—like moving infrequently accessed files from Standard to Infrequent Access or Archive. AWS estimates that customers can save up to 92% this way.

Manual identification is tedious and error-prone—EFS Analyzer automates the process, delivering actionable insights with ease.

Development Prompt 

Python Script for EFS Storage Optimization Analysis

Create a Python script that analyzes an Amazon EFS (Elastic File System) mount point to identify cost optimization opportunities. The script should:

Scan an EFS mount point recursively, with support for parallel processing to handle large file systems efficiently

Categorize files based on last access time (7, 14, 30, 60, 90 days, 1 year, 2 years, and older)

Calculate total storage size for each access time category

Estimate storage costs across different EFS tiers (Standard, Infrequent Access, Archive)

Generate detailed reports in both HTML and plain text formats showing:

File access statistics by time category

Current storage costs (assuming all in Standard tier)

Potential optimized costs using appropriate tiers based on access patterns

Projected monthly savings

Recommendations for tier transitions

Key Requirements:
Handle large file systems efficiently using parallel processing

Provide real-time progress tracking with completion percentage and ETA

Automatically exclude system directories (/proc, /sys, /dev, etc.) to prevent infinite recursion

Detect and avoid symbolic link loops

Support command-line options for:

Specifying the EFS mount point

Setting parallel processing degree (default: number of CPU cores)

Excluding specific directories

Setting maximum scan depth

Controlling whether to follow symbolic links

Redirecting warnings and errors to a log file

Display a clean progress bar that shows:

Percentage completion

Number of directories processed

Number of files scanned

Estimated time remaining

Generate comprehensive reports that include:

File access statistics by category

Storage size distribution

Current vs. optimized cost analysis

Tier distribution recommendations

Potential monthly savings
Enter fullscreen mode Exit fullscreen mode

Iterative Improvements with Amazon Q Developer

Throughout the development process of the AWS EFS Analyzer, I worked iteratively with Amazon Q Developer to refine and enhance the tool. Each step built upon the last, resulting in a more robust and user-friendly solution.

Key Iterative Enhancements

Here are some examples of follow-up prompts that guided the improvement process:

  • Handle System Directory Permissions

    • Prompt: “Fix the permission errors when scanning system directories like /proc.”
    • Result: Amazon Q quickly identified the issue and implemented error handling to avoid crashes when encountering protected system directories.

  • Add User Confirmation for Resource-Intensive Operations

    • Prompt: “Add a confirmation prompt that warns about CPU usage and asks for user confirmation before proceeding.”
    • Result: The tool now includes a warning prompt that ensures users are aware of potential performance impacts during scanning.

  • Improve Documentation

    • Prompt: “Improve documentation with comprehensive docstrings for all functions and classes.”
    • Result: Enhanced readability and maintainability through clear, detailed docstrings and usage explanations.

  • Introduce a Clear Banner Message

    • Prompt: “Add a banner with clear information about the tool's purpose and usage instructions.”
    • Result: Users now see a helpful banner on launch that explains what the tool does and how to use it effectively.

  • Warn About CPU Usage

    • Prompt: “Add a warning about CPU usage during parallel scanning and recommend running during non-peak hours.”
    • Result: A precautionary message was added to guide users on optimal usage timing and prevent performance bottlenecks.

🎥 Demo

Check out the demo video to see the AWS EFS Analyzer in action! The video walks through the key features of the tool, showing how it efficiently scans an EFS mount point, categorizes files based on access patterns, and generates insightful reports for cost optimization.

Watch the Demo

🔑 Features

  • ✅ Real-time progress tracking during file system scan
  • 📊 Interactive HTML reports with visualizations
  • 🔁 Clear storage tier recommendations based on usage
  • 💰 Estimated cost savings based on current storage patterns

📦 Code Repository

You can find the complete source code for the AWS EFS Analyzer on GitHub. The repository includes everything you need to get started, from installation instructions to detailed comments in the code to help you understand the logic behind each part of the tool.

Feel free to explore, contribute, or even fork the project to adapt it for your own needs. The repository is regularly updated, and pull requests are always welcome!

GitHub Repository: aws-efs-analyzer

🚀 How I Used Amazon Q Developer

Amazon Q Developer was instrumental in bringing this project to life, helping at every stage.

1. Rapid Prototyping & Architecture Design

Amazon Q helped define:

  • Parallel processing for faster performance
  • Reliable error handling for file system quirks
  • Modular code with separation of concerns

This eliminated hours of planning.

2. Solving Technical Challenges

When encountering permission errors in /proc, Amazon Q:

  • Diagnosed the cause (special filesystem behavior)
  • Suggested fixes with trade-offs
  • Implemented a safe, scalable solution
  • Added preventative checks elsewhere

3. Iterative Enhancements

Throughout development, Amazon Q helped:

  • Add a progress bar and confirmation prompts
  • Refine error handling
  • Improve documentation with detailed docstrings
  • Include security warnings (e.g., CPU usage alerts)

4. Following Best Practices

Amazon Q ensured:

  • Robust error handling
  • Modular and readable code
  • Clear, helpful CLI outputs
  • Thorough documentation
  • Security-aware design

📚 Educational Value

EFS Optimization Strategies

  • Storage tier comparison and cost benefits
  • How access patterns drive storage decisions
  • Benefits of lifecycle policies
  • Organizing data by usage frequency

Python Development for AWS Tools

  • Handling file system errors
  • Leveraging parallel processing
  • Tracking operation progress
  • Communicating through a friendly CLI
  • Writing maintainable, documented code

AWS Cost Management

  • Analyzing tiered storage costs
  • Identifying savings opportunities
  • Projecting cost reductions
  • Implementing best practices

🧩 Use Case & Impact

EFS Analyzer solves a real pain point:

  • 💸 Reduce AWS bills—potentially thousands/month
  • ⚙️ Automate manual analysis—save time and effort
  • 📁 Encourage lifecycle best practices
  • 📊 Support data-driven decisions

Beyond savings, it promotes smarter data organization and clearer insights into data usage.

🧠 Lessons Learned with Amazon Q Developer

  • Clear input = better output
  • Iterative development works best
  • AI can solve complex problems
  • Context retention leads to cohesive updates

Amazon Q has changed how I develop—faster, better, smarter.

✉️ Contact

For questions or suggestions, reach out via GitHub Issues or open a discussion!

Top comments (0)