DEV Community: Shankar Somasundaram

AlertInsightHub: AI-Powered Cloud Alert Triage & Visualization Platform Using Postmark

Shankar Somasundaram — Sun, 25 May 2025 04:19:34 +0000

This is a submission for the Postmark Challenge: Inbox Innovators.

AlertInsightHub: AI-driven cloud alert triage and visualization for faster incident resolution in hybrid environments.

💡 What I Built

AlertInsightHub is an AI-powered cloud alert triage and visualization platform built for SREs and cloud engineers managing hybrid, multi-cloud, and on-premise infrastructures. It streamlines the processing of infrastructure monitoring alerts, such as AWS SNS email notifications, by seamlessly integrating with Postmark’s inbound webhook system.

AlertInsightHub uses a smart AI agent to extract critical metadata—such as service name, resource, alert metric and resource name —from incoming alert emails. These alerts are transformed into structured, actionable insights, stored in a scalable backend datastore, and visualized through a real-time, interactive dashboard.

This enables engineering teams to triage cloud incidents faster by filtering and investigating alerts based on cloud account, service, instance, and metric type.

📘 Use Case

In hybrid cloud environments, operations teams face a deluge of alerts originating from various monitoring tools. These alerts—typically delivered via email—are unstructured and require manual processing, which is inefficient and error-prone. AlertInsightHub streamlines this process by transforming unstructured alert emails into structured, actionable insights and providing a unified dashboard for triage and resolution.

Cloud and on-premise monitoring tools generate high volumes of alerts, often delivered as plain-text emails. Manual triage of these alerts leads to:

Alert fatigue due to excessive, repetitive, and often low-priority notifications
Delayed response times to critical incidents
Inconsistent resolution paths for similar recurring issues
Lack of historical traceability and difficulty analyzing trends over time

These challenges are amplified in hybrid environments where alert sources span across cloud platforms, legacy systems, and third-party tools.

Target Users

DevOps Engineers **and **SREs in hybrid/multi-cloud environments
NOC teams managing high-volume monitoring and alerting systems
Incident response teams seeking structured, AI-assisted remediation workflows

Solution Summary

AlertInsightHub addresses these challenges by:

Automated Alert Ingestion & Processing
Intelligent Alert Insights
Unified Incident Response

Key Benefits

Reduced MTTR: Faster issue identification and resolution using structured alert data
Operational Efficiency: Frees teams from repetitive manual alert triage
Pattern Visibility: Enables insight into recurring incidents and noisy systems
Institutional Knowledge: Builds a repository of incidents and resolutions for future reference
Hybrid Flexibility: Supports cloud, on-premises, and hybrid environments seamlessly

🏗️ System Design & Architecture

The system is built with a layered architecture:

Data Collection Layer: Captures incoming Postmark webhooks and queues them for processing.
Processing Layer: Converts Postmark webhooks into standardized alerts and generates remediation suggestions using Groqcloud API.
Data Storage Layer: Uses DynamoDB to store raw data, alerts, configurations, and AI-driven recommendations.
Presentation Layer: Offers a web dashboard for real-time alert monitoring, queue tracking, and settings configuration.

🧪 Demo Video

Watch a walkthrough of AlertInsightHub showcasing how incoming AWS cloud SNS Topic alerts are automatically processed via Postmark, parsed using an AI agent, and visualized on an interactive dashboard.

🧪 Testing Instructions

Send a formatted alert email (e.g., SNS notification) to: support@cloudcraftcurator.tech (or Postmark Inbound Email Address)
Postmark routes the email to a webhook endpoint.
AI agent processes the content and stores parsed data.
View the alert on the dashboard.

Refer to the Readme for detailed instructions.

🧰 Code Repository

🔗 GitHub – AlertInsightHub

You can find the complete source code for the project here. Feel free to explore, contribute, or even fork the project to adapt it for your own needs. The repository is regularly updated and pull requests are always welcome!

⚙️ How I Built It

Postmark Inbound Webhook: Receives alerts from support@cloudcraftcurator.tech and triggers webhook delivery.
FastAPI App + AI Agent: Processes inbound JSON payloads using a lightweight AI agent hosted locally. The agent identifies key metadata from the alert—service, resource, metric type, etc.
DynamoDB (Local): Stores structured alert records efficiently with support for fast queries and grouping.
Interactive Dashboard (React/Streamlit): Renders alert summaries, supports drill-down views by account → service → instance → metric.
Devcontainer Support: Local development uses Docker and devcontainer.json to install dependencies (e.g., DynamoDB local, FastAPI app, etc.).
Self-hosted Webhook: Deployed using Docker and exposed securely using services like ngrok or custom domain.

📬 Postmark Features

In the AlertInsightHub project, I've leveraged several key Postmark features to create a robust webhook processing system:

1. 🔗 Webhook Integration

Inbound Webhook Endpoint Implemented a dedicated endpoint (/api/webhook) that receives and processes incoming Postmark webhook payloads.
Webhook URL Display Created a user-friendly display of the webhook URL in the dashboard with a copy button for easy integration with Postmark.

2. ⚙️ Event Processing

Event Queueing All incoming webhook events are immediately stored in a queue system (DynamoDB) for reliable processing.
Asynchronous Processing Implemented a non-blocking design where webhooks are received quickly and processed asynchronously to prevent timeouts.

3. 🗃️ Data Storage & Management

Raw Data Preservation All incoming webhook payloads are stored in their original form in the postmark_data table for audit and debugging purposes.
Structured Data Extracted and stored relevant metadata (timestamp, status, etc.) in the webhook_queue table for efficient querying.

4. 📊 Monitoring & Visualization

Status Tracking Implemented a comprehensive status tracking system (pending, processed, error) for all webhook events.
Dashboard Visualization Created charts showing webhook distribution by status and date.
Filtering Capabilities Added filtering by date and status to help analyze webhook data effectively.

5. 🚨 Error Handling & Resilience

Error Tracking Captured and stored error messages when webhook processing fails.
Reprocessing Capability Implemented a reprocessing feature for failed webhooks.
Graceful Degradation System continues to function even when some components fail.

🔄 Key Iterative Enhancements

Iteration 1: Integrated Postmark with Cloudflare, built alert hub with sample data.
Iteration 2: Integrated GroqCloud AI agent, updated AI roles and processing logic, created webhook_queue table for raw data storage and reprocessing.
Future: Future plans include supporting multiple AI agents to improve alert classification accuracy and provide enhanced remediation suggestions.

💡 Lessons Learned

Managing email routing with Postmark requires careful DNS configuration to avoid conflicts.
Running local development environments with DynamoDB local and FastAPI in Docker greatly improves reproducibility.
AI agents can significantly reduce manual alert triage but require continuous refinement to handle diverse alert formats.
Building an intuitive drill-down dashboard enhances operational visibility and speeds incident investigation.

✉️ Contact

For questions or suggestions, please reach out via GitHub Issues or open a discussion in the repository!. Thank you.

AWS EFS Analyzer: Optimizing Storage Costs with Amazon Q Developer

Shankar Somasundaram — Fri, 09 May 2025 05:57:09 +0000

This is a submission for the Amazon Q Developer "Quack The Code" Challenge: Crushing the Command Line

What I Built

I created the AWS EFS Analyzer, a command-line tool designed to help AWS users optimize their Elastic File System (EFS) storage costs.

The tool identifies cost-saving opportunities by analyzing file access patterns and recommending transitions to more cost-effective storage tiers—like moving infrequently accessed files from Standard to Infrequent Access or Archive. AWS estimates that customers can save up to 92% this way.

Manual identification is tedious and error-prone—EFS Analyzer automates the process, delivering actionable insights with ease.

Development Prompt

Python Script for EFS Storage Optimization Analysis

Create a Python script that analyzes an Amazon EFS (Elastic File System) mount point to identify cost optimization opportunities. The script should:

Scan an EFS mount point recursively, with support for parallel processing to handle large file systems efficiently

Categorize files based on last access time (7, 14, 30, 60, 90 days, 1 year, 2 years, and older)

Calculate total storage size for each access time category

Estimate storage costs across different EFS tiers (Standard, Infrequent Access, Archive)

Generate detailed reports in both HTML and plain text formats showing:

File access statistics by time category

Current storage costs (assuming all in Standard tier)

Potential optimized costs using appropriate tiers based on access patterns

Projected monthly savings

Recommendations for tier transitions

Key Requirements:
Handle large file systems efficiently using parallel processing

Provide real-time progress tracking with completion percentage and ETA

Automatically exclude system directories (/proc, /sys, /dev, etc.) to prevent infinite recursion

Detect and avoid symbolic link loops

Support command-line options for:

Specifying the EFS mount point

Setting parallel processing degree (default: number of CPU cores)

Excluding specific directories

Setting maximum scan depth

Controlling whether to follow symbolic links

Redirecting warnings and errors to a log file

Display a clean progress bar that shows:

Percentage completion

Number of directories processed

Number of files scanned

Estimated time remaining

Generate comprehensive reports that include:

File access statistics by category

Storage size distribution

Current vs. optimized cost analysis

Tier distribution recommendations

Potential monthly savings

Iterative Improvements with Amazon Q Developer

Throughout the development process of the AWS EFS Analyzer, I worked iteratively with Amazon Q Developer to refine and enhance the tool. Each step built upon the last, resulting in a more robust and user-friendly solution.

Key Iterative Enhancements

Here are some examples of follow-up prompts that guided the improvement process:

Handle System Directory Permissions
- Prompt: “Fix the permission errors when scanning system directories like /proc.”
- Result: Amazon Q quickly identified the issue and implemented error handling to avoid crashes when encountering protected system directories.
Add User Confirmation for Resource-Intensive Operations
- Prompt: “Add a confirmation prompt that warns about CPU usage and asks for user confirmation before proceeding.”
- Result: The tool now includes a warning prompt that ensures users are aware of potential performance impacts during scanning.
Improve Documentation
- Prompt: “Improve documentation with comprehensive docstrings for all functions and classes.”
- Result: Enhanced readability and maintainability through clear, detailed docstrings and usage explanations.
Introduce a Clear Banner Message
- Prompt: “Add a banner with clear information about the tool's purpose and usage instructions.”
- Result: Users now see a helpful banner on launch that explains what the tool does and how to use it effectively.
Warn About CPU Usage
- Prompt: “Add a warning about CPU usage during parallel scanning and recommend running during non-peak hours.”
- Result: A precautionary message was added to guide users on optimal usage timing and prevent performance bottlenecks.

🎥 Demo

Check out the demo video to see the AWS EFS Analyzer in action! The video walks through the key features of the tool, showing how it efficiently scans an EFS mount point, categorizes files based on access patterns, and generates insightful reports for cost optimization.

🔑 Features

✅ Real-time progress tracking during file system scan
📊 Interactive HTML reports with visualizations
🔁 Clear storage tier recommendations based on usage
💰 Estimated cost savings based on current storage patterns

📦 Code Repository

You can find the complete source code for the AWS EFS Analyzer on GitHub. The repository includes everything you need to get started, from installation instructions to detailed comments in the code to help you understand the logic behind each part of the tool.

Feel free to explore, contribute, or even fork the project to adapt it for your own needs. The repository is regularly updated, and pull requests are always welcome!

GitHub Repository: aws-efs-analyzer

🚀 How I Used Amazon Q Developer

Amazon Q Developer was instrumental in bringing this project to life, helping at every stage.

1. Rapid Prototyping & Architecture Design

Amazon Q helped define:

Parallel processing for faster performance
Reliable error handling for file system quirks
Modular code with separation of concerns

This eliminated hours of planning.

2. Solving Technical Challenges

When encountering permission errors in /proc, Amazon Q:

Diagnosed the cause (special filesystem behavior)
Suggested fixes with trade-offs
Implemented a safe, scalable solution
Added preventative checks elsewhere

3. Iterative Enhancements

Throughout development, Amazon Q helped:

Add a progress bar and confirmation prompts
Refine error handling
Improve documentation with detailed docstrings
Include security warnings (e.g., CPU usage alerts)

4. Following Best Practices

Amazon Q ensured:

Robust error handling
Modular and readable code
Clear, helpful CLI outputs
Thorough documentation
Security-aware design

📚 Educational Value

EFS Optimization Strategies

Storage tier comparison and cost benefits
How access patterns drive storage decisions
Benefits of lifecycle policies
Organizing data by usage frequency

Python Development for AWS Tools

Handling file system errors
Leveraging parallel processing
Tracking operation progress
Communicating through a friendly CLI
Writing maintainable, documented code

AWS Cost Management

Analyzing tiered storage costs
Identifying savings opportunities
Projecting cost reductions
Implementing best practices

🧩 Use Case & Impact

EFS Analyzer solves a real pain point:

💸 Reduce AWS bills—potentially thousands/month
⚙️ Automate manual analysis—save time and effort
📁 Encourage lifecycle best practices
📊 Support data-driven decisions

Beyond savings, it promotes smarter data organization and clearer insights into data usage.

🧠 Lessons Learned with Amazon Q Developer

Clear input = better output
Iterative development works best
AI can solve complex problems
Context retention leads to cohesive updates

Amazon Q has changed how I develop—faster, better, smarter.

✉️ Contact

For questions or suggestions, reach out via GitHub Issues or open a discussion!