DEV Community: Alex Curtis

From Data to Dashboards: Building an EC2 Cost Analysis Tool with Flask and AWS S3

Alex Curtis — Tue, 01 Apr 2025 19:16:21 +0000

Disclaimer: This article was previously published in Medium.com, link to article: https://medium.com/@alex.curtis_luit/from-data-to-dashboards-building-an-ec2-cost-analysis-tool-with-flask-and-aws-s3-4c0d312ea38f

If you don’t have your own magic hat and need to analyze 100,000 EC2s and provide recommendations….follow along!

How to decode, decipher and recommend changes that will save money on your businesses 100K EC2 operation. No magic hat?! No problem….enter Python!

This article explores how to build a Flask-based web application that analyzes EC2 cost data, generates insightful visualizations, and leverages AWS S3 for storage and retrieval.

The Application’s Core Functionality:

Our application aims to provide a user-friendly interface for analyzing EC2 cost data. It performs the following key tasks:

1️⃣Data Ingestion: Reads EC2 cost data from a CSV file using Pandas.
2️⃣Data Analysis: Performs various cost-related analyses, such as calculating total costs, average costs per instance type, and potential savings.
3️⃣Visualization: Generates visualizations using Matplotlib and Seaborn to represent the analysis results.
4️⃣Storage: Uploads the analysis results and visualizations to AWS S3.
5️⃣Retrieval: Allows users to download the analysis results and view visualizations through a web interface.
Key Components and Code Snippets:

Data Analysis with Pandas:

We use Pandas to read and process the EC2 cost data. The analyze_ec2_costs() function performs the core analysis:

def analyze_ec2_costs(df):
    analysis = {}
    if df.empty:
        logging.error("DataFrame is empty for analysis. Analysis aborted.")
        return analysis
    df.columns = [str(col).strip().lower().replace(" ", "").replace("$", "").replace("(", "").replace(")", "").replace("%", "") for col in df.columns]
    # ... (rest of the analysis logic)
    return analysis

Visualization with Matplotlib and Seaborn:

Visualizations are generated using Matplotlib and Seaborn. The generate_visualizations() function creates plots for instance type distribution, cost per region, CPU utilization, and recommendation breakdown:

import matplotlib.pyplot as plt
import seaborn as sns
import os

and

def generate_visualizations(df):
    # ... (data preprocessing)
    plt.figure(figsize=(10, 6))
    sns.countplot(y="instancetype", data=df, order=df["instancetype"].value_counts().index[:10], palette="pastel", hue="instancetype", legend=False)
    plt.savefig("instance_type_distribution.png")
    # ... (other visualizations)
    plt.close()

AWS S3 Integration:

We use the boto3 library to interact with AWS S3. The upload_to_s3() function uploads files to our S3 bucket: import boto3 import logging

S3_BUCKET_NAME = "alexas-ec2-cost-analysis-bucket"
S3_REGION = "us-east-1"
def upload_to_s3(file_name, object_name=None):
    s3_client = boto3.client('s3', region_name=S3_REGION)
    try:
        logging.info(f"Attempting to upload '{file_name}' to S3: {S3_BUCKET_NAME}/{object_name}")
        s3_client.upload_file(file_name, S3_BUCKET_NAME, object_name)
        logging.info(f"File '{file_name}' uploaded to S3: {S3_BUCKET_NAME}/{object_name}")
    except Exception as e:
        logging.error(f"Error uploading file '{file_name}' to S3: {e}")
        print(f"Upload error: {e}")
        print(f"File path that was attempted to upload: {file_name}")
        print(f"S3 path that was attempted to use: {S3_BUCKET_NAME}/{object_name}")

Flask Web Interface:

We use Flask to create a web interface for our application. The /run_analysis route triggers the analysis and visualization generation:
from flask import Flask, render_template, send_from_directory import os

app = Flask(__name__)
@app.route("/run_analysis")
def run_analysis():
    # ... (read data, analyze, generate visualizations, upload to S3)
    return "Analysis completed and results uploaded to S3."
@app.route("/visualizations/<filename>")
def get_visualization(filename):
    # ... (download visualization from S3 and return)
    return send_from_directory(".", local_filename, as_attachment=False)

Matplotlib Backend Configuration:

To ensure compatibility with our Flask application, we explicitly set Matplotlib to use the “Agg” backend: import matplotlib matplotlib.use('Agg') # Force Matplotlib to use the Agg backend

Results:

A password protected S3 Static Website that serves as the repository for all of the info we need:

The ability to query and download the information, including source .csv and visualizations charts

Key Considerations:

1️⃣Security: Password protection and secure storage of sensitive data are crucial.
2️⃣Scalability: Consider using asynchronous tasks or a message queue for long-running analysis.
3️⃣Error Handling: Implement robust error handling to gracefully handle exceptions and provide informative messages.
4️⃣User Experience: Design a user-friendly interface that provides clear and concise information.

This application provides a foundation for building a comprehensive EC2 cost analysis tool. By leveraging the power of Pandas, Matplotlib, and AWS S3, we can create a powerful and insightful application, and we get to use our magic hat!

Github Link: Python_Applications/EC2_Costs.py at main · alexcurtis1969/Python_Applications

AWSBuilder

AWSCommunityBuilder

@jasondunn

Building Financial Data Analysis with AWS: A Step-by-Step Guide

Alex Curtis — Sun, 23 Mar 2025 16:04:40 +0000

This article was first published on Medium. Check it out here: [https://medium.com/@alex.curtis_luit/grab-the-snake-by-the-tail-and-wag-it-until-you-have-something-useful-a110c108fc53].

Grab the snake by the tail and wag it until you have something useful!

Building a Financial Data Analysis and Reporting Tool with Python

The answer is in the question….shake the Python until you get something useful…are you ready?

In today’s data-driven world, having the right tools to analyze financial data is crucial for decision-making. As part of my ongoing journey in the cloud and DevOps space, I recently created a Python script that fetches market data for stocks and cryptocurrencies, calculates key technical indicators, and generates a comprehensive financial report in PDF format.

In this article, I’ll Walk you through the code I built, its features, and how you can use it to analyze financial market data with Python.

Overview of the Project
This project pulls market data from Yahoo Finance for various tickers (stocks and cryptocurrencies), computes technical indicators like moving averages and Bollinger Bands, and finally outputs the results in a PDF report. The aim is to provide a one-stop tool for both individual investors and professionals to track the performance of various assets and gain insights into market trends.

Key Features:

Data Fetching: The script retrieves historical data for stocks and cryptocurrencies. * The cool thing is you can modify it as you see fit!
Technical Analysis: It calculates standard technical indicators such as the 20-day moving average and Bollinger Bands.
PDF Report Generation: The data and analysis are compiled into a PDF file for easy sharing and viewing.
Code Breakdown
Let’s walk through the key sections of the code and understand how they work.

Fetching Financial Data

We use the yfinance library to fetch market data for a list of tickers. Yahoo Finance provides historical price data for various financial assets, including stocks, indices, and cryptocurrencies.

import yfinance as yf

def fetch_and_process_data(ticker):
    print(f"Fetching data for {ticker}...")
    df = yf.download(ticker, period="1y", interval="1d")
    return df

Here, the fetch_and_process_data function takes a ticker symbol (e.g., "AAPL" for Apple or "BTC-USD" for Bitcoin) and downloads one year of daily historical data.

Calculating Technical Indicators Once we have the data, we perform some technical analysis. For example, we calculate the Rolling Standard Deviation to analyze price volatility and Bollinger Bands to evaluate price movements relative to a moving average.

What does all of this mean?

Rolling Standard Deviation = Measures how much the price is jumping around.

import pandas as pd

def calculate_technical_indicators(df):
    # Calculate rolling standard deviation (volatility)
    df['rolling_std'] = df['Close'].rolling(window=20).std()

    # Calculate Bollinger Bands
    df['UpperBand'] = df['Close'].rolling(window=20).mean() + 2 * df['rolling_std']
    df['LowerBand'] = df['Close'].rolling(window=20).mean() - 2 * df['rolling_std']

    # Fill any missing values with 'N/A'
    df.fillna("N/A", inplace=True)

    return df

In this section, we compute a 20-day rolling standard deviation and use it to calculate the upper and lower Bollinger Bands. Bollinger Bands are used to identify overbought or oversold conditions in an asset.

Generating the PDF Report Once the data is processed and the indicators are calculated, we generate a PDF report using the fpdf library. This report includes the asset's closing price, the calculated technical indicators, and charts for visual representation.

from fpdf import FPDF

def generate_pdf(tickers):
    pdf = FPDF()
    pdf.set_auto_page_break(auto=True, margin=15)
    pdf.add_page()

    for ticker in tickers:
        df = fetch_and_process_data(ticker)
        df = calculate_technical_indicators(df)

        # Write the ticker symbol and the technical indicators to the PDF
        pdf.set_font("Arial", size=12)
        pdf.cell(200, 10, txt=f"Technical Report for {ticker}", ln=True)

        # Add data to the PDF (this is just an example for the first 5 rows)
        for index, row in df.head().iterrows():
            pdf.cell(200, 10, txt=f"{index}: {row['Close']} - UpperBand: {row['UpperBand']} - LowerBand: {row['LowerBand']}", ln=True)

    pdf.output("fintech_market_data.pdf")
    print("PDF generated successfully: fintech_market_data.pdf")

The generate_pdf function adds the processed financial data (including stock prices and technical indicators) to the PDF file. You can extend this part to add more visualizations, such as charts or graphs, to make the report even more insightful.

Handling Data Quality We also handle missing data by filling it with “N/A”. This ensures the script doesn’t break when data is missing for any particular day.

df.fillna("N/A", inplace=True)

This approach prevents errors from occurring when we try to display or process the data further.

Running the Script
Once the script is set up and ready, you simply need to run the main function, which pulls data for each ticker, performs the analysis, and generates the PDF:

def main():
    tickers = ["AAPL", "MSFT", "GOOGL", "BTC-USD", "ETH-USD"]
    generate_pdf(tickers)

if __name__ == "__main__":
    main()

Output
When you run the script, it will download market data for the tickers you specify (Apple, Microsoft, Google, Bitcoin, and Ethereum in this case). It will then generate a PDF file named fintech_market_data.pdf, which contains the financial analysis for each asset, including the calculated technical indicators.

Conclusion
This project is a great starting point for building more advanced financial tools. You can enhance the script further by adding more technical indicators, integrating machine learning models for predictions, or even pushing the data to a cloud platform for real-time analysis.

With Python and a few powerful libraries like yfinance, pandas, and fpdf, you can create powerful financial tools that provide valuable insights. I hope this article inspires you to experiment with financial data analysis and see how Python can simplify the process.

AWS

AWSCommunity

AWSCommunityBuilder

Thanks to @jasondunn @chriscampbell96