Automated Web Scraping and Data Visualization with Python and AI — Part 7: Visualizing Scraped Data with Interactive Dashboards

Automated Web Scraping and Data Visualization with Python and AI — Part 7: Visualizing Scraped Data with Interactive Dashboards and AI‑Driven Storytelling

In the first six parts of this series we built a robust, AI‑assisted scraper (using Claude 4.6 Opus and GPT‑5.4 Pro parallel agents), stored the results in a PostgreSQL data lake, and performed light‑weight cleaning and feature engineering. Now we turn those raw rows into compelling, interactive dashboards and let a large language model write the narrative that makes the data stick.

Why Dashboards Matter in 2026

The Best AI Tools for Data Visualization in 2026 report stresses that “even the smartest AI insights fall flat if they aren’t easy to interpret.” Interactivity—drill‑downs, hover‑tooltips, and on‑the‑fly filtering—has become the lingua franca of business intelligence. Coupled with AI‑generated storytelling, dashboards can now answer “what happened?” and “why it matters?” in a single click.

What We’ll Build

• A Streamlit app that pulls the latest scraped product‑price data from PostgreSQL.
• Dynamic visualizations powered by Plotly Express (time‑series, geo‑maps, and correlation heatmaps).
• An AI‑driven narrative engine that uses Claude 4.6 Opus (via the Anthropic API) and GPT‑5.4 Pro (via OpenAI) to generate natural‑language summaries, highlight anomalies, and suggest actions.
• Parallel‑agent orchestration (Claude + GPT) using the new parallel decorator from the openaigen SDK, demonstrating the “GPT‑5.4 Pro Parallel Agents” capability introduced earlier this year.

Prerequisites

ComponentVersion / Note


Python3.11+
Streamlit1.38.0
Plotly5.22.0
SQLAlchemy2.0.31
psycopg2‑binary2.9.9
anthropic0.5.3 (Claude 4.6 Opus)
openai1.45.0 (GPT‑5.4 Pro)
openaigen0.2.1 (parallel agents)

All of these packages are installable via pip install. If you’re using the Kadoa SDK (the AI‑powered CSS selector generator mentioned in the “Top 7 AI‑Powered Web Scraping Solutions in 2026”), you’ll already have a clean DataFrame ready for ingestion.

Step 1 – Setting Up the Database Connection

We keep the scraped data in a PostgreSQL table called product_prices. The schema is simple but extensible:

CREATE TABLE product_prices (
    id              SERIAL PRIMARY KEY,
    product_id      TEXT NOT NULL,
    product_name    TEXT NOT NULL,
    category        TEXT,
    price_usd       NUMERIC(12,2),
    scraped_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    source_url      TEXT,
    raw_json        JSONB
);

Below is a tiny helper that returns a SQLAlchemy engine using environment variables for credentials. This pattern follows the security recommendations from the Best Web Scraping Courses & Certificates [2026] curricula.

import os
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

def get_engine():
    db_url = os.getenv(
        "DATABASE_URL",
        "postgresql+psycopg2://scraper_user:password@localhost:5432/scraper_db"
    )
    return create_engine(db_url, pool_pre_ping=True)

SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=get_engine())

Step 2 – Pulling the Latest Data

The following function loads the most recent 30 days of price history for a given product_id. It also caches the result with @st.cache_data so the dashboard stays snappy.

import pandas as pd
import streamlit as st
from sqlalchemy import text

@st.cache_data(ttl=300)  # refresh every 5 minutes
def load_price_history(product_id: str) -> pd.DataFrame:
    query = text(
        """
        SELECT product_name, price_usd, scraped_at, source_url
        FROM product_prices
        WHERE product_id = :pid
          AND scraped_at >= NOW() - INTERVAL '30 days'
        ORDER BY scraped_at ASC
        """
    )
    with SessionLocal() as session:
        df = pd.read_sql_query(query, session.connection(), params={"pid": product_id})
    df["scraped_at"] = pd.to_datetime(df["scraped_at"])
    return df

Step 3 – Building the Interactive Visuals

We’ll expose three core charts:

• Time‑Series Price Trend – line chart with hover‑tooltips.
• Geographic Distribution – a choropleth that shows average price by US state (assuming source_url contains a state query param).
• Correlation Heatmap – reveals relationships between price, discount, and rating if those columns exist.

All charts are built with Plotly Express, which streams directly into Streamlit via st.plotly_chart.

import plotly.express as px

def price_trend_chart(df: pd.DataFrame):
    fig = px.line(
        df,
        x="scraped_at",
        y="price_usd",
        title=f"Price Trend for {df['product_name'].iloc[0]}",
        hover_data=["source_url"]
    )
    fig.update_layout(hovermode="x unified")
    return fig

def geo_price_chart(df: pd.DataFrame):
    # Extract state from URL assuming pattern ...?state=CA
    df["state"] = df["source_url"].str.extract(r"state=([A-Z]{2})")
    agg = df.groupby("state")["price_usd"].mean().reset_index()
    fig = px.choropleth(
        agg,
        locations="state",
        locationmode="USA-states",
        color="price_usd",
        color_continuous_scale="Viridis",
        scope="usa",
        title="Average Price by State (Last 30 Days)"
    )
    return fig

def correlation_heatmap(df: pd.DataFrame):
    # Assume df may have extra columns like 'rating' and 'discount_pct'
    numeric_cols = df.select_dtypes(include="number")
    if numeric_cols.shape[1]  str:
    prompt = f"""
    You are a data analyst.  Given the JSON payload below that contains
    product price history for the last 30 days, extract:
    1. The highest price and when it occurred.
    2. The lowest price and when it occurred.
    3. Any price spikes greater than 15 % within a 24‑hour window.
    Return a concise JSON object with keys: highest, lowest, spikes.
    ----
    {df_json}
    """
    response = claude_client.messages.create(
        model="claude-4.6-opus",
        max_tokens=500,
        temperature=0.0,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

@parallel
def craft_narrative(insights_json: str) -> str:
    prompt = f"""
    Using the insights JSON below, write a 150‑word executive summary.
    Highlight any pricing anomalies, suggest possible causes (e.g., promotions,
    supply constraints), and recommend a next step for a product manager.
    Keep the tone professional but engaging.
    ----
    {insights_json}
    """
    response = openai_client.chat.completions.create(
        model="gpt-5.4-pro",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,
        max_tokens=300
    )
    return response.choices[0].message.content

The parallel decorator automatically runs extract_insights and craft_narrative in separate threads, returning a tuple of results once both complete.

Step 5 – Wiring Everything into Streamlit

Below is the full app.py that brings the pieces together. Run it with streamlit run app.py.

import streamlit as st
import pandas as pd
import json

# Import helpers from previous sections
from db import load_price_history
from visuals import price_trend_chart, geo_price_chart, correlation_heatmap
from ai_storytelling import extract_insights, craft_narrative

st.set_page_config(page_title="Product Price Dashboard", layout="wide")

st.title("📈 AI‑Enhanced Product Price Dashboard")
st.caption("Based on my technical understanding as a Lead Programmer Analyst, this app demonstrates how to turn scraped data into interactive visualizations and AI‑driven narratives.")

# Sidebar – product selection
product_id = st.sidebar.text_input("Enter Product ID", value="B07XYZ1234")
if not product_id:
    st.stop()

# Load data
df = load_price_history(product_id)

if df.empty:
    st.warning("No data found for the supplied product ID.")
    st.stop()

# Layout
col1, col2 = st.columns([2, 1])

with col1:
    st.subheader("Price Trend")
    st.plotly_chart(price_trend_chart(df), use_container_width=True)

    st.subheader("Geographic Distribution")
    st.plotly_chart(geo_price_chart(df), use_container_width=True)

    st.subheader("Correlation Heatmap")
    heatmap = correlation_heatmap(df)
    if heatmap:
        st.plotly_chart(heatmap, use_container_width=True)

with col2:
    st.subheader("AI‑Generated Story")
    # Convert DataFrame to JSON for the LLM
    df_json = df.to_json(orient="records", date_format="iso")
    # Run both agents in parallel
    insights_json, narrative = extract_insights(df_json), craft_narrative(insights_json=None)  # placeholder
    # The parallel decorator returns a tuple; we unpack it correctly:
    insights_json, narrative = extract_insights(df_json), None  # placeholder for demo
    # Real execution:
    insights_json, narrative = extract_insights(df_json), craft_narrative(insights_json)
    # Display results
    st.json(json.loads(insights_json))
    st.markdown(narrative)

st.sidebar.markdown("---")
st.sidebar.info(
    "💡 Tip: Adjust the date range or add custom filters using the sidebar to explore other time windows."
)

Note: In the code block above the line insights_json, narrative = extract_insights(df_json), craft_narrative(insights_json) demonstrates the parallel workflow – both calls are dispatched at once, and Streamlit only blocks when the tuple is ready. The SDK handles thread‑pool management, which is a core feature of the “GPT‑5.4 Pro Parallel Agents” architecture released in Q1 2026.

Step 6 – Adding Drill‑Down Interactivity

To make the dashboard truly exploratory, we embed a st.selectbox that lets the user pick a specific date range. Plotly automatically respects the filtered DataFrame, and the AI narrative updates accordingly.

# Add after data load
date_range = st.sidebar.slider(
    "Select date range (days ago)",
    min_value=1,
    max_value=30,
    value=30,
    step=1
)
cutoff = pd.Timestamp.now(tz="UTC") - pd.Timedelta(days=date_range)
filtered_df = df[df["scraped_at"] >= cutoff]

# Use filtered_df in all visual calls
st.plotly_chart(price_trend_chart(filtered_df))
# ... etc.

Step 7 – Deploying the Dashboard

For production you’ll likely push the app to Streamlit Cloud, Azure App Service, or a Docker‑based Kubernetes pod. The following Dockerfile is a minimal, multi‑stage build that respects the 2026 security baseline (non‑root user, pinned dependencies, and read‑only filesystem).

# ---- Build Stage ----
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# ---- Runtime Stage ----
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY . .
RUN useradd -m streamlit && chown -R streamlit:streamlit /app
USER streamlit
EXPOSE 8501
CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.headless=true"]

Deploying this image with docker run -p 8501:8501 ghcr.io/yourorg/price-dashboard:latest gives you a secure, horizontally‑scalable dashboard that can be load‑balanced behind an Ingress with TLS termination.

Best Practices & Gotchas in 2026

• Rate‑limit awareness: Even AI‑assisted scrapers like Kadoa (see the Top 7 AI‑Powered Web Scraping Solutions in 2026) enforce per‑domain quotas. Store the scraped_at timestamp and respect the Retry‑After header to avoid bans.
• Data freshness vs. cost: Claude 4.6 Opus is cheap for extraction, but GPT‑5.4 Pro tokens can add up. Cache the insight JSON for at least 5 minutes (as shown with @st.cache_data) to amortize LLM calls across users.
• Explainability: Include a “Why this chart?” toggle that surfaces the underlying SQL query and the AI reasoning chain. This satisfies the governance demands highlighted in the ThoughtSpot report.
• Privacy compliance: Strip personally‑identifiable information (PII) before sending data to external LLM APIs. The raw_json column should be filtered to only include non‑PII fields.

Extending the Storytelling Engine

Future iterations can incorporate multimodal models (e.g., Claude 4.6 Opus with image generation) to automatically create annotated charts. The openaigen SDK already supports image outputs, so you could ask the LLM to “draw a sparkline highlighting the spike on March 12” and embed the PNG directly in the Streamlit sidebar.

Putting It All Together – A Sample Run

Assume the product ID B07XYZ1234 corresponds to a popular smart‑watch. After a few seconds of loading, the dashboard shows:

• A line chart where the price dips on 2026‑07‑15 (a 20 % discount flash sale) and spikes on 2026‑07‑22 (limited‑edition color release).
• A choropleth indicating the West Coast enjoys an average price $15 lower than the Midwest.
• A heatmap revealing a strong negative correlation (‑0.68) between discount_pct and price_usd.

The AI narrative (generated by GPT‑5.4 Pro) reads:

“Over the past month, the smartwatch’s price exhibited a notable dip of 20 % on July 15, coinciding with a weekend promotion that drove a 12 % surge in sales volume (data not shown). Conversely, a 15 % price spike on July 22 aligns with the

---

Originally published at https://artificial-inteligence.phptutorial.co.in