DEV Community: Amit Wani

DecipherIt: Building a NotebookLM-Inspired AI Research Assistant powered by Bright Data

Amit Wani — Sun, 25 May 2025 22:13:33 +0000

This is a submission for the Bright Data AI Web Access Hackathon

What I Built

DecipherIt is a cutting-edge AI-powered research assistant inspired by Google NotebookLM that revolutionizes how researchers, students, and professionals explore, analyze, and synthesize information from the web. The platform transforms any combination of documents, URLs, or topics into comprehensive research notebooks complete with AI-generated summaries, interactive Q&A capabilities, audio overviews, visual mindmaps, and automatically generated FAQs.

The Problem DecipherIt Solves

Traditional research is time-consuming and fragmented. Researchers often struggle with:

Information Overload: Sifting through countless sources manually
Geo-restrictions: Unable to access content from different regions
Bot Detection: Getting blocked when trying to scrape valuable data
Synthesis Challenges: Difficulty connecting insights across multiple sources
Accessibility: Converting research into different formats for various audiences

DecipherIt addresses these challenges by leveraging Bright Data's MCP Server to provide unrestricted, intelligent web access combined with advanced AI agents that can understand, synthesize, and present information in multiple formats.

Key Features

🔬 Deep Research - Conduct thorough research on any topic with AI-assisted analysis and synthesis
🔍 Multi-Source Research - Seamlessly integrate documents, URLs, and manual text into unified research spaces
🤖 AI-Powered Summaries - Generate comprehensive, well-structured research analyses using advanced AI agents
💬 Interactive Q&A - Chat with your research materials using natural language queries
🎧 Audio Overviews - AI-generated podcast-style audio summaries with multiple voices
❓ Smart FAQ Generation - Automatically create relevant FAQs from your research content
🧠 Visual Mindmaps - Generate interactive, hierarchical mindmaps to visualize research structure and connections
🌐 Global Web Access - Bypass geo-restrictions and bot detection using Bright Data's infrastructure

🔍 Detailed Feature Overview

🔬 Deep Research
DecipherIt's AI agents conduct comprehensive research by strategically planning data collection, discovering diverse sources through Bright Data's global search capabilities, and synthesizing information from multiple perspectives. The system can research any topic from current events to academic subjects, providing thorough analysis that rivals human researchers.

🔍 Multi-Source Research
Users can combine various input types in a single research project: upload documents (PDF, DOCX, PPTX, XLSX), add custom URLs for specific web content, input manual text for direct analysis, or simply enter topics for AI-driven discovery. All sources are processed and integrated into a unified research space.

🤖 AI-Powered Summaries
Specialized CrewAI agents work together to create comprehensive research summaries. The Research Analyst synthesizes information from all sources, while the Content Writer crafts engaging, well-structured analyses that highlight key insights, trends, and connections across the research material.

💬 Interactive Q&A
Using vector embeddings and semantic search through Qdrant database, users can ask natural language questions about their research content. The system provides contextual answers by retrieving relevant information from all processed sources, enabling deep exploration of the research material.

🎧 Audio Overviews
On-demand feature that transforms research into engaging podcast-style audio content. The Podcast Script Generator agent creates conversational scripts, which are then converted to high-quality audio using LemonFox TTS with multiple AI voices, making research accessible in audio format.

❓ Smart FAQ Generation
AI agents automatically analyze research content to generate relevant, insightful questions and comprehensive answers. This feature helps users understand key aspects of their research topic and provides quick access to important information.

🧠 Visual Mindmaps
The Mindmap Creator agent analyzes research structure to generate interactive, hierarchical visualizations with up to 5 levels of depth. Built with react-mindmap-visualiser, these mindmaps help users understand complex topics at a glance and navigate research relationships visually.

🌐 Global Web Access
Powered by Bright Data's MCP Server, DecipherIt bypasses geo-restrictions and bot detection to access content from anywhere in the world. This ensures comprehensive research coverage and access to diverse, authoritative sources that traditional scraping methods cannot reach.

Demo

🚀 Live Demo: https://decipherit.xyz

Demo Credentials:

Email: demo@dev.to
Password: demodemo@123

📂 GitHub Repository:

mtwn105 / decipher-research-agent

Turn topics, links, and files into AI-generated research notebooks — summarize, explore, and ask anything.

🔍 DecipherIt - AI-Powered Research Assistant

Transform your research process with AI-powered intelligence

🚀 Live Demo • 🛠️ Installation • 🤝 Contributing

⭐ Star on GitHub!

✨ Overview

DecipherIt is a cutting-edge AI-powered research assistant inspired by Google NotebookLM that revolutionizes how researchers, students, and professionals explore, analyze, and synthesize information from the web. The platform transforms any combination of documents, URLs, or topics into comprehensive research notebooks complete with AI-generated summaries, interactive Q&A capabilities, audio overviews, visual mindmaps, and automatically generated FAQs.

The Problem DecipherIt Solves

Traditional research is time-consuming and fragmented. Researchers often struggle with:

Information Overload: Sifting through countless sources manually
Geo-restrictions: Unable to access content from different regions
Bot Detection: Getting blocked when trying to scrape valuable data
Synthesis Challenges: Difficulty connecting insights across multiple sources
Accessibility: Converting research into different formats for various audiences

DecipherIt addresses these challenges by leveraging…

View on GitHub

Video Demo

📺 Watch DecipherIt in action:

The video demonstrates:

Setting up a new research notebook
Adding multiple sources (URLs, documents, text)
AI-powered research and analysis process
Exploring generated summaries and insights
Using interactive features like Q&A and mindmaps
Generating audio overviews

Screenshots

How It Works

Input Your Research Sources: Enter any topic, upload documents, add custom URLs, or input manual text
AI Planning: The system creates a strategic research plan using specialized AI agents
Web Discovery: Bright Data's search engine finds relevant sources globally
Intelligent Scraping: Bright Data extracts content and converts it to clean markdown format
AI Analysis: Multiple AI agents analyze, synthesize, and create comprehensive summaries
Multi-Format Output: Get research summaries, FAQs, visual mindmaps, and podcast-style audio overviews

Tech Stack

Frontend

Next.js 15 with App Router
React 19 with concurrent features
TypeScript 5 for type safety
Tailwind CSS 4 for styling
Shadcn/ui component library
Better Auth for authentication
react-mindmap-visualiser for interactive mindmap visualization

Backend

Python 3.12 with FastAPI
CrewAI for multi-agent orchestration
Bright Data MCP Server for web access
Qdrant vector database for semantic search
SQLAlchemy with PostgreSQL
LemonFox TTS for audio generation

AI & ML Services

Google Gemini via OpenRouter for LLM capabilities
OpenAI Embeddings for semantic search
MarkItDown for document processing

CrewAI Crews Overview

DecipherIt employs a sophisticated multi-crew architecture powered by CrewAI:

Planning Crew

Agent: Web Scraping Strategy Expert
Task: Generate 3 targeted search queries for comprehensive topic coverage

Link Discovery Crew

Agent: Link Discovery Specialist
Task: Find authoritative sources using Bright Data search engine

Web Scraping Crew

Agent: Expert Web Scraping Engineer
Task: Extract clean markdown content from URLs using Bright Data scraper

Research Analysis Crew

Agent: Senior Research Analyst
Task: Synthesize multi-source data into comprehensive research insights

Content Creation Crew

Agents: Research Analyst + Content Writer
Tasks: Create engaging blog posts + Generate 10 detailed FAQs

Audio Overview Crew

Agents: Research Analyst + Conversation Planner + Script Writer
Tasks: Analyze content + Plan conversation + Generate 4-5 minute podcast transcript

Mindmap Generation Crew

Agents: Content Analyzer + Mindmap Creator
Tasks: Identify hierarchical themes + Build interactive visualizations (up to 5 levels)

Chat Response Crew

Agent: Decipher (Analytical Assistant)
Task: Answer questions using vector search and chat context with source citations

Architecture

How I Used Bright Data's Infrastructure

Bright Data's MCP (Model Context Protocol) Server is the backbone of DecipherIt's web access capabilities. Here's how I integrated it:

1. Official MCP Server Integration

from mcp import StdioServerParameters
from crewai_tools import MCPServerAdapter

server_params = StdioServerParameters(
    command="pnpm",
    args=["dlx", "@brightdata/mcp"],
    env={
        "API_TOKEN": os.environ["BRIGHT_DATA_API_TOKEN"],
        "BROWSER_AUTH": os.environ["BRIGHT_DATA_BROWSER_AUTH"]
    },
)

2. Two Core Tools Implementation

Search Engine Tool - For discovering relevant sources:

web_scraping_link_collector_tools = [
    tool for tool in tools if tool.name in ["search_engine"]
]

Scrape as Markdown Tool - For extracting clean content:

web_scraping_tools = [
    tool for tool in tools if tool.name in ["scrape_as_markdown"]
]

3. Multi-Agent Workflow

I created specialized CrewAI agents that leverage Bright Data's tools:

Link Collector Agent: Uses search_engine to find relevant sources based on research topics
Web Scraper Agent: Uses scrape_as_markdown to extract clean, structured content from discovered URLs

4. Parallel Processing for Scale

# Execute multiple scraping tasks in parallel
web_scraping_tasks = []
for link in links:
    web_scraping_tasks.append(
        web_scraping_crew.kickoff_async(inputs={
            "url": link.url,
            "current_time": current_time,
        })
    )

web_scraping_results = await asyncio.gather(*web_scraping_tasks)

6. Data Processing & AI Integration

Here's how we process the scraped data and integrate it with our AI agents:

# Process scraped content for AI analysis
async def integrate_scraped_data(web_scraping_results, links):
    scraped_data = []

    # Extract clean content from Bright Data results
    for link, result in zip(links, web_scraping_results):
        scraped_data.append({
            "url": link.url,
            "content": result.raw,  # Clean markdown format
            "title": link.title
        })

    # Create vector embeddings for semantic search
    embeddings = await create_embeddings(scraped_data)

    # Store in Qdrant vector database
    await store_in_vector_db(embeddings, scraped_data)

    # Trigger AI analysis crew
    research_result = await research_content_crew.kickoff_async(inputs={
        "scraped_data": scraped_data,
        "current_time": datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    })

    return research_result

Frontend integration with React and TypeScript:

// Research hook for managing AI-powered research
const useResearch = () => {
  const [isLoading, setIsLoading] = useState(false);
  const [research, setResearch] = useState<Research | null>(null);

  const startResearch = async (sources: ResearchSource[]) => {
    setIsLoading(true);
    try {
      const response = await fetch("/api/research", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ sources }),
      });

      const result = await response.json();
      setResearch(result);
    } catch (error) {
      console.error("Research failed:", error);
    } finally {
      setIsLoading(false);
    }
  };

  return { research, isLoading, startResearch };
};

5. Seamless Integration with AI Pipeline

The scraped data from Bright Data flows seamlessly into DecipherIt's multi-layered AI processing system:

Immediate Processing:

Vector embeddings created using OpenAI embeddings and stored in Qdrant for semantic search capabilities
Contextual analysis by Research Analyst agents to synthesize information from multiple sources
Automatic FAQ generation by analyzing content patterns and extracting key insights

On-Demand Generation:

Audio script creation when users request podcast-style overviews, processed by specialized TTS agents
Mindmap structure analysis for hierarchical visualization when users want visual representations
Interactive Q&A responses powered by vector similarity search through processed content

Performance Improvements

Real-time web data access through Bright Data's infrastructure dramatically enhanced DecipherIt's AI system performance compared to traditional static data approaches:

🚀 Key Improvements

Real-Time Information Access: Unlike AI systems limited by training data cutoffs, DecipherIt's agents access current information including breaking news, latest research papers, and up-to-date statistics.

Global Content Discovery: Bright Data's search engine enables AI agents to discover diverse perspectives from global sources, access region-specific content, and find specialized publications that static systems cannot reach.

Clean Data Processing: The Bright Data's tool provides structured, clean content that AI agents process more effectively, improving analysis accuracy and reducing noise.

Future Enhancements

Interactive Mindmap: Enhanced mindmap features with expandable nodes, custom styling, and export options
Email Notifications: Automated email alerts for research completion and important updates
Mobile App: Native mobile experience for research on-the-go
More Robust Retry Mechanism: Improved background task handling with intelligent retry logic
Live Status Updates: Real-time agent activity monitoring and progress tracking
More Complex Scraping: Advanced web scraping capabilities for dynamic content and complex sites
Social Sign-ins: Integration with Google, GitHub, and other social authentication providers

Conclusion

DecipherIt demonstrates the power of combining Bright Data's robust web access infrastructure with advanced AI agents. By leveraging the Bright Data MCP Server, we've created a research assistant that can access, analyze, and synthesize information from across the global web without the typical limitations of traditional scraping methods.

The platform showcases how real-time web data access can dramatically improve AI system performance, making research faster, more comprehensive, and more reliable than ever before.

Special thanks to DEV.to and Bright Data for organizing this amazing hackathon that made DecipherIt possible! The opportunity to build with Bright Data's powerful infrastructure has been invaluable in bringing this project to life.

Vibe Coded AI Stock Analyst with Amazon Q in 1 Hour!

Amit Wani — Mon, 12 May 2025 06:13:09 +0000

This is a submission for the Amazon Q Developer "Quack The Code" Challenge: Crushing the Command Line & Exploring Possibilities

What I Built 🛠️📈

I have built Stock Pulse. Stock Pulse is a Python tool that analyzes stocks using financial metrics, news trends, and AI-powered sentiment analysis. It provides investment recommendations (BUY, SELL, HOLD) with detailed reasoning based on comprehensive analysis.

I have used Amazon Q Developer CLI to Vibe Code this in under 1 hour without writing a single line of code. I have recorded my full vibe-coding session. It can be found in the Demo section below.

Features ✨

🔍 Analyze multiple stocks at once
💹 Fetch financial metrics and news data using yfinance
🤖 Utilize OpenAI's GPT-4o-mini model for intelligent analysis
📈 Generate investment signals with detailed reasoning
⚠️ Identify key factors and potential risks
💻 Command-line interface and Streamlit web UI

Screenshots 🖼️

Demo 🎬

Streamlit Link: here
Vibe-Coding Session Demo:

Code Repository 🗃️

mtwn105 / stock-pulse

Feel the pulse of the market with AI-powered stock insights. Vibe Coded with Amazon Q Developer CLI in 1 hour!

Stock Pulse

Feel the pulse of the market with AI-powered stock insights.

Vibe Coded with Amazon Q Developer CLI in 1 hour!

Overview

Stock Pulse is a Python tool that analyzes stocks using financial metrics, news trends, and AI-powered sentiment analysis. It provides investment recommendations (BUY, SELL, HOLD) with detailed reasoning based on comprehensive analysis.

Features

Analyze multiple stocks at once
Fetch financial metrics and news data using yfinance
Utilize OpenAI's GPT-4o-mini model for intelligent analysis
Generate investment signals with detailed reasoning
Identify key factors and potential risks
Command-line interface and Streamlit web UI

Installation

Clone the repository:

git clone https://github.com/yourusername/stock-pulse.git
cd stock-pulse

Install dependencies using Poetry:
```
poetry install
```
Create a .env file with your OpenAI API key:
```
cp .env.example .env
```
Then edit the .env file to add your OpenAI API key.

Usage

Command Line Interface

Analyze stocks using the command line:

# Analyze a single stock
poetry run python

…

View on GitHub

How I Used Amazon Q Developer 🤖⚡

I have used Amazon Q Developer CLI to Vibe Code this project in under 1 hour without writing a single line of code manually.

Initial Prompt 📝

I kickstarted the development by giving Amazon Q a single, well-crafted prompt:

i want to create a stock analysis script in python using langchain and langgraph with using openai llm with gpt-4o-mini model. 

use poetry for deps management. poetry is already installed on the system. 

use yfinance tools from langchain for news and price data. use last 1 year price return. 
also use other financial metrics from yfinance like roce, roe, etc. and llm will analyze and return recommendation with signal - BUY,SELL,HOLD and reasoning for it. 

accept list of tickers as argument to the script. 

use poetry to setup project and add deps. dont create pyproject.toml by yourself. 

use python best practices and folder structure for a cli.

Amazon Q responded by generating an organized, ready-to-run CLI project using Poetry, fetching real-time stock data, and integrating GPT-4o-mini for analysis and recommendations.

Iterative Development with Q 🔁💡

I continued to use Amazon Q Developer CLI throughout the process for:

Debugging and resolving errors by simply pasting the error logs into Q.
Requesting code refactors and feature enhancements (e.g., adding streamlit UI or news items in output).
Running Git commands and pushing my code without switching terminals.

Zero Manual Edits ✋❌

From project initialization to the final commit, I did not write or modify any code manually. Every change—big or small—was executed through Amazon Q Developer CLI.

To show full transparency and the incredible capability of Amazon Q, I’ve recorded my entire coding session, which you can find in the demo video above.

Why Amazon Q Was a Game-Changer 💥🎯

Amazon Q isn’t just another coding assistant—it’s an AI developer. It allowed me to stay in flow, innovate rapidly, and push creative boundaries, eliminating friction from the development lifecycle.

From initializing the project with Poetry to handling dependencies, coding logic, bug fixing, git commands, and even feature iterations—Amazon Q handled everything like a true coding copilot. I never had to touch the code manually.

I’ve also recorded my entire coding session (watch the demo video above) to showcase how seamlessly Q guided me through the entire development lifecycle.

Conclusion ✅🎉

Participating in the Amazon Q Developer "Quack The Code" Challenge was a fun and productive experience. I built an intelligent UI & CLI-based stock analysis tool using LangChain, LangGraph, and OpenAI—all within an hour—thanks to the power of Amazon Q Developer CLI.

This project proved that with the right tools and a powerful AI assistant like Amazon Q Developer, building complex and intelligent applications becomes fast, efficient, and even enjoyable.

AudioIntel - Transform Audio into Actionable Intelligence

Amit Wani — Mon, 25 Nov 2024 04:29:09 +0000

This is a submission for the AssemblyAI Challenge: Sophisticated Speech-to-Text & No More Monkey Business. 🏆

What I Built 🛠️

I built AudioIntel - a powerful platform that transforms audio content into actionable intelligence using AssemblyAI's cutting-edge APIs. The platform helps users extract valuable insights from audio content through advanced transcription, analysis, and AI-powered features. ✨

🔗 Live Demo: https://audiointel.amitwani.dev

🎥 Demo Video

Journey 🗺️

The Inspiration 💡

The idea for AudioIntel came from my own struggles with processing audio content efficiently. As someone who consumes a lot of podcasts, interviews, and video content, I often found myself wanting to quickly extract key insights without listening to hours of content. I realized this was a common pain point for many content creators, researchers, and professionals. 🎧

Learning & Iterations 📚

🔄 Integration with AssemblyAI's powerful APIs for transcription and analysis
🗣️ Leveraging AssemblyAI's speaker diarization and sentiment analysis features
🧠 Leveraging AssemblyAI with LeMUR for summarization, question answering, and intelligent content analysis
⚠️ Error handling in audio processing and real-time status updates
🔄 State management for handling complex UI interactions
⚡ Performance optimization for processing large audio files
💾 Database integration using Neon PostgreSQL with Drizzle ORM
🔒 User authentication implementation with Better Auth
🌐 Language translation features using Google Translate API
📤 File upload handling through UploadThing integration

Features Showcase ✨

Multiple Input Sources 📥

📁 File Upload: Support for various audio formats through UploadThing integration
🎙️ Browser Recording: Direct audio capture using the Web Audio API
📺 YouTube Integration: YouTube video to audio conversion and analysis

Real-time Analysis 📊

👥 Speaker diarization with timeline visualization
😊 Sentiment analysis with color-coded segments
🔍 Interactive transcript search and navigation
💬 Interactive chat with the transcript

Smart Content Generation 📝

🤖 AI-powered blog post creation
💭 Context-aware chat interface
📌 Key sections identification with timestamps

Language Translation 🌍

🔄 Translate transcript to multiple languages

Screenshots 📸

Multiple Sources - Audio file, Record file & YouTube 📱

Overview & Analysis 📊

Interactive Features ⚡

Tech Stack 💻

🔥 Framework: Next.js 14 with App Router
📝 Language: TypeScript
💾 Database: Neon PostgreSQL with Drizzle ORM
🎨 UI: Tailwind CSS + shadcn/ui
🎙️ Audio Processing: AssemblyAI
📤 File Upload: UploadThing
📊 Analytics: OpenPanel
🔒 Authentication: Better Auth
🌐 Translation: Google Translate
🚀 Deployment: Vercel

Techincal Archicture 🏗️

Technical Implementation ⚙️

AssemblyAI Integration 🔌

I leveraged several powerful features from AssemblyAI's SDK:

Transcription API

const transcript = await assemblyai.transcripts.transcribe({
  audio: fileUrl,
  speaker_labels: true,
  summarization: true,
  summary_model: "conversational",
  summary_type: "bullets",
  sentiment_analysis: true,
});

LeMUR for Content Generation

// Generate blog post
const { response: blogPostResponse } = await assemblyai.lemur.task({
  transcript_ids: [transcript.id],
  prompt: `Generate a blog post from the transcript in markdown format`,
  final_model: "anthropic/claude-3-5-sonnet",
});

// Generate actionable insights
const { response: insights } = await assemblyai.lemur.task({
  transcript_ids: [transcript.id],
  prompt: `Provide actionable insights from the transcript`,
  final_model: "anthropic/claude-3-5-sonnet",
});

LeMUR for Interactive Chat

const { response: qas } = await assemblyai.lemur.questionAnswer({
  transcript_ids: [transcriptId],
  final_model: "anthropic/claude-3-5-sonnet",
  questions: [{ question: userMessage, answer_format: "short sentence" }],
});

Future Enhancements 🚀

Multi-language support
Advanced analytics dashboard
API endpoints
Custom templates
Advanced search capabilities

Source Code 🔗

mtwn105 / audio-intel

AudioIntel - Audio/Video Intelligence, Transcripts, Summary, and much more

🎙️ AudioIntel

Transform audio into actionable intelligence with our powerful AI platform. AudioIntel helps you extract valuable insights from audio content through transcription, analysis, and AI-powered features.

Live Demo

✨ Features

🎵 Multiple Input Methods
- Upload audio files (MP3, WAV)
- Record directly in browser
- Analyze YouTube videos
🤖 AI-Powered Analysis
- Smart summaries and key takeaways
- Sentiment analysis
- Speaker identification
- Actionable insights generation
📝 Content Generation
- Automatic blog post creation
- Interactive chat with transcripts
- Key sections identification
🔍 Advanced Features
- Timeline view with precise timestamps
- Multi-speaker detection
- Searchable transcripts
- Real-time sentiment tracking

🚀 Getting Started

Prerequisites

Node.js 18+
npm or yarn
AssemblyAI API key

Installation

Clone the repository

git clone https://github.com/yourusername/audio-intel.git
cd audio-intel

Install dependencies

npm install
# or
yarn install

Set up environment variables

cp .env.example .env

Required environment variables:

ASSEMBLYAI_API_KEY=your_api_key
NEXT_PUBLIC_APP_URL=http://localhost:3000
UPLOADTHING_TOKEN=your_uploadthing_token
GOOGLE_GENERATIVE_AI_API_KEY=your_google_generative_ai_api_key
GOOGLE_TRANSLATE_API_KEY=your_google_translate_api_key
BETTER_AUTH_SECRET=your_better_auth_secret
BETTER_AUTH_BASE_URL=http://localhost:3000
DATABASE_URL=your_database_url

Run the development server

npm run dev
# or
yarn dev

Open…

View on GitHub

Submission 📝

This submission was made for the AssemblyAI Challenge for "Sophisticated Speech-to-Text" & "No More Monkey Business" Prompts.

Conclusion 🎉

I had a great time participating in the AssemblyAI Challenge and learned a lot from the experience. I'm looking forward to seeing what other developers come up with! 🚀

Thank you Dev.To & AssemblyAI for organizing this challenge and providing such a great platform for developers to showcase their skills! 🎉

MailTales - Make Emails Fun Again!

Amit Wani — Mon, 02 Sep 2024 05:56:50 +0000

This is a submission for the Nylas Challenge: AI Expedition & Galaxy Brain

What I Built and Why

MailTales is a web app designed to breathe life into your email inbox. With the help of Nylas & AI, MailTales offers unique and creative ways to interact with your emails, transforming them into bite-sized stories, engaging summaries, and even providing insights through sentiment analysis. You can try it https://mailtales.amitwani.dev

Demo

MailTales: You can go to https://mailtales.amitwani.dev

Features

Chat Over Your Inbox: With MailTales, you can chat with your email inbox!

AI-Powered Email Summaries: MailTales provides concise, AI-generated summaries of your emails

Listen to AI-Generated Stories: MailTales can convert the content of your emails into engaging audio stories.

Sentiment Analysis and Categorization: MailTales' sentiment analysis feature helps you understand the tone of your messages. The app also categorizes emails based on their content.

Code

mtwn105 / MailTales

Turn your Emails into Captivating Stories

MailTales

Make emails fun again!

Demo

Youtube - https://youtu.be/zk88KqHMCR0

Features

Chat Over Your Inbox: With MailTales, you can chat with your email inbox!

AI-Powered Email Summaries: MailTales provides concise, AI-generated summaries of your emails

Listen to AI-Generated Stories: MailTales can convert the content of your emails into engaging audio stories.

Sentiment Analysis and Categorization: MailTales' sentiment analysis feature helps you understand the tone of your messages. The app also categorizes emails based on their content.

Tech Stack

Full Stack: NextJS Backend: NodeJS Deployment: Vercel, Fly.io Database: PostgreSQL Queues: Upstash Kafka Email: Nylas AI: Gemini

Architecture

Nylas:…

View on GitHub

Tech Stack

Full Stack: NextJS
Backend: NodeJS
Deployment: Vercel, Fly.io
Database: PostgreSQL
Queues: Upstash Kafka
Email: Nylas
AI: Gemini

Architecture

Nylas: Nylas is used for authentication of user and fetching their emails from inbox
Gemini: Gemini is used for generating AI responses as well as generating embeddings. Model used for generating responses is gemini-1.5-flash and for embeddings it is text-embedding-004
Postgres: Postgres is used to store user and emails data. @vercel/postgres is used for this.
Upstash Kafka: Upstash Kafka is used for the job of generating email embeddings. Generating email embeddings is done by a NodeJS Backend by consuming messages from Upstash Kafka topic.

Your Journey

Nylas

I have used Nylas to get email inbox and its contents. I have used the Nylas NodeJS SDK for this. Logging in the user was very easy using the SDK. The grantId received is then saved to the DB for further requests. Overall it was really interesting experience in building on top of Nylas features. There are many more features which are still left to explore!

Get Emails

const queryParams: ListMessagesQueryParams = {
    in: ["INBOX"],
    limit: limit,
  }

  if (pageToken && pageToken.length > 0) {
    queryParams.pageToken = pageToken;
  }

  const messages = await nylas.messages.list({
    identifier: grantId!,
    queryParams,
  });

Search Emails

const queryParams: ListMessagesQueryParams = {
    in: ["INBOX"],
    limit: limit,
    searchQueryNative: "query"
  }

  if (pageToken && pageToken.length > 0) {
    queryParams.pageToken = pageToken;
  }

  const messages = await nylas.messages.list({
    identifier: grantId!,
    queryParams,
  });

Upstash Kafka

Used Upstash Kafka's REST API to produce message.

 const response = await fetch(`${process.env.UPSTASH_KAFKA_REST_URL}/produce`, {
      method: 'POST',
      headers: {
        Authorization: `Basic ${process.env.UPSTASH_KAFKA_REST_AUTH}`
      },
      body: JSON.stringify({ topic: "generate-email-embeddings", value: user.id })
    })

Used kafkajs in the backend to consume the messages

const kafka = new Kafka({
  brokers: [`${process.env.UPSTASH_KAFKA_BROKERS}`],
  ssl: true,
  sasl: {
    mechanism: 'scram-sha-256',
    username: `${process.env.UPSTASH_KAFKA_USERNAME}`,
    password: `${process.env.UPSTASH_KAFKA_PASSWORD}`
  },
  logLevel: logLevel.ERROR,
});

const consumer = kafka.consumer({ groupId: 'consumer_group_1' });

const run = async () => {
  await consumer.connect();
  await consumer.subscribe({ topic: 'generate-email-embeddings', fromBeginning: true });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        partition,
        offset: message.offset,
        value: message.value?.toString(),
      });

      const userId = Number(message.value?.toString());

      console.log("Generating email embeddings for user", userId);

      await generateEmailEmbeddings(userId);
    },
  });
};

Gemini & Vercel AI SDK

Used Vercel AI SDK to interact with Gemini models.

Get Streaming Response

export async function getGoogleResponse(body: any, system: string = "", tools: any = null) {
  const result = await streamText({
    model: google('gemini-1.5-flash'),
    system,
    messages: body,
    temperature: 0.0001,
    tools,
  });

  return result;
}

Generate Embeddings

export async function generateEmbeddings(emailStrings: string[]) {
  const { embeddings } = await embedMany({
    model: google.embedding('text-embedding-004'),
    values: emailStrings,
  });

  return embeddings;
}

Also, this was the first time of using NextJS and building a proper RAG agent from the scratch using Vercel AI SDK, so overall a great learning experience.

Future Scope

More deeper insights into mailbox
Integration of AI in alot of parts of the app
Crafting beautiful professional replies.

Conclusion

It was a great experience in building this app using Nylas & Gemini over NextJS. I have learnt alot in the last week while building this project. Thanks to Dev.To as well for providing such platforms for building the projects.

Redis Performance Testing

Amit Wani — Sat, 23 Dec 2023 14:18:54 +0000

Redis Benchmark

Redis Benchmark is a command-line tool that comes bundled with Redis. It is designed to simulate various scenarios and workloads, helping you evaluate the performance of your Redis server. Whether you are a developer fine-tuning your application or an administrator optimizing system resources, Redis Benchmark provides valuable insights into your Redis instance's capabilities.

Installation:

If you have Redis installed on your system, Redis Benchmark is likely available. To use it, open a terminal and simply type:

redis-benchmark

This command will execute the benchmark with default settings, providing you with a quick overview of your Redis server's performance.

Benchmarking Workloads

1. Simple Operations:

Start by testing the basic operations, such as GET and SET commands. These operations are fundamental to Redis, and their performance directly impacts the overall responsiveness of your application. Use the following command to run a simple benchmark:

redis-benchmark -t get,set

This will test the throughput and latency of GET and SET commands, giving you an idea of how quickly Redis can handle these operations.

2. Pipelining:

Pipelining allows you to send multiple commands to Redis in a single request, reducing the number of round trips between the client and server. Test the impact of pipelining on performance with the following command:

redis-benchmark -t set,get -P 16

Adjust the value -P to set the number of commands to pipeline.

3. Parallel Connections:

Simulate real-world scenarios by testing the performance of parallel connections. This helps identify how well your Redis instance scales under increased load:

redis-benchmark -t set,get -c 50 -n 100000

In this example, 50 clients will perform 100,000 SET and GET operations concurrently.

Analyzing Results

Once the benchmark is completed, Redis Benchmark provides detailed output, including throughput, latency, and other performance metrics. Analyze this data to identify potential bottlenecks or areas for improvement.

Documentation

More documentation can be found at Redis benchmark | Redis

Shoppable - E-commerce For Everyone

Amit Wani — Mon, 29 Aug 2022 20:57:00 +0000

Overview of My Submission

Shoppable is A Platform to create an online store for your business. Powered by Redis Stack!. You can visit it here

Submission Category:

My category would be Microservice Mavens as this project has 6 decoupled backend microservices and one frontend

Languages/Technologies Used

NodeJS
Angular
Redis Stack (RedisJSON, RedisSearch, Redis Streams)

Demo

Additional Info

Process Flow

Merchant can register on the platform and create a store
He can add the items to inventory
A customer can visit that merchant store
Customer can purchase items listed by merchant
Merchant will receive the order and will fulfill it
Customer will receive notifications regarding the order

Technical Overview

There are 6 microservices (merchant, inventory, customer, order, update, file-upload)
All servers are based on NodeJS
The client is based on Angular and Angular Material
An nginx is used to route request to various microservices as a reverse proxy
Redis Stack is integrated with the help of Redis OM and Node Redis NodeJS libraries
Redis Stack is hosted on Redis Cloud itself
RedisJSON, RedisSearch and Redis Streams has been used
RedisJSON is used to store JSON documents for various schema such as Merchant, Customer, Product, Order, etc.
RedisSearch is used alongside RedisJSON to search through different repositories
Redis Streams is used for publishing and listening to order updates and send email using Mailjet to the customer regarding the update
Apart from this, Google Cloud Storage is also used to store images of the products uploaded by the merchant

Architecture

Application Screenshots

Merchant Login

Merchant Dashboard

Merchant Inventory

Merchant Add/Edit Product

Merchant Orders

Store Home

Product Page

Cart

Order Details

Email Notification

Link to Code

More details are available on the GitHub repo:

mtwn105 / Shopable

A Platform to create an online store for your business. Powered by Redis Stack!

Shoppable - E-commerce for Everyone

A Platform to create an online store for your business Powered by Redis Stack!

Powered By

Redis Stack (RedisJSON, RedisSearch, Redis Streams) NodeJS Angular

Demo Video

Checkout video here

How it works

Process flow

Merchant can register on the platform and create a store
He can add the items to inventory
A customer can visit that merchant store
Customer can purchase items listed by merchant
Merchant will receive the order and will fulfill it
Customer will receive notifications regarding the order

Technical Overview

There are 6 microservices (merchant, inventory, customer, order, update, file-upload)
All servers are based on NodeJS
The client is based on Angular and Angular Material
An nginx is used to route request to various microservices as a reverse proxy
Redis Stack is integrated with the help of Redis OM and Node Redis NodeJS libraries
Redis Stack is hosted on Redis Cloud itself
…

View on GitHub

Check out Redis OM, client libraries for working with Redis as a multi-model database.
Use RedisInsight to visualize your data in Redis.
Sign up for a free Redis database.

Create a Chatbot using WhatsApp Cloud API

Amit Wani — Mon, 27 Jun 2022 05:02:05 +0000

Hello folks, today in this article, we will be going through how you can create a chatbot and integrate it with WhatsApp Cloud API. We will be also giving this chatbot a brain using an OpenAI model.

Prerequisites

To create a WhatsApp chatbot you will need a few Prerequisites:

A Meta Business Account
A Meta Developer Account
An OpenAI Account

You can create these easily using your existing Facebook account or create a new one.

Once you have all of these ready, we can get started by configuring in Meta Developer account to enable WhatsApp integration.

Configuration

Create Meta App

Logon to https://developers.facebook.com/apps/.

Create a New App.

Choose Business

Choose your Business Manager Account and enter other details

Click on Create App and your app will be created.

Add WhatsApp Integration

Open your app page and you will land on this screen.

Scroll down and find WhatsApp. Click on Set up

WhatsApp Configuration

You will see this screen. Here you have been assigned a Temporary Token and a Test WhatsApp Number. We will use these to send and receive messages on that Test WhatsApp Number. In future, if you want, you can connect real numbers as well which will be a proper WhatsApp Business account.

You have to configure a few recipient WhatsApp numbers for testing, up to 5 numbers can be added.

The Phone Number ID shown will be used later to use WhatsApp Cloud API, so keep it handy.

Authentication Token

The final piece of configuration is generation of an authentication token which will be used while communicating with Meta APIs.

Log on to https://business.facebook.com/ and open your Business Settings.

Navigate to the System Users section from left-hand menu

You can add a user or if it already exists, click on Generate New Token

Select the app you created earlier and check whatsapp_business_management and whatsapp_business_messaging permissions and then copy this token.

Application

So we will develop our application using NodeJS and Express

We will install basic dependencies like express morgan helmet cors axios dotenv

We will create two API routes one for verification of the webhook (this will be only used once) and one to receive messages/updates from WhatsApp

GET /api/webhook - To verify this webhook
POST /api/webhook - To receive messages from WhatsApp

So, we can configure a webhook in WhatsApp so that every time a message or update comes we will get notified of that and we can respond accordingly.

This GET API of /api/webhook will be used for verification of this webhook where WhatsApp will send a token and a couple of parameters and we will verify if all those matches whatever is configured on our server. It is called a Verification Token. We create a random long verification token and store it in the environment variables of our server.

This will be our API route for GET /api/webhook

app.get("/api/webhook", (req, res) => {
  const VERIFY_TOKEN = process.env.VERIFY_TOKEN;

  // Parse params from the webhook verification request
  let mode = req.query["hub.mode"];
  let token = req.query["hub.verify_token"];
  let challenge = req.query["hub.challenge"];

  // Check if a token and mode were sent
  if (!mode || !token) {
    return res.status(403).send({ error: "Missing mode or token" });
  }

  // Check the mode and token sent are correct
  if (mode === "subscribe" && token === VERIFY_TOKEN) {
    // Respond with 200 OK and challenge token from the request
    console.log("WEBHOOK_VERIFIED");
    return res.status(200).send(challenge);
  } else {
    // Responds with '403 Forbidden' if verify tokens do not match
    return res.sendStatus(403);
  }
});

To verify our webhook we go to WhatsApp Configuration from the Meta App page

Add the Webhook API URL and the Verification token and click on Verify and Save. You can use a proxy like ngrok if you are still in development. Webhook URL must be an HTTPS one with no Self-signed certificate in the chain.

In Webhook fields, Subscribe to messages

Now for the receiving messages webhook, POST /api/webhook

app.post("/api/webhook", async (req, res) => {
  try {
    // Parse the request body from the POST
    const body = req.body;

    // Check the Incoming webhook message
    console.log("Incoming webhook: " + JSON.stringify(body));

    // Validate the webhook
    if (req.body.object) {
      // Handle the event
      if (req.body.object === "whatsapp_business_account") {
        const entry = req.body.entry[0];

        // Handle the message
        if (entry.changes) {
          for (const change of entry.changes) {
            if (
              change.value &&
              change.field === "messages" &&
              change.value.contacts &&
              change.value.messages
            ) {
              // Handle the value
              const value = change.value;

              const userName = value.contacts[0].profile.name;

              const messages = value.messages;

              // Handle messages
              for (const message of messages) {
                if (
                  message.type === "text" &&
                  message.text &&
                  message.text.body
                ) {
                  const waid = message.from;
                  const text = message.text.body;
                  const msgId = message.id;
                  console.log(
                    "Message from " + waid + " - " + userName + ": " + text
                  );
                }
              }
            }
          }
        }
      }
    }
  } catch (err) {
    console.error("Error occured " + err);
  }
});

This is how we will extract messages from the incoming webhook when the user sends a Text message on the configured number. We can use this text message and then reply accordingly.

For our application, we pass this as a prompt to an OpenAPI model and then get a response from the AI and respond to the user. We are using openai package for this.

For responding to the user, we will use the https://graph.facebook.com/v13.0/<phonenumberid>/messages API endpoint with the authentication token we collected earlier in Headers like below:

  await axios.post(
    process.env.WHATSAPP_SEND_MESSAGE_API,
    {
      messaging_product: "whatsapp",
      recipient_type: "individual",
      to: waid,
      type: "text",
      text: {
        preview_url: false,
        body: reply,
      },
    },
    {
      headers: {
        Authorization: "Bearer " + process.env.WHATSAPP_TOKEN,
      },
    }
  );

By this response will be send to the user.

Conclusion

This is how we will use WhatsApp Cloud API and integrate WhatsApp with your application. There are endless possibilities since Meta recently made this API available to all, which was earlier only available to business owners or BSP (Business Service Partners).

Documentation: https://developers.facebook.com/docs/whatsapp/cloud-api/reference

How to speed up a Web App loading time by 10x faster!

Amit Wani — Mon, 16 May 2022 05:40:38 +0000

Hello folks, Today I will discuss, how I speeded up the loading time of my Web App built using Angular or any SPA (Single Page Application).

Before diving into the article, let's first understand how a Single Page Application (SPA) works be it developed using any web framework (Angular, React, etc.)

Single Page Application

The single-page application is a web application that interacts with the user by dynamically rewriting the current page, rather than loading entire new pages from the server.

In SPA, when the browser makes the first request to the server, the server sends back the index.html file. and that's basically it. that's the only time an HTML file is served. the HTML file has a script tag for the .js file which is going to take control of the index.html page. All subsequent calls return just the data, usually in JSON format. the application uses this JSON data to update the page dynamically. However, the page never reloads.

The client (and not the server) handles the job of transforming data to HTML once the application has started. basically, most of the modern SPA frameworks like Angular, React, and Vue has a templating engine of sorts running in your browser to generate the HTML.

The Problem

Now let's come to the problem or one of the drawbacks of using a SPA. Since the initial index.html is very lightweight, most hard work is being done by the subsequent Javascript files fetched from the server and the client executing that JS in the browser.

As our project starts to become complex with a lot of components and dependencies, the bundled JS file sizes get bigger with time.

Here you can see this is a bundled JS generated by the building of an Angular App which is more than 1 MB 🤯. This file has to get loaded in browsers which takes a good amount of time and the user has to wait for the first painting of the webpage.

Though React is much lighter weight, one can definitely reduce this size by compressing these JS files. That's what we will do next.

The Solution

We will discuss one of the ways to deal with this problem, which is to compress the bundle JS files and serve their compressed versions.

Compress Bundled JS files

In Angular, to generate a compressed version of these JS files, we can use gzipper to compress files to gzip format.

We add a dev dependency in package.json for gzipper.

  "devDependencies": {
   ...
    "gzipper": "^7.0.0",
   ...
  }

We also update our build script in package.json to gzip the files after building using gzipper.

"build": "ng build && gzipper compress --verbose ./dist/client/",

Now when we build our application using npm run build we will get two versions of that JS file. One is a non-compressed one and one is compressed with the .js.gz extension.

As you can see the gzipped version of the Javascript file is almost 4 times smaller (1.2 MB to 300 KB)
Along with JS, all HTML, CSS and asset files are also gzipped reducing their sizes by more than 50%.

Nginx

Now comes the role of Nginx, we need something to serve these compressed versions of Javascript files, we will be using Nginx for this.

In the nginx.conf - The configuration file for Nginx we will these four lines to the server object

    gzip on;
    gzip_min_length 1000;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;

This will enable the gzip compression and when serving the files, if the gzip compression is supported by the browser, Nginx will serve the smaller gzip version thus reducing the loading time by up to 10X.

Conclusion

This marks the end of this article. This was of the many ways to speed up the loading time of a Single Page Application. As your app gets to production-grade at scale, this will be very useful.

For React, you can refer to this article to see how to generate build files in gzipped version.

If you like this article, keep in touch with me on my socials or support me if you want.
If you have any feedback, do let me know in the comments.

Till next time, Peace.!

Read your Favorite Podcasts - PodText

Amit Wani — Mon, 11 Apr 2022 11:58:59 +0000

Hello folks, this post is regarding my submission for Deepgram Hackathon.

Idea

So many of us wants to listen to various amazing podcasts out there, but due to time constraints we are not able to give so much time and attention to listen to them. What if its possible to read the transcript of the podcast in a text version, that's struck my mind.
I decided to create an web app to Read the podcasts - PodText.

Overview of My Submission

PodText will allow us to browse through Top Podcasts of various genres and read transcripts for each episode of a podcast.
Podcast metadata and audio has been fetched using Podcast API by Listennotes. Audio to Text transcription is done using Deepgram's API.

Link: podtext.amitwani.dev

Note: Currently, Podcasts are served from Mock server of Listennotes to avoid exhaustion of API quota

Submission Category

Accessibility Advocates

Tech Stack

NextJS
NextUI
Docker
Railway.app
Listennotes
Deepgram

Details

I have created a NextJS app along with NextUI as a UI library.
Created pages for showing Genres (genres/[id]), Podcasts (podcasts/[id]), Episodes (episodes/[id]) details.
Created an API route api/transcribe which is used to transcribe the audio of Podcast to text using Deepgram. Podcast audio will be fetched from Listennotes' CDN on the fly itself.

Link to Code on GitHub

mtwn105 / podtext

View Text Versions of your favorite podcasts!

Podtext

Read your favorite podcasts

Link: podtext.amitwani.dev

Idea

So many of us wants to listen to various amazing podcasts out there, but due to time constraints we are not able to give so much time and attention to listen to them. What if its possible to read the transcript of the podcast in a text version, that's struck my mind. I decided to create an web app to Read the podcasts - PodText.

Tech Stack

NextJS
NextUI
Docker
Railway.app
Listennotes
Deepgram

View on GitHub

Screenshots/Demo

Future Improvements

There's always room for improvement, here are some things which can be added or done better in this

Better UI/UX
Ability to Play Podcast
Ability to search podcasts
Caching podcast transcriptions
Improved SEO

Conclusion

I have learned a lot while making this project. This has been my first proper project on NextJS, still much improvement is needed on this front. Had fun with using Deepgram's API very easy to use and straight forward.
Looking forward to everyone's submissions and more such hackathons.

Till next time, peace!

Manage your GitHub Starred Repositories

Amit Wani — Tue, 05 Apr 2022 04:22:26 +0000

Hello folks, this is a quick article about how you can manage your GitHub's starred repositories.

You must have come across hundreds of awesome GitHub repositories but the problem is it's very hard to find the one you want at that moment in time.

For this, we usually star our GitHub repositories, but we forget about them.

To solve this, I came across this feature in GitHub (still in beta), where you can create lists to organize your starred repositories.

To do this you just click on the arrow icon next to Star

You will be presented with a dropdown to select an existing list or create a new list

Once you select a list, it will be saved in that list.
You can revisit it by going to Your Stars

And you can see all your lists, you can go through them, search in them, etc.

That's it for this article. Hope you found it useful. Share your thoughts down below.

Till next time, peace.

Journal CLI - Write Journal in Notion using CLI

Amit Wani — Sun, 20 Mar 2022 07:03:22 +0000

Hello folks, today I tried to play around with Notion API and NodeJS to create a CLI which can write Journal for you and store it in a Notion Database.

These days, there is a lot of stuff going on in our lives and we want to focus on the ones which matter most to us. One way to solve this is by writing a journal and sorting this chaotic mess.

Using Notion, we can create beautiful interactive notes and much more. Using the power of notion and efficiency of CLI, I developed this quick way, you can start journaling, by answering just 5 questions at the end of the day and the Journal will be saved in a database in Notion.!

NPM Link - https://www.npmjs.com/package/journal-notion-cli

Duplicate Template

To Get Started Duplicate this Notion Template

Installation

Then Install Journal CLI from NPM using
npm i -g journal-notion-cli

Connect to Notion

Connect to the notion by running journal connect

Write Journal

Write a journal every day using journal write and answer a few questions.

Demo

Source Code

If you like it and want to know more, do check out the GitHub repo at

mtwn105 / journal-cli

A CLI to write journal for you and save it to a Notion database!

Journal CLI

NPM Link

A CLI to write journal for you and save it to a Notion database!

Demo

Installation

Run npm i -g journal-notion-cli to install the cli.

Start using by journal connect

Help Command journal --help

Duplicate this page to your notion

https://amit-wani.notion.site/amit-wani/Journal-a19377f6b17d47768e03d4ffe62a030d

Connect to notion `journal connect`

Write journal every day using `journal write`

Buy me a coffee

coffee

View on GitHub

If you like this tool, do support me by using the "Buy me a coffee" button, and keep reading my content by subscribing to newsletters. Till next time, peace.

Buy me a coffee

https://www.buymeacoffee.com/mtwn105

Correlation Analysis Between Stocks using Python

Amit Wani — Thu, 17 Mar 2022 05:43:22 +0000

Hello everyone, this article is about the tutorial on how to do the correlation analysis of Indian Bank stocks. We will compare major private and public banking stocks listed on the stock exchange.

I will be using Python for data analysis for this one.

Install Dependencies

First, we need to install the required dependencies

pip install pandas
pip install numpy
pip install yfinance
pip install seaborn

Import Dependencies

Now, we will import these required dependencies into our code.

import pandas as pd
import numpy as np
import yfinance as yf
import seaborn as sns

Fetching Stock Price Data

I have taken the below stocks for correlation analysis and stored them in a list. You can see I have added their ticker symbols ending with ".NS" these are picked up from Yahoo Finance since we will be fetching data from Yahoo Finance.

tickers = ['HDFCBANK.NS', 'ICICIBANK.NS', 'AXISBANK.NS', 'SBIN.NS', 'KOTAKBANK.NS', 'INDUSINDBK.NS', 'BANKBARODA.NS']

Now we will fetch data for these stocks from Yahoo Finance using yfinance. I am fetching data of 10 years with auto_adjust=True for getting close price data adjusted for corporate actions. Then storing them into another list called data.

data = []

for ticker in tickers:
    ytick = yf.Ticker(ticker)
    df = ytick.history(period="10y", auto_adjust=True, threads = True)
    df = df[df['Close'] > 0]

    data.append(df)

Manipulating Data

Now, we have a list with multiple panda data frames. For creating a correlation matrix, we need to merge them into a single data frame with only the closing price data of all the stocks.

For this to achieve, we will first use the zip function to zip all the data frames and then use a dictionary to associate the data frame with the stock using the dict function. Finally, we will use the concat function of pandas to concatenate all these data frames from dict with appropriate columns.

mergedDf = pd.concat(dict(zip(tickers, data)), axis=1)

The new data frame should look something like this

Now since here we have multiple columns of data, but we don't need it, that's why we will only keep Close column data.
We will get the Level 1 values and filter only with the Close column.

closeDf = mergedDf.loc[:,mergedDf.columns.get_level_values(1).isin(['Close'])]

It will now look like this

But we also don't need these two levels of headers now. So we can drop Level 1 and only keep symbol tickers as the column header.

closeDf.columns = closeDf.columns.droplevel(1)

It will now look like this

Correlation Analysis

Now we have close price data for all stocks in a single data frame. We now need to calculate daily returns so that we can then create a correlation matrix from that data. But, simple pct_change returns won't work as every stock has a different base as well, so we will need to calculate daily log-returns of all stocks. We can do this by doing the following:

logretDf = np.log(closeDf.pct_change() + 1)

It will now look like this

Now we have a data frame from which we can get various insights regarding correlation, standard variation etc.
We can show the correlation matrix by

logretDf.corr()

You will see something like this

This shows us the correlation percentage of all stocks with each other over the past 10 years of data.
To visualize this better we can use seaborn to plot it in a graphical fashion like below

sns.heatmap(logretDf.corr())

You will see something like this

This will show us a much easier view of the correlation between various banking stocks.

If you got this far, means now you can do correlation analysis of any number of stocks. If you like what I write, please do share it on social media. Till next time, peace.!