DEV Community: Madhurima Rawat

🚀 Project Ideas for GitHub AI, ML and Full Stack

Madhurima Rawat — Fri, 27 Feb 2026 09:08:52 +0000

Contains AI, ML and Full Stack projects with links and also unique ideas without GitHub references. You can pick from here, implement them in your own way, and then add your repository link here later.

So first of all, the reason behind this gist. I thought about something we all have in common as developers, the constant struggle to find project ideas. So for that, I made this gist. It contains all my ideas for some cool, slightly crazy projects.

The goal of this is to solve one common developer problem: finding strong, creative and meaningful project ideas. Star it if you find it useful and feel free to add your ideas too. I would love to include more unique and slightly crazy ideas, and if you have a solid project with a repository and clean code, I will happily add that as well.

First I will add unique project ideas. Then I will add projects with GitHub reference links.

📌 Navigation

🟣 Unique Full Stack
🟢 Unique AI / ML
🔵 Unique Full Stack + AI Combo
📂 Projects With GitHub Links

🟣 UNIQUE FULL STACK

These are ideas I came up with myself. You can choose any of these and build them. They are unique, creative and can really stand out. If implemented nicely, you can even use them as placement projects.

1. Personal Pet Universe

Description: A complete digital ecosystem to manage and gamify your pets' lives.
Features: Feeding schedule + growth charts + personality traits + gamified leveling + login system

2. Digital Memory Capsule Vault

Description: A platform to send messages to your future self.
Features: Time locked letters + mood tagging + timeline visualization

3. Skill Evolution Tracker

Description: Track your coding and learning journey visually.
Features: DSA tracker + GitHub activity integration + analytics dashboard

4. Alternate Reality Resume

Description: A story based interactive resume experience.
Features: Story mode + achievement unlocks + recruiter preview mode

5. Emotional Journal with Dynamic Themes

Description: A journaling platform that adapts visually to your emotions.
Features: Mood based themes + automatic color switching + emotional trend graphs

6. DIY Distraction Free Idea Hub

Description: A distraction free DIY website that fetches simple and easy to build ideas from YouTube and allows users to play the URL directly inside the platform.
Features: Curated beginner DIY bookmarks (DIY book holder + DIY desk organizer + calendar + birthday cards) + highlighted best videos + embedded video player + simple project size suggestion system + minimal no distraction UI

7. Meme Redemption Website

Description: A website that shows trending memes from different months and years without storing images locally, dynamically loading them from external sources.
Features: Monthly and yearly meme archive + live dynamic fetching + trending detection system + no image storage on server + real time updates

8. Barbie Theme Portfolio

Description: A Barbie themed portfolio website where skills are displayed as a dressing table, projects as a closet, and education as pasted pictures.
Features: Dressing table skill layout + closet styled project section + scrapbook style education section + aesthetic pink theme UI + animated transitions

🟢 UNIQUE AI / ML

6. Personality Predictor from Writing

Description: Predict personality traits based on writing patterns.
Tech: NLP + Text Classification
Features: Paragraph input + personality trait prediction

7. Text Based Lie Probability Detector

Description: Analyze linguistic cues to estimate deception probability.
Tech: NLP + Feature Engineering
Features: Pattern detection + probability score output

8. Dream Interpreter AI

Description: Classify psychological themes from dream descriptions.
Tech: NLP + Classification Models
Features: Theme tagging + emotional insight summary

9. AI Study Planner Optimizer

Description: Smart timetable generator with burnout prediction.
Tech: ML + Optimization Algorithms
Features: Exam timeline input + schedule generation + stress prediction

10. Behavior Pattern Analyzer

Description: Predict productivity score from daily routine inputs.
Tech: ML + Data Modeling
Features: Routine analysis + improvement suggestions + scoring system

11. TraumaDump AI

Description: An AI based emotional release platform where users can express trauma through text, image input, or handwriting, interacting with a calm monk or sage style illustration instead of a human.
Tech: NLP + Computer Vision + Handwriting Analysis + Sentiment Analysis
Features: Text based emotional input + image upload for emotional context + handwriting analysis + monk style AI interaction interface + emotional tone detection + supportive coping suggestions + safe and private interaction system

🔵 UNIQUE FULL STACK + AI COMBO

1. AI Pet Personality Analyzer

Description: Predict pet behavior and health risks using AI.
Tech: Full Stack + ML Models
Features: Behavior prediction + training suggestions + analytics dashboard

2. AI Memory Reflection Platform

Description: Intelligent journaling with automated reflection insights.
Tech: NLP + Full Stack
Features: Entry summarization + emotional trend detection + visualization

3. Code Performance Analyzer

Description: Analyze uploaded code and suggest optimizations.
Tech: Static Code Analysis + ML
Features: Complexity detection + performance suggestions

4. AI Career Path Recommender

Description: Suggest career paths based on skills and interests.
Tech: ML + Recommendation Systems
Features: Role suggestions + skill gap analysis + roadmap generation

5. Social Media Reality Analyzer

Description: Detect emotional and psychological patterns in posts.
Tech: NLP + Sentiment Analysis
Features: Toxic positivity detection + emotional tone analysis

6. AI Virtual Family House

Description: A full stack + AI platform that allows users to create virtual family members with customizable appearance and predefined dialogues, designed especially for lonely people or hostelers to experience interactive companionship.
Tech: Full Stack + NLP + Generative AI + Personalization Models
Features: Create and customize family member appearance + predefined and dynamic AI dialogues + interactive emotional conversations + trauma dump interaction mode + religion selection option + festival and celebration simulation based on selected religion + memory based conversation continuity

⭐ Special Suggestion

If you feel there are too many projects to choose from, you could try building the Pet Recommender. It’s an idea I’ve had for a long time and it has huge potential. It gives you space to explore creative UI and UX design, experiment with color palettes, and integrate AI or ML features in a meaningful way.

It can become a very strong Full Stack plus AI combination project. In today’s GPT driven era, unique and slightly crazy ideas usually stand out more. That’s why I suggested this one.

If you choose a project that already has a GitHub reference, make sure you add your own unique improvements. Build something clearly different from the original version so it truly reflects your skills and creativity.

Best of luck with your project 😊

📂 Projects With GitHub Links

🖥 Full Stack

1. Personal Expense Tracker

Description: A simple expense management system with visual analytics.
Tech: MERN / Django + React / Flask
Features: Add + edit + delete expenses + monthly summary + pie chart visualization
Link: https://github.com/rahuldkjain/expense-tracker

2. Blog Website with Authentication

Description: A blogging platform with user authentication and admin control.
Tech: Node.js + Express + MongoDB / Django
Features: Login and signup + create and edit posts + admin panel
Link: https://github.com/bezkoder/node-js-mongodb-auth-jwt

3. Job Portal Website

Description: A recruitment platform connecting recruiters and candidates.
Tech: MERN Stack
Features: Recruiter dashboard + resume upload + job search and filters
Link: https://github.com/bradtraversy/proshop-v2

4. Real Time Chat App

Description: A live messaging system with real time communication.
Tech: React + Node + Socket.io
Features: Private messaging + group chat + user status
Link: https://github.com/socketio/chat-example

5. Production Level E commerce Platform

Description: A scalable e commerce system with payment integration.
Tech: MERN + Stripe
Features: Cart + payment gateway + admin analytics + order tracking
Link: https://github.com/mayankshubham/MERN-Ecommerce

6. SaaS Project Management Tool

Description: A collaborative Kanban style project management system.
Features: Drag and drop boards + teams + deadlines + role based access
Link: https://github.com/atulmy/crater

🤖 AI / ML

1. Movie Recommendation System

Description: Content based recommendation engine for movies.
Tech: Python + Pandas + Scikit learn
Link: https://github.com/nikhilkumarsingh/Movie-Recommender-System

2. Spam Email Classifier

Description: Classifies emails as spam or not spam.
Tech: NLP + Naive Bayes
Link: https://github.com/udacity/machine-learning

3. Fake News Detection

Description: Detect fake vs real news using NLP models.
Tech: TF IDF + Logistic Regression
Link: https://github.com/GeorgeMcIntire/fake_real_news_dataset

4. Image Classification CNN

Description: Deep learning image classification model.
Tech: TensorFlow or PyTorch
Link: https://github.com/keras-team/keras-io

5. Stock Price Prediction LSTM

Description: Time series forecasting using deep learning.
Tech: LSTM + Time Series
Link: https://github.com/llSourcell/Stock_Market_Prediction

6. AI Resume Screening System

Description: Match resumes to job descriptions using similarity scoring.
Tech: NLP + Cosine Similarity
Link: https://github.com/srbhr/Resume-Matcher

7. Sentiment Analysis Web App

Description: Web app that predicts sentiment from text.
Tech: Flask + React + ML
Link: https://github.com/llSourcell/sentiment_analysis

8. AI Chatbot UI

Description: Frontend and backend chat system with AI interaction.
Link: https://github.com/mckaywrigley/chatbot-ui

9. ML Model Playground Internship Project

Description: Interactive model training platform with metrics and graphs.
Tech: Streamlit + ML Models
Link: https://github.com/madhurimarawat/ML-Model-Datasets-Using-Streamlits

🚀 Final Note

I had the idea of this for a long time as whenever I am free I keep thinking of projects I can make and how I can make them. So finally, I decided to create a gist to keep all these ideas in one place: https://gist.github.com/madhurimarawat/98294b842b05aaae00c82ac9335f1b9e you can visit it anytime.

The GitHub section is there so if you want to quickly see the code or are in a hurry, you can just fork the project, run it, and make changes.

If you want to keep in touch or discuss more details about implementation or guidance on these ideas, reach me at:
💼 LinkedIn
📧 Email

Honestly, it took me a lot of time to format and write all this, so do star if you like it. I made this gist public so that more people can benefit from it.

Build something you are proud of.

Consistency > Complexity.

Now go build something crazy good 😊

💡 Bonus: If you have any crazy idea or a good project, drop it in the comments below, I would love to see it and maybe include it in the gist!

💡 Stand Out: Creative Portfolio Concepts for Developers 🎨

Madhurima Rawat — Mon, 21 Jul 2025 04:48:00 +0000

Hi everyone, 👋

I’ve been brainstorming some fun and meaningful ways to design a portfolio that really feels like you. If you want something more personal, visual, and story-driven than a standard template, this post is for you.

About the Cover Image:
The illustration represents the journey of building a portfolio. It starts with a variety of ideas 🎨💡 (represented by the palette and bulb icons), followed by the development phase ⚙️ (gear icon, technologies, and tools). Then comes the identity card icon, symbolizing our digital identity. The network icon shows how a strong portfolio can attract professional connections, and the final two people talking represent collaboration and meaningful interaction 🤝.

Take a look at these portfolio ideas I love, complete with color suggestions, layout tips, and online tools to help you bring them to life.

📚 Table of Contents

Theme	Description
🌤️ Seasons Dynamic	Portfolio theme changes with the seasons
🏠 House Layout	Each room = a portfolio section
🗺️ Roadmap	Your learning journey in map style
🎬 Story Format	A cinematic storytelling experience
🔣 Icon-Based	Minimalist and icon-focused
🛒 Smart Mart	Store-themed skill showcase
📊 Wordcloud Skills	Wordcloud with skill percentage
🤖 Chatbot Portfolio	Chatbot that knows your portfolio
📖 Book Format	Chapters and bookmarks for each section
👗 Wardrobe Layout	Skills = outfits and styles
🌱 Ecosystem Design	Food web of your skillset
🌌 Solar System Theme	Planets and moons as skills

🌤️ Seasons Dynamic

Let your site shift colors or themes depending on the current season. It keeps your portfolio fresh and interesting.

Suggested Colors

Spring: #b2f2bb, #f0fff4
Summer: #ffe066, #ff8787
Monsoon: #4dabf7, #d0ebff
Winter: #dee2e6, #adb5bd

Tools & Resources

JS Date object + CSS classes
Coolors.co for seasonal palettes
CSS Variables

Resources:
I’ve added seasonal themes to my own portfolio. Just go to the Theme section and try the dropdown to explore each look.
If you liked this idea, feel free to use the color styles or design concepts in your own portfolio.

Portfolio Website

Explore my personal portfolio, showcasing education, skills, projects, and achievements.

madhurimarawat.github.io

🏠 House Layout

Each room is a section: Projects in the study, About Me in the living room, Skills in the workshop.
Great for storytelling with a floorplan feel.

Ideas

Create a map-style homepage
Hover to zoom into each room

Inspiration

StoryMapJS
Use SVG + tooltips for interaction

🗺️ Roadmap

Your tech journey shown as a path or roadmap — from where you started to your future goals.

Design Tips

Vertical scroll timeline or curved roadmap
Mark milestones with icons

Tools

TimelineJS
Canva or Figma for custom SVG design

🎬 Story Format

Turn your dev journey into a story. Chapters for each year, goal, or challenge.

Sections

Act I: Learning Phase
Act II: Building Projects
Act III: Real-world Experience

Use

Scroll effects (e.g., AOS)
Typewriter animations

🔣 Icon Based Layout

Minimal text, icon-heavy. One icon = one skill/project/tool.

Fonts & Icons

Font Awesome
Heroicons
Use tooltips to describe icons

Design Color Scheme

Background: #f8f9fa
Icons: #495057, #6c757d

🛒 Smart Mart Style

Think digital mart where each section is a skill category or tech aisle.

Sections

Aisle 1: Frontend Tools
Aisle 2: Backend
Aisle 3: Projects

Design Tools

CSS Grid
Store-style UI: UI Bakery, Squoosh-inspired layouts

📊 Wordcloud Skills

Showcase tech stack with a wordcloud, size reflects confidence or usage.

Tools

WordArt
d3-cloud
Style by years of experience

🤖 Personalized Chatbot

Build a chatbot that introduces your work, answers FAQs, and links to your resume/projects.

Tools to Try

Tidio or BotPress
Custom JS chatbot with hardcoded answers

Tip
Add buttons for “Show Resume,” “Explore Projects,” etc.

📖 Book Style Layout

Turn your portfolio into a digital book. Flip pages or click tabs to explore sections.

Tools

turn.js for page-flip effects
Book-style font: Georgia, Libre Baskerville

👗 Wardrobe Layout

Your skills as wardrobe elements. Choose an "outfit" to see your tech stack mix.

Visuals

Categories like “Frontend Fit”, “Fullstack Casual”
Clothes rack layout with hover cards

Fun Addition
Include personal flair — your “style” in code or design.

🌱 Ecosystem Portfolio

Imagine your skillset as an ecosystem — like a food web. Frontend depends on design, backend powers logic, etc.

Design Ideas

Connect skills using arrows or branches
Use circular node maps

Tools

🌌 Solar System Theme

Planets represent skills, moons for tools or projects. Sun = your core expertise.

Colors

Background: #0b0c10
Planets: gradient or soft neons

Inspiration

Three.js for 3D view
Solar system CSS demo

Resources:
I've created an animated solar system that you can explore for design inspiration. If you're choosing this theme, feel free to reuse or adapt some of the CSS styles and animations.

Solar System

madhurimarawat.github.io

✨ Wrapping Up

These ideas are just a starting point. You can mix and match, maybe a wardrobe-themed portfolio that changes with seasons, or a book-style layout with a chatbot guide. Your portfolio should be more than a template. Let it reflect your creativity, growth, and story.

Build something that's uniquely you.

If you try any of these or have more cool ideas, I’d love to see them! Drop a link or thought in the comments below 🌟

🧰 Developer Resources Hub: Essential Tools and Tech in One Place 📚

Madhurima Rawat — Wed, 16 Jul 2025 07:26:17 +0000

🚀 Introducing Developer Resources Hub

Hey everyone! I hope your code is running smooth like butter 🧈. It's been a while since I posted, so first of all, a big hello to you all 👋

Today I'm excited to share my latest GitHub repo with you:

madhurimarawat / Developer-Resources-Hub

A comprehensive collection of valuable resources for developers, covering job preparation, programming, frontend, backend, IoT, databases, and more.

Developer-Resources-Hub

A comprehensive collection of valuable resources for developers, covering job preparation, programming, frontend, backend, IoT, databases, and more.

Developer Resources Hub brings together well-structured guides, tutorials, and reference materials across key domains in software development:

💼 Job Preparation – Interview questions, coding challenges, and career-building resources
💻 Programming – Language-specific tips, examples, and foundational concepts
🌐 Frontend & Backend Development – Framework insights, tooling, and practical workflows
📡 IoT (Internet of Things) – Architectures, protocols, and application-level references
🗄️ Databases – SQL/NoSQL fundamentals, query optimization, and schema design

📌 Table of Contents

View on GitHub

It's a one-stop place for developers. Whether you're just starting out or already building cool stuff, this collection brings together useful resources on job prep, programming, frontend, backend, databases, AI/ML, IoT, and a lot more.

All current resources are listed here, and new ones are added continuously.

📌 Table of Contents

🔗 Useful Links
📱 Notes and To-do Lists
📚 Free Courses
📘 Books & References
🧠 AI / ML / DS Resources
- 📂 GitHub Repositories
- 📄 PDF Files
- 🔗 AI / ML / DS Reference Links
🎨 AI Art Tools
📲 App Links
🧠 Aptitude & Logical Reasoning
⛓️ Blockchain
📘 Coding Questions
🎨 CSS Formatter
🤖 Custom GPTs
🗄️ Databases
- 🔗 General Links
- 🐬 MariaDB
- 🍃 MongoDB
- 🧮 MySQL
- 🧾 Oracle
- 📊 DS Resources
- 🧑‍💻 Full Stack Development
- 🛠️ Backend
- 🎨 Frontend
- 📚 Books
- 📓 Notes
🧾 Graphic Designing
📡 IoT Resources
🐧 Linux Guides
🧠 LLM (Large Language Models)
📊 Power BI
🧩 Go Programming
📱 Kotlin Programming
🐍 Python Programming
🦀 Rust Programming
🌐 Study Resources
📊 SAS Programming
🧾 HTML to PDF Conversion

🔗 Useful Links

🌐 Online Presence

Boost your visibility and career growth with these essential resources:

🧑‍💼 Mastering LinkedIn – Tips for Building Your Profile and Networking
Learn how to optimize your LinkedIn profile and expand your professional network.
📝 Brag Document – Ace Interviews and Land a Job
Use a brag document to track accomplishments and boost your interview confidence.

✍️ Blogging Platforms

Write articles, share ideas, and grow your developer presence online:

📰 Dev.to
A friendly community of devs writing about code, careers, and life.
✒️ Medium
A global platform for sharing tech blogs, tutorials, and opinions.
🚀 Hashnode
Start your own dev blog on a custom domain with no setup hassle.
📘 Substack
Publish articles or newsletters and build your audience with ease.
📖 Tealfeed
Discover and write insightful content around tech, startups, and growth.
🧵 Bear Blog
A minimalist, privacy-focused blogging platform for developers.

📄 PDF Tools

Convert, scan, and edit PDFs with these handy tools:

✂️ Smallpdf – Crop Tool Crop and resize PDF pages online — quick and user-friendly.

📣 Learning & Community Updates

Prompt Engineering & Tech Updates WhatsApp Channel Stay updated with the latest in prompt engineering, AI tools, and tech trends.

💼 Selling Projects & Digital Products

🛒 Gumroad
A professional platform that empowers creators to monetize their knowledge and digital products, whether it’s code, design assets, or eBooks.
Just start with what you know, share it with the world, and get paid. Simple and effective.
🚀 Product Hunt
A launch platform for makers to showcase their products and gain traction from a tech-savvy audience.
Perfect for getting feedback, early users, and visibility for your digital creations.

⚙️ Utility

🎓 Learning Apps

📚 Edcoupon – Free Udemy Courses Get access to daily updated free Udemy courses in tech, business, design, and more.

✍️ Signature Tools

🖋️ Signature Scanner Scan, create, and digitize your handwritten signature with ease.

🎬 Video Tools

🎞️ AllCut – All-in-One Video Editor
Edit videos, apply effects, cut, merge, and more — all from one app.
📹 Video Compressor & Editor
Compress video size without losing quality and edit videos on the go.

🎨 Image Tools

🧼 Remove.bg – Background Remover Remove image backgrounds instantly using AI.

📥 Download Tools

⬇️ Download4.cc – YouTube Video Downloader
Download YouTube videos quickly and easily without installing software.
🎧 SaveFrom.net – YT Downloader
Another fast and simple option to download YouTube content.

📱 Notes and To-do Lists

✅ To-do List – Daily Planner: Android | iOS

A clean and simple task scheduler and planner for daily productivity.
🗒️ EasyNotes – Notes & Reminders: Android | iOS

Colorful notepad for text and photo notes.
🐱 Meow To-do List (Android)

A fun and quirky task manager app.
🧠 Mind Notes – Smart Notepad (Android)

A visually appealing app for managing sticky notes and tasks.
🗃️ Anki Flashcards: Android | iOS

A powerful, spaced-repetition flashcard app ideal for long-term learning and memory retention.*

📚 Free Courses

🎓 Simplilearn Dashboard – Access your free courses and certifications.
📘 Alison – Explore free online diploma and certificate courses.
💡 Coursera – University-level courses from top institutions.
🧑‍🏫 Udemy – Thousands of affordable courses in tech and more.
🌐 Simplilearn – Professional certificate programs and bootcamps.

📘 Books & References

📖 PDFDrive – Free downloadable eBooks in various domains.
🌊 OceanOfPDF – Library of novels, textbooks, and learning PDFs.
🎯 GoalKicker – Free programming books with detailed guides.

🧠 AI / ML / DS Resources

📂 GitHub Repositories

📊 TheAlgorithms/Python – Collection of algorithms implemented in Python.
🏗️ System Design Primer – Learn system design with real-world use cases.
📄 Machine Learning Cheat Sheet – Quick ML concepts and formulas.
💯 100 Days of ML Code – A step-by-step roadmap to learning ML.
🎯 ML Interview Questions – Comprehensive ML interview prep.
📚 RGPV BTech Notes – Engineering study notes and references.

📄 PDF Files

These PDFs are located in the AI ML DS/AI ML DS/ folder.

📄 AI ML DS.pdf – Overview of AI, ML, and Data Science fields.
📄 AI Vs ML Vs DS FOR INDUSTRY.pdf – Comparison of AI, ML, and DS in practical applications.
📄 Data Science Interview.pdf – Key questions and answers for DS job interviews.
📄 Difference Between DS AI AND DL.pdf – Distinguishes Data Science, AI, and Deep Learning.
📄 Difference Between DS AI AND ML.pdf – Clarifies the overlap and differences among fields.
📄 Difference Between DS AI AND ML-2.pdf – Extended explanation of ML, DS, and AI relationships.

🔗 AI / ML / DS Reference Links

These web links provide additional insights and articles related to AI, ML, DL, and Data Science.

🧠 AI Vs ML Vs DS FOR INDUSTRY – Braincube
A comprehensive guide comparing AI, ML, and Data Science in the manufacturing industry.
🧩 Difference Between AI, ML, and DL – GeeksforGeeks
Breakdown of how Artificial Intelligence, Machine Learning, and Deep Learning differ.
📘 Difference Between DS, AI and ML – MyGreatLearning
Explains how Data Science, AI, and ML intersect and diverge.
🧪 DS vs AI vs ML – Rice University
Insights from Rice University into the distinctions between these fields.

🎨 AI Art Tools

🎨 Bing Image Creator – Generate AI images from text prompts using DALL·E.
🧠 Midjourney – Advanced AI art generation via Discord-based workflow.

📲 App Links

🛠️ Dev Apps

Stay connected, explore resources, and boost productivity with these mobile apps made for developers:

📚 Cheat Sheets
Quickly access programming cheat sheets for multiple languages.
🗓️ Daily.dev
Developer news and articles curated from top sources, delivered daily.
⚡ DevBytes
Byte-sized tech content and developer tips to help you learn something new every day.
A standout feature: it displays the user Streak in the logo to encourage consistent learning.
👩‍💻 Forem (DEV Community)
Engage with devs, read blogs, and join discussions right from your phone.
:octocat: GitHub for Android
Manage repos, issues, pull requests, and more – on the go.

📄 PDF Tools

Convert, scan, and edit PDFs with these handy tools:

📷 Adobe Scan Scan documents and convert them into clean, shareable PDFs easily.

🧠 Aptitude & Logical Reasoning

📘 R. S. Aggarwal Book – Download Link Widely used book for aptitude and reasoning preparation.

📄 PDF Files available in Aptitude and Logical Reasoning/ folder:

⛓️ Blockchain

📖 Concepts and Notes

What is decentralization? Why is it essential to blockchain technology? Learn key principles such as smart contracts, DApps, and privacy in this file:
📄 Blockchain.txt

📘 Books & Learning Resources

📗 Blockchain Basics – Blockchain Training Alliance
Beginner-friendly overview of blockchain concepts and applications.
📕 MRCET Blockchain Lecture Notes (PDF)
Academic-level PDF notes for students learning blockchain.
📘 What is Blockchain – BSV Association eBook
A freely accessible eBook explaining blockchain fundamentals.

📘 Coding Questions

📄 PDF Files available in Coding Questions/ folder:

📄 20 Coding Patterns to Master MAANG Interviews.pdf
Essential coding patterns for cracking top tech interviews.
📄 Zero to Advance DSA Questions.pdf
Comprehensive DSA problem set from basics to advanced.

🎨 CSS Formatter

🎨 ArtDriveThru CSS Formatter Tool Online CSS beautifier and formatter for front-end developers.

🤖 Custom GPTs

📄 PDF Files available in Custom GPT/ folder:

📄 Gesture(12+Custom+GPTs).pdf Descriptions and links to custom GPTs for automation and productivity.

🗄️ Databases

🔗 General Links

📚 Advanced Database Concepts – ExploreDatabase Detailed insights on modern database designs and use cases.

🐬 MariaDB

🍃 MongoDB

📚 The Little MongoDB Book – GitHub
📘 Practical MongoDB PDF – BMC Software
📖 MongoDB Beginner Guide PDF
📝 MongoDB Handwritten Notes – TopperWorld
📘 Free MongoDB PDF Tutorials – FromDev

🧮 MySQL

🔍 SQL Personalities Quiz – LearnSQL Interactive guide to explore your database mindset.

🧾 Oracle

📘 Oracle Database Concepts PDF (Official Docs) Oracle’s official concepts documentation for version 19c.

📊 DS Resources

📬 Free Daily Dose of Data Science PDF Newsletter
Curated daily insights into the world of data science.
🧪 Assessment Quiz – Know What to Learn Next
Take a quick assessment to identify your weak topics.
🧠 5 Ways to Transition Into AI from Non-Tech Background – KDnuggets
Guidance for shifting into AI and ML roles from a non-technical background.
💼 5 Free Resources for DS Job Search – KDnuggets
Free, curated tools to boost your data science job hunt.

🧑‍💻 Full Stack Development

🛠️ Backend

💬 Join the Backend Developers Discord Server
Community for discussing backend frameworks and tools.
🔗 MySQL Search Form with HTML + PHP – Eli The Computer Guy
🔍 PHP MySQL Select – W3Schools
Learn how to query MySQL databases using PHP.
🎥 Lecture 1 – Backend Fundamentals (YouTube)

📓 Notes

📄 PDF Files available in Notes/ folder:

📄 30-Day Challenge to Learn MERN Stack.pdf
Daily learning structure to master MERN stack in a month.
📄 Java Basic Concept Notes.pdf
Key Java concepts in a simplified format.
📄 Java.pdf
Comprehensive guide to Java programming.
📄 Lecture 13 – Backend Development.pdf
Lecture slides covering backend fundamentals.

🎨 Frontend

📘 JavaScript Algorithms GitHub Repo
Collection of JavaScript algorithms explained and implemented.
💻 JavaScript Projects by CodeWithFaraz
🎨 Learn CSS – Web.Dev Interactive Course
❓ Frontend Trivia

🗂️ Questions sourced from Full Stack Development/Frontend/Questions.txt

Q1: How many characters can be written in 1KB?

A) 1048 B) 1024 C) 1042 D) 1000
✅ Answer: B) 1024

📚 Books

📄 PDF Files available in Full Stack Development/Books/ folder:

📄 Web Development with Node & Express.pdf
Learn backend development using Express.js.
📄 The Full Stack Developer Guide (PDFDrive)
Essential full-stack skills for modern web developers.

📓 Notes

📄 Full Stack Introduction.pdf Introductory overview of full stack development and its components.

🧾 Graphic Designing

📄 PDF Files

📄 Graphic Design Basics.pdf

Fundamentals of layout, typography, and visual design principles.
📄 Graphic-Designing-Notes.pdf

Comprehensive notes on various graphic designing topics.

📡 IoT Resources

🔗 Book Link

📘 IoT & Applications Digital Notes – MRCET Core material for understanding IoT fundamentals and its applications.

🐧 Linux Guides

🔗 Website Links

📂 Linux Directory Listing – Part 1
📂 Linux Boot Process Overview Interactive and beginner-friendly Linux tutorials.

🧠 LLM (Large Language Models)

📂 Useful Links

📄 PDF File

📄 LLM Introduction.pdf Beginner-friendly overview of LLM concepts and tools.

📊 Power BI

🔗 Website Resources

📂 PDF Guides

📄 Microsoft Power BI – Mobile PDF Guide
📄 Introduction to Microsoft Power BI – GitHub
📄 Monash Power BI Course
📄 Power BI for Beginners (2020)
📄 End-to-End Analytics with Power BI

🧩 Go Programming

🔗 Project Links

📺 Learn Go Programming by Building 11 Projects

A comprehensive YouTube tutorial for hands-on Go learning.
🧠 Building a Type-Safe Go Backend for React

Dev article explaining how to integrate a robust Go backend with React.

📱 Kotlin Programming

🔗 Project Link

📺 Simple Kotlin App | Android Studio | 2024 Tutorial for building a simple Kotlin Android app using Android Studio.

🐍 Python Programming

🔗 GitHub Repositories

📘 10 GitHub Repositories to Master Python A curated list of top Python learning resources and projects.

📚 Interactive Notebooks

📓 Marimo

A Python-based reactive notebook built for developers—clean, fast, and ideal for dashboards and reports.
📔 Jupyter Notebook

A powerful open-source web app to create and share documents containing live code, equations, and visualizations.
📗 Google Colab

A free cloud service from Google that supports Python notebooks with GPU/TPU support and zero setup.

🧠 Beginner-Friendly Articles

📘 Python for Absolute Beginners – freeCodeCamp

A comprehensive guide to get started with Python from scratch.
📙 Learn Python – W3Schools

Interactive lessons to grasp Python syntax, logic, and functions step-by-step.
📕 Real Python – Beginner’s Roadmap

Structured learning paths for those new to Python, covering basics to real-world projects.

🦀 Rust Programming

🔗 Project Link

📺 Getting Started with Rust – Projects Playlist Video walkthrough on building Rust projects from scratch.

🌐 Study Resources

🌐 Learn Anywhere – Rumie
🌐 GoalKicker – Free Programming Books
🌐 KodeKloud
🌐 KodeKloud Learning Portal
🌐 Engineers Planet – Resources
🌐 Engineers Planet – Learning Portal
🌐 Engineers Planet GitHub – ML/DL Resources

📊 SAS Programming

📄 PDF File

📄 Introduction To SAS Programming.pdf Foundational guide to understanding SAS programming basics.

🔗 Extra Reading

📘 SAS Intro Guide – ECU Official SAS intro document from East Carolina University.

🧾 HTML to PDF Conversion

Convert HTML files to PDF using browser tools, command-line utilities, Python, or online services.

🔹 1. Using Google Chrome or Any Browser (Easy Method)

Open the HTML file in your browser (Chrome, Edge, Firefox).
Press Ctrl + P (Windows/Linux) or Cmd + P (Mac) to open the Print dialog.
In the Destination section, select Save as PDF.
Click Save, choose a location, and save the file.

🔹 2. Using `wkhtmltopdf` (Command Line)

A powerful tool to convert HTML to PDF via terminal.

✅ Install `wkhtmltopdf`

Ubuntu/Debian:

  sudo apt install wkhtmltopdf

Mac (Homebrew):

  brew install wkhtmltopdf

Windows: Download from wkhtmltopdf.org

🔁 Convert HTML to PDF

wkhtmltopdf input.html output.pdf

🔹 3. Using Python (Automated)

Automate the conversion with Python using pdfkit.

✅ Install Dependencies

pip install pdfkit
sudo apt install wkhtmltopdf

💡 Python Script

import pdfkit
pdfkit.from_file("input.html", "output.pdf")

🔹 4. Using an Online Tool

No setup needed—just upload your file and convert:

🌐 Sejda HTML to PDF
🌐 PDFCrowd

🤝 Want to Contribute or Suggest Improvements?

If you'd like to suggest new resources, contribute, or start a discussion, feel free to get involved!

📫 Get in Touch

If you have questions, suggestions, or just want to say hi:

✨ If you're preparing for interviews, exploring new tech, or just looking for helpful guides, this hub is here for you.

Take a look, explore the links, and if you find it helpful, give it a ⭐ on GitHub!

Star 🌟 the repo on GitHub!

🌊 AquaScript: Your Dev Tool for Fake Data, Real Impact ⚡

Madhurima Rawat — Tue, 20 May 2025 06:27:54 +0000

💦 Welcome to AquaScript

AquaScript is your fast, simple, and developer-friendly platform to generate blazing-fast fake JSON data APIs, perfect for seamless development, testing, and prototyping. Built by developers, for developers — with ❤️ by @hanzla-baig , @kelvincode1234 , and @madhurima_rawat .

wecoded-dev / Aquascript

Welcome to 𝗔𝗾𝘂𝗮𝗦𝗰𝗿𝗶𝗽𝘁 — your fast, simple, and developer-friendly platform 🌊 for generating blazing-fast fake JSON data APIs 🚀, perfect for seamless development, testing, and prototyping ✨

AquaScript 🌊✨

Your Favorite Fake JSON API for Testing & Prototyping

Welcome to AquaScript — a fast, simple, and developer-friendly platform to fetch blazing-fast fake JSON data APIs for your web and app projects.

Crafted with passion by Hanzla Baig, Precious Kelvin and Madhurima Rawat (HP&M) with ´❤️´

🚀 About AquaScript

AquaScript is your go-to solution for generating fake, yet realistic, JSON data across a variety of categories. Whether you're building a frontend UI, testing backend endpoints, or prototyping an entire app — AquaScript lets you plug in mock data without the hassle of setting up a database or dealing with real information.

Need mock articles, user profiles, books, jokes, movie listings, or inspirational quotes?
We’ve got you covered. ✅

🎯 Project Goal

To empower developers with a fast, reliable, and easy-to-integrate platform for accessing high-quality fake JSON data — boosting productivity and reducing development friction.

✨ Key

…

View on GitHub

What is AquaScript?
Why AquaScript?
API Categories
Meet the Team
Project Goal
Tech Stack
Live Demo & Contribution
Design & Inspiration
Links & Resources

1. What is AquaScript?

AquaScript provides fake, realistic JSON data across various domains. Whether you're building a frontend UI, testing APIs, or teaching programming, AquaScript is here to make it easier and faster.

2. Why AquaScript?

🌟 Key Highlights:

🆓 Free Access: No signup or API keys required
⚡ Fast: Blazing-fast JSON responses
📚 Diverse Categories: Books, Movies, Quotes, Jokes, Recipes, and more
🔄 Updated: Regularly updated APIs
🔗 Easy Integration: Developer-first approach with clean JSON

3. API Categories

📘 Books — Titles, authors, genres
→ books.json

→ books++.json

🎬 Movies — Ratings, cast, streaming info
→ movies.json

→ movies++.json

💬 Quotes — Inspiring and funny sayings
→ quotes.json

→ quotes++.json

😂 Jokes — General and dev-focused humor
→ jokes.json

→ programming-jokes.json

👤 Users — Random user profiles and metadata
→ users.json

🍽️ Recipes — Ingredients, steps, and nutrition
→ recipes.json

🎵 Songs — Song names, lyrics, and artist info
→ songs.json

Each API returns data in a consistent JSON structure — ideal for quick plug-and-play integration!

4. Meet the Team

Built with ❤️ by:

🧠 Lead Developer

Hanzla BaigFollow

In my projects, I aim to push the boundaries of conventional web design, incorporating advanced animations, responsive layouts, and dynmic features that captivate and retain users. So i like this

(he/him, Founder) — Lead Developer, bringing innovative solutions to life with a forward-thinking mindset and hands-on expertise.
🛠️ Backend Engineer

Precious Kelvin NwaoguFollow

Software Developer skilled in React, Nextjs, Nodejs, Java, HTML5, CSS3 & Material UI. I love building user-centered web applications and am always eager to learn 📚 and collaborate 🤝!

(he/him, Co-Founder) — Full Stack Specialist, architecting robust and scalable APIs that power seamless backend operations.
🧩 Project Coordinator

Madhurima RawatFollow

🚀 Passionate about tech & growth | 💻 Data Scientist & Frontend Dev | C, C++, Python, R | 📊 InfluxDB, Grafana | 🤖 ML Enthusiast | 🐙 GitHub & Open Source | 📚 Learning & Sharing

(she/her, Co-Founder) — Streamlined Git workflows, managed issues and pull requests to ensure smooth, collaborative development.

Together, we aim to empower the developer community with tools that are simple, fast, and powerful.

5. Project Goal

🎯 To provide a free, open-source, and developer-friendly platform for accessing mock data without the overhead of setting up your own infrastructure.

Origin Story

It all started from this post:

AquaScript 🌊✨

Hanzla Baig ・ Apr 12

#programming #productivity #api #discuss

After gathering feedback, we decided to build a GitHub repo and officially launch AquaScript.

🔗 Follow our official DEV profile for updates:

AquaScript Follow

Free JSON APIs Providers

6. Tech Stack

🛠 Built With:

HTML5
CSS3
JavaScript
Bootstrap
jQuery
Node.js
Python
JSON

☁️ More tech and enhancements coming soon...

7. Live Demo & Contribution

🌐 Try it live:

AquaScript | Free JSON APIs for Developers

Access high-quality, reliable JSON APIs for books, movies, quotes, jokes, and more. Easy to integrate, free to use, and constantly updated.

aquascript.xyz

💡 No installation needed — just grab an endpoint and go.

Give the GitHub repo a star! 🌟

8. Design & Inspiration

🖼️ About the Cover Image:

The cover shows a calming blueish gradient, inspired by ocean waves, reflecting the name AquaScript.

The wave motion represents fluidity and speed, key traits of our API platform.

Icons on the cover illustrate all current API categories available.

🧩 The heading says:

AquaScript — JSON Data API

Underneath it, we display:

Fake data, real results — Build faster

I’ve tried to capture the whole concept here — let me know what you think or share any feedback you have!

9. Links & Resources

🌍 Website –

AquaScript | Free JSON APIs for Developers

Access high-quality, reliable JSON APIs for books, movies, quotes, jokes, and more. Easy to integrate, free to use, and constantly updated.

aquascript.xyz
🧑‍💻 GitHub Repo –

wecoded-dev / Aquascript

Welcome to 𝗔𝗾𝘂𝗮𝗦𝗰𝗿𝗶𝗽𝘁 — your fast, simple, and developer-friendly platform 🌊 for generating blazing-fast fake JSON data APIs 🚀, perfect for seamless development, testing, and prototyping ✨

AquaScript 🌊✨

Your Favorite Fake JSON API for Testing & Prototyping

Welcome to AquaScript — a fast, simple, and developer-friendly platform to fetch blazing-fast fake JSON data APIs for your web and app projects.

Crafted with passion by Hanzla Baig, Precious Kelvin and Madhurima Rawat (HP&M) with ´❤️´

🚀 About AquaScript

AquaScript is your go-to solution for generating fake, yet realistic, JSON data across a variety of categories. Whether you're building a frontend UI, testing backend endpoints, or prototyping an entire app — AquaScript lets you plug in mock data without the hassle of setting up a database or dealing with real information.

Need mock articles, user profiles, books, jokes, movie listings, or inspirational quotes?
We’ve got you covered. ✅

🎯 Project Goal

To empower developers with a fast, reliable, and easy-to-integrate platform for accessing high-quality fake JSON data — boosting productivity and reducing development friction.

✨ Key
…

View on GitHub
📝 Project Overview Blog (by Hanzla Baig) –

AquaScript 🌊✨

Hanzla Baig ・ Apr 12

#programming
#productivity
#api
#discuss
📢 AquaScript on DEV –

AquaScript
Follow

Free JSON APIs Providers

👨‍💻 Contributors

🙋‍♂️ Hanzla Baig –

Hanzla BaigFollow

In my projects, I aim to push the boundaries of conventional web design, incorporating advanced animations, responsive layouts, and dynmic features that captivate and retain users. So i like this
🙋‍♀️ Madhurima Rawat –

Madhurima RawatFollow

🚀 Passionate about tech & growth | 💻 Data Scientist & Frontend Dev | C, C++, Python, R | 📊 InfluxDB, Grafana | 🤖 ML Enthusiast | 🐙 GitHub & Open Source | 📚 Learning & Sharing
🙋‍♂️ Precious Kelvin –

Precious Kelvin NwaoguFollow

Software Developer skilled in React, Nextjs, Nodejs, Java, HTML5, CSS3 & Material UI. I love building user-centered web applications and am always eager to learn 📚 and collaborate 🤝!

Let us know what’s next — any API, article, or anything you want!

Some ideas:

New APIs you want us to add
Articles on how we manage the project using issues and PRs
How we handle time zones
Real project use cases
Templates for using this project
Or just any ideas or suggestions you have — we’ve got you covered!

💬 Keep learning, keep building, keep breaking — and most importantly — keep using AquaScript! 🌊💻🚀

We made this for you. Would love to hear your feedback!

✅ Setting Up Cloud-based CI-CD Pipeline 🔄

Madhurima Rawat — Mon, 19 May 2025 16:00:00 +0000

🛰️ Hello Cloud Sentinels!!

May all your logs be clean and your dashboards insightful! 📈☁️

📚 All code, docs, and resources are available in my GitHub repository:

madhurimarawat / Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Tools and Technologies ⚙️💻

1. AWS CLI

AWS Command Line Interface (CLI) is a powerful tool that allows users to interact with AWS services directly from the terminal. It simplifies managing cloud resources by providing commands for a wide range of AWS services, enabling tasks such as provisioning, managing, and automating workflows with ease.

2. LocalStack

LocalStack is a fully functional, local testing environment for AWS services. It enables developers to simulate AWS services on their local machines, facilitating the development and testing of cloud-based applications without needing access to an actual AWS account.

3. Docker

Docker is a containerization platform that allows developers to build, share, and run applications in isolated environments called…

View on GitHub

In the last post,

Madhurima Rawat

May 19 '25

📊 Implementing Cloud Monitoring and Logging 🛠️

#aws #cloud #cloudcomputing #tutorial

Comments 5

5 min read

we previously explored monitoring and logging, and how logs are managed within the cloud.

Today, we’re diving into something equally exciting and essential:
🚀 Cloud-based CI/CD Pipelines!

First, I’ll walk you through what CI/CD really is, why it’s a game-changer in modern development, and then we’ll move on to its hands-on implementation — from seamless code integration to fully automated deployment.

In this experiment, we’ll set up a Continuous Integration/Continuous Deployment (CI/CD) pipeline using GitHub Actions, Docker, and LocalStack. You'll gain practical insights into automating cloud deployments with AWS services, particularly leveraging the AWS CLI and S3 for efficient resource management.

Let’s build pipelines that flow effortlessly — like water through perfectly engineered pipes! ☁️🔄🚀🛠️

🚀 How It Works

1️⃣ GitHub Actions:

Automates build, test, and deployment processes directly from GitHub.
Triggers workflows on code commits, pull requests, or scheduled intervals.

2️⃣ Docker:

Creates containerized environments for running applications.
Ensures that the pipeline runs consistently across different systems.

3️⃣ LocalStack:

Simulates AWS cloud services locally (S3, Lambda, DynamoDB, etc.).
Allows developers to test AWS-related workflows without real AWS costs.

4️⃣ AWS CLI & S3:

AWS CLI automates interactions with AWS services.
S3 (Simple Storage Service) acts as a storage bucket for deployment artifacts.

🔹 Use Cases

✅ Automated Deployment Pipelines

Code is automatically tested, built, and deployed to cloud environments.
Reduces manual intervention, ensuring faster release cycles.

✅ Simulating AWS Services Locally

Developers can test AWS-dependent applications without incurring AWS costs.
Ideal for offline development or local testing of cloud-native applications.

✅ Cloud-Based Workflow Testing

Ensures infrastructure as code (IaC) principles by defining cloud setups in version control.
Useful for DevOps teams deploying applications on AWS.

✅ Disaster Recovery & Backup Automation

CI/CD can automate the creation of S3 backups and deployment rollbacks.
Helps maintain data integrity and business continuity.

✅ Microservices & Serverless Development

Supports Lambda function deployment, API Gateway integration, and event-driven applications.
Helps teams working on serverless computing streamline their workflow.

🌍 Real-Life Examples

📌 E-commerce Platforms

Deploy new features to AWS-hosted websites seamlessly without downtime.
Test changes in a LocalStack AWS simulation before pushing them live.

📌 Financial Services

Automate deployment of fraud detection algorithms in a secure pipeline.
Ensure compliance by testing AWS interactions locally before deploying.

📌 Mobile App Backend Development

Automatically deploy backend APIs (hosted on AWS Lambda) after each successful commit.
Use LocalStack to test S3 storage operations without using real AWS resources.

📌 AI/ML Model Deployment

Automate pushing trained ML models to S3 for cloud inference.
Use GitHub Actions to validate the model before deployment.

🎯 Key Benefits

🔹 Faster Development Cycles → Reduces manual deployment efforts.

🔹 Cost-Efficient Testing → Simulates AWS without incurring costs.

🔹 Reliable Cloud Automation → Ensures seamless integration & deployment.

🔹 Enhanced Security → Controlled CI/CD workflow reduces human errors.

🔹 Scalability → Easily extendable for various AWS services.

📷 Visual Representation

🔗 Version Control in CI/CD

🔗 CI/CD Pipeline Execution

🖼️ About the Cover Image:
The visual begins with a cloud icon surrounded by a 🔄 CI/CD loop, symbolizing continuous integration and deployment within the cloud environment.

From the cloud, the flow branches out into three development lines, each representing different GitHub branches. These branches visually merge into one, signifying the integration into the main branch — a core concept of CI/CD pipelines.

Next, the image shows three connected user icons, representing seamless team collaboration through GitHub and CI/CD processes. It highlights how developers work together, push changes, and build confidently in sync.

🎨 The entire theme follows a black and grey palette, inspired by the GitHub aesthetic.
✅ The icons appear in green, reinforcing the success-driven nature of CI/CD pipelines — where green indicates passing builds, clean merges, and successful deployments.

CI/CD Pipelines with GitHub Actions, Docker, and LocalStack

1. Creating an S3 Bucket

Command:

aws --endpoint-url=http://localhost:4566 s3 mb s3://my-ci-cd-artifacts

Explanation:

aws s3 mb → Creates a new S3 bucket.
s3://my-ci-cd-artifacts → The name of the bucket being created.
--endpoint-url=http://localhost:4566 → Uses LocalStack to simulate AWS services.

Output:

make_bucket: my-ci-cd-artifacts

2. Attempting to Create a CodeCommit Repository

Command:

aws --endpoint-url=http://localhost:4566 codecommit create-
repository --repository-name my-repo

Explanation:

aws codecommit create-repository → Creates a new AWS CodeCommit repository.
--repository-name my-repo → Assigns the repository name as my-repo.
--endpoint-url=http://localhost:4566 → Uses LocalStack.

Error Output:

An error occurred (InternalFailure) when calling the
CreateRepository operation: API for service 'codecommit'
not yet implemented or pro feature - please check
https://docs.localstack.cloud/references/coverage/ for further information

3. Initializing a Git Repository

Command:

git init

Explanation:

git init → Initializes a new Git repository in the current directory.

Output:

Initialized empty Git repository in C:/Users/rawat/Documents/8
SEMESTER/Cloud Computing/Lab/Experiment 10/Codes/.git/

4. Staging and Committing Files

Commands:

git add .
git commit -m "Initial commit"

Explanation:

git add . → Stages all files for commit.
git commit -m "Initial commit" → Commits the staged files with a message.

Output:

[master (root-commit) 2dfb5b6] Initial commit
 3 files changed, 1153 insertions(+)
 create mode 100644 Command Prompt Input and Output Explanation.md
 create mode 100644 Command Prompt Input and Output Explanation.pdf
 create mode 100644 Command Prompt Input and Output.txt

5. Uploading a ZIP File to S3

Command:

aws --endpoint-url=http://localhost:4566 s3 cp my-code.zip
s3://my-ci-cd-artifacts/

Explanation:

aws s3 cp → Copies a file to S3.
my-code.zip → The file being uploaded.
s3://my-ci-cd-artifacts/ → Destination bucket in S3.
--endpoint-url=http://localhost:4566 → Uses LocalStack.

Error Output:

The user-provided path my-code.zip does not exist.

6. Creating a ZIP Archive

Command:

powershell Compress-Archive -Path * -DestinationPath my-code.zip

Explanation:

Compress-Archive -Path * -DestinationPath my-code.zip → Creates a ZIP archive of all files in the directory.

7. Uploading the ZIP File Again

Command:

aws --endpoint-url=http://localhost:4566 s3 cp my-code.zip
s3://my-ci-cd-artifacts/

Output:

upload: .\my-code.zip to s3://my-ci-cd-artifacts/my-code.zip

8. Listing the Uploaded Files in S3

Command:

aws --endpoint-url=http://localhost:4566 s3 ls s3://my-ci-cd-artifacts/

Output:

2025-03-08 10:32:42     289415 my-code.zip

9. Setting Up a Remote Git Repository

Commands:

git remote add origin https://github.com/madhurimarawat/Cloud-
Computing.git
git branch -M main
git push -u origin main

Explanation:

git remote add origin <repo-url> → Links the local repository to GitHub.
git branch -M main → Renames the current branch to main.
git push -u origin main → Pushes the code to GitHub.

Error Output:

To https://github.com/madhurimarawat/Cloud-Computing.git
 ! [rejected]        main -> main (fetch first)
error: failed to push some refs to
'https://github.com/madhurimarawat/Cloud-Computing.git'
hint: Updates were rejected because the remote contains
work that you do not
hint: have locally. This is usually caused by another
repository pushing to
hint: the same ref. If you want to integrate the remote changes, use
hint: 'git pull' before pushing again.

Fix:

To resolve this issue, run:

git pull origin main --rebase
git push -u origin main

10. Pulling the Latest Changes from GitHub

Command:

git pull origin main --rebase

Explanation:

Fetches changes from the remote repository and applies them using rebase instead of a merge.
Ensures a linear commit history by reapplying local changes on top of the latest remote changes.

Output:

remote: Enumerating objects: 240, done.
remote: Counting objects: 100% (240/240), done.
remote: Compressing objects: 100% (212/212), done.
remote: Total 240 (delta 100), reused 43 (delta 21), pack-reused 0
Receiving objects: 100% (240/240), 9.22 MiB | 1.11 MiB/s, done.
Resolving deltas: 100% (100/100), done.
From https://github.com/madhurimarawat/Cloud-Computing
 * branch            main       -> FETCH_HEAD
 * [new branch]      main       -> origin/main
Successfully rebased and updated refs/heads/main.

11. Staging All Changes

Command:

git add .

Explanation:

Stages all modified and newly created files in the current directory for the next commit.

12. Checking for an Ongoing Rebase

Command:

git rebase --continue

Explanation:

Used to continue an ongoing rebase operation if there are conflicts.
In this case, the error means there was no ongoing rebase, so this step was unnecessary.

Output:

fatal: no rebase in progress

13. Pushing Changes to GitHub

Command:

git push -u origin main

Explanation:

Pushes local changes to the remote repository (origin), setting main as the upstream branch.
This makes future git push commands simpler by automatically pushing to origin main.

Output:

Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 282.43 KiB | 31.38 MiB/s, done.
Total 5 (delta 1), reused 1 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/madhurimarawat/Cloud-Computing.git
   b201b02..eb4faf7  main -> main
branch 'main' set up to track 'origin/main'.

Output Breakdown:

Delta compression → Reduces the size of transmitted data.
Objects written successfully → Confirms the push was successful.
Tracking branch set up → Future git push commands will default to origin main.

14. Viewing the YAML Deployment Workflow

Link:

View the deployment YAML file

Purpose:

This GitHub Actions workflow automates a manual deployment process by performing the following steps:

Workflow Breakdown:

Triggering the Workflow
- The workflow is manually triggered using workflow_dispatch, meaning it does not run automatically on commits or merges.
Job Execution
- A single job named deploy is executed on Ubuntu-latest, the default GitHub-hosted runner.
Steps in the Workflow

Checkout Repository

 - Uses `actions/checkout@v4` to fetch the repository contents into the GitHub Actions runner.

(Optional) Install AWS CLI

 - This step is commented out but would install the AWS CLI if needed.

(Optional) Zip the Repository

 - Another commented-out step that creates a ZIP archive of the repository.

(Optional) Upload to LocalStack S3

 - Demonstrates an attempt to upload the ZIP file to a **LocalStack S3 bucket**.
 - **⚠️ This step does NOT work in GitHub Actions**, since LocalStack would need to be running on the same machine.

Print Success Message
- Simply prints "Successfully run!" to indicate that the workflow has been executed.

Key Considerations:

This workflow is primarily a template for deploying to LocalStack S3.
Since GitHub Actions runs on cloud-hosted VMs, it cannot access LocalStack running locally.
We can modify this workflow to deploy to a real AWS S3 bucket by configuring proper AWS credentials.

📄 Want to see how everything came together step by step? Check it out here:
🔗 Experiment 10 Output (PDF)

🧠 Curious about the exact commands and how they function? Explore the detailed input-output flow:
🖥️ CI/CD Setup Input-Output Flow (PDF)

🎉 And that’s a wrap on Cloud-based CI/CD Pipelines!

📚 Got awesome CI/CD tips, resources, or cheat sheets? Feel free to drop them in the comments — I’d love to explore and share them!

📘 Want a compact version of all experiments with explanations and outputs in one place?
Grab the complete PDF here:
📂 Cloud Computing Lab Manual (PDF)

💖 This is the final article in this series.
Thank you all so much for sticking with me and exploring the world of Cloud Computing through each of these posts! It’s been an incredible journey, and I hope you found each article insightful and practical.

💬 I’d love to hear from you:
✨ Which article was your favorite?
✨ How did you like the series overall?
✨ And most importantly — what should I explore next?

Here are a few exciting ideas I’m considering for the next series:

📊 Big Data Analytics using Hadoop
🏢 Data Warehousing using MySQL
🐍🧹 Data Wrangling using Python and MySQL
💻 A community-powered series based on my repo:
madhurimarawat / CodeCulture-Daily

A daily programming challenge repository where fun meets learning! With 39 challenges over 39 days, it helps coders enhance skills through practical tasks and interview prep. While maintenance is paused, contributions are welcome, and future expansions are planned to keep learning ongoing.
🚀 CodeCulture-Daily

A daily programming challenge repository where fun meets learning! Solve challenges, learn new tools and technologies, and get featured as a top contributor. Join now to sharpen your coding skills!

“It's not about ideas. It's about making ideas happen.” – Scott Belsky

Welcome to CodeCulture-Daily, your go-to spot for daily programming and tech challenges! 🎉

By practicing daily, you'll not only sharpen your problem-solving skills but also learn and master various tools, technologies, algorithms, and much more. Whether you're starting as a beginner or pushing through advanced levels, CodeCulture-Daily is the perfect platform to grow your knowledge. 💡

🌟 What to Expect
- 📅 Daily Challenges: Every day at 7 PM, a new challenge will be posted in this repository.
- 🏆 Three Levels: Each challenge is categorized by difficulty
  
  🟢 Beginner
  
  🟡 Intermediate
  
  🔴 Advanced
- 📂 Domain Categories
  
  Tech 💻
  
  🖥️ Programming Languages…
View on GitHub
📌 In this one, I’ll post one question daily — and we’ll solve it together in the comments. How fun would that be?

👉 Let me know in the comments which one you'd love to dive into next!

Until the next article — goodbye for now! 🌟
Let’s meet again in the next series. 🙌💻🚀

📊 Implementing Cloud Monitoring and Logging 🛠️

Madhurima Rawat — Mon, 19 May 2025 06:34:28 +0000

📊 Hello Access Managers!!

From setting secure roles to watching them in action — let’s move from IAM to real-time monitoring and logging! 🔐➡️📊

📚 All code, docs, and resources are available in my GitHub repository:

madhurimarawat / Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Cloud-Computing

Tools and Technologies ⚙️💻

1. AWS CLI

2. LocalStack

3. Docker

Docker is a containerization platform that allows developers to build, share, and run applications in isolated environments called…

View on GitHub

In the last post,

Madhurima Rawat

May 17 '25

👥 Cloud Identity and Access Management (IAM) 🔓

#cloud #cloudcomputing #aws #iam

Comments 4

6 min read

we explored Cloud Identity and Access Management (IAM) and how to secure your cloud infrastructure with fine-grained access control.

Today, we’re gonna dive into something equally vital,
📊 Cloud Monitoring and Logging!

First, I’ll explain what monitoring and logging actually mean, why they’re absolutely essential in any cloud setup, and share some real-life use cases that show their impact.
Then we’ll move ahead with the implementation.

What is AWS CloudWatch?

AWS CloudWatch is a monitoring and observability service that collects and tracks metrics, logs, and events to provide insights into cloud resources and applications.

🔹 Key Features of CloudWatch:

Metrics Monitoring 📊: Collects and analyzes performance metrics for AWS services.
Log Management 📜: Stores and processes logs for applications and infrastructure.
Alarms & Notifications 🚨: Triggers alerts based on threshold values.
Dashboards & Visualization 📈: Provides real-time monitoring dashboards.
Event-driven Automation ⚙️: Automates actions based on events.

Understanding Log Management

Log management is a crucial part of cloud monitoring and involves the collection, storage, and analysis of logs.

💡 Six Components of Log Management:

Log Collection 🏗️: Logs are gathered from various sources (servers, applications, cloud services).
Log Aggregation 🗃️: Logs are combined and stored in a centralized system like CloudWatch Logs.
Log Storage 🏢: Logs are kept in CloudWatch Log Groups and Log Streams.
Log Analysis 🔍: Tools like AWS CloudWatch Insights analyze log data for patterns and trends.
Log Monitoring & Alerts 🚨: CloudWatch can send alerts based on predefined conditions.
Compliance & Retention ✅: Logs are archived based on security and compliance requirements.

What Are Log Streams?

Log streams represent a sequence of log events from the same source (e.g., an application instance, EC2 instance, or microservice).

🔹 Example:

A web server might have separate log streams for each instance it runs on.
CloudWatch logs are organized in log groups, and each log group contains multiple log streams.

Real-life Use Case: CloudWatch Logs & Metrics

Scenario:

A company hosts its web application on AWS EC2 instances and wants to monitor CPU usage and analyze logs for errors.

Steps in AWS CloudWatch:

1️⃣ Enable CloudWatch Logs on EC2 instances to store application logs.

2️⃣ Create a log group /my/app/logs and define log streams for different instances.

3️⃣ Send logs & metrics (e.g., CPU usage, errors, response time) to CloudWatch.

4️⃣ Set up alarms to notify the DevOps team when CPU usage exceeds 80%.

5️⃣ Analyze logs using CloudWatch Insights to detect anomalies.

🖼️ About the Cover Image:
It begins with a speedometer icon, representing metrics and performance monitoring — capturing real-time data like logs, usage stats, and resource health.
Next, a checkbox with a right tick symbolizes that all essential cloud services are being actively monitored and running smoothly ✅.
Following that, a warning icon appears to indicate that if anything goes wrong, the system will trigger alerts and notifications ⚠️.
Finally, a heart with a heartbeat line represents the idea of continuous, real-time monitoring — ensuring your cloud stays alive, healthy, and responsive 💓📡.

AWS CLI Commands for LocalStack (CloudWatch and Logs)

1. Create a Log Group

aws --endpoint-url=http://localhost:4566 logs create-log-group
--log-group-name /my/app/logs

Description:

Creates a log group named /my/app/logs in AWS CloudWatch Logs.

2. Create a Log Stream

aws --endpoint-url=http://localhost:4566 logs create-log-stream
--log-group-name /my/app/logs --log-stream-name my-stream

Description:

Creates a log stream named my-stream within the /my/app/logs log group.

3. Put Log Events

aws --endpoint-url=http://localhost:4566 logs put-log-events --
log-group-name /my/app/logs --log-stream-name my-stream --log-
events "[{\"timestamp\":1741348140000,\"message\":\"Test log entry\"}]"

Description:

Adds a log entry with a timestamp (1741348140000) and message (Test log entry) to the my-stream log stream.

Output:

-----------------------------------------------------------------------------------
|                                  PutLogEvents                                   |
+--------------------+------------------------------------------------------------+
|  nextSequenceToken |  00000000000000000000000000000000000000000000000000000001  |
+--------------------+------------------------------------------------------------+
||                             rejectedLogEventsInfo                             ||
|+-------------------------------------------------------------------+-----------+|
||  tooNewLogEventStartIndex                                         |  0        ||
|+-------------------------------------------------------------------+-----------+|

4. Put Metric Data

aws --endpoint-url=http://localhost:4566 cloudwatch put-metric-
data --namespace "MyApp" --metric-name "CPUUsage" --value 75

Description:

Publishes a custom CloudWatch metric CPUUsage with a value of 75 in the MyApp namespace.

5. List CloudWatch Metrics

aws --endpoint-url=http://localhost:4566 cloudwatch
list-metrics --namespace "MyApp"

Description:

Lists all CloudWatch metrics under the MyApp namespace.

Output:

-------------------------------
|         ListMetrics         |
+-----------------------------+
||          Metrics          ||
|+-------------+-------------+|
|| MetricName  |  Namespace  ||
|+-------------+-------------+|
||  CPUUsage   |  MyApp      ||
|+-------------+-------------+|

6. Describe Log Groups

aws --endpoint-url=http://localhost:4566 logs describe-log-groups

Description:

Retrieves details of all log groups available in CloudWatch Logs.

Output:

-----------------------------------------------------------------------------------------
|                                   DescribeLogGroups                                   |
+---------------------------------------------------------------------------------------+
||                                      logGroups                                      ||
|+-------------------+-----------------------------------------------------------------+|
||  arn              |  arn:aws:logs:us-east-1:000000000000:log-group:/my/app/logs:*   ||
||  creationTime     |  1741322075726                                                  ||
||  logGroupName     |  /my/app/logs                                                   ||
||  metricFilterCount|  0                                                              ||
||  storedBytes      |  0                                                              ||
|+-------------------+-----------------------------------------------------------------+|

7. Retrieve Log Events

aws --endpoint-url=http://localhost:4566 logs get-log-events
--log-group-name /my/app/logs --log-stream-name my-stream

Description:

Fetches the log events from my-stream under /my/app/logs.

Output:

-------------------------------------------------------------------------------------
|                                   GetLogEvents                                    |
+--------------------+--------------------------------------------------------------+
|  nextBackwardToken |  b/00000000000000000000000000000000000000000000000000000000  |
|  nextForwardToken  |  f/00000000000000000000000000000000000000000000000000000000  |
+--------------------+--------------------------------------------------------------+

Want to see how it all worked step by step? Check it out here:
🔗 Experiment 9 Output (PDF)

🧠 Curious about how each command runs and responds? See the detailed input-output flow here:
🖥️ Monitoring Setup Input-Output Flow (PDF)

🎉 And that’s a wrap on Cloud Monitoring and Logging!

📚 Found great monitoring tools, cheat sheets, or observability hacks? Drop them in the comments. I’d love to check them out and share with the community!

🚀 Stay tuned for the next article!
Up next, we’ll dive into Setting Up a Cloud-based CI/CD Pipeline — where you’ll learn how to automate deployment and streamline your cloud workflows like a pro. 🔄☁️💻

👥 Cloud Identity and Access Management (IAM) 🔓

Madhurima Rawat — Sat, 17 May 2025 07:44:39 +0000

🦈 Hello Storage Sharks!!

May all your database data be stored and queried faster than lightning ⚡📊

📚 All code, docs, and resources are available in my GitHub repository:

madhurimarawat / Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Cloud-Computing

Tools and Technologies ⚙️💻

1. AWS CLI

2. LocalStack

3. Docker

Docker is a containerization platform that allows developers to build, share, and run applications in isolated environments called…

View on GitHub

In the last post,

Madhurima Rawat

May 16 '25

🌩️ Cloud Databases and Data Management 📊

#cloud #database #tutorial #docker

Comments

7 min read

we discussed about cloud databases and how data is managed in the cloud.

Today, we’re gonna dive into something even more crucial,
🔐 Cloud Identity and Access Management (IAM)!

So first I’ll explain what IAM is, and then we’ll move forward to its implementation.
Let’s get those access policies tight and secure like pros! ☁️🔐💼

Identity and Access Management (IAM)

Identity and Access Management (IAM) is a framework that ensures the right individuals and entities have appropriate access to technology resources. In AWS IAM, users can securely control access to AWS services and resources.

Key Concepts of IAM

1. Identities in IAM

IAM identities define who can access AWS resources and what actions they can perform. These include:

Users – Individual accounts with login credentials.
Groups – Collections of users sharing permissions.
Roles – Assigned temporary permissions for AWS services or external users.

Example: A developer needs access to an S3 bucket. Instead of assigning permissions directly, they are added to the "Developers" IAM group, which has the required permissions.

2. IAM Policies

IAM policies define permissions using JSON-based policy documents specifying what actions are allowed or denied on AWS resources.

Example Policy (Allow full S3 access to a user):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

This grants full S3 permissions to the user.

3. IAM Roles and Temporary Credentials

IAM roles allow users or AWS services to assume temporary permissions to interact with resources securely.

Example Use Cases:

EC2 Instance Roles – Instead of embedding credentials in an EC2 instance, an IAM role is assigned with the required permissions.
Cross-Account Access – A company allows external contractors to access AWS services via IAM roles without sharing credentials.

4. Multi-Factor Authentication (MFA)

IAM supports MFA, requiring an additional authentication step beyond a username and password.

Example Use Case:

A finance team member accessing sensitive billing data in AWS must verify using Google Authenticator before accessing the console.

5. IAM Best Practices

Follow Least Privilege Principle – Users and roles should only have the minimum permissions needed.
Enable MFA for All Users – Adds an extra security layer.
Use IAM Roles for AWS Services – Avoid storing credentials in applications.
Rotate Access Keys Regularly – If access keys are compromised, rotation prevents misuse.

Real-World IAM Use Cases

1. Enterprise-Level Role-Based Access Control (RBAC)

A large organization has departments like HR, Development, and Finance. Each department has specific access needs:

HR Team: Can access employee databases but not development resources.
Developers: Can deploy applications but cannot modify billing information.
Finance: Can view billing dashboards but cannot access production environments.

IAM ensures proper segregation of duties.

2. IAM in Cloud Security & Compliance

A financial institution must comply with PCI-DSS (Payment Card Industry Data Security Standard):

IAM enforces policies ensuring only authorized personnel access cardholder data.
MFA is required for accessing critical systems.
Audit Logs via AWS CloudTrail help track unauthorized access attempts.

Case Study: IAM in a Tech Startup

Company: CloudTech Solutions

Problem: Developers needed access to AWS services, but granting full access posed security risks.

Solution: The company implemented IAM roles and groups:

Developers: Given read-only access to production and write access to development environments.
Admins: Given full AWS access with MFA enabled.
Automated IAM Policy Auditing detected and removed unused permissions.

Outcome:

Reduced security risks.
Improved compliance with security best practices.
Simplified access management.

🖼️ About the Cover Image:
It starts with a web icon inside a laptop, representing the website or broader cloud ecosystem.
Next, it shows a single user connected to multiple other users through a network and flow icon, illustrating how user roles and policies define connections.
The flow then leads to a lock (🔒) and an identity card icon, symbolizing that only authorized individuals can access specific resources, highlighting the core principles of security and identity management in IAM.

Identity and Access Management (IAM) with Localstack

1. Create a User

aws iam create-user --user-name test-user
--endpoint-url=%AWS_ENDPOINT_URL%

What it does:

Creates a new IAM user named test-user.
This user can be assigned roles and policies later.

Expected Output:

{
  "User": {
    "Path": "/",
    "UserName": "test-user",
    "UserId": "AIDAEXAMPLEID",
    "Arn": "arn:aws:iam::123456789012:user/test-user",
    "CreateDate": "2024-03-06T12:00:00Z"
  }
}

2. Create a Role

aws iam create-role --role-name ec2-role --assume-role-policy-
document file://trust-policy.json --endpoint-url=%AWS_ENDPOINT_URL%

What it does:

Creates an IAM role named ec2-role.
The role uses a trust policy (stored in trust-policy.json) to define which AWS services or users can assume this role.

Explanation of the Trust Policy Document

This Trust Policy is a JSON document that defines who (which AWS service, user, or account) is allowed to assume an IAM role. This is particularly useful when granting permissions to AWS services like EC2, Lambda, or other entities to access AWS resources.

Breakdown of the JSON Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

1. `"Version": "2012-10-17"`

Specifies the version of the IAM policy language being used.
"2012-10-17" is the latest and recommended version for AWS policies.

2. `"Statement": [...]`

Contains one or more statements defining the permission rules.

Inside the `Statement` Block:

`"Effect": "Allow"`

Defines whether the policy allows or denies the action.
"Allow" means this policy grants permissions.

`"Principal": { "Service": "ec2.amazonaws.com" }`

Specifies who is allowed to assume this role.
"Service": "ec2.amazonaws.com" means that the EC2 service (Amazon Elastic Compute Cloud) is allowed to assume this role.

`"Action": "sts:AssumeRole"`

Specifies the action that is permitted.
"sts:AssumeRole" allows the EC2 service to assume this role using the AWS Security Token Service (STS).
This is required when an EC2 instance needs temporary credentials to interact with other AWS services on behalf of the IAM role.

Use Case Example

When you create an IAM Role (e.g., ec2-role) with this trust policy, EC2 instances can assume this role.
This is useful when an EC2 instance needs access to S3, DynamoDB, Lambda, etc., without using long-term credentials.
AWS securely provides temporary credentials to the EC2 instance through this role.

How This Works in Practice

Create an IAM Role with this trust policy.
Attach permissions to the role (e.g., S3 Read Access).
Assign the IAM Role to an EC2 instance when launching it.
The EC2 instance automatically receives temporary credentials and can access AWS resources as defined in the role's permissions.

Expected Output:

{
  "Role": {
    "Path": "/",
    "RoleName": "ec2-role",
    "RoleId": "AROAXAMPLEID",
    "Arn": "arn:aws:iam::123456789012:role/ec2-role",
    "CreateDate": "2024-03-06T12:05:00Z",
    "AssumeRolePolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "ec2.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
  }
}

3. Attach a Policy to the User

aws iam attach-user-policy --user-name test-user
--policy-arn arn:aws:iam::aws:policy/AdministratorAccess
--endpoint-url=%AWS_ENDPOINT_URL%

What it does:

Attaches the AdministratorAccess policy to test-user, giving full access to AWS resources.

Expected Output:

No direct output. The policy is attached successfully.

To verify, use the list-attached-user-policies command.

4. List Users

aws iam list-users --endpoint-url=%AWS_ENDPOINT_URL%

What it does:

Lists all IAM users in the account.

Expected Output:

{
  "Users": [
    {
      "Path": "/",
      "UserName": "test-user",
      "UserId": "AIDAEXAMPLEID",
      "Arn": "arn:aws:iam::123456789012:user/test-user",
      "CreateDate": "2024-03-06T12:00:00Z"
    }
  ]
}

5. List Roles

aws iam list-roles --endpoint-url=%AWS_ENDPOINT_URL%

What it does:

Lists all IAM roles in the account.

Expected Output:

{
  "Roles": [
    {
      "Path": "/",
      "RoleName": "ec2-role",
      "RoleId": "AROAXAMPLEID",
      "Arn": "arn:aws:iam::123456789012:role/ec2-role",
      "CreateDate": "2024-03-06T12:05:00Z"
    }
  ]
}

6. List Attached Policies for a User

aws iam list-attached-user-policies --user-name test-user
--endpoint-url=%AWS_ENDPOINT_URL%

What it does:

Displays the policies attached to test-user.

Expected Output:

{
  "AttachedPolicies": [
    {
      "PolicyName": "AdministratorAccess",
      "PolicyArn": "arn:aws:iam::aws:policy/AdministratorAccess"
    }
  ]
}

These commands help manage IAM users, roles, and permissions efficiently. Using LocalStack, this can be tested in a local environment before deploying in AWS.

📄 Want to see how it all worked step by step? Check it out here:
🔗 Experiment 8 Output (PDF)

🧠 Curious about how each command runs and responds? See the detailed input-output flow here:
🖥️ IAM Setup Input-Output Flow (PDF)

🎉 And that’s a wrap on Identity and Access Management (IAM)!

📚 Found great IAM tutorials, cheat sheets, or guides? Drop them in the comments. I’d love to check them out and share them with others!

🔥 Stay tuned for the next article!
Next up, we’ll explore Cloud Monitoring and Logging — where you'll learn how to track activity and monitor resources effectively in a cloud environment. 📈🔍

🌩️ Cloud Databases and Data Management 📊

Madhurima Rawat — Fri, 16 May 2025 06:45:02 +0000

⚖️ Hello Balancers!

From cloud load to life load, you can balance it all! 🐘📦

📚 All code, docs, and resources are available in my GitHub repository:

madhurimarawat / Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Cloud-Computing

Tools and Technologies ⚙️💻

1. AWS CLI

2. LocalStack

3. Docker

Docker is a containerization platform that allows developers to build, share, and run applications in isolated environments called…

View on GitHub

In the last post,

Madhurima Rawat

May 13 '25

⚖️ Cloud Load Balancing and Auto Scaling 📈

#cloud #tutorial #cloudcomputing #aws

Comments

7 min read

we explored Cloud Load Balancing and Auto Scaling, the dynamic duo behind resilient cloud infrastructure.

Now, before we float too high in the clouds, let’s bring things down to earth and local.

In this post, we’ll set up a PostgreSQL database using Docker on your local machine.
It’s quick, clean, and the perfect way to prototype or develop cloud-ready apps without relying on live cloud resources.

No installations. No clutter. Just a few Docker commands and you're good to go. 🐳🧪

Let’s containerize our data journey from the ground up! 📦🔌

Cloud Databases and Data Management

Overview

This experiment covers setting up and managing cloud-based relational databases using PostgreSQL, Docker, and LocalStack. It provides hands-on experience with database management in a simulated cloud environment, replicating real-world cloud database operations using containerized solutions.

What are Databases?

Databases store, organize, and manage data efficiently. They are crucial in applications ranging from websites to large-scale enterprise systems.

Types of Databases

Databases are broadly categorized into:

Relational Databases (SQL-based) – Structured data stored in tables (e.g., PostgreSQL, MySQL).
NoSQL Databases – Flexible schema for handling unstructured data (e.g., MongoDB, Cassandra).
Cloud Databases – Managed services with scalable storage (e.g., AWS RDS, Google Cloud Firestore).

PostgreSQL Features

PostgreSQL is a powerful, open-source relational database with features like:

ACID Compliance – Ensures data integrity.
Extensibility – Supports custom functions and data types.
Scalability – Handles large volumes of data.

Real-World Application & Case Study

Use Case: Financial Data Management

A leading bank implemented PostgreSQL on the cloud to handle transaction processing, fraud detection, and real-time analytics. With Dockerized deployments, they achieved high availability, ensuring 99.9% uptime and enhanced security.

This experiment provides insights into deploying such systems using Docker and LocalStack to simulate real-world cloud database management.

🖼️ About the Cover Image:
It illustrates the journey of data in a cloud environment. On the left, a PostgreSQL database (🐘🗄️) stores structured information securely. That data flows into a brain (🧠), symbolizing the tools and technologies used for managing, querying, and analyzing the data, like Docker, pgAdmin, and SQL scripts. From there, the insights are transformed into visual representations, a pie chart (📊) and bar graph (📈) displayed on a blackboard (🧾), indicating how data is communicated, presented, and used for decision-making in cloud-native setups.

Database Operations with Postgres

1. Creating an RDS Instance Using LocalStack

Command:

aws rds create-db-instance --db-instance-identifier mydb \
  --db-instance-class db.t3.micro \
  --engine mysql \
  --master-username admin \
  --master-user-password password \
  --allocated-storage 20 \
  --endpoint-url=http://localhost:4566

Error Output:

Could not connect to the endpoint URL: "http://localhost:4566/"

An error occurred (InternalFailure) when calling the CreateDBInstance operation:
API for service 'rds' not yet implemented or pro feature - please check
https://docs.localstack.cloud/references/coverage/ for further information

Explanation:

The command attempts to create an RDS instance in LocalStack.
--endpoint-url=http://localhost:4566 → Uses LocalStack instead of AWS.
Errors indicate:
- LocalStack is either not running or misconfigured.
- RDS API might not be fully implemented in the free version of LocalStack.

Output Breakdown:

Could not connect to the endpoint URL → LocalStack might not be running or accessible.
InternalFailure error → The RDS API might require LocalStack Pro for full functionality.
Possible Fixes:
- Ensure LocalStack is running:
```
docker run --rm -d --name localstack_main -p 4566:4566 localstack/localstack
```
- Check service coverage: LocalStack RDS Coverage

2. Starting a PostgreSQL Container

Command:

docker start my-postgres

docker start postgres

Error Output:

Error response from daemon: No such container: my-postgres
Error: failed to start containers: my-postgres

Error response from daemon: No such container: postgres
Error: failed to start containers: postgres

Explanation:

The containers do not exist under the specified names.
Verify running containers with:

  docker ps -a

If needed, create a new container:

  docker run --name my-postgres -e POSTGRES_USER=admin \
    -e POSTGRES_PASSWORD=password -e POSTGRES_DB=mydb \
    -p 5432:5432 -d postgres:15

Output Breakdown:

No such container: my-postgres → The container was never created or was removed.
failed to start containers → The container name does not match any existing instances.
Possible Fixes:
- Check existing containers: docker ps -a
- Create and start a new PostgreSQL container using docker run (above).

3. Listing Available Docker Images

Command:

docker images

Output:

| REPOSITORY            | TAG    | IMAGE ID     | CREATED      | SIZE   |
| --------------------- | ------ | ------------ | ------------ | ------ |
| my-flask-app          | latest | f5feae0ac7a4 | 6 hours ago  | 139MB  |
| flask-app             | latest | ae4054c49614 | 7 hours ago  | 139MB  |
| hackvortex-backend    | latest | 14e63c26d40b | 21 hours ago | 1.05GB |
| postgres              | 15     | e45d3f5ec589 | 7 days ago   | 430MB  |
| localstack/localstack | latest | b686f3948f42 | 6 weeks ago  | 1.18GB |
| python                | 3.9    | 9f98746e2033 | 3 months ago | 999MB  |
| nginx                 | latest | b52e0b094bc0 | 4 weeks ago  | 192MB  |

Explanation:

Displays available images in the local Docker environment.
PostgreSQL (postgres:15) is available.
LocalStack (localstack/localstack) is present but needs verification (docker ps -a).

Output Breakdown:

postgres:15 is listed → The image exists but the container may not be running.
localstack/localstack exists → LocalStack is installed but may need to be started.

Possible Fixes:

Start PostgreSQL if not running:

docker run --name my-postgres -e POSTGRES_USER=admin \
  -e POSTGRES_PASSWORD=password -e POSTGRES_DB=mydb \
  -p 5432:5432 -d postgres:15

Ensure LocalStack is running:

docker start localstack_main

4. Starting a PostgreSQL Container

Command:

C:\Users\rawat>docker start postgres

Error Output:

Error response from daemon: No such container: postgres
Error: failed to start containers: postgres

5. Listing All Containers

Command:

C:\Users\rawat>docker ps -a

Output:

CONTAINER ID	IMAGE	COMMAND	CREATED	STATUS	PORTS	NAMES
a10c5a71f625	localstack/localstack	"docker-entrypoint.sh"	2 minutes ago	Up 2 minutes (healthy)	127.0.0.1:4510-4560->4510-4560/tcp, 127.0.0.1:4566->4566/tcp, 5678/tcp	localstack-main
7f0fa023ac4f	3a669f02efff	"python app.py"	7 hours ago	Exited (255) 5 minutes ago	8080/tcp, 0.0.0.0:5002->5000/tcp	backend2
9ff472da8892	3a669f02efff	"python app.py"	7 hours ago	Exited (255) 5 minutes ago	8080/tcp, 0.0.0.0:5001->5000/tcp	backend1

6. Running a PostgreSQL Container

Command:

C:\Users\rawat>docker run --name my-postgres -e
POSTGRES_USER=admin -e POSTGRES_PASSWORD=password -e
POSTGRES_DB=mydb -p 5432:5432 -d postgres:15

Error Output:

docker: Error response from daemon: driver failed
programming external connectivity on endpoint my-postgres
(feae7f0fb87909bde1853a7ddefa49bb518f11250e54304f75109
68f7a88cca1): Bind for 0.0.0.0:5432 failed: port is already allocated.

7. Resolving Port Conflict and Running PostgreSQL on a Different Port

Command:

C:\Users\rawat>docker run --name my-new-postgres -e
POSTGRES_USER=admin -e POSTGRES_PASSWORD=password -e
POSTGRES_DB=mydb -p 5433:5432 -d postgres:15

Output:

b2efdca3c6f0af6cf4154fce236f0b66b5efba0f4f9e14972c94b3e0a5afa9de

8. Verifying Running Containers

Command:

C:\Users\rawat>docker ps

Output:

CONTAINER ID	IMAGE	COMMAND	CREATED	STATUS	PORTS	NAMES
b2efdca3c6f0	postgres:15	"docker-entrypoint.s…"	42 seconds ago	Up 41 seconds	0.0.0.0:5433->5432/tcp	my-new-postgres
a10c5a71f625	localstack/localstack	"docker-entrypoint.sh"	3 minutes ago	Up 3 minutes (healthy)	127.0.0.1:4510-4560->4510-4560/tcp, 127.0.0.1:4566->4566/tcp, 5678/tcp	localstack-main

9. Connecting to PostgreSQL and Performing SQL Operations

Command:

C:\Users\rawat>docker exec -it my-
new-postgres psql -U admin -d mydb

Output:

psql (15.12 (Debian 15.12-1.pgdg120+1))
Type "help" for help.

Creating a Table and Inserting Data:

CREATE TABLE students (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    email VARCHAR(100) UNIQUE NOT NULL
);
INSERT INTO students (name, email) VALUES
('Alice Johnson', 'alice@example.com'),
('Bob Smith', 'bob@example.com'),
('Charlie Brown', 'charlie@example.com');

Output:

CREATE TABLE
INSERT 0 3

10. Performing SQL Queries

Selecting Data:

SELECT * FROM students;

Output:

id	Name	Email
1	Alice Johnson	alice@example.com
2	Bob Smith	bob@example.com
3	Charlie Brown	charlie@example.com

Updating Data:

UPDATE students SET email = 'bob.smith@example.com'
WHERE name = 'Bob Smith';

Output:

UPDATE 1

Deleting Data:

DELETE FROM students WHERE name = 'Charlie Brown';

Output:

DELETE 1

Selecting Data with a Condition:

SELECT * FROM students WHERE name LIKE 'A%';

Output:

id	Name	Email
1	Alice Johnson	alice@example.com

Exiting PostgreSQL:

\q

📄 Want to see how it all worked step by step? Check it out here:

🔗 Experiment 7 Output (PDF)

🧠 Curious about how each command runs and responds? See the detailed input-output flow here:

🖥️ PostgreSQL + Docker Setup Flow (PDF)

🎉 And that’s a wrap on Cloud Databases and Data Management!

💡 We went hands-on with setting up PostgreSQL using Docker, a clean, fast, and containerized way to build and test database-backed apps locally before going cloud-native.

💬 Tried other tools like MySQL, MongoDB, or hosted services like Firebase? I’d love to hear what worked best for you and why!

📚 Know a helpful resource or walkthrough?. I’d love to include helpful links in the article for others to explore!

🔥 Coming up next:
We're diving into Cloud Security: Identity and Access Management (IAM), the guardrails of modern cloud architecture. 🔐☁️
Let’s talk users, roles, policies, and how to keep your cloud castle secure!

⚖️ Cloud Load Balancing and Auto Scaling 📈

Madhurima Rawat — Tue, 13 May 2025 15:20:47 +0000

🔁 Hello Clouders!

May your serverless functions always stay within bounds or prepare for a billing surprise! 💸⚡

📚 All code, docs, and resources are available in my GitHub repository:

madhurimarawat / Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Cloud-Computing

Tools and Technologies ⚙️💻

1. AWS CLI

2. LocalStack

3. Docker

Docker is a containerization platform that allows developers to build, share, and run applications in isolated environments called…

View on GitHub

In the last post,

Madhurima Rawat

Apr 27 '25

🛠️ Using Cloud Functions for Serverless Computing 🧩

#aws #lambda #tutorial #cloudcomputing

Comments 13

16 min read

we explored the world of serverless computing, where AWS Lambda took center stage.

Now, it’s time to dive deeper into the magic that makes cloud apps resilient and responsive:
Cloud Load Balancing and Auto Scaling, your dynamic duo for performance and efficiency.
Let’s balance the load, scale the servers, and cloud like pros. ☁️🚀

First, I’ll explain the design behind load balancing and auto scaling.
Then, we’ll implement it using Python and LocalStack — simulating traffic with a simple Flask app instead of using paid services like NGINX.

I’ve also explained all the steps for setting up NGINX-based load balancing, which I initially tried.
So if you’re using the paid version or actual AWS services, everything will work as expected too. ✅

What is Load Balancing?

Load balancing is the process of distributing incoming network traffic across multiple servers to ensure no single server bears too much demand. This improves performance, reliability, and fault tolerance by preventing overload and ensuring continuous availability.

Key Benefits of Load Balancing:

Improved Performance: Spreads traffic efficiently across multiple servers.
High Availability & Redundancy: Ensures uptime even if a server fails.
Scalability: Easily accommodates increased traffic by adding more servers.
Optimized Resource Utilization: Ensures servers are used efficiently.

Types of Load Balancing:

Round Robin: Distributes requests sequentially across servers.
Least Connections: Directs traffic to the server with the fewest connections.
IP Hash: Uses the client's IP address to route requests consistently to the same server.
Weighted Load Balancing: Assigns different weights to servers based on their capacity.

Load Balancing Illustration

This image visually represents load balancing, showing how traffic is distributed among multiple servers to ensure high availability and reliability.

What is Nginx?

Nginx (pronounced "engine-x") is a high-performance web server and reverse proxy designed for speed, stability, and scalability. It is widely used for load balancing, caching, and serving static content efficiently.

Why Use Nginx?

Reverse Proxy & Load Balancing: Distributes requests across multiple backend servers.
High Performance: Handles thousands of concurrent connections efficiently.
Security Features: Protects against DDoS attacks and unauthorized access.
Efficient Static Content Handling: Quickly serves HTML, CSS, JavaScript, and images.
Support for Microservices: Works seamlessly with modern containerized applications.

Nginx Overview

This image provides a high-level view of Nginx’s role in handling requests, acting as a reverse proxy, load balancer, and web server for modern applications.

🖼️ About the Cover Image:
It shows growing user traffic (👥🛒) reaching a load balancer (⚖️), which then distributes tasks efficiently.
Auto scaling (📈📉) ensures resources scale up or down as demand changes — keeping your cloud setup both effective and economical.

Simple Load balancing Function with LocalStack and Flask

1. Creating a Virtual Private Cloud (VPC):

To begin, create a VPC using the AWS CLI.

aws ec2 create-vpc --cidr-block 10.0.0.0/16 --endpoint-
url=http://localhost:4566

2. Creating a Subnet:

Once the VPC is created, define a subnet within it.

aws ec2 create-subnet --vpc-id vpc-66375b0cbe498b519
--cidr-block 10.0.1.0/24 --endpoint-url=http://localhost:4566

3. Creating an Application Load Balancer:

Create an ALB and associate it with the subnet.

aws elbv2 create-load-balancer --name my-load-balancer
--subnets subnet-3f40f7c6e3a26040f --security-groups default
--type application --endpoint-url=http://localhost:4566

Setting Up Nginx Containers with Load Balancing

4. Running Backend Containers:

Navigate to the project directory before running containers.

cd "C:\Users\rawat\Documents\8 SEMESTER\Cloud
Computing\Lab\Experiment 6\Codes"

Run the backend containers using Docker:

docker run -d --name backend1 nginx
docker run -d --name backend2 nginx

If the image is not found locally, Docker will pull it from the repository.

5. Verify Available Docker Images:

To check the downloaded images:

docker images

Expected output:

REPOSITORY                     TAG        IMAGE ID       CREATED        SIZE
nginx                          latest     b52e0b094bc0   4 weeks ago    192MB
flask-app                      latest     269bf42596ed   3 weeks ago    126MB
...

6. Creating and Running a Load Balancer Container:

docker run -d --name load-balancer -p 8080:80 -v "C:/Users/rawat/Documents
/8 SEMESTER/Cloud
Computing/Lab/Experiment 6/Codes/nginx.conf:/etc/nginx/nginx.conf" nginx

Check logs for potential errors:

docker logs load-balancer

7. Fixing 'Host Not Found' Error:

The error "host not found in upstream 'backend1:80'" means Nginx cannot resolve backend1. All containers must be in the same network.

Fix: Use a Custom Docker Network

Step 1: Create a Network

docker network create my-network

Step 2: Start Backend Containers in the Same Network

Remove existing backend containers:

docker rm -f backend1 backend2

Restart them within the custom network:

docker run -d --name backend1 --network my-network nginx
docker run -d --name backend2 --network my-network nginx

Step 3: Update `nginx.conf`

Edit your nginx.conf file to ensure proper load balancing:

events { }

http {
    upstream backend_servers {
        server backend1:80;
        server backend2:80;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://backend_servers;
        }
    }
}

Save the file and restart the load balancer container:

docker rm -f load-balancer
docker run -d --name load-balancer --network my-network -p 8080:80 -v
"C:/Users/rawat/Documents/8 SEMESTER/Cloud
Computing/Lab/Experiment 6/Codes/nginx.conf:/etc/nginx/nginx.conf" nginx

Step 4: Verify Setup

Check if the load balancer is running properly:

docker ps

Access the load balancer at:

http://localhost:8080

If this is not showing up properly, then go to the next step.

Step 5: Restart the Load Balancer

First, remove the existing container:

docker rm -f load-balancer

Then start it again, within the same network:

docker run -d --name load-balancer --network my-network -p 8080:80 -v
"C:/Users/rawat/Documents/8 SEMESTER/Cloud
Computing/Lab/Experiment 6/Codes/nginx.conf:/etc/nginx/nginx.conf" nginx

Step 6: Verify Everything

Check if containers are running:

docker ps

Then try opening:

http://localhost:8080

LocalStack Load Balancer - Flask Simulation

Since Nginx doesn't support load balancing for LocalStack, a Python Flask application is used for round-robin load balancing simulation.

from flask import Flask, render_template_string
import itertools
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import io
import base64

app = Flask(__name__)

# Backend servers
backends = ["Backend 1", "Backend 2"]

# Simulated processes
processes = [f"P{i+1}" for i in range(10)]
load_values = np.random.randint(3, 15, size=len(processes))

# Round-robin assignment
backend_cycle = itertools.cycle(backends)
schedule = {process: next(backend_cycle) for process in processes}

@app.route("/")
def home():
    """Renders the home page with the load balancer visualization."""
    try:
        sns.set_theme(style="darkgrid")
        fig, ax = plt.subplots(figsize=(9, 5))

        x_labels = list(schedule.keys())
        y_labels = list(schedule.values())

        custom_colors = [
            "#52796F", "#E63946", "#6A0572", "#457B9D", "#9C6644",
            "#8D99AE", "#2A9D8F", "#E76F51", "#6D597A", "#B56576"
        ]

        bars = ax.bar(x_labels, load_values, color=custom_colors[: len(processes)], width=0.6)

        for bar, (backend, load) in zip(bars, zip(y_labels, load_values)):
            ax.text(
                bar.get_x() + bar.get_width() / 2,
                bar.get_height() + 0.4,
                backend,
                ha="center", fontsize=10, fontweight="bold",
                bbox=dict(facecolor="white", alpha=0.6, edgecolor="black", boxstyle="round,pad=0.3")
            )

        ax.set_ylabel("Assigned Load")
        ax.set_title("Round Robin Load Balancer Simulation", fontsize=14, fontweight="bold")
        ax.set_xticklabels(x_labels, rotation=30, ha="right")

        img = io.BytesIO()
        plt.savefig(img, format="png", bbox_inches="tight")
        img.seek(0)
        plt.close(fig)

        image_data = base64.b64encode(img.getvalue()).decode()

        html_template = """
        <!DOCTYPE html>
        <html lang="en">
        <head>
            <meta charset="UTF-8">
            <meta name="viewport" content="width=device-width, initial-scale=1.0">
            <title>Load Balancer</title>
            <style>
                body { font-family: Arial, sans-serif; background-color: #f0f2f5; text-align: center; padding: 20px; }
                .container { background: white; padding: 20px; box-shadow: 0 0 15px rgba(0,0,0,0.1); max-width: 850px; margin: auto; border-radius: 10px; }
                h2 { color: #2c3e50; font-size: 24px; margin-bottom: 10px; }
                p { font-size: 16px; color: #555; margin-bottom: 20px; }
                img { max-width: 100%; border-radius: 10px; margin-top: 10px; }
            </style>
        </head>
        <body>
            <div class="container">
                <h2>Round Robin Load Balancer Simulation</h2>
                <p>
                    Load balancing distributes tasks among multiple backend servers to optimize performance and reliability.
                    This simulation demonstrates a Round Robin approach, where each process is assigned to a backend in a cyclic order.
                </p>
                <img src="data:image/png;base64,{{ image_data }}" alt="Load Balancer Graph">
            </div>
        </body>
        </html>
        """
        return render_template_string(html_template, image_data=image_data)

    except Exception as e:
        return f"Error generating graph: {str(e)}", 500

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

This application displays a bar chart illustrating the distribution of processes across backend servers using the round-robin approach. The graph helps in understanding how the tasks are assigned dynamically.

Run the app using:

python app.py

Once the server is running, open a web browser and navigate to http://localhost:5000 to access the application.

📄 Want to see how it all worked step by step? Check it out here:

🔗 Experiment 6 Output (PDF)

🧠 Curious about how each command runs and responds? See the detailed input-output flow here:

🖥️ Load Balancing & Auto Scaling Setup Flow (PDF)

🎉 And that’s a wrap on Load Balancing and Auto Scaling!

💡 I’ve explained both the simulated LocalStack + Flask setup and the full NGINX-based approach, so if you’re using a paid tier, you can follow the complete setup seamlessly.

💬 Do tell me about your own experience, whether you used NGINX, LocalStack, or any other tools for load balancing. What worked, what didn’t?

📚 Found any great resources, tutorials, or articles? Drop them in the comments. I’d love to include helpful links in the article for others to explore!

🔥 Stay tuned for the next article!
We’re diving into Cloud Databases and Data Management, where I’ll talk about storing data the cloud-native way. 🗄️🌩️

✂️ From 2500+ to 10+ Lines: Automated Modular Code Refactor 🤖

Madhurima Rawat — Fri, 02 May 2025 09:48:27 +0000

Hey awesome people! 👋

Thank you so much for the recent engagement on my articles — it really made my day! 💖

So, I had actually planned to write the next article in my ☁️ Local Cloud Computing with LocalStack + Docker + AWS CLI 💡 (6 Part Series) series. But something interesting and super useful came up while working on my major project for college — a Stock Market Prediction Tool — and I thought this deserves a quick write-up!

You can check out the project here 🔗:

madhurimarawat / Stock-Market-Prediction

This repository began as a 7th-semester minor project and evolved into our 8th-semester major project, "Advanced Stock Price Forecasting Using a Hybrid Model of Numerical and Textual Analysis." It utilizes Python, NLP (NLTK, spaCy), ML models, Grafana, InfluxDB, and Streamlit for data analysis and visualization.

Stock-Market-Prediction

This repository began as a 7th-semester minor project and evolved into our 8th-semester major project, "Advanced Stock Price Forecasting Using a Hybrid Model of Numerical and Textual Analysis." It utilizes Python, NLP (NLTK, spaCy), ML models, Grafana, InfluxDB, and Streamlit for data analysis and visualization.

Project Description

The Advanced Stock Price Forecasting Using a Hybrid Model of Numerical and Textual Analysis project involves a comprehensive approach to predicting stock prices using both numerical data and textual analysis. The project components include:

Data Collection and Storage: We gathered historical stock data of major companies and stored it in an InfluxDB database to efficiently handle large-scale time-series data.
Data Visualization: A Grafana dashboard has been set up for real-time visualization of stock prices and analysis results, enhancing data interpretation and decision-making processes.
Textual Analysis for Enhanced Forecasting: We utilized Natural Language Processing (NLP) libraries, such as NLTK…

View on GitHub

📚 Table of Contents

What’s the Context?
Problem Faced 😵
Why Modularization? 💡
Flowchart Overview 📊
Modularization Steps 🧩
Pros and Cons ⚖️
GitHub Issue and PR Links 🔗
Before and After Code 📁
Final Thoughts 💬

What’s the Context?

For my college major project, I built a Stock Market Prediction app using Streamlit. The app includes interactive visualizations, model evaluation, live predictions, and dashboards powered by tools like Grafana and Power BI.

Initially, the Streamlit app had over 2500+ lines of code — all in a single file 😵. I had separate versions for local and deployed apps because some file paths were broken in the production environment.

This made even small changes (like adding a new function) a complete nightmare.

Problem Faced 😵

The original code structure looked like this:
🔗 Original Code (before refactoring)

It had all functions — data handling, UI logic, visualizations, predictions — bundled into one giant file. This made debugging, testing, and adding new features quite challenging and error-prone.

While the code was already modular in terms of function structure, everything lived in a single place. So I decided to properly refactor it — extracting each functional block into its own dedicated file. That’s why the term "refactoring" is used in the cover image, and "modularization steps" are outlined throughout the article.

So, I took the modular functions and placed them in separate files, organizing the code for better structure and clarity. 🚀

Why Modularization? 💡

Codebase Modularization means breaking a large, messy file into independent, reusable, and manageable modules.

🏢 Visualizing the Idea

Imagine your codebase as a large building — with a central hall and multiple rooms where each person (function) lives. Initially, everything is connected, but it’s a bit noisy, and the boundaries are blurry.

🔁 Now, you refactor (transform) the setup:
You redesign the structure into separate flats. Each flat is self-contained, cleaner, and still linked through a central lobby (like a main() function). People (functions) still collaborate, but now with clearer separation, more peace, and organized communication.

This transformation reflects how we modularize code: splitting logic into individual function files, cleaning dependencies, and improving maintainability — all while preserving the original connections.

🖼️ This is what I’m trying to visualize in my cover image — like transforming a shared hall into peaceful, connected flats. 🏢➡️🏘️

Curious if the cover made sense and looked good to you! 👀✨

Pros and Cons ⚖️

✅ Pros (in My App’s Context)

🧪 Easier Debugging and Testing: With functions and modules isolated, testing becomes straightforward, allowing for easier identification and resolution of bugs in specific areas, without impacting other parts of the app.
🧹 Better Readability and Code Structure: Modularization ensures the code is clean and well-organized, improving readability. Each module is purpose-driven, making it simple for anyone — including future developers — to understand the logic and flow.
🔁 Independent Development/Deployment of Modules: Each module is developed and deployed independently, allowing updates or changes in one part of the project without affecting the rest of the app. This separation streamlines the workflow and keeps the process efficient.
📈 Scalable for Future Improvements: The modular structure is built for growth, making it easy to expand the app by adding new features without disturbing the existing code. This design ensures the app is ready for future enhancements and scaling.

❗ Cons of Modularization (in my app’s context)

While modularization comes with a lot of benefits, there are a few challenges I encountered during this refactor:

⏱️ Initial setup time: Setting up the modular structure took quite a bit of effort initially — especially extracting functions, handling dependencies, and preparing import aggregators.
🧵 Managing imports and dependencies: Every individual function file needs to have its required imports — which adds complexity, especially when working with libraries like streamlit, pandas, and plotly.
🌐 Local vs Deployed sync: I had to maintain two separate sets of files — one for local and one for deployment — because some paths and behaviors differ in production (as I explained in this GitHub issue).
* 🔁 Circular dependencies: Refactoring functions into separate modules means being extra cautious to avoid circular imports (e.g., two modules importing from each other), which can break the app.
* 🗂️ Managing directory structures: Maintaining a clean and logical directory hierarchy (like features_functions_local/ and features_functions_deployed/) is necessary but requires planning, otherwise it gets messy fast.

Even though this was from my project app perspective, it’s something that applies to most projects too! 🌍

Flowchart Overview 📊

📌 Codebase Modularization Flowchart

Modularization Steps 🧩

Here’s what I did to cleanly modularize the Streamlit app:

Step 1: `Function_Splitting.py`

📤 Extracts each function into its own file.
📁 Saved in:

features_functions_local/
features_functions_deployed/

Code 👩‍💻

"""
Author: Madhurima Rawat

Script to split top-level functions (excluding 'main') from a large Streamlit app file
into separate Python files, preserving:
- Any comment immediately above the function
- Any global variable used within the function

Each function is saved as a standalone `.py` file inside the `split_functions/` directory.
"""

import ast
import os
import astunparse

# === CONFIGURATION ===
INPUT_FILE = "Streamlit_app_local_combined.py"  # The source Python file to extract functions from

# For Local Running functions
# OUTPUT_DIR = "feature_functions_local"  # Directory to store individual function files

# For Deployed Functions
OUTPUT_DIR = (
    "feature_functions_deployed"  # Directory to store individual function files
)

# Ensure the output directory exists
os.makedirs(OUTPUT_DIR, exist_ok=True)

# Read the entire source file content
with open(INPUT_FILE, "r", encoding="utf-8") as f:
    source = f.read()

# Parse the source code into an abstract syntax tree (AST)
tree = ast.parse(source)
lines = source.splitlines()

# Containers to hold relevant nodes
function_nodes = []  # Functions to extract
global_vars = []  # Global variables (assignments) at top level
import_lines = []  # All top-level import statements

# Classify top-level elements in the AST
for node in tree.body:
    if isinstance(node, (ast.Import, ast.ImportFrom)):
        import_lines.append(node)  # Collect import lines
    elif isinstance(node, ast.Assign):
        global_vars.append(node)  # Track top-level assignments
    elif isinstance(node, ast.FunctionDef) and node.name != "main":
        function_nodes.append(node)  # Collect functions except 'main'


# Extract comment immediately above a function, if present
def get_leading_comment(node):
    lineno = node.lineno - 2
    while lineno >= 0 and lines[lineno].strip().startswith("#"):
        return lines[lineno].strip()
    return ""


# Get all variable names used in a function
def get_used_names(node):
    return {n.id for n in ast.walk(node) if isinstance(n, ast.Name)}


# Prepare a mapping of global variable name -> source code
all_globals = {
    t.targets[0].id: astunparse.unparse(t).strip()
    for t in global_vars
    if isinstance(t.targets[0], ast.Name)
}

# Create a separate file for each function
for func_node in function_nodes:
    func_name = func_node.name
    func_code = ast.get_source_segment(
        source, func_node
    )  # Full source code of the function
    used_names = get_used_names(func_node)  # Variables used in the function

    leading_comment = get_leading_comment(
        func_node
    )  # Optional comment above the function
    globals_needed = [all_globals[name] for name in used_names if name in all_globals]

    file_path = os.path.join(OUTPUT_DIR, f"{func_name}.py")

    with open(file_path, "w", encoding="utf-8") as out_file:
        if leading_comment:
            out_file.write(leading_comment + "\n")  # Write leading comment if present
        for g in globals_needed:
            out_file.write(g + "\n")  # Write any required globals
        out_file.write("\n" + func_code + "\n")  # Finally, write the function code

print(f"✅ Done! Functions split into {OUTPUT_DIR}")

Step 2: `Dependency_Adder.py`

🔍 Uses AST (Abstract Syntax Tree) to detect and inject the right import statements into each function file.

Code 👩‍💻

"""
Author: Madhurima Rawat

Script to analyze individual Python function files inside the 'split_functions' directory
and prepend necessary import statements based on used modules, functions, or objects.

The goal is to ensure each function file is self-contained by including all relevant
imports at the top. This script uses a heuristic approach, looking for known identifiers
to determine which modules are needed.
"""

import os
import ast
from pathlib import Path

# === CONFIGURATION ===

# Path to the folder containing the split function files(Local)
# functions_folder = Path("feature_functions_local")

# Path to the folder containing the split function files (Deployed)
functions_folder = Path("feature_functions_deployed")

# Mapping of commonly used identifiers to their respective import statements
# Includes one-line comments categorized for clarity and maintainability

import_suggestions = {
    # --- STREAMLIT APP and VISUALIZATION FRAMEWORK ---
    "st": "# Importing Streamlit for building the web-based interactive application framework\nimport streamlit as st",
    "plt": "# Importing Matplotlib for generating static plots and charts\nimport matplotlib.pyplot as plt",
    "go": "# Importing Plotly for creating interactive and dynamic visual plots\nimport plotly.graph_objects as go",
    "sns": "# Importing Seaborn for enhanced data visualizations\nimport seaborn as sns",
    # --- DATA HANDLING and MANIPULATION ---
    "pd": "# Importing Pandas for data manipulation and analysis\nimport pandas as pd",
    "np": "# Importing NumPy for numerical computations and array operations\nimport numpy as np",
    "os": "# Importing OS module for handling file and directory paths\nimport os",
    "datetime": "# Importing datetime for working with timestamps and date ranges\nfrom datetime import datetime, timedelta",
    "base64": "# Importing base64 for encoding and decoding binary data\nimport base64",
    # --- MACHINE LEARNING and MODELING ---
    "pickle": "# Importing Pickle for loading/saving pre-trained machine learning models\nimport pickle",
    "LinearRegression": "# Linear Regression model\nfrom sklearn.linear_model import LinearRegression",
    "RandomForestRegressor": "# Random Forest Regressor\nfrom sklearn.ensemble import RandomForestRegressor",
    "SVR": "# Support Vector Machine Regressor\nfrom sklearn.svm import SVR",
    "mean_squared_error": "# Importing evaluation metrics from Scikit-learn\nfrom sklearn.metrics import mean_squared_error, r2_score, precision_score, recall_score, f1_score",
    "MinMaxScaler": "# For scaling data to a 0–1 range\nfrom sklearn.preprocessing import MinMaxScaler",
    "TfidfVectorizer": "# Text feature extraction\nfrom sklearn.feature_extraction.text import TfidfVectorizer",
    # --- DEEP LEARNING (PyTorch) ---
    "torch": "# Importing PyTorch for building and training deep learning models\nimport torch",
    "nn": "# Importing PyTorch's neural network module\nimport torch.nn as nn",
    # --- NATURAL LANGUAGE PROCESSING (NLP) ---
    "TextBlob": "# Importing TextBlob for basic natural language processing tasks\nfrom textblob import TextBlob",
    # --- FINANCIAL DATA and UTILITIES ---
    "yf": "# Importing yfinance for fetching historical stock data from Yahoo Finance\nimport yfinance as yf",
    "webbrowser": "# Importing webbrowser module to open URLs in the default browser\nimport webbrowser",
    "openpyxl": "# Importing openpyxl to enable writing Excel files (.xlsx)\nimport openpyxl",
}


def get_required_imports(source_code):
    """
    Analyze source code of a function to detect required imports
    based on the presence of known identifiers.
    """
    tree = ast.parse(source_code)
    imports = set()

    for node in ast.walk(tree):
        # Detect simple names like 'pd', 'st', etc.
        if isinstance(node, ast.Name):
            if node.id in import_suggestions:
                imports.add(import_suggestions[node.id])
        # Detect attribute access like 'plt.plot'
        elif isinstance(node, ast.Attribute):
            value_id = getattr(node.value, "id", None)
            if value_id in import_suggestions:
                imports.add(import_suggestions[value_id])

    return sorted(imports)


# === MAIN PROCESS ===

changed_files_count = 0  # Counter for changed files
changed_lines_count = 0  # Counter for changed lines

# Iterate through each Python file in the target folder
for filepath in functions_folder.glob("*.py"):
    with open(filepath, "r", encoding="utf-8") as file:
        original_code = file.read()

    # Determine needed imports based on code analysis
    needed_imports = get_required_imports(original_code)

    # Combine imports and original content
    updated_code = "\n".join(needed_imports) + "\n\n" + original_code

    # Only write back to the file if changes were made
    if updated_code != original_code:
        with open(filepath, "w", encoding="utf-8") as file:
            file.write(updated_code)

        # Count the changes
        changed_files_count += 1
        changed_lines_count += updated_code.count("\n")

# Output results
print(
    f"✅ Done! {changed_files_count} file(s) updated. {changed_lines_count} lines changed."
)

Step 3: `Function_Importing.py`

📦 Generates a unified import file to easily access all modular functions in one place.

Code 👩‍💻

"""
Author: Madhurima Rawat

Script to generate a single import file (Import_Functions.py) that aggregates
function imports from individual .py files inside the 'split_functions' directory.

Each file is assumed to define a function with the same name as the filename.

The output file includes:
- A docstring at the top explaining the purpose
- Individual import statements
- A summary comment with total number of imported functions
"""

import os

# === CONFIGURATION ===

# For Local
# FUNCTION_DIR = "feature_functions_local"
# IMPORT_FILE = "Import_Functions_Local.py"

# For Deployment
FUNCTION_DIR = "feature_functions_deployed"
IMPORT_FILE = "Import_Functions_Deployed.py"

# List all Python files in the function directory
function_files = [f for f in os.listdir(FUNCTION_DIR) if f.endswith(".py")]
total_imports = len(function_files)

# === GENERATE THE IMPORT FILE ===
with open(IMPORT_FILE, "w", encoding="utf-8") as f:
    # Write top-level docstring to the output file
    f.write('"""\n')
    f.write(
        f"This file was auto-generated to import all functions from '{FUNCTION_DIR}/'.\n"
    )
    f.write(
        f"Each function file is expected to define a function named after the filename.\n"
    )
    f.write(f"Total functions imported: {total_imports}\n")
    f.write('"""\n\n')

    f.write("# === FUNCTION IMPORTS ===\n")

    # Write each import statement
    for filename in function_files:
        module_name = filename[:-3]  # Remove .py extension
        f.write(f"from {FUNCTION_DIR}.{module_name} import {module_name}\n")

    # Footer summary comment
    f.write(f"\n# ✅ Total functions imported: {total_imports}\n")

print(
    f"✅ '{IMPORT_FILE}' created with {total_imports} function imports from '{FUNCTION_DIR}/'"
)

Step 4: `Split_Clean_Main_Code.py`

🧼 Cleans the main app file (app_cleaned.py) by extracting embedded functions and retaining only the core logic. Also removes unused dependencies for better clarity and maintainability.

Code 👩‍💻:

"""
Author: Madhurima Rawat

Script to clean a Streamlit app by removing all top-level functions except 'main'.
Also removes unused import statements.
Preserves:
- All used import statements
- Global variables
- Top-level code (outside functions)
- The 'main' function (if present)

Saves the result as 'app_cleaned.py'.
"""

import ast
import black

# For Deployed
INPUT_FILE = "Streamlit_app_combined.py"
CLEANED_FILE = "Streamlit_app.py"

# Read source
with open(INPUT_FILE, "r", encoding="utf-8") as f:
    source = f.read()

lines = source.splitlines()
tree = ast.parse(source)

# Step 1: Remove non-main top-level functions and their header comments
lines_to_remove = set()
for node in tree.body:
    if isinstance(node, ast.FunctionDef) and node.name != "main":
        comment_line = node.lineno - 2
        if 0 <= comment_line < len(lines):
            lines_to_remove.add(comment_line)
        for i in range(node.lineno - 1, node.end_lineno):
            lines_to_remove.add(i)

cleaned_lines = [line for i, line in enumerate(lines) if i not in lines_to_remove]
cleaned_code = "".join(line + "\n" for line in cleaned_lines)


# Step 2: Parse cleaned code to remove unused imports
class ImportUsageAnalyzer(ast.NodeVisitor):
    def __init__(self):
        self.imports = {}
        self.used_names = set()

    def visit_Import(self, node):
        for alias in node.names:
            self.imports[alias.asname or alias.name] = node.lineno

    def visit_ImportFrom(self, node):
        for alias in node.names:
            name = alias.asname or alias.name
            self.imports[name] = node.lineno

    def visit_Name(self, node):
        self.used_names.add(node.id)


# Analyze the cleaned code
tree = ast.parse(cleaned_code)
analyzer = ImportUsageAnalyzer()
analyzer.visit(tree)

# Identify unused imports
unused_import_lines = set()
for name, lineno in analyzer.imports.items():
    if name not in analyzer.used_names:
        unused_import_lines.add(lineno - 1)  # Convert to 0-based

# Final clean-up: remove unused imports
final_lines = [
    line
    for i, line in enumerate(cleaned_code.splitlines())
    if i not in unused_import_lines
]
final_code = "".join(line + "\n" for line in final_lines)
formatted_code = black.format_str(final_code, mode=black.FileMode())

# Write the cleaned, formatted file
with open(CLEANED_FILE, "w", encoding="utf-8") as f:
    f.write(formatted_code)

print(
    f"✅ Cleaned and formatted file saved as '{CLEANED_FILE}' "
    "(only 'main' retained, unused imports and comments removed)."
)

Step 5: Run Final Modular App

🎯 Fully cleaned and modularized version — ready for production deployment!

📍 Final updated file:
🔗 Updated Streamlit App

Pros and Cons ⚖️

🟢 Pros	🔴 Cons
Cleaner code, easier to navigate	Initial time investment
Independent testing of modules	Manual path and import checks
Better team collaboration	Requires discipline to maintain separation
Easier debugging and refactoring	May need more files to manage

GitHub Issue and PR Links 🔗

🔎 Issue: #13 – Need to Modularize and Refactor Streamlit Code

📌 Refactor & Modularize Codebase for Streamlit App Deployment #13

madhurimarawat posted on Apr 30, 2025

The current monolithic codebase needs to be modularized to improve readability, maintainability, and deployment workflow. This issue involves splitting out individual functions into logically separated files and preparing the cleaned main app for deployment.

✅ Tasks to be Completed

🔧 Function Splitting
Run Function_Splitting.py to break down the monolithic app into smaller reusable function files saved in:
- features_functions_local (for development)
- features_functions_deployed (for production-ready deployment)
📦 Add Dependencies
Use Dependency_Adder.py to automatically inject all necessary import statements at the top of each function file.
📥 Create Aggregated Import File
Execute Function_Importing/Import_Functions.py to generate a single file for importing all extracted functions efficiently.
🧹 Clean the Main File
Run Split_Clean_Main_Code.py to generate app_cleaned.py, which contains only the cleaned main() logic without function definitions.

🧠 Insert Custom Class
Add the following class to the display_real_time_stock_prediction.py file in both local and deployed folders:

# Class for real time stock data fetching and prediction

# --- CLASS DEFINITION STARTS ---
class StockPricePredictor:
    ...

🔗 Keep Directory Paths Consistent
In the deployed version, make sure these constants are preserved:

DATASET_DIR = "Codes/Historical_Data_Analysis/Preprocessed_Dataset"
DATASET_DIR_1 = "Codes/Historical_Data_Analysis"

🧾 Why This Is Needed

🚀 Improves Deployment Readiness – Easier to push updates without disturbing working components.
🧱 Enforces Separation of Concerns – Logic is modular and testable.
💡 Enhances Developer Experience – Clear structure helps future contributors understand and work on the code quickly.

📎 Additional Notes

This issue will lead to a PR that includes:

Modular function files
A cleaned and refactored app_cleaned.py
Updated deployed code with integrated class structure

View on GitHub

🔧 PR: #14 – Modularize and Refactor Codebase for Local and Deployed Versions

✨ Refactor & Modularization Process for Streamlit App Deployment #14

madhurimarawat posted on Apr 30, 2025

This structure modularizes the codebase while retaining the original functionality

📁 Folder Naming Convention

The function components have been cleanly separated and organized using two key directories:

features_functions_local – for local development and testing.
features_functions_deployed – for final cleaned and deployable code.

⚙️ Step-by-Step Breakdown of the Refactoring Workflow:

🔧 Function Extraction
Run Function_Splitting.py
➤ This extracts all functions from the original app into individual Python files inside the respective features_functions_* folders.
➕ Dependency Injection
Run Dependency_Adder.py
➤ Automatically prepends necessary import statements to each function file by analyzing its contents using ast.
📥 Import Organizer
Navigate to Function_Importing/Import_Functions.py
➤ Generates an aggregated import script from all modularized functions for use in the final build.
🧹 Main Code Cleaner
Run Split_Clean_Main_Code.py
➤ Extracts and saves the core main() logic as app_cleaned.py, removing previously defined function bodies.

🧠 Post-Cleaning Instructions:

After completing the steps above:

✅ Copy and insert this block into the file display_real_time_stock_prediction.py inside both features_functions_local and features_functions_deployed folders:
```
# Class for real time stock data fetching and prediction

# --- CLASS DEFINITION STARTS ---
class StockPricePredictor:
    ...
```
ℹ️ Ensure this is appended after the function definitions inside the same file, or integrated in a logically correct place based on usage.

✅ In the deployed app, ensure the following paths are preserved:

# Directory containing the preprocessed datasets
DATASET_DIR = "Codes/Historical_Data_Analysis/Preprocessed_Dataset"

# Directory containing the original historical datasets
DATASET_DIR_1 = "Codes/Historical_Data_Analysis"

✅ Finally, execute the split main function logic in the cleaned app_cleaned.py within the deployed directory.

✅ Outcome:

We now have a fully modular, self-contained, and deployment-ready Streamlit application with clearly separated concerns for:

Functionality (features_functions_*)
Import management
Reusable core logic
Class-based structure where appropriate

Closes #13

View on GitHub

I’ve explained the reasoning and implementation there in detail from my project’s point of view!

Before and After Code 📁

🧱 Original Code:
Streamlit_app_combined.py
🪄 Modularized Code:
Streamlit_app.py

🛠️ Customize for Your Workflow: Using This for Your Own Project

This was originally tailored to my own project setup, but you can easily adapt it to fit yours. Here's how:

1. Extract Functions

Use the first step to extract all functions and store them in any directory of your choice. Clean them up as needed.

2. Add All Dependencies

In the Dependency_Adder.py file, list all your required libraries in the following format for clear and automatic management of dependencies:

Example Format:

import_suggestions = {
    # --- STREAMLIT APP & VISUALIZATION FRAMEWORK ---
    "st": "# Importing Streamlit for building the web-based interactive application framework\nimport streamlit as st",
    "plt": "# Importing Matplotlib for generating static plots and charts\nimport matplotlib.pyplot as plt",
    "go": "# Importing Plotly for creating interactive and dynamic visual plots\nimport plotly.graph_objects as go",
    "sns": "# Importing Seaborn for enhanced data visualizations\nimport seaborn as sns",

    # --- DATA HANDLING & MANIPULATION ---
    "pd": "# Importing Pandas for data manipulation and analysis\nimport pandas as pd",
    "np": "# Importing NumPy for numerical computations and array operations\nimport numpy as np",
    "os": "# Importing OS module for handling file and directory paths\nimport os",
    "datetime": "# Importing datetime for working with timestamps and date ranges\nfrom datetime import datetime, timedelta",
    "base64": "# Importing base64 for encoding and decoding binary data\nimport base64",

    # --- MACHINE LEARNING & MODELING ---
    # Continue with any additional dependencies...
}

How It Works:

🛠️ Organize Dependencies: Define each library with a user-friendly alias and a short description in the import_suggestions dictionary.
🔄 Auto-Search & Add: The Dependency_Adder.py will scan this dictionary, automatically adding the correct imports to your code files.

3. Import Functions & Organize Main Logic

Once your dependencies are added:

🔄 Run Function_Importing.py — This scans your codebase and inserts the appropriate imports automatically into each function file.
🧹 Clean up your main logic using Split_Clean_Main_Code.py — This separates the logic from the main file and organizes it into modular functions.
🧩 Finally, import all your functions in one line into your main file like so:

# Importing all functions
from Import_Functions_Deployed import *

This way, your main file stays clean, and all logic is neatly modularized and connected!

4. Design Considerations

Ensure you have a central main() function to coordinate the workflow. This helps prevent circular dependencies.
Your code should already be modular — meaning, key logic should be inside functions. If it's not, this process won’t be effective.
This method doesn’t extract classes or global variables used by functions. You’ll need to either:
- Add those manually, or
- Ensure they exist within the main file, or are defined as needed before calling the extracted functions.

5. Run the Main File

Once the functions are organized, you can run your project from the main file just like in my setup.

Final Thoughts 💬

So this was my whole code flow — from my app’s point of view 🔄.
Let me know if you found this helpful! 💬

🔆 Explore the Project

I’ve added comments and documentation to my code, so the article doesn't become a novel. Each file could have its own article! Let me know if you'd like a detailed breakdown of any part.

🛠️ Customize for Your Workflow: Using This for Your Own Project
I’ve also added a dedicated section to help you adapt this setup for your own codebase.
If you have any questions or need help implementing it, feel free to reach out — happy to help! 😊

I'm also in the process of finalizing my thesis. Once it's complete, I'll publish a full series on this stock market prediction project — covering the concept, execution, my teammates, challenges, and more. Let me know if you're excited for it! 🚀

🛠️ Using Cloud Functions for Serverless Computing 🧩

Madhurima Rawat — Sun, 27 Apr 2025 13:43:45 +0000

🔗 Hello Connectors!

May your VPC remain secure and your network forever connected! 🛡️🌐

📚 All code, docs, and resources are available in my GitHub repository:

madhurimarawat / Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Cloud-Computing

Tools and Technologies ⚙️💻

1. AWS CLI

2. LocalStack

3. Docker

Docker is a containerization platform that allows developers to build, share, and run applications in isolated environments called…

View on GitHub

In the previous article,

📡 Setting Up and Configuring Cloud Networking 🔐

Madhurima Rawat ・ Apr 13

#cloud #tutorial #beginners #aws

we talked about cloud networking, exploring how to set up secure and efficient connections with VPCs, subnets, and security groups.

Now, we move on to the backbone of cloud computing: Serverless Lambda — (not your math lambda! 😉)

We’ll explore how AWS Lambda lets you run code without worrying about servers, scaling, or heavy lifting.

Stay tuned — it’s time to go serverless and unleash the power of the cloud! ⚡☁️

What is Serverless Computing and AWS Lambda?

What is Serverless Computing?

Serverless computing is a cloud computing model where the cloud provider manages the underlying infrastructure. The term "serverless" doesn’t mean servers are absent; rather, it means developers don't need to manage them. The provider handles provisioning, scaling, and maintenance, allowing developers to concentrate solely on building and deploying code.

What is AWS Lambda?

AWS Lambda is a serverless compute service by Amazon Web Services (AWS) that allows users to run code without provisioning or managing servers. Developers simply focus on writing application logic, while AWS automatically handles server infrastructure, scaling, and maintenance.

AWS Lambda is a key example of serverless computing. It lets you execute code in response to events—such as HTTP requests, file uploads, or database updates—without worrying about managing servers. Lambda functions integrate seamlessly with services like API Gateway, S3, DynamoDB, and more.

Key Features of AWS Lambda

Event-Driven Execution
- Functions are triggered by events like HTTP requests, S3 uploads, or database changes.
Automatic Scaling
- Lambda automatically scales to handle varying workloads, from a few requests to thousands per second.
Cost-Efficient
- You only pay for the compute time your code uses, rather than pre-paying for idle servers.
No Server Management
- AWS handles all backend server management, freeing developers from infrastructure concerns.
Multiple Language Support
- Supports Python, Node.js, Java, C#, Go, Ruby, and more.
Stateless Functions
- Lambda functions are stateless; they do not retain data between invocations. External storage like databases is used for persistence.

Common Use Cases of AWS Lambda

Real-Time File Processing
- Process images, audio, or documents immediately upon upload to S3.
Microservices Architecture
- Build small, independent services that can be deployed and scaled individually.
API Backends
- Power RESTful APIs by integrating Lambda with API Gateway.
Data Processing and ETL Tasks
- Transform or move data automatically in response to database events or file uploads.
Automation and Scheduled Tasks
- Schedule regular tasks like backups, cleanups, or report generation without server setup.
Event-Driven Applications
- React to events instantly, such as sending alerts when a new user signs up.

Visual Understanding of AWS Lambda

First Image: AWS Lambda Overview

This image provides a snapshot of AWS Lambda's serverless model: developers upload code, and AWS automatically manages provisioning, scaling, and monitoring. Lambda seamlessly interacts with other AWS services, enabling event-driven solutions.

Second Image: AWS Lambda Workflow and Integrations

This deeper dive illustrates how Lambda connects with services like API Gateway, S3, and DynamoDB, showing its ability to handle real-time data processing, microservices communication, and backend automation.

Pros and Cons of AWS Lambda

Pros

Cost-Effective: Only pay for what you use; no charge when idle.
Automatic Scaling: Handles sudden spikes in traffic effortlessly.
Reduced Operational Overhead: No need to manage, patch, or scale servers.
Fast Deployment: Applications can be updated and deployed rapidly.
Security Integration: Works with AWS Identity and Access Management (IAM) for controlled access.

Cons

Cold Start Latency: A delay may occur if a function is triggered after being idle.
Execution Time Limit: Functions can run a maximum of 15 minutes per invocation.
Statelessness: Developers must handle persistence externally.
Resource Constraints: Limited memory and compute power compared to traditional servers.

Practical Real-World Example:

Imagine a photo-sharing application:

A user uploads a photo to an S3 bucket.
An AWS Lambda function triggers automatically to resize the image.
The resized images are saved back to S3 for use across devices.

No server provisioning, scaling, or manual monitoring is required—the entire pipeline runs seamlessly, triggered by simple events.

Real Companies Using AWS Lambda 🚀

Netflix 🎬: Uses Lambda for real-time file encoding, system monitoring, and automation—processing billions of video streams without managing servers.
Airbnb 🏠: Relies on Lambda for image processing, automated backups, and real-time data analysis.
Coca-Cola 🥤: Powers vending machine transactions with Lambda, paying only per use—cutting infrastructure costs.
Reuters 📰: Uses Lambda to resize and format images instantly for fast publishing in their global news delivery system.
iRobot 🤖: Employs Lambda to manage data from its connected home devices, enabling seamless IoT integration and event-driven automation.

Simple Lambda Function with LocalStack

Step-by-Step Creation of a Lambda Function with LocalStack

1. Creating the Lambda Function:

To create a Lambda function using LocalStack, the first step involves writing the Lambda function code. The function will be triggered by an event and should return a simple response.

Example Lambda Function Code (lambda_function.py):

def lambda_handler(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello from LocalStack Lambda!'
    }

This function will output a basic response with a 200 OK status code when invoked.

2. Deploying the Lambda Function:

Before deploying the Lambda function, the Python code needs to be packaged into a ZIP file. This ZIP file will be used to deploy the function to LocalStack.

Steps for Packaging the Lambda Function (on Windows):
1. Navigate to the directory where your lambda_function.py file is located.
2. Right-click the lambda_function.py file and select Send to > Compressed (zipped) folder from the context menu.
3. Rename the resulting .zip file to function.zip.

After the Lambda function is packaged, it is ready for deployment.

3. Checking the Function Creation Status:

After the Lambda function is deployed, its status can be checked to ensure it has been successfully created. Initially, the status might be in a Pending state, indicating that LocalStack is still processing the request to create the function.

Command to Check the Function’s Status:

aws --endpoint-url=http://localhost:4566 lambda get-function
--function-name myLambdaFunction

If the function is still in the Pending state, wait for a few moments before checking again. Once the function’s status is Active or Available, it can be invoked.

4. Invoking the Lambda Function:

After the function is in the Active state, it can be invoked using the AWS CLI. The invocation will trigger the Lambda function and return the response in a file (in this case, output.txt).

Command to Invoke the Function:

aws --endpoint-url=http://localhost:4566 lambda invoke
--function-name myLambdaFunction output.txt

This will execute the Lambda function and store the output in the output.txt file.

Here is the error message:

---

## | GetFunction |

## | Code |

| Location | http://s3.localhost.localstack |
| | .cloud:4566/awslambda-us-east-1 |
| | -tasks/snapshots/000000000000/ |
| | myLambdaFunction-88a260f7-7427 |
| | -46dc-8226-b7f079a11527?AWS |
| | AccessKeyId=949334387222& |
| | Signature=DksEUmpzW5tO%2FXj49v |
| | KMaIqPNwM%3D&Expires=1739893833|

---

## | RepositoryType | S3 |

## | Configuration |

| CodeSha256 | G16xwAmelqgPBfzQVMlj7SR/dZS5l |
| | TV2WSY8mGqvvRw= |

---

## | CodeSize | 312 KB |

## | Description | |

| FunctionArn | arn:aws:lambda:us-east-1:0000 |
| | 00000000:function:myLambdaFunc |
| | tion |

---

## | FunctionName | myLambdaFunction |

| Handler | lambda_function.lambda_handle |
| | r |

---

| LastModified | 2025-02-18T13:15:50.795443+00 |
| | 00 |

---

## | LastUpdateStatus | Failed |

## | MemorySize | 128 MB |

## | PackageType | Zip |

| RevisionId | b27b563d-ede9-4878-bb36-013a3 |
| | 5678f00 |

---

| Role | arn:aws:iam::000000000000:rol |
| | e/execution_role |

---

## | Runtime | python3.8 |

## | State Details |

## | State | Failed |

| StateReason | Error while creating lambda: |
| | public.ecr.aws/lambda/python:3.8 |

---

## | StateReasonCode | InternalError |

Troubleshooting Lambda Creation Issues:

It’s common to face issues when setting up Lambda functions with LocalStack. For example, if LocalStack is unable to fetch the required Lambda runtime (in this case, Python 3.8), the creation will fail. In the example provided, the Lambda function was in a Failed state with the error message:

State: Failed
StateReason: Error while creating lambda: public.ecr.aws/lambda/python:3.8
StateReasonCode: InternalError

This indicates that LocalStack could not retrieve the Python 3.8 runtime image from the repository (public.ecr.aws/lambda/python:3.8). The error is internal, likely caused by an issue with LocalStack’s ability to access or fetch the necessary container image from the registry.

Why Did the Lambda Function Fail?

The Lambda function failed due to an issue where LocalStack was unable to fetch the Python 3.8 runtime from the expected image repository (public.ecr.aws/lambda/python:3.8). This led to an InternalError.

State: Failed
StateReason: Error while creating lambda: public.ecr.aws/lambda/python:3.8
StateReasonCode: InternalError

This error indicates that LocalStack could not access or retrieve the Lambda runtime, which caused the function creation process to fail.

Troubleshooting and Resolving the issue using Docker

How to Resolve This Issue?

1. Ensure the Correct Lambda Runtime is Available

LocalStack uses Docker-based Lambda runtimes. To ensure that LocalStack can properly run the Lambda function, it is necessary to manually pull the required Docker image. In this case, LocalStack requires the Python 3.8 Lambda runtime.

To resolve this, follow these steps to pull the necessary Docker image:

docker pull public.ecr.aws/lambda/python:3.8

Once this command is executed, Docker will download the required Python 3.8 Lambda image. Below is the expected output when the image is successfully pulled:

Expected Output After Pulling the Image:

C:\Users\rawat\Documents\8 SEMESTER\Cloud
Computing\Lab\Experiment 5\Codes>docker pull
public.ecr.aws/lambda/python:3.8
3.8: Pulling from lambda/python
bc2b3a540f9b: Pull complete
7e7a8ab075f3: Pull complete
eb63e7acde8e: Pull complete
35e8cd243a50: Pull complete
d4160264f321: Pull complete
32e94a394cab: Pull complete
Digest: sha256:93e78742873d3ad0c28582366b217
ce5169889f4d63d61179598c2a3dc6142ff
Status: Downloaded newer image for public.ecr.aws/lambda/python:3.8
public.ecr.aws/lambda/python:3.8

2. Verify the Image Availability Locally

To confirm that the image has been successfully downloaded, use the following command to list all the Docker images present on the system:

docker images

The output should list the public.ecr.aws/lambda/python:3.8 image among other available images. Below is an example of the output you should expect:

C:\Users\rawat\Documents\8 SEMESTER\Cloud
Computing\Lab\Experiment 5\Codes>docker images
REPOSITORY                     TAG       IMAGE ID
CREATED       SIZE
flask-app                      latest    269bf42596ed
12 days ago   126MB
<none>                         <none>    44b808030263
2 weeks ago   126MB
<none>                         <none>    063caad47b0a
2 weeks ago   126MB
localstack/localstack          latest    b686f3948f42
3 weeks ago   1.18GB
hello-world                    latest    74cc54e27dc4
3 weeks ago   10.1kB
public.ecr.aws/lambda/python   3.8       348b357f1c82
4 weeks ago   575MB

This verifies that the image is available locally.

3. Retry the Lambda Function Creation

Once the Python 3.8 Lambda runtime image is available locally, LocalStack should be able to retrieve the runtime successfully. Retry the Lambda function creation process, and it should deploy without issues.

By following these steps, the error related to the missing Lambda runtime will be resolved, ensuring that LocalStack can properly run the Lambda function without any further issues.

Running a Simple Lambda Function with LocalStack After Resolving Issues

Recreating and Invoking the Lambda Function with LocalStack

Step 1: Recreate the Lambda Function

The first step is to create a new Lambda function in LocalStack. To do so, the AWS CLI create-function command is used. This command tells LocalStack to create a Lambda function with the specified configurations.

Command to Create the Lambda Function:

aws --endpoint-url=http://localhost:4566 lambda create-function \
    --function-name myLambdaFunction \
    --runtime python3.8 \
    --role arn:aws:iam::000000000000:role/execution_role \
    --handler lambda_function.lambda_handler \
    --zip-file fileb://lambda_function.zip

Here’s a breakdown of what each part of the command does:

aws --endpoint-url=http://localhost:4566:

This specifies that the AWS CLI should interact with LocalStack rather than the actual AWS cloud. localhost:4566 is the default endpoint for LocalStack's services.

lambda create-function:

This is the AWS CLI command used to create a new Lambda function.

--function-name myLambdaFunction:

This specifies the name of the Lambda function being created. In this case, the function is named myLambdaFunction.

--runtime python3.8:

This sets the runtime for the Lambda function to Python 3.8. LocalStack uses Docker to run Lambda functions with specific runtimes, and Python 3.8 is chosen here.

--role arn:aws:iam::000000000000:role/execution_role:

This assigns an IAM role to the Lambda function. The execution_role is required for Lambda functions to execute. In LocalStack, this is a placeholder, and a specific role ARN is not needed for local execution.

--handler lambda_function.lambda_handler:

This specifies the function within the code to be executed when the Lambda function is invoked. The handler format is file_name.function_name, which tells LocalStack to look in the lambda_function.py file for the lambda_handler function to run.

--zip-file fileb://lambda_function.zip:
- This points to the zip file containing the Lambda function code. The fileb:// prefix tells AWS CLI that the file is on the local file system. The lambda_function.zip file must contain the Python script (lambda_function.py).

Step 2: Invoke the Lambda Function

Once the function is created successfully, it can be invoked using the AWS CLI. This triggers the Lambda function and stores the output in a file.

Command to Invoke the Lambda Function:

aws --endpoint-url=http://localhost:4566 lambda invoke
--function-name myLambdaFunction output.txt

Here’s a breakdown of what each part of the invoke command does:

aws --endpoint-url=http://localhost:4566:

Just like before, this specifies that the command should interact with LocalStack on the specified endpoint.

lambda invoke:

This AWS CLI command triggers the Lambda function and executes it.

--function-name myLambdaFunction:

This specifies the name of the Lambda function that should be invoked. In this case, it's myLambdaFunction, which was created in the previous step.

output.txt:
- This is the file where the output of the Lambda function invocation will be stored. After the function runs, its response will be written to output.txt.

Step 3: View the Output

After invoking the Lambda function, the results are stored in the output.txt file. The type command in Windows can be used to display the contents of this file.

Command to View the Output:

type output.txt

This command displays the content of the output.txt file. The expected output looks like this:

{"statusCode": 200, "body": "Hello from LocalStack Lambda!"}

Explanation of the Output:

The content of the output.txt file is a JSON response returned by the Lambda function:

statusCode: 200:

This indicates that the Lambda function executed successfully and returned an HTTP status code of 200, which signifies a successful response.

body: "Hello from LocalStack Lambda!":
- This is the body of the response. It contains the message "Hello from LocalStack Lambda!", which was defined in the Lambda function. This message is returned as part of the response when the function is triggered.

Summary of Commands and Outputs:

Recreate the Lambda Function: The create-function command defines the function, its runtime (Python 3.8), execution role, and handler function. It also links the Lambda function to the lambda_function.zip file containing the code.
Invoke the Lambda Function: The invoke command runs the Lambda function and stores the response in output.txt.
View the Output: The type command shows the contents of output.txt, which contains a JSON response indicating the Lambda function executed successfully.

By following these steps, the Lambda function is successfully created, invoked, and the output is captured and displayed.

Complex Lambda Function with Payload with LocalStack

Detailed Commands, Redeployment, and Output for Lambda Function with Payload Processing

Updated Lambda Function Code:

Here is the updated Lambda function code that processes a payload with an input event and performs some computations:

import json
import logging

# Set up logging for debugging and monitoring
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

# Function versioning for tracking updates
VERSION = "1.0.0"


def lambda_handler(event, context):
    """
    Handles AWS Lambda execution by processing event data
    and performing basic computations.

    **Parameters:**
    - event (dict): The input event data, expected to contain:
        - `"name"` (str, optional): The name of the user.
          Defaults to `"Guest"` if not provided.
        - `"number"` (int, optional): A number to be processed.
          Defaults to `0` if not provided.
    - context (object): AWS Lambda context object
      (not used in this function).

    **Returns:**
    - dict: JSON-formatted response with:
        - `"message"` (str): Greeting message.
        - `"processedNumber"` (int): The input number multiplied by 2.
        - `"version"` (str): Function version identifier.
    """

    # Log event details for debugging
    logger.info(f"Function invoked with event: {json.dumps(event)}")
    logger.info(f"Lambda function version: {VERSION}")

    # Extract 'name' from the event, defaulting to "Guest"
    name = event.get("name", "Guest")
    greeting = f"Hello, {name}!"

    # Extract 'number' from the event and process it
    number = event.get("number", 0)  # Default to 0 if not provided
    result = number * 2  # Double the input number

    # Construct the response payload
    response = {
        "statusCode": 200,  # HTTP status code indicating success
        "body": json.dumps(
            {
                "message": greeting,  # Greeting message
                "processedNumber": result,  # Computed number result
                "version": VERSION,  # Function version for tracking
            }
        ),
    }

    # Log response details before returning
    logger.info(f"Returning response: {json.dumps(response)}")

    return response

Explanation of the Code:

Logging:

The logging module is used to track and debug the Lambda function. It logs the event data received, the function version, and the response returned. The INFO level ensures that relevant details are logged for debugging.
The logger.info statements are used to print the function invocation details and the response.

Complex Logic:

The function checks if the name parameter exists in the event data. If not, it defaults to "Guest".
It retrieves the number parameter and doubles the number as an example of computation (i.e., number * 2).

Versioning:
- The VERSION variable tracks the version of the Lambda function (e.g., "1.0.0"). This can be updated whenever changes are made to the code.

Steps to Fix and Redeploy the Lambda Function:

1. Delete the Existing Lambda Function:

Before updating the function with the new code, it’s essential to delete the old version of the function in LocalStack to avoid conflicts.

Command to Delete the Old Lambda Function:

aws --endpoint-url=http://localhost:4566 lambda delete-function
--function-name myLambdaFunction

This command deletes the function named myLambdaFunction from LocalStack, preparing for the redeployment with updated code.

2. Recreate the Lambda Function with Updated Code:

Once the old function is deleted, zip the updated code and recreate the function. Make sure the updated lambda_function.py is in your current directory.

Step 1: Zip the Updated Code:

zip lambda_function.zip lambda_function.py

This command creates a .zip file containing the updated lambda_function.py file. This zip file will be used to deploy the new version of the Lambda function.

Step 2: Recreate the Lambda Function:

aws --endpoint-url=http://localhost:4566 lambda create-function \
    --function-name myLambdaFunction \
    --runtime python3.8 \
    --role arn:aws:iam::000000000000:role/execution_role \
    --handler lambda_function.lambda_handler \
    --zip-file fileb://lambda_function.zip

This command creates a new Lambda function in LocalStack with the updated code:

--function-name myLambdaFunction: Specifies the function name (myLambdaFunction).
--runtime python3.8: Sets the runtime to Python 3.8.
--role arn:aws:iam::000000000000:role/execution_role: Assigns the execution role (this role can be any placeholder role in LocalStack).
--handler lambda_function.lambda_handler: Defines the entry point for the Lambda function (lambda_handler in the lambda_function.py file).
--zip-file fileb://lambda_function.zip: Points to the .zip file containing the updated code.

3. Invoke the Lambda Function with Payload:

Now that the function is updated, you can invoke it by passing a JSON payload. The payload should be contained in a file (e.g., event.json), which includes input data like "name" and "number".

Example Payload (event.json):

{
  "name": "Madhurima",
  "number": 5
}

Command to Invoke the Lambda Function:

aws --endpoint-url=http://localhost:4566 lambda invoke
--function-name myLambdaFunction --payload file://event.json output.txt

This command invokes the Lambda function with the provided payload and stores the result in the output.txt file.

--function-name myLambdaFunction: Specifies the function to invoke.
--payload file://event.json: Points to the input data (the event.json file) that contains the payload for the function.
output.txt: This file will store the output from the Lambda function.

4. Check the Output:

Finally, to view the result of the Lambda invocation, use the type command to display the content of output.txt.

Command to View Output:

type output.txt

The output will look like this:

{
  "statusCode": 200,
  "body": "{\"message\": \"Hello, Madhurima!\",
  \"processedNumber\": 10, \"version\": \"1.0.0\"}"
}

Explanation of the Output:

statusCode: 200:
- The Lambda function executed successfully and returned an HTTP status code of 200, which signifies success.
body:
- This contains the actual response from the Lambda function in JSON format. The body field includes:
- "message": A greeting message, "Hello, Madhurima!", generated by the function based on the name input from the payload.
- "processedNumber": The result of the computation (doubling the number value from the payload). For the input 5, the result is 10.
- "version": The version of the Lambda function, which is "1.0.0" in this case.

Summary:

Delete Old Function: Use the delete-function command to remove the existing Lambda function.
Recreate Lambda Function: Package the updated code into a zip file and use the create-function command to deploy it.
Invoke the Function: Use the invoke command with a payload to test the updated Lambda function.
Check the Output: View the output in output.txt to ensure the Lambda function processed the payload correctly and returned the expected response.

By following these steps, the updated Lambda function processes the payload, performs the computation, and returns the desired results.

Useful Resources for Learning about AWS Lambda

1. VTI Cloud - AWS Lambda Introduction and Configuration for Beginners

An introduction to AWS Lambda and how to configure it for beginners | VTI CLOUD

We’ve heard a lot about the term “serverless,” how they work, and the benefits that your application brings. A serverless service in general will not require any server provision to run the application. When you run an application on serverless, users will not have to worry about setting up the operating system, patching, or extending […]

vticloud.io

This article offers a beginner-friendly introduction to AWS Lambda, explaining the service and its configuration for those just starting with serverless computing.

2. AWS Lambda Documentation - Welcome to AWS Lambda
docs.aws.amazon.com

The official AWS documentation provides a comprehensive overview of AWS Lambda, including how to get started, manage functions, and integrate with other AWS services.

3. Dev.to - A Beginner's Guide to AWS Lambda

A Beginner's Guide to AWS Lambda

Sedat SALMAN for AWS Community Builders ・ Apr 6 '23

#aws #lambda #cloud #tutorial

This guide provides an easy-to-follow introduction to AWS Lambda, ideal for newcomers who want to understand the basics of Lambda and its use cases.

4. Medium - The Complete Beginner’s Guide to Creating an AWS Lambda Function from Scratch

The Complete Beginner’s Guide to Creating an AWS Lambda Function from Scratch | by Dr. Soufiane El Horma | Medium

Dr. Soufiane El Horma ・ Jan 25, 2024 ・
Medium

This comprehensive guide takes readers through the process of creating AWS Lambda functions, starting from scratch and exploring best practices.

5. Medium - AWS Lambda Functions: A Comprehensive Guide

AWS Lambda Functions: A Comprehensive Guide | by Inexturesolutions | Medium

Inexturesolutions ・ Nov 6, 2023 ・
Medium

A detailed guide that explains AWS Lambda functions, providing insights into their structure, functionality, and use cases in serverless computing.

6. Medium - Learn AWS Lambda in 10 Minutes

Learn AWS Lambda in 5 minutes. Introduction to a famous serverless… | by Jo-Yu Liao | Medium

Jo-Yu Liao ・ Apr 4, 2020 ・
zoejoyuliao.Medium

A quick tutorial for learning AWS Lambda in 10 minutes. This resource is perfect for those who want to quickly understand the basics of Lambda and its use in cloud applications.

📄 Want to see the output step by step? Check it out here:

🔗 Experiment 5 Output (PDF)

🧠 Curious about how each Lambda function runs and responds? See the detailed input-output flow here:

🖥️ AWS Lambda Execution Flow and Output (PDF)

🎉 And that's a wrap! Huge congratulations—you’ve just taken your first steps into the powerful world of serverless computing! 🎖️

You now know how to create, configure, and run AWS Lambda functions — all without touching a single server.

💡 I hope this guide helped you! I added a lot of examples and detailed breakdowns because when I started, serverless felt magical and mysterious! ✨

So, I wanted to make it as clear and beginner-friendly as possible. Would love to hear your thoughts and questions!

🔥 Stay tuned for the next article!

We’re diving into Cloud Load Balancing and Auto Scaling — the dynamic duo that keeps your apps lightning fast ⚡ and super resilient 💪 under any traffic!

💬 Did you enjoy this guide?

Drop a comment if it helped, share your favorite Lambda use-cases, or tell me about your first serverless project!

Let’s keep scaling new heights in the cloud! 🚀☁️

📡 Setting Up and Configuring Cloud Networking 🔐

Madhurima Rawat — Sun, 13 Apr 2025 14:15:55 +0000

🦈 Hello Storage Sharks!

May all your files get accommodated in S3 and your bucket be filled forever! 🪣📁

📚 All code, docs, and resources are available in my GitHub repository:

madhurimarawat / Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Cloud-Computing

Tools and Technologies ⚙️💻

1. AWS CLI

2. LocalStack

3. Docker

Docker is a containerization platform that allows developers to build, share, and run applications in isolated environments called…

View on GitHub

In the previous article,

🗂️ Working with AWS S3 Cloud Storage Services 💾

Madhurima Rawat ・ Apr 4

#tutorial #cloudcomputing #aws #s3

we explored AWS S3, understanding how to create buckets, upload files, and access files.

Now, let’s move on to the next exciting chapter: Cloud Networking and Configuration! 🌐🛠️

We’ll start by breaking down what cloud networking really means — how resources interact, how traffic flows, and how configurations like security groups, subnets, and VPCs come into play. Then we’ll dive straight into hands-on examples to set up and configure networking for your cloud environment.

Stay tuned — we’re about to get connected in the cloud! 🚀🌩️

1. Overview of Virtual Private Cloud (VPC)

A VPC is a customizable virtual network within the AWS cloud. It allows you to manage networking resources securely and privately.

Allows the creation of isolated networks in the cloud.
Provides control over IP address ranges and routing.
Enables creating public and private subnets.
Facilitates secure, private communication between instances.

2. Key Components of a VPC

The VPC is made up of several essential components for creating a robust networking environment:

Subnets: Segment the network into smaller, manageable sections.
Internet Gateway (IGW): Provides internet access for public-facing resources.
Route Tables: Direct traffic within the VPC and between the internet.
NAT Gateway: Allows private subnet instances to access the internet.
Security Groups & Network ACLs: Define inbound and outbound traffic permissions for resources.

3. Networking Flow and Communication

The VPC enables seamless communication between instances and secure internet access.

Public Subnets: These subnets have internet access via an IGW, making them suitable for web servers.
Private Subnets: Isolated subnets that don't directly connect to the internet; they access the internet via a NAT Gateway.
Route Tables: Manage the routing of network traffic within the VPC, to/from the internet or between subnets.
Security Layers: Using Security Groups and Network ACLs to filter traffic and secure resources.

4. LocalStack for VPC Simulation

LocalStack simulates AWS services locally, allowing you to test cloud configurations without needing actual AWS resources.

Simulates AWS Cloud Services: Enables local testing for cloud networking and services like VPC.
Supports Full AWS Environment: Includes EC2, S3, VPC, and more.
Custom Endpoints: Interact with LocalStack through custom endpoint URLs, mimicking real AWS behavior.
Ideal for Development and Testing: Provides a cost-effective solution for developers to test network setups.

5. Use Cases of VPCs

VPCs can be used for various networking purposes to create secure, scalable architectures.

Web Hosting: Deploy secure web applications with public-facing and private backends.
Hybrid Cloud: Extend on-premise networks into the cloud via VPN or Direct Connect.
Big Data & Machine Learning: Isolate workloads for processing and analysis.
Multi-Tier Architectures: Separate frontend, application, and database tiers for better security.

🖥️ All Components of a VPC Implemented in LocalStack

LocalStack is a local AWS cloud emulator that allows developers to create and test AWS services, including Virtual Private Cloud (VPC), on their local machines.

🌐 1. Virtual Private Cloud (VPC)

Definition: A logically isolated network where cloud resources like EC2 instances, databases, and load balancers are deployed.
Key Attributes:
- VPC ID: Unique identifier (e.g., vpc-12345678).
- CIDR Block: Defines the IP range for the VPC (e.g., 10.0.0.0/16).
- Default vs. Custom VPC:
- Default VPC: Automatically provided by AWS.
- Custom VPC: User-defined network configuration.

🛠️ CIDR (Classless Inter-Domain Routing) & CIDR Block Explained

CIDR stands for Classless Inter-Domain Routing, a method used to allocate IP addresses efficiently and manage network traffic. It replaces the traditional class-based addressing (Class A, B, C) with a more flexible prefix-based system.

📌 What is a CIDR Block?

A CIDR block defines a range of IP addresses that can be used within a network, such as a VPC or Subnet. It consists of:

Base IP Address → The starting point of the network (e.g., 192.168.1.0).
Subnet Mask (Slash Notation /X) → Determines how many IPs are in the range.

🔢 How CIDR Works (Subnet Mask & IP Ranges)

CIDR notation looks like:

  192.168.1.0/24

192.168.1.0 → Base IP Address
/24 → Subnet Mask (Means first 24 bits are fixed for network, remaining bits for hosts)
- IP Address Calculation:
/24 allows 256 total IPs (2^(32-24) = 256), but usable IPs are 254 because:
- 1st IP (192.168.1.0) → Network Address
- Last IP (192.168.1.255) → Broadcast Address

📊 Common CIDR Blocks & Their Usages

CIDR Block	Subnet Mask	Total IPs	Usable IPs	Example Usage
`10.0.0.0/8`	255.0.0.0	16,777,216	16,777,214	Large private networks (Enterprise)
`192.168.1.0/24`	255.255.255.0	256	254	Small office/home network
`172.16.0.0/16`	255.255.0.0	65,536	65,534	Medium-sized private networks
`10.0.1.0/28`	255.255.255.240	16	14	Small subnet for internal services

📌 Real-World Example

🏢 Company Network Setup in LocalStack

A company wants two separate environments:
1. Public Subnet for web servers → 192.168.1.0/24
2. Private Subnet for databases → 192.168.2.0/24
The VPC is created in LocalStack, ensuring a controlled environment where services can be tested without real AWS costs.

🌟 Summary

CIDR Block defines IP ranges within a network (VPC, Subnet).
The suffix (/X) determines subnet size, affecting the number of IPs.
Real-world usage: Efficiently dividing networks for different applications (web, database, internal services, etc.).
📌 Real-World Example:
- A company needs a private network for its applications.
- The VPC is created in LocalStack with 192.168.1.0/24 as the IP range, ensuring complete control over networking and security.

📌 2. Subnet (Public & Private)

Definition: A segment of the VPC that organizes resources and determines how IPs are assigned.
Key Attributes:
- Subnet ID: Unique identifier (e.g., subnet-98765432).
- CIDR Block: The subnet's IP range (e.g., 10.0.1.0/24).
- Types of Subnets:
- 🌍 Public Subnet: Connected to the Internet Gateway for external access.
- 🔒 Private Subnet: No direct internet access, used for internal applications.
- Availability Zone (AZ): Each subnet belongs to a specific AWS Availability Zone (e.g., us-east-1a).
📌 Real-World Example:
- A company hosts a web application with two components:
- Public Subnet (10.0.1.0/24) → Hosts a web server that users can access.
- Private Subnet (10.0.2.0/24) → Stores a database that should not be directly exposed to the internet.

🛣️ 3. Route Table

Definition: A set of rules (routes) that direct traffic within the VPC.
Key Attributes:
- Route Table ID: Unique identifier (e.g., rtb-56789).
- Routes: Define paths for network traffic (e.g., 0.0.0.0/0 → Internet Gateway).
- Subnet Association: Subnets are linked to a specific route table.
📌 Real-World Example:
- The public subnet is associated with a route table that contains:
- 0.0.0.0/0 → Internet Gateway (igw-12345), allowing internet access.
- The private subnet is associated with a route table that contains:
- 10.0.0.0/16 → Local Traffic, ensuring internal communication only.

🛜 4. Internet Gateway (IGW) & NAT Gateway

🌍 Internet Gateway (IGW):
- Allows resources in the public subnet to connect to the internet.
- Real-World Example: A web server in a public subnet needs to fetch updates from the internet.
🔒 NAT Gateway (for private subnets):
- Enables outgoing internet traffic from private resources without exposing them to inbound requests.
- Real-World Example: A backend server in a private subnet needs to download software updates, but should not be directly accessible.

⚡ 5. Security Group (SG) & Network ACL (NACL)

🔒 Security Group (SG):
- Acts as a firewall that controls inbound/outbound traffic at the instance level.
- Real-World Example: A web server has an SG allowing inbound traffic on port 80 (HTTP) but blocks everything else.
📜 Network ACL (NACL):
- Controls traffic at the subnet level and applies rules in order.
- Real-World Example: A company sets a rule to block all traffic from a specific IP range to prevent malicious access.

🔗 6. Elastic IP (EIP) & Private IP

📍 Elastic IP (EIP):
- A static public IP assigned to an instance, ensuring it remains the same even after reboot.
- Real-World Example: A company assigns an EIP to a critical web server to maintain a consistent IP for users.
🏠 Private IP:
- Assigned inside a VPC for internal communication.
- Real-World Example: A database only allows connections from a specific private IP assigned to an application server.

🎯 7. VPC Peering & VPN Connection

🤝 VPC Peering:
- Connects two VPCs privately without needing an internet connection.
- Real-World Example: A company connects its production and development environments in different VPCs for seamless data sharing.
🛡️ VPN Connection:
- Connects on-premises networks to AWS securely.
- Real-World Example: A company uses a VPN to access its AWS resources securely from its office.

🛠️ 8. Load Balancer & Auto Scaling

⚖️ Load Balancer (LB):
- Distributes traffic across multiple instances to improve availability.
- Real-World Example: A web application with high traffic uses an Elastic Load Balancer (ELB) to route requests efficiently.
📈 Auto Scaling:
- Dynamically adjusts the number of instances based on demand.
- Real-World Example: An e-commerce website increases server capacity automatically during sales events.

Step-by-Step Guide: Setting Up a VPC in LocalStack

Step 1: Start LocalStack

Start LocalStack using one of the following methods:

Using the CLI

localstack start

Using Docker

docker run --rm -it -p 4566:4566 localstack/localstack

Ensure Docker is Running

Open Docker Desktop and wait until it displays "Docker is running."
LocalStack will simulate AWS services on port 4566, allowing local development without needing an actual AWS account.

Step 2: Create a Virtual Private Cloud (VPC)

Create a VPC (Virtual Private Cloud) to define a private network:

aws ec2 create-vpc --cidr-block 10.0.0.0/16
--endpoint-url=%AWS_ENDPOINT_URL%

What This Does:

Defines a private network (10.0.0.0/16) for AWS resources.
Returns a VPC ID (e.g., "VpcId": "vpc-123456"), which is required for the next steps.

Step 3: Create a Subnet

A subnet allows instances to communicate within the VPC.

aws ec2 create-subnet --vpc-id vpc-123456
--cidr-block 10.0.1.0/24 --endpoint-url=%AWS_ENDPOINT_URL%

What This Does:

Creates a smaller network (10.0.1.0/24) within the VPC for organizing resources.
A Subnet ID (e.g., "SubnetId": "subnet-123456") is returned, which is needed for later steps.

Step 4: Create and Attach an Internet Gateway

An Internet Gateway (IGW) allows resources in the subnet to access the internet.

Create the Internet Gateway:

aws ec2 create-internet-gateway --endpoint-url=%AWS_ENDPOINT_URL%

Attach the Internet Gateway to the VPC:

aws ec2 attach-internet-gateway --internet-gateway-id
igw-123456 --vpc-id vpc-123456 --endpoint-url=%AWS_ENDPOINT_URL%

What This Does:

Enables instances in the VPC to communicate with the internet.
Returns an Internet Gateway ID (e.g., "InternetGatewayId": "igw-123456").

Step 5: Configure Routing for Internet Access

To allow traffic to and from the internet, a Route Table must be created and updated.

Create a Route Table for the VPC:

aws ec2 create-route-table --vpc-id vpc-123456
--endpoint-url=%AWS_ENDPOINT_URL%

Add a Default Route to Enable Internet Access:

aws ec2 create-route --route-table-id rtb-123456
--destination-cidr-block 0.0.0.0/0 --gateway-id
igw-123456 --endpoint-url=%AWS_ENDPOINT_URL%

Associate the Route Table with the Subnet:

aws ec2 associate-route-table --route-table-id rtb-123456
--subnet-id subnet-123456 --endpoint-url=%AWS_ENDPOINT_URL%

What This Does:

Defines a default route (0.0.0.0/0), allowing traffic to flow to the internet.
Links the route table to the subnet so instances can use the IGW.

Step 6: Verify the Configuration

List All VPCs:

aws ec2 describe-vpcs --endpoint-url=%AWS_ENDPOINT_URL%

List All Subnets:

aws ec2 describe-subnets --endpoint-url=%AWS_ENDPOINT_URL%

What This Does:

Confirms the VPC and subnets have been successfully created and configured.
Ensures all components (VPC, subnet, IGW, and routing) are correctly set up.

Useful Resources for Learning about Virtual Private Cloud (VPC)

1. Spiceworks - What is Virtual Private Cloud?

What Is Virtual Private Cloud (VPC)? Definition, Key Components, Best Practices, and Providers - Spiceworks

A virtual private cloud (VPC) is an isolated and secured cloud environment usually created within a public cloud. Click here to understand VPC, its key components, and best practices.

spiceworks.com

A comprehensive guide explaining the concept of Virtual Private Cloud (VPC), its benefits, and how it enhances network security and control over cloud resources.

2. Medium - Deep Dive into Amazon Virtual Private Cloud (VPC)

Amazon VPC: Core Features Explained | Medium

Alexander Obregon ・ Jan 6, 2024 ・
Medium

A detailed exploration of the core features of Amazon VPC, focusing on security, scalability, and flexibility in network configurations.

3. Medium - AWS VPC: Virtual Private Cloud

AWS VPC(Virtual Private Cloud). Amazon Web Services (AWS) Virtual… | by Deepeshjaiswal | Medium

Deepeshjaiswal ・ Feb 19, 2023 ・
Medium

A beginner-friendly article that explains the fundamental concepts of AWS VPC, its components, and how to leverage VPC for secure cloud networking.

4. Dev.to - A Beginner's Guide to AWS Virtual Private Cloud (VPC) Security

A Beginner's Guide to AWS Virtual Private Cloud (VPC) Security

Sedat SALMAN for AWS Community Builders ・ Mar 20 '23

#aws #cloud #security #vpc

This beginner-friendly guide introduces AWS VPC and focuses on its security features. It explains the importance of securing VPCs in cloud environments and provides practical tips for setting up secure VPC architectures on AWS.

📄 Want to see the output step by step? Check it out here:

🔗 Experiment 4 Output (PDF)

🧠 Want to see the input and output for each command in the Command Prompt along with a clear explanation? Check it out here:

🖥️ Command Prompt Input & Output Explanation (PDF)

🎉 And that’s a wrap! Congrats and kudos—you have successfully configured a cloud network! 🎖️

You’ve now understood the foundations of cloud networking and AWS VPC.

💡 I hope this guide helped you! I know this one was a bit long—but that’s because I was super curious and a little confused myself while exploring so many new concepts. 😅

So, I added lots of detailed explanations to make it easier for anyone else diving into it. Do let me know your thoughts and feedback!

🔥 Stay tuned for the next article!

We’re going to explore the heart of the cloud: Serverless Computing with AWS Lambda — the saviour of developers and servers alike! 🦸‍♂️☁️

It’s going to be super exciting, so make sure to stay connected and drop your comments, questions, or even your own cool resources!

💬 Did you like this guide?

Leave a comment if it helped, share your favorite cloud networking diagrams, or anything else you found useful!

Let’s continue exploring the cloud—one byte at a time! 🌟💛