Scraping Social Media with Gemini for Sentiment Analysis
Hackathon project: using Gemini + a user's browser to analyze real sentiment hidden inside social media comment sections.
Inspiration
If you want to understand how people actually feel about a product, brand, or topic, the best data source is often comment sections.
The problem: scraping social platforms is getting harder every year.
Platforms now deploy:
- expensive APIs and paywalls (e.g. Reddit's API changes)
- aggressive bot detection
- rate limits and scraping prevention
Running scrapers from servers or headless browsers usually gets blocked almost immediately.
So we asked:
What if the scraper wasn't a bot at all — but a real user's browser?
Modern browser automation combined with Gemini's computer-use capabilities makes that possible.
A real browser session comes with some powerful advantages:
- ✅ already authenticated to social platforms
- ✅ trusted by anti-bot systems
- ✅ capable of normal browsing behavior
In other words: the ultimate scraping environment already exists — the user's browser.
What It Does
Textpot leverages a user's browser to explore social media and collect comments for sentiment analysis.
The architecture separates browser control from AI decision making.
The system runs a loop (up to 3 turns) to navigate a page and analyze comments:
- Extension captures screenshot
- Screenshot POSTed to Cloud Run
- Gemini analyzes the screen and returns the next action
- Extension executes the action via CDP
The result is a feedback loop where:
- the browser acts
- Gemini decides what to do next
This allows Textpot to automatically explore comment sections and extract sentiment insights.
Architecture
The system is split into two parts.
1. Chrome Extension (User Machine)
The extension owns the browser.
Responsibilities:
- opens the extension popup
- attaches to the page via Chrome DevTools Protocol (CDP)
- performs actions (click, scroll, keypress)
- captures screenshots of the page
Everything runs directly inside the user's local Chrome session.
2. Cloud Run Backend
The backend owns the AI logic.
Responsibilities:
- receives screenshots from the extension
- sends them to Gemini
- stores conversation history across turns
- returns the next action to perform
Importantly:
The backend never directly touches the browser.
It only tells the extension what action to perform next.
Why This Architecture Works
Splitting responsibilities between browser and AI backend solves a major scraping problem.
The browser:
- has real authentication
- has real cookies
- behaves like a normal user
Cloud Run simply tells it:
"Click here."
"Scroll down."
"Open this comment thread."
This approach bypasses many of the traditional scraping roadblocks.
Challenges We Ran Into
The first version of Textpot looked very different.
Initially we built it as a web app running a headless browser on Cloud Run.
That approach quickly failed.
Problems included:
- bot detection blocking the browser
- authentication failures
- restricted access to social media pages
The fix was simple but important:
Move the browser to the user.
Once we pivoted to a Chrome extension, the system could use:
- real user sessions
- real cookies
- normal browsing behavior
That solved most of the blocking issues immediately.
What's Next for Textpot
Next steps include:
- polishing the extension UX
- improving comment extraction
- adding deeper sentiment analysis
- launching on the Chrome Web Store
Because part of the system runs on Google Cloud Run, we'll also need to figure out a sustainable pricing model.
Final Thoughts
AI-powered browser automation opens up a new way to interact with the web.
Instead of fighting platform restrictions with bigger scrapers, we can:
- use real browsers
- keep AI in the backend
- let models like Gemini decide how to navigate
For sentiment analysis and market research, this could unlock data sources that are otherwise extremely difficult to access.
If you're experimenting with Gemini computer use, browser automation, or AI agents, I'd love to hear how you're approaching it.
Top comments (0)