Logging into websites every single run is the fastest way to:
🚫 trigger anti-bot systems
🧩 face endless CAPTCHAs
🔒 get accounts locked
🐌 slow down your scraper
The correct pattern is simple:
Login once → persist browser profile → reuse it forever (until expiration).
In this article I will show how this works in a real project:
👉 https://github.com/AmaLS367/parts_info_collector
This project automates data extraction from Gemini’s web UI and keeps authentication between runs using a persistent browser profile.
🧠 The Core Idea
Instead of exporting cookies manually, Playwright can launch Chromium with a persistent user profile directory.
That directory stores:
cookies 🍪
localStorage
IndexedDB
login tokens
session metadata
Once created, it becomes your reusable authenticated identity.
📂 How the Project Does It
In parts_info_collector, authentication is handled via:
a user-data/ directory
first interactive run via first_start.bat
manual login to Gemini
all next runs reuse the same browser profile
So the flow is:
1️⃣ Run once with UI
2️⃣ Log in manually
3️⃣ Browser profile is saved
4️⃣ Next runs are headless and already authenticated
No repeated login.
No constant CAPTCHA hell 😌
🔐 Persistent Context in Playwright
This is the key API:
browser_context = playwright.chromium.launch_persistent_context(
user_data_dir="user-data",
headless=True
)
That single folder is everything.
If it exists, Playwright loads it.
If not, you run in visible mode and authenticate.
🧑💻 First Run: Create the Session
In the project, the first launch happens with UI enabled so you can log in:
browser_context = playwright.chromium.launch_persistent_context(
user_data_dir="user-data",
headless=False
)
You open Gemini, authenticate manually, and close the browser.
From that moment:
📁 user-data/ contains your session.
♻️ All Next Runs: Fully Headless
Subsequent executions simply reuse the same folder:
browser_context = playwright.chromium.launch_persistent_context(
user_data_dir="user-data",
headless=True
)
You are already logged in.
No forms.
No credentials.
No redirects.
⚠️ What Happens When the Session Expires?
Eventually cookies die.
The project handles this operationally:
you delete user-data/
or rerun first_start.bat
login again
profile is recreated
Simple, manual, and reliable.
You can extend this with:
detecting redirect to /login
checking DOM markers
auto-relogin logic
alerting
🛡️ Why Persistent Profiles Beat Cookie Dumps
Using a persistent profile directory is stronger than just exporting cookies:
✅ keeps IndexedDB tokens
✅ survives browser restarts
✅ mimics real user Chrome profile
✅ less suspicious than scripted logins
✅ perfect for long-running monitors
🚀 When You Should Use This Pattern
This approach is ideal for:
📊 price monitors
🤖 automation bots
🧠 AI web scrapers
📈 background workers
🔁 periodic collectors
Anything that runs for weeks.
🔗 Real Repository
Full project:
👉 https://github.com/AmaLS367/parts_info_collector
🧾 TL;DR
✅ use launch_persistent_context
✅ store profile in user-data/
✅ login once manually
✅ reuse forever
✅ refresh only when expired
Top comments (3)
Solid pattern, and the launch_persistent_context approach is definitely cleaner than dumping cookies manually.
One thing worth knowing for anyone running this at scale or on multiple machines: the persistent profile solves the "login again every run" problem, but a saved user-data dir still produces a browser that looks like a headless Chromium to fingerprinting systems. Canvas hash, WebGL renderer, screen resolution, those come from the machine running it, not from the profile.
So if you're running this on a cloud VM or several parallel instances, sites that use Akamai/Kasada/Cloudflare Bot Management will often still catch you even with a perfectly preserved session. The session keeps you authenticated, it doesn't make the browser look human.
Great writeup though - the "login once, reuse forever" flow is exactly the right mental model for long-running scrapers.
Thanks for sharing your opinion! Appreciate your right thoughts)
Cheers Ama, glad it was useful. It's one of those things that only bites you after you've already built the session management piece and think you're done!