DEV Community: Tech Auto Lab

I Added Retries to My Automation and It Doubled the Failures

Tech Auto Lab — Sun, 04 Oct 2026 15:08:19 +0000

Retries made my Playwright jobs worse, not better: failure rate went from 3.1% to 6.7% in one week. The culprit wasn't flaky code — it was retrying things that should never be retried.

I found this the hard way after wiring a blanket retry wrapper around every step of my account-automation pipeline. Here's what broke and what actually fixed it.

1. Retries turned "already done" into "done twice"

The worst failure class had nothing to do with the network. A step would time out on the response, my wrapper would retry, and the step would run a second time — because the first attempt had actually succeeded server-side.

In two days I double-posted to the same account twice and sent a duplicate signup to one platform. No exception, no error log. Just silent duplicates.

# WRONG: retry on any exception, no idempotency key
def step(action):
    for _ in range(3):
        try:
            return action()
        except Exception:
            time.sleep(2 ** _)
    raise

2. The rule I actually needed: retry only idempotent actions

I split every step into two buckets:

Safe to retry — reads, GETs, checks. Retry freely.
Unsafe to retry — writes, posts, sends. Retry only with an idempotency key, or not at all.

The write steps got an explicit key (account_id + action + date) so a replay would be rejected instead of duplicated.

3. Backoff without jitter synchronized my retries

Every failed job retried on the same exponential schedule, so a brief API hiccup would make 20 workers hammer the endpoint at the exact same second. Adding jitter — a random 0–500 ms — spread them out and cut the second-order failures by more than half.

time.sleep(2 ** attempt + random.uniform(0, 0.5))

4. Numbers after the fix

Metric	Before	After
Failure rate	6.7%	2.9%
Duplicate writes	5 in 2 days	0 in 7 days
Retry storms	~2/day	0

The change that mattered most wasn't the retry count or the backoff curve. It was deciding what deserves a retry at all.

I kept the idempotency-key helper in the shared utils module — it's now the default for every write step in the pipeline. Next I'm applying the same split to the signup flow, which still has one hot spot where a captcha timeout re-submits the whole form.

What's the one place in your automation you retry blindly?

I slowed my scraper down 10x and it finally stopped getting banned

Tech Auto Lab — Thu, 01 Oct 2026 17:06:58 +0000

Faster is not better. I spent two weeks squeezing every millisecond out of my Playwright scraper, and every speedup bought me a new IP ban. The fix was throttling it down to a tenth of the speed — bans dropped to zero.

The number that made me stop and re-read my logs

I was running a headless Playwright loop over a public catalog, ~120 requests a minute, no delay, no jitter. The result was predictable: a fresh datacenter IP died in under 40 minutes, and I was burning residential proxy traffic on a loop that wasn't even finishing.

The log line that broke the illusion: the scraper wasn't failing on the target site at all. It was failing on the rate-limiter page — a 429 that my retry logic treated as a transient error and hammered even harder. I had built a feedback loop that made the ban faster every time it retried.

What I changed, in order

Paced requests instead of bursting them. I capped the loop at 8 requests per minute with a random 2–6 second sleep between hits. Not a fixed sleep(5), but sleep(random.uniform(2, 6)) — fixed delays are a fingerprint too.
Stopped retrying on 429. A 429 is not "try again in a second", it's "back off or leave". I added an exponential backoff that starts at 60 seconds and doubles, capped at 10 minutes, and it stops the loop entirely after three consecutive 429s.
Spread the load across sessions, not threads. I had four concurrent browser contexts sharing one IP. Now one context per IP, one page at a time. Throughput per minute went down, total pages completed per hour went up, because nothing was getting killed mid-run anymore.
Dropped the "faster" micro-optimizations. I had parallelized page loads, disabled images, disabled CSS, and used raw HTTP where I could. All of it made the traffic pattern look less human. I reverted to a normal page load with default timeouts.

The hot-take

The bottleneck was never my code's speed. It was my traffic's shape. A scraper that finishes 100 pages and gets banned collected 100 pages. A scraper that finishes 50 pages an hour and stays alive collects 500 by morning. Rate limits are not a bug to work around — they're the only signal a site gives you before it silently shadow-bans your IP.

What I'd tell past me

Measure pages completed per day, not requests per second. If your success rate is under 100%, you don't have a speed problem — you have a survival problem. Slow the loop down until bans stop, then inch it back up. You'll find the real ceiling is far lower than your code's ceiling, and that's fine.

I stopped logging in every run and my automation stopped getting flagged

Tech Auto Lab — Mon, 28 Sep 2026 15:06:18 +0000

Fresh logins are what get you flagged. Reusing a saved session is what keeps the bot alive. I found this out after 41 accounts: the ones I re-logged into every run died first.

For months my automation treated every browser run like a brand-new visitor. Open MoreLogin, navigate to the login form, type the password, solve the captcha, submit. It worked — until the captchas got harder, the sessions started dying mid-run, and three accounts got flagged in one week. Then I switched to persisting sessions, and the failure rate dropped from roughly 1 in 4 runs to 1 in 30.

Here's the shift, in the order it mattered.

1. A warm cookie jar beats a fresh password every time

Every fresh login is a signal: new device, new IP, a burst of navigation, then a form submission. Anti-bot systems are built to notice exactly that. A persisted session looks like a returning human.

# BEFORE: log in every run, solve the captcha every run
context = browser.new_context()
context.goto("https://example.com/login")
# ... type, solve captcha, submit, hope

# AFTER: inject a saved session, go straight to work
context = browser.new_context(storage_state="session_state.json")
context.goto("https://example.com/dashboard")

The key detail: reuse the same browser fingerprint and the same state file. Changing either one defeats the point.

2. Capture the session once, store it, reuse it until it rots

I capture storage_state after a successful manual login, save it to a file, and reload it until the session dies. Then — and only then — I log in fresh once and capture again.

# capture after the one good login
context.storage_state(path="session_state.json")

# reuse it across runs until a 401/redirect tells me it's stale

A stale session announces itself clearly: a redirect to /login, a 401, or a page that should have my name and doesn't. I detect that and re-capture, instead of logging in preemptively.

3. Keep the session alive with cheap touch requests, not full re-logins

Sessions rot on a schedule, but a light touch resets the clock. A periodic GET on a page the session already owns is far cheaper — and far less suspicious — than a full re-login.

# cheap heartbeat: keep the cookie warm without re-authenticating
page.goto("https://example.com/dashboard")
if "login" in page.url:
    reauthenticate_and_recapture()

4. One session per identity, never shared

The moment I reused a session across two accounts, both got flagged within a day. A session is a fingerprint of one identity. Sharing it is the fastest way to link two profiles to the same operator.

The rule I now follow: one state file per account, stored under the account's own name, never touched by another job.

What I'd do differently next time

Capture the session on the very first login, before doing anything else.
Detect staleness by URL or a marker element, never by a timer guess.
Never share a state file between accounts — isolate by name from the start.

The takeaway that surprised me most: the bot didn't get more stealthy by acting smarter. It got safer by stopping the one thing that screams "automation" — logging in over and over again.

I'm logging this automation series build-in-public — which is cheaper for you in practice: a fresh login per run, or a persisted session you refresh once in a while? Tell me in the comments.

I deleted 200 lines of Playwright waits and my automation got more stable

Tech Auto Lab — Fri, 25 Sep 2026 14:14:43 +0000

Your automation isn't flaky because it's missing sleeps. It's flaky because you keep adding them.

I spent my first weeks of browser automation treating Playwright like Selenium: time.sleep(2) after every click, wait_for_timeout sprinkled everywhere, and a wall of try/except around everything that might race. The scripts worked on my machine and died at 3 a.m. in production.

Then I deleted nearly all of it. The scripts got faster and stopped failing. Here's the shift, in the order it matters.

1. Stop waiting for elements. Wait for state instead.

Every sleep encodes a guess about how long something takes. The guess is always wrong somewhere — a slow CI box, a cold cache, a network hiccup.

Playwright auto-waits for the element to be actionable if you use the right locator. The moment I replaced sleeps with state-based waits, the flakiness dropped.

# BEFORE: guess the timing, hope it's enough
page.click("text=Submit")
page.wait_for_timeout(3000)

# AFTER: wait for the *outcome*, not the clock
page.get_by_role("button", name="Submit").click()
page.get_by_text("Welcome back").wait_for()

2. Role-based locators survive redesigns. CSS selectors don't.

My first selectors were pure CSS: #submit-btn, .login > form > input[3]. Every site redesign broke them, and I was the one who found out at 2 a.m.

Switching to role-based locators — get_by_role, get_by_label, get_by_text — was the single biggest stability win. They describe what a user sees, which barely changes, instead of how the DOM is shaped, which changes constantly.

# BEFORE: breaks when the class is renamed
page.locator(".form__submit--primary").click()

# AFTER: survives a redesign
page.get_by_role("button", name="Publish").click()

3. Web-first assertions catch the failure where it happens, not three steps later

The old way: click, sleep, read some value, compare it yourself, raise your own error. By the time my custom check ran, the actual failure was three steps behind.

expect assertions poll for the condition automatically and fail at the step that broke, with the state of the page in the message. That alone cut my debugging time more than any log line I ever wrote.

from playwright.sync_api import expect

page.get_by_role("button", name="Publish").click()
expect(page).to_have_url(re.compile(r"/myhandle/my-slug"))

4. The retry belongs at the top, not wrapped around every click

I used to wrap individual actions in try/except and retry them in place. That hides the real failure: if a click needs a retry, the page is already in a state you didn't expect.

The correct place for a retry is the whole task — re-run the job from a clean context. If the task is idempotent, a clean retry fixes far more than a hundred in-place excepts.

for attempt in range(3):
    try:
        run_job()
        break
    except JobError:
        if attempt == 2:
            raise

What I'd do differently next time

Write role-based locators from the first line, not as a refactor.
Ban wait_for_timeout in review unless the wait has a comment saying why no state condition exists.
Make every automated task idempotent so the clean retry is actually safe.

The takeaway that surprised me most: less code, fewer sleeps, and a browser that waits for state instead of clocks — that's the whole trick. The flakiness was never about timing. It was about describing what I wanted instead of guessing how long it would take.

Which wait pattern still bites you in production? Tell me in the comments — I'm logging this series build-in-public and I read every one.

The Bugs That Killed My Automation Fleet Never Threw an Error

Tech Auto Lab — Tue, 22 Sep 2026 14:12:21 +0000

The scary part of running an automation fleet isn't the error you see — it's the one that never throws. My accounts didn't crash when they failed. They quietly poisoned themselves, and I only noticed days later when the ban wave landed.

Over the last month I scaled from one headless browser to 40 concurrent accounts. Along the way I found that the three costliest bugs produced no stack trace at all. Here they are, in the order they bit me.

1. reCAPTCHA v3 fails without a checkbox

There is no "I am not a robot" to click. The score just decays silently on every request, and by the time a form rejects you, the account is already flagged. I logged token scores for a week to prove it: clean manual flows sat around 0.9, my automated ones averaged 0.3 — and the worst part is the requests still returned 200 the whole time.

The fix wasn't a solver. It was treating every action like a gesture: slower typing, real pauses, and keeping one session inside one geographic identity for its whole lifetime. The score crawled back up to 0.7 without touching the captcha code.

2. The confirm-by-link flow broke accounts that looked fine

Signup forms were solved long ago. The step that silently killed 6 accounts was email confirmation: I clicked the confirmation link in the wrong browser context, and the platform bound the confirmation to the wrong session. No error, no warning — the account just stayed unverified forever, and every post after that hit a wall I couldn't see.

The rule that stopped it: one context per email, and confirm inside the same session that submitted the form. Boring, but it removed an entire failure class.

3. I had no idea which accounts were actually healthy

After the first ban wave I realized I couldn't answer the simplest question: which of my 40 accounts still worked? I had logs of commands, but nothing tying an account to its IP, its fingerprint, and its last real outcome.

The single most useful thing I built was a 20-line audit log — one row per run: account, platform, IP, fingerprint hash, outcome. When the next wave hit, I diffed locked vs. alive in five minutes instead of guessing for a day. The log didn't make anything faster; it made everything visible.

4. Warm-up that isn't measured isn't warm-up

I "warmed up" accounts by waiting 24 hours and called it done. Then I looked at the data and found the accounts I'd actually used during warm-up — one real read, one profile edit — survived at twice the rate of the ones that just sat idle. Passive waiting did almost nothing; light real activity did the work.

5. Two accounts, one platform, same text = a repeat, not scale

I caught myself cross-posting near-identical text to two accounts on the same site. It wasn't banned, it was just noise — and the moment anyone compared the feeds it would have been an obvious duplicate. Same platform needs a different angle, or it doesn't need to happen.

The lesson I'd compress it all into: the bugs that kill an automation fleet are the ones that never throw an error. If you can't tell which accounts are healthy at a glance, you're running blind no matter how many proxies you have.

If you're scaling your own fleet, build the audit log before you build the automation. Visibility first, proxies later — that order would have saved me a full week.

I Automated reCAPTCHA in Playwright and the Proxy Was the Real Fix

Tech Auto Lab — Sat, 19 Sep 2026 13:26:04 +0000

I spent three days trying to get a headless Playwright bot past reCAPTCHA. I assumed the captcha was the hard part. It wasn't — the IP reputation was. Here are the five things that actually moved the needle, in the order I learned them.

1. Your datacenter IP is already burned before the page loads

The first run failed on a clean datacenter IP before the captcha even rendered. The challenge never appeared; the request just came back flagged. That's the reputation layer doing its job: the moment the datacenter ASN shows up, the score is already too low for any token to matter.

The fix was a residential proxy pinned to the target region. Same browser, same script, same sitekey — one IP swap later and the challenge finally showed up on screen.

2. Solve the token server-side, not with a click simulator

Once the challenge was visible, I dropped the idea of clicking tiles with Playwright. You don't need to interact with the widget at all. You send the sitekey and the page URL to a solving API, get back a response token, and inject it into the hidden field:

token = solver.solve(sitekey=sitekey, url=page.url)
page.evaluate(
    "k => document.getElementById('g-recaptcha-response').value = k",
    token,
)

Then submit the form as a human would. The token is what the backend validates; how the checkbox looks is irrelevant.

3. The token is bound to the IP that requested it

This is the one that cost me a full evening. I solved the token on one connection and submitted the form through a different proxy. Rejected every time. reCAPTCHA binds the token to the IP that opened the challenge, so the solve and the submit have to come from the same exit node.

I moved the solver call to run from the same residential session as the browser, and the rejections stopped cold.

4. Reuse the session, not just the token

A fresh cookie jar gets challenged on every single page load. Once a session passes one challenge, the cookie carries that trust for a while. I started persisting the browser context between runs instead of tearing it down:

context = browser.new_context(storage_state="state.json")
# ... do work ...
context.storage_state(path="state.json")

Warm sessions hit fewer challenges, which means fewer tokens, which means fewer paid solves and a much faster loop.

5. A proxy that dies mid-session is worse than no proxy

I had a residential provider that rotated the exit node every few minutes. Every rotation was a new IP, a new reputation, a new challenge. The "helpful" rotation was actively breaking my solves. I switched to sticky sessions — one IP held for the whole run — and the error rate dropped to near zero.

The lesson I'd compress it all into: the captcha is a reputation check, not a puzzle. Spend your effort on the IP and the session, and let a solver handle the token. That order of priorities saved me more time than any widget interaction ever did.

If you're automating anything with reCAPTCHA, start with sticky residential sessions and server-side token injection — the solver and the clicks come last.

I Automated Cross-Posting to 5 Platforms and Broke 3 Things

Tech Auto Lab — Tue, 15 Sep 2026 14:29:28 +0000

Last month I built a small pipeline that takes one draft and publishes it to five platforms with per-site formatting. The idea was to write once, publish everywhere. What I got instead was three distinct failure modes that only showed up when the volume scaled.

Here's what broke, in order, and what fixed each one.

1. The shared-draft collision

I used one JSON file as the handoff between the writer and the publisher. Two publisher tasks ran in parallel, both reading the same file. One finished, the other overwrote it — and the second platform got the first platform's draft.

The fix was boring and correct: drop the shared file entirely. Each draft now lives in a database row with its own id, and the publisher reads by id. One writer, one row, no shared mutable state.

2. Markdown that isn't Markdown

Each platform parses Markdown slightly differently. Code fences that rendered fine on one site got mangled on another — inline code lost its backticks, headings collapsed, a numbered list became a paragraph.

The fix wasn't a universal formatter. It was a per-platform template: the same body, but headers, fences, and tags rewritten to each site's dialect before send. Formatting is now a field on the mechanic, not an assumption in the code.

3. Rate limits that only fire at cadence

Single posts sailed through. Publishing two or three a week to the same platform was fine — until I hit the same platform twice in one hour. Then the second post silently throttled, no error, just a post that never surfaced.

The fix was a scheduler with a real cadence: each pair gets a minimum gap, and no two posts share an hour. Slowing the loop down fixed more than any retry logic ever did.

The three lessons compress to one rule: never let two parts of the pipeline share mutable state, and never let the scheduler run faster than the platform's patience. Everything else is just plumbing.

If you're building something similar, I'd start with the database row and the per-platform template — the scheduler you can tune later.