DEV Community

Sumit Mishra
Sumit Mishra

Posted on

My Headless LinkedIn Bot Fails Silently - Until I Point SigNoz at It

#signoz #opentelemetry #opensource #python

Subtitle: From ChromeDriver 151 vs Brave 150 to correlated traces: what I learn instrumenting a Selenium cron job with OpenTelemetry

Every morning I open yesterday's log file and scroll, guessing.

Some days the bot sends zero connection requests. Other days it hits 27 of 30 and stops. The script runs headless on a schedule — no browser window, no Slack alert, just a text file that might not tell the whole story.

I built this for a SigNoz hackathon. The goal isn't a polished observability tutorial. It's to stop flying blind on a real Python + Selenium bot running on my Mac. Here's what actually breaks, what I instrument, and what shows up in SigNoz when I finally look.

What I Run

The bot sends LinkedIn connection requests up to a daily cap. I run headless Brave on macOS via launchd:

./schedule/install_daily_schedule.sh install
# At login + hourly while the Mac is on; at most one successful run per day
Enter fullscreen mode Exit fullscreen mode

Relevant settings from .env:

BROWSER_HEADLESS=true
USE_BROWSER_PROFILE=true
USE_NETWORK_GROW=true          # fallback when people search is exhausted
AUTO_ACCEPT_INVITATIONS=true   # accept pending invites on Grow first
LINKEDIN_CONNECTIONS=30
LINKEDIN_DRIVER_AUTO_FIX_ATTEMPTS=2

SIGNOZ_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_SERVICE_NAME=linkedin-connection-bot
Enter fullscreen mode Exit fullscreen mode

When people search runs out, the bot switches to My Network → Grow, accepts incoming invitations, then sends Connect requests. Resume state lets a crashed run pick up on Grow without redoing search. A wrapper script (schedule/daily_run.sh) handles lock files, stale process cleanup, and one chromedriver auto-fix retry.

Simple on paper. Messy in production.

The Failures I Can't See From One Log Line

ChromeDriver 151 vs Brave 150

On July 17, most hourly launchd checks looked like this:

Critical error: Failed to initialize brave: ...
This version of ChromeDriver only supports Chrome version 151
Current browser version is 150.0.7871.125
Enter fullscreen mode Exit fullscreen mode

Brave hadn't auto-updated. Selenium had cached ChromeDriver 151. The bot never got past browser launch — zero connections, 784 lines of stack trace.

I add self-healing in Scripts/browser_utils.py: detect the mismatch, kill stale chromedriver and Brave processes, re-download the matching driver:

def auto_fix_chromedriver_mismatch(browser_name, log_func=None):
    cleanup_stale_bot_processes(browser_name, log_func=log_func)
    path = resolve_chromedriver_path(browser_name, force_refresh=True)
    _log(log_func, f"Auto-fix: resolved chromedriver at {path}")
    return path
Enter fullscreen mode Exit fullscreen mode

That fixed launches once Brave caught up. But I still want to know when it happens without grepping logs at 7 a.m.

Almost Done: 27/30 and a Hangup

Late on July 17, after auto-fix worked, the bot resumed Grow-only mode and sent connections steadily — until:

Progress: 27/30 connections sent (grow)
Progress: 28/30 connections sent (grow)
Cleaning up bot browser processes...
./schedule/daily_run.sh: line 19: 52556 Hangup: 1               "$VENV_PYTHON" run_bot.py
Enter fullscreen mode Exit fullscreen mode

Two short of the cap. Killed by SIGHUP. Without exported counters, the only signal is a progress line buried in a 700-line file.

Stale Locks

While one process is still running, the next hourly check logs: Another bot run is already in progress (pid 51947). Skipping. Earlier that evening I saw: Removing stale lock file. The lock logic is correct. Interrupted runs still leave footguns. Metrics like bot.driver_errors and bot.session_crashes surface this on a dashboard instead of me tailing files.

One Module, Three Signals

Everything lives in Scripts/telemetry.py — OTLP gRPC to port 4317, gated by SIGNOZ_ENABLED. Each run gets a 12-character bot.run_id on traces, metrics, and logs.

Spans: bot.run, bot.configure_browser, bot.login, bot.search_page, bot.accept_invitations, bot.connect_from_grow, bot.recover_session.

Metrics: bot.connections_sent (label source=search|grow), bot.invitations_accepted, bot.driver_errors (error_type=version_mismatch), bot.session_crashes, bot.run_success / bot.run_failure, bot.run_duration_seconds.

Main loop wiring:

tel.init_telemetry()
tel.log_event(logger, "Bot run started", phase="start")

with tel.span("bot.run", phase="run"):
    configurations()
    login(webpage, user)
    tel.record_run_outcome(success=not run_failed, connections_sent=connections_sent)
    tel.record_run_duration(time.time() - run_started_at)
Enter fullscreen mode Exit fullscreen mode

On driver mismatch: tel.record_driver_error(error_type="version_mismatch") before auto-fix runs.

Logs append run_id and phase for trace correlation in SigNoz Logs Explorer:

Verify telemetry smoke test | run_id=87626c6dc93a phase=verify
trace_id=7b69827901b867b9f0658ed774ff684d  span_id=0e08651f63956788
Enter fullscreen mode Exit fullscreen mode

One lesson I keep relearning: logs need explicit OTLP export. telemetry.py wires a LoggingHandler plus OTLPLogExporter — stdout files alone aren't enough.

Three-Pillar Correlation — The Part I Care About Most

SigNoz's real payoff for me isn't three separate tabs. It's jumping between logs, traces, and metrics without rebuilding queries. Unified observability means one run_id on every signal, and SigNoz UI links that carry filters and time range when I drill down (dashboard interactivity).

How my bot wires it

Pillar What I emit Correlation hook
Traces Spans like bot.run, bot.login, bot.connect_from_grow bot.run_id span attribute; root span wraps the whole cron run
Metrics bot.connections_sent, bot.run_success, etc. Same bot.run_id label on every counter/histogram sample
Logs log_event() lines with run_id= and phase= in the body LoggingInstrumentor injects trace_id / span_id when the log fires inside an active span (official guide)

Logs emitted inside a span carry both domain context (run_id=87626c6dc93a phase=verify) and OTel trace context (trace_id=… span_id=…). That second pair is what powers SigNoz's View trace button on a log line. Metrics don't get trace IDs on every sample — I correlate them through the shared bot.run_id label instead. SigNoz also derives APM/RED metrics from span data automatically (APM metrics from traces), which gives me operation-level rates alongside my custom bot.* counters.

Walkthrough: one smoke test, three pillars

After ./venv/bin/python Scripts/verify_telemetry.py prints run_id=87626c6dc93a:

  1. Logs → trace. I open Logs Explorer, filter service.name = linkedin-connection-bot, and find Verify telemetry smoke test | run_id=87626c6dc93a phase=verify. The row shows trace_id and span_id. I click View trace (or the trace ID link). SigNoz opens the bot.verify_telemetry span — the same run I just emitted.

  2. Trace → logs. On that trace's span detail panel I open the Logs tab (span details). SigNoz shows the correlated log line from step 1 without me re-filtering by hand. I can also click Go to related logs from the trace list view.

  3. Trace → metrics. Back in Metrics Explorer I query bot.connections_sent with aggregation Sum, group by bot.run_id, and filter bot.run_id = 87626c6dc93a. I see source = verify with value 1 — the counter incremented inside the same span that produced the log. For a demo run with twelve simulated runs (signoz_demo_data.py), I group by source and spot the grow spike that matches a bot.connect_from_grow span in Traces Explorer.

That's the July 17 hangup story in reverse: instead of grepping 700 lines for Progress: 27/30, I click a log from the failed run, jump to bot.run, and check whether bot.connections_sent stopped climbing for that run_id.

Verify correlation in the SigNoz UI (5 minutes)

Run the smoke test, wait ~10 seconds for export, then check each pillar:

Step Where What to confirm
1 Logs Explorer → filter service.name = linkedin-connection-bot Body contains run_id=; log attributes or body include trace_id and span_id
2 Click a log row → View trace Trace opens with span bot.verify_telemetry and bot.run_id attribute
3 Trace detail → Logs tab Same smoke-test log appears in the correlated panel
4 Metrics Explorer → bot.connections_sent, Sum, group by bot.run_id Sample for your smoke-test run_id with source = verify
5 Traces Explorer → service.name = linkedin-connection-bot, span name bot.verify_telemetry Span attributes include bot.run_id matching the log and metric label

If logs show run_id= but no trace_id, the log probably fired outside an active span — I make sure tel.log_event() calls sit inside with tel.span(...). If View trace is missing, I confirm LoggingInstrumentor ran (SDK OTLP export path per log collection methods) and that I'm on current telemetry.py with OTLP log export enabled.

Standing Up SigNoz and Verifying 

docker compose -f pours/deployment/compose.yaml up -d
curl -s http://localhost:8080/api/v1/health   # {"status":"ok"}
nc -z localhost 4317 && echo "OTLP gRPC reachable"

./venv/bin/python Scripts/verify_telemetry.py
# run_id=87626c6dc93a

# Optional: realistic demo runs for dashboard screenshots (no LinkedIn traffic)
./venv/bin/python Scripts/signoz_demo_data.py --runs 12 --delay 2
Enter fullscreen mode Exit fullscreen mode

A ClickHouse verification script confirmed logs, traces, and metric samples after smoke tests. For blog screenshots I also run signoz_demo_data.py, which emits a dozen simulated runs with varied run_ids, source=search|grow|verify on bot.connections_sent, histogram samples on bot.run_duration_seconds, and a mix of bot.run_success / bot.run_failure.

Tip: SigNoz Metrics Explorer defaults counters like bot.run_success to Rate over Last 30 minutes. Sparse cron jobs look empty even when data exists. Switch to Sum over Last 24 hours (or run demo data recently) to see cumulative run counts. Spans like bot.login and bot.connect_from_grow still need a full run_bot.py run — I skip that during verification to avoid live LinkedIn side effects.

Dashboards and Alerts (Designed, Not Yet Fired)

From my dashboard notes, the panels I build first: run health (bot.run_success vs bot.run_failure), throughput by source, driver errors plus session crashes, run duration p95, and recent bot.run traces.

Alert targets for my setup:

  • bot.run_success sum over 26h = 0 — no successful daily run (26h allows launchd drift)
  • bot.driver_errors where version_mismatch, sum over 1h > 0
  • bot.session_crashes sum over 15m > 0
  • No bot.* metric data for 26h — telemetry pipeline dead

I haven't fired these in production yet. They're the guardrails I wish I'd had before July 17.

What I Do Differently Now

Instrument before the first scheduled run. A verify_telemetry.py smoke test on day one beats log archaeology.

Partial runs need explicit failure signals. Stopping at 27/30 should increment bot.run_failure with connections_sent=27, not look like quiet success.

Put OTEL_* in .env, not just your shell. run_bot.py loads .env with override. My interactive shell had stale exports that confused early tests.

Headless Selenium observability is about phase spans and batch counters, not HTTP latency. But the pattern — one run_id on every signal — transfers to any cron-driven browser job.

The Bot Isn't Mysterious — It's Unobserved

ChromeDriver mismatches, stale locks, and a SIGHUP at 28/30 all leave evidence in flat files. Nothing aggregates it. OpenTelemetry plus SigNoz gives me one place to ask: did today's run succeed? How many connections? Which phase failed? Was it the driver again?

This works for my setup: local SigNoz via Docker, OTLP gRPC on 4317, macOS launchd, headless Brave. Your mileage will vary.

The full project — bot, telemetry module, SigNoz compose stack, verification scripts, and dashboard notes — is on GitHub:

https://github.com/Sumit884-byte/linkedin_connection_bot

If you're running a headless browser job on a schedule and only checking log files afterward, start with a smoke test and three signals. You'll thank yourself the first time ChromeDriver and Brave disagree.

Screenshots


ui-logs-linkedin-bot.png


ui-traces-bot-run.png


ui-metrics-connections-sent.png


ui-metrics-run-success.png


ui-services-overview.png

# File Use in section
1 ui-logs-linkedin-bot.png Instrumentation / SigNoz verification — Logs Explorer with run_id, phase, trace correlation
2 ui-traces-bot-run.png SigNoz verification — bot.verify_telemetry spans
3 ui-metrics-connections-sent.png SigNoz verification — bot.connections_sent by source
4 ui-metrics-run-success.png SigNoz verification — bot.run_success over 24h
5 ui-services-overview.png SigNoz verification — Services page with linkedin-connection-bot

Top comments (0)