I Missed a Hackathon Deadline, So I'm Building a Bot to Never Let It Happen Again

A few weeks ago I missed a deadline for a hackathon I actually wanted to apply to. Not because I didn't have time — I just didn't know it existed until three days after applications closed. I was manually checking Devpost, random Discord servers, and Twitter threads, and something always slipped through.

So I started building opp-radar - a small automation pipeline that fetches hackathons and opportunities, filters out the noise, and pings me on Telegram before I can miss anything again.

This post isn't a "look what I shipped" post. It's still in progress. Right now, only the fetch layer works - and it broke in a way that taught me more than the rest of the build combined. I wanted to write about it while the problem is still fresh, instead of waiting for a polished finish.

The plan (architecture, as designed)

The full pipeline, once done, looks like this:

GitHub Actions (cron scheduler)
        ↓
Devpost API (fetch layer)  ← currently working on this
        ↓
Dedup / filter logic
        ↓
Telegram bot alerts
        ↓
Google Calendar sync

GitHub Actions handles scheduling so I don't need to run a server anywhere. The fetch layer pulls opportunity data from Devpost's API. Everything downstream — dedup, alerts, calendar — depends on that fetch layer being reliable.

Which, right now, it isn't.

The bug: fetches die midway, and retries resurface old results

Here's the actual failure mode I'm hitting:

The fetch script pages through results from the Devpost API. Partway through - not always at the same point - the request errors out. Could be a timeout, could be rate limiting, I haven't fully isolated the cause yet. When that happens, the script just stops. No partial save, no checkpoint.

So naturally, I just re-run it.

Except when I re-run it, it starts from page one again - and shows me hackathons I've already seen and logged from the last successful (or partially successful) run. The dedup step doesn't know a previous run happened, because there's no state being persisted between runs. Every execution is a blank slate.

This is the classic idempotency problem: a script that isn't safe to re-run isn't actually automation - it's a manual process wearing a costume. The whole point of opp-radar is to remove the "did I already check this" mental overhead. Right now it's re-adding that overhead in a different form.

Why this is trickier than it sounds

My first instinct was "just save what you've fetched so far before erroring out." But that alone doesn't fully solve it, because:

• If the API returns items in a different order on retry, "resume from page N" isn't safe - I might skip items or duplicate them anyway.
• Devpost's response schema isn't perfectly stable between calls, so partial-state assumptions can silently break the next run instead of loudly failing.
• GitHub Actions runners are ephemeral — there's no local disk state persisting between scheduled runs, so "just write a checkpoint file" only works if I'm also persisting that file somewhere durable (repo commit, external store, etc.), which adds its own failure surface.

So the real fix isn't "retry more carefully." It's treating every fetched item as independently idempotent - each opportunity needs a stable unique identifier that gets checked against a persisted seen-set before it's added anywhere, regardless of which run or which page it came from. That way, a mid-run crash just means "fewer new items found this time," not "state got corrupted."

Where I'm at now

I'm restructuring the fetch layer around that idea:

• A stable ID per opportunity (probably a hash of title + source URL, since Devpost's own IDs aren't something I want to fully depend on)
• A persisted seen-set (leaning toward a committed JSON file for now, simple and diffable - might move to SQLite if this gets more complex)
• Fetch logic that writes as it goes, not just at the end, so a crash mid-run doesn't throw away work that already succeeded

Nothing glamorous. But it's the difference between "automation" and "a script I still have to babysit."

What's next

Once the fetch layer is actually idempotent, the dedup logic becomes almost trivial - it's really just a lookup against the seen-set. Then Telegram alerts and Calendar sync are comparatively straightforward, since they're just consumers of a clean, deduplicated feed.

I'll write a follow-up once that's working - including whatever breaks next, because something always does.

If you've solved this kind of "safe retry" problem in your own scraping or automation projects, I'd genuinely like to hear how - especially if you've dealt with ephemeral runners like GitHub Actions where local state doesn't persist between runs.