DEV Community: Timothy Kelvin

I Built an Economic Calendar From Government Sources Instead of Scraping ForexFactory

Timothy Kelvin — Fri, 25 Sep 2026 17:48:26 +0000

The most popular economic calendar actors on the Apify Store scrape ForexFactory or Investing.com. Their own docs tell the story: the calendar page blocks datacenter IPs, so you need residential proxies "at any meaningful volume," and a browser for anything past the current week.

But every number on those calendars comes from somewhere. The jobs report comes from the Bureau of Labor Statistics. GDP comes from the Bureau of Economic Analysis. The Fed publishes its own meeting dates. So I built a calendar that reads the agencies directly: no proxies, no browser, and a run finishes in about six seconds.

Getting there took more than five fetch calls. Here's what I ran into.

Five sources, four formats

Source	What it covers	Format
BLS	Jobs report, CPI, PPI, JOLTS	iCalendar feed
BEA	GDP, PCE inflation, trade	iCalendar feed
Census	Retail sales, housing starts, durable goods	HTML table
Federal Reserve	FOMC decisions	HTML page
Treasury	Bill, note and bond auctions	JSON API

Two agencies publishing .ics files was the nicest surprise. That's the format your phone's calendar app subscribes to, and it's meant to be read by software.

1. BLS returns 403 unless you say who you are

My first request to the BLS calendar came back 403 Forbidden. My instinct was that this was bot protection, and my rule for this portfolio is simple: if a site is actively blocking me, I don't try to get around it.

So I tried the honest version first: a User-Agent that names the tool and links to it.

// Agencies block anonymous clients; an honest, identifying User-Agent is what
// their access guidance asks for. This is identification, not disguise.
export const USER_AGENT = 'us-economic-calendar/0.1 (+https://apify.com/m_ctim/us-economic-calendar)';

200 OK. BLS doesn't block scripts, it blocks anonymous scripts. There's a real difference between telling a server who you are and pretending to be Chrome, and it's worth trying the first before assuming you'd need the second.

2. iCalendar lines fold

A long SUMMARY in an .ics file gets split across lines, with the continuation starting with a space:

SUMMARY:Gross Domestic Product\, 3rd Quarter 2025 (Updated Estimate)\, GDP
  by Industry\, and Corporate Profits (Revised)

Split on newlines naively and you get half a title. The fix is one regex before splitting, plus unescaping the commas and semicolons:

// RFC 5545 line folding: a line starting with a space or tab continues the previous one.
const lines = text.replace(/\r\n/g, '\n').replace(/\n[ \t]/g, '').split('\n');
// ...
const value = line.slice(colon + 1).replace(/\\n/gi, '\n').replace(/\\([,;\\])/g, '$1');

3. Two agencies, two ways of writing time

BLS writes local Eastern time with a timezone label:

DTSTART;TZID=US-Eastern:20260103T100000

BEA writes UTC:

DTSTART:20260122T133000Z

Traders want both Eastern (how releases are announced) and UTC (how their systems store time), so every row needs a correct conversion, and daylight saving makes that fiddly. 8:30 AM Eastern is 12:30 UTC in October and 13:30 UTC in November.

Rather than pull in a date library, I used Intl.DateTimeFormat to find the Eastern offset at a given instant. The one subtlety is that you don't know the offset until you know the instant, and you don't know the instant until you know the offset. Two passes settle it, including on the days the clocks change:

export function etToUtcIso(dateStr, timeStr) {
    const [y, m, d] = dateStr.split('-').map(Number);
    const [hh, mm] = timeStr.split(':').map(Number);
    const naive = Date.UTC(y, m - 1, d, hh, mm);
    // Two passes settle the offset correctly on DST transition days.
    let utc = naive - etOffsetMinutes(naive) * 60000;
    utc = naive - etOffsetMinutes(utc) * 60000;
    return new Date(utc).toISOString();
}

The test I trusted: the November 6 jobs report, after the clocks go back on November 1, comes out as 13:30:00Z. October releases come out as 12:30:00Z.

4. BEA names GDP two different ways

Each release gets an impact rating: high for the ones that move markets (jobs, CPI, GDP, the Fed), lower for regional and niche series. My rule for GDP was:

/^Gross Domestic Product/i

The first full-month test looked fine at a glance. Then I listed every row, low impact included, and found Q3 GDP sitting there rated low. BEA's feed calls some releases "Gross Domestic Product, ..." and others "GDP (Advance Estimate), ...". Same release family, two spellings.

// BEA titles it both ways: "Gross Domestic Product, ..." and "GDP (Advance Estimate), ...".
[/^(?:Gross Domestic Product|GDP)\b(?!.*\bby (?:State|County|Metro))/i, 'high', 'growth'],

The lesson I keep relearning: check the rows your filter excluded, not just the ones it kept. The high-impact list looked plausible precisely because nothing in it was wrong. Something was just missing.

5. Some releases belong to two agencies

The monthly trade report is a joint BEA and Census release, and both calendars list it. Without handling that, it shows up twice. Titles differ slightly ("U.S. International Trade in Goods and Services, August 2026" versus the same without the period), so the dedupe key uses the date, the time, and the title up to its first comma:

const normTitle = (t) => t.split(',')[0].toLowerCase().replace(/[^a-z]/g, '').slice(0, 40);
const key = `${raw.date}|${raw.timeEt}|${normTitle(raw.title)}`;

The row that survives records the other agency in an alsoPublishedBy field, so nothing is silently dropped.

6. The Fed's calendar has its own grammar

The FOMC page lists each meeting as a month and a day range. Most are simple, like March and 17-18*. Then there are:

Meetings that span two months: Apr/May with 30-1. The decision comes on the last day, so the date is May 1.
The asterisk: it marks meetings that come with the Summary of Economic Projections, the "dot plot." That goes in the row's title and notes.
Notation votes: entries like 22 (notation vote). There's no 2:00 PM announcement to schedule around, so they're skipped.

const days = dateText.match(/\d+/g);
const lastDay = Number(days[days.length - 1]);
const months = monthText.split('/');
const month = monthIndex(months[months.length - 1]);
const hasProjections = dateText.includes('*');

7. Don't invent data the source doesn't have

Treasury's Fiscal Data API lists announced auctions, with two quirks.

First, it republishes every auction daily under a new record_date, so the same auction appears several times. Keying on CUSIP plus auction date fixes that.

Second, it has no auction time. By convention bills close at 11:30 AM Eastern and notes and bonds at 1:00 PM, and it would have been easy to fill that in. But a convention isn't a published time, and a calendar people schedule trades around shouldn't guess. Auction rows return timeEt: null, and the README says why.

The same rule applies to the biggest thing this calendar lacks: consensus forecasts. Those come from private data vendors, not the agencies. The ForexFactory scrapers have them because ForexFactory compiles them. An official-source calendar can't, so this one doesn't pretend to.

What it returns

{
  "date": "2026-10-29",
  "timeEt": "08:30",
  "timestampUtc": "2026-10-29T12:30:00.000Z",
  "agency": "BEA",
  "title": "GDP (Advance Estimate), 3rd Quarter 2026",
  "period": "3rd Quarter 2026",
  "impact": "high",
  "category": "growth",
  "url": "https://www.bea.gov/news/schedule"
}

Filter by date range, agency, impact or keyword. It charges once per run, so a month of releases costs the same as a day.

Try it: US Economic Calendar on Apify
Source: github.com/timmKal01/us-economic-calendar

If you work with macro data, I'd like to know what you'd add. ECB and Bank of England calendars are the obvious next step.

What crt.sh's Error Pages Taught Me About Retry Logic

Timothy Kelvin — Sun, 20 Sep 2026 14:03:00 +0000

I built a small Apify actor that watches Certificate Transparency logs for a domain, basically "tell me every SSL cert that's been issued for this domain or its subdomains recently." Useful for catching phishing look-alikes and shadow IT before a customer complaint does. The data source is crt.sh, a free community-run search service over CT log data. Getting the actual query working took ten minutes. Making it reliable took a lot longer, and taught me more about "empty result" ambiguity than I expected from what's basically a single GET request.

First problem: 404 doesn't mean "no results."

crt.sh returns bare HTML error pages, not JSON, when it's under load. I first treated a 404 as "no certificates found," which is a completely reasonable read of a 404. Except I'd query the same domain twice in a row and get a 404 once and a real result the next time. Same query, same data, different outcome, because the 404 wasn't about the data at all, it was crt.sh being overloaded. Same story with 502/503/504. None of these are trustworthy "zero results" signals for this service, so they all need to be retried, not accepted.

Second problem: my retry budget wasn't big enough for a real outage.

I started with 4 attempts and exponential backoff, which felt generous. Then during a bad patch I watched it eat 6 consecutive 502s before a success came through, and the actor's rolling 30-day failure rate had climbed to something like 46%. Four attempts just wasn't enough runway to ride out a real bad stretch on a free community service. Bumped it to 8, and capped the backoff instead of letting it grow unbounded, since a successful response can itself take 10-20 seconds once crt.sh is under load, so the retry loop needs real time budget, not just more attempts fired quickly at each other.

Third problem: a hung connection defeats retries entirely.

Plain fetch() has no timeout. One test run just hung, no 502, no slow success, nothing, indefinitely. If a request never resolves, it never reaches the point where my retry logic would even kick in. Added an AbortController with a per-attempt timeout so a dead connection gets treated as a failure and retried like any other, instead of silently stalling the whole run.

Fourth problem, and the sneaky one: a field just disappeared from the response shape.

The actor filters results by date and sorts newest-first, originally using crt.sh's entry_timestamp field. At some point crt.sh's JSON output for this particular query stopped including that field. No error, no warning, just undefined. Which meant new Date(undefined) >= startDate silently evaluated false for every single entry, so the actor returned zero results, every time, regardless of what data actually existed. The scariest kind of bug for a monitoring tool: it fails by going quiet, not by throwing. Switched to not_before (when the cert becomes valid, which is essentially CT-log time anyway) as the log-time proxy instead, since that field is always present.

None of these were exotic failures. Bare error pages under load, a budget that was too small, a fetch with no timeout, a field that quietly vanished. But stacked together they're the difference between "works when I test it" and "actually trustworthy as a monitoring tool," which matters a lot more once the whole point of the thing is telling you about certificates you didn't expect.

If you want to poke at the actual retry code: github.com/timmKal01/certificate-transparency-monitor. The actor itself is live on Apify if you want to point it at a domain.

"Pull a company's open roles straight from their own job board API"

Timothy Kelvin — Fri, 18 Sep 2026 10:05:15 +0000

Most "company hiring signal" tools scrape LinkedIn. company-hiring-tracker does something simpler: most companies that use Greenhouse or Lever for hiring expose their open roles through a public, unauthenticated API on their own job board. No login, no proxy, no headless browser, just a fetch.

Give it a company's ATS (Greenhouse or Lever) and board slug, and it returns each open role: title, location, department, and posting date, as clean JSON.

{ "title": "Senior Backend Engineer", "location": "Remote", "department": "Engineering", "postedAt": "2026-09-10" }

Priced per board queried, not per job returned, so checking a company with 40 open roles costs the same as checking one with 2.

https://apify.com/m_ctim/company-hiring-tracker

"A Raw JSON Textarea Was Quietly Costing Me Users"

Timothy Kelvin — Fri, 18 Sep 2026 09:57:34 +0000

I almost missed this one because the actor itself worked fine. The bug wasn't in the code, it was in the input form.

Looking at usage funnels across a portfolio of small data-fetching actors (Apify calls them Actors — serverless scrapers/API tools), one number stood out: for company-buying-signal-report, 5 people opened the input page, and only 1 actually ran it. Everywhere else in the funnel, the drop-off was gradual. Here it was a cliff.

The only clue was the input schema itself. The actor's one real input field was companies, an array of objects ({ domain, greenhouseSlug, leverSlug }), rendered by Apify's console as a raw JSON editor. Technically correct, since most people running this actor only care about one company at a time, and a blank JSON textarea is not exactly a welcoming way to ask for that.

The fix was small: add flat, single-item fields (domain, greenhouseSlug, leverSlug) alongside the existing array field. In the actor's code, the single fields take priority when filled in, and fall back to the array for anyone who does want to check multiple companies in one run.

const companies = singleDomain
  ? [{ domain: singleDomain, greenhouseSlug, leverSlug }]
  : (companiesInput?.length ? companiesInput : defaultCompanies);

Once I saw it in one actor, I grepped the rest of the portfolio for the same pattern ("editor": "json" on an array-of-objects field with no matching single-item shortcut) and found four more actors with the exact same friction: a hiring tracker, a weather tracker, a risk-briefing tool, and a city council monitor. Same fix, same result, each verified live afterward.

None of this shows up in a code review. It only shows up when you look at where real people stop, not just whether the backend logic is correct.

There are three different things people mean by "check my SSL cert"

Timothy Kelvin — Tue, 15 Sep 2026 04:18:35 +0000

A cert expiring in production is one of those outages that feels avoidable in hindsight. Nobody meant to skip it. It just wasn't anyone's job to check.

The confusing part

"Check my SSL cert" can mean three different things, and they check three different systems.

Is the domain registration itself about to expire? That's a WHOIS/RDAP question, nothing to do with TLS.
Has a new certificate been issued for this domain recently, maybe one you didn't request? That's a certificate transparency log question, a public append-only record of every cert any browser-trusted CA has ever issued.
What certificate is the server actually presenting right now, and how many days until it expires? That's a live TLS handshake question, and it's the one that actually causes an outage when the answer is "zero."

I'd already built tools for the first two. This one covers the third, and it's the one people usually mean when a pager goes off.

How it works

Opens a direct TLS connection to each host, the same handshake a browser performs, and reads the certificate the server presents. Node's built-in tls module does the heavy lifting:

const socket = tls.connect(
  { host, port, servername: host, timeout: 8000, rejectUnauthorized: false },
  () => {
    const cert = socket.getPeerCertificate();
    const authorized = socket.authorized;
    // ...
  },
);

rejectUnauthorized: false looks backwards at first. The point isn't to skip trust checking, it's to keep the connection open long enough to actually read the certificate even when it's self-signed or the chain is broken. socket.authorized and socket.authorizationError still report the trust result separately, so a broken cert shows up as data ("isSelfSigned: true", a trust error message) instead of a connection failure with no detail. Silently failing to connect isn't the same as telling someone their cert is bad.

What it returns

One record per host: issuer, subject, valid dates, days until expiry, expired/self-signed flags, and the trust result. A host that can't be reached at all still returns a record, just with reachable: false and an error message, since "we tried to check and something's wrong" is itself a useful result for a host that should have been reachable.

Pricing shape

Billed per host checked, not per field returned. One charge whether the host is fine, expired, or completely unreachable.

Try it

Source: github.com/timmKal01/ssl-certificate-expiry-checker
Hosted version (Apify actor, no setup): apify.com/m_ctim/ssl-certificate-expiry-checker

If you're already checking domain expiry or watching CT logs, this is the third leg of that stool, the one that actually matches what's live on the wire right now.

Three small API-backed actors I've shipped recently (Docker Hub, lead extraction, SEC Form 4)

Timothy Kelvin — Fri, 11 Sep 2026 19:24:35 +0000

I build small, narrow Apify actors — each one wraps a single official API or data source and does one job well, rather than trying to be a general-purpose scraper. Here are three recent ones, each solving a different, specific problem.

Docker Hub Image Tracker

Docker Hub will repush a tag with a new digest, new architectures, or a different underlying image, and the version string stays the same. This actor hits hub.docker.com/v2/repositories directly and tells you what actually changed for a given repository within a time window — digest, size, supported architectures, last-pushed/last-pulled timestamps.

{ "repository": "postgres", "daysBack": 7, "maxResults": 50 }

Real gotcha from building it: the API's ordering param is backwards from the usual REST convention — ordering=last_updated sorts newest-first, and you need a - prefix to get oldest-first.

apify.com/m_ctim/docker-hub-image-tracker · source

Website Lead Extractor

Built on Crawlee's plain CheerioCrawler instead of a headless browser. Point it at a domain, it crawls within that domain up to a set depth, and pulls every email, phone number, and social profile link (LinkedIn, X, Facebook, Instagram, GitHub) it finds in the raw HTML — no Chromium, no render wait.

{ "startUrls": [{ "url": "https://example.com" }], "maxDepth": 1, "maxPagesPerDomain": 20 }

Tradeoff worth stating plainly: it only sees what's in the initial HTML response, so a client-side-only React SPA with no SSR needs a browser-based crawler instead. For the common case of a static or server-rendered marketing site, it's faster and has nothing to configure around headless-browser flakiness.

apify.com/m_ctim/website-lead-extractor · source

Insider Trading Alert

Every SEC Form 4 filing (executive/director/10%-owner stock buys and sells) is public, structured XML, filed within a couple of business days of the transaction. This actor reads EDGAR's filing feed and each filing's own XML directly — search one ticker or scan the market-wide feed of the newest filings.

{ "ticker": "AAPL", "transactionType": "disposed", "minTransactionValue": 100000 }

Real bug from building it: the boolean "is this person an officer/director" flags get encoded as 1/0 by some filers' software and true/absent by others, on the same form type. A naive === '1' check silently returned false for real officers depending on which software generated the filing.

apify.com/m_ctim/insider-trading-alert · source

All three are pay-per-event on Apify (no subscription), no proxy, and read directly from each source's own official API rather than scraping rendered pages.

Turning a list of company websites into a contact list with a plain HTTP crawl

Timothy Kelvin — Thu, 10 Sep 2026 09:20:16 +0000

Anyone who has done outbound sales or recruiting knows this task. You have a list of company websites and you need the contact info off each one. Open the Contact page, copy the email, check LinkedIn, move to the next site. A 50 domain list eats most of an afternoon done by hand.

The approach

Give it a start URL and it crawls within that same domain, up to a depth you choose, using Crawlee's CheerioCrawler. On each page it checks for:

Email addresses in the raw HTML
Phone numbers
Social profile links (LinkedIn, X, Facebook, Instagram, GitHub)

Everything found gets returned as one record per page, with the URL, domain, and whatever contact info was on it.

{
  "url": "https://example.com/contact",
  "domain": "example.com",
  "emails": ["hello@example.com"],
  "phones": ["+1 415-555-0132"],
  "socialProfiles": {
    "linkedin": "https://linkedin.com/company/example",
    "twitter": "https://x.com/example"
  }
}

What it deliberately doesn't do

No headless browser. It only reads what the site already renders in the raw HTML response to a plain request, the same thing any visitor's browser would see before JavaScript runs. That keeps it fast and cheap to run, and it means it only extracts information the site is already publishing publicly. Nothing behind a login, nothing a normal visitor couldn't already find on a Contact or About page.

Where it falls short, on purpose

Sites that render their content client side with JavaScript are a real weak spot here. A plain HTTP crawl never sees content that only shows up after JS runs, so a heavily client rendered site might come back with far less than it actually has. That's the honest tradeoff for staying fast, cheap, and proxy free. A browser based crawler would handle those sites but costs more to run and adds real complexity. For static or server rendered company sites, which is most small and mid size business sites, the plain HTTP approach works well.

Pricing shape

Billed per page where contact info was actually found. A domain with nothing public anywhere costs nothing to check, and a domain where three pages all list contact info costs about the same as pulling it from all three yourself, just a lot faster.

Try it

Source: github.com/timmKal01/website-lead-extractor
Hosted version (Apify actor, no setup): apify.com/m_ctim/website-lead-extractor

If you run it against a list of your own domains, curious to hear how the hit rate looks on real sites.

I built an Apify Actor that turns a list of company domains into a ranked "who to call first" list

Timothy Kelvin — Mon, 07 Sep 2026 09:14:06 +0000

If you've ever done account-based prospecting, you know the drill: pull open
roles from a company's careers page, eyeball their homepage for what tools
they're running, dig up a contact email, then repeat that for every account
on the list — and manually decide who actually looks worth calling first.

I build small data-focused Apify Actors as a side project, and this one
(Company Buying Signal Report) started from wanting to stop doing that
join by hand.

What it does

Give it a list of company domains. For each one it pulls three signals in
parallel:

Hiring signal — open roles straight from the company's own Greenhouse or Lever job board API (the same public JSON endpoints that power their careers page), matched against a list of role keywords you care about ("account executive", "growth", whatever signals urgency for your product).
Tech stack fingerprint — CMS, ecommerce platform, analytics, payment, and live-chat tools detected from response headers, meta tags, and script sources on the homepage — the same signature-matching approach tools like Wappalyzer use.
Public contact info — email, phone, and social links pulled from the homepage and one linked contact/about page.

Then it computes a low/medium/high buying-signal score with the actual
reasons behind it ("142 open roles (actively scaling)",
"uses paid ecommerce/payment/chat/analytics tooling"), so the output is a
ranked call list, not three raw feeds you still have to reconcile yourself.

The part I actually had to think about: slugs

Greenhouse and Lever both key their public job-board API by a "slug" —
boards-api.greenhouse.io/v1/boards/{slug}/jobs — and that slug usually,
but not always, matches the company's domain name. stripe.com → stripe
works. Plenty of companies don't follow that pattern.

The actor guesses the slug from the domain by default (fast, works most of
the time), but lets you pass greenhouseSlug/leverSlug explicitly per
company when you already know it (it's sitting right there in the careers
page URL). Rather than silently returning nothing for a guess that misses,
it's an explicit opt-in override — a small design choice, but the kind of
thing that's easy to get wrong in either direction (over-trusting a guess,
or forcing every user to look up slugs manually even when the guess would've
worked fine).

No proxy, no login, no headless browser

All three signals come from data companies already publish for anyone to
see — their own careers-page API, their own homepage HTML, their own contact
page. No scraping behind auth, no browser automation, no residential proxy
bill. It's just three passive HTTP lookups run per company and joined into
one row.

Why billed per company, not per signal

It's priced as one charge per company report, regardless of whether that
company has 3 open roles or 300, or how many tech-stack signatures matched.
The point of combining the three lookups was to replace three separate tool
runs (and the spreadsheet join after) with one — pricing it per sub-signal
would've defeated that.

If you're doing target-account prioritization and this saves you a step:
try it on Apify
· source on GitHub

Happy to answer questions about the Greenhouse/Lever API shapes or the
signature-matching approach if anyone's building something similar.

Fingerprinting a website's tech stack from a single HTTP request (no headless browser)

Timothy Kelvin — Sat, 05 Sep 2026 08:47:43 +0000

I wanted a fast way to answer "what's this site built with?" — CMS, ecommerce platform, JS framework, analytics tags, CDN, payment/chat widgets — without spinning up a full headless browser per URL.

The approach

One plain HTTP fetch of the homepage via Crawlee's CheerioCrawler, then check three things against a signature table:

Response headers (Server, X-Powered-By, CF-Ray, X-Vercel-Id, etc.)
The <meta name="generator"> tag
<script src> domains on the page

Same general idea as Wappalyzer, just a much smaller, hand-written signature set — about 30 technologies across 7 categories (CMS, ecommerce, JS frameworks, analytics, CDN/hosting, payment, live chat). Each signature is a test(context) function:

{
  name: 'WordPress',
  category: 'cms',
  test: ({ generator }) => generator.includes('wordpress'),
}

context gives you lowercased headers, html, scriptSrcs, and generator — cheap to check, no DOM traversal needed for most signatures.

What it deliberately doesn't do

No headless browser. Everything it reads is already served to any visitor's plain HTTP request — no JS execution, no login, no bypassing any protection.
No guessing. If nothing in the signature table matches, it returns an empty result for that category rather than a fuzzy best-guess. A smaller-but-honest signature set beats a big one that's confidently wrong.

Where it fell short (and stayed that way, on purpose)

Detection accuracy is bounded by however many signatures I've hand-written — it's ~30 technologies, not the hundreds Wappalyzer tracks. Sites using something niche just come back with an empty category instead of a wrong guess. That's a real limitation, not a bug, and the tradeoff I'd make again: adding a signature is one function, so it grows exactly as fast as real usage demands it, not ahead of it.

Try it

Source: github.com/timmKal01/website-tech-stack-detector
Hosted version (Apify actor, no setup): apify.com/m_ctim/website-tech-stack-detector

If you've got a signature I'm missing, signatures.js is a small file — happy to take PRs or just hear about the gap.