DEV Community

Timothy Kelvin
Timothy Kelvin

Posted on

I Built an Economic Calendar From Government Sources Instead of Scraping ForexFactory

#api #javascript #opendata #showdev

The most popular economic calendar actors on the Apify Store scrape ForexFactory or Investing.com. Their own docs tell the story: the calendar page blocks datacenter IPs, so you need residential proxies "at any meaningful volume," and a browser for anything past the current week.

But every number on those calendars comes from somewhere. The jobs report comes from the Bureau of Labor Statistics. GDP comes from the Bureau of Economic Analysis. The Fed publishes its own meeting dates. So I built a calendar that reads the agencies directly: no proxies, no browser, and a run finishes in about six seconds.

Getting there took more than five fetch calls. Here's what I ran into.

Five sources, four formats

Source What it covers Format
BLS Jobs report, CPI, PPI, JOLTS iCalendar feed
BEA GDP, PCE inflation, trade iCalendar feed
Census Retail sales, housing starts, durable goods HTML table
Federal Reserve FOMC decisions HTML page
Treasury Bill, note and bond auctions JSON API

Two agencies publishing .ics files was the nicest surprise. That's the format your phone's calendar app subscribes to, and it's meant to be read by software.

1. BLS returns 403 unless you say who you are

My first request to the BLS calendar came back 403 Forbidden. My instinct was that this was bot protection, and my rule for this portfolio is simple: if a site is actively blocking me, I don't try to get around it.

So I tried the honest version first: a User-Agent that names the tool and links to it.

// Agencies block anonymous clients; an honest, identifying User-Agent is what
// their access guidance asks for. This is identification, not disguise.
export const USER_AGENT = 'us-economic-calendar/0.1 (+https://apify.com/m_ctim/us-economic-calendar)';
Enter fullscreen mode Exit fullscreen mode

200 OK. BLS doesn't block scripts, it blocks anonymous scripts. There's a real difference between telling a server who you are and pretending to be Chrome, and it's worth trying the first before assuming you'd need the second.

2. iCalendar lines fold

A long SUMMARY in an .ics file gets split across lines, with the continuation starting with a space:

SUMMARY:Gross Domestic Product\, 3rd Quarter 2025 (Updated Estimate)\, GDP
  by Industry\, and Corporate Profits (Revised)
Enter fullscreen mode Exit fullscreen mode

Split on newlines naively and you get half a title. The fix is one regex before splitting, plus unescaping the commas and semicolons:

// RFC 5545 line folding: a line starting with a space or tab continues the previous one.
const lines = text.replace(/\r\n/g, '\n').replace(/\n[ \t]/g, '').split('\n');
// ...
const value = line.slice(colon + 1).replace(/\\n/gi, '\n').replace(/\\([,;\\])/g, '$1');
Enter fullscreen mode Exit fullscreen mode

3. Two agencies, two ways of writing time

BLS writes local Eastern time with a timezone label:

DTSTART;TZID=US-Eastern:20260103T100000
Enter fullscreen mode Exit fullscreen mode

BEA writes UTC:

DTSTART:20260122T133000Z
Enter fullscreen mode Exit fullscreen mode

Traders want both Eastern (how releases are announced) and UTC (how their systems store time), so every row needs a correct conversion, and daylight saving makes that fiddly. 8:30 AM Eastern is 12:30 UTC in October and 13:30 UTC in November.

Rather than pull in a date library, I used Intl.DateTimeFormat to find the Eastern offset at a given instant. The one subtlety is that you don't know the offset until you know the instant, and you don't know the instant until you know the offset. Two passes settle it, including on the days the clocks change:

export function etToUtcIso(dateStr, timeStr) {
    const [y, m, d] = dateStr.split('-').map(Number);
    const [hh, mm] = timeStr.split(':').map(Number);
    const naive = Date.UTC(y, m - 1, d, hh, mm);
    // Two passes settle the offset correctly on DST transition days.
    let utc = naive - etOffsetMinutes(naive) * 60000;
    utc = naive - etOffsetMinutes(utc) * 60000;
    return new Date(utc).toISOString();
}
Enter fullscreen mode Exit fullscreen mode

The test I trusted: the November 6 jobs report, after the clocks go back on November 1, comes out as 13:30:00Z. October releases come out as 12:30:00Z.

4. BEA names GDP two different ways

Each release gets an impact rating: high for the ones that move markets (jobs, CPI, GDP, the Fed), lower for regional and niche series. My rule for GDP was:

/^Gross Domestic Product/i
Enter fullscreen mode Exit fullscreen mode

The first full-month test looked fine at a glance. Then I listed every row, low impact included, and found Q3 GDP sitting there rated low. BEA's feed calls some releases "Gross Domestic Product, ..." and others "GDP (Advance Estimate), ...". Same release family, two spellings.

// BEA titles it both ways: "Gross Domestic Product, ..." and "GDP (Advance Estimate), ...".
[/^(?:Gross Domestic Product|GDP)\b(?!.*\bby (?:State|County|Metro))/i, 'high', 'growth'],
Enter fullscreen mode Exit fullscreen mode

The lesson I keep relearning: check the rows your filter excluded, not just the ones it kept. The high-impact list looked plausible precisely because nothing in it was wrong. Something was just missing.

5. Some releases belong to two agencies

The monthly trade report is a joint BEA and Census release, and both calendars list it. Without handling that, it shows up twice. Titles differ slightly ("U.S. International Trade in Goods and Services, August 2026" versus the same without the period), so the dedupe key uses the date, the time, and the title up to its first comma:

const normTitle = (t) => t.split(',')[0].toLowerCase().replace(/[^a-z]/g, '').slice(0, 40);
const key = `${raw.date}|${raw.timeEt}|${normTitle(raw.title)}`;
Enter fullscreen mode Exit fullscreen mode

The row that survives records the other agency in an alsoPublishedBy field, so nothing is silently dropped.

6. The Fed's calendar has its own grammar

The FOMC page lists each meeting as a month and a day range. Most are simple, like March and 17-18*. Then there are:

  • Meetings that span two months: Apr/May with 30-1. The decision comes on the last day, so the date is May 1.
  • The asterisk: it marks meetings that come with the Summary of Economic Projections, the "dot plot." That goes in the row's title and notes.
  • Notation votes: entries like 22 (notation vote). There's no 2:00 PM announcement to schedule around, so they're skipped. 
const days = dateText.match(/\d+/g);
const lastDay = Number(days[days.length - 1]);
const months = monthText.split('/');
const month = monthIndex(months[months.length - 1]);
const hasProjections = dateText.includes('*');
Enter fullscreen mode Exit fullscreen mode

7. Don't invent data the source doesn't have

Treasury's Fiscal Data API lists announced auctions, with two quirks.

First, it republishes every auction daily under a new record_date, so the same auction appears several times. Keying on CUSIP plus auction date fixes that.

Second, it has no auction time. By convention bills close at 11:30 AM Eastern and notes and bonds at 1:00 PM, and it would have been easy to fill that in. But a convention isn't a published time, and a calendar people schedule trades around shouldn't guess. Auction rows return timeEt: null, and the README says why.

The same rule applies to the biggest thing this calendar lacks: consensus forecasts. Those come from private data vendors, not the agencies. The ForexFactory scrapers have them because ForexFactory compiles them. An official-source calendar can't, so this one doesn't pretend to.

What it returns 

{
  "date": "2026-10-29",
  "timeEt": "08:30",
  "timestampUtc": "2026-10-29T12:30:00.000Z",
  "agency": "BEA",
  "title": "GDP (Advance Estimate), 3rd Quarter 2026",
  "period": "3rd Quarter 2026",
  "impact": "high",
  "category": "growth",
  "url": "https://www.bea.gov/news/schedule"
}
Enter fullscreen mode Exit fullscreen mode

Filter by date range, agency, impact or keyword. It charges once per run, so a month of releases costs the same as a day.

If you work with macro data, I'd like to know what you'd add. ECB and Bank of England calendars are the obvious next step.

Top comments (0)