DEV Community: Your Evenings Data

Checking your customers' EU VAT and UK EORI numbers from code, without treating an outage as "invalid"

Your Evenings Data — Tue, 03 Nov 2026 10:18:21 +0000

Article type: How-to guide (one task: check EU VAT and UK EORI numbers from code)

Prerequisites: Python 3 with requests. No API key is needed for VIES or the HMRC EORI service. An Apify account only if you use the shortcut in section 5.

Tested on: 31 October 2026 (the run in section 4); the Python snippet was re-run on 1 November 2026

If you invoice EU business customers with reverse charge, or ship to the UK, you need to confirm their VAT or EORI number. The official services are free. The tricky part is handling their failures correctly.

1. The services

EU VAT: VIES, the European Commission's service. It has a REST endpoint: POST https://ec.europa.eu/taxation_customs/vies/rest-api/check-vat-number with {"countryCode":"DE","vatNumber":"129273398"}.
GB EORI: HMRC's Check an EORI number API, open access: POST https://api.service.hmrc.gov.uk/customs/eori/lookup/check-multiple-eori with {"eoris":["GB220430231000"]}.
EU/XI EORI: the Commission's EOS validation service.
GB VAT: HMRC's Check a UK VAT number API, which needs an HMRC developer account and application.

2. The mistakes to avoid

An outage is not "invalid". VIES relays each member state's database. When one is down you get errors like MS_UNAVAILABLE or MS_MAX_CONCURRENT_REQ. Record "not checked, try later", and never overwrite the last real answer with it.
Validate locally first. Each country has a format and most have check digits. A typo caught locally never needs a network call.
Go slowly. One number at a time, about one a second, and don't re-check the same number many times a day.
Keep the proof. Pass your own VAT number as the requester and VIES returns a consultation number you can store with the invoice.
Read the terms. VIES is for confirming the numbers of businesses you trade with; it forbids extracting or republishing its content.

3. A minimal VIES check

import requests, time

def check(cc, num):
    r = requests.post("https://ec.europa.eu/taxation_customs/vies/rest-api/check-vat-number",
                      json={"countryCode": cc, "vatNumber": num}, timeout=30)
    if r.status_code >= 500:
        return "not_checked"          # service down: try later
    j = r.json()
    errors = [e.get("error") for e in j.get("errorWrappers", [])]
    if "INVALID_INPUT" in errors:
        return "bad_input"            # fix the number; not an outage
    if errors:
        return "not_checked"          # MS_UNAVAILABLE, MS_MAX_CONCURRENT_REQ...: try later
    return "valid" if j.get("valid") else "invalid"

for cc, num in [("IE", "6388047V"), ("AT", "U13585627"), ("DE", "129273398")]:
    print(cc + num, check(cc, num)); time.sleep(1)

4. A real run, 31 October 2026

Number	Type	Service	Result
IE6388047V	VAT	VIES	valid (cached from an earlier run that day, not re-asked)
ATU13585627	VAT	VIES	invalid ("not a valid VAT number for cross-border trade in AT on this date")
DE129273398	VAT	VIES	valid
GB220430231000	EORI	HMRC	valid

5. The shortcut (we built this)

We built an Apify Actor, VAT Number Checker & EORI Checker, that does the above for up to 500 numbers per run: local format and check-digit checks, outages returned as source_unavailable (free) instead of invalid, at most one VIES check per number per day, the official checkDate on every row, and a monitor mode that flags numbers that became_invalid. It's independent and not affiliated with the European Commission or HMRC, and it's for checking the businesses you trade with.

Pay per event: base price $0.002 per fresh valid or invalid answer, plus $0.00005 per run per GB of memory for the Actor start (one event at the default 256 MB) and $0.01 per monitor run in which a number changed. Higher Apify plans pay less. Cached answers, outages and local failures are free.

VAT Number Checker & EORI Checker on Apify

Your Evenings

Check EU and US product recalls for "charger" and "power bank" in one table

Your Evenings Data — Mon, 02 Nov 2026 10:17:53 +0000

Article type: How-to guide (one task: check recent EU and US recalls for your keywords)

Prerequisites: Python 3 with requests. No API key is needed for Safety Gate, CPSC or openFDA. An Apify account only if you use the shortcut in section 4.

Tested on: 31 October 2026 (the run in section 3); the Python snippet was re-run on 1 November 2026

If you sell electronics in both the EU and the US, a recall can show up in three places: EU Safety Gate, the US CPSC and (for food, drugs and devices) the FDA. All three are public and need no key, but each has its own format. Here's how to read each one, then one schema for all three.

1. The three sources

CPSC: https://www.saferproducts.gov/RestWebServices/Recall?format=json&RecallDateStart=2026-10-17&ProductName=power%20bank
openFDA (food shown; also drug and device): https://api.fda.gov/food/enforcement.json?search=report_date:[20261017+TO+20261031]&limit=100 (With curl, add -g so the square brackets aren't read as a pattern.)
EU Safety Gate: the European Commission publishes weekly reports as XML downloads. There's no keyword search, so you fetch the recent weekly reports and filter them yourself.

Things that catch people out:

No results is normal. openFDA adds enforcement reports weekly, so the last few days are often empty. Treat a 404 "No matches found" from openFDA as zero rows, not an error.
Barcodes don't line up. A 12-digit UPC and the 13-digit EAN with a leading 0 are the same product. Compare digits after stripping leading zeros.
Risk wording differs. Safety Gate says "Serious", CPSC mostly doesn't grade, FDA uses Class I/II/III. Keep the original wording next to any scale you map it to.

2. A minimal CPSC check in Python

import requests
url = "https://www.saferproducts.gov/RestWebServices/Recall"
for kw in ["charger", "power bank"]:
    r = requests.get(url, params={"format": "json", "RecallDateStart": "2026-10-17", "ProductName": kw},
                     headers={"User-Agent": "recall-check/1.0 (you@example.com)"}, timeout=30)
    r.raise_for_status()   # saferproducts.gov refuses the default python-requests User-Agent
    for rec in r.json():
        print(rec["RecallNumber"], rec["RecallDate"][:10], rec["Title"])

3. A real run, 31 October 2026

Last 14 days, keywords "charger" and "power bank", at most 3 per source:

Source	ID	Date	Product	Brand	Risk
EU Safety Gate	SR/03027/26	2026-10-30	LED Football (with charger)	Coolstuff	serious
EU Safety Gate	SR/03069/26	2026-10-30	Power Supply	vhbw	serious
EU Safety Gate	SR/03044/26	2026-10-30	Dual USB-C Charger	X-start	serious
US CPSC	27056	2026-10-29	HAIARA power banks		unknown
US FDA (food, drug, device)			no matches 18-31 Oct

The FDA row is empty because none of the 71 FDA enforcement reports dated in that window (8 food, 5 drug, 58 device) mentioned a charger or power bank. An empty source is a normal answer, not an error.

4. The shortcut (we built this)

We built an Apify Actor, Product Recall API, that reads all three feeds into one schema (riskLevel plus the source's own wording), matches barcodes across UPC/EAN, gives a free no_matches row per empty source, and has a monitor mode that returns only recalls you haven't had before. It's independent and not affiliated with the European Commission, CPSC or FDA.

{ "publishedWithinDays": 14, "keywords": ["charger", "power bank"], "maxResultsPerSource": 3 }

Pay per event: base price $0.002 per recall row returned, plus $0.00005 per run per GB of memory for the Actor start (one event at the default 256 MB) and $0.01 per monitor run that returns new recalls. Higher Apify plans pay less. Status rows are free.

Product Recall API on Apify

Your Evenings

Federal jobs closing this week: a Monday CSV from the USAJOBS API (sorted by close date, in Eastern time)

Your Evenings Data — Mon, 02 Nov 2026 09:01:19 +0000

Article type: How-to guide. You already know USAJOBS and the occupational series you care about. This guide gets you one CSV each Monday of every job USAJOBS lists under its Public filter in those series that closes in the next 7 days.
Prerequisites: Python 3.9+, requests, a free USAJOBS API key (developer.usajobs.gov) and the email you registered it with.
Tested on: 2 November 2026, against the live API.

Career counselors, veterans' transition offices and university career centers often get the same question: "What can I still apply for this week?" The USAJOBS website answers it one search at a time. The API answers it in one run, once you deal with three things:

The default sort isn't by close date, so "first page" doesn't mean "closing soonest".
ApplicationCloseDate has no time zone (2026-11-02T23:59:59.9970), but USAJOBS closes announcements at 11:59 p.m. Eastern Time on the closing date (USAJOBS Help Center).
Some jobs close before that date. Agencies can close or remove an announcement early. "Applicant cut-off" jobs usually close at 11:59 p.m. ET on the day they reach a set number of applications.

The urgency is real. On 2 November 2026, 211 of the 325 announcements USAJOBS listed under USAJOBS's Public filter in series 0343 (Management and Program Analysis) and 0201 (Human Resources Management) closed within 7 days, and 70 of those closed that same day. Half of all these announcements are open for 10 days or less.

Step 1: Put your key in environment variables

export USAJOBS_KEY="your-key"
export USAJOBS_EMAIL="the-email-you-registered@example.com"

USAJOBS wants the email as the User-Agent header and the key as Authorization-Key.

Step 2: Sort by close date and stop at the end of the window

Ask for SortField=closedate with SortDirection=Asc. The results then come back soonest-closing first, so you can stop paging at the first job that closes after your window instead of downloading the whole series.

Read the close date as Eastern time. A Monday-morning run from a server in any time zone then still counts a job that closes at 11:59 p.m. ET that night.

Save this as closing_soon.py:

import csv, os, time, requests
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo

SERIES = ["0343", "0201"]          # Management and Program Analysis, Human Resources Management
DAYS = 7
ET = ZoneInfo("America/New_York")
HEADERS = {
    "Host": "data.usajobs.gov",
    "User-Agent": os.environ["USAJOBS_EMAIL"],
    "Authorization-Key": os.environ["USAJOBS_KEY"],
}

now = datetime.now(ET)
cutoff = now + timedelta(days=DAYS)

def closes_at(job):
    # "2026-11-02T23:59:59.9970", no zone: USAJOBS closes jobs at 11:59 pm Eastern
    return datetime.fromisoformat(job["ApplicationCloseDate"][:19]).replace(tzinfo=ET)

rows = {}                                  # keyed by job ID: one job can list several series
for series in SERIES:
    page = 1
    while True:
        r = requests.get("https://data.usajobs.gov/api/Search", headers=HEADERS, timeout=60, params={
            "JobCategoryCode": series, "WhoMayApply": "Public",
            "SortField": "closedate", "SortDirection": "Asc",
            "ResultsPerPage": 250, "Page": page,
        })
        r.raise_for_status()
        items = r.json()["SearchResult"]["SearchResultItems"]
        past_window = False
        for it in items:
            job = it["MatchedObjectDescriptor"]
            closes = closes_at(job)
            if closes < now:
                continue                      # already closed
            if closes > cutoff:
                past_window = True            # sorted by close date, so the rest are later
                break
            if it["MatchedObjectId"] in rows:
                rows[it["MatchedObjectId"]]["series"] += f";{series}"
                continue
            pay = job["PositionRemuneration"][0]
            rows[it["MatchedObjectId"]] = {
                "closes": closes.strftime("%a %d %b"),
                "days_left": (closes.date() - now.date()).days,
                "series": series,
                "title": job["PositionTitle"],
                "agency": job["OrganizationName"],
                "grade": f'{job["JobGrade"][0]["Code"]}-{job["UserArea"]["Details"]["LowGrade"]}'
                         f'/{job["UserArea"]["Details"]["HighGrade"]}',
                "pay": f'{pay["MinimumRange"]}-{pay["MaximumRange"]} {pay["RateIntervalCode"]}',
                "where": job["PositionLocationDisplay"],
                "url": job["PositionURI"],
            }
        if past_window or len(items) < 250:
            break
        page += 1
        time.sleep(1)

rows = sorted(rows.values(), key=lambda r: (r["days_left"], r["title"]))
with open("closing_this_week.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=list(rows[0]))
    w.writeheader()
    w.writerows(rows)
print(f"{len(rows)} public jobs close by {cutoff:%a %d %b} (ET)")
for d in range(DAYS + 1):
    n = sum(r["days_left"] == d for r in rows)
    if n:
        print(f"  {(now + timedelta(days=d)):%a %d %b}: {n}")

What to change:

SERIES: any four-digit occupational series codes. 0343 and 0201 are just the example.
DAYS: 7 for a weekly list, 14 if your readers need more notice.
Drop "WhoMayApply": "Public" if you also advise current federal employees. Status-only jobs will then be included.

Step 3: Run it

python closing_soon.py

Output on Monday 2 November 2026 (run at 3:46 a.m. ET):

211 public jobs close by Mon 09 Nov (ET)
  Mon 02 Nov: 70
  Tue 03 Nov: 41
  Wed 04 Nov: 25
  Thu 05 Nov: 27
  Fri 06 Nov: 40
  Sat 07 Nov: 6
  Sun 08 Nov: 2

Two lines of closing_this_week.csv from that run:

closes,days_left,series,title,agency,grade,pay,where,url
Mon 02 Nov,0,0201,Compensation and Leave Program Manager,"Treasury, Departmental Offices",GS-14/14,125776-187093 PA,...
Mon 02 Nov,0,0201,HR Policy Specialist (Recruitment and Placement),Office of Human Resources Management,GS-14/14,127829-197200 PA,...

Friday was busy too: 40 of the 211 closed on Friday 6 November.

Step 4: Schedule it for Monday morning

# 07:00 every Monday, server time
0 7 * * 1  cd /path/to/script && USAJOBS_KEY=... USAJOBS_EMAIL=... python closing_soon.py

Then email or share closing_this_week.csv. Re-run it midweek if you need to: a Monday list goes stale faster than you'd think.

Things that will catch you out

Closing early. The close date is the latest a job can close, not a promise. Agencies can close an announcement early or remove it, and an applicant cut-off usually closes it that night, once the limit is reached. Tell people to open the url and check before they spend an evening on an application.
Jobs that stay open for months. 21 of the 325 announcements had application windows longer than six months, some running into fall 2027. They only reach a "closing this week" list near their end date. If your readers want standing openings, run a separate list sorted by opendate instead.
Grades like SL-00/00. Senior-level (SL), Senior Executive (ES) and some other pay plans, such as AD, don't use GS grades, so low and high grade come back as 00. Read the pay range instead.
One job, several series. An announcement can list more than one occupational series. The script keys rows by job ID, so a job found under two of your series appears once with both codes. (In this run none overlapped.)
Pay interval. 210 of the 211 rows were PA (per year). One was PH (per hour). Check RateIntervalCode before you compare salaries.
'Public' isn't always everyone. USAJOBS's Who May Apply: Public filter also returns jobs open only to certain groups, such as veterans or military spouses. Check the job's hiring paths on the announcement before you pass it on.

The shortcut (Disclosure: we built this)

If you'd rather not maintain the script, our Apify Actor USAJOBS API (Unofficial) does the same sort and close-date window as input fields. It is a USAJOBS data tool, not an OPM or USAJOBS product.

curl -X POST "https://api.apify.com/v2/acts/yourevenings~government-jobs-api/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"jobCategoryCodes": ["0343", "0201"], "sortBy": "closingSoonest", "closingWithinDays": 7, "maxResults": 25}'

On 2 November 2026 that returned 25 jobs, all closing 2 November, each with closesAt, title, organization, payPlan, lowGrade, highGrade, salaryMinAnnualUsd, hiringPaths and url. Raise maxResults to get the whole week.

Pricing is pay per event: $0.002 per job returned, plus Apify's $0.00005 Actor start per run. A 25-job run is about $0.05 at base price. Status rows are free. You can schedule it weekly in Apify, or use monitor mode for alerts: later runs return only jobs you haven't had before, at $0.002 per new job plus $0.01 on each run that finds any.

https://apify.com/yourevenings/government-jobs-api?utm_source=yourevenings&utm_medium=devto&utm_campaign=federal-jobs-closing-this-week

Your Evenings

Find a company's subsidiaries from LEI data with Python (no API key)

Your Evenings Data — Sun, 01 Nov 2026 07:59:31 +0000

Article type: How-to guide

Prerequisites: curl and jq, or Python 3 with requests. No API key is needed for GLEIF. An Apify API token for the Actor section.

Tested on: 31 October 2026

If you do KYC or counterparty work you end up asking "which entities report this company as their parent?" The Global LEI Index answers that, it's free, and it needs no key. This post shows the raw API first, then a shortcut.

1. The raw GLEIF API

Every LEI record has relationship links. Direct children of Deutsche Bank AG (LEI 7LTWFZYICNSX8D621K86):

curl -sg "https://api.gleif.org/api/v1/lei-records/7LTWFZYICNSX8D621K86/direct-children?page[size]=10" \
  | jq -r '.data[] | [.id, .attributes.entity.legalName.name, .attributes.entity.jurisdiction] | @tsv'

Things to know:

Results are paged; follow links.next until it's gone. (The -g flag stops curl treating the square brackets in page[size] as a pattern.)
Use ultimate-children for the whole group, not just one level.
Parent info can be missing for a reason: a record may carry a reporting exception such as NO_KNOWN_PERSON or NATURAL_PERSONS instead of a parent. Keep that reason; "no parent" and "parent not reported" are different answers.
Check digits: an LEI is 20 characters with ISO 17442 (mod 97) check digits, so validate before calling.
The data is CC0, which makes it easy to store.

2. Doing it in Python

import requests

def children(lei, level="direct-children"):
    url = f"https://api.gleif.org/api/v1/lei-records/{lei}/{level}?page[size]=100"
    while url:
        r = requests.get(url, timeout=30)
        r.raise_for_status()
        j = r.json()
        for rec in j["data"]:
            e = rec["attributes"]["entity"]
            yield rec["id"], e["legalName"]["name"], e["jurisdiction"]
        url = j.get("links", {}).get("next")

for row in children("7LTWFZYICNSX8D621K86"):
    print(*row, sep="\t")

3. What a real run looked like

We ran the same question on 31 October 2026 and took the first 10 direct subsidiaries (newest first). A few of them:

LEI	Legal name	Jurisdiction	First LEI
9598005PGS3B6FWRJ071	FAIRFIELD INVEST SL	ES	2026-04-17
529900BXMRWF4BF5M980	GAHL HOLDINGS LLC	US-DE	2025-09-18
529900T45ZLIP8J4FH25	DB Advisory Services S.A.S.	CO	2024-08-22
5299007H1EFZEWKADW16	SGI SLP Feeder SCSp	LU	2024-07-11

Each row also had the legal form, register number, addresses, renewal date and the ultimate parent, so you can sort by nextRenewalDate to see which LEIs lapse first.

4. The shortcut (we built this)

We built an Apify Actor, LEI Lookup & Company Ownership API, that wraps the steps above: lookup, search and subsidiaries modes, check-digit validation before any request, parent LEI and name (or the exception reason) in every row, and a monitor mode for scheduled runs that returns only new or changed records. It is independent and not affiliated with GLEIF.

curl -s -X POST "https://api.apify.com/v2/acts/yourevenings~lei-company-lookup/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"mode":"subsidiaries","parentLei":"7LTWFZYICNSX8D621K86","childLevel":"direct","maxResults":10}'

It's pay per event: base price $0.003 per LEI record returned, plus $0.00005 per run per GB of memory for the Actor start (one event at the default 256 MB) and $0.01 per monitor run that returns new or changed records. Higher Apify plans pay less. Status rows are free. The code in section 2 is all you need if you'd rather run it yourself.

LEI Lookup & Company Ownership API on Apify

Your Evenings

Ontario nurse job postings from Canada's Job Bank open data, into a CSV every month

Your Evenings Data — Sat, 31 Oct 2026 17:01:52 +0000

Article type: Tutorial

Prerequisites: curl, jq and awk for the free route. An Apify API token for the scheduled Actor route. cron (or an Apify schedule) to run it monthly or weekly.

Tested on: 31 October 2026

Canada's national Job Bank publishes every posting it carries as open data: one file a month, from January 2023, on open.canada.ca. If you track hiring for one occupation in one province, for a workforce report, a settlement service or a pay comparison, that file has what you need: occupation code, place, vacancies, pay range, hours and the date the job was first posted. It does not include employer names, contact details or apply links.

This post pulls one slice, registered nurses (NOC 2021 code 31301) in Ontario, first with nothing but curl and awk, then as a scheduled job that adds each new month to a CSV.

Disclosure: the hosted Actor in the second half is ours (we built it) and it's paid per row. The free route comes first and is enough for a one-off.

The free route: one file, one awk line

The dataset record lists every monthly file. This gets the newest English one:

URL=$(curl -s "https://open.canada.ca/data/api/action/package_show?id=ea639e28-c0fc-48bf-b5dd-b8899bd43072" \
  | jq -r 'first(.result.resources[] | select(.format=="CSV" and (.language|index("en"))) | .url)')
curl -sL -o latest.csv "$URL"

On 31 October 2026 that was the September 2026 file: 46 MB, UTF-16 and tab-separated, with 65 columns and 52,368 postings. Convert it, then filter on NOC code (column 6) and province (column 17):

iconv -f UTF-16 -t UTF-8 latest.csv | tr -d '\r' > latest.tsv
awk -F'\t' 'NR>1 && $6=="31301" && $17=="Ontario" {print $2"\t"$18"\t"$10"\t"$39"\t"$40}' latest.tsv | head -3

registered nurse - telehealth   NA              10   Hour   38
registered nurse (R.N.)         Lindsay         10   Week   1562.72
registered nurse (R.N.)         Sturgeon Falls  1    Hour   41.15

That month had 83 Ontario postings under 31301. Column 8, External Indicator, splits them by source: 21 were posted on Job Bank by employers (0) and the rest were shared from other job boards (1).

Some things you'll hit if you keep doing this by hand:

The column numbers above are for this file. Columns have been added over time, and some 2024 months are UTF-8 and comma-separated instead, so match on header names, not positions.
Missing values are the text NA.
Rows shared from other job boards carry less detail. In September 2026 nearly all of them had a pay figure, but few had hours and none had an education or experience level.
open.canada.ca asks automated clients to wait 20 seconds between requests.

For one month, that's all you need. To get every month automatically, read on.

The same slice as clean rows

The Actor reads the same official files, filters while it streams and returns one flat JSON row per posting. Here's a real run from 31 October, capped at three rows and limited to postings made on Job Bank itself:

curl -s -X POST \
  "https://api.apify.com/v2/acts/yourevenings~canada-job-bank-data/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H 'Content-Type: application/json' \
  -d '{"months":["latest"],"nocCodes":["31301"],"provinces":["ON"],"postedVia":"jobBank","maxResults":3}' \
  > sample.json

It checked 2,284 rows of the September 2026 file and stopped after the third match. One row, shortened:

{
  "rowType": "posting",
  "postingId": "2026-09:15871793",
  "month": "2026-09",
  "jobTitle": "registered nurse (R.N.)",
  "noc2021Code": "31301",
  "noc2021Name": "Registered nurses and registered psychiatric nurses",
  "postedVia": "Job Bank (employer account)",
  "firstPostingDate": "2026-09-14",
  "vacancies": 1,
  "province": "Ontario",
  "city": "Sturgeon Falls",
  "postalArea": "P2B",
  "economicRegion": "Northeast Region",
  "industry": "Health care and social assistance",
  "employmentType": "Part time",
  "employmentTerm": "Permanent employment",
  "salaryText": "$41.15 to $58.98 hourly (to be negotiated)",
  "salaryPer": "Hour",
  "salaryMin": 41.15,
  "salaryMax": 58.98,
  "hoursPer": "Week",
  "hoursMin": 33.75,
  "hoursMax": 33.75,
  "educationLevel": "College/CEGEP",
  "experienceLevel": "3 years to less than 5 years",
  "sourceFile": "https://open.canada.ca/data/dataset/ea639e28-c0fc-48bf-b5dd-b8899bd43072/resource/03523cbf-a005-40ee-81ad-aca5bb74cf03/download/job-bank-open-data-all-job-postings-en-sept2026.csv",
  "licence": "Contains information licensed under the Open Government Licence - Canada (https://open.canada.ca/en/open-government-licence-canada). Source: Employment and Social Development Canada, Job Bank open data."
}

The run also returned one free status row, {"rowType":"status","status":"max_results", ...}, saying it stopped at the limit. Status rows sit in the same dataset, so filter on rowType when you build the CSV.

Every row keeps a link to the file it came from (sourceFile) and the licence statement (licence), so the attribution goes wherever the data goes. Postal codes are cut to their first three characters.

Every new month into one CSV

Turn on monitor mode and the Actor remembers which months it has already returned. Run it weekly. Most weeks it finds nothing new and returns a free already_reported row. When ESDC publishes the next month's file, usually a few weeks after the month ends, it returns that month's rows once.

#!/bin/sh
# on-nurses.sh: run weekly (cron, or an Apify schedule with the same input)
curl -s -X POST \
  "https://api.apify.com/v2/acts/yourevenings~canada-job-bank-data/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H 'Content-Type: application/json' \
  -d '{"months":["latest"],"nocCodes":["31301"],"provinces":["ON"],"maxResults":1000,
       "monitorMode":true,"stateStoreName":"on-nurses-31301"}' \
  > new.json

[ -f on-nurses.csv ] || echo 'month,firstPostingDate,jobTitle,city,economicRegion,vacancies,employmentType,salaryMin,salaryMax,salaryPer,postedVia' > on-nurses.csv
jq -r '.[] | select(.rowType=="posting")
  | [.month,.firstPostingDate,.jobTitle,.city,.economicRegion,.vacancies,.employmentType,.salaryMin,.salaryMax,.salaryPer,.postedVia] | @csv' \
  new.json >> on-nurses.csv

Some notes on the settings:

First run. It returns the newest month straight away. Set "emitBaselineOnFirstRun": false if you only want months published from now on.
maxResults. A month that maxResults cuts short still counts as read, so set it well above your slice. 1,000 is plenty for 83 postings a month.
stateStoreName. Use a different one for each saved search.
A different slice. Swap nocCodes (a code or its first digits, such as "72" for trades) or provinces, or add cities, keywords, employmentTypes, or salaryPer with minSalary.
Timeouts. Run-sync waits up to 300 seconds. If a call times out, start a normal run and read its dataset afterwards.

The CSV opens as-is in a spreadsheet. If you'd rather pull it straight into Google Sheets or Excel, the run's dataset can also be downloaded as CSV or Excel from the Apify API.

Honest limits

Monthly, not live. The newest file covers last month, so for today's postings use jobbank.gc.ca.
No employer names, contacts or apply links. The open data leaves them out. This is labour-market data, not a way to apply.
English files only. French versions exist with French column names, but the Actor doesn't read them yet.
Externally shared postings have fewer fields. Set "postedVia":"jobBank" if you need hours, education and experience on every row.
Each extra month adds about 20 seconds, because the Actor honours open.canada.ca's request spacing. Up to 12 months per run.
Cost: $0.002 per posting returned and $0.00005 per run start (one start event per GB of memory; the default 1 GB run pays one). Monitor mode adds $0.01 on a run that returns a new month. (These are base prices; higher Apify plans pay less.) Rows your filters drop, status rows and weeks with nothing new carry no posting or monitor charge. For the slice above, a month like September 2026 would be 83 Ã— $0.002 + $0.01 + $0.00005 = $0.17605. If you only want the 21 postings made on Job Bank itself, it's $0.05205.

Contains information licensed under the Open Government Licence – Canada. Source: Employment and Social Development Canada, Job Bank open data. This Actor and this post are not operated, sponsored, approved or endorsed by Employment and Social Development Canada, Job Bank or the Government of Canada.

The Actor, with the inputs above ready to run: Canada Job Bank Data: Job Postings, NOC & Salaries on Apify

Your Evenings

Five things to know before you build on the USAJOBS Search API

Your Evenings Data — Sat, 31 Oct 2026 11:32:33 +0000

Article type: Explanation

Prerequisites: None to read it. To build on the API yourself, you need a free USAJOBS API key.

Tested on: 1 November 2026 (the five points, checked against the live Search API; the Actor input example was not run)

USAJOBS publishes an official Search API (data.usajobs.gov/api/Search) for US federal job announcements. We built a scheduled job feed on it, and these are the details that shaped the code. Disclosure: the hosted version at the end is ours, and we charge for it.

1. Pay comes with an interval code

Each pay range comes with a RateIntervalCode: PA for per year, PH for per hour, plus PD, BW and PM for daily, biweekly and monthly, and PW for piece work. In USAJOBS's own code list, WC means without compensation, so there's no pay to convert. To compare jobs on salary, convert everything to a yearly figure. We use 2,087 hours for a year of hourly pay (the federal standard), 260 days, 26 pay periods or 12 months. Filter on the top of the range, or you'll drop hourly jobs that pay well.

2. One query stops at 10,000 results

The API won't page past 10,000 results for one query. A broad search such as "all jobs posted this month" can hit that limit without telling you. Narrow it by location, agency, occupational series or a posted-within window, and log it whenever a query reaches the limit, so a short result is never mistaken for a complete one.

3. Ask for public jobs explicitly

Many announcements are open only to current federal employees or particular groups. If your users are the general public, request WhoMayApply=Public.

4. Codes need a lookup table

Schedule, appointment, service type and travel come back as codes. Map them to words (Full-time, Permanent, Competitive, Occasional travel) before anyone downstream sees them, or every consumer ends up writing their own mapping.

5. Announcements contain contact details

Rows can carry an agency contact email and phone number, and the announcement text can include more of them, sometimes with a named person. Unless your use case needs them, drop the contact fields and mask emails and phone numbers in free text. Don't store full announcement text by default.

The hosted version (disclosure: we built it)

Our Apify Actor, Government Jobs API: USAJOBS, does all five for you:

Yearly pay is worked out for every row with a standard pay interval, and the salary filter runs before you're charged.
A free scan_limit row tells you when a search hit the 10,000 limit.
Public jobs only (with the default input), with codes turned into words.
Contact details are dropped and masked by default.
No API key is needed on your side.
Monitor mode returns only jobs it hasn't returned before.
Pricing: $0.002 per job, plus $0.01 for a monitor run that finds new jobs (base prices; higher Apify plans pay less).

{ "keywords": ["data scientist"], "locations": ["Washington, District of Columbia"], "radiusMiles": 25, "postedWithinDays": 14, "minSalaryUsd": 100000 }

Not an official USAJOBS or OPM product, and not endorsed by any federal agency. Always check the official announcement on usajobs.gov before you apply.

Yesterday's US private funding rounds from SEC Form D, in about 70 lines of Python (no API key)

Your Evenings Data — Sat, 31 Oct 2026 11:15:42 +0000

Article type: Tutorial

Prerequisites: Python 3 with requests. No API key; the SEC asks you to send a User-Agent with your contact email.

Tested on: 31 October 2026

When a US company raises money privately under Regulation D, it files a Form D with the SEC, usually within 15 days of the first sale. Every filing is public on EDGAR as structured XML, with the amount offered, the amount sold, the industry group and the issuer's state. EDGAR also publishes a daily index of everything filed that day. Put the two together and you get a daily list of new funding rounds, for free and with no API key.

Here's a script that does that and writes a CSV. Disclosure: at the end there's a hosted version that we built and charge for. The script works without it.

The two files you need

The daily form index: https://www.sec.gov/Archives/edgar/daily-index/{year}/QTR{q}/form.{yyyymmdd}.idx. It's a fixed-width text file with form type, company name, CIK, date and file path. It's published after the SEC's business day (US Eastern), and there's none at weekends or on federal holidays.
Each Form D's primary_doc.xml: https://www.sec.gov/Archives/edgar/data/{cik}/{accession without dashes}/primary_doc.xml.

The SEC asks automated tools to declare a User-Agent with a contact email and to stay under 10 requests a second. The script sleeps 0.2 s between requests, which keeps it to 5 a second at most.

The script

import csv, re, sys, time
import xml.etree.ElementTree as ET
from datetime import date, timedelta
import requests

# The SEC asks automated tools to say who they are. Put your own name and email here.
UA = {"User-Agent": "Your Name you@example.com"}
SEC = "https://www.sec.gov"

def get(url):
    time.sleep(0.2)  # at most 5 requests a second; the SEC's limit is 10
    r = requests.get(url, headers=UA, timeout=60)
    r.raise_for_status()
    return r

# 1. Find the latest daily form index (none at weekends or on federal holidays).
day = date.today()
for _ in range(10):
    day -= timedelta(days=1)
    q = (day.month - 1) // 3 + 1
    url = f"{SEC}/Archives/edgar/daily-index/{day.year}/QTR{q}/form.{day:%Y%m%d}.idx"
    r = requests.get(url, headers=UA, timeout=60)
    if r.status_code == 200:
        break
    time.sleep(0.2)
print("index", day)

# 2. Keep new Form D filings (not D/A amendments).
line_re = re.compile(r"^(.*?)\s+(\d{1,10})\s+(\d{8})\s+edgar/data/\d+/(\d{10}-\d{2}-\d{6})\.txt\s*$")
filings = []
for line in r.text.splitlines():
    if line[:17].strip() != "D":
        continue
    m = line_re.match(line[17:])
    if m:
        filings.append((m.group(1).strip(), int(m.group(2)), m.group(4)))
print(len(filings), "Form D filings")

def text(node, path):
    el = node.find(path)
    return el.text.strip() if el is not None and el.text else None

def amount(v):
    try:
        return float(v)
    except (TypeError, ValueError):
        return None  # "Indefinite" or missing

# 3. Read each primary_doc.xml. Company fields only: we never read relatedPersonsList.
rows = []
for name, cik, acc in filings[: int(sys.argv[1]) if len(sys.argv) > 1 else 50]:
    doc = ET.fromstring(get(f"{SEC}/Archives/edgar/data/{cik}/{acc.replace('-', '')}/primary_doc.xml").content)
    industry = text(doc, "offeringData/industryGroup/industryGroupType")
    if industry == "Pooled Investment Fund":
        continue  # keep operating companies
    total = amount(text(doc, "offeringData/offeringSalesAmounts/totalOfferingAmount"))
    sold = amount(text(doc, "offeringData/offeringSalesAmounts/totalAmountSold"))
    if max(total or 0, sold or 0) < 1_000_000:
        continue
    rows.append({"company": name, "state": text(doc, "primaryIssuer/issuerAddress/stateOrCountry"),
                 "industry": industry, "offered": total, "sold": sold,
                 "url": f"{SEC}/Archives/edgar/data/{cik}/{acc.replace('-', '')}/primary_doc.xml"})

with open("form-d.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["company", "state", "industry", "offered", "sold", "url"])
    w.writeheader(); w.writerows(rows)
print(len(rows), "offerings of $1M+ from operating companies")

Install requests (pip install requests), then run python3 formd.py 40 to read the first 40 filings. When I ran it on Monday 26 October 2026, it picked up Friday's index (23 October) and printed:

index 2026-10-23
188 Form D filings
12 offerings of $1M+ from operating companies

Gotchas

"Indefinite" is a valid amount. Funds often put Indefinite as the total offering amount. The script treats it as unknown, and the amount sold usually tells you more.
Many filers are funds. If you want operating companies, drop industry group Pooled Investment Fund, as the script does.
D/A is an amendment. The index lists amendments as D/A. The script skips them, so a raise that's updated isn't counted twice.
Leave the people out. A Form D also lists executives, directors and promoters, with their addresses. The script never reads relatedPersonsList. Unless you have a clear reason to process personal data, don't.
Monday's run sees Friday. There's no index for Saturday or Sunday, so the script steps back to the latest one that exists.

The hosted version (disclosure: we built it)

Our Apify Actor, SEC EDGAR Filings & Form D Funding Monitor, does the same thing in formD mode:

Filters for industry group, issuer state, minimum offering size and securities type. Pooled funds are excluded by default.
Company data only. Related persons, signers and sales-compensation names and addresses are dropped (you get a count), and so are issuer phone numbers and street addresses.
Monitor mode remembers what it has returned and gives you only new filings on later runs.
Pricing: $0.003 per Form D row, plus $0.01 for a monitor run that finds something (base prices; higher Apify plans pay less). Non-matching filings and status rows are free.

{ "mode": "formD", "filedWithinDays": 3, "minOfferingUsd": 1000000, "maxResults": 50 }

It also has company filings, latest filings by form type, full-text search and key financials modes.

Not affiliated with or endorsed by the U.S. Securities and Exchange Commission. Filings are as submitted by the filers. Nothing here is investment advice.

UK Companies House data with curl and Python: new companies by area, and client due dates

Your Evenings Data — Sat, 31 Oct 2026 10:55:30 +0000

Companies House publishes a free snapshot of every live UK company once a month: name, number, status, incorporation date, up to four SIC codes, registered office post town and postcode, and the accounts and confirmation statement due dates. If you sell to new businesses, or just want to know who started up in your sector and area last month, that file is all you need.

This post does it twice: first with pandas on the raw file, then with a hosted API that keeps the monthly comparison for you. At the end there's a due-date check for a list of client companies, the free API way and the hosted way.

Disclosure: the hosted API in the second half is ours, and it's paid per row. The free route comes first and works without it.

The free route: the Basic Company Data file

Download it from the Free Company Data Product page. It's split into 7 zipped CSV files of about 70 MB each (the 1 October 2026 snapshot has 5,704,711 live companies). No key and no sign-up.

Companies that are new, in IT (SIC 62xxx) and registered in the Manchester postcode area (M):

import pandas as pd

cols = ["CompanyName", "CompanyNumber", "CompanyStatus", "IncorporationDate",
        "RegAddress.PostTown", "RegAddress.PostCode", "SICCode.SicText_1"]

df = pd.read_csv("BasicCompanyData-2026-10-01-part1_7.csv", dtype=str,
                 usecols=lambda c: c.strip() in cols)
df.columns = df.columns.str.strip()          # some headers have a leading space
df["inc"] = pd.to_datetime(df["IncorporationDate"], format="%d/%m/%Y", errors="coerce")

new_it_mcr = df[
    df["SICCode.SicText_1"].str.startswith("62", na=False)
    & df["RegAddress.PostCode"].str.match(r"^M\d", na=False)   # M1..M99, not ME/ML/MK
    & (df["inc"] >= "2026-09-01")
]
print(len(new_it_mcr))

On part 1 of 7 that took about 4 seconds and found 20 companies. Loop over the 7 files for the whole register.

Two things to know:

SIC codes are in four columns (SICCode.SicText_1 to _4). The snippet only checks the first. A company that lists "62012" second is missed unless you check all four.
"New" by incorporation date isn't the same as "new this month". If you want what changed between two snapshots (new companies, companies that left the register, status changes such as liquidation, accounts that just became overdue), keep last month's file and diff on CompanyNumber. That's where it gets fiddly: 5.7 million rows a month, and a change in any of a dozen fields.

The API route: one call for a month-over-month change feed

UK Companies House on Apify loads each monthly snapshot, compares it with the one before, and returns typed change rows. The same query as above, as changes:

curl -s -X POST \
  "https://api.apify.com/v2/acts/yourevenings~uk-companies-house/run-sync-get-dataset-items?maxTotalChargeUsd=0.05" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "changes",
    "sicCodes": ["62"],
    "postcodes": ["M"],
    "changeTypes": ["new_on_register"],
    "maxResults": 5,
    "countAllMatches": true
  }' | jq '.[] | {changeType, incorporatedOn, postTown: .registeredOffice.postTown, sicCodes, confirmationStatementNextDue}'

sicCodes is a prefix match across all four SIC codes (62 = all IT, 62012 = software development only). postcodes takes an area (M), a district (M1) or a full postcode.

When I ran this on 31 October 2026, against the 1 October snapshot, 198 companies matched, and maxResults capped the output at 5. One row, trimmed:

{
  "changeType": "new_on_register",
  "incorporatedOn": "2026-09-16",
  "postTown": "MANCHESTER",
  "sicCodes": ["58210", "58290", "62011", "62020"],
  "confirmationStatementNextDue": "2027-09-29"
}

Full rows also have the company name and number, status, type, accounts due dates and overdue flags, charges, previous names, dataAsOf and a sourceUrl to the company's page on the Companies House service.

That run charged 5 change rows and one start: $0.02505. Prices are $0.005 per change row, $0.003 per company row and $0.00005 per run start (base prices; higher Apify plans pay less). maxTotalChargeUsd caps the run, and countAllMatches gives you the full match count in the run summary at no extra charge.

Other change types work the same way: removed_from_register, status_changed, name_changed, sic_changed, registered_office_moved, accounts_now_overdue, accounts_filed, confirmation_statement_now_overdue, confirmation_statement_filed, new_charge and a few more. For scale, the October snapshot had 69,566 companies new on the register and 62,734 whose accounts had just become overdue, across the whole UK. So filter by SIC, postcode or status unless you really want all of them.

Watching your own list

If you already have a list of customers or suppliers, give their numbers and turn on monitorMode:

{
  "companyNumbers": ["00445790", "07524813"],
  "monitorMode": true,
  "stateStoreName": "my-suppliers"
}

The first run saves a free baseline. Schedule it monthly, and later runs return one row per change since then: proposal to strike off, liquidation, accounts now overdue, a new charge, a moved registered office. A run before the next snapshot is out returns a free no_new_snapshot row.

Check confirmation-statement and accounts due dates for a client list

A different job with the same data: you look after a list of client companies and want to know which ones have a confirmation statement or accounts due in the next 30 days, or already late. Two ways to do it.

Free: the Companies House REST API, one call per company

For reminders this is the one to use, because it reads the live register, so a client who filed yesterday won't show as due.

Register on the Companies House developer hub (developer.company-information.service.gov.uk), create an application and add a REST API key. It's free. Auth is HTTP basic with the key as the username and a blank password. The limit is 600 requests per 5 minutes per application; go over it and you get 429 for the rest of that window.

Each GET /company/{company_number} returns the dates you need:

confirmation_statement.next_due and confirmation_statement.overdue
accounts.next_accounts.due_on and accounts.next_accounts.overdue (accounts.next_due still comes back too, but it's marked deprecated)

With your client numbers in a clients.csv with a company_number column:

import csv, os, time, datetime as dt, requests

KEY = os.environ["CH_API_KEY"]                # keep the key out of the script
API = "https://api.company-information.service.gov.uk/company/"
today = dt.date.today()
soon = today + dt.timedelta(days=30)

def profile(num):
    num = num.strip().upper()
    if num.isdigit():
        num = num.zfill(8)                    # 1234567 -> 01234567; SC/NI numbers unchanged
    while True:
        r = requests.get(API + num, auth=(KEY, ""), timeout=30)
        if r.status_code == 429:              # over the limit: wait for the window to reset
            time.sleep(60)
            continue
        time.sleep(0.55)                      # stays under 600 requests per 5 minutes
        return num, (r.json() if r.ok else None)

def day(s):
    return dt.date.fromisoformat(s) if s else None

with open("clients.csv", newline="") as f, open("due.csv", "w", newline="") as out:
    w = csv.writer(out)
    w.writerow(["company_number", "cs_next_due", "accounts_next_due", "chase"])
    for row in csv.DictReader(f):
        num, p = profile(row["company_number"])
        if p is None:
            w.writerow([num, "", "", "check the number"])
            continue
        cs = day(p.get("confirmation_statement", {}).get("next_due"))
        acc = day(p.get("accounts", {}).get("next_accounts", {}).get("due_on"))
        late_or_soon = [d for d in (cs, acc) if d and d <= soon]
        w.writerow([num, cs or "", acc or "", "yes" if late_or_soon else ""])

At that pace 500 clients take about 5 minutes. chase is "yes" for anything due within 30 days or already past due. A "check the number" row is a number the API didn't return, usually a typo in the sheet. If every row says that, check your API key first. Have the script fill in the sheet rather than send anything, and do your reminder emails from the result once you've looked at it.

Hosted: a monthly sweep with the same Actor

If you'd rather not hold a Companies House key, the uk-companies-house Actor from earlier (ours) does a sweep of your list from the monthly snapshot. Use lookup mode with the numbers:

curl -s -X POST \
  "https://api.apify.com/v2/acts/yourevenings~uk-companies-house/run-sync-get-dataset-items?maxTotalChargeUsd=1.60" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "lookup",
    "companyNumbers": ["00445790", "185647", "07524813"],
    "maxResults": 500
  }' > sweep.json

Set maxResults to the size of your list: it's the maximum number of charged rows, and the default is 25. Short numbers are zero-padded, so 185647 is read as 00185647, and SC, NI and OC numbers work as they are. A number listed twice comes back once.

Each company row has the four fields you want, plus dataAsOf (the snapshot date) and a sourceUrl to the company's page on the Companies House service. Trimmed, from a lookup run on 30 October 2026 against the 1 October snapshot:

{
  "recordType": "company",
  "companyNumber": "00445790",
  "accountsNextDue": "2027-08-26",
  "accountsOverdue": false,
  "confirmationStatementNextDue": "2027-07-02",
  "confirmationStatementOverdue": false,
  "lastConfirmationStatementMadeUpTo": "2026-06-18",
  "dataAsOf": "2026-10-01",
  "sourceUrl": "https://find-and-update.company-information.service.gov.uk/company/00445790"
}

The dataset also holds free status rows (not_found, not_on_live_register, invalid_number, duplicate), so filter on recordType. Due within 30 days or already past due, judged against today:

jq --arg soon "$(date -d "+30 days" +%F)" '.[] | select(.recordType == "company")
  | select([.confirmationStatementNextDue, .accountsNextDue]
           | map(select(. != null and . <= $soon)) | length > 0)
  | {companyNumber, confirmationStatementNextDue, accountsNextDue, dataAsOf, sourceUrl}' sweep.json

$soon is today plus 30 days (on macOS use date -v+30d +%F). Don't filter on accountsOverdue or confirmationStatementOverdue for this: both are worked out against the snapshot date, not today, so a date that fell due since the snapshot still reads false.

The catch for reminders is the monthly snapshot. It's compiled to the end of the previous month, so a client who filed after that still shows the old due date. Treat the sweep as a shortlist: before you chase anyone, open their sourceUrl, or run just the flagged numbers through the free script above.

Cost at base prices is $0.003 per company row plus $0.00005 per run start, so a 500-client list is about $1.50 a run. Not-found, invalid and duplicate rows are free. maxTotalChargeUsd caps the run.

If you run it monthly anyway, monitorMode (see "Watching your own list" above) gives you only what changed: confirmation_statement_now_overdue, confirmation_statement_filed, accounts_now_overdue and accounts_filed are among the change types.

Honest limits

Monthly, not live. The snapshot is compiled to the end of the previous month and published within 5 working days of month end. Overdue flags are worked out against the snapshot date. For today's position, follow the row's sourceUrl, or use the free Companies House REST API for a handful of companies.
Company-level records only (people records aren't included). Registered office street lines are dropped (some very small companies use a home address). Post town, postcode and country are kept.
Live companies only. A company that dissolved shows up once as removed_from_register, then it's gone.
The last 3 months are kept (dataMonth picks an older one).
Not affiliated with or endorsed by Companies House.

If you only need this once, the pandas route is fine. If you want the monthly diff without keeping 7 files a month around, that's the job the API does.

Shopify scraper returns 0 products? How to tell which of 6 reasons it is

Your Evenings Data — Sat, 31 Oct 2026 10:44:49 +0000

Article type: How-to guide

Prerequisites: A web browser for steps 1–3. For the code: Python 3 with apify-client and an Apify API token.

Tested on: 1 November 2026 (steps 1–2 checked against a live store's public feed; the Actor call in the code block was not re-run)

You ran a Shopify scraper, it said "Succeeded", and the dataset is empty. Here's how to find out why in two minutes with nothing but a browser, and how to make your own code tell an empty store from a failed read.

Step 1: open the feed yourself

Go to https://STORE-DOMAIN/products.json?limit=250 in a browser.

What you see	What it means	What to do
JSON with `"products": [ … ]`	The feed works. The problem is your tool or your input	Check the input: use the bare domain, not a collection or product URL
`{"products": []}`	It's a Shopify store, but nothing is published to the online store	Nothing to scrape. That's a real zero
A 404, but the HTML contains `cdn.shopify.com` or `myshopify.com`	It's a Shopify store with the public feed closed (headless storefront, or the feed is disabled or redirected)	There's no public catalogue to read. Treat it as "unavailable", not "empty"
Password page	The store is password-protected (pre-launch or wholesale)	It's not public, so stop
403, 429 or 430	You're being rate-limited or blocked	Slow down to about one request a second per store, back off on 429, and don't hammer many stores in parallel from one IP
A plain 404 with no Shopify markers in the HTML	Not a Shopify store	Use a different approach

How to check whether a site is a Shopify store: look for cdn.shopify.com, Shopify.theme or myshopify.com in the page source, or open /meta.json. A Shopify store answers it with its name, currency and published product count.

Step 2: the pagination traps that lose products silently

250 per page is the maximum. Ask for limit=500 and you get 250.
Don't stop at the first short page. Some stores return a short page in the middle of the catalogue. Keep requesting page=N+1 until you get "products": [].
The 25,000-product ceiling. The feed stops at page 100. Page 101 returns an error. For bigger stores, use the public product sitemap and collection feeds (/collections/{handle}/products.json), and dedupe by product id.
Prices are in the store's base currency, and the feed doesn't say which. Read currency from /meta.json.
No stock counts in the bulk feed. It only gives available: true/false. Some stores publish a quantity in each product's own /products/{handle}.json, and many don't.

Step 3: make zeros explain themselves

Whatever you build, return one row per store with a status and a reason, even when there are no products. "0 rows, no reason" is the most common complaint about Shopify scrapers, and it's cheap to fix.

The ready-made version (disclosure: we built this)

Our Apify Actor, Shopify Products Scraper & AI Shopping Readiness Audit, runs Step 1 for every domain you give it. Every store it can't read gets a free status row with a plain-English note:

not_shopify
shopify_products_unavailable (password-protected, headless or blocked)
blocked_by_robots
blocked
no_products
duplicate
invalid_input
error

That means pasting a list of domains also works as a free "is it Shopify?" check.

For stores that work:

Every product. It keeps paging past short pages until the feed is really empty. Above 25,000 products it reads the rest from the sitemap, collection feeds and product pages. Each audit row says whether the whole catalogue was read (catalogueComplete) and the store's known total (totalProductsKnown).
Every image URL and every variant (SKU, price, compare-at price, options, availability), plus a one-row-per-variant view. Stock counts appear only where the store publishes them, and are never estimated.
An optional 0–100 readiness audit of product data (GTINs, descriptions, alt text, SKUs and more).
maxProductsPerStore is a hard cap. robots.txt (including Crawl-delay) is respected, with one request per second per store.

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("yourevenings/shopify-catalogue-audit").call(
    run_input={"stores": ["allbirds.com", "example.com"], "maxProductsPerStore": 100})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["recordType"] == "store":          # stores it couldn't read
        print(row.get("store"), row["status"], row.get("note"))

It costs $0.0008 per product row and $0.02 per store audit, plus a $0.00005 start fee per run (base prices; higher Apify plans pay less). Status rows are free.

It reads only public storefront data and collects no customer data.

What's the strangest "0 products" cause you've hit? Tell us in the comments.

New LLCs in New York, Colorado, Oregon and Connecticut every morning, with curl and jq

Your Evenings Data — Fri, 30 Oct 2026 20:18:27 +0000

Article type: How-to guide

Prerequisites: curl and jq. No API key is needed for the state open-data APIs. For the Actor section: an Apify API token in APIFY_TOKEN.

Tested on: 30 October 2026

A business that registered this week has no reviews, often no website, and hasn't picked its accountant, insurer, bank or agency yet. That makes new filings one of the few lead sources where timing does the qualifying for you. The same records also answer the KYB question "is this entity real, and is it still on the register?".

Several states publish their business register as official open data. This post pulls the last week's filings from four of them into one shape with curl and jq, filters by industry, and then sets up a daily watch on entities you already work with.

Disclosure: the hosted API used here is ours, and it's paid per record. The free open-data queries are shown too, so you can see exactly where the data comes from.

What the raw sources look like

Each state runs a Socrata portal, and you can query them directly with no key. Connecticut, everything registered since 20 October:

curl -sG "https://data.ct.gov/resource/n7gp-d28j.json" \
  --data-urlencode "\$select=accountnumber,name,business_type,status,date_registration,naics_code" \
  --data-urlencode "\$where=date_registration>='2026-10-20T00:00:00'" \
  --data-urlencode "\$order=date_registration DESC" \
  --data-urlencode "\$limit=50000" | jq length

As of 30 Oct 2026 that returned 1,768 entities, and Colorado's dataset (data.colorado.gov, 4ykn-tg5h) returned 6,024 for the same window on the same day. The counts grow as new filings land. For one state that's all you need.

The work starts when you want four: different column names and status words, different date columns, and different ideas of what's in the dataset at all. Connecticut's naics_code arrives as one string like "Other Spectator Sports (711219)". Colorado puts notes like ", Delinquent April 1, 2020" inside the entity name.

Four states, one shape

The Actor normalises all four into one row format. newFilings mode returns entities formed in a window, newest first:

curl -s -X POST \
  "https://api.apify.com/v2/acts/yourevenings~us-state-business-entities/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"mode":"newFilings","states":["NY","CO","OR","CT"],"filedWithinDays":7,
       "maxResults":100,"maxResultsPerState":25}' \
  > week.json

That input is the README's own "new business filings in the last 7 days" example. Raise maxResultsPerState (0 means no limit) once you've seen the volume. Each entity row looks like this (a lookup row from the 30 October test run, shortened):

{
  "rowType": "entity",
  "state": "OR",
  "registerId": "338616",
  "name": "PORTLAND TERMINAL RAILROAD COMPANY",
  "entityType": "DOMESTIC BUSINESS CORPORATION",
  "status": "active",
  "statusDetail": "Listed in the Active Businesses dataset",
  "formationDate": "1882-08-28",
  "address": { "street": "3500 NW YEON AVE", "city": "PORTLAND", "state": "OR", "postalCode": "97210", "full": "3500 NW YEON AVE, PORTLAND, OR 97210" },
  "addressType": "principal_place_of_business",
  "naicsCode": null,
  "recordUrl": "https://egov.sos.state.or.us/br/pkg_web_name_srch_inq.do_name_srch?...",
  "dataset": "https://data.oregon.gov/d/tckn-sxa6",
  "licence": "Public domain (ORS 276A.368)"
}

Status is always one of active, not_in_good_standing, inactive or unknown, with the state's own words in statusDetail. Only Colorado and Connecticut publish status data beyond "active", so not_in_good_standing and inactive come from those two. recordUrl points at the state's own page for that entity, which is useful when a human needs to double-check.

Filter to the leads you actually want

LLCs only, one row per line, ready for a CSV:

jq -r '.[] | select(.rowType=="entity" and (.entityType|test("LIMITED LIABILITY|LLC";"i")))
       | [.state, .registerId, .name, .formationDate, .address.city] | @csv' week.json

For Connecticut you also get an industry code, already split into naicsCode and naicsDescription, so a "new consultancies" or "new restaurants" list is one more filter:

jq -r '.[] | select(.state=="CT" and ((.naicsCode // "") | startswith("5416")))
       | [.name, .naicsCode, .naicsDescription, .formationDate] | @tsv' week.json

Other filters that pay off: drop names that look like property holding vehicles ("123 MAIN ST LLC"), and anything whose jurisdictionOfFormation isn't the state itself if you only want local businesses. That field keeps each state's own wording ("New York" and "Connecticut", but "CO" and "OR"), so match both forms.

A daily change watch for KYB

For customers or suppliers you already have, switch to lookup with monitorMode. Schedule it daily and it returns a row only when something watched changes (name, type, status, closed date, jurisdiction, addresses or NAICS):

curl -s -X POST \
  "https://api.apify.com/v2/acts/yourevenings~us-state-business-entities/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"mode":"lookup","entities":["CO:18961000001","CT:0067276","NY:4424185"],
       "monitorMode":true,"emitBaselineOnFirstRun":false,"stateStoreName":"kyb-us-entities"}' \
  | jq '.[] | select(.changeType=="updated") | {state, registerId, name, changedFields, previousValues}'

If Colorado or Connecticut changes an entity's status, it shows up with changedFields: ["status"] and the old value in previousValues. In New York and Oregon, an entity that leaves the dataset gets a free no_longer_listed row instead (more on why below).

Honest limits

Four states only: NY, CO, OR and CT. They're included because they publish their business registers as open data that allows reuse. Most states don't.
Status data differs per state. Only Colorado and Connecticut publish all statuses. The New York and Oregon datasets list active entities only. A dissolved NY company comes back not_found, which may mean "inactive", not "never existed".
Freshness. New York's dataset can lag the Department of State by a few days, and Oregon refreshes about weekly. "Filed in the last 7 days" can miss the newest filings there. Use a slightly longer window and dedupe.
No people. Officer and owner names aren't requested. Sole proprietorships, DBAs and name reservations are skipped and free. Registered agents are opt-in (includeRegisteredAgent), NY and CO only, and only when the agent is an organisation.
Addresses can be homes. Small LLCs often register at the founder's home address. If you're using this for outreach, treat the address with care and follow the rules for your channel.
Colorado's disclaimer travels with the data. Every Colorado row carries the state's required notice in attribution. Keep it if you republish.
Run-sync waits up to 300 seconds. The portals are paced at up to 2 requests a second per state, so for big windows start an async run and fetch the dataset afterwards.
Cost: $0.0025 per entity row and $0.00005 per run start. Monitor mode adds $0.01 per run that returns at least one new or updated entity. Status rows, skipped records and monitor runs with nothing new carry no record or alert charges. Higher Apify plans get lower prices.

The Actor, with the input examples above ready to run: US State Business Entities API on Apify

KYB checks across six EU company registers from one Python script (FR, NO, FI, CZ, IE, PL)

Your Evenings Data — Fri, 30 Oct 2026 19:19:50 +0000

Article type: How-to guide

Prerequisites: Python 3 with requests, and an Apify API token in APIFY_TOKEN (the script calls the Actor). The free single-country official APIs mentioned near the end need no key.

Tested on: 30 October 2026

If you onboard European suppliers or customers, "does this company exist, is it still active, and is that its registered address?" means six different registers with six different APIs, ID formats and status vocabularies. This post builds a small KYB check that takes a list of register IDs and flags anything that needs a human. Then it turns the same check into a daily watch for liquidations.

Disclosure: the hosted API used here is ours, and it's paid per record. The free official route for each country is noted as we go, so you can build it yourself if you'd rather.

The input: country-prefixed register IDs

Each country has its own ID: a SIREN in France, an organisation number in Norway, a Business ID in Finland, an IČO in Czechia, a CRO number in Ireland and a KRS number in Poland. Keep them as CC:ID strings. That's the only format decision you need:

COUNTERPARTIES = [
    "FR:552032534",   # SIREN, 9 digits
    "NO:923609016",   # organisasjonsnummer, 9 digits
    "FI:0112038-9",   # Y-tunnus
    "CZ:00177041",    # IČO, 8 digits
    "IE:104547",      # CRO number
    "PL:0000028860",  # KRS, 10 digits
]

One call, one schema

The Actor's lookup mode takes up to 500 of these per run. Calling it through the Apify API's run-sync endpoint returns the dataset rows directly:

import os, requests

ACTOR = "yourevenings~eu-company-registers"
URL = f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items"

def lookup(ids):
    r = requests.post(
        URL,
        params={"token": os.environ["APIFY_TOKEN"]},
        json={"mode": "lookup", "companies": ids, "maxResults": 0},
        timeout=310,  # run-sync waits up to 300 s; use async runs for big batches
    )
    r.raise_for_status()
    return r.json()

Every company row has the same fields whichever register it came from: country, registerId, name, legalForm, status, registeredDate, closedDate, address{street, postalCode, city, country, full}, activityCode, vatId, website, recordUrl (the register's own page) plus licence and attribution. status is normalised to active, inactive, in_liquidation, bankrupt or unknown. Where the register gives its own status wording, it's kept in statusDetail (Ireland's "Normal", for example); otherwise statusDetail is null.

For the six IDs above, a run on 30 October 2026 returned:

Country	ID	Name	Status	City
FR	552032534	DANONE	active	PARIS
NO	923609016	EQUINOR ASA	active	STAVANGER
FI	0112038-9	Nokia Oyj	active	ESPOO
CZ	00177041	Škoda Auto a.s.	active	Mladá Boleslav
IE	104547	RYANAIR DESIGNATED ACTIVITY COMPANY	active	— (see below)
PL	0000028860	ORLEN SPÓŁKA AKCYJNA	active	PŁOCK

Ireland's CRO data gives the address as a single line, so address.city is null there and the whole address is in address.full ("RYANAIR DUBLIN OFFICE, AIRSIDE BUSINESS PARK, SWORDS CO. DUBLIN, SWORDS,DUBLIN"). Match on address.full if you compare addresses across countries.

The part that matters for KYB: rows that aren't companies

An empty result is the dangerous case in a KYB check. You can't tell "not registered" from "typo" or "register was down". So the API never returns nothing silently. Each problem comes back as a free status row with rowType: "status", a status code and a plain-English message:

invalid_id: the ID doesn't have the right shape for that register. Usually a typo in your own data.
not_found: well-formed, but not in the register.
source_error: the register was unreachable after retries. The other countries still run.
skipped_individual: a sole trader or natural person. These are skipped, not returned.

So the triage is a small function:

def triage(rows, expected):
    out = {"ok": [], "review": []}
    for row in rows:
        if row["rowType"] == "status":
            out["review"].append((row.get("input") or row.get("country"), f'{row["status"]}: {row["message"]}'))
        elif row["status"] != "active":
            out["review"].append((row["registerId"], f'{row["name"]}: {row["status"]} ({row.get("statusDetail")})'))
        elif expected.get(row["registerId"]) and row.get("vatId") and row["vatId"] != expected[row["registerId"]]:
            out["review"].append((row["registerId"], f'VAT on file {expected[row["registerId"]]} != register {row["vatId"]}'))
        else:
            out["ok"].append(row["registerId"])
    return out

expected is whatever your CRM holds, for example {"923609016": "NO923609016MVA"}. vatId is only filled where the register publishes one. It's null otherwise, so treat a null as "can't check here", not as a mismatch.

Turn it into a daily liquidation watch

A one-off check goes stale. Set monitorMode: true and schedule the same input daily. The Actor stores each company's name, legal form, status, closure date, address, activity code, website and VAT number, and from then on returns only companies whose watched fields changed:

{
  "mode": "lookup",
  "companies": ["FR:552032534", "NO:923609016", "FI:0112038-9"],
  "monitorMode": true,
  "emitBaselineOnFirstRun": false,
  "stateStoreName": "kyb-suppliers"
}

Changed rows carry changeType: "updated", changedFields (e.g. ["status"]) and previousValues, so an active → in_liquidation move is one row you can route to a webhook or Slack. With emitBaselineOnFirstRun: false the first run just records the baseline, and runs where nothing changed return no company rows. Status rows such as source_error still come back, so a register that doesn't answer shows up as a row, not as an empty day.

New registrations (NO, FI, IE)

Norway, Finland and Ireland expose registration dates, so the same API can list companies registered in a window. It's useful for market research, or for checking that a "new" counterparty really is new:

{ "mode": "newCompanies", "countries": ["NO", "FI", "IE"], "registeredWithinDays": 7, "activeOnly": true, "maxResultsPerCountry": 100 }

If you only need one of those countries, the free official APIs are a single call. Brønnøysund's data.brreg.no/enhetsregisteret/api/enheter?fraRegistreringsdatoEnhetsregisteret=2026-10-20 and PRH's avoindata.prh.fi/opendata-ytj-api/v3/companies?registrationDateStart=2026-10-20 both work with no key. The wrapper is worth it when you need several countries in one schema.

Honest limits

Company-level data only. No directors, owners, phone numbers or emails. If your KYB needs beneficial owners, that's a separate source.
Six countries. Poland is lookup only (the KRS API has no name search). newCompanies is Norway, Finland and Ireland only.
Registers page differently. ARES (CZ) returns at most 1,000 matches, and broad searches elsewhere hit a 10,000 cap. Over-broad searches return too_many_results rather than a partial list.
Freshness. Each run reads the registers live, but Ireland's open dataset is refreshed on the CRO's schedule, so very recent changes can show there later than on its website.
Marketing use. French records whose owner opted out of commercial prospecting carry extra.optedOutOfCommercialProspecting: true. Respect it.
Pacing. Requests are rate-limited per register (1–4 a second), so a 500-ID batch can take several minutes, longer than run-sync's 300-second wait. Start an async run and read the dataset when it finishes.
Cost. $0.003 per company row and $0.00005 per run start. Monitor mode adds $0.01 per run that finds a change. Status rows and quiet monitor runs are free. Higher Apify plans get lower prices.

The Actor, with input examples you can run from the browser: EU Company Registers API on Apify

Get an alert when a SAM.gov solicitation you're bidding on is amended (Python, no API key)

Your Evenings Data — Fri, 30 Oct 2026 09:35:27 +0000

Article type: How-to guide

Prerequisites: Python 3 with requests; cron. No SAM.gov API key is needed.

Tested on: 30 October 2026

You've found a solicitation and you're writing the bid. Then the agency posts an amendment that moves the deadline or changes the scope, and you find out late. Here's a small script that checks a watch list of solicitation numbers every morning and tells you what changed. It needs no API key.

Where the data comes from

SAM.gov publishes every active contract opportunity as a public CSV extract, regenerated once a day (around 03:30 UTC recently):

https://sam.gov/api/prod/fileextractservices/v1/api/download/Contract%20Opportunities/datagov/ContractOpportunitiesFullCSV.csv?privacy=Public

The trick is that an amendment shows up as a new notice ID under the same solicitation number (the Sol# column). So to track a solicitation, save its notice IDs (and the fields you care about), then compare them tomorrow:

A new ID means an amendment or follow-on notice.
Same ID, different fields means it was edited in place.
An ID that's gone means the notice left the active list.

Unlike a "new bids" digest, you can't stop reading early: amendments land under solicitations that were first posted weeks ago. So read the whole file. It was about 210 MB on 22 Oct.

The script

import csv, json, re, requests
from pathlib import Path

EXTRACT = ("https://sam.gov/api/prod/fileextractservices/v1/api/download/"
           "Contract%20Opportunities/datagov/ContractOpportunitiesFullCSV.csv?privacy=Public")
WATCH = ["1240LT26Q0109", "W912DY26R0001"]      # the solicitations you're bidding on
STATE = Path("sam_watch.json")
FIELDS = ["Type", "ResponseDeadLine", "SetASideCode", "NaicsCode", "Title", "ArchiveDate"]

norm = lambda s: re.sub(r"[^A-Z0-9]", "", s.upper())
watch = {norm(s) for s in WATCH}

# Amendments land under old solicitation numbers, so read the whole file
# (about 210 MB). Save it to disk first: streaming csv straight off r.raw
# to the end of the file fails with "I/O operation on closed file".
with requests.get(EXTRACT, stream=True, timeout=120) as r:
    r.raise_for_status()
    print("extract last modified:", r.headers.get("Last-Modified"))
    with open("extract.csv", "wb") as f:
        for chunk in r.iter_content(1 << 20):
            f.write(chunk)

current = {}                                    # sol# -> {notice_id: fields}
with open("extract.csv", encoding="cp1252", newline="") as f:
    for row in csv.DictReader(f):
        sol = norm(row["Sol#"])
        if sol in watch and row["Active"] == "Yes":
            current.setdefault(sol, {})[row["NoticeId"]] = {
                "posted": row["PostedDate"], **{k: row[k] for k in FIELDS}}

saved = json.loads(STATE.read_text()) if STATE.exists() else {}
for sol in sorted(watch):
    now, before = current.get(sol, {}), saved.get(sol)
    if before is None:
        print(f"{sol}: tracking started, {len(now)} active notice(s)"
              + ("" if now else " (not in the active extract)"))
        continue
    for nid in now.keys() - before.keys():
        print(f"{sol}: NEW notice {nid} ({now[nid]['Type']}, posted {now[nid]['posted'][:10]}, "
              f"due {now[nid]['ResponseDeadLine'][:10] or 'n/a'}) https://sam.gov/opp/{nid}/view")
    for nid in now.keys() & before.keys():
        diffs = [f for f in FIELDS if now[nid][f] != before[nid][f]]
        if diffs:
            print(f"{sol}: notice {nid} edited: " +
                  "; ".join(f"{f} {before[nid][f]!r} -> {now[nid][f]!r}" for f in diffs))
    for nid in before.keys() - now.keys():
        print(f"{sol}: notice {nid} left the active list (archived, cancelled or removed)")
    if now.keys() == before.keys() and not any(now[n] != before[n] for n in now):
        print(f"{sol}: no change")

STATE.write_text(json.dumps({s: current.get(s, {}) for s in watch}, indent=1))

The first run prints tracking started for each number and saves the state. Later runs print lines like:

1240LT26Q0109: NEW notice cbe71bb9577648beb6b5961f0588c070 (Combined Synopsis/Solicitation, posted 2026-10-21, due 2026-11-17) https://sam.gov/opp/cbe71bb9577648beb6b5961f0588c070/view
1240LT26Q0109: notice c10e3f85ab83436ebc656c3048780c7f edited: ResponseDeadLine '2026-11-04T15:00:00-07:00' -> '2026-11-17T15:00:00-07:00'
W912DY26R0001: no change

Run it daily with cron, after about 04:00 UTC so the day's extract is ready: 15 5 * * * cd ~/sam-watch && python3 track.py.

Gotchas

Save the file, then parse it. Streaming csv straight off r.raw works if you break early. If you read to the end, it fails with ValueError: I/O operation on closed file. Writing to disk first avoids that, and it's only a few seconds.
Windows-1252, not UTF-8. Open the file with encoding="cp1252", or titles with curly quotes will break.
Normalise solicitation numbers. People write W912DY-26-R-0001, and the extract may say W912DY26R0001. The script strips everything but letters and digits on both sides.
Deadlines carry their own time zone, such as -07:00. Compare full strings, or parse them, rather than slicing the date.
It's next-day, not real-time. A change posted today is usually in tomorrow's extract. Check Last-Modified: if it hasn't moved since your last run, there's nothing new to see.
Attachments aren't in the extract. An amendment that only swaps a PDF is invisible unless SAM.gov also posts a notice or edits the fields. For anything you're actively bidding on, keep the SAM.gov "follow" button on as well.
"Left the active list" isn't "cancelled". The extract doesn't say why a notice went. Open it on SAM.gov to check.

The hosted version (disclosure: we built it)

If you'd rather not run a cron job, our Apify Actor, SAM.gov & Grants.gov Scraper, does the same thing with trackSolicitationNumbers (up to 500 numbers):

It returns a change row per new_notice, notice_updated or notice_removed, with a plain-English changeSummary and before and after values for each changed field.
Every tracked number appears in every run: unchanged numbers get a free unchanged row.
It costs $0.003 per change row plus $0.01 for a run that finds changes. Runs with no changes pay only the $0.00005 start fee.

{ "trackSolicitationNumbers": ["1240LT26Q0109", "W912DY26R0001"],
  "stateStoreName": "my-bids-watchlist" }

Not affiliated with GSA or SAM.gov. Always confirm deadlines on the notice itself.