DEV Community: Hay Equipos

How to search open jobs by keyword across thousands of Greenhouse, Lever and Ashby career sites

Hay Equipos — Sat, 24 Oct 2026 19:30:27 +0000

Most job data tools make you bring a list of companies first. That is fine when you already know who you are watching, but it does not answer the more common question: who is hiring a data engineer right now? Job boards like LinkedIn and Indeed answer it, but they are hard to collect from and full of reposts. The cleanest source is the company career site itself, and a large share of startups and mid sized companies run theirs on Greenhouse, Lever, Ashby, Workable, SmartRecruiters or Recruitee.

The Career Site Jobs Search actor, published by Hay Equipos on the Apify Store, searches those six systems by keyword. It holds an index of 19,883 public company job boards, picks the boards that have matching roles, reads them live, and returns each open job in one schema: title, company, location, remote flag, salary range when published, posted date, job link, apply link and the description as plain text. No login, no browser, no company list.

Where the companies come from

No public list of every company on these systems exists, so the index is built from the Common Crawl URL index, a free public archive of web addresses. Every address on boards.greenhouse.io, jobs.lever.co, jobs.ashbyhq.com, apply.workable.com, jobs.smartrecruiters.com and *.recruitee.com in recent crawls was turned into a board name, and each one was checked against that system's public API. Only boards that exist are kept, together with their job titles and locations at build time.

When you search, the actor scores every board by how many of its titles matched, then reads the best boards live, so the results are always current. It is a large sample, not every company in the world, and it leans toward startups and scaleups, because large enterprises mostly use Workday, iCIMS or Taleo, which this actor does not cover. You can add any board it does not know in Extra boards.

What you get back

One row per job (illustrative values):

{
  "source": "ashby",
  "company": "Example Labs",
  "title": "Senior Data Engineer",
  "department": "Engineering",
  "location": "New York, NY",
  "isRemote": true,
  "postedAt": "2026-10-02T15:04:00.000Z",
  "salaryMin": 180000,
  "salaryMax": 220000,
  "salaryCurrency": "USD",
  "jobUrl": "https://jobs.ashbyhq.com/example-labs/1f2e3d4c",
  "applyUrl": "https://jobs.ashbyhq.com/example-labs/1f2e3d4c/application",
  "matchedKeyword": "data engineer",
  "boardUrl": "https://jobs.ashbyhq.com/example-labs",
  "descriptionText": "About the role\n\nYou will build..."
}

A RUN_SUMMARY record in the key value store shows the index date and size, how many boards matched, how many were read and how many jobs came from each system.

How keyword matching works

A title matches when it contains your phrase at the start of a word, with any ending on the last word. data engineer finds "Senior Data Engineer" and "Data Engineering Lead" but not "Data Center Engineer". To require words anywhere in the title, join them with +, for example engineer + kubernetes. Excluded words work the same way, so senior and intern in Exclude titles with remove those roles.

Step by step in the Apify Console

Open the actor on the Apify Store and start it in the Console.
In Job title keywords, add one phrase per line, for example data engineer and analytics engineer.
Add Location contains any of (for example United States, London, Germany), Remote jobs only, or Posted within the last N days if you need them.
Keep Maximum jobs per company at 10 so one big employer does not fill the list, and set Maximum jobs in total.
Click Start and export as CSV, Excel, JSON or HTML.

Calling it from code

curl:

curl -X POST \
  "https://api.apify.com/v2/acts/pistachio_implementation~career-site-jobs-search/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keywords": ["data engineer"], "locations": ["United States"], "postedWithinDays": 14, "maxJobs": 100}'

Python with apify-client:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("pistachio_implementation/career-site-jobs-search").call(
    run_input={
        "keywords": ["product manager", "product owner"],
        "excludeKeywords": ["senior", "intern"],
        "locations": ["London"],
        "postedWithinDays": 30,
        "includeDescription": False,
        "maxJobs": 200,
    }
)

for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(job["company"], "|", job["title"], "|", job["location"], "|", job["jobUrl"])

To watch for new roles, schedule the actor daily with Posted within the last N days set to 1 or 2.

Pricing

Pay per event: $0.002 per job saved on the free tier, which is $2 per 1,000 jobs, falling to $1.25 per 1,000 on Apify's Gold tier. Jobs removed by your filters and searches with no results cost nothing. The optional full scan mode, which also reads boards that had no matching title at index time, adds $0.10 per 1,000 boards checked. You can set a maximum charge per run and the actor stops early when it is reached.

Limits and what it does not do

Six systems only. Workday, iCIMS, Taleo, SuccessFactors, LinkedIn and Indeed are not covered.
The index is a sample. Companies that never appeared in Common Crawl are missing unless you add them in Extra boards.
Location matching is plain text. United States will not match a job listed only as "Austin, TX"; add the cities or states you care about.
Salary appears only when the company publishes it, which is common on Ashby and Greenhouse and rare on Workable and SmartRecruiters.
It does not collect personal data about candidates or recruiters.

Please use the job data in line with each job board's terms.

Try it on the Apify Store: https://apify.com/pistachio_implementation/career-site-jobs-search

How to export all products and prices from a WooCommerce store

Hay Equipos — Mon, 05 Oct 2026 21:24:09 +0000

You want the product catalog of a WooCommerce store in a spreadsheet: a competitor whose prices you track, a supplier whose SKUs you need, or your own store before a migration. Clicking through category pages and copying prices does not scale, and you do not have admin access or API keys for a store you do not own.

Most WooCommerce stores already expose a public, read only Store API at /wp-json/wc/store/v1/products. It is the same API WooCommerce's own cart and checkout blocks use in the browser, and it has been on by default since WooCommerce 4. The WooCommerce Scraper: Store Products, Prices and Stock, published by Hay Equipos on the Apify Store, reads that API for any store address you give it and returns one clean row per product. No login, no API keys, no browser.

What you get back

One row per product. Illustrative example:

{
  "store": "example-store.com",
  "productId": 1042,
  "name": "Classic Cotton T Shirt",
  "slug": "classic-cotton-t-shirt",
  "url": "https://example-store.com/product/classic-cotton-t-shirt/",
  "type": "variable",
  "sku": "TS-CLASSIC",
  "price": 19.99,
  "regularPrice": 24.99,
  "salePrice": 19.99,
  "onSale": true,
  "currency": "USD",
  "minPrice": 19.99,
  "maxPrice": 22.99,
  "inStock": true,
  "stockStatus": "in-stock",
  "rating": 4.6,
  "reviewCount": 31,
  "categories": ["Clothing", "T Shirts"],
  "tags": ["cotton"],
  "brands": [],
  "imageUrl": "https://example-store.com/wp-content/uploads/tshirt.jpg",
  "attributes": [{ "name": "Size", "values": ["S", "M", "L", "XL"] }],
  "variationCount": 4,
  "shortDescription": "Soft everyday tee.",
  "scrapedAt": "2026-09-27T07:28:52.693Z"
}

name	sku	price	regularPrice	onSale	inStock	categories
Classic Cotton T Shirt	`TS-CLASSIC`	19.99	24.99	true	true	Clothing, T Shirts
Canvas Tote Bag	`BAG-TOTE`	15.00	15.00	false	false	Accessories

Prices are converted from the API's minor units to normal amounts (19.99, not 1999), and zero decimal currencies stay whole. For variable products, price is the lowest variant price and minPrice and maxPrice give the range. Rows also carry all image URLs, stock text, low stock count, backorder flag, weight and dimensions when the store provides them. A RUN_SUMMARY record in the run's key value store lists products found and saved per store.

Step by step in the Apify Console

Open the actor on the Apify Store (link at the end) and click Try for free.
In Store URLs, add store home pages, one per line (https://example-store.com or just example-store.com). A product or shop link from the store also works. The actor tries both URL styles the Store API can use, with and without www..
Set Maximum products per store (default 1,000) and Maximum products in total (default 20,000).
Optional filters: Keyword filter (uses the store's own product search), Only products on sale, Only products in stock.
Pick Sort by: store default, newest, price low to high or high to low, best selling, top rated, or name. "Newest" with a small maximum is a cheap way to watch for new products.
Turn on Include full description if you want the long description as plain text. The short description is always included.
Leave Respect robots.txt on. Click Start and export from the Output tab as CSV, Excel or JSON.

Calling it from code

The actor id is pistachio_implementation/woocommerce-store-products. Keep your token in APIFY_TOKEN.

curl:

curl -X POST \
  "https://api.apify.com/v2/acts/pistachio_implementation~woocommerce-store-products/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"storeUrls": ["https://porterandyork.com"], "maxProductsPerStore": 100}'

Python with apify-client, a daily price check:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("pistachio_implementation/woocommerce-store-products").call(
    run_input={
        "storeUrls": ["https://porterandyork.com", "https://www.nutribullet.com"],
        "maxProductsPerStore": 500,
        "onSaleOnly": True,
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if "error" in row:
        print("store skipped:", row["store"], row["error"])
        continue
    print(row["store"], row["sku"], row["name"], row["price"], row["currency"])

Store the results each day and compare price, onSale and inStock by productId to see what changed.

Pricing

Pay per event, with no subscription, no start fee and no platform usage charge on top:

$0.0008 per product saved ($0.80 per 1,000 products).
Store level error rows are free: not a WooCommerce store, API turned off, request blocked, or robots.txt refusal.

For example, a full export of a 2,000 product store costs $1.60.

Limits and what it does not do

WooCommerce only. It works on WooCommerce stores (version 4 or later) that leave the public Store API on, which is the default. A store built on another platform, or one that turned the API off, comes back as one free error row.
Stores behind strong bot protection may block traffic from cloud servers. The actor does not try to get around blocks.
Variations are counted, not listed. variationCount tells you how many variants a product has, and the price range covers them, but each variant is not its own row.
Prices are what the store shows a guest visitor from a US datacenter. Stores that change prices by country or customer group may show you different prices.
No store discovery. You need to supply the store addresses.
It makes one request per second per store, at 100 products per request, so very large catalogs take a few minutes.

This is an independent tool and is not affiliated with WooCommerce or Automattic. Please use the data you collect in line with each store's terms.

Try it on the Apify Store: https://apify.com/pistachio_implementation/woocommerce-store-products

How to export Gumroad products, prices and ratings to a spreadsheet

Hay Equipos — Mon, 05 Oct 2026 21:23:33 +0000

Before you launch a Notion template, a preset pack or an ebook on Gumroad, you want to know what already sells in that niche and at what price. The Discover page shows you products one card at a time, which is fine for browsing and slow for research. What you actually want is a spreadsheet: product, price, rating, number of ratings, creator.

This guide shows how to build that spreadsheet with a small Apify Actor published by Hay Equipos called Gumroad Scraper: Products, Prices, Ratings by Keyword or Tag. It reads the same public search results the Discover page loads. No login and no browser.

What the tool returns

You search by keyword, tag or category. For each product you get one row with:

product name, ID, permalink and link
price and currency (in the product's own currency), a free flag and a pay what you want flag
recurrence for memberships and subscriptions (monthly, quarterly, yearly)
product type
average rating and number of ratings
the creator's storefront name, link and verified flag
a thumbnail link and a short description snippet
the search that found it, its position, and how many matches Gumroad reports in total

A row looks like this (values are illustrative):

{
  "productId": "AbC123xYz==",
  "permalink": "plannerkit",
  "name": "Example Notion Planner Kit",
  "url": "https://examplecreator.gumroad.com/l/plannerkit",
  "price": 12,
  "currency": "USD",
  "isFree": false,
  "isPayWhatYouWant": false,
  "recurrence": null,
  "productType": "digital",
  "rating": 4.8,
  "ratingCount": 312,
  "sellerName": "Example Creator",
  "sellerProfileUrl": "https://examplecreator.gumroad.com/",
  "searchQuery": "notion template",
  "sort": "most_reviewed",
  "position": 7,
  "totalMatches": 18200
}

For pay what you want products, price is the minimum price. A RUN_SUMMARY record in the run's key value store lists each search with its match count and any problem.

What people use it for

Product research before a launch: see what already exists for Notion templates, Lightroom presets, Procreate brushes, icon packs, courses or ebooks, and how many ratings the top products have.
Pricing benchmarks: compare price points, free products and pay what you want use across a niche before you set your own price.
Trend spotting: the newest and hot and new sorts show what creators are shipping this week.
Partner discovery: find the storefronts behind the most reviewed products in a category.

Step by step in the Apify Console

Open the Actor from its Apify Store page and sign in to Apify Console.
In the Input tab, add Search keywords such as notion template or icon pack, and/or Tags such as procreate brushes.
Optionally add Categories such as design, education or software-development. If you combine categories with keywords or tags, each keyword or tag is searched inside each category.
Choose Sort by: featured, newest, hot and new, highest rated, most reviewed, or price in either direction.
Optionally set Minimum price, Maximum price and Minimum rating (1 to 4 stars and up).
Set Maximum results per search and Maximum products for the whole run. A product found by two searches is saved once.
Click Start, then export from the Output tab as CSV, JSON or Excel.

To track new products in a niche, sort by newest and schedule the Actor to run daily.

How to call it from code

With curl:

curl -X POST "https://api.apify.com/v2/acts/pistachio_implementation~gumroad-products-search/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"searchQueries": ["notion template"], "sort": "most_reviewed", "maxResultsPerSearch": 100}'

In Python, with the apify-client package:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("pistachio_implementation/gumroad-products-search").call(
    run_input={
        "tags": ["procreate brushes"],
        "sort": "highest_rated",
        "minPrice": 5,
        "maxResultsPerSearch": 200,
    }
)
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["name"], row["price"], row["currency"], row["rating"], row["ratingCount"])

Pricing

Pay per event: $0.0015 per product saved, which is $1.50 per 1,000 products. There is no start fee and no platform usage charge on top. Duplicates across searches and products skipped by your price filter are free. For example, 10 keywords with 100 products each cost $1.50. You can cap the spend of any run with the maximum charge setting in Apify.

Limits and what it does not do

About 10,000 results per search. Gumroad stops serving results around there, so split very broad searches into narrower keywords, tags or categories.
No sales counts, full descriptions or file lists. They are not part of the search results, so the Actor does not return them.
Prices are compared as listed. Gumroad's own price filter works on internal price tiers, so the Actor checks each listed price again. Prices in other currencies are compared without conversion.
Discover only. Products a creator keeps off Discover do not appear.
No buyer or customer data. Only public product listings and the creator's public storefront name and link.
It checks Gumroad's robots.txt before reading, and spaces requests one second apart.

The Actor is an independent tool and is not affiliated with Gumroad. Please use the data in line with Gumroad's terms.

Try it on the Apify Store: https://apify.com/pistachio_implementation/gumroad-products-search

How to export the Y Combinator company directory to a spreadsheet by batch

Hay Equipos — Sun, 04 Oct 2026 19:43:18 +0000

You want a list of Y Combinator startups in a spreadsheet: every company in the latest batch, every YC fintech company that is hiring, or the whole directory of roughly 6,300 companies for market research. The directory at ycombinator.com/companies is easy to browse but has no export button, and copying company cards by hand is slow and error prone.

The Y Combinator Company Directory: YC Startups by Batch, published by Hay Equipos on the Apify Store, reads the same public search that powers the YC directory page and returns one clean row per company. Founder names, bios, photos and emails are deliberately left out. This is company data, not people data. It is an independent tool and is not affiliated with Y Combinator.

What you get back

One row per company. Illustrative example:

{
  "name": "Example Labs",
  "slug": "example-labs",
  "ycUrl": "https://www.ycombinator.com/companies/example-labs",
  "website": "https://www.examplelabs.ai/",
  "domain": "examplelabs.ai",
  "oneLiner": "Accounting automation for small businesses",
  "description": "Example Labs connects to a company's bank and books...",
  "batch": "Summer 2025",
  "batchCode": "S25",
  "status": "Active",
  "stage": "Early",
  "industry": "Fintech",
  "subindustry": "Fintech -> Accounting",
  "industries": ["Fintech", "Accounting"],
  "tags": ["AI"],
  "regions": ["United States of America", "America / Canada"],
  "locations": "San Francisco, CA, USA",
  "teamSize": 6,
  "isHiring": true,
  "topCompany": false,
  "nonprofit": false,
  "launchedAt": "2025-07-20T16:00:00.000Z",
  "logoUrl": "https://bookface-images.s3.amazonaws.com/small_logos/....png"
}

name	batchCode	industry	teamSize	isHiring	domain
Example Labs	S25	Fintech	6	true	examplelabs.ai
Sample Robotics	S25	Industrials	12	false	samplerobotics.com

With Add company page details turned on, each row also gets yearFounded, city, country, company linkedinUrl, twitterUrl, facebookUrl, crunchbaseUrl, githubUrl, openJobs, up to 20 jobTitles, and jobsUrl. If a company page cannot be read, the row gets a detailsError instead.

Step by step in the Apify Console

Open the actor on the Apify Store (link at the end) and click Try for free.
In Batches, add batch codes such as W24, S25, X25 (Spring) or F25 (Fall), or full names like Winter 2024. Leave it empty for every batch.
Optional filters: Industries (as named in the directory, for example B2B, Fintech, Healthcare; case does not matter), Regions (for example United States of America, Europe, India, Remote), Company status (Active, Acquired, Inactive, Public), Only companies that are hiring, Only YC top companies, and minimum or maximum team size.
Use Search text for a free text search over name, one liner and description, for example AI agents.
Turn on Add company page details if you need social links, year founded and open jobs. It is slower and costs a little extra per company.
Set Maximum companies (default 100, up to 10,000), then click Start and export from the Output tab as CSV, Excel, JSON or HTML.

A filter value that does not exist in the directory is ignored with a warning in the log. If none of your batches, industries or regions exist, the run ends with no rows and a message rather than returning the wrong set.

Calling it from code

The actor id is pistachio_implementation/yc-company-directory. Keep your token in APIFY_TOKEN.

curl:

curl -X POST \
  "https://api.apify.com/v2/acts/pistachio_implementation~yc-company-directory/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"batches": ["S25"], "industries": ["B2B"], "maxItems": 50}'

Python with apify-client, hiring fintech startups from two batches with details:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("pistachio_implementation/yc-company-directory").call(
    run_input={
        "batches": ["W24", "S24"],
        "industries": ["Fintech"],
        "hiringOnly": True,
        "includeDetails": True,
        "maxItems": 200,
    }
)

for c in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(c["name"], c["batchCode"], c["domain"], c.get("openJobs"), c.get("linkedinUrl"))

Schedule the run each season with the new batch code to get each batch as soon as YC lists it. Every run reads the directory live.

Pricing

Pay per event, with no subscription and no platform usage charge on top:

Company saved: $0.0009 per company ($0.90 per 1,000).
Company page details added: $0.001 extra per company, charged only when enrichment is on and the company page was read.
Actor start: Apify's standard start event of $0.00005 per run.

For example, the whole directory without details is about $5.70, and 200 companies with details is about $0.38. You can set a maximum charge per run and the actor stops when it is reached.

Limits and what it does not do

No founder data. Founder names, bios, photos and emails are never collected.
The directory search returns at most 1,000 companies per query, so larger exports are split by batch automatically, and each company appears once.
Detail enrichment opens one YC page per company, about one per second, so 1,000 companies with details take roughly 17 minutes. Without details, thousands of companies take well under a minute.
Team size, status and hiring flags are what YC shows, and they can lag behind reality.
The actor depends on the search settings the directory page publishes to every visitor. It reads them fresh on each run, but if YC removes them, the run fails with a clear message.

Please use the data in line with Y Combinator's terms.

Try it on the Apify Store: https://apify.com/pistachio_implementation/yc-company-directory

How to get all archived URLs and snapshots of a website from the Wayback Machine

Hay Equipos — Sun, 04 Oct 2026 19:37:32 +0000

You need to know what a website used to look like: every page a domain ever had before a migration, the version of a pricing page from last spring, or what an expired domain hosted before you buy it. The Wayback Machine has the answer, but its calendar view shows one page at a time and gives you nothing you can sort or filter. You want a table.

The Wayback Machine Snapshots and Archived URLs actor, published by Hay Equipos on the Apify Store, reads the Internet Archive's own public CDX and availability APIs and returns clean rows. It has three modes:

Snapshots: the capture history of one exact page, optionally thinned to one capture per hour, day, month or year, or to captures where the content changed.
Archived URLs: every distinct URL the archive holds under a domain or path, with its first capture.
Closest: the one archived copy nearest to a date.

What you get back

One row per capture or archived URL. Illustrative example:

{
  "input": "example.com",
  "mode": "snapshots",
  "url": "https://www.example.com/",
  "capturedAt": "2019-03-04T11:22:33Z",
  "timestamp": "20190304112233",
  "statusCode": 200,
  "mimeType": "text/html",
  "digest": "ABCDEFGHIJKLMNOPQRSTUVWXYZ234567",
  "lengthBytes": 8421,
  "archiveUrl": "https://web.archive.org/web/20190304112233/https://www.example.com/",
  "rawArchiveUrl": "https://web.archive.org/web/20190304112233id_/https://www.example.com/",
  "scrapedAt": "2026-09-27T07:21:05.981Z"
}

capturedAt	url	statusCode	mimeType	digest
`2017-01-12T08:00:00Z`	https://www.example.com/	200	text/html	7KQ2...
`2018-02-03T09:30:00Z`	https://www.example.com/	200	text/html	M4XZ...
`2019-03-04T11:22:33Z`	https://www.example.com/	200	text/html	ABCD...

Every row has two links: archiveUrl (the normal Wayback page) and rawArchiveUrl (the archived copy without the Wayback toolbar, which is what you want to feed a parser or an AI model). digest is the Archive's fingerprint of the content, so two captures with the same digest are identical. lengthBytes is the compressed size the Archive stores. In Archived URLs mode each row also has firstCapturedAt. In Closest mode each row also has requestedDate.

Step by step in the Apify Console

Open the actor on the Apify Store (link at the end) and click Try for free.
In URLs or domains, add one per line: a page (example.com/about), a domain (example.com) or a path prefix (example.com/blog/). The scheme and www are optional, and pasted Wayback links work too.
Choose What to get: Snapshots, Archived URLs, or Closest.
Optional: set From date and To date as 2019, 2019-06 or 2019-06-30. A year or month covers the whole period.
For Snapshots, use Keep one snapshot per to thin the history (hour, day, month, year, or content change).
Turn on Only successful captures to drop redirects and errors, and use Content type filter (text/html, application/pdf, or a pattern like image/.*) to narrow results.
For Archived URLs, turn on Include subdomains to also list blog.example.com and similar. For Closest, set Closest to date, or leave it empty for the most recent capture.
Set Maximum rows per URL or domain (default 500) and Maximum rows in total (default 5,000), then click Start and export from the Output tab.

Example: every archived HTML page under a docs folder.

{
  "urls": ["crawlee.dev/docs/"],
  "mode": "urls",
  "mimeType": "text/html",
  "onlySuccessful": true,
  "maxRowsPerInput": 2000
}

Calling it from code

The actor id is pistachio_implementation/wayback-machine-snapshots. Keep your token in APIFY_TOKEN.

curl, one capture per year for a home page:

curl -X POST \
  "https://api.apify.com/v2/acts/pistachio_implementation~wayback-machine-snapshots/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["apify.com"], "mode": "snapshots", "from": "2015", "to": "2025", "collapse": "year", "onlySuccessful": true}'

Python with apify-client, finding the copy of a page closest to a date:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("pistachio_implementation/wayback-machine-snapshots").call(
    run_input={
        "urls": ["example.com/pricing", "example.com/terms"],
        "mode": "closest",
        "closestTo": "2020-01-15",
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["input"], row["capturedAt"], row["rawArchiveUrl"])

summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("RUN_SUMMARY")
print(summary["value"]["errors"])  # inputs with no captures

Pricing

Pay per event, with no start fee, no subscription and no platform usage charged on top:

$0.0005 per archive row saved ($0.50 per 1,000 rows).
Inputs with no captures, invalid inputs and Archive errors are free. They are listed in RUN_SUMMARY in the run's key value store.

For example, the yearly history of 100 home pages over 10 years is about 1,000 rows, so about $0.50. You can set a maximum charge per run and the actor stops cleanly when it is reached.

Limits and what it does not do

Capture records and links only, not page content. Open or fetch rawArchiveUrl to get the page as it was captured.
The Archive's API can be slow, often 10 to 30 seconds per request on a large domain, and sometimes answers that it is busy. The actor makes one request at a time, at least 1.2 seconds apart, and backs off and retries. Very large domains take minutes.
Rows come oldest first. If you only want recent captures, set the From date.
Sites that asked the Archive to exclude them return no rows.
It only knows what the Wayback Machine captured. Pages the Archive never crawled will not appear.

Try it on the Apify Store: https://apify.com/pistachio_implementation/wayback-machine-snapshots

How to scrape Shopify App Store apps, pricing plans and reviews

Hay Equipos — Sun, 04 Oct 2026 19:36:56 +0000

If you build Shopify apps, or you are researching a category before you build one, you keep coming back to the same questions. What do competitors charge on each plan? How fast are their reviews coming in? What do one star reviews complain about? The App Store shows all of this, but one page at a time, with review counts rounded to "47K".

Shopify App Store Scraper by Hay Equipos reads the public pages of apps.shopify.com and returns clean rows: app details with every pricing plan, and optionally the merchant reviews. No login, no browser and no proxies.

What you get back

Each row has a type: app, review or error.

An app row, trimmed, with illustrative values:

{
  "type": "app",
  "handle": "example-reviews-app",
  "url": "https://apps.shopify.com/example-reviews-app",
  "name": "Example Product Reviews",
  "tagline": "Collect photo reviews and show star ratings",
  "developer": "Example Apps Inc",
  "rating": 4.8,
  "reviewCount": 2140,
  "ratingHistogram": { "1": 41, "2": 12, "3": 25, "4": 160, "5": 1902 },
  "pricingSummary": "Free plan available. Free trial available.",
  "hasFreePlan": true,
  "hasFreeTrial": true,
  "plans": [
    { "name": "Free", "priceText": "Free", "price": 0, "billingPeriod": null, "freeTrialDays": null, "features": ["Up to 50 review requests"] },
    { "name": "Growth", "priceText": "$15/month", "price": 15, "billingPeriod": "month", "freeTrialDays": 14, "features": ["Photo reviews", "Review widgets"] }
  ],
  "builtForShopify": true,
  "categories": [{ "name": "Product reviews", "slug": "marketing-and-conversion-social-trust-product-reviews" }],
  "languages": ["English", "French"],
  "worksWith": ["Shopify Flow"],
  "launchedDate": "2019-04-02"
}

App rows also carry description, developerUrl, features (feature tags), iconUrl and, for apps found through a category, category and categoryPosition. The star counts in ratingHistogram are exact: they come from the review filter links on the listing, not the rounded figure shown on screen.

A review row, with illustrative values:

{
  "type": "review",
  "handle": "example-reviews-app",
  "reviewId": "1234567",
  "rating": 2,
  "date": "2026-09-12",
  "text": "Setup was easy but the widget slowed down our product pages...",
  "storeName": "Example Outdoor Co",
  "storeCountry": "Canada",
  "timeUsingApp": "About 2 months",
  "developerReply": "Thanks for the feedback. We have released a lighter widget...",
  "developerReplyDate": "2026-09-13"
}

Apps that do not exist come back as an error row, which is free. A RUN_SUMMARY record in the run's key value store lists any input that returned nothing.

Step by step in the Apify Console

Open the actor on the Apify Store (link at the end) and click Try for free.
In Apps, add app handles (the part after apps.shopify.com/) or full app links, one per line.
In Categories, add category links or slugs to map a whole category. Set Maximum apps per category (default 48). Apps come in the order the category lists them, 24 per category page.
Keep Open each app page on for full details. Turn it off for a fast run that returns only the card data from the category page (name, rating, review count, pricing label, tagline).
To collect reviews, turn on Include reviews and set Maximum reviews per app (default 50, newest first).
Optionally filter with Only reviews with these star ratings (for example 1 and 2 for complaints) and Only reviews newer than (a date like 2026-09-01).
Set Maximum rows (default 5,000, apps plus reviews), click Start and export the table.

Calling it from code

With curl:

curl -X POST \
  "https://api.apify.com/v2/acts/pistachio_implementation~shopify-app-store-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "apps": ["example-reviews-app"],
    "includeReviews": true,
    "maxReviewsPerApp": 20,
    "reviewRatings": ["1", "2"]
  }'

With Python and the apify-client package:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("pistachio_implementation/shopify-app-store-scraper").call(
    run_input={
        "categories": [
            "https://apps.shopify.com/categories/marketing-and-conversion-social-trust-product-reviews"
        ],
        "maxAppsPerCategory": 24,
        "includeReviews": False,
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["type"] == "app":
        prices = [p["price"] for p in row.get("plans") or [] if p.get("price") is not None]
        print(row.get("categoryPosition"), row["name"], row["rating"], row["reviewCount"], prices)

Pricing

Pay per event, with no subscription and no platform usage charge on top:

Event	Price
App saved	$0.0015 per app ($1.50 per 1,000 apps)
Review saved	$0.0003 per review ($0.30 per 1,000 reviews)

There is no start fee, and error rows are free. Example: a weekly check of 50 competitor apps with their 20 newest reviews each is 50 app rows and 1,000 review rows, which costs 50 × $0.0015 + 1,000 × $0.0003 = $0.375.

For review monitoring on a schedule, set "Only reviews newer than" to your last run date. Older reviews are not collected and not charged. Note that each app you list is still saved, and charged, as an app row on every run.

Limits and what it does not do

No keyword search. The App Store's robots.txt asks automated tools not to read search pages, so the actor uses app links and categories instead.
Reviews load 10 per page, at about one page every 1.5 seconds to stay polite, so long review histories take time.
Category order is the App Store's own at the time of the run and can include promoted apps.
Prices are what the listing shows, usually in US dollars. Usage based or extra charges appear only as the listing words them.
No personal contact details. The reviewing store's name and country are only what the App Store publishes on each review.

The actor is an independent tool and is not affiliated with Shopify. Use the data in line with the Shopify App Store's terms.

Try it on the Apify Store: https://apify.com/pistachio_implementation/shopify-app-store-scraper

How to convert any RSS or Atom feed to JSON

Hay Equipos — Sun, 04 Oct 2026 19:31:10 +0000

You want the latest posts from a set of blogs, newsrooms or changelogs as clean JSON, for an AI agent, a RAG pipeline, a Slack alert or a newsletter digest. The catch is that feeds come in four formats (RSS 2.0, RSS 1.0, Atom and JSON Feed), each with its own field names, date styles and character sets. And half the time you do not even know the feed address.

RSS and Atom Feed to JSON by Hay Equipos takes a feed URL, or just a website address, and returns every item in one schema, whatever format the site uses.

What you get back

One row per feed item. Example with illustrative values:

{
  "input": "https://blog.example.com",
  "feedUrl": "https://blog.example.com/rss/",
  "feedTitle": "The Example Blog",
  "feedLink": "https://blog.example.com",
  "feedFormat": "rss2",
  "title": "Shipping our new search API",
  "url": "https://blog.example.com/new-search-api/",
  "id": "post-4821",
  "authors": ["Example Author"],
  "publishedAt": "2026-09-25T14:00:00.000Z",
  "updatedAt": null,
  "categories": ["API", "Product"],
  "summary": "Today we are releasing ...",
  "hasFullContent": true,
  "wordCount": 1180,
  "imageUrl": "https://blog.example.com/images/search-api.png",
  "enclosures": [],
  "text": "Today we are releasing ...",
  "scrapedAt": "2026-09-27T07:18:37.239Z"
}

A few details that save you work:

feedFormat is rss2, rss1, atom or jsonfeed, but every other field is the same for all four.
Dates are ISO 8601 in UTC. HTML entities and character sets such as Latin 1 and Windows 1252 are decoded for you.
hasFullContent tells you whether the publisher put the whole article in the feed or only a summary.
Podcast and video attachments come back in enclosures with type and size.
You can also ask for the publisher's original html.

Feed auto discovery

Paste a home page instead of a feed address and the actor first looks for the feed the page declares in its own header. If there is none, it tries common addresses such as /feed, /rss, /rss.xml, /atom.xml, /index.xml and /feed.json, at most 8 tries per input. Most blog platforms and news sites publish feeds this finds.

The actor respects robots.txt, does not log in, does not use a browser or proxies, and sends at most one request per second to each site.

Step by step in the Apify Console

Open the actor on the Apify Store (link at the end) and click Try for free.
In Feed or website URLs, add feed addresses or ordinary website addresses, one per line.
Leave Find the feed when a website URL is given on, unless you only want to accept feed URLs.
Set Maximum items per feed (default 50, in the publisher's order, which is newest first on almost every site) and Maximum items in total (default 1,000).
For scheduled runs, set Only items published after to a date like 2026-09-01. Older items are skipped and not charged.
Keep Include item text on for plain text bodies, adjust Maximum text length (default 20,000 characters, 0 for no limit) and turn on Include item HTML if you want the raw markup.
Click Start and export the items, or connect the dataset to an integration.

Calling it from code

With curl, using the synchronous endpoint that returns items directly:

curl -X POST \
  "https://api.apify.com/v2/acts/pistachio_implementation~rss-atom-feed-to-json/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://blog.example.com", "https://news.example.org/feed/"],
    "maxItemsPerFeed": 20,
    "publishedAfter": "2026-09-01"
  }'

With Python and the apify-client package:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("pistachio_implementation/rss-atom-feed-to-json").call(
    run_input={
        "urls": [
            "https://blog.example.com",
            "https://news.example.org/feed/",
        ],
        "maxItemsPerFeed": 20,
        "publishedAfter": "2026-09-01",
        "includeContent": True,
        "maxContentChars": 5000,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["publishedAt"], item["feedTitle"], item["title"], item["url"])

summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("RUN_SUMMARY")
for problem in summary["value"]["errors"]:
    print("no items:", problem["input"], problem["error"])

Inputs that give nothing (no feed found, the site blocked the request, robots.txt disallows it) are listed in RUN_SUMMARY and cost nothing.

Pricing

Pay per event, with no subscription and no platform usage charge on top:

Event	Price
Feed item saved	$0.0005 per item ($0.50 per 1,000 items)

There is no start fee. Failed feeds, feeds not found, robots.txt refusals, items older than your date filter and duplicates are free. An agent that checks ten blogs for today's posts pays only for the few new items it gets back. You can set a maximum charge per run and the actor stops cleanly when it is reached.

Limits and what it does not do

A feed carries only what the publisher puts in it, usually the latest 10 to 50 items. The actor reads feeds and does not crawl a site's archive. Run it on a schedule to build a history.
It never visits the article pages. If a site publishes only summaries, text is the summary and hasFullContent is false.
Sites that block cloud servers, or whose robots.txt disallows automated reading of the feed, return a free error entry instead of items.
Feeds larger than 10 MB are skipped.
Discovery stops after 8 candidate addresses per input.

Feeds are published for software to read, but the content still belongs to the publisher. Use it in line with each source site's terms, including copyright in the full text.

Try it on the Apify Store: https://apify.com/pistachio_implementation/rss-atom-feed-to-json

How to export GitHub repository search results with stars, releases and README to a spreadsheet

Hay Equipos — Sun, 04 Oct 2026 19:30:34 +0000

GitHub search is great for browsing and terrible for comparing. If you want the top 100 Python repositories for "vector database" with their stars, license, last push date and latest release in one table, you end up opening tabs and copying numbers by hand. The same goes for keeping an eye on the releases of a list of dependencies.

This guide shows how to get that table in one run with a small Apify Actor published by Hay Equipos called GitHub Repository Search, Stats and Releases. It uses GitHub's official REST API, works without a key, and returns one clean row per repository.

What the tool returns

You can give it search queries, a list of repositories, or both. Search queries accept any GitHub qualifier you would type on github.com, such as topic:rag, org:vercel, license:mit or in:readme.

Each repository row includes the full name, owner, link, description, homepage, main language, topics, stars, forks, open issues, license, fork and archive flags, default branch, and the creation and last push dates. Optionally, it adds:

the latest stable release (tag, name, date and link)
recent releases as separate rows, with release notes, asset count and total download count
the README in Markdown, cut to a length you choose, ready for an LLM or a search index

A repository row looks like this (values are illustrative):

{
  "type": "repository",
  "source": "search",
  "query": "vector database",
  "rank": 1,
  "fullName": "example-org/example-vectordb",
  "url": "https://github.com/example-org/example-vectordb",
  "language": "Python",
  "topics": ["vector-database", "embeddings", "rag"],
  "stars": 18450,
  "forks": 1320,
  "openIssues": 214,
  "license": "Apache-2.0",
  "isArchived": false,
  "createdAt": "2022-05-03T09:12:44Z",
  "pushedAt": "2026-09-29T17:40:02Z",
  "latestRelease": { "tag": "v2.4.0", "publishedAt": "2026-09-20T12:00:00Z" },
  "readme": "# Example VectorDB ..."
}

Searches with no matches and repositories that do not exist are listed in a RUN_SUMMARY record in the run's key value store and cost nothing.

Step by step in the Apify Console

Open the Actor from its Apify Store page and sign in to Apify Console.
In the Input tab, add one or more Search queries, for example vector database, or list Repositories to look up as owner/name or github.com links.
Pick Sort search results by (best match, most stars, most forks or recently updated) and add filters if you want them: Language, Minimum stars, Created on or after and Last code push on or after (dates like 2026-01-01).
Set Maximum repositories per query (up to 1,000, GitHub's own cap for one search).
Switch on Add latest release and Add README text if you need them, and set Release rows per repository above 0 for release notes.
Click Start, then export from the Output tab as CSV, JSON or Excel.

To find new and rising projects, set Created on or after to a recent date and sort by stars.

How to call it from code

With curl:

curl -X POST "https://api.apify.com/v2/acts/pistachio_implementation~github-repo-search-stats/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"searchQueries": ["vector database"], "language": "Python", "minStars": 1000, "sort": "stars", "maxReposPerQuery": 50}'

In Python, with the apify-client package:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("pistachio_implementation/github-repo-search-stats").call(
    run_input={
        "repositories": ["apify/crawlee"],
        "includeLatestRelease": True,
        "includeReadme": True,
    }
)
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["type"], row["fullName"], row.get("stars"), row.get("latestRelease"))

Pricing

Pay per event, with no start fee and no platform usage charge on top:

Repository saved: $0.001, which is $1 per 1,000 repositories. The latest release and the README are included in that price.
Release saved: $0.0003, which is $0.30 per 1,000 release rows.

Searches with no matches, repositories not found, and releases not fetched because of GitHub's rate limit are free. As an example, the top 100 Python repositories for a topic with READMEs cost $0.10. You can cap the spend of any run with the maximum charge setting in Apify.

Limits and what it does not do

GitHub's own limits apply. Without a token, GitHub allows 10 searches a minute and 60 other calls an hour from one address. Plain searches need no extra calls, so they work fine without a token. The latest release and release rows use one extra call per repository, so without a token they cover about 50 repositories an hour. Once that allowance runs out, the remaining repositories are still saved without release data, and RUN_SUMMARY shows coreLimitHit.
Add a token for big runs. A free personal access token with no scopes raises the limits to 30 searches a minute and 5,000 calls an hour. It is a secret input and is never written to the output.
One search returns at most 1,000 repositories. Split large searches by language, date or star range.
Public repositories only. No user profiles, emails or contributor lists are collected.
openIssues is GitHub's own count and includes open pull requests. watchers is filled only for repositories looked up by name.

Please use the data in line with GitHub's terms of service and API terms.

Try it on the Apify Store: https://apify.com/pistachio_implementation/github-repo-search-stats

How to check HTTP status codes and redirect chains for a list of URLs in bulk

Hay Equipos — Sat, 03 Oct 2026 19:43:10 +0000

You just migrated a website, or inherited a spreadsheet of 5,000 backlinks, and you need to know which URLs still work, which ones redirect, and where the redirects end up. Opening them one by one in a browser hides the redirect chain, and most single URL checkers online stop at a handful of links. You want one row per URL with the status code, the full chain of hops and a clear "broken or not" flag.

The Bulk URL Status Code and Redirect Chain Checker, published by Hay Equipos on the Apify Store, does exactly that. It is useful for SEO migrations, broken link audits, backlink and affiliate link checks, and cleaning a URL list before you hand it to another tool.

What you get back

One row per URL. Example (illustrative values):

{
  "url": "http://example.com/old-page",
  "ok": true,
  "isBroken": false,
  "statusCode": 301,
  "finalStatusCode": 200,
  "finalUrl": "https://www.example.com/new-page",
  "redirectCount": 2,
  "redirectChain": [
    { "url": "http://example.com/old-page", "statusCode": 301, "method": "HEAD", "location": "https://example.com/old-page" },
    { "url": "https://example.com/old-page", "statusCode": 301, "method": "HEAD", "location": "https://www.example.com/new-page" },
    { "url": "https://www.example.com/new-page", "statusCode": 200, "method": "HEAD", "location": null }
  ],
  "redirectsToHttps": true,
  "changedDomain": true,
  "redirectLoop": false,
  "tooManyRedirects": false,
  "contentType": "text/html; charset=utf-8",
  "server": "nginx",
  "xRobotsTag": null,
  "hsts": true,
  "responseTimeMs": 840
}

As a table:

url	statusCode	finalStatusCode	isBroken	redirectCount	finalUrl
http://example.com/old-page	301	200	false	2	https://www.example.com/new-page
https://example.com/missing	404	404	true	0	https://example.com/missing
https://no-such-host.example/x	null	null	true	0	(error: ENOTFOUND)

isBroken is true for a final 4xx or 5xx, a redirect loop, too many hops, or no answer at all. ok is true only for a final 2xx. Each row also carries contentLength, lastModified and cacheControl from the final answer. URLs that get no HTTP answer carry an error such as ENOTFOUND, ECONNREFUSED or Timed out.

Step by step in the Apify Console

Open the actor on the Apify Store (link at the end) and click Try for free.
Add your addresses to URLs, one per line, or paste a whole column into Or paste a list (spaces, commas or new lines all work). Duplicates are removed and addresses without a scheme get https:// added.
Leave Follow redirects on to record every hop, and set Maximum redirect hops (default 10, up to 20).
Keep Retry with GET when HEAD is refused on. Some servers reject the lightweight HEAD request but serve the page normally, and the GET retry reads at most 64 KB.
Adjust Requests per second per site (default 3) and Timeout per request (default 20 seconds) if needed.
Leave Respect robots.txt on unless you are checking a site you own or have permission to check.
Click Start and export the results from the Output tab as CSV, Excel or JSON.

Calling it from code

The actor id is pistachio_implementation/url-status-redirect-checker. Keep your token in APIFY_TOKEN.

curl:

curl -X POST \
  "https://api.apify.com/v2/acts/pistachio_implementation~url-status-redirect-checker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["http://apify.com", "https://httpbin.org/status/404"], "followRedirects": true}'

Python with apify-client, checking a migration redirect map:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

old_urls = [
    "https://example.com/blog/post-1",
    "https://example.com/blog/post-2",
]

run = client.actor("pistachio_implementation/url-status-redirect-checker").call(
    run_input={"urls": old_urls, "followRedirects": True, "maxRedirects": 10}
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row.get("isBroken") or row.get("redirectCount", 0) > 1:
        print(row["url"], row.get("finalStatusCode"), row.get("finalUrl"), row.get("error"))

After a migration, a healthy old URL usually shows finalStatusCode 200, redirectCount 1, and the finalUrl you expect.

Pricing

Pay per event: $0.0008 per URL checked ($0.80 per 1,000). A URL is charged when a server answered it with any HTTP status, including 404 and 500, because that answer is the result you asked for. No start fee and no platform usage on top. These rows are free:

invalid URLs
URLs with no HTTP answer (the domain does not exist, the connection is refused, or it times out)
URLs skipped because robots.txt closes them to automated tools

Limits and what it does not do

No JavaScript. Redirects done by JavaScript or by a meta refresh tag are not followed. The page's own status code is reported.
It does not crawl. It checks exactly the URLs you give it and does not discover links on a page. To get every URL of a site, extract its sitemap first and pass that list in.
Some sites answer automated requests from cloud servers with 403 or a challenge page. That status is reported as received. The actor does not try to get around such blocks.
It identifies itself as ApifyLinkChecker and respects robots.txt by default, including rules written for Apify crawlers.
Up to 50,000 URLs per run. Many hosts are checked in parallel (25 at a time), but requests to the same host are spaced out to your per site limit, so a long list on one domain runs slower than a list spread across many.
429 and 5xx answers are retried with backoff before they are reported, and a host that times out twice is treated as not answering for the rest of the run.

Try it on the Apify Store: https://apify.com/pistachio_implementation/url-status-redirect-checker

How to get all page URLs of a website from its sitemap

Hay Equipos — Sat, 03 Oct 2026 19:37:24 +0000

You need a list of every page on a website. Maybe you are auditing your own sitemap, listing a competitor's blog posts with dates, planning a migration, or feeding a crawler or a RAG pipeline. Crawling the whole site link by link is slow and expensive. The site already publishes the list in its sitemap, but large sites split it across dozens of files, nested indexes and gzip archives.

Sitemap URL Extractor by Hay Equipos finds those sitemaps, follows every index, opens the gzip files and returns one clean row per URL.

How it finds the sitemaps

Give it a domain and it reads every Sitemap: line in robots.txt first. If there are none, it tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap.txt. You can also give sitemap links directly: .xml, .xml.gz, .txt, sitemap indexes, robots.txt, or an RSS or Atom feed. Sitemap indexes are followed breadth first.

The pages themselves are never crawled, which is why listing is fast and cheap.

What you get back

One row per unique URL. Example with illustrative values:

{
  "url": "https://www.example.com/blog/how-we-cut-load-time",
  "lastmod": "2026-08-14T09:30:00.000Z",
  "changefreq": "monthly",
  "priority": 0.7,
  "alternates": [
    { "hreflang": "en-us", "url": "https://www.example.com/blog/how-we-cut-load-time" },
    { "hreflang": "de-de", "url": "https://www.example.com/de/blog/how-we-cut-load-time" }
  ],
  "imageCount": 2,
  "videoCount": 0,
  "newsTitle": null,
  "newsPublishedAt": null,
  "sitemapUrl": "https://www.example.com/sitemap-posts.xml",
  "site": "https://www.example.com",
  "statusCode": 301,
  "finalUrl": "https://www.example.com/blog/how-we-cut-load-time/",
  "redirected": true,
  "scrapedAt": "2026-09-27T06:14:24.933Z"
}

statusCode, finalUrl and redirected appear only when the status check is on. A RUN_SUMMARY record in the run's key value store shows, for each site, how the sitemap was found, how many sitemap files were read, any sitemap errors and how many URLs were saved.

Step by step in the Apify Console

Open the actor on the Apify Store (link at the end) and click Try for free.
In Websites or sitemap URLs, add domains or sitemap links, one per line.
Filter if you like: Keep URLs matching any of and Drop URLs matching any of accept plain words or regular expressions (for example /blog/ to keep, /tag/ to drop). Changed on or after keeps only URLs with a lastmod on or after a date. Same host only drops URLs on other hosts.
Turn on Check HTTP status of each URL to find pages that are broken or redirected but still listed in the sitemap.
Set the caps: Maximum sitemap files per site (default 500), Maximum URLs per site (default 50,000) and Maximum URLs in total (default 200,000).
Click Start and export as CSV, Excel or JSON. With the status check on, filter for statusCode 404 or redirected true.

Calling it from code

With curl:

curl -X POST \
  "https://api.apify.com/v2/acts/pistachio_implementation~sitemap-url-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": ["https://www.example.com"],
    "includeUrlPatterns": ["/blog/"],
    "lastmodAfter": "2026-01-01",
    "maxUrlsPerSite": 5000
  }'

With Python and the apify-client package, checking a sitemap for broken pages:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("pistachio_implementation/sitemap-url-extractor").call(
    run_input={
        "startUrls": ["https://www.example.com"],
        "sameHostOnly": True,
        "checkStatus": True,
        "maxUrlsPerSite": 2000,
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    status = row.get("statusCode")
    if status is None or status >= 400 or row.get("redirected"):
        print(status, row["url"], "->", row.get("finalUrl"))

summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("RUN_SUMMARY")
for site in summary["value"]["sites"]:
    print(site["input"], site["discoveredVia"], site["sitemapsRead"], site["urlsSaved"])

Pricing

Pay per event, with no subscription and no platform usage charge on top:

Event	Price
URL saved	$0.00025 per URL ($0.25 per 1,000 URLs)
URL status checked	$0.001 per URL ($1.00 per 1,000 checks), only with the status check on

There is no start fee. Filtered URLs, duplicates and sites without a sitemap cost nothing. For example, listing 20,000 URLs costs $5.00, and adding the status check to those 20,000 adds $20.00. You can set a maximum charge per run and the actor stops when it is reached.

Limits and what it does not do

It lists what the sitemaps say. Pages missing from the sitemap are not found, because pages are not crawled.
Sites that block automated requests to their sitemap return an error in RUN_SUMMARY.
lastmod is whatever the site writes. Some sites set every URL to today's date.
With a date filter set, URLs without a lastmod are dropped, and sitemap files whose own lastmod is older than the date are skipped.
The status check is much slower than listing. It sends a HEAD request (a GET if HEAD is refused) at about three requests a second per site.

Sitemaps exist so that machines can read them, but the content they point to belongs to the site owner. Use the data in line with each source site's terms.

Try it on the Apify Store: https://apify.com/pistachio_implementation/sitemap-url-extractor

How to extract text from PDF URLs to Markdown for an LLM or RAG pipeline

Hay Equipos — Sat, 03 Oct 2026 19:36:48 +0000

You have a list of PDF links (reports, research papers, filings, manuals, price lists) and you need their text in a form an LLM, a search index or a spreadsheet can use. Copying from a PDF viewer scrambles two column layouts and breaks words across lines, and writing your own extraction script means dealing with downloads, redirects, files that are secretly HTML and documents with no text layer at all.

This guide shows how to turn a list of PDF URLs into clean text and Markdown with a small Apify Actor published by Hay Equipos called PDF Text Extractor: PDF URL to Text and Markdown per Page. It downloads each file and reads its text layer with pdf.js, the PDF engine used in Firefox.

What the tool returns

For each PDF you get either one row with the whole document or one row per page. Text comes out in reading order: two column layouts are read column by column, words split across lines are rejoined, larger fonts become Markdown headings, and bullet characters become lists. Each row also carries the document's metadata.

A document mode row looks like this (values are illustrative, text shortened):

{
  "url": "https://www.example.org/reports/annual-report-2025.pdf",
  "finalUrl": "https://www.example.org/reports/annual-report-2025.pdf",
  "success": true,
  "fileName": "annual-report-2025.pdf",
  "fileSizeBytes": 2483112,
  "pageCount": 48,
  "pagesExtracted": 48,
  "title": "Annual Report 2025",
  "author": "Example Organization",
  "createdAt": "2026-03-02T09:15:00.000Z",
  "pdfVersion": "1.7",
  "likelyScanned": false,
  "wordCount": 21450,
  "charCount": 131870,
  "textTruncated": false,
  "text": "Annual Report 2025\n\nLetter from the board...",
  "markdown": "## Annual Report 2025\n\nLetter from the board..."
}

Page mode adds pageNumber and gives the text, Markdown and counts for that page only, while the document fields repeat on every row so each page stands alone. That makes it easy to chunk long documents and keep page numbers for citations.

Files that fail come back with success: false and an error, such as "The server answered HTTP 404", "The URL returned a web page, not a PDF" or "The PDF is password protected".

Step by step in the Apify Console

Open the Actor from its Apify Store page and sign in to Apify Console.
In the Input tab, add direct links under PDF URLs, one per line.
Choose One row per: Document (whole text in one row) or Page.
Keep Include plain text and Include Markdown on, or turn one off.
Set Maximum pages per PDF (300 by default) and Maximum file size (50 MB by default). Larger files are skipped for free.
Optionally set Maximum text length to cut text per row (0 means no limit).
Click Start, then export from the Output tab as JSON, CSV or Excel.

If your PDFs are on your own computer, upload them to an Apify key value store, or anywhere with a public link, and pass those links.

How to call it from code

With curl:

curl -X POST "https://api.apify.com/v2/acts/pistachio_implementation~pdf-text-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://arxiv.org/pdf/1706.03762"], "outputMode": "pages", "maxPagesPerPdf": 50}'

In Python, with the apify-client package, writing one Markdown file per page:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("pistachio_implementation/pdf-text-extractor").call(
    run_input={"urls": ["https://arxiv.org/pdf/1706.03762"], "outputMode": "pages"}
)
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row.get("success"):
        with open(f"page_{row['pageNumber']:03}.md", "w", encoding="utf-8") as f:
            f.write(row["markdown"])

Pricing

Pay per event: $0.002 per PDF processed, which is $2 per 1,000 PDFs, the same in document and page mode. Apify's standard Actor start event is also set, at $0.00005 per run. There is no charge for platform usage on top. Downloads that fail, files that are not PDFs, password protected files, robots.txt skips and files over your size limit are free. You can set a maximum charge per run in Apify, and the Actor stops cleanly when it is reached.

Limits and what it does not do

No OCR. Scanned PDFs without a text layer return little or no text and are flagged with likelyScanned: true (under about 40 characters of text per page). Those need an OCR tool.
Tables come out as text lines in reading order, not as structured cells.
Complex layouts with three or more columns, or text boxes scattered over the page, may read in an imperfect order.
The Markdown is for reading and chunking. Headings are detected from font size and bullets from bullet characters, not from a pixel perfect layout.
Links that need a login, a cookie banner click or a captcha before the file downloads return an error row.
It respects robots.txt by default and spaces downloads from one host one second apart.
Default memory is 512 MB, enough for typical files up to about 50 MB. Give the run more memory for very large files.
Up to 2,000 PDFs per run.

Make sure you have the right to process the documents you send, and use them in line with each source site's terms.

Try it on the Apify Store: https://apify.com/pistachio_implementation/pdf-text-extractor

How to get the latest Medium articles for a tag, author or publication as JSON

Hay Equipos — Sat, 03 Oct 2026 19:31:02 +0000

If you follow a topic, a set of writers or a few company engineering blogs on Medium, checking them by hand every morning gets old fast. What you want is a feed you can push into Slack, a spreadsheet, a newsletter draft or an AI summary: new stories, their links, their authors and, where possible, their text.

This guide shows how to set that up with a small Apify Actor published by Hay Equipos called Medium Articles Feed: Tags, Authors and Publications. It reads Medium's own public RSS feeds, the same feeds news readers use. No login and no browser.

What the tool returns

You give it any mix of tags, authors and publications, including publications on their own domain. For each story you get one row with:

the feed it came from (type, name and title)
story title, link and article ID
the public byline, publish date and last update date
the story's tags
word count and an estimated reading time
the lead image link
the story as plain text, and optionally as Medium's HTML
isFullText, which tells you whether the feed carried the whole story or only a preview

A row looks like this (values are illustrative):

{
  "feedType": "publication",
  "feed": "example-engineering",
  "feedTitle": "Example Engineering on Medium",
  "title": "How we cut our build times in half",
  "url": "https://medium.com/example-engineering/how-we-cut-our-build-times-in-half-1a2b3c4d5e6f",
  "articleId": "1a2b3c4d5e6f",
  "author": "Example Engineering Team",
  "publishedAt": "2026-09-30T14:05:00.000Z",
  "tags": ["devops", "ci", "engineering"],
  "isFullText": true,
  "wordCount": 1640,
  "readingTimeMinutes": 7,
  "imageUrl": "https://cdn-images-1.medium.com/max/1024/example.png",
  "text": "Our builds used to take forty minutes..."
}

The same story found in two feeds is saved once. Feeds that fail, such as a misspelled name or a site that is not a Medium publication, are listed in a RUN_SUMMARY record in the run's key value store.

What people use it for

Content monitoring: watch the topics and writers in your field every hour or every day, and send new stories to Slack, a sheet or a newsletter.
Competitor and brand tracking: follow the engineering or company blogs your competitors publish on Medium.
AI and RAG pipelines: feed the full text of author and publication stories into a summary or search tool.
Trend research: see which tags new stories carry and how long they are.

Step by step in the Apify Console

Open the Actor from its Apify Store page and sign in to Apify Console.
In the Input tab, add Tags such as artificial-intelligence or a tag link.
Add Authors as @handle, a profile link, or a name.medium.com address.
Add Publications as a slug (towards-data-science), a link, or a custom domain such as netflixtechblog.com.
Optionally set Only stories published after to a date like 2026-09-01. Older stories are skipped and not charged.
Keep Include article text on, and switch on Include article HTML if you need images and links inside the story.
Click Start, then export from the Output tab as CSV, JSON or Excel.

To monitor, schedule the Actor hourly or daily, and set Only stories published after to the time of your previous run so you pay only for new stories.

How to call it from code

With curl:

curl -X POST "https://api.apify.com/v2/acts/pistachio_implementation~medium-articles-feed/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"tags": ["artificial-intelligence"], "publications": ["netflixtechblog.com"], "publishedAfter": "2026-09-01"}'

In Python, with the apify-client package:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("pistachio_implementation/medium-articles-feed").call(
    run_input={
        "authors": ["@netflixtechblog"],
        "tags": ["startup"],
        "includeContent": True,
    }
)
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["publishedAt"], row["title"], row["url"], row["isFullText"])

Pricing

Pay per event: $0.001 per article saved, which is $1 per 1,000 articles. There is no start fee and no platform usage charge on top. Failed feeds, duplicates and stories older than your "published after" date are free. For example, following 20 tags and writers every day saves at most 200 stories a day, about $0.20. You can cap the spend of any run with the maximum charge setting in Apify, and the Actor stops cleanly when it is reached.

Limits and what it does not do

Medium's feeds carry the latest 10 stories per tag, author or publication. This tool is built for monitoring what is new, not for downloading a whole archive. Run it on a schedule to build up a history.
Tag feeds carry a short preview, not the full text. Author and publication feeds usually carry the full story. Check isFullText.
No member only content. Those stories appear with whatever Medium puts in the public feed. The Actor never logs in.
No search. Medium search pages are not read, because Medium's robots.txt disallows them. The Actor checks robots.txt for every host it reads.
No claps, responses, followers or profiles. They are not in the feeds, so they are not in the output. The author field is the story's public byline.
Requests are spaced one second apart per host.

The Actor is an independent tool and is not affiliated with Medium. Please use the stories in line with Medium's terms and respect the authors' rights in their work.

Try it on the Apify Store: https://apify.com/pistachio_implementation/medium-articles-feed