DEV Community: Feedsmith

JSON or CSV to Google Sheets on a schedule, without OAuth (service account + one API call)

Feedsmith — Sun, 27 Sep 2026 14:07:04 +0000

Getting data into a Google Sheet is easy once. Doing it every hour, unattended, is where it breaks:

OAuth needs a human. "Sign in with Google" works when you click it in a browser. A cron job, a webhook or an AI agent can't click, and refresh tokens get revoked or expire (7 days for apps in "testing" mode).
Appending is not "write below the last row". The next batch has a new field, or fields in a different order, and your columns silently shift.
Re-running creates duplicates. Daily price or stock checks need update this row if it exists, otherwise add it - which the Sheets API does not do for you.
Limits show up late. A spreadsheet holds at most 10 million cells, write requests are rate limited per minute, and a request body can't be arbitrarily big.

This post shows the service-account route, then a hosted Actor I built that wraps the annoying parts.

Step 1: a service account (5 minutes, once)

A service account is a robot Google identity in your own Google Cloud project. It signs its own
tokens, so there is no consent screen and nothing to refresh by hand.

console.cloud.google.com -> project dropdown -> New Project.
APIs & Services -> Library -> "Google Sheets API" -> Enable.
APIs & Services -> Credentials -> Create Credentials -> Service account (no roles needed).
Open it -> Keys -> Add Key -> Create new key -> JSON. Keep that file private.
In your Google Sheet: Share -> paste the client_email from the JSON -> Editor.

The account can only touch sheets you explicitly share with it. Delete the key in the console and
access stops immediately.

Step 2: send rows

The Actor is feedsmith/google-sheets-sync. From Python, with an Apify API token:

import json, os, requests

rows = [
    {"sku": "A-100", "name": "Desk lamp", "price": {"usd": 24.9}, "tags": ["home", "light"]},
    {"sku": "A-101", "name": "Monitor arm", "price": {"usd": 59.0}, "tags": ["office"]},
]

run = requests.post(
    "https://api.apify.com/v2/acts/feedsmith~google-sheets-sync/run-sync-get-dataset-items",
    params={"token": os.environ["APIFY_TOKEN"]},
    json={
        "serviceAccountKey": open("service-account.json").read(),
        "spreadsheet": "https://docs.google.com/spreadsheets/d/<your-sheet-id>/edit",
        "sheetName": "Prices",
        "mode": "upsert",
        "keyColumns": ["sku"],
        "rawData": rows,
    },
    timeout=300,
)
print(json.dumps(run.json(), indent=2))

The response is one summary item (real run, 8.6 s):

[{"mode": "upsert", "spreadsheetId": "12EqRkXF...", "sheetName": "Prices", "rowsAppended": 2,
  "rowsUpdated": 0, "rowsSkippedDuplicate": 0, "newColumns": ["sku", "name", "price.usd", "tags"],
  "totalCellsAfter": 52104, "warnings": []}]

Next day, send {"sku": "A-100", "price": {"usd": 19.9}} plus a new SKU A-102 with the same call.
The summary says "rowsAppended": 1, "rowsUpdated": 1 and the tab reads:

sku	name	price.usd	tags
A-100	Desk lamp	19.9	home, light
A-101	Monitor arm	59	office
A-102	Cable tray	12.5

A-100 kept its name and tags; only the price changed.

What happened there:

price.usd became its own column (nested objects are flattened to a.b.c), tags became home, light.
upsert matched on sku: existing rows were updated in place, new SKUs were appended. Run it again tomorrow with new prices and you still have one row per SKU.
The tab Prices didn't exist, so it was created.

Other modes: append (never clears anything; unknown keys become new columns on the right, existing
column order is kept), replace (clears the tab and rewrites it, after saving the old values as a JSON
backup) and read (the tab comes back as JSON rows keyed by the header).

Chain it after any scraper

In Apify Console, open the scraper -> Integrations -> Connect Actor -> pick Google Sheets
Import & Export and set datasetId to {{resource.defaultDatasetId}}. Every time the scraper finishes,
its results land in the sheet. No code.

Or let an AI agent do it

Through the Apify MCP server an agent can call the Actor with rawData directly. Because the key does
not need a browser login, it works the same from Claude, Cursor or any MCP client as it does from cron.

Details that bite otherwise

Formula injection. Scraped text like =IMPORTXML(...) or +1-... becomes a live formula if you write with USER_ENTERED. The default is RAW; with USER_ENTERED such values are escaped unless you set allowFormulas. In a real run with USER_ENTERED, =1+1 landed as the text =1+1 (not 2), +84 912 stayed a string, and 2026-09-27 became a real date.
Cell limit. The Actor sums every tab's grid before writing and refuses with "about N rows fit" instead of failing halfway.
Rate limits. Writes are chunked and paced; 429/5xx are retried with backoff. 20,000 rows x 6 columns went into a new tab in 24 s using 50 MB of memory.
Errors you can act on. A sheet that wasn't shared fails with "Share the spreadsheet with sheets-sync@...iam.gserviceaccount.com as Editor (Share button in Google Sheets), then run again."; a disabled API says which project to enable it in.

Cost

$0.002 per successful run plus $0.005 per started 1,000 rows. A failed or dry run is not charged.
A daily sync of 500 rows is about $0.21 a month; an hourly one about $5.

Airbnb has no public API - how to get listing and price data for any city (Python)

Feedsmith — Tue, 22 Sep 2026 14:07:05 +0000

People search "Airbnb API" thousands of times a month, but Airbnb has never had a public API for listing
data. Its partner APIs are for hosts and software vendors managing their own listings. If you're sizing a
short-term-rental market, pricing your own place against the neighbourhood, or doing a data-science project, you
end up doing one of these:

Download a static dataset (Inside Airbnb, Kaggle): free and great for research, but a snapshot, with no prices for your dates.
Pay for a market-analytics subscription: polished dashboards, priced for professionals.
Collect the public search results yourself.

This post is about option 3, done without writing or maintaining a scraper.

The 270-result wall

An Airbnb search tops out at 15 pages of 18 results: about 270 listings per query, however many exist. A city
like Lisbon has thousands. Most tools stop at that wall, and the popular Airbnb scraper on the Apify Store states
a 240-result limit.

The way around it is to split the query until each piece fits under the wall: first by nightly-price band,
then, if a band is still full, into map quadrants. Then de-duplicate by listing id. I built that into an
Apify Actor called "exhaustive mode". In a cloud test on
Porto for four nights it returned 330 unique listings from 28 sub-queries (41 HTTP requests) in 50 seconds,
past the single-query ceiling. A Lisbon test hit the requested 400-listing cap with zero duplicate ids.

Script: a city's listings for your dates

import csv, os, requests

resp = requests.post(
    "https://api.apify.com/v2/acts/feedsmith~airbnb-listings-scraper/run-sync-get-dataset-items",
    params={"timeout": 600},
    headers={"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"},
    json={
        "locations": ["Porto, Portugal"],
        "checkIn": "2026-11-06", "checkOut": "2026-11-09",
        "exhaustive": True, "maxItems": 300, "currency": "USD",
    },
    timeout=630,
)
resp.raise_for_status()
listings = resp.json()

fields = ["id", "name", "roomType", "priceNightly", "priceTotal", "nights",
          "rating", "reviewsCount", "latitude", "longitude", "url"]
with open("airbnb_listings.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
    w.writeheader()
    w.writerows(listings)
print(len(listings), "listings")

Real rows from that run (2026-09-18):

id,name,roomType,priceNightly,priceTotal,nights,rating,reviewsCount,latitude,longitude,url
38350016,São Bento Dream Loft - City's  Historic Center,entire_home,87.67,263,3,4.92,241,41.1444,-8.6113,https://www.airbnb.com/rooms/38350016
19666593,Casas de SantAna - Old town amazing views,entire_home,130.33,391,3,4.98,632,41.1426,-8.6133,https://www.airbnb.com/rooms/19666593

priceTotal is what Airbnb shows for your whole stay. priceNightly is that total divided by the nights, so it
includes cleaning and service fees spread over the stay. That's usually what you want when comparing listings, but
it can sit a bit above the "per night" figure Airbnb's own price filter uses.

A trap when computing market stats

My first version of this script printed median prices from those 300 listings: $158.67 per night for entire homes,
and a p75 of $536. That p75 is wrong as a market figure. Exhaustive mode walks the city price band by price
band, so a run that stops at maxItems has only covered some of the bands. The sample is skewed, not random.

Two ways to get honest numbers:

Raise maxItems until the run finishes below it. Then you have every listing Airbnb returns for those dates and filters.
Or narrow the search (a neighbourhood's map bounds, a price range, roomType) so the full set is small.

The example repo's script now prints a warning when a run was cut off by maxItems.

Details per listing

Set "includeDetails": true to also fetch each listing page: description, the amenities list, house rules, bedrooms,
beds, bathrooms, guest capacity, the host's first name and superhost flag, and six sub-ratings (cleanliness, accuracy,
check-in, communication, location, value). In a four-listing test all four came back complete.

Cost

What	Price
Listing (search data)	$1.20 per 1,000
Listing details (optional)	$2.00 per 1,000

The Porto run above (300 listings) cost the user about $0.36. Duplicates and failed pages are free.

Caveats

Prices depend on dates, guests and currency. Always pass dates if you care about prices.
About one search page in five comes back as an empty page shell. The Actor retries it up to three times, and if it still fails it logs a warning rather than dropping data silently.
It collects public listing data only: no host contact details and no guest reviews' author data.
Airbnb's terms restrict automated access. Use the data responsibly and check what's appropriate for your use case and jurisdiction.

Polymarket API in Python - odds, volume and order books to CSV without writing a scraper

Feedsmith — Mon, 21 Sep 2026 14:07:05 +0000

Polymarket prices are probabilities: a YES share at $0.69 means the market gives the event about a 69% chance.
That makes them handy for dashboards, newsletters, research and model features.

Polymarket's own APIs are public and free: the Gamma API for markets and events, the CLOB API for order
books and price history, and a Data API for trades. If you're writing a trading bot, use them directly.
If you just want clean rows in a CSV, on a schedule, there are a few annoying details:

outcomes, outcomePrices and clobTokenIds come back as JSON strings inside JSON and need a second parse.
Pages cap at 100 markets, so you have to paginate.
There is no server-side text search on /markets.
Sorting "top markets by volume" through /events sorts events, not markets, so small side-markets of a big event jump the queue.
Order books and price history are keyed by outcome token id, not market id.

I packaged all of that into a hosted Actor, so you get one flat record per market.

One request, one CSV

import csv, os, requests

resp = requests.post(
    "https://api.apify.com/v2/acts/feedsmith~polymarket-markets-scraper/run-sync-get-dataset-items",
    params={"timeout": 300},
    headers={"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"},
    json={"status": "open", "search": "bitcoin", "minVolume": 1000, "maxItems": 50},
    timeout=330,
)
resp.raise_for_status()
markets = resp.json()

with open("polymarket.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["question", "yes_probability", "volume_24h", "liquidity", "closes", "url"])
    for m in markets:
        w.writerow([m["question"], m["impliedProbability"], m["volume24h"], m["liquidity"], m["closeTime"], m["url"]])

Output from a real run on 2026-09-18:

   98.9%  Will the price of Bitcoin be above $76,000 on September 18?
  100.0%  Will the price of Bitcoin be above $72,000 on September 18?
   69.5%  Will Bitcoin reach $80,000 in September?

Leave out search and you get the most traded open markets across the exchange, sorted by 24h volume (100
markets in about 8 seconds).

What a record contains

A real record (run on 2026-09-18), with token ids shortened:

{
  "question": "Will Bitcoin reach $80,000 in September?",
  "outcomes": [
    {
      "name": "Yes",
      "price": 0.695,
      "impliedProbability": 0.695,
      "tokenId": "463643032374..."
    },
    {
      "name": "No",
      "price": 0.305,
      "impliedProbability": 0.305,
      "tokenId": "742364722558..."
    }
  ],
  "yesPrice": 0.695,
  "noPrice": 0.305,
  "impliedProbability": 0.695,
  "volume": 470026.909022,
  "volume24h": 73831.21954200002,
  "liquidity": 67523.6986,
  "bestBid": 0.69,
  "bestAsk": 0.7,
  "spread": 0.01,
  "lastTradePrice": 0.7,
  "closeTime": "2026-10-01T04:00:00Z",
  "status": "open",
  "result": null,
  "eventTitle": "What price will Bitcoin hit in September?",
  "tags": [
    "crypto",
    "bitcoin",
    "crypto-prices",
    "recurring",
    "hit-price",
    "monthly"
  ],
  "url": "https://polymarket.com/event/what-price-will-bitcoin-hit-in-september-2026",
  "scrapedAt": "2026-09-18T13:11:59.342Z"
}

Every numeric field is a real number and every JSON-string field is already parsed. Multi-outcome markets keep the
whole outcomes array. impliedProbability is the YES price for binary markets.

Order book, trades and price history

Add any of these flags. Each enriched market costs one extra charge, whichever flags you turn on:

json={
    "search": "bitcoin", "minVolume": 1000, "maxItems": 5,
    "includeOrderbook": True,            # bids/asks per outcome token, best bid/ask, depth
    "includeTrades": True, "maxTradesPerMarket": 50,
    "includePriceHistory": True, "historyInterval": "1w",
}

Privacy note: Polymarket's trade feed includes trader pseudonyms, bios and profile pictures. The Actor keeps only
the public wallet address on each trade.

Schedule it

On Apify, save the input as a task and add a schedule (every 15 minutes, hourly, daily). Each run produces a
dataset you can pull as CSV/JSON from a stable URL, or send to Google Sheets, a webhook, Make, Zapier or n8n.
Every record has scrapedAt, so appending runs gives you a time series of odds.

Cost

$1.50 per 1,000 markets, plus $3 per 1,000 markets enriched with order book, trades or history. A top-100 snapshot
every hour for a month (~72,000 records) comes to about $108. Every 6 hours brings it down to about $18.

Links

Actor: https://apify.com/feedsmith/polymarket-markets-scraper
Runnable example: https://github.com/ankaibua-spec/feedsmith-examples/blob/master/polymarket/top_markets_to_csv.py
Polymarket's own docs: https://docs.polymarket.com/

Not affiliated with Polymarket. Nothing here is trading advice. Disclosure: I built this Actor. This article was drafted with AI assistance (Claude); every command, number and output above comes from real runs on 2026-09-18.

How to find Shopify stores that use Klaviyo (a cheap Wappalyzer/BuiltWith alternative in Python)

Feedsmith — Fri, 18 Sep 2026 13:26:17 +0000

If you sell anything to e-commerce brands (an app, an agency service, an integration), your best leads are
stores that already run the stack you plug into. "Shopify stores that use Klaviyo" is the classic example: they
have an email budget, and they already pay for SaaS.

BuiltWith and Wappalyzer answer "what technology is this website using?", but their lead lists are priced
for bigger teams. This post shows a small pay-as-you-go alternative: give it a list of domains, get back the
technologies each site runs, keep the matches. It costs $0.02 per site analysed, and failed sites are free.

What you need

Python 3.9+ and requests
A free Apify account. The monthly free credit covers a few hundred sites. Copy your API token from Console > Settings > API & Integrations.
A list of domains, one per line (a CSV export from anywhere works)

The script

The detector runs as a hosted Apify Actor. Apify's
run-sync-get-dataset-items endpoint starts it, waits for it to finish and returns the results as JSON in a
single HTTP call, so there is no SDK to install:

import csv, os, requests

REQUIRED = {"Shopify", "Klaviyo"}
domains = [d.strip() for d in open("domains.txt") if d.strip()]

resp = requests.post(
    "https://api.apify.com/v2/acts/feedsmith~tech-stack-detector/run-sync-get-dataset-items",
    params={"timeout": 300},
    headers={"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"},
    json={"urls": domains, "technologies": sorted(REQUIRED), "maxConcurrency": 10},
    timeout=330,
)
resp.raise_for_status()
sites = resp.json()

matches = [s for s in sites if s["status"] == "ok" and REQUIRED <= set(s["technologyNames"])]
with open("matches.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["domain", "final_url", "title"])
    w.writerows([s["domain"], s["finalUrl"], s.get("title") or ""] for s in matches)

print(f"{len(sites)} sites analysed, {len(matches)} run both Shopify and Klaviyo")

The technologies input makes the Actor report only the technologies you care about, so each record stays
small. Leave it out and you get everything it detects: CMS, analytics, ad pixels, payment providers, CDN,
hosting, JavaScript frameworks and more, from about 7,600 fingerprints.

Real run

I ran it on 13 well-known direct-to-consumer brands:

$ python find_shopify_klaviyo.py
13 sites analysed, 4 run both Shopify and Klaviyo -> matches.csv
   www.glossier.com
   kyliecosmetics.com
   www.taylorstitch.com
   www.tentree.com

Most of the other nine are on Shopify too, but Klaviyo wasn't visible in their HTML. That is the main caveat,
covered below.

A full record (without the technologies filter) looks like this, trimmed:

{
  "domain": "www.allbirds.com",
  "status": "ok",
  "httpStatus": 200,
  "technologyNames": ["Apple Pay", "Cloudflare", "Google Tag Manager", "HSTS", "HTTP/3",
                      "Open Graph", "PayPal", "Priority Hints", "Shopify", "..."],
  "categorySummary": { "Ecommerce": ["Shopify"], "Payment processors": ["Apple Pay", "PayPal"], "...": [] }
}

Where the domain list comes from

The detector checks a list you already have. It does not discover stores for you. Common sources:

Exports from your CRM, a conference exhibitor list or a partner directory
Search results you've already collected for your niche ("organic skincare", "running gear", ...)
Customers of a competitor, from their public case-study pages

Run the list through the script once, then again every month to catch stores that switched platforms.

How accurate is it?

I validated it on 28 sites with a publicly known platform: it identified the platform on 16 of the 20 where
there was an unambiguous answer. The misses are informative:

Signals that only exist after JavaScript runs. The detector reads the HTML and HTTP headers, like curl, not a headless browser. A chat widget or email tool injected later by Google Tag Manager won't show up. That's why Klaviyo is missing on some Shopify stores above.
Sites that block non-browser clients return a challenge page with few signals.

The upside of skipping the browser is speed and price: about 20 sites in 35 seconds on Apify, which is why it can
charge $0.02 a site.

Cost

Sites	Price
100	$2
1,000	$20

Sites that fail (dead domain, timeout) aren't charged. With onlyMatchingSites: true, sites that don't match your
technologies filter are neither saved nor charged.