This is a write-up of a small data collector: a Cloudflare Worker on an hourly cron that reads settled funding rates from the public APIs of OKX, Bybit and MEXC and stores them in D1. The code is about 100 lines. Most of the time went into two things that are not in any quick-start: the three APIs disagree on almost every field name, and two of them answered 403 from the Worker while answering 200 from a laptop.
What is being collected
Perpetual futures have a funding rate: a periodic payment between longs and shorts, usually settled every 8 hours. Each exchange publishes a predicted rate that changes until settlement, and a settled rate that is final. For a history table you want the settled one, read from the exchange's own record.
All three exchanges expose that record without an API key:
| Exchange | Endpoint | Time field | Rate field |
|---|---|---|---|
| OKX | /api/v5/public/funding-rate-history?instId=BTC-USDT-SWAP |
fundingTime |
realizedRate |
| Bybit | /v5/market/funding/history?category=linear&symbol=BTCUSDT |
fundingRateTimestamp |
fundingRate |
| MEXC | /api/v1/contract/funding_rate/history?symbol=BTC_USDT |
settleTime |
fundingRate |
Three exchanges, three symbol formats (BTC-USDT-SWAP, BTCUSDT, BTC_USDT), three timestamp names and three response envelopes (data, result.list, data.resultList). The values are strings in some responses and numbers in others.
One function per venue, one shape out
The collector hides all of that behind a single function that returns [{ ts, rate }]:
const UA = { 'User-Agent': 'funding-collector/1.0', Accept: 'application/json' };
async function getJson(u) {
const r = await fetch(u, { headers: UA, signal: AbortSignal.timeout(10000) });
if (!r.ok) throw new Error('http ' + r.status);
return r.json();
}
// Returns [{ ts, rate }] of settled funding, newest first.
async function settled(venue, coin, limit) {
if (venue === 'okx') {
const j = await getJson(`https://www.okx.com/api/v5/public/funding-rate-history?instId=${coin}-USDT-SWAP&limit=${Math.min(limit, 100)}`);
return (j?.data || []).map((d) => ({
ts: Number(d.fundingTime),
rate: Number(d.realizedRate !== undefined && d.realizedRate !== '' ? d.realizedRate : d.fundingRate),
}));
}
if (venue === 'bybit') {
const j = await getJson(`https://api.bybit.com/v5/market/funding/history?category=linear&symbol=${coin}USDT&limit=${Math.min(limit, 200)}`);
return (j?.result?.list || []).map((d) => ({ ts: Number(d.fundingRateTimestamp), rate: Number(d.fundingRate) }));
}
if (venue === 'mexc') {
const j = await getJson(`https://contract.mexc.com/api/v1/contract/funding_rate/history?symbol=${coin}_USDT&page_num=1&page_size=${Math.min(limit, 100)}`);
return (j?.data?.resultList || []).map((d) => ({ ts: Number(d.settleTime), rate: Number(d.fundingRate) }));
}
return [];
}
Two details in there are deliberate.
AbortSignal.timeout(10000) on every request. A cron invocation that waits forever on one slow exchange also delays the other two, so each call gets ten seconds and then fails on its own.
OKX returns both fundingRate and realizedRate in its history rows. The collector prefers realizedRate and falls back to fundingRate only when the realized value is missing or empty, because the goal is the rate that was actually applied at settlement.
Storage: let the primary key do the deduplication
The table has no surrogate id. The natural key is the settlement itself:
CREATE TABLE IF NOT EXISTS funding_history (
venue TEXT NOT NULL,
coin TEXT NOT NULL,
ts INTEGER NOT NULL,
rate REAL NOT NULL,
PRIMARY KEY (venue, coin, ts)
);
With that key, writing is INSERT OR IGNORE. The cron can fetch overlapping windows, run twice, or be re-run by hand after a failure, and the table ends up the same. There is no "have I seen this row" logic in JavaScript at all.
stmts.push(
db.prepare(`INSERT OR IGNORE INTO funding_history (venue, coin, ts, rate) VALUES (?1, ?2, ?3, ?4)`)
.bind(r.venue, r.coin, row.ts, row.rate)
);
// ...
for (let i = 0; i < stmts.length; i += 50) {
const res = await db.batch(stmts.slice(i, i + 50));
out.inserted += res.reduce((a, x) => a + (x?.meta?.changes || 0), 0);
}
meta.changes from each statement in the batch gives the number of rows that were really new, which is what goes into the run report. Rows are also checked before they are queued: a timestamp or rate that is not a finite number is dropped, and so is a timestamp more than a minute in the future.
Ask for less on every run after the first
Each run starts with one query that returns the row count and the newest timestamp per venue and coin:
const counts = new Map((
(await db.prepare(`SELECT venue, coin, COUNT(*) n, MAX(ts) last FROM funding_history GROUP BY venue, coin`).all()).results || []
).map((r) => [r.venue + ':' + r.coin, r]));
for (const venue of VENUES) {
for (const coin of COINS) {
const c = counts.get(venue + ':' + coin);
// Nothing new can exist until one settlement interval (shortest 1 h) after the last stored row.
if (c && now - Number(c.last) < 3600e3) continue;
jobs.push({ venue, coin, limit: c && c.n >= 10 ? 3 : 90 });
}
}
A pair that already has history asks for the last 3 settlements. A pair with fewer than 10 rows asks for 90, which is roughly 30 days at three settlements a day. That backfill is not a reconstruction: it is the same endpoint and the same field, just a longer page. And a pair whose newest row is less than an hour old is skipped entirely, since no exchange here settles more often than hourly.
The fetches then run in parallel, and one failure does not sink the run:
const results = await Promise.all(jobs.map((j) =>
settled(j.venue, j.coin, j.limit)
.then((rows) => ({ ...j, rows }))
.catch((e) => ({ ...j, rows: [], err: String(e?.message || e).slice(0, 60) }))
));
Every job resolves, either with rows or with a short error string. The run report lists the errors next to the insert count, so "Bybit failed for SOL this hour" is visible without stopping OKX and MEXC from being stored. Because writes are idempotent, the failed pair simply catches up on the next run.
The 403 that only happened in production
From a development machine, all three endpoints returned data. Deployed, the cron run stored OKX and reported http 403 for the Bybit and MEXC pairs.
Nothing in the request had changed. Same URL, same headers, no key involved. What had changed was where the request came from. A Worker's outbound fetch leaves from Cloudflare's network near wherever the Worker happens to run, and for a cron trigger that is not near you. The two exchanges were refusing requests from that location. Neither response said why, so this is an inference from the behaviour, not something either exchange documents: the same call succeeded once the Worker ran somewhere else.
The fix was one block in wrangler.toml:
[placement]
region = "aws:ap-northeast-2"
This asks Cloudflare to run the Worker close to that cloud region (Seoul) instead of wherever the trigger fired. After deploying it, responses from the Worker carried the header Cf-Placement: remote-ICN, and the next cron run loaded every pair: 3 exchanges plus a second OKX product, 5 coins, 90 settlements each.
A few things worth knowing before copying that:
- Check the placement options in Cloudflare's current documentation for your plan and wrangler version. The block above is what this project runs with; the available settings have changed over time.
- Look at the
Cf-Placementresponse header to confirm where the Worker ran. Without it you are guessing. - This is a routing preference, not a way around anyone's access rules. If an API's terms restrict where it may be used from, placement does not change what you are allowed to do.
- If the block is ever removed, the 403s come back on the next run. It deserves a comment in the config file saying why it is there.
What the public endpoint looks like
The Worker also serves the stored rows as JSON, with days, venue and coin filters. The filters are checked against the known lists before they reach SQL, and the values are bound parameters:
let sql = `SELECT venue, coin, ts, rate FROM funding_history WHERE ts >= ?1`;
const args = [since];
if (venue && VENUES.includes(venue)) { sql += ` AND venue = ?${args.length + 1}`; args.push(venue); }
if (coin && COINS.includes(coin)) { sql += ` AND coin = ?${args.length + 1}`; args.push(coin); }
sql += ` ORDER BY venue, coin, ts`;
The response is cached for 15 minutes (Cache-Control: public, max-age=900). Settlements arrive at most hourly, so a page that reads this endpoint does not need anything fresher, and the database is not queried once per visitor.
What to keep from this
- Normalize at the edge of the system. One function per source that returns one shape keeps the three APIs' differences out of everything downstream.
- Make the write idempotent with a natural primary key, then stop worrying about overlap and retries.
- Give every outbound call its own timeout and its own
catch, and report failures instead of throwing. - When an API behaves differently in production than on your machine, compare where the request comes from before comparing the request.
The collector has been running hourly since 2026-11-13. On 2026-11-16 it held 1,860 settlements across the four venue feeds.
Disclosure: this post was written by an AI assistant (Claude) from the project's source code and deployment notes. The account owner approved it for publishing without editing the text. The code and the figures are from the running project.
Top comments (0)