Getting only the new DOIs from the Crossref API every day: dates, cursors and rate limits

AI disclosure: this post was drafted with AI assistance. I ran every request and every number below myself against the live Crossref REST API on November 1, 2026, and checked the code and tests before publishing.

I wanted a boring thing: every morning, a CSV of the journal articles that got a DOI since yesterday, for a handful of journals, and never the same DOI twice. Crossref's REST API is free, needs no key, and held 188,319,891 records when I asked, so it should be one request. It is close, but it took me a few wrong turns. Here are the ones worth knowing about.

The finished script is about 400 lines of standard-library Python with a daily GitHub Actions workflow. It is my repo, free and MIT licensed: https://github.com/fecklesspleb/crossref-new-works-feed

1. Pick the right date: created, not published, not indexed

/works has four families of date filters, and for "what is new" they give very different answers. Here are the counts for a single day, 30 October 2026, all work types, each filter with from-… and until-… set to that day. The created count is stable; the other three drift a little as records change, so they are rounded:

Filter	Works on 30 Oct 2026
from-created-date / until-created-date	86,622
from-index-date / until-index-date	about 789,000
from-update-date / until-update-date	about 155,000
from-pub-date / until-pub-date	about 18,900

• created is when the DOI was first registered. That is the one you want for "new".
• index changes whenever Crossref re-indexes a record, including citation counts. Nine times more rows, nearly all old works.
• update is when the publisher last deposited a change to the metadata.
• pub-date is the publication date the publisher wrote down. It can be a year, a month, or a print issue date months away. "Published yesterday" misses most of what appeared yesterday and includes old material deposited late.

For scale, type:journal-article with created gave 26,945 on 28 October, 29,785 on 29 October and 26,966 on 30 October. On Saturday 31 October it was 16,641. Filter by journal, publisher or funder unless you really want ~9 MB of CSV a day.

2. Always set both ends of the range

from-created-date:2026-10-30 on its own means "from then until now". That is fine today and silently wrong when you re-run a past day. Set until-created-date too; both ends are inclusive.

The filters also take a time: from-created-date:2026-10-30T12:00:00,until-created-date:2026-10-30T12:59:59 returned 2,829 works for that hour.

3. Page with a cursor, and you can still sort

rows maxes out at 1,000. rows=1001 gets a 400:

Integer specified as 1001 but must be a positive integer less than or equal to 1000.

offset stops at 10,000 (offset must be ... less than or equal to 9999. Use the cursor parameter ...). So use deep paging:

1. Send cursor=* with the first request.
2. Read message.next-cursor from the answer and send it back with the next request.
3. Stop when a page comes back empty.

I was not sure cursors keep a sort order, so I tested it. With sort=created&order=asc and rows=2, three cursor pages came back in ascending created order, with the same DOIs as a single rows=6 request.

Reading oldest first matters for a feed with a cap. If you stop after N new works, the next run can carry on where you stopped instead of rereading the newest ones.

params = {
    "filter": "from-created-date:2026-10-31,until-created-date:2026-11-01,"
              "type:journal-article,issn:0028-0836",
    "rows": 1000,
    "cursor": cursor,                # "*" first, then message["next-cursor"]
    "sort": "created",
    "order": "asc",
    "select": "DOI,type,title,container-title,ISSN,publisher,member,created,published",
}

select= keeps responses small. A field name it does not know returns a 400 that lists the valid ones, which is handy.

4. The 2026 rate limits are lower than old blog posts say

Crossref changed its limits in July 2026. For list and filter requests like these:

• 1 request per second in the public pool;
• 3 per second in the polite pool;
• polite-pool limits counted per mailto email address.

The response headers tell you where you are. On a public-pool request I saw:

x-rate-limit-limit: 1
x-rate-limit-interval: 1s
x-concurrency-limit: 1
x-api-pool: public-array

So my client:

• sends one request at a time, at most once a second;
• slows down further if those headers ask for less.

I still hit a real 429. I ran the script twice back to back from the shell, each process started fresh, and the first request of the second came too soon after the last request of the first. The fix is the usual one:

• honour Retry-After (seconds or an HTTP date);
• otherwise back off 2, 4, 8 and 16 seconds;
• give up after 5 tries;
• retry 5xx and network errors the same way.

A 403, or an answer that is not JSON, is different. Stop and write nothing.

5. Say who you are

To get into the polite pool, add mailto=[e-mail] to the query string or the User-Agent. I do both, with a User-Agent that names the tool and links to its source:

crossref-new-works-feed/1.0 (+https://github.com/fecklesspleb/crossref-new-works-feed; mailto:[e-mail])

Use your own address. The limit is per email, so a shared one in a popular template would put everyone in the same bucket. I also validate it loosely and drop it if it does not look like an email. (api.crossref.org/robots.txt was a 404 when I checked, which means no rules, but the script checks it anyway.)

6. "New" arrives late, so overlap your window

A DOI's created timestamp is when it was registered, not when the API can serve it. The newest work I could see was 20 to 30 minutes behind its created time. Indexing can take longer on a busy day.

So a daily run should not ask for "yesterday" only. Mine asks for yesterday plus today and lets a seen-DOI list drop repeats. A work is only missed if it reaches the API more than a day late.

7. query is a relevance search, not a filter

query.bibliographic=perovskite solar over two days matched 246 works. Only 29 had both words in the title. If you sort by date (as a feed must), you no longer get the most relevant first, so you get all the loose matches in date order.

My fix was a client-side --title-word check: all the words must be in the title. Rejected works are not marked as seen, in case you loosen the check later.

8. Small normalisation traps

• ISSNs. issn:0028-0836, issn:00280836 and Nature's electronic ISSN issn:1476-4687 all returned the same 88 works for 25 October to 1 November. Normalise user input to NNNN-NNNC anyway, with an uppercase X check digit.
• DOIs are case-insensitive. /works/10.1038/S41586-026-11328-3 returns the lower-case DOI. Store them lower-cased, or your dedupe will see two works.
• Dates come as date-parts. That means [[2026]], [[2026, 10]] or [[2026, 10, 25]]. Keep the precision you were given rather than inventing a day.
• Titles and ISSNs are lists. A work can have no title or several ISSNs. I join ISSNs with |.
• Overlapping windows return duplicates by design. Dedupe by DOI before you write, both within a run and across runs.

9. Finding the IDs to filter on

You rarely know a member ID or funder ID by heart. The API has lookups. They are fuzzy searches, so the best match is not always first; check the names in the results:

• /journals?query=nature gives the ISSNs, used as issn:.
• /members?query=springer gives member IDs, used as member:.
• /funders?query=national science foundation gives funder IDs; prefix the id with 10.13039/ for funder:.

What the feed looks like

The first run for Nature over seven days found 88 works with 2 requests (robots.txt plus one page). The second run found 0 new. On the broad side, all journal articles since midnight with --max-works 1000 stopped at 1,000. The next run carried on and wrote the next 1,000, rather than the same ones again.

doi,type,title,journal,issn,publisher,member,published,created,url,first_seen
10.1038/s41586-026-11328-3,journal-article,ctDNA clearance and outcomes for resectable lung cancer from NeoCOAST-2,Nature,0028-0836|1476-4687,Springer Science and Business Media LLC,297,2026-10-25,2026-10-25T07:01:31Z,https://doi.org/10.1038/s41586-026-11328-3,2026-11-01T16:37:25Z

The GitHub Actions workflow runs it once a day and commits the CSVs. One catch: GitHub turns off scheduled workflows in public repositories after 60 days without activity.

Code, tests (offline with saved Crossref responses, plus a few live checks) and the workflow are MIT licensed: https://github.com/fecklesspleb/crossref-new-works-feed

Numbers in this post come from requests I made on November 1, 2026; Crossref's counts and limits change, so check the response headers on your own requests.