From a $10 One-Day Mistake to About $1.25 a Month: Fetching Only the Previous Day's Posts from the X API

I run a personal daily newspaper, published twice a day: a morning edition at 06:00 and an evening edition at 18:00. It includes posts on X from a streamer I follow. The developer dashboard, as of 2026-10-28, showed 247 billable events over the preceding 30 days for fetching those posts, totaling about $1.25.

That figure comes from deciding the window and the target accounts before fetching: "yesterday, 00:00 to 24:00". On the other hand, on a day when I fetched with no window and no cap on the count (2026-11-06, in work separate from the newspaper), fetching profiles and post lists resulted in about 1,000 reads, including reposts, and a $10 charge.

A pay-per-use API isn't something to fear. It's something to manage with a window and a count. This article walks through the design of the fetcher that makes that management work, and the real code (it runs on the Python standard library alone), one decision at a time. If you want to collect posts from specific accounts regularly with the X API, you can apply these design choices to your own system.

Where the fetcher sits in the publishing pipeline

First, the big picture. The newspaper is published automatically by a scheduled job and deployed to Cloudflare Pages. I started a trial run on 2026-10-26 and have published every day since 2026-11-02.

One issue goes through these steps:

1. Read back the voting data
2. Collect source material
3. Fetch and enrich X posts
4. Build the issue data (JSON)
5. Generate the site
6. Push to GitHub
7. Publish on Cloudflare Pages

This article covers only step 3, "fetch and enrich X posts". I split it out of the newspaper as a standalone tool (I call it x-posts-fetcher) and maintain it locally. Selecting articles, generating pages, and publishing are out of scope here.

The newspaper has a section that gathers posts from the streamer I follow. The targets are two accounts: a main account and a secondary one. I don't name the accounts or handles in this article. The code fragments and JSON values are fictional, there only to show the format.

The design core: window, accounts, and a single JSON output

I made three core design decisions in the fetcher:

• Decide the window first. The fetch period is fixed before any API call.
• Define the target accounts in configuration. Specify whose posts to fetch in the settings.
• Funnel the output into one JSON document. The newspaper reads only that JSON.

These three choices limit the two sources of higher cost — fetching too long a period and fetching too many accounts — and give the downstream step (building the pages) a single point of contact.

I also added two refinements while running it: prefetching the current day's posts at the evening edition and caching them for the next morning, and a step that fills in images and reaction counts from a separate source. I cover these after the core three and the cost section.

The previous-day window: decided in a timezone, sent in UTC

The default fetch window is the previous day in TZ, from 00:00 to 24:00. TZ can be set in the environment, and defaults to Asia/Tokyo if unset. By the time the morning edition goes out, "yesterday" is already over, so fetching the whole previous day gives a clearly bounded period.

You can also give the window explicitly with --from / --to. Write them in ISO 8601; values without a timezone are interpreted as TZ local time. I use this for checks that span dates and for re-fetching.

The start_time / end_time sent to the API are in UTC. The window calculation looks like this (a simplified version to show the shape):

from datetime import datetime, timedelta, timezone
from zoneinfo import ZoneInfo

def default_window(tz_name: str, now: datetime | None = None):
    tz = ZoneInfo(tz_name)
    now = (now or datetime.now(tz)).astimezone(tz)
    today_start = now.replace(hour=0, minute=0, second=0, microsecond=0)
    start = today_start - timedelta(days=1)
    end = today_start
    return start.astimezone(timezone.utc), end.astimezone(timezone.utc)

The period actually sent to the API is recorded in UTC in the output JSON's window, so that later I can check which period a given JSON covers. The window calculation and parsing can be verified with a self-test that uses no network.

Configuration: keys and targets live in .env, and process environment values win

Configuration goes in .env. There are three keys:

Key	Required	Meaning
X_BEARER_TOKEN	Required	Token used to authenticate with the API
X_ACCOUNTS	Required	Target accounts, comma-separated, without @
TZ	Optional	Timezone for the window; defaults to Asia/Tokyo

X_BEARER_TOKEN=xxxxxxxxxxxxxxxx
X_ACCOUNTS=example_user_a,example_user_b
TZ=Asia/Tokyo

.env is assumed to be a file you never publish. If a value with the same name exists in the process environment, that one takes precedence. That way the scheduler can override a value without rewriting the file.

Listing the target accounts in configuration makes it clear which accounts are being fetched. Cost estimates follow the same structure: multiply each account's post count by the post-read unit price, then add the user-read charges.

Fetching: resolve the user ID, paginate, and format the results into one JSON document

The fetcher follows these steps:

1. Resolve the user ID from the username (GET /2/users/by/username/:username).
2. Call GET /2/users/:id/tweets with start_time, end_time, max_results=100, and tweet.fields=created_at,entities.
3. Follow next_token in the response to paginate and get every post in the window.

There is a safety cap of 100 pages. If the cap is exceeded, the fetcher stops with an error telling you to narrow the window (in the code below the message reads "page limit reached; narrow the window", written in Japanese). It is a brake against pulling an unexpected volume.

The pagination skeleton looks like this (simplified):

def fetch_posts(user_id, start_utc, end_utc, token, max_pages=100):
    params = {
        "start_time": start_utc.strftime("%Y-%m-%dT%H:%M:%SZ"),
        "end_time": end_utc.strftime("%Y-%m-%dT%H:%M:%SZ"),
        "max_results": 100,
        "tweet.fields": "created_at,entities",
    }
    posts = []
    for _ in range(max_pages):
        body = api_get(f"/2/users/{user_id}/tweets", params, token)
        posts.extend(body.get("data", []))
        next_token = body.get("meta", {}).get("next_token")
        if not next_token:
            return posts
        params["pagination_token"] = next_token
    raise RuntimeError("ページ数の上限に達しました。窓を狭めてください")

The output is a single JSON document. It contains the fetch time fetched_at, the period actually sent window (UTC), and a list of posts per account. Each post carries id, created_at, text, urls (the links in the body, expanded), and url. The values are fictional examples.

{
  "fetched_at": "2026-11-06T21:00:05Z",
  "window": {
    "start": "2026-11-05T15:00:00Z",
    "end": "2026-11-06T15:00:00Z"
  },
  "accounts": {
    "example_user_a": [
      {
        "id": "1000000000000000001",
        "created_at": "2026-11-06T03:12:45.000Z",
        "text": "（投稿本文がここに入る）",
        "urls": ["https://example.com/page"],
        "url": "https://x.com/example_user_a/status/1000000000000000001"
      }
    ],
    "example_user_b": []
  }
}

The newspaper reads only this JSON. It doesn't need to know anything about X's specifications or billing.

When an error occurs, the fetcher prints a message that includes the HTTP status and exits non-zero. The token is never included in the output. The message adds a hint about the common cause:

• 401: invalid token
• 402: out of credits
• 429: rate limit

Since this component is called from a scheduler, failures show up through the exit code, and the message gives a lead on the cause.

Cost management by window and count: about $1.25 a month, and a $10 failure

Cost control is mostly settled by the design so far (window and count). The X API uses pay-per-use billing. The unit prices I confirmed on 2026-11-07 on the X Developer Platform pricing page are below. Billing is counted by the number of resources fetched, not the number of requests.

• Post reads: $0.005 each
• User reads: $0.010 each

Prices can change, so check the pricing page when you use this yourself. Also, when the credit balance is empty, reads fail (that is the 402 above).

I estimate costs from the number of posts and users read within the chosen window, using these unit prices. For the newspaper's X fetching, the developer dashboard as of 2026-10-28 showed 247 billable events over the past 30 days, about $1.25 in total. "About $1.25 a month" is based on that point in time and that display. It is the result of fetching only the previous day's posts for two accounts every day, and it will change with the number of accounts and how much they post.

The other number is a real failure. On 2026-11-06, in work separate from the newspaper, I fetched the profile and post list of an account outside the target set, with no window and no cap on the count. About 1,000 items, including reposts, were read, and $10 was charged (the breakdown by read type was not kept in my records). Fetching without a window or a cap can incur charges on a very different scale from the daily fetch, all at once.

The lesson is simple. With a pay-per-use API, decide the window and the count before you pull. The fetcher's design reflects this: the window is required, and the page count has a cap. But the cap is only a brake on volume. Reading all 100 pages × 100 results would come to $50 per account in post-read charges at the listed unit price. What keeps the bill small is not the cap but narrowing the window and the targets.

Refinement 1: fetch the current day early at the evening edition, then fetch only the remaining window the next morning

Now the two refinements I added while operating it.

When the morning edition fetches the previous day, it could pull the whole 00:00–24:00 span every time. But posts from 00:00 to 18:00 of that day were already fetched once to produce the evening edition (18:00). I save that fetched data as raw data in a per-date cache file and reuse it the next morning. The morning edition combines two parts:

• The cached part (00:00–18:00)
• Fetching only the remaining window (18:00–24:00)

Cache files are deleted once used. Even though the window is split in two, the parts add up to the previous day's 00:00–24:00, so this stays compatible with the design of deciding the window first.

For the posts dated 2026-10-30, for example, there were 4 cached posts and 4 fetched from the remaining window, for a total of 8.

Whether this reduces the morning fetch volume depends on how a day's posts are distributed across the day. I haven't measured the percentage reduction in my own setup, so I won't quantify the benefit here. The point is the division of work: fetch once at an intermediate point like the evening edition, and the morning only needs to pull a short window.

Refinement 2: fill in what the cards need with fxtwitter

The other refinement is enrichment. This fetch method does not return the images, avatars, or reaction counts needed for the newspaper's cards. So I fill in the following from https://api.fxtwitter.com/status/<post ID>:

• Images
• Avatars
• Counts of likes, reposts, and replies

GET https://api.fxtwitter.com/status/1000000000000000001

The post ID is the id in the X API fetch result. I use the IDs the fetcher put into the JSON and enrich one post at a time. When I introduced this on 2026-10-26, the backfill succeeded for all 18 past posts.

The roles are separate. The X API fetch provides the list of posts in the window, and the fxtwitter step adds the contents that go onto the card.

Who this design fits, and who it doesn't

It fits a use like this:

• The target accounts are fixed.
• The period to fetch can be bounded by day or hour.
• You want to hand the result to another system as JSON.
• You don't want to add external libraries (it runs on the standard library alone).

It doesn't fit in these cases:

• You want to search across topics or keywords. This component assumes fetching per account.
• You need to predict billing precisely. Unit prices can change, and the number of posts in the window varies by day.
• You need reliable fetching. I haven't collected statistics on fetch success rate or failure frequency. My operating records establish only that publication has continued and that the counts in the examples match; I don't yet have statistics that would support a numerical reliability guarantee.

Start with one account and one day

The first thing to do is start small:

1. Narrow the targets to one account.
2. Fix the window at the previous day.
3. Multiply the number of fetched posts by the read unit price, add the user-read charges, and work out an expected cost.
4. Set the page cap by working backward from your own budget.

Once the window and the count cap are fixed, the range your bill can swing over becomes predictable. After that, if you start using it regularly, adding an early fetch at an intermediate point, like the evening edition, plus enrichment with images and reaction counts, is enough.

In my setup, these small, regular fetches cost about $1.25 a month, based on the dashboard snapshot dated 2026-10-28. Start with one day, set a window, and pull.

---

Originally published in Japanese on note. This English edition was translated and edited with AI assistance; all commands and measurements are unchanged from the original.