DEV Community: melonape

A token is not a word, and your chat box is not the bill

melonape — Tue, 03 Nov 2026 07:05:14 +0000

When a tool says a request used "1,000 tokens", it's tempting to read that as 1,000 words. It isn't. A token is the unit a model's tokenizer cuts text into. Sometimes that's a whole word, sometimes a piece of one, sometimes a space glued to a number, and sometimes a few bytes that don't make a readable character on their own.

That gap matters as soon as you start budgeting context windows or reading an API invoice.

What the model actually receives

The model never sees your sentence as typed. The tokenizer splits it into chunks using a fixed vocabulary and rules, then maps each chunk to an integer ID. Those IDs are just positions in a lookup table. A bigger ID doesn't mean a more important word.

Generation runs the other way: the model produces token IDs one step at a time and the app decodes them back to text. The "typing" effect in a chat UI is also shaped by network buffering and rendering, so each visible jump on screen isn't necessarily one token.

One English word, six tokens

Storing every possible word would make the vocabulary huge and still miss new names and typos. Splitting everything into single characters makes common phrases painfully long. Most tokenizers land in between with subword pieces, and those pieces don't have to match the roots and suffixes you learned in school.

OpenAI's tiktoken cookbook uses this example. Under the cl100k_base encoding, antidisestablishmentarianism becomes six tokens:

ant | idis | establish | ment | arian | ism

Under the older r50k_base encoding, the same word is five tokens, with establishment kept as one piece. Neither result is "more accurate". They're different vocabularies, so a count from one encoding can't be reused for a model that runs on the other. BPE, WordPiece and Unigram also build vocabularies differently, which is why the only count that means anything is the one from the model's own tokenizer.

Spaces and digits aren't free

The same cookbook shows 2 + 2 = 4 (with â marking a space):

Encoding	Tokens	Count
`r50k_base`	`2` `â +` `â 2` `â =` `â 4`	5
`cl100k_base`	`2` `â +` `â` `2` `â =` `â` `4`	7

Same string, different grouping of spaces and digits. A leading space, a case change or an extra newline can all shift the count, so if you strip whitespace before counting, you're no longer measuring the text you send.

The "4 characters per token" rule

You've probably seen "1 token is about 4 characters" or "about 3/4 of a word". OpenAI gives that as a rough rule of thumb for English prose. It falls apart on code, URLs, emoji and non-Latin scripts.

Chinese is a good stress test. There are no spaces between words, so a tokenizer might merge two or three common characters into one token, or spend several tokens on one rare character. In UTF-8 a Chinese character usually takes multiple bytes, and byte-level tokenizers can split it so that a single token decodes to half a character. Fewer tokens for the same text also doesn't tell you which model understands it better.

Which tokens are you being charged for?

Usage numbers can mean different things:

Input tokens are everything the request sends: your new message, plus whatever history, system prompt, tool definitions or retrieved documents the app includes.
Output tokens are what the model generates. Some APIs also count hidden reasoning or structural tokens here, so this won't always match the visible reply.
Context window is how many tokens one request can hold. It isn't a character limit you can fill with the same number of characters, and it isn't long-term memory.

So typing "make it shorter" can still be an expensive turn if the app resends the whole conversation with it. Some apps trim, summarize or retrieve history instead. What you can scroll back to and what the model got this turn are two separate things.

On the API side, providers may price input, output and cached tokens separately. Chat subscriptions use their own message limits. Token counts tell you how much you used. Price and quota come from the product, the same way kilowatt-hours don't tell you your bill until you know the rate.

Counting properly

For a short chunk of plain text, use the provider's tokenizer tool or tiktoken with the right encoding for your model, and paste the text exactly, spaces and newlines included. That beats asking a chat model to guess its own token count.

For a real API request, counting the message body isn't enough. Roles, message boundaries, tool schemas, images and audio can all be counted in other ways. OpenAI's docs separate local text tokenization from full-request counting, and if you need to match your bill, check the provider's counting endpoint and usage records.

Last thing: deleting punctuation or compressing prompts into shorthand to save tokens often backfires. The count doesn't drop in proportion, and the instruction gets vaguer first. Cutting irrelevant pasted context and repeated instructions usually saves more.

Examples are from OpenAI's How to count tokens with tiktoken cookbook. Also see OpenAI's Counting tokens guide and Hugging Face's tokenizer summary.

I first wrote this up in Chinese for WarpNav, where we keep plain-language AI explainers.

What I reject before a tool lands in an AI directory

melonape — Sun, 25 Oct 2026 13:13:07 +0000

Most AI directories grow by accepting submissions. That is a product choice. It also explains why so many boards feel interchangeable: the same logo wall, the same one-line blurbs, the same tools that shipped a landing page and never a pricing page that still loads.

If the job is “help someone open the right official site and know what the tool is for,” volume is the wrong scoreboard. The useful work is deciding what not to keep.

Here is the reject list I actually use when evaluating an AI or developer tool for a curated directory. None of these require a fake “I used it for three weeks” story. They are checks against public pages anyone can open.

1. Marketing-only, no task surface

Reject when the homepage is slogans, hero video, and a waitlist — and there is no clear input → output for a real job.

“AI-powered productivity for modern teams” is not a product description. A usable listing needs at least one concrete loop: paste a repo and get a PR draft, upload a PDF and get structured text, pick a model and get an API key that works. If you cannot name the loop after two minutes on the site, the directory entry will not help the next reader either.

2. Pricing that cannot be verified

Reject when pricing is “Contact us” with no public bands, or when the pricing URL 404s / redirects into a broken CMS.

Directories that list “Freemium” without a checkable plan page teach people the wrong habit: trusting a badge instead of a bill. If the official pricing page is gone, stale, or only available after a sales call, say so — or skip the listing until the page exists again.

3. No docs worth linking

Reject when there is no docs root, no GitHub README that explains install, and no API reference for a tool that markets itself as an API.

A directory link is often the last stop before someone wastes an afternoon. Official docs (or a maintained README with install, auth, and limits) are part of the product. “Coming soon” docs are not docs.

4. Clone spam and skin swaps

Reject near-duplicates that only swap the brand color and the LLM logo on the homepage.

The tell is structural: same feature bullets in the same order, same screenshot chrome, same “multi-model” claim with no routing or eval detail. Listing ten clones does not help a developer choose; it trains the board to rank whoever paid for placement first. Keep the clearest official product in the category; drop the rest until one of them shows a real difference (local-only mode, open weights, a distinct workflow, a license you can quote).

5. Star-count theater and borrowed trust

Reject when the pitch is mostly GitHub stars, Product Hunt badges, or “trusted by 10,000 teams” with no link to a primary source you can open.

Stars measure attention, not safety, uptime, or whether the license allows what you need. Same for download counters. They can sit in a “popularity” note. They should not be the reason a tool survives curation.

What a kept listing actually needs

The bar for keeping something is boring on purpose:

A live official URL (not a Medium mirror, not a random Notion dump).
A one-sentence job a stranger could act on.
Enough scope notes that a reader knows platform, account, or license traps before they click through — without pretending the directory audited the vendor’s security program.
A category that matches the job (coding assistant ≠ model gateway ≠ image app), so people are not scrolling a junk drawer.

That is why curated boards stay smaller than submission-fed ones. The empty slots are a feature.

WarpNav’s AI coding tools shelf is built around that filter set: Cursor, Claude Code, Aider, Windsurf, and similar tools sit there because each one has a checkable official surface and a named job — not because a form accepted another logo. The rest of the site follows the same idea across model apps, gateways, and open-source roundups: fewer rows, clearer official doors.

A short checklist before you trust any AI directory

Does the entry link to an official domain you can verify?
Can you state the tool’s job without copying the homepage slogan?
Is pricing or licensing visible without a sales call?
Are docs (or a real README) one click away?
Would removing three near-clones make the category easier to use?

If a directory cannot survive those five questions, it is a billboard with search traffic — not a map.

Coverage is easy. Saying no is the product.

When Codex says "midnight PST," what time is that on your desk?

melonape — Sun, 25 Oct 2026 12:16:59 +0000

When a Codex usage-limit reset shows up on X, the time string is usually not in your timezone.

It says things like “by midnight PST,” “before 10am PST,” or “landing by EOD.” That is fine if everyone on the team lives in California. It is less fine if one person is in Shanghai, another in Berlin, and someone else is reading the same screenshot six hours late.

This is not a /status walkthrough. Your personal 5-hour and weekly windows still live in the Codex UI. This is about the other clock: public reset announcements that teams try to plan around — and how easy those clocks are to misread.

The announcement is Pacific. Your calendar is not.

OpenAI-side Codex reset notes from Tibo often use Pacific labels: PST, “midnight,” “EOD,” “around 2pm.” Those are useful hints. They are not a world clock.

A few conversion habits that cut most of the pain:

Anchor to UTC first. Convert the Pacific time to a single UTC instant, then to each teammate’s local zone. Arguing in three local times at once is how “we thought it was already live” starts.
Treat “PST” as ambiguous until proven otherwise. Literal PST is UTC−8. Pacific local time in summer is often PDT (UTC−7). History of these announcements shows both readings show up in the wild — and they differ by one hour. One hour is enough to miss a reset window you were waiting on.
“EOD” is not a timestamp. End of day in Pacific can mean several different hours depending on who said it. Prefer a confirmation post (“all propagated,” “done”) over an EOD forecast when you need to know it landed.

If you work from China Standard Time (UTC+8), “10am PST” is not a morning event. Literal PST puts that near 02:00 the next calendar day in Beijing; a PDT reading shifts it by another hour. Remote teams that sync on Slack screenshots without naming the zone will disagree about whether the reset “should have happened already.”

Personal reset ≠ public reset

Two clocks get mashed together in Discord and Twitter threads:

Your account windows — 5-hour / weekly (and whatever else your plan shows). Check them in Codex (/status or Settings → Usage). Nobody else’s screenshot is your number.
Public / global-style resets — announced or confirmed for eligible paid users in public posts. Those are events. They are not proof that your bar moved until you verify in-product.

Banked resets are a third mechanism again: stored credit you choose to use, not the same thing as “everyone’s weekly just refilled.” Mixing those three under the word “reset” is how threads talk past each other.

One shared board beats five private conversions

The failure mode is familiar: someone pastes a Pacific time, three people convert it differently, and the channel argues about who is “right” while the useful question — has the public confirmation landed, and against which source post? — goes unanswered.

For that public clock, a small curated page helps more than another CLI one-liner. Codex Reset Watch keeps confirmed and upcoming Codex usage-limit resets on a live UTC countdown, with the original Tibo source posts, timezone notes, a verified history, and browser alerts when a new reset (or a usage-limit rule change) is confirmed on the site. It is unofficial and free to open — it does not log into your account, and the times it shows are public confirmation or observation anchors, not a claim about OpenAI’s exact backend second.

Use it as the team’s shared “has it landed?” board. Keep /status for your remaining percentage.

A short checklist before you schedule around a reset post

Does the post name a timezone, or only a city-less “tonight”?
Is it a forecast (“landing by…”) or a confirmation (“propagated,” “done”)?
Are you converting literal PST or Pacific local (PDT) — and did the whole team pick the same one?
Are you talking about a public reset, your rolling windows, or a banked card?

Get those four straight and most timezone fights disappear. The remaining work is boring on purpose: one UTC reference, one source link, one check in your own Codex UI.

Codex ate my weekly limit in a day — what people are actually doing about it

melonape — Mon, 19 Oct 2026 09:02:35 +0000

If you've been staring at your Codex usage bar and wondering whether you're insane: you're not. Over the last couple of months the shared vibe on X has been less "AI coding is magic" and more "why did half my week disappear before lunch."

I'm not dressing this up as a product review. This is what people have been posting out loud — screenshots, percentages, the messy threads. Community gossip with receipts, not OpenAI docs.

The numbers people are actually posting

Weekly limits going to zero in a single day is not rare anymore.

Several Plus users say they blew the entire weekly allotment in one day — one put it bluntly: a few lines of code and it was gone. On Pro 5x / 20x, people report burning half a week overnight, or emptying the weekly bucket in a day. One person watched ~50% of a Pro 5x weekly limit vanish in an evening.

Single-task stories are worse. A simple UI style change burned through 117% of a 20x plan without finishing. Sol·Ultra·Fast on ten tasks ate ~40% in about an hour. Codex Security on 128 files took 20% of a Plus week. Skip /compact and one person lost their 5-hour window in an hour. Ultrafast reportedly torching a $500 plan's weekly usage in about fifteen minutes also made the rounds.

Compared with earlier this summer, the same work feels more expensive. Old baseline: hammer all day, maybe lose 20–30%. New baseline: use carefully and still see half the week gone — or hear a Pro user claim ~30% in a day and say it feels worse than old Plus endurance. Newer models seem hungrier: three tasks on 6 Sol max burned ~7% of a 20x week in two hours, while 5.6 used to take a full day to hit ~10%. Others put Astra at roughly double 5.6's burn rate.

Rules that keep moving under your feet

It isn't only the burn rate. The rulebook keeps getting rewritten in public.

Late August: weekly limits felt tighter, and Tibo had to respond. Then "you can still finish the task after the limit" got yanked — thousands of likes on that complaint alone. Plus users saw the 5-hour limit return; people called it a paid trial and a double penalty. A batch of usage-eating bugs got fixed and everyone got a reset. Later, Pro $200 chatter shifted from 20x toward 10x for older accounts, with a Pro $500 tier showing up. In October, "Selected model is at capacity" appeared even with ~70% quota left.

None of that is a stable product story. You find out when your bar drops or someone quotes Tibo.

When the model starts phoning it in

Separate from the meter: people say quality got mushy. Same bug three times. Fake progress. Editing files nobody asked for. Spinning up useless tests. Official channels don't own it; support goes quiet; the community invents a "pelican test" because there's nothing better to point at.

I've also seen opposite takes — Codex dumb / web ChatGPT fine, or the reverse — plus theories about marked accounts. Pure vibes, no consensus, so I'm not turning that into advice.

Resets: the part that feels like a lottery

This is what makes people angriest.

Whether you get topped up often depends on whether Tibo posts a reset on X that day. People talk about it like waiting for a favor. There are already several sites whose whole job is "did they reset today?"

Worse: a lot of folks burn a banked reset card, then Tibo announces a global reset anyway — card wasted. Others noticed a banked card can roll your cycle forward, like taking an advance on next week. So the meta-game is ugly: if it "feels like" a reset day, slam Astra; if not, drip-feed 5.6 Sol; save the card until after a natural reset; or refuse to open it and wait for Tibo.

Yes, that is a ridiculous sentence about a paid coding tool. Welcome to 2026.

What people keep repeating (not invented tips)

Patterns other users keep recommending to each other — not a blessing from documentation.

1. Plan in web ChatGPT chat; let Codex only execute.

Regular ChatGPT chat and Codex aren't the same usage pool for a lot of people. Argue the design in the browser, then hand Codex the "change the code / run the commands" part. Same idea inside Codex: Astra or Sol for the plan, Luna (or a Sol sub-agent) for the grind. Counter-anecdote: someone said even web-only planning still killed their 5-hour window.

2. Stop flooring the reasoning dial.

Same tiny 3D game on Astra: Max chewed ~4% of a Pro 5x week in 53 minutes; Medium did ~1% in 25 minutes with similar results. Plenty of people now default to Medium and turn Fast off unless they're in a hurry.

3. Fall back to GPT-5.6 Sol when the new stuff feels wrong.

Complaints about 6.1 Sol being slow or test-happy, and Astra overcomplicating simple asks, keep showing up. The three-task / 7% vs full-day / 10% comparison above is why some people stay on older Sol.

4. Don't let sessions rot.

Past ~150k tokens, people say it gets dumb. Compact, or make it write a handoff doc and start a fresh thread. New task, new thread. Don't swap models mid-conversation.

5. Keep Claude around — carefully.

A chunk bounced back to Claude when Codex felt clogged ("like constipation clearing instantly," someone wrote). Others got Claude-banned and came back to Codex. Dual-wielding is common: Codex by default, Claude when stuck. Mileage varies; bans are real.

I'm skipping the grey stuff — account farms, reverse proxies, gray-market seats, third-party bridges that pipe web ChatGPT quota into Codex. Not going there.

So what do I do about the reset circus?

The banked-card-then-global-reset pattern is what made me stop refreshing Tibo's timeline like a sports score. I put together a small page that watches Tibo's Codex reset posts (and usage-limit rule-change notes that land the same way), shows times in Beijing time with the original timezone notes, keeps a verified history, and can push a browser alert when a new reset or rule change shows up: https://crw.warpnav.com/.

It won't make your quota last longer. It just means the next "Done." from Tibo doesn't have to find you by accident.

If your bar is already on fire: you're in crowded company. Steal the workflows above, ignore anything that sounds like a hustle, and maybe stop opening the banked card five minutes before a global reset lands.