DEV Community

CharlonTank
CharlonTank

Posted on

tokencount: Fast GPT token counting CLI

#rust #cli #ai #productivity

tokencount is a new Rust CLI that helps you answer a deceptively simple question: how many GPT tokens are hidden across this project? If you build AI features, write prompt-heavy docs, or just keep an eye on context windows, this tool makes the audit painless.

Why I built it

Most token counters either run one file at a time or ignore the filesystem realities of big projects. I wanted something that:

  • Walks a codebase quickly (parallel rayon workers + OS-native ignore rules)
  • Respects .gitignore by default and lets me layer custom --exclude globs
  • Talks the same language as OpenAI models (cl100k_base, o200k_base, etc.)
  • Gives a useful summary out of the box: per-file counts, totals, percentiles, and top-N offenders
  • Plays nicely with automation (JSON and NDJSON streaming modes)

Features at a glance

  • Blazing fast scan – ignore::WalkBuilder + Rayon for concurrent IO/tokenization
  • Smart defaults – only scans *.elm unless you add --include-ext flags (good for Elm-heavy repos)
  • Flexible filtering – combine --include-ext, --exclude, --max-bytes, and --follow-symlinks
  • Multiple outputs – table, JSON array with summary, or NDJSON stream for pipelines
  • Rich stats – totals, average per file, and P50/P90/P99 percentiles to spot outliers fast
  • Quiet/verbose modes – keep CI logs clean or turn on detailed warnings locally

Install 

cargo install tokencount
Enter fullscreen mode Exit fullscreen mode

Quick tour 

# default: scan current directory, only *.elm files, table output
tokencount

# include Elm + TypeScript
tokencount ./frontend --include-ext elm --include-ext ts

# show top 10 largest files by tokens
tokencount --top 10

# machine-readable summary for CI
tokencount --format json > tokens.json

# streaming counts for further processing
tokencount --format ndjson

# sort descending by token count
tokencount --sort tokens
Enter fullscreen mode Exit fullscreen mode

Each run ends with a footer like this:

---
total files: 42
total tokens: 128730
average/file: 3065.00
p50: 812
p90: 7194
p99: 24403
Enter fullscreen mode Exit fullscreen mode

Need only the top offenders? Combine --top N with either --sort tokens or the default path sort.

Under the hood

  • Ignore handling uses the ignore crate with .gitignore, .git/info/exclude, and global git ignores respected automatically. I add common junk folders (node_modules, target, .git) so you don’t have to.
  • Tokenization relies on tiktoken-rs, so you get the same counts as OpenAI’s cl100k_base/o200k_base models.
  • Error handling is friendly by default—non UTF-8 files or oversized blobs are skipped with warnings (or silently with --quiet).
  • Percentiles use a nearest-rank approach and degrade gracefully when there are zero files.

Roadmap & feedback

I’m exploring:

  • More encodings (if you need a different tokenizer, open an issue)
  • Optional HTML/Markdown report outputs
  • Built-in file size histogram to complement token stats

Repo & issues live here: github.com/CharlonTank/tokencount

If you try tokencount, I’d love to hear how it fits into your prompt engineering workflow or CI pipelines—reach out in the repo or drop a comment below.

Top comments (0)