<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: iLostCount</title>
    <description>The latest articles on DEV Community by iLostCount (@ilostcount).</description>
    <link>https://dev.to/ilostcount</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4110729%2F51ef02b9-09e5-44b0-8fe4-07b7100d1659.png</url>
      <title>DEV Community: iLostCount</title>
      <link>https://dev.to/ilostcount</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ilostcount"/>
    <language>en</language>
    <item>
      <title>How many tokens is 1,000 words? A conversion cheat sheet for LLM prompts</title>
      <dc:creator>iLostCount</dc:creator>
      <pubDate>Sat, 19 Sep 2026 00:19:02 +0000</pubDate>
      <link>https://dev.to/ilostcount/how-many-tokens-is-1000-words-a-conversion-cheat-sheet-for-llm-prompts-3n17</link>
      <guid>https://dev.to/ilostcount/how-many-tokens-is-1000-words-a-conversion-cheat-sheet-for-llm-prompts-3n17</guid>
      <description>&lt;p&gt;If you only want the number: &lt;strong&gt;1,000 words of ordinary English is roughly 1,300 tokens.&lt;/strong&gt; Going the other way, &lt;strong&gt;1,000 tokens is roughly 750 words, or about 4,000 characters.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That is the whole answer for estimating. The rest of this post is the table, the cases where the ratio breaks, and how to get an exact count when an estimate is not good enough.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cheat sheet
&lt;/h2&gt;

&lt;p&gt;For plain English prose, using the common rule of thumb that 1 token is about 4 characters and about 0.75 words:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;You have&lt;/th&gt;
&lt;th&gt;Roughly this many tokens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1 word&lt;/td&gt;
&lt;td&gt;1.3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100 words&lt;/td&gt;
&lt;td&gt;130&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;500 words (about 1 page)&lt;/td&gt;
&lt;td&gt;650&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1,000 words&lt;/td&gt;
&lt;td&gt;1,300&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10 pages&lt;/td&gt;
&lt;td&gt;6,500&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100 characters&lt;/td&gt;
&lt;td&gt;25&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1,000 characters&lt;/td&gt;
&lt;td&gt;250&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1 paragraph (about 100 words)&lt;/td&gt;
&lt;td&gt;130&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;And in reverse, which is the direction you usually need when you are staring at a model's context limit:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Token budget&lt;/th&gt;
&lt;th&gt;Roughly this much English&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1,000 tokens&lt;/td&gt;
&lt;td&gt;750 words&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4,000 tokens&lt;/td&gt;
&lt;td&gt;3,000 words&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8,000 tokens&lt;/td&gt;
&lt;td&gt;6,000 words&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;128,000 tokens&lt;/td&gt;
&lt;td&gt;96,000 words (a short novel)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Where the ratio stops working
&lt;/h2&gt;

&lt;p&gt;Those numbers are for prose. Tokenizers split on statistical frequency, not on words, so anything unusual costs more:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Code, JSON and long URLs&lt;/strong&gt; break into many small pieces. Punctuation, braces, camelCase and random ID strings all tokenize badly. Budget noticeably more than the prose ratio.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Non-Latin scripts&lt;/strong&gt; are the big one. Arabic, Chinese, Japanese, Hindi and others can run several tokens per character in some tokenizers, so the same meaning costs multiples of the English price.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Numbers and tables&lt;/strong&gt; split in ways that look arbitrary. A long column of figures is not cheap.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Different models, different tokenizers.&lt;/strong&gt; A prompt is not a fixed number of tokens; it is a number per tokenizer.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So: rules of thumb are fine for "will this roughly fit". They are not fine for a hard limit or a cost estimate you are going to rely on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Getting an exact count in code
&lt;/h2&gt;

&lt;p&gt;For OpenAI models, &lt;code&gt;tiktoken&lt;/code&gt; is the direct route:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;tiktoken&lt;/span&gt;

&lt;span class="n"&gt;enc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tiktoken&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_encoding&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;o200k_base&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;your prompt here&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Anthropic and Google both expose token-counting endpoints in their APIs, which is the better option when you want the count for the exact model you are about to call rather than an approximation of it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Getting an exact count without writing anything
&lt;/h2&gt;

&lt;p&gt;Most of the time you are not in a script. You have a block of text in front of you and you want to know whether it fits before you paste it. That is the case I built &lt;a href="https://ilostcount.com" rel="noopener noreferrer"&gt;iLostCount&lt;/a&gt; for: paste the text, read the token, word and character counts as you type. No signup, and the text is not uploaded anywhere, since the counting runs in the page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the number is worth knowing
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The context window is shared.&lt;/strong&gt; It has to hold your system prompt, the conversation so far, anything you retrieved, and the answer. Fill the input and there is no room left for the output.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You pay per token, both directions.&lt;/strong&gt; Re-sending a large document on every turn is the quiet way a bill grows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Truncation is silent.&lt;/strong&gt; Plenty of tools drop the oldest turns when you run over the limit. That does not look like an error. It looks like the model forgetting.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A cheap habit that saves all three: count a long document before it goes into a prompt. If a 40-page PDF turns into 30,000 tokens, you know to chunk it or summarise it first, instead of finding out from a 400 error or an invoice.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Disclosure: this post is from the iLostCount project. The tool is free and the source is public at &lt;a href="https://github.com/ahmad-almazeedi/token-counter" rel="noopener noreferrer"&gt;github.com/ahmad-almazeedi/token-counter&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>beginners</category>
      <category>python</category>
    </item>
    <item>
      <title>How to count the tokens in an LLM prompt (and why the number matters)</title>
      <dc:creator>iLostCount</dc:creator>
      <pubDate>Sat, 05 Sep 2026 06:10:25 +0000</pubDate>
      <link>https://dev.to/ilostcount/how-to-count-the-tokens-in-an-llm-prompt-and-why-the-number-matters-4j1m</link>
      <guid>https://dev.to/ilostcount/how-to-count-the-tokens-in-an-llm-prompt-and-why-the-number-matters-4j1m</guid>
      <description>&lt;p&gt;If a prompt gets rejected as too long, or an API bill looks bigger than expected, the number that matters is tokens, not words. Here is the short version of how to check one.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a token is
&lt;/h2&gt;

&lt;p&gt;Models do not read characters or words. They read tokens: chunks of text that a tokenizer splits your input into. For ordinary English prose, a useful rule of thumb is:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;1 token is roughly 4 characters&lt;/li&gt;
&lt;li&gt;1 token is roughly 0.75 words&lt;/li&gt;
&lt;li&gt;1,000 words is roughly 1,300 tokens&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those ratios shift. Code, JSON and long URLs tokenize worse than prose, because punctuation and odd substrings break into more pieces. Non-Latin scripts such as Arabic, Chinese and Japanese can cost several tokens per character in some tokenizers. Rules of thumb are fine for estimating and useless for a hard limit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Counting in code
&lt;/h2&gt;

&lt;p&gt;For OpenAI models, &lt;code&gt;tiktoken&lt;/code&gt; gives an exact count:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;tiktoken&lt;/span&gt;

&lt;span class="n"&gt;enc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tiktoken&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_encoding&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;o200k_base&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;enc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;your prompt here&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Anthropic and Google both expose token-counting endpoints in their APIs, so you can measure a prompt against the exact model you are calling instead of an approximation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Counting without writing code
&lt;/h2&gt;

&lt;p&gt;Most of the time you are not writing a script. You pasted something into a chat window and you want to know whether it fits. That is what we built &lt;a href="https://ilostcount.com" rel="noopener noreferrer"&gt;iLostCount&lt;/a&gt; for: paste text, read the token, word and character counts as you type. No signup, and nothing is uploaded, since it runs in the page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the number matters
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Context window.&lt;/strong&gt; The window has to hold your system prompt, the conversation so far, any retrieved documents, and the answer. If the input fills the window, there is no room left for the output.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost.&lt;/strong&gt; Input and output are both billed per token, so a prompt that re-sends a large document on every turn adds up quietly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Truncation.&lt;/strong&gt; Some tools silently drop the oldest turns when you run over. That looks like the model forgetting, not like an error.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A cheap habit: count a big document before you paste it into a prompt. If a 40-page PDF turns into 30,000 tokens, you know to chunk or summarise it first.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Disclosure: this post is from the iLostCount project. The tool is free, and the source is public at &lt;a href="https://github.com/ahmad-almazeedi/token-counter" rel="noopener noreferrer"&gt;github.com/ahmad-almazeedi/token-counter&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>webdev</category>
      <category>beginners</category>
    </item>
  </channel>
</rss>
