We hear a lot about tokens in AI these days, in the form of "ChatGPT tokens", "OpenAI tokens" when discussing models like ChatGPT, BERT, and other ...
For further actions, you may consider blocking this person and/or reporting abuse
That’s a nice introduction to tokens in relation to current AI models and LLMs.
Are tokens always just words in the LLM? Or, can a token be a phrase (multiple words)? Also I’m guessing that symbols like +,=,&, could also be tokens that the LLM uses in its attempts to understand the meaning. Is that true? Just curious.
Hey, if you get a chance, would you mind reading my latest article on dev.to and commenting? Software Developer, Are You Just A Hammer?
A token can be as long as a word for an English like language, but for languages like chinese the concept of words separated by spaces doesn't exist and hence the tokenization strategy is difference.
Yes, symbols are considered tokens and hold specific meaning in each context.