Your diagram of the tokens and nodes is not quite right. The tokenizer doesn't form characters into words and the nodes are not broken into words. So the diagram should look like this:
You are right, for each letter of the word, a token is emitted (I'll correct the diagram a bit later). The node representation was an oversight. Thank you for taking the time to modify the diagram :).
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Your diagram of the tokens and nodes is not quite right. The tokenizer doesn't form characters into words and the nodes are not broken into words. So the diagram should look like this:
You are right, for each letter of the word, a token is emitted (I'll correct the diagram a bit later). The node representation was an oversight. Thank you for taking the time to modify the diagram :).