An unfamiliar surname may split into several tokens rather than count as one word.
A tokenizer determines the units and maps them to IDs. Different models can use different tokenizers, so the same paragraph need not produce the same count. Tokens also include special markers used to structure some conversations. Hugging Face documents tokenization.
Do not use “one token equals one word” when planning a long document. Code, punctuation and different languages can change the relationship. Use the chosen model's tokenizer or the application's count when an exact limit matters.
Input tokens are material supplied to the model. Output tokens are what it generates. In a chat, earlier messages may be supplied again as input. When a reply is cut short, check the output limit as well as the overall context. Removing a few visible words may make little difference if the application sends a large history.
Sources
Primary references checked 11 October 2026. This explanation is not a benchmark of a particular model or computer.