In AI language models, what is a "token"?

A token is the basic unit an AI model uses to process text, and it isn't always a whole word — common words might be a single token, while longer words get broken into smaller pieces, and even punctuation can count as its own token. Because models read and generate text token by token, both their context limit and usage cost are measured in token counts rather than letters or whole words.

A digital security key and cryptocurrency both also use the word "token," but describe completely different concepts from unrelated fields, namely authentication and digital currency.

Because different languages or phrasings of the same idea use different numbers of tokens, the same sentence translated or written more verbosely can end up costing noticeably more to process.