Token
A short sequence of characters
A token is a short sequence of characters with meaning, most typically a word or punctuation character. Strings are split into tokens by tokenisation algorithms.
Examples: this, sequence, ?
A token is a short sequence of characters with meaning, most typically a word or punctuation character. Strings are split into tokens by tokenisation algorithms.
Examples: this, sequence, ?