Link Search Menu Expand Document


A short sequence of characters

A token is a short sequence of characters with meaning, most typically a word or punctuation character. Strings are split into tokens by tokenisation algorithms.

Examples: this, sequence, ?


Edit this article β†’

Machine Translate is created and edited by contributors like you!

Learn more about contributing β†’

Licensed under CC-BY-SA-4.0.

Cite this article β†’