Link Search Menu Expand Document

Token

A short sequence of characters


A token is a short sequence of characters with meaning, most typically a word or punctuation character. Strings are split into tokens by tokenisation algorithms.

Examples: this, sequence, ?


↑

Edit this article β†’

Machine Translate is created and edited by contributors like you!

Learn more about contributing β†’

Licensed under CC-BY-SA-4.0.

Cite this article β†’