A short sequence of characters

A token is a short sequence of characters with meaning, most typically a word or punctuation character. Strings are split into tokens by tokenisation algorithms.

Examples: this, sequence, ?

Want to learn more about Token?

Edit this article →

Machine Translate is created and edited by contributors like you!

Learn more about contributing →

Licensed under CC-BY-SA-4.0.

Cite this article →