Token

A short sequence of characters


A token is a short sequence of characters with meaning, most typically a word or punctuation character. Strings are split into tokens by tokenisation algorithms.

Examples: this, sequence, ?


Want to learn more about Token?


Edit this article →

Machine Translate is created and edited by contributors like you!

Learn more about contributing →

Licensed under CC-BY-SA-4.0.

Cite this article →