The bite-sized chunks that language models process text in. Not everything is a full word – ‘uncomfortable’ might be several tokens, while ‘antidisestablishmentarianism’ is apparently just showing off.
Synonyms:
Text Tokens
« Back to Glossary Index
« Back to Glossary Index