C1 noun ˌtoʊ.kə.nɪˈzeɪ.ʃən
tokenization
Meaning
The process of breaking text into smaller units like words, phrases, or symbols for analysis.
Example
NLP uses tokenization to break text into smaller units.