C1 noun ˌtoʊ.kə.nɪˈzeɪ.ʃən
tokenization
将文本分解为较小的单位,如单词、短语或符号进行分析的过程。
Meaning
The process of breaking text into smaller units like words, phrases, or symbols for analysis.
Example
NLP uses tokenization to break text into smaller units.
NLP使用分词法将文本分解为更小的单位。