← Back to Publications
Authors:
Introduces a “transition freedom” metric for unsupervised tokenization that outperforms statistical metrics such as mutual information and conditional probability, reaching F-measure 0.71–1.0 across multilingual corpora, with different metric offshoots needed per language. It finds that compressing models by eliminating statistically weak evidence improves tokenization, and that larger training corpora do not necessarily help.
An unsupervised-language-learning source extending the Link-Grammar / Vepstas–Goertzel interpretable grammar-induction line into tokenization — relevant to the unsupervised-learning pipeline of the Semantic Parsing Deep Dive.