← Back to Publications

Unsupervised Tokenization Learning

Authors:

Year: 2022
Venue: arXiv:2205.11443
Links: arXiv abstract

Summary

Introduces a “transition freedom” metric for unsupervised tokenization that outperforms statistical metrics such as mutual information and conditional probability, reaching F-measure 0.71–1.0 across multilingual corpora, with different metric offshoots needed per language. It finds that compressing models by eliminating statistically weak evidence improves tokenization, and that larger training corpora do not necessarily help.

Relevance to Hyperon

An unsupervised-language-learning source extending the Link-Grammar / Vepstas–Goertzel interpretable grammar-induction line into tokenization — relevant to the unsupervised-learning pipeline of the Semantic Parsing Deep Dive.

Key References

• RawData source
• Semantic Parsing Deep Dive