CFS-based Unigram Language Model, a Brave New Approach to Traditional N-gram Language Models

Abstract

This paper introduces a new concept, the Chinese frequent strings (CFS) based unigram language model, wh ich is in many respects superior to the traditional language model (LM). Important properties of CFSs and applications in Chinese natural language processing (NLP) will be revealed in this paper. We have proposed a methodology for extracting Chinese frequent strings, which contain unknown words, from a Chinese corpus. We found that CFSs contain many 4-gram characters, 3-gram words, and higher n-grams. Such information can only be derived with an extremely large corpus in a traditional language model. In contrast to using a traditional LM, we can achieve high precision and efficiency by using CFSs to solve Chinese toneless phoneme-to-character conversion and to correct Chinese spelling errors with a small training corpus. An accuracy of 92.86% was achieved for Chinese toneless phoneme-to-character conversion. An accuracy of 87.32% was achieved for Chinese spelling error correction. We used a traditional lexicon, namely the ASCED (Academia Sinica Chinese Electronic Dictionary) provided by Academia Sinica, Taiwan, and the word bigram language model to solve the two abovementioned problems. We achieved accuracies of 66.9% and 80.95% respectively for Chinese toneless phoneme-to-character conversion and Chinese spelling error correction.