Survey of the Smoothing Issues on Mandarin Language Models

Abstract

We survey several frequent smoothing methods used by language models for Mandarin. Due to the problem of data sparseness, smoothing techniques are employed to re-estimate the probability for all events while calculating the probability of occurrence. Among well-known smoothing methods, Good-Turing is employed widely. We have proposed a set of properties to analyze the behaviors of Good-Turing in this paper. Two novel smoothing methods are proposed. Finally, we implement three n-gram for Mandarin and then analyze the entropy and related problems of the Good-Turing; such as cut-off value and types of events .