A Novel Concept to Improve the Redistribution Process for Language Models

Abstract

In the paper, a new concept, based on the nonuniform redistribution probability for novel events, to improve the smoothing method in language models is proposed. Basically, there are two processes in the smoothing methods: 1)discounting and 2)redistributing. Instead of uniform probability assignment to each unseen events used by most smoothing methods, we propose new technique to improve the redistribution process. Referring to the probabilistic behavior of all seen events, the redistribution process for novel events in our method is non-uniform. The proposed technique is exploited on well-known and frequently-used Good-Turing smoothing method. The empirical results are demonstrated and analyzed for two n-gram models. The improvement is obvious and effective for smoothing methods, especially on higher unseen event rate.