Simple Structural Information to Optimize Neural Network Architectures

Abstract

In this paper, new information called structural information is proposed. Structural information is composed of two types of information: The first order information and the second order information. The first and second order information represent information on deviation from the equiprobable distribution and the independence respectively. We have so far dealt with total information to be stored in neural networks. However, by introducing structural information, we can control appropriate types of information, depending on methods and problems. In other words, we can control information, taking into account the quality as well as the quantity of information. We applied the structural information to information content in input-hidden connections and hidden units. Then is was applied to XOR problem to show how the structural information control affects the simplification of network architectures. In addition, we applied the methods to language acquisition problems complex enough to test the performance. Experimental results confirmed that generalization is not concerned with total information but with the second order information.