Experimental Study of Ergodic Learning Curve in Hidden Markov Models

Abstract

A number of learning machines used in information science are not regular, but rather singular, because they are non-identifiable and their Fisher information matrices are singular. Even for singular learning machines, the learning theory was developed for the case in which training samples are independent. However, if training samples have time-dependency, then learning theory is not yet established. In the present paper, we define an ergodic generalization error for a time-dependent sequence and study its behavior experimentally in hidden Markov models. The ergodic generalization error is clarified to be inversely proportional to the number of training samples, but the learning coefficient depends strongly on time-dependency.