Statistical inference for the nonparametric and semiparametric hidden Markov model via the composite likelihood approach

In this paper, we propose a new estimation method for a nonparametric hidden Markov model (HMM), in which both the emission model and the transition matrix are nonparametric, and a semiparametric HMM, in which the transition matrix is parametric while emission models are nonparametric. The estimation is based on a novel composite likelihood method, where the pairs of consecutive observations are treated as independent bivariate random variables. Therefore, the model is transformed into a mixture model, and a modified expectation-maximization (EM) algorithm is developed to compute the maximum composite likelihood. We systematically study asymptotic properties for both the nonparametric HMM and the semiparametric HMM. We also propose a generalized likelihood ratio test to choose between the nonparametric HMM and the semiparametric HMM. We derive the asymptotic distribution and prove the Wilk's phenomenon of the proposed test statistics. Simulation studies and an application in volatility clustering analysis of the volatility index in the Chicago Board Options Exchange (CBOE) are conducted to demonstrate the effectiveness of the proposed methods.