Multivariate Structural Analysis of Agreement for Categorical Data

A statistical model is proposed for a multi-dimensional agreement matrix tabulating nominal variables and judgments of two raters in order to analyze the structure of agreement. The model is derived from the notion of the measurement error model in test theory, and it is expressed as a tree with a single divergence. The model is composed of probabilities of agreement, a true score and errors of raters as parameters. It is shown that the parameters can be estimated by the usual maximum likelihood approach and the agreement probabilities are reliability measures. The model is extended to apply to cases of more than three variables and three raters. Partial modifications to the model are discussed for ordered categorical variables and the consistency matrix generated by two sets of answers to a questionnaire. An application example is presented for the consistency matrix.