Reliability of a probabilistic knowledge structure

Indexes for estimating the overall reliability of a test in the framework of knowledge space theory (KST) are proposed and analyzed. First, the possibility of applying in KST the existing classical test theory (CTT) methods, based on the ratio between the true score variance and the total variance of the measure, has been explored. However, these methods are not suitable because in KST error and true score are not independent. Therefore, two new indexes based on the concepts of entropy and conditional entropy are developed. One index is used to estimate the reliability of the response pattern given the knowledge state, while the second one refers to the reliability of the estimated knowledge state of a person. Some theoretical considerations as well as simulations and an empirical example on real data are provided within a study of the behavior of these indexes under a certain number of different conditions.