The subskills of reading: Rule-space analysis of a multiple-choice test of second language reading comprehension

The rule-space methodology is an application of statistical pattern-recognition techniques to diagnosing the cognitive attributes (knowledge, skills, abilities, strategies, etc.) underlying test performance. The methodology provides diagnostic information on individual test-takers on each of these attributes. We used the methodology to analyze performance on a multiple-choice reading comprehension test. Based on a literature search, me hypothesized a set of 27 potentially important attributes. We carried out 4 rule-space analyses of test results for 5,000 Japanese who had taken the reading section of the TOEIC. We modified the set of attributes, deleted attributes, and identified interactions to arrive at the most parsimonious set with the maximum explanatory power. The final analysis contained 16 prime attributes, with 8 interactions. Using these attributes, we could classify 91% of test-takers into their latent knowledge states, and give scores on each attribute (as mastery probabilities). We then regressed total test scores onto these attribute probability scores using multiple regression; the adjusted R-square was .97, indicating that the attributes explained 97% of the variance in total test performance for those 91% of test-takers successfully classified. This suggests that the rule-space methodology can explain performance on complex verbal tasks, such as reading tests, and provide diagnostic scores to test-takers.