A Comparison of the Ability Estimations of Classical Test Theory and the Many Facet Rasch Model in Measurements with Open-ended Questions

The purpose of this study is to compare the ability estimations of classical test theory (CTT) and the Many Facet Rasch model (MFRM) in measurements conducted with open-ended questions. The study was conducted with 100 eighth graders and four mathematics teachers who rated the students' work. The study's data were obtained using an achievement test with 6 open-ended mathematics questions and a holistic rubric for scoring these questions. The data obtained by rating the open-ended mathematics questions were analyzed using both CTT and MFRM. The ability estimations for CTT were calculated using the score averages of the four raters. Afterwards, Many Facet Rasch analysis was conducted using a three-facet pattern including raters, students and items. The ability estimations calculated by these two theories were prepared to be compared by converting ability estimations obtained from the Rasch analysis and reported in the logit scale into the units of rubric used for scoring. This study found that the relative agreement between the ability estimations calculated using CTT and MFRM was extremely high. It was determined that there was a significant difference between the means of the ability estimations of the two theories, and thus no absolute agreement. Research findings revealed that the concurrent validity of the ability estimations of MFRM is higher than that of CTT's ability estimations.