Problem Statement: Reliability is considered the weakest ring when measuring students' achievement through open-ended questions. Despite being an important source of errors that reduce reliability in rating responses given to questions, such factors as tasks or items are other sources of error that are equally important. However, all the methods employed in computing reliability are not handled with all sources of error at the same time. The reliability of measurements conducted with open-ended questions is studied through methods based on three basic theories of measurement: namely, classical test theory, item response theory, and generalizability theory. Purpose of Study: The purpose of the study is to apply classical test theory (CTT) and many facet Rasch model (MFRM) to determine the reliability of the mathematic achievement scores and to compare the results of both theories. Methods: Since the characteristics of CTT and MFRM are discussed and confirmed in this study, this is a descriptive study. Findings and Results: According to CTT, the interconsistency of the mathematic scores was found to be 0.92. Although Kendall's concordance coefficient for four raters was obtained as 0.52, correlation coefficients for four raters were different values between 0.90 and 0.97. According to MFRM, the reliability of the person facet was 0.95, and the reliability of the rater facet was 0.99. For determining the students' mathematic success, the reliability of the mathematic scores was found to be very high. Although there was a difference between the means of the raters' scores, it was determined that the four raters scored the students consistently. Conclusions and Recommendations: With this study, it was seen that the theory to be selected for determining the reliability of the scores depended upon the purpose for which the scores obtained would be used. However, it is concluded that it is more appropriate that at least two theories should be used for determining the reliability of the measurement.
