Comparing Estimated and Real Item Difficulty Using Multi-Facet Rasch Analysis *

This study aimed to compare estimated item difficulty based on expert opinion with real item difficulty based on data by utilizing Rasch analysis. For security reasons, some high-stakes tests are not pre-tested and item difficulty is estimated by teachers in classroom assessments, so it is necessary to examine the extent to which experts make accurate predictions. In this study, we developed a 12-item test in the field of measurement and evaluation similar to those used in the Public Personnel Selection Exam. Item difficulty was estimated and compared separately based on 1165 student responses and the opinions of 12 experts. A multi-facet Rasch analysis was conducted to examine the effects of raters on the test scores. The study revealed that the experts had a good ability to estimate item difficulty for items of moderate difficulty. However, they tended to underestimate item difficulty for items.