Validating items of different modalities to assess the educational technology competency of pre-service teachers

With increasingly more educational tests being administered on computers and other digital devices, the comprehensive validity of tests has more potential to be improved. In the present investigation, 75 items were developed to measure educational technology competency (ETC) for pre-service teachers according to a framework constructed using the Delphi technique. In particular, the appropriateness of items was tested using the multitrait-multimethod (MTMM), higher-order factor model, and multidimensional item response theory (MIRT) to shape a program for developing and validating different modality items. The results were as follows: (1) There was robust convergent validity, weak discriminant validity, and an evident method effect of the items. (2) The graphic modality might explain more variance than the text and video modalities at discriminating examinees' ETC proficiencies. (3) The graphic modality was the most reliable modality among the three different modalities when assessing technology-supported teaching. Thus, this study makes a significant contribution to the design of assessments by developing and validating different modality items. The results could be generalized to some extent when items are developed based on other ETC standards.