Item desirability matching in forced-choice test construction

The forced-choice method has been proposed as a viable strategy to prevent socially desirable responding (SDR) on self-report non-cognitive measures. The ability of the method to eliminate SDR stems from matching items that are perceived as equally desirable into forced-choice item-blocks. The gold standard in quantifying similarity between items in terms of desirability has been the "mean difference index", that is, the absolute difference between items' mean desirability ratings. This index relies on the assumption that items have one true desirability value, as efficiently and unbiasedly estimated by their respective means, and may fail if this assumption does not hold. To circumvent this issue, we propose indexing similarity between items in terms of desirability with several robust measures of absolute agreement (i.e., inter-item agreement indices). Using an empirical example, we show that relying on the mean difference index may lead to suboptimal forced-choice item-block assembly by matching items with a relatively poor inter-item agreement with respect to desirability. R code for computing the proposed agreement indices on a set of desirability ratings is provided, as are recommendations for applied researchers.