Generalisability of a composite student selection programme

The reliability of individual non-cognitive admission criteria in medical education is controversial. Nonetheless, non-cognitive admission criteria appear to be widely used in selection to medicine to supplement the grades of qualifying examinations. However, very few studies have examined the overall test generalisability of composites of non-cognitive admission variables in medical education. We examined the generalisability of a composite process for selection to medicine, consisting of four variables: qualifications (application form information); written motivation (in essay format); general knowledge (multiple-choice test), and a semi-structured admission interview. The aim of this study was to estimate the generalisability of a composite selection. Data from 307 applicants who participated in the admission to medicine in 2007 were available for analysis. Each admission parameter was double-scored using two random, blinded and independent raters. Variance components for applicant, rater and residual effects were estimated for a mixed model with the restricted maximum likelihood (REML) method. The reliability of obtained applicant ranks (G coefficients) was calculated for individual admission criteria and for composite admission procedures. A pre-selection procedure combining qualification and motivation scores showed insufficient generalisability (G = 0.45). The written motivation in particular, displayed low generalisability (G = 0.10). Good generalisability was found for the admission interview (G = 0.86), and for the final composite selection procedure (G = 0.82). This study revealed good generalisability of a composite selection, but indicated that the application, composition and weighting of individual admission variables should not be random. Knowledge of variance components and generalisability of individual admission variables permits evidence-based decisions on optimal selection strategies.