Assessment of multiple choice question exams quality using graphical methods

Exams should be valid, reliable, and discriminative. Multiple informative methods are used for exam analysis. Displaying analysis results numerically, however, may not be easily comprehended. Using graphical analysis tools could be better for the perception of analysis results. Two such methods were employed: standardized x-bar control charts with standard error of measurement as control limits and receiver operator characteristic curves. Exams of two medical classes were analyzed. For each exam, the mean, standard deviation, reliability, and standard error of measurement were calculated. The means were standardized and plotted against the reference lines of the control chart. The means were chosen as cut-off points to calculate sensitivity and specificity. The receiver operator characteristic curve was plotted and area under the curve determined. Standardized control charts allowed clear, simultaneous comparison of multiple exams. Calculating the control limits from the standard error of measurement created acceptable limits of variability in which the standard deviation and reliability were incorporated. The receiver operator characteristic curve graphically showed the discriminative power of the exam. Observations made with the graphical and classical methods were consistent. Using graphical methods to analyse exams could make their interpretation more accessible and the identification of exams that required further investigation easier. Practitioner Notes 1. Exams should be valid, reliable, and discriminative 2. Classical methods to analyze exam quality represent data numerically 3. Numerical representation of data may not be readily understood by department staff 4. Graphical methods to analyze exam represent data in easy-to-understand charts 5. Control charts and receiver operator characteristic curves can be employed for such purposes