Assessments are commonly used to make a decision about an individual, such as grade placement, treatment assignment, job selection, or to inform a diagnosis. A psychometric approach to classify respondents based on the assessment would aggregate items into a score, and then each respondent's score is compared to a cut score. In contrast, a machine learning approach to classify respondents would build a model to predict the probability of belonging to a specific class from assessment items, and then respondents are classified based on their predicted probability of belonging to that class. It remains unclear whether psychometric and machine learning methods have comparable classification accuracy or if 1 method is preferable in all or some situations. In the context of diagnostic assessment, this study used Monte Carlo simulation methods to compare the classification accuracy of psychometric and machine learning methods as a function of the diagnosis-test correlation, prevalence, sample size, and the structure of the diagnostic assessment. Results suggest that machine learning models using logistic regression or random forest could have comparable classification accuracy to the psychometric methods using estimated item response theory scores. Therefore, machine learning models could provide a viable alternative for classification when psychometric methods are not feasible. Methods are illustrated with an empirical example predicting an oppositional defiant disorder diagnosis from a behavior disorders scale in children of age seven. Strengths and limitations for each of the methods are examined, and the overlap between the field of machine learning and psychometrics is discussed. Translational Abstract Assessments and tests are often used to make decisions about an individual, such as deciding who will graduate from high school, who gets hired for a job, and who is referred to more services or treatment. In these tests or assessments, individuals are asked to respond to a series of items, and depending on their item responses, a decision about the individual is made. Traditionally, researchers and assessment specialists have used a psychometric approach to score item responses and then check if the score is above a cut score. However, data-driven, exploratory methods from the area of machine learning could be used for similar purposes. For example, machine learning methods could be used to predict the assessment decision directly from the item responses, without the need to aggregate item responses beforehand. It remains unclear if the classification accuracy of machine learning methods is comparable with the psychometric methods typically used for classification. This study used Monte Carlo simulation methods to study classification accuracy of psychometric and machine learning methods across a variety of assessment conditions, among which are the number of items and response categories of the assessment, the strength of the relationship between the diagnosis and the assessment, sample size, and diagnosis prevalence. The methods were also illustrated in an empirical example predicting an oppositional defiant disorder diagnosis in children of age 7. Strengths and limitations for each of the methods are examined, and the overlap between the field of machine learning and psychometrics is discussed.
