Differentiating expert and novice cognitive structures

A seminal study by Chi et al. firmly established the paradigm that novices categorize physics problems by "surface features" (e.g. "incline," "pendulum," "projectile motion," ... ), while experts use "deep structure" (e.g. "energy conservation," "Newton 2," ... ). Yet, efforts to replicate the study frequently fail, since the ability to distinguish experts from novices is highly sensitive to the problem set being used. But what properties of problems are most important in problem sets that discriminate experts from novices in a measurable way? To answer this question, we studied the categorizations by known physics experts and novices using a large, diverse set of problems, in order to subsequently study how well these two groups can be discriminated using small subsets. Having a large initial set allowed us to form a large number of smaller subsets and study their properties. We found that the number of questions required to accurately classify experts and novices could be surprisingly small so long as the problem set is carefully crafted to be composed of problems with particular pedagogical and contextual features.