Reliability and Consensus of Experienced Wine Judges: Expertise Within and Between?

This paper considers the levels of reliability and consensus of wine quality judgments found in studies of experienced wine judges. Both reliability, which concerns the similarity of repeat judgments of a particular wine by the same judge, and consensus, which concerns the similarity of judgments of a particular wine across judges, are necessary requirements for expertise in wine judging. Reliability and consensus levels found in wine judging are compared to those documented by a large body of research in six other fields: medicine, clinical psychology, business, auditing, personnel management, and meteorology. In all fields, including wine judging, reliability is greater than consensus. Both reliability and consensus are, on average, substantially lower in wine judging than in other fields, although tremendous variability exists across judges in every field. Overall, little support is found for the idea that experienced wine judges should be regarded as experts.