Inter- and Intra-rater Reliability in Lithic Analysis: a Case Study in Handedness Determination Methodologies

Inter- and intra-rater reliability studies in experimental archaeology promote consistency and replicability in the lithic analysis methods that are applied to interpretations of the archaeological record. Replication attempts to classify a knapper's hand preference post-hoc using published methodologies that focus on right- and left-oriented flake features, have been largely unsuccessful. We tested the validity of flake feature categories described in three studies to be useful for determining a knapper's hand preference (Bargallo and Mosquera, Laterality, 19(1), 37-63, 2014; Dominguez-Ballesteros and Arrizabalaga, Journal of Archaeological Science: Reports, 3, 313-320, 2015; Rugg and Mullane, Laterality, 6(3), 247-259, 2001). Five experienced lithic analysts independently made blind predictions of knapper hand preference on an experimental assemblage of mode I flakes produced by 18 knappers (9 left-handed), which included 344 complete flakes from 43 knapped cores. Inter-rater reliability measures (using Fleiss' Kappa) showed significant agreement between raters for only one of the features (eraillure scar), with fair agreement for impact point, and poor agreement for the other features (cone of percussion, hackles, ripples, extraction axis, and platform inclination); poor agreement was found even within raters. Chi squared tests and correspondence analyses show that raters fail to perform significantly better than chance at predicting hand preference. These results suggest not only that these flake features are unreliable predictors of a knapper's hand preference, but also that most of these features do not represent objective categories. We therefore urge caution in applying these methods to archaeological assemblages pending further independent replication.