The Reliability of Multi-Valued Coding of Data

Coding is prevalent in generating analyzable data from all kinds of unstructured observations and meaningful textual or visual phenomena. Coding is essential to content analysis and social research, including computational linguistics, which concern interpretations of meaningful matter. The usual protocol calls for observers or readers to categorize, scale, or measure each of a given set of predefined units of analysis, in effect characterizing them by one value from each variable of analytical interest. However, there are many occasions in which texts and nonverbal phenomena have multiple interpretations. Until now, there has been no practical way to measure their reliability. This paper proposes a coefficient to measure the reliability of multi-valued data. It adds to Krippendorff's family of alpha coefficients, briefly reviewed. We offer a motivation for using the new coefficient, develop it mathematically, exemplify its computation, explore its behavior, and provide access to open source software to compute its most simple case.