Assessing transcription agreement: Methodological aspects

In recent years it has become common practice among speech researchers to report transcription agreement coefficients, when describing findings based on phonetic transcription. Although such coefficients are intended to give an indication of the degree of transcription accuracy, in reality it is not clear to what extent high agreement coefficients do indeed guarantee great transcription accuracy. In this paper it is argued that the most commonly used agreement coefficient, percentage agreement, has three major disadvantages when applied to phonetic transcription: (a) it is based on the assumption that agreement between transcription symbols is all-or-none, (b) it is strongly influenced by chance agreement, and (c) it gives no account of the criteria used for transcription alignment. After a detailed discussion of these drawbacks, an alternative approach to calculating transcription (dis)agreement is proposed. In this approach experimentally derived feature matrices are used as input to a program that aligns transcription pairs automatically and at the same time calculates a (dis)agreement measure for each transcription pair, the average distance. It is argued that this metric is more adequate to express the degree of (dis)similarity between transcriptions than the more usual percentage agreement. The results of an evaluation experiment corroborate this view.