BARS and Those Mysterious, Missing Middle Anchors

Purpose A contributing reason for the common problem of missing middle anchors on behaviorally anchored rating scales (BARS) is the standard deviation (SD) criterion used in scaling phase. An alternative BARS scaling process is proposed based on the a(wg(1)) index of interrater agreement. Design/Methodology/Approach Algebraic principles are used to explicate that the SD criterion is analogous to using a r(wg(1)) interrater agreement statistic with the assumption of a uniform null distribution, and this reliance on r(wg(1)) decreases the likelihood of anchoring behaviors in the mid-range of the rating scale. Archival data from a law enforcement agency were used to compare the success and failure of anchoring a BARS using a SD criterion versus an a(wg(1)) criterion. Findings The a(wg(1)) criterion was successful at anchoring the full range of the rating scale, but only if the cut-off for anchoring behaviors required a "weak" level of interrater agreement. Implications The most surprising finding was that the traditional 1.5 SD criterion on a 9-point rating scale is not a particularly stringent agreement requirement for anchoring behaviors. Although we demonstrated the advantages of using a(wg(1)) to anchor BARS, an equally important conclusion is that incumbents need to be better trained prior to scaling behaviors. Originality/Value We provide a theoretically defensible approach for anchoring BARS that ameliorates the missing middle anchor problem. Further, the utility of a(wg(1)) in the BARS context is yet another example of the limitation of r(wg(1)) when assuming a uniform null distribution.