Checking how fact-checkers check

Fact-checking has gained prominence as a movement for revitalizing truth-seeking ideals in journalism. While fact-checkers are often assumed to code facts accurately, few studies have formally assessed fact-checkers' overall performance. I evaluated the performance of two major fact-checkers in the USA, Fact Checker and Politifact, comparing their inter-rater reliability using a method that is regularly employed across the social sciences. Surprisingly, only one in 10 statements was found to be fact-checked by both fact-checkers. Regarding claims evaluated by both organizations, the fact-checkers performed fairly well on outright falsehoods or obvious truths; however, the agreement rate was much lower for statements in the more ambiguous scoring range (that is, "Half True" or "Mostly False"). The results suggest that fact-checking is difficult, and that validation is challenging. Fact-checkers rarely evaluate statements that are exactly the same, and disagree more often than one might suppose, particularly when politicians craft language to be ambiguous. At least in some cases, the strategic ambiguity of politicians may impede the fact-checking movement's goals.