Evaluation of criteria for information retrieval

We investigate van Rijsbergen's F-measure, break-even point, and 11-point averaged precision, all of which can be translated into 1-dimensional scalar quantity from the precision and the recall. These investigations can be done by comparing to tetrachoric (four fold) correlation coefficient and phi (fourfold point) coefficient, which are often used as the index of statistical association in a 2 x 2 contingency table. The results show that when a fallout rate is less than 0.1, (1) the F-1 measure has similar properties of the phi coefficient, (2) the break-even point is almost equivalent to a phi coefficient, and (3) the 11-point averaged precision should be a measure which is larger than a phi coefficient and has a value smaller than a tetrachoric correlation coefficient.