Policy Evaluation via Composite Indexes: Qualitative Lessons from International Transparency Policy Indexes

International transparency policy indexes (ITPIs) help determine billions in investment and aid, influence "authoritative" scholarship, and shape policy choices. Are ITPIs valid yardsticks.of transparency, or do they encourage dissimulation? Most scholarship on index-based evaluations focuses on "concept indexes" (e.g., governance) from quantitative approaches. This paper presents qualitative insights about ITPIs in specific and "policy indexes" in general, analyzing three measurement-related pitfalls and proposing countermeasures. Most significantly, it shows how indexes presuppose substitutability while policies contain nonsubstitutable 'necessary' policy provisions. This dilemma of "ontological compatibility" means that policies can rank favorably on indexes notwithstanding the absence of lynchpin policy provisions. (C) 2015 Elsevier Ltd. All rights reserved.