Retrospective selection bias (or the benefit of hindsight)

The complexity of geophysical systems makes modelling them a formidable task, and in many cases research studies are still in the phenomenological stage. In earthquake physics, long timescales and the lack of any natural laboratory restrict research to retrospective analysis of data. Such 'fishing expedition' approaches lead to optimal selection of data, albeit not always consciously. This introduces significant biases, which are capable of falsely representing simple statistical fluctuations as significant anomalies requiring fundamental explanations. This paper identifies three different strategies for discriminating real issues from artefacts generated retrospectively. The first attempts to identify ab initio each optimal choice and account for it. Unfortunately, a satisfactory solution can only be achieved in particular cases. The second strategy acknowledges this difficulty as well as the unavoidable existence of bias, and classifies all 'anomalous' observations as artefacts unless their retrospective probability of occurrence is exceedingly low (for instance, beyond six standard deviations). However, such a strategy is also likely to reject some scientifically important anomalies. The third strategy relies on two separate steps with learning and validation performed on effectively independent sets of data. This approach appears to be preferable in the case of small samples, such as are frequently encountered in geophysics, but the requirement for forward validation implies long waiting times before credible conclusions can be reached. A practical application to pattern recognition, which is the prototype of retrospective 'fishing expeditions', is presented, illustrating that valid conclusions are hard to find.