Using the normal range as a criterion for ecological significance in environmental monitoring and assessment

Data from monitoring programs are often used to compare a potentially impacted location with unimpacted: reference locations. Regardless of the experimental design, both statistical and biological criteria are frequently used to judge the significance of observed differences. In this paper, we define ecologically relevant differences as observations from impact locations that fall outside the normal range of variation based on reference-location data. We also define the normal range as the region enclosing 95% of the population of reference-location observations. This 95% region can then be expressed generically as either standard deviations (univariate) or generalized distances (multivariate). Such re-expression allows far the construction of appropriate null hypotheses and statistical tests that determine the probability that a test location Falls within the normal range. We evaluate the ability of three statistical tests (traditional two-sample contrast, and non-traditional equivalence and interval tests) to detect when test locations lie outside of the normal range of variation. For locations that are truly outside of the normal range, traditional two-sample contrasts will lead to erroneous conclusions of no impact about 50% of the time when there are 10-20 reference locations in the design. In contrast, for locations truly outside the normal range of variation, equivalence tests will lead to erroneous conclusions of no impact at most 5% of the time, regardless of sample size. The penalty associated with this test is that locations that are truly just inside the limits of the normal range will have a high probability of being declared impacted. For locations truly inside the normal range of variation, interval tests will lead to a conclusion of impact at most 5% of the time. The penalty associated with this test is chat locations that are truly impacted may not be declared impacted unless effects are very large. When evaluating an impacted location with respect to a set of reference locations, we caution that one must consider these characteristics of the tests when attempting to judge significance.