Evaluating models of expert judgment to inform assessment of ecosystem viability and collapse

Expert judgment underpins assessment of threatened ecosystems. However, experts are often narrowly defined, and variability in their judgments may be substantial. Models built from structured elicitation with large diverse expert panels can contribute to more consistent and transparent decision-making. We conducted a structured elicitation under a broad definition of expertise to examine variation in judgments of ecosystem viability and collapse in a critically endangered ecosystem. We explored whether variation in judgments among 83 experts was related to affiliation and management expertise and assessed performance of an average model based on common ecosystem indicators. There were systematic differences among individuals, much of which were not explained by affiliation or expertise. However, of the individuals affiliated with government, those in conservation and environmental departments were more likely to determine a patch was viable than those in agriculture and rural land management. Classification errors from an average model, in which all individuals were weighted equally, were highest among government agriculture experts (27%) and lowest among government conservation experts (12%). Differences were mostly cases in which the average model predicted a patch was viable but the individual thought it was not. These differences arose primarily for areas that were grazed or cleared of mature trees. These areas are often the target of restoration, but they are also valuable for agriculture. These results highlight the potential for conflicting advice and disagreement about policies and actions for conserving and restoring threatened ecosystems. Although adoption of an average model can improve consistency of ecosystem assessment, it can fail to capture and convey diverse opinions held by experts. Structured elicitation and models of ecosystem viability play an important role in providing data-driven evidence of where differences arise among experts to support engagement and discussion among stakeholders and decision makers and to improve the management of threatened ecosystems.