Evaluating Assessment Tools in Child Protection: A Conceptual Framework of Internal and Ecological Requirements

Despite substantial evidence on the higher predictive validity of empirically derived instruments compared to clinical judgement, the controversy on the best direction in child protection assessment is far from over. We introduce a conceptual framework that may help explain why this controversy continues. The framework distinguishes between internal and ecological requirements of assessment tools. First, existing frameworks have primarily focused on internal requirements that refer to the psychometric qualities of a tool, which are theoretically independent of the organisational context. For these internal requirements, we suggest a distinction between three types of validity: construct validity, predictive validity, and indicative validity. Second, the degree of fit with the ecological requirements determines how well the tool works in a specific organisation: for example, whether a tool makes sense to practitioners, whether they readily adopt or tacitly adapt it, or how well it fits with the objectives of the organisation and the goals of individual workers. We define four such requirements: adequacy, organisational suitability, practicality and utility. The framework is illustrated with data from an ethnographic study in Switzerland. The framework leads to questions that may guide managers and frontline workers in developing, implementing and evaluating standardised risk assessment in child protection. Key Practitioner Messages The value of an assessment tool in child protection practice only partly depends on the tool's internal qualities, such as the tool's validity in predicting future child maltreatment. In addition, the tool must meet ecological requirements: it must be oriented towards an adequate definition of child maltreatment, must support and be supported by organisational procedures and workers' competencies and must support workers in carrying out their goals effectively. When services consider implementing an existing tool or developing a new one, they should assess the tool's internal and ecological qualities with equal care. Paying attention to one set of requirements alone may be a costly mistake.