The assessment of criterion audit cycles by external peer review - when is an audit not an audit?

Introduction Clinical audit has failed to fully deliver the rewards initially envisaged. Contributory factors include: an ill-defined approach to audit; the assumption that health care professionals can intuitively apply audit methods; and the lack of a system to 'quality assure' the process. A method of criterion audit was defined and developed in conjunction with an instrument to facilitate trained General Practitioner (GP) assessors in the review of colleagues' audit projects. Given the potential for improving audit practice, this study aimed to define the methodological factors that contributed to 'unsatisfactory' audits as judged by peer assessors. Methods West of Scotland GPs voluntarily submitted a criterion audit in a standard format for review by two trained colleagues using an assessment instrument. Audits judged unsatisfactory and associated educational feedback were subjected to content analysis. Results Between 1999 and 2004, 336 audits were submitted, of which 132 (39%) were judged to be unsatisfactory. Of these, 118 audits (89%) had a methodological issue identified in the initial project design (e.g. defining criteria) that effectively invalidated the audit. 119 projects (90%) were also judged to have at least one deficiency in the data analysis or change management stages of the audit (e.g. implementing inadequate change). Conclusion A range of audit method issues was found. The proportion of unsatisfactory audits may point to a larger problem beyond this sample, which may have implications for health care quality. If audit practise is to be consistent and rigorous, consideration should be given to assessing the standard of this activity.