Inter-rater reliability and validity of supervision performance assessment and recognition (SPARS) indicators to assess medicines management in public health facilities in Nepal

Background: Nepal's Ministry of Health and Population piloted the Supervision, Performance Assessment, and Recognition Strategy (SPARS), a multipronged strategy to improve medicines management practices in health facilities. Medicines management supervisors (MMS) periodically assess facility performance using 25 indicators grouped into 5 domains - prescribing, dispensing, ordering and reporting quality and stock and storage management - and provide targeted on-the-job coaching. This study aimed to determine inter-rater reliability (IRR) and validity for SPARS indicators in Nepali context. Methods: We assessed IRR and validity scores with three rater teams assessing medicines management in six public health posts. Each team of three MMS and one technical advisor evaluated two facilities, resulting in 24 total assessments. We calculated the mean and median validity and IRR scores for SPARS overall, and by domain and indicator. Acceptable scores were mean validity and IRR scores >= 75%, moderately acceptable were 50-75%, and unacceptable were <50%. Results: The mean overall IRR (60%) and validity (74%) of all 25 SPARS indicators were significantly different (p < 0.003) but not by domain. Three domains related to dispensing, ordering and reporting, and storage had acceptable mean validity scores, while the prescribing quality domain had an unacceptable mean IRR score. We found significant difference (p < 0.0001) between the indicator IRR and validity acceptability scores. Four (16%) and 12 (48%) of the SPARS indicators had acceptable reproducibility and validity, respectively; 8 (33%) IRR scores and 1 (4%) validity score were unacceptable. Simple indicators (67%) had significantly (p < 0.02) higher reproducibility compared to complex indicators (42%), and higher validity of 80% and 72%, respectively, which was not significant. Conclusions: This study demonstrated that the SPARS indicators produce moderately accurate findings. Validity was significantly higher than IRR and both need improvement, especially for complex indicators. As such, future SPARS implementation should endeavour to include validity and IRR assessments to help improve the indicators.