Promoting Responsible Electronic Documentation: Validity Evidence for a Checklist to Assess Progress Notes in the Electronic Health Record

Construct: We aimed to develop an instrument to measure the quality of inpatient electronic health record- (EHR-) generated progress notes without requiring raters to review the detailed chart or know the patient. Background: Notes written in EHRs have generated criticism for being unnecessarily long and redundant, perpetuating inaccuracy and obscuring providers' clinical reasoning. Available assessment tools either focus on outpatient progress notes or require chart review by raters to develop familiarity with the patient. Approach: We used medical literature, local expert review, and attending focus groups to develop and refine an instrument to evaluate inpatient progress notes. We measured interrater reliability and scored the selected-response elements of the checklist for a sample of 100 progress notes written by PGY-1 trainees on the general medicine service. Results: We developed an instrument with 18 selected-response items and four open-ended items to measure the quality of inpatient progress notes written in the EHR. The mean Cohen's kappa coefficient demonstrated good agreement at .67. The mean note score was 66.9% of maximum possible points (SD = 10.6, range = 34.4%-93.3%). Conclusions: We present validity evidence in the domains of content, internal structure, and response process for a new checklist for rating inpatient progress notes. The scored checklist can be completed in approximately 7minutes by a rater who is not familiar with the patient and can be done without extensive chart review. We further demonstrate that trainee notes show substantial room for improvement.