Are we recording postoperative complications correctly? Comparison of NHS Hospital Episode Statistics with the American College of Surgeons National Surgical Quality Improvement Program

Background Hospital Episode Statistics (HES) data are used to measure surgical outcomes, but its quality has been considered inferior to that of clinical databases. We compare the recording accuracy of HES, an administrative database used in the National Health Service (NHS), with that of ACS NSQIP (American College of Surgeons National Surgical Quality Improvement Program), a well-established clinical database. Methods 1323 patient records from our hospital, common to both databases were compared for ten surgical procedures (amputation, appendicectomy, cholecystectomy, femoral hernia repair, Hartmann's procedure, incisional hernia repair, inguinal hernia repair, long saphenous vein surgery, parathyroidectomy and umbilical hernia repair) and nine postoperative complications (acute renal failure, myocardial infarction, pneumonia, pulmonary embolism, urinary tract infection, blood transfusion, septic shock, surgical site infection and wound disruption) using text strings or ICD-10 (International Classification of Diseases) codes. kappa coefficient was calculated as a measure of concordance between HES and ACS NSQIP databases. Results The databases showed perfect or very good agreement in recording a majority of surgical procedures (kappa coefficient range 0.82-1.0), but there was discordance in recording postoperative complications. When HES was investigated using text string or ICD-10 code, the kappa coefficient range for nine postoperative complications was 0.00-0.56, indicating poor to moderate inter-rater agreement. Concordance was even less when searched by HES coder's recommended way to record postoperative complications. Conclusions HES poorly registers postoperative complications. Suggested improvements include addition of dates when a condition is diagnosed, agreed criteria to identify postoperative complications, specifically trained coding staff for surgery and consistent use of the coding guidance.