Automatic embolus detection compared with human experts - A Doppler ultrasound study

Background and Purpose Transcranial Doppler ultrasound (TCD) reliably detects the occurrence of microembolic signals (MES). Unfortunately, TCD monitoring is a time-consuming and mentally strenuous procedure. The purpose of this study was to assess whether automatic embolus detection software devices acting as a ''stand-alone system'' are able to identify MES in patients with solid cerebral microemboli. Methods Ten records of TCD monitoring of the middle cerebral artery in patients with symptomatic high-grade carotid artery stenosis were analyzed for the moments at which MES occurred by four observers and three automatic detection software devices (RB11 on TC2000, Pioneer Version 2.10, and Embotec). The results of the three software systems were assessed on the basic assumption that MES were present if at least three of the four observers agreed. Results The average number of 1-second periods in which MES were detected by the four observers per tape ranged from 5 to 39. The overall kappa values (and SEs) for chance-corrected interobserver agreement between the four observers ranged from .94 (.02) to .99 (.01). The agreement between the software devices and the observers was lower, with kappa values (and SEs) ranging from .18 (.17) to .93 (.07). The RB11 and Embotec systems achieved a kappa value higher than 0.4 in all tapes. The Pioneer system failed to reach a kappa value of 0.4 in three tapes. The RB11 showed a sensitivity of 70% for detecting MES, the Embotec 62%, and the Pioneer 44%. Conclusions In patients with symptomatic high-grade carotid artery stenosis, a high degree of agreement in the detection of moments of MES can be achieved between observers. The three automatic detection software devices reached less agreement. Supervision of TCD monitoring and assessment of MES by an experienced observer is still necessary.