Breaking Down Covariate Shift on Pneumothorax Chest X-Ray Classification

Domain shift poses significant problems to computer-aided diagnostic (CAD) systemswhen deployed in clinical scenarios. There's still no definite fix nor an in-depth understanding of the exact factors driving domain shifts in medical X-rays. Here, we conduct an exploratory study on three covariate shift factors in X-ray classification by controlling for different variables. This is possible by leveraging a homogenously-relabelled mix of public and private X-ray data spanning 23 medical institutions over four continents and 17 classes of pathologies. We show that the acquisition parameter, device manufacturer and geographical shifts degrade out-ofdistribution (OOD) F1 by 6%, 3.2% and 3.3%, respectively. Pneumothorax was found to be themost impaired pathology, suffering a mean F1 generalisation gap of 13.3%, despite being one of themost clinically-consequential radiological findings. To this end, we introduced LISA-topK, a multi-label adaptation of Learning Invariant Predictors with Selective Augmentation (LISA), that we showed to narrow down the OOD gap, surpassing other methods consistently. These pragmatic results shed light on some of the elements of OOD generalisation in X-ray classification, which are essential to researching, understanding and deploying CAD systems. Code is available at https://github.com/RayscapeAI/LISA-topK