Multi-label body constitution recognition via HWmixer-MLP for facial and tongue images

As an important part of intelligent Traditional Chinese Medicine (TCM), automated body constitution recognition (BCR) using biomedical images is valuable for body constitution regulation and disease prevention. According to the composite constitution theory of TCM, we constructed two new multi-label datasets, namely, a facial image multi-label body constitution (FIMBC) dataset and a tongue image multi-label body constitution (TIMBC) dataset. For the BCR task, we proposed a novel MLP-like architecture called HWmixer-MLP to interact with cross-scale width and height features of extracted medical images and fuse them with width and height channel direction features, respectively. To improve the learning ability of HWmixer-MLP, we proposed a binary cognitive gravity loss (BCGL) for the unbalanced labels. Finally, FIMBC and TIMBC datasets were applied to validate the performance of HWmixer-MLP with 5231 facial images and 11636 tongue images, respectively. The experimental results demonstrated that the proposed approach is superior to four SOTA MLP-like models including Wave-MLP, Vip, Cycle-MLP, Active-MLP, and other Transformer-based as well convolutional neural network (CNN)-based methods with mAP values being 79.65% and 48.01% for BCGL-based HWmixer-MLP-T, respectively. Besides, an open medical image dataset was used to verify the performance of the designed HWmixer-MLP and BCGL.