RT-DEKT: real-time object detector with KAN-Transformer

Real-Time Detection Transformer (RT-DETR) is the first Transformer-based detector extended to real-time object detection scenarios. Despite its remarkable performance in both accuracy and real-time inference, the backbone lacks adequate attention to feature channels and spatial dimensions, resulting in suboptimal feature quality. To address this limitation, this paper proposes a hybrid backbone with spatial and channel attention mechanism to enhance the feature extraction capabilities of the backbone. Furthermore, the original Transformer encoder has insufficient processing capabilities for high-dimensional complex image data and poor interpretability. To overcome these limitations, this paper introduces a high-level KAN-Transformer encoder, which utilizes Kolmogorov-Arnold Networks that outperform traditional Multi-Layer Perceptrons. The proposed object detector is named RT-DEKT, with variants based on different backbone architectures: RT-DEKT-R50 and RT-DEKT-R101. Experiments on the COCO dataset demonstrate that RT-DEKT-R50 achieves 53.6% AP and 103 FPS on an NVIDIA T4 GPU, while RT-DEKT-R101 achieves 54.8% AP and 68 FPS. After pretraining on the Objects365 dataset, RT-DEKT-R50 improves to 56.8% AP, and RT-DEKT-R101 further enhances to 58.5% AP. Both variants outperform contemporary detectors in accuracy and real-time performance, offering an effective trade-off between efficiency and accuracy.