Automatic Segmentation using Knowledge Distillation with Ensemble Models (ASKDEM)

Fielding deep learning artificial intelligence (AI) capabilities in environments that are not well represented within training datasets remains a challenge in deploying reliable perception algorithms. Environments with scarce or no representation within training datasets lend to poor semantic scene segmentation and subsequently result in suboptimal autonomous navigation performance in Unmanned Ground Vehicles (UGVs). This can be attributed to multiple technical variables including intrinsic camera properties, lighting, weather, and seasonal differences, all of which pose significant issues related to a model's ability to generalize to diverse environments and hardware configurations. Recently, zero-shot generalization capabilities for scene segmentation have been demonstrated with pre-trained foundational models. Combining the capabilities of such models with state-of-the-art semantic segmentation models can result in semantic representations of scenes with less label noise and better object boundaries. If accurate semantic labels can be applied to unlabeled segments, the resulting pseudo-labeled semantic segmentation data could be used to re-train an existing semantic segmentation model for new environments. To achieve this goal, we develop an architecture based on ensembles of semantic segmentation models to improve inferencing results in new environments by strengthening pixel label predictions used to classify unlabeled segmentation outputs. The process of automatically generating pseudo-labeled data can be computationally intensive and lacks the speed required for online inference on embodied systems. By utilizing the capabilities of pre-trained segmentation models in conjunction with an ensemble of semantic models, we can rapidly label data collected from a UGV in an environment that our fielded lightweight online model has never seen. Once the data is labeled, the original field model is retrained using the AI pseudo-labeled dataset and evaluated against the original field model. This work explores the possibility of a continuous learning framework that applies an ensemble of models to rapidly label data for model retraining. We present results showing that the approach can lead to improved algorithm performance with practical effect on the capabilities of UGVs relying on AI models which were trained on data from domains outside of the current operating environment. We show that models trained using our approach improved overall mIoU by an average of 4.75% on two distinct datasets and provide qualitative results for a third dataset.