Empowering IoT-Based Autonomous Driving via Federated Instruction Tuning With Feature Diversity

Integrating large language models (LLMs) with the Internet of Things (IoT) offer great potential for enhancing vehicle personalization and adaptability in autonomous driving (AD), particularly in open-world scenarios. However, the increasing scarcity of high-quality public data poses a challenge, which could hinder the progress of LLMs in AD. To address this, we propose a novel approach, federated instruction tuning (FIT), that leverages federated learning (FL) to enable collaborative training of a shared model across multiple data owners without sharing raw data, thereby preserving privacy and mitigating data scarcity. Complementing FIT, we introduce a feature diversity (FD) strategy that enriches visual and textual diversity and significantly expands AD data by generating new instruction-following data across key dimensions, such as time, weather, and occlusion. Extensive experiments using LLaMA-Adapter as the base model and four FL methods validate the effectiveness of the FIT framework and the FD strategy. Our analysis also compares LLMs ranging from 1.1 to 7B parameters, with results evaluated using GPT score, demonstrating the potential of FIT in AD. Our findings suggest that FIT and FD can support intelligent network operation and optimization in IoT, benefiting both the AD and artificial intelligence (AI) industries.