DeepVision: Heads-up Computing and AI in Education

Heads-up computing together with AI can enhance in-class learning experiences. In this position paper, we propose the development of a multimodal AI system called DeepVision that integrates Automatic Speech Recognition (ASR), Large Language Models (LLM), Large Vision Models (LVM), Information Retrieval (IR) and Inclusive User Experience Design (IUX) to convert real-time lectures into multiple knowledge representations. These will be visualized on heads-up communication devices such as Augmented Reality (AR) and Mixed Reality (MR) devices. The initiative is a collaboration between Habitat Learn Limited (HLL) and the University of Southampton, leveraging HLL's existing software and extensive data repository to address the challenges of traditional and digital learning environments, especially for students with disabilities or language differences.