Kinect-for-windows with augmented reality in an interactive roleplay system for children with an autism spectrum disorder

Children with autism spectrum disorder (ASD) often find it difficult to communicate using body language and facial expressions to respond appropriately socially. This negatively affects the quality of their everyday social life. Children with ASD can learn to socially interact with virtual three-dimensional (3D) characters in animated games enhanced with augmented reality (AR). They can learn about body language and facial expressions through roleplay. This teaching method can effectively improve the interactive social skills of children with ASD and reduce the fear and anxiety which they typically experience when they face real people. However, most current virtual roleplay models focus only on predefined body language and facial expressions limited by the training system. These systems lack immediacy and a sense of genuine interaction for children with ASD. Therefore, we wanted to create a system that will allow trainers and therapists to control 3D virtual characters that are able to produce in real time the full range of natural body language and facial expressions appropriate for males, females, and even special cartoon characters. Based on the multiple baseline design across subjects, the data indicated that real-time Kinect Skeletal Tracking (KST) System can help the children with ASD improved their social interaction.