A Vision Enriched Intelligent Agent with Image Description Generation

In this paper, we present an intelligent conversational agent enriched with automatic image understanding and facial expression recognition using state-of-the-art machine learning techniques for the advancement of autonomous interaction with the elderly or infirm. The agent is developed to conduct health and emotion well-being monitoring for the elderly. It is not only capable of conducting question-answering via speech-based interaction, but also able to provide analysis of the user's surroundings, emotional states, hazards and fall actions via visual data. The agent is accessible from a web browser and can be communicated with via voice or text means, with a webcam required for the visual analysis functionality. The system has been evaluated with diverse real-life images to prove its efficiency.