Developing multimodal intelligent affective interfaces for tele-home health care

Accounting for a patient's emotional state is integral in medical care. Tele-health research attests to the challenge clinicians must overcome in assessing patient emotional state when modalities are limited (J. Adv. Nurs. 36(5) 668). The extra effort involved in addressing this challenge requires attention, skill, and time. Large caseloads may not afford tele-home health-care (tele-HHC) clinicians the time and focus necessary to accurately assess emotional states and trends. Unstructured interviews with experienced tele-HHC providers support the introduction of objective indicators of patients' emotional status in a useful form to enhance patient care. We discuss our contribution to addressing this challenge, which involves building user models not only of the physical characteristics of users-in our case patients-but also models of their emotions. We explain our research in progress on Affective Computing for tele-HHC applications, which includes: developing a system architecture for monitoring and responding to human multimodal affect and emotions via multimedia and empathetic avatars; mapping of physiological signals to emotions and synthesizing the patient's affective information for the health-care provider. Our results using a wireless non-invasive wearable computer to collect physiological signals and mapping these to emotional states show the feasibility of our approach, for which we lastly discuss the future research issues that we have identified. (C) 2003 Elsevier Science Ltd. All rights reserved.