A Synchronized Multimedia in-Home Therapy Framework in Big Data Environment

Synchronizing multimedia data such as audio and video with the 3D depth skeletal data of a patient performing therapy at home is a challenging task. This is because the media characteristics of video, audio, and skeletal stream data are different. To keep the rich session information and therapeutic semantics, the multiple media streams need to be synchronized temporally in two tiers. At the end of the first-tier synchronization, all the media are synchronized with respect to a global timestamp, which represents a user's in-home therapy session. Once synchronized at the stream level, the media streams have to be further synchronized with respect to a model therapy skeletal stream to make semantic annotation or marker on top of the media streams. This two-tier synchronized multimedia streams represent a patient's in-home therapy session that can be saved to a big data repository for further analysis. The big data repository uses map reduce functions to extract key quality of improvement metrics from the user session such as "a patient could successfully follow the therapist instruction", "how much of user session is done correctly", and "how many gestures were done wrongly" etc. We design a therapy recorder that can perform the two-tier synchronization process and create the synchronized multimedia therapy session file. We also propose a therapy player that can unpack the complex session file and separate the media files while keeping the synchronization among the media. A therapist can use the player to observe the user session, browse the synchronized media, both spatially and temporally, and add his/her comments in the form of audio notes, video notes or text notes on any particular temporal position. A patient can observe the annotations made by the therapist using the playback mode of the player, and visualize multimedia annotated notes to improve future sessions. The query interface is packed with the features to see the statistics or graph plots of any individual session of a patient or a summary of historical session data of a patient or can observe complex relative statistics among a patient group.