MaterialSense: Estimating and utilizing material properties of contact objects in multi-touch interaction

In today's digital art applications, users can play virtual musical instruments or paint on virtual canvas using multi-touch input. Existing touch input devices, however, do not consider that the result of artistic expression changes depending on the material properties of the tool that comes into contact with the art medium. This paper proposes a novel method called MaterialSense that estimates the material properties of objects in contact with a touch input surface. The technique uses a commercial touchpad with six load cells attached, and when up to two objects are in contact, solves the force equilibrium equation to estimate the stiffness and friction coefficient of each contacted object. By analyzing the time-series data of the measured 3-axis force and normal vector for each touchpoint, it can estimate the stiffness and kinetic friction coefficient of the contacted object. Estimated material properties enable novel and realistic artistic expressions in touch-based digital art applications, such as changes in the tone of virtual instruments or the effects of different painting brushes. We present two application scenarios using MaterialSense, along with an in-depth technical evaluation to verify the accuracy and precision of its estimates.