Multimodal Interface for Drawing Diagrams that Does not Interfere with Natural Talking and Drawing

The aim of this research is to support real-time drawing in talking by using multimodal user interface technologies. In this situation, if talking and drawing are considered as commands by mistake during presentation, it will disturb users' natural talking and drawing. To prevent this problem, we introduce two modes of a command mode and a free mode, and explore smooth mode switching techniques that does not interfere with users' natural talking and drawing. We evaluate four techniques. Among them, a technique that specifies the command mode after actions using a pen gesture was the most effective. In this technique, users could quickly draw diagrams, and specifying mode switching didn't interfere with users' natural talk.