Specification and dialogue control of visual interaction through visual rewriting systems

Computers are increasingly being seen not only as computing tools but more so as communication tools, thus placing special emphasis on human-computer interaction (HCI). In this article, the focus is on visual HCI, where the messages exchanged between human and computer are images appearing on the computer screen, as usual in current popular user interfaces. We formalize interactive sessions of a human-computer dialogue as a structured set of legal visual sentences, i.e., as a visual language, and show how rewriting Systems can be generalized to specify both the pictorial and the computational aspects of visual languages. To this end, Visual Conditional Attributed Rewriting (VCARW) systems are introduced, and used for specification of visual languages. These specifications are given as inputs to a procedure illustrated in the article as a system of algorithms, which automatically generates control mechanisms of the interaction, thus favoring the design of more reliable and usable systems.