TON-ViT: A Neuro-Symbolic AI Based on Task Oriented Network with a Vision Transformer

The objective of this paper is to present a neuro-symbolic AI based technique to represent field-medicine knowledge, referred as to TON-ViT. TON-ViT integrates a Deep Learning Model with an explicit symbolic manipulation, a task graph. This task graph describes the steps of each trauma resuscitation as denoted by a verb and noun pair. Through this representation, symbolic processing and manipulation on task graphs, we can find stereotypical procedures, regardless of style of the performer. Furthermore, we can use this technique to find differences in styles, errors, shortcuts and generate procedures never seen before. When used in combination with a transformer, it can help recognize actions in egocentric vision datasets. Last, through symbolic manipulations on the graph, it is possible to generate medical knowledge which the model has not seen before. We present preliminary results after testing the TON-ViT with the Trauma Thompson Dataset.