Extracting Pseudocode from Digital Block Diagram in Technical Documents

The technical document is an entity that consists of several essential and interconnected parts, often referred to as modalities. Despite the extensive attention that certain modalities have already received, say per the textual information, there are several aspects that get disproportional attention throughout the bibliography. An instance of such a modality is the block diagrams found in the technical document, as they can provide crucial information about the functionality and the technical details of both their related parts and the technical document as a whole. This paper deals with the automatic understanding of block diagrams and the extraction of pseudocode associated with the functionality of a given diagram. In particular we present a complete methodology for the formal modelling of digital block diagrams and their elements, then we develop a generative framework and three novel annotated datasets on diagrams classification and captioning. After mapping the initial problem to a block diagram description task, we present several original predictive setups derived from image segmentation, analyze the inference capabilities, and we offer illustrative examples justifying our approach.