Acceleration of Online Recognition of 2D Sequences Using Deep Bidirectional LSTM and Dynamic Programming

In this work, the approach for online recognition of 2D sequences using deep bidirectional LSTM was proposed. One of the complex cases of online sequence recognition is handwritten mathematical expressions (HME). In spite of many achievements in this area, it is a still challenging task as, in addition to character segmentation and recognition, the tasks of structure, relations, and grammar analysis should be resolved. Such a combination of recognizers could lead to an increase in computational complexity for large expressions, which is unacceptable for on-device recognition in mobile applications. As end-to-end neural systems do not achieve plausible accuracy and recognition speed for on-device calculations so far, to overcome this problem we proposed a deeplearning solution that employs recurrent neural networks (RNNs) for structure and character recognition in combination with re-ordering and modified CYK algorithm for expression construction. Also, we explored a variety of structural and optimization enhancements to CYK algorithm that significantly improved the performance in terms of the recognition speed while the recognition accuracy remained at the same level. The ablation study for the introduced optimization techniques demonstrated significant improvement of recognition speed keeping the recognition accuracy comparable with the existing state-of-the-art approaches.