Transformer-based Hierarchical Encoder for Document Classification

Document Classification has a wide range of applications in various domains like Ontology Mapping, Sentiment Analysis, Topic Categorization and Document Clustering, to mention a few. Unlike Text Classification, Document Classification works with longer sequences that typically contain multiple paragraphs. Previous approaches for this task have achieved promising results, but have often relied on complex recurrence mechanisms that are expensive and time-consuming in nature. Recently, self-attention based models like Transformers and BERT have achieved state-of-the-art performance on several Natural Language Understanding (NLU) tasks, but owing to the quadratic computational complexity of the self-attention mechanism with respect to the input sequence length, these approaches are generally applied to shorter text sequences. In this paper, we address this issue, by proposing a new Transformer-based Hierarchical Encoder approach for the Document Classification task. The hierarchical framework we adopt helps us extend the self-attention mechanism to long-form text modelling thereby reducing the complexity considerably. We use the Bidirectional Transformer Encoder (BTE) at the sentence-level to generate a fixed-size sentence embedding for each sentence in the document. A document-level Transformer Encoder is then used to model the global document context and learn the inter-sentence dependencies. We also carry out experiments with the BTE in a feature-extraction and a fine-tuning setup, allowing us to evaluate the trade-off between computation power and accuracy. Furthermore, we also conduct ablation experiments, and evaluate the impact of different pre-training strategies on the overall performance. Experimental results demonstrate that our proposed model achieves state-of-the-art performance on two standard benchmark datasets.