Extract interpretability-accuracy balanced rules from artificial neural networks: A review

Artificial neural networks (ANN) have been widely used and have achieved remarkable achievements. However, neural networks with high accuracy and good performance often have extremely complex internal structures such as deep neural networks (DNN). This shortcoming makes the neural networks as incomprehensible as a black box, which is unacceptable in some practical applications. But pursuing excessive interpretation of the neural networks will make the performance of the model worse. Based on this contradictory issue, we first summarize the mainstream methods about quantitatively evaluating the accuracy and interpretability of rule set. And then review existing methods on extracting rules from Multilayer Perceptron (MLP) and DNN in three categories: Decomposition Approach (Extract rules in neuron level such as visualizing the structure of network), Pedagogical Approach (By studying the correspondence between input and output such as by computing gradient) and Eclectics Approach (Combine the above two ideas). Some potential research directions about extracting rules from DNN are discussed in the last. (C) 2020 Elsevier B.V. All rights reserved.