Reproducing a Neural Question Answering Architecture Applied to the SQuAD Benchmark Dataset: Challenges and Lessons Learned

Reproducibility is one of the pillars of scientific research. This study attempts to reproduce the Gated Self-Matching Network, which is the basis of one of the best performing models on the SQuAD dataset. We reimplement the neural network model and highlight ambiguities in the original architectural description. We show that due to uncertainty about only two components of the neural network model and no precise description of the training process, it is not possible to reproduce the experimental results obtained by the original implementation. Finally we summarize what we learned from this reproduction process about writing precise neural network architecture descriptions, providing our implementation as a basis for future exploration.