Towards a Biologically Inspired Question-Answering Neural Architecture

Though question-answering systems like IBM'sWatson are undoubtedly impressive, their errors are often baffling and inscrutable to onlookers, suggesting that the strategies they use are far different than those that humans employ. Desiring a more biologically inspired approach, we investigate the extent to which a neural network can develop a functional grasp of language by observing question/answer pairs. We present a neural network model that takes questions, as speech-sound sequences, about a visual environment, and learns to answer them with grounded predicate-based meanings. The model must learn to 1) segment morphemes, words, and phrases from the speech stream, 2) map the intended referents from the speech signal onto objects in the environment, 3) comprehend simple questions, recognizing what information the question is asking for, and 4) find and supply that information. Model evaluations suggest that the grounding and question-answering parts of the problem are significantly more demanding than interpreting the speech input.