Sensory Integration in Deep Neural Networks

Two unimodal deep networks and one multimodal deep network are created to test for possible mechanisms of sensory integration that may shed more light on how sensory integration is carried out in biological organisms. One unimodal network is provided with pictures and the other with mel-spectrograms created from sounds. Adapted pre-trained VGG16 network was used for unimodal networks. After training consisting of 30 epochs and repeated for 100 runs the unimodal networks achieved an average accuracy of 0.57 and 0.73 respectively. The multimodal network received processed features from both unimodal networks and after training consisting of 30 epochs and repeated for 100 runs outperformed both unimodal networks with the average accuracy of 0.79. Next, noise was applied to the test data to see how unimodal and multimodal networks compare in noisy environments. Unimodal networks achieved an average accuracy of 0.63 and 0.69 respectively. Again, the multimodal network outperformed both unimodal networks with an average accuracy of 0.73. Pre-trained networks were used and limited training data were provided to the networks to simulate conditions similar to animal brains.