Parallel Implementation of Feedforward Neural Networks on GPUs

Neural networks are often seen as a natural model of parallel computation, especially when contrasted with more traditional sequential models like the Turing Machine. The parallelism of neural networks has become more important in recent years through the confluence of two tendencies in the evolution of computer and information technologies: first, parallel computing devices are now ubiquitous, instead of being relegated to a niche market; and second, the amount of data available to analyze and learn from in machine learning applications has increased at a rapid pace. Graphical Processing Units (GPUs) provide great computational power in standard desktop computers, being composed of many simple execution units. In this paper a technique is presented for the parallel implementation of neural networks in GPUs. The technique is explained in relation to the difficulties imposed by the execution model of GPUs. Experimental results indicate that the proposed implementation techniques can easily attain a performance gain of more than one order of magnitude, and are scalable with the processing power of the GPU used.