Bidirectional Associative Memory with Block Coding: A Comparison of Iterative Retrieval Methods

Recently, Gripon and Berrou (2011) have investigated a recurrently connected Willshaw-type auto-associative memory with block coding, a particular sparse coding method, reporting a significant increase in storage capacity compared to earlier approaches. In this study we verify and generalize their results by implementing bidirectional hetero-associative networks and comparing the performance of various retrieval methods both with block coding and without block coding. For iterative retrieval in networks of size n = 4096 our data confirms that block-coding with the so-called "sum-of-max" strategy performs best in terms of output noise (which is the normalized Hamming distance between stored and retrieved patterns), whereas the information storage capacity of the classical models cannot be exceeded because of the reduced Shannon information of block patterns. Our simulation experiments also provide accurate estimates of the maximum pattern number that can be stored at a tolerated noise level of 1%. It is revealed that block coding is most beneficial for sparse activity where each pattern has only k similar to log n active units.