Neural networks for data mining electronic text collections

The use of neural networks in information retrieval and text analysis has primarily suffered from the issues of adequate document representation, the ability to scale to very large collections, dynamism in the face of new information and the practical difficulties of basing the design on the use of supervised training sets. Perhaps the most important approach to begin solving these problems is the use of 'intermediate entities' which reduce the dimensionality of document representations and the size of documents collections to manageable levels coupled with the use of unsupervised neural network paradigms. This papers describes the issues, a fully configured neural network-based text analysis system - dataHARVEST - aimed at data mining text collections which begins this process, along with the remaining difficulties and potential ways forward.