Text modeling for real-time document categorization

We report on experiments in adapting document categorization techniques to provide for implementation in high-speed hardware.(1,2) Because resources are scarce, it is important to have a small set of robust and maximally informative variables over which learning can occur. We generate variables using information-theoretic clustering. The resulting performance is on par with general-purpose computing implementations which are able to take advantage of large amounts of time and memory. We conclude that custom high-speed hardware for document categorization can be made very accurate. We also believe that some of the strengths of information-theoretic data analysis techniques are brought out.