Automatic target recognition (ATR) evaluation theory: a survey

Proper evaluation of a pattern recognition system in the lab is paramount to its success in the field. In most commercial pattern recognition applications, such as breast cancer detection, optical character recognition, and industrial quality assurance, the boundaries and expectation of the system are well defined. This is due, at least in part, to an excellent understanding of the problem and data space for these applications. For these functions, a method for rigorous evaluation is well understood. However, the size and complexity of the data and problem spaces for automatic target recognition (ATR) systems is enormous. The consequences are that a complete understanding of how an ATR system will perform in practice is extraordinarily difficult to estimate. Thus the act of evaluating an ATR system becomes as important as its design. This paper compiles and reports the techniques used to evaluate ATR system performance. It surveys the specific difficulties associated with ATR performance estimation as well as approaches used to mitigate these obstacles.