Automatic target recognition system evaluation using decision analysis techniques

The United States Air Force is evaluating the development of automatic target recognition technology. Current evaluation techniques do not necessarily ensure that a superior classification system will be delivered to the end-user: the Warfighter. This research presents an evaluation methodology for ATR (Automatic Target Recognition) classification system technology that accounts for both the Evaluator's and the Warfighter's perspectives. The method involves the development of a two-pronged decision analysis model that maps ATR MOPs (Measures of Performance) into value. This is a direct mapping for the Evaluator. However, the Warfighter thinks more in terms of MOEs (Measures of Effectiveness). To incorporate the Warfighter perspective, a combat model, using a notional, unclassified scenario, was exercised in a designed experiment to produce a response surface that could serve a surrogate and intermediate mapping from MOP to MOE. Standard sensitivity analysis techniques were applied and two new methods of performing sensitivity analysis were developed. Finally, recognizing that, in practice, many MOPs were estimated from test data and therefore only estimates, the decision procedure is modified to account for their stochastic nature. The multi-nomial selection procedure is shown to do this effectively.