Classifier performance assessment in social science - does the quality of data matter?

In social science it is common to deal with data of variable quality, as a direct consequence of having human beings as the main source of information. Various measures of data quality have been developed in humanities e.g. validation scales or lie scores. In many real life cases, decisions need to be made or diagnoses assigned based on data of lower quality when data of higher quality are not available - any information on its quality is therefore valuable. Real life decision making problems in social science can be translated into the language of classification. In mathematical classifier design theory various measures of classifier performance have been designed. As far as we know, none of these reflects directly the quality (reliability) of the data. On the example of the receiver operating characteristics (ROC) and the area under the ROC curve (AUC) introduced in signal detection theory we show, how data quality can be incorporated into performance assessment of classifiers. We present a modification of the ROC approach reflecting data quality and discuss possible benefits of its use on artificial data.