Can humans simulate talking like other humans? Comparing simulated clients to real customers in service inquiries

How authentic are inquiry calls made by simulated clients, or 'mystery shoppers', to service organizations, when compared to real callers? We analysed 48 simulated and 63 real inquiry calls to different veterinary practices in the United Kingdom and Ireland. The data were transcribed for conversation analysis, as well as coded for a variety of call categories including reason for the call, call outcome and turn design features. Analysis revealed systematic differences between real and simulated calls in terms of (1) reasons for the call, call outcome and call duration and (2) how callers refer to their pets in service requests and follow-up questions about their animal. Our qualitative analyses were supported with statistical summaries and tests. The findings reveal the limitations of mystery shopper methodology for the assessment of service provision. We also discuss the implications of the findings for the use of simulated encounters and the development of conversational agents.