Quantifying chatbots' ability to learn business processes

Chatbots enable organizations in the business-to-customer domain to respond to repetitive requests efficiently. Extant approaches in Natural Language Processing (NLP) already address the essential requirement of understanding user input and synthesizing a response as close as possible to a response a human interlocutor would give. However, we argue that the organizational adoption of chatbots further depends on the underlying model's capability to learn and comply with organizations' business processes, for example, authenticating a customer before providing sensitive details. To address this issue, we develop an approach that quantifies chatbots' ability to learn business processes using standardized process mining metrics. We demonstrate our approach by training chatbots on a dataset of more than 500,000 customer service conversations from three companies on Twitter and show how our approach supports the quantification of a chatbot's overall ability to learn business processes from the training data. Furthermore, we quantify a chatbot's ability to learn a particular variant of the underlying process and we show how to compare the chatbot's executed steps against a given normative process model. Our approach that seamlessly integrates with existing approaches to evaluate NLP-based chatbots mitigates the current hurdles that practitioners face and, therefore, strives to foster the adoption of chatbots in practice.(c) 2023 Elsevier Ltd. All rights reserved.