Few-shot out-of-scope intent classification: analyzing the robustness of prompt-based learning

被引:0
作者
Yiwei Jiang
Maarten De Raedt
Johannes Deleu
Thomas Demeester
Chris Develder
机构
[1] Ghent University – imec,IDLab
来源
Applied Intelligence | 2024年 / 54卷
关键词
Few-shot learning; Prompt-based models; Outlier/novelty detection; Dialogue intent classification;
D O I
暂无
中图分类号
学科分类号
摘要
Out-of-scope (OOS) intent classification is an emerging field in conversational AI research. The goal is to detect out-of-scope user intents that do not belong to a predefined intent ontology. However, establishing a reliable OOS detection system is challenging due to limited data availability. This situation necessitates solutions rooted in few-shot learning techniques. For such few-shot text classification tasks, prompt-based learning has been shown more effective than conventionally finetuned large language models with a classification layer on top. Thus, we advocate for exploring prompt-based approaches for OOS intent detection. Additionally, we propose a new evaluation metric, the Area Under the In-scope and Out-of-Scope Characteristic curve (AU-IOC). This metric addresses the shortcomings of current evaluation standards for OOS intent detection. AU-IOC provides a comprehensive assessment of a model’s dual performance capacities: in-scope classification accuracy and OOS recall. Under this new evaluation method, we compare our prompt-based OOS detector against 3 strong baseline models by exploiting the metadata of intent annotations, i.e., intent description. Our study found that our prompt-based model achieved the highest AU-IOC score across different data regimes. Further experiments showed that our detector is insensitive to a variety of intent descriptions. An intriguing finding shows that for extremely low data settings (1- or 5-shot), employing a naturally phrased prompt template boosts the detector’s performance compared to rather artificially structured template patterns.
引用
收藏
页码:1474 / 1496
页数:22
相关论文
共 50 条
[41]   Few-Shot Classification Based on Sparse Dictionary Meta-Learning [J].
Jiang, Zuo ;
Wang, Yuan ;
Tang, Yi .
MATHEMATICS, 2024, 12 (19)
[42]   An Optimized Model based on Metric-Learning for Few-Shot Classification [J].
Zhao, Wencang ;
Qin, Wenqian ;
Li, Ming .
PROCEEDINGS OF THE 33RD CHINESE CONTROL AND DECISION CONFERENCE (CCDC 2021), 2021, :1708-1713
[43]   Adaptive Feature Representation Based On Contrastive Learning For Few-Shot Classification [J].
Fang, Jiajie ;
Qiao, Qian ;
Zeng, Ziyin ;
Li, Fanzhang .
2024 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS, IJCNN 2024, 2024,
[44]   Infrared aircraft few-shot classification method based on meta learning [J].
Chen Rui-Min ;
Liu Shi-Jian ;
Miao Zhuang ;
Li Fan-Ming .
JOURNAL OF INFRARED AND MILLIMETER WAVES, 2021, 40 (04) :554-560
[45]   Supervised and Few-Shot Learning for Aspect-Based Sentiment Analysis of Instruction Prompt [J].
Huang, Jie ;
Cui, Yunpeng ;
Liu, Juan ;
Liu, Ming .
ELECTRONICS, 2024, 13 (10)
[46]   Few-Shot Learning Meets Transformer: Unified Query-Support Transformers for Few-Shot Classification [J].
Wang, Xixi ;
Wang, Xiao ;
Jiang, Bo ;
Luo, Bin .
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 2023, 33 (12) :7789-7802
[47]   Towards using Few-Shot Prompt Learning for Automating Model Completion [J].
Ben Chaaben, Meriem ;
Burgueno, Lola ;
Sahraoui, Houari .
2023 IEEE/ACM 45TH INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING-NEW IDEAS AND EMERGING RESULTS, ICSE-NIER, 2023, :7-12
[48]   Hierarchical Prompt Tuning for Few-Shot Multi-Task Learning [J].
Liu, Jingping ;
Chen, Tao ;
Liang, Zujie ;
Jiang, Haiyun ;
Xiao, Yanghua ;
Wei, Feng ;
Qian, Yuxi ;
Hao, Zhenghong ;
Han, Bing .
PROCEEDINGS OF THE 32ND ACM INTERNATIONAL CONFERENCE ON INFORMATION AND KNOWLEDGE MANAGEMENT, CIKM 2023, 2023, :1556-1565
[49]   Ontology-enhanced Prompt-tuning for Few-shot Learning [J].
Ye, Hongbin ;
Zhang, Ningyu ;
Deng, Shumin ;
Chen, Xiang ;
Chen, Hui ;
Xiong, Feiyu ;
Chen, Xi ;
Chen, Huajun .
PROCEEDINGS OF THE ACM WEB CONFERENCE 2022 (WWW'22), 2022, :778-787
[50]   Heterogeneous Few-Shot Learning for Hyperspectral Image Classification [J].
Wang, Yan ;
Liu, Ming ;
Yang, Yuexin ;
Li, Zhaokui ;
Du, Qian ;
Chen, Yushi ;
Li, Fei ;
Yang, Haibo .
IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, 2022, 19