Improving requirements completeness: automated assistance through large language models

被引:9
作者
Luitel, Dipeeka [1 ]
Hassani, Shabnam [1 ]
Sabetzadeh, Mehrdad [1 ]
机构
[1] Univ Ottawa, 800 King Edward Ave, Ottawa, ON K1N 6N5, Canada
基金
加拿大自然科学与工程研究理事会;
关键词
Requirements completeness; Natural language processing (NLP); Machine learning (ML); Large language models (LLMs); BERT;
D O I
10.1007/s00766-024-00416-3
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Natural language (NL) is arguably the most prevalent medium for expressing systems and software requirements. Detecting incompleteness in NL requirements is a major challenge. One approach to identify incompleteness is to compare requirements with external sources. Given the rise of large language models (LLMs), an interesting question arises: Are LLMs useful external sources of knowledge for detecting potential incompleteness in NL requirements? This article explores this question by utilizing BERT. Specifically, we employ BERT's masked language model to generate contextualized predictions for filling masked slots in requirements. To simulate incompleteness, we withhold content from the requirements and assess BERT's ability to predict terminology that is present in the withheld content but absent in the disclosed content. BERT can produce multiple predictions per mask. Our first contribution is determining the optimal number of predictions per mask, striking a balance between effectively identifying omissions in requirements and mitigating noise present in the predictions. Our second contribution involves designing a machine learning-based filter to post-process BERT's predictions and further reduce noise. We conduct an empirical evaluation using 40 requirements specifications from the PURE dataset. Our findings indicate that: (1) BERT's predictions effectively highlight terminology that is missing from requirements, (2) BERT outperforms simpler baselines in identifying relevant yet missing terminology, and (3) our filter reduces noise in the predictions, enhancing BERT's effectiveness for completeness checking of requirements.
引用
收藏
页码:73 / 95
页数:23
相关论文
共 54 条
[1]  
Abbas M, 2021, 27 INT WORKING C REQ
[2]  
Alrajeh D, 2012, PROC INT CONF SOFTW, P705, DOI 10.1109/ICSE.2012.6227147
[3]   AI-Enabled Automation for Completeness Checking of Privacy Policies [J].
Amaral, Orlando ;
Abualhaija, Sallam ;
Torre, Damiano ;
Sabetzadeh, Mehrdad ;
Briand, Lionel C. .
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 2022, 48 (11) :4647-4674
[4]  
[Anonymous], 2019, Speech and Language Processing:An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition
[5]   An empirical study on the potential usefulness of domain models for completeness checking of requirements [J].
Arora, Chetan ;
Sabetzadeh, Mehrdad ;
Briand, Lionel C. .
EMPIRICAL SOFTWARE ENGINEERING, 2019, 24 (04) :2509-2539
[6]   An Active Learning Approach for Improving the Accuracy of Automated Domain Model Extraction [J].
Arora, Chetan ;
Sabetzadeh, Mehrdad ;
Nejati, Shiva ;
Briand, Lionel .
ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY, 2019, 28 (01)
[7]   Automated Extraction and Clustering of Requirements Glossary Terms [J].
Arora, Chetan ;
Sabetzadeh, Mehrdad ;
Briand, Lionel ;
Zimmer, Frank .
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 2017, 43 (10) :918-945
[8]   Automated Checking of Conformance to Requirements Templates Using Natural Language Processing [J].
Arora, Chetan ;
Sabetzadeh, Mehrdad ;
Briand, Lionel ;
Zimmer, Frank .
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 2015, 41 (10) :944-968
[9]  
Bergstra J, 2012, J MACH LEARN RES, V13, P281
[10]  
Berry D.M., 2003, CONTRACT DRAFTING SO