On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot

被引:23
作者
Mastropaolo, Antonio [1 ]
Pascarella, Luca [1 ]
Guglielmi, Emanuela [2 ]
Ciniselli, Matteo [1 ]
Scalabrino, Simone [2 ]
Oliveto, Rocco [2 ]
Bavota, Gabriele [1 ]
机构
[1] Univ Svizzera Italiana USI, SEART Software Inst, Lugano, Switzerland
[2] Univ Molise, STAKE Lab, Campobasso, Italy
来源
2023 IEEE/ACM 45TH INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING, ICSE | 2023年
基金
欧洲研究理事会;
关键词
Empirical Study; Recommender Systems; USAGE;
D O I
10.1109/ICSE48619.2023.00181
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
Software engineering research has always being concerned with the improvement of code completion approaches, which suggest the next tokens a developer will likely type while coding. The release of GitHub Copilot constitutes a big step forward, also because of its unprecedented ability to automatically generate even entire functions from their natural language description. While the usefulness of Copilot is evident, it is still unclear to what extent it is robust. Specifically, we do not know the extent to which semantic-preserving changes in the natural language description provided to the model have an effect on the generated code function. In this paper we present an empirical study in which we aim at understanding whether different but semantically equivalent natural language descriptions result in the same recommended function. A negative answer would pose questions on the robustness of deep learning (DL)-based code generators since it would imply that developers using different wordings to describe the same code would obtain different recommendations. We asked Copilot to automatically generate 892 Java methods starting from their original Javadoc description. Then, we generated different semantically equivalent descriptions for each method both manually and automatically, and we analyzed the extent to which predictions generated by Copilot changed. Our results show that modifying the description results in different code recommendations in similar to 46% of cases. Also, differences in the semantically equivalent descriptions might impact the correctness of the generated code (+/- 28%).
引用
收藏
页码:2149 / 2160
页数:12
相关论文
共 66 条
  • [1] Learning Natural Coding Conventions
    Allamanis, Miltiadis
    Barr, Earl T.
    Bird, Christian
    Sutton, Charles
    [J]. 22ND ACM SIGSOFT INTERNATIONAL SYMPOSIUM ON THE FOUNDATIONS OF SOFTWARE ENGINEERING (FSE 2014), 2014, : 281 - 293
  • [2] Alon U., 2019, ARXIV
  • [3] Nguyen AT, 2016, 13TH WORKING CONFERENCE ON MINING SOFTWARE REPOSITORIES (MSR 2016), P362, DOI [10.1145/2901739.2901759, 10.1109/MSR.2016.044]
  • [4] Nguyen AT, 2012, PROC INT CONF SOFTW, P69, DOI 10.1109/ICSE.2012.6227205
  • [5] [Anonymous], 2006, Proc. MSR
  • [6] On Source Code Completion Assistants and the Need of a Context-Aware Approach
    Arrebola, Fabio Villamarin
    Aquino Junior, Plinio Thomaz
    [J]. HUMAN INTERFACE AND THE MANAGEMENT OF INFORMATION: SUPPORTING LEARNING, DECISION-MAKING AND COLLABORATION, HCI INTERNATIONAL 2017, PT II, 2017, 10274 : 191 - 201
  • [7] Context-Sensitive Code Completion Tool for Better API Usability
    Asaduzzaman, Muhammad
    Roy, Chanchal K.
    Schneider, Kevin A.
    Hou, Daqing
    [J]. 2014 IEEE INTERNATIONAL CONFERENCE ON SOFTWARE MAINTENANCE AND EVOLUTION (ICSME), 2014, : 621 - 624
  • [8] Avazpour I., 2014, Dimensions and Metrics for Evaluating Recommendation Systems, P245, DOI DOI 10.1007/978-3-642-45135-510
  • [9] Automating extract class refactoring: an improved method and its evaluation
    Bavota, Gabriele
    De Lucia, Andrea
    Marcus, Andrian
    Oliveto, Rocco
    [J]. EMPIRICAL SOFTWARE ENGINEERING, 2014, 19 (06) : 1617 - 1664
  • [10] Learning from Examples to Improve Code Completion Systems
    Bruch, Marcel
    Monperrus, Martin
    Mezini, Mira
    [J]. 7TH JOINT MEETING OF THE EUROPEAN SOFTWARE ENGINEERING CONFERENCE AND THE ACM SIGSOFT SYMPOSIUM ON THE FOUNDATIONS OF SOFTWARE ENGINEERING, 2009, : 213 - 222