Better patching using LLM prompting, via Self-Consistency

被引:12
作者
Ahmed, Toufique [1 ]
Devanbu, Premkumar [1 ]
机构
[1] Univ Calif Davis, Davis, CA 95616 USA
来源
2023 38TH IEEE/ACM INTERNATIONAL CONFERENCE ON AUTOMATED SOFTWARE ENGINEERING, ASE | 2023年
基金
美国国家科学基金会;
关键词
LLMs; Self-consistency; Program Repair;
D O I
10.1109/ASE56229.2023.00065
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Large Language models (LLMs) can be induced to solve non-trivial problems with "few-shot" prompts including illustrative problem-solution examples. Now if the few-shots also include "chain of thought" (CoT) explanations, which are of the form problem-explanation-solution, LLMs will generate a "explained" solution, and perform even better. Recently an exciting, substantially better technique, self-consistency [1] (S-C) has emerged, based on the intuition that there are many plausible explanations for the right solution; when the LLM is sampled repeatedly to generate a pool of explanation-solution pairs, for a given problem, the most frequently occurring solutions in the pool (ignoring the explanations) tend to be even more likely to be correct! Unfortunately, the use of this highly-performant S-C (or even CoT) approach in software engineering settings is hampered by the lack of explanations; most software datasets lack explanations. In this paper, we describe an application of the S-C approach to program repair, using the commit log on the fix as the explanation, only in the illustrative few-shots. We achieve state-of-the art results, beating previous approaches to prompting-based program repair, on the MODIT dataset; we also find evidence suggesting that the correct commit messages are helping the LLM learn to produce better patches.
引用
收藏
页码:1742 / 1746
页数:5
相关论文
共 27 条
[1]  
Ahmed T, 2024, Arxiv, DOI arXiv:2304.06815
[2]   Few-shot training LLMs for project-specific code-summarization [J].
Ahmed, Toufique ;
Devanbu, Premkumar .
PROCEEDINGS OF THE 37TH IEEE/ACM INTERNATIONAL CONFERENCE ON AUTOMATED SOFTWARE ENGINEERING, ASE 2022, 2022,
[3]  
Bareiss P, 2022, Arxiv, DOI [arXiv:2206.01335, 10.48550/arXiv.2206.01335, DOI 10.48550/ARXIV.2206.01335]
[4]  
Berabi B, 2021, PR MACH LEARN RES, V139
[5]  
Brown TB, 2020, ADV NEUR IN, V33
[6]   NatGen: Generative Pre-training by "Naturalizing" Source Code [J].
Chakraborty, Saikat ;
Ahmed, Toufique ;
Ding, Yangruibo ;
Devanbu, Premkumar T. ;
Ray, Baishakhi .
PROCEEDINGS OF THE 30TH ACM JOINT MEETING EUROPEAN SOFTWARE ENGINEERING CONFERENCE AND SYMPOSIUM ON THE FOUNDATIONS OF SOFTWARE ENGINEERING, ESEC/FSE 2022, 2022, :18-30
[7]   On Multi-Modal Learning of Editing Source Code [J].
Chakraborty, Saikat ;
Ray, Baishakhi .
2021 36TH IEEE/ACM INTERNATIONAL CONFERENCE ON AUTOMATED SOFTWARE ENGINEERING ASE 2021, 2021, :443-455
[8]  
Chen M., 2021, Evaluating large language models trained on code
[9]   SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair [J].
Chen, Zimin ;
Kommrusch, Steve ;
Tufano, Michele ;
Pouchet, Louis-Noel ;
Poshyvanyk, Denys ;
Monperrus, Martin .
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 2021, 47 (09) :1943-1959
[10]  
Chowdhery A, 2022, Arxiv, DOI arXiv:2204.02311