Fully Autonomous Programming with Large Language Models

被引:8
|
作者
Liventsev, Vadim [1 ,2 ]
Grishina, Anastasiia [3 ,4 ]
Harma, Aki [2 ]
Moonen, Leon [3 ,5 ]
机构
[1] TU Eindhoven, Eindhoven, Netherlands
[2] Philips Res, Eindhoven, Netherlands
[3] Simula, Oslo, Norway
[4] Univ Oslo, Oslo, Norway
[5] BI Norwegian Business Sch, Oslo, Norway
来源
PROCEEDINGS OF THE 2023 GENETIC AND EVOLUTIONARY COMPUTATION CONFERENCE, GECCO 2023 | 2023年
关键词
automatic programming; large language models; program repair;
D O I
10.1145/3583131.3590481
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Current approaches to program synthesis with Large Language Models (LLMs) exhibit a "near miss syndrome": they tend to generate programs that semantically resemble the correct answer (as measured by text similarity metrics or human evaluation), but achieve a low or even zero accuracy as measured by unit tests due to small imperfections, such as the wrong input or output format. This calls for an approach known as Synthesize, Execute, Debug (SED), whereby a draft of the solution is generated first, followed by a program repair phase addressing the failed tests. To effectively apply this approach to instruction-driven LLMs, one needs to determine which prompts perform best as instructions for LLMs, as well as strike a balance between repairing unsuccessful programs and replacing them with newly generated ones. We explore these trade-offs empirically, comparing replace-focused, repair-focused, and hybrid debug strategies, as well as different template-based and model-based prompt-generation techniques. We use OpenAI Codex as the LLM and Program Synthesis Benchmark 2 as a database of problem descriptions and tests for evaluation. The resulting framework outperforms both conventional usage of Codex without the repair phase and traditional genetic programming approaches.
引用
收藏
页码:1146 / 1155
页数:10
相关论文
共 50 条
  • [41] DriveLLM: Charting the Path Toward Full Autonomous Driving With Large Language Models
    Cui, Yaodong
    Huang, Shucheng
    Zhong, Jiaming
    Liu, Zhenan
    Wang, Yutong
    Sun, Chen
    Li, Bai
    Wang, Xiao
    Khajepour, Amir
    IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, 2024, 9 (01): : 1450 - 1464
  • [42] Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
    Sarsa, Sami
    Denny, Paul
    Hellas, Arto
    Leinonen, Juho
    PROCEEDINGS OF THE 2022 ACM CONFERENCE ON INTERNATIONAL COMPUTING EDUCATION RESEARCH, ICER 2022, VOL. 1, 2023, : 27 - 43
  • [43] Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
    Reynolds, Laria
    McDonell, Kyle
    EXTENDED ABSTRACTS OF THE 2021 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS (CHI'21), 2021,
  • [44] Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
    Alkafaween, Umar
    Albluwi, Ibrahim
    Denny, Paul
    JOURNAL OF COMPUTER ASSISTED LEARNING, 2025, 41 (01)
  • [45] Applying Large Language Models to Enhance the Assessment of Parallel Functional Programming Assignments
    Grandel, Skyler
    Schmidt, Douglas C.
    Leach, Kevin
    2024 INTERNATIONAL WORKSHOP ON LARGE LANGUAGE MODELS FOR CODE, LLM4CODE 2024, 2024, : 102 - 110
  • [46] Towards the Integration of Large Language Models in an Object-Oriented Programming Course
    Cipriano, Bruno Pereira
    PROCEEDINGS OF THE 2024 CONFERENCE INNOVATION AND TECHNOLOGY IN COMPUTER SCIENCE EDUCATION, VOL 2, ITICSE 2024, 2024, : 832 - 833
  • [47] Generating and Reviewing Programming Codes with Large Language Models A Systematic Mapping Study
    Lins de Albuquerque, Beatriz Ventorini
    Souza da Cunha, Antonio Fernando
    Souza, Leonardo
    Matsui Siqueira, Sean Wolfgand
    dos Santos, Rodrigo Pereira
    PROCEEDINGS OF THE 20TH BRAZILIAN SYMPOSIUM ON INFORMATIONS SYSTEMS, SBSI 2024, 2024,
  • [48] AutoAlign: Fully Automatic and Effective Knowledge Graph Alignment Enabled by Large Language Models
    Zhang, Rui
    Su, Yixin
    Trisedya, Bayu Distiawan
    Zhao, Xiaoyan
    Yang, Min
    Cheng, Hong
    Qi, Jianzhong
    IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, 2024, 36 (06) : 2357 - 2371
  • [49] Integrating Large Language Models and Metaverse in Autonomous Racing: An Education-Oriented Perspective
    Li, Bai
    Xu, Tian'ao
    Li, Xinyuan
    Cui, Yaodong
    Bian, Xuepeng
    Teng, Siyu
    Ma, Siji
    Fan, Lili
    Tian, Yonglin
    Wang, Fei-Yue
    IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, 2024, 9 (01): : 59 - 64
  • [50] Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles
    Cui, Can
    Ma, Yunsheng
    Cao, Xu
    Ye, Wenqian
    Wang, Ziran
    IEEE INTELLIGENT TRANSPORTATION SYSTEMS MAGAZINE, 2024, 16 (04) : 81 - 94