Feedback Generation for Performance Problems in Introductory Programming Assignments

被引:34
作者
Gulwani, Sumit [1 ]
Radicek, Ivan [2 ]
Zuleger, Florian [2 ]
机构
[1] Microsoft Res, Redmond, WA 98052 USA
[2] TU Wien, Vienna, Austria
来源
22ND ACM SIGSOFT INTERNATIONAL SYMPOSIUM ON THE FOUNDATIONS OF SOFTWARE ENGINEERING (FSE 2014) | 2014年
关键词
Education; MOOCs; performance analysis; trace specification; dynamic analysis;
D O I
10.1145/2635868.2635912
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
Providing feedback on programming assignments manually is a tedious, error prone, and time-consuming task. In this paper, we motivate and address the problem of generating feedback on performance aspects in introductory programming assignments. We studied a large number of functionally correct student solutions to introductory programming assignments and observed: (1) There are different algorithmic strategies, with varying levels of efficiency, for solving a given problem. These different strategies merit different feedback. (2) The same algorithmic strategy can be implemented in countless different ways, which are not relevant for reporting feedback on the student program. We propose a light-weight programming language extension that allows a teacher to define an algorithmic strategy by specifying certain key values that should occur during the execution of an implementation. We describe a dynamic analysis based approach to test whether a student's program matches a teacher's specification. Our experimental results illustrate the effectiveness of both our specification language and our dynamic analysis. On one of our benchmarks consisting of 2316 functionally correct implementations to 3 programming problems, we identified 16 strategies that we were able to describe using our specification language (in 95 minutes after inspecting 66, i.e., around 3%, implementations). Our dynamic analysis correctly matched each implementation with its corresponding specification, thereby automatically producing the intended feedback.
引用
收藏
页码:41 / 51
页数:11
相关论文
共 24 条
[1]   LAURA, A SYSTEM TO DEBUG STUDENT PROGRAMS [J].
ADAM, A ;
LAURENT, JP .
ARTIFICIAL INTELLIGENCE, 1980, 15 (1-2) :75-122
[2]  
[Anonymous], PLDI
[3]  
[Anonymous], 2009, P 36 ANN S PRINC PRO
[4]   Pattern matching for permutations [J].
Bose, P ;
Buss, AF ;
Lubiw, A .
INFORMATION PROCESSING LETTERS, 1998, 65 (05) :277-283
[5]   Looper: Lightweight Detection of Infinite Loops at Runtime [J].
Burnim, Jacob ;
Jalbert, Nicholas ;
Stergiou, Christos ;
Sen, Koushik .
2009 IEEE/ACM INTERNATIONAL CONFERENCE ON AUTOMATED SOFTWARE ENGINEERING, PROCEEDINGS, 2009, :161-169
[6]  
Goldsmith S., 2007, ESEC SIGSOFT FSE
[7]  
Gulwani S., 2014, NON TRADITIONAL REF
[8]  
Gulwani S., 2014, ABS14034064 CORR
[9]   The Reachability-Bound Problem [J].
Gulwani, Sumit ;
Zuleger, Florian .
PLDI '10: PROCEEDINGS OF THE 2010 ACM SIGPLAN CONFERENCE ON PROGRAMMING LANGUAGE DESIGN AND IMPLEMENTATION, 2010, :292-304
[10]   Proving Non-Termination [J].
Gupta, Ashutosh K. ;
Henzinger, Thomas A. ;
Majumdar, Rupak ;
Rybalchenko, Andrey ;
Xu, Ru-Gang .
POPL'08: PROCEEDINGS OF THE 35TH ANNUAL ACM SIGPLAN-SIGACT SYMPOSIUM ON PRINCIPLES OF PROGRAMMING LANGUAGES, 2008, :147-158