Can Large-Language Models Replace Humans in Agile Effort Estimation? Lessons from a Controlled Experiment

Effort estimation is critical in software engineering to assess the resources needed for development tasks and to enable realistic commitments in agile iterations. This study investigates whether generative AI tools, which are transforming various aspects of software development, can improve effort estimation efficiency. A controlled experiment was conducted in which development teams upgraded an existing information system, with the experimental group using the generative-AI-based tool GitLab Duo for estimation and the control group using conventional methods (e.g., planning poker or analogy-based planning). Results show that while generative-AI-based estimation tools achieved only 16% accuracy-currently insufficient for industry standards-they offered valuable support for task breakdown and iteration planning. Participants noted that a combination of conventional methods and AI-based tools could offer enhanced accuracy and efficiency in future planning.