State of Practice: LLMs in Software Engineering and Software Architecture

Large Language Models (LLMs) are finding their way into Software Engineering by assisting with tasks such as code generation. Furthermore, LLMs might have a potential to perform even more complex tasks, such as suggesting architectural design. However, there is a lack of empirical surveys on how software engineering companies use (and plan to use) LLMs and if LLMs truly can provide benefits to software architects. To understand the state of practice considering adoption of LLMs in software engineering, existing challenges, and future trends, we have surveyed 15 different software engineering companies. To understand the ability of LLMs to perform more complex tasks, we report on our experiments with LLM-assisted architectural design. We applied ChatGPT on 5 software projects and in total performed 50 different experiments. Our results capture the state of the practice of LLMs in software engineering and demonstrate how LLMs perform when assisting with (more complex task such as) architectural design. Engineers, architects, and project managers should profit from these results to guide their decision towards targeted adoption of LLMs in their business and engineering domains.