Productivity in high-performance computing

Productivity is an emerging measure of merit for high-performance computing. While pervasive in application, conventional metrics such as flops fail to reflect the complex interrelationships of diverse factors that determine the overall effectiveness of the use of a computing system. As a consequence, comparative analysis of design and procurement decisions based on such parameters is insufficient to deliver highly reliable conclusions and often demands detailed benchmarking to augment the more broad system specifications. Even these assessment methodologies tend to exclude important usage factors such as programmability, software portability and cost. In recent years, the HPC community has been seeking more advanced means of assessing the overall value of high-end computing systems. One approach has been to extend the suite of benchmarks typically employed for comparative examination to exercise more aspects of system operational behavior. Another strategy is to devise a richer metric for evaluation that more accurately reflects the relationship of a system class to the demands of the real-world user workflow. One such measure of quality of computing is 'productivity', a parameter that is sensitive to a wide range of factors that describe the usage experience and effectiveness of a computational workflow. Beyond flops count or equivalent metrics, productivity reflects elements of programmability, availability, system and usage cost and the utility of the results achieved, which may be time critical. In contrast to a single measure, productivity is a class of quantifiable predictors that may be adjusted to reveal best understanding of system merit and sensitivity to configuration choices. This chapter will discuss the set of issues leading to one or more formulations of the productivity, describe such basic formulations and their specific application and consider the wealth of system and usage parameters that may contribute to ultimate evaluation. The paper will conclude with a discussion of open issues that still need to be resolved in order to enable productivity to serve as a final arbiter in comparative analysis of design choices for system hardware and software.