Assessing Inference Time in Large Language Models

Large Language Models have transformed the field of artificial intelligence, yet they are often associated with elitism and inaccessibility. This is primarily due to the large number of their parameters, ranging from 1 billion to 70 billion, making inference on these models costly and resource intensive. To tackle this challenge, various solutions have emerged with the goal of enabling efficient, fast, and resource-constrained inference. This study aims to review and compare these available solutions. The authors conducted a series of experiments that compared the inference speed of the basic HuggingFace transformers library, the HuggingFace Text Generation Inference server, and the open source vLLM library. The findings reveal that vLLM outperforms the other approaches examined. Additionally, the results highlight how relatively straightforward techniques, such as continuous batching, can significantly accelerate inference for large batch sizes.