Operating Conversational Large Language Models (LLMs)in the Presenceof Errors

被引:0
|
作者
Gao, Zhen [1 ]
Deng, Jie [2 ]
Reviriego, Pedro [3 ]
Liu, Shanshan [4 ]
Pozo, Alejando [3 ]
Lombardi, Fabrizio [5 ]
机构
[1] Tianjin Univ, Sch Elect & Informat Engn, Tianjin 300072, Peoples R China
[2] Tianjin Univ, Sch Future Technol, Tianjin 300072, Peoples R China
[3] Univ Politecn Madrid, ETSI Telecomunicac, Madrid 28040, Spain
[4] Univ Elect Sci & Technol China, Sch Informat & Commun Engn, Chengdu 611731, Peoples R China
[5] Northeastern Univ, Dept Elect & Comp Engn, Boston, MA 02215 USA
关键词
Quantization (signal); Benchmark testing; Transformers; Codes; Translation; Memory management; Logic gates; Integrated circuit modeling; Hardware; Computational modeling; Dependability; generative artificial intelligence; large language models; errors;
D O I
10.1109/MNANO.2024.3513112
中图分类号
TB3 [工程材料学];
学科分类号
0805 ; 080502 ;
摘要
Conversational Large Language Models have taken the center stage of the artificial intelligence landscape. As they are pervasive, there is a need to evaluate their dependability, i.e., performance when errors appear due to the underlying hardware implementation. In this paper we consider the evaluation of the dependability of a widely used conversational LLM: Mistral-7B. Error injection is conducted, and the Multitask Language Understanding (MMLU) benchmark is used to evaluate the impact on performance. The drop in the percentage of correct answers due to errors is analyzed and the results provide interesting insights: Mistral-7B has a large intrinsic tolerance to errors even at high bit error rates. This opens the door to the use of nanotechnologies that trade-off errors for energy dissipation and complexity to further improve the LLM implementation. Also, the error tolerance is larger for 8-bit quantization than for 4-bit quantization, so suggesting that there will be also a trade-off between quantization optimizations to reduce memory requirements and error tolerance. In addition, we also show the different impact of errors on different types of weights, which is valuable information for selective protection designs.
引用
收藏
页码:31 / 37
页数:7
相关论文
共 50 条
  • [31] Enhancing Accessibility in Software Engineering Projects with Large Language Models (LLMs)
    Aljedaani, Wajdi
    Eler, Marcelo Medeiros
    Parthasarathy, P. D.
    PROCEEDINGS OF THE 56TH ACM TECHNICAL SYMPOSIUM ON COMPUTER SCIENCE EDUCATION, SIGCSE TS 2025, VOL 2, 2025, : 25 - 31
  • [32] AGE-RELATED VALUE ORIENTATIONS IN LARGE LANGUAGE MODELS (LLMS)
    Zhang, Xin
    Ren, Yuanyi
    Song, Guojie
    INNOVATION IN AGING, 2024, 8 : 1010 - 1010
  • [33] Harnessing large language models (LLMs) for candidate gene prioritization and selection
    Mohammed Toufiq
    Darawan Rinchai
    Eleonore Bettacchioli
    Basirudeen Syed Ahamed Kabeer
    Taushif Khan
    Bishesh Subba
    Olivia White
    Marina Yurieva
    Joshy George
    Noemie Jourde-Chiche
    Laurent Chiche
    Karolina Palucka
    Damien Chaussabel
    Journal of Translational Medicine, 21
  • [34] ChatGPT effects on cognitive skills of undergraduate students: Receiving instant responses from AI-based conversational large language models (LLMs)
    Essel H.B.
    Vlachopoulos D.
    Essuman A.B.
    Amankwa J.O.
    Computers and Education: Artificial Intelligence, 2024, 6
  • [35] Large Language Models as Zero-Shot Conversational Recommenders
    He, Zhankui
    Xie, Zhouhang
    Jha, Rahul
    Steck, Harald
    Liang, Dawen
    Feng, Yesu
    Majumder, Bodhisattwa Prasad
    Kallus, Nathan
    McAuley, Julian
    PROCEEDINGS OF THE 32ND ACM INTERNATIONAL CONFERENCE ON INFORMATION AND KNOWLEDGE MANAGEMENT, CIKM 2023, 2023, : 720 - 730
  • [36] A conversational agent for creating automations exploiting large language models
    Gallo S.
    Paternò F.
    Malizia A.
    Personal and Ubiquitous Computing, 2024, 28 (06) : 931 - 946
  • [37] Visistant: A Conversational Chatbot for Natural Language to Visualizations With Gemini Large Language Models
    Muthumanikandan, V.
    Ram, Santhosh
    IEEE ACCESS, 2024, 12 : 138547 - 138563
  • [38] Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
    Wang, Xiaolei
    Tang, Xinyu
    Xin, Wayne
    Wang, Jingyuan
    Wen, Ji-Rong
    2023 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (EMNLP 2023), 2023, : 10052 - 10065
  • [39] Large Language Models (LLMs) as Graphing Tools for Advanced Chemistry Education and Research
    Subasinghe, S. M. Supundrika
    Gersib, Simon G.
    Mankad, Neal P.
    JOURNAL OF CHEMICAL EDUCATION, 2025,
  • [40] Capabilities and limitations of AI Large Language Models (LLMs) for materials criticality research
    Ku, Anthony Y.
    Hool, Alessandra
    MINERAL ECONOMICS, 2024,