ChatGPT Performance Evaluation on Chinese Language and Risk Measures

被引：0
|
作者：
Zhang H. [1 ]
Li L. [1 ]
Li C. [1 ]
机构：
[1] School of Computer Science, Beijing Institute of Technology, Beijing
来源：
Data Analysis and Knowledge Discovery | 2023年 / 7卷 / 03期
关键词：
Artificial Intelligence; ChatGPT; Language Model;
D O I：
10.11925/infotech.2096-3467.2023.0214
中图分类号：
学科分类号：
摘要：
[Objective] This paper briefly introduces the main technical innovations of ChatGPT, and evaluates the performance of ChatGPT in Chinese on four tasks using nine datasets, analyzes the risk with ChatGPT and proposes our solutions. [Methods] ChatGPT and WeLM models were tested using the ChnSentiCorp dataset, and ChatGPT and ERNIE 3.0 Titan were tested using the EPRSTMT dataset, and it was found that ChatGPT did not differ much from the large domestic models in sentiment analysis tasks. The LCSTS and TTNews datasets were used to test the ChatGPT and WeLM models, and both ChatGPT outperformed the WeLM model; CMRC2018 and DRCD were used for extractive machine reading comprehension(MRC), and the C3 dataset was used for common sense MRC, and it was found that ERNIE 3.0 Titan outperformed ChatGPT in this task. WebQA and CKBQA were used to do Chinese closed-book quiz testing, and it was found that ChatGPT was prone to make factual errors in this task, and the domestic model outperformed ChatGPT. [Results] ChatGPT performed well on classic tasks of natural language processing, such as sentiment analysis with an accuracy rate of more than 85% and a higher probability of factual errors on closed-book questions. [Limitations] The error of evaluation score may be introduced in the process of converting discriminative tasks into generative ones. This paper only evaluated ChatGPT in zero-shot case, so it is not clear how it performs in other cases. ChatGPT may be updated iteratively in subsequent releases, and the profiling results may be time-sensitive. [Conclusions] ChatGPT is powerful but still has some drawbacks, for the large model of Chinese need to be national strategy oriented and pay attention to the limitations of the language model. © 2022 Food and Fermentation Industries. All rights reserved.
引用
收藏
页码：16 / 25
页数：9
相关论文
共 34 条
  • [1] Brown T, Mann B, Ryder N, Et al., Language Models are Few-Shot Learners, Proceedings of the 34th International Conference on Neural Information Processing Systems, 33, pp. 1877-1901, (2020)
  • [2] Thoppilan R, De Freitas D, Hall J, Et al., LaMDA: Language Models for Dialog Applications [OL]
  • [3] Wang S H, Sun Y, Xiang Y, Et al., ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
  • [4] Zeng W, Ren X, Su T, Et al., PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-Parallel Computation
  • [5] Su H, Zhou X, Yu H J, Et al., WeLM: A Well-Read Pre-trained Language Model for Chinese
  • [6] Kiela D, Bartolo M, Nie Y X, Et al., Dynabench: Rethinking Benchmarking in NLP, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4110-4124, (2021)
  • [7] Zhou J, Ke P, Qiu X P, Et al., ChatGPT: Potential, Prospects, and Limitations[J], Frontiers of Information Technology & Electronic Engineering
  • [8] van Dis E, Bollen J, Zuidema W, Et al., ChatGPT: Five Priorities for Research, Nature, 614, 7947, pp. 224-226, (2023)
  • [9] Thorp H H., ChatGPT is Fun, but Not an Author, Science, 379, 6630, (2023)
  • [10] Qin C W, Zhang A, Zhang Z S, Et al., Is ChatGPT a General-Purpose Natural Language Processing Task Solver? [OL]
1234