How accurate are news mentions of scholarly output? A content analysis

News mentions are considered as useful source for measuring the societal impact of scholarly output, meanwhile data quality plays a fundamental role in its research and application. This study is aimed to measure the accuracy of news mentions data in the altmetrics database, in order to inform the reliability and limitation of relevant news altmetrics studies. In total, 5.83 million news mentions records that involve 1.03 million scholarly outputs were extracted from the whole dataset up to December 2019 provided by the Altmetric database. 3000 records were sampled for content analysis using stratified sampling strategy. Results show that: (1) 6 major types and 14 specific error types are identified. (2) Error occurs in 42.5% of the sample records, 27.1% could be attributable to the news platform and 15.4% could be attributable to the Altmetric database. (3) Inaccessibility to the source news article (25.9%), incorrect news link provided by the Altmetric database (6.9%) and inaccurate news mention (7.9%) are found to be the three most common error types. (4) 8.5% of the sample records have errors that would cause miscalculation and undermine the validity of studies based on the data, while 33.8% of the sample records have errors that would influence the reliability and reproductivity. (5) Underlying reasons for the errors are summarized and possible measures to improve the data quality are discussed in an in-depth and systematic way. These results suggest that although the Altmetric database has made great achievements in collecting news altmetrics data, the data collection can be further improved.