Data-Driven Approach for Spellchecking and Autocorrection

被引:3
作者
Toleu, Alymzhan [1 ]
Tolegen, Gulmira [1 ]
Mussabayev, Rustam [1 ]
Krassovitskiy, Alexander [1 ]
Ualiyeva, Irina [2 ]
机构
[1] Inst Informat & Computat Technol, Alma Ata 050010, Kazakhstan
[2] Al Farabi Kazakh Natl Univ, Fac Informat Technol, Alma Ata 050040, Kazakhstan
来源
SYMMETRY-BASEL | 2022年 / 14卷 / 11期
关键词
spellchecking; autocorrection; web; data driven; CONTEXT;
D O I
10.3390/sym14112261
中图分类号
O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];
学科分类号
07 ; 0710 ; 09 ;
摘要
This article presents an approach for spellchecking and autocorrection using web data for morphologically complex languages (in the case of Kazakh language), which can be considered an end-to-end approach that does not require any manually annotated word-error pairs. A sizable web of noisy data is crawled and used as a base to infer the knowledge of misspellings with their correct forms. Using the extracted corpus, a sub-string error model with a context model for morphologically complex languages are trained separately, then these two models are integrated with a regularization parameter. A sub-string alignment model is applied to extract symmetric and non-symmetric patterns in two sequences of word-error pairs. The model calculates the probability for symmetric and non-symmetric patterns of a given misspelling and its candidates to obtain a suggestion list. Based on the proposed method, a Kazakh Spellchecking and Autocorrection system is developed, which we refer to as QazSpell. Several experiments are conducted to evaluate the proposed approach from different angles. The results show that the proposed approach achieves a good outcome when only using the error model, and the performance is boosted after integrating the context model. In addition, the developed system, QazSpell, outperforms the commercial analogs in terms of overall accuracy.
引用
收藏
页数:16
相关论文
共 36 条
  • [1] Abdussaitova A., 2019, P STUD RES WORKSH AS, P1, DOI [10.26615/issn.2603-2821.2019_001, DOI 10.26615/ISSN.2603-2821.2019_001]
  • [2] Atkinson Kevin., 2006, Gnu aspell 0.60. 4
  • [3] Brants T., 2007, P 2007 JOINT C EMPIR, P858
  • [4] An improved error model for noisy channel spelling correction
    Brill, E
    Moore, RC
    [J]. 38TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, PROCEEDINGS OF THE CONFERENCE, 2000, : 286 - 293
  • [5] Church Kenneth W., 1991, Statistics and Computing, V1, P93, DOI DOI 10.1007/BF01889984
  • [6] Deorowicz S, 2005, INT J AP MAT COM-POL, V15, P275
  • [7] Devlin J, 2019, 2019 CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS: HUMAN LANGUAGE TECHNOLOGIES (NAACL HLT 2019), VOL. 1, P4171
  • [8] Flor M, 2012, TRAIT AUTOM LANG, V53, P61
  • [9] Gan ZF, 2021, 2021 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (EMNLP 2021), P3487
  • [10] Jayanthi SM, 2020, PROCEEDINGS OF THE 2020 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING: SYSTEM DEMONSTRATIONS, P158