Development and validity of computerized neuropsychological assessment devices for screening mild cognitive impairment: Ensemble of models with feature space heterogeneity and retrieval practice effect

Objective: This study aimed to develop and validate computerized neuropsychological assessment devices for screening patients with mild cognitive impairment (MCI). Methods: We conducted this study in three phases. Phase I involved the development of a conceptual framework of Memory Guard (MG) based on the principles of the cognitive design system (CDS). Phase II involved three steps of feature engineering: item development, filter, and wrapper. Based on the initial items, the number of items in each dimension was determined through analytic hierarchy process. We constructed an initial set with a total of 198 items with three levels of difficulty. Next, we performed feature selection through comprehensive reliability and validity tests, which resulted in the best item bank of 38 test items. The features for modeling were obtained from the best item bank (option scores, reading time scores and total time scores), demographic variables and their MoCA groups. Regarding the heterogeneity of the feature space, we combined the AdaBoost with the Naive Bayes classification algorithm as the decision model of MG. For the screening tool to be used repeatedly, the retrieval practice effect was considered in the design. Phase III involved the validation of measuring instruments. The features incorporated into the modeling process were optimized based on the classification accuracy and area under curve. We also verified the classification effect of the other three classification models with MG. Results: After three steps of feature engineering, a total of 6 dimensions of cognitive areas were included in MG: orientation, memory, attention, calculation, recall, and language & executive function. 38 features were included in the model (17 features of option score, 20 features of time score, and 1 demographic feature). A total of 333 individuals from two communities in Shanghai and Henan province were included in the measuring instrument verification process. Women accounted for 68.2% of the sample. The median age was 63. 15.3% of the participants had bachelor's degrees or above and 111 participants lived in urban areas (33.3%). The results showed that MG had an accuracy of 93.75% and AUC of 0.923, with a sensitivity of 91.67% and a specificity of 95.45%. Compared to the other three classification models, MG that combined the AdaBoost with the Naive Bayes classification algorithm was the most accurate classifier. Conclusions: MG was proved to be reliable and valid in early screening for patients with MCI. MG that integrated heterogeneous features such as demography, option scores, and time scores had a better predictive performance for screening MCI.