Pitfalls in the assessment of higher education learning gains in Brazil through multilevel regression: handling the ceiling effect

被引:0
作者
Lins, Marcos Estellita [1 ,2 ]
Camara, Musbah Koleilat [2 ,3 ]
de Souza, Ana Luiza Lima [2 ,4 ]
机构
[1] Fed Univ State Rio de Janeiro, Dept Prod Engn, Rio De Janeiro, Brazil
[2] Fed Univ State Rio de Janeiro, COPPE, Prod Engn Program, Rio De Janeiro, Brazil
[3] Fed Ctr Technol Educ Celso Suckow Fonseca, Dept Informat Syst, Rio De Janeiro, Brazil
[4] Fed Ctr Technol Educ Celso Suckow Fonseca, Prod Engn Dept, Rio De Janeiro, Brazil
关键词
Learning gains; multilevel regression; HEI assessment; ceiling effect; VALUE-ADDED MODELS; OUTCOMES; SCHOOL;
D O I
10.1051/ro/2024208
中图分类号
C93 [管理学]; O22 [运筹学];
学科分类号
070105 ; 12 ; 1201 ; 1202 ; 120202 ;
摘要
This work assesses the validity of the Brazilian Indicator of the Difference between Observed and Expected Performances (IDD), which applies a multilevel regression model to rank university courses. It proposes a methodology to test for inconsistencies and research for the causes. With this objective, we develop an experimental methodology, applied to a database with 4432 production engineering students distributed over 329 undergraduate courses, from the government's National Institute for Educational Studies and Research. We display a graphical analysis of the expected and achieved grades in final graduation exams, the gap in which determines the learning gains for every course. Then we apply the multilevel linear regression model of the IDD in a particular scenario, to test for consistency in value-added learning measurements using the technique reductio ad absurdum. The result reveals that the present methodology for the calculation of the IDD is inadequate and inconsistent, and that this can be explained by the ceiling effect. It needs to be radically revised to prevent unfair judgements. Finally, we propose and implement a production frontier technique to correct for the ensuing ceiling effect, namely a variable returns to scale data envelopment analysis model for adjusting the measurements using evidence based benchmarks.
引用
收藏
页码:4955 / 4969
页数:15
相关论文
共 46 条
[1]   ESTIMATED STUDENT SCORE GAIN ON THE ACT COMP EXAM - VALID TOOL FOR INSTITUTIONAL ASSESSMENT [J].
BANTA, TW ;
LAMBERT, EW ;
PIKE, GR ;
SCHMIDHAMMER, JL ;
SCHNEIDER, JA .
RESEARCH IN HIGHER EDUCATION, 1987, 27 (03) :195-217
[2]   Conditions and Consequences of Unequal Educational Opportunities in the Life Course: Results from the Cross-National Comparative eduLIFE Project [J].
Blossfeld, Hans-Peter ;
Kulic, Nevena ;
Skopek, Jan ;
Triventi, Moris ;
Kilpi-Jakonen, Elina ;
de Vilhena, Daniela Vono ;
Buchholz, Sandra .
KOLNER ZEITSCHRIFT FUR SOZIOLOGIE UND SOZIALPSYCHOLOGIE, 2019, 71 (Suppl 1) :399-428
[3]  
Bogoya J.D., 2013, Ingeniera e Investigacin, P76
[4]  
Borden V.M. H., 2008, New Directions for Institutional Research, VS1, P19
[5]  
Brasil L., 2006, Presidencia da Repblica
[6]   The performance of regional governments under the results-based budgeting framework: A two-stage sectoral analysis [J].
Charles, Vincent ;
Diaz, Juan ;
Aparicio, Juan .
RAIRO-OPERATIONS RESEARCH, 2022, 56 (02) :501-528
[7]  
Coates H., 2009, Higher Educ. Manage. Policy, P1
[8]   Measuring value-added in higher education: Possibilities and limitations in the use of administrative data [J].
Cunha, Jesse M. ;
Miller, Trey .
ECONOMICS OF EDUCATION REVIEW, 2014, 42 :64-77
[9]   Preface to the Special issue on Efficiency in Education, Health and Other Public Services [J].
De Witte, Kristof ;
Johnes, Geraint ;
Johnes, Jill ;
Agasisti, Tommaso .
INTERNATIONAL TRANSACTIONS IN OPERATIONAL RESEARCH, 2020, 27 (04) :1819-1820
[10]   Assessment of the European Union's Educational Efficiency [J].
Dinca, Marius Sorin ;
Dinca, Gheorghita ;
Andronic, Maria Letitia ;
Pasztori, Anna Maria .
SUSTAINABILITY, 2021, 13 (06)