What happened with PISA 2018 in Spain? An explanation based on response times to items.

Abstract

In December 2019 OECD decided not to publish Spanish results on Reading for PISA 2018. Apparently, they had found implausible student-response behaviour on a certain number of students. Enough students as to consider not acceptable the results for international comparisons. Months later, they finally published the Reading results, adding a technical note proposing some possible explanations. In this paper, we show that was the test structure what caused the problem. Specifically, the presence at the beginning of the test of the “reading fluency items” and its effect on the final results. We use the time response of those items and the successive performance of the students in the remaining of the test to find which students had an odd behaviour. Later, using a loglinear multilevel model, we found what characterized those students. The application period, the student motivation and the immigration status are relevant variables. The private or public condition of the school, or the gender of the students are not a relevant to predict a student odd behaviour. Finally, the comparison with the results without the reading fluency items, show the repercussion of these for certain CCAA.
 
Keywords: PISA 2018, Reading fluency, rapid guessing, process data, odd behaviour, loglinear model, Reading performance.