Brown Bag Seminar: Kseniia Marcq

Title: The blind side: Exploring item variance in PISA 2018 cognitive domains

Abstract: Communication of International Large-Scale Assessment (ILSA) results is largely dominated by reporting of average country performance that conceals individual differences in performance across pupils, schools, and items. While a great deal of educational research focuses on investigating these differences between pupils and schools, differences between items are overlooked. Given the broadness of the ILSA’s cognitive constructs, however, we hypothesize that the items may present a more substantial source of differences, and consequently, variance, than the well-defined populations of pupils and schools. If true, further inquiry into the item variance and the factors affecting its magnitude is instrumental in promoting a more comprehensive item-level validation framework and providing better diagnostic tools for future test development. Using PISA 2018 as a working example, we test this hypothesis by deconstructing the total response variance into parts attributable to three sources of differences, namely between pupils, schools, and items. Ultimately, we showcase the item variance magnitude as compared to the combined pupil and school variance.

The Brown Bag seminars are given via Zoom and require registration. To register, please visit:

Published June 2, 2021 2:12 PM - Last modified June 2, 2021 2:12 PM