The blind side: Exploring item difficulty in international large-scale assessments (PhD project)
About the project
Research on the International Large Scale Assessments (ILSAs) focuses predominantly on response differences between pupils in their broader school contexts, whereas the investigation of the response differences between items rarely exceeds the original piloting, scaling, and computation of the plausible values. This imbalance is seemingly justified as pupils, teachers, and their respective schools are the key stakeholders in education. However, in assigning such superiority to the analysis of test takers’ performance, a void is created as that of the items is overlooked. Fundamentally, however, an understanding of what makes an item difficult or easy is instrumental to the teachers, curriculum designers, assessment designers and item writers.
This project focuses on the content- or construct-irrelevant factors and their relation to the item difficulty across ILSA studies. The motivation behind the choice of factors to address is twofold. Firstly, if the item difficulty is a function of content-irrelevant factors, the validity of the assessment would be a concern. Furthermore, if those content-irrelevant factors are unstable across countries, they form potential sources of differential item functioning, consequently hindering the comparability of countries’ performance which is at the core of ILSAs. To our knowledge, there exists no comprehensive account on the influence of the content-irrelevant factors on the item difficulty across countries and administrations within one ILSA or across similar domains of different ILSA studies.
The preliminary goals we seek to address are as follows:
• Establish a variance structure in ILSAs by quantifying the magnitude of the variance which can be attributed to items;
• Explore the impact of the content-irrelevant factors on the item difficulty across countries and administrations within one ILSA study;
• Explore whether the impact of the content-irrelevant factors on the item difficulty holds when comparing similar constructs across ILSAs.
The project runs from September 2020.