Trial lecture - time and place
Place: Auditorium 2, Helga Engs hus
Time: 10:00 - 10:45
Title: Automated Item Generation: Evolution, Opportunities, and Challenges in Assessment Programs
Public Defence
Place: Auditorium 2, Helga Engs hus
Time: 12:00 -
Evaluation committee
- 1st opponent: Professor Kajsa Yang Hansen, University of Gothenburg, Sweden
- 2nd opponent: Associate Professor Michalis Michaelides, University of Cyprus, Cyprus
- Chair of committee: Associate Professor Esther Ulitzsch, University of Oslo, Norway
Chair of defence
- Head of Department Rolf Vegar Olsen, Centre for Educational Measurement, University of Oslo, Norway
Supervisors
- Professor Johan Braeken, CEMO - Centre for Educational Measurement, University of Oslo, Norway
Summary
Research on International Large-Scale Assessments (ILSAs), such as the Programme for International Student Assessment (PISA), has traditionally focused on student performance and the associated school or system-level determinants. Within this student-centric approach, the assessment items —the core building blocks of ILSAs—often receive less attention despite their potential to provide insights when examined more closely. This thesis pivots the focus towards these core elements in ILSAs. Using the PISA 2018 response data from over 70 countries, this thesis aims to explore the “item side” of PISA, demonstrating how examining assessment items in ILSAs can offer countless possibilities for item-level validation, test design improvement, and enhancing our understanding of student performance across diverse educational settings.
To situate this item-centric exploration within the broader field of ILSA research, the thesis broadens its scope along three dimensions. First, it encompasses a broad geographic coverage, using response data from every participating country and
jurisdiction in the PISA 2018, allowing for cross-country generalisability and identifying patterns across regions. Second, the thesis spans a wide range of PISA items, both from the achievement tests and the student questionnaire, thus permitting a detailed investigation of various item features. Lastly, the thesis hones in on two outcomes: item difficulty and item nonresponse, illustrating how item features can influence various outcomes.
The thesis is organised into two applications. Application I explores how item
features influence item difficulty within the PISA 2018 achievement tests (Articles I,
II, and III). Here, the findings show (i) considerable differences in item difficulties, a
vast source of largely untapped potential for ILSA research, and (ii) how these differences in item difficulties relate to construct-relevant and -irrelevant item features
to varying extent across different countries. Application II (Articles IV and V) examines how item features such as item position within the student questionnaire and the scale’s format and content relate to item nonresponse patterns in the PISA 2018 student background questionnaire and how these item features interact with student characteristics, such as gender, resulting in systematic patterns of missing data across countries. Both applications have implications for assessment design, validity, and comparability of results across countries.
In summary, this thesis advocates for research on the overlooked ‘item side’ of ILSAs, highlighting the importance of items and their features in shaping various ILSA outcomes and demonstrating the potential of item-centric research to inform future test design, enhance fairness, comparability, and validity of ILSA results, and promote a more nuanced understanding of global educational trends.