Kseniia Marcq

• Braeken, Johan; Lehre, Anne- Catherine W G & Marcq, Kseniia (2025). Pearls and perils when interpreting item-specific results from international large-scale educational assessments: The case of the ‘‘Nordic’’ fractions. Studies in Educational Evaluation. ISSN 0191-491X. 88. doi: 10.1016/j.stueduc.2025.101552. Full text in Research Archive Show summary

  International large-scale assessments (ILSA) in education generate a lot of interest and debate among researchers and stakeholders alike. In this study, we take a deep-dive into a provocative claim that ‘‘Nordic students cannot do fractions’’ based on an item-specific result in the Trends in International Mathematics and Science Study (TIMSS) 2011. Using item response data from five cycles of the TIMSS mathematics achievement test, spanning from 2003 to 2019, across 12 countries representing diverse geographical regions (Africa, Europe, East Asia, and North America) and proficiency levels (from lower-performers to higher-performers), including about 54000 students per cycle and a total of about 800 mathematics items, we investigated the empirical basis and generalizability of the stated claim using an explanatory item response model (IRT) approach. Our study underscores the dangers of miscommunication, oversimplification, and overgeneralization of ILSA results, yet also highlights the hidden potential of deeper item-level analyses for educational researchers and stakeholders to generate critical inquiries into educational practice and our understanding of the subject domain.

• Marcq, Kseniia & Braeken, Johan (2025). From framework to functionality: A cross-country analysis of PISA 2018 reading assessment framework’s item features as determinants of item difficulty. Large-scale Assessments in Education. doi: 10.1186/s40536-025-00261-y. Full text in Research Archive Show summary

  Background Theoretical frameworks excel in conceptualising reading literacy, yet their value hinges on their applicability for real-world purposes, such as assessment. By combining diverse theoretical frameworks, the Programme for International Student Assessment (PISA) 2018 designed an assessment framework for assessing the reading literacy of over 600,000 students globally. The PISA 2018 framework was operationalised into assessment items through construct-relevant item features aimed at determining item difficulty by mirroring the expectations set by the assessment framework. Methods Using cross-classified mixed effects models, we model the PISA 2018 reading item difficulty as a function of the PISA-defined construct-relevant and -irrelevant item features to gauge the extent to which item features determined item difficulty on average and across 71 countries. Results All five, delineated by the PISA framework, construct-relevant item features combined only reduce the variance in item difficulty by an average of 20%, with ‘Text Type’ being the only promising predictor. However, the construct-irrelevant ‘Item Format’ alone has half the predictive value of all construct-relevant features combined (10%), implying that external factors may unduly influence student outcomes. Conclusions These findings suggest that the functionality of item features within the framework leaves much to be desired, pointing to considerable shortcomings that necessitate a re-evaluation of the framework and its more effective implementation. We discuss implications for future PISA design and other stakeholders and call for bridging the gap between the PISA reading assessment framework and its practical application to ensure its validity, effectiveness, and credibility among educational stakeholders worldwide.

• Marcq, Kseniia; Donayre, Erika Jassuly Chalen & Braeken, Johan (2024). The role of item format in the PISA 2018 mathematics literacy assessment: A cross-country study. Studies in Educational Evaluation. ISSN 0191-491X. 83, p. 1–14. doi: 10.1016/j.stueduc.2024.101401. Full text in Research Archive Show summary

  When construct-irrelevant sources affect item difficulty, validity of the assessment is compromised. Using responses of 260000 students from 71 countries to the Programme for International Student Assessment (PISA) 2018 mathematics assessment and cross-classified mixed effects models, we examined three validity concerns associated with the construct-irrelevant factor, item format: whether the format influenced item difficulty, whether item format’s impact on difficulty varied across countries, undermining PISA’s foundational goal of meaningful country comparisons, and whether item format effects differed between genders, affecting assessment fairness. Item format contributed to a substantial average of 12 % of variance in item difficulties. The effect of item format was non-uniform across countries, with 30 % of the variance in item difficulties being due to format in lower-performing countries, and 10 % in higher-performing countries, challenging the comparability of educational outcomes. The impact of gender on item format differences was minor. Implications for secondary research and assessment design are discussed.

• Marcq, Kseniia & Braeken, Johan (2024). Where nonresponse is at its loudest: Cross-country and individual differences in item nonresponse across the PISA 2018 student questionnaire. International Journal of Testing. ISSN 1530-5058. p. 1–28. doi: 10.1080/15305058.2024.2377550. Full text in Research Archive Show summary

  The Programme for International Student Assessment (PISA) student questionnaire, despite being designed for low cognitive demand, may induce test burden due to its 306-item length, resulting in increased item nonresponse toward the questionnaire’s end. Using the PISA 2018 response data from 80 countries and a cross-classified mixed effects model, this study explored the average, cross-country, and individual nonresponse trends throughout the questionnaire. The average nonresponse probability increased from 2% at the questionnaire’s start to 11% at its end. East Asian countries displayed the flattest nonresponse trends, while South American countries showed the steepest. Individual differences were also prominent: students with a lower initial nonresponse propensity experienced a steeper increase in their nonresponse propensity as they progressed through the questionnaire, whereas those with a higher initial propensity remained at a relatively constant level. Our findings highlight the importance of the questionnaire design, the context in which it is administered, and its interaction with individual and cultural factors in shaping student response behavior. We discuss the implications for questionnaire design and researchers using PISA student questionnaire data for secondary analysis.

• Marcq, Kseniia & Braeken, Johan (2023). Gender differences in item nonresponse in the PISA 2018 student questionnaire. Educational Assessment, Evaluation and Accountability. ISSN 1874-8597. 36(3), p. 301–336. doi: 10.1007/s11092-023-09412-7. Full text in Research Archive Show summary

  Abstract Gender differences in item nonresponse are well-documented in high-stakes achievement tests, where female students are shown to omit more items than male students. These gender differences in item nonresponse are often linked to differential risk-taking strategies, with females being risk-averse and unwilling to guess on an item, even if it could gain them credits. In low-stakes settings, similar trends should not apply, as the students carry no consequence for their performance. Instead, test-taking motivation is argued to be the pivoting factor, with female students seen as more motivated and omitting fewer items than male students. In contrast to the high- and low-stakes achievement tests, less is known about gender differences in item nonresponse in student background questionnaires. Using cross-classified mixed effects models, we examined gender differences in item nonresponse on the Programme for International Student Assessment (PISA) 2018 student questionnaire across 80 countries and 71 scales. On average, the odds of male students omitting a questionnaire item were double the odds of female students, consistent with the expected trend in the low-stakes setting. However, we show that gender differences in item nonresponse are not merely a function of the stakes involved for individual students but a more complex phenomenon that is context-dependent and not necessarily stable across countries, scales’ formats, and contents. We argue that examining differences in item nonresponse patterns could serve as a source of additional information about the students’ test-taking behaviour and the quality of the questionnaire.

• Marcq, Kseniia & Andersson, Björn (2022). Standard Errors of Kernel Equating: Accounting for Bandwidth Estimation. Applied Psychological Measurement. ISSN 0146-6216. 46(3). doi: 10.1177/01466216211066601. Full text in Research Archive Show summary

  In standardized testing, equating is used to ensure comparability of test scores across multiple test administrations. One equipercentile observed-score equating method is kernel equating, where an essential step is to obtain continuous approximations to the discrete score distributions by applying a kernel with a smoothing bandwidth parameter. When estimating the bandwidth, additional variability is introduced which is currently not accounted for when calculating the standard errors of equating. This poses a threat to the accuracy of the standard errors of equating. In this study, the asymptotic variance of the bandwidth parameter estimator is derived and a modified method for calculating the standard error of equating that accounts for the bandwidth estimation variability is introduced for the equivalent groups design. A simulation study is used to verify the derivations and confirm the accuracy of the modified method across several sample sizes and test lengths as compared to the existing method and the Monte Carlo standard error of equating estimates. The results show that the modified standard errors of equating are accurate under the considered conditions. Furthermore, the modified and the existing methods produce similar results which suggest that the bandwidth variability impact on the standard error of equating is minimal.

• Marcq, Kseniia & Braeken, Johan (2022). The blind side: Exploring item variance in PISA 2018 cognitive domains. Assessment in Education: Principles, Policy & Practice. ISSN 0969-594X. 29(3), p. 332–360. doi: 10.1080/0969594X.2022.2097199. Full text in Research Archive Show summary

  Communication of International Large-Scale Assessment (ILSA) results is dominated by reporting average country achievement scores that conceal individual differences between pupils, schools, and items. Educational research primarily focuses on examining differences between pupils and schools, while differences between items are overlooked. Using a variance components model on the Programme for International Student Assessment (PISA) 2018 cognitive domains of reading, mathematics, and science literacy, we estimated how much of the response variation can be attributed to differences between pupils, schools, and items. The results show that uniformly across domains and countries, it mattered more for the correctness of an item response which items were responded to by a pupil (27–35%) than which pupil responded to these items (10–12%) or which school the pupil attended (5–7%). Given the findings, we argue that differences between items in ILSAs constitute a source of substantial untapped potential for secondary research.

• Andersson, Björn; Luo, Hao & Marcq, Kseniia (2022). Reliability coefficients for multiple group item response theory models. British Journal of Mathematical & Statistical Psychology. ISSN 0007-1102. doi: 10.1111/bmsp.12269. Full text in Research Archive Show summary

  Reliability of scores from psychological or educational assessments provides important information regarding the precision of measurement. The reliability of scores is however population dependent and may vary across groups. In item response theory, this population dependence can be attributed to differential item functioning or to differences in the latent distributions between groups and needs to be accounted for when estimating the reliability of scores for different groups. Here, we introduce group-specific and overall reliability coefficients for sum scores and maximum likelihood ability estimates defined by a multiple group item response theory model. We derive confidence intervals using asymptotic theory and evaluate the empirical properties of estimators and the confidence intervals in a simulation study. The results show that the estimators are largely unbiased and that the confidence intervals are accurate with moderately large sample sizes. We exemplify the approach with the Montreal Cognitive Assessment (MoCA) in two groups defined by education level and give recommendations for applied work.

View all works in NVA

• Braeken, Johan; Lehre, Anne- Catherine W G & Marcq, Kseniia (2025). The risks and perils of cherry picking results from international large-scale educational assessments: The case of the «Nordic» fractions. Full text in Research Archive

View all works in NVA