─ My research focuses on international large-scale assessments (ILSA) in education, which compare student performance across countries worldwide. These studies are not only an important and valuable source of data for understanding educational systems worldwide, but they are also a considerable financial investment for the countries that participate.
Kseniia explains that despite the scale and expense of these assessments, the research that comes out of them tends to focus almost entirely, and of course, fairly, on the students: how they perform, how their backgrounds shape their results, and how their schools and countries compare to others. This focus on the students leaves out an equally important side of the assessment, the items themselves, i.e., the individual tasks or questions students are asked to complete.
The item side of ILSAs
Kseniia`s research shifts the spotlight to these items and asks: What makes some items more or less difficult for students and in which countries that participate in ILSAs? Are there item design features that systematically lead to students doing better on some questions and less so on others? And how do these patterns vary across countries around the globe? By exploring these questions, her research argues that we are not fully capitalizing on the rich data ILSAs generate. By understanding how items function across different contexts, she argues that we can improve future test design, enhance fairness and comparability between countries, and ultimately make better use of the significant investments countries make when they participate in these studies.
A shift from business to academia
Kseniia explains that her academic path is a bit unusual as she spent nearly a decade working in the corporate world after earning her bachelor`s degree.
─ I’ve been part of the Centre for Educational Measurement (CEMO) for over six years now. Before I joined as a Doctoral Research Fellow, I completed my Master’s degree in Measurement, Assessment and Evaluation at CEMO. In fact, I was part of the very first pilot cohort of CEMO’s Master’s programme. Looking back, I can confidently say that CEMO has more or less single-handedly shaped me into the researcher I am today, and I could not be more grateful for that.
She feels that the move from corporate to academia is somehow reverse of what many people do, but she wouldn’t have done it any other way. Corporate experience taught her a lot, she says: resilience, how to handle high-stakes situations, stick to deadlines, manage stress, and get things done efficiently. All of that has turned out to be incredibly valuable in research too. And academia gave her something corporate life never could - a job that’s also her hobby.
The treasure trove of data left unexplored
When Kseniia was first introduced to ILSAs during her time as student at the Master’s programme, two things immediately caught her attention. First she was fascinated by its sheer scale, with so many countries, students, and contexts all being part of the same assessment. She clarifies that it felt like an absolute treasure trove of data, just waiting to be explored from every possible angle. Second, she noticed how most research on ILSAs focused almost exclusively on students, while the items received far less attention.
─ The items, it seemed, were treated like a bit of a black box, assumed to work equally well across all countries, without much interrogation. That struck me as a missed opportunity, especially since these very items are the foundation for the bold claims ILSAs make about educational quality across the globe. In retrospect, I was drawn to this topic because it allowed me to look into the design of ILSAs, with items as proxies, and, at the same time, to think globally about how these items actually work across different cultural and educational contexts. Noticing curious patterns, asking why certain items behave one way in one region but completely differently in another - that’s exactly the kind of puzzle I love working on.
So what are some of the most significant current findings in Kseniia`s research on this?
At the heart of her research, she elaborates, is a desire to better understand how the items in international assessments actually work across different countries, and what this can tell us about the design, fairness, and validity of these assessments. While items are the building blocks of each assessment, in much of the existing ILSA research, they are treated as neutral carriers of information rather than something worth studying in their own right.
─ If I were to sum up the main takeaway from my research, it would be this: items are far from neutral. Their design, everything from their format, to their position within the assessment, to the cognitive demand they entail, all of this can influence how students engage with them. And importantly, these influences don’t play out the same way in every country. An item that works smoothly in one place might behave very differently somewhere else. And, perhaps, the most useful product of my research has been demonstrating that examining items is actually very useful. We should be doing it far more often if we want to truly understand and improve international assessments.
It would not be an understatement to say that ILSAs are influential. And ILSAs don’t just stay within the research community either. Their results regularly make headlines in the form of league tables, comparing, and often pitting, countries against each other. They have been known to shape national policies and even spark major education reforms. But these comparisons are only as valid as the assessments themselves allow. If the items in these assessments function differently across countries, those comparisons become less reliable, and the conclusions we draw rest on shaky ground.
─ That’s where my research comes in. By showing that items are not neutral and that their design matters for how students in different countries engage with them, my work helps pinpoint potential hidden biases within the data. This is critical if we want international comparisons to be fair, meaningful, and trustworthy.
A day in a PhD life
When asked how it is to be a PhD fellow, and how a typical day looks like, she explains that a lot comes down to being able to plan ahead and, more importantly, having the discipline to actually follow through on those plans. For Kseniia it is about finding that balance between carving out time for deep work and being responsive to the many moving parts that come with academic life.
─ The beauty, and sometimes the curse, of academia is that no two days are quite the same. Some days feel incredibly productive - you sit down, make real progress on a working paper, and it feels like you’re moving mountains. Other days are all about putting out fires: answering emails, solving unexpected problems, or jumping between meetings. But to be fair, that’s true of many professions.
Kseniia`s perception of a good researcher requires a blend of curiosity, persistence, and a healthy dose of critical thinking. Curiosity drives one to ask questions and explore unexpected angles, she explains. Persistence helps one keep going when the analysis doesn’t work, the paper gets rejected, or the statistical models refuse to converge, a situation, she laughs, she knows all too well. And critical thinking keeps one honest, making sure you question your own assumptions and stay open to being wrong. On top of that, good researchers need to be organized, she says, as there’s always more data, more deadlines and more revisions, but also adaptable, because research rarely goes exactly as planned. Lastly, she adds, a sense of humor helps as it is easier to handle “Reviewer 2” comments if you can occasionally laugh about them.
Kseniia`s swap from a corporate to academic path has made her realize that she really enjoys the world of academia, and she would like to continue on that path. As she says:
─ There are still countless puzzles left to solve, and I’d love to keep working on them.
Drawing motivation from everyday worklife
As much of Kseniia`s academic journey has unfolded at CEMO, she explains that her colleagues at the centre have inspired her a lot and that she is constantly amazed by the variety of research projects happening under one roof.
─ One moment you’re in a seminar about machine learning applications in education, the next you’re deep in a conversation about complex statistical modeling, and before you know it, someone’s explaining a theoretical framework with concepts you didn’t even know existed, let alone how to spell them.
It’s this combination of intellectual range and the sheer determination of CEMO staff to push their work forward that makes it such an inspiring place to be, Kseniia says. She ends by describing the research world she has been a part of for the last six years like this:
─ There’s always someone exploring a question from a perspective you never considered, and that kind of environment is both humbling and incredibly motivating.
Master Kseniia Marcq will defend the thesis "Item in Focus: A Large-Scale Examination of Cross-Country Differences in the PISA 2018 Through an Item-Centric Perspective" for the degree of PhD on 21st March. For more information, please check the event page.