Johan Braeken

Braeken, J., & Paap, M. C. S. (2020). Making Fixed-Precision Between-Item Multidimensional Computerized Adaptive Tests Even Shorter by Reducing the Asymmetry Between Selection and Stopping Rules. Applied Psychological Measurement, 44, 531–547.

Braeken, J. & Van Assen, M.A.L.M. (2017). An empirical Kaiser criterion. Psychological Methods, 22, 450-466.

Braeken, J., Kuppens, P., De Boeck, P., & Tuerlinckx, F. (2013). Contextualized personality questionnaires: A case for copulas in structural equation models for categorical data. Multivariate Behavioral Research, 48, 845-870.

Braeken, J. (2011). A boundary mixture approach to violations of conditional independence. Psychometrika, 76, 57-76.

Braeken, J., & Tuerlinckx, F. (2009). A mixed model framework for teratology studies. Biostatistics, 10, 744-755.

Braeken, J., & Tuerlinckx, F. (2009). Investigating latent constructs with item response models: A Matlab IRTm Toolbox. Behavior Research Methods, 41, 1127-1137.

Braeken, J., Tuerlinckx, F., & De Boeck, P. (2007). Copulas for residual dependencies. Psychometrika, 72, 393-411.

• Brinchmann, Ellen Irén & Braeken, Johan (2026). Rock-Paper-Screen: Fifth-graders' comprehension of simple and enhanced digital text. Learning and Individual Differences. ISSN 1041-6080. 128. doi: 10.1016/j.lindif.2026.102896. Full text in Research Archive Show summary

  In this experiment, we compared 338 fifth-grade students' comprehension of text presented in three formats: paper, a simple linear digital format, and an enhanced digital format with an embedded dictionary. We also collected data on the students' prior reading comprehension skills, word reading efficiency, and daily digital device use, which allowed us to investigate whether medium effects varied depending on individual differences in reader characteristics. The results revealed no differences in students' text comprehension across the three reading formats. Additionally, there was no indication that performance across reading formats varied depending on students' prior reading skills or digital experience. Although we found no evidence that the embedded dictionary had an impact on students' text comprehension, the process data suggested that students used the dictionary strategically by clicking on words that were central to understanding the ideas presented in the text.

• Braeken, Johan & Laar, Saskia Van (2026). Assessing the Measurement Equivalence of a set of Items: Item-specific Diagnostics on an Interpretable Metric. Practical Assessment, Research, and Evaluation (PARE). 31(1). doi: 10.7275/PARE.3048. Full text in Research Archive
• Braeken, Johan; Lehre, Anne- Catherine W G & Marcq, Kseniia (2025). Pearls and perils when interpreting item-specific results from international large-scale educational assessments: The case of the ‘‘Nordic’’ fractions. Studies in Educational Evaluation. ISSN 0191-491X. 88. doi: 10.1016/j.stueduc.2025.101552. Full text in Research Archive Show summary

  International large-scale assessments (ILSA) in education generate a lot of interest and debate among researchers and stakeholders alike. In this study, we take a deep-dive into a provocative claim that ‘‘Nordic students cannot do fractions’’ based on an item-specific result in the Trends in International Mathematics and Science Study (TIMSS) 2011. Using item response data from five cycles of the TIMSS mathematics achievement test, spanning from 2003 to 2019, across 12 countries representing diverse geographical regions (Africa, Europe, East Asia, and North America) and proficiency levels (from lower-performers to higher-performers), including about 54000 students per cycle and a total of about 800 mathematics items, we investigated the empirical basis and generalizability of the stated claim using an explanatory item response model (IRT) approach. Our study underscores the dangers of miscommunication, oversimplification, and overgeneralization of ILSA results, yet also highlights the hidden potential of deeper item-level analyses for educational researchers and stakeholders to generate critical inquiries into educational practice and our understanding of the subject domain.

• Chen, Jianan; Brinchmann, Ellen Irén & Braeken, Johan (2025). Setting Discontinue Rules for Progressive Tests: A Practical and Transparent Toolkit. Assessment (ASM). ISSN 1073-1911. doi: 10.1177/10731911251385776. Full text in Research Archive Show summary

  Progressive tests are a popular tailored test format where items are administered in increasing order of difficulty level and discontinued according to a rule system that should counter excessive response burden for test participants and guarantee efficient use of resources for test administrators. To facilitate evidence-based decision-making for setting appropriate discontinue rules, we propose a transparent approach that charts the impact of varied alternative discontinue rules on accuracy and efficiency. These A–E charts are based on retroactively applying discontinue rules to normative item response data. We show that a universal discontinue rule likely does not exist and that the optimal rule varies as a function of the desired efficiency-accuracy trade-off suitable for the intended test use and target population. The proposed approach provides a pragmatic solution for practitioners, researchers, test developers, and test publishers to rethink the existing discontinue rules, systematically evaluate the alternatives, and set appropriate rules.

• Braeken, Johan & Laar, Saskia Van (2025). Reducing Calibration Bias for Person Fit Assessment by Mixture Model Expansion. Educational and Psychological Measurement. ISSN 0013-1644. doi: 10.1177/00131644251364252. Full text in Research Archive Show summary

  Measurement appropriateness concerns the question of whether the test or survey scale under consideration can provide a valid measure for a specific individual. An aberrant item response pattern would provide internal counterevidence against using the test/scale for this person, whereas a more typical item response pattern would imply a fit of the measure to the person. Traditional approaches, including the popular Lz person fit statistic, are hampered by their two-stage estimation procedure and the fact that the fit for the person is determined based on the model calibrated on data that include the misfitting persons. This calibration bias creates suboptimal conditions for person fit assessment. Solutions have been sought through the derivation of approximating bias-correction formulas and/or iterative purification procedures. Yet, here we discuss an alternative one-stage solution that involves calibrating a model expansion of the measurement model that includes a mixture component for target aberrant response patterns. A simulation study evaluates the approach under the most unfavorable and least-studied conditions for person fit indices, short polytomous survey scales, similar to those found in large-scale educational assessments such as the Program for International Student Assessment or Trends in Mathematics and Science Study.

• Hummelen, Benjamin; Langjord, Tuva; Paap, Muirne C. S.; Folmo, Espen Jan; Pedersen, Geir & Braeken, Johan (2025). Differential Item Functioning for Gender and Age of the DSM-IV Borderline Personality Disorder Criteria in a Large Clinical Sample. Journal of Psychopathology and Behavioral Assessment. ISSN 0882-2689. 47(1). doi: 10.1007/s10862-024-10183-8. Full text in Research Archive
• Marcq, Kseniia & Braeken, Johan (2025). From framework to functionality: A cross-country analysis of PISA 2018 reading assessment framework’s item features as determinants of item difficulty. Large-scale Assessments in Education. doi: 10.1186/s40536-025-00261-y. Full text in Research Archive Show summary

  Background Theoretical frameworks excel in conceptualising reading literacy, yet their value hinges on their applicability for real-world purposes, such as assessment. By combining diverse theoretical frameworks, the Programme for International Student Assessment (PISA) 2018 designed an assessment framework for assessing the reading literacy of over 600,000 students globally. The PISA 2018 framework was operationalised into assessment items through construct-relevant item features aimed at determining item difficulty by mirroring the expectations set by the assessment framework. Methods Using cross-classified mixed effects models, we model the PISA 2018 reading item difficulty as a function of the PISA-defined construct-relevant and -irrelevant item features to gauge the extent to which item features determined item difficulty on average and across 71 countries. Results All five, delineated by the PISA framework, construct-relevant item features combined only reduce the variance in item difficulty by an average of 20%, with ‘Text Type’ being the only promising predictor. However, the construct-irrelevant ‘Item Format’ alone has half the predictive value of all construct-relevant features combined (10%), implying that external factors may unduly influence student outcomes. Conclusions These findings suggest that the functionality of item features within the framework leaves much to be desired, pointing to considerable shortcomings that necessitate a re-evaluation of the framework and its more effective implementation. We discuss implications for future PISA design and other stakeholders and call for bridging the gap between the PISA reading assessment framework and its practical application to ensure its validity, effectiveness, and credibility among educational stakeholders worldwide.

• Steinmann, Isa; Chen, Jianan & Braeken, Johan (2024). Who Responds Inconsistently to Mixed-Worded Scales? Differences by Achievement, Age Group, and Gender. Assessment in Education: Principles, Policy & Practice. ISSN 0969-594X. doi: 10.1080/0969594X.2024.2318554. Full text in Research Archive Show summary

  We investigated two research questions: which students are more likely to respond inconsistently to mixed-worded questionnaire scales, and which country samples have larger shares of inconsistent respondents? We defined an inconsistent response as strongly agree- ing or disagreeing with both positively and negatively worded items of the same scale. Since we assumed that inconsistent responding occurs due to a lack of carefulness, reading, or cognitive skills, we expected to find that inconsistent responding was associated with lower achievement, younger age, being a nonnative speaker, and being a boy. We used data from all 38 countries that participated in the fourth- and eighth-grade assessments of TIMSS (Trends in International Mathematics and Science Study) 2019. Using the mean absolute difference method, we identified shares of 1‒21% inconsistent respondents across samples. The results generally sup- ported our hypotheses, especially the hypothesis that inconsistent responding is more common among students and countries with lower mathematics achievement levels.

• Chen, Jianan; Steinmann, Isa & Braeken, Johan (2024). Competing explanations for inconsistent responding to a mixed-worded self-esteem scale: Cognitive abilities or personality? Personality and Individual Differences. ISSN 0191-8869. 222, p. 1–6. doi: 10.1016/j.paid.2024.112573. Full text in Research Archive Show summary

  In survey scale design, a mixed-worded format intends to ensure attentiveness as respondents need to take into account the wording direction when answering an item. However, some respondents tend to deliver inconsistent responses (i.e., agreeing or disagreeing with both positively and negatively-worded items), posing a validity concern. Two potential directions driving inconsistent responding have each individually been put forward: lack of cognitive abilities to effectively deal with the mixed-wording and sheer carelessness while responding. Using a factor mixture approach, we investigated inconsistent responding on Rosenberg’s self-esteem scale as a function of both cognitive ability and personality. Among n = 4938 Grade 5 students from the German National Educational Panel Study (NEPS), 11 % were classified as inconsistent respondents and class memberships were further related to four cognitive abilities (cognitive reasoning, cognitive speed, reading comprehension, reading speed) and five personality traits (conscientiousness, neuroticism, extraversion, agreeableness, openness). Model comparison results indicated that both ability and personality predictors matter with a more prominent role for ability (especially reading comprehension). We discuss the implications of these findings for mixed-wording’s suitability for scale construction and different populations and how researchers can deal with suspected inconsistent respondents in their survey data.

• Koopman, Letty & Braeken, Johan (2024). Investigating the Ordering Structure of Clustered Items Using Nonparametric Item Response Theory. Educational and Psychological Measurement. ISSN 0013-1644. doi: 10.1177/00131644241274122. Full text in Research Archive Show summary

  Educational and psychological tests with an ordered item structure enable efficient test administration procedures and allow for intuitive score interpretation and monitoring. The effectiveness of the measurement instrument relies to a large extent on the validated strength of its ordering structure. We define three increasingly strict types of ordering for the ordering structure of a measurement instrument with clustered items: a weak and a strong invariant cluster ordering and a clustered invariant item ordering. Following a nonparametric item response theory (IRT) approach, we proposed a procedure to evaluate the ordering structure of a clustered item set along this three-fold continuum of order invariance. The basis of the procedure is (a) the local assessment of pairwise conditional expectations at both cluster and item level and (b) the global assessment of the number of Guttman errors through new generalizations of the H-coefficient for this item-cluster context. The procedure, readily implemented in R, is illustrated and applied to an empirical example. Suggestions for test practice, further methodological developments, and future research are discussed.

• Laar, Saskia Van; Chen, Jianan & Braeken, Johan (2024). How Randomly are Students Random Responding to Your Questionnaire? Within-Person Variability in Random Responding Across Scales in the TIMSS 2015 Eighth-Grade Student Questionnaire. Measurement: Interdisciplinary Research and Perspectives. ISSN 1536-6367. 22(1), p. 90–107. doi: 10.1080/15366367.2023.2203972. Full text in Research Archive Show summary

  Questionnaires in educational research assessing students’ attitudes and beliefs are low-stakes for the students. As a consequence, students might not always consistently respond to a questionnaire scale but instead provide more random response patterns with no clear link to items’ contents. We study inter-individual differences in students’ intra-individual random responding profile across 19 questionnaire scales in the TIMSS 2015 eighth-grade student questionnaire in seven countries. A mixture IRT approach was used to assess students’ random responder status on a questionnaire scale. A follow-up latent class analysis across the questionnaire revealed four random responding profiles that generalized across countries: A majority of consistent nonrandom responders, intermittent moderate random responders, frequent random responders, and students that were exclusively triggered to respond randomly on the confidence scales in the questionnaire. We discuss the implications of our findings in light of general data-quality concerns and the potential ineffectiveness of early-warning monitoring systems in computer-based surveys.

• Marcq, Kseniia; Donayre, Erika Jassuly Chalen & Braeken, Johan (2024). The role of item format in the PISA 2018 mathematics literacy assessment: A cross-country study. Studies in Educational Evaluation. ISSN 0191-491X. 83, p. 1–14. doi: 10.1016/j.stueduc.2024.101401. Full text in Research Archive Show summary

  When construct-irrelevant sources affect item difficulty, validity of the assessment is compromised. Using responses of 260000 students from 71 countries to the Programme for International Student Assessment (PISA) 2018 mathematics assessment and cross-classified mixed effects models, we examined three validity concerns associated with the construct-irrelevant factor, item format: whether the format influenced item difficulty, whether item format’s impact on difficulty varied across countries, undermining PISA’s foundational goal of meaningful country comparisons, and whether item format effects differed between genders, affecting assessment fairness. Item format contributed to a substantial average of 12 % of variance in item difficulties. The effect of item format was non-uniform across countries, with 30 % of the variance in item difficulties being due to format in lower-performing countries, and 10 % in higher-performing countries, challenging the comparability of educational outcomes. The impact of gender on item format differences was minor. Implications for secondary research and assessment design are discussed.

• Tesema, Melaku Tesfa; Blömeke, Sigrid & Braeken, Johan (2024). Mapping student enrolment and admission eligibility for higher education in Ethiopia: affirmative action as equity instrument? Frontiers in Education. 9. doi: 10.3389/feduc.2024.1474811. Full text in Research Archive Show summary

  Introduction: UNICEF’s Education For All steers for equal opportunities for all students and diversity in the student population. In Ethiopia, affirmative action is the policy tool addressing equity in enrolment and admission to higher education, targeting underrepresented groups such as females and students from emerging regions. Methods: In a cross-sectional study across two-levels of education, regional differences in enrolment and admission eligibility were explored for two census student cohorts at high school (n = 858,557 and n = 785,053) and at the university preparatory program (n = 199,899 and n = 211,706). These population-level data stem from governmental registers from the mandatory national exams. Enrolment and admission eligibility statistics were computed with special attention to gender imparity. The impact of affirmative action was studied by comparison to the corresponding projected statistics computed as if affirmative action did not apply. Results: In most regions, clearly more male than female students were enrolled across the different educational levels. Affirmative action benefitted a significant percentage of female and emerging-region students. The gender parity in eligibility rates for admission to preparatory program for most regions was restored, but not for admission to university. Discussion: Being implemented as policy directive for nearly three decades, affirmative action at most maintained gender imparity levels in enrolment inherited from earlier educational stages but failed in redressing inequity across gender and regions. Policy implications of the results and recommendations are discussed in light of regional differences and the currently implemented type of affirmative action.

• Marcq, Kseniia & Braeken, Johan (2024). Where nonresponse is at its loudest: Cross-country and individual differences in item nonresponse across the PISA 2018 student questionnaire. International Journal of Testing. ISSN 1530-5058. p. 1–28. doi: 10.1080/15305058.2024.2377550. Full text in Research Archive Show summary

  The Programme for International Student Assessment (PISA) student questionnaire, despite being designed for low cognitive demand, may induce test burden due to its 306-item length, resulting in increased item nonresponse toward the questionnaire’s end. Using the PISA 2018 response data from 80 countries and a cross-classified mixed effects model, this study explored the average, cross-country, and individual nonresponse trends throughout the questionnaire. The average nonresponse probability increased from 2% at the questionnaire’s start to 11% at its end. East Asian countries displayed the flattest nonresponse trends, while South American countries showed the steepest. Individual differences were also prominent: students with a lower initial nonresponse propensity experienced a steeper increase in their nonresponse propensity as they progressed through the questionnaire, whereas those with a higher initial propensity remained at a relatively constant level. Our findings highlight the importance of the questionnaire design, the context in which it is administered, and its interaction with individual and cultural factors in shaping student response behavior. We discuss the implications for questionnaire design and researchers using PISA student questionnaire data for secondary analysis.

• Chen, Jianan; Laar, Saskia van & Braeken, Johan (2023). Who are those random responders on your survey? The case of the TIMSS 2015 student questionnaire. Large-scale Assessments in Education. 11(1). doi: 10.1186/s40536-023-00184-6. Full text in Research Archive Show summary

  A general validity and survey quality concern with student questionnaires under low-stakes assessment conditions is that some responders will not genuinely engage with the questionnaire, often with more random response patterns as a result. Using a mixture IRT approach and a meta-analytic lens across 22 educational systems participating in TIMSS 2015, we investigated whether the prevalence of random responders on six scales regarding students' engagement and attitudes toward mathematics and sciences was a function of grade, gender, socio-economic status, spoken language at home, or migration background. Among these common policy-relevant covariates in educational research, we found support for small group differences in prevalence of random responders (OR ≥ 1.22)(average prevalence of 7%). In general, being a student in grade 8 (vs. grade 4), being male, reporting to have fewer books, or speaking a language different from the test language at home were all risk factors characterizing random responders. The expected generalization and implications of these findings are discussed based on the observed heterogeneity across educational systems and consistency across questionnaire scales.

• Laar, Saskia Van & Braeken, Johan (2023). Prevalence of random responders as a function of scale position and questionnaire length in the TIMSS 2015 eighth-grade student questionnaire. International Journal of Testing. ISSN 1530-5058. doi: 10.1080/15305058.2023.2263206. Full text in Research Archive Show summary

  This study examined the impact of two questionnaire characteristics, scale position and questionnaire length, on the prevalence of random responders in the TIMSS 2015 eighth-grade student questionnaire. While there was no support for an absolute effect of questionnaire length, we did find a positive effect for scale position, with an increase of 5% in random responding over the course of the questionnaire (in both the shorter and the longer version). However, scale character turned out to be an unexpected but more important determinant. Scales about students’ confidence in mathematics or science showed an increase of 9% in random responding, which is double the impact of scale position. Potential mechanisms underlying the confidence case and general implications of the results for questionnaire design are discussed.

• Marcq, Kseniia & Braeken, Johan (2023). Gender differences in item nonresponse in the PISA 2018 student questionnaire. Educational Assessment, Evaluation and Accountability. ISSN 1874-8597. 36(3), p. 301–336. doi: 10.1007/s11092-023-09412-7. Full text in Research Archive Show summary

  Abstract Gender differences in item nonresponse are well-documented in high-stakes achievement tests, where female students are shown to omit more items than male students. These gender differences in item nonresponse are often linked to differential risk-taking strategies, with females being risk-averse and unwilling to guess on an item, even if it could gain them credits. In low-stakes settings, similar trends should not apply, as the students carry no consequence for their performance. Instead, test-taking motivation is argued to be the pivoting factor, with female students seen as more motivated and omitting fewer items than male students. In contrast to the high- and low-stakes achievement tests, less is known about gender differences in item nonresponse in student background questionnaires. Using cross-classified mixed effects models, we examined gender differences in item nonresponse on the Programme for International Student Assessment (PISA) 2018 student questionnaire across 80 countries and 71 scales. On average, the odds of male students omitting a questionnaire item were double the odds of female students, consistent with the expected trend in the low-stakes setting. However, we show that gender differences in item nonresponse are not merely a function of the stakes involved for individual students but a more complex phenomenon that is context-dependent and not necessarily stable across countries, scales’ formats, and contents. We argue that examining differences in item nonresponse patterns could serve as a source of additional information about the students’ test-taking behaviour and the quality of the questionnaire.

• Frans, Niek; Braeken, Johan; Veldkamp, Bernard P. & Paap, Muirne C. S. (2022). Empirical Priors in Polytomous Computerized Adaptive Tests: Risks and Rewards in Clinical Settings. Applied Psychological Measurement. ISSN 0146-6216. 47(1), p. 48–63. doi: 10.1177/01466216221124091. Full text in Research Archive Show summary

  The use of empirical prior information about participants has been shown to substantially improve the efficiency of computerized adaptive tests (CATs) in educational settings. However, it is unclear how these results translate to clinical settings, where small item banks with highly informative polytomous items often lead to very short CATs. We explored the risks and rewards of using prior information in CAT in two simulation studies, rooted in applied clinical examples. In the first simulation, prior precision and bias in the prior location were manipulated independently. Our results show that a precise personalized prior can meaningfully increase CAT efficiency. However, this reward comes with the potential risk of overconfidence in wrong empirical information (i.e., using a precise severely biased prior), which can lead to unnecessarily long tests, or severely biased estimates. The latter risk can be mitigated by setting a minimum number of items that are to be administered during the CAT, or by setting a less precise prior; be it at the expense of canceling out any efficiency gains. The second simulation, with more realistic bias and precision combinations in the empirical prior, places the prevalence of the potential risks in context. With similar estimation bias, an empirical prior reduced CAT test length, compared to a standard normal prior, in 68% of cases, by a median of 20%; while test length increased in only 3% of cases. The use of prior information in CAT seems to be a feasible and simple method to reduce test burden for patients and clinical practitioners alike.

• Busch, Tobias; Brinchmann, Ellen Irén; Braeken, Johan & Wie, Ona Bø (2022). Receptive Vocabulary of Children With Bilateral Cochlear Implants From 3 to 16 Years of Age. Ear and Hearing. ISSN 0196-0202. 43(6), p. 1866–1880. doi: 10.1097/AUD.0000000000001220. Full text in Research Archive Show summary

  Objectives: The vocabulary of children with cochlear implants is often smaller than that of their peers with typical hearing, but there is uncertainty regarding the extent of the differences and potential risks and protective factors. Some studies indicate that their receptive vocabulary develops well at first, but that they fail to keep up with their typical hearing peers, causing many CI users to enter school with a receptive vocabulary that is not age-appropriate. To better understand the receptive vocabulary abilities of children with cochlear implants this study explored age-related differences to matched children with typical hearing and associations between vocabulary skills and child-level characteristics. Design: A retrospective cross-sectional study with matched controls was conducted at the Norwegian national cochlear implant center at Oslo University Hospital. Eighty-eight children (mean age 8.7 years; range 3.2 to 15.9; 43 girls, 45 boys) who had received bilateral cochlear implants before 3 years of age were compared with two groups of children with typical hearing. One group was matched for maternal education, sex, and chronological age, the other group was matched for maternal education, sex, and hearing age. Receptive vocabulary performance was measured with the British Picture Vocabulary Scale. Results: Cochlear implant users’ receptive vocabulary was poorer than that of age-matched children with typical hearing (M = 84.6 standard points, SD = 21.1; children with typical hearing: M = 102.1 standard points, SD = 15.8; mean difference −17.5 standard points, 95% CI [−23.0 to −12.0], p < 0.001; Hedges’s g = −0.94, 95% CI [−1.24 to −0.62]), and children with cochlear implants were significantly more likely to perform below the normative range (risk ratio = 2.2, 95% CI [1.42 to 3.83]). However, there was a significant nonlinear U-shaped effect of age on the scores of cochlear implant users, with the difference to the matched typical hearing children being largest (23.9 standard points, on average) around 8.7 years of age and smaller toward the beginning and end of the age range. There was no significant difference compared with children with typical hearing when differences in auditory experience were accounted for. Variability was not significantly different between the groups. Further analysis with a random forest revealed that, in addition to chronological age and hearing age, simultaneous versus sequential implantation, communication mode at school, and social integration were predictors of cochlear implant users’ receptive vocabulary. Conclusions: On average, the receptive vocabulary of children with cochlear implants was smaller than that of their typical hearing peers. The magnitude of the difference was changing with age and was the largest for children in early primary school. The nonlinear effect of age might explain some of the ambiguity in previous research findings and could indicate that better intervention is required around school entry. The results emphasize that continuous monitoring and support are crucial to avoid far-reaching negative effects on the children’s development and well-being.

• Heltne, Aleksander; Braeken, Johan; Hummelen, Benjamin; Selvik, Sara Germans; Christensen, Tore Buer & Paap, Muirne C. S. (2022). Do Flexible Administration Procedures Promote Individualized Clinical Assessments? An Explorative Analysis of How Clinicians Utilize the Funnel Structure of the SCID-5-AMPD Module I: LPFS. Journal of Personality Assessment. ISSN 0022-3891. doi: 10.1080/00223891.2022.2152344. Full text in Research Archive Show summary

  The current study examined clinicians’ utilization of the SCID-5-AMPD-I funnel structure. Across 237 interviews, conducted as part of the NorAMP study, we found that clinicians administered on average 2-3 adjacent levels under each subdomain, effectively administering only about 50% of available items. Comparing administration patterns of interviews, no two interviews contained the exact same set of administered items. On average, when comparing individual interviews, only about half of the administered items in each interview were administered in both interviews. Cross-classified mixed effects models were estimated to examine the factors affecting item administration. Results indicated that the interplay between patient preliminary scores and item level had a substantial impact on item administration, suggesting clinicians tend to administer items corresponding to expected patient severity. Overall, our findings suggest clinicians utilize the SCID-5-AMPD-I funnel structure to conduct efficient and individually tailored assessments informed by relevant patient characteristics. Adopting similar non-fixed administration procedures for other interviews could potentially provide similar benefits compared to traditional fixed-form administration procedures. The current study can serve as a template for verifying and evaluating future adoptions of non-fixed administration procedures in other interviews.

• Laar, Saskia Van & Braeken, Johan (2022). Caught off Base: A Note on the Interpretation of Incremental Fit Indices. Structural Equation Modeling. ISSN 1070-5511. 29(6), p. 935–943. doi: 10.1080/10705511.2022.2050730. Full text in Research Archive Show summary

  This note serves as a reminder that incremental fit indices are a form of standardized effect sizes and hence, all reservations with respect to interpretations of standardized effect sizes also transfer to their interpretation. Such a realization has major implications for the interpretation and use of incremental fit indices, for the theoretical (im)possibility of default universal rules of thumb in their application, and for simulation studies mapping incremental fit indices as if their value is comparable in an absolute sense across any and all conditions. A small but illustrative working example centered around the alleged impact of model type will drive these points home.

• Laar, Saskia Van & Braeken, Johan (2022). Random Responders in the TIMSS 2015 Student Questionnaire: A Threat to Validity? Journal of Educational Measurement. ISSN 0022-0655. p. 1–32. doi: 10.1111/jedm.12317. Full text in Research Archive Show summary

  The low-stakes character of international large-scale educational assessments implies that a participating student might at times provide unrelated answers as if s/he was not even reading the items and choosing a response option randomly throughout. Depending on the severity of this invalid response behavior, interpretations of the assessment results are at risk of being invalidated. Not much is known about the prevalence nor impact of such random responders in the context of international large-scale educational assessments. Following a mixture item response theory (IRT) approach, an initial investigation of both issues is conducted for the Confidence in and Value of Mathematics/Science (VoM/VoS) scales in the Trends in International Mathematics and Science Study (TIMSS) 2015 student questionnaire. We end with a call to facilitate further mapping of invalid response behavior in this context by the inclusion of instructed response items and survey completion speed indicators in the assessments and a habit of sensitivity checks in all secondary data studies.

• Marcq, Kseniia & Braeken, Johan (2022). The blind side: Exploring item variance in PISA 2018 cognitive domains. Assessment in Education: Principles, Policy & Practice. ISSN 0969-594X. 29(3), p. 332–360. doi: 10.1080/0969594X.2022.2097199. Full text in Research Archive Show summary

  Communication of International Large-Scale Assessment (ILSA) results is dominated by reporting average country achievement scores that conceal individual differences between pupils, schools, and items. Educational research primarily focuses on examining differences between pupils and schools, while differences between items are overlooked. Using a variance components model on the Programme for International Student Assessment (PISA) 2018 cognitive domains of reading, mathematics, and science literacy, we estimated how much of the response variation can be attributed to differences between pupils, schools, and items. The results show that uniformly across domains and countries, it mattered more for the correctness of an item response which items were responded to by a pupil (27–35%) than which pupil responded to these items (10–12%) or which school the pupil attended (5–7%). Given the findings, we argue that differences between items in ILSAs constitute a source of substantial untapped potential for secondary research.

• Steinmann, Isa; Strietholt, Rolf & Braeken, Johan (2021). A Constrained Factor Mixture Analysis Model for Consistent and Inconsistent Respondents to Mixed-Worded Scales. Psychological methods. ISSN 1082-989X. doi: 10.1037/met0000392. Full text in Research Archive Show summary

  Mixed-worded scales require a more careful reading and answering process than scales with only one type of wording. The present study is about the unintended consequences of using such scales, because typically, not all respondents answer positively and negatively worded items consistently. This population heterogeneity—meaning that there are distinct groups of consistently and inconsistently answering respondents—may arguably underlie wording-related effects in mixed-worded scales. We formulated a constrained factor mixture analysis model that operationalized these two assumed classes of respondents. We investigated five data sets that contained four mixed-worded attitude scales, large-scale samples from three countries (Germany, Australia, and the U.S.), and two age groups (children and adolescents). The constrained factor mixture analysis model showed estimated parameter patterns in line with theoretical expectations and consistently outperformed its more traditional competitor, confirmatory factor analysis with one global and one orthogonal method factor across all used data sets. We found proportions of between 7% and 20% of respondents belonging to the inconsistent classes. To further substantiate and validate the interpretation of the proposed model, we related class membership to theoretically relevant respondent characteristics such as reading achievement, cognitive skills, or conscientiousness. Further, we undertook an initial exploration of the overlap in inconsistent respondents’ class membership across time and across scales within a survey. The article discusses implications for future research as well as for the use of mixed-worded scales.

• Karlsen, Jannicke; Hjetland, Hanne Næss; Hagtvet, Bente Eriksen; Braeken, Johan & Melby-Lervåg, Monica (2021). The concurrent and longitudinal relationship between narrative skills and other language skills in children. First language. ISSN 0142-7237. doi: 10.1177/0142723721995688. Full text in Research Archive Show summary

  We examined the concurrent relationship between narrative skills (the Renfrew Bus Story Test) and core language measures (vocabulary, grammar and verbal memory) at age 4 and the longitudinal relationship between core language and listening comprehension skills at age 7 in a sample of 215 children using latent variables and structural equation modelling. Our main purpose was to investigate to what extent narrative retell constitutes a unique influence on later language and listening comprehension skills. The results support a two-factor model of narrative retelling and core language representing different but related constructs at age 4. Narrative retell explained unique variance in later language skills but did not explain additional variance beyond the 58% explained by the age 4 language construct. Similarly, narrative retell predicted unique variance in later listening comprehension, but not beyond what was explained by core language skills at age 4. The strength of the relationship between narrative retelling at age 4 and the age 7 measures was not related to the level of narrative skills. The results indicate that age 4 traditional core language measures capture more of the skills that are important for later language and listening comprehension than narrative skills at the same age.

• Kalandadze, Tamara; Braeken, Johan; Brynskov, Cecilia & Næss, Kari-Anne Bottegaard (2021). Metaphor Comprehension in Individuals with Autism Spectrum Disorder: Core Language Skills Matter. Journal of Autism and Developmental Disorders. ISSN 0162-3257. 52, p. 1–11. doi: 10.1007/s10803-021-04922-z. Full text in Research Archive Show summary

  Poor metaphor comprehension was considered a hallmark of autism spectrum disorder (ASD), but recent research has questioned the extent and the sources of these difculties. In this cross-sectional study, we compared metaphor comprehension in individuals with ASD (N=29) and individuals with typical development (TD; N=31), and investigated the relationship between core language and metaphor comprehension. Individuals with ASD showed more difculty but also a more variable performance in both metaphor and literal items of the task used than individuals with TD did. This indicates that core language ability accounts for metaphor comprehension and should be considered in future research and interventions aiming to improve metaphor comprehension in individuals with ASD.

• Brosseau, Danielle; Braeken, Johan; Carmack, Cindy; Rosberger, Zeev & Körner, Annett (2021). We think we can: development of the Dyadic Efficacy Scale for Cancer. Journal of Psychosocial Oncology Research and Practice. 3(4). doi: 10.1097/OR9.0000000000000066. Full text in Research Archive Show summary

  Background: Measurement advances are needed to enable the study of dyadic-level processes impacting couples coping with cancer. This study sought to develop and empirically examine a Dyadic Efficacy Scale for Cancer (DESC). Cancer-related dyadic efficacy is an individual's confidence to work together with a partner to cope with cancer and its treatment. Methods: The DESC was developed using an exploratory sequential mixed methods design. This paper outlines the psychometric evaluation phase. Individuals with cancer (N = 261) and their partners (N = 217) completed 50 items. Item-level analyses reduced this set to 26 items. Using the dyad as the unit of analysis, confirmatory factor analysis with mirrored patient and partner bifactor structure tested for the presence of a general factor and 3 secondary factors, that is, illness intrusions, patient affect, partner affect. Results: Goodness-of-fit indices supported the identified model, χ2(1170) = 2090, P < .001; RMSEA = .05, P = .14, 90% CI .05–.06; SRMR = .05; CFI = .90. Multidimensionality differed for patients and partners. A general dyadic efficacy factor and secondary factors for managing affect were present for both dyad members, whereas the secondary factor of managing illness intrusions was confirmed for patients only. The model explained 72% and 64% of the variance in patients’ and partners’ dyadic efficacy. Evidence of convergent validity was presented. Conclusions: This study is the first to provide a tool to assess dyadic efficacy among couples coping with cancer. The assessment of cancer-related dyadic efficacy enables new discoveries into couples’ adjustment to cancer.

• Laar, Saskia Van & Braeken, Johan (2021). Understanding the Comparative Fit Index: It's all about the base! Practical Assessment, Research, and Evaluation (PARE). 26(26), p. 1–25. doi: 10.7275/23663996. Full text in Research Archive Show summary

  Despite the sensitivity of fit indices to various model and data characteristics in structural equation modeling, these fit indices are used in a rigid binary fashion as a mere rule of thumb threshold value in a search for model adequacy. Here, we address the behavior and interpretation of the popular Comparative Fit Index (CFI) by stressing that its metric for model assessment is the amount of misspecification in a baseline model and by further decomposition into its fundamental components: sample size, number of variables and the degree of multivariate dependence in the data. Simulation results show how these components influence the performance of CFI and its rule of thumb in practice. We discuss the usefulness of additional qualifications when applying the CFI rule of thumb and potential adjustments to its threshold value as a function of data characteristics. In conclusion, we at a minimum recommend a dual reporting strategy to provide the necessary context and base for meaningful interpretation and even more optimal, a move to using CFI as a real incremental fit index intended to evaluate the relative effect size of cumulative theoretically motivated model restrictions in terms of \% reduction in misspecification as measured by the baseline model.

• Steinmann, Isa; Sanchez, Daniel; Laar, Saskia van & Braeken, Johan (2021). The impact of inconsistent responders to mixed-worded scales on inferences in international large-scale assessments. Assessment in Education: Principles, Policy & Practice. ISSN 0969-594X. doi: 10.1080/0969594X.2021.2005302. Full text in Research Archive Show summary

  Questionnaire scales that are mixed-worded, i.e. include both positively and negatively worded items, often suffer from issues like low reliability and more complex latent structures than intended. Part of the problem might be that some responders fail to respond consistently to the mixed-worded items. We investigated the prevalence and impact of inconsistent responders in 37 primary education systems participating in the joint PIRLS/TIMSS 2011 assessment. Using the mean absolute difference method and three mixed-worded self-concept scales, we identified between 2%‒36% of students as inconsistent responders across education systems. Consistent with expectations, these students showed lower average achievement scores and had a higher risk of being identified as inconsistent on more than one scale. We also found that the inconsistent responders biased the estimated dimensionality and reliability of the scales. The impact on external validity measures was limited and unsystematic. We discuss implications for the use and development of questionnaire scales.

• Braeken, Johan & Paap, Muirne C. S. (2020). Making Fixed-Precision Between-Item Multidimensional Computerized Adaptive Tests Even Shorter by Reducing the Asymmetry Between Selection and Stopping Rules. Applied Psychological Measurement. ISSN 0146-6216. doi: 10.1177/0146621620932666. Full text in Research Archive Show summary

  Fixed-precision between-item multidimensional computerized adaptive tests (MCATs) are becoming increasingly popular. The current generation of item-selection rules used in these types of MCATs typically optimize a single-valued objective criterion for multivariate precision (e.g., Fisher information volume). In contrast, when all dimensions are of interest, the stopping rule is typically defined in terms of a required fixed marginal precision per dimension. This asymmetry between multivariate precision for selection and marginal precision for stopping, which is not present in unidimensional computerized adaptive tests, has received little attention thus far. In this article, we will discuss this selection-stopping asymmetry and its consequences, and introduce and evaluate three alternative item-selection approaches. These alternatives are computationally inexpensive, easy to communicate and implement, and result in effective fixed-marginal-precision MCATs that are shorter in test length than with the current generation of item-selection approaches.

• Hummelen, Benjamin; Braeken, Johan; Christensen, Tore Buer; Nysæter, Tor Erik; Selvik, Sara Germans & Walther, Kristoffer Lygren [Show all 9 contributors for this article] (2020). A psychometric analysis of the structured clinical interview for the DSM-5 alternative model for personality disorders module I (SCID-5-AMPD-I): Level of personality functioning scale. Assessment (ASM). ISSN 1073-1911. 28(5), p. 1320–1333. doi: 10.1177/1073191120967972. Full text in Research Archive Show summary

  The current study aims to examine the psychometric properties of the Structured Clinical Interview for the DSM-5 Alternative Model for Personality Disorders Module I (SCID-5-AMPD-I) assessing the Level of Personality Functioning Scale (LPFS) in a heterogeneous sample of 282 nonpsychotic patients. Latent variable models were used to investigate the dimensionality of the LPFS. The results indicate that the LPFS, as assessed by the SCID-5-AMPD-I, can be considered as a unidimensional construct that can be measured reliably across a wide range of the latent trait. Threshold parameters for the 12 indicators of the LPFS increased gradually over the latent scale, indicating that the five LPFS levels were ordered as predicted by the model. In general, the increase of threshold parameters was relatively small for the shift from Level 2 to Level 3. A better distinction among the different severity levels might be obtained by fine-tuning the interview guidelines or the Level 2 indicators themselves

• Paap, Muirne C.S.; Hummelen, Benjamin; Braeken, Johan; Arnevik, Espen Kristian Ajo; Walderhaug, Espen & Wilberg, Theresa [Show all 9 contributors for this article] (2020). A multi-center psychometric evaluation of the Severity Indices of Personality Problems 118 (SIPP-118): Do we really need all those facets? Quality of Life Research. ISSN 0962-9343. 30(2), p. 567–575. doi: 10.1007/s11136-020-02654-8. Full text in Research Archive Show summary

  Abstract Purpose The Severity Indices of Personality Problems 118 (SIPP-118) is a self-report questionnaire that aims to measure core components of (mal)adaptive personality functioning that can change over time. In this study, we aimed to assess the facet strength of the 16 facets across three large clinical samples. Methods Data from Norwegian and Dutch psychiatric patients were analyzed in this international multi-center study (N 1 = 2814, N 2 = 4751, N 3 = 2217). Bi-factor modeling was used to assess to what degree the SIPP items tap into an overall general factor. The incremental value (distinctiveness) of the facets was studied using proportional reduction in mean squared error (PRMSE) based statistics. Results The estimated model showed adequate fit. The explained common variance (ECV) attributable to the general factor equaled 50% for all three samples. All but two facets (stable self-image and frustration tolerance) showed sufficient levels of distinctiveness. The findings were observed to be comparable across the three samples. Conclusion Our findings showed that the general factor was relatively weak, and the facets had a clear incremental value.

• Paap, Muirne C. S.; Born, Sebastian & Braeken, Johan (2019). Measurement Efficiency for Fixed-Precision Multidimensional Computerized Adaptive Tests: Comparing Health Measurement and Educational Testing Using Example Banks. Applied Psychological Measurement. ISSN 0146-6216. 43(1), p. 68–83. doi: 10.1177/0146621618765719. Full text in Research Archive Show summary

  It is currently not entirely clear to what degree the research on multidimensional computerized adaptive testing (CAT) conducted in the field of educational testing can be generalized to fields such as health assessment, where CAT design factors differ considerably from those typically used in educational testing. In this study, the impact of a number of important design factors on CAT performance is systematically evaluated, using realistic example item banks for two main scenarios: health assessment (polytomous items, small to medium item bank sizes, high discrimination parameters) and educational testing (dichotomous items, large item banks, small- to medium-sized discrimination parameters). Measurement efficiency is evaluated for both between-item multidimensional CATs and separate unidimensional CATs for each latent dimension. In this study, we focus on fixed-precision (variable-length) CATs because it is both feasible and desirable in health settings, but so far most research regarding CAT has focused on fixed-length testing. This study shows that the benefits associated with fixed-precision multidimensional CAT hold under a wide variety of circumstances.

• Tesema, Melaku Tesfa & Braeken, Johan (2018). Regional inequalities and gender differences in academic achievement as a function of educational opportunities: Evidence from Ethiopia. International Journal of Educational Development. ISSN 0738-0593. 60, p. 51–59. doi: 10.1016/j.ijedudev.2017.10.023. Full text in Research Archive Show summary

  This study investigated regional and gender differences in academic achievement in Ethiopia, and examined whether these differences can be explained in terms of unequal educational opportunities (EO). Educational opportunity was operationalized in a broad sense based on a regional differentiation in terms of socio-economic and school environment factors. The study results are based on a multilevel analysis of the 2014 and 2015 national standardized exam for grade 12 students (n = 194503 and n = 205719). Whereas the Central (high EO) regions outperformed the other regions (Cohen’s d = 0.85) as expected, there were some inconsistencies in the comparison between Established (mid EO) regions and Emerging (low EO) regions. Coincidentally, the two Emerging regions that were unexpectedly performing at the level of the Established regions were also the two regions in which there was no evidence for a gender gap in achievement. For other regions, including the Central regions, evidence for a gender gap sometimes as large as the regional gap was identified, with boys having on average higher scores than girls (Cohen’s d = [0.02, 0.92] with an average of 0.50). Plausible explanations and further policy recommendations are discussed.

• Arnesen, Anne; Braeken, Johan; Ogden, Terje & Melby-Lervåg, Monica (2018). Assessing Children’s Social Functioning and Reading Proficiency: A Systematic Review of the Quality of Educational Assessment Instruments Used in Norwegian Elementary Schools. Scandinavian Journal of Educational Research. ISSN 0031-3831. p. 1–26. doi: 10.1080/00313831.2017.1420685. Full text in Research Archive Show summary

  Social functioning and reading proficiency are critical for success in school and society. Therefore, identifying children with such problems is important. This study had 2 parts: first, a random sample of 234 elementary schools was surveyed about which instruments they use to assess reading proficiency and social functioning. Second, a systematic review of the quality of these instruments was conducted using international standards for examining the quality of assessment instruments. The survey showed that schools more often assessed and had more instruments available for reading than for social functioning. The systematic review of the assessment instruments used revealed that the psychometric qualities of many was weak or undocumented, while the dimensions of test material quality were generally good. The findings demonstrate a need for a more thorough examination of the psychometric properties of assessment instruments to be used in school.

• Brinchmann, Ellen Iren; Braeken, Johan & Lyster, Solveig-Alma Halaas (2018). Is there a direct relation between the development of vocabulary and grammar? Developmental Science. ISSN 1363-755X. p. 1–13. doi: 10.1111/desc.12709. Full text in Research Archive Show summary

  Previous studies of individual differences have revealed strong correlations between children's vocabulary and grammatical abilities, and these data have been used to support theoretical accounts positing direct developmental relations between these two areas of language. However, between‐person differences do not necessarily reflect intra‐individual dynamics. Thus, in the present study, we analysed longitudinal data from three annual assessments of vocabulary and grammar in 217 children (Mage = 4 years and 3 months at first assessment) using a modelling strategy with some utility in distinguishing relations at the between‐ and within‐person levels. The results revealed strong correlations between grammar and vocabulary at the between‐person level, but the evidence of direct dependencies between the variables at the within‐person level was rather limited. Specifically, we found a small direct contribution from grammar to vocabulary for children between the ages of 4 and 5, but there was no evidence of any direct contributions from vocabulary to grammar. Further analyses suggested that the home literacy environment may represent a common source of individual differences in children's vocabulary and grammatical skills. In light of these results, we argue that the evidence of direct relations between vocabulary and grammatical development in preschool‐aged children may not be as strong as previously assumed.

• Daus, Stephan & Braeken, Johan (2018). The sensitivity of TIMSS country rankings in science achievement to differences in opportunity to learn at classroom level. Large-scale Assessments in Education. 6(1). doi: 10.1186/s40536-018-0054-1. Full text in Research Archive Show summary

  Background Fair comparisons of educational systems in large-scale assessments can be made only if the differences in curricula have little impact on the outcomes. This study investigated the sensitivity of science achievement rankings to varying degrees of curriculum implementation in the Trends in International Mathematics and Science Study (TIMSS). Methods Country-specific teacher-reported curriculum implementation profiles across the TIMSS science domains were charted including their within-country variability across the classrooms for 33 participating countries of TIMSS 2015. A sensitivity test compared the original ranking to TIMSS curriculum implementation scenarios (a least-possible, a most-possible, and more realistic country-specific median implementation). Results In contrast to expectations, no support was found for a positive relationship between opportunity to learn and science achievement at the between-country level or the within-country level, with only minor exceptions. The sensitivity analysis under different curriculum implementation scenarios also suggests little impact on the rank order of the countries. Conclusions Plausible explanations for this null finding are addressed; attention and research efforts should focus on improving the quality of curriculum implementation indicators in large-scale assessments.

• Daus, Stephan; Nilsen, Trude & Braeken, Johan (2018). Exploring content knowledge: Country profile of science strengths and weaknesses in TIMSS. Possible implications for educational professionals and science research. Scandinavian Journal of Educational Research. ISSN 0031-3831. p. 1–19. doi: 10.1080/00313831.2018.1478882. Full text in Research Archive Show summary

  This study offers curriculum developers, teachers and science education researchers a fine-grained profile on strengths and weaknesses in specific science domains and topics. The study involved a representative sample of 3844 Norwegian pupils in grade 8. Their responses on 216 TIMSS items in 18 topics from 4 science domains were modelled in a hierarchical item response model. An internal comparison identified topics that were relatively harder or easier compared to other topics for the Norwegian pupils, and an external comparison identified topics that were relatively harder or easier for the Norwegian pupils compared with the average of TIMSS participating countries. Interpretation of the profile necessitates contextualisation; hence, these strengths and weaknesses as well as their plausible explanations and possible implications are discussed from the perspectives of curriculum development, teacher training and science education research.

• Paap, Muirne C. S.; Braeken, Johan; Pedersen, Geir Feigum; Urnes, Øyvind; Karterud, Sigmund & Wilberg, Theresa [Show all 7 contributors for this article] (2017). A Psychometric Evaluation of the DSM-IV Criteria for Antisocial Personality Disorder: Dimensionality, Local Reliability, and Differential Item Functioning Across Gender. Assessment (ASM). ISSN 1073-1911. doi: 10.1177/1073191117745126. Full text in Research Archive Show summary

  This study aims at evaluating the psychometric properties of the antisocial personality disorder (ASPD) criteria in a large sample of patients, most of whom had one or more personality disorders (PD). PD diagnoses were assessed by experienced clinicians using the Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, 4th edition, Axis II PDs. Analyses were performed within an item response theory framework. Results of the analyses indicated that ASPD is a unidimensional construct that can be measured reliably at the upper range of the latent trait scale. Differential item functioning across gender was restricted to two criteria and had little impact on the latent ASPD trait level. Patients fulfilling both the adult ASPD criteria and the conduct disorder criteria had similar latent trait distributions as patients fulfilling only the adult ASPD criteria. Overall, the ASPD items fit the purpose of a diagnostic instrument well, that is, distinguishing patients with moderate from those with high antisocial personality scores.

• Pettersen, Andreas & Braeken, Johan (2017). Mathematical Competency Demands of Assessment Items: a Search for Empirical Evidence. International Journal of Science and Mathematics Education. ISSN 1571-0068. 17(2), p. 405–425. doi: 10.1007/s10763-017-9870-y. Full text in Research Archive Show summary

  The implementation of mathematical competencies in school curricula requires assessment instruments to be aligned with this new view on mathematical mastery. However, there are concerns over whether existing assessments capture the wide variety of cognitive skills and abilities that constitute mathematical competence. The current study applied an explanatory item response modelling approach to investigate how teacher-rated mathematical competency demands could account for the variation in item difficulty for mathematics items from the Programme for International Student Assessment (PISA) 2012 survey and a Norwegian national grade 10 exam. The results show that the rated competency demands can explain slightly more than and less than half of the variance in item difficulty for the PISA and exam items, respectively. This provides some empirical evidence for the relevance of the mathematical competencies for solving the assessment items. The results also show that for the Norwegian exam, only two of the competencies, Reasoning and argument and Symbols and formalism, appear to influence the difficulty of the items, which questions to what extent the exam items capture the variety of cognitive skills and abilities that constitute mathematical competence. We argue that this type of empirical data from psychometric modelling should be used to improve assessments and assessment items, as well as to inform and possibly further develop theoretical concepts of mathematical competence.

• Arnesen, Anne; Braeken, Johan; Baker, Scott; Meek-Hansen, Wilhelm; Ogden, Terje & Melby-Lervåg, Monica (2017). Growth in Oral Reading Fluency in a Semitransparent Orthography: Concurrent and Predictive Relations With Reading Proficiency in Norwegian, Grades 2–5. Reading Research Quarterly. ISSN 0034-0553. 52(2), p. 177–201. doi: 10.1002/rrq.159. Full text in Research Archive Show summary

  This study investigated an adaptation of the Oral Reading Fluency (ORF) measure of the Dynamic Indicators of Basic Early Literacy Skills into a European context for the Norwegian language, which has a more transparent orthography than English. Second-order latent growth curve modeling was used to examine the longitudinal measurement invariance of the ORF measure, the growth in oral reading fluency within and across grades 2–5, the relative stability of the ORF measure, and the relationship between the ORF measure and high-stakes national tests of reading proficiency. Results showed that the ORF passages measured the same underlying construct, but some passages stood out regarding the invariance pattern. The oral reading fluency growth curve models demonstrated a linear growth in grades 2 and 3 and a nonlinear growth in grades 4 and 5. Initial individual differences varied more than growth rates, which for all were positive but largest in grades 3 and 4. High relative stability in the ORF measure was found across grades. The concurrent and predictive relations of the ORF measure on the Norwegian national reading tests were moderate to strong (range = .44–.75). Findings indicated that the ORF is a reliable and valid measure of reading in Norwegian grades 2–5 and easy and fast to administer. The ORF measure might contribute to early identification of students at risk for reading difficulties in an orthography more transparent than English. Implications for school practice and future research are discussed.

• Hall, Ruby A.S.; Hoffenkamp, Hannah N.; Braeken, Johan; Tooten, Anneke; Vingerhoets, Ad J.J.M. & Bakel, Hedwig J.A. van (2017). Maternal psychological distress after preterm birth: Disruptive or adaptive? Infant Behavior and Development. ISSN 0163-6383. 49, p. 272–280. doi: 10.1016/j.infbeh.2017.09.012. Full text in Research Archive Show summary

  Background: Maternal postpartum distress is often construed as a marker of vulnerability to poor parenting. Less is known, however, about the impact of postpartum distress on parenting an infant born prematurely. The present study investigated whether high distress levels, which are particularly prevalent in mothers of preterm born infants, necessarily affect a mother’s quality of parenting. Method: Latent Class Analysis was used to group mothers (N = 197) of term, moderately, and very preterm born infants, based on their levels of distress (depression, anxiety, and PTSD symptoms) at one month postpartum, and their quality of parenting at one and six months postpartum. Parenting quality was assessed on the basis of maternal interactive behaviors (sensitivity, intrusiveness, and withdrawal) using observations, and maternal attachment representations (balanced, disengaged, or distorted) using interviews. Results: A 5-Class model yielded the best fit to the data. The first Class (47%) of mothers was characterized by low distress levels and high-quality parenting, the second Class (20%) by low distress levels and low-quality parenting, the third Class (22%) by high distress levels and medium-quality parenting, the fourth Class (9%) by high distress levels and high-quality parenting, and finally the fifth Class (2%) by extremely high levels of distress and low-quality parenting. Conclusions: While heightened distress levels seem inherent to preterm birth, there appears to be substantial heterogeneity in mothers’ emotional responsivity. This study indicates that relatively high levels of distress after preterm birth do not necessarily place these mothers at increased risk with regard to poor parenting. Conversely, low distress levels do not necessarily indicate good-quality parenting. The results of the present study prompt a reconsideration of the association between postpartum distress and parenting quality, and challenge the notion that high levels of maternal distress always result in low-quality parenting practices. The final version of this research has been published in Infant Behaviour and Development. © 2017 Elsevier

• Rutten, Elisabeth A P; Bachrach, Nathan; Balkom, Anton J L M van; Braeken, Johan; Ouwens, Machteld A & Bekker, Marrie H J (2016). Anxiety, depression and autonomy–connectedness: The mediating role of alexithymia and assertiveness. Psychology and Psychotherapy: Theory, Research and Practice. ISSN 1476-0835. 89(4), p. 385–401. doi: 10.1111/papt.12083. Full text in Research Archive
• Wood, Stephen; Niven, Karen & Braeken, Johan (2016). Managerial abuse and the process of absence among mental health staff. Work, Employment and Society. ISSN 0950-0170. 30(5), p. 783–801. doi: 10.1177/0950017015613755. Full text in Research Archive
• Braeken, Johan & Assen, Marcel A. L. M. Van (2016). An empirical Kaiser criterion. Psychological methods. ISSN 1082-989X. 22(3), p. 450–466. doi: 10.1037/met0000074. Full text in Research Archive
• Braeken, Johan & Blömeke, Sigrid (2016). Comparing future teachers’ beliefs across countries: approximate measurement invariance with Bayesian elastic constraints for local item dependence and differential item functioning. Assessment & Evaluation in Higher Education. ISSN 0260-2938. 41(5), p. 733–749. doi: 10.1080/02602938.2016.1161005. Full text in Research Archive
• Hall, Ruby A.S.; Hoffenkamp, Hannah N.; Tooten, Anneke; Braeken, Johan; Vingerhoets, Ad J.J.M. & Bakel, Hedwig J.A. van (2015). The quality of parent–infant interaction in the first 2 years after full-term and preterm birth. Parenting, science and practice. ISSN 1529-5192. 15(4), p. 247–268. doi: 10.1080/15295192.2015.1053333. Full text in Research Archive
• Dang, Cai-Ping; Braeken, Johan; Colom, Roberto; Ferrer, Emilio & Liu, Chang (2015). Do processing speed and short-term storage exhaust the relation between working memory capacity and intelligence? Personality and Individual Differences. ISSN 0191-8869. 74, p. 241–247. doi: 10.1016/j.paid.2014.10.012. Full text in Research Archive
• Hoffenkamp, Hannah; Tooten, Anneke; Hall, Ruby; Braeken, Johan; Eliëns, Marij & Vingerhoets, Ad [Show all 7 contributors for this article] (2015). Effectiveness of Hospital-Based Video Interaction Guidance on Parental Interactive Behavior, Bonding, and Stress After Preterm Birth: A Randomized Controlled Trial. Journal of Consulting and Clinical Psychology. ISSN 0022-006X. 83, p. 416–429. doi: 10.1037/a0038401. Full text in Research Archive
• Traa, Marjan; Braeken, Johan; Vries, Jolanda De; Roukema, Jan; Orsini, Ricardo & Oudsten, Brenda L. Den (2015). Evaluating quality of life and response shift from a couple-based perspective: a study among patients with colorectal cancer and their partners. Quality of Life Research. ISSN 0962-9343. 24, p. 1431–1441. doi: 10.1007/s11136-014-0872-8. Full text in Research Archive
• Hoffenkamp, Hannah N.; Braeken, Johan; Hall, Ruby A.S.; Tooten, Anneke; Vingerhoets, Ad J.J.M. & Bakel, Hedwig J.A. van (2015). Parenting in complex conditions: Does preterm birth provide a context for the development of less optimal parental behavior? Journal of Pediatric Psychology. ISSN 0146-8693. 40(6), p. 559–571. doi: 10.1093/jpepsy/jsv007. Full text in Research Archive
• Braeken, Johan; Mulder, Joris & Wood, Stephen (2015). Relative Effects at Work: Bayes Factors for Order Hypotheses. Journal of Management. ISSN 0149-2063. 41(2), p. 544–573. doi: 10.1177/0149206314525206. Full text in Research Archive
• Hall, RAS; Hoffenkamp, HN; Tooten, A; Braeken, Johan; Vingerhoets, AJJM & Bakel, HJA van (2015). Longitudinal associations between maternal disrupted representations, maternal interactive behavior and infant attachment: a comparison between full-term and preterm dyads. Child Psychiatry and Human Development. ISSN 0009-398X. 46(2), p. 320–331. doi: 10.1007/s10578-014-0473-3. Full text in Research Archive
• Hall, RAS; Hoffenkamp, HN; Tooten, A; Braeken, Johan; Vingerhoets, AJJM & Bakel, HJA Van (2015). Child-Rearing History and Emotional Bonding in Parents of Preterm and Full-Term Infants. Journal of Child and Family Studies. ISSN 1062-1024. 24(6), p. 1715–1726. doi: 10.1007/s10826-014-9975-7. Full text in Research Archive
• Traa, Marjan J.; Braeken, Johan; Vries, Jolanda De; Roukema, Jan A.; Slooter, Gerrit D. & Crolla, Rogier M.P.H. [Show all 8 contributors for this article] (2015). Sexual, marital, and general life functioning in couples coping with colorectal cancer: A dyadic study across time. Psycho-Oncology. ISSN 1057-9249. 24(9), p. 1181–1188. doi: 10.1002/pon.3801. Full text in Research Archive
• Vreeswijk, Charlotte; Maas, A Janneke; Rijk, Catharina; Braeken, Johan & Bakel, Hedwig van (2014). Stability of fathers' representations of their infants during the transition to parenthood. Attachment & Human Development. ISSN 1461-6734. 16, p. 292–306. doi: 10.1080/14616734.2014.900095. Full text in Research Archive
• Maas, A Janneke; Vreeswijk, Charlotte; Braeken, Johan; Vingerhoets, Ad & Bakel, Hedwig van (2014). Determinants of maternal fetal attachment in women from a community-based sample. Journal of Reproductive and Infant Psychology. ISSN 0264-6838. 32, p. 5–24. doi: 10.1080/02646838.2013.853170. Full text in Research Archive
• Dinther, Mart van; Dochy, Filip; Segers, Mien & Braeken, Johan (2014). Student perceptions of assessment and student self-efficacy in competence-based education. Educational Studies. ISSN 0305-5698. 40(3), p. 330–351. doi: 10.1080/03055698.2014.898577. Full text in Research Archive
• Braeken, Johan (2011). A BOUNDARY MIXTURE APPROACH TO VIOLATIONS OF CONDITIONAL INDEPENDENCE. Psychometrika. ISSN 0033-3123. 76, p. 57–76. doi: 10.1007/S11336-010-9190-4. Full text in Research Archive

View all works in NVA

• Ludvigsen, Sten Runar; Nortvedt, Guri A.; Petterson, Astrid; Sollerman, Samuel; Olafsson, Ragnar F. & Taajamo, Matti [Show all 9 contributors for this article] (2016). Northern Lights on PISA and TALIS. Nordisk ministerråd. ISBN 9789289345217. 216 p. Full text in Research Archive

View all works in NVA

• Chen, Jianan & Braeken, Johan (2026). Traditional Progressive Tests with Stopping Rules from Computerized Adaptive Testing: Do Potential Advantages Outweigh Added Complexity? Full text in Research Archive
• Chen, Jianan; Brinchmann, Ellen Irén & Braeken, Johan (2025). Setting Stopping Rules for Progressive Tests: A Practical and Transparent Toolkit. Full text in Research Archive
• Braeken, Johan; Lehre, Anne- Catherine W G & Marcq, Kseniia (2025). The risks and perils of cherry picking results from international large-scale educational assessments: The case of the «Nordic» fractions. Full text in Research Archive
• Steinmann, Isa; Chen, Jianan & Braeken, Johan (2023). Challenging the design principle of mixed-worded questionnaire scales. Full text in Research Archive
• Steinmann, Isa; Ruiz, Jesus Daniel Sanchez; Laar, Saskia Van & Braeken, Johan (2022). How do inconsistent responders to mixed-worded questionnaire scales affect inferences in international large scale assessments? Full text in Research Archive
• Steinmann, Isa; Braeken, Johan & Strietholt, Rolf (2021). Identifying Inconsistent Respondents to Mixed-Worded Scales in Large-Scale Assessments. Full text in Research Archive
• Laar, Saskia Van & Braeken, Johan (2021). Random responders in international large-scale assessments in education: A threat to validity? Full text in Research Archive
• Kalandadze, Tamar; Braeken, Johan; Brynskov, Cecilia & Næss, Kari-Anne Bottegaard (2019). Difficulties in Metaphor Comprehension: A Hallmark of Autism Spectrum Disorder or of Poor Structural Language Ability? Full text in Research Archive
• Laar, Saskia Van & Braeken, Johan (2019). Decomposing the Comparative Fit Index: Effect of model characteristics on CFI performance. Full text in Research Archive
• Laar, Saskia Van & Braeken, Johan (2019). Random responders in international large-scale assessments in education: A threat to validity. Full text in Research Archive
• Braeken, Johan & Laar, Saskia Van (2019). Self-reported personality of Parents & Perceived Temperament of their Infant. Full text in Research Archive
• Daus, Stephan; Stancel-Piatak, Agnes & Braeken, Johan (2019). Curriculum sensitivity of the TIMSS science test: Adjacent-grades within schools design. Full text in Research Archive
• Laar, Saskia Van & Braeken, Johan (2018). Model fit: Rules-of-thumb vs Sampling distributions. Full text in Research Archive
• Laar, Saskia Van & Braeken, Johan (2018). Measurement Invariance in Within-Group Designs. Full text in Research Archive
• Laar, Saskia Van & Braeken, Johan (2018). Measurement Invariance in Within-Subjects Designs. Full text in Research Archive
• Daus, Stephan; Stancel-Piatak, Agnes & Braeken, Johan (2018). Instructional sensitivity of the TIMSS items and test to Norwegian science education instruction. Full text in Research Archive
• Daus, Stephan & Braeken, Johan (2017). Are TIMSS country scores and rankings sensitive to varying opportunities to learn? Full text in Research Archive
• Helland-Riise, Fredrik & Braeken, Johan (2017). The Radicals of Abstract Reasoning Assessments: An Explanatory Item Response Modelling Approach. Full text in Research Archive Show summary

  Abstract reasoning assessments are widely considered the gold standard of intelligence testing. Although Raven Progressive Matrices (RPM) are probably the most prominent example, there are many more less studied and less developed abstract reasoning assessments used in practice. To improve these assessments and generalize RPM study results, we need a better understanding of what factors of the item design drive item response behavior. In modern test design such factors are called radicals, item properties that when modified lead to for instance a change in item difficulty; this in contrast to incidentals, item properties that are mere cosmetic hanges not affecting psychometric item characteristics. Here, we se an explanatory item response modelling approach to examine he effects of potential radical item properties derived from a cognitive lab and from an artificial intelligent item solver to provide validity evidence for an non-Raven-like abstract reasoning assessment. We discuss how this validity evidence can be a first step in a more systematic redesign of the assessment opening up opportunities for automatic item generation and computerized adaptive testing.

• Arnesen, Anne; Smolkowski, Keith; Baker, Scott; Braeken, Johan; Meek-Hansen, Wilhelm & Ogden, Terje [Show all 7 contributors for this article] (2017). Assessing Elementary Students’ Social Functioning and Reading Proficiency. Full text in Research Archive
• Tesema, Melaku Tesfa & Braeken, Johan (2016). Regional and gender differences in academic achievement as a function of opportunities to learn: Evidence from Ethiopia. Full text in Research Archive
• Helland-Riise, Fredrik & Braeken, Johan (2016). Explanatory Item Response Modelling Of An Abstract Reasoning Assessment: A Case For Modern Test Design. Full text in Research Archive Show summary

  General description Reasoning tests are popular components of the assessment toolbox for selection and admission into higher education and job employment (Leighton, 2004; Stanovich, Sá, & West, 2004). Abstract reasoning tests tap into a core reasoning ability (Carpenter, Just, & Shell, 1990) with tasks that require the examinee to generate and apply rules (Wüstenberg, Greiff, & Funke, 2012), but require neither explicit prior contents-specific knowledge of the examinee nor specific language skills (Raven, 2000). Traditionally, test construction and assembly have been the product of creative item writing processes and post-hoc psychometric evaluations, without explicit consideration of cognitive theory (Hunt, Frost, & Lunneborg, 1973). Yet, abstract reasoning provides a case that in principle is ideally suitable for modern test design (e.g. Embretson, 1998; Mislevy, Almond, & Lukas, 2003), combining cognitive theory with a more systematic approach to construction and assembly of test items. Objective and Research Questions. This study is part of a larger project aimed at reverse engineering an existing abstract reasoning test from a modern test design perspective to setup a virtual item bank that does not store individual items, but instead uses automatic item generation rules based on cognitive complexity (see e.g., Gierl & Haladyna, 2013). The objective of the current study represents one step towards such a virtual item bank with research questions focusing on (i) identifying the cognitive relevant item features (i.e. “radicals”) that impact the behaviour of the test and of the participants and (ii) identifying the merely “cosmetic” irrelevant item features (i.e., incidentals). The test. The abstract reasoning test is composed of testlets consisting of items related to the same problem situation from which a set of rules need to be derived that are necessary to solve the individual items. Each testlet is structured around a problem set consisting of a varying number of rows each consisting of a specified input stimulus configuration, an activated set of action buttons and a resulting output stimulus configuration. This problem set allows the examinee to derive the transformations that will happen to the input when a specific action button is activated. This rule knowledge is necessary to solve the connected items. An item consists of a single row with a specified input stimulus configuration, the activated set of action buttons for that item, and four alternative output stimulus configuration possibilities of which the examinee has to decide on the correct one. Theoretical framework. A rational task analysis of the abstract reasoning test proposes an artificial intelligent algorithm (see Newell & Simon, 1972) that consists of 4 core steps. (1) Inventorisation: all the characteristics of input stimulus configurations and output stimulus configurations of the problem set are registered; (2) Matching: an input/output dissimilarity matrix is computed; (3) Rule finding: computationally this would be similar to solving a system of equations or a more greedy version using elimination; (4) Rule application. The test has some characteristics built in by design that can be directly connected to the artificial intelligent algorithm and the related (i) cognitive load of the stimulus material and (ii) cognitive complexity of the rules that need to be derived. Examples of the former characteristics can be as simple as the number of symbols in the input stimulus configuration, examples of the latter characteristics can be whether or not the transformation caused by a specific action button can be derived on its own (i.e., independent of the other action buttons in the problem set). Some theoretically irrelevant item features can also be defined such as the type of symbols used in a stimulus configuration (e.g., triangle or circle).

• Arnesen, Anne; Braeken, Johan; Baker, Scott; Meek-Hansen, Wilhelm & Ogden, Terje (2016). Growth in Oral Reading Fluency in a Semi-Transparent Orthography. Full text in Research Archive
• Daus, Stephan; Nilsen, Trude & Braeken, Johan (2015). Explaining Item Difficulty Heterogeneity Across Content Groups with Curricular Coverage. Full text in Research Archive
• Scherer, Ronny & Braeken, Johan (2014). The issue of item dependencies in problem solving assessments: Comparing different modeling approaches. Full text in Research Archive
• Lehre, Anne- Catherine W G & Braeken, Johan (2023). Using TIMSS data to explore Nordic struggles with fractions. Universitetet i Oslo. Full text in Research Archive Show summary

  Feeling uncertain when you deal with fractions? You are not alone! Dating back more than 4,000 years, fractions have always played a major role in society. Despite its importance, research shows that many pupils, students, and adults struggle with fractions. A blogger pointed to Nordic grade 8 students struggling more than other students around the globe when trying to solve the task: «Which shows a correct method for finding ⅓ – ¼? », taken from an international large-scale assessment in education called Trends in Mathematics and Science Study (TIMSS). Using TIMSS data from several countries and spanning almost 20 years, we explored the claim that Nordic students struggle more with fractions and/or common denominator tasks than students elsewhere in the world. Focusing on tasks in mathematics, the results show that Nordic students, neither struggle more on fractions tasks than non-fraction tasks, nor struggle more on common denominator tasks than non-common denominator tasks. A similar pattern is seen within each of the countries included in this study. However, not limited to the specific student cohort mentioned in the blog, but also other cohorts of Nordic students struggle with specific tasks similar to «Which shows a correct method for finding ⅓ – ¼? ». Future research could explore why: For example, are there noticeable cross-cultural differences in instructions and textbooks used.

View all works in NVA