Jianan Chen

• Chen, Jianan; Brinchmann, Ellen Irén & Braeken, Johan (2025). Setting Discontinue Rules for Progressive Tests: A Practical and Transparent Toolkit. Assessment (ASM). ISSN 1073-1911. doi: 10.1177/10731911251385776. Full text in Research Archive Show summary

  Progressive tests are a popular tailored test format where items are administered in increasing order of difficulty level and discontinued according to a rule system that should counter excessive response burden for test participants and guarantee efficient use of resources for test administrators. To facilitate evidence-based decision-making for setting appropriate discontinue rules, we propose a transparent approach that charts the impact of varied alternative discontinue rules on accuracy and efficiency. These A–E charts are based on retroactively applying discontinue rules to normative item response data. We show that a universal discontinue rule likely does not exist and that the optimal rule varies as a function of the desired efficiency-accuracy trade-off suitable for the intended test use and target population. The proposed approach provides a pragmatic solution for practitioners, researchers, test developers, and test publishers to rethink the existing discontinue rules, systematically evaluate the alternatives, and set appropriate rules.

• Steinmann, Isa; Chen, Jianan & Braeken, Johan (2024). Who Responds Inconsistently to Mixed-Worded Scales? Differences by Achievement, Age Group, and Gender. Assessment in Education: Principles, Policy & Practice. ISSN 0969-594X. doi: 10.1080/0969594X.2024.2318554. Full text in Research Archive Show summary

  We investigated two research questions: which students are more likely to respond inconsistently to mixed-worded questionnaire scales, and which country samples have larger shares of inconsistent respondents? We defined an inconsistent response as strongly agree- ing or disagreeing with both positively and negatively worded items of the same scale. Since we assumed that inconsistent responding occurs due to a lack of carefulness, reading, or cognitive skills, we expected to find that inconsistent responding was associated with lower achievement, younger age, being a nonnative speaker, and being a boy. We used data from all 38 countries that participated in the fourth- and eighth-grade assessments of TIMSS (Trends in International Mathematics and Science Study) 2019. Using the mean absolute difference method, we identified shares of 1‒21% inconsistent respondents across samples. The results generally sup- ported our hypotheses, especially the hypothesis that inconsistent responding is more common among students and countries with lower mathematics achievement levels.

• Chen, Jianan; Steinmann, Isa & Braeken, Johan (2024). Competing explanations for inconsistent responding to a mixed-worded self-esteem scale: Cognitive abilities or personality? Personality and Individual Differences. ISSN 0191-8869. 222, p. 1–6. doi: 10.1016/j.paid.2024.112573. Full text in Research Archive Show summary

  In survey scale design, a mixed-worded format intends to ensure attentiveness as respondents need to take into account the wording direction when answering an item. However, some respondents tend to deliver inconsistent responses (i.e., agreeing or disagreeing with both positively and negatively-worded items), posing a validity concern. Two potential directions driving inconsistent responding have each individually been put forward: lack of cognitive abilities to effectively deal with the mixed-wording and sheer carelessness while responding. Using a factor mixture approach, we investigated inconsistent responding on Rosenberg’s self-esteem scale as a function of both cognitive ability and personality. Among n = 4938 Grade 5 students from the German National Educational Panel Study (NEPS), 11 % were classified as inconsistent respondents and class memberships were further related to four cognitive abilities (cognitive reasoning, cognitive speed, reading comprehension, reading speed) and five personality traits (conscientiousness, neuroticism, extraversion, agreeableness, openness). Model comparison results indicated that both ability and personality predictors matter with a more prominent role for ability (especially reading comprehension). We discuss the implications of these findings for mixed-wording’s suitability for scale construction and different populations and how researchers can deal with suspected inconsistent respondents in their survey data.

• Laar, Saskia Van; Chen, Jianan & Braeken, Johan (2024). How Randomly are Students Random Responding to Your Questionnaire? Within-Person Variability in Random Responding Across Scales in the TIMSS 2015 Eighth-Grade Student Questionnaire. Measurement: Interdisciplinary Research and Perspectives. ISSN 1536-6367. 22(1), p. 90–107. doi: 10.1080/15366367.2023.2203972. Full text in Research Archive Show summary

  Questionnaires in educational research assessing students’ attitudes and beliefs are low-stakes for the students. As a consequence, students might not always consistently respond to a questionnaire scale but instead provide more random response patterns with no clear link to items’ contents. We study inter-individual differences in students’ intra-individual random responding profile across 19 questionnaire scales in the TIMSS 2015 eighth-grade student questionnaire in seven countries. A mixture IRT approach was used to assess students’ random responder status on a questionnaire scale. A follow-up latent class analysis across the questionnaire revealed four random responding profiles that generalized across countries: A majority of consistent nonrandom responders, intermittent moderate random responders, frequent random responders, and students that were exclusively triggered to respond randomly on the confidence scales in the questionnaire. We discuss the implications of our findings in light of general data-quality concerns and the potential ineffectiveness of early-warning monitoring systems in computer-based surveys.

• Chen, Jianan; Laar, Saskia van & Braeken, Johan (2023). Who are those random responders on your survey? The case of the TIMSS 2015 student questionnaire. Large-scale Assessments in Education. 11(1). doi: 10.1186/s40536-023-00184-6. Full text in Research Archive Show summary

  A general validity and survey quality concern with student questionnaires under low-stakes assessment conditions is that some responders will not genuinely engage with the questionnaire, often with more random response patterns as a result. Using a mixture IRT approach and a meta-analytic lens across 22 educational systems participating in TIMSS 2015, we investigated whether the prevalence of random responders on six scales regarding students' engagement and attitudes toward mathematics and sciences was a function of grade, gender, socio-economic status, spoken language at home, or migration background. Among these common policy-relevant covariates in educational research, we found support for small group differences in prevalence of random responders (OR ≥ 1.22)(average prevalence of 7%). In general, being a student in grade 8 (vs. grade 4), being male, reporting to have fewer books, or speaking a language different from the test language at home were all risk factors characterizing random responders. The expected generalization and implications of these findings are discussed based on the observed heterogeneity across educational systems and consistency across questionnaire scales.

• Shang, Wenru; Wan, Yang; Chen, Jianan; Du, Yanqiu & Huang, Jiayan (2021). Introducing the non-invasive prenatal testing for detection of down syndrome in China: A cost-effectiveness analysis. BMJ Open. 11(7), p. 1–11. doi: 10.1136/bmjopen-2020-046582. Full text in Research Archive

View all works in NVA

• Chen, Jianan & Braeken, Johan (2026). Traditional Progressive Tests with Stopping Rules from Computerized Adaptive Testing: Do Potential Advantages Outweigh Added Complexity? Full text in Research Archive
• Chen, Jianan; Brinchmann, Ellen Irén & Braeken, Johan (2025). Setting Stopping Rules for Progressive Tests: A Practical and Transparent Toolkit. Full text in Research Archive
• Steinmann, Isa; Chen, Jianan & Braeken, Johan (2023). Challenging the design principle of mixed-worded questionnaire scales. Full text in Research Archive

View all works in NVA