Esther Ulitzsch

• Ulitzsch, Esther; Nestler, Steffen; Johnson, Sverre Urnes; Snuggerud, Therese Ruud & Lüdtke, Oliver (2026). Capturing Heterogeneity in Levels, Variability, and Couplings across Persons and Time with a Hierarchical Time-Varying Coefficient Formulation of the Multivariate Normal. Psychometrika. ISSN 0033-3123. doi: 10.1017/psy.2026.10125. Full text in Research Archive Show summary

  Time-varying coefficient modeling (TVCM), which represents regression coefficients as smooth functions of continuous time, provides a flexible framework for uncovering complex patterns of change in levels and associations in intensive longitudinal data. However, conventional TVCM remains limited to investigating directional effects across individuals. By introducing a TVCM formulation of the multivariate normal distribution, the present study extends TVCM to explore change in undirected associations (couplings) and variability, thereby broadening its utility for psychological research. We discuss three versions of this approach: an aggregate-level model and two hierarchical versions capturing interindividual differences in unfolding change, either via person-specific intercepts accounting for onset differences or through fully person-specific coefficient functions smoothed via partial pooling. To illustrate the proposed developments, we apply them to six weeks of intensive longitudinal data from 16 anxiety patients undergoing therapy and examine unfolding changes in the level and volatility of nervousness and threat monitoring, their coupling, as well as between-person heterogeneity in each of these. We further show how inspecting first-order derivatives of the coefficient functions supports identifying periods of stability and change. Finally, we discuss extensions incorporating person-level characteristics to explain heterogeneity in patterns of change and predict outcomes.

• Ulitzsch, Esther; Tetzlaff, Leonard; Goldhammer, Frank; Hahnel, Carolin; Kroehne, Ulf & Lüdtke, Oliver (2026). Does sequence mining uncover generalizable behavioral patterns? A methodological case study. Large-scale Assessments in Education. 14(1). doi: 10.1186/s40536-026-00289-8. Full text in Research Archive
• Zhang, Lijin; Rahal, Charles; Kanopka, Klint; Ulitzsch, Esther; Zhang, Zhiyong & Domingue, Benjamin W. (2026). Evaluating Model Predictive Performance in Confirmatory Factor Analysis with Binary Outcomes Using the InterModel Vigorish. Multivariate Behavioral Research. ISSN 0027-3171. doi: 10.1080/00273171.2026.2645212. Full text in Research Archive
• Snuggerud, Therese Ruud; Ulitzsch, Esther; Nordahl, Henrik; Vrabel, Kari-Anne; Hoffart, Asle & Ebrahimi, Omid V. [Show all 9 contributors for this article] (2026). Dynamic changes in metacognitive mechanisms and symptoms during the attention training technique: Insights from ecological momentary assessment. Journal of Anxiety Disorders. ISSN 0887-6185. 119, p. 103136–103136. doi: 10.1016/j.janxdis.2026.103136. Full text in Research Archive Show summary

  Background Anxiety disorders are prevalent and associated with substantial distress and impairment. Within the metacognitive model of emotional disorders, the cognitive attentional syndrome (CAS) and associated metacognitions maintain anxiety. The Attention Training Technique (ATT) targets the CAS and related metacognitions. However, less is known about how symptoms and metacognitive mechanisms evolve across treatment phases. Objective The study aimed to explore temporal dynamics in emotional symptoms and metacognitive mechanisms during group ATT by estimating mean levels, variability, and coupling across the pre-, during-, and post-treatment phases using ecological momentary assessment (EMA). Method Nineteen young adults with social anxiety disorder, panic disorder with agoraphobia, or generalized anxiety disorder participated. EMA was completed four times daily over six weeks: two weeks before treatment, two weeks during treatment, and two weeks after treatment. Each assessment included ten items measuring emotional symptoms (e.g., nervousness) and metacognitive mechanisms (e.g., worry, uncontrollability). Results Three patterns of change emerged: (1) reduced mean levels without change in variability (nervousness, self-consciousness, and worry); (2) reduced mean levels and reduced variability (threat monitoring); and (3) reduced mean levels and reduced variability during treatment, followed by an increase in variability after treatment (uncontrollability, difficulty collecting thoughts, and meta-worry). Conclusions The findings provide new insights into dynamic patterns of change in symptoms and metacognitive mechanisms before, during, and after ATT. Patterns of change in both intensity and stability may be relevant for understanding treatment response and vulnerability following treatment, suggesting the potential clinical relevance of monitoring these mechanisms.

• Domingue, Benjamin W.; Kanopka, Klint; Ulitzsch, Esther & Zhang, Lijin (2025). Implied probabilities of polytomous response functions for model-based prediction and comparison. Behaviormetrika. ISSN 0385-7417. 52(2), p. 705–705. doi: 10.1007/s41237-025-00262-9. Full text in Research Archive
• Bulut, Okan; Yildirim-Erbasli, Seyma Nur; Ulitzsch, Esther & Gorgun, Guher (2025). Measuring Consumers’ Financial Well-Being: Uncovering the Causes and Consequences of Careless and Insufficient Effort in Self-Report Responses. Financial Counseling and Planning. ISSN 1052-3073. p. 0103–0103. doi: 10.1891/jfcp-2024-0103. Full text in Research Archive Show summary

  This study investigates the prevalence and causes of careless and insufficient effort responding (C/IER) in a self-report instrument assessing financial well-being. Using an explanatory mixture item response modeling approach, we analyze data from 6,394 respondents who responded to the Consumer Financial Protection Bureau’s financial well-being and financial skills scales. We identify different patterns of C/IER and examine factors that contribute to its occurrence in the two scales. The results indicate that 14% of the responses were impacted by C/IER, with question characteristics (e.g., negatively worded or frequency-based questions) and respondent demographics (e.g., age, gender, and education level) exhibiting significant relationships with attentiveness. Respondents with higher financial knowledge were more attentive, while younger individuals and those experiencing financial shocks were more prone to careless responding. These findings underscore the importance of carefully designing self-report instruments to minimize response biases and ensure data quality. By addressing different question designs and respondent-level characteristics, future instruments can yield more reliable and valid insights into individuals’ financial well-being.

• Zhang, Lijin; Ulitzsch, Esther & Domingue, Benjamin W. (2025). Bayesian factor mixture modeling with response time for detecting careless respondents. Behavior Research Methods. ISSN 1554-351X. 57. doi: 10.3758/s13428-025-02797-x. Full text in Research Archive Show summary

  Careless respondents inject noise into data which can distort research findings and compromise model fit. To address this, factor mixture modeling (FMM) has been widely used to identify careless respondents. Traditionally, researchers have relied on reverse-worded questions in FMM to facilitate the detection of careless responding. With the rise of online data collection platforms, response time has appeal as a means for understanding careless behavior. We introduce a Bayesian FMM that leverages this rich source of information to identify careless respondents. By jointly modeling responses and response time, this approach effectively identifies careless individuals rushing through the questionnaire without providing responses that reflect the to-be-measured traits. Our simulation studies demonstrate that this model accurately estimates parameters and classifies respondents as either attentive or careless, while maintaining error rates within acceptable limits. Furthermore, integrating response time enhances model convergence and the precision of classification and estimation. Using mediation models as an example, we illustrate how social science researchers can use this FMM approach to address careless responding in substantive research. An empirical study further tests the applicability of the proposed model in real-world scenarios, comparing its conclusions with traditional methods. To support its use, we provide an R function to streamline implementation.

• Zhang, Lijin; Domingue, Benjamin W.; Vogelsmeier, Leonie V. D. E. & Ulitzsch, Esther (2025). A Beta Mixture Model for Careless Respondent Detection in Visual Analogue Scale Data. Psychometrika. ISSN 0033-3123. 90(4), p. 1558–1581. doi: 10.1017/psy.2025.10041. Full text in Research Archive Show summary

  Visual Analogue scales (VASs) are increasingly popular in psychological, social, and medical research. However, VASs can also be more demanding for respondents, potentially leading to quicker disengagement and a higher risk of careless responding. Existing mixture modeling approaches for careless response detection have so far only been available for Likert-type and unbounded continuous data but have not been tailored to VAS data. This study introduces and evaluates a model-based approach specifically designed to detect and account for careless respondents in VAS data. We integrate existing measurement models for VASs with mixture item response theory models for identifying and modeling careless responding. Simulation results show that the proposed model effectively detects careless responding and recovers key parameters. We illustrate the model’s potential for identifying and accounting for careless responding using real data from both VASs and Likert scales. First, we show how the model can be used to compare careless responding across different scale types, revealing a higher proportion of careless respondents in VAS compared to Likert scale data. Second, we demonstrate that item parameters from the proposed model exhibit improved psychometric properties compared to those from a model that ignores careless responding. These findings underscore the model’s potential to enhance data quality by identifying and addressing careless responding.

• Ulitzsch, Esther; Viechtbauer, Wolfgang; Lüdtke, Oliver; Myin-Germeys, Inez; Nagy, Gabriel & Nestler, Steffen [Show all 7 contributors for this article] (2025). Investigating the Effect of Experience Sampling Study Design on Careless and Insufficient Effort Responding Identified With a Screen-Time-Based Mixture Model. Psychological Assessment. ISSN 1040-3590. 37(8), p. 347–359. doi: 10.1037/pas0001379. Full text in Research Archive Show summary

  When using the experience sampling method (ESM), researchers must navigate a delicate balance between obtaining fine-grained snapshots of phenomena of interest and avoiding undue respondent burden, which can lead to disengagement and compromise data quality. To guide that process, we investigated how questionnaire length and sampling frequency impact careless and insufficient effort responding (C/IER) as an important yet understudied aspect of ESM data quality. To this end, we made use of existing experimental ESM data (Eisele et al., 2022) from 163 students randomly assigned to one of two questionnaire lengths (30/60 items) and one of three sampling frequencies (3/6/9 assessments per day). We employed a novel mixture modeling approach (Ulitzsch, Nestler, et al., 2024) that leverages screen time data to disentangle attentive responding from C/IER and allows investigating how the occurrence of C/IER evolved within and across ESM study days. We found sampling frequency, but not questionnaire length, impacted C/IER, with higher frequencies resulting in higher overall C/IER proportions and sharper increases of C/IER across, but not within days. These effects proved robust across various model specifications. Further, we found no substantial relationships between model-implied C/IER and other engagement measures, such as self-reported attentiveness, attention checks, response-pattern-based attentiveness indicators, and compliance. Our findings contrast previous studies on noncompliance, suggesting that respondents may employ different strategies to lower the different types of burden imposed by questionnaire length and sampling frequency. Implications for designing ESM studies are discussed.

• Vogelsmeier, Leonie V. D. E.; Jongerling, Joran & Ulitzsch, Esther (2025). Accounting for Measurement Invariance Violations in Careless Responding Detection in Intensive Longitudinal Data: Exploratory vs. Partially Constrained Latent Markov Factor Analysis. Multivariate Behavioral Research. ISSN 0027-3171. 60(5), p. 878–897. doi: 10.1080/00273171.2025.2492016. Full text in Research Archive Show summary

  Intensive longitudinal data (ILD) collection methods like experience sampling methodology can place significant burdens on participants, potentially resulting in careless responding, such as random responding. Such behavior can undermine the validity of any inferences drawn from the data if not properly identified and addressed. Recently, a confirmatory mixture model (here referred to as fully constrained latent Markov factor analysis, LMFA) has been introduced as a promising solution to detect careless responding in ILD. However, this method relies on the key assumption of measurement invariance of the attentive responses, which is easily violated due to shifts in how participants interpret items. If the assumption is violated, the ability of the fully constrained LMFA to accurately identify careless responding is compromised. In this study, we evaluated two more flexible variants of LMFA—fully exploratory LMFA and partially constrained LMFA—to distinguish between careless and attentive responding in the presence of non-invariant attentive responses. Simulation results indicated that the fully exploratory LMFA model is an effective tool for reliably detecting and interpreting different types of careless responding while accounting for violations of measurement invariance. Conversely, the partially constrained model struggled to accurately detect careless responses. We end by discussing potential reasons for this.

• Ulitzsch, Esther; Belov, Dmitry; Lüdtke, Oliver & Robitzsch, Alexander (2025). Using Item Parameter Predictions for Reducing Calibration Sample Requirements—A Case Study Based on a High-Stakes Admission Test. Journal of Educational Measurement. ISSN 0022-0655. doi: 10.1111/jedm.12426. Full text in Research Archive Show summary

  In item difficulty modeling (IDM), item parameters are predicted from the items' linguistic features, aiming to ultimately render item calibration redundant. Current IDM applications, however, commonly do not yield the required prediction accuracy. To immediately exploit even somewhat inaccurate IDM predictions, we blend IDM with established Bayesian estimation techniques. We propose a two-step approach where (1) IDM predictions are obtained and (2) employed to construct informative prior distributions. We evaluate the approach in a case study on small-sample calibration of the 3PL in a high-stakes test. First, concerning implementation, we find computationally efficient penalized maximum likelihood estimation to be comparable to the best-performing MCMC-based approach. Second, we investigate sample size reductions achievable with state-of-the-art IDM predictions, finding negligible gains compared to merely considering the historical distribution of parameters. Third, we evaluate the prediction accuracy required for a targeted sample size reduction by gradually increasing simulated IDM prediction accuracies. We find that required accuracies can counterbalance each other, allowing calibration sample size to be reduced when either high-quality item difficulty predictions or good predictions of item discriminations and pseudo-guessing parameters are available. We argue that these evaluations provide new, portable IDM benchmarks quantifying performance in terms of achievable sample size reductions.

• Gilbert, Joshua B.; Zhang, Lijin; Ulitzsch, Esther & Domingue, Benjamin W. (2025). Polytomous explanatory item response models for item discrimination: Assessing negative-framing effects in social-emotional learning surveys. Behavior Research Methods. ISSN 1554-351X. 57(4). doi: 10.3758/s13428-025-02625-2. Full text in Research Archive Show summary

  Modeling item parameters as a function of item characteristics has a long history but has generally focused on models for item location. Explanatory item response models for item discrimination are available but rarely used. In this study, we extend existing approaches for modeling item discrimination from dichotomous to polytomous item responses. We illustrate our proposed approach with an application to four social-emotional learning surveys of preschool children to investigate how item discrimination depends on whether an item is positively or negatively framed. Negative framing predicts significantly lower item discrimination on two of the four surveys, and a plausibly causal estimate from a regression discontinuity analysis shows that negative framing reduces discrimination by about 30% on one survey. We conclude with a discussion of potential applications of explanatory models for item discrimination.

• Belov, Dmitry I.; Lüdtke, Oliver & Ulitzsch, Esther (2025). A supervised learning approach to estimating IRT models in small samples. British Journal of Mathematical & Statistical Psychology. ISSN 0007-1102. doi: 10.1111/bmsp.12396. Full text in Research Archive
• Alfers, Tobias; Gittler, Georg; Ulitzsch, Esther & Pohl, Steffi (2024). Assessing the Speed–Accuracy Tradeoff in Psychological Testing Using Experimental Manipulations. Educational and Psychological Measurement. ISSN 0013-1644. 85(2), p. 383–383. doi: 10.1177/00131644241271309. Full text in Research Archive Show summary

  The speed–accuracy tradeoff (SAT), where increased response speed often leads to decreased accuracy, is well established in experimental psychology. However, its implications for psychological assessments, especially in high-stakes settings, remain less understood. This study presents an experimental approach to investigate the SAT within a high-stakes spatial ability assessment. By manipulating instructions in a within-subjects design to induce speed variations in a large sample ( N = 1,305) of applicants for an air traffic controller training program, we demonstrate the feasibility of manipulating working speed. Our findings confirm the presence of the SAT for most participants, suggesting that traditional ability scores may not fully reflect performance in high-stakes assessments. Importantly, we observed individual differences in the SAT, challenging the assumption of uniform SAT functions across test takers. These results highlight the complexity of interpreting high-stakes assessment outcomes and the influence of test conditions on performance dynamics. This study offers a valuable addition to the methodological toolkit for assessing the intraindividual relationship between speed and accuracy in psychological testing (including SAT research), providing a controlled approach while acknowledging the need to address potential confounders. Future research may apply this method across various cognitive domains, populations, and testing contexts to deepen our understanding of the SAT’s broader implications for psychological measurement.

• Ulitzsch, Esther; Buchholz, Janine; Shin, Hyo Jeong; Bertling, Jonas & Lüdtke, Oliver (2024). Using a novel multiple-source indicator to investigate the effect of scale format on careless and insufficient effort responding in a large-scale survey experiment. Large-scale Assessments in Education. 12(1). doi: 10.1186/s40536-024-00205-y. Full text in Research Archive Show summary

  Common indicator-based approaches to identifying careless and insufficient effort responding (C/IER) in survey data scan response vectors or timing data for aberrances, such as patterns signaling straight lining, multivariate outliers, or signals that respondents rushed through the administered items. Each of these approaches is susceptible to unique types of misidentifications. We developed a C/IER indicator that requires agreement on C/IER identification from multiple behavioral sources, thereby alleviating the effect of each source’s standalone C/IER misidentifications and increasing the robustness of C/IER identification. To this end, we combined a response-pattern-based multiple-hurdle approach with a recently developed screen-time-based mixture decomposition approach. In an application of the proposed multiple-source indicator to PISA 2022 field trial data we (a) showcase how the indicator hedges against (presumed) C/IER overidentification of its constituting components, (b) replicate associations with commonly reported external correlates of C/IER, namely agreement with self-reported effort and C/IER position effects, and (c) employ the indicator to study the effects of changes of scale characteristics on C/IER occurrence. To this end, we leverage a large-scale survey experiment implemented in the PISA 2022 field trial and investigate the effects of using frequency instead of agreement scales as well as approximate instead of abstract frequency scale labels. We conclude that neither scale format manipulation has the potential to curb C/IER occurrence.

• Vogelsmeier, Leonie; Uglanova, Irina; Rein, Manuel & Ulitzsch, Esther (2024). Investigating dynamics in attentive and inattentive responding together with their contextual correlates using a novel mixture IRT model for intensive longitudinal data. British Journal of Mathematical & Statistical Psychology. ISSN 0007-1102. doi: 10.1111/bmsp.12373. Full text in Research Archive Show summary

  In ecological momentary assessment (EMA), respondents answer brief questionnaires about their current behaviours or experiences several times per day across multiple days. The frequent measurement enables a thorough grasp of the dynamics inherent in psychological constructs, but it also increases respondent burden. To lower this burden, respondents may engage in careless and insufficient effort responding (C/IER), leaving data contaminated with responses that do not reflect what researchers want to measure. We introduce a novel approach to investigating C/IER in EMA data. Our approach combines a confirmatory mixture item response theory model separating C/IER from attentive behaviour with latent Markov factor analysis. This enables gauging the occurrence of C/IER and studying transitions among states of different response behaviours including their contextual correlates. The approach can be implemented using R packages. An empirical application showcases the approach's efficacy in pinpointing C/IER instances and gaining insights into their underlying causes. We showcase that the approach identifies various C/IER response patterns but requires heterogeneous and negatively worded items to detect straightlining. In a simulation investigating robustness against unaccounted for changes in measurement models underlying attentive responses, the approach proved robust against heterogeneity in loading patterns but not against heterogeneity in factor structures. Extensions to accommodate the latter are discussed.

• Mutak, Augustin; Krause, Robert; Ulitzsch, Esther; Much, Sören; Ranger, Jochen & Pohl, Steffi (2024). Modeling the Intraindividual Relation of Ability and Speed within a Test. Journal of Educational Measurement. ISSN 0022-0655. 61(3), p. 378–407. doi: 10.1111/jedm.12391. Full text in Research Archive Show summary

  Understanding the intraindividual relation between an individual's speed and ability in testing scenarios is essential to assure a fair assessment. Different approaches exist for estimating this relationship, that either rely on specific study designs or on specific assumptions. This paper aims to add to the toolbox of approaches for estimating this relationship. We propose the intraindividual speed-ability-relation (ISAR) model, which relies on nonstationarity of speed and ability over the course of the test. The ISAR model explicitly models intraindividual change in ability and speed within a test and assesses the intraindividual relation of speed and ability by evaluating the relationship of both latent change variables. Model estimation is good, when there are interindividual differences in speed and ability changes in the data. In empirical data from PISA, we found that the intraindividual relationship between speed and ability is not universally negative for all individuals and varies across different competence domains and countries. We discuss possible explanations for this relationship.

View all works in NVA

• Ulitzsch, Esther; He, Qiwei & Pohl, Steffi (2024). Innovations in Exploring Sequential Process Data. Zeitschrift für Psychologie. ISSN 2190-8370. 232(2), p. 71–73. doi: 10.1027/2151-2604/a000560. Full text in Research Archive

View all works in NVA