Full title
Rater Error as a Threat to Evaluation Conclusions: How well do standard methods address this threat?
Abstract
Rater-mediated assessments allow for the direct assessment of the impact of educational interventions on teaching practice, among other important uses. A risk of using rater-mediated assessments is that conclusions are dependent on the specific raters used in the study. This paper reviews the existing methods to address rater error from the perspective of evaluating the robustness of conclusions to the threat of rater error. I then propose a new method that more directly and explicitly explores the robustness of conclusions to the threat of rater error. The proposed method is based on a replication logic that views the use of multiple raters in a study from the perspective of a direct, within-study replication, using multiple imputation to fill in missing ratings that arise from not all raters scoring each case. I present a brief simulation showing multiple imputation works effectively in real data and a case study to demonstrate the logic of the proposed approach. The discussion includes ideas on how to expand the proposed approach (e.g., by including qualitative data on rater error obtained from training/calibration)
Full info
White, M. (2023, August 29). Rater Error as a Threat to Evaluation Conclusions: How well do standard methods address this threat?. https://doi.org/10.31219/osf.io/r7sza