Welcome to the QUINT Observation Systems Seminars (OBS seminars). This series will discuss classroom observation systems as a tool for understanding and improving teaching quality.
This and upcoming OBS seminars are open for all interested parties. We want it to become a meeting arena for scholars genuinely interested in observation systems and related issues. Therefore we recommend that you join our network by subscribing to the network mailing list. You must confirm your email address in the confirmation email you receive to complete signing up to the mailing list.
If you are interested in presenting your research or have questions, please contact the organizer, QUINT Postdoctoral Fellow Mark White.
Program
In this seminar, taking place on May 2, 5:00 PM–6:00 PM CET, Felipe Martinez and Drew Gitomer will be presenting: Taking Stock of Observational Measures of Teaching.
Abstract:
The last 20 years have seen unprecedented interest around the world in developing and extending use of standardized performance measures of teaching based on classroom observation, portfolios and artifacts. Yet, after numerous efforts, including some extraordinarily well-funded and supported, the ability to reliably and validly differentiate teacher quality remains elusive. The quest to measure stable traits attributed to a teacher appears to have inherent limitations in the case of a complex, multidimensional, and highly contextualized performance trait like instruction. Moreover, the evidence available offers limited guidance on how to substantially improve the psychometric quality of measures of instruction at scale, whether through better data collection, lesson sampling, rubric design, rater training, or analytic methods. In this talk, we discuss the implications of these sobering findings by considering a number of related issues and possible directions for research and practice. Specific issues we will raise include:
- Conceptualizations of rater and occasion error in measures of instruction. Inconsistencies in instruction across lessons/occasions as legitimate and even desirable, and inconsistencies across raters as having a legitimate basis as well;
- Conceptual and estimation challenges using standard latent variable modeling techniques to examine factorial structures;
- Implications for traditional predictive or concurrent validity coefficients, which can be typically quite modest.
- Alternative scenarios for uses of observation protocols in strictly formative contexts, with frameworks and protocols as the basis for developing shared understandings of teaching. This requires a different, hermeneutic approach to validity (Moss, 2004) focused on utility to support the development of teaching.
