QUINT Observation Systems Seminars

Seminar on Standardized Observation Systems as a tool for measuring and understanding teaching.


The goal of this seminar is to promote sharing and communication between researchers who work with Observation Systems (Bell et al., 2018) based on the need for greater communication in this area as described by Praetorius and Charalambous (2018, 2021).  The seminar is organized around a very general empirically-grounded framework (Figure 1): Theory should inform Measurement, which generates Data, which informs Theory.  Then, developments in any one area has important implications for each other area. Observation systems are important in that they exist at the intersection of theory and measurement.

The seminar invites researchers to present research that relates Observation Systems broadly.  See Bell et al, 2018 for our conceptualization of observation systems.  Presentations may include: Presentations of empirical results; discussions of instructional theories (as relevant to observation systems; e.g. dimensions/domains to be measured); discussions of measurement/implementation challenges when using observation systems.   Of particular interest, however, will be presentations that address questions spanning the framework (Figure 1): What theoretical implications can we draw from empirical results and what does this imply about measurement approaches? Given a specific theoretical view on teaching quality for example, what measurement implications can we draw and how does this lead to better data on this area? How can alternate measurement approaches improve the quality of the data and thus analyses generated?

Figures of Theory, Measurement, and Data leading from one to the next
Figure 1: Framework for Seminar


Next Seminar (April 13th)

Title: From global teacher observation protocols to fine-grained, automated methods: Strengths and emerging limitations

Speaker: Sean Kelly, Department of Educational Foundations, Organizations, & Policy, School of Education, University of Pittsburgh, Co-Editor, American Educational Research Journal

Abstract:  An essential feature of many modern teacher observation protocols is their “global” approach to measuring instruction, where trained observers provide a summary evaluation of multiple domains of instruction scored over an interval of time or even entire class session. In this talk, I highlight several limitations of global protocols that have emerged, referencing findings from the Measures of Effective Teaching data reported by Kelly et al. (2020; Educational Policy Analysis Archives).  These limitations include: limited discriminatory power to categorize instruction, reliability that is highly contingent on rater training, highly correlated subdomains that make it difficult to distinguish specific practices, the confounding of classroom context with teachers’ own contribution to instruction, and evaluation solely on a continuum of effectiveness rather than measures focused on tradeoffs in time-use or emphasis.  As a frame of reference for understanding global protocols, I also consider the properties of automated systems of observation currently being developed, which address some of these limitations but also introduce new challenges.

Upcoming Seminars

  • May 4th: Julie Cohen and Ethan Hutt will discuss their paper The Change we Cannot See
  • June 9th: Charalambos Charalambous and Anna Praetorius will present work focused on fostering collaborative work around Observation Systems
  • July and August will be taken off for Summer Break
  • September Slot is currently Open
  • October 6th: Courtney Bell will present


Participation is open and widely encouraged.  Those interested in presenting should contact Mark White (mark.white@ils.uio.no).  Preference for presentations will be given to those who routinely and actively participate in the seminars.  Presentations should be no more than 30 minutes and should conclude with some shared points for discussion.  Discussion will then proceed for 30 minutes.  All participants will be muted during the presentation. Clarifying questions can be asked through the chat and a moderator will decide whether to interrupt the speaker to get clarification on the point.  The moderator will be in charge of muting/unmuting participants during the discussion.

Mailing List

To stay up-to-date about upcoming presentations and receive Zoom invitations feel free to join our network and subscribe to the network mailing list using your institutional email address where possible. For security reasons, we cannot accept subcriptions submitted from free email accounts. Please contact the organisers if you have questions. You must confirm your email address in the confirmation email you receive to complete signing up to the mailing list.  If you are already subscribed to our mailing list and would like to unsubscribe, you may unsubcribe here.


These seminars are organized by the QUINT Nordic Center of Excellence in an effort to encourage joint learning about the affordances and limitations of Observation Systems and with the goal of allowing frequent communication by scholars engaged with these tools.  Mark White is responsible for organizing seminars. Feedback and suggestions can be directed to him at mark.white@ils.uio.no. 


Bell, C. A., Dobbelaer, M. J., Klette, K., & Visscher, A. (2018). Qualities of classroom observation systems. School Effectiveness and School Improvement, 0(0), 1–27. https://doi.org/10.1080/09243453.2018.1539014

Praetorius, A. K., & Charalambous, C. Y. (2018). Classroom observation frameworks for studying instructional quality: Looking back and looking forward. ZDM, 50(3), 535–553. https://doi.org/10.1007/s11858-018-0946-0

Published Jan. 26, 2021 3:57 PM - Last modified Mar. 17, 2021 10:08 AM