Assessing the Reliability of the Five-in-20 Classroom Observation Tool

Abstract

Effective classroom management practices are crucial to fostering a positive learning environment, student achievement, and student social-emotional development, as well as teacher job satisfaction. The Five-in-20 Classroom Observation Tool (FCOT) was developed from the 21 evidence-based strategies identified by Simonsen and colleagues (2008) to assess and support teachers’ classroom management; however, its psychometric properties are unknown. The present study looks at the interobserver agreement of the FCOT, specifically at how consistent observers’ ratings are with each other and how consistent observers’ ratings are with a standard coded copy of the FCOT. Forty-four participants, using the FCOT, rated the same 20-minute video of classroom instruction. Their ratings were compared using intraclass correlation coefficient and alpha coefficient. Intraclass correlation coefficient yielded excellent agreement for average measures and moderate agreement for single measures for both strategy endorsement and quality rating. Cronbach’s alpha also suggested strong reliability for strategy and quality. The participants’ ratings were compared to the standard coded FCOT using Cohen’s kappa and percent agreement. The kappa value for strategy endorsement suggested substantial agreement, while the kappa value for quality rating suggested moderate agreement. Percent agreement for strategy endorsement was calculated at 85.37% and for quality rating at 60.24%. Implications and suggestions for future research are provided.

Date of Award	2023
Original language	American English
Awarding Institution	Eastern Illinois University
Supervisor	Margaret Floress (Supervisor)