Variability and generalizability of ratings for Quality of math classes between and within lessons

Measuring instructional quality with observer ratings has received great attention within the last years, resulting in first evidence on the variability between lessons. However, there is a lack of research on the extent to which observer ratings vary during a lesson. Furthermore, there are hardly any studies that compared the stability of generic and subject-specific characteristics of instructional quality. The study presented here departs from these desiderata and investigates into the stability of generic and subject-specific characteristics of instructional quality, both between and within lessons. We measured instructional quality in 37 classes during two lessons of mathematics instruction and four segments per lesson. Trained observers rated five characteristics of instructional quality, three of which represent the basic dimensions (classroom management, student support, cognitive activation) and two of which are subject-specific (subject-related and teaching-related mathematics educational quality). The results indicate acceptable reliabilities for four of the aforementioned five characteristics. In addition, variability differed relevantly between generic and subject-specific characteristics of instructional quality. We discuss our findings against various conceptualisations of instructional quality.