Different teacher-level effectiveness estimates, different results : Inter-model concordance across six generalized value-added models (VAMs)

Journal article

Sloat, Edward, Amrein-Beardsley, Audrey and Holloway, J.. (2018). Different teacher-level effectiveness estimates, different results : Inter-model concordance across six generalized value-added models (VAMs). Educational Assessment, Evaluation and Accountability. 30(4), pp. 367-397. https://doi.org/10.1007/s11092-018-9283-7
AuthorsSloat, Edward, Amrein-Beardsley, Audrey and Holloway, J.
Abstract

In this study, researchers compared the concordance of teacher-level effectiveness ratings derived via six common generalized value-added model (VAM) approaches including a (1) student growth percentile (SGP) model, (2) value-added linear regression model (VALRM), (3) value-added hierarchical linear model (VAHLM), (4) simple difference (gain) score model, (5) rubric-based performance level (growth) model, and (6) simple criterion (percent passing) model. The study sample included fourth to sixth grade teachers employed in a large, suburban school district who taught the same sets of students, at the same time, and for whom a consistent set of achievement measures and background variables were available. Findings indicate that ratings significantly and substantively differed depending upon the methodological approach used. Findings, accordingly, bring into question the validity of the inferences based on such estimates, especially when high-stakes decisions are made about teachers as based on estimates measured via different, albeit popular methods across different school districts and states.

Keywordsteacher accountability; teacher effectiveness; teacher evaluation; teacher quality; validity; value-added models
Year2018
JournalEducational Assessment, Evaluation and Accountability
Journal citation30 (4), pp. 367-397
PublisherSpringer Netherlands
ISSN1874-8597
Digital Object Identifier (DOI)https://doi.org/10.1007/s11092-018-9283-7
Scopus EID2-s2.0-85050908415
Research or scholarlyResearch
Page range367-397
Publisher's version
License
All rights reserved
File Access Level
Controlled
Output statusPublished
Publication dates
Online28 Jul 2018
Publication process dates
Accepted13 Jul 2018
Deposited06 Sep 2021
Permalink -

https://acuresearchbank.acu.edu.au/item/8wv71/different-teacher-level-effectiveness-estimates-different-results-inter-model-concordance-across-six-generalized-value-added-models-vams

Log in to edit

Restricted files

Publisher's version

  • 78
    total views
  • 0
    total downloads
  • 0
    views this month
  • 0
    downloads this month
These values are for the period from 19th October 2020, when this repository was created.

Export as

Related outputs

Expertise : Keywords in teacher education
Gerrard, Jessica and Holloway, Jessica. (2023). Expertise : Keywords in teacher education Bloomsbury Academic. https://doi.org/10.5040/9781350238268
Belonging as flickering and in flux in academic work : A collective biography
Gravett, Karen, Ajjawi, Rola, Bearman, Margaret, Holloway, Jessica, Olson, Rebecca and Winstone, Naomi. (2023). Belonging as flickering and in flux in academic work : A collective biography. International Journal of Qualitative Studies in Education. pp. 1-16. https://doi.org/10.1080/09518398.2023.2258095
Knowledge-based resistance :The role of professional organisations in the struggle against statutory assessments in England
Santori, Diego and Holloway, Jessica. (2023). Knowledge-based resistance :The role of professional organisations in the struggle against statutory assessments in England. Pedagogy Culture and Society. pp. 1-20. https://doi.org/10.1080/14681366.2023.2254782
Technical agonism : Embracing democratic dissensus in the datafication of education
Holloway, Jessica, Lewis, Steven and Langman, Sarah. (2023). Technical agonism : Embracing democratic dissensus in the datafication of education. Learning, Media and Technology. 48(2), pp. 253-265. https://doi.org/10.1080/17439884.2022.2160987
Reviewing the evidence base : Attraction, pathways and retention : A focus on attracting candidates
Wyatt-Smith, Claire, Holloway, Jessica, Alexander, Colette, Harris, Lois and Marcy, Ariel. (2022). Reviewing the evidence base : Attraction, pathways and retention : A focus on attracting candidates Sydney, Australia: Association of Independent Schools New South Wales.
Anti-standardization and testing opt-out movements in education : Resistance, disputes and transformation
Campos Martinez, Javier, Falabella, Alejandra, Holloway, Jessica and Santori, Diego. (2022). Anti-standardization and testing opt-out movements in education : Resistance, disputes and transformation. Education Policy Analysis Archives. 30(132), pp. 1-14. https://doi.org/10.14507/epaa.30.7506
Teachers and educational policy : Markets, populism, and im/possibilities for resistance
Stacey, Meghan, Gavin, Mihajla, Gerrard, Jessica, Hogan, Anna and Holloway, Jessica. (2022). Teachers and educational policy : Markets, populism, and im/possibilities for resistance. Education Policy Analysis Archives. 30(93), pp. 1-15. https://doi.org/10.14507/EPAA.30.7407
Humanising feedback encounters : A qualitative study of relational literacies for teachers engaging in technology-enhanced feedback
Payne, Ameena L., Ajjawi, Rola and Holloway, Jessica. (2022). Humanising feedback encounters : A qualitative study of relational literacies for teachers engaging in technology-enhanced feedback. Assessment and Evaluation in Higher Education. 48(7), pp. 903-914. https://doi.org/10.1080/02602938.2022.2155610
Governing teachers through datafication : Physical–virtual hybridity and language interoperability in teacher accountability
Holloway, Jessica and Lewis, Steven. (2022). Governing teachers through datafication : Physical–virtual hybridity and language interoperability in teacher accountability. Big Data and Society. 9(2), pp. 1-14. https://doi.org/10.1177/20539517221137553
Emergent developments in the datafication and digitalization of education
Lewis, Steven, Holloway, Jessica and Lingard, Bob. (2022). Emergent developments in the datafication and digitalization of education. In In Rizvi, Fazal, Lingard, Bob and Rinne, Risto (Ed.). Reimagining globalization and education pp. 62-78 Routledge. https://doi.org/10.4324/9781003207528-5
Metrics, standards and alignment in teacher policy : Critiquing fundamentalism and imagining pluralism
Holloway, Jessica. (2021). Metrics, standards and alignment in teacher policy : Critiquing fundamentalism and imagining pluralism Springer Nature. https://doi.org/10.1007/978-981-33-4814-1
Metrics, standards and alignment in teacher policy: critiquing fundamentalism and imagining pluralism
Holloway, Jessica. (2021). Metrics, standards and alignment in teacher policy: critiquing fundamentalism and imagining pluralism Springer Nature. https://doi.org/10.1007%2F978-981-33-4814-1
Teachers and teaching : (Re) thinking professionalism, subjectivity and critical inquiry
Holloway, Jessica. (2021). Teachers and teaching : (Re) thinking professionalism, subjectivity and critical inquiry. Critical Studies in Education. 62(4), pp. 411-421. https://doi.org/10.1080/17508487.2021.1966065
Democracy and teachers : The im/possibilities for pluralisation in evidence-based practice
Holloway, Jessica and Larsen Hedegaard, Maria Louise. (2021). Democracy and teachers : The im/possibilities for pluralisation in evidence-based practice. Journal of Education Policy. 38(3), pp. 432-451. https://doi.org/10.1080/02680939.2021.2014571
Re-professionalizing teaching : The new professionalism in the United States
Brass, Jory and Holloway, Jessica. (2021). Re-professionalizing teaching : The new professionalism in the United States. Critical Studies in Education. 62(4), pp. 519-536. https://doi.org/10.1080/17508487.2019.1579743
Datafication and surveillance capitalism : The Texas Teacher Evaluation and Support System (T-TESS)
Holloway, Jessica and Lewis, Steven. (2021). Datafication and surveillance capitalism : The Texas Teacher Evaluation and Support System (T-TESS). In In Wyatt-Smith, Claire, Lingard, Bob and Heck, Elizabeth (Ed.). Digital disruption in teaching and testing : Assessments, big data, and the transformation of schooling pp. 181-194 Routledge, Taylor & Francis Group.
Aligning teacher preparation, professional development and evaluation : The orthodoxy of tap teachers and teaching
Holloway, Jessica. (2020). Aligning teacher preparation, professional development and evaluation : The orthodoxy of tap teachers and teaching. Studia Paedagogica. 25(2), pp. 117-137. https://doi.org/10.5817/SP2020-2-6
Distributing leadership within rural schools : Sharing responsibility for diverse students needs 
Uysal, Huseyin and Holloway, Jessica. (2020). Distributing leadership within rural schools : Sharing responsibility for diverse students needs . In In Slapac, Alina and Coppersmith, Sarah (Ed.). Beyond language learning instruction : Transformative supports for emergent bilinguals and educators pp. 126-145 IGI Global. https://doi.org/10.4018/978-1-7998-1962-2.ch006
Teacher accountability, datafication and evaluation : A case for reimagining schooling
Holloway, Jessica. (2020). Teacher accountability, datafication and evaluation : A case for reimagining schooling. Education Policy Analysis Archives. 28(56), pp. 1-12. https://doi.org/10.14507/epaa.28.5026
Using test scores to evaluate and hold school teachers accountable in New Mexico
Geiger, Tray J., Amrein-Beardsley, Audrey and Holloway, Jessica. (2020). Using test scores to evaluate and hold school teachers accountable in New Mexico. Educational Assessment, Evaluation and Accountability. 32(2), pp. 187-235. https://doi.org/10.1007/s11092-020-09324-w
School testing culture and teacher satisfaction
Smith, William C. and Holloway, Jessica. (2020). School testing culture and teacher satisfaction. Educational Assessment, Evaluation and Accountability. 32(4), pp. 461-479. https://doi.org/10.1007/s11092-020-09342-8
School autonomy, school accountability and social justice : Stories from two Australian school principals
Keddie, Amanda and Holloway, Jessica. (2020). School autonomy, school accountability and social justice : Stories from two Australian school principals. School Leadership and Management. 40(4), pp. 288-302. https://doi.org/10.1080/13632434.2019.1643309
Discourse analysis as theory, method, and epistemology in studies of education policy
Anderson, Kate T. and Holloway, Jessica. (2020). Discourse analysis as theory, method, and epistemology in studies of education policy. Journal of Education Policy. 35(2), pp. 188-221. https://doi.org/10.1080/02680939.2018.1552992
Competing locals in an autonomous schooling system : The fracturing of the ‘social’ in social justice
Holloway, Jessica and Keddie, Amanda. (2020). Competing locals in an autonomous schooling system : The fracturing of the ‘social’ in social justice. Educational Management, Administration & Leadership. 48(5), pp. 786-801. https://doi.org/10.1177/1741143219836681
The performative to the datafied teacher subject : Teacher evaluation in Tennessee
Holloway, Jessica and Goel La Londe, Priya. (2020). The performative to the datafied teacher subject : Teacher evaluation in Tennessee. In In Grek, Sotiria, Maroy, Christian and Verger, Antoni (Ed.). World yearbook of education 2021 : Accountability and datafication in the governance of education pp. 252-278 Routledge.
How is an inclusive agenda possible in an excluding education system? Revisiting the Danish Dilemma
Colling Larsen, Tobias, Holloway, Jessica and Hamre, Bjørn. (2019). How is an inclusive agenda possible in an excluding education system? Revisiting the Danish Dilemma. International Journal of Inclusive Education. 23(10), pp. 1049-1064. https://doi.org/10.1080/13603116.2019.1626497
Examining the influence of international large-scale assessments on national education policies
Fischman, Gustavo E., Topper, Amelia Marcetti, Silova, Iveta, Goebel, Janna and Holloway, Jessica L.. (2019). Examining the influence of international large-scale assessments on national education policies. Journal of Education Policy. 34(4), pp. 470-499. https://doi.org/10.1080/02680939.2018.1460493
‘Make money, get money’ : How two autonomous schools have commercialised their services
Holloway, Jessica and Keddie, Amanda. (2019). ‘Make money, get money’ : How two autonomous schools have commercialised their services. Discourse : Studies in the Cultural Politics of Education. 40(6), pp. 889-901. https://doi.org/10.1080/01596306.2018.1451305
Risky teachers : Mitigating risk through high-stakes teacher evaluation in the USA
Holloway, Jessica. (2019). Risky teachers : Mitigating risk through high-stakes teacher evaluation in the USA. Discourse : Studies in the Cultural Politics of Education. 40(3), pp. 399-411. https://doi.org/10.1080/01596306.2017.1322938
Value-added models for teacher evaluation and accountability : Commonsense assumptionsValue-added models for teacher evaluation and accountability : Commonsense assumptions
Amrein-Beardsley, Audrey and Holloway, Jessica. (2019). Value-added models for teacher evaluation and accountability : Commonsense assumptionsValue-added models for teacher evaluation and accountability : Commonsense assumptions. Educational Policy: an interdisciplinary journal of policy and practice. 33(3), pp. 516-542. https://doi.org/10.1177/0895904817719519
Teacher evaluation as an onto-epistemic framework
Holloway, Jessica. (2019). Teacher evaluation as an onto-epistemic framework. British Journal of Sociology of Education. 40(2), pp. 174-189. https://doi.org/10.1080/01425692.2018.1514291
The OECD's campaign for distributed leadership : The risks of pushing for more accountability and teacher responsibility
Holloway, Jessica. (2019). The OECD's campaign for distributed leadership : The risks of pushing for more accountability and teacher responsibility. In In Ydesen, Christian (Ed.). The OECD’s historical rise in education the formation of a global governing complex pp. 199-217 Springer International Publishing. https://doi.org/10.1007/978-3-030-33799-5_10
Teachers as leaders : Accountability era school leadership and the limits of democratic participation
Holloway, J., Kerr, Jessica and Zacharakis, Jeffrey. (2019). Teachers as leaders : Accountability era school leadership and the limits of democratic participation. In In McMahon, Brenda J. and Merriweather, Lisa R. (Ed.). Convictions of conscience : How voices from the margins inform public actions and educational leadership pp. 181-197 Information Age Publishing, Inc..
Prescribed distributed leadership in the era of accountability : The experiences of mentor teachers
Holloway, Jessica, Nielsen, Ann and Saltmarsh, Sarah. (2018). Prescribed distributed leadership in the era of accountability : The experiences of mentor teachers. Educational Management, Administration & Leadership. 46(4), pp. 538-555. https://doi.org/10.1177/1741143216688469
Making accountable teachers : The terrors and pleasures of performativity
Holloway, Jessica and Brass, Jory. (2018). Making accountable teachers : The terrors and pleasures of performativity. Journal of Education Policy. 33(3), pp. 361-382. https://doi.org/10.1080/02680939.2017.1372636
School reforms, market logic, and the politics of inclusion in the United States and Denmark
Holloway, Jessica and Hamre, Bjørn. (2018). School reforms, market logic, and the politics of inclusion in the United States and Denmark. In In Hamre, Bjørn, Morin, Anne and Ydesen, Christian (Ed.). Testing and inclusive schooling : International challenges and opportunities pp. 105-120 Routledge.
Global perspectives on high-stakes teacher accountability policies : An introduction
Holloway, Jessica, Sorensen, Tore Bernt and Verger, Antoni. (2017). Global perspectives on high-stakes teacher accountability policies : An introduction. Education Policy Analysis Archives. 25(85), pp. 1-13. https://doi.org/10.14507/epaa.25.3325
The murky waters of neoliberal marketization and commodification on the education of adults in the United States
Zacharakis, Jeff and Holloway, Jessica. (2016). The murky waters of neoliberal marketization and commodification on the education of adults in the United States. The European Journal for Research on the Education and Learning of Adults. 7(2), pp. 223-236. https://doi.org/10.3384/rela.2000-7426.rela9083
Validating “value added” in the primary grades: one district’s attempts to increase fairness and inclusivity in its teacher evaluation system
Amrein-Beardsley, Audrey, Polasky, Sarah and Holloway-Libell, Jessica. (2016). Validating “value added” in the primary grades: one district’s attempts to increase fairness and inclusivity in its teacher evaluation system. Educational Assessment, Evaluation and Accountability. 28(2), pp. 139-159. https://doi.org/10.1007/s11092-015-9234-5
What counts as good teaching? How a student growth percentile model has defined teacher quality at one urban middle school
Holloway, Jessica. (2016). What counts as good teaching? How a student growth percentile model has defined teacher quality at one urban middle school. In In Kappler Hewitt, Kimberly and Amrein-Beardsley, Audrey (Ed.). Student growth measures in policy and practice : Intended and unintended consequences of high-stakes teacher evaluations pp. 137-152 Palgrave Macmillan Ltd.. https://doi.org/10.1057/978-1-137-53901-4_7