Using pairwise comparison and ordered exemplars as a basis for a novel method of standard setting in narrative writing

Journal article

Humphry, Stephen, Adie, Lenore, Maxwell, Carolyn and Sappl, Sonia. (2023). Using pairwise comparison and ordered exemplars as a basis for a novel method of standard setting in narrative writing. Frontiers in Education. 8, p. Article 1183695. https://doi.org/10.3389/feduc.2023.1183695
AuthorsHumphry, Stephen, Adie, Lenore, Maxwell, Carolyn and Sappl, Sonia
Abstract

Standard setting of extended performances is fraught with difficulties due to the subjective nature of the scoring decision. This article introduces and reports results of a novel systematic method for setting standards for assessments that require extended performances: the Scaled Exemplar Standard Setting method (SESS). This two-stage method brings together (1) pairwise comparison of samples to scale performances and (2) standard setting involving multiple standards with tasks presented to judges in a validated order. By utilizing paired comparison methodology in the first stage, the judgment is made more manageable by chunking it into a series of pairs. This method produces scaled performances that can be checked for internal consistency of the judgments. The validated, ordered performances are used in the standard setting stage, making the task for judges as transparent and straightforward as possible. Recommendations and implications for standard setting are discussed in light of the results.

Keywordsassessment; teacher judgment; standard setting; pairwise comparison; extended performance; scaled exemplars; reliability
Year2023
JournalFrontiers in Education
Journal citation8, p. Article 1183695
PublisherFrontiers Media S.A.
ISSN2504-284X
Digital Object Identifier (DOI)https://doi.org/10.3389/feduc.2023.1183695
Scopus EID2-s2.0-85168296926
Open accessPublished as ‘gold’ (paid) open access
Page range1-13
FunderAustralian Research Council (ARC)
Publisher's version
License
File Access Level
Open
Output statusPublished
Publication dates
Online02 Aug 2023
Publication process dates
Accepted20 Jul 2023
Deposited07 Nov 2023
ARC Funded ResearchThis output has been funded, wholly or partially, under the Australian Research Council Act 2001
Grant IDLP180100046
Permalink -

https://acuresearchbank.acu.edu.au/item/8zy69/using-pairwise-comparison-and-ordered-exemplars-as-a-basis-for-a-novel-method-of-standard-setting-in-narrative-writing

Log in to edit

Download files

Publisher's version
OA_Humphry_2023_Using_pairwise_comparison_and_ordered_exemplars.pdf
License: CC BY 4.0
File access level: Open

  • 77
    total views
  • 41
    total downloads
  • 0
    views this month
  • 0
    downloads this month
These values are for the period from 19th October 2020, when this repository was created.

Export as

Related outputs

Evidence of teacher assessment work and its relationship to their assessment identity
Adie, Lenore Ellen, Wyatt-Smith, Claire Maree, Finch, Mary and DeLuca, Christopher. (2024). Evidence of teacher assessment work and its relationship to their assessment identity. Teaching and Teacher Education. 141, pp. 1-10. https://doi.org/10.1016/j.tate.2024.104518
Mediating teachers’ assessment work
Adie, Lenore Ellen, Gallagher, Jeanine, Wyatt-Smith, Claire Maree, Spina, Nerida and DeLuca, Christopher. (2023). Mediating teachers’ assessment work. The Australian Educational Researcher. pp. 1-21. https://doi.org/10.1007/s13384-023-00675-z
Students drawing conclusions about assessment to inform school change
Willis, Jill, Adie, Lenore, Addison, Bruce and Kimber, Megan. (2023). Students drawing conclusions about assessment to inform school change. Cambridge Journal of Education. pp. 1-23. https://doi.org/10.1080/0305764X.2023.2228224
Professionalizing teacher education : Performance Assessment, Standards, Moderation, and Evidence
Wyatt-Smith, Claire Maree, Adie, Lenore Ellen, Haynes, Michele and Day, Chantelle. (2022). Professionalizing teacher education : Performance Assessment, Standards, Moderation, and Evidence Routledge. https://doi.org/10.4324/9780429318504
Learning progression–based assessments : A systematic review of student and teacher uses
Harris, Lois Ruth, Adie, Lenore and Wyatt-Smith, Claire. (2022). Learning progression–based assessments : A systematic review of student and teacher uses. Review of Educational Research. 92(6), pp. 996-1040. https://doi.org/10.3102/00346543221081552
The conceptualisation of a teaching performance assessment : Designing for evidence of graduate competence
Adie, Lenore and Wyatt-Smith, Claire. (2021). The conceptualisation of a teaching performance assessment : Designing for evidence of graduate competence. In In Wyatt-Smith, Claire, Adie, Lenore and Nuttall, Joce (Ed.). Teaching performance assessments as a cultural disruptor in initial teacher education : Standards, evidence and collaboration pp. 19-37 Springer Nature. https://doi.org/10.1007/978-981-16-3705-6_2
Introducing a new model for online cross-institutional moderations
Wyatt-Smith, Claire and Adie, Lenore. (2021). Introducing a new model for online cross-institutional moderations. In In Wyatt-Smith, Claire, Adie, Lenore and Nuttall, Joce (Ed.). Teaching performance assessments as a cultural disruptor in initial teacher education : Standards, evidence and collaboration pp. 39-58 Springer Nature. https://doi.org/10.1007/978-981-16-3705-6_3
Disrupting teacher education for sustainable change
Wyatt-Smith, Claire and Adie, Lenore. (2021). Disrupting teacher education for sustainable change. In In Wyatt-Smith, Claire, Adie, Lenore and Nuttall, Joce (Ed.). Teaching performance assessments as a cultural disruptor in initial teacher education : Standards, evidence and collaboration pp. 217-229 Springer Nature. https://doi.org/10.1007/978-981-16-3705-6_17
Provocation 5 : COVID triggered disruption in teacher education and resultant actions
Wyatt-Smith, Claire, Day, Chantelle and Adie, Lenore. (2021). Provocation 5 : COVID triggered disruption in teacher education and resultant actions. In In Wyatt-Smith, Claire, Adie, Lenore and Nuttall, Joce (Ed.). Teaching performance assessments as a cultural disruptor in initial teacher education : Standards, evidence and collaboration pp. 185-192 Springer Nature. https://doi.org/10.1007/978-981-16-3705-6_14
Professional standards, evidence and collaboration : Cultural disruption in teacher education
Wyatt-Smith, Claire and Adie, Lenore. (2021). Professional standards, evidence and collaboration : Cultural disruption in teacher education. In In Wyatt-Smith, Claire, Adie, Lenore and Nuttall, Joce (Ed.). Teaching performance assessments as a cultural disruptor in initial teacher education : Standards, evidence and collaboration pp. 3-18 Springer Nature. https://doi.org/10.1007/978-981-16-3705-6_1
The application of pairwise comparisons to form scaled exemplars as a basis for setting and exemplifying standards in teacher education
Wyatt-Smith, Claire, Humphrey, Stephen, Adie, Lenore and Colbert, Peta. (2020). The application of pairwise comparisons to form scaled exemplars as a basis for setting and exemplifying standards in teacher education. Assessment in Education. 27(1), pp. 65-86. https://doi.org/10.1080/0969594X.2020.1712326
Examining research into the use of data walls for teaching and learning : How are they being implemented within data use cycles?
Adie, Lenore, Harris, Lois and Wyatt-Smith, Claire. (2020). Examining research into the use of data walls for teaching and learning : How are they being implemented within data use cycles? Teaching and Teacher Education. 89, p. Article: 103012. https://doi.org/10.1016/j.tate.2019.103012
Using data walls to display assessment results : A review of their effective impacts on teachers and students
Harris, Lois, Wyatt-Smith, Claire and Adie, Lenore. (2020). Using data walls to display assessment results : A review of their effective impacts on teachers and students. Teachers and Teaching: Theory and Practice. 26(1), pp. 50-66. https://doi.org/10.1080/13540602.2020.1739018
The rubric as a moderating tool in tertiary contexts
Adie, Lenore. (2020). The rubric as a moderating tool in tertiary contexts. In In Grainger, Peter and Weir, Katie (Ed.). Facilitating student learning and engagements in higher education through assessment rubrics pp. 87-96 Cambridge Scholars Publishing.
Synergy and tension between large-scale and classroom assessment: International trends
Louis Volante, Christopher DeLuca, Lenore Adie, Eva Baker, Heidi Harju-Luukkainen, Margaret Heritage, Christoph Schneider, Gordon Stobart, Kelvin Tan and Claire Wyatt-Smith. (2020). Synergy and tension between large-scale and classroom assessment: International trends. Educational Measurement: Issues and Practice. 39(4), pp. 21-29. https://doi.org/10.1111/emip.12382
Contesting educational assessment policies in Australia
Cumming, Joy, van der Kleij, Fabienne Schoenmaker and Adie, Lenore. (2019). Contesting educational assessment policies in Australia. Journal of Education Policy. 34(6), pp. 836 - 857. https://doi.org/10.1080/02680939.2019.1608375
Fidelity of summative performance assessment in initial teacher education: The intersection of standardisation and authenticity
Adie, Lenore and Wyatt-Smith, Claire. (2019). Fidelity of summative performance assessment in initial teacher education: The intersection of standardisation and authenticity. Asia-Pacific Journal of Teacher Education. https://doi.org/10.1080/1359866X.2019.1606892
Feasibility and value of using a GoPro camera and Apple iPad to study teacher-student assessment feedback interactions
van der Kleij, Fabienne Schoenmaker, Adie, Lenore and Cumming, Joy. (2019). Feasibility and value of using a GoPro camera and Apple iPad to study teacher-student assessment feedback interactions. In In B. P. Veldkamp and C. Sluijter (Ed.). Theoretical and Practical Advances in Computer-based Educational Measurement pp. 339 - 359 SpringerOpen. https://doi.org/10.1007/978-3-030-18480-3_18
A meta-review of the student role in feedback
van der Kleij, Fabienne Schoenmaker, Adie, Lenore and Cumming, Joy. (2019). A meta-review of the student role in feedback. International Journal of Educational Research. 98, pp. 303 - 323. https://doi.org/10.1016/j.ijer.2019.09.005
Diverse perspectives on student agency in classroom assessment
Adie, Lenore, Willis, Jill and van der Kelij, Fabienne S.. (2018). Diverse perspectives on student agency in classroom assessment. The Australian Educational Researcher. 45(1), pp. 1 - 12. https://doi.org/10.1007/s13384-018-0262-2
Research-informed conceptualization and design principles of teacher performance assessments: wrestling with system and site validity
Adie, Lenore and Wyatt-Smith, Claire Maree. (2018). Research-informed conceptualization and design principles of teacher performance assessments: wrestling with system and site validity. In In C. Wyatt-Smith and L. Adie (Ed.). Innovation and Accountability in Teacher Education: Setting directions for new cultures in teacher education pp. 115 - 132 Springer Nature. https://doi.org/10.1007/978-981-13-2026-2
Formative assessment and feedback using information technology
van der Kleij, Fabienne S. and Adie, Lenore. (2018). Formative assessment and feedback using information technology. In In J. Voogt, G. Knezek and R. Christensen and K. Lai (Ed.). Second Handbook of Information Technology in Primary and Secondary pp. 601 - 615 Springer International Publishing. https://doi.org/10.1007/978-3-319-71054-9
The development and application of coding frameworks to explore dialogic feedback interactions and self-regulated learning
Adie, Lenore, Van der Kleij, Fabienne Michelle and Cumming, Joy Joy. (2018). The development and application of coding frameworks to explore dialogic feedback interactions and self-regulated learning. British Educational Research Journal. 44(4), pp. 704 - 723. https://doi.org/10.1002/berj.3463
New cultures in teacher education
Wyatt-Smith, Claire Maree and Adie, Lenore. (2018). New cultures in teacher education. In In C. Wyatt-Smith and L. Adie (Ed.). Innovation and Accountability in Teacher Education: Setting directions for new cultures in teacher education pp. 1 - 15 Springer Nature. https://doi.org/10.1007/978-981-13-2026-2
2018 Queensland NAPLAN review: School and system perceptions report and literature review
Cumming, Joy, Jackson, Christine, Day, Chantelle, Maxwell, Graham, Adie, Lenore, Lingard, Bob, Haynes, Michele and Heck, Elizabeth. (2018). 2018 Queensland NAPLAN review: School and system perceptions report and literature review Australia: Institute for Learning Sciences and Teacher Education, Australian Catholic University.
Validity and participation: implications for school comparison of Australia's national assessment program
Thompson, Greg, Adie, Lenore and Klenowski, Val. (2018). Validity and participation: implications for school comparison of Australia's national assessment program. Journal of Education Policy. 33(6), pp. 759 - 777. https://doi.org/10.1080/02680939.2017.1373407
Assessment: the trilogy of standards, evidence and judgement in australian education reform
Wyatt-Smith, Claire Maree and Adie, Lenore. (2018). Assessment: the trilogy of standards, evidence and judgement in australian education reform. In In A. Reid and D. Price (Ed.). The Australian Curriculum: Promises, problems and possibilities pp. 163 - 176 Australian Curriculum Studies Association.
Assessment moderation in an Australian context: processes, practices, and challenges
Beutel, Denise, Adie, Lenore and Lloyd, Margaret. (2017). Assessment moderation in an Australian context: processes, practices, and challenges. Teaching in Higher Education. 22(1), pp. 1 - 14. https://doi.org/10.1080/13562517.2016.1213232
Using video technology to enable student voice in assessment feedback
van der Kleij, Fabienne S., Adie, Lenore and Cumming, Joy. (2017). Using video technology to enable student voice in assessment feedback. British Journal of Educational Technology. 48(5), pp. 1092 - 1105. https://doi.org/10.1111/bjet.12536
Assessment
Wyatt-Smith, Claire Maree, Adie, Lenore, Van der Kleij, Fabienne Michelle and Cumming, Joy. (2017). Assessment. In In D. Pendergast, K. Main and N. Bahr (Ed.). Teaching Middle Years: Rethinking curriculum, pedagogy and assessment pp. 301 - 320 Allen & Unwin.
Teacher educators' critical reflection on becoming and belonging to a community of practice
Adie, Lenore, Mergler, Amanda, Alford, Jennifer, Chandra, Vinesh and Hepple, Erika. (2017). Teacher educators' critical reflection on becoming and belonging to a community of practice. In In J. McDonald and A. Cater-Steel (Ed.). Communities of Practice: Facilitating Social Learning in Higher Education pp. 403 - 422 Springer.
Making meaning of assessment policy in Australia through teacher assessment conversations
Adie, Lenore Ellen and Willis, Jill. (2016). Making meaning of assessment policy in Australia through teacher assessment conversations. In In D. Laveault and L. Allal (Ed.). Assessment for learning: Meeting the challenge of implementation pp. 35 - 53 Springer International Publishing. https://doi.org/10.1007/978-3-319-39211-0_3
Quality assurance of assessment and moderation discourses involving sessional staff
Grainger, Peter, Adie, Lenore and Weir, Katie. (2016). Quality assurance of assessment and moderation discourses involving sessional staff. Assessment and Evaluation in Higher Education. 41(4), pp. 548 - 559. https://doi.org/10.1080/02602938.2015.1030333
What's the point of moderation? A discussion of the purposes achieved through contemporary moderation practices
Bloxham, Sue, Hughes, Clair and Adie, Lenore Ellen. (2016). What's the point of moderation? A discussion of the purposes achieved through contemporary moderation practices. Assessment and Evaluation in Higher Education. 41(4), pp. 638 - 653. https://doi.org/10.1080/02602938.2015.1039932
The prospects for assessment for learning in higher education in Asian settings: The case of Vietnam
Ho, Nhat, Adie, Lenore and Klenowski, Val. (2016). The prospects for assessment for learning in higher education in Asian settings: The case of Vietnam. Assessment Matters. 10, pp. 5 - 26. https://doi.org/10.18296/am.0015
Schooling elsewhere: Rurality, inclusion and education
Pini, Barbara, Carrington, Suzanne and Adie, Lenore Ellen. (2015). Schooling elsewhere: Rurality, inclusion and education. International Journal of Inclusive Education. 19(7), pp. 677 - 684. https://doi.org/10.1080/13603116.2014.964489
The use of multimodal technologies to enhance reflective writing in teacher education
Adie, Lenore and Tangen, Donna. (2015). The use of multimodal technologies to enhance reflective writing in teacher education. In In M. E. Ryan (Ed.). Teaching reflective learning in higher education: A systematic approach using pedagogic patterns pp. 127 - 138 Springer. https://doi.org/10.1007/978-3-319-09271-3_9
International representations of inclusive education : How is inclusive practice reflected in the Professional Teaching Standards of China and Australia?
Carrington, Suzanne, Saggers, Beth, Adie, Lenore, Zhu, Nan, Gu, Dingqian, Hu, Xiaoyi, Wang, Yan, Deng, Meng and Mu, Guanglun. (2015). International representations of inclusive education : How is inclusive practice reflected in the Professional Teaching Standards of China and Australia? International Journal of Disability, Development and Education. 62(6), pp. 556 - 570. https://doi.org/10.1080/1034912X.2015.1077933
The School-Community Integrated Learning pathway : Exploring a new way to prepare and induct final-year preservice teachers
Hudson, Suzanne, Hudson, Peter and Adie, Lenore. (2015). The School-Community Integrated Learning pathway : Exploring a new way to prepare and induct final-year preservice teachers. Improving Schools (print). 18(3), pp. 221 - 235. https://doi.org/10.1177/1365480215596223
Analysis of moderation practices in a large STEM-focused faculty
Czaplinski, Iwona, Senadji, Bouchra, Adie, Lenore and Beutel, Denise. (2014). Analysis of moderation practices in a large STEM-focused faculty. 2014 International Conference on Teaching, Assessment and Learning (TALE). https://doi.org/10.1109/TALE.2014.7062561
Towards a theoretical framework for online professional discussions
Adie, Lenore. (2014). Towards a theoretical framework for online professional discussions. Journal of Learning Design. 7(3), pp. 54 - 66.
Teachers using annotations to engage students in assessment conversations : Recontextualising knowledge
Willis, Jill and Adie, Lenore. (2014). Teachers using annotations to engage students in assessment conversations : Recontextualising knowledge. Curriculum Journal. 25(4), pp. 495 - 515. https://doi.org/10.1080/09585176.2014.968599
Moderation in higher education: Four discourses
Beutel, Denise A., Adie, Lenore E. and Lloyd, Margaret M.. (2014). Moderation in higher education: Four discourses. Research and Development in Higher Education: Higher Education in a Globalized World. Hong Kong: Hong Kong Baptist University.
Using annotations to inform an understanding of achievement standards
Adie, Lenore and Willis, Jill. (2014). Using annotations to inform an understanding of achievement standards. Assessment Matters. 6, pp. 112-136.
How do preservice teacher education students move from novice to expert assessors?
Grainger, Peter and Adie, Lenore. (2014). How do preservice teacher education students move from novice to expert assessors? Australian Journal of Teacher Education (online version). 39(7), pp. 89 - 105. https://doi.org/10.14221/ajte.2014v39n7.9
The development of shared understandings of assessment policy : Travelling between global and local contexts
Adie, Lenore. (2014). The development of shared understandings of assessment policy : Travelling between global and local contexts. Journal of Education Policy. 29(4), pp. 532 - 545. https://doi.org/10.1080/02680939.2013.853101
Moderation practices in a faculty of education : The four discourse model
Adie, Lenore E.. (2013). Moderation practices in a faculty of education : The four discourse model. Brisbane, Australia: Queensland University of Technology. pp. 1-6
An investigation of moderation practices within a faculty of education
Beutel, Denise, Adie, Lenore and Lloyd, Margaret M.. (2013). An investigation of moderation practices within a faculty of education. ICEASS International Conference on Education and Social Sciences. Singapore: pp. 43 - 49
Negotiating the meaning of achievement standards in the Australian curriculum
Willis, Jill and Adie, Lenore. (2013). Negotiating the meaning of achievement standards in the Australian curriculum. Curriculum Perspectives (Journal Edition). 33(1), pp. 52 - 62.
Enhancing teacher education: the School-Community Integrated Learning (SCIL) pathway
Hudson, Sue, Hudson, Peter and Adie, Lenore. (2013). Enhancing teacher education: the School-Community Integrated Learning (SCIL) pathway. Annual Australian Teacher Educators Association Conference : Knowledge Makers and Notice Takers : Teacher Education Research Impacting Policy and Practice. Brisbane, Australia:
Conceptualising teachers' assessment literacies in an era of curriculum and assessment reform
Willis, Jill, Adie, Lenore and Klenowski, Val. (2013). Conceptualising teachers' assessment literacies in an era of curriculum and assessment reform. The Australian Educational Researcher (print). 40(2), pp. 241 - 256. https://doi.org/10.1007/s13384-013-0089-9
The development of teacher assessment identity through participation in online moderation
Adie, Lenore. (2013). The development of teacher assessment identity through participation in online moderation. Assessment in Education: Principles, Policy & Practice. 20(1), pp. 91 - 106. https://doi.org/10.1080/0969594X.2011.650150
Identifying discourses of moderation in higher education
Adie, Lenore, Lloyd, Margaret and Beutel, Denise. (2013). Identifying discourses of moderation in higher education. Assessment and Evaluation in Higher Education. 38(8), pp. 968 - 977. https://doi.org/10.1080/02602938.2013.769200
Towards an understanding of teacher judgement in the context of social moderation
Adie, Lenore, Klenowski, V and Wyatt-Smith, Claire Maree. (2012). Towards an understanding of teacher judgement in the context of social moderation. Educational Review. 64(2), pp. 223 - 240. https://doi.org/10.1080/00131911.2011.598919
Urban pre-service teachers' conceptions of teaching in rural communities
Adie, Lenore and Barton, Georgina. (2012). Urban pre-service teachers' conceptions of teaching in rural communities. Australian Journal of Teacher Education. 37(6), pp. 111 - 123. https://doi.org/10.14221/ajte.2012v37n6.7
Learning as identity and practice through involvement in online moderation
Adie, Lenore. (2012). Learning as identity and practice through involvement in online moderation. Educational Assessment, Evaluation and Accountability (print). 24(1), pp. 43 - 56. https://doi.org/10.1007/s11092-011-9132-4
Promoting rural and remote teacher education in Australia through the Over The Hill project
Beutel, Denise, Adie, Lenore and Hudson, Suzanne. (2011). Promoting rural and remote teacher education in Australia through the Over The Hill project. The International Journal of Learning. 18(2), pp. 377 - 388.
An investigation into online moderation
Adie, Lenore. (2011). An investigation into online moderation. Assessment Matters. 3, pp. 5 - 27.
Incorporating authentic assessment into different university learning scenarios
Adie, Lenore, Wharton, Lee and Hee, Lisa. (2010). Incorporating authentic assessment into different university learning scenarios. Tunku Abdul Rahman College International Conference on Learning and Teaching. Malaysia: Tunku Abdul Rahman College. pp. 82 - 90
Developing consistency in teacher judgement formation through online meeting centres
Adie, Lenore. (2010). Developing consistency in teacher judgement formation through online meeting centres. Journal of Learning Design. 3(2), pp. 14 - 26. https://doi.org/10.5204/jld.v3i2.47
Moderation as judgement practice : Reconciling system level accountability and local level practice
Klenowski, Val and Adie, Lenore. (2009). Moderation as judgement practice : Reconciling system level accountability and local level practice. Curriculum Perspectives (Journal Edition). 29(1), pp. 10 - 28.
Changing assessment practices : the case for online moderation
Adie, Lenore. (2009). Changing assessment practices : the case for online moderation. Australian Association for Reseach in EducatIon. Australia: Australian Association for Reseach in Education (AARE). pp. 1 - 14
The hegemonic positioning of 'Smart State' policy
Adie, Lenore. (2008). The hegemonic positioning of 'Smart State' policy. Journal of Education Policy. 23(3), pp. 251 - 264. https://doi.org/10.1080/02680930801923831
Moderation as judgement practice: Reconciling system level accountability and local level practice
Klenowski, Valentina, Adie, Lenore, Gunn, Stephanie, Looney, Anne, Elwood, Jannette, Wyatt-Smith, Claire Maree and Colbert, Peta Jane. (2007). Moderation as judgement practice: Reconciling system level accountability and local level practice. In P L Jeffery (Ed.). AARE 2007 International Educational Research Conference Papers. Australia: Australian Association for Research in Education. pp. 1 - 29