Loading...
Using pairwise comparison and ordered exemplars as a basis for a novel method of standard setting in narrative writing
Humphry, Stephen Adie, Lenore Maxwell, Carolyn Sappl, Sonia
Humphry, Stephen
Adie, Lenore
Maxwell, Carolyn
Sappl, Sonia
Abstract
Standard setting of extended performances is fraught with difficulties due to the subjective nature of the scoring decision. This article introduces and reports results of a novel systematic method for setting standards for assessments that require extended performances: the Scaled Exemplar Standard Setting method (SESS). This two-stage method brings together (1) pairwise comparison of samples to scale performances and (2) standard setting involving multiple standards with tasks presented to judges in a validated order. By utilizing paired comparison methodology in the first stage, the judgment is made more manageable by chunking it into a series of pairs. This method produces scaled performances that can be checked for internal consistency of the judgments. The validated, ordered performances are used in the standard setting stage, making the task for judges as transparent and straightforward as possible. Recommendations and implications for standard setting are discussed in light of the results.
Keywords
assessment, teacher judgment, standard setting, pairwise comparison, extended performance, scaled exemplars, reliability
Date
2023
Type
Journal article
Journal
Frontiers in Education
Book
Volume
8
Issue
Page Range
1-13
Article Number
Article 1183695
ACU Department
Institute for Learning Sciences and Teacher Education (ILSTE)
Faculty of Education and Arts
Faculty of Education and Arts
Relation URI
Source URL
Event URL
Open Access Status
Published as ‘gold’ (paid) open access
License
CC BY 4.0
File Access
Open