Using pairwise comparison and ordered exemplars as a basis for a novel method of standard setting in narrative writing
Journal article
Humphry, Stephen, Adie, Lenore, Maxwell, Carolyn and Sappl, Sonia. (2023). Using pairwise comparison and ordered exemplars as a basis for a novel method of standard setting in narrative writing. Frontiers in Education. 8, p. Article 1183695. https://doi.org/10.3389/feduc.2023.1183695
Authors | Humphry, Stephen, Adie, Lenore, Maxwell, Carolyn and Sappl, Sonia |
---|---|
Abstract | Standard setting of extended performances is fraught with difficulties due to the subjective nature of the scoring decision. This article introduces and reports results of a novel systematic method for setting standards for assessments that require extended performances: the Scaled Exemplar Standard Setting method (SESS). This two-stage method brings together (1) pairwise comparison of samples to scale performances and (2) standard setting involving multiple standards with tasks presented to judges in a validated order. By utilizing paired comparison methodology in the first stage, the judgment is made more manageable by chunking it into a series of pairs. This method produces scaled performances that can be checked for internal consistency of the judgments. The validated, ordered performances are used in the standard setting stage, making the task for judges as transparent and straightforward as possible. Recommendations and implications for standard setting are discussed in light of the results. |
Keywords | assessment; teacher judgment; standard setting; pairwise comparison; extended performance; scaled exemplars; reliability |
Year | 2023 |
Journal | Frontiers in Education |
Journal citation | 8, p. Article 1183695 |
Publisher | Frontiers Media S.A. |
ISSN | 2504-284X |
Digital Object Identifier (DOI) | https://doi.org/10.3389/feduc.2023.1183695 |
Scopus EID | 2-s2.0-85168296926 |
Open access | Published as ‘gold’ (paid) open access |
Page range | 1-13 |
Funder | Australian Research Council (ARC) |
Publisher's version | License File Access Level Open |
Output status | Published |
Publication dates | |
Online | 02 Aug 2023 |
Publication process dates | |
Accepted | 20 Jul 2023 |
Deposited | 07 Nov 2023 |
ARC Funded Research | This output has been funded, wholly or partially, under the Australian Research Council Act 2001 |
Grant ID | LP180100046 |
https://acuresearchbank.acu.edu.au/item/8zy69/using-pairwise-comparison-and-ordered-exemplars-as-a-basis-for-a-novel-method-of-standard-setting-in-narrative-writing
Download files
Publisher's version
OA_Humphry_2023_Using_pairwise_comparison_and_ordered_exemplars.pdf | |
License: CC BY 4.0 | |
File access level: Open |
81
total views43
total downloads0
views this month0
downloads this month