Validating a forced‑choice method for eliciting quality‑of‑reasoning judgments
Journal article
Marcoci, Alexandru, Stelmach, Margaret E., Rowe, Luke, Barnett, Ashley, Primoratz, Tamar, Kruger, Ariel, Karvetski, Christopher W., Stone, Benjamin, Diamond, Michael L., Saletta, Morgan, van Gelder, Tim, Tetlock, Philip E. and Dennis, Simon. (2023). Validating a forced‑choice method for eliciting quality‑of‑reasoning judgments. Behavior Research Methods. 56, pp. 4958-4973. https://doi.org/10.3758/s13428-023-02234-x
Authors | Marcoci, Alexandru, Stelmach, Margaret E., Rowe, Luke, Barnett, Ashley, Primoratz, Tamar, Kruger, Ariel, Karvetski, Christopher W., Stone, Benjamin, Diamond, Michael L., Saletta, Morgan, van Gelder, Tim, Tetlock, Philip E. and Dennis, Simon |
---|---|
Abstract | In this paper we investigate the criterion validity of forced-choice comparisons of the quality of written arguments with normative solutions. Across two studies, novices and experts assessing quality of reasoning through a forced-choice design were both able to choose arguments supporting more accurate solutions—62.2% (SE = 1%) of the time for novices and 74.4% (SE = 1%) for experts—and arguments produced by larger teams—up to 82% of the time for novices and 85% for experts—with high inter-rater reliability, namely 70.58% (95% CI = 1.18) agreement for novices and 80.98% (95% CI = 2.26) for experts. We also explored two methods for increasing efficiency. We found that the number of comparative judgments needed could be substantially reduced with little accuracy loss by leveraging transitivity and producing quality-of-reasoning assessments using an AVL tree method. Moreover, a regression model trained to predict scores based on automatically derived linguistic features of participants’ judgments achieved a high correlation with the objective accuracy scores of the arguments in our dataset. Despite the inherent subjectivity involved in evaluating differing quality of reasoning, the forced-choice paradigm allows even novice raters to perform beyond chance and can provide a valid, reliable, and efficient method for producing quality-of-reasoning assessments at scale. |
Keywords | reasoning; quality of reasoning; comparative judgment; forced choice; automatic reasoning assessment |
Year | 2023 |
Journal | Behavior Research Methods |
Journal citation | 56, pp. 4958-4973 |
Publisher | Springer |
ISSN | 0743-3808 |
Digital Object Identifier (DOI) | https://doi.org/10.3758/s13428-023-02234-x |
PubMed ID | 37833511 |
Scopus EID | 2-s2.0-85174061724 |
Web address (URL) | https://link.springer.com/article/10.3758/s13428-023-02234-x |
Open access | Published as ‘gold’ (paid) open access |
Research or scholarly | Research |
Page range | 4958-4973 |
Funder | Office of the Director of National Intelligence (ODNI), United States of America |
Intelligence Advanced Research Projects Activity (IARPA), United States of America | |
The British Academy | |
The Leverhulme Trust | |
Publisher's version | License File Access Level Open |
Output status | Published |
Publication dates | |
Online | 13 Oct 2023 |
Publication process dates | |
Accepted | 02 Sep 2023 |
Deposited | 27 Nov 2023 |
Grant ID | 16122000002 |
SRG2223\231699 | |
Additional information | © The Author(s) 2023. |
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ |
https://acuresearchbank.acu.edu.au/item/8zzq0/validating-a-forced-choice-method-for-eliciting-quality-of-reasoning-judgments
Download files
Publisher's version
OA_Marcoci_2023_Validating_a_forced_choice_method_for.pdf | |
License: CC BY 4.0 | |
File access level: Open |
84
total views38
total downloads0
views this month0
downloads this month