The generalizability of online experiments conducted during the COVID-19 pandemic
Journal article
Peyton, Kyle, Huber, Gregory A. and Coppock, Alexander. (2021). The generalizability of online experiments conducted during the COVID-19 pandemic. Journal of Experimental Political Science. https://doi.org/10.1017/XPS.2021.17
Authors | Peyton, Kyle, Huber, Gregory A. and Coppock, Alexander |
---|---|
Abstract | The COVID-19 pandemic imposed new constraints on empirical research, and online data collection by social scientists increased. Generalizing from experiments conducted during this period of persistent crisis may be challenging due to changes in how participants respond to treatments or the composition of online samples. We investigate the generalizability of COVID era survey experiments with 33 replications of 12 pre-pandemic designs, fielded across 13 quota samples of Americans between March and July 2020. We find strong evidence that pre-pandemic experiments replicate in terms of sign and significance, but at somewhat reduced magnitudes. Indirect evidence suggests an increased share of inattentive subjects on online platforms during this period, which may have contributed to smaller estimated treatment effects. Overall, we conclude that the pandemic does not pose a fundamental threat to the generalizability of online experiments to other time periods. |
Keywords | generalizability; experiment; online research; survey research; replication; COVID-19; pandemic; attentiveness |
Year | 2021 |
Journal | Journal of Experimental Political Science |
Publisher | Cambridge University Press |
ISSN | 2052-2630 |
Digital Object Identifier (DOI) | https://doi.org/10.1017/XPS.2021.17 |
Research or scholarly | Research |
Page range | 1-16 |
Publisher's version | License All rights reserved File Access Level Controlled |
Output status | Published |
Publication dates | |
Online | 02 Jul 2021 |
Publication process dates | |
Deposited | 25 Oct 2021 |
https://acuresearchbank.acu.edu.au/item/8wx8x/the-generalizability-of-online-experiments-conducted-during-the-covid-19-pandemic
Restricted files
Publisher's version
56
total views0
total downloads2
views this month0
downloads this month