The generalizability of online experiments conducted during the COVID-19 pandemic

Journal article


Peyton, Kyle, Huber, Gregory A. and Coppock, Alexander. (2021). The generalizability of online experiments conducted during the COVID-19 pandemic. Journal of Experimental Political Science. https://doi.org/10.1017/XPS.2021.17
AuthorsPeyton, Kyle, Huber, Gregory A. and Coppock, Alexander
Abstract

The COVID-19 pandemic imposed new constraints on empirical research, and online data collection by social scientists increased. Generalizing from experiments conducted during this period of persistent crisis may be challenging due to changes in how participants respond to treatments or the composition of online samples. We investigate the generalizability of COVID era survey experiments with 33 replications of 12 pre-pandemic designs, fielded across 13 quota samples of Americans between March and July 2020. We find strong evidence that pre-pandemic experiments replicate in terms of sign and significance, but at somewhat reduced magnitudes. Indirect evidence suggests an increased share of inattentive subjects on online platforms during this period, which may have contributed to smaller estimated treatment effects. Overall, we conclude that the pandemic does not pose a fundamental threat to the generalizability of online experiments to other time periods.

Keywordsgeneralizability; experiment; online research; survey research; replication; COVID-19; pandemic; attentiveness
Year2021
JournalJournal of Experimental Political Science
PublisherCambridge University Press
ISSN2052-2630
Digital Object Identifier (DOI)https://doi.org/10.1017/XPS.2021.17
Research or scholarlyResearch
Page range1-16
Publisher's version
License
All rights reserved
File Access Level
Controlled
Output statusPublished
Publication dates
Online02 Jul 2021
Publication process dates
Deposited25 Oct 2021
Permalink -

https://acuresearchbank.acu.edu.au/item/8wx8x/the-generalizability-of-online-experiments-conducted-during-the-covid-19-pandemic

Restricted files

Publisher's version

  • 56
    total views
  • 0
    total downloads
  • 2
    views this month
  • 0
    downloads this month
These values are for the period from 19th October 2020, when this repository was created.

Export as

Related outputs

Building mass support for global pandemic recovery efforts in the United States
Nair, Gautam and Peyton, Kyle. (2022). Building mass support for global pandemic recovery efforts in the United States. Proceedings of the National Academy of Sciences of USA. 1(4), pp. 1-9. https://doi.org/10.1093/pnasnexus/pgac123
Beliefs about minority representation in policing and support for diversification
Peyton, Kyle, Weiss, Chagai and Vaughn, Paige. (2022). Beliefs about minority representation in policing and support for diversification. Proceedings of the National Academy of Sciences of USA. 119(52), p. Article e2213986119. https://doi.org/10.1073/pnas.2213986119
Mass support for proposals to reshape policing depends on the implications for crime and safety
Vaughn, Paige E., Peyton, Kyle and Huber, Gregory A.. (2022). Mass support for proposals to reshape policing depends on the implications for crime and safety. Criminology and Public Policy. 21(1), pp. 125-146. https://doi.org/10.1111/1745-9133.12572
Racial resentment, prejudice, and discrimination
Peyton, Kyle and Huber, Gregory A.. (2021). Racial resentment, prejudice, and discrimination. The Journal of Politics. 84(3), pp. 397-403. https://doi.org/10.1086/711558
Corruption in emerging and developing economies : Evidence from a pooled cross-section
Peyton, Kyle and Belasen, Ariel R.. (2012). Corruption in emerging and developing economies : Evidence from a pooled cross-section. Emerging Markets Finance and Trade. 48(2), pp. 29-43. https://doi.org/10.2753/REE1540-496X480202