The effect of workshop training on rater variability in children’s oral narrative assessment

Journal article


Karusoo-Musumeci, Ava, Pearce, Wendy M. and Donaghy, Michelle. (2022). The effect of workshop training on rater variability in children’s oral narrative assessment. Child Language Teaching and Therapy. 38(1), pp. 8-21. https://doi.org/10.1177/02656590211023839
AuthorsKarusoo-Musumeci, Ava, Pearce, Wendy M. and Donaghy, Michelle
Abstract

Oral narrative assessments are important for diagnosis of language disorders in school-age children so scoring needs to be reliable and consistent. This study explored the impact of training on the variability of story grammar scores in children’s oral narrative assessments scored by multiple raters. Fifty-one speech pathologists and 19 final-year speech pathology students attended training workshops on oral narrative assessment scoring and analysis. Participants scored two oral narratives prompted by two different story stimuli and produced by two children of differing ages. Demographic information, story grammar scores and a confidence survey were collected pre- and post-training. The total story grammar score changed significantly for one of the two oral narratives. A significant effect was observed for rater years of experience and the change in total story grammar scores post training, with undergraduate students showing the greatest change. Two story grammar elements, character and attempt, changed significantly for both stories, with an overall trend of increased element scores post-training. Confidence ratings also increased post-training. Findings indicated that training via an interactive workshop can reduce rater variability when using researcher-developed narrative scoring systems.

Keywordsadult learning; monitoring change; oral narrative; professional development; story grammar
Year2022
JournalChild Language Teaching and Therapy
Journal citation38 (1), pp. 8-21
PublisherSAGE Publications
ISSN0265-6590
Digital Object Identifier (DOI)https://doi.org/10.1177/02656590211023839
Scopus EID2-s2.0-85108808430
Page range8-21
Publisher's version
License
All rights reserved
File Access Level
Controlled
Output statusPublished
Publication dates
Online27 Jun 2021
Publication process dates
Deposited24 Nov 2022
Permalink -

https://acuresearchbank.acu.edu.au/item/8y75w/the-effect-of-workshop-training-on-rater-variability-in-children-s-oral-narrative-assessment

Restricted files

Publisher's version

  • 1
    total views
  • 0
    total downloads
  • 1
    views this month
  • 0
    downloads this month
These values are for the period from 19th October 2020, when this repository was created.

Export as

Related outputs

Pathways to paediatric speech pathology services in Australia
Just, Elena, Pearce, Wendy M. and Crosbie, Sharon. (2022). Pathways to paediatric speech pathology services in Australia. Speech, Language and Hearing. 25(2), pp. 192-203. https://doi.org/10.1080/2050571X.2020.1846837
Narrative comprehension skills of Indigenous and non-Indigenous Australian children in their first year of school
Shoebridge, Sarah J., Flanagan, Kieran J. and Pearce, Wendy M.. (2021). Narrative comprehension skills of Indigenous and non-Indigenous Australian children in their first year of school. International Journal of Speech-Language Pathology. pp. 1-16. https://doi.org/10.1080/17549507.2021.1914729
Speech-language pathologists’ perceptions and experiences when working with Aboriginal and Torres Strait Islander children
Zingelman, Sally, Pearce, Wendy M. and Saxton, Kate. (2021). Speech-language pathologists’ perceptions and experiences when working with Aboriginal and Torres Strait Islander children. International Journal of Speech-Language Pathology. 23(3), pp. 225-235. https://doi.org/10.10.1080/175249507.2020.1779345
Speech-language pathologists’ perceptions and experiences when working with Aboriginal and Torres Strait Islander children
Zingelman, Sally, Pearce, Wendy M. and Saxton, Kate. (2021). Speech-language pathologists’ perceptions and experiences when working with Aboriginal and Torres Strait Islander children. International Journal of Speech-Language Pathology. 23(3), pp. 225-235. https://doi.org/10.1080/17549507.2020.1779345
Verbal contingencies in the Lidcombe Program : A noninferiority trial
Donaghy, Michelle, O'Brian, Sue, Onslow, Mark, Lowe, Robyn, Jones, Mark and Menzies Ross, G.. (2020). Verbal contingencies in the Lidcombe Program : A noninferiority trial. Journal of Speech, Language, and Hearing Research. 63(10), pp. 3419-3431. https://doi.org/10.1044/2020_JSLHR-20-00155
Perceptions of a culturally responsive school-based oral language and early literacy programme
Norman, Taneal, Pearce, Wendy M. and Eastley, Fiona. (2020). Perceptions of a culturally responsive school-based oral language and early literacy programme. The Australian Journal of Indigenous Education. pp. 1-10. https://doi.org/10.1017/jie.2019.25
First year allied health student transition to a regional university
Gray, Marion, Gordon, Susan, O'Neill, Maureen and Pearce, Wendy M.. (2019). First year allied health student transition to a regional university. Australian Journal of Rural Health. 27(6), pp. 497 - 504. https://doi.org/10.1111/ajr.12581
Language abilities of Indigenous and non-Indigenous Australian children from low socioeconomic backgrounds in their first year of school
Pearce, Wendy M. and Flanagan, Kieran. (2019). Language abilities of Indigenous and non-Indigenous Australian children from low socioeconomic backgrounds in their first year of school. International Journal of Speech-Language Pathology. 21(2), pp. 212-223. https://doi.org/10.1080/17549507.2018.1444091
Pragmatic language outcomes of children with attention deficit hyperactivity disorder after therapist- and parent-delivered play-based interventions: Two one-group pretest-posttest studies with a longitudinal component
Wilkes-Gillan, Sarah, Munro, Natalie, Cordier, Reinie, Cantrill, Alycia and Pearce, Wendy. (2017). Pragmatic language outcomes of children with attention deficit hyperactivity disorder after therapist- and parent-delivered play-based interventions: Two one-group pretest-posttest studies with a longitudinal component. American Journal of Occupational Therapy. 71(4), pp. 7104220030p1 - 7104220030p10. https://doi.org/10.5014/ajot.2017.019364
Evaluating the pragmatic language skills of children with ADHD and typically developing playmates following a pilot parent-delivered play-based intervention
Cordier, Reinie, Munro, Natalie, Wilkes-Gillan, Sarah, Ling, Lydia, Docking, Kimberley and Pearce, Wendy. (2017). Evaluating the pragmatic language skills of children with ADHD and typically developing playmates following a pilot parent-delivered play-based intervention. Australian Occupational Therapy Journal. 64(1), pp. 11 - 23. https://doi.org/10.1111/1440-1630.12299
Psychometric properties of language assessments for children aged 4-12 years: a systematic review
Denman, Deborah, Speyer, Renee, Munro, Natalie, Pearce, Wendy, Chen, Yu-Wei and Cordier, Reine. (2017). Psychometric properties of language assessments for children aged 4-12 years: a systematic review. Frontiers in Psychology. 8, pp. 1 - 28. https://doi.org/10.3389/fpsyg.2017.01515
Management options for pediatric patients who stutter: Current challenges and future directions
Donaghy, Michelle and Smith, Kylie A.. (2016). Management options for pediatric patients who stutter: Current challenges and future directions. Pediatric Health, Medicine and Therapeutics. https://doi.org/10.2147/PHMT.S77568
An investigation of the role of parental request for self-correction of stuttering in the Lidcombe Program
Donaghy, Michelle, Harrison, Elisabeth, O'Brian, Sue, Menzies, Ross, Onslow, Mark, Packman, Ann and Jones, Mark. (2015). An investigation of the role of parental request for self-correction of stuttering in the Lidcombe Program. International Journal of Speech-Language Pathology. 17(5), pp. 511 - 517. https://doi.org/10.3109/17549507.2015.1016110
Dialectal grammatical differences in oral narratives of school-aged Indigenous Australian children
Pearce, Wendy, Williams, Cori and Steed, William. (2015). Dialectal grammatical differences in oral narratives of school-aged Indigenous Australian children. International Journal of Speech-Language Pathology. 17(4), pp. 335 - 345. https://doi.org/10.3109/17549507.2014.979878
Reliability and validity of the Pragmatics Observational Measure (POM) : A new observational measure of pragmatic language for children
Cordier, Reinie, Munro, Natalie, Wilkes-Gillan, Sarah, Speyer, Renee and Pearce, Wendy. (2014). Reliability and validity of the Pragmatics Observational Measure (POM) : A new observational measure of pragmatic language for children. Research in Developmental Disabilities. 35(7), pp. 1588 - 1598. https://doi.org/10.1016/j.ridd.2014.03.050
Effect of dialect on the identification of speech impairment in indigenous children
Laffey, Kate, Pearce, Wendy and Steed, William. (2014). Effect of dialect on the identification of speech impairment in indigenous children. Australian Review of Applied Linguistics. 37(2), pp. 161 - 177. https://doi.org/10.1075/aral.37.2.05laf
Reliability and validity of the Pragmatics Observational Measure (POM): A new observational measure of pragmatic language for children
Cordier, Reinie, Munro, Natalie, Wilkes-Gillan, Sarah, Speyer, Renee and Pearce, Wendy. (2014). Reliability and validity of the Pragmatics Observational Measure (POM): A new observational measure of pragmatic language for children. Research in Developmental Disabilities. 35(7), pp. 1588 - 1598. https://doi.org/10.1016/j.ridd.2014.03.050
First-time mothers’ knowledge and beliefs regarding early communication development
Williams, Vicki, Pearce, Wendy and Devine, Sue. (2014). First-time mothers’ knowledge and beliefs regarding early communication development. Early Child Development and Care. 184(1), pp. 15 - 31. https://doi.org/10.1080/03004430.2013.769213
The cultural appropriateness and diagnostic usefulness of standardized language assessments for Indigenous Australian children
Pearce, Wendy M. and Williams, Cori. (2013). The cultural appropriateness and diagnostic usefulness of standardized language assessments for Indigenous Australian children. International Journal of Speech-Language Pathology. 15(4), pp. 429 - 440. https://doi.org/10.3109/17549507.2012.762043
Oral narratives produced by Aboriginal Australian children: Dilemmas with normative comparisons
Pearce, Wendy and Stockings, Emma. (2011). Oral narratives produced by Aboriginal Australian children: Dilemmas with normative comparisons. ACQuiring Knowledge in Speech, Language and Hearing. 13(3), pp. 126 - 131.
A comparison of oral narratives in children with specific language and non-specific language impairment
Pearce, Wendy, James, Deborah and McCormack, Paul. (2010). A comparison of oral narratives in children with specific language and non-specific language impairment. Clinical Linguistics & Phonetics. 24(8), pp. 622 - 645. https://doi.org/10.3109/02699201003736403
Prioritising intervention for children with developmental language impairments: does cognition matter?
James, Deborah G. H., McCormack, Paul F. and Pearce, Wendy. (2007). Prioritising intervention for children with developmental language impairments: does cognition matter? Journal of Clinical Practice in Speech-Language Pathology. 9(1), pp. 5 - 8.
Does the choice of stimulus affect the complexity of children's oral narratives?
Pearce, Wendy. (2003). Does the choice of stimulus affect the complexity of children's oral narratives? International Journal of Speech-Language Pathology. 5(2), pp. 95 - 103.
Exploring the boundaries of SLI: findings from morphosyntactic and story grammar analyses
James, Deborah G. H., McCormack, Paul F. and Pearce, Wendy. (2003). Exploring the boundaries of SLI: findings from morphosyntactic and story grammar analyses. Clinical Linguistics & Phonetics. 17(4), pp. 325 - 334. https://doi.org/10.1080/0269920031000080019