Rating of physiotherapy student clinical performance : Is it possible to gain assessor consistency?

Journal article


Kirwan, Garry W., Clark, Courtney R. and Dalton, Megan. (2019). Rating of physiotherapy student clinical performance : Is it possible to gain assessor consistency? BMC Medical Education. 19, p. 32.
AuthorsKirwan, Garry W., Clark, Courtney R. and Dalton, Megan
Abstract

Background
Reliable interpretation of the Assessment of Physiotherapy Practice (APP) tool is necessary for consistent assessment of physiotherapy students in the clinical setting. However, since the APP was implemented, no study has reassessed how consistently a student performance is evaluated against the threshold standards. Therefore, the primary aim of this study was to determine the consistency among physiotherapy educators when assessing a student performance using the APP tool.

Methods
Physiotherapists (n = 153) from Australia with a minimum 3 years clinical experience and who had supervised a physiotherapy student within the past 12-months were recruited. Three levels of performance (not adequate, adequate, good/excellent) were scripted and filmed across outpatient musculoskeletal, neurorehabilitation, cardiorespiratory and inpatient musculoskeletal. In the initial phase of the study, scripts were written by academic staff and reviewed by an expert panel (n = 8) to ensure face and content validity as well as clinical relevance prior to filming. In the second phase of the study, pilot testing of the vignettes was performed by clinical academics (n = 16) from Australian universities to confirm the validity of each vignette. In the final phase, study participants reviewed one randomly allocated vignette, in their nominated clinical area and rated the student performance including a rationale for their decision. Participants were blinded to the performance level. Percentage agreement between participants was calculated for each vignette with an a priori percentage agreement of 75% considered acceptable.

Results
Consensus among educators across all areas was observed when assessing a performance at either the ‘not adequate’ (97%) or the ‘good/excellent’ level (89%). When assessing a student at the ‘adequate’ level, consensus reduced to 43%. Similarly, consensus amongst the ‘not adequate’ and ‘good/excellent’ ranged from 83 to 100% across each clinical area; while agreement was between 33 and 46% for the ‘adequate’ level. Percent agreement between clinical educators was 89% when differentiating ‘not adequate’ from ‘adequate’ or better.

Conclusion
Consistency is achievable for ‘not adequate’ and ‘good/excellent’ performances, although, variability exists at an adequate level. Consistency remained when differentiating an ‘adequate’ or better from a ‘not adequate’ performance.

Keywordsphysiotherapy; consensus; assessment; students; APP
Year2019
JournalBMC Medical Education
Journal citation19, p. 32
Open accessPublished as ‘gold’ (paid) open access
Research or scholarlyResearch
Page range1-8
Publisher's version
License
File Access Level
Open
Output statusPublished
Publication dates
Online24 Jan 2019
Publication process dates
Accepted08 Jan 2019
Deposited17 May 2021
Permalink -

https://acuresearchbank.acu.edu.au/item/8w12w/rating-of-physiotherapy-student-clinical-performance-is-it-possible-to-gain-assessor-consistency

Download files


Publisher's version
  • 150
    total views
  • 40
    total downloads
  • 0
    views this month
  • 1
    downloads this month
These values are for the period from 19th October 2020, when this repository was created.

Export as

Related outputs

Fostering patient uptake of recommended health services and self-management strategies for musculoskeletal conditions : A Delphi study of clinician attributes
O'Leary, Shaun, Gale, Janette, Volker, Glen, Kuipers, Pim, Dalton, Megan and McPhail, Steven. (2020). Fostering patient uptake of recommended health services and self-management strategies for musculoskeletal conditions : A Delphi study of clinician attributes. Musculoskeletal Care. 18(2), pp. 161-168. https://doi.org/10.1002/msc.1448
Acceptance of the 'Assessment of Physiotherapy Practice (Chinese)' as a standardised evaluation of professional competency in Chinese physiotherapy students : An observational study
Jia Hu, Jones, Alice Y. M., Zhou, Xuelian, Zhai, Hua, Ngai, Shirley P. C., Siu, Ka-Chun and Dalton, Megan. (2020). Acceptance of the 'Assessment of Physiotherapy Practice (Chinese)' as a standardised evaluation of professional competency in Chinese physiotherapy students : An observational study. BMC Medical Education. 20(1), p. 108. https://doi.org/10.1186/s12909-020-02026-3
Physiotherapy students' DiSC behaviour styles can be used to predict the likelihood of success in clinical placements
Milne, Nikki, Louwen, Chanelle, Reidlinger, Dianne, Bishop, Jo, Dalton, Megan and Crane, Linda. (2019). Physiotherapy students' DiSC behaviour styles can be used to predict the likelihood of success in clinical placements. BMC Medical Education. 19(1), p. 19. https://doi.org/10.1186/s12909-019-1825-2
Simulation-based education improves student self-efficacy in physiotherapy assessment and management of paediatric patients
Hough, Judith, Levan, Daniel Anthony, Steele, Michael, Kelly, Kristine and Dalton, Megan. (2019). Simulation-based education improves student self-efficacy in physiotherapy assessment and management of paediatric patients. BMC Medical Education. 19(1), pp. 1 - 11. https://doi.org/10.1186/s12909-019-1894-2
Using interprofessional learning for continuing education : development and evaluation of the graduate certificate program in health professional education for clinicians
Henderson, Saras, Dalton, Megan and Cartmel, Jennifer. (2016). Using interprofessional learning for continuing education : development and evaluation of the graduate certificate program in health professional education for clinicians. Journal of Continuing Education in the Health Professions. 36(3), pp. 211-217. https://doi.org/10.1097/CEH.0000000000000093
Measuring safety climate in acute hospitals: Rasch analysis of the safety attitudes questionnaire
Soh, Sze E., Barker, Anna, Morello, Renata T., Dalton, Megan and Brand, Caroline. (2016). Measuring safety climate in acute hospitals: Rasch analysis of the safety attitudes questionnaire. BMC Health Services Research. 16(1), pp. 1 - 9. https://doi.org/10.1186/s12913-016-1744-4
Identifying educator behaviours for high quality verbal feedback in health professions education: Literature review and expert refinement
Johnson, Christina E., Keating, Jennifer L., Boud, David J., Dalton, Megan, Kiegaldie, Debra, Hay, Margaret, McGrath, Barry, McKenzie, Wendy A., Nair, Kichu Balakrishnan R., Nestel, Debra, Palermo, Claire and Molloy, Elizabeth K.. (2016). Identifying educator behaviours for high quality verbal feedback in health professions education: Literature review and expert refinement. BMC Medical Education. 16(96), pp. 1 - 11. https://doi.org/10.1186/s12909-016-0613-5
Identifying educator behaviours for high quality verbal feedback in health professions education: Literature review and expert refinement
Johnson, Christina E., Keating, Jennifer L., Boud, David, Dalton, Megan, Kiegaldie, Debra, Hay, Margaret, McGrath, Barry, McKenzie, Wendy A., Nair, Kichu Balakrishnan R., Nestel, Debra, Palermo, Claire and Molloy, Elizabeth K.. (2016). Identifying educator behaviours for high quality verbal feedback in health professions education: Literature review and expert refinement. BMC Medical Education. 16(1), pp. 1 - 11. https://doi.org/10.1186/s12909-016-0613-5
Validation of the Australian Nursing Standards Assessment Tool (ANSAT): A pilot study
Ossenberg, Christine, Dalton, Megan and Henderson, Amanda. (2016). Validation of the Australian Nursing Standards Assessment Tool (ANSAT): A pilot study. Nurse Education Today. 36, pp. 23 - 30. https://doi.org/10.1016/j.nedt.2015.07.012
Rating of physiotherapy student clinical performance: Is it possible to gain assessor consistency?
Clark, C., Kirwan, G. M. and Dalton, Megan Bernadette. (2016). Rating of physiotherapy student clinical performance: Is it possible to gain assessor consistency? The 4th European Congress of the ER-WCPT. United Kingdom: Elsevier. pp. eS33 - eS34
Determining attainment of nursing standards: The use of behavioural cues to enhance clarity and transparency in student clinical assessment
Ossenberg, Christine, Henderson, Amanda and Dalton, Megan Bernadette. (2015). Determining attainment of nursing standards: The use of behavioural cues to enhance clarity and transparency in student clinical assessment. Nurse Education Today. 35(1), pp. 12 - 15. https://doi.org/10.1016/j.nedt.2014.09.002
www.applinkup.com - An online system to manage and collate workplace based assessment outcomes for physiotherapy students
Dalton, Megan, Russell, T. and Keating, Jennifer L.. (2015). www.applinkup.com - An online system to manage and collate workplace based assessment outcomes for physiotherapy students. 17th World Confederation for Physical Therapy Congress. Singapore: Elsevier. pp. S288 - S289
Consensus between assessors regarding ratings of competency of physiotherapy students: Is it possible?
Clark, C., Kirwan, G. M. and Dalton, Megan Bernadette. (2015). Consensus between assessors regarding ratings of competency of physiotherapy students: Is it possible? APA National Conference. Australia: Elsevier. pp. S254 - S255 https://doi.org/10.1016/j.physio.2015.03.437
Assessing physical therapy students' performance during clinical practice
Murphy, Sue, Dalton, Megan and Dawes, Diana. (2014). Assessing physical therapy students' performance during clinical practice. Physiotherapy Canada. 66(2), pp. 169 - 176. https://doi.org/10.3138/ptc.2013-26
Assessing physical therapy students' performance during clinical practice
Murphy, Sue, Dalton, Megan and Dawes, Diana. (2014). Assessing physical therapy students' performance during clinical practice. Physiotherapy Canada. 66(2), pp. 169 - 176. https://doi.org/10.3138/ptc.2013-26
The Assessment of Physiotherapy Practice (APP) is a reliable measure of professional competence of physiotherapy students: A reliability study
Dalton, Megan, Davidson, Megan and Keating, Jennifer L.. (2012). The Assessment of Physiotherapy Practice (APP) is a reliable measure of professional competence of physiotherapy students: A reliability study. Journal of Physiotherapy. 58(1), pp. 49 - 56. https://doi.org/10.1016/S1836-9553(12)70072-3
Building sustainable capacity in physiotherapy clinical education: An innovative Queensland state-wide collaboration
Dunwoodie, R., Moller, M., Teys, P., Blackwell, L., Dalton, Megan Bernadette, Kirwan, G. M., Offord, Sonia S., Pellatt, A. and Stuart, R.. (2011). Building sustainable capacity in physiotherapy clinical education: An innovative Queensland state-wide collaboration. World Physical Therapy 2011. Amsterdam: Elsevier. pp. eS1474 - eS1475
A systematic review of instruments for the assessment of professional competence of physiotherapy students
Dalton, Megan, Davidson, Megan and Keating, Jennifer L.. (2011). A systematic review of instruments for the assessment of professional competence of physiotherapy students. World Physical Therapy 2011. Amsterdam: Elsevier. pp. eS256 - eS257
The Assessment of Physiotherapy Practice (APP) is a valid measure of professional competence of physiotherapy students: A cross-sectional study with Rasch analysis
Dalton, Megan, Davidson, Megan and Keating, Jennifer L.. (2011). The Assessment of Physiotherapy Practice (APP) is a valid measure of professional competence of physiotherapy students: A cross-sectional study with Rasch analysis. Journal of Physiotherapy. 57(4), pp. 239 - 245. https://doi.org/10.1016/S1836-9553(11)70054-6
Promoting and providing expert guidance in work-intensive clinical settings
Henderson, Amanda, Alexander, Heather, Haywood, Alison, Stapleton, Peta, Cooke, Marie, Patterson, Elizabeth, Dalton, Megan and Creedy, Debra K.. (2010). Promoting and providing expert guidance in work-intensive clinical settings. Vocations and Learning. 3(2), pp. 141 - 156. https://doi.org/10.1007/s12186-010-9036-3
The objective structured clinical examination (OSCE): Optimising its value in the undergraduate nursing curriculum
Mitchell, Marion L., Henderson, Amanda, Dalton, Megan, Groves, Michele and Nulty, Duncan. (2009). The objective structured clinical examination (OSCE): Optimising its value in the undergraduate nursing curriculum. Nurse Education Today. 29(4), pp. 398 - 404. https://doi.org/10.1016/j.nedt.2008.10.007
Boosting morale and improving performance in the nursing setting
Stapleton, Peta, Henderson, Amanda, Creedy, Debra K., Cooke, Marie, Patterson, Elizabeth, Alexander, Heather, Haywood, Alison and Dalton, Megan. (2007). Boosting morale and improving performance in the nursing setting. Journal of Nursing Management. 15(8), pp. 811 - 816. https://doi.org/10.1111/j.1365-2934.2007.00745.x