Skip to main content
Download PDF

The impact of a targeted education package on the knowledge, attitudes, and utilisation of patient reported outcome measures amongst chiropractors in Australia

Chiropractic & Manual Therapies volume 30, Article number: 44 (2022) Cite this article

Abstract

Background

Patient Reported Outcome Measures (PROMs) have been shown to be valid and reliable indicators of health status and treatment outcomes, however, the current knowledge, understanding, and utilisation of PROMs within the Australian Chiropractic profession is limited. This study used the New World Kirkpatrick Model (NWKM) to evaluate whether an online PROM education package could improve knowledge, confidence, attitude, and utilisation of PROMs by chiropractors in Australia.

Methods

A longitudinal cohort interventional study of chiropractors in Australia. The recruitment phase and data collection period occurred from November 2020 – May 2021. Participants completed three online surveys two weeks before, four weeks after, and 12 weeks after receiving an online education package that included ten evidence-based region-specific modules on PROMs. Survey questions were grouped into five subthemes for analysis according to the NWKM levels: (1) Reaction; (2) Learning - knowledge; (3) Learning – confidence; (4) Behaviour – attitude; (5) and Behaviour - utilisation).

Results

Of the 116 participants that enrolled in the study, 43 completed all three survey and were included in the analysis. There was very positive reaction to the education package with mean response scores (1–5 Likert scale) for the reaction questions ranging from 3.75 to 4.43. There was a small, but significant, increase in knowledge (out of 32) at four weeks (24.3 ± 6.1) and 12 weeks after receiving the education package (27.2 ± 5.5), compared to baseline (27.4 ± 5.1). There was no effect of the intervention on clinician confidence or attitude towards PROMs. Utilisation of function- and pain-related PROMs did not change after the intervention. There was a small and significant (p < 0.05) increase in utilisation of health-related PROMs 12 weeks after the intervention.

Conclusion

Despite modest improvements in knowledge, which were retained 12 weeks after the educational package was provided, there was no evidence that participant confidence, attitude, or utilisation of PROMs changed because of the intervention. While the respondents’ have positive attitudes and beliefs regarding PROMs use, further education surrounding the clinical translation process into clinical practice is required. Caution is advised when interpreting these findings due to the low participant response and completion rate with the potential for selection bias and the inability to generalise the results.

Background

Patient reported outcome measures (PROMs) are validated questionnaires and survey instruments used to measure the status of a patient’s health condition using information that comes directly from the patient [1, 2]. PROMs were initially used as epidemiological surveys to identify patterns of symptomatology or health status, however, they have evolved to be tools that can also be used to support clinical practice and research [3]. In the clinical setting, PROMs can assist the decision-making process, improve therapist-patient communication, monitor treatment progress, and/or facilitate patient-centered care [4,5,6,7]. PROMs are therefore considered an integral part of evidence-based health care and therefore it is important that clinicians are equipped with the requisite skills and knowledge needed to apply PROMs in clinical practice.

A recent survey of chiropractors in Australia found that 72% of respondents reported using PROMs for the management of low back pain (LBP) [8]. Similar rates of PROM utilisation were reported for physiotherapists in the United Kingdom, where 72% of physiotherapists, and 71% of physiotherapy practices, routinely used PROMs [9]. Despite these relatively high rates, it has been suggested there is a lack of knowledge and understanding of PROMs amongst chiropractors, which may limit the full clinical implementation of PROMs and their potential benefits [5, 8,9,10,11]. For example, a survey of chiropractors in Australia found that 19% of respondents reported that a lack of knowledge regarding PROMs prevented clinical implementation, while time required to administer and score PROMs in a clinical setting (17%) was also a barrier to their utilisation [8]. The results of studies by Holmes, Bishop [5] and Antunes, Harding [12] mirrored these findings, suggesting that a clinician’s lack of PROM knowledge was a significant barrier to implementation.

Although chiropractors are increasingly encouraged to use PROMs [13], a lack of knowledge transfer from literature, education, and training into clinical practice are likely barriers to their successful implementation into clinical practice [5, 14]. Furthermore, even though chiropractors place high value on the importance of PROMs in clinical practice, these beliefs do not appear to translate to increased frequency of PROM usage, suggesting other factors, including knowledge, may be central to changing behaviors [15].

To improve the transfer of knowledge to practice, a number of theories, models and frameworks have been developed [16]. Within allied health settings, the most highly cited and accepted model is the Knowledge to Action framework (KTA) [17,18,19,20] created by Graham and Tetroe [21]. KTA is a conceptual framework made up of two distinct components - knowledge creation and the action cycle. The knowledge creation phase is represented as a tunnel and the information becomes more refined and specific as it passes through this phase to uncover the most valid and useful knowledge [22]. The action phase is a cycle leading to implementation or application of knowledge uncovered in the knowledge creation phase. During this step, adapting knowledge to meet the needs within particular contexts or populations occurs [23].

The aim of this study was to evaluate the impact of an educational package developed using the New World Kirkpatrick Model (NWKM) according to four levels; reaction, learning, behaviour and results [24]. The implementation was assessed using KTA framework to improve PROM knowledge and utilisation in a cohort of chiropractors in Australia. We hypothesised that a tailored educational package intervention would increase participant knowledge of PROMs, which would translate to a positive change in attitude and frequency of PROM utilisation in clinical practice.

Methods

Study design

A longitudinal interventional cohort survey with repeated measures was used to gain an understanding of the respondent’s knowledge, attitudes, and utilisation of PROMs in the Chiropractic profession of Australia. An online educational package was designed, delivered and the findings evaluated using the New World Kirkpatrick Model (NWKM) [24,25,26] using three of the four key themes of the model: reaction, learning, and behaviour. The study was conducted over 7 months and included three online surveys administered at specific timepoints using the Qualtrics software. Full ethical approval was granted by the Human Research Ethics Committee of Central Queensland University, reference number 0000022391.

Participants

Chiropractors who were current members of the Australian Chiropractic Association (ACA) or Chiropractic Australia (CA) were invited to participate in the study via member emails and advertisement in the association’s newsletters. The sample population consisted of 3,215 ACA members and 1,163 CA members, totaling 4,378 potential participants, representing the majority of Chiropractors registered and practicing in Australia in 2020 [27]. The recruitment phase and data collection period occurred from November 2020 – May 2021.

Longitudinal surveys

Participants were surveyed on three occasions: (1) two weeks before the educational package (Survey 1); (2) four weeks after receiving the educational package (Survey 2); and (3) 12 weeks after receiving the educational package (Survey 3). Survey 1 collected demographic information including age, sex, country of graduation, highest level of education, in addition to the participant’s knowledge and attitudes about PROMs and their preferred mode of delivery (e.g., audio, audio-visual or transcript) for the educational package. Survey 2 and 3 repeated the knowledge and attitude questions from Survey 1, while omitting the demographic questions, and including items assessing the participant’s reaction to the educational package.

Educational package

The content and design of the educational package was informed by data obtained from Survey 1, and therefore, being co-designed [28]. Respondents were given multiple-choice questions with options of content to be included in the educational package. There was also an open text box to allow them to formulate their own answers and request specific information to be included, which is recommended when using survey methods [29]. Survey 1 also asked the respondents about their preference of mode for the delivery of the educational package. Of the 116 respondents, 81% were very likely/likely to view via text (electronic), 57% via audio-video format, 55% via text (paper), and 35% via audio. Given the spread of preferred formats the educational package was adapted and made available via for all three modes – audio-visual, audio and text [30].

When asked about what topics should be included in the package, most respondents requested ‘Examples of the best PROMs for major conditions’ (85%), ‘How PROMs can be used in Chiropractic clinical practice (83%)’, and ‘Explanation of categories of PROMs (80%)’. The subsequent educational package consisted of 10 modules: 1) introduction; 2) background of PROMs; 3) PROMs in Chiropractic practice; 4) non-location specific PROMs; 5) PROMs for headache, whiplash, temporomandibular joint and dizziness; 6) PROMs for cervical spine; 7) PROMs for thoracic spine; 8) PROMs for lumbar spine; 9) PROMs for upper extremity; and 10) PROMs for lower extremity. For modules 4–10, the following topics were included (where available and/or appropriate) for each example PROM: category, overview, scoring, area of assessment, method of delivery, and clinical condition(s). The duration of the ten audio-visual and audio modules ranged from approximately one to fifteen minutes. The full educational package is available in all formats.

Data analysis

Survey data was exported from Qualtrics software and analysed using the Statistical Package for the Social Sciences (SPSS version 28). A one-way repeated measure analysis of variance was used to assess the effect of time (two weeks before, four weeks after, and 12 weeks after) on knowledge scores, with post-hoc paired t-tests used to compare time points. For non-parametric data obtained from Likert responses, the effect of time was assessed using the Friedman test with post-hoc Wilcoxon signed-rank test used to compare time points. All post-hoc tests were corrected for multiple comparisons using Bonferroni alpha corrections. To aid interpretation and discussion survey questions were thematically grouped a-priori according to each level within the NWKM - reaction, learning, and behavior (see Supplementary Table 1). The correlation between response data of thematically grouped questions were assessed using Cronbach’s alpha with α scores > 0.7 considered satisfactory [31].

Results

A total of 116 participants completed Survey 1 and were provided with the educational package and an invitation to complete Survey 2 and 3. Of these participants, 52 completed Survey 2 (47% retention) and 43 completed Survey 3 (37% retention from those who completed the initial survey). A summary of participant characteristics is presented in Table 1.

Table 1 Descriptive summary of participant characteristics for each survey
Full size table

Level 1—reaction to the educational package

Participant reaction to the educational package was evaluated using eight questions from Survey 3 (n = 43) grouped according to four subthemes – knowledge, behaviour, relevance, and satisfaction (Table 2). Most respondents viewed the educational package favorably with mean response scores ranging from 3.75 (‘I would recommend changes to my practice procedures after viewing this educational package’) and 4.43 (‘The educational package was effective at increasing my knowledge’) on a 5-point Likert scale. Most participants were either somewhat or strongly satisfied with the overall educational package (81%), and either strongly or somewhat agreed that the educational package improved their knowledge of PROMs (87%). Similarly, 72% of participants were satisfied with the duration of the package and 99% felt that the package was relevant to their needs, with 88% reporting they would recommend it to their colleagues. The correlation between subtheme questions ranged between α = 0.71 and 0.91.

Table 2 Reaction to the education package
Full size table

Level 2—learning (knowledge about PROMs)

Participant knowledge of PROMs was evaluated in each survey using 32 questions that covered topics including PROM definitions, categories, and specific examples. The mean (SD) knowledge scores (out of 32) for Survey 1, 2, and 3 were 24.2 (6.1), 27.2 (5.5), and 27.4 (5.1), respectively. The one-way repeated measured analysis of variance (n = 43) revealed a significant effect of time on knowledge (F41,1 = 21.198, p < 0.001). Post-hoc t-tests indicate a significant improvement in knowledge from Survey 1 to Survey 2 (mean difference ± 95%CI = 3.0 ± 2.5, p = 0.013) and Survey 1 to Survey 3 (mean difference ± 95%CI = 3.1 ± 1.7, p < 0.001), with no significant difference in knowledge between Survey 2 and 3 (p = 1.00).

Level 2 learning—confidence in understanding and using PROMs

The related samples Friedman’s analysis of variance by ranks test (n = 39) revealed no significant effect of time on clinician confidence in: (1) understanding what a PROM is (χ2 (2) = 0.607, p = 0.738); (2) understanding the significance of using PROMs (χ2 (2) = 0.157, p = 0.924); (3) recognising when to apply PROMs (χ2 (2) = 0.157, p = 0.924); (4) implementing PROMs (χ2 (2) = 0.171, p = 0.918); or (5) knowing what patient reported measures are available (χ2 (2) = 0.639, p = 0.726). The correlation between the questions α = 0.899. The mean ranks for each question at each time point are included in Table 3.

Table 3 Change in participant confidence using patient reported outcome measures before and after receiving the education package
Full size table

Level 3—behaviour (attitude towards PROMs)

There was no significant effect of time on how influential PROMs were to the treatment plan and patient management (χ2 (2) = 3.644, p = 0.162) (see Table 4). There was a significant effect of time on whether health professional should use PROMs using valid and reliable tools (χ2 (2) = 6.982, p = 0.03), however, the post-hoc Wilcoxon rank-tests revealed no significance difference between Survey 1 and 2 (Z = 1.670, p = 0.095) or between Survey 1 and 3 (Z = 1.485, p = 0.138). There was no effect of time on clinician attitudes to PROMs based on the remaining statements: (1) ‘PROMs enable you to get a better understanding of your patient’s progress’ (χ2 (2) = 1.627, p = 0.443); (2) ‘The use of validated PROMs is clinically helpful in an increasing medicolegal environment’ (χ2 (2) = 1.853, p = 0.396); (3) ‘The use of patient reported outcome measures could be helpful in justifying ongoing treatment to third parties’ (χ2 (2) = 4.680, p = 0.096); (4) ‘My patients are all different; therefore patient reported outcome measures would not be useful’ (χ2 (2) = 0.628, p = 0.731); (5) ‘Available patient reported outcome measures are unsuitable for the type of patients I treat’ (χ2 (2) = 2.086, p = 0.352); (6) ‘If I had more time, I would be interested in using patient reported outcome measures’ (χ2 (2) = 0.609, p = 0.738); (7) ‘I do not see the use of patient reported outcome measures as a priority’ (χ2 (2) = 2.385, p = 0.304); (8) ‘Patient reported outcome measures are unpopular with patients’ (χ2 (2) = 1.681, p = 0.432); (9) ‘Patient satisfaction is the most important outcome’ (χ2 (2) = 4.978, p = 0.083); (10) ‘I do not know enough about patient reported outcome measures to feel comfortable/confident using them’ (χ2 (2) = 0.828, p = 0.661); 11) ‘The patient discontinuing treatment puts me off using patient reported outcome measures’ (χ2 (2) = 0.023, p = 0.998); 12) ‘There is no need to change from the way that we have to assess/assessed patients’ (χ2 (2) = 2.141, p = 0.343); 13) ‘If I had to use patient reported outcome measures, I would prefer to choose which ones I used’ (χ2 (2) = 2.049, p = 0.359); 14) ‘Access to information about patient reported outcome measures is limited in my work environment’ (χ2 (2) = 1.162, p = 0.559); and 15) ‘It is not necessary to measure functional outcomes’ (χ2 (2) = 0.667, p = 0.717). The correlation between the questions α = 0.655.

Table 4 Change in participant attitude towards patient reported outcome measures before and after receiving the education package
Full size table

Level 3—behaviour (Utilisation of PROMs)

There was no significant effect of time on the frequency of pain-related (χ2 (2) = 0.192, p = 0.909) or functional-related (χ2 (2) = 3.69, p = 0.158) PROM use (Table 5). There was, however, a significant effect of time on the frequency of health-related PROM use (χ2 (2) = 8.310, p = 0.016). Post-hoc Wilcoxon signed-rank tests revealed a significant increase in the mean rank score for health-related PROM use from Survey 1 (1.79) to Survey 3 (2.01) (Z = 2.707, p = 0.007, ES = 0.41) with 14 positive differences, 3 negative differences, and 27 ties (Table 3). There was no difference in the mean rank score between Survey 1 to Survey 2 (Z = 0.125, p = 0.901), with 10 positive differences, 7 negative differences, and 35 ties. When asked about whether the educational package increased their use of patient reported outcome measures (PROMs) in clinical practice, 39% of participants in Survey 2 answered ‘yes’ (41% No, 17% Unsure), compared to 56% in Survey 3 (28% No, 16% Unsure).

Table 5 Changes in participant utilisation of patient reported outcome measures before and after receiving the education package
Full size table

Discussion

This longitudinal study evaluated the impact of an educational package developed and implemented using the KTA framework, and which aimed to increase PROM knowledge and utilisation in a small cohort of chiropractors in Australia. The results are presented and discussed according to three of the four levels of the NWKM – reaction, learning, and behavior [32]. Overall, the reaction to the educational package was very positive, with most participants noting improved knowledge and a high level of satisfaction with the material. Although there was a significant improvement in knowledge after receiving the educational package, which was retained at 12 weeks, these changes were small and did not translate to improvements in confidence, attitude, or frequency of PROM usage. The findings suggest that changes in knowledge alone may be insufficient to influence the frequency of PROM utilisation by chiropractors.

Level 1—reaction

It is important to evaluate the participant’s reactions to the course or training when evaluating the success of the educational intervention [33]. Overall, the findings confirmed a positive reaction to the educational package. Most respondents reported that the intervention was favourable, engaging, and relevant to their needs. These findings could therefore be important to create clinical behaviour change as Oreg, Vakola [34] found that organizational and behaviour change are not well implemented unless there is a positive reaction.

Survey 1 included items which allowed respondents to communicate directly with the content creators and inform design and content of the educational package [28, 35]. Importantly, 98% of those surveyed felt that the educational package they engaged with was highly relevant to their needs and matched their learning style; the latter of which has been associated with improved learning outcomes in educational [36]. Furthermore, 81% of the survey respondents were either somewhat or strongly satisfied with the overall educational package and 88% stated that the educational package was effective at increasing their knowledge and they would recommend it to their colleagues. Interestingly, only 70% of the respondents noted that they would recommend changes to their practice procedures after viewing the educational package. While research suggests that participant reaction to learning is important such measures may not correlate to how much participants learnt, or whether their behaviour changed [37,38,39,40]. For example, a recent study of acupuncturists noted that while the reaction to the training is important, levels 2 (learning) and 3 (behaviour) of the NWKM are the most relevant to determine the success of the program [41].

Level 2—learning

Level two of the NWKM is concerned with determining the degree of learning, knowledge, confidence, and skill acquisition because of training [26, 32, 42, 43]. To assess knowledge, evaluation of the participant’s recall, understanding and application of the learning should be measured [44]. Recall and understanding were assessed using three questions included in each survey and covered a range of topics including the definition of PROMs, categories of PROMs, and the identification of PROMs examples using multiple-choice questions. The results show a significant improvement in knowledge scores between survey 1 and survey 2, which were retained at survey 3, suggesting the intended learning outcomes were achieved and the educational package was successful [45, 46]. Nevertheless, the improvements in knowledge were modest (~ 10%), and while statistically significant, may be insufficient to affect a change in participant behavior and attitude towards PROMs. Despite a recent survey of chiropractors in Australia suggesting that improving clinician understanding of PROMs and why/when to use PROMs would improve utilisation rates, our study, together with others, suggest that knowledge, in and of itself, may not be sufficient to change behavior [47, 48]. Similarly, confidence of the learner, which has been suggested to be a gap between learning and behaviour [49], was also not changed as a result of the educational package. We found no difference in confidence scores between survey 1, and surveys 2 and 3. Research suggests that motivation and confidence are key determinants to create behavioral change [50]. Boyce, Robertson [51] further suggest that building confidence is important if intentions are to be translated into behaviour change. Nevertheless, while confidence may bridge the gap between knowledge and behavior, improving knowledge alone seems insufficient to change confidence. Overall, this study suggests that the provision of an online educational package aimed to improve knowledge of PROMs in chiropractors had no effect on their confidence in understanding what PROMs were available or when and/or how to use them.

Level 3—behaviour

Level 3 of the NWKM evaluates to what extent the newly acquired knowledge or skills have been practically applied [42, 52]. This level also assesses the degree to which the participants apply what they learned and modified their behaviour based on the intervention [53]. In the current study we evaluated behavior change using questions related to two subthemes: (1) attitude toward PROMs; and (2) frequency of PROM utilisation. Participant’s attitude toward PROMs were evaluated using 16 questions, which covered a variety of attitudinal aspects relating to the importance and influence of PROMs in clinical practice. Of the 16 questions, only one question was significantly different after the provision of the educational package. Participants more strongly agreed with the statement: ‘Health professionals should use patient reported outcome measures to monitor treatment outcomes using valid and reliable tools’ in survey 2, compared to survey 1. The educational package may have highlighted the arrival of the “era of accountability” with more pressure placed on health care providers to provide treatment evidence [54]. The last decade has seen an increasing focus and interest not only in a patients symptoms, but also in documenting the patient experience and their interactions with healthcare providers [55]. Nevertheless, the overall results do not support a change in clinician attitudes toward PROMs after receiving the educational package. Whilst difficult to determine, it is likely that respondents already had a very favorable view towards PROMs and so there may have been a ceiling effect limiting the impact of the educational package on participant attitudes. Previous studies do show that most chiropractors have a favorable view of PROMs [5, 15], however, these views do not necessarily correlate to higher levels of usage [15].

Although 70% of the respondents reported that they would recommend changes to their practice procedures after viewing the educational package, the data did not support that those changes occurred. Furthermore, despite most respondents acknowledging the importance of PROMs and the need for clinical change to include PROMs, the clinical implementation did not increase after receiving the educational package. Except for health-related PROMs, the reported frequency of PROM utilisation did not change from survey 1 to surveys 2 and 3. We found a small but significant increase in the reported use of health-related PROMs from survey 1 to survey 3. Although there was a similar trend toward increased use of functional PROMs, these differences were not significant. These improvements may be explained, at least in part, by two factors. First, the current study, as well as previous studies [8], show that functional and health-related PROMs are less frequently used by chiropractors in clinical practice, compared to pain-related PROMs, and so their frequency of use may be more likely changed with an educational package that promotes awareness of these PROMs. Secondly, and further to this point, the educational package was specifically designed to address these practice gaps by providing a comprehensive summary of numerous health- and functional-related PROMs relevant to each body region. It is possible that much of this content may have been perceived as ‘new and novel’ and therefore may be more likely to elicit a measurable change in behaviour, compared to the more commonly used pain-related PROMs.

Overall, the educational package did not affect a measurable change in participant’s confidence, attitude, or frequency of PROM utilisation. Although we used the KTA framework to design the content of the educational package, we did not include any specific strategies to promote translation, but rather relied on knowledge to drive behavior change. Clearly, knowledge alone is not sufficient to affect behavioral change. Although knowledge has been consistently identified as a barrier/facilitator to PROM utilisation in healthcare, multiple other barriers exist which were not addressed in the current study and may be more influential in changing behavior. Time was the most reported barrier to PROM usage by chiropractors [5, 8]. Although the educational package provided links to source material there were no specific implementation strategies included in the package that may have eased the time burden of using PROMs in clinical practice. The provision of all relevant PROMs to chiropractic care, including background information, via a single electronic application may be a future strategy to better align PROM knowledge with PROM access, and reduce the time the burden of accessing PROMs in a clinical setting.

It is possible that if this study had continued over a longer period of time, this may have increased the likelihood of measuring behavioral change. Axtell, Maitlis [56] found that the amount of learning transferred into practice one month post intervention was a strong predictor for learning transferred one year post intervention. According to Kirkpatrick D [57] the most accurate time to evaluate behaviour change is at least three months after the training was applied, although literature suggests that behaviour change may not occur until six months, to enable learners sufficient time to put their new skills into practice [58]. Therefore, a longer-term follow-up (up to 12 months) may have been needed to identify whether the short-term changes in knowledge changed behaviour.

Future directions

The findings from this study have implications for future research in this field, specific to clinical implementation and for future contributions to literature.

A modification to the current education package could include the addition of a module specific to implementation of PROMs in clinical practice, e.g. the addition of a multi-modal (video, text and audio files) module outlining the exact steps to apply PROMs would be warranted. Using more technology-based apps, electronic links to PROMs (copyright pending) would possibly assist chiropractors by making PROMs more accessible and less time consuming. The education package could be offered to Chiropractors through chiropractic professional organisations/governing bodies as continuing professional development and ensure chiropractors are receiving the most up to date information. Additionally, the education package could be embedded into Chiropractic universities curriculum to ensure graduating Chiropractors are proficient in the implementation of PROMs at the beginning of their clinical careers.

Limitations

In addition to the short follow-up period, there are several other limitations to this study. First, the low response rate to the cohort survey may limit the strength and generalisability of the findings. Although online surveys are a common research method, their success, particularly amongst health care professionals has been questioned [59]. Cunningham, Quan [59] imply that low survey response rates are common within the health care profession. Another limitation is bias, which is an inherent issue in the design of surveys [60]. Although the authors aimed to reduce selection bias and population bias by inviting all members of Australia’s main chiropractic associations to participate, an unintentional responder bias may have occurred. Due to the low response rate, the survey respondents may not be representative of the entire chiropractic profession in Australia. A further limitation of this study was that there was no control group. Additionally, given the substantial drop-out between survey 1 and survey 2, it is possible the participants who completed all three surveys may have already had favorable views of PROMs, and relatively high utilisation rates, and so there may have been a potential ceiling effect associated with the educational package that affected changes in confidence, attitude, and frequency of PROM usage.

Conclusion

An online educational package delivered to chiropractors in Australia was effective at improving knowledge of PROMs, including what PROMs are and why, how, and when they should be used in clinical practice. Despite modest improvements in knowledge, which were retained 3-months after the educational package was provided, there was no evidence that participants confidence, attitudes, or frequency of PROM use changed because of the intervention. The study findings suggest that knowledge of PROMs alone may be insufficient to change the frequency of their use in clinical practice by chiropractors in Australia. However, caution is advised when interpreting these findings due to the low participant response and completion rate with the potential for selection bias and the inability to generalise the results.

Data availability

Not applicable.

References

  1. Santana MJ, Haverman L, Absolom K, Takeuchi E, Feeny D, Grootenhuis M, et al. Training clinicians in how to use patient-reported outcome measures in routine clinical practice. Qual Life Res. 2015;24(7):1707–18. doi:https://doi.org/10.1007/s11136-014-0903-5.

    Article  PubMed  Google Scholar 

  2. Thompson L, Bekelman D, Allen L, Peterson P. Patient-Reported Outcomes in Heart Failure: Existing Measures and Future Uses. Curr Heart Fail Rep. 2015;12(3):236–46. doi:https://doi.org/10.1007/s11897-015-0253-9.

    Article  PubMed  Google Scholar 

  3. Wolpert M. Uses and Abuses of Patient Reported Outcome Measures (PROMs): Potential Iatrogenic Impact of PROMs Implementation and How It Can Be Mitigated. Adm Policy Mental Health Mental Health Serv Res. 2014;41(2):141–5. doi:https://doi.org/10.1007/s10488-013-0509-1.

    Article  Google Scholar 

  4. Clohesy N, Schneiders A, Eaton S. Utilization of Low Back Pain Patient Reported Outcome Measures Within Chiropractic Literature: A Descriptive Review. J Manip Physiol Ther. 2018;41(7):628–39. doi:https://doi.org/10.1016/j.jmpt.2017.11.007.

    Article  Google Scholar 

  5. Holmes M, Bishop F, Newell D, Field J, Lewith G. Chiropractors’ views on the use of patient-reported outcome measures in clinical practice: a qualitative study. Chiropr Man Therap. 2018;26:50. doi:https://doi.org/10.1186/s12998-018-0219-6.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Weldring T, Smith SMS. Patient-Reported Outcomes (PROs) and Patient-Reported Outcome Measures (PROMs). Health Serv Insights. 2013;6:61–8. doi:https://doi.org/10.4137/HSI.S11093.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Yeomans S, Liebenseon C. Applying outcome measures. JNMS. 1997;5.

  8. Clohesy N, Schneiders A. A preliminary investigation examining patient reported outcome measures for low back pain and utilisation amongst chiropractors in Australia: facilitators and barriers to clinical implementation. Chiropr Man Therap. 2018;26:38. doi:https://doi.org/10.1186/s12998-018-0208-9.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Brinkman M, Barten D, Pisters M, Verheij R. Current use of PROMs and factors associated with their use in patients with nonspecific low back pain. Learn Health Syst. 2019;3(4):e10194. doi:https://doi.org/10.1002/lrh2.10194.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Duncan E, Murray J. The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review. BMC Health Serv Res. 2012;12(1):96. doi:https://doi.org/10.1186/1472-6963-12-96.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract. 1999;5(4):401–16. doi:https://doi.org/10.1046/j.1365-2753.1999.00209.x.

    Article  CAS  PubMed  Google Scholar 

  12. Antunes B, Harding R, Higginson I. Implementing patient-reported outcome measures in palliative care clinical practice: a systematic review of facilitators and barriers. Palliat Med. 2014;28(2):158–75. doi:https://doi.org/10.1177/0269216313491619.

    Article  PubMed  Google Scholar 

  13. Chapman-Smith D. The Role of Patient Questionnaires. The Chiropractic Report. 2012;26(5).

  14. Eilayyan O, Visca R, Zidarov D, Ware P, Bussières A, Ahmed S. Developing theory-informed knowledge translation strategies to facilitate the use of patient-reported outcome measures in interdisciplinary low back pain clinical practices in Quebec: mixed methods study. BMC Health Serv Res. 2020;20(1):789. doi:https://doi.org/10.1186/s12913-020-05616-5.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Clohesy N, Schneiders A, Barbery G, Obst S. Clinicians’ perceived value and demographic factors that predict the utilisation of patient reported outcome measures for low back pain amongst chiropractors in Australia. Chiropr Man Therap. 2021;29(1):42. doi:https://doi.org/10.1186/s12998-021-00399-w.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Nilsson E, Orwelius L, Kristenson M. Patient-reported outcomes in the Swedish National Quality Registers. J Intern Med. 2016;279(2):141–53. doi:https://doi.org/10.1111/joim.12409.

    Article  CAS  PubMed  Google Scholar 

  17. Lynch E, Mudge A, Knowles S, Kitson A, Hunter S, Harvey G. “There is nothing so practical as a good theory”: a pragmatic guide for selecting theoretical approaches for implementation projects. BMC Health Serv Res. 2018;18(1):857. doi:https://doi.org/10.1186/s12913-018-3671-z.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Field B, Booth A, Ilott I, Gerrish K. Using the Knowledge to Action Framework in practice: a citation analysis and systematic review. Implement Sci. 2014;9(1):172. doi:https://doi.org/10.1186/s13012-014-0172-2.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Davison C, Ndumbe-Eyoh C. C. Critical examination of knowledge to action models and implications for promoting health equity. Int J Equity Health. 2015;14(49). doi:https://doi.org/10.1186/s12939-015-0178-7.

  20. Moore J, Virva R, Henderson C, Lenca L, Butzer J, Lovell L, et al. Applying the Knowledge-to-Action Framework to Implement Gait and Balance Assessments in Inpatient Stroke Rehabilitation. Arch Phys Med Rehabil. 2020;103(7S):230-54. doi:https://doi.org/10.1016/j.apmr.2020.10.133.

    Article  Google Scholar 

  21. Graham I, Tetroe J. Some theoretical underpinnings of knowledge translation. Acad Emerg Med. 2007;14(11):936–41. doi:https://doi.org/10.1197/j.aem.2007.07.004.

    Article  PubMed  Google Scholar 

  22. Graham I, Logan J, Harrison M, Straus S, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13–24. doi:https://doi.org/10.1002/chp.47.

    Article  PubMed  Google Scholar 

  23. Sinden K, MacDermid J. Does the knowledge-to-action (KTA) framework facilitate physical demands analysis development for firefighter injury management and return-to-work planning? J Occup Rehabil. 2014;24(1):146–59. doi:https://doi.org/10.1007/s10926-013-9442-0.

    Article  PubMed  Google Scholar 

  24. Kirkpatrick W, Kirkpatrick J. Four levels of training evaluation. United States, ATD Press; 2016.

  25. Gill S, Sharma G. Evaluation of Vocational Training Program from the Trainees’ Perspective: An Empirical Study. Pac Bus Rev Int. 2013;6(5):35–43.

    Google Scholar 

  26. Rouse D. Employing Kirkpatrick’s evaluation framework to determine the effectiveness of health information management courses and programs. Perspect Health Inf Manag. 2011;8.

  27. AHPRA. Chiropractic board statistics 2020. https://www.chiropracticboard.gov.au/About-the-Board/Annual-report.aspx. Accessed 25 March 2022.

  28. Bradwell P, Marr S. Making the most of collaboration: an international survey of public service co-design, Demos Report 23. London: 2008.

  29. O’Cathain A, Thomas KJ. “Any other comments?“ Open questions on questionnaires – a bane or a bonus to research? BMC Med Res Methodol. 2004;4(1):25. doi:https://doi.org/10.1186/1471-2288-4-25.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Smith A, Ling P, Hill D. The Adoption of Multiple Modes of Delivery in Australian Universities. J Univ Teach Learn Pract. 2006;3(2). doi:https://doi.org/10.53761/1.3.2.2.

  31. Bland JM, Altman DG. Statistics notes: Cronbach’s alpha. BMJ. 1997;314(7080):572. doi:https://doi.org/10.1136/bmj.314.7080.572.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Cahapay M. International Journal of Assessment Tools in Education. Int J Assess Tools Educ. 2021;8(1):135–44. doi:https://doi.org/10.21449/ijate.856143.

    Article  Google Scholar 

  33. Gökhan Ö. Program Evaluation through Kirkpatrick’s Framework. Pacific Business Review International. 2015;8(1).

  34. Oreg S, Vakola M, Armenakis A. Change Recipients’ Reactions to Organizational Change: A 60-Year Review of Quantitative Studies. J Appl Behav Sci. 2011;47(4):461–524. doi:https://doi.org/10.1177/0021886310396550.

    Article  Google Scholar 

  35. Shahvari Z, Raisi F, Firoozi Khojastefar R, Yahyavi S, Mirsepassi Z. Memos on Sexuality Education Effectiveness; A Qualitative Study in Severe Mental Illness in Iran. Int J Sex Health. 2020;32(3):254–66. doi:https://doi.org/10.1080/19317611.2020.1793047.

    Article  Google Scholar 

  36. Baykan Z, Naçar M. Learning styles of first-year medical students attending Erciyes University in Kayseri, Turkey. Adv Physiol Educ. 2007;31(2):158–60. doi:https://doi.org/10.1152/advan.00043.2006.

    Article  PubMed  Google Scholar 

  37. Alliger G, Janak E. Kirkpatrick’s levels of training criteria: Thirty years later. Pers Psychol. 1989;42(2):331–42. https://doi.org/10.1111/j.1744-6570.1989.tb00661.x. doi: doi/.

    Article  Google Scholar 

  38. Arthur W, Bennett W, Edens P, Bell S. Effectiveness of Training in Organizations. J Appl Psychol. 2003;88(2):234–45. doi:https://doi.org/10.1037/0021-9010.88.2.234.

    Article  PubMed  Google Scholar 

  39. Colquitt J, LePine J, Noe R. Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. J Appl Psychol. 2000;85(5):678–707. doi:https://doi.org/10.1037//0021-9010.85.5.678.

    Article  CAS  PubMed  Google Scholar 

  40. Tannenbaum SI, Yukl G. Training and development in work organizations. Ann Rev Psychol. 1992;43:399–441. doi:https://doi.org/10.1146/annurev.ps.43.020192.002151.

    Article  Google Scholar 

  41. Liao S-C, Hsu S-Y. Evaluating A, Continuing Medical Education Program: New World Kirkpatrick Model Approach. Int J Manage Econ Social Sci. 2019;8. doi:https://doi.org/10.32327/IJMESS.8.4.2019.17.

  42. Phillips J, Phillips P. Handbook of Training Evaluation and Measurement Methods (4th). London: 2016. doi: https://doi.org/10.4324/9781315757230.

  43. Allen LM, Hay M, Palermo C. Evaluation in health professions education—Is measuring outcomes enough? Med Educ. 2022;56(1):127–36. doi:https://doi.org/10.1111/medu.14654.

    Article  PubMed  Google Scholar 

  44. Kirkpatrick J, Kirkpatrick W. Introduction to the New World Kirkpatrick Model. 2019. https://www.kirkpatrickpartners.com/wp-content/uploads/2021/11/Introduction-to-the-Kirkpatrick-New-World-Model.pdf. Accessed on 6 January 2022.

  45. Deodhar M, Powdwal S. Impact of continuing education programs (CEPs) on LIS professionals in academic libraries in Mumbai, India. Libr Manage. 2017;38(2/3):117–30.

    Article  Google Scholar 

  46. Mohamed R, Ariyani A, Alias S. Evaluating the Effectiveness of a Training Program Using the Four Level Kirkpatrick Model in the Banking Sector in Malaysia: In: 3rd International Conference on Business and Economic Research, 12-13 March. Bandung, Indonesia2012.

  47. Arlinghaus KR, Johnston CA. Advocating for Behavior Change With Education. Am J Lifestyle Med. 2017;12(2):113–6. doi:https://doi.org/10.1177/1559827617745479.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Worsley A. Nutrition knowledge and food consumption: can nutrition knowledge change food behaviour? Asia Pacific journal of clinical nutrition. 2002(11):579–85. doi: https://doi.org/10.1046/j.1440-6047.11.supp3.7.x.

  49. Piryani RM, Dhungana GP, Piryani S, Sharma Neupane M. Evaluation of teachers training workshop at Kirkpatrick level 1 using retro-pre questionnaire. Adv Med Educ Pract. 2018;9:453–7. doi:https://doi.org/10.2147/AMEP.S154166.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Lubman D, Hall K, Gibbie T. Motivational interviewing techniques. Australian J Gen Practitioners. 2012;41:660–7.

    Google Scholar 

  51. Boyce T, Robertson R, Dixon A. Commissioning and behaviour Change Kicking Bad Habits final report. 2008.

  52. Baldwin T, Ford J. Transfer of Training: A Review and Directions of Future Research. Pers Psychol. 1988;41(1):63–105. doi:https://doi.org/10.1111/j.1744-6570.1988.tb00632.x.

    Article  Google Scholar 

  53. Cheng E, Hampson I. Transfer of training: A review and new insights. J Manage Reviews. 2008;10(4):327–41. doi:https://doi.org/10.1111/j.1468-2370.2007.00230.x.

    Article  Google Scholar 

  54. Relman A. Assessment and accountability: the third revolution in medical care. 2012. 87-90. doi: https://doi.org/10.1056/NEJM198811033191810.

  55. Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open. 2013;3(1):e001570. doi:https://doi.org/10.1136/bmjopen-2012-001570.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Axtell CM, Maitlis S, Yearta SK. Predicting immediate and longer-term transfer of training. Personnel Rev. 1997;26(3):201–13. doi:https://doi.org/10.1108/00483489710161413.

    Article  Google Scholar 

  57. Kirkpatrick D. Great Ideas Revisited. Techniques for Evaluating Training Programs. Revisiting Kirkpatrick’s Four-Level Model. Training and Development. 1996;50(1).

  58. May G, Kahnweiler W. The effect of a mastery practiice design on learning and transfer in behavior modelling training Personnel psychology. 2000;53(2):353–73. doi: https://doi.org/10.1111/j.1744-6570.2000.tb00205.x.

  59. Cunningham CT, Quan H, Hemmelgarn B, Noseworthy T, Beck CA, Dixon E, et al. Exploring physician specialist response rates to web-based surveys. BMC Med Res Methodol. 2015;15. doi:https://doi.org/10.1186/s12874-015-0016-z.

  60. Choi B, Pak A. A catalog of biases in questionnaires. Prev Chronic Dis. 2005;2.

Download references

Acknowledgements

Not applicable.

Funding

Not applicable.

Author information

Authors and Affiliations

  1. College of Health Sciences, School of Health, Medical and Applied Sciences, Central Queensland University, Bundaberg, QLD, Australia

    Natalie Clohesy, Anthony Schneiders & Steven Obst

  2. School of Psychology, Griffith University, South Brisbane, QLD, Australia

    Gaery Barbery

Authors
  1. Natalie Clohesy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anthony Schneiders
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gaery Barbery
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Steven Obst
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Concept development: NCC, SO, AGS. Design (planned the methods to generate the results): NCC, SO, AGS, GB. Ethics submission/approval NCC, AGS. Analysis and interpretation of data: SO, NCC, AGS, GB. Manuscript preparation NCC, SO, AGS, GB. Critical review: SO, AGS, GB. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Natalie Clohesy.

Ethics declarations

Ethical approval and consent to participate

Full ethical approval was granted by the Human Research Ethics Committee of Central Queensland University, reference number 0000022391.

Consent for publication

Not applicable.

Competing interests

The authors declare they have no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Clohesy, N., Schneiders, A., Barbery, G. et al. The impact of a targeted education package on the knowledge, attitudes, and utilisation of patient reported outcome measures amongst chiropractors in Australia. Chiropr Man Therap 30, 44 (2022). https://doi.org/10.1186/s12998-022-00450-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12998-022-00450-4

Keywords

Chiropractic & Manual Therapies

ISSN: 2045-709X