Quality of reporting in chiropractic mixed methods research: a methodological review protocol
Chiropractic & Manual Therapies volume 29, Article number: 35 (2021)
Mixed methods designs are increasingly used in health care research to enrich findings. However, little is known about the frequency of use of this methodology in chiropractic research, or the quality of reporting among chiropractic studies using mixed methods.
To quantify the use and quality of mixed methods in chiropractic research, and explore the association of study characteristics (e.g., authorship, expertise, journal impact factor, country and year of publication) with reporting quality.
We will conduct a systematic search of MEDLINE, EMBASE, CINAHL, and the Index to Chiropractic Literature to identify all chiropractic mixed methods studies published from inception of each database to December 31, 2020. Articles reporting the use of both qualitative and quantitative methods, or mixed qualitative methods, will be included. Pairs of reviewers will perform article screening, data extraction, risk of bias with the Mixed Methods Appraisal Tool (MMAT), and appraisal of reporting quality using the Good Reporting of A Mixed Methods Study (GRAMMS) guideline. We will explore the correlation between GRAMMS and MMAT scores, and construct generalized estimating equations to explore factors associated with reporting quality.
This will be the first methodological review to examine the reporting quality of published mixed methods studies involving chiropractic research. The results of our review will inform opportunities to improve reporting in chiropractic mixed methods studies. Our results will be disseminated in a peer-reviewed publication and presented publicly at conferences and as part of a doctoral thesis.
Mixed methods designs, which include qualitative and quantitative methods, have been increasingly used in health care research to enrich findings [1, 2]. The explicit mixing or linking of qualitative and quantitative components within a mixed methods study allows researchers to answer questions with a greater breadth and depth of understanding than would be possible with only one methodology alone [1, 2]. This integration of methods, which is central to mixed methods research [1,2,3], is distinct from “multi-method” research, where investigators use quantitative and qualitative methods in a study without linking or integrating the two components (e.g., adding a series of open-ended questions to the end of a quantitative survey). An integrated mixed methods approach is particularly useful for investigating complex, multilevel programs and interventions [3,4,5,6], and is therefore well-suited to address research problems involving knowledge translation, program evaluations, or comparisons of therapeutic interventions within the chiropractic profession. However, little is known about the frequency of use of this methodology in chiropractic research, or the quality of reporting among chiropractic studies using mixed methods.
Researchers conducting mixed methods studies need to make decisions regarding the sequencing or timing of the qualitative and quantitative components (i.e., concurrent or sequential data collection and analysis), as well as the priority or “emphasis” that will be given to each method [1, 2, 7]. For example, Stuber et al.  used a sequential, quantitative dominant  mixed methods design where supplemental interviews and focus groups (qualitative) were conducted to help explain initial survey results (quantitative) in a study of patient perceptions toward patient-centered care in chiropractic practice. In addition to the mixing or linking of two unique research paradigms—qualitative and quantitative—mixed methods studies may also involve data transformation (i.e., converting qualitative data into quantitative data [‘quantitizing’] or vice versa [‘qualitizing’] in order to further integrate the data ). As such, mixed methods studies can become complex investigations that require additional time and resources, and a team of researchers with expertise in quantitative, qualitative, and mixed methodologies [1, 2].
Previous methodological reviews have examined the mixed methods literature [9,10,11,12,13,14] and highlighted areas for improvement in the quality of reporting. For instance, a review of health services research  found that authors of mixed methods studies in health services research typically did not describe or justify the need for a mixed methods design, or integrate data and findings from the individual quantitative and qualitative components. This lack of integration inhibits new insights from being generated within mixed methods studies (i.e., beyond the results obtained from the two separate components), thereby limiting the methodological potential of this research strategy [1,2,3]. Currently, the quality of reporting in chiropractic mixed methods research is unknown as no methodological reviews have examined this literature.
The primary purpose of our methodological review is to examine the quality of reporting and characteristics (e.g., authorship, expertise, journal impact factor, country and year of publication) of chiropractic mixed methods studies. In addition, we will assess the risk of bias of included articles and examine the correlation between reporting quality and risk of bias. We will also use multivariable regression analysis to explore possible factors influencing reporting quality. Our findings will be of interest to educators, researchers, publishers, editors, and consumers of the chiropractic and allied health literature.
This methodological review was registered with the Open Science Framework (OSF) on December 14, 2020 (https://osf.io/).
A systematic search of multiple databases including MEDLINE, EMBASE, CINAHL, and the Index to Chiropractic Literature (ICL) will be conducted to identify all published chiropractic mixed methods articles, without time limits to December 31, 2020. Our search strategy was developed by an academic librarian (RJC) (Fig. 1). The reference lists of eligible articles will also be hand-searched, and contact will be made with experts in the chiropractic mixed methods field, to identify additional eligible studies not identified in our electronic database searches. We will update our literature search if more than six months elapses between the time of our database searches and submission of results.
Articles that meet the following criteria will be included: (1) published in a peer-reviewed journal; (2) written in the English language; (3) authored by one or more chiropractic researchers (i.e., someone with chiropractic credentials or affiliation with a chiropractic educational institution); (4) involved any type of chiropractic intervention (e.g., therapeutic, educational) or non-intervention research (e.g., policy, scope of practice); (5) reported a mixed methods approach (i.e., the use of both qualitative and quantitative methods [1, 2], or mixed qualitative methods , in the same research article); and (6) reports primary research. For the analysis, studies that report quantitative and qualitative results in separate papers will be combined and considered as one study. In addition, ‘mixed’ surveys (i.e., those with both closed- and open-ended questions) will only be included when the use of “mixed methods” was explicitly stated in the title or abstract. Protocols, letters, editorials, commentaries, books and book chapters, grey literature (e.g., conference proceedings, abstracts, lectures, dissertations or unpublished manuscripts), and secondary sources of evidence, e.g., clinical practice guidelines or any type of review article will be excluded.
Each title and abstract retrieved from the database searches, as well as those identified through bibliographic-searching and contact with experts, will be screened by pairs of independent reviewers. Online systematic review software (DistillerSR, Evidence Partners, Ottawa, Canada; https://www.evidencepartners.com) will be used to facilitate the literature screening process. We will resolve discrepancies on decisions to include or exclude potentially eligible articles by discussion to achieve consensus or, when not possible, adjudication by a third reviewer. We will also complete full-text screening independently and in duplicate, with discrepancies resolved as previously described. Agreement for full-text screening will be assessed using the kappa (κ) statistic, and the strength of agreement will be interpreted as: poor (κ ≤ 0.2), fair (0.21 ≤ κ ≤ 0.4), moderate (0.41 ≤ κ ≤ 0.6), substantial (0.61 ≤ κ ≤ 0.8), or almost perfect (κ > 0.8) .
Data collection process and assessment of reporting quality
Pairs of reviewers will independently extract data and assess reporting quality of included articles using standardized, pilot-tested data extraction forms. Discrepancies will be resolved by consensus or adjudication by a third reviewer. If a reviewer was an author on an included article, the study will be reviewed by another member of the research team. The following six items from the Good Reporting of A Mixed Methods Study (GRAMMS) criteria  will be used to assess reporting quality: (1) describes the justification for using a mixed methods approach to the research question; (2) describes the mixed methods design (i.e., the purpose, priority, and sequence of methods); (3) describes each method in terms of its sampling, data collection, and analysis; (4) describes the integration of the quantitative and qualitative components (i.e., where integration has occurred, how it has occurred, and who among the research team has participated in it); (5) describes any limitation of one method associated with the presence of the other method; and (6) describes any insights gained from mixing or integrating methods. An author checklist of the GRAMMS criteria is provided in Additional file 1. Selected articles will be evaluated on an item-by-item basis, with reviewers rating each item as “yes” (if the item was reported), “yes, but improvements are possible” (if the item was incompletely reported), or “no” (if the item was not reported). For the analysis (see ‘Synthesis of Results’ below), we will sum the scores for each item (1 = “yes”; 0.5 = “yes, but improvements are possible”; 0 = “no”).
The following information will also be extracted from all eligible articles: (1) the first author, (2) year of publication, (3) journal name, (4) number of authors, (5) country where the study was conducted (or, if not available, country of residence of the corresponding author), (6) type of mixed methods design, and (7) whether the list of authors included a mixed methodologist (i.e., graduate-level training or expertise in mixed methods research). We will determine methodological expertise by examining the authors’ published details and affiliations, along with any other information given in the article that explicitly describes an author as having expertise in this area. In addition, the impact factor will be obtained at the year of publication for each journal, either directly from the journal’s website, or from the Journal Citation Report by Thompson Reuters on the Web of Science (https://jcr.clarivate.com/). Journals without an available impact factor will be assigned a value of zero .
Risk of bias of individual studies
We will assess risk of bias of included articles with the Mixed Methods Appraisal Tool (MMAT) (version 2011), which has been validated for systematic reviews of mixed studies (i.e., qualitative, quantitative, and mixed methods studies) [18,19,20] (Additional file 2). The MMAT is comprised of 11 items in three sections, including: (1) four items for appraising the qualitative component of a mixed methods study or a primary qualitative study; (2) four items for appraising the quantitative component of a mixed methods study or a primary quantitative study (i.e., randomized controlled, non-randomized, or descriptive); and (3) three items for appraising the mixed methods component of a mixed methods study. Pairs of reviewers will appraise each article according to the MMAT, using a similar process as described above.
Synthesis of results
We will assess all included articles for completeness of reporting on the six items of the GRAMMS instrument (i.e., 0–6 items, where 0 = no reporting of any items and 6 = complete reporting of all six items), and data will be presented as the mean (with standard deviation) or median (with inter-quartile range) number of quality items reported depending on whether the distribution is normal. Risk of bias scores obtained from the MMAT (i.e., 0–11 items) will also be summarized and presented. In addition, we will generate frequencies for individual reporting items on the GRAMMS instrument (i.e., the number of articles reporting a particular item) (Table 1) and describe extracted study characteristics as counts and percentages.
To examine correlation between the GRAMMS and MMAT instruments, we will compare the item scores for each article between the two instruments (i.e., 0–6 for GRAMMS, 0–11 for MMAT), using Pearson’s r or Spearman’s ρ for parametric and non-parametric distributions, respectively. Data distributions will be analyzed for normality by visual inspection of histograms, probability plots, and quantile–quantile plots, and then confirmed with the Kolmogorov–Smirnov test. Based on previous findings from research on randomized controlled trials and adherence to CONSORT guidelines [21,22,23], we predict that chiropractic mixed methods studies with a lower risk of bias (i.e., higher MMAT score) will be correlated with higher reporting quality.
We will use generalized estimating equations (GEEs) to explore the association between reporting quality and article characteristics (i.e., publication date, authorship, and journal impact factor). We will model the dependent variable as the number of GRAMMS items for which complete reporting occurred (maximum value of six) divided by the total number of GRAMMS items (six). The dependent variable will be regressed on the year of article publication (post-2009 versus pre-2009), journal impact factor (higher versus lower), number of co-authors (higher versus lower), and inclusion of an author with training in mixed methods (yes versus no) (Table 2). These factors have previously been shown to be associated with methodological reporting quality [24, 25].
We will employ a binomial distribution and logit link function in our GEE models to generate crude and adjusted odds ratios, with corresponding 95% confidence intervals and p-values. Goodness-of-fit will be assessed by comparing our model’s deviance to its degrees of freedom and by examining the residual plot. We will address over- or under-dispersion by re-running the model with a scale parameter calculated by dividing the deviance by its degrees of freedom. We will also incorporate the journal name as a grouping factor to account for potential similarity or clustering of articles published in the same journal.
We hypothesize that studies published since 2009 (i.e., ≥ 1 year after publication of the GRAMMS criteria ), studies published in higher impact journals, those with a greater number of authors, and those that included a mixed methodologist, will be associated with higher reporting quality. A minimum sample of 40 chiropractic mixed methods articles will be needed to guard against over-fitting of our regression model (i.e., minimum of 10 observations per independent variable) . Variance inflation factors (VIFs) will also be explored to assess for multicollinearity among independent variables . If we detect multicollinearity between two or more variables (i.e., VIFs ≥ 10), we will remove the variable(s) that we deem of lower importance . The two-sided statistical significance level (\(\alpha\)) will be 5%, and all data and comparative analyses will be performed using SPSS v26.0 (IBM SPSS Statistics ©).
Our review will be reported in accordance with an adapted version of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for meta-epidemiological research .
This study is a methodological literature review of previously published articles and does not require ethics approval.
This will be the first methodological review to examine the reporting quality of published mixed methods studies involving chiropractic research. The results of this review are important because they will inform areas for improvement regarding reporting of chiropractic mixed methods studies. This may lead to important changes in the quality of evidence generated from these studies, with consequent implications for chiropractic policy, research, editorial, and clinical practice.
Mixed methods research can serve as a powerful tool for investigating complex therapeutic interventions, educational programs or knowledge translation strategies for improving clinical practice [1,2,3,4,5,6]. However, mixed methods research also requires specialized skills in qualitative and quantitative data integration and analysis [1,2,3]. Therefore, chiropractors conducting these types of studies should undertake graduate-level training in mixed methods research or, at a minimum, collaborate with researchers possessing mixed methodological expertise.
Previous research has shown that certain authorship factors, such as methodological expertise, as well as having multiple authors on a research project, can significantly improve the reporting quality and conduct of studies [24, 25, 30]. For example, a study on reporting quality among systematic reviews  found that including methodologists on research teams was associated with greater concordance with reporting guidelines. Likewise, more recent articles or those published in journals with higher impact factors tend to meet better reporting standards [24, 25]. Yet, no research to date has investigated the influence of these or other factors on the quality of reporting among published chiropractic mixed methods studies.
Strengths and limitations
Our methodological review has several strengths. First, we will conduct a comprehensive and exhaustive search to identify all eligible studies involving chiropractic mixed methods research. To reduce errors in our methodological procedures, we will perform article screening, data extraction and quality appraisals in duplicate. Moreover, statistical adjustments will be applied at the analysis stage to control for between-group differences when exploring associations, and GEE modelling will be used to account for hierarchical clustering of articles within journals. For our regression model, we have prespecified the anticipated direction of association for each independent variable a priori to give reassurances that associations are unlikely to be spurious if detected. A limitation of this review is we will exclude non-English publications, which may lead to selection bias.
Dissemination of our review will occur via a peer-reviewed publication and a conference presentation. The review findings will also be presented publicly and defended as part of a doctoral thesis.
Availability of data and materials
The datasets to be used and/or analyzed for the current study will be available from the corresponding author on reasonable request.
Cumulative Index to Nursing and Allied Health Literature
Consolidated Standards of Reporting Trials
Excerpta Medica Database
Generalized Estimating Equation
Good Reporting of A Mixed Methods Study
Index to Chiropractic Literature
Medical Literature Analysis and Retrieval System Online
Mixed Methods Appraisal Tool
Open Science Framework
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Statistical Program for the Social Sciences
Variance Inflation Factor
Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 3rd ed. Thousand Oaks: Sage; 2018.
NIH Office of Behavioral and Social Sciences. Best practices for mixed methods research in the health sciences. 2nd ed. Bethesda: National Institutes of Health; 2018. https://www.obssr.od.nih.gov/wp-content/uploads/2018/01/Best-Practices-for-Mixed-Methods-Research-in-the-Health-Sciences-2018-01-25.pdf. Accessed 22 Dec 2020.
Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs–principles and practices. Health Serv Res. 2013;48(6 Pt 2):2134–56.
Oakley A, Strange, V, Bonell C, Allen E, Stephenson J, RIPPLE Study Team. Process evaluation in randomised controlled trials of complex interventions. BMJ. 2006;332(7538):413–6.
Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ. 2009;339:b3496.
Drabble SJ, Ocathain A. Moving from randomized controlled trials to mixed methods intervention evaluations (part 3). In: Nagy Hesse-Biber S, Burke Johnson R, editors. The Oxford handbook of multimethod and mixed methods research inquiry. Oxford: Oxford University Press; 2018.
Leech NL, Onwuegbuzie AJ. A typology of mixed methods research designs. Qual Quant. 2009;43(2):265–75.
Stuber KJ, Langweiler M, Mior S, McCarthy PW. Assessing patient-centered care in patients with chronic health conditions attending chiropractic practice: protocol for a mixed-methods study. Chiropr Man Ther. 2016;24:15.
O’Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008;13(2):92–8.
Bishop FL, Holmes MM. Mixed methods in CAM research: a systematic review of studies published in 2012. Evid Based Complement Alternat Med. 2013;2013:187365.
Brown KM, Elliott SJ, Leatherdale ST, Robertson-Wilson J. Searching for rigour in the reporting of mixed methods population health research: a methodological review. Health Educ Res. 2015;30(6):811–39.
Pluye P, Bengoechea EG, Granikov V, Kaur N, Tang D. A world of possibilities in mixed methods: review of the combinations of strategies used to integrate the phases, results and quantitative and qualitative data. Int J Mult Res Approaches. 2018;10(1):41–56.
Kaur N, Vedel I, El Sherif R, Pluye P. Practical mixed methods strategies used to integrate qualitative and quantitative methods in community-based primary health care research. Fam Pract. 2019;36(5):666–71.
Fàbregues S, Hong QN, Escalante-Barrios EL, Guetterman TC, Meneses J, Fetters MD. A methodological review of mixed methods research in palliative and end-of-life care (2014–2019). Int J Environ Res Public Health. 2020;17(11):3853.
Morse JM. Simultaneous and sequential qualitative mixed method designs. Qual Inq. 2010;16(6):483–91.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74.
Froud R, Bjørkli T, Bright P, Rajendran D, Buchbinder R, Underwood M, Evans D, Eldridge S. The effect of journal impact factor, reporting conflicts, and reporting funding sources, on standardized effect sizes in back pain trials: a systematic review and meta-regression. BMC Musculoskelet Disord. 2015;16:370.
Pluye P, Gagnon M, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews. Int J Nurs Stud. 2009;46(4):529–46.
Pace R, Pluye P, Bartlett G, Macaulay AC, Salsberg J, Jagosh J, Seller R. Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. Int J Nurs Stud. 2012;49(1):47–53.
Pluye P, Robert E, Cargo M, Bartlett G, O’Cathain A, Griffiths F, Boardman F, Gagnon MP, Rousseau MC. Proposal: a mixed methods appraisal tool for systematic mixed studies reviews; 2011. http://mixedmethodsappraisaltoolpublic.pbworks.com. Accessed 16 Feb 2021.
Bath FJ, Owen VE, Bath PM. Quality of full and final publications reporting acute stroke trials: a systematic review. Stroke. 1998;29(10):2203–10.
Liu LQ, Morris PJ, Pengel LHM. Compliance to the CONSORT statement of randomized controlled trials in solid organ transplantation: a 3-year overview. Transpl Int. 2013;26(3):300–6.
McCormick F, Cvetanovich GL, Kim JM, Harris JD, Gupta AK, Abrams GD, Romeo AA, Provencher MT. An assessment of the quality of rotator cuff randomized controlled trials: utilizing the Jadad score and CONSORT criteria. J Shoulder Elbow Surg. 2013;22(9):1180–5.
Jin Y, Sanger N, Shams I, Luo C, Shahid H, Li G, Bhatt M, Zielinski L, Bantoto B, Wang M, Abbade LP, Nwosu I, Leenus A, Mbuagbaw L, Maaz M, Chang Y, Sun G, Levine MA, Adachi JD, Thabane L, Samaan Z. Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? A systematic review of reviews: an update. J Multidiscip Healthc. 2018;11:495–510.
Mbuagbaw L, Lawson DO, Puljak L, Allison DB, Thabane L. A tutorial on methodological studies: the what, when, how and why. BMC Med Res Methodol. 2020;20(1):226.
Katz MH. Multivariable analysis: a primer for readers of medical research. Ann Intern Med. 2003;138(8):644–50.
Kleinbaum DG, Kupper LL, Nizam A, Rosenberg ES. Applied regression analysis and other multivariable methods. 5th ed. Boston: Cengage Learning; 2014.
Mulla SM, Maqbool A, Sivananthan L, Lopes LC, Schandelmaier S, Kamaleldin M, Hsu S, Riva JJ, Vandvik PO, Tsoi L, Lam T, Kebrahim S, Johnston BC, Olivieri L, Montoya L, Kunz R, Scheidecker A, Buckley DN, Sessler DI, Guyatt GH, Busse JW. Reporting of IMMPACT-recommended core outcome domains among trials assessing opioids for chronic non-cancer pain. Pain. 2015;156(9):1615–9.
Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–42.
Fleming PS, Koletsi D, Pandis N. Blinded by PRISMA: are systematic reviewers focusing on PRISMA and ignoring other guidelines? PLoS ONE. 2014;9(5):e96407.
This project was supported by a research grant from D’Youville College.
Ethics approval and consent to participate
Consent for publication
PCE is supported by grants from McMaster University, the NCMIC Foundation, and the Canadian Chiropractic Research Foundation outside of the submitted work. All authors have no competing interests to declare.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Emary, P.C., Stuber, K.J., Mbuagbaw, L. et al. Quality of reporting in chiropractic mixed methods research: a methodological review protocol. Chiropr Man Therap 29, 35 (2021). https://doi.org/10.1186/s12998-021-00395-0
- Study protocol
- Mixed methods research
- Reporting quality
- Methodological review