Edited by: Dâmaris Silveira, University of Brasilia, Brazil
Reviewed by: Hong Weng, Wuhan University, China
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Systematic reviews of prospective controlled clinical therapy trials are one of the most important sources of information in modern medicine. Besides the systematic search for and statistical pooling of current clinical trial data for a particular type of therapy, systematic reviews also have the task of appraising the quality of trial results. The quality of trial results may be diminished by low internal trial validity, due to systematic error (bias). A high risk of bias may likely cause the reported trial results to be diverted from the actual true therapeutic effect and thus render it unsuitable for clinical guidance. According to the Cochrane Collaboration, the risk of bias in clinical therapy trials should be assessed using its Risk of Bias tool, Version 2 (RoB 2). However, the tool has been established to have poor inter-rater reliability, with a limited empirical evidence base and described as complex and demanding. Against this background, the composite quality score (CQS) has been developed as a possible alternative trial appraisal tool, characterised by high epistemic rigour, empirical evidence base, inter-rater reliability and ease of use. This article presents the current evidence of the CQS and its limitations.
According to the Cochrane Collaboration, the risk of bias in clinical therapy trials should be assessed using its Risk of Bias tool, Version 2 (RoB 2). The tool consists of 22 signalling questions to five bias domains (randomisation process, deviation from intended interventions, missing outcome data, measurement of outcome, and selection of the reported result) (
The tool was developed between 2015–2019, based on a lengthy process of expert consensus, repeated cycles of discussion, piloting and rewording. The development was further supported by the results of a systematic review of meta-epidemiological studies concerning empirical evidence of bias effect on clinical trial results (
Despite its rigorous development process, the RoB 2 tool has to date been shown to have poor inter-rater reliability (IRR: Fleiss’ Kappa 0.16; 95% CI: 0.08–0.24) and its application has been described as complex and demanding, therefore requiring intensive formal training and the conduct of pilot runs before it may correctly be applied (
During its developmental process, the RoB 2 tool is claimed to have been based on empirical evidence from meta-epidemiological studies (
Besides its poor inter-rater reliability (IRR) and limited evidence base, its complex and demanding application may be the reason why a low adherence to the RoB 2 tool among Cochrane protocols and systematic reviews was observed and why most Cochrane reviewers themselves choose not to use the tool (
In addition to the above, the RoB 2 tool assigns an overall “low-bias risk” status to trials that have been judged to be of “low-bias risk” in all of its five single bias domains, even though any methodological trial error that lies outside its five domains may completely invalidate the reported trial results.
Against this background, the composite quality score (CQS) has been developed as a possible alternative for the appraisal of prospective controlled clinical therapy trials in systematic reviews.
The development of the CQS as a possible trial appraisal tool started in 2019 with the epistemic consideration that regardless of how many trial appraisal criteria a trial has fully complied with, the judgment of such a trial as being of “low-bias risk” cannot be justified but a single methodological error may render a trial to actually be of “high-bias risk.” Accordingly, three evidence-based criteria describing trial design requirements (random allocation sequence generation, allocation sequence concealment, minimum sample size) that are essential (albeit not sufficient) for the trial results to reflect therapeutic truth were established. The criteria were intentionally worded in a simplified and least restrictive manner, based on the rationale that the lower the stringency of an appraisal criterion, the higher the certainty of “high-bias risk” when such criterion is not met (
To explore the applicability of the novel CQS approach for trial appraisal, it was applied in the field of restorative dentistry. Based on a systematic literature search, 683 prospective clinical controlled trial reports were identified out of a total of 14,694 citations. Of these, the bias risk of 99.7% of trials could be appraised and identified as high, despite the low stringency of the applied criteria (
The findings of all three preliminary investigations (
In order to extend the applicability and evidence base of the CQS-1 for all fields of clinical therapy, a systematic review of meta-epidemiological studies was conducted. Based on its results, one new criterion concerning double-blinding was added, and the original criteria concerning random allocation sequence generation and minimum sample size were amended (
Following the extension of the original CQS-1, the CQS-2 was assessed to establish whether its extended format extension would have negatively affected the high inter-rater reliability of the first version. The results of this study showed that CQS extension had no negative effect and that the CQS-2 version was also associated with very high inter-rater reliability (BPC 1.00; 95% CI: 0.94–1.00) and did not statistically differ significantly (
The result of this assessment led to the development of a CQS version (CQS-2B) that included four criteria related to the random allocation to treatment groups, concealment of such allocation, double-blinding and sample size minimum (
CQS-2B appraisal criteria.
| Criterion I | “Randomisation” for allocation to treatment groups is in some form reported in the text |
|---|---|
| Criterion II | Any assurance that the patient allocation to treatment groups according to the random sequence was applied by an independent agent or agency, not otherwise involved in the trial, is in some form reported in the text |
| Criterion III | Double-blinding or the blinding of at least two out of the three groups: trial participants, trial personnel and trial outcome assessors in some form reported in the text |
| Criterion IV | The sample size of any particular treatment group reported in the trial is not less than |
Its application comprised: (i) binary trial report rating per appraisal criterion (scores: 0 = no/invalid/falsified, 1 = yes/corroborated); (ii) multiplication of all criterion scores to an overall appraisal score, (iii) identification of invalid/falsified trial reports based on a zero overall appraisal score.
During its application, several corroboration (C-) levels are recognised. These levels indicate the number of consecutive criteria that a trial has complied with (e.g., level C3 indicates compliance with criterion I–III; level C4 indicates compliance with criterion I–IV, etc.). A corroboration level for a particular trial is reached before one criterion is rated with a 0-score or when all criteria are rated with a 1-score, for example corroboration level C3: criterion I–III = 1-score, criterion IV = 0-score; corroboration level C4: all criteria = 1-score. After a criterion has been rated with a 0-score, the C-level of a trial remains the same even if a following criterion is rated with a 1-score, for example corroboration level C1: criterion I = 1-score, criterion II = 0-score, criterion III = 1-score. No certainty of “low-bias risk” is ascribed to an overall 1-score appraisal result, which indicates only that during the appraisal process no evidence for high-bias risk could be established so far.
It was explored whether systematic review conclusions originally based on Cochrane’s second version of its Risk of Bias tool (RoB 2) would differ when the CQS-2B is used for trial appraisal instead. The results of this study provided justification for the testable hypothesis that trial appraisal using the CQS-2B provides more conservative conclusions based on similar data than trial appraisal using Cochrane’s RoB 2 tool (
Despite its high epistemic rigour, evidence base and inter-rater reliability, the current version of the CQS has certain limitations that need to be taken into account when considering its application in systematic reviews of prospective controlled clinical therapy trials. These include the following items.
As per the design, the CQS-2B is not a tool for verifying low-bias risk. Trials that are awarded 1-scores for all four appraisal criteria may still have serious flaws in other aspects of their methodology. This is a fully intended aspect of the CQS approach in line with the epistemic principles of deductive falsification (
One example is the clinical trial by Sitthisettapong et al. (
However, subsequent further in-depth appraisal reveals that patient allocation was neither random nor concealed but alternated with patients “with odd ID numbers assigned to the experimental treatment, and those with even ID numbers to the control treatment.” This form of alternation may cause the allocation of patients to either group to be visible, which renders the study open to a high risk of selection bias. Furthermore, the important baseline variable “falling asleep with a bottle” was statistically significantly more distributed among the experimental group (
This example shows that corroborated trials according to the CQS-2B cannot be considered as low-bias risk. Instead, corroborated trials may be considered eligible for further in-depth appraisal as shown in the example above. Such appraisal may not be confined within the criteria of any specific trial appraisal tool but may need to be tailored specifically to the particularities of each trial. In contrast, a further in-depth appraisal may not be warranted for trials that have been already falsified at C1–4 levels since the appraisal process has already provided sufficient reason to consider such trials as of high-bias risk. In that way, the CQS-2B may contribute to a more timely and efficient trial appraisal, particularly of a large volume of prospective controlled clinical therapy trials (
Not all actual trial characteristics that were performed during the conduct of a trial, required by the CQS criteria, may have been reported by trial authors. For example, trial randomisation may have included central allocation, but this was not made explicit in the published trial report. Trial appraisal using the CQS-2B would thus have assigned an erroneous 0–score to its criterion II.
Contacting trial authors during a systematic review of trials may appear to be a possible solution but can be affected by recall bias, the fact that such author responses have not been part of the original peer-review process and the reduction of possible reproducibility of the systematic review results at a later stage. Trial appraisal remains by its nature qualitative text analysis only. This is valid for any trial appraisal tool, such as the CQS-2B or Cochrane’s RoB 2. The consequence is false-negative trial appraisal results that may not be remedied.
The lack of essential trial characteristics may not always lead to systematic error. This limitation is also valid for any trial appraisal tool. Trial appraisal based on qualitative text analysis can assume whether a trial is susceptible to systematic error only when essential trial characteristics are missing. Quantitative tests that can be applied during the systematic review of clinical trials may solve this problem. Currently, there are only a few tests available for this purpose, such as the highly accurate Berger–Exner test for 3rd-order selection bias (
The CQS has been developed based on high epistemic rigour, meta-epidemiological evidence and inter-rater reliability. However, like all trial appraisal tools, it has limitations that need to be considered during its application in systematic reviews of prospective controlled clinical therapy trials.
SM contributed to conception and design of the review and wrote the first draft of the manuscript. SM, SR, and VY commented and improved the manuscript. All authors contributed to the article and approved the submitted version.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.