| Sign In to gain access to subscriptions and/or personal tools. |
Quality of RCTs in Periodontology— A Systematic Review
1 Departments of Periodontology and Correspondence: *corresponding author, i.needleman{at}eastman.ucl.ac.uk
Randomization, concealment of treatment allocation, blinding, and patient follow-up are key quality components of randomized controlled trials (RCTs). The objective of this study was to assess the quality of RCTs in periodontology using these evidence-based components. Following a detailed search, screening and quality assessments of RCTs were conducted in duplicate and independently. The results showed that although 91% of trials were described as randomised, adequate methods for randomization and allocation concealment were found in 17% and 7% of studies, respectively. Blinding was adequate for the caregiver in 17% and for the examiner in 55% of studies. A clear accounting of all participants was present in 56% of reports. This rigorous systematic review revealed that the quality of RCTs in periodontology, judged by their publications, frequently does not meet recommended standards. If this quality is reflected in actual study conduct, fundamental errors could have a significant impact on the outcomes of these trials.
Key Words: systematic review • clinical trials • periodontal diseases
The randomized controlled trial (RCT) provides the highest level of evidence for medical interventions (Sackett et al., 2000). The elements inherent in a carefully designed and conducted RCT will minimize bias, balance confounders, and hence produce the most reliable estimate of treatment effect. However, unraveling which components are most significant to trial quality is a recent development. Observational studies show these to be; randomization (the generation of a true randomization sequence [Roberts and Torgerson, 1998]), allocation concealment (concealing the randomization sequence during patient recruitment (Torgerson and Roberts, 1999), blinding (Day and Altman, 2000), and handling of withdrawals and losses to follow-up (May et al., 1981). A meta-analysis of these observational studies showed that improper allocation concealment produced the largest impact on size of treatment effect (Juni et al., 2001b). Treatment was found to be 30% more beneficial if concealment was inadequate or unclear (combined odds ratio 0.70; 95% CI, 0.62, 0.80). While overall, an inadequate method for generating a random sequence did not produce an increased estimate, one study (Kjaergard et al., 1999) demonstrated a 51% greater treatment effect (OR 0.49; 95% CI, 0.3, 0.81). In addition, where allocation concealment was adequate but the method used to generate the randomization sequence was not, the treatment effect was exaggerated compared with adequate methods. In other words, with a sequence that is not truly random, the allocation of the treatment group for the next individual could be predicted, even if it is properly concealed. Blinding is less straightforward to investigate, partly because it is dependent on the objectivity of the outcome assessed (e.g., tooth loss vs. gingival redness), and partly because it is not always achievable. The impact of blinding within the meta-analysis was variable, although three of the four included studies were in the direction of exaggerating the outcome. The fourth study showed the widest confidence interval and a negative estimate of effect (Moher et al., 1998), although it was based mainly on trials with objective outcomes such as mortality, where examiner blinding is irrelevant. Evidence that inadequate blinding can lead to larger treatment effects is also available from other studies (Colditz et al., 1989). Only two studies addressed the effect of losses to follow-up, and neither showed a significant effect of inadequate methodology on size of treatment outcome. Unfortunately, both studies used inadequate methods to assess patient attrition, which might explain their lack of effect (Juni et al., 2001b). Evidence for bias from incomplete follow-up is, however, available elsewhere (May et al., 1981). It should be noted that these studies are observational and cannot provide definitive evidence of effect. However, the consistency in direction of effect is notable, and a review of results provides compelling evidence for their importance to study quality. Quality assessment of trials in periodontology has been reported before (Hayes et al., 1992; Antczak-Bouckoms, 1993; Knoll-Kohler, 1999). However, the use of checklists and scales as reported by these authors has become discouraged, partly because many of the included items have no evidence of an effect and partly because estimated quality can vary widely depending on which scale is used (Moher et al., 1996; Juni et al., 1999). In summary, current best evidence highlights the importance of certain recognized aspects of trial quality. Assessing these components is now recommended in quality appraisal of clinical trials (Cochrane Reviewers Handbook, 2001; Juni et al., 2001a; NHS Centre for Reviews and Dissemination, 2001). Our objective was systematically to assess the quality of RCTs in periodontology using evidence-based criteria shown to have an impact on the size of treatment outcome, namely, randomization, allocation concealment, blinding, and handling of withdrawals.
Protocol Development This was a systematic review of randomized controlled trials in periodontology addressing intervention studies in humans. The first phase involved development of the review protocol detailing all methods, including: aim, search, eligibility criteria for study inclusion, methods for screening and data abstraction, and data analysis.
Search Strategy
Quality Assessment
Pilot Study
Screening and Data Abstraction Screening of titles and abstracts was conducted in duplicate by two reviewers independently (RM & IN). Agreement was measured and Kappa scores calculated. Disagreement was resolved by subsequent discussion. Screening and data abstraction of the full-text articles were also performed in duplicate. One reviewer screened all reports (RM) and three other reviewers were allocated samples of studies. Agreement was measured as before and disagreement resolved by discussion. One reviewer performed computer data entry, and analysis consisted of descriptive statistics for prevalence of quality components by means of SPSS Version 10.0 (SPSS Inc., Chicago, IL, USA).
Search The search revealed 283 articles for screening. The distribution among journals was: Journal of Clinical Periodontology, 155 (54.8%); Journal of Periodontology, 126 (44.5%); and Journal of Periodontal Research, 2 (0.7%). The flow of articles through the screening process is given in the Fig.  Reasons for study exclusion are given in Table 1.
Study Quality Components Inter-reviewer agreement Agreement between and among reviewers was good. The range of Kappa values for screening and quality components was 0.95-1.0 (Table 2 ).
Quality scores Most studies described themselves as randomized (91%) (Table 3 ). However, in the great majority of studies (83%), there was either no method reported or it was inadequately explained. A clearly inadequate randomization method was utilized in one study only. Allocation concealment was particularly poorly reported, with 93% of the studies giving either no method or an unclear description.
Patient and caregiver blinding was absent in 43% and 57% of studies, respectively. Approximately one-quarter of studies provided inadequate information to determine whether these types of blinding had been used. Only 17% of studies positively demonstrated caregiver blinding. Examiner blinding was not possible for 16% of studies, mainly due to very clear differences in treatments between test and control groups. Out of the remaining studies, 55% had definite examiner blinding, 7% had no blinding, and 38% provided inadequate information to determine. Approximately half of the studies (56%) demonstrated proper accounting for all participants at the end of the trial. For the rest, accounting of participants was absent in 14% of studies and impossible to determine for 30% of trials. The use of analyses to take into account withdrawals (drop-outs, losses to follow-up, excluded patients) was not applicable for 43% of studies where follow-up was deemed complete. For the rest, only 11% of analyses took withdrawals into account, although a larger proportion (56%) provided inadequate information to determine whether this had been done.
This systematic review has demonstrated that RCTs in periodontology frequently do not meet current recommendations on study quality. The randomization method could be judged adequate in only 17% of trials and the concealment of the allocation sequence adequate in 7%. While examiner blinding was more frequently reported (55%), both patient and caregiver blinding were not commonly used. We were surprised how often the accounting of patient follow-up was not given or unclear (44%). It is also evident that few analyses currently attempt to assess the impact of patients lost to follow-up. However, it is important to point out that such an analysis is only feasible due to the large numbers of RCTs published in periodontology compared with other dental specialties (Niederman et al., 2002). Attention to aspects of design and reporting will help to enhance the impact of this specialty in oral health care. Several components of quality assurance were included in this review, since the possibility of reviewer bias must be minimized. Thus, screening of abstracts/titles and full-text articles was always performed in duplicate and independently. The same approach was taken for assessment of study quality. In this way, the possibility of bias from one reviewer has been reduced. In addition, the level of agreement was checked for each of these stages and is reported above. The level of agreement was very good, despite the frequent difficulty in finding data in trial reports. We attribute this level of agreement to several aspects. First, reviewers were carefully trained in the methods of the review. Second, piloting of the screening and data abstraction forms reduced ambiguities in language. Third, a clear key to categorizing the quality components was designed and provided to all reviewers. It must be remembered that, in this study (and others in medicine), we have used the trial publication as a surrogate for actual study conduct. There may be reasons why this could be inaccurate. Most likely, authors may omit important data or be unaware of the need to include them. This could be particularly true for randomization, where researchers tend not to separate this process into (1) generation of the random sequence and (2) the concealment of the sequence (Roberts and Torgerson, 1998). Assessment of trial protocols or ethics committee submissions might provide a means of assessing this, although they too may lack detail. Contact with the original study authors to clarify unclear methods is a further method to check protocol details. We did not undertake this approach because of the numbers of trials involved in this review, and the conclusions should reflect this limitation. However, there is compelling evidence from two studies that poor reporting does reflect poor study quality (Liberati et al., 1986; Schulz et al., 1995). In one, the impact of inadequate allocation concealment was substantial, whether the methods were clearly inadequate or whether the reviewers were uncertain due to insufficient information (OR exaggerated by 41% and 30%, respectively (Schulz et al., 1995). This study did not attempt to quantify the impact of trial quality on the outcome of clinical research, although this will be the subject of a future project. However, if we accept the findings from studies in other fields in medicine, we can speculate on the possible findings of such research. In the studies presented above, inadequate allocation concealment had the greatest potential to exaggerate the size of the treatment effect. In this study, adequate allocation concealment was found in only 7% of RCTs in periodontology. It is likely, therefore, that the effect of selection bias introduced by inadequate allocation concealment could engender an important overestimation of treatment effects in these trials. Furthermore, ascertainment bias (systematic distortion of results by knowledge of the intervention received by participants) could be introduced at several stages (Jadad, 1998). Blinding aims to minimize this bias, and the results of this study showed a poor level of blinding of the examiner and, particularly, the caregiver. Outcomes typically assessed in periodontology, such as probing changes, may be particularly prone to this type of bias. Therefore, an exaggeration of treatment effect is a likely consequence. While caregiver blinding was assessed only where we judged that it could be achieved, no such distinction was made for examiner blinding, and this might have underestimated the occurrence of appropriate examiner blinding. With regard to bias from missing patient data (losses to follow-up and withdrawals), the current recommendation is that statistical analyses model the impact of missing data with intention-to-treat or worst-case-scenario approaches. It is clear that few studies in periodontology have attempted to examine this effect. As with the other quality components, failure to investigate this bias may lead to an inaccurate estimate of treatment effect. In experimental therapeutic studies, this could again lead to overestimation of treatment effect, since patients with adverse or poor outcomes might preferentially be lost to the study (Jadad, 1998). Incomplete reporting of trials has been a common finding in medicine. To tackle this problem, investigators have proposed the CONSORT statement, which has recently been revised and updated (Moher et al., 2001b). CONSORT provides authors with a checklist and flow diagram to help improve the quality of reports of randomized controlled trials. The aim is to facilitate the appraisal of trial quality, although a further outcome might be an improvement in future study design. A recent study of the quality of reports before and after adoption of the CONSORT statement, and in comparison with those in a non-adopting journal, indicated a significant increase in reporting quality (Moher et al., 2001a). So far, the British Dental Journal (Needleman, 1999) and the Journal of Orthodontics are the only oral health journals to adopt this guideline. This study suggests the need for wider acceptance of CONSORT by oral health journals. In conclusion, within the limitations of this study, we have shown that greater attention to quality aspects of design and reporting of RCTs in periodontology is needed, and it is likely that other fields in dentistry are similarly affected. Since many of these aspects of quality are likely to have an impact on the size of the reported treatment effect, the dental research community should view these results with concern. Further research is needed to determine the impact of quality on treatment outcomes in periodontology. Better reporting of clinical trials in periodontology is needed, and the adoption of the CONSORT statement should be a first step for many dental journals.
We are grateful to the Cochrane Oral Health Group for help with searching and for access to the specialized trials register. No external support was received for this study. University College London paid salaries to IGN, DM, and MT. Received for publication April 3, 2002. Revision received July 24, 2002. Accepted for publication September 6, 2002.
Journal of Dental Research, Vol. 81, No. 12,
866-870 (2002)
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
|
 |
|
Sign In to gain access to subscriptions and/or personal tools.
TOP
ABSTRACT
INTRODUCTION
scores)