Evaluation of Statistically Significant Findings in Orthopaedic Trials
Evaluation of Statistically Significant Findings in Orthopaedic Trials
Background: Evidence-based medicine posits that health care research is founded upon clinically important differences in patient centered outcomes. Statistically significant differences between two treatments may not necessarily reflect a clinically important difference. We aimed to quantify the sample sizes and magnitude of treatment effects in a review of orthopaedic randomized trials with statistically significant findings.
Methods: We conducted a comprehensive search (PubMed, Cochrane) for all randomized controlled trials between 1/1/95 to 12/31/04. Eligible studies include those that focused upon orthopaedic trauma. Baseline characteristics and treatment effects were abstracted by two reviewers. Briefly, for continuous outcome measures (ie functional scores), we calculated effect sizes (mean difference/standard deviation). Dichotomous variables (ie infection, nonunion) were summarized as absolute risk differences and relative risk reductions (RRR). Effect sizes >0.80 and RRRs >50% were defined as large effects.
Using regression analysis we examined the association between the total number of outcome events and treatment effect (dichotomous outcomes).
Results: Our search yielded 433 randomized controlled trials (RCTs), of which 76 RCTs with statistically significant findings on 184 outcomes (122 continuous/62 dichotomous outcomes) met study eligibility criteria. The mean effect size across studies with continuous outcome variables was 1.7 (95% confidence interval: 1.43-1.97). For dichotomous outcomes, the mean risk difference was 30% (95%confidence interval:24%-36%) and the mean relative risk reduction was 61% (95% confidence interval: 55%-66%; range: 0%-97%). Fewer numbers of total outcome events in studies was strongly correlated with increasing magnitude of the treatment effect (Pearson's R = -0.70, p < 0.01). When adjusted for sample size, the number of outcome events revealed an independent association with the size of the treatment effect (Odds ratio = 50, 95% confidence interval: 3.0-1000, p = 0.006).
Conclusion: Our review suggests that statistically significant results in orthopaedic trials have the following implications-1) On average large risk reductions are reported 2) Large treatment effects (>50% relative risk reduction) are correlated with few number of total outcome events. Readers should interpret the results of such small trials with these issues in mind.
Evidence-based medicine posits that health care research is founded upon clinically important differences in patient centered outcomes. Randomized trials continue to represent the reference standard for the comparison of surgical interventions. Although fundamentally the most important for guiding clinical practice, few randomized trials are conducted in orthopaedic surgery. Current estimates suggest that less than 5% of the orthopaedic literature represent randomized trials. Nevertheless, the impact of randomized trials, especially those with statistically significant findings, is large.
Statistically significant differences between two treatments may not necessarily reflect a clinically important difference. Although it is well known that orthopaedic studies with small sample sizes risk underpowered false negative conclusions (Beta-errors), statistically significant findings in small trials can occur at the consequence of very large differences between treatments (treatment effect). It is not uncommon for randomized trials to report relative risk reductions larger than 50% when comparing one treatment with another.
Devereaux and colleagues caution the interpretation of small trials in cardiology. For example, the peri-operative beta-blocker evidence suggests large treatment effects (i.e., relative risk reductions >75%) but these results are inconsistent with beneficial cardiovascular therapies established in trials with 10,000s of patients, which generally demonstrate moderate relative risk reductions in the order of 15 to 35%.
Our study had 2 objectives: 1) To determine the magnitude of treatment effects in a sample of orthopaedic randomized trials with statistically significant results and 2) to examine the association between the number of outcome events (a measure of study sample size) and the size of the treatment effect. We conducted a systematic review to identify randomized trials in orthopaedic trauma with the following hypotheses: 1) statistically significant studies would not always report large treatment effects and 2) studies with smaller sample sizes (and few outcome events) would be more likely to report larger treatment effects than those with larger sample sizes.
Background: Evidence-based medicine posits that health care research is founded upon clinically important differences in patient centered outcomes. Statistically significant differences between two treatments may not necessarily reflect a clinically important difference. We aimed to quantify the sample sizes and magnitude of treatment effects in a review of orthopaedic randomized trials with statistically significant findings.
Methods: We conducted a comprehensive search (PubMed, Cochrane) for all randomized controlled trials between 1/1/95 to 12/31/04. Eligible studies include those that focused upon orthopaedic trauma. Baseline characteristics and treatment effects were abstracted by two reviewers. Briefly, for continuous outcome measures (ie functional scores), we calculated effect sizes (mean difference/standard deviation). Dichotomous variables (ie infection, nonunion) were summarized as absolute risk differences and relative risk reductions (RRR). Effect sizes >0.80 and RRRs >50% were defined as large effects.
Using regression analysis we examined the association between the total number of outcome events and treatment effect (dichotomous outcomes).
Results: Our search yielded 433 randomized controlled trials (RCTs), of which 76 RCTs with statistically significant findings on 184 outcomes (122 continuous/62 dichotomous outcomes) met study eligibility criteria. The mean effect size across studies with continuous outcome variables was 1.7 (95% confidence interval: 1.43-1.97). For dichotomous outcomes, the mean risk difference was 30% (95%confidence interval:24%-36%) and the mean relative risk reduction was 61% (95% confidence interval: 55%-66%; range: 0%-97%). Fewer numbers of total outcome events in studies was strongly correlated with increasing magnitude of the treatment effect (Pearson's R = -0.70, p < 0.01). When adjusted for sample size, the number of outcome events revealed an independent association with the size of the treatment effect (Odds ratio = 50, 95% confidence interval: 3.0-1000, p = 0.006).
Conclusion: Our review suggests that statistically significant results in orthopaedic trials have the following implications-1) On average large risk reductions are reported 2) Large treatment effects (>50% relative risk reduction) are correlated with few number of total outcome events. Readers should interpret the results of such small trials with these issues in mind.
Evidence-based medicine posits that health care research is founded upon clinically important differences in patient centered outcomes. Randomized trials continue to represent the reference standard for the comparison of surgical interventions. Although fundamentally the most important for guiding clinical practice, few randomized trials are conducted in orthopaedic surgery. Current estimates suggest that less than 5% of the orthopaedic literature represent randomized trials. Nevertheless, the impact of randomized trials, especially those with statistically significant findings, is large.
Statistically significant differences between two treatments may not necessarily reflect a clinically important difference. Although it is well known that orthopaedic studies with small sample sizes risk underpowered false negative conclusions (Beta-errors), statistically significant findings in small trials can occur at the consequence of very large differences between treatments (treatment effect). It is not uncommon for randomized trials to report relative risk reductions larger than 50% when comparing one treatment with another.
Devereaux and colleagues caution the interpretation of small trials in cardiology. For example, the peri-operative beta-blocker evidence suggests large treatment effects (i.e., relative risk reductions >75%) but these results are inconsistent with beneficial cardiovascular therapies established in trials with 10,000s of patients, which generally demonstrate moderate relative risk reductions in the order of 15 to 35%.
Our study had 2 objectives: 1) To determine the magnitude of treatment effects in a sample of orthopaedic randomized trials with statistically significant results and 2) to examine the association between the number of outcome events (a measure of study sample size) and the size of the treatment effect. We conducted a systematic review to identify randomized trials in orthopaedic trauma with the following hypotheses: 1) statistically significant studies would not always report large treatment effects and 2) studies with smaller sample sizes (and few outcome events) would be more likely to report larger treatment effects than those with larger sample sizes.
