Generalizability of the findings from a randomized controlled trial of a web-based substance use disorder intervention

BACKGROUND AND OBJECTIVES: There is growing concern regarding the generalizability of findings from randomized controlled trials (RCTs) of interventions for substance use disorders (SUDs). This study used a selection model approach to assess and improve the generalizability of an evaluation for a web-based SUD intervention by making the trial sample resemble the target population. METHODS: The sample of the web-based SUD intervention (Therapeutic Education System vs. Treatment-as-usual; n = 507) was compared with the target population of SUD treatment-seeking individuals from the Treatment Episodes Data Set-Admissions (TEDS-A). Using weights based on the probabilities of RCT participation, we computed weighted treatment effects on retention and abstinence. RESULTS: Substantial differences between the RCT sample and the target population was demonstrated in significant difference in the mean propensity scores (1.62 standard deviations at p < .001). The population effect on abstinence (12 weeks and 6 months) was statistically insignificant after weighting the data with the generalizability weight. DISCUSSIONS AND CONCLUSIONS: Generalizability of the findings from the RCT could be limited when the RCT sample does not well represent the target population. SCIENTIFIC SIGNIFICANCE: Application of generalizability weights can be a potentially useful tool to improve generalizability of RCT findings. (Am J Addict 2018;27:231-237).