Applying the ROBINS-I tool to natural experiments: an example from public health.

Item request has been placed! ×
Item request cannot be made. ×
loading   Processing Request
Read More Add to Saved list
  • Additional Information
    • Source:
      Publisher: BioMed Central Country of Publication: England NLM ID: 101580575 Publication Model: Electronic Cited Medium: Internet ISSN: 2046-4053 (Electronic) Linking ISSN: 20464053 NLM ISO Abbreviation: Syst Rev Subsets: MEDLINE
    • Publication Information:
      Original Publication: London : BioMed Central
    • Subject Terms:
      Bias* ; Confounding Factors, Epidemiologic* ; Public Health*; Research Design/*standards; Humans ; Reproducibility of Results ; Retrospective Studies ; Surveys and Questionnaires
    • Abstract:
      Background: A new tool to assess Risk of Bias In Non-randomised Studies of Interventions (ROBINS-I) was published in Autumn 2016. ROBINS-I uses the Cochrane-approved risk of bias (RoB) approach and focusses on internal validity. As such, ROBINS-I represents an important development for those conducting systematic reviews which include non-randomised studies (NRS), including public health researchers. We aimed to establish the applicability of ROBINS-I using a group of NRS which have evaluated non-clinical public health natural experiments.
      Methods: Five researchers, all experienced in critical appraisal of non-randomised studies, used ROBINS-I to independently assess risk of bias in five studies which had assessed the health impacts of a domestic energy efficiency intervention. ROBINS-I assessments for each study were entered into a database and checked for consensus across the group. Group discussions were used to identify reasons underpinning lack of consensus for specific questions and bias domains.
      Results: ROBINS-I helped to systematically articulate sources of bias in NRS. However, the lack of consensus in assessments for all seven bias domains raised questions about ROBINS-I's reliability and applicability for natural experiment studies. The two RoB domains with least consensus were selection (Domain 2) and performance (Domain 4). Underlying the lack of consensus were difficulties in applying an intention to treat or per protocol effect of interest to the studies. This was linked to difficulties in determining whether the intervention status was classified retrospectively at follow-up, i.e. post hoc. The overall risk of bias ranged from moderate to critical; this was most closely linked to the assessment of confounders.
      Conclusion: The ROBINS-I tool is a conceptually rigorous tool which focusses on risk of bias due to the counterfactual. Difficulties in applying ROBINS-I may be due to poor design and reporting of evaluations of natural experiments. While the quality of reporting may improve in the future, improved guidance on applying ROBINS-I is needed to enable existing evidence from natural experiments to be assessed appropriately and consistently. We hope future refinements to ROBINS-I will address some of the issues raised here to allow wider use of the tool.
    • References:
      Health Place. 2007 Mar;13(1):99-110. (PMID: 16343973)
      J Eval Clin Pract. 2012 Feb;18(1):12-8. (PMID: 20698919)
      J Epidemiol Community Health. 2015 Feb;69(2):189-95. (PMID: 25481532)
      BMJ. 1998 Jan 24;316(7127):285. (PMID: 9472515)
      Int J Epidemiol. 2007 Jun;36(3):666-76. (PMID: 17470488)
      J Epidemiol Community Health. 2009 Jan;63(1):12-7. (PMID: 18801798)
      BMJ. 2016 Oct 12;355:i4919. (PMID: 27733354)
      J Epidemiol Community Health. 2013 Jan;67(1):98-104. (PMID: 22851579)
      Public Health. 2000 Nov;114(6):434-9. (PMID: 11114752)
      Int J Behav Nutr Phys Act. 2016 Oct 7;13(1):107. (PMID: 27717360)
      BMJ. 2018 May 16;361:k1079. (PMID: 29769210)
      J Epidemiol Community Health. 2012 Dec;66(12):1182-6. (PMID: 22577181)
      Annu Rev Public Health. 2017 Mar 20;38:39-56. (PMID: 28125392)
      J Clin Epidemiol. 2013 Sep;66(9):982-93. (PMID: 23683848)
      Int J Behav Nutr Phys Act. 2017 Apr 19;14(1):49. (PMID: 28424086)
      BMJ. 2014 Mar 07;348:g1687. (PMID: 24609605)
      Am J Public Health. 2009 Nov;99 Suppl 3:S681-92. (PMID: 19890174)
      Cochrane Database Syst Rev. 2013 Feb 28;(2):CD008657. (PMID: 23450585)
      PLoS One. 2014 May 13;9(5):e96920. (PMID: 24824199)
      J Clin Epidemiol. 2013 Sep;66(9):973-81. (PMID: 22981249)
      BMJ. 2009 Oct 19;339:b4012. (PMID: 19841007)
    • Grant Information:
      1517742 International Medical Research Council (GB) & CSO; SPHSU15 United Kingdom CSO_ Chief Scientist Office; SPHSU13 United Kingdom CSO_ Chief Scientist Office; MC_UU_12017/15 United Kingdom MRC_ Medical Research Council; SCAF/15/02 United Kingdom CSO_ Chief Scientist Office; MC_UU_12017/13 United Kingdom MRC_ Medical Research Council
    • Contributed Indexing:
      Keywords: Controlled before and after; Methodological quality; Natural experiments; Non-randomised studies; Public health; Risk of bias
    • Publication Date:
      Date Created: 20180126 Date Completed: 20190107 Latest Revision: 20240327
    • Publication Date:
      20240327
    • Accession Number:
      PMC5784724
    • Accession Number:
      10.1186/s13643-017-0659-4
    • Accession Number:
      29368630