Bayesian safety surveillance with adaptive bias correction.

Publisher: Wiley Country of Publication: England NLM ID: 8215016 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1097-0258 (Electronic) Linking ISSN: 02776715 NLM ISO Abbreviation: Stat Med Subsets: MEDLINE

Original Publication: Chichester ; New York : Wiley, c1982-

Adverse Drug Reaction Reporting Systems* ; Vaccines*/adverse effects ; Product Surveillance, Postmarketing*; Humans ; Bayes Theorem ; Bias ; Probability

Postmarket safety surveillance is an integral part of mass vaccination programs. Typically relying on sequential analysis of real-world health data as they accrue, safety surveillance is challenged by sequential multiple testing and by biases induced by residual confounding in observational data. The current standard approach based on the maximized sequential probability ratio test (MaxSPRT) fails to satisfactorily address these practical challenges and it remains a rigid framework that requires prespecification of the surveillance schedule. We develop an alternative Bayesian surveillance procedure that addresses both aforementioned challenges using a more flexible framework. To mitigate bias, we jointly analyze a large set of negative control outcomes that are adverse events with no known association with the vaccines in order to inform an empirical bias distribution, which we then incorporate into estimating the effect of vaccine exposure on the adverse event of interest through a Bayesian hierarchical model. To address multiple testing and improve on flexibility, at each analysis timepoint, we update a posterior probability in favor of the alternative hypothesis that vaccination induces higher risks of adverse events, and then use it for sequential detection of safety signals. Through an empirical evaluation using six US observational healthcare databases covering more than 360 million patients, we benchmark the proposed procedure against MaxSPRT on testing errors and estimation accuracy, under two epidemiological designs, the historical comparator and the self-controlled case series. We demonstrate that our procedure substantially reduces Type 1 error rates, maintains high statistical power and fast signal detection, and provides considerably more accurate estimation than MaxSPRT. Given the extensiveness of the empirical study which yields more than 7 million sets of results, we present all results in a public R ShinyApp. As an effort to promote open science, we provide full implementation of our method in the open-source R package EvidenceSynthesis.
(© 2023 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.)

Anderson Roy M, May RM. Directly transmitted infections diseases: control by vaccination. Science. 1982;215:1053-1060.
Francis Donald P, Hadler Stephen C, Thompson Sumner E, et al. The prevention of hepatitis B with vaccine: report of the Centers for Disease Control multi-center efficacy trial among homosexual men. Ann Intern Med. 1982;97:362-366.
Centers for Disease Control and Prevention. National Immunization Program (Centers for Disease Control and Prevention), Education and Information and Partnership Branch National Immunization Program (Centers for Disease Control and Prevention). Epidemiology and Prevention of Vaccine-Preventable Diseases. Atlanta, GA: Department of Health & Human Services, Public Health Service, Centers for Disease Control and Prevention; 2005.
Nowak Glen J, Kristine S, Kelli B, Smith Teresa M, Michelle B. Promoting influenza vaccination: insights from a qualitative meta-analysis of 14 years of influenza-related communications research by US Centers for Disease Control and Prevention (CDC). Vaccine. 2015;33:2741-2756.
Thomas V, Frank DS, Chen Robert T, Elizabeth M. Vaccine safety surveillance using large linked databases: opportunities, hazards and proposed guidelines. Expert Rev Vaccines. 2003;2:21-29.
Lieu Tracy A, Martin K, Davis Robert L, et al. Real-time vaccine safety surveillance for the early detection of adverse events. Med Care. 2007;45:S89-S95.
Baker Meghan A, Michael N, Cole David V, Lee Grace M, Lieu TA. Post-licensure rapid immunization safety monitoring program (PRISM) data characterization. Vaccine. 2013;31:K98-K112.
Andreia L, Andrews Nick J, Thomas SL. Near real-time vaccine safety surveillance using electronic health records-a systematic review of the application of statistical methods. Pharmacoepidemiol Drug Saf. 2016;25:225-237.
Moro Pedro L, Rongxia L, Penina H, Eric W, Maria C. Surveillance systems and methods for monitoring the post-marketing safety of influenza vaccines at the Centers for Disease Control and Prevention. Expert Opin Drug Saf. 2016;15:1175-1183.
Lee Grace M, Romero José R, Bell BP. Postapproval vaccine safety surveillance for COVID-19 vaccines in the US. JAMA. 2020;324:1937-1938.
EMA European Medicine Agency. Pharmacovigilance Plan of the EU Regulatory Network for COVID-19 Vaccines (EMA/333964/2020). 2020. https://www.ema.europa.eu/en/documents/other/pharmacovigilance-plan-eu-regulatory-network-covid-19-vaccines_en.pdf.
Centers for Disease Control and Prevention VAERS. Vaccine Adverse Event Reporting System (VAERS). 2021. https://www.cdc.gov/vaccinesafety/ensuringsafety/monitoring/vaers/index.html.
Centers for Disease Control and Prevention VSD. Vaccine Safety Datalink (VSD). 2021. https://www.cdc.gov/vaccinesafety/ensuringsafety/monitoring/vsd/index.html.
Centers for Disease Control and Prevention CISA. Clinical Immunization Safety Assessment (CISA) Project. 2021. https://www.cdc.gov/vaccinesafety/ensuringsafety/monitoring/cisa/index.%html?CDC_AA_refVal=https%3A%2F%2Fwww.cdc.gov%2Fvaccinesafety%2Factivities2FCISA.html.
World Health Organization (WHO). Establishing Surveillance Systems in Countries Using COVID-19 Vaccines. 2021. https://cdn.who.int/media/docs/default-source/covid-19-vaccines-safety-surveillance-manual/training-slides_covid-19_vs_surveillance_systemsafa53e9c-0bde-4765-90fa-2dedf3b6da72.pdf?sfvrsn=6ccff509_5&Status=Master.
Rodrigues LC, Smith PG. Use of the case-control approach in vaccine evaluation: efficacy and adverse effects. Epidemiol Rev. 1999;21:56-72.
Glanz Jason M, McClure David L, Stanley X, et al. Four different study designs to evaluate vaccine safety were equally validated with contrasting limitations. J Clin Epidemiol. 2006;59:808-818.
Newcomer Sophia R, Martin K, Stan X, et al. Bias from outcome misclassification in immunization schedule safety research. Pharmacoepidemiol Drug Saf. 2018;27:221-228.
Martin K, Davis RL, Margarette K, Edwin L, Tracy L, Richard P. A Maximized Sequential Probability Ratio Test for Drug and Vaccine Safety Surveillance. Seq Anal. 2011;30:58-78.
Schuemie Martijn J, Faaizah A, Nicole P, et al. Vaccine Safety Surveillance Using Routinely Collected Healthcare Data-An Empirical Evaluation of Epidemiological Designs. Front Pharmacol. 2022;13:2532.
Schuemie Martijn J, Fan B, Akihiko N, Suchard Marc A. Adjusting for both sequential testing and systematic error in safety surveillance using observational data: Empirical calibration and MaxSPRT. Stat Med. 2023;42:619-631.
Barnard GA. Sequential tests in industrial statistics. Suppl J R Stat Soc. 1946;8:1-21.
Wetherill GB. Bayesian sequential analysis. Biometrika. 1961;48:281-292.
Berger James O, Brown Lawrence D, Wolpert RL. A unified conditional frequentist and Bayesian test for fixed and sequential simple hypothesis testing. Ann Stat. 1994;22:1787-1807.
Berger James O, Wolpert RL. The likelihood principle. Institute of Mathematical Statistics Lecture Notes-Monograph. Beachwood, OH: IMS; 1988.
Berger James O, Ben B, Yinping W. Unified frequentist and Bayesian testing of a precise hypothesis. Stat Sci. 1997;12:133-160.
Berger James O, Benzion B, Yinping W. Simultaneous Bayesian-frequentist sequential testing of nested hypotheses. Biometrika. 1999;86:79-92.
Jha Sumit K, Clarke Edmund M, Langmead Christopher J, Axel L, André P, Paolo Z. A bayesian approach to model checking biological systems. International Conference on Computational Methods in Systems Biology. Hanover, PA: Springer; 2009:218-234.
Jerome C. A Bayesian test of some classical hypotheses-with applications to sequential clinical trials. J Am Stat Assoc. 1966;61:577-594.
Thall Peter F, Simon Richard M, Estey EH. Bayesian sequential monitoring designs for single-arm clinical trials with multiple outcomes. Stat Med. 1995;14:357-379.
Smith Michael K, Ieuan J, Morris Mark F, Grieve Andrew P, Keith T. Implementation of a Bayesian adaptive design in a proof of concept study. Pharm Stat J Appl Stat Pharm Ind. 2006;5:39-50.
Xian Z, Suyu L, Kim Edward S, Herbst Roy S, Jack LJ. Bayesian adaptive design for targeted therapy development in lung cancer-a step toward personalized medicine. Clin Trials. 2008;5:181-193.
Berry Scott M, Carlin Bradley P, Jack LJ, Peter M. Bayesian adaptive methods for clinical trials. Boca Raton, FL: CRC press; 2010.
Rongxia L, Brock S, Charles R. A Bayesian approach to sequential analysis in post-licensure vaccine safety surveillance. Pharm Stat. 2020;19:291-302.
Belongia Edward A, Irving Stephanie A, Shui Irene M, et al. Real-time surveillance to assess risk of intussusception and other adverse events after pentavalent, bovine-derived rotavirus vaccine. Pediatr Infect Dis J. 2010;29:1-5.
Xintong L, Anna O, Rupa M, et al. Characterising the background incidence rates of adverse events of special interest for covid-19 vaccines in eight countries: multinational network cohort study. BMJ. 2021;373.
Klein Nicola P, Ned L, Kristin G, et al. Surveillance for adverse events after COVID-19 mRNA vaccination. JAMA. 2021;326:1390-1399.
Whitaker HJ, Farrington CP, Spiessens B, Musonda P. Tutorial in biostatistics: the self-controlled case series method. Stat Med. 2006;25:1768-1797.
Paddy F, Rush M, Miller E, et al. A new method for active surveillance of adverse events from diphtheria/tetanus/pertussis and measles/mumps/rubella vaccines. Lancet. 1995;345:567-569.
Farrington CP. Control without separate controls: evaluation of vaccine safety using case-only methods. Vaccine. 2004;22:2064-2070.
Tchetgen E. The control outcome calibration approach for causal inference with unobserved confounding. Am J Epidemiol. 2014;179:633-640.
Schuemie MJ, Ryan PB, DuMouchel W, Suchard MA, Madigan D. Interpreting observational studies: why empirical calibration is needed to correct p-values. Stat Med. 2014;33:209-218.
Schuemie MJ, Hripcsak G, Ryan PB, Madigan D, Suchard MA. Robust empirical calibration of p-values using observational data. Stat Med. 2016;35:3883-3888.
George H, Duke Jon D, Shah Nigam H, et al. Observational Health Data Sciences and Informatics (OHDSI): opportunities for observational researchers. MEDINFO 2015: eHealth-enabled Health. Amsterdam, the Netherlands: IOS Press; 2015:574-578.
Overhage J, Marc RPB, Reich Christian G, Hartzema Abraham G, Stang Paul E. Validation of a common data model for active safety surveillance research. J Am Med Inform Assoc. 2012;19:54-60.
Schuemie Martijn J, Ryan Patrick B, George H, David M, Suchard MA. Improving reproducibility by using high-throughput observational studies with empirical calibration. Philos Transact A Math Phys Eng Sci. 2018;376:20170356.
Ravi G, Bradley L, Jonathan D, et al. Risk of Guillain-Barré Syndrome Following Recombinant Zoster Vaccine in Medicare Beneficiaries. JAMA Intern Med. 2021;181:1623-1630.
Anna O, George H. COVID-19 vaccination effectiveness rates by week and sources of bias: a retrospective cohort study. BMJ Open. 2022;12:e061126.
Suchard Marc A, Schuemie Martijn J, Krumholz Harlan M, et al. Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: a systematic, multinational, large-scale analysis. Lancet. 2019;394:1816-1826.
George H, Ryan Patrick B, Duke Jon D, et al. Characterizing treatment pathways at scale using the OHDSI network. Proceedings of the National Academy of Sciences. Washington, D.C: National Academy of Sciences, Vol 113; 2016:7329-7336.
Clair B, Defalco Frank J, Ryan Patrick B, Rijnbeek Peter R. Increasing trust in real-world evidence through evaluation of observational data quality. J Am Med Inform Assoc. 2021;28:2251-2257.
Steven B, Juhani E, Claire-Anne S, et al. Importance of background rates of disease in assessment of vaccine safety during mass immunisation with pandemic H1N1 influenza vaccines. Lancet. 2009;374:2115-2122.
Katherine YW, Nordin James D, Martin K, et al. An assessment of the safety of adolescent and adult tetanus-diphtheria-acellular pertussis (Tdap) vaccine, using active surveillance for adverse events in the Vaccine Safety Datalink. Vaccine. 2009;27:4257-4262.
Katherine YW, Martin K, Fireman Bruce H, et al. Active surveillance for adverse events: the experience of the Vaccine Safety Datalink project. Pediatrics. 2011;127:S54-S64.
Buttery JP, Danchin MH, Lee KJ, et al. Intussusception following rotavirus vaccine administration: post-marketing surveillance in the National Immunization Program in Australia. Vaccine. 2011;29:3061-3066.
Leonoor W, Coralie L, Corinne V, et al. The incidence of narcolepsy in Europe: before, during, and after the influenza A (H1N1) pdm09 pandemic and vaccination campaigns. Vaccine. 2013;31:1246-1254.
Barker Charlotte IS, Snape MD. Pandemic influenza A H1N1 vaccines and narcolepsy: vaccine safety surveillance in action. Lancet Infect Dis. 2014;14:227-238.
Salmon Daniel A, Michael P, Richard F, et al. Association between Guillain-Barré syndrome and influenza A (H1N1) 2009 monovalent inactivated vaccines in the USA: a meta-analysis. Lancet. 2013;381:1461-1468.
Clémence G, Pauline B, Jérémie R, et al. Seasonal influenza vaccine and Guillain-Barré syndrome: a self-controlled case series study. Neurology. 2020;94:e2168-e2179.
Faaizah A, Schuemie Martijn J, Fan B, et al. Serially Combining Epidemiological Designs Does Not Improve Overall Signal Detection in Vaccine Safety Surveillance. Drug Saf. 2023;797-807:1-11.

R01 AG068002 United States AG NIA NIH HHS; R01 LM006910 United States LM NLM NIH HHS; R01 AG068002 United States AG NIA NIH HHS; R01 LM006910 United States LM NLM NIH HHS

Keywords: Bayesian sequential testing; postmarket safety surveillance; real-world evidence; systematic error

0 (Vaccines)

Date Created: 20231127 Date Completed: 20231226 Latest Revision: 20240123

20240123

10.1002/sim.9968

38010062