1 List of Abbreviations

AUC Area Under the receiver-operator Curve
CCAE IBM MarketScan Commercial Claims and Encounters
CDM Common Data Model
CIOMS Council for International Organizations of Medical Sciences
COVID-19 COronaVIrus Disease 2019
CPRD Clinical Practice Research Datalink
CRAN Comprehensive R Archive Network
EHR Electronic Health Record
EMA European Medicines Agency
ENCEPP European Network of Centres for Pharmacoepidemiology and Pharmacovigilance
H1N1pdm Hemagglutinin Type 1 and Neuraminidase Type 1 (2009 pandemic influenza)
HPV Human PapillomaVirus
IRB Institutional review board
JMDC Japan Medical Data Center
LLR Log Likelihood Ratio
MDCR IBM MarketScan Medicare Supplemental Database
MDCD IBM MarketScan Multi-State Medicaid Database
MSE Mean Squared Error
OHDSI Observational Health Data Science and Informatics
OMOP Observational Medical Outcomes Partnership
MaxSPRT MAXimized Sequential Probability Ratio Test
PS Propensity score
RCT Randomized controlled trial
SCCS Self-Controlled Case Series
SCRI Self-Controlled Risk Interval
WHO World Health Organization

2 Responsible Parties

2.1 Investigators

Version 2.0 Investigator Institution/Affiliation
Fan Bu Department of Biostatistics, University of Michigan, Ann Arbor, MI, USA
Shounak Chattopadhyay Department of Biostatistics, University of California, Los Angeles, Los Angeles, CA, USA
George Hripcsak Department of Biomedical Informatics, Columbia University, New York, NY, USA
Patrick B. Ryan Observational Health Data Analytics, Janssen Research and Development, Titusville, NJ, USA
Martijn J. Schuemie Observational Health Data Analytics, Janssen Research and Development, Titusville, NJ, USA
Marc A. Suchard * Department of Biostatistics, University of California, Los Angeles, Los Angeles, CA, USA
* Principal Investigator

2.2 Disclosures

This study is undertaken within Observational Health Data Sciences and Informatics (OHDSI), an open collaboration. GH receives grant funding from the US National Institutes of Health and the US Food & Drug Administration. PBR and MJS are employees of Janssen Research and Development and shareholders in John & Johnson. MAS receives grant funding from the US National Institutes of Health and the US Food & Drug Administration and contracts from the US Department of Veterans Affairs and Janssen Research and Development.

3 Abstract

Background and Significance

As recently approved COVID-19 vaccines are rolled out globally, it is likely that safety signals will be identified from spontaneous reports and other data sources. Although some work has been done on the best methods for vaccine safety surveillance, there is a scarcity of information on how these perform in analyses of real-world data.

Study Aims

To study the comparative performance (bias, precision, and timeliness) of different analytical methods for the study of comparative vaccine safety.

Study Description

  • Design: Cohort, self-controlled, and case-control studies
  • Exposures: previous viral vaccines including 2017-2018 flu, H1N1pdm flu, Human Papillomavirus (HPV) and Varicella-Zoster, and BNT162b2 and mRNA-1273 mRNA vaccines against COVID-19.
  • Outcomes: selected adverse events of special interest (e.g., myocarditis or pericarditis); negative control outcomes; imputed positive control outcomes
  • Analyses:
    1. Historical rate comparisons.
    2. Cohort analyses using a contemporary non-user comparator, with large-scale propensity score matching
    3. Self-controlled case series with variations
    4. Case-control analyses
    5. Concurrent comparator
  • Metrics:
    • Area Under the receiver-operator Curve (AUC). The ability to discriminate between positive controls and negative controls based on the point estimate of the effect size. Will be stratified by true effect size of the positive controls.
    • Coverage. How often the true effect size is within the 95% confidence interval.
    • Mean precision, computed as 1 / (standard error)2
    • Mean squared error (MSE). Mean squared error between the log of the effect size point-estimate and the log of the true effect size.
    • Type 1 error. For negative controls, how often was the null rejected (at alpha = 0.05). This is equivalent to the false positive rate and 1 - specificity.
    • Type 2 error. For positive controls, how often was the null not rejected (at alpha = 0.05). This is equivalent to the false negative rate and 1 - sensitivity. Will be stratified by true effect size of the positive controls.
    • Non-estimable. Measure for how many of the controls was the method unable to produce an estimate
    • Sensitivity and specificity based on the MaxSPRT decision rule
    • Detection time: the number of months until 80% of positive controls exceeds the critical value. Will be stratified by true effect size of the positive controls.

4 Amendments and Updates

Table 4.1 lists any protocol amendments made over time.

Table 4.1: Protocol amendments
Number Date Section of study protocol Amendment or update Reason
1.1.0 2021-04-09 Methods to evaluate Added historic comparator, cohort method, and SCCS variations The historic comparator variation was based on initial EMAEUS results showing (preventable) outliers. The cohort method and SCCS variants are based on 3rd-party protocols for COVID-19 vaccine safety surveillance.
1.2.0 2021-06-29 Exposure-outcome pairs Switched from synthetic to imputed positive controls Positive control synthesis requires a minimum number of outcomes in the data, precluding many negative controls and leading to only ‘highly’ powered positive controls. This limited the ability to measure performance of methods in low-power settings. Positive control imputation is more simplistic but can be applied in low-power settings.
2.0.0 2023-11-06 Abstract Added BNT162b2 and mRNA-1273 vaccines and concurrent comparator design Second EUMAEUS study (EUMAEUS v2) to evaluate concurrent comparator design using negative and imputed positive control outcomes and myocarditis or pericarditisis real-world positive control outcome.
2.0.0 2023-11-06 Investigators Updated authors EUMAEUS v2
2.0.0 2023-11-06 Milestones Added new milestone dates EUMAEUS v2
2.0.0 2023-11-06 Exposure-outcome pairs Added BNT1262b2 and mRNA-1273 COVID-19 vaccines EUMAEUS v2
2.0.0 2023-11-06 Real-world positive control outcome Added myocarditisis or pericarditisis for COVID-19 vaccines EUMAEUS v2
2.0.0 2023-11-06 Methods to evaluation Added concurrent comparator with three target/comparator design choices EUMAEUS v2
2.0.0 2023-11-06 Time-at-risk Added concurrent comparator time-at-risk definitions for consistency with method use-cases and EUMAEUS v1 choices EUMAEUS v2
2.0.0 2023-11-06 Exposure cohort definitions Added BNT162b2 and mRNA-1273 COVID-19 vaccine cohort definitions EUMAEUS v2
2.0.0 2023-11-06 Real-world outcome cohort definition Added myocarditis or pericarditis outcome cohort defintion EUMAEUS v2
2.0.0 2024-01-30 Overview of analyses Updated total number of estimates to include real-world positive control outcome EUMAEUS v2
2.0.0 2024-01-30 Appendix Corrected age restriction for BNT162b2 and EUA date for mRNA-1273 EUMAEUS v2
2.0.0 2024-02-05 Appendix Corrected all COVID vaccines concept set EUMAEUS v2

5 Milestones

Table 5.1 lists the study milestones.

Table 5.1: Study milestones
Milestone Planned / actual date
EU PAS Registration 2021-03-23 / 2021-03-23
Start of analysis 2021-03-23 / 2021-03-25
End of analysis 2021-05-01 / 2021-05-31
Results presentation 2021-05-01 / 2021-06-29
Start of 2nd analysis 2024-02-12
End of 2nd analysis 2024-04-23
Results presentation 2024-05-21

6 Rationale and Background

A total of 3 COVID-19 vaccines have been approved for clinical use in Europe, and 3 in the USA. Many more are in pipeline, and at least two more have reported to date on phase 3 efficacy data. Although safe and effective based on large randomised controlled trials, COVID-19 vaccines will be subject to post-marketing safety studies, including both analyses of spontaneous reports (pharmacovigilance) as well as longitudinal analyses in the form of post-authorisation safety studies.

The ENCEPP (European Network of Centres for Pharmacoepidemiology and Pharmacovigilance) methodological guidelines, in their 8th revision,[1] mention a few documents that set out standards for the conducting of vaccine safety studies. Specific aspects related to vaccine safety research are discussed in detail in different materials, including the Report of the CIOMS/WHO Working Group on Definition and Application of Terms for Vaccine Pharmacovigilance (2012), the CIOMS Guide to Active Vaccine Safety Surveillance (2017), the CIOMS Guide to Vaccine Safety Communication (2018), the Brighton Collaboration resources, the Module 4 (Surveillance) of the e-learning training course Vaccine Safety Basics by the World Health Organization (WHO), or the recommendations on vaccine-specific aspects of the EU pharmacovigilance system outlined in the Module P.I: Vaccines for prophylaxis against infectious diseases of the Good pharmacovigilance practices (GVP). Additionally, the Accelerated Development of VAccine beNefit-risk Collaboration in Europe (ADVANCE) project has summarized methods for vaccine safety in a bespoke report [2] covering multiple study designs, both experimental and observational in nature. The EMA (European Medicines Agency) has also issued guidances [3] and a plan [4] for pharmacovigilance of COVID-19 vaccines. Despite this plethora of literature and guidance, there is a scarcity of methodological studies on the performance of different methods for vaccine safety.

Given the quick and increasingly global rollout of COVID-19 vaccines internationally, it is highly likely that potential safety signals will emerge, which will need a timely but robust evaluation in ‘real world’ observational studies. It is therefore urgent that we conduct large-scale evaluations of methods for vaccine safety, similar to previous work on methods for drug safety. [5] The results of this evaluation will help us understand how these methods will perform when applied to COVID-19 vaccines.

7 Study Objectives

The overarching aim is to identify the best methods for the generation of evidence of vaccine safety in observational, real-world data. Specific aims:

  • To estimate the bias and precision associated with the use of different methods (historic rate, cohort, self-controlled, and case-control) for the study of vaccine safety compared
  • To compare the ‘timeliness’ of these methods for the identification of vaccine safety signals

8 Research Methods

8.1 Exposure-outcome pairs

8.1.1 Exposures

The evaluation will center on six existing (groups of) vaccines, for specific time periods (start date to end date), as shown in Table 8.1.

Table 8.1: Exposures of interest.
Exposure Name Start Date End Date History Start Date History End Date
H1N1pdm vaccination 01-09-2009 31-05-2010 01-09-2008 31-05-2009
Seasonal flu vaccination (Fluvirin) 01-09-2017 31-05-2018 01-09-2016 31-05-2017
Seasonal flu vaccination (Fluzone) 01-09-2017 31-05-2018 01-09-2016 31-05-2017
Seasonal flu vaccination (All) 01-09-2017 31-05-2018 01-09-2016 31-05-2017
Zoster vaccination (Shingrix) 01-01-2018 31-12-2018 01-01-2017 31-12-2017
HPV vaccination (Gardasil 9) 01-01-2018 31-12-2018 01-01-2017 31-12-2017
COVID-19 vaccination (BNT126b2) 11-12-2020 30-06-2021 01-01-2019 30-06-2019
COVID-19 vaccination (mRNA-1273) 18-12-2020 30-06-2021 01-01-2019 30-06-2019

For some methods the period between historic start and historic end date will be used to estimate the historic incidence rate. For analyses executed on data in the southern hemisphere (if any) the flu seasons are different, and the study periods will need to be adjusted accordingly. The formal cohort definitions of each exposure can be found in Appendix A.

8.1.2 Negative control outcomes

Negative controls are outcomes believed not to be caused by any of the vaccines, and therefore ideally would not be flagged as a signal by a safety surveillance system. Any effect size estimates for negative control ideally should be close to the null.

A single set of negative control outcomes is defined for all six vaccine groups. To identify negative control outcomes that match the severity and prevalence of suspected vaccine adverse effects, a candidate list of negative controls was generated based on similarity of prevalence and percent of diagnoses that were recorded in an inpatient setting (as a proxy for severity). Manual review of this list by clinical experts created the final list of 93 negative control outcomes. The full list of negative control outcomes can be found in Appendix C

Negative control outcomes are defined as the first occurrence of the negative control concept or any of its descendants.

8.1.3 Imputed positive control outcomes

Positive controls are outcomes known to be caused by vaccines, and ideally would be detected as signals by a safety surveillance system as early as possible. For various reasons, real positive controls are problematic.[6] Instead, here we will rely on imputed positive controls, created by shifting the estimated effect sizes for the negative controls. We assume the negative controls have a true effect size of 1, so to simulate the estimated effect size when the true effect size is \(\theta\) we multiply the estimate by \(\theta\). For example, if for a negative control a method produces an effect size estimate of 1.1, for a positive control with true effect size of 2 the estimated effect size becomes 1.1 x 2 = 2.2. This approach makes strong assumptions on the nature of the systematic error, most importantly that systematic error does not change as a function of the true effect size. Although this assumption is likely not to hold in the real world, imputing positive controls allows us to provide some indication of what type 2 error to expect for various true effect sizes. For each negative control we will impute positive controls with true effect sizes of 1.5, 2, and 4, so using the 93 negative controls we are able to construct 93 \(\times\) 3 = 279 positive control outcomes. This increased true effect is applied both for the first and second injection of multi-dose vaccines.

8.1.4 Real-world positive control outcome for COVID-19 vaccines

In addition to the negative control and imputed positive control outcomes, we will further investigate the risk of myocarditis or pericarditis following COVID-19 vaccination as a real-world positive control outcome [8]. Lee et al. used a historical comparator design and demonstrated an increased risk after vaccination across several administrative claims data sources. Goddard et al. used a concurrent comparator design and found an elevated risk ratio 0 - 7 days post vaccination of 3.3 (95% confidence interval, 1.5 - 7.0) summing across both BNT162b2 and mRNA-1273 vaccines. We will use this positive control to evaluate each method’s detection statistical performance and timeliness to detect an increased association. Our cohort definition (see Appendix) for myocarditis or pericarditis was developed and validated across 26 data sources [10]

8.2 Data sources

We will execute EUMAEUS as an OHDSI network study. All data partners within OHDSI are encouraged to participate voluntarily and can do so conveniently, because of the community’s shared Observational Medical Outcomes Partnership (OMOP) common data model (CDM) and OHDSI tool-stack. Many OHDSI community data partners have already committed to participate and we will recruit further data partners through OHDSI’s standard recruitment process, which includes protocol publication on OHDSI’s GitHub, an announcement in OHDSI’s research forum, presentation at the weekly OHDSI all-hands-on meeting and direct requests to data holders.

Table 8.2 lists the 5 already committed data sources for EUMAEUS; these sources encompass a large variety of practice types and populations. For each data source, we report a brief description and size of the population it represents. All data sources will receive institutional review board approval or exemption for their participation before executing EUMAEUS.

Table 8.2: Committed EUMAEUS data sources and the populations they cover.
Data source Population Patients History Data capture process and short description
Administrative claims
IBM MarketScan Commercial Claims and Encounters (CCAE) Commercially insured, < 65 years 142M 2000 – Adjudicated health insurance claims (e.g. inpatient, outpatient, and outpatient pharmacy) from large employers and health plans who provide private healthcare coverage to employees, their spouses and dependents.
IBM MarketScan Medicare Supplemental Database (MDCR) Commercially insured, 65+ years 10M 2000 – Adjudicated health insurance claims of retirees with primary or Medicare supplemental coverage through privately insured fee-for-service, point-of-service or capitated health plans.
IBM MarketScan Multi-State Medicaid Database (MDCD) Medicaid enrollees, racially diverse 26M 2006 – Adjudicated health insurance claims for Medicaid enrollees from multiple states and includes hospital discharge diagnoses, outpatient diagnoses and procedures, and outpatient pharmacy claims.
Optum Clinformatics Data Mart (Optum) Commercially or Medicare insured 85M 2000 – Inpatient and outpatient healthcare insurance claims.
Electronic health records (EHRs)
Optum Electronic Health Records (OptumEHR) US, general 93M 2006 – Clinical information, prescriptions, lab results, vital signs, body measurements, diagnoses and procedures derived from clinical notes using natural language processing.

8.3 Methods to evaluate

Vaccine safety surveillance methods can be broken down in to four components: construction of a counterfactual (often referred to as the ‘expected count’), a time-at-risk, the statistic to estimate, and potentially a decision rule on the estimate to classify signals from non-signals.

8.3.1 Counterfactual construction

Historic rates

Traditionally, vaccine surveillance methods compute an expected count based an incidence rate estimated during some historic time period, for example in the years prior to the initiation of the surveillance study. We will use the historic period indicated in Table 8.1. We will evaluate two variations:

  • Unadjusted, entire year. Using a single rate computed across the entire historic year for the entire population.
  • Age and sex adjusted, entire year. Using a rate stratifying by age (in 10 year increments) and sex, computed across the entire historic year. This allows the expected rate to be adjusted for the demographics of the vaccinated.
  • Unadjusted, time-at-risk relative to outpatient visit. Using a single rate computed during the time-at-risk relative to a random outpatient visit in the historic year.
  • Age and sex adjusted, time-at-risk relative to outpatient visit. Using a rate stratifying by age and sex, computed during the time-at-risk relative to a random outpatient visit in the historic year.

Initial results show that this counterfactual approach is sensitive to changes in coding practices. We therefore introduce a study diagnostic: the percent change in overall incidence rate (across the entire population) between the historic and current time period. For each of the four variations listed above, we add a new variation where effect-size estimates are removed if the change in incidence rate is greater than 50%.

Cohort method using a contemporary non-user comparator

A comparator cohort study most closely emulates a randomized clinical trial, comparing the target cohort (those vaccinated) to some comparator cohort. We define two types of non-user comparator cohort, one having an outpatient visit on the index date, and another having a random date as the index date. For both comparator variants we exclude subjects having a vaccinations for the same disease as the target vaccine on or before the index date. When doing unadjusted comparisons, the comparator cohort will be a random sample of equal size as the target cohort. When doing propensity score (PS) adjusted comparisons, the comparator cohort will be a stratified (by age and sex) random sample of four times the size of the target cohort (two times for the Seasonal Flu Vaccination (all) target cohort for computational reasons). Propensity models will use a large generic set of covariates, including demographics and covariates per drug, condition, procedure, measurement, etc., and will be fitted using large-scale regularized regression as described previously. [11] We will evaluate 10 method variations:

Anchoring the comparator on a random outpatient visit:

  • Unadjusted comparison.
  • 1-on-1 PS matching.
  • PS stratification. Five equally-sized strata will be defined in the target (vaccinated) population.
  • Inverse Probability of Treatment Weighting (IPTW). We will use stabilized weights to compute the average treatment effect in the treated (ATT). Weights will be truncated to a maximum value of 10, similar to Izurieta et al. (2020). [12]
  • 1-on-1 PS matching within each period, using only the ‘new’ data in that period to fit the propensity model. Once a population is matched in a period, that matching will be carried forward to subsequent periods. This method will only be evaluated using the H1N1pdm vaccinations for computational reasons.

Anchoring the comparator on a random date:

  • Unadjusted comparison.
  • 1-on-1 PS matching.
  • PS stratification.
  • IPTW with trimming.
  • 1-on-1 PS matching within each period. This method will only be evaluated using the H1N1pdm vaccinations for computational reasons.

Self-Controlled Case Series (SCCS) / Self-Controlled Risk Interval (SCRI)

The SCCS and SCRI designs are self-controlled, comparing the time-at-risk (the time shortly following the vaccination) to some other time in the same patient’s record. The SCCS design uses all patient time when not at risk as the control time. [13] The SCRI design uses a pre-specified control interval relative to the vaccination date as the control time. [14] This unexposed time can be both before or after the time at risk. We will evaluate five variations:

  • A simple SCCS, using all patient time when not at risk as the control time, with the exception of the 30 days prior to vaccination which is excluded from the analysis to avoid bias due to contra-indications.
  • An SCCS adjusting for age and season. Age and season will be modeled to be constant within each calendar month, and vary across months as bicubic splines.
  • A simple SCCS discarding all time prior to vaccination.
  • An SCRI, using a control interval of 43 to 15 days prior to vaccination.
  • An SCRI, using a control interval of 43 to 71 days after to vaccination.

Case-control

The case-control design compares cases (those with the outcome) to controls (those that do not have the outcome), and looks back in time for exposures to a vaccine. We will evaluate two variants:

  • Using up to four age and sex matched controls per case. For age we will use a two-year caliper.
  • By sampling controls from the general non-case population, and adjusting for age and sex in the outcome model. The control sample will be four times the number of controls. Age will be modeled as one variable per 5-year age category.

Concurrent comparator

Risk of confounding and biased estimation due to differences between patients who choose to receive a vaccine or when they choose to obtain the vaccine remain.
The concurrent comparator method aims to control for this bias by predefining a risk interval and designating patients in the exposure target cohort as vaccinated patients who were more recently vaccinated at the time of observation and patients in the control group as vaccinated comparators who had been vaccinated at an earlier time [15]. We will evaluate three variants that cover two different uses of the concurrent comparator method in the literature [8] and have a time-at-risk equal to our prior methods comparison [16]:

  • Using target patients who had been vaccinated 0-7 days prior as compared to comparator patients on the same day who had been vaccinated 22-42 days
  • Target: 1-21 days vs comparator: 22-42 days
  • Target: 1-28 days vs comparator: 29-56 days

8.3.2 Time-at-risk

The time-at-risk is the time window, relative to the vaccination date, when outcomes will potentially be attributed to the vaccine. We define three time-at-risk windows: 1-28 days, 1-42 days, and 0-1 days after vaccination for the historical rates, cohort, SCCS and case-control methods. For the concurrent comparator method we define 0-7 days, 1-21 days and 1-28 days time-at-risk windows for greater comparability with the literature. Time-at-risk windows will be constructed both for the first and second dose. The time-at-risk for one dose will be censored at the time of the next dose.

8.3.3 Statistic

  • Effect-size estimate. Each method can be used to produce an effect-size estimates such as a hazard ratio, incidence rate ratio, or odds ratio. For example, when using a historic rate we can compute the observed to expected ratio, which can be interpreted as the incidence rate ratio.

  • Log likelihood ratio (LLR). A common practice in vaccine safety surveillance is to computer the LLR, which is the log of the ratio between the likelihood of the alternative hypothesis (that there is an effect) and the likelihood of the null hypothesis (of no effect). The LLR is a convenient statistic when performing sequential testing, where the LLR can be compared to a pre-computed critical value, as is done in the MaxSPRT method. [17] Although typically MaxSPRT uses a historic rate as counterfactual, any counterfactual can be used to compute the LLR and can be used in MaxSPRT.

Effect-size estimates will be computed both with and without empirical calibration. [18,19] Empirical calibration will be done using leave-one-out: when calibrating the estimate for a control, the systematic error distribution will be fitted uses all controls except the one being calibrated.

8.3.4 Decision rule

To identify ‘signals’ we need a decision rule, for example in the shape of a threshold value on one of the estimates statistics. In our experiment we will consider one decision rule, which is the critical value computed for the LLR at an alpha of 0.05. For the historical rates method we will use a Poisson model assuming the counterfactual is known without uncertainty. For all other methods we will use a binomial model. All critical values will be computed using the Sequential package in CRAN.

8.4 Metrics

Similar to our previous study, we will compute the following metrics based on the effect size estimates: [20]

  • Area Under the receiver-operator Curve (AUC). The ability to discriminate between positive controls and negative controls based on the point estimate of the effect size. This will be stratified by true effect size of the positive controls.
  • Coverage. How often the true effect size is within the 95% confidence interval.
  • Mean precision. Precision is computed as 1 / (standard error)2, higher precision means narrower confidence intervals. We use the geometric mean to account for the skewed distribution of the precision.
  • Mean squared error (MSE). Mean squared error between the log of the effect size point-estimate and the log of the true effect size.
  • Type 1 error. For negative controls, how often was the null rejected (at alpha = 0.05). This is equivalent to the false positive rate and 1 - specificity.
  • Type 2 error. For positive controls, how often was the null not rejected (at alpha = 0.05). This is equivalent to the false negative rate and 1 - sensitivity. This will be stratified by true effect size of the positive controls.
  • Non-estimable. For how many of the controls was the method unable to produce an estimate? There can be various reasons why an estimate cannot be produced, for example because there were no subjects left after propensity score matching, or because no subjects remained having the outcome.

In addition, based on the MaxSPRT decision rule, we will compute sensitivity, specificity, as well as the number of months until 80% of all positive controls exceeds the critical value (detection time). These will be stratified by true effect size of the positive controls.

8.4.1 Timeliness

To understand the time it takes for a method the identify signals, the study period for each vaccine will be divided into calendar months. For each month the methods will be executed using the data that had accumulated up to the end of that month, and the performance metrics will be reported for each month.

8.4.2 Multiple doses

For the zoster and HPV vaccines requiring multiple doses separated by multiple months, metrics will be computed three times:

  • Treating all doses the same, so computing statistics using both doses without distinguishing between first and second.
  • Using the first dose only
  • Using the second dose only

8.5 Overview of analyses

In total, we will evaluate:

  • 14 counterfactuals
  • 3 times at risk (0-1, 1-28 and 1-42 days, or 0-7, 1-21, 1-28 days for the concurrent comparator)
  • 6 vaccines, with a total of 9 + 9 + 9 + 9 + 12 + 12 = 60 time periods
  • 93 negative controls
  • 3 \(\times\) 93 = 279 synthetic positive controls
  • 1 real-world positive control outcome
  • 3 dose definitions (both, first, second) for the zoster and HPV vaccines, 1 for H1N1pdm, seasonal flu, the two COVID-19 vaccines

Resulting in a total of 14 \(\times\) 3 \(\times\) [(9 + 9 + 9 + 9) \(\times\) 1 + (12 + 12) \(\times\) 3] \(\times\) (93 + 279 + 1) = 118,117,440 effect-size estimates. Each estimate will contain:

  • The effect-size estimate (e.g. hazard ratio, incidence rate ratio, odds ratio) with 95% confidence interval and p-value.
  • The empirically calibrated effect-size estimate and p-value
  • The LLR

This will be computed for each database.

9 Strengths and Limitations

9.1 Strengths

  • Cohort studies allow direct estimation of incidence rates following exposure of interest, and the new-user design can capture early events following treatment exposures while avoiding confounding from previous treatment effects; new use allows for a clear exposure index date.
  • Large-scale propensity score matching and stratification create balance on a large number of baseline potential confounders and have been found in the past to balance unmeasured confounders.
  • Systematic processes including a pre-specified selection of covariates avoids investigator-specific biases in variable selection.
  • Use of real negative and imputed positive control outcomes provides an independent estimate of residual bias in the experiment.
  • The fully specified study protocol is being published before analysis begins.
  • Dissemination of the results will not depend on estimated effects, avoiding publication bias.
  • All analytic methods have previously been verified on real data.
  • All software is freely available as open source.
  • Use of a common data model allows extension of the experiment to future databases and allows replication of these results on licensable databases that were used in this experiment, while still maintaining patient privacy on patient-level data.
  • Use of multiple databases allows estimating consistency to add credibility and supports generalizability.

9.2 Limitations

  • Even though many potential confounders will be included in this study, there may be residual bias due to unmeasured or misspecified confounders, such as confounding by indication, differences in physician characteristics that may be associated with drug choice, concomitant use of other drugs started after the index date, and informative censoring at the end of the on-treatment periods. To minimize this risk, we used methods to detect residual bias through our negative and positive controls.
  • Our follow-up times are limited and variable, potentially reducing power to detect differences in effectiveness and safety.
  • We assume hazards are not time varying.
  • Misclassification of study variables is unavoidable in secondary use of health data, so it is possible to misclassify treatments, covariates, and outcomes; we do not expect differential misclassification, so bias will most likely be towards the null.
  • The electronic health record databases may be missing care episodes for patients due to care outside the respective health systems; bias will most likely be towards the null.

10 Protection of Human Subjects

EUMAEUS does not involve human subjects research. The project does, however, use de-identified human data collected during routine healthcare provision. All data partners executing the EUMAEUS studies within their data sources will have received institutional review board (IRB) approval or waiver for participation in accordance to their institutional governance prior to execution (see Table 10.1). EUMAEUS executes across a federated and distributed data network, where analysis code is sent to participating data partners and only aggregate summary statistics are returned, with no sharing of patient-level data between organizations.

Table 10.1: IRB approval or waiver statement from partners.
Data source Statement
IBM MarketScan Commercial Claims and Encounters (CCAE) New England Institutional Review Board and was determined to be exempt from broad IRB approval, as this research project did not involve human subject research.
IBM MarketScan Medicare Supplemental Database (MDCR) New England Institutional Review Board and was determined to be exempt from broad IRB approval, as this research project did not involve human subject research.
IBM MarketScan Multi-State Medicaid Database (MDCD) New England Institutional Review Board and was determined to be exempt from broad IRB approval, as this research project did not involve human subject research.
Optum Clinformatics Data Mart (Optum) New England Institutional Review Board and was determined to be exempt from broad IRB approval, as this research project did not involve human subject research.
Optum Electronic Health Records (OptumEHR) New England Institutional Review Board and was determined to be exempt from broad IRB approval, as this research project did not involve human subject research.

11 Management and Reporting of Adverse Events and Adverse Reactions

EUMAEUS uses coded data that already exist in electronic databases. In these types of databases, it is not possible to link (i.e., identify a potential causal association between) a particular product and medical event for any specific individual. Thus, the minimum criteria for reporting an adverse event (i.e., identifiable patient, identifiable reporter, a suspect product and event) are not available and adverse events are not reportable as individual adverse event reports. The study results will be assessed for medically important findings.

12 Plans for Disseminating and Communicating Study Results

Open science aims to make scientific research, including its data process and software, and its dissemination, through publication and presentation, accessible to all levels of an inquiring society, amateur or professional [21] and is a governing principle of EUMAEUS. Open science delivers reproducible, transparent and reliable evidence. All aspects of EUMAEUS (except private patient data) will be open and we will actively encourage other interested researchers, clinicians and patients to participate. This differs fundamentally from traditional studies that rarely open their analytic tools or share all result artifacts, and inform the community about hard-to-verify conclusions at completion.

12.1 Transparent and re-usable research tools

We will publicly register this protocol and announce its availability for feedback from stakeholders, the OHDSI community and within clinical professional societies. This protocol will link to open source code for all steps to generating diagnostics, effect estimates, figures and tables. Such transparency is possible because we will construct our studies on top of the OHDSI toolstack of open source software tools that are community developed and rigorously tested [20]. We will publicly host EUMAEUS source code at (https://github.com/ohdsi-studies/Eumaeus), allowing public contribution and review, and free re-use for anyone’s future research.

12.2 Continous sharing of results

EUMAEUS embodies a new approach to generating evidence from healthcare data that overcome weaknesses in the current process of answering and publishing (or not) one question at a time. Generating evidence for thousands of research and control questions using a systematic process enables us to not only evaluate that process and the coherence and consistency of the evidence, but also to avoid \(p\)-hacking and publication bias [6]. We will store and openly communicate all of these results as they become available using a user-friendly web-based app that serves up all descriptive statistics, study diagnostics and effect estimates for each cohort comparison and outcome. Open access to this app will be through a general public facing EUMAEUS web-page.

12.3 Scientific meetings and publications

We will deliver multiple presentations at scientific venues and will also prepare multiple scientific publications for clinical, informatics and statistical journals.

12.4 General public

We believe in sharing our findings that will guide clinical care with the general public. EUMAEUS will use social-media (Twitter) to facilitate this. With dedicated support from the OHDSI communications specialist, we will deliver regular press releases at key project stages, distributed via the extensive media networks of UCLA and Columbia.

References

1
The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP). Guide on Methodological Standards in Pharmacoepidemiology (Revision 8). 2010.
2
Accelerated Development of VAccine beNefit-risk Collaboration in Europe (ADVANCE). D4.2 Report on appraisal of vaccine safety methods. 2014.
3
European Medicines Agency (EMA). Consideration on core requirements for RMPs of COVID19 vaccines. 2020.
4
European Medicines Agency (EMA). Pharmacovigilance Plan of the EU Regulatory Network for COVID-19 Vaccines. 2020.
5
Schuemie MJ, Soledad Cepede M, Suchard MA, et al. How confident are we about observational findings in health care: A benchmark study. 2.1. 2020.
6
Schuemie MJ, Ryan PB, Hripcsak G, et al. Improving reproducibility by using high-throughput observational studies with empirical calibration. Philosophical transactions Series A, Mathematical, physical, and engineering sciences. 2018;376.
7
Wong H-L, Hu M, Zhou CK, et al. Risk of myocarditis and pericarditis after the COVID-19 mRNA vaccination in the USA: A cohort study in claims databases. The Lancet. 2022;399:2191–9.
8
Goddard K, Lewis N, Fireman B, et al. Risk of myocarditis and pericarditis following BNT162b2 and mRNA-1273 COVID-19 vaccination. Vaccine. 2022;40:5153–9.
9
Li X, Ostropolets A, Makadia R, et al. Characterising the background incidence rates of adverse events of special interest for covid-19 vaccines in eight countries: Multinational network cohort study. BMJ. 2021;373.
10
Voss EA, Shoaibi A, Yin Hui Lai L, et al. Contextualising adverse events of special interest to characterise the baseline incidence rates in 24 million patients with COVID-19 across 26 databases: A multinational retrospective cohort study. eClinicalMedicine. 2023;58.
11
Tian Y, Schuemie MJ, Suchard MA. Evaluating large-scale propensity score performance through real-world and synthetic data experiments. International journal of epidemiology. 2018;47:2005–14.
12
Izurieta HS, Lu M, Kelman J, et al. Comparative effectiveness of influenza vaccines among U.S. Medicare beneficiaries ages 65 years and older during the 2019-20 season. Clin Infect Dis. 2020.
13
Whitaker HJ, Farrington CP, Spiessens B, et al. Tutorial in biostatistics: the self-controlled case series method. Stat Med. 2006;25:1768–97.
14
Glanz JM, McClure DL, Xu S, et al. Four different study designs to evaluate vaccine safety were equally validated with contrasting limitations. J Clin Epidemiol. 2006;59:808–18.
15
Klein NP, Lewis N, Goddard K, et al. Surveillance for adverse events after COVID-19 mRNA vaccination. JAMA: the journal of the American Medical Association. 2021;326:1390–9.
16
Schuemie MJ, Arshad F, Pratt N, et al. Vaccine safety surveillance using routinely collected healthcare Data—An empirical evaluation of epidemiological designs. Frontiers in pharmacology. 2022;13.
17
Kulldorff M, Davis RL, Kolczak† M, et al. A maximized sequential probability ratio test for drug and vaccine safety surveillance. Sequential Analysis. 2011;30:58–78.
18
Schuemie MJ, Ryan PB, DuMouchel W, et al. Interpreting observational studies: Why empirical calibration is needed to correct p-values. Statistics in medicine. 2014;33:209–18.
19
Schuemie MJ, Hripcsak G, Ryan PB, et al. Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare data. Proceedings of the National Academy of Sciences of the United States of America. 2018;115:2571–7.
20
Schuemie MJ, Cepeda MS, Suchard MA, et al. How confident are we about observational findings in health care: A benchmark study. Harvard Data Science Review. 2020;2.
21
Woelfle M, Olliaro P, Todd MH. Open science is a research accelerator. Nature chemistry. 2011;3:745–8.

Appendix

A Exposure Cohort Definitions

A.1 H1N1pdm Vaccines

A.1.1 Cohort Entry Events

People enter the cohort when observing any of the following:

  1. drug exposures of ‘H1N1 vaccine’, starting between September 1, 2009 and May 31, 2010.

Limit cohort entry events to the earliest event per person.

A.1.2 Cohort Exit

The cohort end date will be offset from index event’s start date plus 0 days.

A.1.3 Cohort Eras

Entry events will be combined into cohort eras if they are within 0 days of each other.

A.1.4 Concept set: H1N1 vaccine

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
40213187 Novel influenza-H1N1-09, all formulations 128 CVX NO YES NO
40166607 influenza A-California-7-2009-(H1N1)v-like virus vaccine 0.03 MG/ML Injectable Suspension 864704 RxNorm NO YES NO
40166130 0.25 ML influenza A-California-7-2009-(H1N1)v-like virus vaccine 0.03 MG/ML Prefilled Syringe 864781 RxNorm NO YES NO
40166144 0.5 ML influenza A-California-7-2009-(H1N1)v-like virus vaccine 0.03 MG/ML Prefilled Syringe 864797 RxNorm NO YES NO
42902936 influenza A-California-7-2009-(H1N1)v-like virus vaccine 0.03 MG/ML Prefilled Syringe 1360049 RxNorm NO YES NO
40240135 influenza A-California-7-2009-(H1N1)v-like virus vaccine 0.09 MG/ML 1111367 RxNorm NO YES NO
40225009 influenza A-California-7-2009-(H1N1)v-like virus vaccine 0.12 MG/ML 1005949 RxNorm NO YES NO
40166608 influenza A-California-7-2009-(H1N1)v-like virus vaccine 158000000 UNT/ML 864812 RxNorm NO YES NO
45776785 influenza A-California-7-2009-(H1N1)v-like virus vaccine 50000000 MG/ML 1543758 RxNorm NO YES NO
40166609 influenza A-California-7-2009-(H1N1)v-like virus vaccine Injectable Suspension 864703 RxNorm NO YES NO
40166611 influenza A-California-7-2009-(H1N1)v-like virus vaccine Prefilled Syringe 864780 RxNorm NO YES NO

A.2 Seasonal Flu Vaccines (Fluvirin)

A.2.1 Cohort Entry Events

People enter the cohort when observing any of the following:

  1. drug exposures of ‘Fluvirin’, starting between September 1, 2017 and May 31, 2018.

Limit cohort entry events to the earliest event per person.

A.2.2 Cohort Exit

The cohort end date will be offset from index event’s start date plus 0 days.

A.2.3 Cohort Eras

Entry events will be combined into cohort eras if they are within 0 days of each other.

A.2.4 Concept set: Fluvirin

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
1593906 influenza A virus A/Hong Kong/4801/2014 (H3N2) antigen 0.03 MG/ML / influenza A virus A/Singapore/GP1908/2015 (H1N1) antigen 0.03 MG/ML / influenza B virus B/Brisbane/60/2008 antigen 0.03 MG/ML [Fluvirin 2017-2018] 1928971 RxNorm NO YES NO

A.3 Seasonal Flu Vaccines (Fluzone)

A.3.1 Cohort Entry Events

People enter the cohort when observing any of the following:

  1. drug exposures of ‘Fluzone’, starting between September 1, 2017 and May 31, 2018.

Limit cohort entry events to the earliest event per person.

A.3.2 Cohort Exit

The cohort end date will be offset from index event’s start date plus 0 days.

A.3.3 Cohort Eras

Entry events will be combined into cohort eras if they are within 0 days of each other.

A.3.4 Concept set: Fluzone

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
1593354 influenza A virus A/Hong Kong/4801/2014 (H3N2) antigen 0.12 MG/ML / influenza A virus A/Michigan/45/2015 (H1N1) antigen 0.12 MG/ML / influenza B virus B/Brisbane/60/2008 antigen 0.12 MG/ML [Fluzone 2017-2018] 1928341 RxNorm NO YES NO

A.4 Seasonal Flu Vaccines (All)

A.4.1 Cohort Entry Events

People enter the cohort when observing any of the following:

  1. drug exposures of ‘Seasonal flu vaccine’, starting between September 1, 2017 and May 31, 2018.

Limit cohort entry events to the earliest event per person.

A.4.2 Cohort Exit

The cohort end date will be offset from index event’s start date plus 0 days.

A.4.3 Cohort Eras

Entry events will be combined into cohort eras if they are within 0 days of each other.

A.4.4 Concept set: Seasonal flu vaccine

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
40213145 influenza, injectable, quadrivalent, contains preservative 158 CVX NO YES NO
42903442 influenza B virus 1312376 RxNorm NO YES NO
40213150 influenza, live, intranasal, quadrivalent 149 CVX NO YES NO
40213159 influenza virus vaccine, whole virus 16 CVX NO YES NO
40225028 influenza virus vaccine, inactivated A-Victoria-210-2009 X-187 (H3N2) (A-Perth-16-2009) strain 1005931 RxNorm NO YES NO
40213156 influenza virus vaccine, split virus (incl. purified surface antigen)-retired CODE 15 CVX NO YES NO
40213151 Seasonal, trivalent, recombinant, injectable influenza vaccine, preservative free 155 CVX NO YES NO
40213327 influenza nasal, unspecified formulation 151 CVX NO YES NO
40213148 influenza, intradermal, quadrivalent, preservative free, injectable 166 CVX NO YES NO
40213158 influenza virus vaccine, unspecified formulation 88 CVX NO YES NO
36878713 Influenza Virus Fragmented, Inactivated, Strain B / Phuket / 3073/2013 OMOP989577 RxNorm Extension NO YES NO
42873961 influenza B virus vaccine, B-Wisconsin-1-2010-like virus 1303855 RxNorm NO YES NO
40225038 influenza virus vaccine, live attenuated, A-Perth-16-2009 (H3N2) strain 1005911 RxNorm NO YES NO
40213146 Influenza, injectable, quadrivalent, preservative free 150 CVX NO YES NO
40213143 Influenza, injectable, Madin Darby Canine Kidney, preservative free, quadrivalent 171 CVX NO YES NO
36879025 Influenza Virus Surface Antigens, strain A / Switzerland / 9715293/2013 H3N2 - Analogue Strain Nib-88 OMOP991645 RxNorm Extension NO YES NO
40213157 Seasonal trivalent influenza vaccine, adjuvanted, preservative free 168 CVX NO YES NO
45776076 influenza A virus vaccine, A-Texas-50-2012 (H3N2)-like virus 1541617 RxNorm NO YES NO
40213149 influenza virus vaccine, live, attenuated, for intranasal use 111 CVX NO YES NO
40213147 Influenza, injectable,quadrivalent, preservative free, pediatric 161 CVX NO YES NO
40213152 Seasonal, quadrivalent, recombinant, injectable influenza vaccine, preservative free 185 CVX NO YES NO
42903441 influenza A virus 1312375 RxNorm NO YES NO
40213141 influenza, high dose seasonal, preservative-free 135 CVX NO YES NO
40213153 Influenza, seasonal, injectable 141 CVX NO YES NO
40213144 Influenza, injectable, Madin Darby Canine Kidney, quadrivalent with preservative 186 CVX NO YES NO
40213142 Influenza, injectable, Madin Darby Canine Kidney, preservative free 153 CVX NO YES NO
40213155 seasonal influenza, intradermal, preservative free 144 CVX NO YES NO
40164828 influenza B virus vaccine B/Brisbane/60/2008 antigen 857921 RxNorm NO YES NO

A.5 HPV Vaccines

A.5.1 Cohort Entry Events

People enter the cohort when observing any of the following:

  1. drug exposures of ‘Gardasil 9’, starting between January 1, 2018 and December 31, 2018.

A.5.2 Cohort Exit

The cohort end date will be offset from index event’s start date plus 0 days.

A.5.3 Cohort Eras

Entry events will be combined into cohort eras if they are within 0 days of each other.

A.5.4 Concept set: Gardasil 9

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
36248866 Gardasil 9 Injectable Product 1597098 RxNorm NO YES NO
45892513 L1 protein, human papillomavirus type 11 vaccine / L1 protein, human papillomavirus type 16 vaccine / L1 protein, human papillomavirus type 18 vaccine / L1 protein, human papillomavirus type 31 vaccine / L1 protein, human papillomavirus type 33 vaccine / 1597102 RxNorm NO YES NO
45892514 0.5 ML L1 protein, human papillomavirus type 11 vaccine 0.08 MG/ML / L1 protein, human papillomavirus type 16 vaccine 0.12 MG/ML / L1 protein, human papillomavirus type 18 vaccine 0.08 MG/ML / L1 protein, human papillomavirus type 31 vaccine 0.04 MG/ML / 1597103 RxNorm NO YES NO
45892510 0.5 ML L1 protein, human papillomavirus type 11 vaccine 0.08 MG/ML / L1 protein, human papillomavirus type 16 vaccine 0.12 MG/ML / L1 protein, human papillomavirus type 18 vaccine 0.08 MG/ML / L1 protein, human papillomavirus type 31 vaccine 0.04 MG/ML / 1597099 RxNorm NO YES NO
40213322 Human Papillomavirus 9-valent vaccine 165 CVX NO YES NO

A.6 Zoster Vaccines

A.6.1 Cohort Entry Events

People enter the cohort when observing any of the following:

  1. drug exposures of ‘Shingrix’, starting between January 1, 2018 and December 31, 2018.

A.6.2 Cohort Exit

The cohort end date will be offset from index event’s start date plus 0 days.

A.6.3 Cohort Eras

Entry events will be combined into cohort eras if they are within 0 days of each other.

A.6.4 Concept set: Shingrix

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
792784 varicella zoster virus glycoprotein E Injection [Shingrix] 1986828 RxNorm NO YES NO
792783 varicella zoster virus glycoprotein E, recombinant 0.1 MG/ML [Shingrix] 1986827 RxNorm NO YES NO
792788 varicella zoster virus glycoprotein E, recombinant 0.1 MG/ML Injection [Shingrix] 1986832 RxNorm NO YES NO
36421491 Varicella-Zoster Virus Vaccine Live (Oka-Merck) strain Injectable Solution [Shingrix] OMOP4763774 RxNorm Extension NO YES NO
792785 Shingrix Injectable Product 1986829 RxNorm NO YES NO
706103 zoster vaccine recombinant 187 CVX NO YES NO

A.7 COVID-19 mRNA vaccine (BNT162b2)

A.7.1 Cohort Entry Events

People may enter the cohort when observing any of the following:

  1. drug exposure of ‘Comirnaty Pfizer COVID-19 vaccine for initial dose adult’ for the first time in the person’s history; having no drug exposures of ‘All COVID-19 vaccines’, starting anytime prior to ‘Comirnaty Pfizer COVID-19 vaccine for initial dose adult’ start date; allow events outside observation period.

  2. drug exposures of ‘Comirnaty Pfizer COVID-19 vaccine for initial dose adult’; with all of the following criteria:

  3. having at least 1 drug exposure of ‘Comirnaty Pfizer COVID-19 vaccine for initial dose adult’ for the first time in the person’s history, starting between 48 days before and 14 days before ‘Comirnaty Pfizer COVID-19 vaccine for initial dose adult’ start date; having no drug exposures of ‘All COVID-19 vaccines’, starting anytime prior to ‘Comirnaty Pfizer COVID-19 vaccine for initial dose adult’ start date; allow events outside observation period.

  4. having no drug exposures of ‘Comirnaty Pfizer COVID-19 vaccine for initial dose adult’, starting anytime prior to ‘Comirnaty Pfizer COVID-19 vaccine for initial dose adult’ start date; having at least 1 drug exposure of ‘Comirnaty Pfizer COVID-19 vaccine for initial dose adult’ for the first time in the person’s history, starting between 48 days before and 14 days before ‘Comirnaty Pfizer COVID-19 vaccine for initial dose adult’ start date.

A.7.2 Inclusion Criteria

A.7.2.1 1. age >= 16 with start after 11Dec2020 (EUA date)

Entry events with the following event criteria: who are >= 16 years old; starting on or after December 11, 2020.

A.7.2.2 2. has >365d prior observation

Entry events having at least 1 observation period, starting anytime up to 365 days before cohort entry start date and ending between 0 days before and all days after cohort entry start date.

A.7.3 Cohort Exit

The cohort end date will be offset from index event’s start date plus 1 day.

A.7.4 Cohort Eras

Entry events will be combined into cohort eras if they are within 0 days of each other.

A.7.5 Concept set: All COVID-19 vaccines

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
702664 SARS-COV-2 COVID-19 Non-US Vaccine, Specific Product Unknown 500 CVX NO YES NO
702665 SARS-COV-2 COVID-19 Inactivated Virus Non-US Vaccine Product (QAZCOVID-IN) 501 CVX NO YES NO
702666 SARS-COV-2 COVID-19 Inactivated Virus Non-US Vaccine Product (COVAXIN) 502 CVX NO YES NO
702667 SARS-COV-2 COVID-19 Viral Vector Non-replicating Non-US Vaccine Product (Sputnik Light) 504 CVX NO YES NO
702668 SARS-COV-2 COVID-19 Viral Vector Non-replicating Non-US Vaccine Product (Sputnik V) 505 CVX NO YES NO
702669 SARS-COV-2 COVID-19 Viral Vector Non-replicating Non-US Vaccine Product (CanSino Biological Inc./Beijing Institute of Biotechnology) 506 CVX NO YES NO
702670 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (Anhui Zhifei Longcom Biopharmaceutical + Institute of Microbiology, Chinese Academy of Sciences) 507 CVX NO YES NO
702671 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (EpiVacCorona) 509 CVX NO YES NO
702672 SARS-COV-2 COVID-19 Inactivated Virus Non-US Vaccine Product (BIBP, Sinopharm) 510 CVX NO YES NO
702673 SARS-COV-2 COVID-19 Inactivated Virus Non-US Vaccine Product (CoronaVac, Sinovac) 511 CVX NO YES NO
702676 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, preservative free, 3 mcg/0.2mL dose, tris-sucrose formulation 219 CVX NO YES NO
702677 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, preservative free, 30 mcg/0.3mL dose, tris-sucrose formulation 217 CVX NO YES NO
702678 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, preservative free, 10 mcg/0.2mL dose, tris-sucrose formulation 218 CVX NO YES NO
702680 SARS-COV-2 COVID-19 Live Attenuated Virus Non-US Vaccine Product (COVIVAC) 503 CVX NO YES NO
702681 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (Jiangsu Province Centers for Disease Control and Prevention) 508 CVX NO YES NO
722117 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, bivalent booster, preservative free, 10 mcg/0.2 mL dose, tris-sucrose formulation 301 CVX NO YES NO
722118 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, bivalent booster, preservative free, 10 mcg/0.2 mL dose 230 CVX NO YES NO
722119 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, bivalent, preservative free, 3 mcg/0.2 mL dose, tris-sucrose formulation 302 CVX NO YES NO
724904 SARS-COV-2 (COVID-19) vaccine, UNSPECIFIED 213 CVX NO YES NO
724905 SARS-COV-2 (COVID-19) vaccine, vector non-replicating, recombinant spike protein-ChAdOx1, preservative free, 0.5 mL 210 CVX NO YES NO
739902 SARS-COV-2 (COVID-19) vaccine, vector non-replicating 2479831 RxNorm NO YES NO
778259 SARS-COV-2 COVID-19 Virus Like Particle (VLP) Non-US Vaccine Product (Medicago, Covifenz) 512 CVX NO YES NO
778260 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (Anhui Zhifei Longcom, Zifivax) 513 CVX NO YES NO
778261 SARS-COV-2 COVID-19 DNA Non-US Vaccine Product (Zydus Cadila, ZyCoV-D) 514 CVX NO YES NO
778262 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (Medigen, MVC-COV1901) 515 CVX NO YES NO
778263 SARS-COV-2 COVID-19 Inactivated Non-US Vaccine Product (Minhai Biotechnology Co, KCONVAC) 516 CVX NO YES NO
778264 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (Biological E Limited, Corbevax) 517 CVX NO YES NO
780152 SARS-COV-2 (COVID-19) vaccine, subunit, recombinant spike protein 2606074 RxNorm NO YES NO
905418 SARS-COV-2 (COVID-19) vaccine, D614, prefusion spike recombinant protein subunit (CoV2 preS dTM), AS03 adjuvant added, preservative free, 5mcg/0.5mL dose 225 CVX NO YES NO
905419 SARS-COV-2 (COVID-19) vaccine, D614, prefusion spike recombinant protein subunit (CoV2 preS dTM), AS03 adjuvant added, preservative free, 10mcg/0.5mL dose 226 CVX NO YES NO
35894915 COVID-19 vaccine OMOP5042939 RxNorm Extension NO YES NO
36118948 COVID-19 vaccine, whole virus, inactivated, adjuvanted with Alum and CpG 1018 OMOP5051441 RxNorm Extension NO YES NO
36118949 COVID-19 vaccine, recombinant, full-length nanoparticle spike (S) protein, adjuvanted with Matrix-M OMOP5051442 RxNorm Extension NO YES NO
36126197 COVID-19 vaccine, recombinant, plant-derived Virus-Like Particle (VLP) spike (S) protein, adjuvanted with AS03 OMOP5051443 RxNorm Extension NO YES NO
37003432 SARS-CoV-2 (COVID-19) vaccine, mRNA spike protein 2468231 RxNorm NO YES NO
739902 SARS-COV-2 (COVID-19) vaccine, vector non-replicating 2479831 RxNorm NO YES NO
780152 SARS-COV-2 (COVID-19) vaccine, subunit, recombinant spike protein 2606074 RxNorm NO YES NO

A.7.6 Concept set: Comirnaty Pfizer COVID-19 vaccine for initial dose adult

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
37003433 SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.1 MG/ML 2468232 RxNorm NO YES NO
702117 SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.05 MG/ML 2583742 RxNorm YES YES NO
742039 SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.0075 MG/ML 2623380 RxNorm YES YES NO
779947 SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.015 MG/ML 2603832 RxNorm YES YES NO
742008 SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 0.025 MG/ML 2621899 RxNorm YES YES NO
742040 SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 OMICRON (BA.4/BA.5) 0.0075 MG/ML 2623381 RxNorm YES YES NO
1525544 SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 OMICRON (BA.4/BA.5) 0.05 MG/ML 2610346 RxNorm YES YES NO
742007 SARS-CoV-2 (COVID-19) vaccine, mRNA-BNT162b2 OMICRON (BA.4/BA.5) 0.025 MG/ML 2621898 RxNorm YES YES NO

A.8 COVID-19 mRNA vaccine (mRNA-1273)

A.8.1 Cohort Entry Events

People may enter the cohort when observing any of the following:

  1. drug exposure of ‘Spikevax Moderna COVID-19 vaccine for initial dose adult’ for the first time in the person’s history; having no drug exposures of ‘All COVID-19 vaccines’, starting anytime prior to ‘Spikevax Moderna COVID-19 vaccine for initial dose adult’ start date; allow events outside observation period.

  2. drug exposures of ‘Spikevax Moderna COVID-19 vaccine for initial dose adult’; with all of the following criteria:

  3. having at least 1 drug exposure of ‘Spikevax Moderna COVID-19 vaccine for initial dose adult’ for the first time in the person’s history, starting between 48 days before and 14 days before ‘Spikevax Moderna COVID-19 vaccine for initial dose adult’ start date; having no drug exposures of ‘All COVID-19 vaccines’, starting anytime prior to ‘Spikevax Moderna COVID-19 vaccine for initial dose adult’ start date; allow events outside observation period.

  4. having no drug exposures of ‘Spikevax Moderna COVID-19 vaccine for initial dose adult’, starting anytime prior to ‘Spikevax Moderna COVID-19 vaccine for initial dose adult’ start date; having at least 1 drug exposure of ‘Spikevax Moderna COVID-19 vaccine for initial dose adult’ for the first time in the person’s history, starting between 48 days before and 14 days before ‘Spikevax Moderna COVID-19 vaccine for initial dose adult’ start date.

A.8.2 Inclusion Criteria

A.8.2.1 1. age >= 18 with start after 18Dec2020 (EUA date)

Entry events with the following event criteria: who are >= 18 years old; starting on or after December 18, 2020.

A.8.2.2 2. has >365d prior observation

Entry events having at least 1 observation period, starting anytime up to 365 days before cohort entry start date and ending between 0 days before and all days after cohort entry start date.

A.8.3 Cohort Exit

The cohort end date will be offset from index event’s start date plus 1 day.

A.8.4 Cohort Eras

Entry events will be combined into cohort eras if they are within 0 days of each other.

A.8.5 Concept set: All COVID-19 vaccines

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
702664 SARS-COV-2 COVID-19 Non-US Vaccine, Specific Product Unknown 500 CVX NO YES NO
702665 SARS-COV-2 COVID-19 Inactivated Virus Non-US Vaccine Product (QAZCOVID-IN) 501 CVX NO YES NO
702666 SARS-COV-2 COVID-19 Inactivated Virus Non-US Vaccine Product (COVAXIN) 502 CVX NO YES NO
702667 SARS-COV-2 COVID-19 Viral Vector Non-replicating Non-US Vaccine Product (Sputnik Light) 504 CVX NO YES NO
702668 SARS-COV-2 COVID-19 Viral Vector Non-replicating Non-US Vaccine Product (Sputnik V) 505 CVX NO YES NO
702669 SARS-COV-2 COVID-19 Viral Vector Non-replicating Non-US Vaccine Product (CanSino Biological Inc./Beijing Institute of Biotechnology) 506 CVX NO YES NO
702670 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (Anhui Zhifei Longcom Biopharmaceutical + Institute of Microbiology, Chinese Academy of Sciences) 507 CVX NO YES NO
702671 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (EpiVacCorona) 509 CVX NO YES NO
702672 SARS-COV-2 COVID-19 Inactivated Virus Non-US Vaccine Product (BIBP, Sinopharm) 510 CVX NO YES NO
702673 SARS-COV-2 COVID-19 Inactivated Virus Non-US Vaccine Product (CoronaVac, Sinovac) 511 CVX NO YES NO
702676 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, preservative free, 3 mcg/0.2mL dose, tris-sucrose formulation 219 CVX NO YES NO
702677 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, preservative free, 30 mcg/0.3mL dose, tris-sucrose formulation 217 CVX NO YES NO
702678 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, preservative free, 10 mcg/0.2mL dose, tris-sucrose formulation 218 CVX NO YES NO
702680 SARS-COV-2 COVID-19 Live Attenuated Virus Non-US Vaccine Product (COVIVAC) 503 CVX NO YES NO
702681 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (Jiangsu Province Centers for Disease Control and Prevention) 508 CVX NO YES NO
722117 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, bivalent booster, preservative free, 10 mcg/0.2 mL dose, tris-sucrose formulation 301 CVX NO YES NO
722118 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, bivalent booster, preservative free, 10 mcg/0.2 mL dose 230 CVX NO YES NO
722119 SARS-COV-2 (COVID-19) vaccine, mRNA, spike protein, LNP, bivalent, preservative free, 3 mcg/0.2 mL dose, tris-sucrose formulation 302 CVX NO YES NO
724904 SARS-COV-2 (COVID-19) vaccine, UNSPECIFIED 213 CVX NO YES NO
724905 SARS-COV-2 (COVID-19) vaccine, vector non-replicating, recombinant spike protein-ChAdOx1, preservative free, 0.5 mL 210 CVX NO YES NO
739902 SARS-COV-2 (COVID-19) vaccine, vector non-replicating 2479831 RxNorm NO YES NO
778259 SARS-COV-2 COVID-19 Virus Like Particle (VLP) Non-US Vaccine Product (Medicago, Covifenz) 512 CVX NO YES NO
778260 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (Anhui Zhifei Longcom, Zifivax) 513 CVX NO YES NO
778261 SARS-COV-2 COVID-19 DNA Non-US Vaccine Product (Zydus Cadila, ZyCoV-D) 514 CVX NO YES NO
778262 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (Medigen, MVC-COV1901) 515 CVX NO YES NO
778263 SARS-COV-2 COVID-19 Inactivated Non-US Vaccine Product (Minhai Biotechnology Co, KCONVAC) 516 CVX NO YES NO
778264 SARS-COV-2 COVID-19 Protein Subunit Non-US Vaccine Product (Biological E Limited, Corbevax) 517 CVX NO YES NO
780152 SARS-COV-2 (COVID-19) vaccine, subunit, recombinant spike protein 2606074 RxNorm NO YES NO
905418 SARS-COV-2 (COVID-19) vaccine, D614, prefusion spike recombinant protein subunit (CoV2 preS dTM), AS03 adjuvant added, preservative free, 5mcg/0.5mL dose 225 CVX NO YES NO
905419 SARS-COV-2 (COVID-19) vaccine, D614, prefusion spike recombinant protein subunit (CoV2 preS dTM), AS03 adjuvant added, preservative free, 10mcg/0.5mL dose 226 CVX NO YES NO
35894915 COVID-19 vaccine OMOP5042939 RxNorm Extension NO YES NO
36118948 COVID-19 vaccine, whole virus, inactivated, adjuvanted with Alum and CpG 1018 OMOP5051441 RxNorm Extension NO YES NO
36118949 COVID-19 vaccine, recombinant, full-length nanoparticle spike (S) protein, adjuvanted with Matrix-M OMOP5051442 RxNorm Extension NO YES NO
36126197 COVID-19 vaccine, recombinant, plant-derived Virus-Like Particle (VLP) spike (S) protein, adjuvanted with AS03 OMOP5051443 RxNorm Extension NO YES NO
37003432 SARS-CoV-2 (COVID-19) vaccine, mRNA spike protein 2468231 RxNorm NO YES NO
739902 SARS-COV-2 (COVID-19) vaccine, vector non-replicating 2479831 RxNorm NO YES NO
780152 SARS-COV-2 (COVID-19) vaccine, subunit, recombinant spike protein 2606074 RxNorm NO YES NO

A.8.6 Concept set: Spikevax Moderna COVID-19 vaccine for initial dose adult

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
1525542 SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 OMICRON (BA.4/BA.5) 0.05 MG/ML 2610327 RxNorm YES YES NO
37003517 SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.2 MG/ML 2470233 RxNorm NO YES NO
779678 SARS-CoV-2 (COVID-19) vaccine, mRNA spike protein Injectable Suspension [Spikevax] 2601551 RxNorm NO YES NO
779413 SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.1 MG/ML 2598699 RxNorm YES YES NO
1525540 SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.05 MG/ML 2610324 RxNorm YES YES NO
742037 SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 OMICRON (BA.4/BA.5) 0.025 MG/ML 2623377 RxNorm YES YES NO
742036 SARS-CoV-2 (COVID-19) vaccine, mRNA-1273 0.025 MG/ML 2623376 RxNorm YES YES NO

B Real-world outcome cohort definition for COVID-19 vaccines

B.1 Adverse event outcome - myocarditis or pericarditis

B.1.1 Cohort Entry Events

People may enter the cohort when observing any of the following:

  1. condition occurrences of ‘Myocarditis Pericarditis’.

B.1.2 Inclusion Criteria

B.1.2.1 1. has no events in prior ‘clean window’ - 365 days

Entry events having no condition occurrences of ‘Myocarditis Pericarditis’, starting in the 365 days prior to cohort entry start date; allow events outside observation period.

B.1.3 Cohort Exit

The cohort end date will be offset from index event’s start date plus 1 day.

B.1.4 Cohort Eras

Entry events will be combined into cohort eras if they are within 0 days of each other.

B.1.5 Concept set: Myocarditis Pericarditis

Concept ID Concept Name Code Vocabulary Excluded Descendants Mapped
4231274 Viral myocarditis 89141000 SNOMED NO YES NO
4289908 Viral pericarditis 70189005 SNOMED NO YES NO
4138837 Pericarditis 3238004 SNOMED NO YES NO
314383 Myocarditis 50920009 SNOMED NO YES NO
4149913 Systemic lupus erythematosus with pericarditis 309762007 SNOMED NO YES NO
318072 Histoplasmosis with pericarditis 187059008 SNOMED NO YES NO
44782774 Chest pain due to pericarditis 34791000119103 SNOMED NO YES NO

C Negative controls

Table C.1: Negative control outcomes.
Outcome Id Outcome Name
438945 Accidental poisoning by benzodiazepine-based tranquilizer
434455 Acquired claw toes
316211 Acquired spondylolisthesis
201612 Alcoholic liver damage
438730 Alkalosis
441258 Anemia in neoplastic disease
432513 Animal bite wound
4171556 Ankle ulcer
4098292 Antiphospholipid syndrome
77650 Aseptic necrosis of bone
4239873 Benign neoplasm of ciliary body
23731 Benign neoplasm of larynx
199764 Benign neoplasm of ovary
195500 Benign neoplasm of uterus
4145627 Biliary calculus
4108471 Burn of digit of hand
75121 Burn of lower leg
4284982 Calculus of bile duct without obstruction
434327 Cannabis abuse
78497 Cellulitis and abscess of toe
4001454 Cervical spine ankylosis
4068241 Chronic instability of knee
195596 Chronic pancreatitis
4206338 Chronic salpingitis
4058397 Claustrophobia
74816 Contusion of toe
73302 Curvature of spine
4151134 Cyst of pancreas
77638 Displacement of intervertebral disc without myelopathy
195864 Diverticulum of bladder
201346 Edema of penis
200461 Endometriosis of uterus
377877 Esotropia
193530 Follicular cyst of ovary
4094822 Foreign body in respiratory tract
443421 Gallbladder and bile duct calculi
4299408 Gouty tophus
135215 Hashimoto thyroiditis
442190 Hemorrhage of colon
43020475 High risk heterosexual behavior
194149 Hirschsprung’s disease
443204 Human ehrlichiosis
4226238 Hyperosmolar coma due to diabetes mellitus
4032787 Hyperosmolarity
197032 Hyperplasia of prostate
140362 Hypoparathyroidism
435371 Hypothermia
138690 Infestation by Pediculus
4152376 Intentional self poisoning
192953 Intestinal adhesions with obstruction
196347 Intestinal parasitism
137977 Jaundice
317510 Leukemia
765053 Lump in right breast
378165 Nystagmus
434085 Obstruction of duodenum
4147016 Open wound of buttock
4129404 Open wound of upper arm
438120 Opioid dependence
75924 Osteodystrophy
432594 Osteomalacia
30365 Panhypopituitarism
4108371 Peripheral gangrene
440367 Plasmacytosis
439233 Poisoning by antidiabetic agent
442149 Poisoning by bee sting
4314086 Poisoning due to sting of ant
4147660 Postural kyphosis
434319 Premature ejaculation
199754 Primary malignant neoplasm of pancreas
4311499 Primary malignant neoplasm of respiratory tract
436635 Primary malignant neoplasm of sigmoid colon
196044 Primary malignant neoplasm of stomach
433716 Primary malignant neoplasm of testis
133424 Primary malignant neoplasm of thyroid gland
194997 Prostatitis
80286 Prosthetic joint loosening
443274 Psychostimulant dependence
314962 Raynaud’s disease
37018294 Residual osteitis
4288241 Salmonella enterica subspecies arizonae infection
45757269 Sclerosing mesenteritis
74722 Secondary localized osteoarthrosis of pelvic region
200348 Secondary malignant neoplasm of large intestine
43020446 Sedative withdrawal
74194 Sprain of spinal ligament
4194207 Tailor’s bunion
193521 Tropical sprue
40482801 Type II diabetes mellitus uncontrolled
74719 Ulcer of foot
196625 Viral hepatitis A without hepatic coma
197494 Viral hepatitis C
4284533 Vitamin D-dependent rickets