The study adhered to the principles set out in the Declaration of Helsinki, with informed consent obtained from all participants. The Columbia University Health Sciences Institutional Review Board (IRB) reviewed and approved the study (IRB AAAR3954). The analysis in this study was conducted on the deidentified data.

We used EHR data obtained from the NYP/CUIMC data warehouse. The NYP/CUIMC is a quaternary care academic medical center that includes an academic hospital, a children’s hospital, and a community-based hospital serving a diverse patient population in northern Manhattan, New York City (NYC). EHR data were collected and stored in the data warehouse during routine clinical care at the CUIMC/NYP. The EHR data were converted to the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) version 5.0 [38]. All data involved in this analysis were collected up to September 21, 2021, which captured the B.1.1.7 (alpha; January 2021-June 2021) and B.1.617.2 (delta; June 2021-December 2021) variant waves but did not include data from the B.1.1.529 (omicron; December 2021-present) wave [39]. Due to the insufficient sample size of individuals vaccinated with non-mRNA vaccines, and the different mechanisms between the mRNA vaccine and adenovector vaccines (such as Johnson & Johnson) [40], we only investigated breakthrough infections in the fully mRNA-vaccinated individuals.

Individuals over the age of 18 years who resided in NYC were included in this study. OMOP concepts related to vaccines were used to identify vaccinated individuals who received 2 doses of Pfizer/BNT162b2 or Moderna/mRNA-1273. To minimize potential bias resulting from missing vaccination records, vaccines records in our data warehouse were obtained from both CUIMC EHR data and the NYC vaccine registry. We required individuals to complete their 2-dose administration with a time interval of 20-23 days for Pfizer/BNT162b2 and 27-31 days for Moderna/mRNA-1273; individuals with 2 doses with 14 days of available follow-up after their second dose were considered fully vaccinated. Individuals who received doses from more than 1 manufacturer or only received 1 vaccine dose were excluded. We defined COVID-19-positive cases by using the OMOP measurement concepts and corresponding value concepts related to detect positive RNA using polymerase chain reaction (PCR). Individuals with at least 1 positive SARS-CoV-2 PCR test were flagged as COVID-19 positive. To balance the cofounding between positive cases and negative cases, we adopted a test-negative design—only individuals with at least 1 negative PCR test were included as COVID-19-negative cases. To reduce the potential false positives in the negative cohort, we additionally established stringent criteria to further exclude individuals with any evidence of a prior SARS-CoV-2 infection: (1) a positive SARS-CoV-2 PCR test, (2) a positive SARS-CoV-2 antibody test, or (3) a concept indicating SARS-CoV-2 infection. The details of OMOP concepts used for the cohort definition are available in Multimedia Appendix 1.

Based on the vaccine and SARS-CoV-2 status, we then constructed 6 cohorts based on evidence of COVID-19 breakthrough infection (ie, positive or negative) and vaccination status (ie Vax, Prevax, and Unvax), as shown in Figure 1. For example, “Vax positive” is a collection of individuals who were vaccinated but later experienced breakthrough infections. The vaccination status was classified into “Vax” (those who were fully vaccinated), “Prevax” (those during a period when vaccines were unavailable), and “Unvax” (those who were not vaccinated during the period when vaccines were available). Individuals who were in the Prevax infection–negative cohort could also be in a Vax cohort later. Of note, if an individual receives a first dose for vaccination, that individual exits the Unvax cohort (and may later become part of a Vax cohort if fully vaccinated). More details about the cohort definitions can be found in the Results section.

For each cohort, we extracted individuals' demographic data, including age, gender, ethnicity, and race. For the vaccinated cohorts, the vaccine brand and corresponding administration dates were also extracted. To approximate the available observation time, we extracted the total number of prior EHR visits and days of observation periods between clinical encounters for each individual. We extracted all previous condition and drug concepts from the condition_era and drug_era tables. To avoid extracting condition/drug concepts potentially caused by the SARS-CoV-2 infection itself, we added a 90-day washout period (ie, ignore all the concepts within the 90-day window prior to the PCR test regardless of its result). To identify individuals who might have compromised immune systems, we compiled a list of conditions and drugs, including active solid tumors and hematologic malignancies (within 2 years), solid-organ or hematopoietic stem cell transplant, primary immunodeficiencies, HIV infection, immunosuppressive therapies (eg, cancer chemotherapeutic agents, certain biologic agents, rituximab), and CKD [41]; see Multimedia Appendices 2 and 3. Individuals could fall into multiple immunocompromised subgroups. To adjust for the caseload in NYC, a 7-day rolling average of cases was applied [42].

We compared the Vax-positive and Vax-negative cohorts to identify potential risk factors associated with breakthrough infections (Multimedia Appendix 4). The entry date was defined as the fully vaccinated date, and individuals were then followed until the first positive PCR date (or the end of the study for Vax-negative individuals). For each risk factor (eg, vaccine brand, demographics, immunocompromised status), a univariate Poisson regression was fit to assess the incidence rate ratio (IRR; ie, breakthrough per 1000 person-days) against the reference status. To minimize potential bias resulting from daily caseload, viral mutations, and EHR data quality, the Poisson regression was adjusted for (1) the total number of observation days in the EHR before the entry date, (2) the total number of visits in the EHR before the entry date, and (3) the calendar month of the PCR test date. We further applied a non-hypothesis-driven approach to uniformly evaluate the risk effect for each historical condition and drug by fitting a univariate Poisson regression with similar adjustment. Condition and drug concepts significantly associated with the breakthrough infections were identified as a <.05 Bonferroni-adjusted P value [43].

For the Vax cohorts, the entry date was defined similarly to the entry date for the risk factor analysis. For the Unvax cohorts, the entry date was defined as January 18, 2021 (14 days after the first individual received their second dose at the CUIMC/NYP), and individuals were then followed up until the first positive PCR test (latest for negative individuals) or the date when they received their first dose, whichever came first (Multimedia Appendix 5). We 1:1-matched vaccinated individuals to unvaccinated individuals using a nearest-neighbor search based on (1) observation days, (2) visit count, (3) calendar week of the PCR test (earliest positive PCR or latest negative PCR), (4) demographics (eg, gender, age, race, ethnicity), and (5) immunocompromised status (binary). The IRRs for the vaccine were estimated via Poisson regressions.

As shown in Multimedia Appendix 6, we further identified 1:1-matched individuals in the Prevax cohort based on the same covariates, except for the calendar week of the PCR test, which was replaced by the 7-day rolling average of cases in NYC at the PCR testing date. Given the difficulty in identifying an appropriate entry date for the prevaccinated cohort, we applied a case-control design to calculate the odds ratio (OR) of contracting COVID-19 infection between the Vax cohort and the Prevax cohort using logistic regressions.

Table 1 provides baseline characteristics of the 6 cohorts (note: some individuals are in multiple cohorts at different times). For the Vax-positive (ie, breakthrough) cohort, the median age was 60 years (IQR 40.7-75.4). Of 198 individuals in the Vax-positive cohort, 156 (78.8%) received Pfizer/BNT162b2, while 42 (21.2%) received Moderna/mRNA-1273. In addition, 65 (45.5%) had underlying immunocompromised conditions, and 120 (60.6%) of the patients with breakthrough infections were hospitalized. In general, PCR-positive individuals had a higher number of prior visits and observational days compared to unvaccinated individuals. For later analyses, we used a matching strategy to balance the covariates between the cohorts.

The overall estimated breakthrough infection rate was 0.16 (95% CI 0.14-0.18). Table 2 summarizes risk factors associated with breakthrough infections. We found a significantly higher incidence rate in vaccinated males than in females (IRR=1.47, 95% CI 1.11-1.94). We did not find any significant change in the incidence rate associated with other demographics, though Black individuals are likely to have a higher incidence rate and Asian individuals are likely to have a lower incidence rate. However, given the large portion of unknown race/ethnicity in the EHRs, our study was unable to estimate this association with meaningful accuracy. There was a significantly higher rate of breakthrough infections among those vaccinated with Pfizer/BNT162b2 compared to Moderna/mRNA-1273 (adjusted IRR=1.66, 95% CI 1.17-2.35). An immunocompromised state was significantly associated with a higher incidence rate among the vaccinated (adjusted IRR=1.48, 95% CI 1.09-2.00). Those with primary immunodeficiency, a history of organ transplant, an active tumor, and use of immunosuppressant medications were at the highest risk.

For the underlying conditions and drug usage analysis, a total of 1359 and 536 unique candidate conditions and drugs were available for investigation, respectively. Concepts needed a minimum of 100 individuals to be considered. Table 3 summarizes the top 10 breakthrough infection-associated condition and drug concepts. In addition to previously known conditions and drugs related to immunocompromised status (eg, immunodeficiency disorder, valganciclovir), we found that prior conditions and drugs related to pulmonary disease (eg, postinflammatory pulmonary fibrosis, albuterol) were also among those significantly associated with an increased breakthrough infection rate. The full list of associated conditions and drug concepts is provided in Multimedia Appendices 2 and 3.

We analyzed the protective effect of vaccination in the Vax cohort using 2 matched Prevax and Unvax cohorts. When comparing the Vax cohort with the Prevax cohort, the risk of COVID-19 infection in vaccinated individuals was significantly lower (adjusted OR 0.12, 95% CI 0.10-0.13), which was also the case when stratifying by age, gender, and immunocompromised status (Table 4). Similarly, we found a significant reduction in the incidence rate (adjusted IRR=0.42, 95% CI 0.36-0.49) when comparing the Vax cohort with the Unvax cohort (Table 5); similar observations were found across age, gender, and immunocompromised status subgroups.

By comparing the breakthrough cohort (ie, Vax positive) against the no-breakthrough cohort (ie, Vax negative), we found a number of medical commodities were associated with an increased risk of breakthrough infection. First, we found that immunosuppressive therapy is associated with higher rates of breakthrough infection. Individuals with active tumors also had higher rates of breakthrough infection, suggesting that the effects of active malignancy or chemotherapy lead to a reduced immune response. There was a statistically not significant increase in individuals with a history of tumors, suggesting that individuals whose cancers are in remission are more similar to the average population in terms of immune response. Our findings are in line with prior studies of solid organ transplant recipients who have shown weaker immune responses in patients who are immunosuppressed and undergo vaccination against COVID-19 [44,45]. For example, valganciclovir is a drug used commonly to prevent cytomegalovirus disease after solid organ transplantation [46], and we found it was significantly associated with the increased risk of breakthrough infection, indicating individuals who underwent solid organ transplant were among those at high risk of breakthrough infections. We also observed an increased risk of infection in individuals with prior lung infection. A potential explanation is the microbiome changes within the lung that play a key role in the initiation and progression of COVID-19 [47,48]. In addition, studies have shown that patients with COVID-19 and preexisting interstitial lung disease (ILD) had a poorer prognosis [49,50], which highlights the importance of staying vigilant and continued use of personal protective and social measures, even with vaccination among those individuals. Furthermore, in individuals with Alzheimer disease who were vaccinated, there was an increased risk of infection, which might be due to their frailty and medical vulnerability, and nonadherence to infection control measures, such as physical distancing [51]. This is also confirmed by the finding that donepezil is a high-risk factor, which is used to treat confusion (dementia) related to Alzheimer disease [52]. We did not find a significantly increased risk of breakthrough infection in individuals with CKD. An ongoing study (the Renal Patients COVID-19 Vaccination Immune Response [RECOVAC-IR] study) aims to provide further guidance regarding the efficacy of vaccines in patients with CKD or whether other measures, such as booster vaccinations, are required [53].

Although our findings reaffirmed the high protection of mRNA vaccines against COVID-19 infection, we found that Moderna/mRNA-1273 had an overall higher effectiveness in preventing SARS-CoV-2 infections. A previous high-quality prospective study [54] involving 3975 individuals observed through April 2021 demonstrated similar vaccine effectiveness between the mRNA vaccines, but it was underpowered and did not perform statistical analysis. A more recent Mayo Clinic study of data collected through July 2021 [55] was consistent with our findings despite differences in cohort definitions and geography. Another recent study comparing the SARS-CoV-2 antibody response following vaccination similarly found higher antibody titers in participants vaccinated with mRNA-1273 compared with those vaccinated with Pfizer/BNT162b258. A more recent meta-analysis study of data collected through September 2021 showed that the estimated long-term vaccine effectiveness for COVID-19 hospitalization was 85.4% (95% CI 84.8%-86.0%) with the Pfizer/BNT162b2 vaccine and 89.8% (95% CI 89.2%-90.4%) with Moderna/mRNA-1273. Additional studies should be considered to provide further guidance on effectiveness differences between vaccine brands and booster shot prioritization. Although individuals with immunosuppressed disorders are at higher risk of developing breakthrough infection, the adjusted IRR in immunocompromised individuals is 0.43 (95% CI 0.34-0.55), supporting the conclusion that vaccination can still greatly reduce the infection rate among this subgroup [56]. Our study supports the current policies recommending that immunocompromised individuals receive booster doses [57].

It is important to provide constant public health surveillance of vaccine protection. By leveraging EHR data from various health systems, we can provide more robust and generalizable evidence of vaccine effectiveness. Unfortunately, it is not always easy to aggregate the medical data from multiple institutions due to Health Insurance Portability and Accountability Act of 1996 (HIPPA) constrains. Therefore, we developed an entirely CDM-based analysis pipeline, making it easily transferable to dozens of other health care databases compatible with it [58]. We have provided our OMOP-compatible analysis pipeline on the GitHub repository [59]. Other institutions that have implemented their OMOP instance can download the code and easily replicate the analysis using their own institution’s OMOP instance and share the evidence in a timely manner.

Given the high level of missingness typically found in EHR data, it is challenging to estimate the absolute incidence rate of breakthrough infections. In our study, the incidence rate among the vaccinated cohort was estimated to be 0.16 per 1000 person-days. This potentially overestimates the incidence rate (particularly in comparison to 0.031 in Israel’s national surveillance data [60], ~0.01 in the original Pfizer/BNT162b2 and Moderna/mRNA-1273 trials [1,2]) because we imposed a criterion to only include those who have at least 1 PCR test available, which is also called test-negative design [61]. If we remove this requirement, the incidence rate among the vaccinated cohort becomes ~0.007 per 1000 person-days. However, this is an underestimation of the true rate because some of the SARS-CoV-2-infected individuals might have been tested elsewhere or not at all. In addition, despite adopting a test-negative design, we were still unable to confirm whether negative cases were truly negative (eg, tested positive elsewhere). Similarly, some patients may be incorrectly labeled as unvaccinated if their vaccinations took place outside of NYC or the NYP health system, which could lower effectiveness estimations. However, our main focus in this study was to identify the risk subgroups at increased risk of breakthrough infections, and a cofounding-aligned comparative design can achieve this goal by matching the patient’s demographics and their tendency in seeking health care in our medical center.

Another limitation of this study is we could not stratify breakthrough infections by variant type due to limitations in testing data. Our study used the EHR data collected through September 2021, which covers periods where the B.1.1.7 (alpha) and B.1.617.2 (delta) variants were prevalent. Our findings cannot be generalized to other newly emerged variants of concerns, including B.1.1.529 (omicron) for which the existing mRNA vaccines may have differing effectiveness. However, this is not unique to our study, as the pandemic has often evolved faster than high-quality analyses can be performed. Even a recent systematic review of the efficacy and effectiveness of the COVID-19 vaccines published in January 2022 included only papers before April 2021 (the data collected in those papers can be from even earlier). The conflict between the speed of scientific publication and the rapid evolution of the pandemic remains a significant challenge for the overall research community.

Finally, the CUIMC/NYP is an academic medical center in NYC, which might not represent the general American population or other potential patient groups of interest. In particular, the overall population in our study is sicker than the general population, as evidenced by the high rate of comorbidities and older age of our patient cohort.

We performed a retrospective analysis to investigate risk factors contributing to COVID-19 breakthrough infections among vaccinated individuals. We found those who are male, immunocompromised, or have preexisting pulmonary disease are at a higher risk of COVID-19 breakthrough infection. Although both vaccines are highly effective in preventing SARS-CoV-2 infection, Moderna/mRNA-1273 is associated with a lower risk of breakthrough infection than Pfizer/BNT162b2. Multiple medical institutions’ data are warranted to better link the PCR test results and vaccination information. Those with an OMOP instance of their data can reapply our analysis to check the robustness of our results [59].