This study used a nationwide database combining the COVID-19 registry from the Korea Disease Control and Prevention Agency (KDCA) and the health insurance claims data from the National Health Insurance Service (NHIS). Korea has a universal health insurance system that provides coverage for the entire population. Claims data are generated when health care providers deliver services to patients and subsequently file claims with the NHIS. The NHIS compiles a comprehensive database that includes information on insurance eligibility, premiums, medical services, and health care institutions [31]. This database encompasses records of inpatient, outpatient, and emergency department visits. Since the launch of the national COVID-19 vaccination program on February 26, 2021, the KDCA has managed vaccination records for the entire population [32]. The linked database includes information on COVID-19 vaccinations, laboratory-confirmed SARS-CoV-2 infections, diagnoses, prescribed medications (eg, drug codes, daily dosage, and duration), and health care procedures. It also contains timestamps for data recording, enabling us to track the dates when the data were accrued.

For this study, we used the KDCA COVID-19 registry data from February 26, 2021, to June 30, 2023, and NHIS health insurance claims data from January 1, 2017, to June 30, 2023. The study population consisted of individuals aged 12 years and older who received either monovalent or bivalent COVID-19 vaccines in Korea. Only individuals with Korean nationality who were enrolled in the national health insurance during the study period were included. Individuals participating in clinical trials for COVID-19 vaccines, those vaccinated abroad, or those with erroneous or duplicate vaccination records were excluded.

The exposures of interest included all COVID-19 monovalent vaccines used for primary doses (ie, first and second doses) and booster doses, as well as bivalent COVID-19 vaccines administered in Korea. Korea introduced 6 types of COVID-19 monovalent vaccines, including those based on mRNA, nonreplicating viral vector, and recombinant protein platforms (Table S1 in Multimedia Appendix 1). Our analysis focused on all primary and booster doses of monovalent COVID-19 vaccines administered from February 26, 2021, to September 30, 2022. During the 2022-2023 winter season, 4 types of mRNA bivalent vaccines were introduced in Korea (Table S2 in Multimedia Appendix 1). As part of the COVID-19 winter booster vaccination program, 1 dose of a bivalent vaccine was recommended regardless of the specific vaccine type. This study included all bivalent COVID-19 vaccines administered from October 11, 2022, to March 31, 2023.

To account for vaccine platforms and recommended dose schedules, we categorized the exposure groups based on the type of vaccines and dose number: mRNA monovalent vaccine (doses 1, 2, and 3), non-replicating viral vector monovalent vaccine (doses 1 and 2), recombinant protein–based monovalent vaccine (doses 1, 2, and 3), and mRNA bivalent vaccine doses. All analyses were stratified by platform and dose number.

This study selected outcomes of interest by considering both the associations between COVID-19 vaccines and AEs, and the timing of regulatory actions. The selection process involved reviewing the priority lists of adverse events of special interest (AESIs) for COVID-19 vaccines published by the Safety Platform for Emergency Vaccines [33], the VSD [17], and the BEST Initiative [34]. We also considered a list of AEs deemed causally or suspiciously related to COVID-19 vaccines by the KDCA (Table S3 in Multimedia Appendix 1). Through this process, 3 conditions were chosen as the outcomes of interest based on different causal association scenarios: anaphylaxis, myocarditis, and acute myocardial infarction (AMI). Each of these conditions was commonly listed as AESIs by the Safety Platform for Emergency Vaccines, VSD, and BEST Initiative. Anaphylaxis was considered an AESI that was causally associated with COVID-19 vaccines by the KDCA even before vaccine introduction, given that it can potentially be induced by all vaccine types [35]. Subsequent literature further supported this association [36-38]. Myocarditis was included in the AESI lists and, following the vaccine rollout, causal associations were recognized for mRNA and NVX-CoV2373 vaccines [39]. AMI, initially listed as an AESI, has not shown a definitive causal relationship with vaccination in postvaccination epidemiological studies.

In addition, a negative control event was selected—an event for which there is no evidence of a causal association with vaccination and that has not been flagged as a signal [40]. Colonic diverticulitis was chosen as the negative control event, as it was not included in COVID-19 AESI lists and is generally considered unrelated to vaccination. Moreover, this acute condition typically requires immediate treatment, making it unlikely to show significant variations in health care use between the prepandemic and pandemic periods [41].

Table S4 in Multimedia Appendix 1 presents the operational definitions for the selected AESIs and the negative control event. To maintain sensitivity and enable timely analysis for NRT sequential monitoring, operational definitions were based solely on diagnostic codes. The diagnostic codes and the lengths of risk periods were determined by referencing definitions employed by VSD [17] and BEST [34], as well as expert opinions from clinicians in the relevant medical fields.

This study conducted monthly sequential testing to compare the background incidence rates of outcomes of interest with the incidence rates observed during the risk period following COVID-19 vaccination. The study design consisted of three steps: (1) estimation of background incidence rates for the outcomes of interest, (2) analysis of their incidence rates during the postvaccination risk periods, and (3) application of the sequential testing method. Sensitivity analyses were performed when a signal met the statistical threshold.

Sequential testing, which adjusts for multiple comparisons through repeated analyses over time, can rapidly identify potential safety concerns [1,42]. It generates a statistical signal as soon as there is sufficient evidence to reject the null hypothesis of no excess risk, thereby providing timely safety information that can enhance public confidence in vaccines [42]. Sequential testing methods are categorized into continuous sequential methods, which analyze data frequently (eg, weekly or monthly), and group sequential methods, which analyze data at discrete intervals after a specific amount of data has been accumulated [43]. This study adopted a continuous sequential method, known for its superior performance in using data aggregated over short intervals [44]. Among the continuous sequential methods, the maximized sequential probability ratio test (MaxSPRT), developed by VSD for sequential monitoring, is particularly suited for ongoing surveillance as it continues monitoring until the statistical threshold is reached or the maximum sample size is attained [44]. This study used Poisson MaxSPRT (PMaxSPRT) to compare the observed number of cases of AEs with the expected number derived from historical controls, especially for rare events during a large-scale vaccination campaign.

This study was approved by the institutional review board of Ewha Womans University (ewha-202404-0014-01). Informed consent was not required due to the utilization of anonymized patient data.

Table 1 summarizes the sociodemographic and vaccination characteristics of the Korean population who received COVID-19 vaccines during the study period, stratified by vaccine platform and dose number. Over 43 million individuals received monovalent vaccines, with mRNA vaccines being the most frequently administered across all dose levels. Among individuals aged 65 years and older, ChAdOx1 was the vaccine most commonly administered. Since NVV was recommended only for those aged 18 years or older, most individuals aged 12 to 17 years received other vaccine platforms. Recombinant protein–based vaccines, introduced later than other platforms, had fewer recipients. Nonreplicating viral vector vaccines for booster doses were excluded from sequential monitoring as they were not recommended for booster doses.

Approximately 6.3 million individuals received bivalent vaccines, with the majority being aged 65 years or older, reflecting the prioritization of older adults, immunocompromised individuals, and high-risk groups. BNT162b2 BA.4/5 was the most frequently administered bivalent vaccine. Figures S5 and S6 in Multimedia Appendix 1 illustrate the weekly distribution of vaccinations by vaccine type and dose number for both monovalent and bivalent vaccines.

Table 2 presents the annual background rates of AESI derived from data for 2018, 2019, and 2020. For most AESI across all age groups, the background rates in 2020 were either statistically significantly lower or similar to those in the pre-COVID-19 period. This study used the lower incidence rate between 2018 and 2019 as the background rate for each AESI and age group.

Figures S6-S9 in Multimedia Appendix 1 present the distribution of delays in data accrual for claims made before the introduction of COVID-19 vaccination. For all AESIs, comparisons revealed that diagnoses from inpatient settings had faster data accrual than those from combined inpatient and outpatient settings (Figures S7-S10 in Multimedia Appendix 1). Consequently, the study used data delay distributions based on inpatient diagnostic claims. During the COVID-19 pandemic, over 95% of claims data were accrued faster compared to the prepandemic period. Thus, this study used data accrual delay distributions from inpatient diagnoses recorded between February 2020 and June 2020 to calculate the contribution of the risk period to sequential testing (Table S6 in Multimedia Appendix 1).

The person-time at risk and the number of events at each sequential testing point were calculated, and the expected number of events within the risk period was estimated using background rates. These calculations were performed separately for each age group, vaccine platform, and dose at each testing.

For AMI and colonic diverticulitis, none of the analyses stratified by age group, vaccine platform, or dose met the statistical signal threshold (Tables S7 and S8 in Multimedia Appendix 1).

For myocarditis, the statistical signal threshold was met for mRNA vaccines in the 12‐17 and 18‐39 age groups across all doses and for the first dose in the 40‐64 age group. The threshold was also met for recombinant protein vaccines in the 40‐64 age group for both the first and second doses. Significant results from sequential testing for myocarditis are summarized in Table 3, while the complete results for all age and dose groups are provided in Table S9 in Multimedia Appendix 1.

For anaphylaxis, the statistical signal threshold was exceeded for mRNA vaccines in the 18‐39 age group across all doses and for the first and second doses in individuals aged 40 years and older. For nonreplicating viral vector vaccines, the threshold was met following the first and second doses in the 18‐39 and 40‐64 age groups, as well as after the first dose in individuals aged 65 years and older. A summary of significant sequential testing results for anaphylaxis is presented in Table 4, and detailed findings across all age and dose groups are available in Table S10 in Multimedia Appendix 1.

Sensitivity analyses were conducted for myocarditis and anaphylaxis to account for temporal trends in monthly incidence rates. For myocarditis, no differences were found between the sensitivity and primary analyses (Table S11 in Multimedia Appendix 1). For anaphylaxis, however, the sensitivity analysis showed that the signal threshold was no longer exceeded for the second dose of non-replicating viral vector vaccines in the 40‐64 age group or the first dose in individuals aged 65 years and older (Table S12 in Multimedia Appendix 1).

The timing of statistical signal detection using the sequential monitoring method was compared to the timing of safety measures implemented by the KDCA (Figure S11 in Multimedia Appendix 1).

For myocarditis, the KDCA initially did not recognize an association between COVID-19 vaccines and myocarditis during the early vaccination phase. However, international reports of myocarditis following mRNA COVID-19 vaccination and reviews by global regulatory agencies prompted further investigation [45,46]. Based on domestic causality assessments, the KDCA announced on March 14, 2022, that myocarditis was causally associated with mRNA COVID-19 vaccination. Subsequently, on December 29, 2022, myocarditis was identified as a potential AE related to recombinant protein–based vaccines [39].

In this study, sequential testing detected statistical signals for myocarditis in different age groups after receiving mRNA vaccines. For the 12‐17 and 18‐39 age groups, the signals were detected in January 2022 and September 2021, respectively. In the 40‐64 age group, signals were detected for mRNA vaccines in December 2021 and for recombinant protein–based vaccines in June 2022. The earliest signal was observed on September 30, 2021, for mRNA vaccines in the 18‐39 age group.

For anaphylaxis, the KDCA recognized a causal relationship with COVID-19 vaccination from the beginning of the vaccination program. Sequential testing detected signals for nonreplicating viral vector vaccines in the 18‐39 age group in April 2021 and for mRNA vaccines in June 2021. In the 40‐64 age group, signals for nonreplicating viral vector vaccines were detected in April 2021, while mRNA vaccines reached the threshold in June 2021, after approximately 80,000 doses. Among individuals aged 65 years and older, signals for both nonreplicating viral vector and mRNA vaccines were detected in July 2021. The earliest signal was observed on April 30, 2021, for the first dose of nonreplicating viral vector vaccines in the 18‐39 and 40‐64 age groups.

This study conducted an exploratory analysis to evaluate the applicability of NRT surveillance for recently developed COVID-19 vaccines using a sequential testing method. Monthly analyses were performed using PMaxSPRT to compare background rates with postvaccination incidence rates, accounting for data accrual delays. All COVID-19 vaccines introduced in Korea were analyzed by vaccine platform, age group, and dose for 4 outcomes of interest.

The results showed that the statistical signal threshold for myocarditis was met in individuals younger than 64 years after mRNA vaccination and in those aged 40 to 64 years after recombinant protein–based vaccines. For anaphylaxis, the statistical signal threshold was met following mRNA and nonreplicating viral vector vaccines in individuals aged 18 years and older. No statistical signals were detected for AMI or colonic diverticulitis by the end of surveillance.

Comparison with prior studies revealed consistent findings for myocarditis and anaphylaxis. Lloyd et al [22] identified statistical signals for myocarditis, pericarditis, and anaphylaxis following COVID-19 vaccination in the US population aged 12‐64 years. Similarly, studies analyzing BNT162b2 in adolescents aged 12 years and older also detected signals for myocarditis and pericarditis [23,24]. Wong et al [47], studying US adults aged 65 years and older, reported signals for AMI and other outcomes, but sensitivity analyses accounting for time trends and pandemic background rates showed no signals. These findings align with our study, which detected statistical signals for myocarditis and anaphylaxis but not for AMI.

This study compared the timing of statistical signals with the regulatory actions taken by the KDCA. In Korea, the KDCA delegated COVID-19 vaccine safety monitoring to CoVaSC, which made causality assessments prior to regulatory decisions [32]. Anaphylaxis was recognized as causally associated with COVID-19 vaccination early in the campaign. Sequential testing revealed statistical signals for nonreplicating viral vector vaccines in April 2021 and for mRNA vaccines in June 2021. Subsequent epidemiological studies confirmed the association between COVID-19 vaccines and incident anaphylaxis risk [36,39]. For myocarditis, causality was recognized for mRNA vaccines in March 2022 and for recombinant protein–based vaccines in December 2022 [39]. Sequential testing showed that statistical signals were first detected in September 2021 for mRNA vaccines and in June 2022 for recombinant protein–based vaccines. These findings suggest that NRT sequential testing could have facilitated earlier epidemiological evaluations of high-priority conditions using domestic data.

Sequential monitoring conducted early during the COVID-19 vaccination campaign in countries such as the United States demonstrated the rapid detection of safety signals when applying NRT methods. For instance, sequential testing in the US FDA surveillance program identified statistical signals for AMI, pulmonary embolism, disseminated intravascular coagulation, immune thrombocytopenia, myocarditis, and pericarditis after BNT162b2 vaccination. It also detected a signal for anaphylaxis after mRNA-1273 vaccination [22,47]. In a US pediatric population aged 5‐17 years, sequential testing detected signals for myocarditis and pericarditis [24]. Initial VSD analyses in the United States did not detect signals for AESIs [17], but extended studies later identified signals for Guillain-Barré syndrome with Ad26.COV2.S and myocarditis or pericarditis in younger adults after mRNA vaccination [48,49].

Countries with established NRT sequential monitoring systems identified safety concerns relatively early in their vaccination campaigns. In contrast, Korea’s vaccination campaign began later without a sequential monitoring system in place. As a result, regulatory actions largely relied on international findings, such as safety concerns about myocarditis raised abroad before vaccines were administered to Korean adolescents and young adults. This reliance may have influenced diagnosis coding practices. Cases resembling myocarditis might have been coded as myocarditis, which could have impacted studies based on claims data. Our study, which used a health insurance claims database for sequential analysis, may also have been influenced by these coding practices in analyzing myocarditis, one of our outcomes of interest. Such diagnostic awareness and heightened attention following international safety reports may have increased the likelihood of coding myocarditis, potentially inflating the observed signal strength. If an NRT monitoring system had been established before these issues were raised internationally, or if sequential analysis had been conducted globally at the onset of vaccinations, the time required to detect a myocarditis signal in our study might have been longer. In that case, a lower relative risk might have been estimated. However, had sequential monitoring been implemented domestically, safety signals might have been detected earlier, as demonstrated by this study.

To implement NRT monitoring effectively within the country, further exploration of data sources capable of supporting NRT monitoring is needed. In this study, we used health insurance claims data and addressed data latency by calculating delay distributions based on recent historical data. One advantage of using domestic health insurance claims data is that they offer medical information for the entire population, allowing for the surveillance of even rare AEs. However, given the nature of claims data, inherent delays may occur during several stages of data processing. These include the provision of medical services, submission of claims by medical institutions, review procedures, and subsequent reimbursement by the NHIS. This process can take approximately 3 months or, in some cases, more than 6 months before the data becomes available. Therefore, to enhance NRT monitoring in the future, collaboration with sample hospitals might be considered to collect data based on EHRs, instead of claims data, which may accelerate the data accrual process. Integrating multiple data sources, such as NRT hospital data and EHRs, would allow timely validation of diagnostic codes and help mitigate potential diagnostic biases and misclassification. Furthermore, to prevent spurious signal detection driven by heightened diagnostic awareness, such as increased hospital visits and changes in coding practices observed for myocarditis, future surveillance systems should incorporate clinical validation processes. These should include case verification using detailed medical record review or laboratory findings. Linking diagnostic data with laboratory results or chart reviews could help distinguish genuine cases from those arising due to increased diagnostic activity.

To our knowledge, this is the first study in Korea to apply NRT sequential monitoring based on a nationwide vaccination registry and health care data. We used a data accrual delay adjustment method tailored to claims data. This approach enabled early safety surveillance shortly after vaccine rollout. However, this study has several limitations. First, while analyses were stratified by age group, vaccine platform, and dose—key factors influencing safety and timing of COVID-19 vaccination—the study did not account for sex differences. This was because vaccine recommendations and rollout timing differed across age groups. These factors were expected to have a greater impact on safety signal detection during the signal detection phase. Considering potential biological and behavioral differences between sexes, future studies should incorporate subgroup analyses by sex to identify potential variations. Second, as this study was designed as a feasibility assessment, the scope was intentionally limited to 3 AESIs and 1 negative control outcome. The selected outcomes were mostly diagnosed in Korea after being reported as AEs in countries that began vaccination earlier. Such international reports may have influenced health care–seeking behavior, potentially leading to overdiagnosis of conditions with similar clinical presentations. For instance, myocarditis cases in Korea were diagnosed predominantly after international reports, where heightened awareness may have contributed to an increase in the diagnosis of related conditions. Future research should expand sequential monitoring to a broader range of AESIs, such as Guillain-Barré syndrome or thrombosis with thrombocytopenia, to evaluate its applicability across diverse clinical presentations. Third, the operational definitions of AESIs relied solely on diagnostic codes for timely signal detection, which may limit diagnostic validity. For AESIs that met the statistical signal threshold in sequential testing, follow-up epidemiological studies with refined operational definitions are necessary to improve diagnostic accuracy. Fourth, the average data accrual rate, calculated from historical data, may not fully reflect actual delays or unexpected changes during the study period. However, this study minimized potential inaccuracies by using recent historical data. Further efforts should be made to continuously update and incorporate monthly accrual rates from the most recent data to enhance accuracy. Finally, although the sequential analysis controlled for repeated testing within each analysis stratum, no adjustment was made for multiple testing across all strata. As a result, the possibility of an inflated type I error rate at the overall study level cannot be entirely ruled out, and the findings should be interpreted as hypothesis-generating rather than confirmatory.

In conclusion, this study demonstrated that sequential monitoring can be effectively applied using Korean data sources in situations that require active surveillance, such as monitoring newly developed vaccines. Comparisons between signal detection timelines and KDCA regulatory actions indicated that NRT sequential monitoring could have enabled earlier recognition of safety concerns, such as anaphylaxis and myocarditis, thereby supporting more timely regulatory decisions. Future studies should explore alternative data sources, such as EHRs, to incorporate a broader range of clinical information and improve monitoring timeliness.