We systematically searched 3 databases, PubMed, Embase, and Web of Science, from January 1, 2018, to September 7, 2022, to identify all the cohort studies and case-control studies of AVSS. The search strategy was registered on the PROSPERO (International Prospective Register of Systematic Reviews) database (CRD42023449920). The details are shown in Supplementary Materials (Multimedia Appendix 1).

There were two stages in the literature screening. In the first step, we selected studies with descriptions of AVSS or systems and excluded: (1) basic experiments (in vitro and animal) and basic research (pathology); (2) vaccine management and economics studies, such as policy, health economics, and sociomedical assessment, quality of life; (3) case reports, case series studies, nursing experiences, which primarily described the characteristics of individual patients or small groups; (4) vaccine effectiveness evaluation; (5) studies not in English; (6) studies belonging to other categories; and (7) uncertain studies, requiring full-text reading for clarification in the next step.

During the secondary step, we included only postmarketing association studies, such as cohort studies and case-control studies that were conducted with control groups from the same source population, regardless of whether databases were used. The following publications were excluded: (1) passive surveillance, (2) clinical trials, (3) theoretical or methodological studies for vaccine safety surveillance, (4) studies not classified as association studies, including correlational studies, signaling reports, descriptive studies, or those lacking a comparison group, (5) studies not related to vaccine intrinsic safety, (6) nonoriginal studies, such as reviews, conference abstracts, correspondence, errata, or protocols, (7) studies not in English, or (8) studies without full text. Table 1 presents our inclusion and exclusion criteria.

For the above steps, 2 trained researchers independently reviewed the title and abstract of each paper, if any disagreement occurred, a senior researcher would participate in the discussion and reach a final agreement.

To formulate a standard, reasonable, and validated MDS for AVSS, 3 stages were undertaken to ensure its scientific and practical validity. First, a preliminary framework for the MDS was developed by referencing relevant variables from the Council for International Organizations of Medical Sciences (CIOMS) Guide to Active Vaccine Safety Surveillance and the COVID-19 Vaccines: Safety Surveillance Manual [25,26]. This initial framework comprised four dimensions: “Vaccine,” “Outcome,” “Demographic Data,” and “Covariate.” In addition, metadata including data sources, encoding, quality control, and assurance were also taken into consideration. Second, to improve the rationality and feasibility of the drafted framework, experts from diverse fields including immunization, epidemiology, clinical medicine, and databases were consulted to review the framework. A total of 30 eligible studies were randomly selected to refine the framework. Third, a standardized questionnaire for information extraction was developed based on the validated framework. Relevant variables were extracted from the methodology, results, and limitations sections of each study. During this process, any overlooked elements were identified and supplemented through discussion to finalize the framework. Furthermore, basic characteristics of the included studies (authors, publication year, country or region, study design, and sample size) were also collected.

The data extraction and management were completed parallelly by 2 trained researchers using Microsoft Excel 2019, with oversight from a senior researcher responsible for quality control. Subsequently, descriptive statistical analysis was performed using R 4.3.2 (R Core Team), and data visualization was carried out using both R 4.3.2 and OriginPro (Origin Laboratories) 2021.

In the study, the key steps from study design, literature searching, framework formulation, information extraction, and analysis to interpretation have been supervised by external experts to strictly ensure high quality. Meanwhile, as for literature selection and data extraction, every participant was trained and qualified, and the information was double entry independently and parallelly, and any disagreement was ruled out by another senior researcher.

This study is the first to systematically clarify the minimum data requirements for AVSS by conducting a comprehensive review aligned with WHO and CIOMS guidelines, focusing on minimizing data range. The proposed MDS and Metadata framework for AVSS encompasses four key dimensions: “Vaccine,” “Outcome,” “Demographic Data,” and “Covariate.” Across these dimensions, 68 variables were identified: 13 in “Vaccine,” 12 in “Outcome,” 11 in “Demographic Data,” and 32 in “Covariate.” Certain variables were deemed essential for all vaccine safety association studies, including unique and anonymized individual identifiers; vaccine name and vaccination date from the “Vaccine” category; diagnosis and diagnostic date from the “Outcome” category; age and sex from the “Demographic Data” category; and history and drugs from the “Covariate” category. These aspects of data collection, quality control, and assurance must be transparently declared in AVSS protocols; however, regrettably, only a minority of studies addressed them adequately. Therefore, we strongly advocate for the establishment of a globally standardized and widely recognized MDS and Metadata framework. Such a standard would enable countries and regions, particularly LMICs, to rapidly develop AVSS systems or conduct related research. In addition to facilitating data comparability, this effort would significantly strengthen the global vaccine safety ecosystem.

The study design was carefully developed to ensure the reliability and representativeness of the findings. First, the study population covered the whole life course, and the number of studies from various groups, including the general population (43/125, 34.4%), maternal and infants (31/125, 24.8%), children (25/125, 20%), and special patients (20/125, 16%) was relatively balanced. Second, the investigated vaccines spanned a wide spectrum, comprising newly emergency-marketed COVID-19 vaccines, annually administered influenza vaccines, globally disseminated human papillomavirus vaccines, diphtheria, tetanus, acellular pertussis vaccines that often raise concerns, and more than 15 other vaccines in total. Finally, this review spanned over 23 countries and regions, with a notable proportion of 13% (n=16) of included studies from LMICs.

The 49 studies examining the safety of COVID-19 vaccines encompassed diverse populations, including the general population, infants, children and adolescents, pregnant women, and special patients. Despite this breadth, only 54 variables were used, likely reflecting the challenges of limited data availability during the rollout of an emergency vaccine. Previous studies have underscored several challenges in evaluating COVID-19 vaccine safety, including insufficient sample sizes and lower study quality [148-150]. Furthermore, misleading evidence has, at times, hindered efforts to increase vaccine coverage [151]. It is imperative for us to establish a stable AVSS system to provide reliable and sufficient data for the safety studies of emergency vaccines.

The maternal population, which includes 96 variables, reflects a broad spectrum of data, likely due to the inclusion of infant-related variables at this level. Pregnant women are frequently excluded from randomized controlled trials of drug safety because of their unique physiological characteristics [152]. Therefore, the real-world evidence from postmarketing surveillance plays a critical role in evaluating vaccine safety for this population. Furthermore, establishing continuous active surveillance systems facilitates the investigation of the long-term effects of prenatal exposure on offspring.

The standard MDS could provide precise guidance and concise extents of data requirements for the AVSS system. However, no relevant guidelines have yet been released. Although both the CIOMS and WHO guidelines briefly outline the data elements of AVSS, their contents lack consistency and empirical validation in real-world applications [153]. This study is the first to validate these concepts and develop a comprehensive framework. When compared with the common data models of established systems like the VSD and Sentinel, the MDS shares several core strengths. Like these systems, the MDS encompasses the entire lifecycle population, covers a wide array of vaccines, and accounts for a broad spectrum of events. Furthermore, the MDS in our study demonstrated the other two advantages. First, it focuses exclusively on vaccine safety surveillance, minimizing unnecessary data collection efforts and addressing the resource constraints often faced in such endeavors. Second, the MDS incorporates a wider range of data sources, including registration and survey data, which better reflect the diversity of real-world settings and enhance the representativeness of AVSS systems.

A well-defined MDS is crucial for ensuring data quality, comparability, and efficient monitoring in public health surveillance. Within an AVSS system, an effective MDS helps identify key indicators, enhances data accessibility, and strengthens risk assessment [22]. By standardizing data collection, it reduces fragmentation, ensures dataset consistency, and enables timely, automated signal detection, allowing policymakers to respond swiftly to emerging safety concerns. In resource-constrained regions, prioritizing key variables such as vaccination date, vaccine name, outcome diagnosis, and diagnosis date, among the most frequently reported in our study, ensures feasibility while maintaining surveillance effectiveness. From a public health perspective, our study aims to establish a robust AVSS that balances data accessibility and efficiency. By integrating elements from established frameworks such as CIOMS and WHO guidelines, we validate their applicability through a comprehensive literature review. Our findings offer a foundational reference for resource-limited regions, with future efforts focusing on refining MDS definitions through expert consultation and real-world application.

Metadata, which reflects data quality and applicability, is essential for ensuring that real-world data can generate real-world evidence. Unfortunately, only 14.6% of the 103 studies on databases provided Metadata information, indicating that most studies overlooked the importance of data quality control and assurance. This raises concerns about the overall quality of evidence in AVSS from previous studies. It’s imperative to increase awareness and provide clear guidelines in the future. Certainly, the MDS and Metadata are essential components of the information infrastructure for scientific research on vaccine safety vigilance. Prioritizing their importance ensures the establishment of high-quality evidence, enhances evidence transparency, and facilitates evidence translation, particularly in LMICs [154].

There were several limitations in our study. First, our review focused exclusively on association studies in vaccine safety from the past 5 years, which may only represent the latest progress in this field but not its full picture. Second, the variables mentioned in previous studies may not fully correspond to those actually used, potentially introducing some bias in the frequency description. Third, non-English studies were excluded from our analysis. However, given that AVSS studies mostly originate from high-income countries, the impact of this publication bias is likely minimal and can be disregarded. Fourth, our analysis was limited to comparative observational studies (eg, cohort studies and case-control studies) with explicit comparison groups, excluding self-controlled designs (eg, self-controlled case series, and case-crossover studies). While this exclusion represents a limitation, Self-Controlled Case Series (SCCS) studies typically require a smaller set of predefined variables, and the key variables used in SCCS designs are already encompassed within cohort and case-control studies. Therefore, although excluding SCCS studies may slightly reduce the frequency counts of certain variables, it does not affect the overall data scope of the identified MDS. Fifth, this study included variables with a frequency of ≥5% as part of the MDS. While variables with <5% frequency, although less commonly reported, may still hold significant importance, such as “place of vaccination” for detecting clustering of adverse events. For transparency and future reference, a complete list of variables identified in this study is provided in Table S5 in Multimedia Appendix 1.

In conclusion, following WHO and CIOMS guidance and adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Checklist 1), this study presents an initial framework of MDS and Metadata for AVSS. This MDS can provide precise guidance and concise requirements for the data scope of the AVSS system and studies. This framework is particularly valuable for resource-limited regions, providing a foundation for implementing AVSS systems effectively and efficiently. Therefore, it is crucial to establish a standardized MDS globally based on these findings to enhance the global vaccine safety ecosystem.