This systematic review and meta-analysis was registered with PROSPERO (International Prospective Register of Systematic Reviews; ID CRD42021276492) and conducted by adhering to the Cochrane Handbook for Systematic Reviews of Interventions and the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The MEDLINE/PubMed, EMBASE, CINAHL, and SCOPUS databases were searched from their inception until December 2023 by 2 researchers (authors DTAP and PJ). Additional relevant papers were also obtained through a manual search by checking each included study’s references. The gray literature, such as abstracts, conference proceedings, and editorials, was excluded. The keywords and their synonyms for the search strategy were as follows: (“pharmacovigilance” OR “post-marketing surveillance” OR “adverse event reporting” OR “adverse event reporting system” OR “adverse event reports” OR “self-reporting”) AND (“herbal medicine” OR “herbal remedies” OR “medicinal herb” OR “herbal drugs” OR “herbal products” OR “botanical medicines” OR “phytomedicine” OR “phytotherapy” OR “traditional medicine”). In this study, the term “herbal medicines” was defined as “herbs, herbal materials, herbal preparations and finished herbal products. Herbs include crude plant material (leaves, flowers, fruit, seeds, stems, wood, bark, roots, rhizomes or other plant parts, which may be entire, fragmented or powdered). Herbal materials include, alongside herbs, fresh juices, gums, fixed oils, essential oils, resins and dry powders of herbs. Herbal preparations include comminuted or powdered herbal materials, or extracts, tinctures and fatty oils of herbal materials. Finished herbal products consist of herbal preparations made from one or more herbs” [2].

Primary studies were included in this research if they were original studies reporting AEs associated with HMs through a voluntary reporting scheme, with no restriction of language. Studies were excluded if they (1) were reviews or systematic reviews, case reports, conference abstracts, or editorials; (2) were studies reporting AEs on animals; (3) did not provide AEs in detail; or (4) were studies reporting AEs associated with complementary and alternative medicines.

The primary outcome was the reporting rate of the AEs of HMs, calculated as the number of reports referred to AEs associated with HMs (numerator) divided by the total number of reports of AEs associated with all medicines (denominator).

Studies that could not be included in the meta-analysis because of incomplete data were reviewed narratively. Endnote 20 (Clarivate Analytics) was used as a reference manager to import citations and remove duplicate publications.

Titles and abstracts were screened independently by 2 researchers (DTAP and PJ). The full text was then examined in detail by the same 2 researchers. Non-English papers were translated using Google Translate [17].

The following data were independently double-extracted into evidence tables by 2 researchers (authors CK and DTAP): general characteristics of the studies (eg, country of study, source of databases, study period) and outcome data (eg, reporting rate of AEs of HMs, severity of AEs, causality assessment, affected body parts or systems most involved in AEs).

Two independent investigators (DTAP and PJ) assessed the risk of bias (ROB) using Crombie’s checklist for cross-sectional studies [18,19]. The following items were appraised: (1) appropriate design, (2) adequate description of data, (3) reported response rates, (4) adequate representation of the total sample, (5) clearly stated aims and the likelihood of reliable and valid measurements, (6) statistical significance, and (7) adequate description of analyses. Each item scored 1 point for yes, 0.5 points for unclear, and 0 for no. The total score was modified as high ROB (0-4), moderate ROB (>4 and <6), and low ROB (6-7). Any discrepancies during data extraction and ROB assessment were resolved by consensus between these investigators and the principal investigator (CK). The κ statistic was used to evaluate the degree of agreement between investigators. A negative κ indicated a lack of agreement, while the following ranges were used to interpret the level of agreement: 0.01-0.20 as none to slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0.81-1.00 as almost perfect agreement [20].

Pooled effect estimates for the reporting rate of the AEs of HMs across the included studies with corresponding 95% CIs were calculated as a percentage using the DerSimonian-Laird random effects model [21]. To assess the heterogeneity among studies, standard χ² tests and the I² statistic were used [22]. Where high heterogeneity was indicated (I²≥75%), the results across studies were summarized using the median reporting rate and the IQR. To explore the possible sources of heterogeneity, subgroup analyses were performed by continent (North America, Europe, Asia, Africa), the source of the reporter (consumer, HCPs, or all stakeholders), source of databases (poison control center, national PV center, or regional PV center), and the ROB. Sensitivity analysis was performed to examine the influence of studies with a high ROB and the source of the reporter to the pooled estimate of the reporting rate of AEs associated with HMs. A univariate random effects meta-regression was also used to investigate heterogeneity. Egger’s asymmetry test was conducted to look for signs of publication bias [23]. If significant publication bias existed, the trim-and-fill method was performed to adjust the publication bias [23]. Statistical tests were 2-sided with a significance of P<.05. All statistical analyses were performed with STATA 17.0 (Stata Corp LLC).

There were 3864 papers identified from the databases (n=3857, 99.8%, papers) and manual searching (n=7, 0.2%, papers). After removing duplicates, 3145 (81.4%) papers were screened against the eligibility criteria. Screening of titles or abstracts resulted in 3058 (97.2%) papers being excluded. A total of 87 (2.8%) papers remained for full-text screening. Of those, 61 (70.1%) studies were excluded because of AEs not collected through voluntary reporting schemes (n=33, 54.1%, papers), detailed AEs not provided (n=14, 23%, papers), duplicated data sources included from other reports (n=2, 3.3%, papers), and information about complementary and alternative medicines provided (n=12, 19.6%, papers). See Figure 1 for the PRISMA flowchart of the study selection process.

In total, 26 studies were included in the systematic review [16,24-48] (Table 1). These studies were published between 2005 and 2021, with 20 (76.9%) studies in English [16,24-32,35,36,38-42,44,45,47] 3 (11.5%) in Chinese [33,46,48] 2 (7.7%) in Spanish [34,37], and 1 (3.8%) in Dutch [43]. Most of the studies were conducted in Asia (n=12, 46.2%) [27,30-33,38,39,41,45-48], followed by Europe (n=7, 26.9%) [16,28,29,35,36,42,43], North America (n=5, 19.2%) [24,25,34,37,44], and 1 (3.8%) each in Africa [40] and Australia [26]. In terms of the source of the reporter, AEs were reported only by HCPs in 4 (15.4%) studies [29,35,38,41], including physicians, pharmacists, nurses, or herbal practitioners; by HCPs, consumers, and other stakeholders (pharmaceutical companies, manufacturers) in 18 (69.2%) studies; and by only consumers in 2 (7.7%) studies [24,26]; the remaining 2 (7.7%) studies [34,37] had no information on reporters. Data of AEs were acquired from the spontaneous reporting system of the national PV center (n=21, 80.8%) [16,26,28-31,33,35-48], the regional PV center (n=3, 11.5%) [27,32,34], and the poison control center (n=2, 7.7%) [24,25].

Of the 26 studies reviewed, 24 (92.3%) provided information about the severity of AEs but only 9 (34.6%) studies specifically mentioned the assessment scale. Among these, the WHO scale was the most commonly used, mentioned in 4 (15.4%) studies [24,34,41,45], while 1 (3.8%) study used a modified Hartwig scale [26]. In contrast, 4 (15.4%) studies [25,38,42,43] defined their own severity classification, distinguishing between nonserious and serious events or categorizing them as minor, moderate, or major.

The most frequently affected systems as per the median reporting rate were (1) the skin and appendage (median 22.3%, IQR 13.3%-35.1%), with AEs such as rash, itching, erythema, urticaria, and pruritus [16,27-30,33-35,37,38,40-43,46,47]; (2) the gastrointestinal system (median 17.5%, IQR 9.0%-36.9%), with AEs such as abdominal pain, nausea, and vomiting [16,27-30,33-35,37,38,40-42,44,46,47]; (3) the central nervous system (median 12.5%, IQR 6.9%-17.9%), with AEs such as dizziness, fatigue, and headache [16,26-28,30,33-35,37,38,40-44,46,47]; and (4) the cardiovascular system (median 5.0%, IQR 1.8%-19.0%), with AEs such as hypertension, heart flutter, and tachycardia [16,28,29,35,37,40,41,44,47]. One study [36] reported AEs pertaining to the hepatobiliary system, including hepatitis (66.7%), hepatic necrosis (16.7%), and increasing hepatic enzymes (33.3%).

Of the 26 studies, 11 (42.3%) reported AE outcomes. The reported outcomes after treatment were “not recovered” (median 0.6%, IQR 0.3%-11.8%) in 5 (45.5%) studies [27,31,46-48], “recovered” (median 46.4%, IQR 12.5%-66.8%) in 10 (90.9%) studies [16,27,28,31-33,40,46-48], “cured” (median 54.2%, IQR 23.8%-63.8%) in 5 (45.5%) studies [27,32,33,46,48], and “fatal/death” (median 1.5%, IQR 0.7%-6.6%) in 6 (54.5%) studies [27,29,31,32,40,47]. Table S1 in Multimedia Appendix 1 shows a detailed summary.

Of the 26 studies, 16 (61.5%) reported the causality assessment scale used, where the WHO Uppsala Monitoring Centre (UMC) was the most common in 8 (50%) studies [16,29-31,34,36,40,42], followed by the Naranjo algorithm in 2 (12.5%) studies [38,43]; the Karch and Lasagna criteria in 1 (6.3%) study [37]; a combination of the WHO-UMC, the Naranjo algorithm, and the local Thai algorithm in 1 (6.3%) study [45]; and a predefined scale (definite, probable, possible, doubtful) in 1 (6.3%) study [24]. In addition, 3 (18.8%) of the 16 studies [33,46,48] conducted in China applied data mining methods, including the proportional reporting ratio (PRR) and the Bayesian confidence propagation neural network (BCPNN), for signal monitoring to detect the correlation between AEs and HMs (Table S1 in Multimedia Appendix 1).

The causality assessment ranged mostly from “possible” to “probable” (Table S1 in Multimedia Appendix 1). Notably, by using data mining methods (the PRR and the BCPNN), the 3 (18.8%) studies conducted in China could clarify the warning signals for specified HMs: Dengzhan Xixin (headache, dizziness, itching, chills, heart palpitations, fever, flushing) [33], Guizhi Fuling (gastric dysfunction, abdominal pain [46], and Shujinjianyao (rash, nausea, abdominal pain, headache, vomiting) [48].

None of included studies provided full details of herbal products, with deficiency noted in the omission of batch numbers, identified as crucial elements [49]. Among the reviewed studies, a minority (n=11, 42.3%) mentioned specific brand names [16,24,27-30,33,35,43,46,48], 1 (3.8%) study identified the manufacturer [24], and 9 (34.6%) detailed the pharmaceutical form [27,28,32-34,40,41,46,48]. Furthermore, there was a notable absence of data pertaining to the quality assessment of herbal medicinal products, including information about testing for contamination, the presence of adulterants, or purity levels.

Applying Crombie’s checklist for ROB assessment, 3 (11.5%) of the 26 included studies [27,29,31] fulfilled most of the criteria and were classified as showing low ROB (6-7 points), 8 (30.8%) studies [26,28,32-34,37,45,47] showed a moderate ROB (>4 to <6 points), and 15 (57.5%) studies [16,24,25,30,35,36,38-44,46,48] were classified as showing a high ROB (≤4 points). For ROB assessments, the interrater agreement was high (Cohen κ=0.94).

The evidence of publication bias was detected by performing the Egger test (P=.01). The trim-and-fill method of calibrating the publication bias identified 3 (11.5%) imputed studies (Figure 2).

A total of 14 (53.8%) of the 26 studies providing sufficient data for calculating the reporting rate of AEs were submitted for meta-analyses.

The increasing worldwide trend of HM usage reflects the growing number of people placing faith in the efficacy of HMs, emphasizing the urgent need to address concerns about the risks and benefits of HMs. This systematic review and meta-analysis critically assessed the reporting rate of AEs associated with HMs using data sourced from PV databases. Our findings underscore the presence of potential risks, alongside benefits, prompting continued concerns regarding the safety of HMs in clinical applications. The reported median rate of the AEs of HMs was calculated as 1.42%, with the highest AE reported rate of 29.84% and the reported outcomes of AEs being as serious as death (1.5%). With the increased trend in the consumption of HMs, an effective ongoing process to detect, assess, and prevent adverse effects from HMs is essential, allowing us to understand more about their benefits and risks. This understanding is crucial for promoting informed decision-making among health care providers and consumers, ensuring safer use of these products. Additionally, it facilitates the development of evidence-based guidelines and recommendations for integrating HMs into health care practices, while addressing gaps in knowledge regarding their pharmacological interactions and long-term effects.

The meta-regression and sensitivity analysis revealed a significant impact, with the highest reporting rate (median 29.84%, IQR 27.30%-32.51%) observed for AEs reports submitted by consumers. This highlights the importance of direct consumer reporting in identifying the adverse effects of HMs and underscores the necessity of integrating such reporting into the PV system [2]. Since the 20th century, numerous countries have established consumer reporting systems, with Canada pioneering this initiative in 1965 [50]. Various measures have been undertaken to motivate consumers to report, including providing feedback to reporters and promoting the reporting system through media, social media, and health care providers [50]. In addition, consumer organizations play a crucial role in bridging the gap between consumers and national PV centers, thereby fostering the use of these reporting systems [2,50]. However, since only medical professionals can accurately establish causality, relying solely on information provided by consumers does not validate the occurrence of an AE caused by a specific product [51]. Hence, effort is needed to develop and enhance more effective processes for collecting, detecting, and assessing the quality of consumer reports. First, the reporting form should be simplified for consumers to complete by using layperson language and including consumer-specific questions, especially other aspects of medicine use, such as experiences of ineffectiveness [52]. Their experiences might provide important insight, emphasizing aspects consumers are unable to communicate to their doctors [53]. Since there has been no standardized consumer reporting form for the AEs of HMs, the promotion and emphasis of international guidelines for a standard patient reporting form are essential [2,50]. Second, promotion and education of using the spontaneous reporting system for consumers are required, even though consumers may be aware of the self-reporting system and prepared to use it [2,54]. Third, a follow-up strategy should be implemented to obtain more medical confirmations, which can aid in analyzing the cause-and-effect relationship and ensuring the data elements of reports are as complete and accurate as possible [2].

A significant challenge in summarizing data in this systematic review and meta-analysis was the lack of comprehensive information about the HMs cited in the AE reports. Only the main ingredients were disclosed, while other potentially harmful components were not listed. Clearly stating all herbal components in AE reports is essential for ensuring accurate and comprehensive documentation of the potential risks associated with HMs [6]. Understanding specific botanical ingredients helps HCPs and researchers analyze adverse reactions more effectively. Identifying these components also aids in assessing interactions between herbs and medications, improving comprehension of AE mechanisms [15,55]. This knowledge supports creating clearer guidelines and warnings, enhancing transparency and reliability in reporting AEs, and promoting safer use and informed decision-making in the herbal product industry [6]. AE reports for HMs should include the brand name, manufacturer, pharmaceutical form, extract amount per dose, ingredients, excipients, regulatory status, and test results for contamination, adulterants, or purity [49].

Although numerous HMs and AEs were screened in this systematic review and meta-analysis, this is deemed insufficient, as many unregulated or self-prescribed HMs remain unaccounted for by existing PV systems [6]. Challenges in HM monitoring include diversity in the classification of HMs across nations, a lack of rigorous quality management, and insufficient collaboration among stakeholders. Therefore, a PV system for HMs should raise awareness of PV activities for HMs in the public domain. The involvement of all relevant stakeholders, including HCPs, consumers/patients, manufacturers, complementary practitioners, and sellers/distributors, is crucial for the effective engagement of the system in actively monitoring and reporting AEs [6,55].

This study has many strengths, including a comprehensive literature search without language restrictions, double-data extraction, and ROB assessments. Our systematic review and meta-analysis also has some limitations. First, due to limited available publications, only 1 study targeting consumer reports was included in the quantitative analysis as a representative of the AE-reporting rate from consumer reports. The result therefore needs to be interpreted with caution. Additional studies gathering data on consumer reports should be conducted to obtain more accurate estimates of the reporting rate. Second, this systematic review and meta-analysis excluded studies on prevalence or incidence rates that did not collect AE reports from PV databases. Future studies should examine the prevalence rate of AEs associated with HMs in various settings (eg, hospitals, community pharmacies) to provide an overview of real-world data on these AEs.

This systematic review and meta-analysis highlighted HM risks with a wide range of AE-reporting rates, depending on the source of the reporter, and revealed deficiencies in detailed HM component information provided within AE reports. Continuous efforts are necessary to standardize consumer reporting systems in terms of the reporting form, education, and follow-up strategy to improve data quality assurance measures, aiming to enhance the reliability and utility of PV data for monitoring the safety of HMs. Achieving effective monitoring and reporting of these AEs necessitates collaborative efforts from diverse stakeholders, including patients/consumers, manufacturers, physicians, complementary practitioners, sellers/distributors, and health authorities.