This paper is in the following e-collection/theme issue:

Reviews on Public Health Technology and Innovation (69) Pharmacovigilance (134) Adverse Drug Events Detection, Pharmacovigilance and Surveillance (156) Pharmacology and Pharmaceutics (66)

Published on in Vol 10 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/63808, first published .
Estimating Adverse Events Associated With Herbal Medicines Using Pharmacovigilance Databases: Systematic Review and Meta-Analysis

Estimating Adverse Events Associated With Herbal Medicines Using Pharmacovigilance Databases: Systematic Review and Meta-Analysis

Estimating Adverse Events Associated With Herbal Medicines Using Pharmacovigilance Databases: Systematic Review and Meta-Analysis

Authors of this article:

Chuenjid Kongkaew1, 2 Author Orcid Image ;   Dang Thuc Anh Phan1, 3 Author Orcid Image ;   Prathan Janusorn1, 4 Author Orcid Image ;   Pajaree Mongkhon1, 5 Author Orcid Image
Article Authors Cited by Tweetations (1) Metrics Author Video

    Review

    1Centre for Safety and Quality in Health, Department of Pharmacy Practice, Faculty of Pharmaceutical Sciences, Naresuan University, Phitsanulok, Thailand

    2Department of Pharmaceutical and Biological Chemistry, School of Pharmacy, University College London, London, United Kingdom

    3Faculty of Pharmacy, Hue University of Medicine and Pharmacy, Hue City, Vietnam

    4Pharmacy Department, Soidao Hospital, Chantaburi, Thailand

    5School of Pharmaceutical Sciences, University of Phayao, Phayao, Thailand

    Corresponding Author:

    Chuenjid Kongkaew, PhD

    Centre for Safety and Quality in Health

    Department of Pharmacy Practice

    Faculty of Pharmaceutical Sciences, Naresuan University

    99 Moo 9

    Thapo Sub-district, Muang District

    Phitsanulok, 65000

    Thailand

    Phone: 66 9332 92451

    Email: chuenjidk@nu.ac.th


    Background: Herbal medicines (HMs) are extensively used by consumers/patients worldwide. However, their safety profiles are often poorly reported and characterized. Previous studies have documented adverse events (AEs) associated with HMs, such as hepatotoxicity, renal failure, and allergic reactions. However, the prevalence rate of AEs related to HMs has been reported to be low. To date, no systematic review and meta-analysis has comprehensively analyzed the AEs of HMs using published data acquired from pharmacovigilance (PV) databases.

    Objective: This study aimed to (1) estimate the reporting rate of the AEs of HMs using PV databases and (2) assess the detailed data provided in AE reports.

    Methods: In this systematic review and meta-analysis, MEDLINE/PubMed, SCOPUS, EMBASE, and CINAHL were systematically searched for relevant studies (until December 2023). The DerSimonian-Laird random effects model was used for pooling the data, and subgroup analyses, the meta-regression model, and sensitivity analysis were used to explore the source of heterogeneity. Crombie’s checklist was used to evaluate the risk of bias (ROB) of the included studies.

    Results: In total, 26 studies met the eligibility criteria. The reporting rate of the AEs of HMs ranged considerably, from 0.03% to 29.84%, with a median overall pooled estimate of 1.42% (IQR 1.12%-1.72%). Subgroup analyses combined with the meta-regression model revealed that the reporting rate of the AEs of HMs was associated with the source of the reporter (P=.01). None of the included studies provided full details of suspected herbal products, only the main ingredients were disclosed, and other potentially harmful components were not listed.

    Conclusions: This systematic review and meta-analysis highlighted risks related to HMs, with a wide range of reporting rates, depending on the source of the reporter. Continuous efforts are necessary to standardize consumer reporting systems in terms of the reporting form, education, and follow-up strategy to improve data quality assurance, aiming to enhance the reliability and utility of PV data for monitoring the safety of HMs. Achieving effective monitoring and reporting of these AEs necessitates collaborative efforts from diverse stakeholders, including patients/consumers, manufacturers, physicians, complementary practitioners, sellers/distributors, and health authorities.

    Trial Registration: PROSPERO (Prospective International Register of Systematic Reviews) CRD42021276492

    JMIR Public Health Surveill 2024;10:e63808

    doi:10.2196/63808

    Keywords

    herbal medicine; pharmacovigilance; adverse event; spontaneous reporting system; meta-analysis 


    The World Health Organization (WHO) defined herbal medicines (HMs) as substances that “[…]contain active ingredient parts of plants or other plant materials or combinations thereof” [1]. These substances have been used as part of traditional (folk) medicine over the millennia, and they are becoming increasingly popular in recent years [1,2]. More than 80% of the population worldwide relies on traditional herbal treatment, and those estimates differ by country, ethnicity, age group, gender, or clinical condition [3-5]. There are many reasons for this high consumption, including escalating costs of health care, barriers to physician consultations, personal preferences, and perceived safety and health benefits of HMs. Most consumers/patients assume that HMs are of natural origin and believe them to be harmless [6]. Contrary to this fallacy, there have been reports of serious adverse events (AEs) associated with HMs, including hepatotoxicity, renal failure, allergic reactions, colon perforation, carcinoma, coma, and even death [7,8]. These AEs can be attributed to overdosing, adulteration, or contamination of HMs, herb-drug interactions, and herb-herb interactions [9-12].

    Pharmacovigilance (PV) is the science and activities relating to the detection, assessment, understanding, and prevention of the adverse effects of drugs or any other possible drug-related problems [2]. Essentially, a PV system aims to avert AEs resulting from medication use and implement measures to minimize the consequences of potential adverse effects [2,13]. Over recent decades, the assessment of medication safety and benefits has been significantly transformed by the development of large databases and statistical programs, enhancing the use and rapid analysis of data [14]. Collecting and analyzing individual reports on AEs remains the most cost-effective and straightforward method for drug safety assessment and new signal detection [13].

    Individuals reporting AEs have been used as sources of data on the safety of medicinal products. These reporters may include health care professionals (HCPs), such as physicians, dentists, pharmacists, and nurses, who possess medical qualifications. Additionally, consumers (non-HCPs), such as patients, patients’ relatives, and caregivers, are now recognized as valuable sources of safety information about medicinal products [2,13].

    In light of increasing safety concerns, many researchers advocate for the integration of herbal products into the existing PV system and the use of a single reporting form [15]. Recently, the PV of HMs, known as phytovigilance, has garnered attention [6], with spontaneous reporting identified as the primary method for monitoring these products [2,16].

    To the best of our knowledge, no similar systematic review and meta-analysis evaluating the reporting rate of the AEs of HMs using the data acquired from PV databases has been published. In addition, little is known about the quality of information provided within the AE reports through a spontaneous reporting system. Therefore, the aim of this systematic review and meta-analysis was to estimate the reporting rate of the AEs of HMs using data from PV databases and to assess the detailed information provided in AE reports. The study also aimed to scrutinize the characteristics of these AEs, including their severity, causality, and outcomes.


    Search Strategy

    This systematic review and meta-analysis was registered with PROSPERO (International Prospective Register of Systematic Reviews; ID CRD42021276492) and conducted by adhering to the Cochrane Handbook for Systematic Reviews of Interventions and the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The MEDLINE/PubMed, EMBASE, CINAHL, and SCOPUS databases were searched from their inception until December 2023 by 2 researchers (authors DTAP and PJ). Additional relevant papers were also obtained through a manual search by checking each included study’s references. The gray literature, such as abstracts, conference proceedings, and editorials, was excluded. The keywords and their synonyms for the search strategy were as follows: (“pharmacovigilance” OR “post-marketing surveillance” OR “adverse event reporting” OR “adverse event reporting system” OR “adverse event reports” OR “self-reporting”) AND (“herbal medicine” OR “herbal remedies” OR “medicinal herb” OR “herbal drugs” OR “herbal products” OR “botanical medicines” OR “phytomedicine” OR “phytotherapy” OR “traditional medicine”). In this study, the term “herbal medicines” was defined as “herbs, herbal materials, herbal preparations and finished herbal products. Herbs include crude plant material (leaves, flowers, fruit, seeds, stems, wood, bark, roots, rhizomes or other plant parts, which may be entire, fragmented or powdered). Herbal materials include, alongside herbs, fresh juices, gums, fixed oils, essential oils, resins and dry powders of herbs. Herbal preparations include comminuted or powdered herbal materials, or extracts, tinctures and fatty oils of herbal materials. Finished herbal products consist of herbal preparations made from one or more herbs” [2].

    Study Selection

    Primary studies were included in this research if they were original studies reporting AEs associated with HMs through a voluntary reporting scheme, with no restriction of language. Studies were excluded if they (1) were reviews or systematic reviews, case reports, conference abstracts, or editorials; (2) were studies reporting AEs on animals; (3) did not provide AEs in detail; or (4) were studies reporting AEs associated with complementary and alternative medicines.

    The primary outcome was the reporting rate of the AEs of HMs, calculated as the number of reports referred to AEs associated with HMs (numerator) divided by the total number of reports of AEs associated with all medicines (denominator).

    Studies that could not be included in the meta-analysis because of incomplete data were reviewed narratively. Endnote 20 (Clarivate Analytics) was used as a reference manager to import citations and remove duplicate publications.

    Data Screening and Extraction

    Titles and abstracts were screened independently by 2 researchers (DTAP and PJ). The full text was then examined in detail by the same 2 researchers. Non-English papers were translated using Google Translate [17].

    The following data were independently double-extracted into evidence tables by 2 researchers (authors CK and DTAP): general characteristics of the studies (eg, country of study, source of databases, study period) and outcome data (eg, reporting rate of AEs of HMs, severity of AEs, causality assessment, affected body parts or systems most involved in AEs).

    Risk-of-Bias Assessment

    Two independent investigators (DTAP and PJ) assessed the risk of bias (ROB) using Crombie’s checklist for cross-sectional studies [18,19]. The following items were appraised: (1) appropriate design, (2) adequate description of data, (3) reported response rates, (4) adequate representation of the total sample, (5) clearly stated aims and the likelihood of reliable and valid measurements, (6) statistical significance, and (7) adequate description of analyses. Each item scored 1 point for yes, 0.5 points for unclear, and 0 for no. The total score was modified as high ROB (0-4), moderate ROB (>4 and <6), and low ROB (6-7). Any discrepancies during data extraction and ROB assessment were resolved by consensus between these investigators and the principal investigator (CK). The κ statistic was used to evaluate the degree of agreement between investigators. A negative κ indicated a lack of agreement, while the following ranges were used to interpret the level of agreement: 0.01-0.20 as none to slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0.81-1.00 as almost perfect agreement [20].

    Data Synthesis and Statistical Analysis

    Pooled effect estimates for the reporting rate of the AEs of HMs across the included studies with corresponding 95% CIs were calculated as a percentage using the DerSimonian-Laird random effects model [21]. To assess the heterogeneity among studies, standard χ2 tests and the I2 statistic were used [22]. Where high heterogeneity was indicated (I2≥75%), the results across studies were summarized using the median reporting rate and the IQR. To explore the possible sources of heterogeneity, subgroup analyses were performed by continent (North America, Europe, Asia, Africa), the source of the reporter (consumer, HCPs, or all stakeholders), source of databases (poison control center, national PV center, or regional PV center), and the ROB. Sensitivity analysis was performed to examine the influence of studies with a high ROB and the source of the reporter to the pooled estimate of the reporting rate of AEs associated with HMs. A univariate random effects meta-regression was also used to investigate heterogeneity. Egger’s asymmetry test was conducted to look for signs of publication bias [23]. If significant publication bias existed, the trim-and-fill method was performed to adjust the publication bias [23]. Statistical tests were 2-sided with a significance of P<.05. All statistical analyses were performed with STATA 17.0 (Stata Corp LLC).


    Search Outcomes

    There were 3864 papers identified from the databases (n=3857, 99.8%, papers) and manual searching (n=7, 0.2%, papers). After removing duplicates, 3145 (81.4%) papers were screened against the eligibility criteria. Screening of titles or abstracts resulted in 3058 (97.2%) papers being excluded. A total of 87 (2.8%) papers remained for full-text screening. Of those, 61 (70.1%) studies were excluded because of AEs not collected through voluntary reporting schemes (n=33, 54.1%, papers), detailed AEs not provided (n=14, 23%, papers), duplicated data sources included from other reports (n=2, 3.3%, papers), and information about complementary and alternative medicines provided (n=12, 19.6%, papers). See Figure 1 for the PRISMA flowchart of the study selection process.

    ‎
    Figure 1. PRISMA flow diagram. AE: adverse event; PRISMA; Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

    Characteristics of Included Studies

    In total, 26 studies were included in the systematic review [16,24-48] (Table 1). These studies were published between 2005 and 2021, with 20 (76.9%) studies in English [16,24-32,35,36,38-42,44,45,47] 3 (11.5%) in Chinese [33,46,48] 2 (7.7%) in Spanish [34,37], and 1 (3.8%) in Dutch [43]. Most of the studies were conducted in Asia (n=12, 46.2%) [27,30-33,38,39,41,45-48], followed by Europe (n=7, 26.9%) [16,28,29,35,36,42,43], North America (n=5, 19.2%) [24,25,34,37,44], and 1 (3.8%) each in Africa [40] and Australia [26]. In terms of the source of the reporter, AEs were reported only by HCPs in 4 (15.4%) studies [29,35,38,41], including physicians, pharmacists, nurses, or herbal practitioners; by HCPs, consumers, and other stakeholders (pharmaceutical companies, manufacturers) in 18 (69.2%) studies; and by only consumers in 2 (7.7%) studies [24,26]; the remaining 2 (7.7%) studies [34,37] had no information on reporters. Data of AEs were acquired from the spontaneous reporting system of the national PV center (n=21, 80.8%) [16,26,28-31,33,35-48], the regional PV center (n=3, 11.5%) [27,32,34], and the poison control center (n=2, 7.7%) [24,25].

    Table 1. Characteristics of included studies (N=26).
    StudyCountrySource of databasesReport collection durationSource of the reporter
    Mazzanti et al [16]ItalyPhytovigilance system database from the Italian National Institute of Health7 yearsAnyone observing a suspected adverse reaction
    Dennehy et al [24]United StatesCalifornia Poison Control System (CPCS)6 monthsConsumers
    Gryzlak et al [25]United StatesAmerican Association of Poison Control Center1 yearHCPsa, consumers
    Hoban et al [26]AustraliaTherapeutic Goods Administration (TGA)14 yearsHCPs
    Huang et al [27]ChinaHubei Adverse Drug Reaction Monitoring Center6 yearsMedical institutions, enterprises, consumers
    Ippoliti et al [28]ItalyItalian Medicines Agency18 yearsPhysicians, pharmacists, consumers
    Jacobsson et al [29]SwedenSwedish Medical Products Agency (MPA)19 yearsSenior physicians (75%), pharmacists (4%), drug industry staff (2%), nurses (1%), dentists (0.3%)
    Kalaiselvan et al [30]IndiaNational Coordination Centre (NCC) for the Pharmacovigilance Programme of India2 years 6 monthsHCPs, consumers, pharmaceutical companies
    Lee et al [31]MalaysiaMalaysian Adverse Drug Reactions Advisory Committee (MADRAC)18 yearsPharmacists (n=196, 83.4%), physicians (n=27, 11.5%), pharmaceutical companies (n=2, 0.9%), others (n=10, 4.3%)
    Li et al [32]ChinaChina Guangdong Provincial Center of ADR Monitoring, district hospitals12 yearsHCPs, manufacturers, consumers
    Li et al [33]ChinaNational Drug ADR Monitoring Center4 yearsHCPs, consumers, others
    Lores and Lazo [34]Republic of CubaProvincial Coordinating Unit of Pharmacovigilance Santiago de Cuba5 yearsN/Rb
    Menniti-Ippolito et al [35]ItalyItalian National Institute of Health5 yearsHospital doctors (43%), pharmacists (22%), general practitioners (16%), specialist physicians (8%), herbalists (2%), others (5%), patients (1 report)
    Petronijevic et al [36]Republic of SerbiaNational Pharmacovigilance Centre14 yearsPhysicians (81.4%), pharmacists (5.7%), marketing authorization holders (12.9%)
    Salvador et al [37]Republic of CubaCuban Pharmacovigilance System Vigibase8 yearsN/R
    Saokaew et al [38]ThailandHealth Products Vigilance Center (HPVC)9 yearsHCPs (100%)
    Shin et al [39]Republic of KoreaKorean Food and Drug Administration, Consumer Agency, Association of Traditional Korean Medicine12 yearsConsumers, pharmaceutical companies
    Skalli et al [40]MoroccoMoroccan Pharmacovigilance Herbal Center9 yearsN/R
    Suwankesawong et al [41]ThailandThailand Health Product Vigilance Center (HPVC)1 year, 11 monthsHCPs (100%)
    Svedlund et al [42]SwedenSwedish Medical Products Agency9 yearsHCPs, patients
    van Hunsel and van Grootheest [43]NetherlandDutch Updates Center Lareb12 yearsPharmacies (44%), general practitioners (27%), consumers (14%), specialists (14%), hospital pharmacists (0.8%)
    Wallace et al [44]United StatesCenter for Food Safety and Applied Nutrition’s Adverse Event Reporting System (CAERS)5 yearsHCPs, consumers, manufacturers
    Wechwithan et al [45]ThailandNational pharmacovigilance database12 yearsHCPs, consumers
    Wei et al [46]ChinaNational Adverse Reaction Monitoring Center9 yearsReported by everyone
    Xu et al [47]SingaporeVigilance & Compliance Branch of Health Sciences Authority7 yearsDoctors (87%), pharmacists (10%), traditional Chinese medicine practitioners (0.9%), manufacturers (1.1%), nurses (0.5%), others (0.3%)
    Zhang et al [48]ChinaNational Center for Adverse Drug Reaction Monitoring12 yearsReported by everyone

    aHCP: health care professionals.

    bN/R: not reported.

    Severity, Affected Body Systems, and Outcomes Involving AEs of HMs

    Of the 26 studies reviewed, 24 (92.3%) provided information about the severity of AEs but only 9 (34.6%) studies specifically mentioned the assessment scale. Among these, the WHO scale was the most commonly used, mentioned in 4 (15.4%) studies [24,34,41,45], while 1 (3.8%) study used a modified Hartwig scale [26]. In contrast, 4 (15.4%) studies [25,38,42,43] defined their own severity classification, distinguishing between nonserious and serious events or categorizing them as minor, moderate, or major.

    The most frequently affected systems as per the median reporting rate were (1) the skin and appendage (median 22.3%, IQR 13.3%-35.1%), with AEs such as rash, itching, erythema, urticaria, and pruritus [16,27-30,33-35,37,38,40-43,46,47]; (2) the gastrointestinal system (median 17.5%, IQR 9.0%-36.9%), with AEs such as abdominal pain, nausea, and vomiting [16,27-30,33-35,37,38,40-42,44,46,47]; (3) the central nervous system (median 12.5%, IQR 6.9%-17.9%), with AEs such as dizziness, fatigue, and headache [16,26-28,30,33-35,37,38,40-44,46,47]; and (4) the cardiovascular system (median 5.0%, IQR 1.8%-19.0%), with AEs such as hypertension, heart flutter, and tachycardia [16,28,29,35,37,40,41,44,47]. One study [36] reported AEs pertaining to the hepatobiliary system, including hepatitis (66.7%), hepatic necrosis (16.7%), and increasing hepatic enzymes (33.3%).

    Of the 26 studies, 11 (42.3%) reported AE outcomes. The reported outcomes after treatment were “not recovered” (median 0.6%, IQR 0.3%-11.8%) in 5 (45.5%) studies [27,31,46-48], “recovered” (median 46.4%, IQR 12.5%-66.8%) in 10 (90.9%) studies [16,27,28,31-33,40,46-48], “cured” (median 54.2%, IQR 23.8%-63.8%) in 5 (45.5%) studies [27,32,33,46,48], and “fatal/death” (median 1.5%, IQR 0.7%-6.6%) in 6 (54.5%) studies [27,29,31,32,40,47]. Table S1 in Multimedia Appendix 1 shows a detailed summary.

    Causality Assessment Between AEs and HMs

    Of the 26 studies, 16 (61.5%) reported the causality assessment scale used, where the WHO Uppsala Monitoring Centre (UMC) was the most common in 8 (50%) studies [16,29-31,34,36,40,42], followed by the Naranjo algorithm in 2 (12.5%) studies [38,43]; the Karch and Lasagna criteria in 1 (6.3%) study [37]; a combination of the WHO-UMC, the Naranjo algorithm, and the local Thai algorithm in 1 (6.3%) study [45]; and a predefined scale (definite, probable, possible, doubtful) in 1 (6.3%) study [24]. In addition, 3 (18.8%) of the 16 studies [33,46,48] conducted in China applied data mining methods, including the proportional reporting ratio (PRR) and the Bayesian confidence propagation neural network (BCPNN), for signal monitoring to detect the correlation between AEs and HMs (Table S1 in Multimedia Appendix 1).

    The causality assessment ranged mostly from “possible” to “probable” (Table S1 in Multimedia Appendix 1). Notably, by using data mining methods (the PRR and the BCPNN), the 3 (18.8%) studies conducted in China could clarify the warning signals for specified HMs: Dengzhan Xixin (headache, dizziness, itching, chills, heart palpitations, fever, flushing) [33], Guizhi Fuling (gastric dysfunction, abdominal pain [46], and Shujinjianyao (rash, nausea, abdominal pain, headache, vomiting) [48].

    Quality of Information Provided in AE Reports

    None of included studies provided full details of herbal products, with deficiency noted in the omission of batch numbers, identified as crucial elements [49]. Among the reviewed studies, a minority (n=11, 42.3%) mentioned specific brand names [16,24,27-30,33,35,43,46,48], 1 (3.8%) study identified the manufacturer [24], and 9 (34.6%) detailed the pharmaceutical form [27,28,32-34,40,41,46,48]. Furthermore, there was a notable absence of data pertaining to the quality assessment of herbal medicinal products, including information about testing for contamination, the presence of adulterants, or purity levels.

    Risk-of-Bias Assessment and Publication Bias

    Applying Crombie’s checklist for ROB assessment, 3 (11.5%) of the 26 included studies [27,29,31] fulfilled most of the criteria and were classified as showing low ROB (6-7 points), 8 (30.8%) studies [26,28,32-34,37,45,47] showed a moderate ROB (>4 to <6 points), and 15 (57.5%) studies [16,24,25,30,35,36,38-44,46,48] were classified as showing a high ROB (≤4 points). For ROB assessments, the interrater agreement was high (Cohen κ=0.94).

    The evidence of publication bias was detected by performing the Egger test (P=.01). The trim-and-fill method of calibrating the publication bias identified 3 (11.5%) imputed studies (Figure 2).

    ‎
    Figure 2. Funnel plots testing the publication bias before (left) and after (right) applying the trim-and-fill method.

    Quantitative Synthesis

    A total of 14 (53.8%) of the 26 studies providing sufficient data for calculating the reporting rate of AEs were submitted for meta-analyses.

    Pooled Reporting Rate of AEs of HMs Using PV Databases

    The meta-analysis of the 14 (53.8%) studies showed a pooled reporting rate of the AEs of HMs at a median of 1.42% (IQR 1.12%-1.71%). There was significant heterogeneity (χ213=33,650.09, P<.001, I2=99.96%) as the reporting rates ranged considerably from 0.03% to 29.84% (Figure 3).

    ‎
    Figure 3. Forest plot showing the pooled reporting rate of the AEs associated with HMs. AE: adverse event; HM: herbal medicine.
    Subgroup Analysis

    A subgroup analysis by continent showed that there was a significant difference between Asia and Europe (Table 2), with median estimates of 1.12% (IQR 0.74%-1.50%) and 0.83% (IQR 0.28%-1.37%), respectively (P<.001). A subgroup analysis by ROB showed that “moderate ROB” studies detected a higher reporting rate compared to “high ROB” studies (Table 2), with median estimates of 0.95% (IQR 0.33%-1.57%) and 0.41% (IQR 0.25%-0.57%), respectively (P<.001).

    Table 2. Subgroup analysis testing heterogeneity by continent, source of databases, source of the reporter, and ROBa.
    CovariatesStudies (n=14), n (%)Reporting rate estimate (%), median (IQR)Heterogeneity test

    		χ2 (df)P valueI2 (%)
    Continent

    		Asia8 (57.1)1.12 (0.74-1.50)32445.48 (7)<.00199.9

    		Africa1 (7.1)2.40 (1.68-3.40)—b——

    		Europe4 (28.6)0.83 (0.28-1.37)350.85 (3)<.00199.1

    		North America1 (7.1)29.84 (27.30-32.51)———
    Source of databases

    		Poison control center1 (7.1)29.84 (27.30-32.51)———

    		National PVc center11 (78.6)0.40 (0.30-0.50)2538.21 (10)<.00199.6

    		Regional PV center2 (14.3)2.58 (2.54-2.62)———
    Source of the reporter

    		Consumer only1 (7.1)29.84 (27.30-32.51)———

    		HCPsd only3 (21.4)0.44 (0.14-0.73)1044.49 (2)<.00199.8

    		All stakeholders10 (71.4)1.32 (0.80-1.85)29250.89 (9)<.00199.9
    ROB

    		High7 (50.0)0.41 (0.25-0.57)1527.25 (6)<.00199.6

    		Moderate5 (35.7)0.95 (0.33-1.57)4475.91 (4)<.00199.9

    		Low2 (14.3)0.84 (0.82-0.87)———

    aROB: risk of bias.

    bNot applicable.

    cPV: pharmacovigilance.

    dHCP: health care professional.

    Meta-Regression Model

    We conducted meta-regression analysis to identify potential sources of heterogeneity in the reporting rate of the AEs of HMs. The covariates considered included the continent, the source of the reporter, source of databases, and the ROB. From the model, we found that the source of the reporter was associated with the pooled reporting rate of the AEs of HMs (P=.01), whereas there was no relationship between the location, databases, or ROB and the reporting rate of AEs (Table 3).

    Table 3. Results of univariate meta-regression analysis and the potential sources of heterogeneity in the reporting rate of AEsa associated with HMsb.
    CovariateCoefficient (SE)t13 (95% CI)P value
    Continent0.032 (0.016)1.94 (–0.004 to 0.668).076
    Source of the reporter–0.083 (0.026)–3.19 (–0.140 to –0.026).008
    Source of databases–0.084 (0.040)–2.08 (–0.173 to 0.004).059
    ROBc0.016 (0.029)0.52 (–0.0492 to 0.803).061

    aAE: adverse event.

    bHM: herbal medicine.

    cROB: risk of bias.

    Sensitivity Analysis

    There was little difference in the pooled reporting rate when studies with a high ROB were added to the analysis. The reporting rate slightly decreased from 1.58% to 1.42%. In contrast, of the 14 (53.8%) studies included in the analysis, 1 (7.1%) [24] with only consumer reports impacted the pooled reporting rate. When this study [24] was added, the pooled reporting rate rose significantly from 1.09% to 1.42% (Table 4).

    Table 4. Results of sensitivity analysis to examine the influence of studies with a high ROBa and source of the reporter to the pooled estimate of the reporting rate of AEsb associated with HMsc.
    AnalysisStudies (n=14), n (%)Reporting rate estimate (%), median (IQR)
    All studies14 (100.0)1.42 (1.12, 1.72)
    No studies with a high ROB7 (50.0)1.58 (0.68, 2.48)
    No study with only consumer reports13 (92.9)1.09 (0.79, 1.39)

    aROB: risk of bias.

    bAE: adverse event.

    cHM: herbal medicine.


    Principal Findings

    The increasing worldwide trend of HM usage reflects the growing number of people placing faith in the efficacy of HMs, emphasizing the urgent need to address concerns about the risks and benefits of HMs. This systematic review and meta-analysis critically assessed the reporting rate of AEs associated with HMs using data sourced from PV databases. Our findings underscore the presence of potential risks, alongside benefits, prompting continued concerns regarding the safety of HMs in clinical applications. The reported median rate of the AEs of HMs was calculated as 1.42%, with the highest AE reported rate of 29.84% and the reported outcomes of AEs being as serious as death (1.5%). With the increased trend in the consumption of HMs, an effective ongoing process to detect, assess, and prevent adverse effects from HMs is essential, allowing us to understand more about their benefits and risks. This understanding is crucial for promoting informed decision-making among health care providers and consumers, ensuring safer use of these products. Additionally, it facilitates the development of evidence-based guidelines and recommendations for integrating HMs into health care practices, while addressing gaps in knowledge regarding their pharmacological interactions and long-term effects.

    The meta-regression and sensitivity analysis revealed a significant impact, with the highest reporting rate (median 29.84%, IQR 27.30%-32.51%) observed for AEs reports submitted by consumers. This highlights the importance of direct consumer reporting in identifying the adverse effects of HMs and underscores the necessity of integrating such reporting into the PV system [2]. Since the 20th century, numerous countries have established consumer reporting systems, with Canada pioneering this initiative in 1965 [50]. Various measures have been undertaken to motivate consumers to report, including providing feedback to reporters and promoting the reporting system through media, social media, and health care providers [50]. In addition, consumer organizations play a crucial role in bridging the gap between consumers and national PV centers, thereby fostering the use of these reporting systems [2,50]. However, since only medical professionals can accurately establish causality, relying solely on information provided by consumers does not validate the occurrence of an AE caused by a specific product [51]. Hence, effort is needed to develop and enhance more effective processes for collecting, detecting, and assessing the quality of consumer reports. First, the reporting form should be simplified for consumers to complete by using layperson language and including consumer-specific questions, especially other aspects of medicine use, such as experiences of ineffectiveness [52]. Their experiences might provide important insight, emphasizing aspects consumers are unable to communicate to their doctors [53]. Since there has been no standardized consumer reporting form for the AEs of HMs, the promotion and emphasis of international guidelines for a standard patient reporting form are essential [2,50]. Second, promotion and education of using the spontaneous reporting system for consumers are required, even though consumers may be aware of the self-reporting system and prepared to use it [2,54]. Third, a follow-up strategy should be implemented to obtain more medical confirmations, which can aid in analyzing the cause-and-effect relationship and ensuring the data elements of reports are as complete and accurate as possible [2].

    A significant challenge in summarizing data in this systematic review and meta-analysis was the lack of comprehensive information about the HMs cited in the AE reports. Only the main ingredients were disclosed, while other potentially harmful components were not listed. Clearly stating all herbal components in AE reports is essential for ensuring accurate and comprehensive documentation of the potential risks associated with HMs [6]. Understanding specific botanical ingredients helps HCPs and researchers analyze adverse reactions more effectively. Identifying these components also aids in assessing interactions between herbs and medications, improving comprehension of AE mechanisms [15,55]. This knowledge supports creating clearer guidelines and warnings, enhancing transparency and reliability in reporting AEs, and promoting safer use and informed decision-making in the herbal product industry [6]. AE reports for HMs should include the brand name, manufacturer, pharmaceutical form, extract amount per dose, ingredients, excipients, regulatory status, and test results for contamination, adulterants, or purity [49].

    Although numerous HMs and AEs were screened in this systematic review and meta-analysis, this is deemed insufficient, as many unregulated or self-prescribed HMs remain unaccounted for by existing PV systems [6]. Challenges in HM monitoring include diversity in the classification of HMs across nations, a lack of rigorous quality management, and insufficient collaboration among stakeholders. Therefore, a PV system for HMs should raise awareness of PV activities for HMs in the public domain. The involvement of all relevant stakeholders, including HCPs, consumers/patients, manufacturers, complementary practitioners, and sellers/distributors, is crucial for the effective engagement of the system in actively monitoring and reporting AEs [6,55].

    Strengths and Limitations

    This study has many strengths, including a comprehensive literature search without language restrictions, double-data extraction, and ROB assessments. Our systematic review and meta-analysis also has some limitations. First, due to limited available publications, only 1 study targeting consumer reports was included in the quantitative analysis as a representative of the AE-reporting rate from consumer reports. The result therefore needs to be interpreted with caution. Additional studies gathering data on consumer reports should be conducted to obtain more accurate estimates of the reporting rate. Second, this systematic review and meta-analysis excluded studies on prevalence or incidence rates that did not collect AE reports from PV databases. Future studies should examine the prevalence rate of AEs associated with HMs in various settings (eg, hospitals, community pharmacies) to provide an overview of real-world data on these AEs.

    Conclusion

    This systematic review and meta-analysis highlighted HM risks with a wide range of AE-reporting rates, depending on the source of the reporter, and revealed deficiencies in detailed HM component information provided within AE reports. Continuous efforts are necessary to standardize consumer reporting systems in terms of the reporting form, education, and follow-up strategy to improve data quality assurance measures, aiming to enhance the reliability and utility of PV data for monitoring the safety of HMs. Achieving effective monitoring and reporting of these AEs necessitates collaborative efforts from diverse stakeholders, including patients/consumers, manufacturers, physicians, complementary practitioners, sellers/distributors, and health authorities.

    Acknowledgments

    We express our gratitude to Professor Michael Heinrich from University College London, United Kingdom, for contributing valuable knowledge and offering suggestions to enhance the robustness of our paper and Dr Pawel Posadzki from the University of Physical Education, Faculty of Rehabilitation, Cracow, Poland, for his guidance on technical issues of systematic reviews and meta-analyses.

    This work was funded by the International Research Network for Pharmacovigilance in Herbal Medicines (contract number IRN62W0003) and Thailand Science Research and Innovation (TSRI) 2019 (R2566A049 and R2566A050 NU World Class, Food and Herb for Health and Beauty).

    Data Availability

    The data sets analyzed during this study are available from the corresponding author upon reasonable request.

    Authors' Contributions

    CK conceptualized the study, while DTAP and CK verified the methodology, wrote the first draft, and reviewed the final manuscript. DTAP, PJ, PM, and CK collected data and carried out statistical analyses.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Summary of severity assessment, causality assessment, affected body systems, and outcomes of AEs of HMs. AE: adverse event; HM; herbal medicine.

    DOCX File , 34 KB
    Multimedia Appendix 2

    PRISMA checklist for reporting systematic review.

    PDF File (Adobe PDF File), 132 KB
    1. World Health Organization. Traditional medicines, global situation, issues, and challenges. In: The World Medicines Situation (3rd ed.). Geneva. World Health Organization; 2011:14.
    2. World Health Organization. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance systems. World Health Organization. 2004. URL: https://iris.who.int/bitstream/handle/10665/43034/9241592214_eng.pdf [accessed 2024-08-20]
    3. Toklu HZ. Pharmacovigilance of herbal medicine: herbavigilance. Adv Pharmacoepidemiol Drug Saf. 2016;5(4):208. [CrossRef]
    4. Woo CSJ, Lau JSH, El-Nezami H. Chapter 10 - herbal medicine: toxicity and recent trends in assessing their potential toxic effects. Adv Botan Res. 2012;62:365-384. [CrossRef]
    5. Kim M, Han C. Adverse drug reactions in Korean herbal medicine: a prospective cohort study. Eur J Integr Med. Jan 2017;9:103-109. [CrossRef]
    6. Skalli S, Wang J-B, Hugman B. Chapter 9 - pharmacovigilance of herbal medicines: the vigilance of tomorrow. In: Mukherjee PK, editor. Evidence-Based Validation of Herbal Medicine (Second Edition). Amsterdam, Netherlands. Elsevier; 2022:239-250.
    7. Shaw D, Graeme L, Pierre D, Elizabeth W, Kelvin C. Pharmacovigilance of herbal medicine. J Ethnopharmacol. Apr 10, 2012;140(3):513-518. [FREE Full text] [CrossRef] [Medline]
    8. Posadzki P, Watson LK, Ernst E. Adverse effects of herbal medicines: an overview of systematic reviews. Clin Med (Lond). Feb 2013;13(1):7-12. [FREE Full text] [CrossRef] [Medline]
    9. Pan H, Wu L, Wang P, Chiu P, Wang M. Real-world evidence of the herb-drug interactions. J Food Drug Anal. Sep 14, 2022;30(3):316-330. [CrossRef]
    10. Loo SC, Chong CP. Exploring the concurrent use and potential interactions between prescription drugs and health supplement products among patients with chronic diseases. Eastern J Med. 2021;26(4):606-617. [CrossRef]
    11. Oura K, Tanaka M, Matsumoto K, Satake R, Inoue M, Yoshida Y, et al. Analysis of drug-induced interstitial lung disease caused by herbal medicine using the Japanese Adverse Drug Event Report database. BMC Complement Med Ther. Mar 14, 2024;24(1):121. [FREE Full text] [CrossRef] [Medline]
    12. Steinhoff B. Pyrrolizidine alkaloid contamination in medicinal plants: regulatory requirements and their impact on production and quality control of herbal medicinal products. Planta Med. Feb 26, 2022;88(2):125-129. [FREE Full text] [CrossRef] [Medline]
    13. Sienkiewicz K, Burzyńska M, Rydlewska-Liszkowska I, Sienkiewicz J, Gaszyńska E. The importance of direct patient reporting of adverse drug reactions in the safety monitoring process. Int J Environ Res Public Health. Dec 31, 2021;19(1):413. [FREE Full text] [CrossRef] [Medline]
    14. Bihan K, Lebrun-Vignes B, Funck-Brentano C, Salem J. Uses of pharmacovigilance databases: an overview. Therapie. Nov 2020;75(6):591-598. [CrossRef] [Medline]
    15. Skalli S, Soulaymani Bencheikh R. Safety monitoring of herb-drug interactions: a component of pharmacovigilance. Drug Saf. 2012;35(10):785-791. [CrossRef]
    16. Mazzanti G, Vitalone A, Da Cas R, Menniti-Ippolito F. Suspected adverse reactions associated with herbal products used for weight loss: spontaneous reports from the Italian phytovigilance system. Eur J Clin Pharmacol. Nov 19, 2019;75(11):1599-1615. [CrossRef] [Medline]
    17. Jackson JL, Kuriyama A, Anton A, Choi A, Fournier J, Geier A, et al. The accuracy of Google Translate for abstracting data from non-English-language trials for systematic reviews. Ann Intern Med. Jul 30, 2019;171(9):677-679. [CrossRef]
    18. Crombie IK. The Pocket Guide to Critical Appraisal. Oxford, UK. John Wiley & Sons; 1996.
    19. Ma L, Wang Y, Yang Z, Huang D, Weng H, Zeng X. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Mil Med Res. Feb 29, 2020;7(1):7. [FREE Full text] [CrossRef] [Medline]
    20. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. 2012;3(22):276-282. [CrossRef]
    21. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. Sep 1986;7(3):177-188. [FREE Full text] [CrossRef] [Medline]
    22. Borenstein M, Hedges L, Higgins J, Rothstein H. Introduction to Meta-Analysis. Oxford, UK. John Wiley & Sons; 2009.
    23. Mavridis D, Salanti G. How to assess publication bias: funnel plot, trim-and-fill method and selection models. Evid Based Ment Health. Feb 29, 2014;17(1):30. [CrossRef] [Medline]
    24. Dennehy CE, Tsourounis C, Horn AJ. Dietary supplement-related adverse events reported to the California Poison Control System. Am J Health Syst Pharm. Jul 15, 2005;62(14):1476-1482. [CrossRef] [Medline]
    25. Gryzlak BM, Wallace RB, Zimmerman MB, Nisly NL. National surveillance of herbal dietary supplement exposures: the poison control center experience. Pharmacoepidemiol Drug Saf. Sep 06, 2007;16(9):947-957. [FREE Full text] [CrossRef] [Medline]
    26. Hoban CL, Byard RW, Musgrave IF. A comparison of patterns of spontaneous adverse drug reaction reporting with St. John's Wort and fluoxetine during the period 2000-2013. Clin Exp Pharmacol Physiol. Jul 2015;42(7):747-751. [CrossRef] [Medline]
    27. Huang R, Cai Y, Yang L, Shangguan X, Ghose B, Tang S. Safety of traditional Chinese medicine injection based on spontaneous reporting system from 2014 to 2019 in Hubei Province, China. Sci Rep. Apr 23, 2021;11(1):8875. [FREE Full text] [CrossRef] [Medline]
    28. Ippoliti I, Menniti-Ippolito F, Mazzanti G, Di Giacomo S. Suspected adverse reactions to performance enhancing dietary supplements: spontaneous reports from the Italian phytovigilance system. Phytother Res. Jun 10, 2021;35(6):3246-3261. [CrossRef] [Medline]
    29. Jacobsson I, Jönsson AK, Gerdén B, Hägg S. Spontaneously reported adverse reactions in association with complementary and alternative medicine substances in Sweden. Pharmacoepidemiol Drug Saf. Nov 03, 2009;18(11):1039-1047. [CrossRef] [Medline]
    30. Kalaiselvan V, Saurabh A, Kumar R, Singh G. Spontaneous reporting of adverse events due to herbal products in Pharmacovigilance Programme of India. 2014. Presented at: 14th ISoP Annual Meeting on New Ideas in Ancient Cultures - Advancing; October 19-22, 2014; Tianjin, China. URL: https://link.springer.com/content/pdf/10.1007/s40264-014-0208-1.pdf
    31. Lee FY, Wong H, Chan H, Mohamed Ali N, Abu Hassan MR, Omar H, et al. Hepatic adverse drug reactions in Malaysia: an 18-year review of the national centralized reporting system. Pharmacoepidemiol Drug Saf. Dec 02, 2020;29(12):1669-1679. [CrossRef] [Medline]
    32. Li H, Deng J, Yue Z, Zhang Y, Sun H. Detecting drug-herbal interaction using a spontaneous reporting system database: an example with benzylpenicillin and qingkailing injection. Eur J Clin Pharmacol. Sep 11, 2015;71(9):1139-1145. [CrossRef] [Medline]
    33. Li Y-Y, Xie Y-M, Shen H, Xiang Y-Y. Study of analysis 1 390 adverse drug reactions cases of parenterally administered Dengzhan Xixin based on China’s spontaneous response system. CJCMM. Sep 15, 2013;18(38):2998-3002. [CrossRef]
    34. Lores D D, Lazo R Y. [Characterization of suspected adverse reactions to herbal medicines reported to the Provincial Coordinating Unit of Pharmacovigilance Santiago de Cuba]. Mexican Journal of Pharmaceutical Sciences. Mar 2011. URL: https://www.scielo.org.mx/scielo.php?script=sci_arttext&pid=S1870-01952011000100004 [accessed 2024-08-20]
    35. Menniti-Ippolito F, Mazzanti G, Santuccio C, Moro PA, Calapai G, Firenzuoli F, et al. Surveillance of suspected adverse reactions to natural health products in Italy. Pharmacoepidemiol Drug Saf. Jun 10, 2008;17(6):626-635. [CrossRef] [Medline]
    36. Petronijevic M, Ilic K, Suzuki A. Drug induced hepatotoxicity: data from the Serbian pharmacovigilance database. Pharmacoepidemiol Drug Saf. Apr 02, 2011;20(4):416-423. [CrossRef] [Medline]
    37. Salvador AKR, Milian A, Carbonell LA, López GJ, Orta IA, Lee AC. Vigilance of adverse reactions of herbal medicines in Cuba for the period 2003-2010. Rev Cubana Plant Med. 2015;20(1):14-24. [FREE Full text]
    38. Saokaew S, Suwankesawong W, Permsuwan U, Chaiyakunapruk N. Safety of herbal products in Thailand: an analysis of reports in the Thai Health Product Vigilance Center database from 2000 to 2008. Drug Saf. Apr 01, 2011;34(4):339-350. [CrossRef] [Medline]
    39. Shin H, Jeong S, Lee MS, Ernst E. Adverse events attributed to traditional Korean medical practices: 1999–2010. Bull World Health Org. May 31, 2013;91(8):569-575. [CrossRef]
    40. Skalli S, Chebat A, Badrane N, Bencheikh RS. Side effects of cade oil in Morocco: an analysis of reports in the Moroccan herbal products database from 2004 to 2012. Food Chem Toxicol. Feb 2014;64:81-85. [CrossRef] [Medline]
    41. Suwankesawong W, Saokaew S, Permsuwan U, Chaiyakunapruk N. Characterization of hypersensitivity reactions reported among Andrographis paniculata users in Thailand using Health Product Vigilance Center (HPVC) database. BMC Complement Altern Med. Dec 24, 2014;14(1):515. [FREE Full text] [CrossRef] [Medline]
    42. Svedlund E, Larsson M, Hägerkvist R. Spontaneously reported adverse reactions for herbal medicinal products and natural remedies in Sweden 2007-15: report from the Medical Products Agency. Drugs Real World Outcomes. Jun 28, 2017;4(2):119-125. [FREE Full text] [CrossRef] [Medline]
    43. van Hunsel FPAM, van Grootheest ACK. [Adverse reactions to herbal remedies: analysis of reported adverse reactions in the Netherlands]. Ned Tijdschr Geneeskd. 2013;157(47):A6615. [Medline]
    44. Wallace RB, Gryzlak BM, Zimmerman MB, Nisly NL. Application of FDA adverse event report data to the surveillance of dietary botanical supplements. Ann Pharmacother. Apr 15, 2008;42(5):653-660. [CrossRef]
    45. Wechwithan S, Suwankesawong W, Sornsrivichai V, McNeil EB, Jiraphongsa C, Chongsuvivatwong V. Signal detection for Thai traditional medicine: examination of national pharmacovigilance data using reporting odds ratio and reported population attributable risk. Regul Toxicol Pharmacol. Oct 2014;70(1):407-412. [CrossRef] [Medline]
    46. Wei R-L, Xie Y-M, Zhang W-L. [Analysis on 1 500 adverse reactions of Guizhi Fuling Capsules based on spontaneous response system]. Zhongguo Zhong Yao Za Zhi. Apr 2019;44(7):1497-1502. [CrossRef] [Medline]
    47. Xu Y, Patel DN, Ng SP, Tan S, Toh D, Poh J, et al. Retrospective study of reported adverse events due to complementary health products in Singapore from 2010 to 2016. Front Med (Lausanne). Jun 12, 2018;5:167. [FREE Full text] [CrossRef] [Medline]
    48. Zhang C, Li P-P, Wang L-X, Xie Y-M, Li L. [ADR/AE early warning analysis of Shujin Jianyao Pills based on spontaneous reporting system]. Zhongguo Zhong Yao Za Zhi. Aug 2020;45(15):3533-3538. [CrossRef] [Medline]
    49. Heinrich M, Jalil B, Abdel-Tawab M, Echeverria J, Kulić ?, McGaw LJ, et al. Best practice in the chemical characterisation of extracts used in pharmacological and toxicological research-the ConPhyMP-guidelines. Front Pharmacol. Sep 13, 2022;13:953205. [FREE Full text] [CrossRef] [Medline]
    50. Worakunphanich W, Youngkong S, Suwankesawong W, Anderson C, Thavorncharoensap M. Comparison of patient adverse drug reaction reporting systems in nine selected countries. Int J Environ Res Public Health. Apr 07, 2022;19(8):4447. [FREE Full text] [CrossRef] [Medline]
    51. Hugman B. From the Uppsala Monitoring Centre : a review of viewpoint part 1 and part 2. Drug Saf. Jul 2005;28(7):645-646. [CrossRef] [Medline]
    52. van Geffen ECG, van der Wal SW, van Hulten R, de Groot MCH, Egberts ACG, Heerdink ER. Evaluation of patients' experiences with antidepressants reported by means of a medicine reporting system. Eur J Clin Pharmacol. Dec 15, 2007;63(12):1193-1199. [FREE Full text] [CrossRef] [Medline]
    53. Vilhelmsson A. Consumer narratives in ADR reporting: an important aspect of public health? Experiences from reports to a Swedish consumer organization. Front Public Health. Sep 01, 2015;3:211. [FREE Full text] [CrossRef] [Medline]
    54. Robertson J, Newby DA. Low awareness of adverse drug reaction reporting systems: a consumer survey. Med J Aust. Nov 18, 2013;199(10):684-686. [CrossRef] [Medline]
    55. Choudhury A, Singh PA, Bajwa N, Dash S, Bisht P. Pharmacovigilance of herbal medicines: concerns and future prospects. J Ethnopharmacol. Jun 12, 2023;309:116383. [CrossRef] [Medline]

    ‎
    AE: adverse event
    BCPNN: Bayesian confidence propagation neural network
    HCP: health care professional
    HM: herbal medicine
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
    PRR: proportional reporting ratio
    PV: pharmacovigilance
    ROB: risk of bias
    UMC: Uppsala Monitoring Centre
    WHO: World Health Organization

    Edited by A Mavragani; submitted 01.07.24; peer-reviewed by TQ Vo, M Behzadifar; comments to author 17.07.24; revised version received 03.08.24; accepted 16.08.24; published 29.08.24.

    Copyright

    ©Chuenjid Kongkaew, Dang Thuc Anh Phan, Prathan Janusorn, Pajaree Mongkhon. Originally published in JMIR Public Health and Surveillance (https://publichealth.jmir.org), 29.08.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on https://publichealth.jmir.org, as well as this copyright and license information must be included.