Published on in Vol 7 , No 6 (2021) :June

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/30137, first published .
The Use of Social Media in Detecting Drug Safety–Related New Black Box Warnings, Labeling Changes, or Withdrawals: Scoping Review

The Use of Social Media in Detecting Drug Safety–Related New Black Box Warnings, Labeling Changes, or Withdrawals: Scoping Review

The Use of Social Media in Detecting Drug Safety–Related New Black Box Warnings, Labeling Changes, or Withdrawals: Scoping Review

Review

1College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea

2College of Pharmacy, Ewha Womans University, Seoul, Republic of Korea

3Graduate School of Pharmaceutical Sciences, Ewha Womans University, Seoul, Republic of Korea

*these authors contributed equally

Corresponding Author:

Myeong Gyu Kim, PhD

College of Pharmacy

Ewha Womans University

52, Ewhayeodae-gil, Seodaemun-gu

Seoul, 03760

Republic of Korea

Phone: 82 02 3277 3102

Email: kimmg@ewha.ac.kr


Background: Social media has become a new source for obtaining real-world data on adverse drug reactions. Many studies have investigated the use of social media to detect early signals of adverse drug reactions. However, the trustworthiness of signals derived from social media is questionable. To confirm this, a confirmatory study with a positive control (eg, new black box warnings, labeling changes, or withdrawals) is required.

Objective: This study aimed to evaluate the use of social media in detecting new black box warnings, labeling changes, or withdrawals in advance.

Methods: This scoping review adhered to the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews checklist. A researcher searched PubMed and EMBASE in January 2021. Original studies analyzing black box warnings, labeling changes, or withdrawals from social media were selected, and the results of the studies were summarized.

Results: A total of 14 studies were included in this scoping review. Most studies (8/14, 57.1%%) collected data from a single source, and 10 (71.4%) used specialized health care social networks and forums. The analytical methods used in these studies varied considerably. Three studies (21.4%) manually annotated posts, while 5 (35.7%) adopted machine learning algorithms. Nine studies (64.2%) concluded that social media could detect signals 3 months to 9 years before action from regulatory authorities. Most of these studies (8/9, 88.9%) were conducted on specialized health care social networks and forums. On the contrary, 5 (35.7%) studies yielded modest or negative results. Of these, 2 (40%) used generic social networking sites, 2 (40%) used specialized health care networks and forums, and 1 (20%) used both generic social networking sites and specialized health care social networks and forums. The most recently published study recommends not using social media for pharmacovigilance. Several challenges remain in using social media for pharmacovigilance regarding coverage, data quality, and analytic processing.

Conclusions: Social media, along with conventional pharmacovigilance measures, can be used to detect signals associated with new black box warnings, labeling changes, or withdrawals. Several challenges remain; however, social media will be useful for signal detection of frequently mentioned drugs in specialized health care social networks and forums. Further studies are required to advance natural language processing and mine real-world data on social media.

JMIR Public Health Surveill 2021;7(6):e30137

doi:10.2196/30137

Keywords



Clinical trials are a primary measure of the efficacy and safety of drugs before they are marketed. However, the limited number of subjects and study period makes it difficult to detect rare adverse drug reactions (ADRs) [1]. For example, 10,000 subjects are required to detect a very rare ADR (with a frequency of <1 in 10,000 individuals). More subjects are needed to obtain significant results. Furthermore, the controlled environment of clinical trials does not fully reflect the effects of different ages, comorbidities, and drug-drug interactions on ADRs in the real world. Hence, it is important to investigate the real-world data (RWD) of marketed drugs for mining ADR signals. Pharmacovigilance is defined as the science of detecting, assessing, understanding, and preventing drug-related adverse effects or problems [2]. Postmarketing surveillance studies and mining of spontaneous adverse event reporting systems are the 2 principal methods of pharmacovigilance.

In recent years, social media has become an essential part of everyday life. Social media platforms (eg, Facebook, Twitter, and patient forums) are where people share experiences and opinions. Patients use social media to increase their health knowledge and exchange advice and information [3]. Social media has become a new source of RWD on ADRs. Researchers expect social media to identify signals that conventional pharmacovigilance methods (eg, postmarketing surveillance studies and spontaneous adverse event reporting systems) have not identified or to identify signals earlier than conventional methods. In addition, social media can be monitored in real time, and adverse events caused by off-label use, unknown in clinical trials, can be detected.

Many studies have investigated the detection of early ADR signals on social media. Tricco et al [4] conducted a scoping review, which included 77 pharmacovigilance studies that used social media, from 2001 to 2016. Another review by Pappa and Stergioulas [5] qualitatively analyzed 100 pharmacovigilance studies that used social media, from 2007 to 2018. Both these reviews summarized the social media platforms used in research and data analysis methods, and the advantages and challenges of pharmacovigilance using social media. Compared to conventional pharmacovigilance methods, pharmacovigilance studies using social media are still in their infancy [4,5]. However, social media has the potential to complement existing pharmacovigilance systems [6].

The reliability of signals derived from social media is questionable [7]. Positive controls are essential for ensuring signal accuracy and verifying the usefulness of social media for mining previously unknown ADRs. Considering the purpose of pharmacovigilance, new black box warnings, labeling changes, and withdrawals are appropriate as positive controls, rather than well-known ADRs already on the drug label. Patients usually post mild and common ADRs (eg, pain or fatigue) on social media platforms; hence, testing whether mining social media data can detect significant ADRs (appearing as new black box warnings, labeling changes, or withdrawals) is necessary [8].

This scoping review aimed to determine how well social media helps detect new black box warnings, labeling changes, or withdrawals in advance.


Search Strategy

This study followed a preplanned protocol and adhered to the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews checklist [9]. Two biomedical databases, PubMed and EMBASE, were searched to identify relevant literature. Search terms (Multimedia Appendix 1) consisted of 15 popular social networks worldwide and sites mentioned in previous scoping reviews [4,10]. There were no restrictions on publication language and publication year. The search was conducted on January 7, 2021. We searched the reference lists of relevant literature (eg, recent reviews and selected articles in this scoping review) for inclusion. Additional Google Scholar searches have been made to include grey literature. EndNote X9 (Clarivate Analytics) was used to manage the literature.

Study Selection

Two authors independently evaluated the eligibility of the studies. Articles were included if they focused on the analysis of drug safety–related new black box warnings, labeling changes, or withdrawals from social media data. Studies were excluded if they did not study new black box warnings, labeling changes, or drug withdrawals, did not use social media data (eg, data of the US Food and Drug Administration [FDA] Adverse Event Reporting System [FAERS]), or were not original articles (eg, reviews, protocols, and conference abstracts). If the same data were assessed using the same analytic method, an article that could encompass other studies was selected.

Owing to the nature of computer science research, conference proceedings that contained data were included as an exception. Unlike most other academic disciplines, computer science often considers conference proceedings as a last-resort means of reporting research findings [11]. Conference proceedings are typically between 4000 and 7000 words in length, similar to journal articles, which allows sufficient details of the study to be reported [11].

Study selection proceeded in 2 stages. First, titles and abstracts were screened for eligibility. Second, full texts were reviewed to finalize the articles for analysis. Any discrepancy between the 2 authors was resolved through discussion.

Data Extraction

Data were extracted by an author and recorded in a preprepared data extraction table. The extracted data were reviewed by other authors, and the data extraction form was continuously modified. The extracted data included article characteristics (eg, author and publication year), target social media platforms, study drugs, analytical methods, main results, and limitations mentioned by the authors.

Results Synthesis

The study results were aggregated or summarized during qualitative synthesis. Additionally, descriptive statistical analysis was conducted for the frequency and proportion of the social media platforms and analytical methods used in the studies. Social media platforms were categorized in accordance with the classification of Pappa and Stergioulas [5]: generic social networking sites (SNS) and specialized health care social networks and forums. Generic SNS include Facebook and Twitter, and specialized health care social networks and forums include generic health-centered SNS (eg, PatientsLikeMe, DailyStrength, MedHelp, and WebMD), medicine-focused sharing platforms (eg, Ask a Patient and Medications.com), and disease-specific web-based health forums.


Characteristics of Selected Studies

Figure 1 shows the study selection process. We identified 75 articles from PubMed and 169 from EMBASE. Google Scholar and bibliographic searches yielded 87 and 15 studies, respectively. Through the deduplication and 2-step selection process, 14 studies were finally included in this scoping review [12-25]. Table 1 provides the details of the included studies and the social media sources used in each study.

Figure 1. Flowchart of the protocol of the scoping review.
View this figure
Table 1. Characteristics of social media sources.
NumberAuthor (year)Social media platformsDurationStudy drugsNumber of posts
1Caster et al (2018) [12]Twitter, Facebook, and 407 patient forumsMarch 2012-March 201575 drugs (Harpaz et al [26] reference set and WEB-RADR reference set)6,279,424 posts from Twitter and Facebook (690,492 posts with an indicator score threshold of 0.4); 42,721 posts from 407 patient forums using with an indicator score threshold of 0.7
2Pierce et al (2017) [13]Facebook and TwitterMarch 2009-October 201410 drugs935,246 posts (704,283 nonspam posts)
3Duh et al (2016) [14]Ask a Patient2001-2014Sibutramine and atorvastatin270 posts on sibutramine and 998 posts on atorvastatin
4aYang et al (2015) [15]MedHelp1997-201120 drugs with >500 threads for each 16,344 posts (8053 posts on 10 drugs that were on alert or had a revised label)
5aYang et al (2015) [16]MedHelp1997-201120 drugs with >500 threads for each16,344 posts
6Feldman et al (2015) [17]MedHelp, exchanges.webmd.com, HealthBoards, and ehealthforum.com1999-2013Cholesterol-lowering drugs and antidepressants41,086 posts for cholesterol-lowering drugs and 273,990 for antidepressants
7Coloma et al (2015) [18]Facebook, Google+, and TwitterUntil September 2014Rosiglitazone2537 posts related to rosiglitazone and cardiovascular events
8Patki et al (2014) [19]DailyStrengthb20 normal and 18 black box drugsc20,486 posts (normal: 10,399, black box: 7327, withdrawn: 2760)
9Abou Tamm et al (2014) [20]Three French websites (Doctissimo, Atoute.org, and Vivelesrondes)Benfluorex220 initial posts and 660 secondary posts
10Adjeroh et al (2014) [21]Twitter and general web search queries2008-201246 drugs that had a Food and Drug Administration alert2 million posts on Twitter
11Wu et al (2013) [22]Online discussions using forum search engines such as Google Discussion Search2000-20114 drugs178,871 posts
12Liu et al (2013) [23]Diabetes patient forumFebruary 2009-November 2012Antidiabetic drugs185,874 posts
13Chee et al (2011) [24]Yahoo GroupsNot prespecifiedNot mentionedd
14Chee et al (2009) [25]Yahoo GroupsNatalizumab, rofecoxib, and celecoxib20,000 posts on natalizumab and 867,659 posts on rofecoxib and celecoxib

aSame data but different analytical methods were used.

b—: data not available.

cThe number of drugs withdrawn was not mentioned.

dThere is a total of 12,519,807 messages in 27,290 public Health & Wellness Yahoo! groups.

Figure 2 shows the number and type of social media sources. Most studies (n=8 of 14, 57.1%) collected data from a single source. Meanwhile, Caster et al [12] collected posts from Twitter, Facebook, and 407 patient forums. Specialized health care social networks and forums were used in 10 (71.4%) studies, and 3 (21.4%) studies used generic SNS. MedHelp (n=4 of 14, 28.6%), Twitter (n=4 of 14, 28.6%), Facebook (n=3 of 14, 21.4%), and Google (n=3 of 14, 21.4%) were frequently used sites.

Figure 2. Number and type of social media sources.
View this figure

The number of posts collected varied from hundreds to tens of millions according to the type of sources and study drugs (Table 1). Studies using generic SNS platforms collected a relatively large number of posts compared to those using specialized health care social networks and forums. However, the large volume of posts collected on generic SNS are noisy and difficult to process [5]. For example, Pierce et al [13] sampled 935,246 posts that named 10 selected drugs from Facebook and Twitter over 5.5 years. Among them, 98,252 posts resembled adverse events (Proto-AEs), and only 6 posts described certain, probable, and possible cases of interest [13]. Yang et al [15] collected 8053 posts that named 10 drugs over 4 years on MedHelp.

Methods Used for Detecting New Black Box Warnings, Labeling Changes, or Withdrawals

The analytical methods used in the studies varied considerably. Figure 3 indicates the analytical methods used in each study (Multimedia Appendix 1). Three (21.4%) studies conducted by Duh et al [14], Coloma et al [18], and Abou Taam et al [20] manually annotated posts for the presence or absence of selected adverse events. After manual annotation, a time-series analysis or frequency analysis was conducted. This analysis was possible because the number of posts used in these 3 studies ranged from hundreds to thousands. However, this analytical method is difficult to implement with a higher number of posts.

Figure 3. Analytical methods in the studies included in this scoping review.
View this figure

Five (35.7%) studies adopted machine learning algorithms for detecting ADRs from social media. Feldman et al [17] utilized an unsupervised relation extraction for mining drug-ADR relations. Patki et al [19] used machine learning algorithms for the binary classification of social media posts as ADR or non-ADR. Liu et al [23] used a kernel-based learning method to extract adverse events in patient forums and then used semisupervised learning algorithms to classify report sources into patients’ experiences or not. Chee et al [24] used machine learning algorithms to classify drugs as watchlist and nonwatchlist drugs. Pierce et al [13] used automated classifiers that identified Proto-AEs and then manually reviewed the cases and assessed causality.

Caster et al [12] conducted a disproportionality analysis to detect signals and compared the area under the receiver operating characteristic curve with a World Health Organization global pharmacovigilance database (VigiBase database). Yang et al [15], Yang et al [16], and Feldman et al [17] computed a lift measure for association rule mining to evaluate the likelihood of a particular drug-ADR relationship. Other methods include a tensor-based technique [16], discriminative classification and generative modeling [22], sentiment analysis [25], and peak-labeling signal fusion [21].

Detection Performance

Nine (64.2%) studies reported that social media monitoring allowed for the detection of black box warnings, labeling changes, or withdrawals 3 months to 9 years in advance [14-17,20-24]. Most (88.9%) of these studies were conducted on specialized health care social networks and forums (Figure 4). One post that mentioned the occurrence of benfluorex (Mediator)-cardiac valvulopathy in a woman was posted in a patient forum 7 months before withdrawal [20]. Chee et al [24] identified 4 drugs (hydromorphone [proprietary name: Palladone], cerivastatin [Baycol], trovafloxacin [Trovan], and rofecoxib [Vioxx]) that were withdrawn by analyzing Yahoo Groups. Furthermore, posts related to sibutramine, which was not a watchlist drug at the time of the study but was later withdrawn owing to the risk of cardiac arrest and stroke, were observed more than a year ago [24]. Sibutramine was the fifth-highest-risk drug in the study [24]. In Duh et al’s [14] study on sibutramine, social media mentions of sibutramine-related cardiac issues helped predict those in FAERS 11 months later. In a study by Adjeroh et al [21], 30 of 46 (65.2%) drugs were detectable prior to FDA alert. Depending on the drug, adverse events could be detected 3 months (fluvastatin [Lescol]) to 35 months (codeine and atorvastatin [Lipitor]) before the FDA alert [21]. Wu et al [22] reported that the discussion frequency of arrhythmia due to propoxyphene/acetaminophen (Darvocet obviously exceeded the threshold since 2006, 4 years before the drug’s recall. Adverse events of 3 other drugs, namely simvastatin (Zocor) drospirenone/ethinyl estradiol (Yaz and Yasmin), were also detected 4-6 years prior to FDA action [22]. Yang et al [15] and Yang et al [16] analyzed the same data by association rule mining and the tensor-based technique. Association rule mining detected 6 of 14 ADRs, which included fluoxetine (Prozac)–induced suicidal thoughts (1 year in advance), methylphenidate (Concerta)–induced blurred vision (2 years in advance), fluoxetine (Prozac)–induced depression (5 years in advance), fluvoxamine (Luvox)–induced suicidal thoughts (5 years in advance), simvastatin–induced kidney disease (6 years in advance), and lansoprazole-induced diarrhea (13 years in advance) [15]. The tensor-based technique allowed more ADRs (11 of 14) to be detected 1-7 years earlier than association rule mining [16]. Feldman et al [17] reported that unsupervised relation extraction from web-based forums identified statin-induced cognitive impairment (labeling changes in 2012) as early as 9 years prior to the FDA label change. In the same study, Feldman et al [17] identified a significant relationship between bupropion (Wellbutrin) and agitation 7 years before FDA action. AZDrugMiner, developed by Liu et al [23], could detect rosiglitazone (Avandia)–induced myocardial infarction (18% of adverse events) and cardiac disorder (13% of adverse events) at high frequency from patient forums.

Figure 4. Perspectives on pharmacovigilance using social media and sources of social media.
View this figure

Five (35.7%) studies reported modest or negative results. Of these, 2 (40%) used generic SNS, and 1 (20%) used generic SNS and specialized health care social networks and forums (Figure 4). Caster et al [12] reported that signal detection in social media performed poorly and was not recommended. The area under the receiver operating characteristic curve of social media varied between 0.47 and 0.53 for the reference sets, while that of VigiBase ranged between 0.64 and 0.69 [12]. On Twitter and Facebook, only 16% and 6% of positive controls were detected before their index dates in the WEB-RADR and Harpaz references, respectively [12]. In a study by Pierce et al [13], proto-AEs were observed for only 2 of 10 drugs studied. Dronedarone-induced vasculitis reported by the FDA in April 2012 was indicated on social media in December 2010. However, as FAERS received its first report in July 2010, social media was 5 months late [13]. Only 1 positive signal was identified on social media before FAERS. The correlation between Banana Boat sunscreen and skin burns was observed on social media on June 2, 2012, and the FAERS first report appeared 17 days later [13]. Coloma et al [18] reported that the number of posts on rosiglitazone-induced cardiovascular events tended to increase with regulatory action. However, only 10 of 2537 posts described personal experiences of rosiglitazone-induced cardiovascular events. Patki et al [19] classified posts with or without ADR and classified the drugs as black box or normal drugs on the basis of the number of ADR-related posts. The classifier for black box drugs revealed a modest F-score of 0.6 [19]. Levofloxacin (Levaquin) and baclofen were misclassified as normal drugs because the number of posts was small, and the posts mainly referred to usefulness and not ADRs [19].

Challenges

Table 2 categorizes the study limitations. There are currently several challenges associated with the use of social media for pharmacovigilance in terms of social media coverage, data quality, and analytic processing.

Table 2. Challenges associated with the use of social media for pharmacovigilance.
ChallengesReferences
Coverage

Limited social media coverage (generalizability to other data sources)[12-14,18]

Lack of user population representation[18]

Not a balanced coverage of all drugs and medical conditions[14,18,24]

Limited coverage period[12,14]
Data quality

Use of colloquial language: misspellings or use of nonmedical terms and slang[13,15,16]

Data duplication (double-counting)[13,14]

Lack of medical and demographic information[14]

Lack of causality information[25]

Nonvalidated or incomplete data or misinformation[14]

Low signal-to-noise ratio[12,14,18,19,25]
Processing

Curation burden due to data volume[13]

Word-level analysis or does not reach semantic or discourse levels[15,18,22-24]

Different types of social media platforms have different user characteristics and types of data. Since studies have been conducted on limited social media platforms, analytical methods that were applicable to one platform may not be suitable on another platform. Typically, younger individuals use social media; therefore, adverse events that occur mainly in older people or those associated with drugs used for geriatric diseases can be underestimated. Data obtained from patient forums tend to be biased toward specific patients or drugs.

The low signal-to-noise ratio of the data makes the preprocess and analytical process burdensome. Social media, of course, is noisier than databases that collect only adverse drug events. In particular, the number of posts on generic social networking sites that are unrelated to adverse events is much higher than that of specialized health care social networks and forums.

There are extensive social media data to classify manually, but an understanding of natural language has not yet reached human levels. Nonetheless, many studies analyzed whether drug and adverse event–related terms are in the same sentence or post.


Principal Findings

As interest in the use of social media in pharmacovigilance has increased, there have been steadily review papers on pharmacovigilance using social media. Most of the reviews focused on the data sources or analytical methods used in individual studies. Positive controls have allowed us to assess whether social media can detect significant unknown signals. A recent review [27] summarized studies that determined whether signals can be detected through social media before the actions of regulatory authorities. In this scoping review, we reviewed 14 studies on the application of social media in detecting new black box warnings, labeling changes, or withdrawals. Some studies in our review are not included in other previous reviews such as the one by Caster et al, who expressed a negative opinion. Our scoping review has the advantage of being the most state-of-the-art review that analyzed detection performance (eg, time interval between signals detected on social media and regulatory authority action).

Most studies have reported that meaningful signals could be detected before regulatory authorities take action. Signals were identified 3 months to 9 years in advance on social media, depending on social media sources and drug-ADR pairs. Most studies that have reported the usefulness of social media have used specialized health care social networks and forums. These sources have more patient experiences while fewer unrelated posts, which have helped overcome the disadvantages of low signal-to-noise ratios. Therefore, specialized health care social networks and forums are suitable sources for pharmacovigilance using social media. Furthermore, drugs such as benfluorex, sibutramine, drospirenone/ethinyl estradiol, methylphenidate, and fluoxetine, which may interest social media users, have yielded favorable outcomes.

A recent study by Caster et al [12], which used a large number of social media platforms, revealed a poor performance. Study drugs were not actively included on social media in their study. The detection performance of social media varies depending on the study drug, and social media–based pharmacovigilance will be useful for restrictive drugs. Thus, an approach to select drugs to monitor adverse events with social media will need to be developed. Differences in reference sets and statistical analysis also explain why Caster et al’s findings differed from those of other studies. The study used the WEB-RADR reference (independent of regulatory action) as well as the Harpaz reference (based on FDA labeling changes). Disproportionality analysis, which is commonly used for signal detection in spontaneous adverse event–reporting systems, was used instead of other methods tailored to the analysis of social media. The results of a pilot study conducted in 2018 are in line with those of the study by Caster et al [12]. Studies have reported 6 proto-AEs on Twitter and patient forums that meet the criteria for disproportionality [7]. Five of 6 selected adverse events were observed a median of 252 (range 144-367) days later on social media [7]. Negative views in recent studies have made the application of social media hesitant. We eagerly anticipate the results of the Adverse Drug Reactions from Patient Reports in Social Media study, which is currently underway in France [28].

Several challenges are associated with the use of social media for pharmacovigilance. Studies have reported a coverage problem, which makes their findings difficult to generalize. Therefore, pharmacovigilance cannot be entirely dependent on social media, but rather social media can be used to supplement existing pharmacovigilance measures. The challenges of low signal-to-noise ratios and word-level analysis might be resolved by enhancing natural language processing (NLP) and machine learning algorithms. While there have been recent rapid advances, computers still cannot fully understand human language at a semantic or discourse level. NLP techniques will need further improvements to meaningfully analyze the vast amount of RWD in social media.

Limitations

This study has several limitations. First, while there was no restriction on the language of the papers retrieved from PubMed and EMBASE and references to related papers were searched, there may still be studies that we missed. Second, social media platforms that were widely used or derived from previous reviews were used as search terms. Therefore, we may have possibly missed information on minor social media platforms. Lastly, this is a rapidly developing field. Although we updated our search before submission, new studies may be published as of this publication. In particular, the Adverse Drug Reactions from Patient Reports in Social Media study is ongoing; hence, our results will need to be updated.

Conclusions

Social media, along with conventional pharmacovigilance measures, can be used to detect signals associated with new black box warnings, labeling changes, or withdrawals. Several challenges remain; however, social media will be useful for signal detection for frequently mentioned drugs in specialized health care social networks and forums. Further studies are required to advance NLP and mine RWD on social media.

Acknowledgments

This study was supported by a research fund from Chungnam National University.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Supplementary Tables. Table 1: Search terms. Table 2: Analytical methods.

DOCX File , 22 KB

References

  1. Singh S, Loke YK. Drug safety assessment in clinical trials: methodological challenges and opportunities. Trials 2012 Aug 20;13:138 [FREE Full text] [CrossRef] [Medline]
  2. Pharmacovigilance. World Health Organization.   URL: https://www.paho.org/en/topics/pharmacovigilance [accessed 2021-03-10]
  3. McDonald L, Malcolm B, Ramagopalan S, Syrad H. Real-world data and the patient perspective: the PROmise of social media? BMC Med 2019 Jan 16;17(1):11 [FREE Full text] [CrossRef] [Medline]
  4. Tricco AC, Zarin W, Lillie E, Jeblee S, Warren R, Khan PA, et al. Utility of social media and crowd-intelligence data for pharmacovigilance: a scoping review. BMC Med Inform Decis Mak 2018 Jun 14;18(1):38 [FREE Full text] [CrossRef] [Medline]
  5. Pappa D, Stergioulas LK. Harnessing social media data for pharmacovigilance: a review of current state of the art, challenges and future directions. Int J Data Sci Anal 2019 Feb 12;8(2):113-135 [FREE Full text] [CrossRef]
  6. Sarker A, Ginn R, Nikfarjam A, O'Connor K, Smith K, Jayaraman S, et al. Utilizing social media data for pharmacovigilance: A review. J Biomed Inform 2015 Apr;54:202-212 [FREE Full text] [CrossRef] [Medline]
  7. Rees S, Mian S, Grabowski N. Using social media in safety signal management: is it reliable? Ther Adv Drug Saf 2018 Oct;9(10):591-599 [FREE Full text] [CrossRef] [Medline]
  8. Smith K, Golder S, Sarker A, Loke Y, O'Connor K, Gonzalez-Hernandez G. Methods to Compare Adverse Events in Twitter to FAERS, Drug Information Databases, and Systematic Reviews: Proof of Concept with Adalimumab. Drug Saf 2018 Dec;41(12):1397-1410 [FREE Full text] [CrossRef] [Medline]
  9. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med 2018 Oct 02;169(7):467-473 [FREE Full text] [CrossRef] [Medline]
  10. Most popular social networks worldwide as of July 2020, ranked by number of active users(in millions). Statista.   URL: https://www.statista.com/statistics/272014/global-social-networks-ranked-by-number-of-users/ [accessed 2021-10-09]
  11. Vrettas G, Sanderson M. Conferences versus journals in computer science. J Assn Inf Sci Tec 2015 Apr 27;66(12):2674-2684. [CrossRef]
  12. Caster O, Dietrich J, Kürzinger ML, Lerch M, Maskell S, Norén GN, et al. Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project. Drug Saf 2018 Dec;41(12):1355-1369 [FREE Full text] [CrossRef] [Medline]
  13. Pierce CE, Bouri K, Pamer C, Proestel S, Rodriguez HW, Van Le H, et al. Evaluation of Facebook and Twitter Monitoring to Detect Safety Signals for Medical Products: An Analysis of Recent FDA Safety Alerts. Drug Saf 2017 Apr;40(4):317-331 [FREE Full text] [CrossRef] [Medline]
  14. Duh MS, Cremieux P, Audenrode MV, Vekeman F, Karner P, Zhang H, et al. Can social media data lead to earlier detection of drug-related adverse events? Pharmacoepidemiol Drug Saf 2016 Dec;25(12):1425-1433 [FREE Full text] [CrossRef] [Medline]
  15. Yang H, Yang CC. Using Health-Consumer-Contributed Data to Detect Adverse Drug Reactions by Association Mining with Temporal Analysis. ACM Trans Intell Syst Technol 2015 Aug 13;6(4):1-27. [CrossRef]
  16. Yang C, Yang H. Exploiting Social Media with Tensor Decomposition for Pharmacovigilance. 2015 Presented at: 2015 IEEE International Conference on Data Mining Workshop; November 14-17, 2015; Atlantic City, NJ. [CrossRef]
  17. Feldman R, Netzer O, Peretz A, Rosenfeld B. Utilizing Text Mining on Online Medical Forums to Predict Label Change due to Adverse Drug Reactions. 2015 Presented at: 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; August 10-13, 2015; Sydney, NSW. [CrossRef]
  18. Coloma PM, Becker B, Sturkenboom MCJM, van Mulligen EM, Kors JA. Evaluating Social Media Networks in Medicines Safety Surveillance: Two Case Studies. Drug Saf 2015 Oct;38(10):921-930 [FREE Full text] [CrossRef] [Medline]
  19. Patki A, Sarker A, Pimpalkhute P, Nikfarjam A, Ginn R, O'Connor K, et al. Mining adverse drug reaction signals from social media: going beyond extraction. Proceedings of BioLinkSig 2014:1-8.
  20. Abou Taam M, Rossard C, Cantaloube L, Bouscaren N, Roche G, Pochard L, et al. Analysis of patients' narratives posted on social media websites on benfluorex's (Mediator® ) withdrawal in France. J Clin Pharm Ther 2014 Feb;39(1):53-55. [CrossRef] [Medline]
  21. Adjeroh D, Beal R, Abbasi A, Zheng W, Abate M, Ross A. Signal Fusion for Social Media Analysis of Adverse Drug Events. IEEE Intell Syst 2014;29(2):74-80.
  22. Wu H, Fang H, Stanhope SJ. Exploiting online discussions to discover unrecognized drug side effects. Methods Inf Med 2013;52(2):152-159. [CrossRef] [Medline]
  23. Liu X, Chen H. AZDrugMiner: An Information Extraction System for Mining Patient-Reported Adverse Drug Events in Online Patient Forums. 2013 Presented at: ICSH 2013: International Conference on Smart Health; August 3-4, 2013; Beijing. [CrossRef]
  24. Chee BW, Berlin R, Schatz B. Predicting adverse drug events from personal health messages. AMIA Annu Symp Proc 2011;2011:217-226 [FREE Full text] [Medline]
  25. Chee B, Berlin R, Schatz B. Measuring population health using personal health messages. AMIA Annu Symp Proc 2009 Nov 14;2009:92-96 [FREE Full text] [Medline]
  26. Harpaz R, Odgers D, Gaskin G, DuMouchel W, Winnenburg R, Bodenreider O, et al. A time-indexed reference standard of adverse drug reactions. Sci Data 2014 Nov 11;1:140043 [FREE Full text] [CrossRef] [Medline]
  27. Convertino I, Ferraro S, Blandizzi C, Tuccori M. The usefulness of listening social media for pharmacovigilance purposes: a systematic review. Expert Opin Drug Saf 2018 Nov;17(11):1081-1093. [CrossRef] [Medline]
  28. Arnoux-Guenegou A, Girardeau Y, Chen X, Deldossi M, Aboukhamis R, Faviez C, et al. The Adverse Drug Reactions From Patient Reports in Social Media Project: Protocol for an Evaluation Against a Gold Standard. JMIR Res Protoc 2019 May 07;8(5):e11448 [FREE Full text] [CrossRef] [Medline]


ADR: adverse drug reaction
FAERS: FDA Adverse Event Reporting System
FDA: Food and Drug Administration
NLP: natural language processing
Proto-AEs: posts that resembled adverse events
RWD: real-world data
SNS: social networking site


Edited by G Eysenbach; submitted 03.05.21; peer-reviewed by D Huang, S Matsuda; comments to author 10.05.21; revised version received 22.05.21; accepted 30.05.21; published 28.06.21

Copyright

©Jae-Young Lee, Yae-Seul Lee, Dong Hyun Kim, Han Sol Lee, Bo Ram Yang, Myeong Gyu Kim. Originally published in JMIR Public Health and Surveillance (https://publichealth.jmir.org), 28.06.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on https://publichealth.jmir.org, as well as this copyright and license information must be included.