We defined a set of common and rare ADRs to study for their association with two classes of drugs: an EGFR inhibitor, erlotinib, and the immune checkpoint programmed cell death 1 (PD-1) inhibitors, nivolumab and pembrolizumab. We focused on eight skin-related ADRs in this work: rash, acne, pruritus (ie, itchy skin), paronychia (ie, nail changes), xerosis (ie, dry skin), hypohidrosis, bullous eruption (ie, blister), and psoriasis.

The Inspire dataset consists of 7,320,546 discussion posts from 2005 to 2016. For each drug in the study, we generated a corpus defined as a collection of user posts from Inspire, with every post containing at least one mention of a keyword corresponding to one of the drugs of interest. To retrieve relevant posts from the Inspire dataset, we used regular expressions, a simple text-processing method for string matching. We created a regular expression pattern that implemented the exact string match. We have provided the list of the drugs and the related regular expression in Multimedia Appendix 1.

We identified 55,778 posts for erlotinib and 15,738 for the combined PD-1 inhibitors, nivolumab and pembrolizumab. Because nivolumab and pembrolizumab were more recently introduced and, therefore, have less discussion content but essentially identical mechanisms, we combined posts for the two drugs referencing the same ADR.

To extract ADR mentions from user posts, we used DeepHealthMiner (DHM), a neural network-based named entity recognition (NER) system that is specifically trained to extract drug safety-related entities from user-generated content in social media [29]. DHM is a supervised feed-forward neural network that is trained to identify two different entity types: ADRs and drug indications. The original, labeled training data is based on a dataset from DailyStrength [30], an online health community that contains patient-generated reports of treatment experiences very similar to those from Inspire. The labeled training data are the sentences from patient posts manually annotated for the entity span (ie, start and end position offsets) and the type of entity (ie, ADR or indication). More detailed information about the data and annotation details can be found in prior publications [31,32]. The unlabeled sentences from user posts are also used for unsupervised training of the word embeddings, which are used as vectors representing the input tokens to the NER system.

We retrained DHM using both labeled and unlabeled posts from Inspire. We retrained 150-dimensional word embeddings by adding the 7 million Inspire posts to the original unlabeled sentences. We manually labeled an additional 200 Inspire posts, following the original annotation guideline [31,32], to retrain DHM using Inspire content. Some of the rare ADRs in this study are more frequently discussed as indication mentions (eg, psoriasis or blistering) rather than ADRs in user postings; therefore, to improve our system’s ability to correctly recognize such cases, we created a subset of 200 Inspire posts from our corpus that particularly contained keywords from the rare seed ADRs. We annotated all the ADR and indication mentions following the original annotation guidelines to retrain DHM using Inspire content. The preliminary results showed that the system could successfully extract mentions of rare ADRs after retraining on 200 posts.

To evaluate the performance of the retrained NER model in identifying ADR entities from Inspire posts, we created a test set—the Inspire test set—consisting of 50 randomly selected posts. Two human annotators labeled all mentions of ADRs, not limited to the selected ADRs in this study, corresponding to any treatment reported in the posts (interannotator agreement: 90.9%). We used the F-measure to quantify the interannotator agreement by considering one annotator as the gold standard [33].

The extracted ADR mentions along with the information about the related drugs were stored in a database for further analyses of the associations among drugs and ADRs.

To map the extracted ADR mentions to Unified Medical Language System (UMLS) Concept Unique Identifiers (CUIs), we generated a lexicon of the eight ADR concepts. We defined a set of seed CUIs for these ADRs as shown in Multimedia Appendix 2. Using the UMLS hierarchy, we expanded every seed concept by adding synonyms and alternative names. For “nail changes,” we further expanded by including the children of the seed concepts (ie, is-a relation in the hierarchy). Every lexicon entry has a name, UMLS CUI, related seed ADR concept, and an identifier that we defined to group all items corresponding to a specific ADR. As an example, the lexicon entry “ingrown toenail (C0027343)” is an expanded concept based on “Disorder of nail (C0027339),” and “nail changes” is the human readable identifier for the ADR. Additionally, for every ADR, we added a set of colloquial phrases to the lexicon (see Multimedia Appendix 2). The list of colloquial phrases was generated by physicians, representing the list of words and phrases that they had seen patients use to describe their own cutaneous ADRs. We included these to better capture how patients describe ADRs in their own words.

Finally, we used Lucene [34] to index the lexicon. Lucene has been utilized as a successful tool in previous studies for normalizing biomedical entities [35,36]. We built a Lucene repository from the lexicon by creating one Lucene document for each lexicon entry, indexing the corresponding UMLS concept name and the CUI for each lexicon entry. The concept names are tokenized and lemmatized for generalization.

For mapping an extracted ADR mention to a UMLS concept, we searched the index using the text span extracted from the post by our NER system. We tokenized and lemmatized the extracted text span in the same manner as the indexed concept names and generated a Lucene query to search the index. For example, to map the extracted ADR mention “finger nails have been peeling” to a UMLS concept, we search the index using the query keywords “finger nail have be peel” and retrieve “peel nail (Peeling of nails: C0263531).” We retrieved a ranked list of the matched, relevant ADR concepts and chose the top-ranked concept from this list.

The co-occurrence of a drug and a reported ADR in a post can be considered as a potential association. To quantify the strength of an association between a drug and an ADR, we calculated the proportional reporting ratio (PRR), a statistic widely used for ADR signal generation from spontaneous reporting databases [26,37] and originally introduced by Evans et al [29]. The PRR quantifies ADR signals by comparing the frequency at which an ADR is reported with a drug of interest compared to the frequency at which the ADR is reported with other drugs (ie, comparison drugs) in the database.

We defined the comparison drugs by sorting drugs mentioned in Inspire by discussion frequency and selecting those most highly discussed in Inspire discussion posts (frequency >5000). To reduce noise, we excluded over-the-counter drugs with common indications (eg, pain and allergy drugs), as well as several anti-inflammatory drugs (eg, prednisone and Humira) that have indications that match some of our rare ADRs (eg, bullous eruption and psoriasis), given that patients discussing these drugs were almost exclusively discussing their indications rather than experienced ADRs. The final comparison drug list included 27 drugs, which are listed in Multimedia Appendix 3.

The PRR for an adverse reaction, R, and a drug, D, was calculated based on the following formula:

We defined count(D∩R) as the number of unique users that have reported both D and R in a post; count(D) as the total number of users reporting any ADR for drug D, including all extracted mentions by DHM (eg, weight gain or fatigue); count(!D∩R) as the total number of unique users that reported R for comparison drugs, excluding D; and count(!D) as the total number of unique users that reported at least one ADR for any drug except D.

We performed empirical calibration to find the threshold at which the PRR may represent a true ADR signal [38,39]. We considered the distribution of PRR scores for a set of drug-condition pairs with no known associations. To create this negative control set, we first paired all 27 comparison drugs and the eight target ADRs, totaling 216 drug-condition pairs. We then excluded the drug-condition pairs with known associations (ie, ADR or indication) based on the Medi-Span Adverse Drug Effects Database (Wolters Kluwer Health) and side-effect resource (SIDER) [40], leaving 86 pairs without any documented associations. Based on the PRR formula, we expected and validated that drug-ADR pairs with known and common associations have PRR values greater than 1 (see Figure 1). If a drug-ADR pair has a known association, then the corresponding PRR will be relatively higher than that for pairs with no associations. Our goal in excluding these known pairs from the negative control set was to illustrate the distribution of PRRs for drug-ADR pairs with no known associations and use only these to calibrate the threshold at which a PRR for an unknown drug-ADR pair would represent a significant, novel result.

It is important to note that a drug and a condition can be associated for reasons other than an ADR or indication and may, therefore, co-occur frequently within the negative control set. We identified two common reasons that a drug (drug x) and a condition (condition y) with no documented association co-occur frequently in user posts, therefore representing false positives. The first reason is related to prescribing patterns: if a drug or a set of similar drugs are commonly prescribed with drug x, while the reported ADR condition is not attributed to drug x, the ADR will appear more frequently with all coprescribed drugs. For example, mentions of nail changes happen frequently with zoledronic acid; however, while a true ADR is associated with capecitabine, which is frequently prescribed with zoledronic acid, zoledronic acid is not responsible for the ADR. The second reason is that if condition y is one of the syndromic conditions related to the indication for drug x, the related condition will co-occur with the drug at a similar rate to the true indication. For example, metformin can be prescribed for polycystic ovary syndrome (PCOS), and patients with PCOS often have acne. Acne is not an ADR associated with metformin but will co-occur frequently with mention of metformin by way of acne’s association with PCOS.

Because of the potential for observing such drug-condition associations, we manually reviewed the negative control set to exclude them, leaving 81 drug-ADR pairs with no association. The negative control set is available in Multimedia Appendix 4.

Figure 2 depicts our overall pipeline and graphical depiction of the above methods.

We identified 50,574 Inspire posts related to EGFR inhibitors and 16,598 related to checkpoint inhibitors; a total number of 13,600 ADR concepts extracted from the former and 812 concepts from the latter mapped to the eight target ADRs. We evaluated the performance of the presented NER method in identifying ADR mentions from the Inspire test set and achieved a microaveraged F-measure of 0.738 (0.731 precision and 0.745 recall).

To assess system performance in extraction and normalization of ADR concepts, we performed additional manual validation of the extracted ADR concepts as described in the Methods. Our final validation set for the entire pipeline for identifying drug-ADR pairs consisted of an additional 120 posts which had 15 randomly selected posts for every ADR that the system extracted for one of our target drugs.

We considered an extracted concept as a true positive if the system correctly detected the relevant span of text; classified the entity as an ADR rather than indication; and normalized the ADR to the correct concept. Benchmarking results show that our system achieved a high performance (microaverage precision of 0.90) in recognizing and normalizing the ADRs in the user posts.

The distribution of the frequencies and the calculated PRR of the skin ADRs extracted from the EGFR inhibitor-related posts correlates closely with published observed rates in patients (see Figure 1), further validating the pipeline for identifying ADRs. The PD-1 inhibitor ADR landscape is largely limited to case reports; forum frequencies and PRR values correlate with published rates, suggesting that those described to date accurately represent the emerging toxicity landscape (see Figure 1).

To determine the threshold at which PRR represents true signal warranting further investigation, we plotted the PRR distribution for the 81 drug-ADR pairs in the negative control set (see Figure 3). More than 95% of the pairs in this set have a PRR of less than 0.82; therefore, drug-ADR pairs with a PRR greater than 1 can be considered as signaling true ADRs.

To compare the timing of online ADR reports to initial ADR reports in the literature, we looked at both common and rare events associated with our target drugs. Papulopustular (ie, acneiform) rash and fingernail changes are well-reported ADRs associated with erlotinib and were first described in published case reports in September 2005 [41] and September 2006 [42], respectively. Inspire posts for these reactions appeared 5 and 3 months in advance of publication, respectively. Psoriasis in the setting of PD-1 inhibitor treatment was first documented in case reports in July 2015 [43] and May 2016 [44]. Inspire forum posts describing this ADR preceded the 2015 case report by 9 months (see Figure 4A). Blistering reactions with PD-1 inhibitors were initially published as a case report in June 2015 [45] and as a three-case series in May 2016 [46]. Forum descriptions preceded the first case report by 9 months (see Figure 4B). Taken together, our data suggest a significant and consistent reporting lead-time advantage in online health forum posts. Cutaneous ADRs in both EGFR and PD-1 inhibitors have been reported to be associated with cancer response to therapy [47-50], highlighting the clinical utility of early detection.

In early 2017, we documented three patients on erlotinib reporting hypohidrosis. Our system detected 23 unique Inspire users reporting hypohidrosis in causal association with erlotinib as early as 2006, based on manual review of user posts, with a PRR score of 1.90 (see Figure 3). This predicted ADR is absent from over 15 years of EGFR inhibitor literature, implicating a novel ADR detected through our system extractions from online posts. EGFR is expressed in sweat glands [51] and decreases with pharmacologic EGFR inhibition [52]. Interestingly, the hypohidrotic ectodermal dysplasia phenotype is partially mediated by decreased EGFR signaling [53], which can possibly cause hypohidrosis. These findings demonstrate the potential of ADR extraction from Internet forums to not only detect ADRs earlier, but also to contribute to novel ADR discovery.