On December 31, 2019, the World Health Organization (WHO) was made aware of a cluster of cases of viral pneumonia of unknown origin in Wuhan, Hubei Province, China [1]. The WHO reported this cluster via Twitter on January 4, 2020, saying, “#China has reported to WHO a cluster of #pneumonia cases —with no deaths— in Wuhan, Hubei Province. Investigations are underway to identify the cause of this illness [2].” On January 19, the WHO Western Pacific Regional Office tweeted evidence of human-to-human transmission, saying, “According to the latest information received and @WHO analysis, there is evidence of limited human-to-human transmission of #nCOV. This is in line with experience with other respiratory illnesses and in particular with other coronavirus outbreaks [3].” The first case in the United States was reported the next day. Five days later, on January 26, 2020, GreatGameIndia published the article “Coronavirus Bioweapon–How China Stole Coronavirus From Canada And Weaponized It,” which claimed that the coronavirus was leaked into China from a Canadian laboratory [4]. The original article received 1600 likes on its first day of publication; it was then reposted verbatim but with the more provocative headline “Did China Steal Coronavirus From Canada And Weaponize It” on the website ZeroHedge [5]. This version was reposted by the website RedStateWatcher.com, one of the 140 most popular sites in the United States, with more than 4 million followers on Facebook; from there, the story quickly went viral [6].

Misinformation surrounding pandemics is not unique to SARS-CoV-2, the virus that causes COVID-19. At least as far back as the Russian flu pandemic of 1889, pandemic spread of misinformation, claims of fact that are either demonstrably false or unverifiable [7], has been concomitant with disease spread [8]. People are susceptible to misinformation when trust in authoritative sources is low, which can occur when officials provide conflicting information and guidance [9]. Misinformation will also include conspiracy theories, which posit explanations of events or circumstances based primarily on a conspiracy [10] (ie, an agreement between a small group of people to commit an illegal act). Although some conspiracies, such as Watergate or the Tuskegee experiments, may eventually be proven to be true criminal acts, the vast majority of conspiracy theories are not true, and their spread can undermine public health efforts [11]. Some conspiracy theories may be better classified as disinformation—false or misleading information that is intentionally passed to a target group [12] with its true source concealed [13].

The COVID-19 outbreak has left many people isolated within their homes, and these people are turning to social media for news and social connection. Thus, they are especially vulnerable to believing and sharing conspiracy theories [14]. This study examines four oft-repeated and long-lived conspiracy theories surrounding COVID-19: 5G technology is somehow associated with the disease; Bill Gates or the Bill & Melinda Gates Foundation created or patented the virus; the virus is human-made and was released from a laboratory; and a COVID-19 vaccine will be harmful. None of these conspiracy theories are unique, nor are they entirely distinct.

Cellular carriers began a limited rollout of 5G cellular service in 2018 [15], which required the installation of new cell towers [16]. These new towers were already the source of a more general conspiracy theory that the signal is harmful to humans and that its dangers were being “covered up” by “powerful forces in the telecommunications industry” [17]. Wireless technology has consistently been blamed for causing immune damage in humans, and similar theories were seen with the rollouts of 2G, 3G, 4G, and Wi-Fi service [17]. Even the 1889 Russian flu was purported to be caused by the then-new technology of electric light [8]. The COVID-19–related 5G conspiracy theory emerged in the first week of January, and it may not have evolved past a fringe view into a trending hashtag without being shared by websites with the primary aim of spreading conspiracy theories on Twitter or by people aiming to denounce the theory [18].

J Uscinski stated that conspiracy theories often “are about accusing powerful people of doing terrible things” [19]. The Bill & Melinda Gates Foundation is arguably the largest philanthropic venture ever attempted, and it has proven to be fertile ground for the development of conspiracy theories, ranging from misinterpretations of a “patent on COVID-19” [20] to incorporation of vaccine-averse concerns. For example, the Bill & Melinda Gates Foundation funded research to develop injectable invisible ink to serve as a permanent record of vaccination in developing countries [21,22]. This technology was announced in December 2019, the same month that SARS-CoV-2 emerged in Wuhan, China, and a conspiracy theory emerged suggesting that the COVID-19 vaccine would be used to microchip individuals with the goal of population control [20].

Associations between HIV and other infectious diseases consistently re-emerge, including associations with polio [23], Ebola virus [24], and COVID-19. The COVID-19–related HIV conspiracy theory began on January 31, 2020, with the preprint publication of “Uncanny similarity of unique inserts in the 2019-nCoV spike protein to HIV-1 gp120 and Gag” ([25], withdrawn paper), which was quickly retweeted by Anand Ranganathan, a molecular biologist with over 200,000 followers on Twitter. He cited the preprint as evidence of a potential laboratory origin with a now-deleted Tweet: “Oh my god. Indian scientists have just found HIV (AIDS) virus-like insertions in the 2019-nCoV virus that are not found in any other coronavirus. They hint at the possibility that this Chinese virus was designed…” Within two hours, Ross Douthat, a prominent New York Times opinion columnist, retweeted Ranganathan to his >140,000 followers, further legitimizing the theory through a reputable news outlet and greatly furthering the reach of the story outside the scientific community [26]. Three days after the initial release of the preprint, the original paper was retracted.

Laboratory origin theories have also garnered political attention; then-US President Donald Trump claimed to have evidence of a Chinese laboratory origin of SARS-CoV-2 [27], prompting a Twitter response from a Chinese government account [28] that was flagged by Twitter as misinformation [29]. Additional laboratory-related conspiracy theories quickly emerged, including theories that the virus was created to achieve global population reduction or to impose quarantines, travel bans, and martial law, all of which were previously seen during the 2014 Ebola virus outbreak [24] and the 2015-2016 Zika virus outbreak [30].

Vaccine-related social media articles are often shared by people who are relatively knowledge-deficient and vaccine-averse compared to nonsharers [31], with content consisting of debunked associations with autism and general mistrust of government or the pharmaceutical industry. With newly emergent diseases such as HIV and Ebola, conspiracy theories quickly followed regarding the ability to profit off of vaccines while conspiring with American pharmaceutical companies [24].

In the past year, substantial work has emerged investigating the onslaught of misinformation related to COVID-19. Multiple studies have found that misinformation is common; both social media platforms [32-34] and web pages returned results for common COVID-19 queries at the beginning of the pandemic [35], including scientific journals without sufficiently rigorous review processes [36].

Social media studies have so far indicated that original tweets present false information more often than evidence-based information, but that evidence-based information is more often retweeted [32]; therefore, during the first three months of the outbreak, the volume of misinformation tweets was small compared to that of the overall conversation [37]. The amount of Twitter data related to COVID-19 dwarfed that of other health-related content, but proportionally more of the data originated from credible websites [33].

Researchers have also attempted to characterize the people who are likely to believe misinformation. One nationally representative study in the United States found that some myths (eg, that the virus was created or spread on purpose) were believed by over 30% of respondents [38]. Evidence across several countries shows that people who believe misinformation are more likely to obtain information from social media or have a self-perceived minority status [39], and characteristics such as “trusting scientists” and obtaining information from the WHO had a negative relationship with belief in misinformation [40].

With the above framing in mind, this paper seeks to answer the following research questions:

1. Can conspiracy theories identified a priori be automatically identified using supervised learning techniques?

We used a large corpus of Twitter data (120 million initial tweets and 1.8 million tweets after our initial regular expression filtering step) and random forest models to classify tweets associated with the four conspiracy theories described above.

2. Can identified tweets about defined conspiracy theories be characterized by existing methodologies?

We used tweet sentiment to assess the emotional valence in conspiracy theory tweets compared to their non–conspiracy theory counterparts. We used dynamic topic modeling, an unsupervised learning approach, to explore the changes in word importance among the topics within each theory.

3. Can our findings inform public health messaging to reduce the effects of misinformation found on social media?

We compared the results of the preceding research questions to identify commonalities and connections between early conspiracy theories that can be addressed by initial public health messaging to prevent further misinformation spread. We additionally showed that theories evolve to include real-world events and incorporate details from unrelated conspiracy theories; therefore, later public health messaging will also need to evolve.

We filtered the data into four data sets using regular expressions (see Figure 2) to increase the number of relevant tweets in each category of interest [43-47]. The four data sets are hereafter referred to using the following terms:

Within each regular expression-filtered conspiracy theory data set, we randomly sampled 1000 tweets to create the training data. After sampling, duplicate tweets were removed. Two authors coded each set of tweets and established agreement by jointly coding a subset of tweets (see Table 1). Any tweet promoting or engaging with misinformation, even to refute it, was labeled as COVID-19 misinformation. This labeling was performed with the rationale that tweeting about misinformation, even in the context of a correction, increases the size of the audience exposed to that misinformation. In prior work on COVID-19 conspiracy theories, it was found that engaging with a theory to correct it can indeed increase the overall visibility of the theory [18]. Interrater analysis found relatively high agreement and reasonable Cohen κ scores (mean 0.759, Table 1). However, the effort demonstrated the difficulty of reliably identifying misinformation; in many cases, oblique references and jokes fell in a gray area that raters labeled “uncertain” (~6.1% of the coded tweets). A second pass was made over tweets labeled “uncertain” by comparing rater assessments and marking these tweets as “COVID-19 misinformation” or “not COVID-19 misinformation” based on rater agreement. For example, if annotators 1 and 2 had high agreement when labeling 5G tweets, a tweet labeled by annotator 1 as “uncertain” could be relabeled as “COVID-19 misinformation”. Using this approach, we were able to avoid removing data and thus shrinking the amount of available training data.

The tweets were tokenized, and both URLs and stop words were removed. Unigrams and bigrams were used as features in a document-term matrix, and the most sparse (<0.05% populated) terms were removed. Additionally, we added Boolean features describing relationships to domains identified by NewsGuard as sources of misinformation. This was achieved by linking associated Twitter accounts to tracked websites. Features included (1) a tweet originating from a misinformation-identified domain, (2) a tweet replying to an originating tweet, (3) a tweet retweeting an originating tweet, or (4) a tweet that was otherwise linked (eg, replying to a retweet of a tweet from a misinformation source). As noted elsewhere, only English tweets were used in this analysis.

The data were partitioned into a two-thirds/one-third training-test split. Data were sampled so that the training data had an equal sample distribution (50% misinformation, 50% nonmisinformation). The testing data used the remaining available data; thus, the sample distribution was uneven.

Classifiers were built using R, version 3.6.3 (R Project); the randomForest package, version 4.6-14, was used to train random forest models with 150 trees up to 25 terminal nodes (and at least 3 terminal nodes), and 25 variables were randomly sampled at each split. Case sampling was performed with replacement. We used an active learning approach in which after each run of the random forest classifier, the calculated posterior entropy was used to select the three unlabeled tweets that caused the most uncertainty in the model. These were then hand-labeled by an author (DG) and applied to the next run of the model. We applied 9 cycles of active learning to each model. Additionally, for each hand-labeled tweet, highly similar tweets (string similarity ≥0.95) were identified and given the same label. This approach was implemented using the R activelearning package, version 0.1.2. The models that performed the best (measured by F1 score) were used to assign labels to the regular expression-filtered tweets.

Two well-documented sentiment dictionaries were used to label the tokenized tweets. The first, AFINN [48], provided an integer score ranging from –5 (negative sentiment) to +5 (positive sentiment) for each word in the dictionary. The second dictionary, the National Research Council (NRC) Word-Emotion Association Lexicon [49], was used to tag words with categories of emotion, providing labels for 8 emotions of anger, anticipation, disgust, fear, joy, sadness, surprise, and trust in addition to an overall “positive” or “negative” sentiment. We then compared the sentiment for each classified data set over time. For each tweet, aggregate sentiment metrics were calculated, including the sum of integer scores and the counts for each emotion label.

Dynamic topic modeling (DTM) was used to characterize themes and analyze temporal changes in word importance [50]. DTM divides tweets into weekly time slices based on the time they were generated. The set of topics at each time slice is then assumed to evolve from the set of topics at the previous time slice using a state space model. The result is an evolving probability distribution of words for each topic that shows how certain words become more or less important over time for the same topic. Traditional topic models, such as latent Dirichlet allocation [51], assume that all the documents (which are here equivalent to tweets) are drawn exchangeably from the same topic distribution, irrespective of the time when they were generated. However, a set of documents generated at different times may reflect evolving topics.

Dynamic topic models were trained for each conspiracy theory, with the number of topics ranging from 2-5. Small numbers of topics were chosen because these tweets were already classified to be relevant for individual misinformation topics, and because our goal was to identify potential subtopics that evolved over time. The optimal number of topics was assessed qualitatively by reviewing the topic modeling results. DTM was implemented in Python using the gensim [52] wrapper (“ldaseqmodel”) for the DTM model [50,53].

After filtering using regular expressions, our corpus included roughly 1.8 million unique tweets across the four conspiracy theories (Table 2). The relative volume of tweets in each data set is shown in Figure 3. The number of tweets appearing in multiple data sets corresponds to the edge thickness. All the data sets showed some degree of overlap between categories, with Gates showing the most overlap and 5G showing the least. 5G additionally had a low volume of tweets compared to the other theories.

The model performance metrics for each theory are presented in Tables 3 and 4. Class proportions were roughly balanced in the 5G, Gates, and Lab theories. The Vax tweets were heavily imbalanced, with only ~18% labeled as COVID-19 misinformation (Table 3). The best performing models were the 5G and Lab theories, with F1 scores of 0.804 and 0.857, respectively (Table 4). Although the results for the Gates theory were weaker (F1 score=0.654), Vax scored the lowest (F1 score=0.347). This could be due to the imbalanced nature of the data set.

The range in sentiment was significantly greater for COVID-19 misinformation, with tweets more consistently showing increased negative sentiment, especially in April and May 2020. Figure 4 shows Gates-related tweets by net sentiment score over time. See Multimedia Appendix 1 for additional figures related to other conspiracy theories (Figures S1-S3).

Figure 5 shows the sentiments of tweets (with daily average sentiment scores for each category averaged across all dates in the study range) in each conspiracy theory subset across eight emotions and the general negative or positive sentiment. Although tweets related to 5G conspiracies show similar results for misinformation and nonmisinformation, there are clear differences in the other four conspiracy theories. In general, tweets classified as misinformation tend to rate higher on negative sentiment, fear, anger, and disgust compared to tweets not classified as misinformation.

For each conspiracy theory data set, DTM was used to identify 2-5 potential subtopics and understand their evolution over time. The optimal model was assessed qualitatively by reviewing the results. Models with 2 topics led to optimal results (qualitatively coherent topics with the least amount of overlap) for Gates, 5G, and Lab theories, while the model with 3 topics qualitatively led to optimal results for Vax theories. The results for the Gates theory are visualized here, and the remaining theories are visualized in Multimedia Appendix 1.

The Gates theory was optimally represented by 2 topics. Both topics showed peaks of increased Twitter discussion in mid-January to mid-February and a second peak in April (Figure 6). The initial peaks in Topic 1 corresponded to high weighting of the words predicted, kill_65m, event, and simulation, while the later spike in April showed higher weights for words such as fauci and buttar (Figure 7). The model identified a second topic that referred to several conspiracy theories about Bill Gates, SARS-CoV-2, and vaccines. This second topic initially focused on theories about the origins of the virus, with highly weighted words including pirbright and patent. In late April, higher-weighted words included kennedy, jr, and fauci.

The Vax data showed high weighting for the word bakker in Topic 1 and a brief increase in the word microchip in early April within Topic 2 (Multimedia Appendix 1, Figure S6). The term bakker refers to the tele-evangelist Jim Bakker, who promoted myths about possible COVID-19 cures, including the use of colloidal silver, on his show [54]. A linguistic shift in referring to the virus was also observable within the vaccine theory, with coronavirus highly weighted until mid-March, when COVID became more frequently used.

In the Lab data, words such as biosafety, biowarfare, warned, and laboratory were more highly weighted early in the outbreak, suggesting that people were discussing a malicious laboratory release [63] (Multimedia Appendix 1, Figure S5, topic 2). The weight of words such as escaped, evidence, and originated increased as the theory evolved over time. Overlap was seen between the Lab theory and the Gates theory, including words such as kill, kill_65m, and kill_forget. In addition, we observed terms related to other, older theories, such as ebola in Topic 2 in mid-January, and terms related to Jeffrey Epstein and conspiracy theories associated with his death (epstein, forget_epstein) [40].

The ongoing COVID-19 pandemic clearly illustrates the need to identify health-related misinformation, especially with a lens toward improving communication strategies to combat it. We focused on four specific conspiracy theories and fused existing methods to identify relevant tweets and characterize the language used over time. This is especially important in the context of COVID-19 as an emerging infectious disease, when much of the scientific knowledge about its risks, transmission, and mitigation may be quickly evolving [56]. With this context in mind, we address our findings with respect to each research question below.

A major limitation of any work on misinformation is that we obviously cannot examine all relevant theories, or even all of the nuance in our four identified public health-related theories, in any single study. Conspiracy theories are continuous in nature, as demonstrated here, whereas we can only observe a discrete sample within any single study. Because of this, we must aim for internal validity within any single, well-defined study and hope that many such studies will contribute to a “big picture” of social media misinformation and its effects. Not only has COVID-19 misinformation continued to spread past the end of our analysis in May 2020, but emerging conspiracy theories and topics continue to relate back to the conspiracy theories presented here. For instance, our research into claims about a laboratory origin of SARS-CoV-2 focused on popular conspiracy theories around a Chinese laboratory in Wuhan, a Canadian laboratory, and Fort Detrick in the United States. However, even at the time of this writing, two additional theories have gained traction. One indicates that the virus originated from the French Pasteur Institute; another suggests that it originated in a laboratory at the University of North Carolina [20]. We hope that results captured at the time of this analysis can inform subsequent investigations.

Second, our labeled training data explicitly labeled attempts to correct or refute misinformation as misinformation. Although this approach more accurately captured the exposure a given conspiracy might have in social media, it likely led to overestimation of the number of individuals supporting any particular theory. Excluding corrections could also have produced subtly different sentiment and dynamic topic model results, as people promoting conspiracy theories will likely differ in sentiment and word usage from those attempting to refute them. We chose to include corrections to avoid attempting to infer tweet context (eg, sarcasm is difficult to distinguish in an individual tweet) and because retweeting inaccurate information, even to correct it, still increases the number of individuals who see inaccurate content [18]. Prior work has identified both rumor-correcting and rumor-promoting tweets during crises using Twitter data [62]. Future work would benefit from considering these separately.

Additionally, our exclusive use of Twitter data fails to capture the entirety of the spread of misinformation. Social media platforms have broadly faced significant challenges in identifying and containing the spread of misinformation throughout the course of the COVID-19 pandemic [7]. Twitter users are also known to be a demographically biased sample of the US population [63-65]. Future research would benefit from analysis of misinformation on other social media platforms. Our findings are thus not generalizable to the US population as a whole. However, we emphasize that the goal of this study is not to achieve generalizability but rather to achieve internal validity by accurately categorizing sentiment and describing misinformation patterns within this population.

Characterizing misinformation that poses concerns to public health is a necessary first step to developing methods to combat it. The ability to assess conspiracy theories before they become widespread would enable public health professionals to craft effective messaging to preempt misperceptions rather than to react to established false beliefs. Health officials too often fail to craft effective messaging campaigns because they target what they want to promote rather than addressing the recipients’ existing misperceptions [66]. Misinformation can spread rapidly and without clear direction; this is evidenced by one tweet we uncovered while conducting this research, which shared an article promoting a conspiracy theory with the commentary that the user had not established credibility but rather “thought I’d share first” (tweet anonymized for privacy). An understanding of the appearance, transmission, and evolution of COVID-19 conspiracy theories can enable public health officials to better craft outreach messaging and to adjust those messages if public perceptions measurably shift. This study demonstrates that identifying and characterizing common and long-lived COVID-related conspiracy theories using Twitter data is possible, even when those messages shift in content and tone over time.