The Reliability and Quality of YouTube Videos as a Source of Public Health Information Regarding COVID-19 Vaccination: Cross-sectional Study

Introduction

The recent emergency authorization and rollout of COVID-19 vaccines by regulatory bodies has generated global media attention. Unsurprisingly, internet searches related to COVID-19 vaccination increased drastically during November-December 2020, as the public attempted to source information amid a surge in media coverage [1]. Many internet users turned to YouTube, the second-most visited website globally after Google, for further information [2].

YouTube is the most popular video-sharing platform worldwide. Over 1 billion hours’ worth of video is streamed each day on the website, and it is visited by over 2 billion unique users monthly [3]. It has strong penetrance globally and across all major sociodemographic groups. YouTube provides a potent means of disseminating real-time information across a population; users are able to curate video content from sources varying from individual users, through celebrities, to media outlets. Aware of their central role in the dissemination of key public health information, YouTube has implemented a COVID-19 medical misinformation policy, which forbids COVID-19–related content that contradicts local health authorities and risks public safety [4].

There have been, however, high-profile instances of internet-propagated misinformation regarding COVID-19, including the ingestion of cleaning products as potential treatment, which have had severe consequences [5]. Despite the aforementioned measures implemented by YouTube, there remains a concern for COVID-19 vaccination programs to remain an easy target for misinformation content. Previous studies have highlighted that vaccination programs, such as the human papillomavirus vaccination program, have been a common target of high-profile YouTube videos propagated by a community of vocal users who are critical of vaccination programs [6]. Furthermore, there remains a concern that anti–COVID-19 vaccination videos could (1) pose a significant threat to compliance with the vaccination program, especially among those who are disproportionately affected by the illness; (2) create spill-over dissonance toward other critical COVID-19 public health measures; and (3) displace notifications regarding other emerging time-sensitive information from official public health sources. In fact, from the beginning of the pandemic until now, COVID-19 vaccine hesitancy has been steadily increasing [7]. Understanding the nature of available content regarding the COVID-19 vaccination program on this widely used platform is, therefore, of substantial public health interest and forms a foundation on which strategies for misinformation counteraction can be based.

To date, no studies have evaluated the quality and reliability of COVID-19 vaccination–related information available on YouTube. The objective of this study, therefore, was to evaluate the reliability and quality of information of YouTube’s most prominent videos on COVID-19 vaccination, using 2 validated criteria (DISCERN and HONcode).

Methods

Methods Overview

Ethical approval for this study was waived because all gathered data are freely available in the public domain. The phrases “coronavirus vaccine” and “COVID-19 vaccine” were searched on the UK version of YouTube on December 10, 2020. The search was conducted in an incognito browser (Google Chrome) to avoid biased suggestions based on cookies. Search results were sorted by view count to identify videos that had achieved the greatest impact and were most likely to trend, thereby reaching further viewers. The 200 most viewed videos (10 pages) of each search were subsequently extracted.

Video titles and channels were first screened for relevance and English language before full-video screening. Videos were included if they described 1 or more of the following: mechanisms of action of vaccines, clinical trial procedures, manufacturing processes, side effects or safety, and vaccine efficacy. Descriptions of the criteria are provided in Multimedia Appendix 1, Table S1. If there was uncertainty regarding whether a video should be included, a consensus was sought between authors, with a predisposition to include the video for full assessment. Additionally, videos were assessed for their promotion of public health measures such as hand-washing, wearing masks, or social distancing. Finally, instances of nonfactual content in these videos were noted. Nonfactual information (ie, misinformation) was defined as nonscientifically corroborated content that contradicted medical information provided by the current local health authority or the World Health Organization. Examples of misinformation are available on YouTube’s medical misinformation policy [4]. Duplicate videos and non-English–language videos were excluded. Video content screening was completed independently by 2 authors (CC and ED). Any discrepancies were resolved through discussion with a third author (VS).

Characteristics (video URL, channel, country of origin, view count, duration, video age, and the number of likes, dislikes, and comments) of the included videos were extracted. Videos were placed in 6 main categories by YouTube channel type: educational channels produced by medical professionals, educational channels produced by nonmedical individuals (eg, science education or explanatory media), independent nonmedical users (eg, vloggers with no obvious affiliations), internet media (eg, newsmagazine shows or talk shows), news agencies (ie, clips uploaded from network news), and nonprofit or medical organizations (eg, hospitals, government organizations, or universities). Descriptions and examples of channel types are provided in Multimedia Appendix 1, Table S2.

Reliability of the video content (ie, the extent to which the source of information, and therefore the information itself, could be relied upon, evident from clearly referenced and scientifically corroborated content) was assessed against a modified Health on the Net Foundation Code of Conduct (HONcode) checklist [8] and modified DISCERN quality criteria for consumer health information [8,9], which have previously been used to assess the quality of health information on YouTube. The quality of video content (ie, completeness, understandability, relevance, depth, and accuracy of information provided) was also assessed using the DISCERN quality criteria. Video rating was completed independently by 2 authors (CC and ED).

HONcode consists of 8 principles that evaluate the reliability and credibility of health information [10]. Videos were rated with a score of 1 (adherent) or 0 (nonadherent) for each of the 8 principles. The DISCERN instrument consists of 16 questions rated from 1 to 5, which assess health content across 3 domains including video reliability (8 questions, 40 points), treatment information quality (7 questions, 35 points), and an overall reviewer rating (5 points), thus yielding a maximum cumulative score of 80 (Textbox 1) [11]. The first 8 questions were applied to all videos, and 8 additional questions were applied to videos specifically on vaccine science (ie, how the treatment works). The questionnaire was adapted from Goobie et al [8] and Loeb et al [9].

Statistical Analysis

Statistical analysis was performed using Stata (version 13, StataCorp). Intercategory differences were assessed using Kruskal–Wallis tests and the post hoc Dunn test. Interrater reliability was assessed using the Cohen κ statistic. A DISCERN score of +1 or –1 point was considered agreement. Associations between engagement metrics and DISCERN scores were evaluated using linear regression. Significance was set at P<.05. Data are presented as median (IQR) values.

Availability of Data and Material

The data sets used or analyzed in this study are available from the corresponding author on reasonable request.

Results

Video Characteristics

The video review process is illustrated in Figure 1. From the 200 results of each search, 52 duplicate videos were removed, yielding 348 unique videos. After the video title and channel were screened and the full video was assessed, 48 videos were included for data extraction, with a combined total view count of 30,100,561. Videos that were not in English (n=62) or did not meet the study inclusion criteria (n=225)—describing topics such as vaccination priority, national distribution plans, politics, or pandemic mortality figures—were excluded. The characteristics of the included videos are summarized in Table 1. The majority (75%) of videos were produced by US channels. The median number of views per video was 236,064 (IQR 152,082-596,234).

‎View this figure

Source

The 48 videos were segregated into 6 categories by their YouTube channel type. The contribution of each channel type toward the total view count is detailed in Table 2. The most viewed video (6,668,737 views, 22% of total views) described the mechanisms of action of COVID-19 vaccines and was produced by a medical organization (JAMA Network).

Content

Twenty-eight of 48 (58%) videos addressed vaccine science and mechanisms of action. Twenty-eight videos also discussed vaccine trials, and 23 videos discussed vaccine safety or side effects. Ten videos advocated the importance of continued traditional public health measures to reduce COVID-19 transmission (eg, hand-washing, mask-wearing, and social distancing).

Regarding nonfactual content, 2 videos (1 by internet media and 1 independently produced) contained unsubstantiated vaccine safety concerns, despite YouTube’s aforementioned COVID-19 misinformation policy. Both videos were interviews with a single prominent antivaccination advocate. These 2 nonfactual videos accounted for 390,927 views (1.3% of total viewership).

Adherence With HONcode and DISCERN Principles

There was strong interrater agreement for both HONcode principles (median 94%, IQR 93%-97%; median κ=0.81, IQR 0.73-0.87) and DISCERN (median 88%, IQR 82-94%; median κ=0.83, IQR 0.76-0.91).

Forty-seven of 48 (98%) videos had either low (56%) or moderate (42%) adherence with HONcode principles. In general, videos scored poorly regarding the disclosure of financial sources and advertising (Table 3). Regarding the “authoritative” domain, only approximately half of the videos had an input from a medical professional or relevant scientist. Additionally, only a minority of videos fulfilled criteria relating to the “attribution,” “justifiability,” and “transparency” of the data presented.

There were significant differences in DISCERN reliability and overall scores among different channel types (P<.001 and P=.03, respectively). Educational channels produced by medical professionals attained the greatest median (IQR) DISCERN scores for reliability (median 35.5, IQR 32.0-39.4), quality (median 21.3, IQR 20.1-25.8), user judgement (median 4.8, IQR 4.1-5), and overall score (median 64.3, IQR 58.5-66.3). Videos produced by independent nonmedical individuals achieved the lowest reliability score (median 18, IQR 18-20). The post hoc Dunn test revealed that educational channels produced by medical professionals attained significantly higher overall DISCERN scores than nonmedical educational (P=.01), independent (P=.02), internet media (P=.01), and news channels (P<.001). Additionally, both educational channels produced by medical and nonmedical professionals achieved significantly higher DISCERN reliability scores than those produced by independent nonmedical users (P<.001 and P<.001, respectively), internet media (P=.007 and P=.04, respectively), and news channels (P<.001 and P=.003, respectively).

Regression analysis revealed no significant association between engagement metrics and DISCERN or HONcode scores (Multimedia Appendix 1, Table S3). However, there was a significant positive association between DISCERN and HONcode ratings for all videos included in the study (P<.001; r²=.583).

Discussion

Principal Findings

This study highlights the importance of YouTube as a medium for sharing of curated COVID-19–related information. We demonstrate growing public interest in extracting vaccine-related content from this resource, with the videos shortlisted in this study having been viewed over 30 million times globally thus far, and an average of 1890 comments in the discussion thread of each video. The available content appears favorably received, with a mean of 14.3 likes per dislike per video. This study, however, demonstrates the varying quality of information provided on YouTube, with 98% of reviewed content with low to moderate adherence to HONcode principles and DISCERN reliability scores ranging from 18 (nonmedical individuals) to 35.5 (educational channels) out of 40.

Despite variable video quality, our search identified only 2 (4.2%) videos that would constitute mis- or disinformation, which accounted for only 1.3% of the total viewership. In comparison, studies evaluating misinformation on YouTube, published in March and June 2020, highlighted a significantly higher proportion of videos containing misleading or nonfactual information [12,13]. This is probably owing to YouTube’s COVID-19 medical misinformation policy, which came into effect on October 14, 2020. YouTube now operates a “three strikes” system to prevent users from uploading unsubstantiated videos, which parallels the COVID-19 misinformation policies of Facebook and Twitter [14,15]. While the policy is explicitly directed at videos that contain unreal claims, such as the COVID-19 vaccines “kill people who receive them” or “contain a microchip,” tackling more insidious forms of misinformation in a timely manner has proven difficult. Recent criticism of these policies has highlighted their reliance on scientific consensus from health authorities to determine what exactly constitutes misinformation [16]. In such a rapidly developing field, with limited longitudinal evidence, this consensus cannot be readily achieved, which allows time for inaccurate social media content to be shared. This is of particular concern since previous studies on vaccine hesitancy have reported that videos of a negative tone are more likely to be shared and liked, thus perpetuating misinformation and confirmation biases [17]. In an effort to rapidly combat this, there has been increased government engagement and centralization of initiatives to reduce and prevent the spread of misinformation. The World Health Organization, in partnership with government agencies, has introduced several initiatives to improve public awareness of and to tackle vaccine-related misinformation (the so-called “infodemic”) on the internet [18,19]. Additionally, social media companies and the government of the United Kingdom have agreed on a battery of measures to reduce vaccine disinformation through rapid removal of flagged content and increased cooperation with public health bodies to ensure that authoritative messages regarding vaccine safety are disseminated to as many individuals as possible [20].

Along with user-directed policies, several other strategies have been suggested to limit the dissemination of false health information on social media platforms. These include mobilizing medical professionals as advocates to counter the propagation of misinformation [21]. Among the videos reviewed in our study, less than one-third were posted by nonprofit or medical organizations or medical professionals, which accounts for a lower proportion than that posted by news agencies. Furthermore, videos from established health-related organizations such as JAMA Network or World Health Organization only accounted for 25% of the total viewership. While there has been an exponential growth in medical YouTubers, and despite the fact that videos produced by these individuals achieved the highest reliability and quality scores, concurrent with other studies, we found that their current role remains limited [22]. Of note, the most viewed shortlisted video was developed by JAMA Network, which may suggest the importance of brand recognition or marketing in attracting audiences.

In addition to the varied provenance of available vaccine-related content, we identified a paucity of reliable information on YouTube. Information in videos produced by reputable health-related organizations was significantly more reliable than that obtained from videos produced by only nonmedical individual users (P=.007), and its reliability was comparable to videos from all other categories. The majority also only achieved “low adherence” to HONcode principles. Even though most of the videos produced by these nonprofit or medical organizations explained vaccine concepts in a clear and approachable manner, often utilizing the “infographic” format, they did not cite sources or provide links to further information, a common phenomenon in videos produced by established educational channels on YouTube. Thus, these videos were unable to fulfil both DISCERN reliability indicators (eg, “referencing of information” and “directing viewers to additional sources of knowledge”) and HONcode principles (eg, attribution and transparency) and were unable to attain high scores.

While nonfactual claims were limited to a small minority of videos, the absence of key information—particularly regarding basic vaccinology and the importance of concurrent public health measures—currently limits the utility of YouTube videos as robust sources of public health information. These findings echo those of previous studies reporting that reliability and quality of non–vaccine-related information on COVID-19 on YouTube is unsatisfactory [13,23]. As such, viewers are provided with incomplete evidence as to how the COVID-19 vaccine fits into the larger public health effort and are not provided with curated resources that could potentially provide these pertinent details.

Limitations and Future Directions

There are some limitations to this study. First, the subset of videos examined were limited to the English language only, given that 88% of the videos were from the YouTube channels from the United States or the United Kingdom. While this represents a language bias and limits the generalizability of our findings to different languages and non–English-speaking countries, we note that similar findings have been reported with respect to COVID-19–related information in other languages [13].

Second, the search strategy was limited to 2 search phrases (“coronavirus vaccine” and “COVID-19 vaccine”). These phrases would not encompass the various searches the general public may make on this topic (eg, “covid vaccine,” “coronavirus vaccination,” or “covid vaccination”), which could yield different video results. Additionally, the search terms used were “neutral” and may not reflect searches made by individuals who (1) have already been previously subjected to misinformation, (2) have a network that shares similar misinformation content, or (3) are part of groups that are more likely to seek misinformation.

Third, videos were first screened by title relevance and for the purpose of pragmatism, only those videos that had titles relevant to 5 domains of COVID-19 vaccine information (mechanisms of action of vaccines, clinical trial procedures, manufacturing processes, side effects and safety, and vaccine efficacy) were considered for full video analysis. However, videos with nonrelevant titles may still contain relevant information on COVID-19 vaccination and given that they are accessible by the public through neutral search terms, they could contribute toward the dissemination of incorrect or low-quality information. Additionally, a large majority of search results were excluded at the screening stage, which resulted in a relatively small sample size. This process may have introduced a selection bias, limiting the generalizability of our findings. For completeness of reporting in the future, all videos in the search results should be analyzed for low-quality or incorrect information.

Fourth, the search was conducted at a single timepoint (December 10, 2020), which was relatively early in the timeline of global COVID-19 vaccine distribution. Given the dynamic nature of the COVID-19 pandemic and vaccine development, knowledge and attitudes regarding vaccination may evolve with increased scientific understanding and public health interventions. Moreover, the search results used in this study led to the inclusion of videos produced in late March and April when vaccine development was still in its early stages. Therefore, topics such as vaccine manufacturing, efficacy, or safety were not discussed in this early sample. These videos instead concentrated on the explanation of vaccine science and the methodology of clinical trials. A cross-sectional analysis of YouTube videos through searches at multiple timepoints or of those stratified by the age of the video could be conducted in the future to assess the progression of video content and quality.

Fifth, although used in previous studies that assessed YouTube as a source of medical information, DISCERN and HONcode principles were developed and validated for the assessment of written medical information. However, a strong interrater agreement between the scoring systems suggests they are reasonable tools to use in the absence of a validated alternative.

Finally, although efforts were made to reduce selection bias by performing the search in an incognito window, the physical search location could still be revealed to YouTube through the IP address. As such, further studies should consider assessing the nature of content that users are exposed to at different locations, perhaps carrying out stratifying analysis using socioeconomic markers such as index of multiple deprivation. Additionally, routes of misinformation may vary depending on culture, education level, and even at a national level. Social and ethnic determinants have been demonstrated to impact vaccine hesitancy [24,25]. It is important to understand the drivers of vaccine hesitancy and develop high-quality, widely available educational resources to target these demographic groups and improve vaccine uptake. Creating videos in collaboration with medical professionals and taking advantage of YouTube’s widespread reach represents one potential solution.

Conclusions

Our findings demonstrate that YouTube videos produced by educational channels, especially those produced by medical professionals, achieve the highest quality and reliability metrics. Consistent with previous similar studies, this suggests that there is currently a missed opportunity in collaboration between respected health-related organizations and established educational YouTube content producers to disseminate high-quality information on COVID-19 vaccination [26]. This could potentially be a rapidly implementable public health intervention to engage a wider audience and increase public awareness and knowledge.