The open and actively updated data set named Coronavirus (COVID-19) Vaccinations [15], hosted by the Our World in Data database, was accessed on July 8, 2021. There were 58 countries/regions with their total vaccination count exceeding 4 million doses. For these 58 countries/regions the total vaccination count and vaccination dose per 100 people were noted. The governmental or official vaccination website was searched via Google with the phrases “official,” “COVID,” and “vaccine.” After entering these three terms, the first 10 result pages were examined to identify the relevant official vaccination website. The “frequently asked questions” (FAQs) or “questions and answers” (Q&As) section was identified and analyzed. This particular section was selected assuming that this would be the page where citizens sought for official information in the first place. Together with the websites of the World Health Organization (WHO) and European Union, we attempted to collect information from a total of 60 websites. As many European countries are in the European Union, the public living in Europe may look for information not only from their own country’s website but also from the EU website. Similarly, the WHO is the global health authority, and many people may look for information from its website as well. Therefore, websites from the WHO and European Union were included in this study. If there were multiple websites for one country (eg, vaccination websites from different governmental departments), then the one with FAQs targeting the public was selected. If such a section was not available from any of the websites, then the country would be coded as “cannot locate the vaccination FAQ website” (n=10; see Figure 1). The total vaccination count and vaccination dose per 100 people would be assessed with the website metrics to reveal if a better prepared website would relate to better vaccination coverage.

The website identification was performed by four authors (AWKY, EK, EDP, and AGA). The data extraction was performed by AWKY and double-checked by AGA. Disagreements in these procedures were solved by discussion and consensus reached between AWKY and AGA.

For readability assessment (limited to websites with English versions only), the first 5 answers from the FAQs or Q&As page were transferred to Office Word software (Microsoft Corporation). The first 5 answers were used for evaluation, as online health information seekers rarely go beyond the first page of a search [16], and we reasoned that 5 FAQs would occupy a similar page space. The Flesch Reading Ease score and Flesch-Kincaid Grade Level statistic [17] were computed automatically by its built-in Spelling and Grammar function. A higher Flesch Reading Ease score indicates the text is more readable: reading difficulty is considered very easy if the score approaches 100, whereas the score range of 60 to 70 is considered standard [18,19]. In contrast, the value for the Flesch-Kincaid Grade Level indicates the grade level required for a reader to understand the text [17]. The American Medical Association and National Institutes of Health recommended that patient information materials should be written in the third to seventh grade level [20,21]. In the United States, the third grade means the third school year after kindergarten (third year of primary school) when students are aged around 8 to 9 years. Similarly, seventh grade means the seventh school year after kindergarten when students are aged around 12 to 13 years. The readability of websites without the English version were not evaluated, but they were still evaluated for the subsequent analyses (quality assessment and listing of protection duration and side effects) following Google Translation to English.

Content quality was first assessed by the Health On the Net Foundation code (HONcode) certification status (ie, whether a website bore the HONcode badge). The Health on the Net Foundation is affiliated with the United Nations and issues the HONcode for health care websites that filed an application and met the required standard [22]. A dedicated web browser toolbar can be installed [23] to identify the certification status of a website, as used by previous researchers [24]. Having a HONcode certification is good, as it indicates the website reaches a standard of offering quality health information.

As this HONcode only gave a binarized certification status (certified or not) to the websites, we also used the Quality Evaluation Scoring Tool (QUEST) to evaluate the FAQ sections in seven aspects, namely, authorship, attribution to sources/references, type of study attributed to, conflict of interest, currency of presented information, complementarity of the patient-doctor relationship (support or no support of the relationship), and tone [25]. Readers can refer to the second figure by Robillard et al [25], published with open-access, for the complete description of the coding and scoring of QUEST. There is no cutoff score for QUEST, but the original research team reported that a score ≤11 indicated the worst scores from a survey of 290 online articles [26].

We recorded the number of Q&As from each website. The stated protection duration and the side effects of vaccinations listed on the websites were also recorded.

It was not possible to collect all data within a single day. Data were collected since the day of the database search (July 8, 2021). On the last day of data collection (September 29, 2021), the publication dates and dates of the last update from all websites were collected to ensure a fairer evaluation of recency.

The Pearson correlation test was conducted to evaluate if the two readability metrics, QUEST score, and number of Q&As were correlated with the total vaccination count and vaccination dose per 100 people. Data from the WHO and European Union were not included in the correlation test or regional counts. Statistical tests were performed with SPSS (version 26.0; IBM Corp). Results were deemed significant if P<.05.

The 60 investigated vaccination FAQs websites are listed in Table 1. By the United Nations geoscheme, 22 of the 58 selected countries/regions were in Asia, 22 in Europe, 4 in North America, 7 in South America, 2 in Africa, and 1 in Oceania. There were 10 countries for which we could not locate a vaccination FAQs website. There were 20 countries that provided non-English websites only. Therefore, quality and content data were available from 50 websites, whereas readability data were available from 30 websites.

The Flesch Reading Ease score of the vaccination FAQs websites ranged between 11.2 and 69.5, with a mean of 40.9 (SD 13.2). The websites of Sweden and the United Kingdom had a score above the threshold of 60 (Figure 2). In particular, the most difficult website was from the Philippines, with a score of 11.2.

Meanwhile, the Flesch-Kincaid Grade Level of the websites ranged between 6.5 to 17.6, with a mean of 12.1 (SD 2.8). Only the website of Sweden was within the targeted range (Flesch-Kincaid Grade Level 6.5; Figure 2).

Only the websites from Switzerland and the WHO were HONcode certified. The QUEST score of the analyzed websites ranged between 7 and 20, with a mean of 15.3 (SD 3.1). Only 5 websites did not reach the recommended score of 11 (Figure 3). A detailed breakdown of the score revealed that only Sweden and Canada explicitly mentioned the authorship. None of the websites listed references to identifiable scientific studies. Besides, half of the websites (25/50) did not present a publication date or date of the last update. For the 25 websites that listed a date of the last update, all of them were updated in the year 2021 (n=12 in September, n=2 in August, n=3 in July, n=2 in June, n=3 in April, n=2 in February, and n=1 in January). Four of them also listed a publication date, ranging from October 23, 2020, to April 7, 2021.

The number of Q&As listed in the websites ranged from 6 to 150, with a mean of 36.5 (SD 35.1). Regarding the duration of protection offered by the vaccines, 46% (23/50) of the websites stated that they do not know, and another 40% (n=20) did not address this aspect. The remaining websites stated that the vaccine could provide protection for at least 6 months (n=2), 6 to 8 months (n=1), at least 9 months (n=1), at least 9 to 12 months (n=1), and at least 12 months (n=2). Five side effects were most frequently mentioned, namely, fever/chill (n=41, 82%), various injection site discomfort events (eg, swelling, redness, or pain; n=39, 78%), headache (n=36, 72%), fatigue (n=33, 66%), and muscle/joint pain (n=31, 62%). Other side effects were mentioned much less frequently (<28%). Figure 4 lists the frequency counts of mentioned side effects.

Pearson correlation tests revealed that the Flesch Reading Ease score, Flesch-Kincaid Grade Level, QUEST score, and number of Q&As had no significant relationship with total vaccination count (P=.35, .29, .47, and .23, respectively) and vaccination dose per 100 people (P=.30, .44, .33, and .59, respectively).

As information on COVID-19 vaccination might influence vaccination progress, we set out to assess governmental websites providing information on COVID-19 vaccination in 60 countries/regions and organizations, and finally identified and analyzed 50 websites that promote COVID-19 vaccination programs.

Readability remained an issue for most of the analyzed websites. Even though we analyzed the FAQs section of the websites, which are supposed to be reader friendly, all but 1 failed to meet the recommendation by the American Medical Association and National Institutes of Health of being easily understood by a seventh grader or below [20,21], and only 2 met the recommendation of a Flesch Reading Ease score of 60 to 70 [18,19]. One of the potential reasons could be that the information disseminated in these vaccination campaign websites must be accurate and explanatory to minimize any risks of legal consequences. Hence, the choice of words would be careful and explanatory clauses added to lengthen the sentences. For example, the Philippines website was as difficult to read as a standard auto insurance policy (see Flesch [77] for more examples). It listed a simple question “are there enough vaccines for filipinos” with an answer composed of 2 long sentences with many polysyllabic words:

Certainly, readability and content quality are two different aspects valued by online health information seekers. HONcode certification was reported as an indicator correlated to the content accuracy of online health information [78]. In this context, it is interesting to observe that the surveyed vaccination websites did not present certifications. Since the Health On the Net Foundation is affiliated with the United Nations and its certification is globally recognized, more efforts should be made to get certified, especially as these vaccination websites are accessed by many people on a daily and global basis. However, most of them actually had a QUEST score of ≥11, meaning that their content quality was good. The most frequently listed side effects in the websites were consistent to the ones most frequently reported by people receiving the COVID-19 vaccines [79,80]. This consistency is beneficial, as information seekers will feel that the website is trustworthy and provides accurate descriptions of what to be expected if they want to be vaccinated. Although none of the websites listed references to identifiable scientific studies, it could be argued that the FAQs websites were primarily intended for provision of the most important and updated information; references might not be highly sought after by the information seekers in this scenario.

The majority of the websites either did not mention the duration of protection or stated that they did not know how long vaccination could protect citizens. Despite this particular missing (or unknown) piece of information, these 48 countries/regions still had the highest total vaccination counts. To get vaccinated may be influenced by many different considerations, wherefore readability, quality, and content of the vaccination websites were not the only potential influencing factors. Higher frequency of exposure to positive information about COVID-19 vaccination was associated with a higher intention to be vaccinated [81]. Meanwhile, public health authorities should also monitor social media continuously to identify newly developed antivaccine arguments and provide updated information to clear the doubts and debunk the myths [82]. These aspects should be carefully considered by the authorities as they update/improve the campaign websites. This survey found that only a small proportion of the websites were last updated in September 2021. It suggested that the rest of these websites should be updated more frequently to provide up-to-date information.

Initially, it was a bit surprising to find that there was no relationship between vaccination counts and website metrics. However, it could be explained by the fact that people could access vaccine information via many sources such as newspapers, magazines, YouTube, and Facebook. A previous study [83] revealed that the commonest reasons for agreeing to be vaccinated included “to protect themselves and others” (29%), “belief in vaccination and science” (16%), and “to help stop the virus spread” (15%), which could be intuitive and might not be affected by vaccination website readability.

This study found that only 2 of the 50 surveyed websites were HONcode certified. This was consistent to a recent survey on online misinformation about COVID-19, for which the certification rate was reported to be 1.8% (2/110) [84]. It was similarly reported that it was difficult to comprehend the privacy policies of COVID-19 contact tracing apps, with their reading level between 7th and 14th grade [85]. Considered together, the take-home message for the authoritative bodies and policy makers is to further improve the COVID-19 vaccination campaign items (eg, websites or apps) by making them more comprehensible and more trustworthy with certification.

This study surveyed the FAQs section of official vaccination websites. However, people may seek online information from other sources such as online discussion forums, news outlets, and blogs. Readability and content quality of such sources was outside the scope of this study and might provide an interesting topic for further research. Furthermore, some official vaccination websites did not have FAQs sections, and thus, no data were extracted. Meanwhile, the initial decision to target 60 websites/countries was relatively arbitrary. Therefore, readers should be aware that not all websites across all countries were included. For the non-English language websites, content were only analyzed after translation to English via Google Translate. Therefore, readability could not be assessed. Even for the websites that presented English materials, some biases or issues may still exist. In particular, because English could be a primary language for some countries but not the others or even for certain populations within a country instead of the entire population, bias might be introduced. Hence, the content could be written as a primary version, a translated version, or even provided by automated website translation beyond our notice. Instead of choosing the desired language by clicking a button, a country might provide FAQs sections in other languages at completely different websites not evaluated in this study. In addition, the potential reuse of content from EU/WHO websites by country websites was not assessed. Readers should be aware of these issues during the interpretation of the readability assessment results.

The following points could be concluded: the content quality was good, HONcode certification should be considered, content should be written in a more readable manner, and web pages should be updated more frequently to keep the information up to date. In short, COVID-19 vaccination FAQs websites basically provide good quality information, but more efforts should be paid to make them more readable and updated.