This study focused on public agencies in the state of Texas. Texas was chosen because this state became one of the disease epicenters following the enforcement of Governor Abbott’s state reopening measures in April 2020. At the time of data collection (mid-July 2020), Texas was facing the second peak of COVID-19 cases and had the highest 7-day average number of daily new cases (n=15,038) [27]. In addition, with Texas being the second largest and second most populous state in the United States, its public agencies may face the particularly challenging task of reaching out to the diverse population and coordinating with peer agencies. Since this study examined the public tweets of governmental agencies, it was exempt from human subjects ethics review.

We conducted the following steps to select the sample tweets for analysis. First, we identified all of the active Twitter accounts of public health departments and Office of Emergency Management (OEM) organizations at the city, county, and state levels in Texas. To identify public health departments, we obtained a list of health department directories from the Centers for Disease Control and Prevention and the US Department of Health and Human Services. Additionally, a list of local-level health agencies was obtained from the National Association of County and City Health Officials. In this step, we identified a total of 26 Texas public health departments that actively tweeted during the studied period. We also used a list of Texas city and county names to conduct searches on Twitter and identified an additional 56 official OEM organization Twitter accounts, which yielded a total of 82 organizations. Second, we created a list of 25 COVID-19–related keywords (“covid,” “corona,” “koronavirus,” “ncov,” “sars,” “pandemic,” “epidemic,” “quarantine,” “outbreak,” “handwash,” “wuhan,” “panic,” “chinese virus,” “lock down,” “sheltering in place,” “shelter in place,” “flatten the curve,” “safer at home,” “stay home,” “face covering,” “wear mask,” “get tested,” “quarantine,” “ppe,” and “n95”). All tweets from the 82 organizations that contained at least one of these keywords and were published between January 1 and June 30, 2020, were downloaded using Twitter’s developer application programming interface (n=15,382).

A codebook was developed to guide the coding of the training data set. It included the following variables: functions, types of recommended actions, and HBM variables. Each tweet was coded in terms of the presence or absence of COVID-19–related content. Tweets that contained COVID-19–related content were further coded.

First, each tweet was coded in terms of the functions it served. Tweets served the information function if they shared information about COVID-19, such as COVID-19 symptoms, risks of the disease, prevention information, current infection rates or case numbers, and testing information, or if they described actions that agencies were taking to contain COVID-19 spread. Tweets served the action function if they urged readers to adopt a certain health behavior. Tweets served the community function if they built community by asking readers to interact with each other and with the sender, providing emotional support, and boosting morale. These descriptions of functions were adapted from Kang [25]. Each tweet was evaluated in terms of whether it contained any of these three types of information.

Second, each tweet was coded in terms of whether it included one or more of the following actions: (1) handwashing, (2) social distancing, (3) mask wearing or face covering, (4) staying at home or sheltering in place, (5) getting tested, (6) learning more information, and (7) other behaviors.

Finally, HBM variables, including severity (any reference to the magnitude and seriousness of COVID-19), susceptibility (the likelihood that a person, a group, or the public in general will contract COVID-19), benefits (the benefits of recommended behaviors and their effectiveness in preventing or treating COVID-19 or containing the pandemic on the societal level), barriers (the difficulties associated with adopting or implementing the recommended behaviors), and self-efficacy (one’s ability to engage in recommended behaviors) were coded. TThe coding of these health beliefs was adapted from Tang and Park [12], respectively).

Several rounds of training sessions were conducted to assist two coders with understanding each item in the codebook. Afterward, around 20% (3000/15,000) of the tweets were used for the development of a training data set. Two coders coded 150 tweets that were randomly selected from the remaining 80% (12,000/15,000) of the tweets. These tweets achieved satisfactory intercoder reliability (Cohen κ: mean 0.83; range 0.56-0.96). Two items (barriers and self-efficacy) were dropped from the codebook because they were nearly completely absent from the collected tweets. Afterward, each coder independently coded half of the training data set.

Data cleaning was conducted by following the steps laid out by Du et al [28]. The bidirectional encoder representations from transformers (BERT), a natural language processing program developed by Google, was trained to automatically classify tweets [29]. The pretrained BERT-large model from Huggingface was used [30]. We divided the initial, manually coded data sets (3000 tweets) into a training data set (number of tweets: 2400/3000, 80%) and a testing data set (number of tweets: 600/3000, 20%). In our training set, some labels had a relatively low frequency (<250 occurrences), which resulted in these labels being mostly ignored in the model’s training process. To train such low-frequency categories, we doubled all instances of tweets with minority labels to give them a stronger signal in the model. The model was trained for 3 epochs by using the AdamW optimizer with a learning rate of 2e − 5.

Precision, recall, and overall F1 score (the harmonic mean of precision and recall) were calculated for each variable. We also calculated the microaveraging F1 score and macroaveraging F1 score to evaluate variables’ performance in each classification task. We summed up all of the individual true positives, false positives, and false negatives for the microaveraged score. For the macroaveraged score, we used the average of the F1 scores of different categories. Overall, our model achieved good results (Table 1). Afterward, we used the program to automatically classify all of the tweets in the sample.

Hierarchical linear regressions or stepwise linear regressions were used to answer RQ3 (ie, how various tweet features predicted the numbers of favorites and retweets). This method enabled the assessment of separate effects from different blocks of variables. Since both variables for measuring engagement were highly skewed, we adopted the standard practice of log-transforming these metrics before they were entered into regression models. In the two regression models, the independent variables consisted of the following three blocks: (1) the information, action, and community message types; (2) the dichotomous thematic categories, which included social distancing, face covering, sheltering in place, getting tested, information seeking, and other behaviors; and (3) the health belief variables, which included severity, susceptibility, and benefits. To control for the effect of account popularity (popular accounts were more likely to promote greater public engagement), we entered the log-transformed number of followers as a control variable in each model.

Governmental agencies are among the most trusted sources of COVID-19–related information [31]. Public health agencies shoulder the responsibility of promptly providing locally relevant pandemic updates, prevention guidelines, and relevant policies to the public during the COVID-19 pandemic. This study uses the HBM and examines social media functions to understand how public health agencies in Texas communicate COVID-19 pandemic–related information to the public via Twitter and assesses the empirical relationships between various message features and social media engagement outcomes. We found that public health agencies mostly used Twitter to share information and they used Twitter to promote action and community with less frequency. Tweets that served the action function were the most likely to be retweeted and liked. Beliefs about susceptibility, severity, and benefits were the most frequently covered health beliefs. Tweets that provided susceptibility and severity information resulted in more public engagement in terms of the number of retweets and endorsements.

The information function was the most prominent function among the studied tweets, followed by the action and community functions. This is consistent with the findings of an earlier study that examined the tweets of Canadian public health agencies [32]. Information is of paramount importance to the public, especially during the early stages of an infectious disease outbreak, which are characterized by a lack of information and a high level of uncertainty. In terms of public engagement, tweets that served the information function were more likely to be retweeted. An existing study with a smaller sample size than that of our study has also shown that science-based tweets about COVID-19 are more likely to be retweeted than tweets that contain false information [33]. This means that useful information can be further disseminated through retweeting. Furthermore, tweets that promoted different preventive measures were the most likely to be retweeted and liked, which shows that the Twitter users are spreading such recommendations through retweeting. Finally, retweets that served the action and community functions were more likely to be liked. This means that readers tend to respond favorably to such tweets to show their support.

Although the HBM has been traditionally used to study psychological predictors of individuals’ adoption of preventative behaviors, it was used in this study to examine the public’s collective responses to health messages in terms of public engagement. Beliefs about susceptibility, severity, and benefits were the most frequently covered health beliefs, whereas information about barriers and self-efficacy was absent from most tweets. This means that communicating the risks of COVID-19 to the public was the priority of Texas public health agencies. Emphasizing health benefits is conducive to the adoption of preventative behaviors [11]. In addition, we found that tweets containing beliefs about susceptibility often resulted in more public engagement in terms of the number of favorites and retweets, whereas the benefits of prevention methods did not increase public engagement. It appears that the public is more interested in learning about the risks of COVID-19 than in learning about preventive behaviors during the early stage of a public health crisis, as the Crisis and Emergency Risk Communication Model has indicated previously [34]. Although this study focused on how message characteristics affect public engagement, other research has shown that public health agencies’ positions in a network (eg, whether the organization occupies a “star” position, which represents their network centrality) also affected the two-way communication between agencies and the public [35].

Our findings identified several strategies that public agencies could adopt to more effectively communicate risk information during an unfolding pandemic. First, the fact that informative tweets were more likely to be retweeted suggests that public agencies should continue to use Twitter as an information dissemination tool to increase their community outreach efforts. The sharing and retweeting function of social media can allow public health agencies to disseminate timely, credible, and easy-to-share information at a large scale, which directly and indirectly helps combat health misinformation [21]. Furthermore, as action-oriented messages were more likely to be favored, public agencies should consider incorporating specific action items into their tweets. In other words, the public needs not only factual information about the pandemic but also specific guidance and concrete action items, which can further boost the public support of public agencies.

Second, although emphasizing the susceptibility and severity of the disease increased public engagement, directly communicating the benefits of preventive behaviors was less effective in promoting public engagement. Given the importance of educating the public about prevention behaviors for infectious diseases, public agencies need to be more creative when designing, framing, and implementing social media messages about preventive behaviors. Furthermore, self-efficacy information was almost completely absent from the tweets of public health agencies. Telling the public that they are capable of performing a recommended behavior is essential in increasing the adoption of such behaviors.

In terms of methodology, this study demonstrates the feasibility of using natural language processing to identify theoretical constructs such as social media functions and health beliefs. We showed that a relatively small training data set could be used to create algorithms for the classification of a much larger corpus of Twitter data. The method established in this study can be easily used to classify COVID-19–related tweets according to different types of organizations (eg, hospitals, community organizations, and media) and individuals (eg, politicians and physicians) in and beyond the state of Texas.

This study only examined public health agencies’ tweets from a single state in the United States, and our data only covered the first wave of the COVID-19 outbreak in the United States. According to the Crisis and Emergency Risk Communication Model, the public has different informational and emotional needs during different stages of an outbreak [34]. It is important to examine agencies’ Twitter content during the later stages of the outbreak. Fortunately, our research method can be easily and longitudinally scaled to study more Twitter content from different parts of the United States. Future studies may examine how message features may vary across different stages of the pandemic and how their resulting public engagement outcomes shift over time. We only examined the text of tweets but did not examine pictures and videos. Future studies should examine how pictures or videos affect public engagement. Additionally, in terms of the communication functions of governmental organizations, an earlier study has suggested that their communication efforts are often fragmented; there is a lack of Twitter mentions, coordination, and mutual retweets among different governmental organizations [36]. Future research could examine the coordination and inconsistency among public health agencies at the local, state, national, and international levels. This approach was piloted in a recent study [37].

This study examines the content of COVID-19–related tweets that were published by the public health agencies in Texas during the first 6 months of 2020. We found that although public health agencies mostly used Twitter to disseminate pandemic-related information, they could use the Twitter platform to further promote preventative actions, since in this study, the public positively responded to tweets that promoted actions. Furthermore, the public was most likely to engage with tweets that described people’s susceptibility to contracting COVID-19, as such information helped them to understand the risk of the disease. However, there was a lack of information that convinced the public of the high feasibility of proposed preventative behaviors and increased the public’s confidence. Public health agencies can vastly expand their reach during public health crises by steadily building up their follower bases.