Since our primary goal in this study was to understand how people express stress and relaxation through Twitter, we developed annotation guidelines for both stress and relaxation tweets based on reports from the American Psychological Association [7], CDC [8,44], and medical websites [6,45,46]. Following these guidelines, we classified tweets by both genre and theme. Genre reflects the format of the tweet (eg, personal experience), and theme reflects the domain of the actual content conveyed (including such categories as stress symptoms and stress topics).

Details for each genre and theme for stress and relaxation tweets were as follows.

Leveraging the annotated data derived from our content analysis of dataset 1, we applied and evaluated machine learning algorithms for classification of stress versus nonstress tweets and relaxation versus nonrelaxation tweets (on dataset 1). To apply the classifier trained on dataset 1 to the unseen, much larger dataset 2 (cities dataset), we first filtered tweets by keeping only the tweets that contained stress- or relaxation-related hashtags in Textbox 1 or the keywords “stress” or “relax” for each city in dataset 2. After this step, dataset 2 contained only tweets with stress- or relaxation-related keywords or hashtags. To calculate the proportion of stress or relaxation tweets at the city level, we used the stress or relaxation classifier trained on dataset 1 to filter stress or relaxation tweets, and then applied the classifier for first-hand experiencer to tweets from each city in dataset 2. Figure 4 shows a flowchart describing our machine learning design.

Our study focused on 2 machine learning-based classification tasks. First, tweets were classified into the appropriate stress and relaxation category (ie, is it stress or relaxation related?). Second, first-hand experience tweets versus nonfirst-hand experience tweets were classified. We used 2 machine learning algorithms: naive Bayes and support vector machines (SVMs), which were implemented on dataset 1 using 10-fold cross-validation. We used both the naive Bayes and SVM algorithms, as both these algorithms have been used extensively for text classification tasks [50-52]. We used the Rainbow package [51] for implementing both naive Bayes and SVMs (linear kernel). We used “bag-of-words” as feature sets for both algorithms. The reason we used the bag-of-word representation is that this feature representation is considered as a baseline and the most common text representation in text classification in general [50-52]. To the best of our knowledge, this is the first study on classifying tweets on stress and relaxation tweets.

We applied the 2-step classification to each city in dataset 2 to automatically identify stress and relaxation tweets. We calculated the proportions of stress and relaxation tweets to the total number of tweets in each city.

For both stress or relaxation and first-hand experience classifications, we used accuracy, sensitivity, specificity, and positive predictive values (PPVs) as metrics [53-55]. They were defined as follows: sensitivity = TP/(TP + FN); PPV = TP/(TP + FP); specificity = TN/(FP + TN); and accuracy = (TP + TN)/(TP +TN + FP + FN), where TP is the number of tweets that are correctly classified as true, FP is the number of tweets that are incorrectly classified as true, FN is the number of tweets that are true but incorrectly classified as false, and TN is the number of tweets that are correctly classified as false.

To compare data among cities, we used Pearson chi-square test and reported significance if the P value was less than .05 [56]. Statistical analyses were performed using the publicly available R package software version 3.2.3 (R Foundation). Note that, to preserve the anonymity of Twitter users, all example tweets reported in this paper are paraphrases of original tweets.

Figure 5 shows the distribution of themes in first-hand experience stress tweets. The highest-frequency theme in stress tweets was topic, followed by nonspecific (eg, “#stressed!!!”), action taken (eg, “I need a drink #sostressed”), and symptoms (eg, “Not sure what to do...#stressed #worried #lost”). This suggests that Twitter users who posted about stress usually posted more about the cause or topic of their stress and less about actions and symptoms associated with stress.

Among the total number of stress-related tweets, as Figure 5 shows, the most frequent topic was education, followed by other topic, work, and social relationships. This is interesting because many of Twitter’s users are young people who attend school [57,58]. It seems that education and issues related to education, such as exams and finals, were of the utmost concern for Twitter users. Examples of the education topic are “Never doing a session B math course ever again #sostressful” and “my exam in less than a month?! #stressing.” Figure 6 shows the topic distribution of first-hand experience stress tweets.

Relaxation-related tweets encompassed a wider range of topics than stress-related tweets. The most frequent topic of relaxation tweets was rest & vacation, followed by nature and water. Figure 7 shows topic distribution of first-hand experience of relaxation tweets.

Table 1 shows cross-validated classification results. Our results indicated that both algorithms achieved high accuracy (range 78.08%-85.64%), sensitivity (range 90.26%-99.09%), and PPV (range 70.68%-89.32%). Specificity was rather lower, especially with first-hand relaxation classification (naive Bayes: 11.67%, SVM: 18.33%).

Of the 2 machine learning algorithms used, SVM (with linear kernel) performed better than naive Bayes in classifying stress versus nonstress tweets (81.66% vs 78.64% accuracy, 92.73% vs 91.97% sensitivity, 70.61% vs 65.30% specificity, 76.07% vs 72.69% PPV). SVM was also better than naive Bayes in classifying relaxation versus nonrelaxation tweets in accuracy (83.72% vs 78.08%), specificity (77.18% vs 60.00%), and PPV (79.86% vs 70.68%) but slightly lower in sensitivity (90.26% vs 96.15%).

Table 1 also indicates that naive Bayes had better accuracy and sensitivity than SVM in identifying first-hand experience stress and relaxation tweets: 87.58% versus 85.61% (accuracy) and 95.53% versus 90.64% (sensitivity) for stress; 85.64% versus 83.85% (accuracy) and 99.09% versus 95.76% (sensitivity) for relaxation tweets. In contrast, SVM performed better in specificity and PPV in classifying first-hand experience stress and relaxation tweets.

Table 2 shows the terms that had the highest information gain for stress and relaxation classification. Interestingly, we found that most terms characteristic of the stress class were related to the term “stress,” such as “stressed” or “stressin,” In contrast, the terms most characteristic of the relaxation class were “vacation,” “water,” or “beach,” which are related to the topics as categorized in our relaxation schema.

Using an SVM algorithm trained on our annotated data (dataset 1), we automatically classified the much larger dataset 2 (cities dataset). We used a 3-step classification process. First, we filtered by the keywords “stress” and “relax.” Second, we applied the stress or relaxation classifier to these filtered data. Third, we used the first-hand classifier to identify first-hand stress and relaxation tweets. In both steps, we used SVM (linear kernel) trained on dataset 1 as the classifier. We used SVM because it had advantages in stress and relaxation classification in comparison with naive Bayes in dataset 1. Table 3 shows the number of tweets after each step.

To evaluate the performance of stress and relaxation classification in dataset 2, we randomly sampled 2 sets of 100 tweets, with each set consisting of 100 tweets containing either the keyword “stress” (set 1) or “relax” (set 2) from a city in dataset 2. We chose New York for evaluation, since New York had the greatest number of tweets. Then 100 tweets from set 1 were manually annotated (conducted by author SD) as stress or nonstress and first-hand experience stress or nonfirst-hand experience stress class. Similarly, 100 tweets from set 2 were also manually annotated as relaxation or nonrelaxation and first-hand relaxation experience or nonfirst-hand experience relaxation class.

Table 4 shows the results of classification of set 1 and set 2 using the SVM algorithm. It indicated fair accuracy (66.0%-92.0%) and high PPV (84.6%-100.0%); however, it had lower sensitivity in first-hand stress classification (44.0%) and specificity in relaxation classification (57.1%). The results of the SVM algorithm in dataset 2 were different from those in dataset 1, perhaps due to different data distribution. Figure 8 shows the descriptions of manual annotation of 100 random tweets of set 1 and set 2.

Figure 9 shows the proportion of stress and relaxation tweets out of all tweets by city in dataset 2. The number of stress tweets was twice that of the number of relaxation tweets, indicating that Twitter users were more likely to tweet about stress than relaxation.

To evaluate theme classification by keyword matching, we randomly sampled 50 classified tweets for each theme from New York. Manual review showed that keyword classification achieved a PPV from 60% to 90% for relaxation tweets and 40% to 80% for stress tweets. Themes that had high PPV in relaxation tweets were alcohol & drugs (94%), entertainment & hobbies (94%), and water (92%). Themes having lower PPV were nature (60%) and food & drink (78%). For stress tweets, themes having high PPV are finances (84%), education (82%), and behavioral (82%), while travel (50%) and temporal (62%) had lower PPV. Multimedia Appendix 3 shows the numbers of classified first-hand stress and relaxation tweets by theme for each city.

First-hand classification results from dataset 2 showed that cities manifested a uniform pattern of stress and relaxation tweets. We found that the singular first-person pronoun “I” was consistently used the most across all cities when expressing stress, found in approximately 4% of all stress tweets, while in relaxation tweets “I” was used less often (ranked 7), at around 2.4%. Multimedia Appendix 4 shows details of the 30 highest-frequency keywords in first-hand experience stress and relaxation tweets for Los Angeles, New York, San Diego, and San Francisco.

We also found that linguistic expressions of negation such as “not,” “but,” and “don’t” or quantifying words such as “much” were among the 30 unigrams most characteristic of stress-related tweets. In addition, users often used emotionally laden swearwords when expressing stress. It is important to note, however, that the affective polarity of certain swearwords can be highly context dependent (“it’s shit” vs “it’s the shit”) [59]. Relaxation tweets, on the other hand, tended to contain words indicating relaxation and time, such as “relax,” “home,” “time,” “day,” and “now.” We found that “home” was among the highest-frequency terms in relaxation tweets, as was “weekend.” Multimedia Appendix 5 depicts tag clouds of stress and relaxation tweets for each city.

Figure 10 shows the theme distributions of stress tweets among the 4 cities. Education was the highest-frequency topic (12%-14%), followed by work (4%-5%) and travel (4%) (data presented in Multimedia Appendix 3). Interestingly, we found that tweets describing action taken and psychological and emotional symptoms also had relatively high frequencies (8%-10%). This indicates that, besides topic, people often posted about their emotional state and reaction to stress.

The topic distributions of relaxation tweets were also consistent across cities. Figure 11 shows that rest & vacation was the highest-frequency topic (27%-31%), followed by entertainment & hobbies (13%-14%), food & drink (9%-10%), and nature (9%-10%). Multimedia Appendix 3 shows detailed numbers of stress and relaxation tweets for each city.

Although we did not find statistically significant differences in theme distributions among cities for stress tweets, there were significant differences between New York and the other cities in the topics of nature and water in relaxation tweets. This may indicate the different activities taken for relaxation between the east coast (New York) and the west coast (Los Angeles, San Diego, and San Francisco). We found that high-frequency terms for relaxation tweets in New York included “watching,” while in San Diego “beach” was more common. This intuitively suggests that San Diegans more often relaxed by going to the beach, while New Yorkers relaxed by enjoying indoor (or spectator) entertainment (“watching,” “listening”).

Compared with 2 public surveys on the most stressful cities in the United States by Forbes [40] in 2011 and CNN [41] in 2014, the proportion of stress tweets found here were different. Both surveys ranked New York and Los Angeles among the most stressful cities in the country, while San Diego and San Francisco were categorized as less stressful. Our city ranking based on the proportion of first-hand experience stress tweets was New York followed by San Diego, Los Angeles, and San Francisco (Table 5 and Figure 9). While we found no significant difference between New York and San Diego, we did find significant differences (P<.001) in pairwise comparisons between San Diego, Los Angeles, and San Francisco (Table 5).

Differences between results found in public stress surveys and our automatic classification of Twitter messages could be due to differences in methodology and population when collecting data. Public surveys collect data using telephones and paper-based reports, while Twitter messages are user generated, are naturalistic, and reflect personal thoughts.

The distribution of stress topics across cities shows an interesting finding: peoples’ reactions to stress were more positive than negative. Figure 10 shows that, for all cities, 8%-10% of tweets reported positive action taken in response to stress, while only 1%-2% reported negative action (see Multimedia Appendix 3 for details). This suggests that people may react to stress positively, or that people are more likely to publicly report positive rather than negative actions. Examples of positive reaction in stress tweets are rest (“Rest is best when you are stressed”) and exercising (“I’m so stressed, thank god I’m heading to yoga now”).

Relaxation can be considered a stress management activity. Figure 9 shows that the numbers of relaxation tweets were consistently proportional across all cities to those of stress tweets, indicating that Twitter users were consistently more inclined to post about stressful life events or experiences than about relaxing experiences. Examples of stress relief from relaxation tweets are personal contact (“I don’t need anything but a hug...”), exercising (“Went for a run, feel awesome, now time to relax”), shopping (“Last day in #SanDiego Just relaxing, shopping and say bye to friends”), and entertainment (“Relaxing watching a movie:-) :-)”). Figure 7 and Figure 10 also indicate that rest & vacation was the highest-frequency topic within relaxation tweets, followed by entertainment & hobbies, nature, and water. These topics can be considered common activities for stress relief.

Our research addressed several aspects of the use of Twitter as a medium of expression of stress and relaxation by users. First, we created schema for categorizing stress- and relaxation-related tweets based on previously published psychological guidelines. By categorizing first-hand experience tweets into the primary themes of content topics, symptoms, and actions taken, we gained further insight into the common patterns of expressions of stress.

Second, we analyzed in detail the contents of tweets based on our annotation scheme and found both similarities and differences in the prevalence and characteristics of stress and relaxation tweets across cities on the east and west coasts of the United States. The most frequent topic of stress tweets in our datasets was education, which likely reflects the younger demographic of Twitter users [57,58], but work and travel were also common topics. It is notable that, despite poverty rates, unemployment rates, and cost of living being significant factors in the methodology of CNN’s and Forbes’s stress ranking systems of the most stressful cities, finances were not a major content topic of the stress tweets in any city in our studies. Although this result could be partially attributable to the need for either computer or mobile phone access in order to use Twitter and may cause underrepresentation in lower-income groups, it may also indicate that certain topics, such as personal finances, still remain relatively taboo in social media settings. Regarding positive and negative actions regarding stress, positive actions far outnumbered more destructive behavior. The use of Twitter in itself to discuss feelings of stress and stress management can be seen as a constructive manner of dealing with stress by expressing these feelings and using the support of “followers” and friends. Social media platforms are increasingly being used as support networks in the management of chronic health conditions as varied as cancer, depression, and obesity. A recent systematic review by Patel et al found that the impact of social media use on those experiencing chronic disease was positive in 48% of studies reviewed, neutral in 45%, and harmful in only 7% [60].

Third, our study indicated that words most associated with relaxation strategies (see Table 2) fell into 3 main groups: (1) bathing and personal care (eg, “bath,” “shower”), (2) vacationing (“vacation,” “pool,” “beach”), and (3) watching sports or television (“videos,” “sitting,” “watching”), indicating that relaxation strategies involved purposefully taking time away from work-based activities and daily responsibilities. A further key theme that emerged from a qualitative analysis of the data was the idea of nature—in this case, particularly water (eg, “pool,” “beach,” “rain”)—as being of key importance for relaxation. This result is consistent with recent research demonstrating the link between stress reduction and exposure to the natural environment (eg, [61]).

Finally, we showed that machine learning algorithms could be employed to achieve good accuracy for the automatic classification of stress and relaxation tweets.

This study has several limitations. First, we obtained dataset 2 from the Twitter API’s 1% sample. Second, the annotation scheme we developed, although well suited for our purpose, could benefit from further refinement. For example, we found that many tweets were categorized as topic “other.” Third, it is likely that classification results could be improved given the availability of additional training data, in particular for first-hand experience classification of stress and relaxation tweets. Furthermore, using additional feature sets, such as ngrams, emotions, and negations, could help improve accuracy. Fourth, Twitter reports of stress and relaxation may be influenced by self-presentation issues (eg, stress related to excessive workload can be used as a status indicator in some contexts). Finally, as with all social media-based research, the population studied is unlikely to be a representative sample of the general population.

This research showed that Twitter can be a useful tool for the analysis of stress and relaxation levels in the community, and has the potential to provide a valuable supplement to social and psychological studies of stress and stress management.