The study was approved by the institutional review board for adherence to ethical principles of research. The images were anonymized, and aggregated statistics for states were calculated. There was no attempt to recruit subjects, reidentify subjects, or link the images with other personal information in order to maintain confidentiality. Individual-level mask use adherence was not analyzed. Anonymized images were stored on secure servers as the following 3 different data set collections:

1. Mask-unmask classifier data set: This data set was used for training a model that classifies whether the person in the image is wearing a mask or not. For training, we needed images in which people were wearing masks (masked images) and images in which people were not wearing masks (unmasked images) so that our model could learn to distinguish between the 2 categories. We collected around 30,000 images of people wearing masks from Google Search images using the tags “people wearing masks” and “children wearing masks.” Images from Instagram were also collected with the tag explore feature, using the following 3 tags: “mask,” “masked,” and “covidmask.” Although the tagging algorithms used by Google are expected to capture most of the images, some images may have been missed. There is further scope for expanding our set of tags chosen to capture the entire population of images in which people are wearing masks, which is a limitation of our current approach. However, this will need more research, as capturing other scenarios may also lead to noisier sets. After data collection, images with a width and height of at least 50 pixels were kept to ensure decent image quality. Then, Face Detector was used on these images to extract faces. The images of extracted faces were distributed among 5 annotators, and the annotators were asked to classify the faces as either “masked” or “unmasked.” After the annotations, images of 9055 masked faces were obtained. For the unmasked face images, we created a random sample (without replacement) of 9055 faces from the VGGFace2 data set [27], which is a large-scale face recognition data set. The samples from VGGFace2 and the mask-unmask classifier data set were used to train the unmask-mask classifier, whose details are given in the Proposed Framework section.

2. Fit score data set: Out of 9055 masked faces that we obtained from the previous data set, we selected 504 images with different poses and a wide variety of mask designs. Then, we annotated these images using Label Studio [28] for getting pixel-level annotations of the mask region on the face. This data set was then used to train a semantic segmentation model, whose details are in the Proposed Framework section.

3. USA cities Instagram data set: For the analysis phase, we collected location-tagged public posts from Instagram between February 1, 2020, and May 31, 2020, for the following 6 cities: New York City, Seattle, Dallas, New Orleans, Minneapolis, and Boston. The first COVID case was reported in January 2020 [29], and till July 2020, the United States was still in the first wave of the COVID-19 pandemic [30]. Hence, the chosen time frame captures the beginning and growth of the COVID-19 pandemic in the United States. This collection was done for 6 major US cities, and these 6 cities were selected to represent different geographical sections of the country. We collected a total of 2.04 million public posts from these 6 cities. These posts were collected via Instagram’s explore location feature. Instagram’s GraphQL application programming interface was employed for the data collection. The tools used have been published as a python PyPI package [31]. We also collected 195,452 posts for New York City and Minneapolis from May 25, 2020, to July 15, 2020, which had major protests [32]. We curated a list of trending tags and keywords (“blm,” “blacklivesmatter,” “georgefloyd,” “justiceforgeorgefloyd,” “policebrutality,” and “protest”) during this period. We refer to the posts whose captions included these tags as BLM posts and the rest as non-BLM posts.

This article proposes a mask-unmask classification framework (for classifying masked and unmasked images) and a fit score analysis framework (for evaluating whether the masks are being worn effectively or not in the given image). The mask-unmask classification model is used to analyze the USA cities Instagram data set. The fit score analysis framework is just used for BLM posts to capture the mask use patterns during a huge social gathering.

Images obtained from sources mentioned in the previous section consisted of various individuals. Thereby, to detect face masks in these images, the first task of both frameworks was face detection. This was done using the pretrained model Retinaface [33], which is one of the top performing models on Face Detection on the WIDER Face (Hard) data set [34]. Next, facial landmarks were obtained using Dlib’s implementation [35] (proposed by Kazemi et al [36]), which was used to extract the regions of interest (ROIs), as shown in Figure 1A. The landmarks 5-13, 31-36, and 49-68 (Figure 1B) were used to filter the face’s jaw region. This jaw region obtained was then used as input in the classification model for the mask-unmask classification framework. The landmarks 32-36 and 49-68 (Figure 1B) were used to filter the nose-mouth region from the face. This nose-mouth region obtained was then used for calculating the fit score.

The jaw region was then classified on the basis of whether it contained a mask over it or not (Figure 1A) using a classification model. The following architectures were experimented with while training the mask-unmask classification model: MobileNet V2, Nas Net, EffecientNet B0, EffecientNet B1, EffecientNet B2, and DenseNet121. These architectures were selected since they have significantly fewer parameters than most other architectures (Multimedia Appendix 1). The input image size for all the models was 224×224. Transfer learning was used, and weight initialization of all models was done using ImageNet. All models were truncated at the last fully connected layer. The following layers were added: (1) average pooling with 5×5 pool size, (2) flatten layer, (3) dense layer with 128 hidden units and reLU activation, (4) dropout layer of 0.5, and (5) dense layer of 2 hidden units and Softmax activation. Adam optimizer with an initial learning rate of 1e-4 was used, and each model was trained for 30 epochs with a batch size of 64, with binary cross-entropy as the loss function. For training the classifier, the total data from the mask-unmask classifier data set consisted of 9055 masked and 9055 unmasked samples, which were split in an 80:20 ratio for training and validation sets. Five-fold cross-validation was used to evaluate the trained models’ performance, and the different model results can be found in Multimedia Appendix 1.

The evaluation indicated that EfficientNet B0 was the best performing model, with an overall accuracy of 0.98 (SD 0.01). EfficientNet [37] is a convolutional neural network architecture and scaling method that uniformly scales all depth/width/resolution dimensions. Efficient B0 has 5.3 million parameters with 18 layers. Its architecture consists of an initial 3×3 convolution layer, followed by a series of MBconv layers with different kernel sizes and number of channels. The series of MBconv layers are followed by a convolution layer, a pooling layer, and a fully connected layer. It uses linear activation in the last layer in each block to prevent loss of information from ReLU. Compared with conventional convolutional neural network models, the main building block for EfficientNet is MBConv, an inverted bottleneck conv, known initially as MobileNetV2. Before EfficientNet came along, the most common way to scale up ConvNet was by one of the following 3 dimensions: depth (number of layers), width (number of channels), and image resolution (image size). EfficientNet, on the other hand, performs compound scaling, that is, scaling of all 3 dimensions while maintaining a balance between all dimensions of the network. We used this trained model for further analysis using the mask-unmask classification framework.

To calculate the fit score of the appropriate region covering the nose and mouth regions of the face, we used a semantic segmentation-based model (Figure 1A). We defined the fit score as shown in Equation 1. Using the data from the fit score data set, we trained a U-Net–based model [38] for segmenting images of faces into the masked and unmasked regions. The model uses ResNet 32 and 50 encoders pretrained on ImageNet data [39]. The layers were trained progressively using cyclical learning rates [40]. Different model variations were experimented with using different encoders and input image sizes (Multimedia Appendix 2). Using the output of this model (true positive [TP] + false positive [FP]) and the nose-mouth region’s facial landmarks, we calculated the fit score of an image of a face. The fit score is the percentage of ROI area covered by the mask on the face (Figure 1A). We employed the fit score analysis framework on BLM posts to understand how well people wore masks in groups during large events like protests. City-wise analysis was done to observe mask fit differences across major states of protest.

We used the Mann-Kendall trend test to look for monotonic increasing trends in the daily percentage of mask users.

We also used Pearson correlation, Spearman rank-sum correlation, and the Welch t test to perform our analysis. Although Pearson correlation assumes normal distribution for both variables [41], it has been shown to reveal hidden correlations even when data are not normally distributed [42].

We performed Pearson and Spearman correlations to decide whether the value of the correlation coefficient r between lagged COVID-19 cases and the daily percentage of people wearing masks is significantly different from 0 at a threshold of P<.01 [43].

We then conducted the Welch t test to assess whether the daily posting is significantly affected by stay-at-home laws. The Welch t test requires normal distribution as a prerequisite, but since we were comparing mean values and our underlying series length was large (>30), this assumption could be bypassed [44]. We assessed the before and after posting effects of the application of stay-at-home laws in New York, Dallas, Seattle, Boston, Minneapolis, and New Orleans (Multimedia Appendix 3) [45-50]. We perform the Welch t test to test the following hypotheses for the 6 cities and calculate the P values with an alpha of .01: H0, μ0=μ1 (the daily percentage of group posting is not affected by the stay-at-home laws) and H1, μ0≠μ1 (the daily percentage of group posting is affected by the stay-at-home laws), where μ0 and μ1 are the mean percentages of daily group posting.

In addition, we performed the Welch t test to test the effect on the percentage of masked faces from mask mandates for Boston, Minneapolis, and New York City (mask mandate dates for the other 3 cities did not lie in our chosen timeframe) (Multimedia Appendix 4) [45,51,52]. The hypotheses are as follows: H0, μ0=μ1 (the daily percentage of masked faces is not affected by the mask mandates) and H1, μ0≠μ1 (the daily percentage of masked faces is affected by the mask mandates), where μ0 and μ1 are the daily mean percentages of masked faces. We calculated the associated P value for significance testing, with an alpha of .01.

To visually inspect what the trained EfficientNet B0 in the mask-unmask classifier had learned, we implemented GradCam [53] on the network. GradCam assesses which parts of the input image have the highest activation values, given a target class. In this case, we passed the jaw region (ROI) after facial landmark detection as the input image to the GradCam network for 3 examples (2 masked and 1 unmasked) (Figure 2A). We also inspected the segmentation model on the corpus of BLM posts collected from social media.

The COVID-19 pandemic has given the entire research community and governments a chance to reflect on what kind of system needs to be in place to handle such catastrophes. A renewed focus is emerging in infodemiology [54], especially leveraging mass surveillance data [55,56]. Location tracking [57], periodic self-checks, and image recognition systems have been deployed by many governments [58-61] to get a handle on the pulse of the pandemic in their states. This study suggests another such approach, which can be applied to specific demographics to achieve similar near–real-time tracking of the pandemic’s spread. Instagram and other social media platforms have been very successful in tracking the number of visits to public places [62]. In the context of COVID-19, public places are the focal points for the spread of the virus. It has been well-documented that face masks and social distancing are the 2 most effective nonpharmaceutical interventions to curb the spread of COVID-19 [7]. However, the use of masks and effective social distancing are often self-reported [63], without any proof to corroborate the claims. Models built on image data with location data [64] can be a powerful tool for the authorities to keep track of the pandemic’s pulse.

An overall decrease in group posting was found as the pandemic grew and lockdowns were put in place. These group pictures posted online can be used as an estimator for the percentage of people spending time in groups. A sudden spike in group posting was found from May 16, 2020, to May 22, 2020 (week 15 of our timeline). This can be linked to the easing lockdown restrictions through weeks 13, 14, and 15 [65-68]. As the pandemic spread, the percentage of mask users saw a significant increasing trend for all the 6 cities through the months, suggesting that more people started wearing masks as the pandemic spread. Significant positive Spearman and Pearson correlations were found between the daily COVID-19 cases and the percentage of mask users for all the cities, except New York City, with an alpha of .01 (Multimedia Appendices 12 and 13). The maximum correlation was found to be with lag as 1 for all cities, except Minneapolis, as seen in Figure 4. As seen through social media, the stay-at-home state policies were successful as a significant decrease in group posting was observed on the adoption of stay-at-home laws for all 6 analyzed cities. After the mask use mandates were applied, a significant increase in the percentage of mask users was seen in Boston and Minneapolis, and a slight increase was observed in New York City. These results indicate adherence to nonpharmaceutical interventions in the 6 cities, with varying percentages of changes and effects. The trends of increasing mask use with the pandemic and positive changes with mask mandates corroborate with self-reported number-based survey methods in the United States [7]. Although a significant increase was seen in the percentages, the growth could have increased separately from the mandates. With an insignificant increase seen for New York City, supplemental public health interventions can be applied to maximize the adoption of such methods.

A large difference was observed in the percentage of group pictures between the posts that talked about the BLM protests. This difference can be explained by the huge collection of people in protests, with a lack of social distancing measures causing a high percentage of posts to involve group pictures. On the contrary, mask use was found to be much higher for BLM-related posts as compared to non-BLM posts. The mask fit score distribution of the protestors showed that only 35% of mask users had more than 80% of their nose/mouth region covered. This indicates that, while social distancing measures were not appropriately followed due to the nature of such large gatherings, protestors were more likely to wear a mask than the general public, but only a small percentage covered their faces properly, as seen through social media posts.

Models built on image data with location data can be powerful tools for authorities to keep track of the pandemic’s pulse. This study provides a new method for governments and organizations to monitor policy decisions indirectly.

The images present in our data sets have a high definition and are in RGB mode. If the trained models have to be deployed in a new setting (eg, CCTV feed), certain image augmentation techniques like gray scaling and rescaling might be needed to fine-tune the models. For the analysis, we chose the image data from 6 major US cities by population and correlated the data with state-wide COVID-19 cases. Since these cities are some of the most populated cities of their respective states, it is reasonable to assume that they will be the hotbeds of COVID-19 spread in their respective states.

The analysis was conducted using images obtained from the social media platform Instagram. We understand that these images might not represent the entire city population [69]. However, they do represent a wide demographic of internet users [70]. With recent studies conducted on geo-tagged text data present in Instagram posts [71], our assumption of a fair population representation in the Instagram data might not be too far-fetched. Among the mask users, celebrity posts (posts with likes greater than 10,000) contributed to only 0.2% of the total posts, which contained at least one mask, showing that the collected data mainly involved posts from the general population. However, we do acknowledge that capturing metadata for users while performing similar studies might yield a conclusive answer to the question of fair representation. Capturing and using metadata can be future work, which will build on our results.

An addition to the modeling pipeline could be an indoor/outdoor environment detector, similar to that in the study by Zhou et al [72]. Another addition to the analysis could be selectively looking at the specific activity of users, who are deemed as “influencers” on the network. Their activity on the network can be analyzed in conjunction with the activity of their followers. This can help determine the role of social networks and the power of certain influential nodes in that network over other people’s behavior during critical times such as a pandemic. Dynamic location relationships present in mobility data [73] can be further used to understand the pandemic’s spread with higher location precision and even recognize malevolent actors in the system. A natural extension of this work is its replication across different social media platforms like Twitter, Facebook, and Baidu.

Models built on image data with location data can be powerful tools for authorities to keep track of the pandemic’s pulse. This study examined 2.04 million posts collected from 6 US cities between February 1, 2020, and May 31, 2020, for adherence to mask use and social distancing, as seen through social media.

This study found a general increasing trend in mask use and a decreasing trend in group pictures as the pandemic spread. The stay-at-home laws caused a significant drop in group posting for all 6 cities, while the mask mandates caused a significant increase in mask use for 2 of the 3 cities analyzed. Although these results suggest an upward trend in the adoption of preventive methods, a large portion of nonadopters seen online indicates a need for supplemental measures to increase the effectiveness of such methods.

Posts related to protests were found to capture the lack of attention given to safety measures, with high percentages of detected group pictures and incorrect mask use. The methodology used provides a directional indication of how government policies can be indirectly monitored. The findings can help governments and other organizations as indicators for the successful implementation of nonpharmaceutical interventions for the COVID-19 pandemic.