Adaptation and Utilization of a Postmarket Evaluation Model for Digital Contact Tracing Mobile Health Tools in the United States: Observational Cross-sectional Study

Introduction

Background

During periods of high COVID-19 incidence, US health departments were unable to scale up case management staff to deliver effective and timely contact-tracing services [1]. This is expected considering that prior to the COVID-19 outbreak, US health departments were grossly understaffed and needed a minimum of 80,000 more full-time equivalent positions to provide adequate infrastructure and a minimum package of public health services [2].

Digital contact tracing (DCT) apps are intended to supplement health department case managers by automatically and more efficiently estimating the proximity and duration of an individual’s exposure to patients diagnosed with COVID-19. A model developed to predict the impact of conventional and mobile DCT apps on the SARS-CoV-2 pandemic estimated that the disease transmission rate could be reduced by adding either contact tracers or effective widely used DCT apps (eg, Bluetooth, GPS) [3]. DCTs reduce disease spread by alerting the user if they have been near an infected person who is also using DCT, followed by user isolation and testing for infection. However, a key DCT effectiveness assumption is that there are sufficient people adopting the app to signal disease exposure. Ongoing monitoring of population uptake is necessary to address concerns that users are relying on DCT apps that have not met minimum population-level app adoption rates [4,5].

Health Technology Assessments

Health technology assessments (HTAs) of public health applications are uncommon and challenged by lack of data, conflicting stakeholder priorities, and methodological issues [6]. The surge in COVID-19 cases in the United States in the summer of 2021 (ie, the “third wave”) driven by the SARS-CoV-2 Delta variant marks an urgent time to perform an assessment of DCT apps to inform future public health contact-tracing programs and app design [7]. By 2020, an estimated 2.6 billion people globally were expected to use mobile health (mHealth) apps, generating an estimated US $31 billion in revenue [8].

Despite the large market of mHealth apps, there is no US regulatory authority in place to evaluate if DCT apps are safe and beneficial [9]. Section 522 of the Federal Food, Drug, and Cosmetic Act provides the Food and Drug Administration (FDA) with the authority to require manufacturers to conduct postmarket surveillance to address important public health questions on the effectiveness and safety of a health device [10]. However, the FDA generally does not regulate products that are intended solely to track locations or contacts associated with public health surveillance [11]. Public health DCT developers could voluntarily implement HTAs using the FDA Software Precertification Program; however, the FDA guidance does not provide a framework to conduct the postmarket evaluations [12].

There are a wide variety of frameworks available for evaluating health applications, but they may not be suitable for HTAs [13,14]. The clinical applications marketplace conducts HTAs, but they have not been frequently performed in the public health market [6]. This study adapted the American Psychological Association (APA) App Evaluation Model (AEM) hierarchical rating system because it was developed by harmonizing application evaluation questions from 45 frameworks [15]. The AEM provides a comprehensive evaluation framework for clinicians and patients to identify high-quality mHealth apps. The framework may work well at a population level with hierarchical layers to explain adoption decisions [16,17].

DCT only functions when all people near each other are using the app. Individuals install DCT apps on their own mobile devices, and these apps must be adopted by a sufficient percentage of a population to be effective [16,17]. In this study, we set the adoption effectiveness rate at 56% of the national population [18]. We define our DCT adoption as the percentage of the population using a DCT plus those that are interested in using a DCT but have not yet adopted it. Our assumption is that public health campaigns could convert many of the interested users into adopters through public health communication campaigns.

Following public health safety guidance while using DCT is important, and the AEM framework provides health and technology evaluation elements to evaluate functional use. For example, DCT requires users to voluntarily report their positive COVID-19 test results to public health authorities; therefore, user behavior is part of DCT functionality. After DCT adoption, users should not cease their social distancing behaviors in a way that increases disease exposures.

DCT represents a new technology with a short research history, and the AEM provides a structure for measure selection [19]. The AEM has five categories for evaluation: Access and Background, Privacy and Safety, Clinical Foundation, Usability, and Data Integration Toward a Therapeutic Goal. The AEM provides a way to critically assess mHealth and does not include a minimum or maximum score to be considered “good” or “useful” [15,20].

Objective

The main objective of this study was to evaluate COVID-19 DCT apps deployed in the United States during the COVID-19 pandemic under the hypothesis that DCT requires a minimal level of uptake by a population to be effective and DCT users should follow social distancing policies. Accordingly, the specific study aims were as follows.

Aim 1: Determine if there is a sufficient percentage of people using or interested in DCT to reach the minimum critical mass (56%) [18].

Aim 2: Assess the effectiveness and safety of DCT apps using the APA AEM hierarchical rating system [16].

The AEM is a conceptual framework for developing key study measures and was used to select variables. The measures derived from the framework assess adoption rate characteristics and behaviors necessary for the safe use of DCT apps.

Methods

Data Source

We approximated a nationally representative sample of US adults (aged 18+ years), with a stratified nonprobability sample to survey 3853 online panel participants between December 22, 2020, and January 2, 2021. Respondents were recruited by Climate Nexus Polling, using several market research panels as described elsewhere [21]. Participants were recruited using stratified sampling methods [21]. Compensation for participants depended on the specific market research panel and respondents’ preferences (eg, cash, gift cards, reward points) and was valued below US $4.00. Quotas were set to match the US Census Bureau’s Voting and Registration Supplement to the Current Population Survey parameters for age, gender, race, educational attainment, census region, and Hispanic ethnicity. Sampling weights were used to account for small deviations from the preselected census parameters. The participation rate of the survey was 68.5% (qualified respondents that completed the survey). The 95% credibility interval for this survey was ±1.7%. The survey data are provided in Multimedia Appendix 1.

DCT apps are new technology and there were no established and validated survey instruments to measure their use; therefore, Climate Nexus Polling developed DCT app survey questions based on a previous Pew Research survey on digital privacy [22].

Key Study Measures Aligned to the APA AEM

Model Variable Selection

Demographic measures were based on a literature review to assess if app adoption applied to a diverse range of people [23,24]. The following background for each category links the AEM to the study’s DCT postmarket evaluation. The AEM framework was used to select variables a priori from data available in the survey (Figure 1). The detailed process is described in Multimedia Appendix 2.

‎

Access and Background

This category is concerned with the transparency of app governance and ensuring that the benefits of mHealth apps are available to a diverse range of people, regardless of background. The US DCT apps are based on mobile phone Bluetooth proximity tracking using the Google/Apple Exposure Notification (GAEN) platform [25], and are cosponsored by US state government public health organizations [25]. Adoption of DCT apps in the United States is voluntary, and varies by demographics and behavior [23]. Furthermore, DCT app adoption may be subject to health equity issues since mHealth adoption is generally associated with younger, educated populations [23,26].

The “state of residence” measure was used to demonstrate the role of a person’s state government in driving local adoption [25,27,28].

Privacy and Safety

There is an expectation of confidentiality by DCT users, and privacy policies should ensure that COVID-19 exposure proximity measurement is accurate. Concern over privacy and data breaches is a genuine concern. Experts collaborated to develop secure decentralized contract-tracing platform architectures [29]; however, some GAEN implementations may have exposed user data. A class action lawsuit was filed in April of 2021 against Google, claiming that their COVID-19 contact-tracing platform exposed unwitting Android adopters’ sensitive personal information [30].

Concerns over government surveillance and mobile device hacking measures were selected from the survey [23,28,30].

Clinical Foundation

There is no agreed-upon definition of effectiveness for measuring the clinical foundation of DCT. In January 2021, the National Institute of Standards and Technology held a workshop to discuss successes and challenges associated with implementing proximity detection technologies [31]. For public health, effectiveness is defined as reducing transmission and quicker notification of exposure. From an adopter perspective, effectiveness lies in addressing privacy concerns and finding a value proposition. From a technology perspective, effectiveness is how well the DCT apps measure proximity and if they can function at current adoption rates. A systematic review of DCT also found this lack of definition for effectiveness [18].

Also of potential concern is the “risk compensation” theory, which suggests that individuals will increase risky social distancing behavior after a COVID-19 public health intervention is implemented [32,33]. Researchers are concerned that DCT app adoption could create a false sense of security and safety, leading to behavior that increases infections [34]. Additionally, the personality of the adopter may be an issue because the extroversion personality type is associated with lower compliance with social distancing [35].

Clinical foundation measures were selected to inform future work on the definition of DCT public health effectiveness. Measures were selected to assess social distancing behavior (mask, visiting friends/family, attending religious services) [23,35]. Psychological distress that could impact adoption and measures were included to assess a user’s worry about COVID-19 [36]. Measures on motivations for adoption and interest were selected from the survey [23,24].

Usability

Grounded in the principle of autonomy, an mHealth app should enable the user to take an educated and active role in their health decisions. For example, DCT apps could help essential workers take an active role in managing their exposures. However, a German study showed that there was not greater uptake of DCT apps by people working during the early pandemic [23]. In addition, the readability of DCT apps is higher than US reading levels and potentially less accessible for the general population [27]. There is also little known about DCT use by people in good versus poor health. Studies show that older people with chronic diseases adopt self-care and vitals mHealth apps [26]; however, their chronic disease management is a very different purpose than tracking infectious disease exposure.

Usability is especially important for populations with high exposure risk or at high risk of poor outcomes. Measures were selected for essential worker, health, and health insurance status [23,24].

Data Integration Toward a Therapeutic Goal

This category is related to the principle of shared decision-making through appropriate information sharing with public health departments and health care systems. Some DCT apps do offer adopters links to testing resources, local health department resources, and quarantine social support [27]. In the US GAEN DCT systems, adopters are solely in control of reporting their positive test results through the app; therefore, designers have limited ability to automate the contact-tracing reporting data path. Unfortunately, a Morbidity and Mortality Weekly Report found that 35% to 48% of individuals with COVID-19 did not report any of their contacts to public health department contact tracers [37]. DCT apps provide anonymous COVID-19–positive lab test reporting capability, but there is no evidence showing that DCT adoption improves the likelihood of reporting.

User infection status and sharing a positive test result measures assess if apps ultimately provide exposure data to public health authorities [24,37].

Statistical Analysis

Market segments of the survey population were divided into three categories: (1) the “adopted” category includes the population self-declaring use of a DCT app; (2) the “interested-not-adopted” category includes the population self-declaring interest in, but not using a DCT app; and (3) the “not interested” category describes the remaining population. All three categories were included in the descriptive statistics table using frequencies and survey weighted percentages. Only the adopted and interested-not-adopted (“interested”) categories were included in the statistical analysis because these populations would be targets for uptake communications campaigns. Logistic regression models were applied to the adopted and interested-not-adopted segments to compare against the key measures. Regression results for the two models are presented with unweighted percentages and adjusted odds ratios (ORs) with 95% CIs. Both model results and key measures are presented in tables with variables organized by AEM categories. Health care workers (n=1029) were excluded from the analysis because health systems had contact-tracing protocols in place before the COVID-19 pandemic to meet health care–acquired infections regulations. Health care workers may answer survey questions inconsistently because they follow workplace infection contact-tracing protocols that are not applicable to the general public. Multicollinearity of adopted and interested-not-adopted models were checked through correlation analysis. Descriptive and statistical analyses were conducted using STATA version 16.1 (StataCorp, College Station, TX).

To mitigate the risk of model overfitting given the large number of measures, variables were selected a priori through a literature review and allocated to the APA framework as described in the Key Measures section above and Multimedia Appendix 2. By using measures with statistical significance from other data sets, we reduced the likelihood that our results were based on overfitting in our survey sample. Results were also analyzed for overfit risk by setting a minimum measure subsample frequency threshold (n>15) [38].

Ethical Considerations

This project was considered exempt for review by the George Mason University Institutional Review Board (1684418-3).

Results

Descriptive Analysis

Out of 3853 respondents, 2824 were included in the analysis, with 1029 excluded because they self-identified as health care workers. Among the total sample of 2824 people, 490 (17.4%) reported using a DCT app, 697 (24.7%) reported interest, and 1637 (58.0%) were not interested (Table 1). All measures exceed the minimum frequency threshold for assessing the risk of overfitting, with most of the measures having over 100 survey respondents.

Older age groups adopted the apps less frequently, where the ≥65-year-old group are at higher risk of poor COVID-19 outcomes but represent a lower rate of adoption (10.2%). Those not interested in adopting the app were disproportionately concerned with government surveillance postepidemic (43.2%).

DCT app adopters more frequently visited friends and family outside the home each week (61.6%) compared to the overall survey rate (47.6%). Those interested in the app visited friends and family outside the home less frequently (43.0%). Adopters also had high rates of concern about losing their home (43.3%), being lonely (41.2%), and feeling afraid (41.6%). Despite having a higher rate of thinking they have or had the coronavirus infection (28.6%), their rate of telling neighbors and friends was similar to that of the base population (67.4%).

There was also a difference in the rate of wearing masks in the overall study population (84.5%) versus a lower rate in the adoption segment (74.9%). Adopters showed a higher rate of very good/excellent health (54.3%) compared to the overall population and other categories.

Logistic Regression Analysis

There were differences in the frequency and association between the adopted and interested segment populations across all five AEM categories (Table 2). Covariate-adjusted ORs show how the adoption and interest vary by study subject characteristics. The OR is interpreted as the odds of a covariate being the same (OR=1.0) for adopted versus not adopted groups. For example, people who live in a state that offers a DCT adopt the apps 1.47 times more often (OR>1.0) than people in states not offering DCTs. People who wore a mask in public to help protect themselves or others from getting sick adopt DCT apps 0.52 times less often (OR<1.0) than people that do not wear masks.

Essential worker status was not associated with being in the adoption or interest segments, and education was not a significant factor. Being an adopter was negatively associated with having health insurance (16.1%; OR 0.65, 95% CI 0.48-0.89) and being interested was associated with being in fair to poor health (31.0%; OR 1.42, 95% CI 1.08-1.85). Older age was negatively associated with both adoption and interest. Having an annual income over US $100,000 was associated with both adoption and interest. Political party affiliation was not a statistically significant factor in adoption, although those self-declared as Democratic party members had the highest adoption frequency.

Not adopting a DCT app due to concerns about government surveillance (25.0%; OR 0.65, 95% CI 0.52-0.81) and that their phone will be hacked (27.7%; OR 0.78, 95% CI 0.62-0.99) was negatively associated with interest. Adoption was positively associated with visiting friends and family (22.9%; OR 1.63, 95% CI 1.28-2.09), whereas interest was negatively associated with visits (28.1%; OR 0.81, 95% CI 0.66-0.99). Religious services attendance was associated with interest (32.5%; OR 1.29, 95% CI 1.06-1.58). Adopters tended to express feeling lonely (24.9%; OR 1.31, 95% CI 1.00-1.70), while being interested was positively associated with feeling afraid (39.3%; OR 1.38, 95% CI 1.10-1.73). Motivation measures showed the highest frequencies in the adoption (38.3%-41.3%) and interest segments.

Adopters were more likely to think they have or had the coronavirus (28.6%; OR 1.60, 95% CI 1.21-2.12). Adoption was not positively associated with telling friends and neighbors about a positive COVID-19 lab result.

Discussion

Principal Findings

Overall, each layer of the hierarchical AEM framework shows barriers to the ability of DCT to effectively and safely supplement understaffed manual contact tracers and protect the public. With respect to aim 1, we found that a small percentage of the eligible population reported adoption of the app (17.4%), which is lower than estimated adoption rates in the United Kingdom (28%) [39] and New Zealand (31%) [40]. Programs to encourage adoption among those interested in a DCT app (24.7%) could increase adoption, but still fall well below the minimum target threshold of adoption. The not-interested segment is large enough (58.0%) to prevent DCT apps from reaching the population adoption minimum. Low adoption and interest rates are problematic because in order for DCT apps to be effective, high population uptake is needed alongside other control measures [18]. With respect to aim 2, inconsistent associations across prior DCT studies may be due to variable selection differences, which shows the benefit of using the AEM framework to develop study measures a priori [19]. Our results show that DCT apps may not be clinically effective in protecting adopters’ friends and family from their frequent maskless contact outside the home given low adoption rates and DCT technology false-negative detection issues. Adopter and interested segments were positively associated with motivations to protect friends and family, community, and the older population; however, this study found several areas of concern regarding beneficence.

Essential workers are at greater risk of exposure as part of their employment, but they adopted DCT apps at a low rate (23.6%). Essential workers were not associated with being part of either the adoption or interest segment, and other studies also found that they had lower intent to adopt and report positive COVID-19 lab tests [24]. Future DCT app design should consider that infection risk varies across subpopulations, and DCT solution design should focus on adoption with higher-risk subpopulations. A study on DCT app user sentiment suggested that DCT apps are largely passive in nature, and could be improved by adding features such as proactive reminder notifications and links to detailed disease-spread information [30].

The US public health department sponsorship of DCT apps was considered a positive attribute in the AEM Access and Background category; however, in the not-interested segment, this was a negative attribute due to the association with distrust of government surveillance. Other studies indicated concerns about privacy that varied by political affiliation [28], whereas concerns about postpandemic surveillance and hacking were not associated with adoption hesitancy in the interested segment. Resolving this conundrum requires further study of the not-interested segment.

DCT apps are available for free on Google Play and Apple Store, and the adoption rate is higher in the 26 US states sponsoring apps [25]. Early studies before DCTs were deployed could only survey people about their intent to adopt [24], and our findings show differences in adoption versus interest subpopulations across the AEM categories. The influence of age, gender, race, and income was found to be inconsistent across studies [24] and variation in culture is suggested to play a role [23]. Our results support previous findings that DCT adoption is associated with a younger population, but did not support the association with well-educated populations [23,26].

Safe use of DCT apps after adoption is a concern. The high volume of social visits warrants further investigation into the influence of risk compensation behavior and an extraversion personality trait. The COVID-19 pandemic has resulted in ongoing related psychological stress accompanying DCT exposure monitoring, reporting, quarantine, and isolation [41]. Adoption and interest are associated with mental health stressors, with worries of losing a home and feelings of loneliness. More research is needed to determine if there are causal links with loneliness, extroversion personality type, DCT adoption, and poor social distancing compliance [35]. Our study suggests that DCT apps should not be considered as passive public health infection reporting tools because of the psychological and behavioral associations [36]. Our findings support a prior recommendation to investigate DCT adoption psychological impacts [36].

Neither adoption nor interest was associated with telling neighbors or friends about testing positive for COVID-19. This is concerning considering that adoption was associated with adopters thinking they had been infected with the coronavirus. DCTs may not be an effective addition to manual contact tracing given that they are not associated with improving upon the rate of infected people sharing contacts with public health contact tracers. Lessons from other stigmatized diseases such as HIV may offer DCT designers insight to encourage the sharing of exposure information to slow transmission [42]. Findings from an HIV mHealth prevention and testing study recommended using evidence-based methods for app development, including links to geospatial prevention information and the ability to access self-testing kits [43]. There is a clinical trial underway for an HIV mHealth app that may produce useful information to DCT developers [44].

Limitations

Adoption and interest were measured by self-described use at the time of the survey and did not include questions about duration of use. The survey was national and did not ask the respondents which of the DCT apps they considered when answering survey questions, although most states use the same GAEN proximity detection platform. The study results occasionally varied from those reported by international DCT deployments, which suggests that local culture can play a role in adoption. The data are representative of the US adult population but not of all app adopters, and we were able to identify important differences between adopters and nonadopters. Importantly, this study is the first to create a postmarket assessment approach for DCTs, and points to the need to prioritize development of multidisciplinary teams at each step of the development, implementation, and evaluation continuum.

This study did not assess the political party control of DCT sponsoring states, which controls if a DCT is offered and how energetically it is marketed. An individual’s political party affiliation was not associated with adoption, but political party in control of a regional government is likely a different factor than individual user political affiliation.

Conclusions

With low adoption rates nationally, the DCT apps may only be useful as part of pandemic management programs in smaller, more targeted populations such as company employees, university employees and students, or health system workers and clients.

The AEM highlighted key clinical foundation issues that should be addressed in exposure contact-tracing system design. For example, DCT app adoption is associated with frequent visits with friends and family members outside the home, not wearing masks, not having health insurance, feeling lonely, and adopters thinking they have/had the coronavirus. Designing for behavioral risks requires public health and technology organizations to codevelop multidisciplinary definitions of DCT effectiveness along with measurement methods.

The APA AEM framework combined with observational data analysis is suited to assess end-user mobile app experience and can help with future study measure selection. This postmarket beneficence and safety evaluation approach using the AEM and observational data could be applied to other COVID-19 pandemic public health applications that manage disease exposure, promote personal health tracking, monitor health, and raise awareness [45].