Published on in Vol 9 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/38633, first published .
Adaptation and Utilization of a Postmarket Evaluation Model for Digital Contact Tracing Mobile Health Tools in the United States: Observational Cross-sectional Study

Adaptation and Utilization of a Postmarket Evaluation Model for Digital Contact Tracing Mobile Health Tools in the United States: Observational Cross-sectional Study

Adaptation and Utilization of a Postmarket Evaluation Model for Digital Contact Tracing Mobile Health Tools in the United States: Observational Cross-sectional Study

Authors of this article:

Kevin E Cevasco 1 Author Orcid Image ;   Amira A Roess 1 Author Orcid Image

Original Paper

College of Health and Human Services, George Mason University, Fairfax, VA, United States

Corresponding Author:

Amira A Roess, MPH, PhD

College of Health and Human Services

George Mason University

4400 University Dr

Fairfax, VA, 22030

United States

Phone: 1 703 993 1923

Email: aroess@gmu.edu


Background: Case investigation and contact tracing are core public health activities used to interrupt disease transmission. These activities are traditionally conducted manually. During periods of high COVID-19 incidence, US health departments were unable to scale up case management staff to deliver effective and timely contact-tracing services. In response, digital contact tracing (DCT) apps for mobile phones were introduced to automate these activities. DCT apps detect when other DCT users are close enough to transmit COVID-19 and enable alerts to notify users of potential disease exposure. These apps were deployed quickly during the pandemic without an opportunity to conduct experiments to determine effectiveness. However, it is unclear whether these apps can effectively supplement understaffed manual contact tracers.

Objective: The aims of this study were to (1) evaluate the effectiveness of COVID-19 DCT apps deployed in the United States during the COVID-19 pandemic and (2) determine if there is sufficient DCT adoption and interest in adoption to meet a minimum population use rate to be effective (56%). To assess uptake, interest and safe use covariates were derived from evaluating DCTs using the American Psychological Association App Evaluation Model (AEM) framework.

Methods: We analyzed data from a nationally representative survey of US adults about their COVID-19–related behaviors and experiences. Survey respondents were divided into three segments: those who adopted a DCT app, those who are interested but did not adopt, and those not interested. Descriptive statistics were used to characterize factors of the three groups. Multivariable logistic regression models were used to analyze the characteristics of segments adopting and interested in DCT apps against AEM framework covariates.

Results: An insufficient percentage of the population adopted or was interested in DCTs to achieve our minimum national target effectiveness rate (56%). A total of 17.4% (n=490) of the study population reported adopting a DCT app, 24.7% (n=697) reported interest, and 58.0% (n=1637) were not interested. Younger, high-income, and uninsured individuals were more likely to adopt a DCT app. In contrast, people in fair to poor health were interested in DCT apps but did not adopt them. App adoption was positively associated with visiting friends and family outside the home (odds ratio [OR] 1.63, 95% CI 1.28-2.09), not wearing masks (OR 0.52, 95% CI 0.38-0.71), and adopters thinking they have or had COVID-19 (OR 1.60, 95% CI 1.21-2.12).

Conclusions: Overall, a small percentage of the population adopted DCT apps. These apps may not be effective in protecting adopters’ friends and family from their maskless contacts outside the home given low adoption rates. The public health community should account for safe use behavioral factors in future public health contact-tracing app design. The AEM framework was useful in developing a study design to evaluate DCT effectiveness and safety.

JMIR Public Health Surveill 2023;9:e38633

doi:10.2196/38633

Keywords



Background

During periods of high COVID-19 incidence, US health departments were unable to scale up case management staff to deliver effective and timely contact-tracing services [1]. This is expected considering that prior to the COVID-19 outbreak, US health departments were grossly understaffed and needed a minimum of 80,000 more full-time equivalent positions to provide adequate infrastructure and a minimum package of public health services [2].

Digital contact tracing (DCT) apps are intended to supplement health department case managers by automatically and more efficiently estimating the proximity and duration of an individual’s exposure to patients diagnosed with COVID-19. A model developed to predict the impact of conventional and mobile DCT apps on the SARS-CoV-2 pandemic estimated that the disease transmission rate could be reduced by adding either contact tracers or effective widely used DCT apps (eg, Bluetooth, GPS) [3]. DCTs reduce disease spread by alerting the user if they have been near an infected person who is also using DCT, followed by user isolation and testing for infection. However, a key DCT effectiveness assumption is that there are sufficient people adopting the app to signal disease exposure. Ongoing monitoring of population uptake is necessary to address concerns that users are relying on DCT apps that have not met minimum population-level app adoption rates [4,5].

Health Technology Assessments

Health technology assessments (HTAs) of public health applications are uncommon and challenged by lack of data, conflicting stakeholder priorities, and methodological issues [6]. The surge in COVID-19 cases in the United States in the summer of 2021 (ie, the “third wave”) driven by the SARS-CoV-2 Delta variant marks an urgent time to perform an assessment of DCT apps to inform future public health contact-tracing programs and app design [7]. By 2020, an estimated 2.6 billion people globally were expected to use mobile health (mHealth) apps, generating an estimated US $31 billion in revenue [8].

Despite the large market of mHealth apps, there is no US regulatory authority in place to evaluate if DCT apps are safe and beneficial [9]. Section 522 of the Federal Food, Drug, and Cosmetic Act provides the Food and Drug Administration (FDA) with the authority to require manufacturers to conduct postmarket surveillance to address important public health questions on the effectiveness and safety of a health device [10]. However, the FDA generally does not regulate products that are intended solely to track locations or contacts associated with public health surveillance [11]. Public health DCT developers could voluntarily implement HTAs using the FDA Software Precertification Program; however, the FDA guidance does not provide a framework to conduct the postmarket evaluations [12].

There are a wide variety of frameworks available for evaluating health applications, but they may not be suitable for HTAs [13,14]. The clinical applications marketplace conducts HTAs, but they have not been frequently performed in the public health market [6]. This study adapted the American Psychological Association (APA) App Evaluation Model (AEM) hierarchical rating system because it was developed by harmonizing application evaluation questions from 45 frameworks [15]. The AEM provides a comprehensive evaluation framework for clinicians and patients to identify high-quality mHealth apps. The framework may work well at a population level with hierarchical layers to explain adoption decisions [16,17].

DCT only functions when all people near each other are using the app. Individuals install DCT apps on their own mobile devices, and these apps must be adopted by a sufficient percentage of a population to be effective [16,17]. In this study, we set the adoption effectiveness rate at 56% of the national population [18]. We define our DCT adoption as the percentage of the population using a DCT plus those that are interested in using a DCT but have not yet adopted it. Our assumption is that public health campaigns could convert many of the interested users into adopters through public health communication campaigns.

Following public health safety guidance while using DCT is important, and the AEM framework provides health and technology evaluation elements to evaluate functional use. For example, DCT requires users to voluntarily report their positive COVID-19 test results to public health authorities; therefore, user behavior is part of DCT functionality. After DCT adoption, users should not cease their social distancing behaviors in a way that increases disease exposures.

DCT represents a new technology with a short research history, and the AEM provides a structure for measure selection [19]. The AEM has five categories for evaluation: Access and Background, Privacy and Safety, Clinical Foundation, Usability, and Data Integration Toward a Therapeutic Goal. The AEM provides a way to critically assess mHealth and does not include a minimum or maximum score to be considered “good” or “useful” [15,20].

Objective

The main objective of this study was to evaluate COVID-19 DCT apps deployed in the United States during the COVID-19 pandemic under the hypothesis that DCT requires a minimal level of uptake by a population to be effective and DCT users should follow social distancing policies. Accordingly, the specific study aims were as follows.

Aim 1: Determine if there is a sufficient percentage of people using or interested in DCT to reach the minimum critical mass (56%) [18].

Aim 2: Assess the effectiveness and safety of DCT apps using the APA AEM hierarchical rating system [16].

The AEM is a conceptual framework for developing key study measures and was used to select variables. The measures derived from the framework assess adoption rate characteristics and behaviors necessary for the safe use of DCT apps.


Data Source

We approximated a nationally representative sample of US adults (aged 18+ years), with a stratified nonprobability sample to survey 3853 online panel participants between December 22, 2020, and January 2, 2021. Respondents were recruited by Climate Nexus Polling, using several market research panels as described elsewhere [21]. Participants were recruited using stratified sampling methods [21]. Compensation for participants depended on the specific market research panel and respondents’ preferences (eg, cash, gift cards, reward points) and was valued below US $4.00. Quotas were set to match the US Census Bureau’s Voting and Registration Supplement to the Current Population Survey parameters for age, gender, race, educational attainment, census region, and Hispanic ethnicity. Sampling weights were used to account for small deviations from the preselected census parameters. The participation rate of the survey was 68.5% (qualified respondents that completed the survey). The 95% credibility interval for this survey was ±1.7%. The survey data are provided in Multimedia Appendix 1.

DCT apps are new technology and there were no established and validated survey instruments to measure their use; therefore, Climate Nexus Polling developed DCT app survey questions based on a previous Pew Research survey on digital privacy [22].

Key Study Measures Aligned to the APA AEM

Model Variable Selection

Demographic measures were based on a literature review to assess if app adoption applied to a diverse range of people [23,24]. The following background for each category links the AEM to the study’s DCT postmarket evaluation. The AEM framework was used to select variables a priori from data available in the survey (Figure 1). The detailed process is described in Multimedia Appendix 2.

Figure 1. App Evaluation Model (AEM) with study variables. APA: American Psychological Association; DCT: digital contact tracing.
Access and Background

This category is concerned with the transparency of app governance and ensuring that the benefits of mHealth apps are available to a diverse range of people, regardless of background. The US DCT apps are based on mobile phone Bluetooth proximity tracking using the Google/Apple Exposure Notification (GAEN) platform [25], and are cosponsored by US state government public health organizations [25]. Adoption of DCT apps in the United States is voluntary, and varies by demographics and behavior [23]. Furthermore, DCT app adoption may be subject to health equity issues since mHealth adoption is generally associated with younger, educated populations [23,26].

The “state of residence” measure was used to demonstrate the role of a person’s state government in driving local adoption [25,27,28].

Privacy and Safety

There is an expectation of confidentiality by DCT users, and privacy policies should ensure that COVID-19 exposure proximity measurement is accurate. Concern over privacy and data breaches is a genuine concern. Experts collaborated to develop secure decentralized contract-tracing platform architectures [29]; however, some GAEN implementations may have exposed user data. A class action lawsuit was filed in April of 2021 against Google, claiming that their COVID-19 contact-tracing platform exposed unwitting Android adopters’ sensitive personal information [30].

Concerns over government surveillance and mobile device hacking measures were selected from the survey [23,28,30].

Clinical Foundation

There is no agreed-upon definition of effectiveness for measuring the clinical foundation of DCT. In January 2021, the National Institute of Standards and Technology held a workshop to discuss successes and challenges associated with implementing proximity detection technologies [31]. For public health, effectiveness is defined as reducing transmission and quicker notification of exposure. From an adopter perspective, effectiveness lies in addressing privacy concerns and finding a value proposition. From a technology perspective, effectiveness is how well the DCT apps measure proximity and if they can function at current adoption rates. A systematic review of DCT also found this lack of definition for effectiveness [18].

Also of potential concern is the “risk compensation” theory, which suggests that individuals will increase risky social distancing behavior after a COVID-19 public health intervention is implemented [32,33]. Researchers are concerned that DCT app adoption could create a false sense of security and safety, leading to behavior that increases infections [34]. Additionally, the personality of the adopter may be an issue because the extroversion personality type is associated with lower compliance with social distancing [35].

Clinical foundation measures were selected to inform future work on the definition of DCT public health effectiveness. Measures were selected to assess social distancing behavior (mask, visiting friends/family, attending religious services) [23,35]. Psychological distress that could impact adoption and measures were included to assess a user’s worry about COVID-19 [36]. Measures on motivations for adoption and interest were selected from the survey [23,24].

Usability

Grounded in the principle of autonomy, an mHealth app should enable the user to take an educated and active role in their health decisions. For example, DCT apps could help essential workers take an active role in managing their exposures. However, a German study showed that there was not greater uptake of DCT apps by people working during the early pandemic [23]. In addition, the readability of DCT apps is higher than US reading levels and potentially less accessible for the general population [27]. There is also little known about DCT use by people in good versus poor health. Studies show that older people with chronic diseases adopt self-care and vitals mHealth apps [26]; however, their chronic disease management is a very different purpose than tracking infectious disease exposure.

Usability is especially important for populations with high exposure risk or at high risk of poor outcomes. Measures were selected for essential worker, health, and health insurance status [23,24].

Data Integration Toward a Therapeutic Goal

This category is related to the principle of shared decision-making through appropriate information sharing with public health departments and health care systems. Some DCT apps do offer adopters links to testing resources, local health department resources, and quarantine social support [27]. In the US GAEN DCT systems, adopters are solely in control of reporting their positive test results through the app; therefore, designers have limited ability to automate the contact-tracing reporting data path. Unfortunately, a Morbidity and Mortality Weekly Report found that 35% to 48% of individuals with COVID-19 did not report any of their contacts to public health department contact tracers [37]. DCT apps provide anonymous COVID-19–positive lab test reporting capability, but there is no evidence showing that DCT adoption improves the likelihood of reporting.

User infection status and sharing a positive test result measures assess if apps ultimately provide exposure data to public health authorities [24,37].

Statistical Analysis

Market segments of the survey population were divided into three categories: (1) the “adopted” category includes the population self-declaring use of a DCT app; (2) the “interested-not-adopted” category includes the population self-declaring interest in, but not using a DCT app; and (3) the “not interested” category describes the remaining population. All three categories were included in the descriptive statistics table using frequencies and survey weighted percentages. Only the adopted and interested-not-adopted (“interested”) categories were included in the statistical analysis because these populations would be targets for uptake communications campaigns. Logistic regression models were applied to the adopted and interested-not-adopted segments to compare against the key measures. Regression results for the two models are presented with unweighted percentages and adjusted odds ratios (ORs) with 95% CIs. Both model results and key measures are presented in tables with variables organized by AEM categories. Health care workers (n=1029) were excluded from the analysis because health systems had contact-tracing protocols in place before the COVID-19 pandemic to meet health care–acquired infections regulations. Health care workers may answer survey questions inconsistently because they follow workplace infection contact-tracing protocols that are not applicable to the general public. Multicollinearity of adopted and interested-not-adopted models were checked through correlation analysis. Descriptive and statistical analyses were conducted using STATA version 16.1 (StataCorp, College Station, TX).

To mitigate the risk of model overfitting given the large number of measures, variables were selected a priori through a literature review and allocated to the APA framework as described in the Key Measures section above and Multimedia Appendix 2. By using measures with statistical significance from other data sets, we reduced the likelihood that our results were based on overfitting in our survey sample. Results were also analyzed for overfit risk by setting a minimum measure subsample frequency threshold (n>15) [38].

Ethical Considerations

This project was considered exempt for review by the George Mason University Institutional Review Board (1684418-3).


Descriptive Analysis

Out of 3853 respondents, 2824 were included in the analysis, with 1029 excluded because they self-identified as health care workers. Among the total sample of 2824 people, 490 (17.4%) reported using a DCT app, 697 (24.7%) reported interest, and 1637 (58.0%) were not interested (Table 1). All measures exceed the minimum frequency threshold for assessing the risk of overfitting, with most of the measures having over 100 survey respondents.

Older age groups adopted the apps less frequently, where the ≥65-year-old group are at higher risk of poor COVID-19 outcomes but represent a lower rate of adoption (10.2%). Those not interested in adopting the app were disproportionately concerned with government surveillance postepidemic (43.2%).

Table 1. Frequencies of survey population and survey segments.
Survey segmentStudy population (N=2824), n (wt%a)Adopted (n=490), n (%)Interested-not-adopted (n=697), n (%)Not interested (n=1637), n (%)
Access and background: State of residence offers a DCTb app (yes)1534 (55.3)310 (63.3)376 (54.0)848 (51.8)
Security and privacy

Do not use because of concerns with government surveillance postepidemicN/AcN/A236 (33.9)707 (43.2)

Do not use because my phone might get hackedN/AN/A155 (22.2)405 (24.7)
Clinical foundation

Social distancing


Visit friends and family weekly (yes)1321 (47.6)302 (61.6)286 (43.0)733 (45.1)


Worn a mask in public to help protect yourself or others from getting sick? (yes)2409 (84.5)367 (74.9)611 (87.7)1431 (87.4)


Attends religious services (yes)1309 (46.5)285 (58.2)333 (47.8)691 (42.2)

Feelings


Worried about losing your home (yes)784 (28.5)212 (43.3)215 (30.9)357 (21.8)


Feel afraid (yes)893 (31.9)204 (41.6)271 (38.9)418 (25.5)


Feel lonely (yes)812 (28.6)202 (41.2)230 (33.0)380 (23.2)

Motivation for use


Use to protect family and friendsN/A279 (56.9)430 (61.7)N/A


Use to help stop the pandemic/reduce deaths in older American adultsN/A229 (46.7)326 (46.8)N/A


Use as a responsibility to my communityN/A219 (44.7)320 (45.9)N/A


Use to let me know my risk of infection/help peace of mind/help me stay healthyN/A268 (54.7)432 (62.0)N/A


Not use because it would make me feel more anxiousN/AN/A227 (32.6)552 (33.7)


Not use because I would not benefit/I won’t be infectedN/AN/A122 (17.5)446 (27.2)
Usability

Essential worker664 (24.5)157 (32.0)173 (24.8)334 (20.4)

Health status


Very good or excellent1288 (45.7)266 (54.3)317 (45.5)705 (43.1)


Good967 (34.2)151 (30.8)226 (32.4)590 (36.0)


Fair or poor569 (20.2)73 (14.9)154 (22.1)342 (20.9)


Has health insurance (yes)2421 (85.3)389 (79.4)616 (88.4)1416 (86.5)

No use because too much hassle to installN/AN/A187 (26.8)389 (23.8)
Therapeutic goal

Would you tell your neighbors or friends if you become COVID-19–positive? (yes)1949 (69.1)330 (67.4)502 (72.0)1117 (68.2)

Do you think you have already had the coronavirus or currently have it? (yes)490 (18.2)140 (28.6)115 (16.5)235 (14.4)
Demographics

Age (Gallup divisions, years)


18-29510 (19.0)155 (31.6)144 (20.7)211 (12.9)


30-49948 (33.2)214 (43.7)283 (40.6)451 (27.6)


50-64686 (24.2)71 (14.5)154 (22.1)461 (28.2)


65+680 (23.6)50 (10.2)116 (16.6)514 (31.4)

Gender


Female1491 (51.7)245 (50.0)393 (56.4)853 (52.1)


Male1333 (48.3)245 (50.0)304 (43.6)784 (47.9)

Race


White2126 (72.1)319 (65.1)506 (72.6)1301 (79.5)


Black322 (10.2)69 (14.1)84 (12.1)169 (10.3)


Hispanic232 (14.1)68 (13.9)68 (9.8)96 (5.9)


Other144 (3.7)34 (6.9)39 (5.6)71 (4.3)

Political party affiliation


Republican1066 (38.1)154 (31.4)205 (29.4)707 (43.2)


Democratic1273 (44.5)269 (54.9)383 (55.0)621 (37.9)


Independent485 (17.4)67 (13.7)109 (15.6)309 (18.9)

Annual income level (US $)


<50,0001606 (57.4)244 (49.8)366 (52.5)996 (60.8)


50,000-99,000778 (27.4)137 (28.0)201 (28.8)440 (26.9)


≥100,000440 (15.2)109 (22.2)130 (18.7)201 (12.3)

Population density


Urban814 (29.5)185 (37.8)236 (33.9)393 (24.0)


Semiurban1326 (46.8)218 (44.5)322 (46.2)786 (48.0)


Rural684 (23.8)87 (17.8)139 (19.9)458 (28.0)

Education


No high school176 (6.7)38 (7.8)44 (6.3)94 (5.7)


At least some high school718 (26.4)107 (21.8)147 (21.1)464 (28.3)


Some college1025 (37.7)158 (32.2)246 (35.3)621 (37.9)


Bachelor’s degree or above905 (29.3)187 (38.2)260 (37.3)458 (28.0)

awt%: weighted percentage.

bDCT: digital contact tracing.

cN/A: not applicable.

DCT app adopters more frequently visited friends and family outside the home each week (61.6%) compared to the overall survey rate (47.6%). Those interested in the app visited friends and family outside the home less frequently (43.0%). Adopters also had high rates of concern about losing their home (43.3%), being lonely (41.2%), and feeling afraid (41.6%). Despite having a higher rate of thinking they have or had the coronavirus infection (28.6%), their rate of telling neighbors and friends was similar to that of the base population (67.4%).

There was also a difference in the rate of wearing masks in the overall study population (84.5%) versus a lower rate in the adoption segment (74.9%). Adopters showed a higher rate of very good/excellent health (54.3%) compared to the overall population and other categories.

Logistic Regression Analysis

There were differences in the frequency and association between the adopted and interested segment populations across all five AEM categories (Table 2). Covariate-adjusted ORs show how the adoption and interest vary by study subject characteristics. The OR is interpreted as the odds of a covariate being the same (OR=1.0) for adopted versus not adopted groups. For example, people who live in a state that offers a DCT adopt the apps 1.47 times more often (OR>1.0) than people in states not offering DCTs. People who wore a mask in public to help protect themselves or others from getting sick adopt DCT apps 0.52 times less often (OR<1.0) than people that do not wear masks.

Table 2. Logistic regression results for adopted and interested-not-adopted segments.
Survey segmentsAdoptedInterested-not-adopted

Respondents (n=2824), %aAdjusted ORb (95% CI) (n=490)Respondents (n=2334), %aAdjusted OR (95% CI) (n=697)
Access and background: state of residence offers a DCTc app (yes)63.3d1.47 (1.16-1.86)53.90.98 (0.81-1.19)
Security and privacy

Do not use because of concerns with government surveillance postepidemicN/AeN/A25.00.67 (0.54-0.83)

Do not use because my phone might get hackedN/AN/A27.70.78 (0.62-0.99)
Clinical foundation

Social distancing


Visit friends and family weekly (yes)22.91.63 (1.28-2.09)28.10.81 (0.66-0.99)


Worn a mask in public to help protect yourself or others from getting sick? (yes)15.20.52 (0.38-0.71)29.91.05 (0.77-1.44)


Attends religious services (Yes)21.81.26 (0.99-1.61)32.51.29 (1.06-1.58)

Feelings


Worried about losing your home (yes)27.01.52 (1.18-1.95)37.61.19 (0.94-1.49)


Feel afraid (yes)22.81.04 (0.80-1.35)39.31.38 (1.10-1.73)


Feel lonely (yes)24.91.31 (1.00-1.70)37.71.14 (0.90-1.44)

Motivation for use


Use to protect family and friends39.42.65 (2.00-3.51)61.7f


Use to help stop the pandemic/reduce deaths in older American adults41.32.04 (1.52-2.73)46.8


Use as a responsibility to my community40.61.49 (1.10-2.03)45.9


Use to let me know my risk of infection/help peace of mind/help me stay healthy38.32.09 (1.57-2.78)62.0


Do not use because it would make me feel more anxiousN/AN/A29.10.59 (0.47-0.73)


Do not use because I would not benefit/I won’t be infectedN/AN/A21.51.14 (0.90-1.44)
Usability

Essential worker (yes)23.61.12 (0.87-1.46)34.11.06 (0.84-1.34)

Health status


Very good or excellent20.7131.01


Good15.60.93 (0.72-1.22)27.71.10 (0.88-1.38)


Fair or poor12.80.73 (0.52-1.03)31.01.42 (1.08-1.85)


Has health insurance (yes)16.10.65 (0.48-0.89)30.31.24 (0.92-1.68)

Do not use because too much hassle to installN/AN/A32.50.93 (0.73-1.17)
Therapeutic goal

Would you tell your neighbors or friends if you become COVID-19–positive? (yes)67.40.84 (0.65-1.08)72.01.20 (0.97-1.50)

Do you think you have already had the coronavirus or currently have it? (yes)28.61.60 (1.21-2.12)32.90.96 (0.74-1.26)
Demographics

Age (years)


18-2930.4140.61


30-4922.60.93 (0.69-1.25)38.60.86 (0.65-1.15)


50-6410.30.69 (0.47-1.00)25.00.47 (0.34-0.64)


65+7.40.63 (0.41-0.98)18.40.30 (0.21-0.43)

Gender: male18.41.14 (0.90-1.44)27.90.89 (0.73-1.08)

Race


White15.0128.01


Black21.41.06 (0.74-1.53)33.20.87 (0.63-1.20)


Hispanic29.31.25 (0.86-1.83)41.51.30 (0.90-1.87)


Other23.61.45 (0.90-2.34)35.51.01 (0.65-1.57)

Political party affiliation


Republican14.4122.51


Democratic21.11.05 (0.80-1.39)38.11.62 (1.29-2.03)


Independent13.80.85 (0.59-1.22)26.10.97 (0.73-1.30)

Annual income level (US $)


<50,00015.2126.91


50,000-99,00017.61.10 (0.82-1.47)31.41.13 (0.89-1.43)


≥100,00024.81.45 (1.02-2.06)39.31.42 (1.05-1.91)

Population density


Urban22.7137.51


Semiurban16.40.98 (0.75-1.28)29.10.86 (0.68-1.08)


Rural12.70.93 (0.66-1.31)23.30.73 (0.55-0.96)

Education


No high school21.6131.91


Some high school14.91.02 (0.62-1.67)24.10.76 (0.49-1.16)


Some college15.40.86 (0.53-1.40)28.40.96 (0.63-1.46)


Bachelor’s degree and above20.70.92 (0.55-1.53)36.21.24 (0.80-1.93)

aFrequency is presented as a percentage of adopted or interested-not-adopted against the total population of each variable.

bOR: odds ratio.

cDCT: digital contact tracing.

dStatistically significant associations are in italics.

eN/A: not applicable.

fRemoved from model due to collinearity with being in the interested-not-adopted segment.

Essential worker status was not associated with being in the adoption or interest segments, and education was not a significant factor. Being an adopter was negatively associated with having health insurance (16.1%; OR 0.65, 95% CI 0.48-0.89) and being interested was associated with being in fair to poor health (31.0%; OR 1.42, 95% CI 1.08-1.85). Older age was negatively associated with both adoption and interest. Having an annual income over US $100,000 was associated with both adoption and interest. Political party affiliation was not a statistically significant factor in adoption, although those self-declared as Democratic party members had the highest adoption frequency.

Not adopting a DCT app due to concerns about government surveillance (25.0%; OR 0.65, 95% CI 0.52-0.81) and that their phone will be hacked (27.7%; OR 0.78, 95% CI 0.62-0.99) was negatively associated with interest. Adoption was positively associated with visiting friends and family (22.9%; OR 1.63, 95% CI 1.28-2.09), whereas interest was negatively associated with visits (28.1%; OR 0.81, 95% CI 0.66-0.99). Religious services attendance was associated with interest (32.5%; OR 1.29, 95% CI 1.06-1.58). Adopters tended to express feeling lonely (24.9%; OR 1.31, 95% CI 1.00-1.70), while being interested was positively associated with feeling afraid (39.3%; OR 1.38, 95% CI 1.10-1.73). Motivation measures showed the highest frequencies in the adoption (38.3%-41.3%) and interest segments.

Adopters were more likely to think they have or had the coronavirus (28.6%; OR 1.60, 95% CI 1.21-2.12). Adoption was not positively associated with telling friends and neighbors about a positive COVID-19 lab result.


Principal Findings

Overall, each layer of the hierarchical AEM framework shows barriers to the ability of DCT to effectively and safely supplement understaffed manual contact tracers and protect the public. With respect to aim 1, we found that a small percentage of the eligible population reported adoption of the app (17.4%), which is lower than estimated adoption rates in the United Kingdom (28%) [39] and New Zealand (31%) [40]. Programs to encourage adoption among those interested in a DCT app (24.7%) could increase adoption, but still fall well below the minimum target threshold of adoption. The not-interested segment is large enough (58.0%) to prevent DCT apps from reaching the population adoption minimum. Low adoption and interest rates are problematic because in order for DCT apps to be effective, high population uptake is needed alongside other control measures [18]. With respect to aim 2, inconsistent associations across prior DCT studies may be due to variable selection differences, which shows the benefit of using the AEM framework to develop study measures a priori [19]. Our results show that DCT apps may not be clinically effective in protecting adopters’ friends and family from their frequent maskless contact outside the home given low adoption rates and DCT technology false-negative detection issues. Adopter and interested segments were positively associated with motivations to protect friends and family, community, and the older population; however, this study found several areas of concern regarding beneficence.

Essential workers are at greater risk of exposure as part of their employment, but they adopted DCT apps at a low rate (23.6%). Essential workers were not associated with being part of either the adoption or interest segment, and other studies also found that they had lower intent to adopt and report positive COVID-19 lab tests [24]. Future DCT app design should consider that infection risk varies across subpopulations, and DCT solution design should focus on adoption with higher-risk subpopulations. A study on DCT app user sentiment suggested that DCT apps are largely passive in nature, and could be improved by adding features such as proactive reminder notifications and links to detailed disease-spread information [30].

The US public health department sponsorship of DCT apps was considered a positive attribute in the AEM Access and Background category; however, in the not-interested segment, this was a negative attribute due to the association with distrust of government surveillance. Other studies indicated concerns about privacy that varied by political affiliation [28], whereas concerns about postpandemic surveillance and hacking were not associated with adoption hesitancy in the interested segment. Resolving this conundrum requires further study of the not-interested segment.

DCT apps are available for free on Google Play and Apple Store, and the adoption rate is higher in the 26 US states sponsoring apps [25]. Early studies before DCTs were deployed could only survey people about their intent to adopt [24], and our findings show differences in adoption versus interest subpopulations across the AEM categories. The influence of age, gender, race, and income was found to be inconsistent across studies [24] and variation in culture is suggested to play a role [23]. Our results support previous findings that DCT adoption is associated with a younger population, but did not support the association with well-educated populations [23,26].

Safe use of DCT apps after adoption is a concern. The high volume of social visits warrants further investigation into the influence of risk compensation behavior and an extraversion personality trait. The COVID-19 pandemic has resulted in ongoing related psychological stress accompanying DCT exposure monitoring, reporting, quarantine, and isolation [41]. Adoption and interest are associated with mental health stressors, with worries of losing a home and feelings of loneliness. More research is needed to determine if there are causal links with loneliness, extroversion personality type, DCT adoption, and poor social distancing compliance [35]. Our study suggests that DCT apps should not be considered as passive public health infection reporting tools because of the psychological and behavioral associations [36]. Our findings support a prior recommendation to investigate DCT adoption psychological impacts [36].

Neither adoption nor interest was associated with telling neighbors or friends about testing positive for COVID-19. This is concerning considering that adoption was associated with adopters thinking they had been infected with the coronavirus. DCTs may not be an effective addition to manual contact tracing given that they are not associated with improving upon the rate of infected people sharing contacts with public health contact tracers. Lessons from other stigmatized diseases such as HIV may offer DCT designers insight to encourage the sharing of exposure information to slow transmission [42]. Findings from an HIV mHealth prevention and testing study recommended using evidence-based methods for app development, including links to geospatial prevention information and the ability to access self-testing kits [43]. There is a clinical trial underway for an HIV mHealth app that may produce useful information to DCT developers [44].

Limitations

Adoption and interest were measured by self-described use at the time of the survey and did not include questions about duration of use. The survey was national and did not ask the respondents which of the DCT apps they considered when answering survey questions, although most states use the same GAEN proximity detection platform. The study results occasionally varied from those reported by international DCT deployments, which suggests that local culture can play a role in adoption. The data are representative of the US adult population but not of all app adopters, and we were able to identify important differences between adopters and nonadopters. Importantly, this study is the first to create a postmarket assessment approach for DCTs, and points to the need to prioritize development of multidisciplinary teams at each step of the development, implementation, and evaluation continuum.

This study did not assess the political party control of DCT sponsoring states, which controls if a DCT is offered and how energetically it is marketed. An individual’s political party affiliation was not associated with adoption, but political party in control of a regional government is likely a different factor than individual user political affiliation.

Conclusions

With low adoption rates nationally, the DCT apps may only be useful as part of pandemic management programs in smaller, more targeted populations such as company employees, university employees and students, or health system workers and clients.

The AEM highlighted key clinical foundation issues that should be addressed in exposure contact-tracing system design. For example, DCT app adoption is associated with frequent visits with friends and family members outside the home, not wearing masks, not having health insurance, feeling lonely, and adopters thinking they have/had the coronavirus. Designing for behavioral risks requires public health and technology organizations to codevelop multidisciplinary definitions of DCT effectiveness along with measurement methods.

The APA AEM framework combined with observational data analysis is suited to assess end-user mobile app experience and can help with future study measure selection. This postmarket beneficence and safety evaluation approach using the AEM and observational data could be applied to other COVID-19 pandemic public health applications that manage disease exposure, promote personal health tracking, monitor health, and raise awareness [45].

Acknowledgments

The authors wish to thank our colleagues at Climate Nexus for their support, including James Wyatt, Molly Fisch-Friedman, and Reece Rushing, and Dr. Gary Kreps for his insights on health communication. The authors would like to thank the Institute for Biohealth Innovation at George Mason University and the TD Charitable Foundation for financial support.

Data Availability

A subset of survey data that were used in this study and the data dictionary are available in Multimedia Appendix 1.

Authors' Contributions

KEC and AAR had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: KEC. Acquisition, analysis, and interpretation of data: KEC, AAR. Drafting of the manuscript: KEC. Critical revision of the manuscript for important intellectual content: AAR. Statistical analysis: KEC. Obtained funding: AAR. Study supervision: AAR.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Survey data used in a postmarket evaluation model for digital contract tracing (January 2021).

XLSX File (Microsoft Excel File), 496 KB

Multimedia Appendix 2

App evaluation model variable selection process.

DOCX File , 24 KB

  1. Spencer KD, Chung CL, Stargel A, Shultz A, Thorpe PG, Carter MW, et al. COVID-19 case investigation and contact tracing efforts from health departments - United States, June 25-July 24, 2020. MMWR Morb Mortal Wkly Rep 2021 Jan 22;70(3):83-87. [CrossRef] [Medline]
  2. Staffing Up: workforce levels needed to provide basic public health services for all Americans. de Beaumont Foundation. 2021 Oct.   URL: https://debeaumont.org/wp-content/uploads/2021/10/Staffing-Up-FINAL.pdf [accessed 2022-12-18]
  3. Kretzschmar ME, Rozhnova G, Bootsma MCJ, van Boven M, van de Wijgert JHHM, Bonten MJM. Impact of delays on effectiveness of contact tracing strategies for COVID-19: a modelling study. Lancet Public Health 2020 Aug;5(8):e452-e459 [FREE Full text] [CrossRef] [Medline]
  4. Morley J, Cowls J, Taddeo M, Floridi L. Ethical guidelines for COVID-19 tracing apps. Nature 2020 Jun 28;582(7810):29-31. [CrossRef] [Medline]
  5. Trang S, Trenz M, Weiger WH, Tarafdar M, Cheung CM. One app to trace them all? Examining app specifications for mass acceptance of contact-tracing apps. Eur J Inf Syst 2020 Jul 27;29(4):415-428. [CrossRef]
  6. Stojanovic J, Wübbeler M, Geis S, Reviriego E, Gutiérrez-Ibarluzea I, Lenoir-Wijnkoop I. Evaluating public health interventions: a neglected area in health technology assessment. Front Public Health 2020 Apr 22;8:106 [FREE Full text] [CrossRef] [Medline]
  7. Tarricone R, Petracca F, Ciani O, Cucciniello M. Distinguishing features in the assessment of mHealth apps. Expert Rev Pharmacoecon Outcomes Res 2021 Aug 04;21(4):521-526. [CrossRef] [Medline]
  8. mHealth app developer economics 2016: the current status and trends of the mHealth app market. Research2Guidance. 2016.   URL: https://research2guidance.com/product/mhealth-app-developer-economics-2016/ [accessed 2022-12-18]
  9. Shuren J, Patel B, Gottlieb S. FDA regulation of mobile medical apps. JAMA 2018 Jul 24;320(4):337-338. [CrossRef] [Medline]
  10. Postmarket surveillance under Section 522 of the Federal Food, Drug, and Cosmetic Act. US Food and Drug Administration. 2022.   URL: https://www.fda.gov/media/81015/download [accessed 2022-12-18]
  11. Digital health policies and public health solutions for COVID-19. US Food and Drug Administration. 2022 Apr 28.   URL: https:/​/www.​fda.gov/​medical-devices/​coronavirus-covid-19-and-medical-devices/​digital-health-policies-and-public-health-solutions-covid-19 [accessed 2022-12-18]
  12. Developing a software precertification program: a working model v1. US Food and Drug Administration. 2019 Jan.   URL: https://www.fda.gov/media/119722/download [accessed 2022-12-18]
  13. Hensher M, Cooper P, Dona S, Angeles M, Nguyen D, Heynsbergh N, et al. Scoping review: development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers. J Am Med Inform Assoc 2021 Jun 12;28(6):1318-1329 [FREE Full text] [CrossRef] [Medline]
  14. Moshi MR, Tooher R, Merlin T. Suitability of current evaluation frameworks for use in the health technology assessment of mobile medical applications: a systematic review. Int J Technol Assess Health Care 2018 Jan 11;34(5):464-475. [CrossRef] [Medline]
  15. App Advisor: An American Psychiatric Association Initiative. The App Evaluation Model. American Psychiatric Association.   URL: https://www.psychiatry.org/psychiatrists/practice/mental-health-apps/the-app-evaluation-model [accessed 2022-12-18]
  16. Henson P, David G, Albright K, Torous J. Deriving a practical framework for the evaluation of health apps. Lancet Digit Health 2019 Jun;1(2):e52-e54 [FREE Full text] [CrossRef] [Medline]
  17. Torous J, Andersson G, Bertagnoli A, Christensen H, Cuijpers P, Firth J, et al. Towards a consensus around standards for smartphone apps and digital mental health. World Psychiatry 2019 Feb;18(1):97-98 [FREE Full text] [CrossRef] [Medline]
  18. Braithwaite I, Callender T, Bullock M, Aldridge RW. Automated and partly automated contact tracing: a systematic review to inform the control of COVID-19. Lancet Digit Health 2020 Nov;2(11):e607-e621 [FREE Full text] [CrossRef] [Medline]
  19. Hafermann L, Becher H, Herrmann C, Klein N, Heinze G, Rauch G. Statistical model building: background "knowledge" based on inappropriate preselection causes misspecification. BMC Med Res Methodol 2021 Sep 29;21(1):196 [FREE Full text] [CrossRef] [Medline]
  20. Lagan S, Aquino P, Emerson MR, Fortuna K, Walker R, Torous J. Actionable health app evaluation: translating expert frameworks into objective metrics. NPJ Digit Med 2020;3:100 [FREE Full text] [CrossRef] [Medline]
  21. Goldberg M, Gustafson A, Maibach E, Ballew M, Bergquist P, Kotcher J, et al. Mask-wearing increased after a government recommendation: a natural experiment in the US during the COVID-19 pandemic. Front Commun 2020 Jun 17;5:5. [CrossRef]
  22. Auxier B. How Americans see digital privacy issues amid the COVID-19 outbreak. Pew Research Center. 2020 May 04.   URL: https://pewrsr.ch/2SCKgoW [accessed 2020-03-01]
  23. Munzert S, Selb P, Gohdes A, Stoetzer LF, Lowe W. Tracking and promoting the usage of a COVID-19 contact tracing app. Nat Hum Behav 2021 Feb;5(2):247-255. [CrossRef] [Medline]
  24. Li T, Cobb C, Yang J, Baviskar S, Agarwal Y, Li B, et al. What makes people install a COVID-19 contact-tracing app? Understanding the influence of app design and individual difference on contact-tracing app adoption intention. Pervasive Mob Comput 2021 Aug;75:101439 [FREE Full text] [CrossRef] [Medline]
  25. Johnson B. The Covid Tracing Tracker: What's happening in coronavirus apps around the world. MIT Technology Review. 2020 Dec 16.   URL: https://www.technologyreview.com/2020/12/16/1014878/covid-tracing-tracker/ [accessed 2022-12-18]
  26. Bol N, Helberger N, Weert JCM. Differences in mobile health app use: a source of new digital inequalities? Inf Soc 2018 Apr 26;34(3):183-193. [CrossRef] [Medline]
  27. Blacklow S, Lisker S, Ng M, Sarkar U, Lyles C. Usability, inclusivity, and content evaluation of COVID-19 contact tracing apps in the United States. J Am Med Inform Assoc 2021 Aug 13;28(9):1982-1989 [FREE Full text] [CrossRef] [Medline]
  28. Zhang B, Kreps S, McMurry N, McCain RM. Americans' perceptions of privacy and surveillance in the COVID-19 pandemic. PLoS One 2020 Dec 23;15(12):e0242652 [FREE Full text] [CrossRef] [Medline]
  29. Rivest R, Canetti R. The PACT protocol specifications. 2020.   URL: https://pact.mit.edu/wp-content/uploads/2020/04/The-PACT-protocol-specification-ver-0.1.pdf [accessed 2021-09-19]
  30. Mitchell H. Lawsuit alleges Google left millions of users' contact tracing data public. Beckers Hospital Review. 2020 Apr.   URL: https:/​/www.​beckershospitalreview.com/​cybersecurity/​lawsuit-alleges-google-left-millions-of-users-contact-tracing-data-public.​html [accessed 2022-12-18]
  31. Stephens M, Cala G, Greene K, Keenan K, Robinson A, Shi L, et al. Challenges for digital proximity detection in pandemics: Privacy, accuracy, and impact. 2001 Presented at: National Institute of Standards and Technology Workshop; January 26-28, 2001; online. [CrossRef]
  32. Hedlund J. Risky business: safety regulations, risks compensation, and individual behavior. Inj Prev 2000 Jun;6(2):82-90 [FREE Full text] [CrossRef] [Medline]
  33. Yan Y, Bayham J, Richter A, Fenichel EP. Risk compensation and face mask mandates during the COVID-19 pandemic. Sci Rep 2021 Feb 04;11(1):3174. [CrossRef] [Medline]
  34. Vandamme AM, Nguyen T, Denis M, Depypere M, Kiekens A, Naesens L, on behalf of the Coronavirus Pandemic Preparedness team. Factors that should shape the decision-making to deploy contact tracing apps for pandemic containment measures. KU Leuven Institute for the Future. 2020.   URL: https://rega.kuleuven.be/if/tracing-tools-for-pandemics [accessed 2022-12-18]
  35. Carvalho LDF, Pianowski G, Gonçalves AP. Personality differences and COVID-19: are extroversion and conscientiousness personality traits associated with engagement with containment measures? Trends Psychiatry Psychother 2020 Jun;42(2):179-184. [CrossRef] [Medline]
  36. Kawakami N, Sasaki N, Kuroda R, Tsuno K, Imamura K. The effects of downloading a government-issued COVID-19 Contact tracing app on psychological distress during the pandemic among employed adults: prospective study. JMIR Ment Health 2021 Jan 12;8(1):e23699 [FREE Full text] [CrossRef] [Medline]
  37. Lash RR, Donovan CV, Fleischauer AT, Moore ZS, Harris G, Hayes S, Contact Tracing Assessment Team, et al. COVID-19 contact tracing in two counties - North Carolina, June-July 2020. MMWR Morb Mortal Wkly Rep 2020 Sep 25;69(38):1360-1363. [CrossRef] [Medline]
  38. Babyak MA. What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosom Med 2004 May 01;66(3):411-421. [CrossRef] [Medline]
  39. Lewis D. Contact-tracing apps help reduce COVID infections, data suggest. Nature 2021 Mar;591(7848):18-19. [CrossRef] [Medline]
  40. Gasteiger N, Gasteiger C, Vedhara K, Broadbent E. The more the merrier! Barriers and facilitators to the general public's use of a COVID-19 contact tracing app in New Zealand. Inform Health Soc Care 2022 Apr 03;47(2):132-143. [CrossRef] [Medline]
  41. Lohiniva A, Dub T, Hagberg L, Nohynek H. Learning about COVID-19-related stigma, quarantine and isolation experiences in Finland. PLoS One 2021 Apr 14;16(4):e0247962 [FREE Full text] [CrossRef] [Medline]
  42. Eaton LA, Kalichman SC. Social and behavioral health responses to COVID-19: lessons learned from four decades of an HIV pandemic. J Behav Med 2020 Jun 25;43(3):341-345 [FREE Full text] [CrossRef] [Medline]
  43. Biello KB, Hill-Rorie J, Valente PK, Futterman D, Sullivan PS, Hightow-Weidman L, et al. Development and evaluation of a mobile app designed to increase HIV testing and pre-exposure prophylaxis use among young men who have sex with men in the United States: open pilot trial. J Med Internet Res 2021 Mar 24;23(3):e25107 [FREE Full text] [CrossRef] [Medline]
  44. Liu A, Coleman K, Bojan K, Serrano PA, Oyedele T, Garcia A, et al. Developing a mobile app (LYNX) to support linkage to HIV/sexually transmitted infection testing and pre-exposure prophylaxis for young men who have sex with men: protocol for a randomized controlled trial. JMIR Res Protoc 2019 Jan 25;8(1):e10659 [FREE Full text] [CrossRef] [Medline]
  45. Almalki M, Giannicchi A. Health apps for combating COVID-19: descriptive review and taxonomy. JMIR Mhealth Uhealth 2021 Mar 02;9(3):e24322 [FREE Full text] [CrossRef] [Medline]


AEM: App Evaluation Model
APA: American Psychological Association
DCT: digital contact tracing
FDA: Food and Drug Administration
GAEN: Google/Apple exposure notification
HTA: health technology assessment
mHealth: mobile health
OR: odds ratio


Edited by T Sanchez, A Mavragani; submitted 10.04.22; peer-reviewed by A Teles, M Cozad; comments to author 14.12.22; revised version received 13.01.23; accepted 19.01.23; published 22.03.23

Copyright

©Kevin E Cevasco, Amira A Roess. Originally published in JMIR Public Health and Surveillance (https://publichealth.jmir.org), 22.03.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on https://publichealth.jmir.org, as well as this copyright and license information must be included.