This paper is in the following e-collection/theme issue:

Vaccination and Immunization in the Digital Age (401) Mobile Health (mhealth) (3158) mHealth for Wellness, Behavior Change and Prevention (3099) Use and User Demographics of mHealth (365) Demographics of Users, Social & Digital Divide (834)

Published on in Vol 11 (2025)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/67370, first published .
Greater Improvements in Vaccination Outcomes Among Black Young Adults With Vaccine-Resistant Attitudes in the United States South Following a Digital Health Intervention: Latent Profile Analysis of a Randomized Control Trial

Greater Improvements in Vaccination Outcomes Among Black Young Adults With Vaccine-Resistant Attitudes in the United States South Following a Digital Health Intervention: Latent Profile Analysis of a Randomized Control Trial

Greater Improvements in Vaccination Outcomes Among Black Young Adults With Vaccine-Resistant Attitudes in the United States South Following a Digital Health Intervention: Latent Profile Analysis of a Randomized Control Trial

Authors of this article:

Noah Mancuso1, 2 Author Orcid Image ;   Jenna Michaels1 Author Orcid Image ;   Erica N Browne1 Author Orcid Image ;   Allysha C Maragh-Bass3 Author Orcid Image ;   Jacob B Stocks4 Author Orcid Image ;   Zachary R Soberano4 Author Orcid Image ;   C Lily Bond4 Author Orcid Image ;   Ibrahim Yigit4 Author Orcid Image ;   Maria Leonora G Comello5 Author Orcid Image ;   Margo Adams Larsen6 Author Orcid Image ;   Kathryn E Muessig4 Author Orcid Image ;   Audrey Pettifor5 Author Orcid Image ;   Lisa B Hightow-Weidman4 Author Orcid Image ;   Henna Budhwani4 Author Orcid Image ;   Marie C D Stoner1 Author Orcid Image
Article Authors Cited by Tweetations Metrics

    1Women's Global Health Imperative, Research Triangle Institute (RTI) International, 3040 East Cornwallis Road, Research Triangle Park, NC, United States

    2Department of Epidemiology, Emory University, Atlanta, GA, United States

    3FHI 360, Durham, NC, United States

    4Institute on Digital Health and Innovation, College of Nursing, Florida State University (FSU), Tallahassee, FL, United States

    5Department of Epidemiology, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States

    6Virtually Better, Inc, Decatur, GA, United States

    Corresponding Author:

    Noah Mancuso, MSPH


    Background: Negative attitudes toward vaccines and suboptimal vaccination rates among African American and Black (Black) Americans have been well documented, due to a history of medical racism and human rights violations in the United States. However, digital health interventions (DHI) have been shown to address racial disparities in several health outcomes, such as cardiovascular disease, HIV, and maternal health. The Tough Talks COVID (TT-C) study was a randomized controlled trial of a DHI designed to empower Black young adults in the United States South to make informed, autonomous decisions about COVID-19 vaccine uptake by addressing structural barriers and misinformation about vaccines.

    Objective: Our objective was to identify subgroups of Black young adults with various vaccine attitudes at baseline and determine the subgroups for which the TT-C DHI was most impactful.

    Methods: Black young adults aged 18‐29 years in Alabama, Georgia, and North Carolina who were unvaccinated or insufficiently vaccinated against COVID-19 completed three online surveys over three months (N=360). Latent profile analysis was used to identify subgroups based on general vaccine attitudes at baseline, including hesitancy, confidence, knowledge, conspiracy beliefs, and mistrust. Logistic regression was used to examine the associations between latent profiles and vaccine uptake, and linear regression was used to examine changes in vaccine attitudes at three months post-randomization. Modification of the TT-C DHI’s effects was assessed by latent profiles.

    Results: Three latent profiles emerged: vaccine-receptive (n=124), vaccine-neutral (n=155), and vaccine-resistant (n=81). Political affiliation, income, social support, and recent flu vaccination differed significantly between the three subgroups (P<.05). Vaccine uptake was not significantly different by subgroup, and the TTC-DHI did not have differing effects on uptake across subgroups. However, the DHI had the strongest effect—with statistically significant measures of association (P<.05) and interaction P values (P<.10)—among the baseline vaccine-resistant and vaccine-neutral subgroups compared to the vaccine-receptive subgroups at three months in improving vaccine hesitancy, confidence, and conspiracy beliefs at three months: vaccine-resistant difference: −0.40 (−0.76 to −0.37), 0.39 (0.02 to 0.75), and −0.47 (−0.86 to −0.09); vaccine neutral difference: −0.36 (−0.52 to −0.19), 0.35 (0.18 to 0.51), and −0.24 (−0.44 to −0.03). The DHI had no effects on these outcomes among the vaccine-receptive subgroup.

    Conclusions: Our findings revealed subgroups of Black young adults in the United States South with different vaccination attitudes, for which the TT-C intervention had differing effects. Black young adults who are vaccine-resistant or vaccine-neutral may experience larger gains from a digital vaccine intervention. Future work aimed at improving vaccination outcomes could target these populations to maximize resource efficiency and drive the greatest improvements in vaccine outcomes.

    Trial Registration: ClinicalTrials.gov NCT05490329; https://clinicaltrials.gov/study/NCT05490329

    JMIR Public Health Surveill 2025;11:e67370

    doi:10.2196/67370

    Keywords

    COVID-19; mHealth; African American and Black; young adults; vaccination; mobile health 


    The World Health Organization declared the outbreak of novel COVID-19, a global pandemic in early 2020, caused by SARS-CoV-2 [1]. By the end of the year, two COVID-19 vaccines had been issued emergency use authorizations in the United States [2,3]. At the time, nearly three in five Americans reported intentions to be vaccinated for COVID-19; however, African American or Black (Black) Americans reported intentions to vaccinate approximately 30% lower than that of White Americans and 50% lower than that of Asian Americans [4]. Hesitancy toward COVID-19 vaccination was particularly high among Black young adults and residents of the United States South [5,6], which translated into lower uptake of the COVID-19 vaccine and a disproportionately high mortality rate for Black Americans early in the pandemic [7,8].

    Negative attitudes toward vaccines and suboptimal vaccination rates among Black Americans have been influenced by a history of medical racism and human rights violations in the United States [9-14]. Continued harmful experiences with the health care system and government motivate hesitancy toward public health programs, such as vaccination, as a self-protective coping mechanism [15]. Studies have consistently shown this hesitancy, with reported lower vaccine uptake among Black communities for the annual flu shot due to distrust of medical institutions and concerns for safety [16-18]. Additionally, disparities in human papillomavirus vaccination have been observed among young Black girls, which is reflected in higher rates of cervical cancer incidence and mortality later in life [19,20]. During the COVID-19 pandemic, these experiences were exacerbated for Black young adults by the spread of misinformation and conspiracy theories online and on social media [21,22].

    Digital health interventions (DHIs) have been successful in addressing racial disparities in health by creating culturally tailored messaging, inclusive programming, and more accessible services for disease prevention, ranging from cardiovascular disease to HIV to maternal health [23-27]. DHIs are particularly helpful for young adults, given the high proportion of Americans aged 18‐28 years with smartphones who report using web-based news and social media sites to access information about COVID-19 [28,29]. However, few DHIs to date have been developed specifically for COVID-19 vaccination among Black young adults in the United States South.

    The Tough Talks COVID (TT-C) study was a randomized controlled trial of a DHI designed to empower Black young adults in the United States South to make informed, autonomous decisions about COVID-19 vaccine uptake (ClinicalTrials.gov NCT05490329) [30]. The TT-C intervention was cocreated using community-based participatory research methods to address structural barriers and misinformation about vaccines through interactive activities and digital storytelling [31,32]. Assuming heterogeneity among the study population, we aimed to use a latent profile analysis (LPA) to identify subgroups of Black young adults with different baseline vaccine attitudes and identify subgroups for whom TT-C was most impactful, to allow for better-informed community-based policies, practices, and messaging related to vaccine uptake and attitudes.


    Participants

    We used data from the TT-C randomized control trial, which included Black young adults aged 18‐29 years who were unvaccinated or insufficiently vaccinated against COVID-19 in Alabama, Georgia, and North Carolina. Participants were recruited online or via social media and randomized 1:1 to either the TT-C DHI, a self-directed asynchronous app developed to enable Black young adults in Southern United States to make decisions about COVID-19 vaccine receipt via nonstigmatizing and tailored messaging, or an information standard-of-care control consisting of COVID-19 vaccine materials adapted from the Centers for Disease Control and Prevention for three months. Surveys were completed at baseline, one month, and three months after randomization. Additional details about the study, which was conducted between March and December 2023, have been published previously [30] .

    Latent Indicators

    Five baseline variables related to general vaccine attitudes at baseline were used as latent indicators in the LPA: vaccine hesitancy, vaccine confidence, vaccine-related conspiracy beliefs, vaccine knowledge, and group-based medical mistrust. Each variable was assessed using a previously validated scale with a standardized continuous score as input using z-scores, such that the mean score transformed to zero to remove the impact of any baseline differences in means or variances across the variables before they were used in the LPA. This approach helps mitigate the influence of baseline distribution differences in the clustering process by making the variables comparable. The measures used in this LPA were prespecified in the study protocol and tested in formative surveys with Black young adults [30,33]. Vaccine hesitancy was a mean score of a 9-item scale with Likert responses ranging from 1‐5, where higher scores indicated greater hesitancy [34]. Vaccine confidence was a mean score of a 7-item scale with Likert responses ranging from 1‐5, where higher scores indicated greater confidence [35]. Vaccine-related conspiracy beliefs were a mean score of an adapted 5-item scale with Likert responses ranging from 1‐4, where higher scores indicated greater belief in conspiracy [36]. Vaccine knowledge was a sum score of an adapted 4-item scale with “yes” or “no” responses for each item, where higher scores indicated greater knowledge [37]. Group-based medical mistrust was a sum score of a 12-item scale with Likert responses ranging from 1‐5, where higher scores indicated greater mistrust [38]. For mean scores, if a response was missing for more than one scale item, the mean score was set as missing and not calculated. For summed scores, if a response was missing for any scale item, the summed score was set as missing and not calculated.

    Additional Measures

    Once latent profiles were established, differences in several baseline sociodemographic variables and other measures were assessed between the profiles. These included self-reported: sex assigned at birth, gender, age, education (ie, 9th-12th grade; no diploma; high school graduate or GED completed; some college level or technical or vocational degree; bachelor’s degree; other advanced degree), political affiliation (ie, Democrat, Republican, Independent, other), employment status (eg, full-time; part-time; student; not working; disabled, permanently or temporarily), annual income, insurance coverage (eg, public, private, none), social support (continuous scale) [39], and previous flu and human papillomavirus vaccination.

    Outcomes

    The primary outcome was uptake of a COVID-19 vaccine anytime during the 3-month follow-up period, based on uploaded vaccine cards or, if not uploaded, self-report at one month and three months. Secondary outcomes included vaccine hesitancy, vaccine confidence, vaccine-related conspiracy beliefs, and vaccine knowledge, which were reported at 3 months. These secondary measures were assessed in the same manner as the baseline latent indicators explained above. Latent profiles were used to assess effect measure modification (EMM) of the TT-C DHI on both primary and secondary outcomes.

    Statistical Analysis

    Latent profile analysis allows us to gain insights into “hidden” subgroups in a study sample using person-centered mixture modeling. This approach models how indicator variables relate to each other by estimating different profiles (or classes) and determining the best-fitting model [40,41]. In LPA, model parameters include the probability of an individual belonging to a given profile, and each profile can have its own set of means and variances.

    We evaluated two different model specifications: (1) equal variance and covariance across profiles and (2) varied variances and covariances across profiles. For each specification, we tested models with two, three, and four profiles. Model fit was assessed using an analytical hierarchy process based on the following information criteria: Akaike information criterion, Bayesian information criterion, Approximate Weight of Evidence, consistent likelihood criterion, and Kullback information criterion [42]. Lower values indicate a better fit for each criterion. Additional criteria including the sample size-adjusted Bayesian information criterion, entropy, and the bootstrap likelihood ratio (BLRT) were used to assess overall model quality. For these criteria, a lower sample size-adjusted Bayesian information criterion, higher entropy, and statistically significant BLRT indicate a good fit. The best model was selected using an analytical hierarchical process that balanced low values of Akaike information criterion, Bayesian information Criterion, Approximate Weight of Evidence, consistent likelihood criterion, and Kullback information criterion, which indicate a more parsimonious model, with fewer parameters while still accounting for data complexity, with higher entropy to ensure distinct profile separation and a significant BLRT to confirm robustness [42]. After model selection, we assigned study participants to the profile with the highest posterior probability based on their pattern of responses across the variables. Descriptive statistics were then calculated for the sociodemographic and additional variables within each profile. Differences in continuous variables across profiles were tested using ANOVA, while differences in categorical variables were assessed with the χ2 test of independence.

    The primary outcome of vaccine uptake at three months was assessed using a logistic regression model. The odds ratio approximates the risk ratio, as the probability of vaccine uptake was not common (<15%). For secondary outcomes—vaccine hesitancy, vaccine confidence, and vaccine-related conspiracy beliefs—we used linear regression to determine differences in outcomes at three months. Poisson regression was used to test for differences in the secondary outcome of vaccine knowledge score at three months. The main effects of the TT-C DHI intervention on vaccine uptake and vaccine attitudes will be reported in the primary manuscript of the study (Budhwani et al, PhD, unpublished data, 2024). The EMM of the effect of the TT-C DHI intervention on primary and secondary outcomes by profile group was assessed in this study by including the study arm and latent profile at baseline as both main effects and interaction terms. The EMM assesses whether an intervention effect (ie, the effect of the intervention on the outcome in the intervention group compared to the control group) differs across a variable of interest (in this case, the LPA groups). Interaction terms with a P value <.10 were considered indicators of significant effect measure modification. For all models, the state of residence was controlled to account for its inclusion in the stratified randomization study design. We did not control for other variables as they were balanced by the randomized design. All analyses were conducted using R software (version 4.4.1; R Foundation for Statistical Computing) [43].

    Sensitivity Analysis

    A sensitivity analysis was conducted using latent class analysis (LCA), typically used with categorical variables, where all indicator variables were dichotomized at appropriate cut-points for each validated scale [44]. We tested LCA models with two, three, and four classes and used similar criteria to assess fit. We then compared the number of classes identified by LCA to the number of profiles identified by LPA, as well as the total variance explained and general sociodemographic differences across the classes/profiles. We chose to report LPA as the main analysis to maintain the continuous nature of the originally validated measures used as indicator variables and because of the added model flexibility of LPA with varying variances and covariances across profiles.

    Ethical Considerations

    All study materials and procedures were reviewed and approved by the Institutional Review Boards of the University of North Carolina, Chapel Hill (21‐1746) and Florida State University (STUDY00003617). Informed consent was collected digitally from all study participants before data collection and randomization. All data were deidentified to safeguard participant information.


    A total of 360 Black young adults were included in this study, of whom 76% (n=272) identified as cisgender or transgender women. The median age was 24 (IQR 21‐27) years, and approximately one-third of participants (n=122) had a bachelor’s degree or higher. Overall vaccine uptake at month 3 was low (n=21) and did not significantly differ between the intervention and control arms (odds ratio [OR] 1.88, 95% CI 0.76 to 4.69).

    Latent Profile Analysis

    Using the prespecified fit criteria with the analytic hierarchy process, we found that the model with three profiles and varied variances and covariances across profiles provided the best fit (Table 1). The profiles in Figure 1 were named based on the mean values of the five indicator variables, all of which were significantly different (P<.001) between the profiles (Table 2). The vaccine-receptive group (Profile 2) included 124 participants (34%) and had the lowest mean scores for medical mistrust, conspiracy beliefs, and vaccine hesitancy and the highest mean score for vaccine confidence and knowledge of vaccines compared to other profiles. The vaccine-resistant group (Profile 3) included 81 participants (23%) and had the highest mean score for medical mistrust, conspiracy beliefs, and vaccine hesitancy and the lowest mean score for confidence and knowledge of vaccines compared to other profiles. The vaccine-neutral group (Profile 1) included 155 participants (43%) and had mean scores between those of the vaccine-receptive and vaccine-resistant groups for all indicator variables.

    Table 1. Fit indices for the latent profile analysis of Model 1 (equal variance and covariance across profiles) and Model 2 (varied variances and covariances across profiles) for 2‐4 profiles of baseline vaccine attitudes in the TT-Ca study
    ModelNumber of profilesAICbBICcAWEdCLCeKICfSABICgEntropyBLRThP value
    124439.424501.594642.034409.154458.424450.830.87<.001
    134245.054330.554524.384202.714270.054260.750.83<.001
    144218.364327.174574.384163.964249.364238.340.80<.001
    223889.834049.164412.273809.063933.833919.090.61<.001
    233844.344035.284634.793721.773909.343888.590.65<.001
    243843.894180.444916.483693.393943.893917.120.71<.001

    aTT-C: Tough Talks COVID.

    bAIC: Akaike information criterion.

    cBIC: Bayesian information criterion.

    dAWE: Approximate Weight of Evidence.

    eCLC: consistent likelihood criterion.

    fKIC: Kullback information criterion.

    gSABIC: Sample size-adjusted Bayesian information criterion.

    hBLRT: Bootstrap likelihood ratio.

    ‎
    Figure 1. Standardized values of each indicator variable included in the three-profile model that best fit the latent profile analysis of baseline vaccine attitudes in the TT-C study. Points represent the mean standardized value for each profile, with bars showing ± 1 SD. The vaccine-resistant group (Profile 3 in blue) showed the highest medical mistrust, vaccine conspiracy beliefs, and vaccine hesitancy, coupled with the lowest vaccine confidence and knowledge. TT-C: Tough Talks COVID.
    Table 2. Sociodemographic and baseline variables of TT-Ca study participants by vaccine attitude profile.
    Baseline variableProfile 2 Vaccine-receptive (n=124)Profile 1 Vaccine-neutral (n=155)Profile 3 Vaccine-resistant (n=81)P value
    Vaccine hesitancy (ranges from 0‐5)<.001
    Median (IQR)1.8 (1.4-2.0)2.5 (2.2-2.8)3.3 (3.0-3.8)
    Mean (SD)1.7 (0.3)2.5 (0.5)3.4 (0.6)
    Vaccination knowledge score (ranges from 0‐4)<.001
    Median (IQR)4 (3-4)2 (2-3)1 (0-1)
    Mean (SD)3.4 (0.6)2.2 (0.9)0.8 (0.8)
    Vaccine confidence (ranges from 0‐5)<.001
    Median (IQR)4.1 (4.0-4.4)3.4 (3.1-3.7)2.6 (2.1-2.9)
    Mean (SD)4.2 (0.4)3.4 (0.4)2.5 (0.5)
    Vaccine conspiracy beliefs (ranges from 0‐5)<.001
    Median (IQR)1.6 (1.2-1.8)2.2 (2.0-2.6)3.1 (3.0-3.6)
    Mean (SD)1.6 (0.4)2.3 (0.4)3.2 (0.5)
    Medical mistrust (ranges from 0‐60)<.001
    Median (IQR)32 (29-35)35 (32-38)39 (36-42)
    Mean (SD)31.9 (4.6)34.6 (5.0)39.4 (6.3)
    Randomization arm, n (%).20
    Intervention55 (44)78 (50)47 (58)
    Control69 (56)77 (50)34 (42)
    Sex assigned at birth, n (%).08
    Male19 (15)28 (18)14 (17)
    Female105 (85)127 (82)67 (83)
    Gender identity, n (%).70
    Man (cis or trans)17 (14)25 (16)15 (19)
    Woman (cis or trans)93 (75)119 (77)60 (74)
    Gender fluid, gender queer, gender nonconforming, nonbinary, agender or other gender14 (11)11 (7)6 (7)
    Age (years).05
    Median (IQR)23.0 (20.0-27.0)24.0 (21.0-26.0)24.0 (23.0-27.0)
    Mean (SD)23.3 (3.7)23.6 (3.3)24.5 (3.1)
    Hispanic/Latino ethnicity, n (%)5 (4)9 (6)2 (3).50
    Flu vaccination (ever), n (%).05
    No23 (19)33 (21)27 (33)
    Yes97 (78)114 (74)48 (59)
    Flu vaccination (past 12 months), n (%).01
    No55 (57)72 (63)40 (83)
    Yes40 (41)33 (29)7 (15)
    Human papillomavirus (any), n (%).12
    No23 (19)40 (26)25 (31)
    Yes83 (67)86 (55)39 (48)
    Human papillomavirus vaccination (full series), n (%).20
    No8 (9.6)16 (19)4 (10)
    Yes73 (88)62 (72)33 (85)
    Political affiliation, n (%).001
    Democrat74 (60)70 (45)27 (33)
    Republican2 (1.6)6 (3.9)3 (3.7)
    Independent27 (22)46 (30)28 (35)
    Other8 (6.5)3 (1.9)1 (1.2)
    Social support (ranges from 0‐5), n (%).02
    Median (IQR)4.2 (3.5-4.8)3.9 (3.0-4.7)3.8 (2.6-4.8)
    Mean (SD)4.1 (0.8)3.8 (0.9)3.6 (1.1)
    Annual income (USD), n (%).01
    ≤$24,99931 (25)63 (42)32 (40)
    $25,000 - $49,99941 (34)46 (29)24 (30)
    $50,000 - $74,99920 (16)17 (11)9 (11)
    $75,000 - $99,99911 (8.9)11 (7.1)3 (3.7)
    $100,00012 (9.7)5 (3.2)1 (1.2)
    Insurance coverage, n (%)<.001
    None12 (9.7)18 (12)12 (15)
    Private81 (65)66 (43)23 (28)
    Public27 (22)59 (38)40 (49)
    Education level, n (%).20
    9th to 12th grade, no diploma8 (6.5)6 (3.9)2 (2.5)
    High school graduate or GED completed22 (18)42 (27)24 (30)
    Some college level/technical/vocational degree45 (36)50 (32)36 (44)
    Bachelor’s degree36 (29)39 (25)13 (16)
    Other advanced degree (Master’s, Doctoral degree)12 (9.7)17 (11)5 (6.2)
    Employment status, n (%).30
    Working, full-time54 (44)64 (41)35 (43)
    Working, part-time20 (16)19 (12)12 (15)
    Not working14 (11)26 (17)11 (14)
    Student34 (27)38 (25)14 (17)
    Disabled, permanently or temporarily0 (0)4 (2.6)3 (3.7)
    Other2 (1.6)3 (1.9)5 (6.2)

    aTT-C: Tough Talks COVID

    Individuals in the vaccine-receptive group was significantly more likely to have been vaccinated for the flu in the past 12 months (n=40, 41%) compared with those in the vaccine-neutral (n=33, 29%) and the vaccine-resistant (n=7, 15%) groups (P=.01). Additionally, the vaccine-receptive group was more likely to report Democratic political affiliation (n=74, 60%) compared to the vaccine-neutral (n=70, 45%) and vaccinate-resistant (n=27, 33%) groups (P=.001). Reported annual income and insurance coverage were highest among the vaccine-receptive group; however, there were no significant differences in educational attainment or employment status across profiles. The vaccine-receptive group also tended to be younger (mean age, 23.3, SD 3.7) years and reported more social support (mean score, 4.1, SD 0.8) compared with both the vaccine-neutral (mean age, 23.6, SD 3.3 years; mean social support score, 3.8, SD, 0.9) and vaccine-resistant (mean age, 24.5, SD 3.1 years; mean social support score, 3.6, SD 1.1) groups (P=.05).

    EMM by Baseline Vaccine Attitude Profile

    Effect measure modification analysis showed that the TT-C DHI did not have a different effect on vaccine uptake across latent profile subgroups (Table 3). For secondary outcomes, EMM showed that the TT-C DHI had differing effects among those identified as vaccine-resistant compared with those classified as vaccine-receptive at baseline for the outcomes of vaccine hesitancy (interaction P=.07), vaccine confidence (interaction P=.06), and vaccine-related conspiracy beliefs (interaction P=.04) at 3 months (Table 3). There were also differing effects between vaccine-neutral and vaccine-receptive groups from baseline for the outcomes of vaccine hesitancy (interaction P=.05) and vaccine confidence (interaction P=.05) at 3 months. The vaccine-resistant group showed the greatest effect from the TT-C DHI, with a reduced mean vaccine hesitancy score of −0.40 (95% CI −0.76 to −0.37), an increased mean vaccine confidence score of 0.39 (95% CI 0.02 to 0.75), and a reduced conspiracy belief score of −0.47 (95% CI −0.86 to −0.09). The vaccine-neutral subgroup experienced moderate effects, including a reduced mean vaccine hesitancy score of −0.36 (95% CI −0.52 to −0.19) and an increased mean vaccine confidence score of 0.35 (95% CI 0.18 to 0.51). The TT-C DHI had no significant effect on any of the secondary vaccine-related outcomes among the vaccine-receptive subgroup.

    Table 3. Results of the effect measure modification assessment for the impact of the TT-Ca DHIb on primary and secondary outcomes at month 3 for the three vaccine attitude profiles at baseline.
    OutcomesOdds ratio (95% CI) or Diff (95% CI)cInteraction P valued
    Primary outcomes
     Vaccine uptake (3 months).79
      Vaccine-resistant1.81 (0.41-7.90)
      Vaccine-neutralnot enough data
      Vaccine-receptive1.32 (0.35-5.04)
    Secondary outcomes
     Vaccine hesitancy (3 months)
      Vaccine-resistant−0.40 (−0.76 to −0.37).07
      Vaccine-neutral−0.36 (−0.52 to −0.19).05
      Vaccine-receptive−0.08 (−0.26 to 0.09)–e
     Vaccine confidence (3 months)
      Vaccine-resistant0.39 (0.02-0.75).06
      Vaccine-neutral0.35 (0.18-0.51).05
      Vaccine-receptive0.06 (−0.11 to 0.24)–
     Vaccine knowledge (3 months)
      Vaccine-resistant0.90 (0.27-1.53).12
      Vaccine-neutral0.81 (0.41-1.20).17
      Vaccine-receptive0.16 (0.07-0.40)–
     Vaccine-related conspiracy beliefs (3 months)
      Vaccine-resistant−0.47 (−0.86 to −0.09).04
      Vaccine-neutral−0.24 (−0.44 to −0.03).36
      Vaccine-receptive−0.09 (−0.26 to 0.08)–

    aTTC: Tough Talks COVID

    bDHI: digital health interventions

    cOdds ratios (95% CI) are presented for primary outcomes and Diff (95% CI) for secondary outcomes.

    dReference for the interaction P-value is the vaccine-receptive subgroup with significance indicated by P<.10

    enot applicable

    Sensitivity Analysis

    When using the prespecified fit criteria with the analytic hierarchy process, we found the LCA model with the best fit to include three classes, similar to the LPA results. Similar to our LPA results, the three classes represented subgroups of study participants who were either: (1) vaccine-receptive, with low mean scores for mistrust, conspiracy beliefs, or hesitancy and high mean scores for knowledge and confidence; (2) vaccine-resistant, with high mean scores for mistrust, conspiracy beliefs, and hesitancy and low mean scores for knowledge or confidence; or (3) somewhere in between the two, as vaccine-neutral. The classes from the LCA had similar differences as the profiles in the LPA analysis with respect to political affiliation, income, insurance coverage, and flu vaccination. Thus, LCA results reinforce the findings from the LPA.


    Our LPA of Black young adults participating in an RCT for the TT-C DHI found three distinct subgroups of individuals with different baseline vaccine attitudes. The three subgroups ranged from vaccine-receptive to vaccine-resistant and differed on several sociodemographic variables, such as political affiliation, income, and social support. While there were no significant differences in vaccine uptake by intervention arm between the three subgroups, we identified modification whereby the TT-C DHI had the largest improvement in hesitancy, confidence, and conspiracy beliefs at 3 months among the subgroups who were vaccine-resistant and vaccine-neutral at baseline compared to those who were vaccine-receptive.

    To our knowledge, this is the first study to use LPA to explore the efficacy of DHIs for improving vaccine outcomes among Black young adults in the United States South, a population understudied and underserved in vaccine research. Our results from TT-C align with those of several social media and video-based educational DHIs across different populations, which showed the biggest improvements in vaccine outcomes among those with poor vaccine attitudes at baseline [45,46]. Because the TT-C DHI was cocreated with youth from the community and incorporated digital storytelling—a method well suited for marginalized communities [32]—we believe we prevented experiences of “backfiring” or reverse effects seen in more generalized vaccine-related DHIs where vaccine attitudes worsened among those with poor attitudes at baseline [47,48]. Additionally, the vaccine-receptive subgroup in this study had high confidence and knowledge in vaccines, with little hesitancy, mistrust, or conspiracy beliefs despite being insufficiently vaccinated. These baseline scores left limited room for improvement at month 3, which likely explains why TT-C differs from some other vaccine DHIs that saw the largest improvements in vaccine outcomes among those with more positive attitudes at baseline [49,50].

    The success of this community-based participatory research project in developing a DHI for Black young adults in the United States South mirrors that of other DHIs aimed at acknowledging and addressing the historical injustices and ongoing inequities that drive distrust within communities of color, while using culturally appropriate messages and existing community infrastructure to address concerns about COVID-19 vaccine safety and access [51-55]. By using digital storytelling to convey accurate, relatable, and culturally resonant messages, DHIs can foster trust and engagement in positive vaccine beliefs, which can serve as a model for improving vaccination among other underserved or vaccine-resistant populations, both for COVID-19 and other vaccines. This work provides a blueprint for public health initiatives to adapt and implement culturally sensitive vaccine interventions that help build the capacity to identify inaccurate or misleading information online and to encourage individuals to make informed decisions. However, it also highlights the challenges of translating changed attitudes into actions.

    Despite the significant improvements seen in vaccine attitudes, the lack of impact from the TTC-DHI on actualized vaccination behavior among any subgroup of Black young adults in this study may be attributed to a number of external factors. One key factor was the availability and access of COVID-19 vaccines during the study period of late 2023, when primary series promotion was shifting to boosters. Changing public health guidelines and the short 3-month timeframe in the study may have resulted in a lack of opportunity for behavioral action to get vaccinated, as the recommended frequency of boosters was greater than 3 months and it may have been less likely for clinics to offer the primary vaccination series. Indeed, only 21 (6.4%) participants received a new COVID-19 vaccine over the 3-month follow-up period. Additionally, the Health Belief Model has shown that a change in attitudes is not always enough to effect behavior change; certain conditions, such as perceived risk of infection and resource mobilization, are also required [56,57]. Lastly, other external influences, such as ongoing misinformation and continued structural inequalities in access, may have dampened the direct impact of the intervention on behavior, even as attitudes shifted positively.

    This study draws on a large sample of Black young adults from the United States South, and we used a comprehensive range of validated vaccine scales as constructs to conduct the LPA. However, there are several limitations to this work. First, the landscape of COVID-19 science was changing quickly while the app was being developed and implemented (2021‐2024); although app content was responsive to these changes, not all components were adjustable once developed, which may limit the generalizability and overall effectiveness of app messaging. This evolving context may have influenced participants’ attitudes and behaviors in ways that are difficult to fully capture, as new information or policy changes could have overshadowed the effects of the intervention itself. Second, information about vaccine attitudes was collected via self-report, which may be misreported due to social desirability bias. Lastly, the use of LPA does not guarantee proper creation or assignment of subgroups and overlap between identified profiles may impact findings. However, our sensitivity analysis resulted in the same number of subgroups with similar distributions and effect measures, strengthening our confidence in these results and our study conclusions.

    This analysis found that Black young adults who are vaccine-resistant or vaccine-neutral may experience larger improvements in vaccine outcomes when using DHIs like TT-C. Future researchers and policymakers can use this knowledge to create better-informed community-based policies, practices, and messaging related to vaccine uptake and attitudes while understanding subpopulations to target for resource efficiency and to drive the biggest improvement in vaccine outcomes. However, vaccine-receptive individuals may experience other benefits from these types of DHIs not assessed, such as improved linkage to vaccine resources or social support and thus should not be excluded. Vaccine-receptive individuals could also be leveraged as peer leaders or vaccine advocates in their respective communities to further increase uptake and acceptability of DHIs among Black young adults in the United States South.

    Acknowledgments

    The research reported in this publication was supported by the National Institute of Minority Health Disparities of the National Institutes of Health under award number R01MD016834. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

    Disclaimer

    Research reported in this publication was supported by the National Institute on Minority Health And Health Disparities of the National Institutes of Health under Award Number R01MD016834. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

    Authors' Contributions

    Conceptualization: NM, LHW, HB, MS

    Data curation: JM, JS

    Formal analysis: NM, JM, EB, MS

    Funding acquisition: LHW, HB

    Investigation: AMB, JS, ZS, LB, MLC, MAL, KM, LHW, HB, MS

    Methodology: NM, EB, MS

    Project administration: JS, ZS, LB, LHW, HB

    Validation: IY, EB

    Writing - original draft: NM, JM, EB, MS

    Writing - review & editing: NM, JM, EB, AMB, JS, ZS, LB, IY, MLC, MAL, KM, AP, LHW, HB, MS

    Conflicts of Interest

    None declared.

    1. Timeline of WHO’s response to COVID-19. World Health Organization. URL: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/interactive-timeline [Accessed 2024-10-02]
    2. Oliver SE, Gargano JW, Marin M, et al. The Advisory Committee on Immunization Practices’ Interim Recommendation for use of Moderna COVID-19 vaccine — United States, December 2020. MMWR Morb Mortal Wkly Rep. Jan 1, 2021;69(5152):1653-1656. [CrossRef] [Medline]
    3. Parums DV. Editorial: First full regulatory approval of a COVID-19 vaccine, the BNT162b2 Pfizer-BioNTech vaccine, and the real-world implications for public health policy. Med Sci Monit. Sep 6, 2021;27:e934625. [CrossRef] [Medline]
    4. Funk C, Tyson A. Intent to get a COVID-19 vaccine rises to 60% as confidence in research and development process increases. Pew Research Center; 2020. URL: https:/​/www.​pewresearch.org/​science/​2020/​12/​03/​intent-to-get-a-covid-19-vaccine-rises-to-60-as-confidence-in-research-and-development-process-increases/​ [Accessed 2024-10-02]
    5. Gurley S, Bennett B, Sullivan PS, et al. COVID-19 vaccine perceptions, intentions, and uptake among young adults in the United States: prospective college-based cohort study. JMIR Public Health Surveill. Dec 15, 2021;7(12):e33739. [CrossRef] [Medline]
    6. Willis DE, Presley J, Williams M, Zaller N, McElfish PA. COVID-19 vaccine hesitancy among youth. Hum Vaccin Immunother. Dec 2, 2021;17(12):5013-5015. [CrossRef] [Medline]
    7. Gold JAW, Rossen LM, Ahmad FB, et al. Race, ethnicity, and age trends in persons who died from COVID-19 - United States, May-August 2020. MMWR Morb Mortal Wkly Rep. Oct 23, 2020;69(42):1517-1521. [CrossRef] [Medline]
    8. Cheng Z, Li Y. Racial and ethnic and income disparities in COVID-19 vaccination among Medicare beneficiaries. J Am Geriatr Soc. Sep 2022;70(9):2638-2645. [CrossRef] [Medline]
    9. Nuriddin A, Mooney G, White AIR. Reckoning with histories of medical racism and violence in the USA. Lancet. Oct 3, 2020;396(10256):949-951. [CrossRef] [Medline]
    10. Bogart LM, Dong L, Gandhi P, et al. COVID-19 vaccine intentions and mistrust in a national sample of Black Americans. J Natl Med Assoc. Jan 2022;113(6):599-611. [CrossRef] [Medline]
    11. Tsui J, Martinez B, Shin MB, et al. Understanding medical mistrust and HPV vaccine hesitancy among multiethnic parents in Los Angeles. J Behav Med. Apr 2023;46(1-2):100-115. [CrossRef] [Medline]
    12. Bogart LM, Ojikutu BO, Tyagi K, et al. COVID-19 related medical mistrust, health impacts, and potential vaccine hesitancy among Black Americans living with HIV. J Acquir Immune Defic Syndr. Feb 1, 2021;86(2):200-207. [CrossRef] [Medline]
    13. Wall LL. J. Marion Sims and the vesicovaginal fistula: historical understanding, medical ethics, and modern political sensibilities. Female Pelvic Med Reconstr Surg. 2018;24(2):66-75. [CrossRef] [Medline]
    14. Alang S, McAlpine DD, Hardeman R. Police brutality and mistrust in medical institutions. J Racial Ethn Health Disparities. Aug 2020;7(4):760-768. [CrossRef] [Medline]
    15. Bogart LM, Takada S, Cunningham WE. Medical mistrust, discrimination, and the domestic HIV epidemic. In: Ojikutu BO, Stone VE, editors. HIV in US Communities of Color. Springer International Publishing; 2021:207-231. [CrossRef]
    16. Shon EJ, Lee L. Effects of vaccine literacy, health beliefs, and flu vaccination on perceived physical health status among under/graduate students. Vaccines (Basel). Mar 30, 2023;11(4):765. [CrossRef] [Medline]
    17. Jamison AM, Quinn SC, Freimuth VS. “You don’t trust a government vaccine”: narratives of institutional trust and influenza vaccination among African American and White adults. Soc Sci Med. Jan 2019;221:87-94. [CrossRef] [Medline]
    18. Quinn SC, Hilyard KM, Jamison AM, et al. The influence of social norms on flu vaccination among African American and White adults. Health Educ Res. Dec 1, 2017;32(6):473-486. [CrossRef] [Medline]
    19. Cohen CM, Wentzensen N, Castle PE, et al. Racial and ethnic disparities in cervical cancer incidence, survival, and mortality by histologic subtype. J Clin Oncol. Feb 10, 2023;41(5):1059-1068. [CrossRef] [Medline]
    20. Cook RL, Zhang J, Mullins J, et al. Factors associated with initiation and completion of human papillomavirus vaccine series among young women enrolled in Medicaid. J Adolesc Health. Dec 2010;47(6):596-599. [CrossRef] [Medline]
    21. Skafle I, Nordahl-Hansen A, Quintana DS, Wynn R, Gabarron E. Misinformation about COVID-19 vaccines on social media: rapid review. J Med Internet Res. Aug 4, 2022;24(8):e37367. [CrossRef] [Medline]
    22. Gabarron E, Oyeyemi SO, Wynn R. COVID-19-related misinformation on social media: a systematic review. Bull World Health Organ. Jun 1, 2021;99(6):455-463A. [CrossRef] [Medline]
    23. Jack BW, Bickmore T, Yinusa-Nyahkoon L, et al. Improving the health of young African American women in the preconception period using health information technology: a randomised controlled trial. Lancet Digit Health. Sep 2020;2(9):e475-e485. [CrossRef] [Medline]
    24. Cyriac J, Jenkins S, Patten CA, et al. Improvements in diet and physical activity-related psychosocial factors among African Americans using a mobile health lifestyle intervention to promote cardiovascular health: the FAITH! (Fostering African American Improvement in Total Health) app pilot study. JMIR Mhealth Uhealth. Nov 12, 2021;9(11):e28024. [CrossRef] [Medline]
    25. Zhou ES, Ritterband LM, Bethea TN, Robles YP, Heeren TC, Rosenberg L. Effect of culturally tailored, internet-delivered cognitive behavioral therapy for insomnia in Black women: a randomized clinical trial. JAMA Psychiatry. Jun 1, 2022;79(6):538-549. [CrossRef] [Medline]
    26. Mancuso N, Mansergh G, Stephenson R, et al. Factors associated with mobile app-based ordering of HIV self-test kits among men who have sex with men in Atlanta, Detroit and New York City: an exploratory secondary analysis of a randomized control trial. J Int AIDS Soc. May 2023;26(5):e26100. [CrossRef] [Medline]
    27. Stoner MCD, Smith L, Ming K, et al. Results from a pilot study of an automated directly observed therapy intervention using artificial intelligence with conditional economic incentives among young adults with HIV. J Acquir Immune Defic Syndr. Jun 1, 2024;96(2):136-146. [CrossRef] [Medline]
    28. Faverio M. Share of those 65 and older who are tech users has grown in the past decade. Pew Research Center; 2022. URL: https:/​/www.​pewresearch.org/​short-reads/​2022/​01/​13/​share-of-those-65-and-older-who-are-tech-users-has-grown-in-the-past-decade/​ [Accessed 2024-10-02]
    29. Hill PW, Diamond J, Spiegel AN, VanWormer E, Leadabrand M, McQuillan J. Accuracy of COVID-19 relevant knowledge among youth: number of information sources matters. PLOS ONE. 2022;17(12):e0267871. [CrossRef] [Medline]
    30. Budhwani H, Maragh-Bass AC, Tolley EE, et al. Tough Talks COVID-19 digital health intervention for vaccine hesitancy among Black young adults: protocol for a hybrid type 1 effectiveness implementation randomized controlled trial. JMIR Res Protoc. Feb 13, 2023;12(1):e41240. [CrossRef] [Medline]
    31. Wallerstein N, Duran B, Oetzel JG, Minkler M. Community-Based Participatory Research for Health: Advancing Social and Health Equity. 3rd ed. Wiley; 2017. URL: https:/​/www.​wiley.com/​en-dk/​Community-Based+Participatory+Research+for+Health%3A+Advancing+Social+and+Health+Equity%2C+3rd+Edition-p-9781119258858 [Accessed 2024-10-09]
    32. de Jager A, Fogarty A, Tewson A, Lenette C, Boydell K. Digital storytelling in research: a systematic review. TQR. 2017;22(10):2548-2582. [CrossRef]
    33. Stoner MCD, Tweedy D, Comello MGL, et al. Using narratives to inform the development of a digital health intervention related to COVID-19 vaccination in Black young adults in Georgia, North Carolina and Alabama. Vaccine (Auckl). Nov 15, 2022;40(48):6908-6916. [CrossRef] [Medline]
    34. Shapiro GK, Tatar O, Dube E, et al. The vaccine hesitancy scale: psychometric properties and validation. Vaccine (Auckl). Jan 29, 2018;36(5):660-667. [CrossRef] [Medline]
    35. Gilkey MB, Reiter PL, Magnus BE, McRee AL, Dempsey AF, Brewer NT. Validation of the vaccination confidence scale: a brief measure to identify parents at risk for refusing adolescent vaccines. Acad Pediatr. 2016;16(1):42-49. [CrossRef] [Medline]
    36. Shapiro GK, Holding A, Perez S, Amsel R, Rosberger Z. Validation of the vaccine conspiracy beliefs scale. Papillomavirus Res. Dec 2016;2:167-172. [CrossRef] [Medline]
    37. Zingg A, Siegrist M. Measuring people’s knowledge about vaccination: developing a one-dimensional scale. Vaccine (Auckl). May 28, 2012;30(25):3771-3777. [CrossRef] [Medline]
    38. Shelton RC, Winkel G, Davis SN, et al. Validation of the group-based medical mistrust scale among urban Black men. J Gen Intern Med. Jun 2010;25(6):549-555. [CrossRef] [Medline]
    39. Sherbourne CD, Stewart AL. The MOS social support survey. Social Science & Medicine. Jan 1991;32(6):705-714. [CrossRef]
    40. Muthén B, Muthén LK. Integrating person-centered and variable-centered analyses: growth mixture modeling with latent trajectory classes. Alcohol Clin Exp Res. Jun 2000;24(6):882-891. [CrossRef] [Medline]
    41. Jason L, Glenwick D. Handbook of Methodological Approaches to Community-Based Research: Qualitative, Quantitative, and Mixed Methods. Oxford University Press; 2016. [CrossRef]
    42. Akogul S, Erisoglu M. An approach for determining the number of clusters in a model-based cluster analysis. Entropy (Basel). 2017;19(9):452. [CrossRef]
    43. R: A language and environment for statistical computing. R Foundation for Statistical Computing. 2021. URL: https://www.r-project.org/ [Accessed 2024-08-01]
    44. Scotto Rosato N, Baer JC. Latent class analysis: a method for capturing heterogeneity. Soc Work Res. Mar 1, 2012;36(1):61-69. [CrossRef]
    45. Vandormael A, Adam M, Greuel M, et al. The effect of a wordless, animated, social media video intervention on COVID-19 prevention: online randomized controlled trial. JMIR Public Health Surveill. Jul 27, 2021;7(7):e29060. [CrossRef] [Medline]
    46. Bender FL, Rief W, Brück J, Wilhelm M. Effects of a video-based positive side-effect information framing: an online experiment. Health Psychol. Aug 2023;42(8):593-602. [CrossRef] [Medline]
    47. van Stekelenburg A, Schaap G, Veling H, Buijzen M. Investigating and improving the accuracy of US citizens’ beliefs about the COVID-19 pandemic: longitudinal survey study. J Med Internet Res. Jan 12, 2021;23(1):e24069. [CrossRef] [Medline]
    48. Freeman D, Loe BS, Yu LM, et al. Effects of different types of written vaccination information on COVID-19 vaccine hesitancy in the UK (OCEANS-III): a single-blind, parallel-group, randomised controlled trial. Lancet Public Health. Jun 2021;6(6):e416-e427. [CrossRef] [Medline]
    49. Johnson V, Butterfuss R, Kim J, Orcutt E, Harsch R, Kendeou P. The “Fauci Effect”: Reducing COVID-19 misconceptions and vaccine hesitancy using an authentic multimodal intervention. Contemp Educ Psychol. Jul 2022;70:102084. [CrossRef] [Medline]
    50. Amazeen MA, Krishna A, Eschmann R. Cutting the bunk: comparing the solo and aggregate effects of prebunking and debunking Covid-19 vaccine misinformation. Sci Commun. Aug 2022;44(4):387-417. [CrossRef]
    51. Strully KW, Harrison TM, Pardo TA, Carleo-Evangelist J. Strategies to address COVID-19 vaccine hesitancy and mitigate health disparities in minority populations. Front Public Health. 2021;9:645268. [CrossRef] [Medline]
    52. Ford KL, West AB, Bucher A, Osborn CY. Personalized digital health communications to increase COVID-19 vaccination in underserved populations: a double diamond approach to behavioral design. Front Digit Health. 2022;4:831093. [CrossRef] [Medline]
    53. Koskan A, Larkey L, Todd M, Kim SW. Protocol for a community-based digital storytelling pilot intervention to reduce Hispanic parents’ vaccine hesitancy to immunize their children against COVID-19. PLOS ONE. 2024;19(3):e0299787. [CrossRef] [Medline]
    54. Castellon-Lopez YM, Klomhaus AM, Garcia C, et al. MivacunaLA (MyshotLA): a community-partnered mobile phone intervention to improve COVID-19 vaccination behaviors among low-income, Spanish-speaking, and immigrant Latino parents or caregivers. Vaccines (Basel). May 8, 2024;12(5):511. [CrossRef] [Medline]
    55. AuYoung M, Rodriguez Espinosa P, Chen WT, et al. Addressing racial/ethnic inequities in vaccine hesitancy and uptake: lessons learned from the California alliance against COVID-19. J Behav Med. Apr 2023;46(1-2):153-166. [CrossRef] [Medline]
    56. Limbu YB, Gautam RK, Pham L. The Health Belief Model applied to COVID-19 vaccine hesitancy: a systematic review. Vaccines (Basel). Jun 18, 2022;10(6):973. [CrossRef] [Medline]
    57. Zampetakis LA, Melas C. The health belief model predicts vaccination intentions against COVID-19: a survey experiment approach. Appl Psychol Health Well Being. May 2021;13(2):469-484. [CrossRef] [Medline]

    ‎
    BLRT: bootstrap likelihood ratio
    DHI: digital health intervention
    EMM: effect measure modification
    LCA: latent class analysis
    LPA: latent profile analysis
    TT-C: Tough Talks COVID

    Edited by Amaryllis Mavragani; submitted 15.10.24; peer-reviewed by Edem Gavor, Kola Adegoke, Mary-Gloria Chidiebere Iwunwa, Qiong Fang, Yewei Xie; final revised version received 27.02.25; accepted 04.03.25; published 16.04.25.

    Copyright

    © Noah Mancuso, Jenna Michaels, Erica N Browne, Allysha C Maragh-Bass, Jacob B Stocks, Zachary R Soberano, C Lily Bond, Ibrahim Yigit, Maria Leonora G Comello, Margo Adams Larsen, Kathryn E Muessig, Audrey Pettifor, Lisa B Hightow-Weidman, Henna Budhwani, Marie C D Stoner. Originally published in JMIR Public Health and Surveillance (https://publichealth.jmir.org), 16.4.2025.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on https://publichealth.jmir.org, as well as this copyright and license information must be included.