This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on http://publichealth.jmir.org, as well as this copyright and license information must be included.
Web-based approaches, specifically social media sites, represent a promising approach for recruiting young gay and bisexual men for research studies. Little is known, however, about how the performance of social media advertisements (ads) used to recruit this population is affected by ad content (ie, image and text).
The aim of this study was to evaluate the effects of different images and text included in social media ads used to recruit young gay and bisexual men for the pilot test of a Web-based human papillomavirus (HPV) vaccination intervention.
In July and September 2016, we used paid Facebook advertisements to recruit men who were aged 18-25 years, self-identified as gay or bisexual, US resident, and had not received HPV vaccine. A 4x2x2 factorial experiment varied ad image (a single young adult male, a young adult male couple, a group of young adult men, or a young adult male talking to a doctor), content focus (text mentioning HPV or HPV vaccine), and disease framing (text mentioning cancer or a sexually transmitted disease [STD]). Poisson regression determined whether these experimental factors affected ad performance.
The recruitment campaign reached a total of 35,646 users who viewed ads for 36,395 times. This resulted in an overall unique click-through rate of 2.01% (717/35,646) and an overall conversion rate of 0.66% (241/36,395). Reach was higher for ads that included an image of a couple (incidence rate ratio, IRR=4.91, 95% CI 2.68-8.97,
Facebook ads are a convenient and cost-efficient strategy for reaching and recruiting young gay and bisexual men for a Web-based HPV vaccination intervention. To help optimize ad performance among this population, researchers should consider the importance of the text and image included in the social media recruitment ads.
Human papillomavirus (HPV) infection is the most common sexually transmitted infection in the United States [
Efforts are therefore needed to increase HPV vaccination among young gay and bisexual men. However, reaching and recruiting this population may pose a challenge to HPV vaccination interventions. Recruitment via community settings (eg, community events and organizations) has been a frequently used approach for recruiting gay and bisexual men into research studies, but it is an approach that may oversample individuals who are more involved with the lesbian, gay, bisexual, and transgender (LGBT) community [
Web-based approaches, specifically social media sites, represent a newer and promising recruitment strategy [
The content of the recruitment ads likely plays a role in the success of a Facebook recruitment campaign. Indeed, past studies involving smoking cessation and mental health have shown that both the images and text included in Facebook ads affect ad performance and recruitment metrics (eg, number of Facebook users who click on an ad) [
“Outsmart HPV” is a mobile-friendly Web-based HPV vaccination intervention developed for young gay and bisexual men. Men were eligible for this project if they (1) were in the age range of 18-25 years, (2) self-identified as gay or bisexual, (3) were a US resident, and (4) had not received any doses of HPV vaccine. Following intervention development, we conducted a pilot test of this project. Recruitment for the pilot test occurred in two waves, with the first wave occurring in July 2016 and the second in September 2016. All recruitment occurred via paid Facebook ads. This report includes recruitment results from the pilot test. The Institutional Review Board at The Ohio State University approved this study. The parent randomized trial is registered at ClinicalTrials.gov (identifier NCT02835755).
We created recruitment ads using Facebook’s Ads Manager program. All ads included a headline, main text, image, Outsmart HPV project logo, and weblink to the project website. Ads adhered to Facebook’s requirements at the time of ad development, including character limits (25 characters for the headline and 90 characters for the main text) and image restrictions (any image used could not include more than 20% text) [
The headline (“Earn Up To $95 Online!”), Outsmart HPV logo, and weblink to the project website were identical for all ads. The main text for all ads started with “Online HPV study for gay & bisexual men,” but the remaining main text and image varied across the ads using a 4x2x2 factorial experiment design. Experimental factors included image type, content focus, and disease framing. Image type (4 conditions) refers to the primary image featured in the ad. All ad images featured at least one male who appeared to be a young adult in the target age group, but ads varied on the number and type of people in the image: (1) a single man (“single person”), (2) a male couple (“couple”), (3) a group of more than 2 men (“group”), or (4) a man interacting with a health care provider (“doctor”). The people included in the images were from a range of racial or ethnic groups. Content focus (2 conditions) involved whether the ad’s remaining main text mentioned (1) HPV, or (2) HPV vaccine. Disease framing (2 conditions) involved whether the ad’s remaining main text mentioned (1) cancer, or (2) a sexually transmitted disease (STD). We created 2 ads for each of the 16 experimental conditions (ie, 2 variations of each image type), resulting in a total of 32 ads.
Example Facebook recruitment ads for Outsmart HPV (human papillomavirus). All ads were News Feed ads, and experimental factors included: image type (single person, couple, group, or doctor), content focus (HPV or HPV vaccine), and disease framing (cancer or sexually transmitted disease [STD]).
We targeted all ads to Facebook users by sex (male), age (18-25 years), location (United States), and language (English). We further targeted the ads on keywords generated from information that Facebook users add to their Timeline, pages they “like,” or ads they have previously clicked on. The keywords used in our campaign included “bisexuality,” “homosexuality,” “same-sex relationship,” “genderqueer,” “gay pride,” “LGBT community,” “LGBT culture,” or “rainbow flag (LGBT movement).” The chance that a given ad appeared on a user’s page was determined by a Facebook algorithm that considers several factors (eg, the spending limit specified for the ad, competition from other ads).
Ads linked potential participants to the project website or the project’s Facebook page (which in turn directed potential participants to the project website). We collaborated with the Center for Health Communications Research (CHCR) at the University of Michigan to develop and maintain the project website, which was a mobile-friendly website accessible by desktop or laptop, tablet computer, or smartphone (iOS and Android). Once on the project website, potential participants first completed a project eligibility screener. Those determined to be eligible were then asked to provide informed consent and create a project account. Following account creation, participants completed a preintervention Web-based survey, were randomly assigned to either the intervention or control arm, and then completed remaining study activities (eg, viewing intervention or control materials on the Web about HPV vaccine and completing 3 additional Web-based surveys over the course of 7 months). Participants could earn up to US $95 in gift cards during the course of the project.
Facebook recruitment campaign results for Outsmart HPV (human papillomavirus).
Facebook provided standard data on several metrics related to ad performance. Similar to past studies involving Facebook recruitment campaigns [
We calculated descriptive statistics for all Facebook metrics and survey data. We used multivariable Poisson models to determine the effects of our experimental factors (image type, content focus, and disease framing) on ad reach, UCTR, and conversion rate. The multivariable models included all three experimental factors and controlled for recruitment wave. Poisson models for UCTR and conversion rate included an offset or exposure term due to differences in the denominators of these quantities. The analytic dataset used for Poisson analyses included 64 observations (32 created ads that were used in two recruitment waves). We report incidence rate ratios (IRRs) and 95% CI from the multivariable Poisson models. Robust sandwich standard errors were used in calculating the 95% CIs and
Our Facebook recruitment campaign reached a total of 35,646 users who viewed ads for 36,395 times (
The recruitment campaign resulted in 241 conversions (76.0% of potential participants found to be eligible [241/317]) and an overall conversion rate of 0.66% (241/36,395). A total of 150 individuals provided consent and were enrolled into the project (47.3% of potential participants found to be eligible [150/317]). It took about 4 days in July 2016 (ie, recruitment wave 1) and about 2 days in September 2016 (ie, recruitment wave 2) to reach this enrollment goal. The total cost of the campaign was US $413.72. This translates into a cost of US $0.58 per unique click on an ad, US $1.72 per conversion, and US $2.76 per enrolled participant.
The image included in ads affected ad performance. Reach was higher for ads that included an image of a couple (IRR=4.91, 95% CI 2.68-8.97) or a group (IRR=2.65, 95% CI 1.08-6.50) compared with those that included an image of a single person (
Results of Facebook advertisement factorial experiment. The analytic dataset included 64 observations (32 created ads that were used in two recruitment waves). Multivariable models included all variables in the table and controlled for recruitment wave.
Factor | Reacha | UCTRb,c | Conversion ratec,d | ||||||||||
n | Mean | IRRe (95% CI) | Mean | IRR (95% CI) | Mean | IRR (95% CI) | |||||||
Single person | 16 | 210.88 | reff | 1.51 | ref | 0.35 | ref | ||||||
Couple | 16 | 1034.56 | 4.91 (2.68-8.97) | <.001 | 2.52 | 1.66 (0.99-2.80) | .06 | 0.85 | 2.56 (1.13-5.77) | .02 | |||
Group | 16 | 557.94 | 2.65 (1.08-6.50) | .03 | 1.73 | 1.03 (0.62-1.71) | .91 | 0.59 | 1.76 (0.77-4.03) | .18 | |||
Doctor | 16 | 424.50 | 2.01 (0.88-4.62) | .10 | 1.40 | 0.91 (0.50-1.67) | .77 | 0.45 | 1.26 (0.52-3.09) | .61 | |||
HPVg | 32 | 558.34 | ref | 2.05 | ref | 0.62 | ref | ||||||
HPV vaccine | 32 | 555.59 | 1.00 (0.53-1.86) | .99 | 1.97 | 0.98 (0.79-1.22) | .86 | 0.71 | 1.15 (0.93-1.43) | .19 | |||
Cancer | 32 | 528.38 | ref | 1.79 | ref | 0.68 | ref | ||||||
STDh | 32 | 585.56 | 1.11 (0.59-2.07) | .75 | 2.21 | 1.34 (1.06-1.69) | .01 | 0.65 | 0.98 (0.79-1.22) | .85 |
aReach was defined as number of Facebook users ad was shown to.
bUnique click-through rate (UCTR) was defined as the number of unique users who clicked on an ad divided by reach.
cDisplayed as a percentage.
dConversion rate was defined as the number of unique users who reached the study consent form divided by the number of times an ad was viewed.
eIRR: incidence rate ratio.
fref: reference group.
gHPV: human papillomavirus.
hSTD: sexually transmitted disease.
Enrolled participants were from 31 states and the District of Columbia. Most participants were in the age range of 22-25 years (58.7%, 88/150), non-Hispanic white (56.7%, 85/150), and not married or living with a partner (80.0%, 120/150;
Reaching and recruiting young gay and bisexual men may be a challenge faced by interventions to increase HPV vaccination. Social media sites represent a potentially effective strategy for overcoming this challenge, yet little is known about the effectiveness of this strategy in recruiting for such interventions. We successfully enrolled 150 eligible young gay and bisexual men via Facebook ads for a pilot test of Outsmart HPV. Recruitment occurred over a short time period and cost less than US $500 total, translating into less than US $3 spent on recruitment per enrolled participant. The recruitment metrics for our study (eg, UCTR and cost per enrolled participant) are similar to those from past studies that recruited via Facebook ads [
Findings from our study also identify potential strategies for optimizing the performance of Facebook ads. Similar to a past study [
The text included in Facebook ads also affected ad performance in our study. Ads with main text mentioning an STD had a higher UCTR compared with ads mentioning cancer. Past research suggests that adolescents and young adults are less future-orientated (eg, tend to focus on potential short-term vs long-term consequences) than older adults [
The characteristics of participants in our study were comparable with past samples of gay and bisexual men who were recruited through various other strategies. For example, the demographic characteristics of participants in our study (eg, age, race or ethnicity, education level) are highly similar to those of young sexual minority men from the National Survey of Family Growth and the National Longitudinal Study of Adolescent Health [
Study strengths include the use of an experimental design, recruitment of a national sample, and examining several metrics related to ad performance. Our study also has several limitations. Data were not available on the characteristics of Facebook users who were shown an ad but did not click on it. Similarly, we were not able to assess reasons why Facebook users did not click on an ad or clicked on an ad but did not complete the project eligibility screener. We were also not able to link ad performance data with data from study surveys to determine how ad performance may have differed across demographic groups. Future research should identify strategies for linking these data and explore potential differences in ad performance. Fraudulent accounts (eg, multiple accounts) are a growing concern for Web-based research [
Facebook ads are a convenient and cost-efficient strategy for reaching and recruiting young gay and bisexual men for a Web-based HPV vaccination intervention. Importantly, the characteristics of young gay and bisexual men enrolled via this recruitment strategy are similar to those enrolled via other recruitment strategies. Future Facebook recruitment efforts for this population should strongly consider the importance of ad content. Specifically, it may be beneficial to include ads that feature a young adult male couple and that frame HPV and HPV vaccine in the context of an STD.
Demographic and health-related characteristics of young gay and bisexual men enrolled in Outsmart HPV (human papillomavirus; n=150).
Advisory Committee on Immunization Practices
human papillomavirus
Center for Health Communications Research
incidence rate ratio
lesbian, gay, bisexual, and transgender
sexually transmitted disease
unique click-through rate
This study was supported by the National Cancer Institute of the National Institutes of Health under Award Number R21CA194831. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
PLR had received research grants from Merck Sharp & Dohme Corp. and Cervical Cancer-Free America, via an unrestricted educational grant from GlaxoSmithKline. EDP had received research grants from Merck Sharp & Dohme Corp. These funds were not used to support this research study.