“Outsmart HPV” is a mobile-friendly Web-based HPV vaccination intervention developed for young gay and bisexual men. Men were eligible for this project if they (1) were in the age range of 18-25 years, (2) self-identified as gay or bisexual, (3) were a US resident, and (4) had not received any doses of HPV vaccine. Following intervention development, we conducted a pilot test of this project. Recruitment for the pilot test occurred in two waves, with the first wave occurring in July 2016 and the second in September 2016. All recruitment occurred via paid Facebook ads. This report includes recruitment results from the pilot test. The Institutional Review Board at The Ohio State University approved this study. The parent randomized trial is registered at ClinicalTrials.gov (identifier NCT02835755).

We created recruitment ads using Facebook’s Ads Manager program. All ads included a headline, main text, image, Outsmart HPV project logo, and weblink to the project website. Ads adhered to Facebook’s requirements at the time of ad development, including character limits (25 characters for the headline and 90 characters for the main text) and image restrictions (any image used could not include more than 20% text) [28]. We designed all ads to appear in the “News Feed” on Facebook, which is a streaming list of updates from the user’s connections (eg, friends) and advertisers. We focused solely on News Feed ads, as opposed to other Facebook ad locations (eg, right column), since News Feed ads are more effective in terms of recruitment metrics for research studies [26].

The headline (“Earn Up To $95 Online!”), Outsmart HPV logo, and weblink to the project website were identical for all ads. The main text for all ads started with “Online HPV study for gay & bisexual men,” but the remaining main text and image varied across the ads using a 4x2x2 factorial experiment design. Experimental factors included image type, content focus, and disease framing. Image type (4 conditions) refers to the primary image featured in the ad. All ad images featured at least one male who appeared to be a young adult in the target age group, but ads varied on the number and type of people in the image: (1) a single man (“single person”), (2) a male couple (“couple”), (3) a group of more than 2 men (“group”), or (4) a man interacting with a health care provider (“doctor”). The people included in the images were from a range of racial or ethnic groups. Content focus (2 conditions) involved whether the ad’s remaining main text mentioned (1) HPV, or (2) HPV vaccine. Disease framing (2 conditions) involved whether the ad’s remaining main text mentioned (1) cancer, or (2) a sexually transmitted disease (STD). We created 2 ads for each of the 16 experimental conditions (ie, 2 variations of each image type), resulting in a total of 32 ads. Figure 1 shows example ads across various experimental conditions. Facebook approved all ads before their use.

We targeted all ads to Facebook users by sex (male), age (18-25 years), location (United States), and language (English). We further targeted the ads on keywords generated from information that Facebook users add to their Timeline, pages they “like,” or ads they have previously clicked on. The keywords used in our campaign included “bisexuality,” “homosexuality,” “same-sex relationship,” “genderqueer,” “gay pride,” “LGBT community,” “LGBT culture,” or “rainbow flag (LGBT movement).” The chance that a given ad appeared on a user’s page was determined by a Facebook algorithm that considers several factors (eg, the spending limit specified for the ad, competition from other ads).

Ads linked potential participants to the project website or the project’s Facebook page (which in turn directed potential participants to the project website). We collaborated with the Center for Health Communications Research (CHCR) at the University of Michigan to develop and maintain the project website, which was a mobile-friendly website accessible by desktop or laptop, tablet computer, or smartphone (iOS and Android). Once on the project website, potential participants first completed a project eligibility screener. Those determined to be eligible were then asked to provide informed consent and create a project account. Following account creation, participants completed a preintervention Web-based survey, were randomly assigned to either the intervention or control arm, and then completed remaining study activities (eg, viewing intervention or control materials on the Web about HPV vaccine and completing 3 additional Web-based surveys over the course of 7 months). Participants could earn up to US $95 in gift cards during the course of the project.

Facebook provided standard data on several metrics related to ad performance. Similar to past studies involving Facebook recruitment campaigns [17,20,25-27], we examined 3 metrics as outcomes: (1) reach (ie, the number of Facebook users an ad was shown to), (2) unique click-through rate (UCTR; number of unique Facebook users who clicked on an ad divided by reach), and (3) conversion rate (number of unique Facebook users who reached the study consent form divided by the number of times an ad was viewed). We express UCTRs and conversion rates as percentages for the remainder of this paper. We also report information on the cost of the recruitment campaign, as well as the demographic and health-related characteristics of enrolled participants (using data from the project eligibility screener and preintervention survey).

We calculated descriptive statistics for all Facebook metrics and survey data. We used multivariable Poisson models to determine the effects of our experimental factors (image type, content focus, and disease framing) on ad reach, UCTR, and conversion rate. The multivariable models included all three experimental factors and controlled for recruitment wave. Poisson models for UCTR and conversion rate included an offset or exposure term due to differences in the denominators of these quantities. The analytic dataset used for Poisson analyses included 64 observations (32 created ads that were used in two recruitment waves). We report incidence rate ratios (IRRs) and 95% CI from the multivariable Poisson models. Robust sandwich standard errors were used in calculating the 95% CIs and P values. We conducted analyses using Stata version 12.0 (Stata Corp), and all statistical tests were 2-tailed with a critical alpha of .05.

Our Facebook recruitment campaign reached a total of 35,646 users who viewed ads for 36,395 times (Figure 2). There were 717 unique clicks on the ads, resulting in an overall UCTR=2.01% (717/35,646). A total of 551 potential participants completed the project eligibility screener, of whom 317 (57.5%) were eligible for participation in the study. Among those found to be ineligible, reasons for ineligibility (more than one reason could apply per person) included: already received HPV vaccine or did not know vaccination status (90.5%, 212/234), outside of the project’s age range (8.1%, 19/234), did not self-identify as gay or bisexual (6.0%, 14/234), female gender (0.9%, 2/234), and did not confirm US residence (0.4%, 1/234).

The recruitment campaign resulted in 241 conversions (76.0% of potential participants found to be eligible [241/317]) and an overall conversion rate of 0.66% (241/36,395). A total of 150 individuals provided consent and were enrolled into the project (47.3% of potential participants found to be eligible [150/317]). It took about 4 days in July 2016 (ie, recruitment wave 1) and about 2 days in September 2016 (ie, recruitment wave 2) to reach this enrollment goal. The total cost of the campaign was US $413.72. This translates into a cost of US $0.58 per unique click on an ad, US $1.72 per conversion, and US $2.76 per enrolled participant.

The image included in ads affected ad performance. Reach was higher for ads that included an image of a couple (IRR=4.91, 95% CI 2.68-8.97) or a group (IRR=2.65, 95% CI 1.08-6.50) compared with those that included an image of a single person (Table 1). Ads that included an image of a couple also had a higher conversion rate (IRR=2.56, 95% CI 1.13-5.77) and UCTR (OR=1.66, 95% CI 0.99-2.80) than ads that included an image of a single person, though the latter association was of borderline statistical significance in the multivariable model (P=.06). For disease framing, ads with main text mentioning an STD had a higher UCTR compared with ads with main text mentioning cancer (IRR=1.34, 95% CI 1.06-1.69). The content focus of ads did not affect ad performance.

Enrolled participants were from 31 states and the District of Columbia. Most participants were in the age range of 22-25 years (58.7%, 88/150), non-Hispanic white (56.7%, 85/150), and not married or living with a partner (80.0%, 120/150; Multimedia Appendix 1). About 82.7% (124/150) of participants self-identified as gay. Most participants did not have a college degree (62.7%, 94/150) and reported a household income of less than US $50,000 (76.0%, 114/150). Most participants had some form of health insurance (82.0%, 123/150), though fewer than half reported having a routine medical check-up in the last year (46.7%, 70/150). About half of the participants (51.3%, 77/150) reported they were younger than 18 years of age at sexual debut (first vaginal, anal, or oral intercourse), and most reported having at least six male sexual partners during their lifetime (64.7%, 97/150). Few participants reported being positive for human immunodeficiency virus (HIV; 5.3%, 8/150) or a history of either genital warts (6.0%, 9/150) or another STD (18.0%, 27/150).

Reaching and recruiting young gay and bisexual men may be a challenge faced by interventions to increase HPV vaccination. Social media sites represent a potentially effective strategy for overcoming this challenge, yet little is known about the effectiveness of this strategy in recruiting for such interventions. We successfully enrolled 150 eligible young gay and bisexual men via Facebook ads for a pilot test of Outsmart HPV. Recruitment occurred over a short time period and cost less than US $500 total, translating into less than US $3 spent on recruitment per enrolled participant. The recruitment metrics for our study (eg, UCTR and cost per enrolled participant) are similar to those from past studies that recruited via Facebook ads [16,17,25,29]. Our findings not only add to the growing body of literature that shows Facebook ads are a convenient and cost-efficient recruitment strategy for research studies but specifically suggest that they are an effective strategy for recruiting young gay and bisexual men for Web-based HPV vaccination interventions.

Findings from our study also identify potential strategies for optimizing the performance of Facebook ads. Similar to a past study [27], ad images had a meaningful effect on ad performance. Ads that featured an image of a young adult male couple, and to a lesser degree an image of a group of young adult men, performed better than ads that featured a single young adult male. Ads that included an image of a couple may have appealed to the importance of romantic relationships during adolescence and young adulthood [30]. Furthermore, these ads could have prompted potential participants to think about the health of their sexual partners, which may have been motivating since previous research suggests that young sexual minority men view the protection of their partners’ health as an advantage of HPV vaccination [31]. Ads that included an image of a group of young adult men may have affected potential participants’ perceived social norms about the topic and the study. Positive social norms have been previously associated with HPV vaccination behaviors among young gay and bisexual men [7]. Future studies that recruit through Facebook should use ads that feature multiple individuals in the target age range to help improve ad performance and recruitment. Furthermore, we recommend that recruitment ads in future research feature individuals who are racially or ethnically diverse, as we did in this study, so that the ads are relatable and appealing to a wide audience.

The text included in Facebook ads also affected ad performance in our study. Ads with main text mentioning an STD had a higher UCTR compared with ads mentioning cancer. Past research suggests that adolescents and young adults are less future-orientated (eg, tend to focus on potential short-term vs long-term consequences) than older adults [32], and potential participants likely perceived STDs to be a more relevant and immediate health outcome than cancer. Indeed, nearly half of all incident STDs in the United States occur among adolescents and young adults [33], whereas cancer is much less common among young adults than older ages [34]. Our finding is also consistent with past research examining the effects of message framing on HPV vaccine acceptability, as young adults tended to be more receptive to HPV vaccine when it was presented in the context of an STD or genital warts [35,36]. This is in contrast to older adults, who tended to be more receptive to HPV vaccine when it was presented as preventing cancer [37,38]. It is therefore important that future efforts consider framing HPV and HPV vaccine in the context of an STD when targeting materials including study recruitment materials, toward young adults.

The characteristics of participants in our study were comparable with past samples of gay and bisexual men who were recruited through various other strategies. For example, the demographic characteristics of participants in our study (eg, age, race or ethnicity, education level) are highly similar to those of young sexual minority men from the National Survey of Family Growth and the National Longitudinal Study of Adolescent Health [39,40]. Furthermore, about 80% of participants in our study self-identified as gay, which closely resembles other national samples of gay and bisexual men [7,41-43]. Health-related characteristics (eg, lack of health insurance and receipt of routine medical check-up in the last year) were also similar between participants in our study and those from past studies [7,40,44,45]. These similarities should not be altogether surprising given the ubiquitous nature of Facebook use among young adults [46], and they suggest that Facebook ads can recruit diverse samples of young gay and bisexual men that are comparable with what would be obtained through more traditional recruitment methods. Future research should further explore these comparisons, as well as continue to monitor how samples of young gay and bisexual men recruited via Facebook compare with those recruited through other social media sites that may have different audiences (eg, Instagram, Grindr, and MiGente). Data from the American Men’s Internet Survey suggest that characteristics of sexual minority men differ by the social media site used of recruitment [47], and it is important to continue to monitor these potential differences as social media evolve.

Study strengths include the use of an experimental design, recruitment of a national sample, and examining several metrics related to ad performance. Our study also has several limitations. Data were not available on the characteristics of Facebook users who were shown an ad but did not click on it. Similarly, we were not able to assess reasons why Facebook users did not click on an ad or clicked on an ad but did not complete the project eligibility screener. We were also not able to link ad performance data with data from study surveys to determine how ad performance may have differed across demographic groups. Future research should identify strategies for linking these data and explore potential differences in ad performance. Fraudulent accounts (eg, multiple accounts) are a growing concern for Web-based research [48], and we used several recommended strategies to reduce this possibility. We used a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) during the project account creation process, inspected participants’ account and contact information (eg, email address) for similarities between accounts, and inspected survey data for inconsistent or illogical responses [48]. Our pilot test had a somewhat modest sample size, and future efforts should examine how ad performance and metrics may change over time when recruiting large samples of young gay and bisexual men.

Facebook ads are a convenient and cost-efficient strategy for reaching and recruiting young gay and bisexual men for a Web-based HPV vaccination intervention. Importantly, the characteristics of young gay and bisexual men enrolled via this recruitment strategy are similar to those enrolled via other recruitment strategies. Future Facebook recruitment efforts for this population should strongly consider the importance of ad content. Specifically, it may be beneficial to include ads that feature a young adult male couple and that frame HPV and HPV vaccine in the context of an STD.