Published on 12.09.16 in Vol 2, No 2 (2016): Jul-Dec
How Best to Obtain Valid, Verifiable Data Online From Male Couples? Lessons Learned From an eHealth HIV Prevention Intervention for HIV-Negative Male Couples
This is a corrected version. See correction statement: http://publichealth.jmir.org/2016/2/e155/
Background: As interest increases in the development of eHealth human immunodeficiency virus (HIV)-preventive interventions for gay male couples, Web-based methods must also be developed to help increase the likelihood that couples enrolled and data collected from them represent true unique dyads. Methods to recruit and collect reliable and valid data from both members of a couple are lacking, yet are crucial for uptake of novel sexual health and HIV-prevention eHealth interventions. Methods to describe best practices to recruit male couples using targeted advertisements on Facebook are also lacking in the literature, yet could also help in this uptake.
Objective: The objective of our study was to describe challenges and lessons learned from experiences from two phases (developmental phase and online randomized controlled trial [RCT]) of an eHealth HIV-prevention intervention for concordant HIV-negative male couples in terms of (1) recruiting male couples using targeted advertisements on Facebook, (2) validating that data came from two partners of the couple, and (3) verifying that the two partners of the couple are in a relationship with each other.
Methods: The developmental phase refined the intervention via in-person focus groups, whereas the pilot-testing phase included an online RCT. For both phases, couples were recruited via targeted Facebook advertisements. Advertisements directed men to a study webpage and screener; once eligible, participants provided consent electronically. A partner referral system was embedded in the consenting process to recruit the relationship partner of the participant. Both men of the couple had to meet all eligibility criteria—individually and as a couple—before they could enroll in the study. Verification of couples’ relationships was assessed via the concurrence of predetermined screener items from both partners, done manually in the developmental phase and electronically in the pilot-testing phase. A system of decision rules was developed to assess the validity that data came from two unique partners of a couple.
Results: Several important lessons were learned from these experiences, resulting in recommendations for future eHealth studies involving male couples. Use of certain “interests” and types of images (eg, shirtless) in targeted Facebook advertisements should be avoided or used sparingly because these interests and types of images may generate adverse reactions from a broader audience. Development of a systematic approach with predetermined criteria and parameters to verify male couples’ relationships is strongly recommended. Further, researchers are encouraged to develop a system of decision rules to detect and handle suspicious data (eg, suspicious email addresses/names, multiple entries, same IP address used in multiple entries) to help validate the legitimacy of male couples’ relationships online.
Conclusions: These lessons learned combined with recommendations for future studies aim to help enhance recruitment efforts and the validity and reliability of collecting dyadic data from male couples for novel eHealth HIV-preventive interventions.
JMIR Public Health Surveill 2016;2(2):e152
Use of the Internet, including social media, has dramatically increased within the past decade. As evidenced in the Pew Research Center’s Internet and American Life Project, the prevalence of Internet use among US adults rose from 14% to 87% between 1995 and 2014 , and social media use increased from 8% to 76% between 2005 and 2015 [ ]. Not only has the Internet allowed individuals and communities to connect and communicate via various multimedia features (eg, chatting, picture sharing, messaging), it has also increasingly been used as a platform to disseminate health information and promote the uptake of healthy behaviors across all scientific disciplines [ - ]. These online environments have created a wide yet complex venue to recruit and collect psychosocial and behavioral data to further health promotion and prevention efforts across different populations.
For example, the Internet has increasingly been used for human immunodeficiency virus (HIV) prevention research with various populations due to the efficiency of targeting and collecting data from specific subpopulations, such as gay, bisexual, and other men who have sex with men (MSM) [- ]. Despite this efficiency, characteristics of MSM who are recruited online versus in person may differ for some characteristics and not in others. For example, MSM recruited online versus venue-based, time-spaced sampling (ie, offline) did not statistically differ regarding their HIV and sexually transmitted infection (STI) testing patterns, prevalence of HIV and other STIs, and study retention rates up to 24 months [ ]. However, online studies with MSM and male couples suggest that Internet-based samples of MSM tend to self-report as being more educated and non-Hispanic white [ , ]. As such, online studies with MSM and male couples may limit the potential for generalizability depending on who participates in the research study.
With respect to prevention, some MSM use the Internet to meet other MSM for sex, friendships, relationships, and for various types of social support [, , ]. Although MSM are not the only population to use the Internet for these reasons, research has also noted that Internet use has been associated with having higher numbers of sex partners, greater frequency of condomless anal sex (receptive and insertive), and increased levels of recreational drug use (eg, methamphetamines) among MSM [ , ]. Given these reasons, HIV-prevention efforts that are conducted online may be beneficial to reach some of these MSM in addition to the efforts that are provided in person.
Although methods of recruiting online are more time efficient than recruiting in person, there are challenges related to the reliability and validity of collecting data online from participants . For example, anonymity and lack of direct face-to-face contact with participants prohibit researchers to know who and where data are originating from when it is collected online. In addition, randomly generated responses to survey items may occur in order to help expedite completion of the survey, particularly if an incentive is involved. Monetary incentives have been frequently associated with increased participant misrepresentation for eligibility [ , ] and multiple data entry [ , ] by individuals who wish to increase the amount of incentives and/or probabilities of winning. Survey response rates may also decrease over time, which could threaten the validity of generalizing the results to a broader audience. Bauermeister and colleagues [ ] examined how invalid data collected online may influence statistical relationships, decrease statistical power, and increase the likelihood of biased conclusions. They did not recommend using a conservative approach and excluding suspicious entries because valid data could accidentally be removed in the process. Instead, they recommended developing both pre and post hoc decisions to handle all data, such as grouping entries into different categories (eg, valid, suspicious, and invalid) and obtaining a population list of all eligible participants for verification processes, respectively [ ].
These issues of reliability and validity of online data become more complex when conducting online research with dyads (eg, gay male couples). By nature, handling dyadic data requires at least double the amount of time and effort than that required of individual-level data. The criteria used to determine the reliability and validity of an individual participant must be assessed not only twice, but must also be crosschecked with that of that participants’ partner’s data to (1) validate the data are coming from two unique individuals and (2) verify that these two unique individuals are in a relationship together. The eligibility criteria used in couples Web-based research is also inherently more complex due to the need to ensure that both individual- and couple-level eligibility requirements are met. However, the anonymity that the Internet warrants makes it easier for participants to pose as a fraudulent “couple” by pretending to be both partners in the relationship. If incentives are used, then this further increases the need to ensure that dyadic data collected online are valid and reliable because some couples’ motivation to become eligible may be less than honest. For these reasons and due to the increased use of the Internet among adults, methods are needed to help ensure that dyadic data collected online from couples are valid and verifiable of their relationships.
In recent years, attention toward male couples for HIV prevention has increased because many MSM in the United States—up to 67%—acquire HIV while in a same-sex relationship [, ]. In addition, few preventive interventions approved by the Centers for Disease Control and Prevention currently exist for this population [ - ]. Despite how common Internet use is among MSM and its efficiency to enhance recruitment and data collection efforts for prevention research, few studies have fully harnessed the capabilities of eHealth (ie, Web-based research and/or preventive programs hosted online) to help advance HIV-prevention efforts for male couples [ ]. Moreover, the majority of HIV-preventive interventions for MSM have focused on behavior change at the individual level [ - ], thereby emphasizing the need to not only develop eHealth HIV-preventive interventions for male couples, but to also develop methods to maximize recruitment efforts and verification of valid data from both members of the couple (ie, dyadic data) online. Methods to recruit and collect reliable and valid dyadic data online are lacking, yet are crucial for development and uptake of future eHealth HIV-preventive interventions for male couples.
To help fill this critical gap, this paper aims to provide methodological recommendations in three distinct areas to (1) maximize efficiency and accuracy of using targeted, online Facebook advertisements to recruit male couples; (2) facilitate the collection of valid dyadic data; and (3) ensure verification that dyadic data collected are representative of two males who are in a relationship together (ie, male couple). To accomplish these aims, data and related experiences captured from the development and pilot testing of an eHealth HIV-prevention intervention with concordant HIV-negative male couples in the United States are used. The methods used and lessons learned from these experiences provide concrete suggestions and recommended safeguards that may benefit future eHealth endeavors targeting similar populations of male couples.
The University of Miami Institutional Review Board approved all study procedures. For both the development and pilot testing of the eHealth HIV-prevention intervention, interested men who clicked on the Facebook advertisement were directed to the study webpage and an eligibility screener via SurveyGizmo, a Health Insurance Portability and Accountability Act-compliant Web-based survey tool and database server. Once eligible, participants provided consent electronically. A partner referral system was embedded in the consent process to recruit the relationship partner of the index participant. Both men of the couple had to meet all eligibility criteria and be deemed valid—as an individual and as a couple—before they could enroll into either the development or pilot-testing portion of the study; the development portion included both partners of the couple participating in one in-person focus group, whereas the pilot testing of the intervention included a Web-based randomized controlled trial (RCT) with couples participating online.
Targeted Recruitment via Facebook
Male couples were recruited via targeted advertisements placed on Facebook. All advertisements targeted potential research participants who were male, living in the United States, at least 18 years of age, interested in men, and had a relationship status of either being married, engaged, in a relationship, domestic partnership, or civil union. Each advertisement included a picture of a male couple with a brief title, message, and a Web link to the study eligibility screener. In total, 13 advertisement campaigns were conducted for the focus group, development phase, and RCT pilot-testing phase of the project, and each Facebook advertisement campaign lasted for 72 hours. The campaigns used to recruit for the focus groups occurred between March 2015 and May 2015, whereas the campaigns used to recruit for the online RCT occurred between October 2015 and March 2016. Following Facebook’s guidelines and word limits, all advertisements contained titles similar to “In a relationship?” with a brief message such as “Male couples wanted to try out a cool, new online health & relationship program. Earn $$!” Each campaign’s total cost ranged from US $499.97 to US $3997.62 depending on performance, audience, and placement of the advertisements (eg, desktop newsfeed, mobile newsfeed). Per campaign, the advertisements resulted in a mean of 110,478 people reached and a mean of 3534 clicks to the study website. Each targeted Facebook advertisement campaign provided useful metrics about how well the targeted advertisement performed, including (1) the total number of people reached, (2) number of people who clicked on the advertisement, (3) how relevant the targeted audience thought the advertisement pertained to them (ie, relevancy score ranging from 0 to 10 with 10 being most relevant), and (4) negative and positive feedback about the advertisement obtained from those who were shown the advertisement while using Facebook.
Both members of the male couple had to meet the following eligibility criteria to participate in the focus group or RCT of the intervention project: (1) self-report as male, (2) be at least 18 years of age, (3) be in a current sexual relationship with a main relationship partner for at least 6 months, (4) self-report as HIV-negative, (5) practice condomless anal sex with the main relationship partner for at least 6 months, (6) self-report no recent history of intimate partner violence or coercion within the previous year, (7) have not established a sexual agreement in the relationship, and (8) own a mobile phone and have an alternate method to access the Internet (eg, computer).
Procedures Used to Verify Couples’ Relationships and Validity of Dyadic Data Collected
Both individual- and couple-level criteria were used to deem couples’ eligibility to participate in the study. Based on our prior work with male couples [, ], we developed a system of decision rules to assess the verification of a couples’ relationship based on both partners’ responses to screener items (ie, to verify that a couple is in a relationship). Verification of couples’ relationships was based on data collected from both members of the couple and whether they concurred (ie, agreed) on certain predetermined eligibility screener items (see ), including (1) matching their relationship length within a margin of 1 month, (2) exact matching of each other’s birthday months, (3) matching each other’s ages within a margin of one year, and (4) exact matching of at least one of their two contact information options (eg, phone number, email address). All couples had to concur on all four items to be considered eligible. Each participant and his partner’s data were crosschecked to match and concur on all four of the predetermined screener items. Suspicious cases or cases in which couples did not concur on one or two items were assessed a second time after calling or emailing the potential participant to verify their corresponding partners’ information. Verification of couples’ relationships was done manually by entering data into a secure database during the developmental focus group phase; for the pilot RCT, this verification step was done electronically via an embedded Web-based system that contained an algorithm.
For both phases of the intervention project, validation of both partners’ data collected was assessed on a case-by-case basis. For the development phase, multiple entries of the same Internet Protocol (IP) address were examined to validate whether or not one or both partners were unique individuals. For example, each entry of the eligibility screener had an IP address associated with it. Some IP addresses were associated with several entries, potentially suggesting that individuals had attempted to complete the eligibility screener multiple times as either the same person or pretending to be different people. Eligibility screener entries with the same IP address were flagged and noted for further investigation by using the other validation markers.
Because the online pilot RCT aimed to recruit a large number of male couples (compared to the focus groups), an electronic algorithm was created to electronically match both partners as a couple based on predetermined eligibility criteria and verification rules. For the online RCT phase, suspicious names and/or email addresses, duplicate entries (ie, same two email addresses used for two different sets of couples), and similar back-to-back screener entries were examined in addition to the validation markers used in the developmental focus group phase.
Lessons learned from this intervention project are an example of the challenges associated with collecting and verifying valid dyadic data from male couples online. The lessons learned from this project are not exhaustive, yet provide insights and suggestions to help improve these efforts, particularly with the increased attention toward developing eHealth and mHealth preventive interventions for male couples [- ]. Specifically, lessons learned from this project entail (1) advantages and disadvantages of using certain selection criteria to target advertisements on Facebook, (2) selection of images that resonate with the targeted audience, (3) use of an algorithm versus manual input of data to verify male couples’ relationships via predetermined decision rules, and (4) monitoring of dyadic data collected to assess validity of unique responses obtained from both partners of the male couple.
Targeted Advertisements for Male Couples on Facebook
Some images used in the advertisements received higher relevancy scores than others and received less negative feedback from supportive individuals who were not lesbian, gay, bisexual, and transgender (LGBT). Negative feedback was received in three forms: (1) public comments posted on the Facebook advertisements and study Facebook community page, (2) messages sent privately to the study Facebook community page, and (3) voicemail.provides examples of the negative feedback received from some of the images used for the targeted advertisements placed on Facebook.
After some investigation and consultation with Facebook (private telephone communication with Facebook, January 13, 2016), interests that were used to target Facebook members, such as the LGBT community or gay news, included members who had positive views of this interest as well as those who had negative views of this same interest. Because the negative-view individuals were also targeted, this resulted in a flood of backlash in the form of unsupportive and homophobic comments posted on the study Facebook webpage, study team’s voicemail, and as comments left in association with the targeted advertisement. After these experiences, targeting criteria were adjusted for the remainder of the advertisement campaigns by actively excluding Facebook members who had the following information on or associated with their profiles: interested in women or both men and women; had a relationship status as single, unspecified, open, complicated, separated, divorced, or widowed; and had moderate, conservative, and very conservative political views. Targeted advertisements were also changed by using less overtly sexual images of male couples (eg, clothed male couple slightly embracing).provides examples of Facebook advertisements that were used before and after these modifications were made.
Verification of Couples’ Relationships
A systematic manual (ie, color-coded) approach was used during the project’s developmental phase to verify whether both partners of the male couple were in a relationship (after both men had electronically consented) using predetermined eligibility screener items. This post hoc comparative method of the dyadic data collected from the eligibility screener enabled the characterization of couples into one of three groups: (1) eligible and verified couples who met all predetermined decision rules, (2) eligible but unverified couples because only two or three of the predetermined decision rules were met, and (3) ineligible couples. This systematic manual method illuminated the strengths and limitations of this approach to verify couples’ relationships. Although manually matching entries allowed for greater control over the assessment of couples’ relationships, this method proved inefficient in terms of time and resources.
Based on this experience, the amount of time required to verify couples’ relationships with this approach emphasized the necessity to electronically automate this process, particularly if data were to be collected online from large samples of male couples such as during an eHealth intervention. As such, a Web-based electronic algorithm was created to automatically verify couples’ relationships based on data collected from both members of the couple and whether they concurred on predetermined eligibility screener items. The algorithm was developed based on the results and experiences obtained during the developmental focus group phase. However, the algorithm had an important limitation. The system would report couples as partially matched when their contact information mismatched (eg, one partner reported his partners’ email address different from what his partner reported as his actual email address). This mismatch required a manual verification of the couples’ relationship by contacting the potential participants to further assess the couples’ relationship and their eligibility to enroll into the project.
Validation of Dyadic Data Collected
For time- and resource-efficiency reasons, validation of couples’ data (ie, data came from two unique individuals) occurred after the algorithm electronically matched and verified that a couple was in a legitimate relationship. This approach helped to expedite the recruitment process by filtering out ineligible couples and couples whose relationships were not verified. Verified couples could then be validated. Validation of couples’ data was still required because some instances noted that the same IP addresses were being used in multiple screener entries. In some of these sets of entries with the same IP addresses, it was clear that the respondent(s) attempted the screener multiple times to determine the eligibility criteria to enroll in the study. Some individuals also created generic email addresses to appear as a couple for inclusion into the study. Because of these instances, the awareness of fraudulent yet eligible entries increased, thus warranting the creation of additional safeguards for a post hoc assessment of verified screener entries. Specifically, similar and/or suspicious email addresses (eg, “email@example.com” and reporting partner’s to be “firstname.lastname@example.org”) or names (eg, Melissa Wise) were noted and flagged. Also noted were similar yet different screener back-to-back entries to determine eligibility, same IP addresses used in multiple screener entries, and duplicate entries (ie, same two emails used for two different sets of couples). For these cases, the individuals and their “partners” were called and asked to verify, via phone, specific questions to assess their validity.illustrates the difference between the manual and electronic methods of matching couples (verification of couples’ relationships) as well as examples of the post hoc assessment of screener entries (validation that data were from two unique individuals).
|Information used to verify and validate couples’ relationships||Developmental focus group phase||Pilot-testing randomized controlled trial phase|
|Number of screener items||17||17|
|Number of items used to determine eligibility||10||11|
|Predetermined items used to assess verification of relationship via comparison of self-reports by both partners of the couple||Individual’s birthday month; partner’s birthday month; individual’s age; partner’s age; relationship length; sexual agreement; individual’s email; partner’s email; individual’s phone number; partner’s phone number||Post hoc assessment|
|Method used to match and crosscheck predetermined items for relationship verification||Manual||Electronic via algorithm|
|Post hoc assessment of screener entries for validation purposesa||Similar/suspicious email addresses and names (eg, Kylemcap1@gmail.com and email@example.com; Melissa Wise); back-to-back completions: time-stamped as 20-30 minutes apart (duration of survey); similarity of responses: consistent extreme/haphazard responses throughout survey; same IP addresses used in multiple entries: same IP address appearing in multiple screener entries with different responses (potentially to determine and pass eligibility criteria); duplicate entries: same two emails used for two different sets of couple IDs (eg, same two emails used for two different IDs: 2452EW and 2864OD)|
a Experience suggested the continued need for post hoc assessment when using an electronic method and/or dealing with large sample sizes.
Several important lessons were learned about recruiting and collecting data from male couples from this project, resulting in recommendations for future Web-based studies specific to male couples. Because many MSM and/or male couples used the study Facebook webpage to ask questions or share stories/details, it was important to provide a friendly and supportive environment for these men to connect and communicate, free from public hostile comments.
For placement of targeted advertisements on Facebook, certain “interests” (eg, LGBT news) and images (ie, shirtless) should be avoided or used sparingly because they may generate adverse reactions from a broader audience. Although all images used in the advertisements were appropriate, some images may have been too explicit or hypersexualized for those outside the targeted audience, and especially to those who explicitly have a negative attitude toward the LGBT community. Researchers may also want to consider contacting established Facebook community groups (eg, Gay Miami Beach) who may have members of the targeted population to help promote the study. With respect to future work, researchers should consider how other characteristics of partnered MSM and male couples may influence the likelihood of them responding to targeted advertisements placed on Facebook. For instance, some images and recruitment text may resonate better with certain subgroups of male couples (eg, younger vs older male couples, ethnicity/race) and not others. Another consideration includes the target population’s computer and research literacy. As such, formative work is recommended to investigate which images, text, and placement of the advertisement (eg, desktop newsfeed, mobile newsfeed) would best help enhance the target populations’ response to clicking on the Facebook advertisement.
Experiences associated with screening potential male couples and verifying their relationships reinforces the need to develop and use a systematic approach that includes predetermined criteria with decision rules. These parameters will help enhance the efficiency as well as the reliability of collecting data from both members of the male couple. However, although time efficient, algorithms to electronically verify couples’ relationships may be imperfect and unable to capture certain nuances of their relationships. For example, two partners may define their relationship length differently if they had a break or period of separation occur during their relationship. Due to the discordance in their self-reports about relationship length, the predetermined decision rule associated with this questionnaire item would result in classifying this particular couples’ relationship as unverified. Based on the algorithm, this couple would then be deemed as unverified because of this one criterion they disagreed on. Researchers should consider which criteria they plan to use with corresponding decision rules because these decisions may affect the eligibility and verification of couples’ relationships in their Web-based projects that are specific to male couples.
It also is important to note that validation of dyadic data should not be assessed in isolation. Validation of dyadic data collected online should be based on examining multiple, if not all, predetermined criteria. Examining only one criterion (eg, same IP addresses, suspicious emails) may lead to disqualifying a potentially eligible verified couple. For instance, both partners of a male couple may live together and use the same computer, which would report them having the same IP address when completing a Web-based questionnaire. Email addresses that appear deceiving or misrepresentative of a potential participant warrant further investigation. For instance, an email address of “firstname.lastname@example.org” could be flagged as suspicious because the first name was Heather, a name typically associated with being a female. However, further investigation of this participant could reveal that his name was “Heath” and not “Heather.”
The Internet is a useful platform to target and enroll at-risk MSM and male couples from a variety of locales. However, it is evident that recruiting and collecting valid and reliable dyadic data online is more complex than that at the individual level. The varying results of the Facebook advertisement campaigns to recruit male couples further stresses the need to monitor posted Facebook advertisements to help minimize the receipt of negative feedback and that the target audience is being reached. Developing a systematic approach with predetermined criteria and parameters is strongly recommended to verify male couples’ relationships as efficiently as possible. Researchers are encouraged to develop a system of decision rules to detect and handle suspicious data to help validate the legitimacy that the dyadic data are coming from two unique partnered individuals. With the decision rules and criteria set, researchers should be cautious in observing multiple criteria as a whole, rather than in isolation. These lessons learned combined with recommendations for future studies may help other researchers enhance recruitment efforts and the validity and reliability of collecting dyadic data from male couples for novel preventive eHealth interventions specific to this population.
Special thanks are extended to the participants for their time and effort in participating in this study. Without their participation, improving these methods would not have been possible. This work was supported by the National Institute of Mental Health under Grant R34 MH102098 (PI Mitchell J).
Conflicts of Interest
- Pew Research Center. 2015. Internet use over time URL: http://www.pewinternet.org/data-trend/internet-use/internet-use-over-time/ [accessed 2016-07-06] [WebCite Cache]
- Pew Research Center. 2015. Social media use over time URL: http://www.pewinternet.org/data-trend/social-media/social-media-use-all-users/ [accessed 2016-07-06] [WebCite Cache]
- Kubicek K, Carpineto J, McDavitt B, Weiss G, Kipke MD. Use and perceptions of the internet for sexual information and partners: a study of young men who have sex with men. Arch Sex Behav 2011 Aug;40(4):803-816 [FREE Full text] [CrossRef] [Medline]
- Magee JC, Bigelow L, Dehaan S, Mustanski BS. Sexual health information seeking online: a mixed-methods study among lesbian, gay, bisexual, and transgender young people. Health Educ Behav 2012 Jun;39(3):276-289. [CrossRef] [Medline]
- Minichiello V, Rahman S, Dune T, Scott J, Dowsett G. E-health: potential benefits and challenges in providing and accessing sexual health services. BMC Public Health 2013;13:790 [FREE Full text] [CrossRef] [Medline]
- Grossman CI, Forsyth A, Purcell DW, Allison S, Toledo C, Gordon CM. Advancing novel HIV prevention intervention research with MSM--meeting report. Public Health Rep 2011;126(4):472-479 [FREE Full text] [Medline]
- Rosser BR, Wilkerson JM, Smolenski DJ, Oakes JM, Konstan J, Horvath KJ, et al. The future of Internet-based HIV prevention: a report on key findings from the Men's INTernet (MINTS-I, II) Sex Studies. AIDS Behav 2011 Apr;15 Suppl 1:S91-S100 [FREE Full text] [CrossRef] [Medline]
- Strömdahl S, Lu X, Bengtsson L, Liljeros F, Thorson A. Implementation of Web-based respondent driven sampling among men who have sex with men in Sweden. PLoS One 2015;10(10):e0138599 [FREE Full text] [CrossRef] [Medline]
- Thériault N, Bi P, Hiller JE, Nor M. Use of web 2.0 to recruit Australian gay men to an online HIV/AIDS survey. J Med Internet Res 2012;14(6):e149 [FREE Full text] [CrossRef] [Medline]
- Zou H, Wu Z, Yu J, Li M, Ablimit M, Li F, et al. Internet-facilitated, voluntary counseling and testing (VCT) clinic-based HIV testing among men who have sex with men in China. PLoS One 2013;8(2):e51919 [FREE Full text] [CrossRef] [Medline]
- Hernandez-Romieu AC, Sullivan PS, Sanchez TH, Kelley CF, Peterson JL, Del RC, et al. The comparability of men who have sex with men recruited from venue-time-space sampling and facebook: a cohort study. JMIR Res Protoc 2014;3(3):e37 [FREE Full text] [CrossRef] [Medline]
- Mitchell JW, Sophus AI, Lee J, Petroll AE. Anal douche practices and willingness to use a rectal microbicide enema for HIV prevention and associated factors among an internet sample of HIV-negative and HIV-discordant male couples in the US. AIDS Behav 2015 Nov 23:1250-1254. [CrossRef] [Medline]
- Sanchez T, Zlotorzynska M, Sineath C, Kahle E, Sullivan P. The annual American Men's Internet Survey of Behaviors of Men Who have Sex with Men in the United States: 2014 key indicators report. JMIR Public Health Surveill 2016;2(1):e23 [FREE Full text] [CrossRef] [Medline]
- Al-Tayyib AA, McFarlane M, Kachur R, Rietmeijer CA. Finding sex partners on the internet: what is the risk for sexually transmitted infections? Sex Transm Infect 2009 Jun;85(3):216-220. [CrossRef] [Medline]
- Kakietek J, Sullivan PS, Heffelfinger JD. You've got male: internet use, rural residence, and risky sex in men who have sex with men recruited in 12 U.S. cities. AIDS Educ Prev 2011 Apr;23(2):118-127 [FREE Full text] [CrossRef] [Medline]
- Benotsch EG, Kalichman S, Cage M. Men who have met sex partners via the Internet: prevalence, predictors, and implications for HIV prevention. Arch Sex Behav 2002 Apr;31(2):177-183. [Medline]
- Ko N, Tseng P, Huang Y, Chen Y, Hsu S. Seeking sex partners through the internet and mobile phone applications among men who have sex with men in Taiwan. AIDS Care 2016 Jul;28(7):927-931. [CrossRef] [Medline]
- Pequegnat W, Rosser BR, Bowen AM, Bull SS, DiClemente RJ, Bockting WO, et al. Conducting Internet-based HIV/STD prevention survey research: considerations in design and evaluation. AIDS Behav 2007 Jul;11(4):505-521. [CrossRef] [Medline]
- Kramer J, Rubin A, Coster W, Helmuth E, Hermos J, Rosenbloom D, et al. Strategies to address participant misrepresentation for eligibility in Web-based research. Int J Methods Psychiatr Res 2014 Mar;23(1):120-129 [FREE Full text] [CrossRef] [Medline]
- Quach S, Pereira JA, Russell ML, Wormsbecker AE, Ramsay H, Crowe L, et al. The good, bad, and ugly of online recruitment of parents for health-related focus groups: lessons learned. J Med Internet Res 2013 Jan;15(11):e250 [FREE Full text] [CrossRef] [Medline]
- Morabia A, Zheng Y. On the influence of a raffle upon responses to an urban transportation survey in New York City. Int J Public Health 2009;54(1):31-34. [CrossRef] [Medline]
- Bauermeister J, Pingel E, Zimmerman M, Couper M, Carballo-Diéguez A, Strecher VJ. Data quality in web-based HIV/AIDS research: handling invalid and suspicious data. Field Methods 2012 Aug 1;24(3):272-291 [FREE Full text] [CrossRef] [Medline]
- Goodreau SM, Carnegie NB, Vittinghoff E, Lama JR, Sanchez J, Grinsztejn B, et al. What drives the US and Peruvian HIV epidemics in men who have sex with men (MSM)? PLoS One 2012;7(11):e50522 [FREE Full text] [CrossRef] [Medline]
- Sullivan PS, Salazar L, Buchbinder S, Sanchez TH. Estimating the proportion of HIV transmissions from main sex partners among men who have sex with men in five US cities. AIDS 2009 Jun 1;23(9):1153-1162. [CrossRef] [Medline]
- Burton J, Darbes LA, Operario D. Couples-focused behavioral interventions for prevention of HIV: systematic review of the state of evidence. AIDS Behav 2010 Feb;14(1):1-10 [FREE Full text] [CrossRef] [Medline]
- Grossman CI, Forsyth A, Purcell DW, Allison S, Toledo C, Gordon CM. Advancing novel HIV prevention intervention research with MSM--meeting report. Public Health Rep 2011;126(4):472-479 [FREE Full text] [Medline]
- Herbst JH, Beeker C, Mathew A, McNally T, Passin WF, Kay LS, Task Force on Community Preventive Services. The effectiveness of individual-, group-, and community-level HIV behavioral risk-reduction interventions for adult men who have sex with men: a systematic review. Am J Prev Med 2007 Apr;32(4 Suppl):S38-S67. [CrossRef] [Medline]
- El-Bassel N, Gilbert L, Witte S, Wu E, Hunt T, Remien RH. Couple-based HIV prevention in the United States: advantages, gaps, and future directions. J Acquir Immune Defic Syndr 2010 Dec;55 Suppl 2:S98-S101. [CrossRef] [Medline]
- Wu E, El-Bassel N, Donald ML, Fontaine Y, Hess L. Adaptation of a couple-based HIV intervention for methamphetamine-involved African American men who have sex with men. Open AIDS J 2010;4:123-131 [FREE Full text] [CrossRef] [Medline]
- Hirshfield S, Chiasson MA, Joseph H, Scheinmann R, Johnson WD, Remien RH, et al. An online randomized controlled trial evaluating HIV prevention digital media interventions for men who have sex with men. PLoS One 2012;7(10):e46252 [FREE Full text] [CrossRef] [Medline]
- Wei C, Herrick A, Raymond HF, Anglemyer A, Gerbase A, Noar SM. Social marketing interventions to increase HIV/STI testing uptake among men who have sex with men and male-to-female transgender women. Cochrane Database Syst Rev 2011(9):CD009337. [CrossRef] [Medline]
- Ybarra ML, Bull SS, Prescott TL, Birungi R. Acceptability and feasibility of CyberSenga: an Internet-based HIV-prevention program for adolescents in Mbarara, Uganda. AIDS Care 2014 Apr;26(4):441-447 [FREE Full text] [CrossRef] [Medline]
- Young SD, Zhao M, Teiu K, Kwok J, Gill H, Gill N. A social-media based HIV prevention intervention using peer leaders. J Consum Health Internet 2013 Oct 1;17(4):353-361 [FREE Full text] [CrossRef] [Medline]
- Mitchell JW. Characteristics and allowed behaviors of gay male couples' sexual agreements. J Sex Res 2014;51(3):316-328 [FREE Full text] [CrossRef] [Medline]
- Mitchell JW, Lee JY, Woodyatt C, Bauermeister J, Sullivan P, Stephenson R. Perceived challenges and rewards of forming a sexual agreement among HIV-negative male couples. Arch Sex Behav 2016 Aug;45(6):1525-1534. [CrossRef] [Medline]
- Cordova D, Bauermeister JA, Fessler K, Delva J, Nelson A, Nurenberg R, et al. A community-engaged approach to developing an mHealth HIV/STI and drug abuse preventive intervention for primary care: a qualitative study. JMIR Mhealth Uhealth 2015;3(4):e106 [FREE Full text] [CrossRef] [Medline]
- Bailey JV, Webster R, Hunter R, Freemantle N, Rait G, Michie S, et al. The Men's Safer Sex (MenSS) trial: protocol for a pilot randomised controlled trial of an interactive digital intervention to increase condom use in men. BMJ Open 2015;5(2):e007552 [FREE Full text] [CrossRef] [Medline]
- Muessig KE, Baltierra NB, Pike EC, LeGrand S, Hightow-Weidman LB. Achieving HIV risk reduction through HealthMpowerment.org, a user-driven eHealth intervention for young Black men who have sex with men and transgender women who have sex with men. Digit Cult Educ 2014;6(3):164-182 [FREE Full text] [Medline]
|HIV: human immunodeficiency virus|
|IP: Internet Protocol|
|LGBT: lesbian, gay, bisexual, and transgender|
|MSM: men who have sex with men|
|RCT: randomized controlled trial|
|STI: sexually transmitted infection|
Edited by G Eysenbach; submitted 22.07.16; peer-reviewed by V Minichiello, S Stahlman; comments to author 17.08.16; revised version received 21.08.16; accepted 21.08.16; published 12.09.16
©Jason Mitchell, Ji-Young Lee, Rob Stephenson. Originally published in JMIR Public Health and Surveillance (http://publichealth.jmir.org), 12.09.2016.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on http://publichealth.jmir.org, as well as this copyright and license information must be included.