Let’s Talk aBOT Scam Online Survey Completions in Health Behavior Research: Tutorial With Case Studies, Practical Guidance, and a Checklist for Researchers

Introduction

Online data collection is an efficient and cost-effective method to reach a wide audience [1]. However, reports of online surveys being completed by bots (software programs that can complete repeated tasks at speed [2]); by individuals fraudulently claiming to be eligible participants (scammers); and by people completing the survey with malicious intent to sabotage the project or system (bad actors) are becoming widespread and can lead to economic loss, loss of trust, and identity theft [3,4]. In research [2,5-7], these practices raise data integrity concerns, impact the ability to draw accurate conclusions or take appropriate actions based on survey findings, and strain scarce research resources, especially when an incentive (eg, a voucher) is offered for completion. While most survey platforms provide some bot or scam prevention methods [8], these alone may be insufficient [5], and researchers using online methods need practical recommendations to ensure the rigor of their data [9]. The purpose of this paper is to describe case studies and experiences of potential bot or scam completions of online surveys (for either recruitment or data collection purposes) within the health behavior field, specifically physical activity and nutrition; and to provide a checklist of practical strategies for researchers to help reduce and identify bot or scam completions before, during, and after data collection, informed by these case studies and the wider literature [2,5,6,10]. The checklist may also support submissions for journals, grant agencies, and ethics committees so that publications and research funders can take precautions to avoid bot or scam completions.

Ethical Considerations

Ethics approval for the included case studies was obtained from the following institutes: case study 1, ethics approval granted from Deakin University Human Research Ethics–Health (HEAG-H 103_2024); case study 2, ethics approval granted from Deakin University Human Research Ethics (2024/HE000064); case study 3, ethics approval granted from Deakin University Human Research Ethics (2024-209); and case study 4, ethics approval granted from Deakin University Human Research Ethics (2022-311) and Brazil (CAE:63025822.8.1001.5346). Participants were provided with a digital plain language statement outlining the study requirements, privacy and confidentiality of their data and any compensation provided (specific to the case study). Informed consent (digital) was then obtained from participants.

Recent Case Studies

We present 4 recent case studies of surveys conducted in the health (specifically physical activity and nutrition) research field in Australia where bot or scam completions were detected (Textboxes 1-4). Three case studies offered a supermarket voucher to each participant, and 1 case study entered participants into an e-gift card draw. The estimated percentage of nonlegitimate participants and differences in results between the included and excluded participants (where available) are presented for each case study in Textboxes 1-4.

Considerations and Checks to Reduce the Incidence of and Identify Bot or Scam Completions

Based on the learnings from these case studies and the wider literature [2,5,6,10], the following considerations and checks have been compiled to assist researchers when designing and conducting their research to help reduce the incidence of, and identify bot or scam completions. These have been developed into a checklist shown in Multimedia Appendix 1.

Some considerations or checks may be sufficient on their own to suggest a nonlegitimate completion (eg, when the honeypot question is completed), whereas a combination of different checks may be needed to identify a potential nonlegitimate completion (eg, a run of similar start times, plus a questionable email address, plus similar responses to open-ended questions) to avoid the exclusion of genuine respondents. The following considerations and checklist can be used by researchers to develop a study-specific list of considerations and checks, including hierarchies and combinations of criteria, which can then be applied to each phase of their study to help identify potentially nonlegitimate completions.

Before Data Collection: Creating Study Collateral

Information can be included in the plain language statements, consent forms, and advertisements to help deter bots or scams and indicate how potentially fraudulent data will be dealt with. For example, researchers can reduce the prominence of advertising vouchers or payment as compensation, specify the type of voucher and where it can be redeemed (eg, for a country-specific retailer), state that completions are limited to 1 per household, notify participants that they may be asked to confirm their identity via their listed contact method (eg, email or phone number), and indicate that suspected fraudulent completions will not receive the voucher.

Survey Design and Development 1: Built-In Survey Platform Functions

When creating the survey, enable the built-in bot or scam detection functions of the survey platform. These tools may automatically navigate respondents out of the survey if deemed to be fraudulent (bot or scam) or add a variable to the collected data that can be later reviewed and interpreted as potentially problematic.

In Qualtrics [8], Bot detection provides a Q_RecaptchaScore (a score of <0.5 is likely to be a bot); and Prevent multiple submissions adds a Q_BallotBoxStuffing value (a score of 1 is likely to be a duplicate) for each completion. Enabling the Relevant ID functions (Duplicate, Duplicate Score, Fraud Score, and Last Start Date) helps detect multiple responses by looking at respondents’ metadata. The Duplicate field will show “True” with the corresponding date and time the last attempt was started in the Last Start Date field. Duplicate Scores of ≥75 indicate duplicate responses, while Fraud Scores of ≥30 indicate the response is both fraudulent and likely a bot. The reCAPTCHA feature, where the participant needs to correctly click the squares containing an item (eg, motorcycles), can also be added to a survey.

In REDCap (Research Electronic Data Capture), reCAPTCHA is also available [11]. Similarly, a similar IP address or geolocation tracking is available through 3 options: set-up of consent forms as e-consent forms, where the IP address is automatically recorded; use of geolocation action tags in survey fields, to identify the longitude and latitude of the survey respondent; and external modules that identify IP address [12]. However, each option has limitations: reCAPTCHA may be passed by sophisticated bots; IP addresses may be falsified with virtual private network services or proxy servers; use of geolocation tags may be blocked by respondents’ browsers; and use of external modules is determined by individual REDCap server managers [13].

Survey Design and Development 2: Online Survey Question and Structure

Several precautions can be added to surveys during the design phase to exclude or identify bot or scam completions. These may include the following measures:

• Including a reCAPTCHA variable
• Applying branching logic to exclude ineligible responses from progressing, ensuring that a participant must correctly answer the eligibility question to continue
• Using open-ended questions which provides an additional check as they must type the correct response
• Adopting a 2-step process separating screening and consent form completion from the study survey (eg, eligible participants are sent a unique link to the full survey via their email address, only after responses to the screening questions have been assessed by both the survey platform functions and the researcher)
• Adding automated date and time survey fields, which can be coded to be hidden from respondents (eg, action tags in REDCap can record the respondent’s current date and time based on the host REDCap server, and current date and time based on Coordinated Universal Time); discrepancies in timestamps can indicate responses from outside the target region or time zone.
• Repeating questions where the response would not typically change to enable cross-checking (eg, name, date of birth, email, and postcode, in separate consent and survey sections)
• Adding dummy “honeypot” questions (eg, “Please select your favorite option: 1, 2, or 3”), coded to be hidden to humans and only visible to bots. Completion of the question indicates the respondent is nonlegitimate
• Including attention check questions to progress (eg, “Select a color from this list to continue,” response options: cat, blue, broccoli)
• Requesting additional contact information to verify participant eligibility

Following Data Collection: Identifying Indicators of Potential Bot or Scam Completions

Researchers should promptly and continuously review their data to identify potential bot or scam completions. The following are factors based on embedded data (eg, IP address), participant responses, and response patterns, which can help identify potentially fraudulent responses:

• Completed responses to the honeypot question (ie, competed by a bot).
• Incorrect geolocation for the study sample (eg, latitude or longitude out of range).
• Built-in bot detection function or variables (eg, Qualtrics Q_RecaptchaScore<0.5, Duplicate Score of ≥75, Fraud Score of ≥30).
• Duplicate IP addresses: however, this may not be a reliable indicator where participants could be at the same location (eg, workplace, school, public library, or internet café) or where multiple responses are permitted from the same address or family or household.
• Similar IP addresses (ie, all but the final sequence of numbers are identical) may indicate repeated submissions from similar sources.
• Several failed attempts at passing a reCAPTCHA test, followed by a successful completion from the same IP address; however, this requires the researcher to examine and compare incomplete and complete surveys.
• Unrealistic survey completion time (eg, less than 10 min when the median time was 33 min).
• Multiple consecutive responses with similar survey start and end time, or in a run of start and end times (eg, a survey with a median completion length of 30 min starts at 5:00 PM and takes 7 min to complete; another survey starts at 5:07 PM, and so on).
• Multiple participants with the same personal details (eg, first name and surname, email address, phone or mobile number, home address).
• Questionable email address or names, often with a similar pattern and completed at a similar time (eg, a participant called Jane Doe provides an email address of john.smith123@...; a run of similar name formats, such as, including a middle initial or extra spaces between names; email addresses contain a random mix of letters and numbers with similar pattern between participants, such as bgckts541236@... or 74sqwpc.hx54612@...).
• Unusual responses for study context (eg, in studies recruiting parents or perinatal women, this may include an unusually high number of children, particularly over a short or unrealistic time frame, or unrealistic age of childbirth, such as more than 9 children from live births, child birth dates only 6 months apart, child birth at parent age less than 12 years).
• Mismatch between data provided at 2 locations of the participation process (eg, mismatch between date of birth reported in the consent form and survey or state and postcode provided do not match).
• Identical or very similar responses to an open-ended question. This should not be applied to likely common responses (eg, “none,” “no benefits,” “schoolwork,” etc).
• Questionable date format used for date-based responses (eg, including full stops when backslashes are requested or writing the month instead of the number format).
• Providing a landline phone number when a mobile number is requested, or providing landline phone numbers that do not match the area or city they are claiming to live in.
• Providing a full name when only a first name was requested.
• Nonresponse to requested contact information, which would allow researchers to check and confirm details.
• Completed at unusual or unlikely times, such as between 12 AM and 5 AM. Use participant-specific context as needed (eg, consider postnatal women with disrupted sleep patterns, or night shift workers).

Analysis and Write-Up: Reporting of Potential Bot or Scam Completions

Researchers should include a description of any suspected bot or scam completions in reporting their findings using the checklist in Multimedia Appendix 1. This should include how they were identified, the number identified, and reporting confirmation that they were excluded. This provides greater confidence that the participants included and the subsequent findings of the study are legitimate.

Additional Considerations

It is important to note that with increasingly sophisticated bots or scams, not all potential bot or scam completions may be identified, and some legitimate completions may be incorrectly excluded (eg, case study 3). These preventive measures may also inadvertently contribute to the exclusion of some groups, for example, those with low literacy or digital skills. As the technology landscape is rapidly changing, so too are prevention and identification techniques. The use and publication of the checklist in Multimedia Appendix 1 of this paper can lead to more transparency around the experiences of different studies. This may allow for future exploration of variation in fraudulent rates, and possible reasons for these (eg, platform, incentive, or recruitment method). Purpose-designed empirical studies are needed to determine the most effective platform-, methodology-, and population group–specific strategies for preventing and identifying bot or scam completions.

Conclusions

Researchers using online recruitment and data collection methods should consider the checklist of strategies provided to reduce and identify potential bot or scam completions at each stage of their project including in the creation of study collateral, survey design and development, data review, and write-up. Continued exploration and sharing of strategies are needed as technology and bots or scams evolve. Details of the steps planned or taken should be included in grant and ethics applications. While these steps may be time- and resource-consuming, they may help prevent and identify potentially nonlegitimate survey responses and provide greater confidence in the research findings.