Choosing a mode of data collection is one of the most important decisions that researchers make when designing a research survey. The choice of mode may be shaped by feasibility, cost, response rates, and potential for achieving a more representative sample. Currently, two of the most frequently used administration modes are telephone and Web [1,2]; the former is often interviewer-administered and the latter is usually self-administered. Often, telephone questionnaires have higher response rates and can also allow interviewers to clarify questions and motivate participants. However, telephone questionnaires can be relatively resource intensive and may have difficulty using visual aids [3,4]. By contrast, Web questionnaires are often less expensive and easier to implement but may be affected by lower response rates, item nonresponse, greater potential for fraud, and Internet accessibility issues that affect representativeness [3,5-7]. Given the compromises between modes, another option is a mixed-mode survey, which is administered using two or more data collection modes to allow participants an element of convenience and choice and to also compensate for the limitations of a single mode [1,2,8-10]. In each case, the choice of mode, or modes, may influence respondents’ answers, and accordingly, the conclusions that researchers can draw about a study population.

One recurring concern in the literature is the relationship between survey mode and the accuracy of the information provided by the respondent because survey mode may influence respondents’ willingness to answer certain questions at all, truthfully, or with socially desirable but inaccurate answers. Social desirability bias may be heightened in interviewer-administered telephone and face-to-face surveys but reduced in self-administered Web surveys [11,12]. Furthermore, although Web surveys typically have lower response rates than other modes of survey research, they can have lower item nonresponse rates as well, possibly because of the absence of interviewers whose characteristics can influence the kinds of attitudes and behaviors that people report [12-14]. In the context of mixed-mode surveys, these challenges can lead to inaccurate conclusions if researchers aggregate data that were collected through different modes [10], or if they compare results across multiple surveys that were collected by different modes [12]. In addition, if respondents are given the choice of mode, respondents may select a mode where the provision of socially desirable answers is more prevalent, or choose to not respond to certain questions, which may also prevent researchers from making accurate comparisons across subgroups [15,16].

The emergence of Web surveys has provided researchers with technology that creates new opportunities and challenges for addressing response bias. Accordingly, there has been much recent research on Web surveys and how mode effects may shape responses to survey questions [5,9,10,17-23]. Although most research shows that social desirability bias is lower in Web surveys compared with other modes [4,12,24], other work has shown no difference [25,26]. This suggests that Web surveys may reduce social desirability bias only under certain circumstances and potentially only with certain types of questions. Alternatively, as Internet use is increasingly ubiquitous in society, the literature may be starting to reflect the fact that Web surveys may no longer evoke differential responses from survey participants.

In particular, how willing respondents are to report personal, private, or sensitive matters in a survey setting can affect data quality [9,17,18,22,27,28]. In a recent meta-analysis of 10 experimental studies that looked at Web administration versus interviewer-administered modes for collecting potentially embarrassing information, the authors concluded that self-administration via the Web improved reporting accuracy for socially undesirable responses [18]. Similarly, recent findings from the British National Survey of Sexual Attitudes and Lifestyles compared computer-assisted personal interviews with self-interview modes and found that reporting of sensitive information was overall higher in the Web/self-administered survey mode [21]. This could make Web surveys the preferred mode for asking highly sensitive personal questions.

This body of research is encouraging for research fields that investigate potentially sensitive topics. HIV and AIDS–related research often addresses particularly sensitive issues, and avoiding sensitive questions is not possible in most HIV- or AIDS-related surveys. Given the sensitive content, it is likely that studies of HIV and AIDS face particularly strong challenges arising from reporting biases, including social desirability and item nonresponse biases. To our knowledge, no national population-based experimental studies have looked at response differences between telephone and Web questionnaires for extremely sensitive health content, such as HIV and AIDS. Therefore, in this area of health research, observational studies that do not randomize participants to a completion mode can provide a helpful foundation for which to further explore the issue of response bias by mode. Indeed, it may be advantageous for health researchers considering mixed-mode survey designs to learn whether their specific topic is prone to response bias by mode, even when that data are nonexperimental in nature, such as 2 recent observational studies that explored response biases for specific health issues [26,27]. Understanding the extent to which responses may differ and how they differ depending on mode is essential given the use of mixed-mode designs and for the comparability of studies on sensitive health topics that use different data collection modes.

A national HIV and AIDS survey was conducted among Canadians in 2011 to determine public knowledge, attitudes, and behaviors about HIV and AIDS. The survey used a mixed-mode data collection method, whereby participants selected whether they would prefer to complete the questionnaire by telephone or the Internet, which allows us to compare responses between (persons who chose) telephone and Web completion modes. We seek to determine whether the mode of questionnaire completion influences responses in terms of social desirability and missing data. To explore whether any observed associations vary by question type, we select different types of questions including potentially sensitive, stigma-related, and less-sensitive questions.

We sought to determine whether there are differences in social desirability and missing data between people who chose to complete an HIV- and AIDS-related questionnaire by telephone versus the Web. We anticipated that due to the sensitive and personal nature of some of the questions, we would see differences in responses between the telephone interviewer-administered questionnaire and the Web-based self-completed questionnaire. Although we saw large sociodemographic differences between completion modes, for most of the questions (23 of 28) we studied, there were no significant differences in responses by mode. Overall, 2 of 13 questions assessed for social desirability, and 3 of 15 questions assessed for item nonresponse were significantly associated with choice of mode in the multivariate analysis. However, despite finding few significant response differences, those we found were large in magnitude; ranging from 1.6- to 8-fold difference.

Participants were given a choice to complete the questionnaire through the Internet or by telephone. As expected, and similar to other mixed-mode studies that used a non-random method for selecting participants [17,27,38], we see large differences in sociodemographic characteristics between the Web and telephone groups. The telephone group was older and had a larger proportion of women. Similar to other Canadian studies, we also found that Internet respondents reported higher incomes and more education [27,38]. Other studies have found younger age, higher incomes, greater education, and race to be strongly associated with the Internet and email access [6,39-41]. Although we do not know whether that telephone respondents in our study have access to the Internet, our results are consistent with what these findings suggest about Internet access. The exception is that we found no difference in the proportion of visible minorities between the two modes; this finding may be unique to the Canadian population as compared with the American population, owing to more targeted social policies in Canada to expand Internet access and reduce the digital divide for disadvantaged and racialized groups [42].

We thought that sensitive and stigma-related questions might be differentially affected by social desirability between completion modes, but a pattern in responses by question type was not apparent. We saw some evidence of social desirability for sensitive questions, with telephone respondents reporting lower numbers to questions about sexual partners they had in the past year and in answering whether any of these sexual partners were casual partners. This is in line with previous research that has found that Web questionnaires are better at eliciting truthful responses to sensitive questions than telephone questionnaires [9,20,22,27,28]. However, only the question about number of sexual partners showed a statistically significant difference, a difference that remained after multivariate adjustment. This suggests that the social interaction with the interviewer may have resulted in respondents stating a lower number of sexual partners—a more socially appropriate response.

Four other questions that we tested for social desirability showed significant differences in responses by mode, but not in the anticipated direction. If social desirability was at play, telephone respondents would be expected to provide more tolerant responses about their attitudes toward people living with HIV and AIDS because these attitudes are more socially appropriate. However, we found that telephone respondents gave less tolerant responses than Web respondents when asked if they felt afraid of people living with HIV and AIDS and when rating their comfort level with shopping at a grocery store owned by someone who is HIV seropositive. Yet, these differences did not remain in the multivariate analysis and are therefore attributed to sample differences between mode groups. Similarly, telephone respondents reported less knowledge about HIV and AIDS and less charitable giving than Web respondents, which are considered socially undesirable responses, yet only, the difference in charitable giving persisted in the multivariate analysis. This result suggests that when reporting their charitable donations to a live interviewer, telephone respondents were not affected by social desirability and did not feel socially obligated to appear more charitable. Although we adjusted for income and education in our models, Web respondents may be different from telephone respondents in other important characteristics that could account for the counterintuitive result to this question, such as religious affiliation, awareness of need, altruism, and personal values, all of which are known motivators for charitable giving [43]. Another possible explanation is that some telephone respondents may fear that the live interviewer will ask them to donate to a charity at the end of the questionnaire if they appear charitable and therefore choose to answer “no” to avoid the perceived solicitation.

With respect to the missing data objective, we also saw no clear pattern of response differences between question types. Among the questions we considered, there was generally a greater frequency of item nonresponse to sensitive and stigmatizing questions and among the Web sample. One question from each category was found to have significant differences in item nonresponse between the telephone and Web groups, with a greater frequency of missing responses in the Web questionnaire. These differences also remained in the multivariate analysis, which suggests that they are attributable to the mode and not to sociodemographic differences between the samples. The greatest amount of item nonresponse was for annual household income (15.2%), with more nonresponse among telephone participants (17.7%) than Web participants (14.6%), although this difference was not statistically significant. Questions about income are well known to generate a large frequency of missing data in most surveys.

Our findings for item nonresponse are consistent with those of other experimental and observational studies that looked at missing data by mode and also found that Web questionnaires produced more missing data [9,27,44,45]. It is thought that telephone interviewers may inadvertently persuade respondents to answer, or that they may further explain or clarify questions. Some of these studies found that differences were particularly evident for complicated or difficult questions [9], although neither question that we saw differences for is considered difficult. Due to the nature of how the data were entered and coded, we are unable to distinguish between different types of missing data (eg, “do not know” responses vs not answered questions), and therefore, we cannot tell where “do not know” responses may be genuine answers. The concerning illness or disease question was an open-ended question, and this likely contributed to item nonresponse among the Web sample as typing was required, and no interviewer was present to prompt for a response. This is consistent with other studies that have also found increased missing data for open-ended questions on Web surveys [22,27]. In contrast, the HIV testing question provided response categories, but some respondents may not have known if they had ever been tested for HIV and legitimately selected the “do not know” response, which would have contributed to item nonresponse for this question.

Our study benefits from several strengths. We conducted a large national survey, and so, we anticipate that our results have fair generalizability to the Canadian population. Our sample was recruited from a respondent panel that was constructed using random-digit-dial, and this strengthens our ability to make causal inferences [46]. We also performed a multivariate regression analysis to control for sample differences between the telephone and Web groups and included a propensity score that can help adjust for mode selection effects [10]. After accounting for sociodemographic disparities between the groups, it is more likely that any remaining differences in responses can be attributed to the mode of completion.

The results of our observational study should be considered along with its limitations. First, and most importantly, we did not randomize respondents to mode, and therefore, our estimates may be subject to selection bias. We attempted to control for the risk of selection bias by including a propensity score in our regression models. The inclusion of the propensity score neither changed the significance of our results from the sociodemographic-adjusted models nor did it greatly change their magnitude. This indicates that relatively little selection bias is present. Furthermore, the fact that our results persist in the face of the propensity score control would suggest that our results do describe the effect of survey mode on response patterns. However, there may be other unmeasured characteristics that we are unable to control for, such as differences in people’s experiences and opinions between the groups, and if these characteristics also affect mode selection, this could be confounding our results. Although, if these characteristics are unrelated to mode selection, they would only produce nondifferential misclassification, which would indicate our estimates are conservative. Furthermore, the regression model that generated the propensity score was limited to variables contained within our questionnaire, and as such, there may be variables that help explain why one chose a given survey mode that are missing from the model specification. The propensity score is therefore not a perfect predictor of mode selection, and there is some degree of misspecification as the probability of choosing a mode and actually choosing it are distinct constructs. In addition, because the sample size of telephone respondents was much smaller than the sample of Web respondents, the propensity score may not be able to adequately balance sociodemographic variables across the response groups. Second, the participation rate from the random-digit-dial panel was low, and more people who opted to complete by telephone finished the questionnaire (31.1%) than those who opted to complete through the Internet (18.4%). Although this participation rate limits the external validity of our results, particularly for the Web sample, it is nevertheless consistent with response rates from similar surveys in Canada that report response rates between 14% and 28% [31,47]. Finally, beyond social desirability and missing data, we did not consider other potential ways that responses could differ between modes, such as nondifferentiation bias (eg, answering the same response across a series of items), acquiescence bias (eg, agreeing across questions), and avidity bias (eg, disproportionate representation of those interested or invested in the survey topic of HIV and AIDS), due to study scope and design of the questionnaire.

In summary, there has been considerable concern in the survey methodology literature about the use of Web questionnaires to conduct research. Primary concerns include low response rates, low representativeness, sampling issues, and the comparability of Web-collected data to data collected by other modes [6,44,46,48]. Although these are important concerns that are not fully addressed by our study, we have shown that in a nonrandomized mixed-mode survey for a particularly sensitive health topic, the differences between telephone- and Web-collected responses were minimal in terms of social desirability bias and item nonresponse. This is promising given that Web surveys are significantly cheaper and faster to implement. On the basis of our analysis in this nonrandomized study, we cautiously suggest a comparative strength of Web surveys is that they may provide more truthful results, particularly for questions about sexual behaviors. Although we did see more item nonresponse among the Web group overall, it was mostly nonsignificant and may be addressed through improved Web questionnaire design, such as using prompts that alert respondents if they attempt to move forward without completing a question [2,22].

Our results, although limited by the observational study design, may be important to consider in light of the increasing use of mixed-mode surveys that combine telephone and Web completion modes to reduce costs and increase validity and may help inform future experimental survey methods studies and population-based research in the area of HIV and AIDS.