Paper-Based Versus Web-Based Versions of Self-Administered Questionnaires, Including Food-Frequency Questionnaires: Prospective Cohort Study

Introduction

Background

A key aspect in epidemiology is the adequate classification of exposure [1]. For this reason, the valid estimation of usual dietary intake in nutritional epidemiology studies is a topic of emerging interest, posing a complex and challenging task [2,3]. In general, the selection of the most appropriate instrument for assessing usual food intake in large-scale epidemiological studies depends on the research purpose. In this context, several dietary assessment methods are available, all of which have their own advantages and limitations [4]. There are 2 methods that prevail: the food-frequency questionnaire (FFQ), most frequently used in studies assessing the association between diet and health-related outcomes, and the 24-hour dietary recalls, primarily used in nutrition surveillance research [2]. Traditional approaches for gathering information from study participants include face-to-face or telephone interviews administered by trained dieticians, as well as paper or printed questionnaires, usually self-reported data [5,6]. These methods require a great deal of resources in terms of personnel, logistics, and materials [5,7]. However, in the past years, advances in technology and the wide use of the internet have allowed researchers to collect data quickly, with minimal costs from large sample groups, through the use of Web-based self-administered questionnaires [8-10]. It has been suggested that a Web-based FFQ can increase the response rate, which may result in greater validity of the data collected, compared with paper-based response rates [10]. However, most of the major limitations of conventional FFQs are similar to Web-based FFQs; therefore, the measurement errors in both approaches may remain essentially equivalent [11]. In fact, the cognitively complex completion process, inherent to the FFQ method (eg, averaging the usual frequency and portion size for each food group or specific food item), seems to be similar for paper-based and interactive computer-based or Web-based formats. The effectiveness of these types of data collection has previously been tested in different fields of health, several populations, and a variety of settings [12-18]. On the other hand, the integration of new technologies, including computer software and Web-based apps, has created novel and unique opportunities to conduct research focused on data collection in nutritional epidemiology [19]. In fact, there are several Web-based cohort studies in which the participants enroll via internet and all data are collected by Web-based questionnaires [5,7,17,20-23]. However, until now, there has been a scarcity of evidence from large-scale epidemiological studies and nutrition surveys that evaluate the comparison between traditional and new technologies to measure dietary intake. In the past 5 years, research in this area has attracted a growing interest [6,24]. The Seguimiento Universidad de Navarra (SUN) project offers an excellent opportunity to study this purpose, given that it is a large and well-known European cohort that uses both paper-based and Web-based FFQs, at baseline and/or during the follow-up, among participants with a high education level.

Objectives

Therefore, we aimed to compare results from the general baseline questionnaire (Q_0) and the 10-year follow-up questionnaire (Q_10) of the SUN prospective cohort obtained in different subjects, some of whom used a paper-based version, and others used a Web-based version.

Methods

Study Population

The SUN project is a multipurpose and prospective Spanish cohort of university graduates, designed to study the impact of several sociodemographic, nutrition, and lifestyle characteristics on the prevention of noncommunicable diseases. Open enrollment began in 1999. The design and methods used in the SUN project have been formerly described in detail elsewhere [25]. The study protocol was supported by the Institutional Review Board of the University of Navarra. We considered a response to the initial questionnaire as informed consent to participate. Participation in this cohort is only by invitation or registration. Currently, all questionnaires of the SUN cohort can be filled by paper or Web-based questionnaires. Web-based questionnaires were available since 2004, using a password-protected area of the SUN website. Starting at the beginning of 2004, participants were offered the possibility of answering their questionnaires either on the paper-based version or on the Web-based version. Thus, in each baseline or follow-up questionnaire since 2004, they can choose how to complete this questionnaire. For example, if a participant received the paper-based questionnaire, he or she can request his or her password to complete the Web-based version of the questionnaire on the SUN website. The participants who have access to the questionnaires do not necessarily have to complete all the questions. Diet in this cohort is evaluated through a repeatedly validated baseline semiquantitative FFQ [26,27]. At this time, the full-length FFQ has been administered only twice throughout the study, at baseline (FFQ_0) and after 10 years of follow-up (FFQ_10). Only those participants who have completed the Q_10 after 10 years on the internet can also fill the FFQ_10, although its filling is absolutely voluntary. For the present analysis, we used the latest available database from March 1, 2017, which included those participants who had responded to the Q_0. Of the 22,564 participants who completed the Q_0, 22,225 (98.5%) of them reported their data using the paper-based version; meanwhile, only 339 (1.5%) participants completed the Web-based version (see Figure 1). After a 10-year follow-up period, we collected questionnaires from 4379 (39.3%) participants by using the paper version and from 6765 (60.7%) participants by using the Web-based questionnaire. Among these 6765 participants who answered the 10-year questionnaire using the Web-based version, 5882 participants also completed a full-length FFQ via the internet (52.8% of the completed Q_10).

‎View this figure

Exposure Assessment

In the SUN cohort, the Q_0, and follow-up questionnaires are self-administered. The Q_0 includes information about sociodemographic variables (eg, sex, age, marital status, and employment status), lifestyle-related variables (eg, smoking status, physical activity, and special diets), anthropometric variables (weight, height), and clinical variables (medical history, family health history, blood pressure levels, medication use, and not gaining more than 5 kg of weight in previous years). On the other hand, the Q_10 collected information on a wide array of characteristics, including weight, height, marital status, and diagnosis of several diseases. Both of these general questionnaires can be filled out in paper form or on the Web, and they are exactly equivalent. The paper-based FFQ comprises 136 food and beverage items, categorized into 9 frequency categories of consumption. These items capture the usual consumption of listed foods during the previous year. There are 9 options for the average frequency of consumption (never or almost never: at least 6 times per day). The Web-based FFQ is a Web-based questionnaire, with a format very similar to that of the paper-based version including the same items.

Measures of Data Quality

Nutrient intakes were calculated as the frequency multiplied by the nutrient composition of specified portion sizes for each food item by using an ad hoc computer program that was specifically developed for this aim. A trained dietitian updated the nutrient database using the latest available information from food composition tables for Spain [28,29]. We considered total energy intakes lower than 800 or 500 kcal/d for men and women, respectively, and greater than 4000 or 3200 kcal/d for men and women, as proposed by Willett as implausible energy intake [30]. In addition, we calculated the outliers of energy intake, defined as either percentiles 1 and 99 or 5 and 95. For each food group, the consumption was considered implausible if it fell outside the 25th percentile minus 3 times this interquartile range or the 75th percentile plus 3 times this interquartile range. For the sake of comparing the proportion of missing responses to the paper-based versus the Web-based questionnaires, we calculated the mean number of missing values in 6 sections of the baseline assessment of the cohort: (1) FFQ (136 items), (2) healthy eating attitudes questions (10 items), (3) alcohol consumption questions (3 items), (4) physical activity during leisure time questions (17 items), (5) other activities questions (24 items), and (6) personality traits questions (3 items). Finally, we evaluated differences in the following potential internal inconsistencies in reporting between paper-based and Web-based questionnaires: (1) eating fruit for dessert but reporting a total fruit consumption equal to 0, (2) eating fried foods at home but reporting olive oil consumption or other vegetable oil consumption equal to 0, (3) drinking alcohol sometimes but reporting total alcohol consumption equal to 0, and (4) drinking wine sometimes at lunch or dinner but reporting a total alcohol consumption equal to 0.

Assessment of Other Variables

Physical activity information is obtained at baseline through a questionnaire validated in Spain [31], which collects information on 17 sports participated in, in the past year (with 10 answer options from never to more than 11 hours a week). The physical activity level during leisure time was quantified by assigning metabolic equivalents (METs) to each activity. Total METs-hours/week for each participant was calculated as the sum of the number of hours spent in each activity multiplied by the specific METs of that activity. Healthy eating attitudes are evaluated through 10 questions, asking the participants if they tried to eat more fruit, more vegetables, more fish, less meat, less sweets and pastries, more fiber and less fat, and if they tried to avoid the consumption of butter, removed fat from meat, and did not add sugar to drinks. We developed a score to capture the gathered information from these 10 questions, which was used in previous publications of the SUN cohort [32,33].

Data Analysis

The aim of our study was not to compare repeated measurements of the same variables within the same subjects using 2 different methods; the aim of this study was to compare the reliability and comprehensiveness of the information gathered with each of both methods (paper-based or Web-based) from different subjects. Descriptive statistics were used to describe the differences at baseline and follow-up between participants who completed the questionnaire in paper-based version and among those who filled out the Web-based version. We used means and SDs for continuous variables or percentages for categorical variables. Multiple linear regression models were used to assess the association between the type of questionnaire (paper or Web-based version) and the differences in the mean number of missing values at baseline. Logistic regression models were run to assess the relationship between the type of questionnaire at baseline and the risk of having implausible data of food items and mismatches or inconsistencies in dietary variables. Both analyses were adjusted for sex, age, level of education (bachelor, graduate, postgraduate, and doctorate), and year entering the cohort (1999-2000, 2001, 2002-2003, 2004, 2005-2007, and 2008-2017). The Q_0 paper-based version was always considered as the reference category. Analyses were performed with STATA version 12 (STATA Corp). All P values are 2 tailed, and statistical significance was set at the conventional cut-off of P<.05.

Results

Baseline Characteristics

We have compared several potential measures of data quality between Web-based and paper-based data collection at baseline and at 10-year follow-up. Overall, the sociodemographic, lifestyle, and health characteristics were well balanced between the 2 approaches for administering the questionnaires, particularly Q_10, which had a higher number of participants and similar year of completion (Table 1).

Subjects who fulfilled the paper-based Q_0 were more likely to be older, married, workers, current smokers and ex-smokers, and they were more likely to have a lower level of university education. Moreover, they were less active and showed a higher prevalence of hypertension, cancer, diabetes, and weight gain in the past 5 years. On the other hand, participants who completed the Web-based Q_10 were more likely to be men, younger, physically active during leisure time, not married, workers, never smokers, had a higher adherence to the Mediterranean diet, and generally had less prevalence of chronic disease related to diet. Beneficial changes in the consumption of most food and macronutrients and a positive response to dietary attitudes were observed after 10 years of follow-up, mainly when comparing the paper FFQ_0 and FFQ_10 (Table 2).

Principal Results

We have found similar results to those previously published [32,33]. In Table 3, we provide the adjusted differences in the mean number of missing values in several items of the general questionnaire at baseline.

Generally, the adjusted differences in the 6 sections evaluated at Q_0 were low. In all cases, they were lower than 3.5%, even for questions on healthy eating attitudes, alcohol consumption, and personality traits; they were very low, lower than 0.5%. However, when we compared participants who completed Q_0 at baseline using the Web-based versus paper-based version, we found some contradictory data. Thus, the mean amount of missing data in the FFQ_0 and physical activity during leisure time questions was significantly higher among subjects who had completed the Web-based version (ß=3.3, 95% CI 2.03-4.64 and ß=2.01, 95% CI 1.45-2.57, respectively) but significantly lower in the other activity questions (ß=–2.01, 95% CI –3.04 to –1.35). Overall, the percentage of participants with implausible data on food consumption in the FFQ was always lower than 3% in participants who used the paper-based version at baseline or at 10-year follow-up (Table 4).

However, we found the following exception: the implausible reporting of nuts consumption, accounting for more than 16% of participants. On the other hand, when we considered all participants, generally, no significant differences among food group consumption were found, except for the implausible reporting of nuts consumption, more probable in these same subjects—odds ratio (OR) 1.54 (95% CI 1.20-1.99). When we classified participants according to the type of questionnaire chosen at 10-year follow-up, the implausible reporting of legume and meat consumption was less frequent in those who collected the Web-based version. The number of participants with implausible olive oil consumption was 1 among those who used the Web-based version and 0 among those who used the paper-based version. For this reason, we could not present the OR for Web-based versus paper-based version. Finally, overall, in both the paper-based and Web-based FFQ_0, the items with higher frequency of mismatches and inconsistencies were total energy intake (near 10%) and self-reported alcohol consumption (approximately 2%; Table 5).

On the contrary, mismatches and inconsistencies were lower in reported consumption of fruit dessert versus total fruit consumption and fried food versus oil consumption. In addition, subjects who filled the Q_0 via internet exhibited a significantly higher risk of presenting inconsistencies in reporting alcohol consumption and wine consumption versus total alcohol intake, compared with those who chose the paper-based version: adjusted OR 3.58 (95% CI 2.01-6.22) and OR 3.01 (95% CI 1.21-7.48), respectively, although the mismatches in relation to total energy intake outside of the reference subset had a lower risk: OR 0.31 (95% CI 0.17-0.56).

Discussion

Principal Findings

To the best of our knowledge, our research is the first to compare the quality of the answers, at baseline and after a 10-year follow-up period, in different subjects in a large prospective cohort, where some of them used the traditional version (paper-based) and others used a new method (Web-based) for data collection in a large prospective cohort. Our most important objective was to compare the respective ability of each method (Web-based or paper-based) to gather the relevant information in a reliable and comprehensive manner. The overall response rates in Q_0 were higher for the paper-based version than the Web-based version, 98.5% and 1.5%, respectively, as the electronic version of the questionnaire has only been available since 2004. In addition, the paper-based version was always offered as the first choice. However, in Q_10, the proportion of participants choosing the Web-based version was 61%, because of the fact that if participants are given a choice, they prefer the electronic version. A previous publication suggested that it could be useful to offer all subjects both a paper-based and a Web-based version of a long-term instrument, such as the FFQ, to avoid selection bias [34]. We found that in both methods, baseline characteristics, including food consumption and energy and nutrient intake, were comparable across a range of parameters, with a few exceptions. Although our present assessment does not represent a proper validation study, these findings are in agreement with previous studies that have shown few differences in health questionnaire scores or measures among different methods of administration [13,14]. For example, the NutriNet-Santé study published that the Web-based sociodemographic and economic questionnaire provided information of similar-to-superior quality compared with the traditional paper-based version [17]. Overall, the percentage of missing values in FFQ_0, with 136 items, was higher than 9%, whereas the missing information on physical activity during leisure time and activity questions, with 17 items, was higher than 17%, despite these questionnaires being shorter and simpler than the FFQ. On the other hand, although several studies in different areas have suggested that the validity and reliability of data obtained on the Web are comparable with those obtained by classical methods [35], our findings reveal that the quality of the data has not worsened with the incorporation of the Web-based version in the SUN cohort of highly educated adults. In fact, for some measures of data quality, the results have been more favorable among subjects who filled in questionnaires using the Web-based version, but for other measures, the data quality was worse. An exception is in relation to the implausible values of nut consumption, greater than 24% in FFQ_10. This is probably because of the fact that the majority of these participants reported their FFQ_10 after 2013, when the evidence on the health effects of this food group in the cardiovascular prevention of Primary Prevention of Cardiovascular Disease with a Mediterranean Diet trial had been published [36]. Moreover, the prominence of nuts in the latest published dietary guidelines has also contributed to the remarkable increase in their consumption. In addition, it is possible that SUN participants with longer follow-up were more aware of their diet because of general study participation and the administration of dietary recall methods, which resulted in improvements in their dietary habits (Hawthorne effect).

Strengths and Limitations

The strengths of the SUN study are its high retention rate (91%) [25], a relatively long follow-up, its prospective design, the large sample, and its coordination in a single center, as well as the fact that many of the measurements have been validated. In addition, this study is a comparison and not a validation. Thus, we compared within the same study the self-reported Q_0 and Q_10 and the same FFQ_0 and the FFQ_10. Both FFQs had an identical format and did not incorporate photographs of serving sizes. For this reason, all answers to Q_O and Q_10 are most definitely comparable among themselves. We acknowledge that this study has several limitations. First, this study is not a validation of the same method of dietary assessment; rather, it has two different versions of assessment: paper-based or Web-based. In addition, the aim of this publication was not to compare repeated measurements of the same variables within the same subjects using 2 different methods. Consequently, our findings should be interpreted with caution, and they should never be analyzed as results of a validation study. Second, our sample of Web-based responders of FFQ_10 was small, with respect to the responders of the general Q_0. Thus, among participants with Q_10 (n=11,144), 6765 filled it out using internet, and among them, only 5882 completed the full-length FFQ_10, approximately 53% of participants with 10-year assessment. In addition, only 1.5% of Q_0 was filled out using the internet, and the results of the comparisons of this questionnaire, according to the form of administration, should be interpreted with caution. Third, Web-based cohorts were used to include motivated internet-skilled volunteers; this selection bias is not likely in our cohort, as there is wide access to the internet in all subgroups of the population [7]. On the other hand, despite a selection bias that could be the major factor limiting the generalizability of results, because of the nonrepresentative nature of the internet population, this bias is not very likely, as in Spain, 84.6% of the population between 16 and 74 years had easy access to the internet in the last 3 months [37]. However, it is possible that some older study participants who have greater difficulty using the computer did not fill out the Web-based questionnaires, as they needed a completion guide. Although in the SUN project all participants are university graduates with total access to internet and computers, without apparent difficulties of use, we could have hoped for higher response rates for the Web-based questionnaires [38]. Fourth, there was a difference in the way the Q_0 (paper) and the Q_10 (by internet) were filled out, which could have introduced a systematic error in our results. Although they contained exactly the same items and portions and only varied in format of presentation, it is possible that the previous knowledge of participants when they completed the follow-up questionnaire may have also interfered in their answers of follow-up questionnaires, but this bias affects both groups. Fifth, in the SUN cohort, if participants forget to click on an answer, an error message does not appear before they can go to the next page. For this reason, in some questions of Q_0 or Q_10 by internet, the average number of data missing is high. Sixth, this cohort was formed by graduates from University of Navarra, as well as from other different Spanish universities and professional associations, limiting the external validity of our results, which is required to extrapolate the present findings to the general population. However, epidemiology cohorts are usually nonrepresentative, and generalization should be based on biological plausibility. Seventh, participants could complete the Web-based questionnaires only since 2004, and this fact might result in underestimation of the effects of administration mode on the survey metrics. Finally, the SUN study is based on self-reported information; however, because of the high motivation of the participants (only 10% of the participants accepted to enter to the cohort) [39] and their high education level, we can assume high quality data. This paper provides an overview of the implementation of novel technologies in a large-sample cohort. The main lesson learned from methodological research in the context of the SUN cohort is the following: there is a need for the electronic version to be validated against the original paper-based format. On the other hand, the Web-based form should be designed to be completed in parts, should automatically reject incomplete questionnaires, and should also point out missing or contradictory items [35]. However, these messages may increase respondent frustration and thus decrease completion rates [40]. Minimal technical problems still need improvement before these new methods become common practice [41]. In conclusion, in the digital era, technological progress is having a significant impact on all aspects of our lives, and it is also accelerating scientific discoveries and changing research methods [2]. Several methods for the assessment of dietary intake are currently available in nutritional epidemiology, all have their own limitations and advantages. Thus, the dietary assessment methods should always be selected with caution, considering the research, objective, hypothesis, design, and available resources [10]. Finally, although the performance of innovative dietary assessment technologies has been investigated, more research is needed in regard to their validity and the most effective future strategies that could be incorporated into nutritional epidemiology [20,40,41].