We chose a between-subjects design with 1 three-level independent variable: participants had to solve fictional case vignettes while receiving different types of decision support (no support in the control group, a DST in the form of a static flowchart in the first intervention group, and an interactive DST in the second intervention group). Participants were randomly assigned to 1 of the 3 groups. Decision support quality was evaluated by analyzing the tools’ effects on decision making and the participants’ judgement of the tools with multiple dependent variables: Regarding effects on decision making, the number of correct appraisals (using official CDC recommendations from October 14, 2020, as the standard [28]), perceived certainty regarding the appraisal, and mental effort in making the appraisals were examined. Participants' judgments were collected on tool usefulness, ease of use, trust in the tools’ recommendations, and intention to use the tools in the future.

The Ethics Committee of the Department of Psychology and Ergonomics (IPA) at the Technische Universität Berlin approved this study (tracking number ROEB_01_20200715). Participants volunteered to take part in the survey and provided informed consent before participating.

We designed 2 mock DSTs. The design was intended to mimic existing tools available on the internet. Features of various tools related to COVID-19 were analyzed. Relevant common features of these tools were the use of short, precise questions or statements to which the participants could respond in agreement (yes) or disagreement (no), and sometimes also with uncertainty (unsure). These questions guided participants through a decision tree’s nodes step-by-step until a recommendation could be provided. Input requirements and output content were mostly based on recommendations from public health agencies, such as the CDC. For the purpose of our study, the John Hopkins University Coronavirus Self-Checker was identified as a most suitable template because it is a decision tree–based interactive tool that requires a small amount of information at each step as input [18]. This facilitated the construction of a second DST providing identical advice in the form of a static flowchart, outlining the entire decision tree at once. In both cases, the content was based on the CDC recommendations from October 14, 2020 [28].

Considering these aspects, we developed 2 mock DSTs with identical content that differed only with respect to the interactivity of their user interface: a static flowchart where participants could follow the arrows of the appropriate path using their gaze (Multimedia Appendix 1). This flowchart was developed using Microsoft PowerPoint [29]. The other tool was implemented as a simple, interactive conversational agent, where participants clicked buttons to respond to questions, guiding them through the nodes of the decision tree step-by-step (Multimedia Appendix 2). The individual screens of the interactive DST were also designed using Microsoft PowerPoint and then linked together using InVision [30], enabling dynamic interaction. Mimicking the output of existing tools, 5 different recommendations were possible: “emergency care is required,” “self-care and physical distancing are sufficient,” “self-care is sufficient and quarantine necessary,” “self-care is sufficient and isolation is necessary,” and “nonemergency care and isolation are required.” We anticipated that laypersons would struggle comprehending the difference between “quarantine” and “isolation,” yet we decided to differentiate between them, as some COVID-19 DSTs use these terms without explaining them [7,11]. During data analyses, however, we rated answers as correct when participants confused these 2 terms (see the Data Analysis section). We developed mock tools rather than choosing existing DSTs from the internet in order to ensure comparability of the algorithm and design of the tools, as well as stable integration into the survey throughout the surveying period.

We created 7 short patient descriptions (case vignettes). In this process, we extracted decision criteria identified by the CDC and its recommendations concerning help-seeking and social behavior in the case of a potential COVID-19 infection [28]. Decision criteria included the presence of typical COVID-19 symptoms with the following expressions: 0, no symptoms; 1, primary symptoms (typical for a COVID-19 infection); and 2, secondary symptoms (can also occur in a COVID-19 infection but are less typical) (see Multimedia Appendix 3) [28]. Another decision criterion represented the potential contact with the pathogen SARS-CoV-2 and the following expressions: 1, critical contact with a confirmed infected person has occurred; 2, contact may have occurred; or 3, critical contact can be excluded. The last important criterion is the presence or absence of risk factors classified by the Robert Koch Institute (RKI), Germany’s federal public health authority (eg, certain pre-existing conditions such as chronic lung disease; see Multimedia Appendix 4) [3]. Through a pretest with 15 participants, in which we evaluated the response behavior of the participants as well as their free-text feedback, we revised the wording of the vignettes, improving intelligibility and removing errors spotted by the pretest participants. The vignettes included more information than necessary for the appraisal to better simulate a real-life decisional context and increase ecological validity. We also ensured that all information asked for in the developed DSTs was included in the vignettes and that the fictitious patients represented diverse age and gender groups. In addition, each possible outcome of the mock tools was covered at least once as a correct solution. Multimedia Appendices 5 and 6 contain an outline of the 7 case vignettes.

Data were cleaned and explored using R 4.0.2 [43] and the tidyverse packages [44]. After examining the central tendencies and distributions of the variables separately for the 3 groups and plotting them using the ggplot2 package [45], we performed a one-way analysis of variance to compare the 3 conditions separately for the dependent measures accuracy, decisional certainty, and mental effort. If significant group differences were present, we followed up with Bonferroni-corrected post hoc pairwise t tests. For the dependent variables’ usefulness, ease of use, trust in tool recommendations, and future use intention, we conducted Welch two-sample t tests to compare the 2 intervention groups because the sample sizes of the groups were unequal [46]. Effect sizes were quantified with Cohen d and calculated using the rstatix package [47]. When results were not statistically significant, we performed a post hoc power analysis using the pwr package [48] with corresponding group sizes, an α level of .05, and a power 1-β of .80.

In total, our survey was accessed 233 times during the 4 days it was available. Overall, 37 participations could not be used for data analysis because they dropped out of the questionnaire (n=12, 32%), exceeded the maximum survey completion time (n=10, 27%), failed attention checks (n=6, 16%), or did not meet eligibility criteria (n=9, 25%). The remaining participants all affirmed that they had paid close attention during the survey and answered honestly. This yielded a total of 196 (84.1%) participants, who assessed all 7 case vignettes and could be included in the analysis (see Table 1 for details). The mean time for completion of the survey was 22 minutes and 3 seconds. Across the 3 groups, the participant characteristics were similar; see Table 1.

Omnibus ANOVAs detected group differences with respect to decision accuracy (overall and separately for decisions about help-seeking behavior and social behavior) and perceived certainty but not for mental effort in decision making; see Table 2.

We investigated participants’ perceptions of the mock tools with 4 metrics: perceived usefulness, perceived ease of use, trust, and intention to use the tools again in the future. On average, participants perceived the DSTs positively: their self-reported trust in the tools was generally high in both experimental groups as was the number of DST recommendations the participants followed (on average, in more than 12 [86%] of a total of 14 recommendations); see Table 3. Accordingly, almost all (126/130, 95%) participants judged the mock tools to be useful (ie, indicated a perceived usefulness score of above 3.0 on a scale from 1 to 5), with half of the participants scoring the usefulness at or above 4.75. Ease of use was rated similarly positive (see Table 3), and their intention to use the tools in the future was high, on average. Regarding differences between the 2 DSTs, the mock users’ scores on perceived usefulness, trust, and future use intention were higher in the static than in the interactive DST group. Ease of use was perceived higher in the interactive DST group. However, all effect sizes of differences between the 2 DST groups were small (Cohen d≤0.35) and proved significant only for future use intention in a Welch two-sample t test; see Table 3.

Based on a post hoc power analysis, we estimated that on a population level, the real difference in these variables between both groups was less than Cohen d=0.5 (equivalent to a difference of 0.34 for perceived usefulness and perceived ease of use, 0.42 for self-reported trust, 0.96 for followed decisions, and 0.51 for future use intention), with a power of 0.8.

In our study, DSTs increased laypersons’ accuracy and certainty in decision making. Potentially, DSTs can also reduce the mental effort in decision making, but this effect was statistically nonsignificant in our experiment. Thus, our experiment confirms the benefit of DSTs for COVID-19-related self-triage discussed in prior studies [4,23].

Regarding the question of whether 1 mode of presentation of DSTs (ie, static or interactive presentation) is superior to the other, our experiment produced evidence that differences in measures of the quality of decision support and participants’ perception of the tools are small. Interactive DSTs are potentially more convenient to use. Users of the interactive DST rated the mental effort as lower and the perceived ease of use as higher than the users of the static DST. However, these differences were not significant and did not translate into a higher level of trust, greater perceived usefulness, or greater intention to use the tool in the future. On the contrary, users rated the static tool more favorably on these measures than the interactive DST. Overall, the effect sizes for differences in these measures were low and statistically nonsignificant.

To the best of our knowledge, our study is among the first to directly compare the effectiveness of different modes of presentation on the quality of support received from web-based, patient-facing clinical DSTs. Our results are in line with findings from similar research in a different field of application: In their 2018 meta-analysis on the use of PtDAs to support decisions concerning prostate cancer screening, Baptista et al [27] concluded that web-based decision aids reduce decisional conflict compared to no decision aid, but no more than static, printed-out decision aids. Comparability between Baptista et al [27] and our study is limited, as web-based decision aids and their paper-based printed-out versions assessed by Baptista et al [27] do not directly match the mock conversational agent and the static flowchart we presented digitally in our study. However, both results suggest that for a decision scenario with clear-cut options, the mode of presentation with a DST has little to no effect on how helpful it is for its users.

First, our results underline the benefits of making DSTs available to laypersons for decisions with clear-cut options, such as those encountered in the COVID-19 pandemic, as both decisional accuracy and trust in the decision increased when laypersons were supported by both mock DSTs. Second, the quality of decision support provided by the static flowchart and the interactive tool did not differ significantly, and post hoc sensitivity analyses on effect sizes indicated that we can rule out large effects. Thus, factors such as development effort might potentially weigh more than the format of interaction when deciding on how to present decision support to laypersons.

When complexity is low, as in the case of the COVID-19 decisions, the static version may provide a better overview and thus make the decision more transparent to its users and be quicker and more cost-efficient to develop and publish than an interactive conversational agent. In contrast, for decisions with higher complexity, full transparency might compromise ease of use of a static DST. Thus, for more complex decisions, interactive tools that guide the user step-by-step may be more user friendly. However, our results suggest that interactivity is not an effective means in itself to increase the usefulness of a DST: First, the higher convenience of using the interactive DST did not translate into greater trust or perceived usefulness, nor did it yield a greater increase in decision accuracy or certainty than static flowcharts. As the latter entail less development efforts, while increasing transparency, they might be the preferable mode of DSTs for public health officials to implement on their websites to provide guidance to the public on decisions of low complexity and limited decision space.

Our study also raises 2 topics for further research. First, which factors increase users’ trust in DSTs, and how are they weighed? From our results, we can only speculate that users prefer transparency over convenience. Second, would subgroups of the population prefer 1 type of interface over the other? Although our results showed only minor differences between sample averages of users of the static and interactive DSTs, potentially subgroups of the population benefit more from 1 type of interface than the other. For example, users with low technological affinity might prefer a static flowchart over a conversational agent.

The results of our study are mainly limited by concerns about external validity. That is, the interactive mock DST we developed does not fully represent the whole variety of interactive tools that are available on the internet. Although our DST mimics the interactive tool from Johns Hopkins University, other tools incorporate significantly more decision factors and possible outcomes. Second, the sampled study population’s composition of educational background is not representative of the adult US population. Our study sample included a higher proportion of highly educated persons, with only 4 (2%) of 196 participants being nonhighschool graduates as compared to 9.8% in the US adult population [49]. In addition, the mock interactive tool we developed for this paper does not fully exhaust the potential of interactivity, as our participants only followed prompts to respond to binary questions by clicking either Yes or No buttons, while more interactive tools also require users to enter information manually (eg, age or current location). Finally, participants in this study did not use the tool for self-assessment but to appraise cases with fictitious patients. This means that they did not experience personal concerns, as would likely be the case with COVID-19 suspicion in a real-world use situation. To promote external validity, the recommendations of the DSTs we developed conform with all CDC guidelines and their interactive capacities are basic but mimic those of existing DSTs.

When the decision space is limited, a static flowchart potentially performs just as well as an interactive tool in enhancing the decision quality of laypersons with symptoms indicative of a COVID-19 infection. As static flowcharts reveal their underlying decision algorithm more transparently, they might prove to be more suitable in not only guiding laypersons through the health care system but also communicating the reasoning and thereby empowering patients. Further research should validate our findings on different use cases, elaborate on the trade-off between transparency and convenience in DSTs, and investigate whether subgroups of users benefit more from 1 type of user interface than the other by assessing interactions between outcome variables (eg, accuracy, mental effort, perceived usefulness) and participant characteristics (eg, age, eHealth literacy). We have made all data necessary to conduct these exploratory analyses and to reproduce our reported findings publicly available [50].