Development and Actionability of the Dutch COVID-19 Dashboard: Descriptive Assessment and Expert Appraisal Study

Introduction

In response to the COVID-19 pandemic, caused by the infection of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), that emerged in 2019, governments worldwide have been forced to take measures that impact the lives of individuals in order to protect the health of their citizens. Making the right COVID-19 policy decisions requires a balanced trade-off between protecting the population from infection and its consequences, ensuring that non-COVID-19 care needs are met (dual-track capacity monitoring), and minimizing socioeconomic impacts [1]. Web-based public reporting by means of dashboards has become an essential tool for monitoring COVID-19 information and communicating it to the public [2-4]. Moreover, dashboards are used to support individuals in informed decision making, for instance to educate citizens on whether they need to adapt their behaviors to minimize individual and population risk [5]. As such, dashboards are a powerful communication tool, and they are also frequently used by media as key information sources. If, however, information in dashboards is based on suboptimal reporting practices (eg, incomplete or unreliable reporting of data), it could produce undesirable effects, such as misleading perceptions, stress, or anxiety [6-9]. Users of dashboards must therefore be assured of complete, valid, reliable, and balanced information that can support them in making informed decisions as they deal with the pandemic. Information can be actionable only if it is fit for purpose and fit for use. Important determinations in the development of dashboards therefore include the selection and standardization of indicators at national and international levels, sources of data collection, analysis of data, and visualization techniques chosen to display the data [9,10]. The World Health Organization (WHO) advised 4 key types of information needed to effectively manage transitions and modulate restrictive measures over the course of the COVID-19 pandemic: (1) public health and epidemiological, (2) health system management, (3) behavioral insights, and (4) social and economic impact [11].

In mid-2020, an international network of performance intelligence researchers [12], to which the authors of this paper belong, carried out a global study to assess the actionability of 158 COVID-19 dashboards at international, national, and regional levels in 53 countries worldwide [13]. Actionability refers to a dashboard’s potential to inform decision making by way of providing information that is both fit for purpose—meeting a specific information need—and fit for use—placing the right information into the right hands at the right time and in a manner that can be understood [14]. The study identified 7 features that were common to highly actionable dashboards: (1) knowing the audience and their information needs; (2) managing the type, volume, and flow of displayed information; (3) reporting data sources and methods clearly; (4) linking time trends to policy decisions; (5) providing “data close to home”; (6) disaggregating the information into relevant subgroups; (7) using storytelling and visual cues.

The international dashboard study also found that there was overlap in indicators between the international dashboards, being that most dashboards portrayed case numbers, hospital admissions, and deaths due to COVID-19. Yet, little is currently known about how dashboards have improved their content and actionability over the course of the pandemic and what guided the decisions that were made in the development of the dashboards. This will very likely be context-dependent, thus a closer look at a specific country context can provide insight into the “why” of COVID-19 dashboard development. One study has looked at the development of Canadian COVID-19 dashboards and concluded that data availability and dashboard technology witnessed the most improvements. However, no improvements were found in communicative elements [15]. The development of the Dutch government’s COVID-19 dashboard, led by the Ministry of Health, Welfare and Sport, is continuously evolving in which publicly available feedback from expert panels [16] and from surveys among the public [17,18] are used to further develop the dashboard’s content and its communication strategy. This makes it an interesting case study for research. With this study, we set out to evaluate how the Dutch COVID-19 dashboard developed from its launch on June 5, 2020 through January 31, 2021, to assess its actionability, and to understand what factors guided decisions during the dashboard’s development.

Those objectives translated into the following research questions: (1) How did the Dutch COVID-19 dashboard develop over the course of the pandemic from its launch on June 5, 2020 to January 31, 2021? (2) How actionable is the Dutch COVID-19 dashboard, based on its December 2020 version, reflecting on features of highly actionable COVID-19 dashboards? (3) What decisions were made in the development of the COVID-19 dashboard by its developers and why?

This study aimed to improve the Dutch COVID-19 dashboard’s actionability in order to enhance public understanding of the pandemic.

Methods

Scope and Study Setting

This study focused on the development of the Dutch government’s COVID-19 dashboard [19] from June 5, 2020 (launch) to January 31, 2021. The first confirmed COVID-19 patient in the Netherlands was reported on February 27, 2020 to the House of Representatives by the Minister for Medical Care [20]. Hygiene and safety recommendations announced on March 9, 2020 at a press conference by the Dutch prime minister, accompanied by the director of the National Institute for Public Health and the Environment (RIVM), were rapidly followed by a number of policy measures to enforce physical distancing, impose travel restrictions, and establish case management and quarantine policies to contain the spread of the SARS-CoV-2 virus. In the weekly COVID-19 mortality numbers, a first peak was seen from mid-March to late May 2020, followed by relatively low numbers during the summer months of June to September, with numbers increasing again from October 2020 [21]. The COVID-19 measures taken by the government reflect these 2 high-burden periods, showing a pause or partial loosening for most measures in the June to September period [22,23]. In the management of the pandemic, the national government is advised by an Outbreak Management Team coordinated by the RIVM and works closely together with the 25 safety regions that, by law, have a responsibility for regional disaster management [24].

In May 2020, in addition to existing Dutch websites providing information on the pandemic [25-27], the Minister of Health, Welfare and Sport proposed a dedicated COVID-19 dashboard with the aim of keeping track of the pandemic and informing policy decisions to the House of Representatives [21]. On June 5, 2020, a test version of the official Dutch government dashboard for COVID-19 (entitled Coronadashboard) was launched for the general public [21].

Descriptive Assessment of the Development of the COVID-19 Dashboard by the Netherlands Ministry of Health, Welfare and Sport Over Time

To examine the development of the Dutch COVID-19 dashboard, 3 points in time were chosen for our descriptive assessment. The first was conducted on July 16, 2020, approximately 1 month after the dashboard’s launch, and was part of our international study of COVID-19 dashboards [28]. The second was on December 23, 2020, amid new containment measures, and the third was on January 31, 2021, just after vaccinations had commenced. The dashboard website was archived at these time points (Multimedia Appendix 1). We used the descriptive checklist developed and validated in the international dashboard study (see Multimedia Appendix 2 for the checklist and its measures) [28]. The tool was based on communication theory (notably on Lasswell’s Model from 1948 [29]), the discipline of performance intelligence in health [30,31], and existing evidence on performance indicators, public reporting, and the use of dashboards [28]. In addition to general and context aspects, the tool focuses on whether the purpose of the dashboard and the type of users are noted, what indicators are presented, whether metadata and the sources are made available, what disaggregation options are presented, and the use of visualization techniques.

Actionability Appraisal of the COVID-19 Dashboard

To appraise the actionability of the Dutch COVID-19 dashboard, we rated it in terms of the 7 common features of highly actionable COVID-19 dashboards (Table 1) [28].

Two individual researchers (VB and TJ) marked these as either not present, somewhat present, or clearly present and provided argumentation for the ratings in free text. The actionability scoring was performed on December 17, 2020 (see Multimedia Appendix 2 for links to the archive). After their individual scorings, they shared and discussed their findings. Discrepancies in scorings were discussed between the 2 researchers and resulted in either an agreement for the score or an agreement to clarify discrepancies in the concluding qualitative appraisal for each of the 7 features.

Appraisal Validation and Expert Reflections on the Decisions Made Throughout the Development Process of the COVID-19 Dashboard

A reflection meeting with the project team of the Ministry of Health, Welfare and Sport responsible for the COVID-19 dashboard was organized on April 23, 2021, with the following aims: to verify the appraisal of the dashboard’s actionability, to reflect on the development of the COVID-19 dashboard, and to understand decisions that were made towards the development of the dashboard. The project lead and 2 members of the team were present at the meeting. First, the study team presented the previously identified 7 key features of actionable dashboards. Second, the dashboard team shared their general reflection on the development of the dashboard from launch to date. Finally, the actionability appraisal of the 7 features of actionable dashboards were discussed in-depth. Due to COVID-19 restrictions, the meeting was conducted digitally. The meeting was moderated by NK, and notes were taken by TJ and VB. The Dutch dashboard scoring and appraisal for the 7 features were shared with the team in advance.

Results

Development of the COVID-19 Dashboard Over Time

Table 2 shows the findings of the descriptive assessment of the Dutch COVID-19 dashboard at 3 points in time.

Purpose and Users

A general statement of purpose was found at all 3 points in time. The statement highlighted the severity of the COVID-19 pandemic and explained the need to create a dashboard in order to use information to monitor the virus and to ensure that measures would be taken only where appropriate. The statement also emphasized that a political assessment of all relevant aspects was to include broader social and economic interests (see Multimedia Appendix 1 for archives). The target audience for the dashboard was not explicitly stated.

Content and Data

The number of indicators available on the dashboard increased over time, from 10 to 63 indicators. Both new themes were included (such as behavior and vaccinations), and the level of detail and stratification of previously available indicator themes was expanded (such as deaths, with an added indicator of excess mortality). The dashboard’s focus was mainly on epidemiological indicators (at different time points: 80%, 78%, and 79%, respectively), such as infection incidence numbers, reproduction rate, estimated number of infectious persons, and deaths. Health system capacity indicators included COVID-19 patients (such as numbers admitted to hospitals) and did not address non-COVID care needs. During our study, 4 social and behavioral indicators were added (added from December 10, 2020), such as percentage supporting wearing masks in public transport and percentage adhering to wearing a mask in public transport. These indicators showed self-reported survey outcomes on the compliance with and support for the COVID-19 measures taken by the government, based on data collected by the RIVM. No indicators relating to the social and economic impacts of the pandemic were observed.

Data sources were noted at all 3 points in time, and the data used in the dashboard were largely open-source. Metadata were provided in a separate document.

Analysis and Display of Data

The time trend information became more detailed over time. Customizable time trends and comparisons of indicator outcomes with outcomes for previous weeks, days, 2-day periods, or 3-day periods were provided. Geographic information became more granular over time. For instance, additional indicators on municipal and safety region levels alongside the primarily available national level were provided at the last 2 assessment dates. Disaggregation options were also expanded over time, as age groups and multiple vulnerable groups (eg, disability care homes and people over 70 years old living at home) were added.

Although visualization options were not adapted, during the second and third descriptive assessments, we observed additional interactive options providing change of information and display and more use of simplification techniques through small icons. For example, absolute numbers of infections and a relative number per 100,000 inhabitants were used at all 3 time points of the descriptive assessment. At all 3 time points, a scale was used to visualize the relative numbers, but at the last 2 time points, interpretation of these numbers was simplified by adding a norm value and color coding (red colors for “bad” and green colors for “good”) to the scale.

Actionability of the COVID-19 Dashboard

Table 3 provides the initial researcher ratings and our consolidated appraisal of the Dutch government’s COVID-19 dashboard, version December 17, 2020, in terms of the 7 established features of highly actionable COVID-19 dashboards.

Decisions Made in the Development of the COVID-19 Dashboard by its Developers

Various reflections were made in the meeting with the COVID-19 dashboard team of the Ministry of Health, Welfare and Sport in the spring of 2021. The following sections include the considerations the dashboard development team made in their decisions to shape the COVID-19 dashboard, ordered by the 7 features of actionable dashboards.

Knowing the Audience and Their Information Needs

At the outset of the dashboard, policy makers, notably in the 25 safety regions, were the primary target audience. Even though much of the same indicators were used over time, there is currently a different medium to provide daily updates to policy makers. Thus, the dashboard was found to be gradually adjusted to addressing the general public. The dashboard has been designed for the purpose of high-frequency (daily) updates in a focused area, which, from the perspective of the COVID-19 dashboard team, is less fitting for the purpose of more broadly reporting on (health) systemwide performance to a wider public. The data infrastructure of the dashboard can be re-used in future situations in which high-frequency daily data updates might be necessary for reporting on other (health) system performance areas.

Managing the Type, Volume, and Flow of the Information

The letter to the House of Representatives in May 2020 has been the leading document in guiding choices on type, volume, and flow of information displayed on the COVID-19 dashboard. The information on the dashboard was found to be organized in chronological order of the infection and successive COVID-19 disease process (estimate infected persons, COVID-19 positive test results, hospital admissions, intensive care unit admission, deaths). To increase quick access to the most important information, a homepage was added when the audience of the dashboard changed from policy makers to the broader public. To facilitate navigation, visits to the different subtabs of the dashboards were reported to be monitored. When a certain subtheme had become more popular, the sidebar could be amended to prioritize that particular subject. For example, when the vaccination subtab was visited more often, it was moved to the top of the page.

Making Data Sources and Methods Clear

It was a political decision to foster transparency in data sources and methods. To ensure that the data sources and methods would not yield discussion, transparency of data was maximized by making it largely available as open source.

Linking Time Trends to Policy Decisions

The dashboard development team was very careful not to link time trends to policy decisions. A caveat of linking policy decisions to time trends is that correlation is suggested. From the perspective of the dashboard development team, it was a deliberate choice not to make these assumptions. Instead, formal interpretations were handled by political decision makers, who were advised by multidisciplinary experts via the Outbreak Management Team.

Providing Data “Close to Home”

The dashboard team explained that, in some phases of the pandemic, as in the summer 2020 period when case numbers were low, more disaggregated information should ideally be provided in order to act locally to prevent the spread of the virus. However, the team did not have access to neighborhood-level data from all municipalities. Another consideration was that the more an individual zooms in at the local level, the lower the numbers become. For the perception of severity of the transmission and due to privacy issues, disaggregation could challenge its usefulness. The dashboard team did not think it was wise in this stage of the pandemic to invest in more locally available data. The team anticipated that in the future, when most of the population of the Netherlands is vaccinated, attention will turn to investing in information on the international status of the virus. Doing so may provide insights into possible mutation effects that can arise from international travel.

Breaking Down Population Data Into Relevant Subgroups

Disaggregating the data, for example by sex, has not been high on the agenda for the dashboard’s development. The dashboard team explained that data for socioeconomic status or ethnicity are not available in their database and would require integration of data on a personal identifier level, which was not available to the dashboard team.

Using Storytelling and Visual Cues

News-like items were added when the audience of the dashboard changed from policy makers to the broader public. This decision was made to enhance public understanding of the information provided, especially for visitors to the dashboard with less affinity towards numbers. During the development of the dashboard, ongoing research on public interpretation and understanding of different features of the dashboard were conducted in order to enhance actionability.

Discussion

Principal Findings

We assessed the COVID-19 dashboard provided by the Dutch government in terms of its development during the pandemic and its actionability for conveying information on the pandemic and supporting data-driven decision making. We found that the dashboard predominantly showed epidemiological information on COVID-19. It had been developed and adapted by adding more in-depth indicators, more geographic disaggregation options, and new indicator themes. It also changed in target audience from policy makers to the general public; thus, a homepage was added with the most important information, using news-like items to explain the provided indicators and conducting research to enhance public understanding of the dashboard.

Two of the 4 key components advised by the WHO Regional Office for Europe for monitoring and managing the pandemic were still missing from the dashboard 10 months after the first patient with COVID-19 was reported in the Netherlands: (1) indicators of available capacity for dual-track health system management to safeguard non-COVID care and (2) indicators of social and economic impact. As COVID-19 care needs draw on limited health system resources, non-COVID care needs must also be taken into account when weighing options to take policy decisions [32,33]. Although information on health care service utilization for other conditions (dual-track monitoring) as well as socioeconomic impact is available via a variety of other government-linked organizations—including the Dutch Health Authority (NZA), Statistics Netherlands (CBS), and the Netherlands Institute for Social Research (SCP)—such information is not provided on the COVID-19 dashboard, thus constraining an overall assessment of the situation. As the pandemic persists, the monitoring of both non-COVID care access and socioeconomic impact is of growing importance to adequately inform policy decisions [11]. Several Dutch political parties have emphasized the need to obtain and discuss such information in a more “overall” format.

Experiences from other countries have shown the importance of relevant disaggregation into subgroups, such as socioeconomic status and ethnicity, in order to highlight inequalities within populations [34-38]. Reported data from the United States on negative changes in life expectancy and the substantial differences between ethnic groups have underscored the relevance of transparent, differential data [39,40]. Ethnicity is not represented in the established data infrastructure on health; thus, the Dutch COVID-19 dashboard does not provide this disaggregation option. In the Netherlands, cohort studies are used to gain insights into ethnic inequalities [41], but such studies carry significant time delays for accessing valuable information needed at the time of a pandemic. Two policy briefs released in May 2021 on the COVID-19 impact among ethnic groups underscore that disaggregating data on COVID-19 impact for ethnic groups can lead to valuable insights for policy decision making [42,43]. The dashboard development team also noted that disaggregation of data by sex and gender was not high on their agenda. COVID-19 outcomes have shown to be affected by sex and gender differences, for example in increased mortality in men compared with women [44]. Taking into account data on sex and gender is needed in order to distinguish potential sex-based immunological differences or gender-based differences in behavior that require tailored care or policy actions [45,46].

The dashboard requires timely data to inform the public of the constantly evolving context, as was the case with the indicators on administered vaccinations and percentages of infected individuals with mutated virus variants. Communicating a constantly changing message to a wide public is complex. Dashboards therefore need to promptly adapt and evolve so as to take the most important considerations of “today” into account, while also providing “stable” trend indicators to highlight the changing circumstances. When data reporting or data infrastructure lagged behind, the Dutch government chose in many instances to show estimated or not yet contextualized data, for example on estimated numbers of infectious people or on sewage water testing [47]. By our third assessment of the dashboard, news-like text items had been added to bridge gaps in still-unavailable data and to increase public understanding of the provided indicators, making use of textual explanations such as what impact the new variant might have and why caution was needed. Experts are divided as to whether it is wise to publish work-in-progress data for indicators such as sewage water testing. Some ask why an indicator should be shown if it does not lead to consequences in policy decisions [48], while others argue that it could provide early warnings to the general public [49,50]. Experts in health communication emphasize that information should be presented in a way that assumes that citizens have a choice in their behavior [51,52]. Then, a dashboard can not only communicate a general picture of fear but also acknowledge the freedom of individuals to make a choice, which appeals to their responsibility for the collective.

The duration of the COVID-19 pandemic and the pressing need for immediate action have exposed weaknesses in the health and social care data infrastructure of the Netherlands in terms of its timeliness and accessibility to (and interoperability of) health and social care data and socioeconomic data for the benefit of public health. These gaps are evident in the limited availability of timely dual-track indicators, data on social and economic impact, and possibilities for relevant population breakdowns. We advise (1) establishing timely indicators relating to health system capacity in the Dutch health information infrastructure, (2) including relevant data disaggregation options, and (3) enabling interoperability between social, health, and economic data sources.

Strengths and Limitations

A particular strength of this study is that the checklist and the features of actionable dashboards that we applied were developed on the basis of a large sample of international COVID-19 dashboards (158 dashboards in 53 countries) [28]. A number of limitations has to be taken into account in interpreting the presented findings. Both the pandemic and the dashboard are constantly developing. As 3 discrete points in time were chosen for the descriptive checklist, some back-and-forth changes to the dashboard between time points could have been missed. Another limitation is that, although the same researcher (VB) filled out the checklist at all 3 time points, the archiving of the dashboard during the first checklist was limited to a copy of its main page, so that the in-depth tabs could not be revisited and checked by other researchers.

Conclusion

The Dutch COVID-19 dashboard developed over time to be fit for purpose and fit for use in terms of providing epidemiological information to the general public as the target audience. This can help to transform the dashboard from a tool of public accountability to an instrument for community action to contain the pandemic. However, to enhance its actionability for monitoring COVID-19 and its social and economic impact, including a broader set of indicators may be considered. Therefore, gaps in the Dutch health information infrastructure need to be addressed. To strengthen the Dutch health system’s ability to cope with upcoming phases of the COVID-19 pandemic or future public health emergencies, we advise (1) establishing timely indicators related to health system capacity, (2) including relevant data disaggregation options (eg, sex, socioeconomic status), and (3) enabling interoperability between social, health, and economic data sources.