Handling Missing Data in COVID-19 Incidence Estimation: Secondary Data Analysis

Introduction

Surveillance data are vital for public health policy and resource allocation [1]. During the COVID-19 pandemic, the rapid analysis of incomplete data led to potential biases, affecting our understanding of COVID-19 knowledge, attitudes, and behaviors [2]. Additionally, a study using US infectious disease surveillance data demonstrated that missing data can impact measured health disparities, emphasizing the need to consider this limitation when interpreting disparity metrics [3].

The absence of standardized and systematically collected surveillance data during the COVID-19 outbreak has necessitated the use of robust statistical tools and approaches to address these data gaps. Despite the availability of various analytical techniques, the application of statistical modeling processes has been limited [4]. Moreover, when imputation methods have been used, they have often lacked detailed descriptions and transparency [5].

Addressing the problem of missing data in public health surveillence systems requires system-level solutions, such as collecting more complete laboratory data, improving data linkage, and designing more efficient data collection procedures [3]. The analytical challenges posed by the current pandemic present an important opportunity to assess the utility of available statistical methods. Regardless of data quality, missing data and suboptimal analytical strategies can reduce a study’s statistical power and lead to biased estimates, resulting in erroneous conclusions. Robust statistical methods are crucial to enhance future data collection efforts, data interpretation, and their clinical and public health implications [6,7].

Gaps in the existing literature lie in the inadequate use of statistical modeling approaches to address the problem of missing data in disease and risk factor monitoring systems, particularly during public health emergencies such as the COVID-19 pandemic [8]. This shortfall is critical because missing data can significantly hinder the accurate monitoring of disease trends and the formulation of effective public health policies [9]. While various imputation methods exist, their application in this context has been limited, leading to uncertainties in disease trend forecasting and policy recommendations. These limitations can result in skewed data interpretations, which may, in turn, affect resource allocation, emergency response strategies, and overall public health outcomes.

In the present study, we used several theoretical approaches based on statistical modeling and epidemiological concepts to address the challenge of using different statistical methods for handling missing data in the interpretation of community surveillance information collected during different pandemic periods. We evaluated the performance of several imputation strategies to determine the best approaches for dealing with missing data in disease monitoring, showing how handling missing data can affect estimates of the COVID-19 incidence rate (CIR) in different pandemic situations.

Methods

Study Context and Data Source

This study used data collected in Bac Ninh Province, Vietnam, during the calendar year 2021 from the surveillance system for patients with COVID-19/SARS-CoV-2 who were admitted to the National Institute of Hygiene and Epidemiology in Hanoi, Vietnam.

The database included information on 13,727 patients with COVID-19 collected from the beginning of the 2021 outbreak in Bac Ninh Province, from January 1, 2021, to December 31, 2021, without any missing data. Based on the information contained in this data set, and because we wanted to restrict our study population to cases that could be transmitted to the broader community, we calculated the CIR only for confirmed cases of COVID-19 (n=10,599; this represents 77% of the data set) that were diagnosed in each community from each district in Bac Ninh Province (Multimedia Appendix 1).

We decided to focus exclusively on community cases to understand the transmission dynamics in the broader community. We focused on 3 specific variables in the data set: the date of each community-acquired case of COVID-19 that was forwarded to the surveillance system, the community code, and the number of daily cases at the community level (Multimedia Appendix 2).

Overview

We conducted a simulation to calculate various statistical and epidemiological indices of this community epidemic, assessing the effectiveness of different methods for handling missing data across differing missingness proportions and pandemic periods for each of the 7 missing-data analytic methods. The simulation steps began with generating a reference data set by separating the data set into different periods. Subsequently, for each missingness proportion in each period, steps 2 through 4 were repeated, during which statistical and epidemiological indices were calculated for the 7 missing-data handling methods.

Step 1: Separating by Period

We separated the COVID-19 pandemic that was occurring in Bac Ninh Province into 3 distinct time periods using the following working definitions: the first period, the zero COVID-19 period, ran from January 1 to July 4, 2021. This was when the local government had tightened prevention policies and the primary goal was to stop the community transmission of COVID-19. During this period, there were multiple short-range waves of COVID-19 outbreaks, with the peak CIR ranging from 150 to 250 cases daily [10].

The next period, the transition period, took place between July 5 and October 22, 2021. During this period, the local government used a flexible pandemic policy with the goal of controlling community transmission of COVID-19 and minimizing the importation of new cases from affected provinces while increasing the population level of COVID-19 vaccine coverage. During this period, the highest CIR was more than 200 cases per day, but there were many days in Bac Ninh province with no notification of cases (CIR=0), with the longest range of zero notification days being more than 2 weeks [10].

The final study period, the new normal, ran from October 23, 2021, until the end of the study on December 31, 2021. During this period, the primary goal of public health officials was to open social facilities and terminate all isolation policies. The CIR in this period fluctuated, with multiple long-range waves of outbreaks; during the highest peak, there were more than 600 daily cases of COVID-19 [10].

Step 2: Generating Simulated Data Sets

We assumed that there were values missing completely at random in our study, so that the data values missing in our simulation data sets were unrelated to any observed or unobserved data in the data set. In other words, the missing data points did not depend on the values of other variables or the values of the missing variable itself. Inasmuch, we randomly changed the missing data percentage from 5% to 30%, with intervals of 5%, for each time point for the variable “cases per day at the community level.” This was defined as the total number of confirmed cases of COVID-19 that were diagnosed and reported daily at each community in Bac Ninh Province [8], resulting in 6 levels of cutoff percentages for missing data sets during each of the 3 distinct periods. We used the missMethods R package to generate missing values based on previous research that has shown the effectiveness of generating missing values in data sets [11]; 18 simulated databases were created in our study.

Step 3: Handling the Missing Data

The methods for handling missing data were based on a previous literature review of the techniques used in ecological data sets [12]. We selected 7 methods that we deemed to be suitable for imputing missing values from the number of daily cases of COVID-19 occurring in each study community.

Backfill Imputation

We used the number of daily cases from the previous day for each community unit as the value for imputation for the missing values of that community. If there were no cases on the previous day to impute, we assumed a missing value of 0 because when no data were available from the previous day, assuming a value of 0 was a conservative approach, indicating no new cases reported. We used the “na. locf()” function in the zoo package of R to conduct this imputation process [13].

Moving Average

We used the mean of the last 14 days of COVID-19 as the average for imputation. The cutoff time of 14 days served as the reference for the minimum time for a COVID-19 containment cycle (CCC) [10]. We created a function to carry out this process.

Median Imputation

We created a function in R to use the value of the number of daily cases of COVID-19 during the last 14 days in each community as the reference to find the median for imputing missing values for that community.

Maximum Likelihood

We used maximum likelihood estimation (MLE), which is based on a normal distribution. We created a function to conduct this process. First, we calculated the MLE for the mean (μ) and SD (σ) of the last 14 days of nonmissing values in the input variable x. Then, for each missing value, we randomly sampled a value from a normal distribution with mean (μ) and SD (σ), effectively replacing the missing value.

Linear Interpolation

We use the “na_interpolation()” function in the imputed package of R [14]. Missing values were replaced by values estimated by linear interpolation, which created a linear relationship between neighboring known data points (the last day and the next day).

Autoregressive Integrated Moving Average Model

We used the “auto. arima()” function in the forecast package of R for calculating imputed missing values [15]. The autoregressive integrated moving average (ARIMA) model combines 3 key components: AR (the “autoregressive” term), I (the “differencing” term), and MA (the “moving average” term). The AR term refers to the past values used for forecasting the next value while the MA term is used to define the number of past forecast errors used to predict future values. The order of “differencing” specifies the number of times the differencing operation is performed on a series to make it stationary. In the default figures, the maximum number of historical observations was set to the last 5 days. The ARIMA model subsequently determined the order of these components (from 1 to 5 previous days might be possibly related to the current data), and imputation values were chosen through data analysis and model selection techniques.

K-Nearest Neighbor Imputation

We used the closest data points to the one with missing values. In our study, we used the “kNN()” function in the VIM package of R to fill in missing daily COVID-19 case counts at the community level by K-nearest neighbor (KNN) imputation [16]. This method estimates missing values based on nearby data points. We applied KNN with a set number of neighbors, in our example 14 days, representing the minimum time for a CCC in each community [10].

Step 4: Estimating the Effectiveness

To illustrate the efficacy of various missing data handling methods in estimating the CIR, we implemented the 7 imputation techniques to address missing data during different study periods and levels of missing data.

On the statistical side, to assess the extent to which these missing data handling methods mitigated the effects on estimating the CIR, we examined bias and the root mean square error (RMSE) resulting from direct comparisons between the imputed and original values of the daily CIR for a population of 1,000,000 people. We computed the mean absolute crude bias (${\displaystyle \overline{ACB}}$) and the mean crude RMSE (${\displaystyle \overline{RMSE}}$) as indicators of performance [8]. To quantify the alterations in CIR between the original and the imputed data sets, we employed the mean absolute percentage change (APC) in the CIR, denoted as ${\displaystyle {\overline{APC}}_{CIR}}$ (Multimedia Appendix 3).

From the epidemiological perspective, we used the average number of confirmed cases in each CCC as the reference index to measure the effectiveness of the imputation data. The CCC consisted of several nonpharmacological control strategies aimed at managing the COVID-19 pandemic within each community in Bac Ninh Province [10]. We used the mean APC of the mean of the average confirmed cases of COVID-19 for each CCC, referred to as ${\displaystyle {\overline{APC}}_{cases}}$, to discern differences in confirmed cases for each CCC at the community level between the original and imputed data sets (Multimedia Appendix 3).

R (version 4.2.2; R Foundation for Statistical Computing) was used for all data analyses that were carried out.

Ethical Considerations

The study received approval in accordance with decision 4326/QD-DHYHN by the Institutional Review Board of Hanoi Medical University. All methods were conducted in compliance with the committee’s guidelines and regulations. We received permission for all the data sets in this study from the Vietnam National Institute of Hygiene and Epidemiology for use and analysis. All personal information and identifiers were removed from the data set prior to analysis.

Results

Zero COVID-19 Period

Figure 1 shows the results of the imputation methods used to address missing data in the context of the CIR during the zero COVID-19 period. Among these methods, KNN imputation showed the lowest mean ACB and mean crude RMSE values from 5% to 20% missing-data levels. In the 25% to 30% missing-data levels, while KNN imputation and median imputation consistently yielded lower mean ACBs than the other methods, linear interpolation imputation had the lowest mean crude RMSE.

‎

Table 1 provides an assessment of the mean APC in the CIR and the average number of confirmed cases in each CCC during the zero COVID-19 period using the 7 imputation methods to address missing data. Median imputation and KNN imputation consistently exhibited the lowest mean APC values for both CIR and for the average number of confirmed cases during each CCC. The moving average imputation method followed as the second-lowest performer for APC in CIR, with the mean APC increasing gradually as the level of missing data increased. Backfill imputation was the second-lowest performer in APC in terms of the average number of confirmed cases during each CCC, with the mean APC value rising nearly similarly to the median imputation results. Both backfill imputation and median imputation had APCs on average for the number of confirmed cases of COVID-19 during each CCC higher than KNN imputation at all levels of missing data. In contrast, the linear interpolation imputation method consistently exhibited the highest mean APC values across the specified levels of missing data. Lastly, the ARIMA model imputation and maximum likelihood imputation methods demonstrated the second-highest mean APC values when missing data levels increased; the ARIMA model imputation had the highest APC on average for the number of confirmed cases of COVID-19 during each CCC.

Transition Period

Figure 2 shows the results of the different imputation methods used to address missing data in the context of the CIR during the transition period. The ARIMA model and KNN imputation methods consistently demonstrated the lowest mean ACB across all levels of missing data; the ARIMA model and median imputation methods had the same results in terms of the mean ACB and mean RMSE. With regards to the mean crude RMSE, the moving average and ARIMA model imputation methods consistently yielded lower values than the other methods across varying levels of missing data. On the other hand, the maximum likelihood imputation method generally resulted in higher mean ACBs and mean crude RMSEs compared with alternative methods. The backfill imputation method exhibited the second-highest mean crude RMSE, particularly at the 20% to 30% level of missing data.

Table 2 presents an overview of the 7 imputation methods used to address missing data in the CIR and the average number of confirmed cases during each CCC. The median and ARIMA model imputation methods consistently displayed relatively lower mean APC values for both the CIR and average of confirmed cases in each CCC than the other analytic methods. The backfill imputation and KNN imputation methods provided the second-lowest mean APC values as the level of missing data increased. In contrast, the maximum likelihood and moving average imputation methods displayed comparatively higher mean APC values than the other methods of imputation.

‎

New Normal Period

Figure 3 illustrates the mean ACB and mean crude RMSE of the 7 imputation methods used to address missing data in the CIR during the new normal period. Both the backfill and linear interpolation imputation methods consistently demonstrated the lowest mean ACB across all levels of missing data. The ARIMA model imputation and KNN imputation methods provided the second-lowest absolute mean ACB and mean crude RMSE compared with the other analytic methods across different levels of missing data. On the other hand, the maximum likelihood and moving average imputation methods showed the highest mean ACB and mean crude RMSE as the level of missing data increased.

Table 3 displays the mean APC between the original and imputation data sets when we addressed varying levels of missing data in the CIR and in the average number of confirmed cases in each CCC during the new normal period. Three statistical methods, namely the backfill, linear interpolation, and KNN imputation methods, consistently exhibited relatively lower mean APC values compared with the other imputation methods. While the ARIMA model imputation method provided the second-lowest mean APC in CIR values as the level of missing data increased, median imputation had the second-lowest mean APC in terms of the average number of confirmed cases in each CCC at all levels of missing data. In contrast, the maximum likelihood and moving average imputation methods consistently displayed higher mean APC values in the CIR and in the average number of confirmed cases during each CCC than the other methods of imputing missing data.

‎

Discussion

Principal Results

In examining our study’s primary objective, which was to demonstrate how different methods of handling missing data affect estimation of the CIR, we highlight how the ongoing pandemic, as well as the preventive measures and health policy recommendations that were used to control future cases of COVID-19 in the community, could affect the effectiveness of different analytical methods. After examining 7 imputation approaches, we found that KNN and median imputation performed the best during the zero COVID-19 period, with KNN also having the lowest mean APC in terms of the CIR. ARIMA and median imputation were the most successful analytic approaches used during the transition period, whereas backfill, linear interpolation, and KNN performed the best during the new normal phase. Inasmuch, our findings show that one’s selection of the different imputation methods that could be used must take into account the specific pandemic conditions to increase the accuracy of predicted incidence rate estimates.

Comparisons With Prior Work

Several of our findings differ from those of a study that was designed to find the best way to handle missing data for estimating a wellness index over the lifetime based on panel data from smart devices that collected various types of life logs, such as steps walked and sleep duration [17]. Our findings also differ from a study that examined how well artificial neural networks handle missing data collected in a pediatric intensive care unit [18]. The differences between our results and these previous studies were due to the performance of different imputation methods and a focus on different pandemic time periods. This underscores the importance of understanding the particular pandemic situation and developing and using health policy measures considering the potential biases and effectiveness of these analytic techniques. During periods of strict population-based control, such as the zero COVID-19 period, simpler methods, such as KNN and median imputation methods, which rely on recent data, could be used. In contrast, during more volatile periods of viral infections, such as the transition and new normal periods, methods that model temporal dependencies or use neighboring data points, namely the ARIMA model and KNN imputation methods, are more effective.

Our results also highlight the limitations of certain analytic methods, such as the maximum likelihood and moving average, which generally showed higher mean ACB and crude RMSE values, indicating less robustness in handling variability in the extent of missing data during different pandemic phases. These methods are, however, often used to handle missing data in medical data sets. For example, in a study involving 50 individuals selected from a 2 × 2 randomized controlled trial, the moving average method showed the best agreement with observed values [19]. This study compared various data imputation methods for calculating body weight variability using both linear and nonlinear approaches. Moreover, maximum likelihood imputation methods have been used for handing missing data at random in a number of randomized controlled trials [20]. The limitations of these methods in our study may be attributed to their underlying assumptions, which might not hold in the rapidly changing context of a pandemic, leading to increased bias and error in the calculation and interpretation of imputed data and illness incidence rates.

The effectiveness of each imputation method that we used in this study was influenced by the underlying data structure and characteristics of missingness during each pandemic period. For example, the backfill method, which assumes that the last observed value can be carried forward, may work during periods of low variability but can introduce significant bias during high variability periods, such as the new normal period. Similarly, moving average methods might not capture true variability in the number of cases of disease that may occur during rapid changes in transmission dynamics. Our findings differ from an observational study in 2023 that used moving average imputation for 3 public, completed time-series data sets that were collected from power equipment [21]. This study aimed to create a customized methodology that combined an asymmetric denoising autoencoder and a moving average filter to impute missing data in time-series monitoring data. When choosing different imputation methods, it is crucial to consider epidemic-specific and contextual factors. Data may be missing due to overwhelmed health care systems or reporting delays in the number of cases of confirmed illness, leading to errors in data interpretation and policy recommendations to contain the spread of disease. In addition, the stage of the epidemic and extent of use and effectiveness of public health interventions can impact the suitability of different imputation techniques. Understanding these factors is essential to selecting methods that minimize bias and accurately reflect underlying trends in disease magnitude and health-related outcomes.

These findings emphasize the need for transparency and detailed reporting in the application of data imputation methods. The lack of detailed descriptions and transparency in the reporting and application of these methods in previous studies has been a significant limitation in interpreting the published literature. By providing a comprehensive analysis of various imputation techniques and their performance across different pandemic phases, this study contributes to a better understanding of how to more effectively handle missing data in disease surveillance. The detailed comparison of methods and the consideration of different pandemic phases provide valuable insights for future research and public health practice.

Study Strengths and Limitations

The main strength of this study is that we used individual data to calculate the number of new cases of COVID-19 that were diagnosed and reported to public health authorities on a daily basis in each of the communities studied. Moreover, we were able to compare the original values with the imputed estimates that were collected during the 3 periods of this ongoing epidemic in a large Vietnamese province.

There are some limitations of our study, however, that need to be kept in mind in the interpretation of our principal study findings. Because we targeted an extensive range of missing values greater than 5%, we did not use any methods to ignore missing data or delete the missing values, such as listwise deletion or pairwise deletion. In addition, we used large imputed data sets and did not use methods useful for handling missing data in studies with small sample sizes, such as data augmentation [22]. Furthermore, our results are primarily limited to handling missing data with missing completely at random patterns without a need to account for potential biases that may have been introduced by nonrandom missing data. Scenarios in which data were missing at random or missing not at random were not addressed in this study [23]. Future investigations will be needed to analyze these types of missing-data scenarios. Another limitation is that our study did not account for unexpected cases, as the data were produced based on existing data and therefore may not represent some unforeseen phenomena.

Conclusions

This study illustrates that the choice of imputation method used should be tailored to the specific epidemiological context and data collection environment. Statistical modeling and a thorough understanding of local pandemic dynamics are essential for improving the accuracy of incidence rate estimates and, in turn, public health responses to ongoing disease trends and the development and application of disease control measures. Future research should continue to refine these methods, ensuring that they can adapt to the evolving challenges of disease surveillance in public health emergencies. By improving currently available imputation methods, we can facilitate more accurate and dependable public health responses in future situations, ultimately contributing to better resource allocation, emergency response strategies, and community health outcomes.