We obtained the weekly number of HFMD notifications from outpatient clinics from the Korean HFMD surveillance system from January 1, 2017, to December 31, 2022 [8]. The data included the number of weekly reports and number of weeks published in the surveillance report. Furthermore, we reviewed the electronic records for the weekly number of notifications from the sentinel sites and surveillance operations to identify changes in the number of KDCA surveillance officers. Furthermore, we collected the monthly number of patients with HFMD from the Korean Health Insurance Review and Assessment Service (KHIRA) [15] from January 1, 2017, to December 31, 2022. The data from the KHIRA were based on the national health insurance reimbursement data of more than 80,000 health care providers in South Korea, covering around 97% of the South Korean population (46 million). It includes patient diagnoses using the International Classification of Diseases, Clinical Modification, 10th Revision (ICD-10-CM) [16].

The data were collected from KDCA’s web-based reporting system and include no personally identifiable information. Ethical approval was waived by the Institutional Review Board of Konyang University (KYU 2023-10-030).

We evaluated a nationwide HFMD surveillance system using a structured framework based on the updated CDC guidelines [10]. The evaluation attributes included timeliness, stability, completeness, sensitivity, and representativeness (Multimedia Appendix 3). These attributes have been widely used to evaluate infectious disease surveillance performance at the country level [17,18]. For consistency and comparability, we used a quantitative scoring scale from 1 to 3 for each indicator: weak (1), moderate (2), and good (3), from January 1, 2017, to December 31, 2022.

For timeliness, we measured the mean time lag for each year during the study period between the HFMD case notification (from the physician at the sentinels to the KDCA), acquired from case-report records, and dissemination (from the KDCA to the public), collected from the HFMD surveillance reports (Multimedia Appendix 4). As HFMD sentinel sites report weekly, all data were formatted in weeks. We calculated the mean and SD of the time lag of the surveillance system’s weekly notifications each year. Statistically, this is expressed by the following equation:

where x is each time lag between the case notification and results dissemination and n is the number of weeks in that year (ie, 52 wk). Furthermore, we provided the SD by using the following equation:

We considered the timeliness of the surveillance performance as weak (score 1), moderate (score 2), and good (score 3) using a 3-point scale (score 1: a mean time lag ≥5 wk; score 2: a mean time lag of 3‐4 wk; and score 3: a mean time lag of ≤2 wk).

We defined stability as how stable the surveillance system operates without interruption [9]. In this evaluation, we divided the monitoring system into 3 detailed categories: frequency of program suspension, staff replacement, and KDCA personnel training. We reviewed the surveillance operation records to identify the number of times that the surveillance system was halted for technical issues, including the blackout of the HFMD surveillance database system at the KDCA (score 1: >10 times; score 2: 6‐10 times; and score 3: <6 times). The web-based surveillance system was automatically programmed to record any technical issues and changes in the surveillance officers in charge. We reviewed the KDCA surveillance officer turnover each year (score 1: >300%; score 2: 100‐299%, and score 3:<100%). This is expressed by the following equation:

We also reviewed the number of times that the KDCA officers received surveillance-related training per year (score 1: none; score 2: 1‐2 times; and score 3: ≥3 times). These data were acquired from the KDCA web-based reporting and surveillance system’s electronic records. We calculated the mean of these 3 indicators each year to identify the overall stability (mean score between 1 and <2: weak; mean score of between 2 and <3: moderate; and mean score of 3: good).

In our assessment of completeness, we calculated the submission rate of complete notification reports out of the overall number of notification reports submitted by the sentinels to the KDCA each year. This is expressed by the following equation:

This value is automatically calculated in the web-based system. No reports from any sentinel site were considered missing reports. We considered the completeness of the surveillance performance as weak (score 1), moderate (score 2), and good (score 3) using a 3-point scale (score 1: <80%; score 2: 80‐90%; and score 3: >90%).

To assess sensitivity for the incidence trends, we obtained the monthly number of reported HFMD cases in the surveillance system from the KDCA [8]. Due to the limited public health resources for identifying the nationwide incidence of HFMD, we assumed that the number of patients from the KHIRA represents the HFMD incidence in the community. We collected the monthly number of patients clinically diagnosed with HFMD (ICD-10-CM: B08.4) from January 1, 2017, to December 31, 2022 [15]. Then, we calculated the correlation coefficient for each year between the monthly number of notified HFMD cases in the surveillance system and the number of new patients with HFMD from the KHIRA. We first conducted a Shapiro-Wilk test to determine if the variables were normally distributed. Then, we conducted either the Pearson or Spearman test, where appropriate. The Pearson test is expressed by the following equation:

where ρ is the Pearson correlation coefficient between x (ie, the monthly number of notified HFMD cases from the KDCA) and y (ie, the monthly number of patients with HFMD from the KHIRA), cov is the covariance, σx is the SD of x, and σy is the SD of y. The Spearman test is defined as the Pearson test between the rank variables. We considered the sensitivity of the surveillance performance as weak (score 1), moderate (score 2), and good (score 3) using a 3-point scale (score 1: <0.7; score 2: 0.7‐0.79; and score 3: ≥0.8).

For representativeness (ie, age representativeness), we compared the proportion of 0‐ to 6-year-olds in the surveillance data, which is the age group most affected by HFMD [19], with the proportion of 0‐ to 6-year-olds from KHIRA data on patients with HFMD. We considered the representativeness of the surveillance performance as good (score 3) if the difference in the proportion was less than 10%, moderate (score 2) if the difference was from 10% to 20%, and weak (score 1) if the difference was more than 20%. Furthermore, we assessed the geographic representativeness of the sentinel surveillance sites by comparing the geographical distribution of the sentinel sites with the population distribution obtained from the Korean Statistical Information Service [20]. Using the overall designated number of sentinel sites in South Korea, we calculated the expected number of sentinel sites in 17 regions, including 8 metropolitan cities and 9 provinces, by the regional population. This is expressed by the following equation:

We then conducted a chi-square test to determine the statistical difference between the expected number of sentinels and the observed number of sentinels in the region. This can be expressed by the following equation:

where O_i is the observed number of sentinels in each region, E_i is the expected number of sentinels in each region, and k is the number of regions. We considered the geographic representativeness of the surveillance performance to be good (score 3) if the P value of the test is above .05 and weak (score 1) if the P value of the test is less than .05. We conducted all analyses using RStudio (version 4.1.0; Posit).

In the timeliness assessment, we found a shorter than 1.9-week lag between the case notification and data dissemination during the study period. This was deemed good (score 3) based on our criteria.

For stability, we could not identify any HFMD surveillance system cessation during the study period (score 3, good performance). However, there was no regular surveillance training program for surveillance staff at the KDCA (score 1, weak performance). Due to data limitations, we only assessed the turnover rate of the KDCA surveillance staff as 260% during 2022 (score 1, weak performance). Therefore, the mean stability was moderate (mean score of 2) from 2017 to 2021 and weak (mean score of 1.6) in 2022 (Table 1).

Regarding data completeness, the complete report rate, including zero case reports, was above 90% (good performance) from 2017 to 2019 and in 2022. It was 88.1% and 88.6% (moderate performance) in 2020 and 2021, respectively (Table 1).

For the sensitivity, the correlation coefficient between the number of monthly HFMD case notifications from the sentinel sites and the monthly number of patients with HFMD from KHIRA was >0.97 (good performance) from 2017 to 2019 and in 2022, while it was 0.70 and 0.77 (moderate performance) in 2020 and 2021, respectively (Table 1 and Multimedia Appendix 5).

We identified that the difference between the proportions of 0‐ to 6-year-olds from the HFMD surveillance system by the KDCA and among the patients with HFMD from the KHIRA ranged from 3% to 6% (score 3, good performance) from 2017 to 2019 and in 2022 and ranged from 12% to 13% (score 2, moderate performance) from 2021 to 2022 (Table 2). In our assessment of the representativeness of the geographic distributions of the sentinel sites, the P values from chi-square tests were between .95 and .99 from 2017 to 2022 (good performance) (Table 3).

Timeliness is an important measure to evaluate in any surveillance system. Case notification is the result of a chain of events from infection to report dissemination by public health authorities [21]. Delays in the processes originate from patient delay (ie, delays in visiting the clinics), doctor delay (ie, time delay from consultation to diagnosis), laboratory delay (ie, time delay from the laboratory test to confirmed test result), and notification delay (ie, time delay from the physician to the appropriate public health authority) [22]. The Korean HFMD surveillance system collects the information regarding the notification date and dissemination of the case report (Multimedia Appendix 4). Therefore, in this study, we only accounted for the time lag from the case notification to results dissemination (ie, time elapsed between the notification from the physician to the dissemination of the surveillance report). As HFMD reporting involves a web-based reporting system in the KDCA, there was a shorter than 2-week time lag between case notification from the sentinel clinics and weekly surveillance report dissemination. A previous study demonstrated that quantifying the reporting delay is necessary to manage outbreaks and communicate with the public [21,22]. Therefore, this finding provides valuable input to develop real-time epidemic models using surveillance data.

As surveillance system timeliness differs by country and disease [23], additional study of HFMD surveillance systems in other countries is necessary to compare results. The Korean HFMD surveillance system depends on sentinel participation, and there is no penalty for delayed reporting. An electronic reporting system is now standardized in many countries, and it improves the timeliness and completeness of infectious disease surveillance [24]. Additional evaluation of timeliness is warranted to identify the specific time lag in each reporting step [23].

We identified that the operational stability of the surveillance system was good. However, due to the replacement of surveillance officers and lack of KDCA training programs, it was evaluated as moderate from 2017 to 2021. The more frequently infectious disease surveillance personnel are replaced, the more likely it is that the surveillance system stability will be affected. It may become difficult to derive improvement measures because the relevant indicators cannot be calculated. Therefore, personnel turnover must be minimized and an adequate staffing level secured to manage surveillance system stability.

KDCA surveillance officers conduct routine public health activities. Many countries, including South Korea, have initiated programs to train field epidemiologists in public health surveillance and outbreak responses [25]. However, we identified no regular training program for surveillance staff in South Korea. Many countries struggle to attract and retain public health personnel in the field of infectious diseases due to the high level of responsibility and the principle of learning by doing [26,27]. Furthermore, limited academic resources, including a population-based training program and a lack of proper teaching methodologies, result in insufficient opportunities for professional development [27]. Therefore, to improve stability, tying surveillance officers to the academic field will support professional development and sustainability.

The notification data quality is important to enable public health authorities to respond in a timely and appropriate manner. Therefore, it is necessary to complete all data fields in the notification system [24,28]. The current Korean HFMD electronic surveillance system is mainly operated by web-based reporting records to track reporting at each sentinel site. The system identifies missing fields on the notification form, and an alarm is sent to the sentinel sites to complete the missing data. In this study, we could not identify any missing data fields in the reporting system. Thus, we considered the completeness of data as good during the study period. However, we identified that the completeness of notification was decreased from 2020 to 2021 and recovered in 2022. This is likely due to surges in the number of COVID-19 notifications.

Sensitivity evaluation using community-based investigation requires additional resources, and it is difficult to conduct regularly. Previous studies demonstrated alternative methods for evaluating sensitivity by comparing other sources of surveillance data, including public health laboratories, hospitals, and the national reimbursement system [14,29,30]. Previous studies report that the sensitivities of HFMD surveillance were from 38% to 57% and 76% in Hong Kong and Thailand, respectively [14,29]. However, the study design, including data sources, population coverage, and sensitivity measuring methods, differs between studies, warranting caution when comparing the results. A surveillance system with low sensitivity can be useful for monitoring disease outbreak trends if sensitivity remains constant over time [10]. In our study, to overcome the limitation of alternative methodologies and identify sensitivity changes over time, we evaluated 6 consecutive years (2017‐2022) of data. We identified lower sensitivity during the 2020‐2021 period compared to other years. This is likely due to the significantly reduced number of HFMD cases due to different public health and social distancing measures in different locations (ie, different measures in metropolitan areas and provincial regions). Additional study is necessary to determine the association between surveillance sensitivity and behavior changes, such as visiting clinics to treat HFMD. As HFMD is normally not a life-threatening disease, patient visits to clinics may be affected by the overall number of COVID-19 cases in the country.

The nationwide reimbursement data have been widely used to identify the incidence of diseases in South Korea [5,16,31]. To determine age-specific representativeness of the surveillance system, we compared the age-specific proportion of the incidence data between the KDCA and KHIRA. From the national health insurance reimbursement data, we identified that the proportion of 0‐ to 9-year-olds accounted for more than 91% to 98% of HFMD incidence in South Korea during the study period. Children aged 0‐6 years old accounted for 89% to 92% of the HFMD incidence from 2017 to 2019 and in 2022 and 79% to 80% from 2020 to 2021. The lower HFMD incidence in the 0‐6 years age group from 2020 to 2021 compared to other years is likely due to the strict public health measures in place during the COVID-19 pandemic from 2020 to 2021, including closing kindergartens and wearing face masks. This result is in line with a previous Korean study that determined that public health and social measures are strongly associated with reduced HFMD transmissibility, particularly in kindergarteners [5].

It is important to select sentinel locations to effectively meet surveillance objectives and save public health resources. We identified that the geological distribution of HFMD sentinels in South Korea was good during the study period. However, having more sentinels in underrepresented areas will enhance the local resolution of HFMD occurrence. Even if the current sentinel sites are based on population, other factors, including location, distance between sites, and population density, should be considered when designating new sentinel sites [32].

There are several limitations to this study. First, we could not evaluate other surveillance performance attributes, including simplicity, flexibility, and acceptability. An additional survey using a questionnaire is warranted to evaluate these other attributes. Second, we could not evaluate the positive predictive value. As HFMD surveillance is based on the reporting of clinical symptoms, further study is necessary to identify its positive predictive value (ie, the proportion of cases with confirmed pathogens out of the number of reported cases).

In conclusion, the completeness, sensitivity, and age representativeness of the HFMD surveillance system were temporarily reduced during the acute period of the COVID-19 pandemic. Continuous evaluation of surveillance performance will provide robust and reliable data to support public health policy development.