Performance of the Severe Acute Respiratory Illness Sentinel Surveillance System in Yemen: Mixed Methods Evaluation Study

Background: The national severe acute respiratory illness (SARI) surveillance system in Yemen was established in 2010 to monitor SARI occurrence in humans and provide a foundation for detecting SARI outbreaks. Objective: To ensure that the objectives of national surveillance are being met, this study aimed to examine the level of usefulness and the performance of the SARI surveillance system in Yemen. Methods: The updated Centers for Disease Control and Prevention guidelines were used for the purposes of our evaluation. Related documents and reports were reviewed. Data were collected from 4 central-level managers and stakeholders and from 10 focal points at 4 sentinel sites by using a semistructured questionnaire. For each attribute, percent scores were calculated and ranked as follows: very poor (≤20%), poor (20%-40%), average (40%-60%), good (60%-80%), and excellent (>80%). Results: As rated by the evaluators, the SARI surveillance system achieved its objectives. The system’s flexibility (percent score: 86%) and acceptability (percent score: 82%) were rated as “excellent,” and simplicity (percent score: 74%) and stability (percent score: 75%) were rated as “good.” The percent score for timeliness was 23% in 2018, which indicated poor timeliness. The overall data quality percent score of the SARI system was 98.5%. Despite its many strengths, the SARI system has some weaknesses. For example, it depends on irregular external financial support. Conclusions: The SARI surveillance system was useful in estimating morbidity and mortality, monitoring the trends of the disease, and promoting research for informing prevention and control measures. The overall performance of the SARI surveillance system was good. We recommend expanding the system by promoting private health facilities’ (eg, private hospitals and private health centers) engagement in SARI surveillance, establishing an electronic database at central and peripheral sites, and providing the National Central Public Health Laboratory with the reagents needed for disease confirmation. (JMIR Public Health Surveill 2021;7(7):e27621) doi: 10.2196/27621


Introduction
Worldwide, acute lower respiratory infection is the second commonest cause of morbidity and the third commonest cause of mortality in all age groups [1]. A significant proportion of the global burden of acute lower respiratory infection, especially in children and older adults, is attributable to influenza and respiratory syncytial viruses. to 5 million cases of severe illness and about 250,000 to 500,000 deaths. In early 2019, the Global Burden of Disease study estimated that 99,000 to 200,000 annual deaths resulting from lower respiratory tract infections are directly attributable to influenza [1,2].
Estimates are rare in many countries, including countries in the Eastern Mediterranean Region. The influenza A (H1N1) pandemic highlighted the necessity of reliable estimates for the disease burden of severe acute respiratory illness (SARI) and influenza-associated SARI (F-SARI) in all countries and regions of the world [3].
Many countries have established sentinel sites for influenza epidemiological surveillance. The data captured from sentinel sites have been used by WHO member states to estimate disease burden at the national level and to compare data between countries.
Due to the fact that many emerging and reemerging diseases classified under the International Health The Surveillance Department of the Ministry of Public Health proposed the inclusion of influenza in the Joint Program Review and Planning Mission [6]. The Central Public Health Laboratory was recognized as a national influenza center where facilities for carrying out polymerase chain reaction and serology tests are available and functional [6,7].
The WHO recommends that countries should perform surveillance for SARI and F-SARI and that surveillance systems should undergo periodic comprehensive evaluations. To ensure that the objectives of national surveillance are being met, this study aimed to examine the level of usefulness and the performance of the SARI surveillance system.

Study Design
A descriptive study was conducted to evaluate the SARI surveillance system from October to December 2018 based on the updated Centers for Disease Control and Prevention guidelines for the evaluation of a public health surveillance system [8]. Mixed methods with quantitative and qualitative components were used for the evaluation. SARI sentinel sites at four public hospitals-Al Joumhouri, Al Wahda, Al Swaide, and Al Thawra-in four governorates (Sana'a city, Aden, Taiz, and Al Hodeida) were studied. All possible stakeholders, including National Influenza Control Program managers, data entry staff, Ministry of Public Health and Population staff, and members of focal points in sentinel sites, were enrolled in this study.

Data Collection
A desk review of all documents, guidelines, strategies, and pertinent scientific literature on influenza programs was conducted. Data were collected via in-depth interviews and semistructured questionnaires (Multimedia Appendix 1) with stakeholders and members of focal points at the sentinel sites, respectively. In addition, a review of the SARI system database was conducted.

System Attributes
A total of 9 surveillance system attributes that can affect usefulness were assessed. Quantitative analysis was used to assess data quality, timeliness, sensitivity, and positive predictive values. Qualitative analysis was used to assess representativeness, simplicity, flexibility, acceptability, and stability.

Analysis Methods
To determine the level of usefulness, the system was considered useful if it met at least one of its objectives and one of its planned uses. With regard to qualitative attributes, stakeholders were asked to rate the degree to which they agreed with attributes' specific indicators by using a 5-point Likert scale (1=strongly disagree; 2=disagree; 3=neutral; 4=agree; 5=strongly agree). Higher scores indicated better performance in terms of the studied attribute. The scores of all indicators for each attribute were summed and divided by the maximum scores to produce a percent score. The percent score was used to rank each attribute. The final rank of each attribute was classified as follows: excellent (attribute score: >80%), good (attribute score: 60%-80%), average (attribute score: 40%-60%), poor (attribute score: 20%-40%), and very poor (attribute score: ≤20%).
With regard to quantitative attributes, data quality was assessed by measuring the completeness of patient interview forms, form transmission data, and respiratory specimen collection and testing data and by checking signs and symptoms records and primary diagnoses to determine whether surveillance case definitions had been adhered to properly. The data collected by the system were compared against the minimum data collection standards for SARI surveillance. Timeliness was assessed by calculating the percentage of specimens that were collected and sent to the laboratory within 72 hours.

The Main Purpose and Objectives of the SARI System
The National Influenza Sentinel Surveillance system was established in 2010. The main purpose and objectives of the SARI system are to monitor influenza occurrence in humans and to provide a foundation for detecting outbreaks of novel strains of influenza.

SARI Case Definition
A case of SARI was defined as a person meeting the case definition of influenza-like illness (ie, the sudden onset of a fever of >38 °C and at least 1 of the following respiratory symptoms: dry cough, sore throat in the absence of another diagnosis, and shortness of breath or difficulty in breathing requiring hospital admission).

Sources of Data for the SARI System
Surveillance for influenza-like illness and SARI was carried out in 4 sentinel sites. Aggregated data were collected from and reported by each sentinel site. The data included the following:

Data Flow and Feedback in the SARI Surveillance System
The system was designed so that each sentinel site could send its reports to central sites within 1 week. Further, samples sent to National Central Public Health laboratories within 72 hours are sent to US Naval Medical Research Unit Number 3 for confirmation, as shown in Figure 1. At the central sites, data are reviewed, organized, and analyzed, as required. Feedback is then sent to sentinel sites.

Demographic Characteristics of the Participants
A total of 3 persons at the central sites and 10 persons at the peripheral sites (3 pediatricians, 3 nurses, 2 laboratorian specialists, 3 health system directors, and 2 medical specialists) evaluated the system. The mean age of the participants was 44.5 years.

Usefulness
Of the 4 SARI stakeholders at the central sites, 3 (75%) agreed that the SARI surveillance system met its objectives. The usefulness percent score was 75%, indicating that the usefulness of the SARI system was good.

Flexibility
All 13 respondents agreed that the system could easily adapt to changes in the SARI case definition, include other diseases, and accommodate any changes in data with less effort and minimal costs. The overall percent score for flexibility was 86%, which indicated excellent performance (Table 1). At sentinel sites, the percent score, as determined by 10 stakeholders, was 71%. This indicated that the system had good performance at the peripheral sites. The system adapted to accommodate new, additional information (eg, variation in resources) a The total score, percent score, and rank of the system were 120, 86, and excellent, respectively.

Stability
All 4 stakeholders at the central sites agreed that the system could adapt to changes in resources, but half of the stakeholders (2/4, 50%) stated that the system mainly depends on external funds. With regard to the scoring system, the overall stability percent score was 75% ( Table 2). This indicated that the stability of the SARI system was good. The system can adapt to funding withdrawal a The total score, percent score, and rank of the system were 6, 75, and good, respectively.

Simplicity
In total, 2 indicators of simplicity were ranked as excellent, 3 were ranked as good, and 1 was ranked as average. The simplicity percent score of the system was 74% (Table 3), indicating that the SARI system's simplicity attribute was good. The trainees had training a The total score, percent score, and rank of the system were 186, 74, and good, respectively.

Acceptability
The two indicators of acceptability (ie, the willingness to participate among people within the system and satisfaction with the SARI surveillance system) were ranked as excellent. The acceptability percent score of the SARI system was 82% (ie, the SARI system had excellent performance; Table 4).

Data Quality
Data quality was evaluated by assessing the percentage of complete forms and the missing variable data in the forms. All of the patients included in the central sites' database (N=245) had complete forms (245 forms; 100% completeness). With regard to missing data, 22 case report forms from 2018 were selected randomly and reviewed. No missing variables were found, and the variables in these forms were in line with those of the database (completeness: 100%; accuracy: 97%). The overall data quality percent score was 98.5% (excellent).

Timeliness
The percentage of collected specimens at the health facilities that sent samples to the laboratory within 72 hours was used as an indicator of timeliness. Of the 182 collected samples, 42 (23.1%) samples were sent to the laboratory within 72 hours. The percent score for timeliness was 23% in 2018, indicating that the system's timeliness was poor.

Overall Performance of the SARI Surveillance System
The overall performance of the SARI surveillance system had a percent score of 79% (ie, the SARI system had good performance; Table 5).

Strengths and Weaknesses of the SARI Surveillance System
Despite its many strengths, the SARI system has some weaknesses. For example, it depends on irregular external financial support. To act as a platform for the surveillance of other respiratory illnesses, the SARI surveillance system integrates an influenza-like illness surveillance system with the electronic Disease Early Warning System. This has several benefits. First, it allows for efficient laboratory data collection and transportation. Second, the SARI system uses resources more efficiently than other systems. These widespread benefits enhance the usefulness of the system and allow the system to meet its own surveillance objectives and address broader national priorities.

Principal Findings
The surveillance systems at sentinel sites are tools for the early detection of disease, the monitoring of trends in the burden of diseases, and the generation of recommendations for the prevention and control of diseases. The evaluation of surveillance systems helps decision makers to set priorities for future planning, resource allocation, and future interventions for preventing the spread of diseases.
Overall, this study showed that the performance of the SARI surveillance system was good. Studies from Zambia [9] and the Democratic Republic of Congo [10] have reported similar findings. The SARI system was found to be useful in detecting trends and signal changes in the occurrence of SARI, estimating the magnitude of morbidity and mortality related to SARI, and promoting research for informing prevention and control measures for SARI. Similarly, a previous evaluation in Zambia demonstrated the usefulness of the system [9]. The SARI surveillance system was shown to be simple. The case definition and case report forms were available and easy to use. Similarly, the simplicity of the SARI surveillance system was documented in previous evaluations conducted in the Democratic Republic of Congo [10] and Zambia [10]. The SARI surveillance system's flexibility was excellent. The system appeared to be able to adapt easily to changes in the SARI case definition and accommodate changes in data with less effort and minimal costs. This finding is consistent with the findings of a study from South Africa [11]. However, it is not consistent with the findings of studies from Zambia [9] and the Democratic Republic of Congo [10], which reported that the flexibility of the evaluated systems ranged from moderate to good. The acceptability of the SARI system was excellent, as reflected by the willingness of stakeholders to participate in the system and their satisfaction with the SARI surveillance system.
The stability of the SARI surveillance system was good. It was found that the system was stable and could adapt to changes in resources (eg, donors withdrawing their support). This finding is in line with the findings of previous studies from South Africa [11], the Democratic Republic of Congo [10], and Zambia [9]. However, it is not in line with the findings of a study from Pakistan [12], which reported average stability.
Our findings showed that the timeliness of the SARI system was very poor. This might be due to the lack of laboratory components that are essential for sampling. Previous evaluations in South Africa [11], China [13], Zambia [9], and the Democratic Republic of Congo [10] reported moderate to good timeliness. The quality and completeness of SARI surveillance system data were excellent.

Limitations
We could not calculate positive predictive values and assess sensitivity because the samples have not been tested since 2016 due to a lack of reagents.

Conclusion
Overall, the SARI surveillance system was useful in estimating morbidity and mortality, monitoring the trends of the disease, and promoting research for informing prevention and control measures. The overall performance of the SARI surveillance system was good. We recommended expanding the system by promoting private health facilities' (eg, private hospitals and private health centers) engagement in SARI surveillance, establishing an electronic database at central and peripheral sites, and providing the National Central Public Health Laboratory with the reagents needed for disease confirmation.