Evaluation of the National Tuberculosis Surveillance System in Sana’a, Yemen, 2018: Observational Study

Introduction

Tuberculosis is an infectious disease caused by mycobacterium tuberculosis bacteria [1,2]. It remains a major cause of ill health, one of the top 10 causes of death worldwide, which has taken a higher ranking than HIV/AIDS [3]. According to World Health Organization (WHO) estimates, nearly 10 million people were infected with tuberculosis in 2019 [3]: 5.6 million men, 3.2 million women, and 1.2 million children. There were 1.4 million that died with tuberculosis among HIV-negative people, while 208,000 deaths were among HIV-positive people.

Tuberculosis can affect anyone anywhere, but almost 90% of those who fall sick with tuberculosis have been living in one of the 30 high tuberculosis burden countries. Eight countries account for two-thirds of the total, with India leading the count, followed by Indonesia, China, the Philippines, Pakistan, Nigeria, Bangladesh, and South Africa. The WHO Eastern Mediterranean Region contributes 819,000 (8.5%) of the people who developed the disease, with an estimated incidence rate of 114 per 100,000 populations [4].

Yemen has a moderate tuberculosis burden. According to WHO estimates, 14,000 patients with tuberculosis were reported in 2019, and the incidence rate was estimated as 48 per 100,000 population in 2019 [5]. The surveillance system must be evaluated periodically and produce recommendations for improving performance and usefulness. Therefore, this evaluation aims to assess the usefulness and performance of the tuberculosis surveillance system attributes and to identify the strengths and weaknesses of the system.

Methods

Study Design

A quantitative and qualitative evaluation was conducted to assess the performance of the tuberculosis surveillance system using the Centers for Disease Control and Prevention’s (CDC) updated guidelines for Evaluating Public Health Surveillance Systems [6].

Study Setting and Duration

The study was carried out in 28 tuberculosis sentinel sites for the whole population in their assigned catchment areas in Sana’a City during October 1 to December 31, 2018.

Study Population

All participants have interviewed based on their involvement with key aspects of tuberculosis surveillance activities: seven managers and one data entry personnel at a central level and one tuberculosis coordinator and one lab supervisor at the governorate level, and at the peripheral level, 10 tuberculosis coordinators from the district level and 11 medical officers and 24 lab technicians from the health facilities.

Data Collection and Analysis

The National Tuberculosis Control Program (NTCP) documents (strategic plan, guidelines, annual reports, and databases) were reviewed to describe the system. In-depth interviews were used with participants based on their involvement with key aspects of tuberculosis surveillance activities. Verbal consent was obtained from all respondents who participated in the study. Semistructured questionnaires were used to collect information related to surveillance system attributes including flexibility, stability, simplicity, and acceptability at the four levels. The usefulness level was assessed using questions with (yes 1 or no 0) answers, while the other surveillance attributes were assessed using a three-point Likert scale (3 agree, 2 neutral, 1 disagree).

Scoring System

Specific indicators were used to assess each performance attributes.

The score percent was calculated by the following:

The overall score percent was calculated by the following:

The score percent of each attribute was interpreted as the following: greater than 80% was ranked as good, between 60% to 80% was ranked as average, and lesser than 60% was ranked as poor [7,8]. Data quality was assessed by reviewing documents such as quarter report, while the sensitivity and positive predictive value (PPV) were calculated using the following equation:

Sensitivity = (true positive A / true positive A + false negative C) (3)

Predictive value = (true positive A / true positive A + false negative B) (4)

The date was analyzed using Excel 2013 (Microsoft Corporation) and Epi Info version 7.2 (CDC) to calculate frequency and percentage.

Results

Desk Review Findings

Description of the Tuberculosis Surveillance System

The NTCP was established in 1970 in the primary health care sector at the Ministry of Public Health and Population (MoPHP). The Tuberculosis Control Program is a tool for tuberculosis control strategy implementation within a national health system. As such, the NTCP, a vehicle for the directly observed treatment short course (DOTS) strategy since 1995, was used to reach the global targets for detection of at least 70% of cases and treatment success of at least 85%. The NTCP expanded the DOTS strategy gradually to cover all the existing 333 districts until the DOTS coverage in the population reached 100% by the end of December 2007. The MoPHP was established and organized as a central unit of the NTCP in the framework of a national health program in 1995 [9].

As of 2006, the NTCP has adopted the WHO Stop Tuberculosis Strategy and has initiated the development of components. In late 2015, the NTCP adopted the WHO End Tuberculosis Strategy as a national policy to prevent, manage, and control tuberculosis [10]. There are four tuberculosis centers located in major governorates (Sana’a City, Aden, Taiz, and Al Hudaydah). They provided outpatient clinical, radiological, and bacteriological culture services. The laboratory network of tuberculosis control consisted of 271 microscopy laboratories [11].

Data Flow of the Tuberculosis Surveillance System

The following diagram of data flow (Figure 1) describes four levels of responsibility to ensure tuberculosis prevention, care, and control services from peripheral basic management units to the center of the NTCP within the tuberculosis system. Each of these levels has well-defined tasks. The NTCP has 13 forms, registers, and reports. Those were designed for paper-based recording and reporting systems that are used by the surveillance department for management of data at all districts. The tuberculosis control program activities were highly dependent on international donors, the WHO, Japan International Cooperation Agency, and the Global Fund to Fight AIDS.

‎View this figure

Participants’ Characteristics

A total of 54 participants were involved, 57% (n=31) were males and 43% (n=23) females. Almost half of the participants 50% (n=27) were lab technicians.

Findings From Qualitative Data

Usefulness

Table 1 shows that the overall score percent for usefulness was 71% (57/80), indicating an average rank. Only 40% (4/10) of participants mentioned that the tuberculosis surveillance data provided estimates of the tuberculosis magnitude, incidence, prevalence, and mortality, and helped in resource planning, prevention, care, and control.

Some participants mentioned:

The lack of reporting from the private sector leads to the presence of a gap between the estimated number of new cases and the number of actually detected cases.

The Tuberculosis Surveillance System Attributes

Flexibility

The overall score of flexibility was 75% (30/40) that reveals an average performance. Two out of four indicators of the flexibility attribute, “The system can accommodate any changes in reporting method” and “The system can integrate the surveillance of another disease,” had good rank. The indicator “The system can accommodate data changes with minimum cost and efforts” was poorly ranked (Table 2).

Stability

The overall stability of the system scored 15% (3/20), which indicates a poor performance. Both indicators of stability (“The system is stable after sponsors withdraw their support” and “The system does not require time to manage the data”) had a poor rank (Table 3).

Simplicity

Regarding simplicity, out of 11 indicators, 6 indicators had a poor ranking. Two of the indicators had a good rank. The overall simplicity of the system scored 59% (863/1452), which indicates a poor performance (Table 4).

Acceptability

Regarding the acceptability attribute, the statement related to willingness to participate in the tuberculosis surveillance system had an average rank of 88% (116/132). However, the satisfaction with the tuberculosis surveillance system had a poor rank of 44% (58/132). The overall acceptability score was 66% (174/264), which indicates average performance (Table 5).

Findings From Quantitative Data

Data Quality

Regarding data quality, we have reviewed four tuberculosis reports; each report includes six format types. These formats were compared with the database. Four out of six forms were complete and accurate. However, the completeness and accuracy of the other two forms were zero. The overall data quality score was 67% (4/6), which ranked as average (Table 6).

Sensitivity

The sensitivity of the tuberculosis surveillance system was 96%, which ranked as good. The sensitivity calculated by using the formula of (A / A + C) was as follows: sensitivity = 78 / 81 × 100 = 96% (Table 7).

Positive Predicative Value

The PPV of the tuberculosis surveillance system was 73%, which ranked as average. It was calculated by using the formula of (A / A + B), that is, the following: PPV = 78 / 107 × 100 = 73% (Table 7).

Strengths and Weaknesses of the Tuberculosis Surveillance System

Although the tuberculosis surveillance system had strength points that included the presence of an infrastructure system, qualified human resources, availability of antituberculosis drugs, and the presence of the coordination for tuberculosis activities at all levels, it had many weak points such as depending on external donors and a bureaucrat structure (a large number of hard copies) that lead to delay in its implemented activities in addition to a lack the motivation of human resources, lack of refreshment training for medical officers, poor coordination with other health sectors (private), and high turnover among medical officers at a peripheral level due to political crises and security situations.

Discussion

Principal Findings

Evaluation of any surveillance system is the cornerstone for its improvement and ensuring proper morbidity and mortality indicators. It is important to identify the weaknesses and strengths of the system and provide decision makers with evidence-based data to decide on its continuity.

Our study findings showed that the tuberculosis surveillance system was useful. It helps in monitoring trends of tuberculosis and estimates the need for antituberculosis drugs and laboratory reagents at all levels. These findings are similar to other studies in Afghanistan, Yemen, and Pakistan [7,8,12].

The flexibility of the tuberculosis surveillance system was ranked as average. It has been integrated with HIV and AIDS. The NTCP has a counseling clinic where each tuberculosis case was referred to undergo voluntary counseling and testing. Our findings are consistent with the results found in previous evaluations carried out in Pakistan and Zimbabwe [12,13].

Regarding the stability of the tuberculosis surveillance system, it was poor because of its total dependency on the donors in addition to the government’s support for first-line antitubercuolosis drugs and the salary of the staff being completely suspended since 2014, as consequences of political and security crises. In addition, prolonged procedures to release funds from the Global Fund to implement tuberculosis control activities may lead to abandoning the activity implementation. Although, this finding is similar to what has been reported in Yemen and Zimbabwe [8,13]. Other studies showed stable tuberculosis surveillance systems [7,14].

The simplicity of the system was poorly rated for several reasons including data collection being time-consuming, the staff not receiving adequate training on tuberculosis surveillance, training courses not performed frequently, and report forms not easy to fill. Similar findings have been reported in Afghanistan and Zimbabwe [7,13]. However, this finding is not consistent with the findings in previous studies in Yemen, Pakistan, and South Africa [8,12,14].

This evaluation shows that the acceptability was average, as the focal points are willing to participate in the dengue surveillance system; however, they were poorly satisfied. This finding agrees with the findings of others in Afghanistan and Yemen [7,8] but disagrees with other studies in Harare City [15].

The data quality of the tuberculosis system was average. These findings were consistent with findings of other studies conducted in Harare City [15] but inconsistent with another study in the Republic of South Africa [14,16].

The sensitivity and the PPV of the system were good and average, respectively. These results are aline with a previous study in Afghanistan and Eden District [7,14]. However, they disagree with a study in Pakistan [12].

Limitations

This evaluation was carried out in Sana’a City and targeted only the health facilities providing tuberculosis services (purposive sample) due to time and funds constraints.

Conclusions

The tuberculosis surveillance system was found useful. Flexibility, PVP, and data quality were average. Stability, acceptability, and simplicity were poor. The sensitivity was good.

The main weaknesses in the tuberculosis surveillance system included a lack of governmental finances, a paper-based system, and a lack of regular staff training. Developing an electronic system, securing governmental financial support, and training the staff on tuberculosis surveillance are strongly recommended to improve the system performance.