Background: Tuberculosis remains a public problem that is considered one of the top causes of morbidity and mortality worldwide. The National Tuberculosis Control Program in Yemen was established in 1970 and included in the national health policy under the leadership of the Ministry of Public Health and Population to monitor tuberculosis control. The surveillance system must be evaluated periodically to produce recommendations for improving performance and usefulness.
Objective: This study aims to assess the usefulness and the performance of the tuberculosis surveillance system attributes and to identify the strengths and weaknesses of the system.
Methods: A quantitative and qualitative evaluation of the national tuberculosis surveillance system was conducted using the Centers for Disease Control and Prevention’s updated guidelines. The study was carried out in 10 districts in Sana’a City. A total of 28 public health facilities providing tuberculosis services for the whole population in their assigned catchment areas were purposively selected. All participants were interviewed based on their involvement with key aspects of tuberculosis surveillance activities.
Results: The tuberculosis surveillance system was found to have an average performance in usefulness (57/80, 71%), flexibility (30/40, 75%), acceptability (174/264, 66%), data quality (4/6, 67%), and positive predictive value (78/107, 73%), and poor performance in simplicity (863/1452, 59%) and stability (15%, 3/20). In addition, the system also had a good performance in sensitivity (78/81, 96%).
Conclusions: The tuberculosis surveillance system was found to be useful. The flexibility, positive predictive value, and data quality were average. Stability and simplicity were poor. The sensitivity was good. The main weaknesses in the tuberculosis surveillance system include a lack of governmental financial support, a paper-based system, and a lack of regular staff training. Developing an electronic system, securing governmental finances, and training the staff on tuberculosis surveillance are strongly recommended to improve the system performance.
Tuberculosis is an infectious disease caused by mycobacterium tuberculosis bacteria [, ]. It remains a major cause of ill health, one of the top 10 causes of death worldwide, which has taken a higher ranking than HIV/AIDS [ ]. According to World Health Organization (WHO) estimates, nearly 10 million people were infected with tuberculosis in 2019 [ ]: 5.6 million men, 3.2 million women, and 1.2 million children. There were 1.4 million that died with tuberculosis among HIV-negative people, while 208,000 deaths were among HIV-positive people.
Tuberculosis can affect anyone anywhere, but almost 90% of those who fall sick with tuberculosis have been living in one of the 30 high tuberculosis burden countries. Eight countries account for two-thirds of the total, with India leading the count, followed by Indonesia, China, the Philippines, Pakistan, Nigeria, Bangladesh, and South Africa. The WHO Eastern Mediterranean Region contributes 819,000 (8.5%) of the people who developed the disease, with an estimated incidence rate of 114 per 100,000 populations .
Yemen has a moderate tuberculosis burden. According to WHO estimates, 14,000 patients with tuberculosis were reported in 2019, and the incidence rate was estimated as 48 per 100,000 population in 2019 . The surveillance system must be evaluated periodically and produce recommendations for improving performance and usefulness. Therefore, this evaluation aims to assess the usefulness and performance of the tuberculosis surveillance system attributes and to identify the strengths and weaknesses of the system.
A quantitative and qualitative evaluation was conducted to assess the performance of the tuberculosis surveillance system using the Centers for Disease Control and Prevention’s (CDC) updated guidelines for Evaluating Public Health Surveillance Systems .
Study Setting and Duration
The study was carried out in 28 tuberculosis sentinel sites for the whole population in their assigned catchment areas in Sana’a City during October 1 to December 31, 2018.
All participants have interviewed based on their involvement with key aspects of tuberculosis surveillance activities: seven managers and one data entry personnel at a central level and one tuberculosis coordinator and one lab supervisor at the governorate level, and at the peripheral level, 10 tuberculosis coordinators from the district level and 11 medical officers and 24 lab technicians from the health facilities.
Data Collection and Analysis
The National Tuberculosis Control Program (NTCP) documents (strategic plan, guidelines, annual reports, and databases) were reviewed to describe the system. In-depth interviews were used with participants based on their involvement with key aspects of tuberculosis surveillance activities. Verbal consent was obtained from all respondents who participated in the study. Semistructured questionnaires were used to collect information related to surveillance system attributes including flexibility, stability, simplicity, and acceptability at the four levels. The usefulness level was assessed using questions with (yes 1 or no 0) answers, while the other surveillance attributes were assessed using a three-point Likert scale (3 agree, 2 neutral, 1 disagree).
Specific indicators were used to assess each performance attributes.
The score percent was calculated by the following:
The overall score percent was calculated by the following:
The score percent of each attribute was interpreted as the following: greater than 80% was ranked as good, between 60% to 80% was ranked as average, and lesser than 60% was ranked as poor [, ]. Data quality was assessed by reviewing documents such as quarter report, while the sensitivity and positive predictive value (PPV) were calculated using the following equation:
Sensitivity = (true positive A / true positive A + false negative C) (3)
Predictive value = (true positive A / true positive A + false negative B) (4)
The date was analyzed using Excel 2013 (Microsoft Corporation) and Epi Info version 7.2 (CDC) to calculate frequency and percentage.
Desk Review Findings
Description of the Tuberculosis Surveillance System
The NTCP was established in 1970 in the primary health care sector at the Ministry of Public Health and Population (MoPHP). The Tuberculosis Control Program is a tool for tuberculosis control strategy implementation within a national health system. As such, the NTCP, a vehicle for the directly observed treatment short course (DOTS) strategy since 1995, was used to reach the global targets for detection of at least 70% of cases and treatment success of at least 85%. The NTCP expanded the DOTS strategy gradually to cover all the existing 333 districts until the DOTS coverage in the population reached 100% by the end of December 2007. The MoPHP was established and organized as a central unit of the NTCP in the framework of a national health program in 1995 .
As of 2006, the NTCP has adopted the WHO Stop Tuberculosis Strategy and has initiated the development of components. In late 2015, the NTCP adopted the WHO End Tuberculosis Strategy as a national policy to prevent, manage, and control tuberculosis . There are four tuberculosis centers located in major governorates (Sana’a City, Aden, Taiz, and Al Hudaydah). They provided outpatient clinical, radiological, and bacteriological culture services. The laboratory network of tuberculosis control consisted of 271 microscopy laboratories [ ].
Data Flow of the Tuberculosis Surveillance System
The following diagram of data flow () describes four levels of responsibility to ensure tuberculosis prevention, care, and control services from peripheral basic management units to the center of the NTCP within the tuberculosis system. Each of these levels has well-defined tasks. The NTCP has 13 forms, registers, and reports. Those were designed for paper-based recording and reporting systems that are used by the surveillance department for management of data at all districts. The tuberculosis control program activities were highly dependent on international donors, the WHO, Japan International Cooperation Agency, and the Global Fund to Fight AIDS.
A total of 54 participants were involved, 57% (n=31) were males and 43% (n=23) females. Almost half of the participants 50% (n=27) were lab technicians.
Findings From Qualitative Data
shows that the overall score percent for usefulness was 71% (57/80), indicating an average rank. Only 40% (4/10) of participants mentioned that the tuberculosis surveillance data provided estimates of the tuberculosis magnitude, incidence, prevalence, and mortality, and helped in resource planning, prevention, care, and control.
Some participants mentioned:
The lack of reporting from the private sector leads to the presence of a gap between the estimated number of new cases and the number of actually detected cases.
|The system data provide estimates of the TBa magnitude, incidence, prevalence, and mortality||4||40||Poor|
|The system data were used to monitor trends of TB over time||10||100||Good|
|The system data can identify areas with anti-TB drugs failure||2||20||Poor|
|The system data were used for procurement of anti-TB drugs and laboratory reagents when needed||10||100||Good|
|The system data can be used to recognize the high-risk groups||10||100||Good|
|The system data helped in resource planning, prevention, care, and control||4||40||Poor|
|The system data were used to update and develop the national policy strategy for TB control||7||70||Average|
|The system data were used to assess the impact of interventions||10||100||Good|
The Tuberculosis Surveillance System Attributes
The overall score of flexibility was 75% (30/40) that reveals an average performance. Two out of four indicators of the flexibility attribute, “The system can accommodate any changes in reporting method” and “The system can integrate the surveillance of another disease,” had good rank. The indicator “The system can accommodate data changes with minimum cost and efforts” was poorly ranked ().
|The system can accommodate with changes in case definition||7||70||Average|
|The system can accommodate any changes in reporting method||10||100||Good|
|The system can integrate the surveillance of other disease||10||100||Good|
|The system can accommodate data changes with minimum cost and efforts||3||30||Poor|
The overall stability of the system scored 15% (3/20), which indicates a poor performance. Both indicators of stability (“The system is stable after sponsors withdraw their support” and “The system does not require time to manage the data”) had a poor rank ().
|The system is stable after sponsors withdraw their support||0||0||Poor|
|The system does not require time to manage the data||3||30||Poor|
Regarding simplicity, out of 11 indicators, 6 indicators had a poor ranking. Two of the indicators had a good rank. The overall simplicity of the system scored 59% (863/1452), which indicates a poor performance ().
|The system has standard case definitions for TBa||121||92||Good|
|The case definition for TB is easy to use||111||84||Good|
|Report forms are available||97||73||Average|
|Report forms are easy to fill||70||53||Poor|
|Data collection is not time consuming||61||46||Poor|
|Transmitting data to the central level is easy||59||45||Poor|
|Follow-up of cases is easy||48||36||Poor|
|Anti-TB drugs and laboratory reagents are available in a health facility to confirm diagnosis||99||75||Average|
|Staff received training for TB surveillance||64||48||Poor|
|Training courses are performed frequently||47||36||Poor|
|The system is responsive to suggestions||86||65||Average|
Regarding the acceptability attribute, the statement related to willingness to participate in the tuberculosis surveillance system had an average rank of 88% (116/132). However, the satisfaction with the tuberculosis surveillance system had a poor rank of 44% (58/132). The overall acceptability score was 66% (174/264), which indicates average performance ().
|I am willing to participate in the system||116||88||Average|
|I am satisfied with the TBa surveillance system||58||44||Poor|
Findings From Quantitative Data
Regarding data quality, we have reviewed four tuberculosis reports; each report includes six format types. These formats were compared with the database. Four out of six forms were complete and accurate. However, the completeness and accuracy of the other two forms were zero. The overall data quality score was 67% (4/6), which ranked as average ().
|Forms||Completeness (%)||Accuracy (%)|
|TBa case finding||100||100|
|Sputum smear microscopy conversion||100||100|
|TB treatment outcomes||100||100|
The sensitivity of the tuberculosis surveillance system was 96%, which ranked as good. The sensitivity calculated by using the formula of (A / A + C) was as follows: sensitivity = 78 / 81 × 100 = 96% ().
|Positive||78 (true positive A)||29 (false positive B)||107|
|Negative||3 (false negative C)||9 (true negative D)||12|
aDSM: direct smear microscopy.
Positive Predicative Value
The PPV of the tuberculosis surveillance system was 73%, which ranked as average. It was calculated by using the formula of (A / A + B), that is, the following: PPV = 78 / 107 × 100 = 73% ().
Strengths and Weaknesses of the Tuberculosis Surveillance System
Although the tuberculosis surveillance system had strength points that included the presence of an infrastructure system, qualified human resources, availability of antituberculosis drugs, and the presence of the coordination for tuberculosis activities at all levels, it had many weak points such as depending on external donors and a bureaucrat structure (a large number of hard copies) that lead to delay in its implemented activities in addition to a lack the motivation of human resources, lack of refreshment training for medical officers, poor coordination with other health sectors (private), and high turnover among medical officers at a peripheral level due to political crises and security situations.
Evaluation of any surveillance system is the cornerstone for its improvement and ensuring proper morbidity and mortality indicators. It is important to identify the weaknesses and strengths of the system and provide decision makers with evidence-based data to decide on its continuity.
Our study findings showed that the tuberculosis surveillance system was useful. It helps in monitoring trends of tuberculosis and estimates the need for antituberculosis drugs and laboratory reagents at all levels. These findings are similar to other studies in Afghanistan, Yemen, and Pakistan [, , ].
The flexibility of the tuberculosis surveillance system was ranked as average. It has been integrated with HIV and AIDS. The NTCP has a counseling clinic where each tuberculosis case was referred to undergo voluntary counseling and testing. Our findings are consistent with the results found in previous evaluations carried out in Pakistan and Zimbabwe [, ].
Regarding the stability of the tuberculosis surveillance system, it was poor because of its total dependency on the donors in addition to the government’s support for first-line antitubercuolosis drugs and the salary of the staff being completely suspended since 2014, as consequences of political and security crises. In addition, prolonged procedures to release funds from the Global Fund to implement tuberculosis control activities may lead to abandoning the activity implementation. Although, this finding is similar to what has been reported in Yemen and Zimbabwe [, ]. Other studies showed stable tuberculosis surveillance systems [ , ].
The simplicity of the system was poorly rated for several reasons including data collection being time-consuming, the staff not receiving adequate training on tuberculosis surveillance, training courses not performed frequently, and report forms not easy to fill. Similar findings have been reported in Afghanistan and Zimbabwe [, ]. However, this finding is not consistent with the findings in previous studies in Yemen, Pakistan, and South Africa [ , , ].
This evaluation shows that the acceptability was average, as the focal points are willing to participate in the dengue surveillance system; however, they were poorly satisfied. This finding agrees with the findings of others in Afghanistan and Yemen [, ] but disagrees with other studies in Harare City [ ].
The data quality of the tuberculosis system was average. These findings were consistent with findings of other studies conducted in Harare City  but inconsistent with another study in the Republic of South Africa [ , ].
The sensitivity and the PPV of the system were good and average, respectively. These results are aline with a previous study in Afghanistan and Eden District [, ]. However, they disagree with a study in Pakistan [ ].
This evaluation was carried out in Sana’a City and targeted only the health facilities providing tuberculosis services (purposive sample) due to time and funds constraints.
The tuberculosis surveillance system was found useful. Flexibility, PVP, and data quality were average. Stability, acceptability, and simplicity were poor. The sensitivity was good.
The main weaknesses in the tuberculosis surveillance system included a lack of governmental finances, a paper-based system, and a lack of regular staff training. Developing an electronic system, securing governmental financial support, and training the staff on tuberculosis surveillance are strongly recommended to improve the system performance.
The authors would like to acknowledge the Global Health Development| Eastern Mediterranean Public Health Network for their technical support.
Conflicts of Interest
- Tuberculosis. World Health Organization. URL: https://www.who.int/news-room/fact-sheets/detail/tuberculosis [accessed 2020-04-25]
- Republic of Yemen. National Guideline for Tuberculosis Management in Yemen. Yemen: Ministry of Public Health; Jun 15, 2018.
- Global tuberculosis report. World Health Organization. URL: https://apps.who.int/iris/bitstream/handle/10665/336069/9789240013131-eng.pdf [accessed 2020-04-24]
- Tuberculosis profile: WHO Eastern Mediterranean Region. World Health Organization. URL: https://worldhealthorg.shinyapps.io/tb_profiles/?_inputs_&lan=%22EN%22&entity_type=%22group%22&group_code=%22EMR%22 [accessed 2020-04-25]
- Tuberculosis profile: Yemen. World Health Organization. 2020. URL: https://worldhealthorg.shinyapps.io/tb_profiles/?_inputs_&entity_type=%22country%22&lan=%22EN%22&iso2=%22YE%22 [accessed 2020-04-25]
- German R, Lee L, Horan J, Milstein R, Pertowski C, Waller M, Guidelines Working Group Centers for Disease Control and Prevention (CDC). Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR Recomm Rep 2001 Jul 27;50(RR-13):1-35. [Medline]
- Saeed K, Bano R, Asghar R. Evaluation of the national tuberculosis surveillance system in Afghanistan. East Mediterr Health J 2013 Feb;19(2):200-207 [FREE Full text] [Medline]
- Abdulmughni J, Mahyoub EM, Alaghbari AT, Al Serouri AA, Khader Y. Performance of multidrug-resistant tuberculosis surveillance in Yemen: interview study. JMIR Public Health Surveill 2019 Oct 03;5(4):e14294 [FREE Full text] [CrossRef] [Medline]
- Al-Akhali A, Ohkado A, Fujiki A, Mitarai S, Yamada N, Masui T, et al. Nationwide survey on the prevalence of anti-tuberculosis drug resistance in the Republic of Yemen, 2004. Int J Tuberc Lung Dis 2007 Dec;11(12):1328-1333. [Medline]
- Republic of Yemen. National Strategic Plan for Tuberculosis Prevention Care and Control in Yemen. Yemen: Ministry of Public Health; Jun 2016.
- World Health Organization. Report of Desk Review of the National Tuberculosis Program of Yemen. Cario: World Health Organization; 2014.
- Asif M, Baig M, Shah M. Evaluation of the Tuberculosis Surveillance System in District Hyderabad, Province Sindh-Pakistan, 2012. Int J Trop Dis Health 2015 Jan 10;9(1):1-8. [CrossRef]
- Makurumidze R, Gombe N, Bangure D, Takundwa L, Tshuma C, Magure T, et al. Evaluation of the Tuberculosis Surveillance System in Shamva District, Zimbabwe, 2014. J US-China Med Sci 2017 Apr 28;14(2):64-71. [CrossRef]
- Mlotshwa M, Smit S, Williams S, Reddy C, Medina-Marino A. Evaluating the electronic tuberculosis register surveillance system in Eden District, Western Cape, South Africa, 2015. Glob Health Action 2017;10(1):1360560 [FREE Full text] [CrossRef] [Medline]
- Matambo R, Duri C, Ruhanya V, Mungofa S, Chadambuka E, Nyandoro G, et al. Evaluation of a tuberculosis surveillance system in the Harare City Health Department in Zimbabwe. Int J Trop Dis Health 2016 Jan 10;14(4):1-10. [CrossRef]
- Podewils LJ, Bantubani N, Bristow C, Bronner LE, Peters A, Pym A, et al. Completeness and reliability of the Republic of South Africa National Tuberculosis (TB) Surveillance System. BMC Public Health 2015 Aug 11;15:765 [FREE Full text] [CrossRef] [Medline]
|CDC: Centers for Disease Control and Prevention|
|DOTS: directly observed treatment short course|
|MoPHP: Ministry of Public Health and Population|
|NTCP: National Tuberculosis Control Program|
|PPV: positive predictive value|
|WHO: World Health Organization|
Edited by M Algunaid; submitted 31.01.21; peer-reviewed by F Lami; comments to author 21.03.21; revised version received 29.04.21; accepted 15.08.21; published 30.11.21Copyright
©Fadwa Salem Ahmed Al kalali, Essam Mahyoub, Abdulbary Al-Hammadi, Labiba Anam, Yousef Khader. Originally published in JMIR Public Health and Surveillance (https://publichealth.jmir.org), 30.11.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on https://publichealth.jmir.org, as well as this copyright and license information must be included.