The Surveillance Outbreak Response Management and Analysis System (SORMAS): Digital Health Global Goods Maturity Assessment

Background: Digital health is a dynamic field that has been generating a large number of tools; many of these tools do not have the level of maturity required to function in a sustainable model. It is in this context that the concept of global goods maturity is gaining importance. Digital Square developed a global good maturity model (GGMM) for digital health tools, which engages the digital health community to identify areas of investment for global goods. The Surveillance Outbreak Response Management and Analysis System (SORMAS) is an open-source mobile and web application software that we developed to enable health workers to notify health departments about new cases of epidemic-prone diseases, detect outbreaks, and simultaneously manage outbreak response. Objective: The objective of this study was to evaluate the maturity of SORMAS using Digital Square’s GGMM and to describe the applicability of the GGMM on the use case of SORMAS and identify opportunities for system improvements. Methods: We evaluated SORMAS using the GGMM version 1.0 indicators to measure its development. SORMAS was scored based on all the GGMM indicator scores. We described how we used the GGMM to guide the development of SORMAS during the study period. GGMM contains 15 subindicators grouped into the following core indicators: (1) global utility, (2) community support, and (3) software maturity. Results: The assessment of SORMAS through the GGMM from November 2017 to October 2019 resulted in full completion of all subscores (10/30, (33%) in 2017; 21/30, (70%) in 2018; and 30/30, (100%) in 2019). SORMAS reached the full score of the GGMM for digital health software tools by accomplishing all 10 points for each of the 3 indicators on global utility, community support, and software maturity. Conclusions: To our knowledge, SORMAS is the first electronic health tool for disease surveillance, and also the first outbreak response management tool, that has achieved a 100% score. Although some conceptual changes would allow for further improvements to the system, the GGMM already has a robust supportive effect on developing software toward global goods maturity. (JMIR Public Health Surveill 2020;6(2):e15860) doi: 10.2196/15860 JMIR Public Health Surveill 2020 | vol. 6 | iss. 2 | e15860 | p. 1 https://publichealth.jmir.org/2020/2/e15860 (page number not for citation purposes) Tom-Aba et al JMIR PUBLIC HEALTH AND SURVEILLANCE


Introduction
Overview Digital health is a dynamic field with a rapidly growing number of initiatives and tools, many of which operate in certain geographic areas and for a limited period [1]. This lack of sustainability may be due to a variety of reasons, including the lack of integration in standard frameworks, lack of diverse and continuous donor support, lack of serving the objectives, and limited generalizability of its application among others [2][3][4]. In the field of public health, funding for such initiatives relies mostly on scarce public resources. Many such parallel initiatives do not seem to be mature enough to survive their pilot phase, which makes it particularly unfortunate. It is in this context that the concept of global goods maturity is gaining importance [5,6].

Global Goods
The concept of global goods stems from guidelines influencing health policies to support technologies that are meant to assist government agencies and policy makers in launching, scaling, and sustaining digital health innovations [7]. Different ministries of health convened advisory committees, including members of the digital health ecosystem and new multifaceted enterprises such as the Digital Impact Alliance and Digital Square [7]. These included global agencies, governments, philanthropies, funders, and academics to improve health data through shared investments in global goods and to fast-track the development and scale-up of successful digital health solutions to campaign for improvements in the facilitating environment for digital health [6,8]. To address the current lack of donor coordination, the committees created a framework to allow shared investments in specific countries and global goods. Health strengthening through enhancing country information systems would contribute to better decision making and, ultimately, better health [6]. The digital investment principles state that the funding donors within countries who are looking to prioritize their needs to improve the health of populations should align their resources around scalable, sustainable, accessible, interoperable, and evidence-based digital health global goods [6].

Surveillance Outbreak Response Management and Analysis System
The Surveillance Outbreak Response Management and Analysis System (SORMAS) is an open-source mobile and web application software that we developed to enable health workers to notify health departments about new cases of epidemic-prone diseases, detect outbreaks, and manage outbreak response at the same time. SORMAS is a management process system that supports supervisors to validate cases and control the spread of disease. As a multifunctional software, it can be used for case surveillance, laboratory data management, contact tracing, and disease detection to prevent and manage outbreaks that may occur. SORMAS also uses a bidirectional information exchange synchronizing user requests as well as sending feedback to the different users within the existing surveillance system [9][10][11].

Digital Square Global Goods Maturity Model
Digital Square, which is an initiative aimed at coordinating international efforts to develop and broadly share useful, free, and open-source digital tools, included 18 mobile health (mHealth) tools into the database of digital health software referred to as "global goods software" [12,13]. Digital Square developed a global good maturity model (GGMM) for digital health tools, which engages the digital health community to identify areas of investments for global goods [12,13]. We identified the GGMM to be a suitable concept in assessing the maturity of SORMAS for the following reasons: (1) GGMM has a particular focus on health software and on that being used in low-resource settings [12,13], (2) the objectives of GGMM matches well with the mission of SORMAS [12,13], and (3) the scope of most of the tools included in the GGMM guidebook fits that of SORMAS [12,13].
The following key concepts are prevalent throughout the study in determining the maturity of SORMAS according to the GGMM: global utility, community support, and software maturity. The objective of this study was to assess the level of global goods maturity that SORMAS has attained using the GGMM version 1.0.

Methods
We applied the GGMM version 1.0 on SORMAS from November 2017 until October 2019 to assess the level of global goods maturity that it has attained [13]. The GGMM contains 15 subindicators grouped into the following 3 core indicators: (1) global utility, (2) community support, and (3) software maturity. Each subindicator is divided into 3 possible values from −1 for "low," 0 for "medium," to 1 for "high." Each value contains a definition that is listed in

Global utility
At least ten countries or states actively use the tool for use as part of their At least four countries or states actively use the Less than two countries or states actively use the tool for use as part of their health information system Country utilization health information system, with at least 30% of total nation-wide or tool for use as part of their health information state-wide target users routinely using product/service as intended system, with at least 20% of total nation-wide or state-wide target users routinely using product or service as intended Software productization A robust API is available for key data and metadata exchange needs for the primary business domain with functional requirements for the API having been developed in conjunction with appropriate country, regional, and global stakeholders. API endpoints exist for core data and metadata elements that adhere to standards developed by an appropriate Standards Development Organization relevant to the tools business domain. Standards-based API endpoints are used in at least four jurisdictions (eg, countries or states) Some application programming interfaces (APIs) are available for accessing and managing data. There are user-facing interfaces to export core data and metadata in the system (eg, in CSV format) for further analysis and data transfer purposes Extract or import data into the system usually requires looking at source code and/or directly accessing database Interoperability and data accessibility Role-based authorization exists, if appropriate. All remote access (web interface and APIs) are encrypted by default using current best practices. An independent security audit of the software has taken place within the last 12 months There is at least one jurisdictions (eg, country, state) deployment for which 30% of all "entities" are managed within the software. Performance and load testing is a part of routine releases, and results are publicly available There is at least one jurisdiction (eg, country and state) deployment for which 20% of all "entities" are managed within the software. There has been at least one evaluation of software performance/load testing There are no jurisdictions (eg, country and state) that manage 10% of their "entities" within the tool, and no performance and load statistics exist

Scalability
We put together a GGMM assessment group consisting of a public health expert from the Nigerian Centre for Disease Control, a medical epidemiologist, an international health expert, an information technology specialist and a statistician from the Helmholtz Centre for Infections Research, and 2 software engineers from Symeda, a company for developing health software. This group periodically assessed the completion of SORMAS in the 15 subindicators. Subsequently, the software roadmap and the work plan of SORMAS was reprioritized to dedicate resources to and obtain progress in those subindicators that did not reach a full score. To reduce selection bias, as members of the assessment group were part of the development and deployment of SORMAS, and to reduce the potential partiality resulting from it, we also asked 2 external experts to review the findings made by the assessment group. These external experts had contributed to the development of the GGMM and were not involved in the development of SORMAS. LGAs and 85 health facilities [14,15]. By February 2018, SORMAS was deployed in 3 additional federal states (71 LGAs) for meningitis outbreak. In March and April 2018, SORMAS was further deployed in 3 additional states (49 LGAs; 5 health facilities) for the Lassa fever outbreak. As of October 2019, SORMAS has been fully established for all epidemic-prone diseases in 15 federal states (including the federal capital), 287

Results
LGAs, 37 health facilities, and approximately 700 users covering a population of 75 million. SORMAS managed multiple outbreaks simultaneously across the country. The assessment of SORMAS applying the GGMM showed that SORMAS had a 10-point score each in global utility, community support, and software maturity (see Table 2).  SORMAS has a multilingual support mechanism for English and French on its platform. The language translation component in  [10]. There is no registry or publicly available documentation of tools that have applied the GGMM assessment except the District Health Information System 2 [17]. Of all the mHealth tools selected as global goods software listed in the GGMM guidebook, none have accomplished over 90% of the full score nor are we aware of any other tool that has obtained it [13]. Thus, it appears very likely that SORMAS may be the first and, so far, only digital tool that has reached the full score of the GGMM. SORMAS has multiple revenue streams and funding mechanisms that have been enabling progress in its development since 2016. SORMAS requires funding for software adaptations to country-specific requests, personnel for training, and supervision and maintenance of the tool.
The outline and the scoring principle of the GGMM version 1.0 was easy to apply. However, there were some limitations concerning the clarity of the definition of each of the 3 possible criteria of the subindicators. Many of those definitions contained a combination of several contextually independent items, for which the wording did not clearly distinguish between "AND" or "OR" combinations. Some definitions left room for interpretation, which may be necessary for some but also be too ambiguous in other situations. The 30 subindicators grouped into 3 core indicators all have equal weights keeping the model and its handling simple.
On the other hand, it may also not represent the difference in importance that different indicators may have. For example, it may be considered essential for global goods maturity to have repetitive external security tests implemented than multilinguarity. Once the full score has been accomplished, these differences are no longer relevant. However, as long as the total score is only partially completed, the quantitative value may be very misleading in the attempt to compare tools. A way to improve this dilemma would be to add weights to each indicator. A more natural way would be to discourage the computation of proportional completion and to categorically apply a dichotomous all-or-nothing principle, by which all requirements have been either fully fulfilled or not fulfilled [18]. Some indicators address items that may develop into two directions. For example, the number of countries using the tool may not always diminish but also increase, or an external penetration test may not be repeated as required. It may increase scaling if those subindicators that do require a regular renewal or update be explicitly highlighted to facilitate monitoring of the status. Another difficulty appeared with the subindicator for global utility, "Source Code Accessibility," as it was not entirely clear what "new modules" and "functionality" meant in the context of this model: In contrast to modular systems, some mHealth apps are built specifically to deliver a ready-to-go solution for a specific task, and they are defined by external standards that govern how the internal processes of the end users work. In such a situation, the requirement for facilitating options for developing new modules and functionalities without prematurely forking the code may not be applicable.
For the time being, the GGMM serves as a self-assessment system. In that, the authors can confirm that in the case of SORMAS, it has dramatically guided the prioritization and acceleration of its development. It appears to be the main objective of the model. On the other hand, self-assessment also allows for a certain degree of subjectivity or bias [19]. To reduce this risk, we contacted 2 external experts who were not involved in the development and deployment of SORMAS but were instead instrumental in the development of the GGMM. Through this we aimed to reduce conflicts of interest and maximize the expertise on SORMAS as well as on the interpretation of the GGMM. We believe that inviting independent external experts in reviewing this assessment may be a model for other projects and tools to apply the GGMM assessment. We do not entirely recommend an external assessment, as practiced in many accreditation procedures, because it carries the risk of creating a business that will only but draw resources needed for the actual development of the tools.

Conclusions
To our knowledge, SORMAS is the first electronic health tool for disease surveillance, and also the first outbreak response management tool, that has reached the full score (100%) of the GGMM. The GGMM is clear in most of its definitions and is easy to apply for self-assessment, although some indicators require more resources for completion than others. Some conceptual modifications would allow for further improvements to the system. Nevertheless, it already has a supportive effect on developing software toward global goods maturity.