This paper is in the following e-collection/theme issue:

Viewpoint and Opinions on Technology and Innovation in Public Health (53) Innovative Methods in Public Health and Surveillance (336) Surveillance Systems (466) Health Services in Resource-Poor Settings and LMICs (423)

Published on in Vol 7, No 8 (2021): August

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/29309, first published .
Substandard and Falsified Medicines: Proposed Methods for Case Finding and Sentinel Surveillance

Substandard and Falsified Medicines: Proposed Methods for Case Finding and Sentinel Surveillance

Substandard and Falsified Medicines: Proposed Methods for Case Finding and Sentinel Surveillance

Authors of this article:

Elizabeth Pisani1, 2, 3 Author Orcid Image ;   Amalia Hasnida1 Author Orcid Image ;   Mawaddati Rahmi3 Author Orcid Image ;   Maarten Olivier Kok1, 4 Author Orcid Image ;   Steven Harsono5 Author Orcid Image ;   Yusi Anggriani3 Author Orcid Image
Article Authors Cited by (11) Tweetations (5) Metrics

    Viewpoint

    1Erasmus School of Health Policy and Management, Erasmus University, Rotterdam, Netherlands

    2School of Public Health, Imperial College, London, United Kingdom

    3Faculty of Pharmacy, Universitas Pancasila, Jakarta, Indonesia

    4Department of Health Sciences, Vrije Universiteit Amsterdam, Amsterdam, Netherlands

    5IQVIA Public Health, Singapore, Singapore

    Corresponding Author:

    Elizabeth Pisani, MA, MSc, PhD

    Erasmus School of Health Policy and Management

    Erasmus University

    3000 DR

    Rotterdam

    Netherlands

    Phone: 31 2072541654

    Email: pisani@ternyata.org


    The World Health Organization and others warn that substandard and falsified medicines harm health and waste money, especially in low- and middle-income countries. However, no country has measured the market-wide extent of the problem, and no standardized methods exist to estimate the prevalence of either substandard or falsified medicines. This is, in part, because the task seems overwhelming; medicine markets are huge and diverse, and testing medicines is expensive. Many countries do operate some form of postmarket surveillance of medicine, but their methods and goals differ. There is currently no clear guidance on which surveillance method is most appropriate to meet specific public health goals. In this viewpoint, we aimed to discuss the utility of both case finding and risk-based sentinel surveillance for substandard and falsified medicines, linking each to specific public health goals. We posit that choosing the system most appropriate to the goal, as well as implementing it with a clear understanding of the factors driving the production and sale of substandard and falsified medicines, will allow for surveillance resources to be concentrated most efficiently. We adapted principles used for disease outbreak responses to suggest a case-finding system that uses secondary data to flag poor-quality medicines, proposing risk-based indicators that differ for substandard and falsified medicines. This system potentially offers a cost-effective way of identifying “cases” for market withdrawal, enhanced oversight, or another immediate response. We further proposed a risk-based sentinel surveillance system that concentrates resources on measuring the prevalence of substandard and falsified medicines in the risk clusters where they are most likely to be found. The sentinel surveillance system provides base data for a transparent, spreadsheet-based model for estimating the national prevalence of substandard and falsified medicines. The methods we proposed are based on ongoing work in Indonesia, a large and diverse middle-income country currently aiming to achieve universal health coverage. Both the case finding and the sentinel surveillance system are designed to be adaptable to other resource-constrained settings.

    JMIR Public Health Surveill 2021;7(8):e29309

    doi:10.2196/29309

    Keywords

    substandard drugs; falsified medicine; counterfeit medicine; medicine quality; sentinel surveillance; public health surveillance; substandard; pharmaceuticals; surveillance; public health 


    Background on Substandard and Falsified Medicines

    In late 2017, World Health Organization’s (WHO’s) press department issued a press release with the bold headline: “1 in 10 medical products in developing countries is substandard or falsified” [1]. More recently, with governments scrambling to secure supplies of diagnostic tests, medicines, and vaccines to cope with the COVID-19 pandemic, WHO and others have issued new warnings stating that the world may face an increased threat of poor-quality medicines [2-5]. These include substandard medicines, which are made by registered pharmaceutical companies in regulated factories but do not meet the quality standards set out in their market authorization paperwork, either because they were poorly made or because they have degraded since manufacture. This increased threat of poor-quality medicines also includes falsified medicines, which are made, repackaged, or sold by criminals who seek deliberately to misrepresent the identity, composition, or source of the product [6].

    Poor-quality medicines can use up family and national budgets without curing patients; indeed, they sometimes poison or kill people instead of curing them. Underdosing infectious pathogens also allows drug-resistant infections to spread [7,8]. Thus, if these “medicines” are indeed common, they may substantially undermine physical and financial health. Estimates based on available data for particularly well-studied molecules provide an order of magnitude: poor-quality antimalarials were estimated to cost US $130 million per year in a single region of the Democratic Republic of Congo, US $141.5 million in Zambia, and US $830 million across Nigeria. In the latter country, substandard antimalarials are estimated to contribute to 12,300 deaths per year [9-11]. A 2015 study reported that, in 39 sub-Saharan African countries, there were 122,350 deaths attributable to poor-quality antimalarials among children under 5 years of age. However, the authors noted that “there is considerable uncertainty surrounding our results because of gaps in data on case fatality rates and prevalence of poor-quality antimalarials” [12]. Similarly, the meta-analysis that gave rise to WHO's press release, which said that 10% of medical products in developing countries are substandard or falsified, is careful to note the many limitations of that estimate. This meta-analysis was based on studies of uneven sizes and methods, conducted largely in low-income countries with limited domestic pharmaceutical industries, and heavily skewed toward antimalarials and a few other medicines that most interest global health agencies. Even within that constrained pool and looking only at studies that included sample sizes of 50 or more, reported prevalence of substandard or falsified medicines ranged from 0% to 91% [13]. Reviews have reported similar data constraints and findings [14-18]. For example, Ozawa and colleagues [18] found that studies reported a prevalence of substandard or falsified medicines between 0.8% and 89% in Africa and a prevalence of between 0.7% and 50% in Asia.

    WHO actively maintains a case-reporting system for substandard and falsified medical products, including medicines, contraceptives, vaccines, and point-of-sale diagnostics. For brevity, we use the term medicines throughout this paper to cover all these medical products. Regular training provided to individuals designated as in-country focal points increases the use of the system, but, similar to all case-reporting systems, it provides no information on denominators (the number of products inspected or tested), so interpretation of trends and comparisons between countries is difficult.

    It may be that the problem of medicine quality is understated because of a vicious cycle of limited systematic measurement leading to limited visibility and limited awareness of the problem that in turn restricts resources available for systematic measurement. Alternatively, the problem may be that WHO and researchers are cherry-picking data to overstate the problem, perhaps for reasons of self-interest, as Hodges and Garnett [19] suggest.

    We do not know which of these dynamics holds true. There is, to our knowledge, no clear understanding of the prevalence of substandard or falsified medicines in any single country, let alone across all the “developing countries,” as suggested by the press release’s headline. No country has yet made systematic estimates of the prevalence of substandard or falsified medicines across all therapeutic categories in its medicine market, and no standardized methods for calculating such an estimation yet exist.

    In this viewpoint, we aimed to briefly review different approaches to surveillance and estimation in public health, discuss their relevance in the context of medicine quality, and lay out ideas for 2 potentially cost-minimizing methods that may improve our ability to measure or reduce the prevalence of poor-quality medicines, especially, in low- and middle-income settings.

    Approaches to Surveillance

    Overview of Surveillance Systems

    We follow WHO, United States Centers for Disease Control and Prevention, the World Bank, and others in defining public health surveillance as the ongoing and systematic collection and use of data to inform policy, plan and evaluate interventions, and improve health outcomes [20,21]. Surveillance systems monitor the prevalence of infectious and noncommunicable diseases; of disability; and, increasingly, of the behavioral, social, corporate, and environmental causes of ill-health. In addition, surveillance systems have, in recent decades, expanded to include the systematic monitoring of health system factors such as service use, prescription practices, or access to medicine.

    Surveillance can take many forms, each serving a slightly different purpose within the catch-all definition of “improving health outcomes.” However, most can be categorized into either “passive” or “active” surveillance. Passive surveillance involves reporting events such as disease diagnoses as they arise. An early example of passive sentinel surveillance in the United States was the weekly reporting of diseases by designated physicians, which began in Massachusetts in 1874 [22]. The informatics era has greatly expanded the potential for secondary data to be used to inform public health decision-making. Examples include the use of both retail data of over-the-counter medicine sales and data from internet searches to flag potential disease outbreaks, and the use of medical-claims data to track trends in noncommunicable diseases [23,24]. Active surveillance tends to be more resource intensive, usually involving purposive data collection—often the collection and screening of blood or other biological samples or, more recently, medical imaging.

    Active surveillance systems systematically collect and test samples for the purpose of tracking ill-health or health-related risks. Some active surveillance, such as active sentinel surveillance, test a cross-section of a defined population to establish disease prevalence. Others, such as case finding, specifically, target individuals at highest risk of needing services. These 2 types of active surveillance have different purposes. Sentinel surveillance, similar to many other surveillance systems, such as those that track noncommunicable diseases, determinants of health, and health system factors, provides data intended to guide medium- or longer-term health program planning. Case-finding systems, frequently used in infectious disease outbreaks but also used for early detection of treatable noncommunicable conditions, provide data intended to inform immediate therapeutic or preventative action. These different goals, upon which this paper focuses in the context of medicine quality, affect the design and use of surveillance systems and data, as show in Table 1.

    Table 1. Major types of surveillance systems in public health.
    PurposeOutbreak responseHealth program planning
    System design:Case finding: identify infected individualsSentinel surveillance: track prevalence over time
    Resulting action:Isolate and treatAdjust policies and programs
    Key characteristic:Specific: pinpoint individuals for rapid follow-upComparable: standardized methods allowing comparison over time
    Cannot be used to:Estimate prevalence; track trends over timeRespond at individual level
    Case Finding

    The control of outbreaks and epidemics of infectious diseases requires that chains of transmission be broken. In these circumstances, surveillance systems try to identify infected individuals, isolating and, if possible, treating them to interrupt transmission. We term these systems “case finding.” They are relatively rare but have seen a resurgence during the COVID-19 pandemic.

    Sentinel Surveillance

    Sentinel surveillance systems are more common. Designed to track trends in infection over time, they use standardized methods to measure the prevalence of a disease within a defined population, comparing the result with prevalence measured in the same way in earlier years or in different locations. Sentinel surveillance is used to estimate the burden of disease, to target prevention and treatment interventions, and to monitor the impact of these interventions.

    While passive surveillance can achieve these goals, it is of limited use for tracking rare diseases, which are easily missed by these systems [25]. While epidemiological orthodoxy holds that active surveillance involving regular screening of randomly selected samples provides the best approximation of disease trends across a population as a whole, this is also impractical for rare conditions that would require very large samples.

    The HIV pandemic entrenched the idea of active sentinel surveillance in populations defined not by geography but by risk of exposure to the virus [26,27]. This allowed health authorities to focus surveillance resources in subpopulations where the majority of cases of the largely invisible disease were to be found while still producing comparable data and tracking trends over time. In many countries, those groups included people who inject drugs, gay men, sex workers of all genders, and sex workers’ most frequent clients.

    Figure 1 illustrates randomized and risk-based approaches to HIV sentinel surveillance. For the same limited resources (in this simplification, 5 tests), random sampling, on the left, yields just 1 positive test, while sentinel surveillance, on the right, yields 3. Combined with robust estimates of the size of those subpopulations, this approach will provide a more accurate estimate of the prevalence of infection nationally, the data produced will more accurately reflect the effect of targeted risk-reduction interventions, and these benefits will be achieved at a lower cost compared with random sampling.

    The HIV example is of considerable relevance when thinking about surveillance of substandard and falsified medicines, because it is largely invisible until actively tested and clusters around known risk factors. A similar model for medicine quality is presented in this paper’s section “Proposed Method for Sentinel Surveillance.”

    ‎
    Figure 1. Illustrative difference between random and risk-based surveillance for HIV.
    View this figure

    Existing Surveillance of Medicine Quality

    At the international level, surveillance of medicine quality takes the form of passive reporting of detected cases. As with disease case reporting, this provides information by demographic, geographic, environmental, or other factors, which is invaluable in helping to identify clusters of risk. However, case reporting does not provide any idea of the number of products tested. No case reports may mean there is no problem in a particular country, but it may also mean there is no capacity or willingness to detect or report cases.

    At the national level, medicine regulators in many countries conduct some form of postmarket surveillance. In some countries, this is largely passive, limited to collating reports of adverse events submitted by health care providers through pharmacovigilance systems. Again, this means the denominator is unknown. However, in other countries, the regulator actively samples medicines from supply chains for inspection and testing. Where they report the number of products inspected or tested, as well as the number of out-of-specification products, this active surveillance allows for the calculation of prevalence in the segment of the market from which samples were drawn.

    Sample selection in active surveillance varies widely, from random to convenience sampling, although regulators do not always state which method they use. Academic groups and WHO have published recommended methods for conducting surveys of medicine quality [28,29], as well as for sampling high-risk medicines from the internet [30]. While not focused specifically on sentinel surveillance, these methods have informed the guidance provided by technical and regulatory agencies on surveillance approaches that focus on selecting products at the highest risk for inspection and testing, including at the point of import [31-36]. To date, the criteria for determining risk have focused largely on the risk of impact to public health, factors intrinsic to the molecules (eg, stability and therapeutic index), and regulatory history. Not all agencies share information about risk profiling, for fear of helping those who produce poor-quality medicines to circumvent targets. However, as far as we know, market-related drivers of the risk of falsification are rarely considered. Furthermore, none of the guidelines or tools currently in the public domain explicitly differentiate between the risks for falsification and the risks for substandard production or degradation. However, attention paid to risk-based, postmarket surveillance is growing; the WHO Member State Mechanism on falsified and substandard medicines chose the development of methods and tools for risk-based surveillance as a prioritized activity in its current workplan [6], and work is ongoing.

    Sentinel surveillance of substandard or falsified medicines is a form of postmarket surveillance designed explicitly to select samples in reproducible ways over time, so that trends can reliably be measured. This is rare in the case of substandard or falsified medicines, although some repeat random surveys have been conducted [37]. Site-based sentinel sampling has also been attempted in some locations. This may suffer from bias if people, including falsifiers, become aware of the practice and change behavior to avoid supplying known sentinel sites [29].

    In the context of medicine quality, case-finding efforts focus on trying to identify individual products that are most likely to be substandard or falsified, so that they can quickly be recalled or otherwise removed from the market. In some high-income countries with strong pharmacovigilance systems, these efforts coexist with active sampling from the supply chain.

    Market-wide estimates of the prevalence of substandard or falsified medicines are virtually nonexistent. Reasons for this include the apparent complexity of the task and the expense of pharmacopeial testing. Even small countries will typically have many thousands of registered medicines and vaccines on the market. Meanwhile, well-staffed medicine-testing laboratories are scarce—there are fewer than 50 WHO-prequalified drug-testing laboratories across all low- and middle-income settings [38]. Local pharmaceutical reference standards and reagents that allow for testing of the content and quality of medicines are often unavailable, while international, gold-standard products can cost several hundred dollars for even the most common molecules [39]. In addition, medicine regulators may be wary of systematic approaches, seeing transparent surveillance and robust estimation processes as an unwelcome evaluation of regulatory performance.

    In short, while postmarket surveillance exists in different forms, there is currently no global guidance on the purpose or shape of national surveillance systems for substandard or falsified medicines and no standardized methods for translating the results of surveillance into market-wide estimates of prevalence.

    The remainder of this paper proposes candidate methods, expanding on existing risk-based approaches. We aimed to review the risk factors that underlie (1) substandard and (2) falsified medicines; to propose a method for case finding based on the identified risk factors; to propose a sentinel surveillance method based on the identified risk factors; and to propose a method for developing nation-wide estimates for the prevalence of substandard and falsified medicines, based on sentinel surveillance. Our proposal is based on exploratory work undertaken in Indonesia. We believe the proposed methods are feasible in many resource-limited settings.


    Overview of Risk Factors for Poor-Quality Medicines

    Both falsified and substandard medicines exist because there is money to be made selling them. In the same way that a spike in opportunistic infections once signalled a potential cluster of undetected HIV infections, dymanics in medicine markets can act as crude predictors of clustered cases of substandard or falsified medicines. In earlier works, we reviewed available academic literature; examined reports of the case-reporting database, WHO Global Surveillance and Monitoring System for substandard and falsified medical products; and conducted detailed case studies in 4 middle-income countries, using an epidemiological approach to identify risk factors associated with substandard and falsified medicines [6,40]. We identified a limited number of market-related factors that combine to increase the possibility that certain products in a market will be substandard, as shown in Table 2.

    Table 2. Market risk factors for substandard medicines.
    Risk factorRisks to quality
    High or rising pressure on profit marginsIncentivizes cost cutting
    Stretched technical capacityIncreases risk of production errors or degradation during distribution
    Limited oversightAllows substandard products to flow through the supply chain
    Low risk of damage to corporate reputationReduces incentive to invest in quality assurance

    Some of these market factors operate at the level of a particular brand, others operate at the company level, others relate to the level of the supply chain, and others relate to a specific molecule. These market factors interact and further combine with other factors already considered in risk-based surveillance for medicine quality, such as the stability of a molecule or the complexity of the production or packaging process, to signal the likelihood that a medicine will be substandard or falsified.

    A comprehensive review of academic literature describing interventions to control falsified medicines found few studies that addressed market drivers of falsification [41]. However, we find market-related factors strongly shape incentives for falsifiers, leading to increased risk of falsification, as shown in Table 3.

    Table 3. Market risk factors for falsified medicines.
    Risk factorFalsifier incentive
    Shortage of (or restricted access to) affordable, desired productCriminals prefer to make and sell products where there is a ready market (where demand exceeds accessible supply)
    High-priced medicine or relatively high-priced brandProfit opportunity influences choice of product and brand falsification
    Limited risk of discovery or punishmentRisk of retribution shapes choice of distribution channel

    We propose adding indicators of the market factors shown in Tables 2 and 3 to increase the specificity of existing risk-based sampling and to more easily distinguish between products at risk for falsification and those more likely to be substandard.

    Proposed Method for Case Finding: Sample Based on an Index of Risk

    Effective case-finding systems may appeal to regulators, politicians, and the public, because they inform product recalls and other immediate actions to protect patients. On the downside, these systems are data-hungry, and sampling is relatively resource intensive. They do not systematically test a specific number of samples from a well-defined population, and, thus, cannot easily be used to measure trends over time or to estimate the magnitude of the problem.

    These limitations notwithstanding, many sources of routinely collected data related to medicine markets do exist, including in middle- and some lower-income countries. These include data collected by medicine regulators, health authorities and insurers, customs and excise departments, and market research firms. We propose to use these data to guide case finding, as shown in Textbox 1. Steps 2-5 should be undertaken separately for substandard and falsified medicines.

    Steps for systematic case finding for falsified and substandard medicines.
    • Step 1: Define indicators of public health importance (eg, burden of disease, vulnerability of affected population, narrow therapeutic index, sales volume of brand, or dosage form).
    • Step 2: Define 1 or more objective indicators for each of the risk factors for substandard medicines and for falsified medicines, specifying the level at which it operates. Identify potential collinearity, and eliminate duplication.
    • Step 3: Create risk scores for each numeric indicator (eg, none, minimal, some, or high), and calculate indicators and scores for each product (examples in Supplementary Tables A and B in [42]).
    • Step 4: For each product, add risk scores to create a total index of risk. Select products to be sampled, prioritizing those with the highest risk index.
    • Step 5: For sampled products, weigh by risks related to geography and supply chain, and draw up a sample frame.
    • Step 6: Sample selected products (from specified locations, if indicated in Step 5).
    • Step 7: Test sampled products. For potentially falsified products, screen visually and using rapid or low-cost devices such as a hand-held spectrometer or field-based thin layer chromatography. For potentially substandard products, perform quantitative assay and dissolution tests.
    Textbox 1. Steps for systematic case finding for falsified and substandard medicines.

    Step 1 is carried out in consultation with health authorities, while steps 2 and 3 take into account available data sources and the opinion of experts from the many sectors involved in the production, procurement, sale, and use of medicines. In Table 4, we provided a single example of a possible indicator for each area of risk for substandard production or degradation. These suggestions derive from ongoing exploratory work in Indonesia, a large middle-income country with substantial domestic medicine production and a single-payer health insurance system. Exact specifications of the indicators, as well as decisions about potential weighting, may differ by country and will be determined, in large part, by the data available. A more comprehensive list of alternative indicators, together with suggested data sources, is provided in the supplementary tables in [42].

    Because the medicine market is extremely heterogeneous, several indicators use relative measures, such as ratios compared with the median. These indicators must then be turned into scores that can be added together to create a total index of risk as described in Textbox 1. Supplementary Table A in [42] suggests methods for turning indicators into risk scores.

    To provide a single example for the first indicator in Table 4, examine the ratio of price to weighted market median price for the same product. If the ratio is above 1, the product is priced above the market median and, thus, not deemed irrationally cheap or at risk of cost cutting. Deciles of risk are calculated only for those products with a price–to–median-price ratio of less than one. Products closest to the median (deciles 7-10) may also be considered at no risk. Those in deciles 5 and 6 may score at 1 risk point (at minimal risk for cost cutting), and those in the second to fourth deciles score 2 points (at some risk). Brands (or nonbranded products from a specific market authorization holder) that fall into the first decile—the products selling at the deepest discount—are awarded 3 risk points (at high risk for cost cutting). Narrower gradations would allow for greater specificity; expert committees may decide what is most appropriate in the local context.

    A similar process can be undertaken for products at risk for falsification, but the indicators will be different. Table 5 provides examples for each of the major risk-factor groupings. Again, a more comprehensive list of alternative indicators, together with suggested data sources, is provided in the supplementary tables in [42].

    The success of the case-finding approach will depend, to a significant extent, on the willingness of data custodians to share these data with those conducting case finding. The sensitivity and specificity of case finding will additionally depend on the ways in which indicators are combined. While Textbox 1 describes a simple index, weighting is possible. If weighting is used, it is likely that brand-specific indicators, which have greater specificity, will carry a greater weight than market-wide indicators relating to molecules. However, we propose working with regulators to use retrospective data to find the model that best predicts poor-quality products in specific markets. Regulators with higher capacity for analysis may wish to develop more complex algorithms, including “big data” approaches, that combine price and volume data in ways that more closely pinpoint risk in specific markets.

    Table 4. Indicative components to flag potentially substandard medicines.
    IndicatesIndicatorLevel at which indicator appliesRationale
    Profit pressure: cost cuttingRatio of price to weighted market median for same product (same molecule and dosage form)Brand and dosage formAlthough premium brands are usually available, products produced by a large number of companies will tend toward the lowest cost of quality-assured production plus a fair profit [43]. If a particular product sells significantly below the market median, it may signal insufficient investment in quality assurance or other cost-cutting measures.
    Technical limitation: production errorsNumber of years continuously producing this moleculeManufacturer (per molecule)As companies and their staff gain experience and streamline their standard operating procedures in the production of a new medicine, the risk of production errors falls. Mistakes in production are more common among newly registered manufacturers.
    Limited oversightTime since most recent GMPa inspection of any facilityManufacturer (by
    production site)    		Medicine regulators aim to inspect production facilities on a regular basis; some additionally include risk-based inspection. In practice, frequency of inspection depends on regulatory capacity, and intervals may vary. The risk of detectable deviations from GMP grows with time since last inspection.
    Production historyNumber of regulatory warnings or sanctions given to manufacturer over reference periodManufacturer (all
    products)    		Investment in quality assurance is embedded in corporate culture. Manufacturers who repeatedly receive warnings for GMP violations may systematically underinvest in quality assurance, meaning all their products are at higher risk.
    Reputational riskNumber of years of MAb holder in marketMA holder (all products)Most companies are incentivized to invest in QAc, in part, because they wish to maintain their reputation as a provider of quality goods. New companies may be established opportunistically, especially, in rapidly growing markets. With less investment in building a reputation than older firms, new companies may have less to lose if found to be marketing substandard products.
    Intrinsic risk:
    degradation    		Stability of moleculeMolecule (all products)Some molecules are less stable than others and more sensitive to variations in humidity, temperature, light, or other factors. Less stable molecules are more likely to degrade, becoming substandard before consumption.
    Ecological risk: degradationClassification of district
    accessibility    		All products (or less stable products), by point of saleLong supply chains and poor infrastructure pose challenges for maintaining temperature and humidity and may also reduce frequency of distribution. These factors increase the risk of degradation, especially, for less stable products.

    aGMP: good manufacturing practice.

    bMA: market authorization.

    cQA: quality assurance.

    Table 5. Indicative components to flag potentially falsified medicines.
    IndicatesIndicatorLevel at which indicator appliesRationale
    Market opportunity: limited affordabilityProduct is on patent but not listed in current national formularyBrandOn-patent products usually have premium prices. When they are not listed in current national formularies, they are usually not covered in the national insurance scheme, indicating limited affordability for patients. Patients or health care providers may seek these products at cut prices outside of the regulated supply chain.
    Market opportunity: desirabilityMolecule is used recreationally or off-labelMoleculeSome narcotics and psychotropic medicines are used recreationally or otherwise abused, including use for purposes for which they are not licensed. Additionally, access to some medicines is tightly restricted for political reasons, such as their potential use as abortifacients. Since the sale of these products is regulated, users without prescriptions commonly seek them outside of the regulated supply chain or from vendors who do not observe due diligence.
    ProfitabilityRatio of (price × retail channel sales volume) to market median, for the same dosage formBrandFalsifiers want to sell products for which there is a lucrative market, for which a large number of patients are prepared to pay a high price. For any given medicine for which there is a choice of brands, those brands with a combination of a relatively high retail price and a relatively large sales volume will be attractive targets.
    Low risk of detectionNumber of listings for product on 2 largest internet marketplacesBrandGeneral internet marketplaces provide an unregulated but commonly used space for trading medicines without official licenses. The vast number of online transactions creates difficulty for regulatory monitoring, and anonymity limits the possibility of repercussions. More listings of products on the largest general online marketplaces also indicate high demand.

    Proposed Method for Sentinel Surveillance: Tracking Trends in Risk Groups

    Sentinel surveillance is a form of postmarket surveillance that is less data-intensive than case finding and has a different purpose. Systematic testing of comparable samples over time allows health authorities to: establish the likely prevalence of substandard medicines and, separately, of falsified medicines; inform estimates of the health and economic impacts of these medicines; make a case for additional investment in quality assurance in production or procurement, if necessary, including more investment in regulatory enforcement; plan and implement policies and programs to reduce prevalence of poor-quality medicines; and track progress over time toward achieving that goal.

    The principal challenge in developing a robust, risk-based sentinel surveillance system for substandard and falsified medicines is in identifying “risk groups” of medicines, within which most poor-quality medicines cluster. These risk groups are the functional equivalent of the risk behaviors that circumscribe sentinel populations for another invisible threat to health, HIV infection.

    Similar to HIV sentinel surveillance, the specific sentinel groups may vary from country to country, depending on market dynamics and the risks and opportunities they create. The critical point is that groups are defined based on the feasibility of drawing samples and in ways that are replicable over time.

    Drawing from existing risk-based approaches and the additional market risk factors identified in Table 2 and Table 3, and, again, with reference to ongoing exploratory work in the Indonesian market, we propose sentinel groups for both substandard medicines (Table 6) and falsified medicines (Table 7). We underline that these groupings are not intended to encompass all at-risk products, nor do we suggest that all of the products in these groupings are at risk. Rather, these factors act as proxies that may yield a higher concentration of at-risk products, compared with a random sample.

    Table 6. Suggested sentinel groups for substandard medicines.
    Sentinel groupDefinitionSignals potential problem ofRationale
    Irrationally low-priced essential medicinesIn public systems, medicine price <75% of international reference price; in retail pharmacies, cheapest available version of target medicineCost cuttingIrrationally low prices are a strong predictor for cost cutting. Selection of samples from the public system can be brand specific, and price data are available in advance, so a clear price threshold can be set. For private provision and retail sampling, brand-specific sampling is not feasible, so the cheapest medicine available should be sampled. Molecules should be selected for local public health importance.
    Contract-
    manufactured medicines    		Products randomly selected from those that are manufactured by a company other than market authorization holderReduced oversightContract manufacturing is a frequent means of lowering production costs (by outsourcing to companies that can achieve economies of scale). The market authorization holder does not always have clear oversight of quality assurance practices at contract manufacturers.
    Poor regulatory historyMedicines randomly selected from those made by companies with history of regulatory violations and involuntary recalls within a time reference periodInadequate quality assuranceThough regulators work with past violators to improve manufacturing and distribution practice, corporate culture and incentives appear to act as enablers of cost cutting and other practices that increase the risk of substandard production or degradation.
    Technically
    vulnerable    		Medicines randomly selected from those that are technically challenging to manufacture or distribute, including those with unstable molecules, limited therapeutic index, or sterile formsHigher potential for production errorsThese product-specific characteristics require particular investment in quality assurance. It would be possible to restrict this sentinel group to newer, less experienced manufacturers or market authorization holders. For unstable molecules, samples may be drawn from outlets in geographically remote areas.
    Table 7. Suggested sentinel groups for falsified medicines.
    Sentinel groupDefinitionSignals potential problem ofRationale
    High irrational demandMedicines that are used for recreation or other off-label purposes, for which alternatives are restricted or expensive; randomly sampled from retail outletsMarket opportunityDemand planning is based on authorized uses only. Off-label use creates shortages, which provide market opportunities for falsifiers. Sample frame may be weighted toward independent pharmacies and medicine shops.
    Life-saving but unaffordableMedicines that are known or reputed to be life-saving, that retail at >10% of average per capita household spending, but that are not covered by national insurers; brand-specific sample from retail outletsMarket opportunityPatients with life-threatening conditions are highly motivated to acquire these medicines. High profit margins incentivize their sale, which may diminish due diligence even in the regulated supply chain. This sample may include products not locally authorized; these should also be screened for falsification.
    Sold on unregulated internet platformsRandom sample of “prescription-only” medicines sold through unlicensed internet sellers (sentinel group may be combined with signals of profit potential though purposive sampling of brands retailing at >200% of market median for the dosage form)Evasion of regulationFalsifiers favor internet sales because the potential for detection and successful prosecution is low. While the sample may be weighted toward medicines with high irrational demand, it should include other medicines of public health importance, such as antibiotics.

    Most sentinel groups should be sampled in the public, private, and (if applicable) nonprofit sectors; the obvious exceptions are samples specific to unregulated channels, which should not exist in the public sector. A sentinel surveillance system may be established following a process similar to that described in Textbox 2.

    Steps to establish sentinel surveillance for falsified and substandard medicines.
    • Step 1: Define indicators of public health importance (eg, burden of disease, vulnerability of population, sales volume of brand or dosage form, or publicly procured); list medicines by public health importance.
    • Step 2: Review local data sources and market conditions to define proxy “sentinel groups,” in which the highest concentrations of (A) falsified and (B) substandard medicines of public health importance are likely to be found.
    • Step 3: Define indicators (or combinations of indicators) that best circumscribe those sentinel groups.
    • Step 4: Draw up a sample frame for each sentinel group, including in it the public and private sectors, as appropriate, and sample a predetermined number of products for testing.
    • Step 5: Test samples. For sentinel groups containing potentially falsified products, screen visually and using rapid or low-cost devices such as hand-held spectrometers or field-based thin layer chromatography. For sentinel groups containing potentially substandard products, perform quantitative assays and dissolution tests.
    Textbox 2. Steps to establish sentinel surveillance for falsified and substandard medicines.

    Pharmacopeial testing is expensive. The risk-based sentinel approach aims to reduce costs of routine surveillance in 2 ways. First, it increases the “yield” of testing by focusing it on the clusters of medicines most likely to be at risk (Figure 2). Second, it provides an initial triage for testing technologies. Products selected in sentinel groupings for falsification risk can be screened visually and using lower-cost field-based devices [44]; only those at high risk of substandard production need to undergo assay and dissolution testing.

    ‎
    Figure 2. Illustration of risk-based sentinel surveillance for substandard and falsified medicines.
    View this figure

    Developing National Estimates for the Prevalence of Substandard and Falsified Medicines

    Steps to Estimate the Prevalence of Poor-Quality Medicines

    In the same way that HIV prevalence among sex workers or drug injectors does not represent the prevalence of the virus in a whole population, the prevalence of poor-quality medicines in risk-based sentinel groups does not represent medicine quality across a whole national market. However, if the size of each of those sentinel groups can be calculated and if assumptions can be made about their relationships with the wider population, then sentinel surveillance can provide a starting point for making robust estimates of national prevalence [45]. The same is true for medicine surveillance groups and their relationships with the wider market. Textbox 3 suggests a process for developing such estimates.

    Steps to estimate the national prevalence of falsified and substandard medicines.
    • Step 1: Conduct risk-based sentinel surveillance for falsified and substandard medicines to obtain prevalence estimates for each sentinel group.
    • Step 2: Calculate the interaction between sentinel groups.
    • Step 3: Use market, procurement, and regulatory data to estimate the total volumes of medicines in each sentinel group.
    • Step 4: Apply the prevalence estimates (step 1) to the volume data (step 3) to estimate the total number of substandard medicines across all sentinel groups for substandard medicines, correcting for interactions where necessary. Repeat for falsified medicines.
    • Step 5: Use market, procurement, and regulatory data to estimate the total volumes of medicines in (lower-risk) market subsectors that do not fall in the sentinel groups.
    • Step 6: Use all available data sources (eg, regulatory data, academic studies, case-reporting data, and data from other countries) to make assumptions about the residual prevalence in these nonsentinel sectors, comparing with prevalence in sentinel groups. Document each assumption, and, then, estimate the prevalence of substandard and falsified medicines in each sector.
    • Step 7: Apply the prevalence estimates (step 6) to volumes (step 5) to estimate the total number of poor-quality medicines outside of sentinel groups.
    • Step 8: Calculate the national prevalence estimate, where national prevalence estimate = (step 4 + step 6)/(step 3 + step 5).
    Textbox 3. Steps to estimate the national prevalence of falsified and substandard medicines.

    The approach in Textbox 3 has the great advantage of transparency. In addition, the process will highlight important data gaps that can be rectified over time. Assumptions can easily be corrected as more complete data are collected or shared. For example, step 6 might initially include the assumption that imported, on-patent oncology drugs that do not fall into any of the groups in Table 6 are substandard 0% of the time. The assumption may change if regulators in producing countries issue product recalls for products that are also exported.

    Estimation Process

    Health-related estimates tend to improve in accuracy and local relevance (and, thus, potential utility) if potential end users are involved in their production [46]. This is, in part, because these actors can identify and contribute data to the process, and their expertise provides critical insights for informing necessary assumptions [47]. Estimation of the prevalence of substandard and falsified medicine should be led by medicine regulators and ministries of health. We would strongly encourage active consultation with pharmaceutical manufacturers and distributors, insurers, procurement authorities, consumer and patient advocates, and professional associations (eg, doctors and pharmacists) for deciding on methods and assumptions. The policies, decisions, and behaviors of these actors shape markets and influence the quality of medicines that patients consume [48]. Besides enriching the process, their participation increases the likelihood that estimates will be broadly accepted and acted upon. However, the motivations and interests of these groups are rarely aligned; careful annotation and transparent publication of all assumptions and data sources used in the estimation process can guard against capture by any interest group, protecting the integrity of the estimates [47].

    Next Steps

    The national estimates that result from this process will not capture the full complexity of medicine markets, especially, in initial rounds of estimation. However, we think it is important to begin to work toward that goal with tools that are most likely to be adopted by regulators in resource-constrained settings. We believe that a feasible and important first step in better quantifying the threat posed by substandard and falsified medicines includes simple, spreadsheet-based national models that clearly document all data sources and assumptions and that are based on clearly defined and repeatable sentinel surveillance. Later, more sophisticated models may embrace more complexity and be expanded to estimate the extent to which these products undermine health and well-being and the damage they do to family and national budgets.

    Our suggested methods may seem complex, and the processes may seem institutionally daunting, but, again, we draw inspiration from the experience of HIV surveillance. The current state of information systems for medicine quality closely resembles HIV surveillance systems circa 1995. Many low- and middle-income countries had no system at all. In those that did, incomplete case reporting was the norm; sentinel surveillance focused mainly on pregnant women even in countries where most infections were in men; behavioral risk surveillance was in its infancy; and estimation of population size was unheard of. Meanwhile, most estimates of national prevalence were made by a handful of people working for international organizations, using assumptions that did not reflect the diversity of national epidemics [49].

    Now, the picture is very different. Most countries have developed surveillance systems based around the specific risks that drive their national epidemic and gather data related to risk behaviors and treatment outcomes in ways that are comparable across time. Population size estimation allows for the development of prevalence estimates that are useful in informing programming and measuring risks. Many different sectors cooperate in the implementation of these systems, which are largely country-led [50].

    This transition in HIV surveillance was made possible by the vast, disease-specific investments in HIV seen from the early 2000s, investments that were themselves triggered, in part, by findings in countries such as Thailand, an “early-adopter” of risk-based surveillance for HIV. While we do not imagine that similar investments will be forthcoming in the field of medicine quality, we believe that increased national investments in medicine procurement through expanded efforts to achieve universal health coverage will increase the urgency of ensuring that public money is invested in medicines that actually cure patients or prevent disease, rather than in their substandard or falsified doppelgangers. It is, thus, a good time to start building the capacities and systems that will improve the ways in which health systems measure; understand; and, ultimately, curtail the extent and distribution of substandard and falsified medicines.

    Prioritizing Surveillance Approaches for Medicine Quality

    As with infectious disease surveillance, the 2 approaches we have suggested for surveillance of medicine quality—case finding and sentinel surveillance—have different purposes. Case finding aims to pinpoint problems for an immediate response, while sentinel surveillance allows for more reliable quantification of the problem and for the monitoring of the effectiveness of interventions. It is unclear in situations where resources are constrained, which one a regulator should prioritize.

    In the short term, especially, from the point of view of the medicine regulator who will be held responsible if substandard or falsified medicines are shown to harm patients, case finding will probably be the more attractive option. This is true despite the fact that case finding is more data intensive and, likely, more technically challenging to implement, because it requires greater specificity to succeed than sentinel surveillance. Case finding is likely to be the more valuable approach in settings where the regulator is well resourced and where substandard and falsified medicines are comparatively rare.

    Where it is suspected that falsified and, especially, substandard medicines may be rather more prevalent, however, the calculus changes, at least, from a broader public health point of view. Here, a more robust understanding of the extent of the problem, provided by estimates based on sentinel surveillance, may be more valuable. In such settings, which may include many low- and middle-income countries, the prevalence of substandard and, to a lesser extent, falsified medicines will likely have a system-wide effect on health outcomes, as well as on public and private finances. Only after estimating prevalence can one reliably estimate impact. Robust estimates of impact are politically persuasive and may encourage policy makers to change health financing, procurement, and industrial policies in ways intended to erode the factors that incentivize the production and sale of substandard medicines and to shrink the market for falsified products. In addition, a clear understanding of the magnitude of the problem may prove a powerful argument for adequate resourcing for medicine regulators, especially, in lower-income settings.

    We are currently consulting closely with the Indonesian national medicine regulator to trial risk-based case finding, as well as to plan and implement sentinel surveillance and develop national estimates using the methods suggested in this paper. We welcome challenges to our thinking and suggestions to improve the proposed methods, and we look forward to continued debate on the subject.

    Acknowledgments

    AH’s work was supported through a fellowship from the United States Pharmacopeia Quality Institute, which also provided support for supervision to EP and MOK. MOK’s work was also supported by the Netherlands Research Excellence Initiative (REI). These funders have had no part in preparing this manuscript.

    We thank Annie Wang for contributing to initial definitions for case finding. We thank Michael Deats, Sara de Valente, Katharina Hauk, Paul Newton, and Koray Parmaksiz for helpful comments on an earlier draft.

    Authors' Contributions

    The paper was conceived and drafted by EP. EP, SH, AH, YA, and MR, together, developed the case finding indicators. MOK provided supervision for AH. All authors commented on the manuscript and approved the final version.

    Conflicts of Interest

    AH’s work on risk-based surveillance was supported by a research fellowship from the United States of Pharmacopeial Convention (USP) Quality Institute; USP Quality Institute also provided supervision support for EP and MK. In addition, EP, YA, AH, and MR currently receive research funding from UK National Institute for Health Research (GHPSR Project: NIHR131145) to develop and trial sentinel surveillance and estimation methods related to medicine quality. MOK receives research support from the Netherlands Research Excellence Initiative. SH is an employee of the public health division of health care data science firm IQVIA. IQVIA provides data and insights on pharmaceutical market dynamics in over 100 countries. The company contributed research and staff time for exploration of the ideas proposed in this paper at no cost.

    1. 1 in 10 medical products in developing countries is substandard or falsified. World Health Organization. 2017 Nov 28.   URL: https:/​/www.​who.int/​news/​item/​28-11-2017-1-in-10-medical-products-in-developing-countries-is-substandard-or-falsified [accessed 2020-12-21]
    2. Newton PN, Bond KC, Adeyeye M, Antignac M, Ashenef A, Awab GR, et al. COVID-19 and risks to the supply and quality of tests, drugs, and vaccines. The Lancet Global Health 2020 Jun;8(6):e754-e755. [CrossRef]
    3. Willmer, G. New alliance seeks to fight plague of fake medicines. SciDevNet. 2021 Dec 14.   URL: https://www.scidev.net/global/news/new-alliance-seeks-to-fight-plague-of-fake-medicines/ [accessed 2020-12-21]
    4. World Health Organization. Medical Product Alert N°3/2020. WHO Rapid Alerts. 2020.   URL: https://www.who.int/news-room/detail/31-03-2020-medical-product-alert-n-3-2020 [accessed 2020-04-09]
    5. World Health Organization. Medical Product Alert N°4/2020. WHO Rapid Alerts. 2020.   URL: https://www.who.int/news-room/detail/09-04-2020-medical-product-alert-n4-2020 [accessed 2020-04-09]
    6. Report of the fifth meeting of the Member State mechanism on substandard/spurious/falsely-labelled/falsified/counterfeit medical products. World Health Organization. 2017 Jan 17.   URL: https://apps.who.int/gb/sf/pdf_files/MSM5/A_MSM5_8-en.pdf [accessed 2017-02-14]
    7. WHO Global Surveillance Monitoring System for Substandard and Falsified Medical Products. World Health Organization. 2017.   URL: http://www.who.int/medicines/regulation/ssffc/publications/GSMS_Report.pdf [accessed 2018-02-09]
    8. Buckley G, Gostin L. Countering the Problem of Falsified and Substandard Drugs: Committee on Understanding the Global Public Health Implications of Substandard, Falsified, and Counterfeit Medical Products. In: B,Lawrence O. Gostin. Washington DC: Institute of Medicine of the National Academies; 2013.
    9. Jackson KD, Higgins CR, Laing SK, Mwila C, Kobayashi T, Ippolito MM, et al. Impact of substandard and falsified antimalarials in Zambia: application of the SAFARI model. BMC Public Health 2020 Jul 09;20(1):1083 [FREE Full text] [CrossRef] [Medline]
    10. Beargie SM, Higgins CR, Evans DR, Laing SK, Erim D, Ozawa S. The economic impact of substandard and falsified antimalarial medications in Nigeria. PLoS One 2019 Aug 15;14(8):e0217910 [FREE Full text] [CrossRef] [Medline]
    11. Ozawa S, Haynie D, Bessias S. Modeling the Economic Impact of Substandard and Falsified Antimalarials in the Democratic Republic of the Congo. Am J Trop Med Hyg ? 2019;100:57. [CrossRef]
    12. Renschler R, Walters K, Newton P, Laxminarayan R. Estimated under-five deaths associated with poor-quality antimalarials in sub-Saharan Africa. Am J Trop Med Hyg 2015 Jun;92(6 Suppl):119-126 [FREE Full text] [CrossRef] [Medline]
    13. World Health Organization. A study on the public health and socioeconomic impact of substandard and falsified medical products. World Health Organization. 2017.   URL: http://who.int/medicines/regulation/ssffc/publications/Layout-SEstudy-WEB.pdf [accessed 2017-11-28]
    14. Johnston A, Holt D. Substandard drugs: a potential crisis for public health. Br J Clin Pharmacol 2014 Aug;78(2):218-243 [FREE Full text] [CrossRef] [Medline]
    15. McManus D, Naughton B. A systematic review of substandard, falsified, unlicensed and unregistered medicine sampling studies: a focus on context, prevalence, and quality. BMJ Glob Health 2020 Aug;5(8):a [FREE Full text] [CrossRef] [Medline]
    16. Rojas-Cortés R. Substandard, falsified and unregistered medicines in Latin America, 2017-2018. Rev Panam Salud Publica 2020;10:44. [CrossRef]
    17. Nayyar GM, Breman J, Mackey T. Falsified and substandard drugs: stopping the pandemic. Am J Trop Med Hyg 2019:8. [CrossRef]
    18. Ozawa S, Evans DR, Bessias S, Haynie DG, Yemeke TT, Laing SK, et al. Prevalence and Estimated Economic Burden of Substandard and Falsified Medicines in Low- and Middle-Income Countries: A Systematic Review and Meta-analysis. JAMA Netw Open 2018 Aug 03;1(4):e181662 [FREE Full text] [CrossRef] [Medline]
    19. Hodges S, Garnett E. The ghost in the data: Evidence gaps and the problem of fake drugs in global health research. Global Public Health 2020 Mar 31;15(8):1103-1118. [CrossRef]
    20. Garcia-Abreu A, Halperin S, Danel I. Public health surveillance toolkit: a guide for busy task managers. World Bank. 2002.   URL: https:/​/openknowledge.​worldbank.org/​bitstream/​handle/​10986/​20817/​563720WP0REPLA00Box349502B00PUBLIC0.​pdf [accessed 2020-08-20]
    21. Choi BCK. The Past, Present, and Future of Public Health Surveillance. Scientifica 2012;2012:1-26. [CrossRef]
    22. Choi BC, Pak A. Lessons for surveillance in the 21st century: A historical perspective from the past five millennia. Soz Präventivmed 2001 Nov;46(6):361-368. [CrossRef]
    23. Simonsen L, Gog JR, Olson D, Viboud C. Infectious Disease Surveillance in the Big Data Era: Towards Faster and Locally Relevant Systems. J Infect Dis 2016 Dec 01;214(suppl_4):S380-S385 [FREE Full text] [CrossRef] [Medline]
    24. Kirian ML, Weintraub JM. Prediction of gastrointestinal disease with over-the-counter diarrheal remedy sales records in the San Francisco Bay Area. BMC Med Inform Decis Mak 2010 Jul 20;10(1):39 [FREE Full text] [CrossRef] [Medline]
    25. Nsubuga P, White M, Thacker S. Public Health Surveillance: A Tool for TargetingMonitoring Interventions. In: Jamison DT, Breman JG, Measham AR, editors. Disease Control Priorities in Developing Countries. Washington DC: World Bank; 2006.
    26. Schwartländer B, Ghys PD, Pisani E, Kiessling S, Lazzari S, Caraël M, et al. HIV surveillance in hard-to-reach populations. AIDS 2001;15:S1-S3. [CrossRef]
    27. Pisani E, Lazzari S, Walker N, Schwartländer B. HIV surveillance: a global perspective. J Acquir Immune Defic Syndr 2003 Feb;32 Suppl 1:S3-11. [CrossRef] [Medline]
    28. Newton PN, Lee S, Goodman C, Fernández FM, Yeung S, Phanouvong S, et al. Guidelines for field surveys of the quality of medicines: a proposal. PLoS Med 2009 Mar 24;6(3):e52 [FREE Full text] [CrossRef] [Medline]
    29. WHO Expert Committee on Specifications for Pharmaceutical Preparations. Guidelines on the Conduct of Surveys of the Quality of Medicines. WHO Technical Report Series, No. 2016.   URL: http://apps.who.int/medicinedocs/documents/s22404en/s22404en.pdf [accessed 2017-12-07]
    30. Vida RG, Merczel S, Jáhn E, Fittler A. Developing a framework regarding a complex risk based methodology in the evaluation of hazards associated with medicinal products sourced via the internet. Saudi Pharm J 2020 Dec;28(12):1733-1742 [FREE Full text] [CrossRef] [Medline]
    31. General European OMCL Network (GEON). Incorporation of a risk based approach in market surveillance testing at OMCLs. General European OMCL Network. 2007.   URL: https:/​/www.​edqm.eu/​sites/​default/​files/​omcl_incorporation_of_a_rb_approach_in_ms_testing_at_omcls.​pdf [accessed 2020-08-24]
    32. Aroca Á, Guzmán J. Model for a risk-focused approach to health inspection, surveillance, and control in Colombia. Rev Panam Salud Publica Pan Am J Public Health (41) 2017:e105. [CrossRef]
    33. United States Pharmacopeial Convention, Babigumira J, Stegarchis A. A risk-based resource allocation framework for pharmaceutical quality assurance for medicines regulatory authorities in low- and middle-income countries. In: USP Promoting Quality of Medicines. Washington DC: United States Pharmacopeial Convention; Jun 2018.
    34. United States Pharmacopeial Convention, Nkansah P, Smine K. Guidance for implementing risk-based post-marketing quality surveillance in low- and middle-income countries. In: USP Promoting Quality of Medicines. Washington DC: United States Pharmacopeial Convention; 2018:2018.
    35. United States Food and Drug Administration. Predictive Risk-based Evaluation for Dynamic Import Compliance Targeting (PREDICT). FDA. 2014.   URL: https://www.fda.gov/media/83668/download [accessed 2021-04-04]
    36. FDA Center for Drug Evaluation Research. Drug Quality SamplingTesting Programs. FDA. 2021 Mar 02.   URL: https://www.fda.gov/drugs/science-and-research-drugs/drug-quality-sampling-and-testing-programs [accessed 2021-06-09]
    37. Tabernero P, Mayxay M, Culzoni M, Dwivedi P, Swamidoss I, Allan EL, et al. A Repeat Random Survey of the Prevalence of Falsified and Substandard Antimalarials in the Lao PDR: A Change for the Better. Am J Trop Med Hyg 2015 Jun;92(6 Suppl):95-104 [FREE Full text] [CrossRef] [Medline]
    38. World Health Organization. WHO List of Prequalified Quality Control Laboratories, 51st Edition. World Health Organization. 2021.   URL: https://extranet.who.int/pqweb/medicines/medicines-quality-control-laboratories-list [accessed 2021-02-03]
    39. US Pharmacopeia. USP Reference Standards Catalog. USP. 2020.   URL: https://static.usp.org/doc/referenceStandards/dailycatalog.pdf [accessed 2020-12-21]
    40. Pisani E, Nistor A, Hasnida A, Parmaksiz K, Xu J, Kok MO. Identifying market risk for substandard and falsified medicines: an analytic framework based on qualitative research in China, Indonesia, Turkey and Romania. Wellcome Open Res 2019;4:70 [FREE Full text] [CrossRef] [Medline]
    41. Hamilton WL, Doyle C, Halliwell-Ewen M, Lambert G. Public health interventions to protect against falsified medicines: a systematic review of international, national and local policies. Health Policy Plan 2016 Dec;31(10):1448-1466. [CrossRef] [Medline]
    42. Pisani E, Hasnida A, Rahmi M, Harsono S, Kok MO, Anggriani Y. Supplementary material for Pisani et al: Substandard and falsified medicines: proposed methods for case finding and sentinel surveillance. Harvard Dataverse, V2 2021 [FREE Full text] [CrossRef]
    43. Hill AM, Barber MJ, Gotham D. Estimated costs of production and potential prices for the WHO Essential Medicines List. BMJ Glob Health 2018 Jan 29;3(1):e000571 [FREE Full text] [CrossRef] [Medline]
    44. Vickers S, Bernier M, Zambrzycki S, Fernandez FM, Newton PN, Caillet C. Field detection devices for screening the quality of medicines: a systematic review. BMJ Glob Health 2018;3(4):e000725 [FREE Full text] [CrossRef] [Medline]
    45. World Health Organization, UNAIDS. Guidelines on estimating the size of populations most at risk to HIV. In: Guidelines on estimating the size of populations most at risk to HIV. Geneva: World Health Organization; 2010.
    46. Pisani E, Kok M. In the eye of the beholder: to make global health estimates useful, make them more socially robust. Glob Health Action 2017;10(sup1):1266180 [FREE Full text] [CrossRef] [Medline]
    47. UNAIDS, World Health Organization. Case study on estimating HIV infection in a concentrated epidemic: lessons from Indonesia. In: Case study on estimating HIV infection in a concentrated epidemic: lessons from Indonesia. Geneva: UNAIDS; 2004.
    48. Hasnida A, Kok MO, Pisani E. Challenges in maintaining medicine quality while aiming for universal health coverage: a qualitative analysis from Indonesia. BMJ Glob Health 2021 May 28;6(Suppl 3):e003663 [FREE Full text] [CrossRef] [Medline]
    49. Pisani E. The wisdom of whores: Bureaucrats, brothels and the business of AIDS. In: The wisdom of whores: Bureaucrats, brothels and the business of AIDS. London: Granta; 2008.
    50. Mahy M, Brown T, Stover J, Walker N, Stanecki K, Kirungi W, et al. Producing HIV estimates: from global advocacy to country planning and impact measurement. Glob Health Action 2017;10(sup1):1291169 [FREE Full text] [CrossRef] [Medline]

    ‎
    WHO: World Health Organization

    Edited by Y Khader; submitted 01.04.21; peer-reviewed by H Akram, P Nguyen, R Bright; comments to author 13.05.21; revised version received 09.06.21; accepted 27.06.21; published 16.08.21

    Copyright

    ©Elizabeth Pisani, Amalia Hasnida, Mawaddati Rahmi, Maarten Olivier Kok, Steven Harsono, Yusi Anggriani. Originally published in JMIR Public Health and Surveillance (https://publichealth.jmir.org), 16.08.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on https://publichealth.jmir.org, as well as this copyright and license information must be included.