This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on http://publichealth.jmir.org, as well as this copyright and license information must be included.
Estimates of the sizes of hidden populations, including female sex workers (FSW), men who have sex with men (MSM), and people who inject drugs (PWID), are essential for understanding the magnitude of vulnerabilities, health care needs, risk behaviors, and HIV and other infections.
This article advances the successive sampling-population size estimation (SS-PSE) method by examining the performance of a modification allowing visibility to be jointly modeled with population size in the context of 15 datasets. Datasets are from respondent-driven sampling (RDS) surveys of FSW, MSM, and PWID from three cities in Armenia. We compare and evaluate the accuracy of our imputed visibility population size estimates to those found for the same populations through other unpublished methods. We then suggest questions that are useful for eliciting information needed to compute SS-PSE and provide guidelines and caveats to improve the implementation of SS-PSE for real data.
SS-PSE approximates the RDS sampling mechanism via the successive sampling model and uses the order of selection of the sample to provide information on the distribution of network sizes over the population members. We incorporate visibility imputation, a measure of a person’s propensity to participate in the study, given that inclusion probabilities for RDS are unknown and social network sizes, often used as a proxy for inclusion probability, are subject to measurement errors from self-reported study data.
FSW in Yerevan (2012, 2016) and Vanadzor (2016) as well as PWID in Yerevan (2014), Gyumri (2016), and Vanadzor (2016) had great fits with prior estimations. The MSM populations in all three cities had inconsistencies with expert prior values. The maximum low prior value was larger than the minimum high prior value, making a great fit impossible. One possible explanation is the inclusion of transgender individuals in the MSM populations during these studies. There could be differences between what experts perceive as the size of the population, based on who is an eligible member of that population, and what members of the population perceive. There could also be inconsistencies among different study participants, as some may include transgender individuals in their accounting of personal network size, while others may not. Because of these difficulties, the transgender population was split apart from the MSM population for the 2018 study.
Prior estimations from expert opinions may not always be accurate. RDS surveys should be assessed to ensure that they have met all of the assumptions, that variables have reached convergence, and that the network structure of the population does not have bottlenecks. We recommend that SS-PSE be used in conjunction with other population size estimations commonly used in RDS, as well as results of other years of SS-PSE, to ensure generation of the most accurate size estimation.
Having accurate estimates of the sizes of hidden populations, including female sex workers (FSW), men who have sex with men (MSM), and people who inject drugs (PWID), are essential for understanding the magnitude of vulnerabilities, health care needs, risk behaviors, and HIV and other infections. In addition, population size estimations (PSEs) are used to inform resource allocation to develop programs to support sexual health and well-being, counseling and treatment for drug use, to advance social and economic justice, and to respond to and monitor critical health needs and epidemics. However, measuring a hidden population is extremely challenging and current methods contain numerous biases [
Currently, many PSEs of FSW, MSM, and PWID are conducted in conjunction with HIV biobehavioral surveys (BBS) using respondent-driven sampling (RDS) [
RDS is a probability-based sampling method which, when implemented and analyzed correctly, can yield findings representing the network of the population sampled [
One of the PSE methods being commonly used in conjunction with RDS surveys is successive sampling-population size estimation (SS-PSE) [
This article describes the use of SS-PSE in three rounds of BBS, conducted in 2012, 2014, and 2016, among FSW, MSM, and PWID in three cities in Armenia: Yerevan, the capital city (2016 population: 467,087 females and 373,903 males, aged 18 years or older); Gyumri, the second-largest city, located in the northwest of Armenia (2016 population: 49,482 females and 41,535 males, aged 18 years or older); and Vanadzor, the third-largest city, located in the north of Armenia (2016 population: 26,052 females and 28,962 males, aged 18 years or older) [
Respondent-driven sampling (RDS) recruitment chains for selected populations: (a) female sex workers (FSW), Yerevan 2016 and (b) men who have sex with men (MSM), Yerevan 2014. Seeds are indicated by red squares and the waves of recruitment are shown vertically.
Standard RDS methods were used to recruit FSW, MSM, and PWID in 2012, 2014, and 2016 in Yerevan as well as in 2016 in Gyumri and Vanadzor [
The network size question is crucial to RDS studies as a proxy for a person’s propensity to be included in the sample. Participants were asked how many individuals they know who meet the study eligibility requirements and then, as a follow-up, how many of them they have seen in the previous month. An individual’s network size is considered to be the second, more restrictive, of these numbers. For example, the precise question for FSW in Vanadzor was “How many women do you know, whom also know you, are 18 years of age and older, are living in Yerevan, and have exchanged vaginal or anal sex for money or other reward? How many of them have you seen in the past month?”
Population size estimations were conducted using SS-PSE [
The successive sampling model assumes that individuals with a higher degree are more likely to be recruited earlier in the RDS process, since they are more connected and easily accessible in the social network. Thus, if there are fewer large-degree individuals in later waves than earlier waves, this suggests a depletion of members of the population and a large sample fraction; the population is likely not much larger than the sample. However, if the reported degrees stay roughly the same across recruitment waves, the sample size is likely a smaller portion of the population. If the reported degrees increase notably across waves, this may be an indication that the RDS recruitment process is not operating as expected and would merit caution when interpreting the results of various estimators.
The original SS-PSE method relies on self-reported network sizes. However, these values are subject to bias due to heaping or rounding and both intentional and unintentional misreporting; additionally, they may contain missing or impossibly low or high values [
Plots of enrollment date versus self-reported network size for selected populations: (a) female sex workers (FSW), Yerevan 2014, showing a depletion in mean reported degree over the study period; (b) men who have sex with men (MSM), Vanadzor 2016, showing an increasing trend over the study period; and (c) FSW, Yerevan 2016, showing a constant trend over the study period. Note that the magnitude of trends is not comparable across plots due to different reported degree values for the different populations.
Imputed visibility SS-PSE is a Bayesian method, where information about unknown parameters is expressed through probability distributions over their possible values. Thus, the resulting estimates take the form of a distribution called the posterior distribution. We estimate the posterior distribution for the population size N, given our prior belief about the population size and observed data. The prior information used for each of the imputed visibility SS-PSE models of the 15 Armenian datasets was a median, obtained as the average of two medians for that population and city provided by local experts in 2016 through a consensus and extrapolation led by the second author (LGJ) [
We applied the imputed visibility SS-PSE method to 15 datasets of FSW, MSM, and PWID populations from Armenia.
Prior values and quantiles of the posterior distribution for population size obtained from imputed visibility SS-PSE (successive sampling-population size estimation) estimates from 15 datasets of female sex workers (FSW), men who have sex with men (MSM), and people who inject drugs (PWID) populations in Armenia.
Population | Expert values, n | Posterior, n | |||||||
Low^{a} | Median (prior)^{b} | High^{a} | 5% | 25% | Median | 75% | 95% | Assessment^{c} | |
FSW, Yerevan 2012 | 1500 | 3143 | 9900 | 1243 | 2235 | 3734 | 6975 | 16,599 | Great |
FSW, Yerevan 2014 | 1500 | 3143 | 9900 | 397 | 421 | 445 | 469 | 542 | Bad |
FSW, Yerevan 2016 | 1500 | 3143 | 9900 | 784 | 1340 | 2090 | 4169 | 12,924 | Okay |
FSW, Gyumri 2016 | 165 | 351 | 1089 | 126 | 152 | 171 | 217 | 340 | Okay |
FSW, Vanadzor 2016 | 115 | 239 | 759 | 133 | 164 | 205 | 280 | 551 | Great |
MSM, Yerevan 2012 | 2420 | 4202 | 6667 | 943 | 1550 | 2407 | 4335 | 14,296 | Bad |
MSM, Yerevan 2014 | 2420 | 4202 | 6667 | 836 | 1407 | 2299 | 4477 | 17,973 | Bad |
MSM, Yerevan 2016 | 2420 | 4202 | 6667 | 871 | 1121 | 1550 | 2264 | 4870 | Bad |
MSM, Gyumri 2016 | 176 | 306 | 485 | 127 | 176 | 249 | 409 | 974 | Okay |
MSM, Vanadzor 2016 | 123 | 214 | 339 | 323 | 506 | 666 | 836 | 993 | Bad |
PWID, Yerevan 2012 | 1667 | 5842 | 14,473 | 1196 | 2041 | 3236 | 5823 | 17,117 | Good |
PWID, Yerevan 2014 | 1667 | 5842 | 14,473 | 1196 | 2091 | 3435 | 3569 | 19,008 | Good |
PWID, Yerevan 2016 | 1667 | 5842 | 14,473 | 698 | 947 | 1245 | 2091 | 6072 | Bad |
PWID, Gyumri 2016 | 167 | 584 | 1446 | 197 | 354 | 596 | 1141 | 2960 | Great |
PWID, Vanadzor 2016 | 117 | 409 | 1013 | 201 | 318 | 477 | 812 | 1775 | Great |
^{a}The prior low and high values are, respectively, the minimum of the expert prior lows and the maximum of the expert prior highs.
^{b}The prior median is the average of the expert prior medians.
^{c}The
Posterior distributions for the size of selected populations: (a) people who inject drugs (PWID), Vanadzor 2016; (b) female sex workers (FSW), Yerevan 2016; (c) PWID, Yerevan 2016; and (d) men who have sex with men (MSM), Vanadzor 2016. The 90% credible interval is indicated by the shaded blue region and the posterior median by a vertical red line.
Several example posterior distributions are provided in
Each PSE was assessed by comparing the posterior median with the low and high values provided by experts. A
The posterior distributions for population size, like those shown in
Panel (c) PWID, Yerevan 2016, provides an example of a case that is more difficult to interpret. Although the shape of the posterior distribution is acceptable and does not indicate problems with convergence of the SS-PSE method, it is clear that much of the mass from the posterior distribution falls below the prior distribution. This indicates that the PSE is much smaller than the prior median specified. Upon examining these data, we did not observe RDS assumption violations. A possible explanation is that a bottleneck in the underlying social network affected recruitment, making it difficult or impossible to sample from a portion of the population. This means that, in practice, the PSE is only for a subgroup within the overall PWID population. When size estimates are much smaller than experts expect, this could be indicative of a disjoint network, bottleneck, or other reason why only a subset of the population was reachable in the sample. In this case, we advise study officials to return to the formative research study protocol to consider whether any of these scenarios were possible [
In contrast to panel (c), where the estimate was
Overall, many of the point estimates tend to be lower than the expert prior median. This scenario may reflect the reality that RDS surveys may not be reaching the full hidden population, perhaps due to bottlenecks, clustering, or isolated individuals, resulting in a PSE only for the subpopulation that is reachable by RDS.
To place the estimates obtained using imputed visibility SS-PSE in context, we compare the posterior medians to PSEs obtained using service and unique object multiplier methods and wisdom of the crowds for the nine datasets in 2016; we also compare the posterior medians to SS-PSE without visibility imputation for all 15 datasets. The service multiplier method requires two overlapping data sources, including a count of nonduplicated clients accessing a service and a probability-based survey. For these estimations, the service data were unique counts of key populations who received an HIV test between January 1 and June 30, 2016. The PSE is this count divided by the proportion who reported having an HIV test in the probability-based survey (ie, the RDS surveys, also used for the SS-PSE models). Similarly, the unique object multiplier estimate is the number of unique objects distributed to the key population divided by the proportion who reported receiving that object in the probability-based survey. The unique object distributed was a leather bracelet for all populations in 2016, given out one week prior to the start of the survey by outreach workers. Multiplier methods rely on several assumptions, including that no individual is counted more than once in each multiplier, that there is limited in-and-out-migration, that the two data sources are independent of each other, and that the probability-based survey is representative of the hidden population. Wisdom of the crowds assumes that, in aggregate, the responses of a sufficient number of key population members about the size of their population will provide a good estimate of the actual size of their population. Participants in the RDS survey were asked for their best guesstimate on the population size and the average was computed.
The data considered included three rounds—2012, 2014, and 2016—of BBS for FSW, MSM, and PWID in Yerevan, Armenia. Imputed visibility SS-PSE models were fit for each year using the same prior median population size for each population, based on consultation with local experts. We compared the size estimates, descriptively, over these three years for each population. We present a visual inspection of trend in population size over time, as three years of data are not enough to do a time series analysis and a hypothesis test for equality depends on assumptions that may not be met by the RDS sampling process.
Comparison of imputed visibility SS-PSE (successive sampling-population size estimation) posterior medians with other population size estimation methods for the 15 Armenia datasets.
Population | Expert values (n) | Object multiplier | Service multiplier | WOC^{a} (best mean) | SS-PSE median (no visibility) | SS-PSE median (visibility) | ||
Low | Median | High | ||||||
FSW^{b}, Yerevan 2012 | 1500 | 3143 | 9900 | N/A^{c} | N/A | N/A | 2041 | 3734 |
FSW, Yerevan 2014 | 1500 | 3143 | 9900 | N/A | N/A | N/A | 469 | 445 |
FSW, Yerevan 2016 | 1500 | 3143 | 9900 | 571 | 1283 | 1615 | No fit^{d} | 2090 |
FSW, Gyumri 2016 | 165 | 351 | 1089 | 150 | 92 | 196 | 277 | 171 |
FSW, Vanadzor 2016 | 115 | 239 | 759 | 204 | 156 | 67 | 275 | 205 |
MSM^{e}, Yerevan 2012 | 2420 | 4202 | 6667 | N/A | N/A | N/A | No fit | 2407 |
MSM, Yerevan 2014 | 2420 | 4202 | 6667 | N/A | N/A | N/A | No fit | 2299 |
MSM, Yerevan 2016 | 2420 | 4202 | 6667 | 749 | 8300 | 11,900 | 1121 | 1550 |
MSM, Gyumri 2016 | 176 | 306 | 485 | 3659 | N/A | 138 | 168 | 249 |
MSM, Vanadzor 2016 | 123 | 214 | 339 | 150 | N/A | 40 | No fit | 666 |
PWID^{f}, Yerevan 2012 | 1667 | 5842 | 14,473 | N/A | N/A | N/A | 1245 | 3236 |
PWID, Yerevan 2014 | 1667 | 5842 | 14,473 | N/A | N/A | N/A | 1743 | 3435 |
PWID, Yerevan 2016 | 1667 | 5842 | 14,473 | 9000 | N/A | 19,342 | 997 | 1245 |
PWID, Gyumri 2016 | 167 | 584 | 1446 | 3000 | 6800 | 26 | No fit | 596 |
PWID, Vanadzor 2016 | 117 | 409 | 1013 | 3000 | 7000 | 198 | No fit | 477 |
^{a}WOC: wisdom of the crowd estimates.
^{b}FSW: female sex workers.
^{c}N/A: not applicable. Information was not collected at the time the study was implemented that would enable calculation of these values.
^{d}SS-PSEs without visibility imputation where the value is
^{e}MSM: men who have sex with men.
^{f}PWID: people who inject drugs.
Therefore, even if the size of, for example, the Yerevan MSM population remained exactly constant over the period from 2012 to 2016, we would expect to get different estimates each year due to sampling. We used the overall variability of the estimates, indicated by the 90% credible intervals, to assess how unusual any particular year’s estimate was, and if it was actually indicative of a trend.
Panels (b) and (c) in
Trend analysis plots showing the prior and posterior distributions for 2012, 2014, and 2016 biobehavioral surveys in Yerevan of (a) female sex workers (FSW), (b) men who have sex with men (MSM), and (c) people who inject drugs (PWID). The median population size estimate is indicated by a red point for each year and the prior median for all three years is shown as a horizontal dotted line.
Imputed visibility SS-PSE provides an estimate for the size of a hidden population using data already routinely collected in an RDS survey. Unlike many other PSE methods, imputed visibility SS-PSE relies on only one data source and can therefore be performed retroactively, after an RDS study has already been conducted. Further, the visibility imputation modification allows for potentially erroneously self-reported network sizes to be modeled, making the method more robust to misreporting, missing values, and extreme values. However, given the difficulty measuring a hidden population and the potential for biases at many levels, including undetected bottlenecks in the network structure of the population, problems with the RDS sample, and errors in the prior size estimations, some estimates may not make sense. It is therefore always important to assess the quality of the PSE, rather than treating it as innately correct. Diagnostic plots, such as the plots of social network size by enrollment date, are useful tools to assess RDS and SS-PSE assumptions. The posterior distribution should also be examined to assess possible issues with model fit, which could be indicated by a flat distribution or one with a spike at large values of N.
When fitting the imputed visibility SS-PSE model, prior belief about the population size is specified. For these 15 datasets we used the prior median, as this was the information available. It is also possible to use the first and third quartiles or other distribution summary measures, based on available knowledge. Prior values should be ascertained before fitting the model and not altered when an estimate does not make sense, in order to avoid introducing bias from the researcher. Instead, when the posterior distribution has the appropriate shape, but the estimate does not align with expert knowledge, it is advisable to engage additional stakeholders and examine the study in more detail. In this exercise, we found that the MSM populations in all three cities considered had inconsistencies with the expert prior values provided. The maximum low prior value was larger than the minimum high prior value, making a
To examine the sensitivity of the imputed visibility SS-PSE model fits to the choice of prior median, we fit each model with three different prior medians: the average of two expert values and each expert value individually. The average of the two expert values was the final prior median used for the models presented in the Results section. Using the other prior medians does not drastically change the PSE. Although the point estimates are slightly larger for the larger prior median and slightly smaller for the smaller prior median, the values are very similar given the overall variability of the distribution. Superimposed posterior distributions for SS-PSE fits using these three prior medians for each of the 15 datasets are provided in
Evaluating the results from the imputed visibility SS-PSE, as well as other PSEs used in conjunction with RDS (ie, unique object and service multipliers, wisdom of the crowds), is essential given that they are prone to biases, which may lead to unrealistic over- and underestimations. Many size estimation techniques can be used as part of each survey to triangulate and validate the most optimal size estimation [
The imputed visibility SS-PSE method of PSE can be used with existing RDS data sources to obtain reasonable estimates when benchmarked against prior expert knowledge. We demonstrate the performance of this method on 15 datasets of FSW, MSM, and PWID populations from three waves of BBS studies conducted using RDS from three cities in Armenia. This is the first assessment of the modification to the imputed visibility SS-PSE methodology on such a large variety of data and the first to consider trend analysis for the same population over three time points. This is also the first presentation of how to interpret different outputs from SS-PSE in real data. These studies cover a variety of recruitment structures and sizes coming from nine different underlying social networks. The results provide examples of good model fits, where the RDS assumptions appear to be satisfied and the resulting posterior distributions are of the appropriate shape, and bad model fits, where the RDS assumptions appear to be violated in diagnostic plots or the posterior distributions depart greatly from expert opinions. We find that the imputed visibility SS-PSE model performs favorably compared to other PSE methods for these populations; these other methods have no basis on which to assess bias and often give impossibly large or small estimates or produce no estimate at all. Because SS-PSE does not rely on data from multiple surveys or census information, it is a valuable method of PSE. However, there are limitations to its use. If RDS assumptions are violated or there are issues with convergence in the model, results from SS-PSE should be interpreted with caution. To this end, we also provide guidance and suggested methods for goodness of fit to assess the SS-PSE methodology and the overall quality of the estimates. We recommend that SS-PSE be used in conjunction with other PSE techniques commonly used in RDS to ensure generation of the most accurate and acceptable size estimation.
Diagnostic and sensitivity plots.
Armenian Dram
biobehavioral surveys
female sex workers
men who have sex with men
population size estimation
people who inject drugs
respondent-driven sampling
successive sampling-population size estimation
The Joint United Nations Programme on HIV/AIDS
wisdom of the crowd estimate
We would like to thank everyone who participated in the surveys. In addition, we would like to thank Mark Handcock, University of California, Los Angeles, co-founder of the Hidden Population Methods Research Group and co-developer of the successive sampling population size estimation, for his consultation on this manuscript. We also want to thank Seda Abgaryan, Lilith Hovhannisyan, Ruben Hovhannisyan, and Tigran Hovsepyan of the National Center for AIDS Prevention, Yerevan, Armenia.
None declared.