A convenience sample of SGM was recruited during September 2020 to complete an online survey through advertisements on two social networking apps (Grindr and Hornet). Grindr is a location-based social networking and online dating app launched in 2009 that has since become the largest and most popular gay mobile app in the world. Hornet, another location-based social networking and online dating platform, was launched in 2011 and is available as an app and on the web. For Grindr, advertisement banners were randomly displayed to users for 2 weeks. Hornet users received 1 inbox message with a link to the survey. Participants needed to provide electronic informed consent before initiating the survey. No compensation was provided, and no personally identifiable information was collected except for IP address. Participant eligibility included age 18 years and older and residency in Brazil. Exclusion criteria were self-identifying as a cisgender woman and an incorrect response to any of 3 attention questions that were included throughout the survey instrument at approximately every 15 items [25]. A full version of the survey instrument is provided in Multimedia Appendix 1.

This study was approved by the National Institute of Infectious Diseases Programa Inova at the Oswaldo Cruz Foundation (FIOCRUZ) institutional review board (#CAAE 01777918.0.0000.5262) in accordance with all applicable regulations.

The survey was programmed on Alchemer survey software. The survey was in Portuguese and contained 55 questions, with certain questions conditionally presented using branching logic. Survey links remained active for 1 month. Respondents were able to change and review answers. Four authors systematically checked the usability and technical functionality of the electronic questionnaire on different platforms and operating systems before starting the survey.

The survey instrument was divided into 3 sections. Section 1 included items on sociodemographic information (age, gender, sexual orientation, race/skin color, education, family monthly income, and state of residence). Section 2 included items referring to prior HIV testing and HIV test results. HIV-negative and unknown-status participants were questioned about preexposure prophylaxis (PrEP) use (current, never, or past), and those who were not currently using PrEP responded to 2 questions about sexual behavior during the previous 6 months (condomless anal sex and condomless receptive anal sex). HIV-positive participants were questioned about use of antiretroviral treatment (ART), and if in use, adherence (prior 7 days) was measured using the 3-item WebAd-Q instrument [26]. Section 3 included items of 3 HIV knowledge measures: HIV-KA [11], HIV-KQ [7], and WHO-KI [13]. These 3 measures use the same response format, which includes the options “true,” “false,” and “I don’t know.” The total score of each instrument was calculated by summing across all items that the participant answered correctly (“I don’t know” was coded as an incorrect response). Higher scores reflect greater knowledge. We also included a question about perceived accuracy of the Undetectable Equals Untransmittable (U=U) slogan [27-29]. Individuals were able to provide an email address if they wished to receive the correct answers to HIV knowledge questions 1 month after the study was terminated.

HIV-KA [11] is a 12-item measure previously used to assess knowledge among 4176 MSM in 12 Brazilian cities with a respondent-driven sampling methodology used for recruitment. The tool was evaluated using item response theory with difficulty and discrimination parameters estimated by marginal maximum likelihood and the knowledge score (theta) estimated by the expected a posteriori method based on Bayesian statistical principles.

HIV-KQ [7] is an 18-item measure evaluated for its psychometric properties among low-income US adults (n= 1019). Results indicated strong levels of internal consistency and test-retest stability. For this study, we excluded 3 items: one was considered not relevant to the Brazilian context (“A natural skin condom works better against HIV than does a latex condom”), and 2 were deemed less relevant for SGM populations (“A woman cannot get HIV if she has sex during her period” and “There is a female condom that can help decrease a woman’s chance of getting HIV”).

WHO-KI [13] is a 5-item measure developed to assess progress in building knowledge of essential facts about HIV transmission among key populations. It is endorsed by the WHO and recommended as a tool to monitor key populations’ knowledge.

We assessed whether respondents correctly perceived the accuracy of the prevention benefits of U=U through the question “With regard to HIV-positive individuals transmitting HIV through sexual contact, how accurate do you believe the slogan Undetectable=Untransmissible is?” as used in previous studies [27-29]. Response options were based on a Likert-type scale from 1 (completely inaccurate) to 4 (completely accurate) plus a fifth option (I don’t know what “undetectable” means).

Descriptive statistics of the study population are provided. We randomly split the study population in half and assessed the factor structure of all items of the HIV-KA together as a single measure with exploratory factor analysis (EFA) in the first half followed by confirmatory factor analysis (CFA) in the second half. EFA was used to identify the number of factors and assess item factor loadings. EFA was performed using robust weighted least squares estimator given the categorical nature of the survey items, chi-square test statistic, and geomin oblique rotation [30]. A Cattell scree test on the sedimentation graph was examined. The number of factors was chosen based on the scree plot (eigenvalues), model adequacy, and overall interpretability. CFA used a weighted least squares estimator with a diagonal weight matrix, robust standard errors, and a mean- and variance-adjusted chi-square statistic with delta parameterization [31]. To assess model fit, the chi-square test, Tucker-Lewis Index (TLI) [32], comparative fit index (CFI) [33], root mean square error of approximation (RMSEA) [34], and standardized root mean residual (SRMR) [33] were used. Since the chi-square test is highly sensitive to sample size, it can lead to the rejection of well-fitting models [35]. Therefore, the TLI, CFI, and RMSEA fit indices were emphasized. Good fitting models may be indicated by a TLI and CFI ≥ 0.95, RMSEA ≤ 0.06, and SRMR < 0.08 [36].

Then, with the whole sample, a generalized partial credit item response theory model (GPCM) [37] was fit to all 12 items of the HIV-KA. The GPCM estimates 2 types of parameters for each item: a threshold parameter, which measures the level of knowledge at which people are more likely to answer the question correctly than incorrectly, and a discrimination parameter, which measures the strength of the association between that item and the underlying construct (HIV knowledge). From these item-level parameters, item information functions were estimated for each item and summed pointwise to obtain the test information function (TIF). The TIF measures the total amount of Fisher information in each item and is inversely related to the standard error of measurement of the underlying construct—that is, greater precision in the measurement of the underlying construct [38].

We used methods described by Harel et al [24]. Briefly, OTA systematically explores the space of all possible shortened versions of a fixed length to optimize the height of the TIF, thus minimizing the standard error of measurement of the underlying construct [24,39,40]. Here, for each possible length of shortened form (3 to 11), the OTA procedure created a candidate shortened version of the HIV-KA. Forms of length 1 and 2 were not generated because a minimum of 3 items are needed for the single-factor model to be identifiable [41]. Based on previously established guidelines [24], the OTA procedure was anchored at 5 points across the spectrum of the underlying construct (–5, –3, –1, 1, 3), jointly maximizing the shortened form’s TIF at these points [17].

Each of the candidate short forms and the full-length form were scored using two procedures to obtain estimates of each participant’s level of HIV knowledge. First, the summed scores across all items included in the form were calculated by adding item scores for each item included in the form. Second, factor scores, which estimate a level of a latent construct, were estimated from the GPCM for each participant for each form through an application of Bayes theorem. Both summed scores and factor scores were used due to the reliance on the former in research and the improved measurement properties of the latter [42,43].

Resulting from the OTA procedure are candidate shortened forms, each with the applicable optimal items. Removal of items implies a reduced amount of test information as compared to the full-length form. The selection of the final form was based on 5 criteria: reliability, concurrent validity based on summed scores, concurrent validity based on factor scores, convergent validity based on summed scores, and convergent validity based on factor scores. Applying these 5 criteria concurrently ensured that the final selected shortened version maintained desirable measurement properties across these categories.

Accordingly, we first generated the candidate shortened forms and assessed each form’s reliability against the full-length form using a Cronbach alpha coefficient [44]. The shortened version was required to maintain at least 95% of the value of Cronbach alpha for the full-length form. Second, we estimated concurrent validity for both summed and factor scores by calculating a Pearson correlation coefficient between the scores on each candidate shortened version and the scores on the full-length form. For both the summed and factor scores, these correlations were required to be at least 0.90, ensuring that the shortened version demonstrated high concurrent validity.

Next, for convergent validity, we used two different criteria based on 2 HIV knowledge measures (HIV-KQ and WHO-KI, Multimedia Appendix 2, Figure S1), and, in so doing, created 2 potential shortened forms—one validated against the HIV-KQ and the other validated against the WHO-KI. We assessed convergent validity through the correlation between participant scores on short forms and the 2 HIV knowledge measures. The candidate short forms were required to demonstrate statistical equivalence with the convergent validity of the full-length of each measure through an application of equivalence testing. Equivalence testing assesses whether the difference between 2 correlations is within a prespecified range, in this case set at .05 [45]. To assess statistical significance, we applied the Benjamini-Hochberg correction procedure for each of the hypothesis tests used (candidate shortened versions × 2 scoring procedures) [46].

Finally, post hoc convergent and concurrent validity of the shortened forms were evaluated. We calculated the mean scores and 95% confidence intervals when using the 2 candidate shortened forms according to the participant’s response to the perceived accuracy of the U=U slogan (dichotomized into completely accurate vs partially accurate, inaccurate, completely inaccurate, or “I don’t know what undetectable means”). Additionally, we calculated mean scores with 95% confidence intervals of the candidate shortened forms according to select variables. As observed in a prior study [11], we hypothesized that the scores of the shortened forms would be associated with age, race, education, and income. We also hypothesized that participants reporting access to HIV prevention services, measured by prior HIV testing [47] and PrEP use, would score higher in the shortened forms of the HIV-KA.

All analyses were conducted in R software (version 4.0.2, R Foundation for Statistical Computing) [48]. The GPCM was fit using the ltm package [49], and the OTA analysis was conducted using the lpSolveApi package [50].

Of 3368 participants who initiated the survey, 2552 answered all items of the 3 HIV knowledge measures of interest and were included in analyses (Multimedia Appendix 2, Figure S2 and Table S1). Mean age of study participants was 35.1 years, with the overwhelming majority being cisgender men self-identified as gay or bisexual. Most participants self-reported as White, had a college education or higher, and were earning more than US $400 per month. Most (1510/2552, 59.2%) reported prior HIV testing and a negative HIV status; 31.0% (792/2552) self-reported as HIV positive. The vast majority (785/792, 99.1%) of HIV-positive participants had initiated ART, and, among these, most (472/785, 60.1%) were ART-adherent. Most (1534/1760, 87.2%) of HIV-negative or unknown participants were not currently using PrEP and reported condomless anal sex in prior 6 months (Table 1).

The EFA of the 12-item HIV-KA (EFA sample; n=1276) yielded 1 eigenvalue of factor greater than 1 (Factor 1 eigenvalue 2.04). Based on examination of the scree plot, we judged that a 1-factor solution provided the most interpretable model. Results from the CFA (CFA sample; n=1276) indicated that a 1-factor structure showed reasonably good fit: χ²₆₆=1352.1, P<.001; CFI=0.94; TLI=0.93; RMSEA=0.03; SRMR=0.11. The item loadings ranged from 0.18 (item 8) to 0.75 (item 5).

The GPCM was fit on the 12 items of the HIV-KA. Item content with the discrimination parameters estimated from the GPCM are provided in Table S2 of Multimedia Appendix 2. The 3 items with the highest amount of discriminative ability and, therefore, the most influential on the TIF were items 2, 4, and 5. The items with the least amount of discriminative ability and, therefore, the least influential on the TIF were items 8, 9, and 11. Individual item information functions generated from the estimates of the GPCM and the test information function for the full-length form and the 2 short forms are provided in Figures S3 and S4, respectively, of Multimedia Appendix 2.

Short forms with at least 8 items had Cronbach α=.59 or higher, suggesting a moderate level of reliability compared with full-length (for which Cronbach α=.64, Table 2). Short forms with the applicable items resulting from the OTA procedure are shown in Table S3 of Multimedia Appendix 2. Concurrent validity correlations were high for all short forms with 6 items or more based on factor score correlations (Table 2). For convergent validity, all versions with at least 9 items demonstrated statistically significant equivalency for the correlations between the summed and factor scores with the HIV-KA (Table 3). The 9-item shortened form was the shortest candidate version to fulfill our requirements when validating against the HIV-KQ for the equivalency analysis, while the 10-items shortened form was the shortest candidate when validating against the WHO-KI.

The selected short forms include, for example, the items addressing PrEP (item 1: “There are medications for HIV-negative people to take before having sex with other people to prevent HIV infection”), treatment as prevention (item 2: “An HIV-infected person who is taking HIV/AIDS medications has a lower risk of transmitting the virus to another person”), and postexposure prophylaxis (item 4: “There are medications for HIV/AIDS to be used after a situation of risk of infection [ie, unprotected sex, sexual violence]”; Table 3). The 3 items that were consistently dropped from the selected short versions were item 8 (“When having intercourse with only one faithful partner, not infected with HIV, the risk of contracting the virus is lower”), item 9 (“There is a cure for HIV”), and item 10 (“A healthy-looking person may be infected with the HIV virus”).

HIV knowledge scores using both the 9-item and 10-item short forms were higher among participants who perceived the U=U slogan as completely accurate versus not accurate (Table 4). Participants of younger age, of Black, Pardo, or indigenous race, and reporting lower education and lower income scored lower on HIV knowledge. Participants who never tested for HIV scored lower than those who tested negative or positive, while those currently using PrEP scored higher than those reporting past or never use. Last, knowledge scores were very similar for those reporting condomless receptive anal sex or not as well as among those reporting as ART-adherent or not.

In this study, we used a novel OTA method to generate valid short-form measures of the HIV-KA. The OTA procedure generated 9-item and 10-item shortened forms that satisfied our prespecified criteria in terms of reliability, concurrent validity, and convergent validity. These versions maintained high reliability and high concurrent validity with the full-length form, as well as statistically equivalent convergent validity correlations with HIV-KQ and WHO-KI. Our results indicate that 9-item and 10-item HIV-KA versions could be used among Brazilian SGM to assess HIV knowledge. Nonetheless, unless the number of items is a critical issue for a particular study, we argue that the shortened forms were not short enough (only shortened by 2 to 3 items) to justify recommendation as these would not be directly comparable to existing studies that have used the full-length version.

The OTA method is a replicable method that maintains performance standards based on objective criteria and, as such, selected short forms can be said to fulfill prespecified reliability and concurrent and convergent validity. The reliability of the shortened forms was of the same magnitude as the full-length form while showing high concurrent validity with the full-length form. Statistically equivalent convergent validity correlations using 2 HIV knowledge measures (HIV-KQ and WHO-KI) were also shown for both summed and factor scores.

The selected short forms included the items addressing recent paradigms of HIV prevention and treatment, such as preexposure prophylaxis, treatment as prevention, and postexposure prophylaxis. The 3 items removed from the short forms showed the lowest discriminative ability, indicating that they were less useful in the construction of the HIV knowledge score. Notwithstanding, semantic aspects of the items may have impacted the results. Particularly, for item 8, the item with the lowest discriminative ability, the wording may have caused confusion. Item 8 states that the risk of infection is lower when it would have been more appropriate to say that the risk is null (“When having intercourse with only one faithful partner, not infected with HIV, the risk of contracting the virus is lower”). Moreover, the use of the word lower suggests that it is lower than some other situation that the item does not specify. For example, compared to complete sexual abstinence, the risk may be higher, although it is lower compared to other sexual behaviors.

Our results also indicate post hoc convergent validity for the shortened forms using the 1-item measure on the perceived accuracy of U=U, a slogan launched in 2016 by the Prevention Access Campaign to translate scientific evidence into a community message that highlights how people living with HIV on antiretroviral treatment with suppressed viral load cannot transmit HIV to their sexual partners [51]. The scientific evidence supporting U=U has accumulated over the past decade and is contingent on the body of knowledge showing effectiveness of treatment as prevention, in which the use of ART among people living with HIV reduces HIV transmission yielding public health as well as personal health benefits [14]. The observed convergent validity thus reinforces the value of the proposed shortened forms to measure HIV knowledge among SGM. Of note, mean HIV knowledge score was high considering all 3 measures, and the proportion of individuals perceiving U=U as accurate was higher than observed in a 2019 survey conducted among Brazilian gay, bisexual, and other MSM [29]. This may indicate that perceived accuracy of the U=U slogan actually increased among SGM from Brazil over time, an important positive finding as understanding the accuracy of U=U empowers those living with HIV, improving treatment adherence, and decreasing HIV-related stigma [52]. Additionally, it may also enhance scale-up of PrEP, which is available at no cost through the Brazilian public health system. However, there are significant sociodemographic differences between the sample populations, with the current sample having more participants from the South/Southeast of Brazil and with higher education and income. As such, these differences may also play a role explaining the increased perceived accuracy of U=U.

Although only slightly shorter than the original 12-item HIV-KA measure, the 9-item or 10-item shortened forms may be preferred as they reduce participant burden, which is particularly important for participants who may have difficulty completing self-reported questionnaires. As observed in this study (Multimedia Appendix 2, Table S1) and in previous online surveys conducted by our group among SGM, those aged 18 to 24 years reporting lower income and lower education who had never been tested for HIV were more likely to not complete the survey [16]. Future studies should assess whether using the 9-item or 10-item shortened forms ultimately reach SGM populations that are more sociodemographically diverse. The number of HIV cases among young gay, bisexual, and other MSM continues to rise in Brazil [53] and, although scarce, age-dependent HIV incidence rate estimates also show that younger gay, bisexual, and other MSM are the most vulnerable [54]. In this regard, the unbiased representation of gay, bisexual, and other MSM on surveys addressing HIV transmission knowledge and sexual behavior is paramount to improve and aid development of prevention campaigns to these groups. Furthermore, beyond objective knowledge, we echo the recent call for promoting prevention literacy, whereby knowledge of the multiple prevention modalities is promoted to allow individuals to make the decisions that are optimal for their health while also promoting community advocacy and mobilization [55]. Shorter questionnaires with accessible and appealing language constructed with community participation may increase completion rate, and, consequently, the value of the collected information particularly as applicable to vulnerable groups.

Ad hoc construct validity of the HIV-KA shortened forms showed that those of younger age, non-White race, reporting lower education and lower income scored lower on HIV knowledge. A study from Brazil on 4129 MSM recruited by respondent-driven sampling in 12 Brazilian cities in 2016 observed that not only was the prevalence of unprotected receptive anal intercourse higher among younger participants, they scored lower on HIV knowledge and were less likely to have been tested for HIV in the past, despite having more years of schooling [47]. This lower HIV knowledge may be a contributing factor to higher vulnerability to HIV infection. Indeed, multiple studies using different measures of HIV-related knowledge have shown a link between testing and knowledge with a gradient of increased knowledge as you move from the categories of never tested to HIV-negative and HIV-positive [28,29,56]. One hypothesis for this finding is that exposure to the health care setting and counseling during testing may increase HIV-specific knowledge. This rationale could also possibly explain why PrEP users who routinely have to access health services to refill their prescriptions also scored higher. That said, it is impossible to determine temporality and it may well be that those who are more knowledgeable about HIV are also more likely to get tested or use PrEP. We observed no correlation between HIV knowledge scores and report of condomless receptive anal sex, a finding also reported previously [56]. Future studies could explore motivations for engagement in high-risk sexual behavior and perhaps how HIV knowledge could help inform safer sexual practices.

There are several study limitations that must be considered. This study used cross-sectional data, and therefore the sensitivity to change and test-retest reliability of the HIV-KA short forms could not be assessed. All collected data were self-reported by participants and may be subject to measurement errors that can arise in the collection, recall, or recording of information. Participants were recruited from a convenience online sample and may not reflect other SGM populations in Brazil. As for any study design, online samples have strengths that should be acknowledged which include the reaching geographically diverse populations as well as individuals from remote regions and those completely disconnected from HIV prevention services [57]. The challenges include the need for testing of survey instrument on a variety of hardware devices and software platforms, for effective means of advertising to diverse populations and to maintain participant anonymity, among others [57]. Our participants were mostly cisgender, and additional studies should include a greater representation of transgender and nonbinary individuals. Additionally, the survey was advertised as about HIV knowledge and this may have influenced participant selection, possibly overrepresenting those already living with HIV. The OTA procedure is sensitive to the investigator-defined choice of decision criteria in the selection of the final shortened version. These decision criteria, when applied in future studies, must be carefully considered by researchers. Furthermore, the OTA method treats the 12 items of the HIV-KA as if they represented a full item bank of possible items. It is possible that if other items were considered than a different set of items would have been selected into the final short form. Lastly, this analysis should be replicated in other samples of SGM populations, as well as other populations, to increase the generalizability and to confirm that the selected short forms are optimal for other populations.

In conclusion, this study showed how OTA methods might be used to shorten the 12-item HIV-KA to 9-item and 10-item versions while maintaining comparable reliability and validity among a large sample of Brazilian SGM. While OTA was primarily used for the development of high-stakes educational testing, it has now been used, as well, to successfully shorten patient-reported outcome measures in several patient populations. This study is the first, to our knowledge, to use OTA to shorten a knowledge assessment tool. Although the shortened forms in this study did not represent substantial reductions in items, OTA represents an important methodology as we attempt to maximize the information we collect in the least burdensome way possible, which can be supported by reducing the number of items in surveys.