This instrument development and descriptive study involved the development of an HLS-MW and the evaluation of the validity and reliability. The study included 2 phases—instrument development and instrument validation.

We developed the HLS-MW based on Baker’s health literacy framework [24]. According to this framework, health literacy involves one’s skills in different dimensions, such as health-related print (ie, documents, prose, and numeracy) and oral literacy. Within our HLS-MW scale, we focused on health-related print literacy—documents (ie, understanding forms and tables) and prose (ie, understanding health-related terms presented in sentences). The scale did not include numeracy as a separate dimension. Numeracy refers to the application of arithmetic operations and the understating of quantitative information [24]. We considered numeracy as a part of the “document” dimension in that it operationalizes one’s ability to understand written “quantitative” information. Given that the HLS-MW was not specific to a certain disease (eg, diabetes) and was intended to be used to assess general health literacy in migrant workers, we decided that there was a limited need to address specific arithmetic skills.

We reviewed quotes and themes from a qualitative study involving migrant workers to identify relevant venues in which migrant workers experience and encounter health literacy tasks in South Korea [20]. We also examined other empirical referents, such as health education materials, existing health literacy instruments [25-27], textbooks on the EPS—Test of Proficiency in Korean (EPS-TOPIK), resources for immigrants provided by the National Institute of Korean Languages, and relevant studies on immigrants’ health care in South Korea [20,28,29]. We extracted 20 passages and 709 health-related terms for migrant workers from these sources.

Document literacy included medical forms commonly used when visiting a clinic, such as appointment instructions for outpatient and lab visits. We used existing health literacy scales written in Korean [26,27] to modify the items that were salient to migrant workers, such as adapting items into plain language use based on word difficulty [30]. For prose literacy, the initial pool of 709 terms was categorized based on their characteristics (ie, body-related terms, disease- and symptom-related terms, treatment-related terms, and healthy life and miscellaneous) and difficulty (ie, beginner, intermediate, and advanced) to develop 35 items [30].

To examine the content validity of the 229 terms, a review was carried out by a panel comprising 7 health professionals, including 3 nurses, a social worker, a pharmacist, and 2 doctors, with at least 3 years of experience as volunteers at a free clinic for migrant workers.

Using a 4-point scale, they were asked to rate each document and prose item for its appropriateness (1=not appropriate to 4=very appropriate) and importance (1=not important to 4=very important). The content validity index (CVI) was 0.76 (range 0.43‐1), and any item with a CVI <0.7 was modified or removed [31], resulting in a final set of 28 items (7 document literacy items and 21 prose items). These items comprised 89 health-related terms embedded across 11 passages that addressed checking in at the clinic desk or doctor’s office and instructions for the next appointment. Subsequently, we conducted a field test with 10 migrant workers to confirm this initial version’s readability, understandability, and cultural relevance before proceeding with a validation process for psychometric testing. The process of the HLS-MW development is presented in Figure 1.

We used web-based sampling to recruit participants for the validation study. First, in collaboration with the Support Center for Foreign Workers in Daegu, South Korea, we identified 6 migrant community leaders who could speak fluent Korean, representing 6 South Asian countries—Indonesia, Vietnam, Nepal, Cambodia, Pakistan, and the Philippines. Subsequently, we held a 1-hour briefing session with these 6 community leaders at the Daegu Center, where we explained the study’s purpose and how we planned to recruit study participants. The 6 leaders were granted access to a website written in their own language and asked to check the survey’s logic and any errors. If no errors were identified, each leader was provided with a link to the study survey written in their own language to recruit their peers in South Korea by sharing the link through mobile phone texts or social media between August 19 and September 2, 2021. Upon clicking the study link, potential participants were presented with a brief eligibility screening questionnaire, and after verifying their eligibility, eligible migrant workers were directed to a consent script followed by the survey. In addition to the HLS-MW, the web-based survey included sociodemographic (eg, nationality, age, sex, education, job, length of stay in South Korea, type of visa, health insurance, and Korean language proficiency) and medical status questions (eg, health problems and use of medical services in Korea). Participants’ Korean language ability was self-rated on a 4-point scale (0=not proficient to 3=proficient) across four areas (ie, reading, writing, listening, and speaking).

A total of 402 migrant workers residing in South Korea responded to the web-based survey. Inclusion criteria encompassed (1) being previously or currently employed through the EPS for migrant workers, (2) having lived in South Korea for at least 6 months, (3) providing informed consent to participate in this study using their phone number, and (4) being able to read Korean. Potential participants were excluded if they were (1) international students, (2) earning an income from temporary economic activities on a tourist visa, or (3) Koreans living abroad (H-2 visa holders), since their Korean language proficiency, status of residence, and employment standard and procedures differed from those of unskilled migrant workers.

Of the 40 workers who completed the study survey and were asked to complete the HLS-MW again 2 weeks later, 23 complied (response rate of 57.5%).

We undertook a preprocessing step to identify and remove incomplete or invalid data, excluding 68 out of 402 surveys that were completed in less than 3 minutes. The 3-minute cutoff was based on the time it took for 5 native Korean workers with similar ages, education levels, and job categories to complete the study survey. Consequently, the final analytic sample included 334 complete surveys. The HLS-MW comprised 35 true-or-false items. An incorrect answer for each item was assigned a score of 0 and a correct answer was assigned a score of 1.

A descriptive analysis was conducted to summarize participants’ demographics and health literacy scores. Exploratory factor analysis (EFA) was performed via the Mplus 8.2 program (Muthén & Muthén). Weighted least square mean and variance adjusted (WLSMV) was determined using M-plus. The WLSMV approach is generally applied for dichotomous variables [32]. To determine the number of factors that should be retained in EFA [33], a scree plot of the eigenvalues of the tetrachoric correlation matrix was graphed. This study assessed the model fit by using root mean square error of approximation (RMSEA), the Tucker-Lewis index (TLI), the comparative fit index (CFI), and standardized root mean square residual (SRMR). The overall fit of the structural model was good, with the CFI and TLI above the recommended value of 0.9 [34], RMSEA less than 0.08 [35], and SRMR less than 0.08 [34]. Items with factor loadings of less than 0.4 were deleted [36].

The Rasch model, known as the 1-parameter logistic model under the item response theory (IRT) model, was applied to assess the adequacy of items in the measurement instrument; that is, all test items were intended to measure a single latent trait or construct [37]. Within the Rasch model, trait ability, item difficulty, and item-fit indices can be used to examine the psychometric properties of an instrument within a given population. In this view, a response to an item is caused by the (1) latent trait level of the subject (ability) and (2) agreement tendency of the item (difficulty). Conversely, item-fit indices are used to indicate the degree to which individual items define the unidimensionality of a construct.

where P(x=1) is the probability of an endorsed response (a yes response to an item), B_n is the trait (or ability) parameter of person n, and D_i is the difficulty of endorsing item i. When B_n>D_i, B_n=D_i, and B_n<D_i, the chances of a yes response are greater than 50%, equal to 50%, and less than 50%, respectively [38].

There was an examination of the unidimensionality of items within the 2 dimensions of the HLS-MW (the assumption that items measure a single latent trait) and those items’ local independence (the assumption that responses to items are unrelated). To provide evidence in support of scale unidimensionality, item-fit indices statistics were computed using the jMetrik 4.0 program (Informer Technologies). Item-fit indices indicate the degree to which individual items define the unidimensionality of a construct [39]. Item fit was evaluated using outfit mean-square fit statistics (MNSQ) instead of standardized fit statistics (ZSTD) and infit MNSQ. MNSQ values of 1.0 are ideal, while values greater than 1.3 or less than 0.7 may indicate unsatisfactory model-data fit [40]. In the case of overfitting and underfitting items and the standardized transformation of the mean-square to approximate a t statistic (ZSTD), values greater than ±1.96 were considered potentially misfitting [37]. We examined the residual correlation matrix to check for values <0.2, indicative of potential local dependence [41].

We evaluated item difficulty and discrimination using classical test theory (CTT). Item difficulty levels range from 0 to 1, with higher values representing easier questions (ie, <0.25: difficult, 0.25-0.75: moderately difficult, and >0.75: easy) [42]. By contrast, a discrimination index closer to 1 indicates better discrimination (ie, >0.2: no item discrimination, 0.2-0.29: little discrimination [item revision warranted], 0.3-0.39: moderate discrimination [little or no revision required], >0.4: satisfactory discrimination).

Based on Everitt’s recommendations [43] for sample size in factor analysis (participant to variable ratio of 10:1), a minimum sample size of 280 was needed for the study. Additionally, with the Rasch model’s application, at least 200 respondents were needed for successful estimation [38]. Therefore, a sample size of 334 was considered adequate. Final items were selected based on item difficulty, model composition, goodness-of-fit, factor loadings, content validity, and item redundancy. In particular, the number of items with similar difficulty levels was reduced by discarding some redundant items [44].

Finally, test-retest reliability, item-total correlations, internal consistency, and intraclass correlation coefficients (ICCs) were examined to address the reliability of the HLS-MW. Specifically, item-total correlations greater than 0.3 were considered adequate [45], and the Kuder-Richardson index was used to estimate internal consistency reliability.

For construct validity testing, we hypothesized that health literacy scores would be associated with sex, education, and Korean language proficiency. We performed 2-tailed t tests and ANOVAs to explore the relationships between these factors and health literacy. Based on existing research [22], we hypothesized that men with higher education and those with higher Korean language proficiency would have significantly higher health literacy scores using the HLS-MW.

This study was approved by the institutional review board of Daegu University (IRB no. 1040621‐202101HR-005) before data collection. All participants were informed of the purpose of the study. Written informed consent was obtained from every participant before beginning data collection.

Table 1 shows the sample characteristics (N=334). The sample comprised mostly males (n=277, 82.9%) in their 30s (mean age 31.89, SD 5.55 years). The majority of participants were from Indonesia (99/334, 29.6%) and Vietnam (93/334, 27.8%), followed by the Philippines (50/334, 15%), Nepal (41/334, 12.3%), Cambodia (35/334, 10.5%), and Pakistan (14/334, 4.2%). In terms of length of residence, most had resided in Korea for 5‐10 years (122/334, 36.5%) or less than 5 (183/334, 54.8%) years. More than half (170/334, 50.9%) of the sample had completed high school. Most participants (279/334, 83.5%) did not have current health problems and had health insurance (303/334, 90.7%). Regarding their Korean language proficiency, only 9.6% (32/334) were fluent, whereas 38.3% (128/334) had difficulty with Korean, and 52.1% (174/334) had average proficiency. Out of 28 items, 11 (3.3%) answered all items incorrectly; no one answered all items correctly, and 4 (1.2%) people answered 27 questions correctly (Multimedia Appendix 1).

An EFA was conducted to examine the factor structure and dimensionality of the data, with the findings showing that the unidimensionality assumption was not violated. The EFA revealed three different models (1-, 2-, and 3-factor). All models had a good fit with no significant differences (Table 2). However, the 1-factor and 3-factor models did not match the conceptual structure of the HLS-MW. In the 2-factor model, most items were significantly loaded on each factor (ie, factor loading>0.4) except for items 24 and 28 (Table 2). Nevertheless, the EFA fit indices overall supported the 2-factor structure of the HLS-MW, with the first factor accounting for 35.4% of the total variance (Multimedia Appendix 2).

The unidimensional assumption should exceed 20% of the accounted variance—as per Reckase’s rule [46]—in the first factor, and the scree plot of the eigenvalues should be greater than 1.0 [34]. The scree plot appeared to show one dominant factor; hence, the scale was deemed sufficiently unidimensional (Multimedia Appendix 2). The absolute value of the correlation coefficient between the residuals of the item was less than 0.2; therefore, local independency was not violated [41].

Table 3 shows item difficulty, item discrimination, and item fit based on CTT and the Rasch model. According to CTT, item difficulty in the prose literacy section ranged from 0.33 to 0.7, and item discrimination ranged from 0.39 to 0.69. Among the 21 items, most showed moderate difficulty with acceptable discrimination. In the document literacy section, item difficulty ranged from 0.12 to 0.63, and item discrimination ranged between −0.15 and 0.64; item 24 had a negative value (−0.15), indicating low validity. Additional results of the theta (IRT ability scores) and IRT test information function are presented in Multimedia Appendices 3 and 4.

According to the Rasch model for item fit, 17 prose items and 5 document items showed a satisfactory fit, while an unsatisfactory fit was found in 6 items (items 1, 2, 4, 18, 24, 28). However, item 2 was retained because the degree of deviation from the fit indices and ZSTD value was marginal, increasing the content validity. Items 7, 13, and 21 were removed as these items had difficulty values similar to those of item 3. Additionally, item 28 was removed due to its difficulty value for migrant workers.

To address item redundancy, items 22, 26, 6, and 10 (−1.19, −0.89, −0.05, −0.05, respectively) and items 12, 11, and 14 (0.15, 0.31, and 0.35, respectively) were eliminated due to similar difficulty levels. Considering the item difficulty, composition, goodness-of-fit, and factor loading, the final HLS-MW consisted of 13 items (10 items addressing prose literacy and 3 items addressing document literacy).

Table 4 shows a comparison of HLS-MW scores according to key demographic characteristics. We found that age was significantly associated with health literacy scores among migrant workers (F_2,331=3.11, P=.04). Specifically, those aged 30‐39 years had a significantly higher health literacy score (mean 7.03, SD 4.18) compared to those aged 20‐29 (mean 6.84, SD 3.66) years and those aged above 40 (mean 5.09, SD 4.26) years. Participants who worked in manufacturing had significantly higher HLS-MW scores than those who worked in agriculture groups (t_2,331=3.64, P=.02). Additionally, migrant workers residing in South Korea for 5‐10 years had the highest HLS-MW scores, followed by those with shorter (<5 years) and longer (>10 years) years of residence (F_2,327=3.15, P=.03). HLS-MW scores were high in the order of Korean language proficiency (F_2,331=3.54, P=.004). Finally, participants who completed high school had the highest HLS-MW scores (mean 7.04, SD 4.02), followed by those who completed college or higher (mean 6.79, SD 4.09) and middle school (mean 6.02, SD 3.89) levels of education. However, these differences in health literacy scores by educational level were not statistically significant.

The HLS-MW exhibited good internal consistency. The Cronbach α coefficient was 0.88 for the overall scale, 0.86 for the prose literacy section, and 0.67 for the document section. The ICC was 0.94 (95% CI 0.93‐0.95) and the test-retest reliability was r=0.74 (95% CI 0.57‐0.92).

This study is one of the first to systematically assess health literacy among unskilled migrant workers. We developed a theory-based health literacy scale—HLS-MW—and validated it using EFA, Rasch analysis, and hypothesis testing with a sample of migrant workers in South Korea. Our study indicated that the HLS-MW exhibited good reliability and valid psychometric properties.

We found that more than one-third of the migrant workers had limited Korean language proficiency. This finding is consistent with previous research and a recent national survey reporting that 44%‐52% of migrant workers have limited Korean proficiency [17,47]. Insufficient Korean language proficiency leads to difficulty in understanding and obtaining health-related information, using medical services, and communicating with health providers [23]. We discovered that only 34.1% (114/334) to 56% (187/334) of the survey respondents understood basic health-related words in Korean, such as “blood pressure” (item 2) and “temperature” (item 3),” and along with “sleep” (item 19) was the most difficult for workers to grasp in prose literacy.

The national Korean learning program for migrant workers primarily focuses on work-related issues (eg, understanding the Labor Standards Act and the Industrial Safety and Health Act). For this reason, migrant workers often encounter barriers to understanding and communicating with health care providers even after they pass EPS-TOPIK [20]. A recent analysis of Korean language textbooks for immigrants [48] also revealed that health-related topics were mainly confined to 1‐2 units of brief conversations at hospitals and pharmacies for the beginner level. Indeed, it is critical to include educational content addressing medical terminologies and health service use within the Korean language curriculum for migrant workers. In particular, migrant workers commonly experience musculoskeletal problems and allergies due to their working conditions and sleep disturbance in relation to working hours and shifts [49,50]. The retained items of the HLS-MW were selected to focus on situations encountered at the reception when visiting a doctor’s clinic for treatment or outpatient appointments and clarifying a doctor’s prescription for health problems. Additionally, questions were posed regarding situations in which medication is prescribed. These represent typical situations migrant workers living in Korea experience when facing health problems. Screening immigrants with low health literacy using short-form tools could be useful for health care providers who are unfamiliar with other ethnic populations, and could also serve as a resource for migrant workers to inform them of their health care weaknesses and allow them to receive support. In the future, it is necessary to develop educational content for Korean language and medical services in textbooks and educational programs for migrant workers, tailored to their level of Korean proficiency.

Women and those with a longer residence, younger age, higher education, and greater language proficiency have been associated with higher health literacy among migrant workers [29,51-54]. While longer residency in South Korea and Korean language proficiency were significantly associated with HLS-MW scores in this study, sex, age, and education were not. These results may be a simple artifact of sampling bias. For example, most participants in the study were men (82.9% or 277/334) and youths in their 20s and 30s. It is unclear why education was not associated with health literacy in the study. One possibility might be that migrant workers under the EPS in South Korea are primarily employed in small workplaces, doing similar jobs regardless of their education level; therefore, their exposure to Korean words may have been similarly limited, independent of their educational level. Further research is warranted to include, in the study sample, diverse migrant workers with more women and older workers.

There are certain study limitations to note. This validation study used a convenience sample of migrant workers; almost all (97% or 324/334) were documented workers. While there are no accurate statistics regarding the number of undocumented migrant workers in South Korea, one available government report estimates that there are nearly 390,000 illegal immigrants in South Korea, approximately 19% of all immigrants in the country [55]. Undocumented migrant workers are more likely to have frequent health problems and unmet health care needs compared to native workers [56]. Additionally, the proportion of H-2 migrant workers who responded to our survey with proficient Korean skills was 57.2% (191/334), compared to 4.2% (14/334) of E-9 migrant workers. H-2 visa holders (eg, Korean-Chinese) have higher Korean language proficiency than those with E-9 visas [10]. The sample also consisted of mostly male (277/334, 82.9%). Future validation of the HLS-MW should explore whether visa type or any other characteristics, such as documentation status and gender, are associated with health literacy among migrant workers.

Additionally, the HLS-MW does not suggest a cutoff point for detecting limited health literacy; therefore, more research is needed to develop a cutoff point. Since there exists no gold standard that determines who belongs to the low versus high health-literacy group in South Korea, research identifying a criterion for health literacy in Korean health care settings may be useful. Lastly, we conducted this study exclusively on the web. Considering that this study was conducted during the COVID-19 pandemic and that nearly 90% (288/331) of migrant workers in the sample had graduated from high school or higher, had an average age in the 30s, and depended on diverse information from EPS apps and web-based services, the study’s participants primarily responded using their mobile phones. Future research is necessary to increase the validity and dimensionality of the scale using an offline survey. Health literacy for migrant workers has not been studied in South Korea partly due to the lack of a scale to adequately measure their health literacy. We hope this scale will encourage researchers to explore the significance of problems regarding low health literacy in the health care context and enable health care providers to develop strategies to enhance migrant workers’ literacy skills.

Despite the limitations, the HLS-MW is a valid and reliable scale for measuring the health literacy of migrant workers in South Korea. In particular, our findings shed light on the health experience among unskilled migrant workers from South Asia. Additional studies are warranted to further test the psychometric properties of the HLS-MW among diverse groups of migrant workers in South Korea while also establishing cutoff points to help identify those requiring health literacy support.