The MHLS-C was developed in two phases. Phase One was for item translation adapted from the MHLS (O’Connor and Casey, 2015). Permission to translate and validate the MHLS was obtained from Dr. O’Connor, the original designer. We translated the MHLS into the MHLS-C based on an adapted Brislin’s translation model for cross-cultural translation, which included translation, back-translation, comparison, linguistic adaptation and pilot testing (Brislin, 1970). First, two bilingual researchers independently translated the 35-item MHLS from English to Chinese and combined these two versions. Then, a third bilingual researcher back-translated the Chinese version into English. Later, a fourth researcher compared the back-translated English version with the original English scale. Disagreements were mainly derived from the different expression habits with respect to word order. A consensus was reached via a group discussion. This process yielded Chinese version 1 of the MHLS-C.

Individual cognitive interviews were conducted by the principal investigator with 20 clinical nurses. This method was used to how well participants understood the questions and could provide valid responses reflecting their own symptom experiences (Reeve et al., 2017). The interview used structured questions to detect how residents interpreted the items and thereby tested their comprehensibility and readability. Example questions include “Tell me in your own words what this question is asking” and “What does this item mean to you?” The information from the interviews was discussed in a group meeting, and we made appropriate changes to a few synonyms, such as by changing “紊乱” to “障碍” and “政客” to “政界人士.” This process yielded Chinese version 2 of the MHLS-C for validation.

Phase Two tested the psychometric properties using a cross-sectional survey. The inclusion criteria were (1) registered nurses with a working experience over 1 year (usually finishing fresh nurse training), (2) aged 18–60 years old, and (3) voluntary participation. In April 2021, a stratified-clustered sample of participants was recruited. The researchers distributed the e-invitation and questionnaire link via the widely used smartphone-based investigation tool Wenjuanxing.¹ Using stratified cluster sampling, we chose 15 out from 35 tertiary and 15 out from 54 secondary public hospitals in Shanghai, including comprehensive, psychiatric and other specialized hospitals. In each hospital, we randomly selected 2 to 3 wards. The link was disseminated by the head nurse in the selected wards, and all nursing staff who met the inclusion criteria were willing to participate in the survey.

Sample size was determined based on the subject-to-item ratio of 5–10:1 (Streiner et al., 2015) based on a set of 35 items. The sample size ranged from 350 to 700 for exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), respectively. When the sample size neared the required number, the researcher ended the questionnaire. A total of 872 registered nurses were recruited. The sample was split into two parts for analysis as stated above. The characteristics of the nurses are shown in Table 1, and there was no significant difference between the two randomly selected groups in any of the sociodemographic or family characteristics. To evaluate the test-retest reliability, after 2 weeks, 40 participants in two wards were asked to complete the MHLS-C again.

A questionnaire consisting of several sociodemographic questions and several scales was used. Completion of the questionnaire required approximately 3–10 min. The JSE-HP, PHQ-2, and GAD-2 were applied to evaluate convergent validity.

The study was reviewed by the Institutional Review Board at the researchers’ institute, and informed consent was obtained from all participants. The survey was anonymized. Data were downloaded, and only the researchers had access to them.

The data were analyzed using SPSS 21.0 (IBM Corp. Released 2010. Armonk, NY: IBM Corp.) and Mplus 7.0 (Muthén and Muthén). The methodological quality of the MHLS was examined using the COSMIN checklist (COnsensus-based Standards for the selection of health status Measurement INstruments) (Mokkink et al., 2010). In total, six of the nine domains were determined to be adequately assessed: item analysis, internal consistency, retest reliability, concurrent validity, structural validity, and hypothesis testing. The sample of nurses was randomly split into two parts for EFA and CFA by using “select cases” function in SPSS, which meant 50 percentages of the total sample was randomly selected for EFA and the left was for CFA. The validation procedure is shown in Figure 1.

(a) Item analysis: We deleted an item if it met one of following criteria: (1) factor loading < 0.4 or cross-loading or false loading which could not be theoretically explained via EFA, (2) critical ratio (CR) value was non-significant with insufficient power of discrimination (Nunnally and Bernstein, 1994), (3) Cronbach’s α coefficient of the scale increased if this item was deleted (Nguyen et al., 2014; Huang et al., 2017), and (4) Pearson correlation coefficient between a certain item with total score was very small or non-significant, which reflected its representativeness and common variance with the whole scale is poor in explain that the connotation of the measurement (Nunnally and Bernstein, 1994).

In EFA analysis for detecting the number of factors and item loadings, we first tested whether the KMO value was >0.7 and whether the P value of Bartlett’s/df was <0.05, indicating suitability for factor analysis. Then, principal component analysis with the maximum variance method was conducted, and the solutions were examined theoretically (Floyd and Widaman, 1995). The number of extracted factors was based on an eigenvalue >1.0 and factor loading above 0.4 and the percentage of the explained variance.

(b) Structural validity: The EFA-derived structure was investigated using Velicer’s minimum average partial (MAP) test and a parallel analysis (PA) for a retest (O’Connor, 2000). According to Velicer, the smallest average squared partial correlation or the smallest average 4th-power partial correlation best indicates the number of factors (Velicer et al., 2000). One hundred random datasets were generated, and when the eigenvalue of the actual data became smaller than the corresponding eigenvalue of the random data, the number of factors was retained (O’Connor, 2000).

CFA was then conducted, and the parameters used to appraise the model were χ²/df (> 5) and several model fit indicators, including comparative fit index (CFI), Tucker–Lewis index (TLI), the Akaike information criterion (AIC), Bayesian information criterion (BIC) and root mean square error of approximation (RMSEA) (Kline, 2011). Comparative fit index and TLI values > 0.90 and RMSEA values < 0.08 suggest good fit, while the value for the AIC and BIC should be smaller to obtain the most parsimonious model fit (Kline, 2011). The first 15 items of the MHLS had a 4-point scaling, with the others were 5-point scaling. Thus, the robust weighted least squares with mean and variance adjustment (WLSMV) estimator was used in present study (Flora and Curran, 2004; Wang et al., 2013).

(c) Concurrent validity: Pearson correlation analysis was used to detect the relation of the total score and each factor of the MHLS-C with the JSE-HP, PHQ-2, and GAD-2. We anticipated that the mental health literacy score would have a positive relationship with empathy ability Jefferson Scale of Empathy-Health Professionals (JSE-HP) and a negative relationship with depression (PHQ-2) and anxiety (GAD-2).

(d) Reliability: Cronbach’s α coefficient was applied to determine the internal consistency of the whole scale and its dimensions. The intraclass correlation coefficient (ICC) was used to detect the correlation in 40 nurses at a two-week interval.

(e) Statistical description was used to show the norm, and regression analysis was performed to detect the variables influencing mental health literacy. For unranked variables, such as marital status and working hospital, dummy variables were created.

The KMO value was 0.92, and Bartlett’s test of sphericity was significant (chi square = 21613.87, P < 0.001). Then, in EFA analysis, four factors were extracted that accounted for 66.33% of the variance. The loading coefficient from EFA, CR values, Cronbach’s α coefficient after deleting an item, and r values of each item are summarized in Table 2. According to the criteria for item retention. I-10, I-15, I-20, I-21, I-22, and I-23 were deleted for false loading not complied with theoretical understanding, cross-loading, the CR value was non-significant, and the correlation coefficient was low.

In the MAP test, when the root was 4, we obtained the smallest average squared partial correlation and the average 4th-power partial correlation get larger in root 5. In the parallel analysis, when the root was 5, the mean random-data eigenvalue was larger than its eigenvalue from the actual dataset. Therefore, based on the MAP test and PA shown in Table 3, a 4-factor model is recommended and statistically stable.

The 4-factor solution was then validated using the CFA sample. The CFA confirmed the 4-factor structure. To improve the model fit based on modification indices, several pairs of residual correlations within one factor were added, indicating that each element of a pair had the same facet as its attributed factor. As factor 4 was correlated with other factors with weak even negative correlation, we further detected the higher order structure. The fit indicators were summarized in Table 4. The 1 factor model based on original scale showed unsatisfactory model fit, and the 4 factor model as well as 4 first-order 1 second-order fit much better. Further, we detected 4 first-order 2 second-order model, of which factor 1, factor 2 and factor3 were summarized together while factor 4 was set alone. This 4 first-order 2 second-order structure showed equal indicators with 4 first-order 1 second-order. Thus, based on theoretical analysis, the 4 first-order 2 second-order structure is recommended.

The final structure is shown in Figure 2, with regression weights that were significant. Following theoretical analysis, we named factor 1 “knowledge of mental disorder (knowledge),” factor 2 “ability of seeking information and help (ability),” factor 3 “recognition of mental disorder (recognition),” and factor 4 “acceptance of patients with mental illness (acceptance).” The factor 1, factor 2 and factor 3 contributed to a higher latent variable defined as “Core literacy,” while factor 4 contributed to “Social acceptance,” which could be seen as two subscales, MHLS-Core and MHLS-SA.

Pearson correlation analysis showed that MHLS-C and MHLS-Core were moderately negatively correlated with the PHQ-2 and GAD-2 and highly positively correlated with the JSE-HP. While MHLS-SA was not significantly correlated with PHQ-2, GAD-2, JES-HP, and MHLS-Core but with total score. More specifically, MHLS-SA and knowledge were not significantly correlated with PHQ-2 and GAD-2 scores. All factors were significantly correlated with JES-HP, except MHLS-SA. More information could be found at Table 5.

The Cronbach’s α was 0.85 for the overall scale, with 0.89 and 0.93 for two subscales. The test-retest reliability was good, with ICCs of 0.80 for the whole scale and 0.79 and 0.94 for two subscales.

The mean scores was 93.02 ± 10.76, The basic statistics and percentiles of each subscale are summarized in Table 6. The skewness and kurtosis of each factor and whole scale indicated that they were not strictly normally distributed. Thus, scores in each quartile of each factor and whole scale were used to describe the cut-off, as shown in Table 6. The 50^th percentile for the MHLS-C was 99, with 74 and 20 for MHLS-Core and MHLS-SA.

Setting the total MHLS-C score as the dependent variable, the regression analysis showed that professional credentials (β = 0.26, P < 0.001), hospital hierarchy (β = 0.13, P < 0.001), position (β = 0.12, P < 0.001), other specialist hospital (β = 0.10, P = 0.009), psychiatric hospital (β = 0.19, P < 0.001), entered into the model. The other sociodemographic variables had no statistical significance.

Robust psychometric characteristics and practicality are the basis for the clinical application of a tool. At the item level, three items were deleted for poor item information: I-10 “men are more likely to experience an anxiety disorder compared to women,” I-15 “a mental health professional can break confidentiality if your problem is not life-threatening and they want to assist others to better support you,” I-20 “People with a mental illness could snap out if it if they wanted,” I-21 “A mental illness is a sign of personal weakness,” I-22 “A mental illness is not a real medical illness” and I-23 “People with a mental illness are dangerous.” In the Iranian version, six questions (20, 21, 22, 26, 27, 28) were deleted by CFA (Nejatian et al., 2021), while in the Persian version, five items (10, 12, 20, 21, 22) were deleted from item analysis (Heizomi et al., 2020). We could see that I-10, I-20, I-21, and I-22 generally had poor item information in various versions. I-10 refers to gender risk factors for anxiety disorder, but the incidence between genders may be different in different countries, and thus it was deleted. I-20 refers to recognition of mental disorders, but it seems to be too absolute because mental illness could be self-cured in some cases, such as self-practiced mindfulness therapy (Parsons et al., 2017), and I-21, I-22 conveyed vogue content in Chinese context, and thus they had poor performance. I-15 did not perform well only in MHL-C, which may be due to cultural differences in China, where family members of patients with mental disorders highly participate in shared decision-making regarding admission, treatment, and discharge compared to European and Chinese Americans (Gao et al., 2019). I-23 was summarized to acceptance in Persian version and in our study, it was cross-loaded to ability to seeking help, which could not be explained theoretically. Therefore, 6 items deleted in the Chinese version were statistically, theoretically, and culturally appropriate.

Henson and Robert suggest that when performing EFA, the number of factors extracted should be considered based on multiple criteria rather than relying on a single standard alone (Reise et al., 2000; Henson and Roberts, 2006). The five-factor structure in MHLS-C was validated by EFA, CFA, MAP and PA. O’Connor’s and Jorm’s concept of MHL included several attributes, as mentioned in the introduction (Jorm et al., 1997; O’Connor and Casey, 2015). In the original English version of MHLS, as the communalities were low, the structure was univariate (O’Connor and Casey, 2015). In the Iranian version, six factors were the ability to recognize disorders, knowledge of self-treatment, knowledge of professional help available, knowledge of risk factors and causes, knowledge of where to seek information, and attitudes that promote recognition or appropriate help-seeking behavior (Nejatian et al., 2021). In the Persian version, five factors were the ability to recognize mental disorders, confidentiality of mental health practitioners, skills of mental health information seeking, beliefs about mental illnesses, and attitudes toward patients with mental illness (Heizomi et al., 2020). Compared with them, 4 factors in MHLS-C are more similar to the Persian version. Factor 1, “knowledge of mental disorder (knowledge),” refers to the ability to recognize specific disorders and the confidentiality of mental health practitioners in the Persian version (Heizomi et al., 2020) and all knowledge-related factors in the Iranian version (Nejatian et al., 2021). Factor 2 “ability to seek information and help (ability),” factor 3 “recognition of mental disorder (recognition)” and factor 4 “acceptance of patients with mental illness (acceptance)” were similar to the remaining factors in the Persian version accordingly (Heizomi et al., 2020). We could see the homogeneity and heterogeneity of structure in different cultures.

To be noted, acceptance with knowledge and recognition were either non-significant or negative, which was similar to the Persian version (Heizomi et al., 2020). The finding in this study complied with the findings in Persian version and also the definition of mental health literacy, which refers to knowledge and attitudes regarding mental health that aid in the recognition, management and prevention of mental health issues (Jorm et al., 1997). The criterion validity of MHLS-SA scores is poor both in our study and in Persian version, but good in the Iranian version. Further to compare the participants, the Persian version was among mothers of female high school children, whereas the Iranian version was among general population. Previous study has found that Chinese primary healthcare providers hold negative attitudes to mental health patients, especially with regard to engaging in closer personal relationships with psychiatric patients (Ma et al., 2018; Yin et al., 2020). The poor criterion validity of MHLS-SA may due to the testing participants, which may hold different level of acceptance and influence the correlation between acceptance and other factors. This implied that further validation of Chinese version among the general public may be necessary. In addition, this implies promoting knowledge and help-seeking, and an objective view does not guarantee people’s social acceptance of patients with mental disorders. When designing interventions, this attribute should be given independent attention. We further detected the higher 4 first-order 2 second-order structure, which showed that MHLS-C could be split into two subscales, core literacy and social acceptance. Even though criterion validity of MHLS-SA scores is poor, we recommended to keep it for future validation among other population group. The former core literacy subscale could be considered to be used independently to measure the core content of mental health literacy.

The concurrent validity of the MHLS-C was supported by its correlation with mental health status and empathy. In a previous study, there was no significant relationship between MHL and psychological distress (O’Connor and Casey, 2015), while other studies found similar results among university nursing students (Al-Yateem et al., 2018) and teachers (Chen et al., 2021). Our study found a moderately negative correlation with depression and anxiety, which implied that higher MHL would promote self-management and help-seeking and thus help to promote mental health status to some extent (Jorm, 2012). In addition, we found that MHL was highly positively correlated with empathy (0.586), which indicated that the promotion of MHL among health professionals would greatly promote their ability to observe and identify mental disorders and thus provide humanistic care ability. The Cronbach’s α coefficients and retest reliability of the whole scale and each factor were good, which indicated that the MHLS-C had good reliability.

As different versions contained different items, we used the ratio of the total score and mean score to compare the MHL level, which were 71.2% in our study, 71.5% in the Iranian public (Nejatian et al., 2021), 60.9% in Persian/Farsi speaking people (Heizomi et al., 2020), 72.0% in Chinese teachers (Chen et al., 2021) and 90.9% in Australian health professionals (O’Connor and Casey, 2015). We could see that the ratio in this study was moderately high but lower than that in Australian health professionals. A previous survey revealed that less than 50% of student nurses (Al-Yateem et al., 2018) and 54.3% of pediatric hospital staff in United Arab Emirates (Al-Yateem et al., 2017) could correctly identify the disorders presented and 38.9%, 56.2, and 17.5% for schizophrenia, depression, and GAD, respectively, in Chinese nurses in general hospitals (Hao et al., 2020). The results found that higher position, higher professional credentials, higher hospital hierarchy, other specialist hospital, psychiatric hospital and unmarried status were positive predictors of MHL. The results were in accordance with previous studies (Poon et al., 2019; Hao et al., 2020).