VH is a multifaceted concept, encompassing different behavioral categories such as confidence (referring to trust in the vaccine or healthcare provider), complacency (perceiving the need for vaccination or valuing it), and convenience (relating to accessibility) (13). A variety of conceptual frameworks have been used to develop measurements for vaccine hesitancy in previous studies. The widely used models include the Health Belief Model (HBM), the Theory of Planned Behavior (TPB), and the Trust, Confidence, and Cooperation (TCC) Model (2).

For our instrument, we adopted a hybrid conceptual framework that integrates the core dimensions of VH (confidence, complacency, and convenience), the HBM, and the TCC model. We decided to use the HBM as vaccine hesitancy is primarily affected by an individual’s perceived beliefs, including perceived benefit, perceived susceptibility, perceived severity, and perceived barriers (14, 15). These perceptions are closely tied to trust in vaccines, an essential determinant of VH (3, 14, 15). The TCC model was incorporated to capture the role of trust more fully in promoting cooperation and vaccine acceptance (14–16).

This integrated conceptual framework, with two main constructs, Vaccine Hesitancy and Perceptions of Vaccines, guided both our literature search strategy and the identification of relevant VH measurement items. The Vaccine Hesitancy construct includes three subdomains (confidence, complacency, and convenience), while Perceptions of Vaccines encompasses five subdomains (perceived benefit, perceived barrier, perceived susceptibility, perceived severity, and trust) (14–16).

The purpose of this study was to develop an instrument to be used to measure vaccine hesitancy of COVID-19 vaccines. Health professionals and other responsible groups can use the information to distribute and administer the COVID-19 vaccines. Furthermore, this will benefit the development of health education programs toward vaccine hesitancy, specifically for COVID-19 vaccines. To date, there is no gold standard COVID-19 vaccine hesitancy scale. Hundreds of scales have been developed to identify key factors of vaccine hesitancy for COVID-19 (8). The gold standard vaccine hesitancy scale (VHS) was created in 2012 by the WHO SAGE Working Group, answered by parents about vaccination practices for their children (17). The questions were developed from several global pilot tests and literature reviews of similar instruments, including the Parent Attitude About Childhood Vaccines (PACV). This scale design allows for the deployment of the scale in diverse groups and cultures (18). However, a general vaccine hesitancy scale based on the perceptions of parents regarding vaccinating their children is not sufficient to address the unique hesitancies surrounding the COVID-19 vaccine. With its rapid development and distribution, the global nature of the pandemic, and its politicization, a COVID-19 vaccine hesitancy scale that captures the unique circumstances of COVID-19 has yet to be developed.

After the comprehensive literature review guided by the conceptual framework, the research team combed through hundreds of pertinent questions and statements related to vaccine hesitancy. Literature for COVID-19 vaccine hesitancy demonstrated a theme of seven constructs: benefits, cost, confidence, convenience, complacency, risk, and perception (3, 8, 9, 19). The items were developed based on previous scales, including the WHO-SAGE vaccine hesitancy measurement, generalized vaccine hesitancy scale and HPV vaccination attitudes and beliefs scale, and other relevant literature (2–4, 8, 9, 19). Using common themes found in the literature, the research team narrowed down the number of questions to 71 questions deemed vital and relevant. After the items were generated, duplicate items and items that were deemed irrelevant were removed.

To validate the VHAPS-CV19, a content validity analysis was conducted in partnership with subject matter experts (SMEs). This phase’s primary focus is on determining the content validity of this survey tool. The items in this survey were based on literature collected during the initial phase of the study. Each item was clearly written and determined to be representative of the hesitancies and perceptions of interest identified by the research team. Feedback from SMEs would inform the inclusion, revision, and exclusion of items from the finalized tool.

SMEs were instructed to rate the relevance of each item directly in a Google Sheet, ranging from not relevant to highly relevant. 10 SMEs were recruited and asked to review the content validity of the survey tool. SMEs across the United States were chosen based on their demonstrated expertise and mastery related to COVID-19 and associated vaccination. The SMEs include professors, clinicians, and public health professionals, with experience ranging from 2 years to more than 10 years. Demographic information was not collected as part of this process.

The initial VHPSC-19 contained 41 items and was developed to identify factors related to COVID-19 vaccine hesitancy. The items were further reviewed by subject matter experts in the content validity index (CVI) phase using a Likert scale rating. The following formulas were used to calculate results for I-CVI and S-CVI scores (20):

I-CVI = N of experts rating items a 3 or 4 / Total N of experts

To test whether each item was measuring what the research intended, face validity was also performed using self-identified COVID-19 hesitant and non-hesitant individuals. The laypersons who were asked to provide feedback on the items were previously known to the research team. Individuals provided feedback about the wording, structure, and content of the survey. The research team considered the feedback for the final items.

For the pilot testing, American adults aged 18 years of age or older living in the United States were eligible to complete the VHPSC-19 via a Qualtrics online survey. Participants’ recruitment was conducted through Social Media pages (which will include Twitter, Facebook, Instagram, LinkedIn) and through institutional listservs. The surveys were completed anonymously. Upon completion, participants were asked voluntarily to provide name and email address for the raffle purpose. There was no way to link survey responses to the name/email provided. Once the winners were selected, all names and emails were permanently destroyed. Participation in this study was voluntary. All respondents gave their online consent before completing the survey. Data were collected from the end of December 2022 to May 1, 2023. Ethical approval was obtained from the institutional review board of our institution. The total number of respondents was 352. Due to the convenience sample and snowballing recruitment method, we are unable to confirm an accurate response rate. Participants were also invited to participate in a test–retest survey 2 weeks following their initial survey submission to assess the test–retest reliability of the tool. An alpha value of over 0.7 was used to determine which items to retain (21), 73 respondents fully completed the test–retest survey 2 weeks after the initial survey. Following reliability analysis testing with the alpha values over 0.7, 29 items were retained. Six items were dropped, which were repetitive or did not meet the alpha value cutoff of 0.7 (21).

All responses were used for analysis after management procedures were applied for missing data. The authors used SPSS, version 28.0, software (SPSS Inc.) to perform descriptive analysis of demographic characteristics and EFA and internal consistency and scale reliability testing. The researchers utilized principal axis factoring, a prevalent estimation approach in exploratory factor analysis (EFA), to offer an elucidation for the underlying composition among the items (22). The researchers employed the varimax rotation technique, which is widely recognized in exploratory factor analysis (EFA), with the underlying assumption that there was no correlation among the items. This choice was made to enhance the presentation of the factors inherent in the VHPSC-19, thereby contributing to a more meaningful representation (23).

The scale level (S-CVI) was 0.87. The item level (I-CVI) ranged from 0.71 to 1.00. The CVI process led to the retention of 35 highly relevant items. The remaining 35 items were tested with a survey that asked a diverse sample of adult Americans living in the United States to specify their level of agreement or disagreement on a Likert scale of 1 to 5 (strongly disagree to strongly agree) with statements that led to assess a person’s hesitancy towards the COVID-19 vaccine. Demographic questions were included.

352 participants completed the survey. Most of the participants were Female (71.6%), Married (42.6%), White (79.5%), had at least a bachelor’s degree (78.0%) and working full-time (62.2%). Their ages ranged from 24 to 76 years (M = 46.14 years; SD = 12.02 years). Half (49.7%) aligned with democratic political views. Ninety-three percent of participants had at least one COVID-19 vaccine, and 86.9% received a COVID-19 vaccine booster. Almost two-thirds of the participants reported they received positive COVID-19 test results (Table 1).

An EFA with varimax rotation was performed to determine the correlations and underlying structure among the factors. All 352 participants were included in the EFA of the 29 items within the VHPSC-19. All factors with eigenvalues of 0.30 or greater were retained and further evaluated. Pearson correlation coefficients for the items ranged from 0.980 to 0.986 (p < 0.001). The Kaiser-Meyer-Olkin Measure of Sampling Adequacy was 0.961 and Bartlett’s Test of Sphericity was significant at x²_{(406, 352)} = 8,025.78, p < 0.001, validating that factor analysis was suitable for this study sample (23). Following the EFA with varimax rotation, a three-factor solution emerged and explained 62.3% of the total variance. The loadings in the factor component matrix were between −0.831 and 0.881, while the loadings in the factor rotated component matrix were between 0.351 and 0.838; both indicated minimal to excellent loading (24). While three factors emerged, Factor 3 did not consist of a minimum of three unique statements. Items 16 and 26 were dropped from the tool as they only appeared in Factor 3. This rationale supported the choice to maintain the two-factor solution. Items with double or triple loadings were placed in the most appropriate factor, given their importance to understanding factors and existing theoretical foundations. Next, Cronbach’s alpha and item-total correlation were used to measure reliability and internal consistency. For the 14 items from factor 1, the internal consistency value ranged from 0.948 to 0.962, and the alpha value was 0.955. For the 12 items from factor 2, the internal consistency value ranged from 0.917 to 0.939 and the alpha value was 0.929, indicating excellent reliability (25) (see Table 2).

After finalizing the VHPSC-19, the authors reviewed and named the two factors. Scores are determined by summing up the ratings of each item on the 5-point Likert scale (where 1 = lowest, 2 = low, 3 = middle, 4 = high, and 5 = highest). Factor 1 included 14 items regarding COVID-19 vaccine perception and trust, which are interconnected. Poor perception and lack of trust of a vaccine can be due to misinformation, poor health literacy, mistrust in government officials, and improper health communication for the given audience. The score of Factor 1 ranged from 14 (lowest concerns for vaccine perception and trust) to 70 (highest concerns for vaccine perception and trust). Factor 2 included 12 items related to COVID-19 vaccine hesitancy. This factor measures a person’s reluctance or hesitation towards getting the COVID-19 vaccine due to cost, complacency, confidence, and convenience. (Table 3) The score of Factor 2 ranged from 12 (lowest vaccine hesitancy) to 60 (highest vaccine hesitancy).

As literature indicates, there is no gold standard to measure COVID-19 vaccine hesitancy and perception currently in the United States. To overcome COVID-19 vaccine hesitancy, public health practitioners first need to be able to identify those who are hesitant or have negative perception. This instrument assist public health professionals in their ability to identify a person’s COVID-19 vaccine hesitancy status as well as COVID-19 vaccine perception. Based on how a person responds, a different intervention may need to be used to overcome COVID-19 vaccine hesitancy.