The questionnaire used as part of a Steinbaum et al., (2023) study consisted of 22 items and two open-ended questions adapted from a previously published survey [herein referred to as pilot survey] on parent acceptance of dentist roles in HPV vaccination (Supplementary Appendix A) (18). Of the 22 items, 6 were demographic questions; 8 were related to COVID-19 vaccinations and 8 were related to the HPV vaccination. Questions between the COVID-19 and HPV vaccinations were similar in content assessing whether participants were up to date in their vaccinations, their knowledge of the vaccines and their beliefs and comfort with dental professionals administering vaccines. For all analyses below, responses of “unsure” or “prefer not to answer” were coded as missing, making all responses binary (yes/no).

The quantitative study aimed to evaluate the psychometric properties of the pilot questionnaire, including construct validity assessment using exploratory factor analysis (EFA) and computerized adaptive test (CAT) simulation (19). We used the data from the 163 participants in the Steinbaum et al., (2023) study to calibrate the items. The senior author on the current paper is a co-author on the Steinbaum et al. (2023) paper providing access to the data for 163 participants The data for these participants were used to conduct the psychometric analyses. For the 163 participants in the Steinbaum et al., (2023) study, patients ages 18–45 with dental appointments and no limitations according to medical condition, race, other demographic factors, or dental history were recruited from an academic dental institution in Pennsylvania using a convenience sampling method.

Participants were primarily female (67%) with a mean age of 31.67 (7.67). Forty-four percent of participants identified as African American/Black followed by Whites (32%). Slightly more than half had less than a high school degree (26%) or some college (27%) with one-third having an income less than $20,000 and one-third between $35,000 and $75,000. About one-half of participants were vaccinated against each of COVID-19 (55%) and HPV (51%). See Figure 1 for pilot survey distribution results. Data were collected on sex assigned at birth, gender identity, age, race, education, household income, vaccination status and plans, education about the vaccines, comfort, and attitudes about the dental provider's role in vaccinations for both COVID-19 and HPV. For analysis, age and race was simplified into 3-level categorical variables (younger than 24, ages 24–40, and 40 and older and White, African American/Black, and others, respectively). Education level was simplified into less than high school, some college, bachelor's degree, and graduate degrees. When analyzing responses for whether patients would be comfortable receiving a vaccine today (HPV or COVID-19), individuals already vaccinated were removed from analysis of this question only. Other than noting that the percentages did not add to 100 due to missingness, no further information was provided. See Table 1 for data regarding missing questions.

Details of the CAT simulation method of test abbreviation are available (20). Briefly, CAT in its full implementation is a method of “catering” a test or scale to an examinee by continually updating (estimating and re-estimating) his/her trait level during the administration to determine the optimal next item (question) to administer (21, 22). A secondary use of CAT, when it is not desirable or feasible to develop a full CAT implementation system, is to simulate what would happen if a particular fixed-length scale (as here) were administered adaptively. The benefit of these simulations is that they reveal which items within a scale are most informative overall; for example, if there is an item that is never (or very rarely) administered in the simulations, one can infer that said item is “low quality” and could possibly be removed from the scale without noticeable loss of information. This approach to test and scale abbreviation has been used successfully in both cognitive (23, 24) and non-cognitive (25, 26) domains.

The qualitative study aimed to determine the clarity and accessibility of the pilot survey and to gain a deeper understanding of the perceptions of dental administration of vaccines through a focus group discussion (27). Focus groups are increasingly used in dental studies seeking a depth of understanding not captured in quantitative approaches (28, 29). The use of a focus group instead of individual interviews was selected so participants could provide their views and learn how their views were shaped by the synergy of the focus group (30). Focus groups allow for times when an individual participant may not have an immediate response but end up having one based on the conversation, which can produce rich, detailed data (31).

Using convenience sampling, we recruited 9 participants for a focus group using flyers distributed within Philadelphia, PA and online using a waiting room television screen where the recruitment flyer was advertised. Our target population was adults ages 18–45 with dental appointments. There were no limitations according to medical condition, race, other demographic factors, or dental history. Participants who met the following eligibility criteria were included: (1) patients between the ages of 18 and 45 to match possible eligibility for HPV vaccination; (2) ability to speak and read English; and (3) ability to independently consent to participate. A focus group facilitated by a trained moderator explores the perspectives of a group of participants about a particular problem—in this case—the clarity and accessibility of our pilot survey. Focus groups of 8–12 can provide information about the processes participants use to answer survey questions and assist researchers in learning how participants perceive items. In this case, 8 participants is considered an adequate size to facilitate discussion. Participants were asked to commit one hour to the review of a pilot survey and were offered a $50 gift card and a meal as incentives for participation.

Guided by the Health Belief Model (32), we gathered participants’ feedback on the questions in the pilot survey to understand if the questions were clear, how they thought about the questions, and their opinions about the questions. See Supplementary Table 2: Appendix B. All patients were consented prior to starting the focus group. The focus group was recorded for transcription and coding purposes. A moderator along with a lead and an assistant notetaker facilitated the focus group. Using the think-aloud method, the focus group discussion centered on comprehension, understanding, meaning of questions and satisfaction with the questionnaire (33). The think-aloud approach allows researchers to understand how patients perceive and interpret the information presented and to identify any potential problems in the material. Specifically, we reviewed the questionnaire in detail and asked participants to say everything that passed through their minds as they read. Participants were probed in any areas where there seemed to be confusion.

For the psychometric analyses, we conducted exploratory item-factor analysis and computerized adaptive test (CAT) simulation (20). Item covariance structure was assessed using exploratory item-factor analysis (item-wise EFA). Inter-item tetrachoric correlations were calculated, and 1-, 2-, and 3-factor solutions were estimated using least-squares extraction and promax rotation. Four methods for determining the optimal number of factors were used: the minimum average partial (MAP) method (34), parallel analysis with Glorfeld correction at the 95th percentile (35), the minimum Bayesian Information Criterion (36), and subjective evaluation of the scree plot to determine where the successively decreasing eigenvalues begin to form a linear trend (the “elbow”). These methods suggested 1-factor, 1-factor, 3-factor, and 1-factor solutions, respectively. The 1-factor (unidimensional) result was supported by the ratio of first to second eigenvalues of 8.3/1.4 = 5.9, which is beyond a commonly used cutoff of 3.0 (23). EFA solutions from one to three factors were estimated for thoroughness. For the above factor analytic procedures, missing data were handled with pairwise deletion.

For theoretical reasons, and to suit the goals of the original protocol, it was necessary to keep the COVID-related items separate from the HPV-related items. The goal of the study was to compare attitudes between the different disease states (not knowledge, which we used to support attitudes and not behavior but rather intention) thus despite the EFA results that suggest a single factor. The second step was therefore to develop two abbreviated forms of the 8-item scales using computerized adaptive test (CAT) simulation (19). Items were calibrated in preparation for CAT-simulation using the 2-parameter logistic item response theory model (2PLM) using the mirt package in R, with missing data handled using full-information maximum likelihood (FIML) (37). Details of IRT are beyond the present scope (38), but the goal of the method is to characterize each item (question) using some quantifiable parameters. The 2PLM characterizes items in terms of their discrimination (α) and difficulty/extremity (β). The discrimination parameter indicates how precisely an item can distinguish among individuals on a particular trait continuum (here, negative attitude toward COVID or HPV vaccines), where highly discriminating items are generally desirable (high quality). The difficulty/extremity parameter indicates how “far up” the trait continuum someone needs to be to have a 50% probability of endorsing the item. An important feature of IRT models is that both the trait dimension and item extremity parameters are on a z metric, meaning someone with a trait level of 1.0 is one standard deviation above the mean, and therefore administering that person an item with an extremity parameter of 1.0 would be the exact extremity for which the person has a 50% probability of endorsement. Because items provide maximum information when the probability of endorsement is 50%, it is always most informative to administer items with extremity parameters as close as possible to a person's estimated trait standing. With items calibrated using the 2PLM, they could be used in CAT simulation. Here, 5,000 simulated examinees were administered exactly five items using the catR package in R, where the five items administered to one examinee often differed from those administered to another (39). The frequency with which each item was administered was used to determine the item's quality and suitability for an abbreviated form.

The transcription of the focus group was analyzed following the procedures of Miles, Huberman, and Saldana (40), where the research team first reviewed the transcript line by line to identify comments related to the HBM (32). Four of the team members (TC, KF, YL, MF) independently reviewed and coded the transcript using a provisional code list with HBM constructs, then met as a group to review their findings. Codes were revised and the team independently reviewed and recoded the transcript and met four more times to review findings. The codes were revised throughout the process. Discrepancies in coding were resolved through consensus among the first two authors. During second order coding using our final code list, the codes were reviewed to identify semantic relationships in our findings, which were used to group them into themes. Attitudes and knowledge emerged as the two major themes from the analysis.

We used standard merging and narrative weaving techniques to compare and contrast quantitative and qualitative data (41, 42). Specifically, we examined how changes to the pilot survey resulting from our psychometric analyses did or did not align with recommended changes offered by participants during our qualitative focus groups.

Two findings from the psychometric analyses were unexpected. First, the empirical methods strongly suggested a single factor was sufficient to explain inter-item covariance, rather than the two hypothesized factors of “COVID-related” and “HPV-related.” Second, the 2- and 3-factor EFA solutions suggested that, to the extent multiple factors were necessary to explain the covariance, they were better characterized as attitudes/knowledge (Factor 1) and willingness/intention to accept a vaccine (Factor 2). When a third factor was extracted, the additional factor related to knowledge about cancer prevention (cancer_HPV) and intention to seek an HPV vaccine.

Table 2 shows the results of the unidimensional, 2-, and 3-factor solutions of the combined item set. The highest loadings on the unidimensional model were “administer_hpv” (0.95) and “accept_covid” (0.94) indicating that these were the strongest indicators of the measured trait. The lowest loadings were “Intends_No_HPV_Vaccine” (0.21) and “inform_COVID” (0.40), indicating they were the weakest indicators.

Table 3 shows the IRT parameters estimates and results of the CAT simulations when the COVID- and HPV-related item sets were used separately as item banks [J] (see Supplementary Table 1 for estimates with confidence intervals). For the COVID items, discrimination parameters were highest for “administer_COVID” (8.15) and “accept_COVID” (9.48), and extremity ranged from −0.21 (“Vaccine_Today_No”) to 5.23 (“inform_COVID”). Note that because items were coded such that endorsement indicated a “no” response, the extreme difficulty of the “inform_COVID” item indicates that almost no one said s/he was not informed about COVID. For the HPV items, discrimination parameters were highest for “administer_HPV” (5.53) and “comfort_HPV” (5.51), and extremity ranged from −0.24 (“Vaccine_Today_No”) to 1.80 (“Intends_No_HPV_Vaccine”). For the item administration frequency (quality), the top four COVID-related items were “administer_covid”, “dental_covid”, “Intends_No_COVID_Vaccine”, and “Vaccine_Today_No”. The top four HPV-related items were “accept_hpv”, “administer_hpv”, “Vaccine_Today_No”, and “comfort_hpv.” [J] Notably, variability of these discrimination parameter estimates across items provides support for the use of the 2PL model rather than a 1PL model in which the discrimination parameter is constrained to equality across items.

Supplementary Table 2 shows the fit statistics for the COVID and HPV models. The M2 statistic (44) is non-significant, suggesting acceptable fit, and the CFIs/TLIs (all > 0.95) and RMSEAs (both < 0.06) are consistent with this (43, 44). However, the SRMRs (both > 0.08) are higher than usually considered acceptable; this would suggest some highly correlated residuals. Supplementary Table 3 shows the item residual covariances and local dependency statistics, which give some clues as to what is causing the high SRMR. There are indeed some “doublets”, or pairs of items that have significant covariance even after controlling for the latent trait. Examples include dental covid with accept_covid and vaccine_today, admin_hpv with vaccine_today, and cancer_hpv with dental_hpv and int_no_hpv. Notably, the residual covariance between accept_covid and admin_covid was only moderate (0.13), suggesting it is likely not the sole cause of the very high discrimination parameters of those two items.

Table 4 details demographic characteristics of focus group participants. Of the nine focus group participants, (56%) identified as female. Most participants were 40 years of age or younger (89%), and the mean age of participants was 30 years. Participants identifying as White made up 44% of the focus group (n = 4) and participants identifying as Asian made up 44% (n = 4).

Using the think aloud method to get feedback from the participants about the pilot survey, participants noted the following problems: (1) the phrase completely vaccinated is unclear; (2) the answer choices yes, no and unsure did not account for every vaccination status and intention—there should be a prefer not to answer choice; (3) lack of knowledge on the HPV vaccine name; and 5. lack of clarity on what dental professional meant.

As a result of the feedback, changes were made to the pilot survey see Table 5 for details. The term “completely vaccinated” was clearly defined in both the COVID-19 and HPV portion of the final survey. The number of doses required to be considered “up to date” on each type of vaccine was included in the final survey. The pilot survey question “If no, do you plan on receiving the COVID-19 vaccine(s)?” was also combined with the above question and language was added to clarify variations in vaccination status and intention using a single question. A “prefer not to answer” choice was added. The HPV vaccine name, “Gardasil 9” was added. The term “dental professional” was changed to “dentist” for clarification fort the final survey, despite the ability for auxiliary dental personnel to educate and play other roles in vaccination in certain areas.

The two overarching themes to better understand perceptions of dentist involvement in vaccine efforts identified from the focus group were: (1) attitudes related to appropriateness and acceptability of dental providers educating and administering vaccinations; and (2) how lack of knowledge hindered intentions to seek vaccination from dental providers.

Based on the integrated results of the psychometric and focus group analysis, the pilot survey was adjusted. See Table 5 for a joint display of final survey changes based on psychometrics and focus group findings. Items were considered for inclusion and edited based on a combination of qualitative and quantitative metrics, and the goal of the survey. The length of the final survey was organized according to factor analysis and modified to reflect the recommendations provided by focus group participants (see Supplementary Appendix C). These changes reduced the survey from 22 questions to 16 questions, a 25% decrease. The final survey included one question about both actual behavior and intention, one question about knowledge, and three questions about attitudes, compared to one question each about actual behavior, intention, and knowledge in the original survey and 5 questions about attitudes. Patients were also provided free-text space to provide comments at the end of the COVID-19 and HPV vaccine questions. A version history table that reconciles the 1. original pilot items and response options; 2. The exact item set used for the EFA/IRT/CAT; and 3. The final 16 item instrument are presented in Table 6.

Given the number of people in the US that visit their dental providers more (47, 51) than their physicians, and the ongoing work needed to increase vaccination coverage, we need valid surveys that assess attitudes related to dentist roles in preventive healthcare. Testing surveys with target populations provides a crucial opportunity to engage the population and understand their perceptions to develop interventions. If findings from future research using this survey indicate that patient acceptance is high for dentists to provide vaccinations, there are opportunities to educate legislators and advocate for changes in what professions are allowed to vaccinate. Expanding the scope of dentistry to include vaccine education and administration could increase access to care, vaccine uptake, and disease prevention. This study not only provides evidence for a valid survey to test without concerns regarding understandability and clarity but also provides evidence that in our efforts to create a comprehensive public health system. We need all health care providers involved in vaccination efforts. Every clinical encounter is an opportunity to vaccinate (52).