We included all published studies in which HCPs had to perform oral intubation on COVID-19 patients during simulation training. No restrictions were placed on study design. The co-primary outcomes were the perceived usefulness of the simulation for pandemic preparedness, confidence gained through the simulation, and improved preparedness to treat COVID-19 patients. We excluded studies that examined (a) contamination in simulation scenarios with different devices, mostly aerosol boxes; (b) non-oral airway procedures (mostly tracheostomy); (c) no simulation of airway scenarios, only providing algorithms or simulation scenarios; (d) ventilator-related issues; (e) single case reports; (f) conference abstracts or posters; and (g) lists of airway equipment for COVID-19 patients.

No language restrictions were applied in the search. We did not include gray literature. The last search was conducted on 25th August 2022. The inclusion criteria focused on studies evaluating simulation-based training for airway management in the context of COVID-19, while exclusion criteria included studies without primary data or those not addressing simulation in this context.

On 25th August 2022, we conducted a search on PubMed, Medline, the Cochrane Library, and Embase using the keywords “airway,” “management,” “COVID,” and “simulation.” We excluded publications containing the keyword “box,” as there have been many publications investigating the use of aerosol boxes for intubation of COVID-19 patients. We refrained from applying any other limitations, especially language restrictions or time cutoffs. The search algorithm for each database can be found in Supplementary Digital Content 1. The aim was to find articles that compared and assessed simulation as a training method to prepare HCPs for the COVID-19 pandemic. We aimed to compare studies in which simulation training for airway management in COVID-19 patients was performed. We could not find a review article or meta-analysis on this important subject at the time of our search.

Study selection was conducted by two reviewers, MK and BK, who independently screened the titles and abstracts of all publications identified during the literature search. In case of insufficient information in the abstract, the publications were included for full-text review. Disagreements were resolved by consensus. During the full-text review, eligibility was again independently assessed by both reviewers. Disagreements were resolved through arbitration by a third reviewer, BG.

Data collection was conducted independently by four reviewers. The first round was completed by MK, and the second round was divided among SA, AB, and LS using a customized Excel database. In case of discrepancies in data extraction, discussion was used to resolve them. Any measure of effect related to simulation training was eligible for inclusion. We categorized outcomes according to the four levels of the Kirkpatrick model—reaction, learning, behavior, and results—to provide a structured framework for interpretation (17). Level 1 represents reaction (e.g., the satisfaction or confidence of the participants), Level 2 reflects learning (e.g., acquisition of knowledge and skills), Level 3 assesses changes in behavior (application of the acquired knowledge or skills at work), and Level 4 measures results (better patient outcomes at work due to the training). Applying the Kirkpatrick model allowed us to contextualize improvements in knowledge and confidence as indicators of both learning and behavioral changes, supporting the robustness of our conclusions.

No restrictions were imposed regarding the number of participants in the included studies.

To assess the risk of bias, we used the Scottish Intercollegiate Guidelines Network (SIGN) checklist for cohort studies (18). Most well-known checklists for assessing risk of bias are designed for studies with control and intervention groups. Due to their emphasis on blinding and differences between these groups, they were not suitable for the design of the included simulation studies. All reviewed publications had only one group of participants, with data regarding their knowledge or confidence collected before and after the simulation training. Ratings were assigned independently by two reviewers (MK and AB), and discrepancies were resolved by consensus. A graphic overview was created to provide an outline of the study quality.

Two outcomes were investigated: The level of comfort, which was captured as confidence in the data set, and the level of knowledge after the simulation-based training.

Some parameters necessary for meta-analysis were not reported directly in the publication but could be retrieved from other information that was reported in the publication. In these situations, the Wald confidence interval was used, along with the approximation of the standard deviation (SD) and mean as described by Wan et al. (19).

Descriptive statistics, stratified by publication year, are reported as follows: Mean and standard deviation (SD) for continuous variables, median and interquartile range (IQR) for ordinal or skewed continuous variables, and frequencies and percentages for categorical variables.

As the different studies measured different outcomes (e.g., only confidence, only knowledge, or both) in different ways (5-point Likert scale, 7-point Likert scale, Multiple Choice Questions), we had a large amount of missing data. For the meta-analysis, a random-effects model was employed due to the expected substantial between-study heterogeneity in the meta-analysis of observational studies. The meta-analysis was performed using the function rma() from the package metafor. As the studies applied different scales of measurement for the primary outcomes, no direct overall effect measure could be estimated. However, forest plots were generated to provide a visual representation of the primary outcomes. Change scores were calculated for the studies to be able to compute a random effects model, as well as standardized mean differences (SMDs) for paired data. A correlation coefficient ρ of 0.5 was assumed for these calculations. A random effects model was used for meta-analysis. The heterogeneity parameter τ² was computed by default using the restricted maximum likelihood (REML) estimator for the random effects models. The number of studies in the meta-analyses was low; therefore, meta-regression could not be performed to evaluate sources of heterogeneity. To do this, a minimum of 10 studies would be necessary. To still evaluate sources of heterogeneity, graphical assessments of possible sources of heterogeneity were performed.

I² and χ² are reported. The χ² test assesses the significance of heterogeneity but does not provide measurements. An index of heterogeneity is the I² by Higgins, see Bland (20), chapter 17.5. The I² is the percentage of the χ² statistic that is not explained by the variation within the studies. An I² value from 0% to 40% might not be important, a value from 30% to 60% may represent moderate heterogeneity, a value from 50% to 90% may represent substantial heterogeneity, and a value from 75% to 100 % indicates considerable heterogeneity. Funnel plots were used to assess possible publication bias.

For all analyses, a nominal level (α) of 5% was used. Accordingly, the default confidence interval width was set to 95%.

As different measurement types (e.g., 5-point Likert scales, numeric rating scales from 0 to 10, surveys without scales, 7-point scales) were used to report the confidence and knowledge gained through training, it was not possible to derive overall effects from the given data. Therefore, the change score between the post- and pre-simulation results was derived. Consequently, the data were to some degree standardized as the differences were being investigated and not the raw scores. However, as the scores did not have the same range (from 2 to 15 points possible), it was not possible to completely standardize the values in this way.

The mean change in each group was calculated by subtracting the post-simulation mean from the baseline mean, see the following formula:

The SD of a change score was computed in the following way [Higgins et al. (21), chapter 6.5.2.8].

Where ρ = 0.5 was set by definition.

The correlation parameter ρ was changed in a sensitivity analysis to quantify the effect of the choice on the results. Moreover, the estimation method for τ² was changed to the DerSimonian–Laird (DL) estimator in a sensitivity analysis. If the results based on the sensitivity analysis remained unchanged, they were considered robust.

All analyses were performed using the R programming language (22) in combination with dynamic programming with knitr in a fully scripted way.

In our search of the electronic databases, we found 141 articles after removing duplicates, of which 32 met the inclusion criteria. Inter-reviewer reliability indicated substantial agreement and was calculated using means of Cohen's Kappa (κ = 0.72; n = 141 articles, 129 agreements and 12 disagreements).

After the full-text review, we excluded 13 publications because they did not meet the inclusion criteria after all or were posters from conferences or letters to the editor. In addition, we searched the reference lists of the included articles to find other relevant publications. In this way, we found another article that met the inclusion criteria. Therefore, in total, we found 20 articles that met our inclusion criteria, as shown in our PRISMA flow chart (see Supplementary Digital Content 2).

A list of the screened publications, including the 20 studies that met our inclusion criteria, can be found in Supplementary Digital Content 3.

We screened the risk of bias according to the SIGN checklist for cohort studies (18). For ease of visualization, we created a graphic diagram. The following tables show our assessment process and our results according to the checklist's questions (Table 1):

The complete checklist included questions that did not apply to any of our studies due to their study design. The cohort studies checklist was the best fit, but as we did not compare two different groups, the questions regarding homogeneity or dropouts from each arm of the study could not be answered. In addition, there was no blinding, as the participants knew about the simulation training and the aim of the training. To make our table clearer and for ease of visualization, we excluded the following questions. The answer to all of them would have been “does not apply” in all of our included studies.

Among the included studies, the majority were rated as moderate quality, with fewer studies rated as high or low quality; a summary distribution is provided in Table 2.

Of all 20 studies, five provided numeric data on the confidence of the participants before and after the simulation training. In addition, four studies provided data on knowledge before and after the training. The remaining studies provided only post-training data, so we could not compare changes in knowledge or confidence, or did not provide numeric data at all.

Across the studies, “confidence” referred to various aspects, including self-confidence in technical airway skills, team performance, handling of PPE, and stress management. When different data were provided regarding the confidence of various skills, we focused on confidence related to airway management.

Figures 1, 2 illustrate the mean confidence pre- and post-simulation, including the 95% confidence interval. Pre- and post-training knowledge scores are provided in Figures 3, 4. Figures 5, 6 compare the means and 95% confidence intervals before and after the simulation for the two primary outcomes, respectively.

As described in the methods, the studies were heterogeneous; therefore, we compared the change score of the different studies. The studies and the results of the meta-analyses are illustrated in Figures 7, 8. As the observed post-simulation confidence and knowledge means were subtracted from the baseline before training, a score less than zero indicated a positive training effect. A score of zero demonstrated no training effect, while a value greater than zero indicated a negative training effect, meaning reduced confidence or knowledge after the simulation.

The standardized mean difference (SMD) of a paired sample is also called Cohen's d, and together with its standard error, it can be derived from the given data. This is implemented in the metafor package in R (23). Therefore, a random effects model was fitted for both outcomes using the standardized mean difference as the effect measure.

The formula for the calculation of the SMD reads is as follows:

With the given formula, a negative effect was expected. The post-simulation results were subtracted from the pre-simulation results. A value equal to zero indicated no effect, while a value greater than zero indicated a reduction in confidence or knowledge following the simulation/training. A negative effect indicated a training effect, as confidence or knowledge after training was higher than before (see Figures 9, 10).

Figures 11, 12 show the mean of the change scores and the estimated confidence intervals, setting ρ to {0.25, 0.5, 0.75} for both outcome variables separately. It can be seen that the smaller the ρ, the larger the confidence interval, as the standard deviation of a change score decreases with an increasing correlation. The smaller the ρ, the smaller the probability of obtaining a significant result, as the SD decreases with increasing ρ.

For the different given correlations, ρ, the confidence interval does not intersect zero. Therefore, it can be stated that the choice of a correlation ρ ε [0.25, 0.75] does not alter the results.

τ² can be estimated using different methods. Applying both the restricted maximum likelihood (REML) estimator and the DL estimator led to similar estimates for τ² in the random effects models for the SMD for knowledge. The estimates for τ² confidence differed. However, the estimates for I ² were similar. Figures 13, 14 show the forest plot for the models using the REML estimator, along with the results from the DL approach (highlighted in red).

Figure 15 shows the funnel plots for the change score models and the SMD models. The white funnel plot illustrates the 95% confidence interval around the effect estimate. It is notable that many studies fall outside the funnel plot. Moreover, the studies are distributed asymmetrically in the funnel plot, indicating publication bias.

The certainty of evidence across the included studies varied. While most studies demonstrated improvements in confidence and knowledge post-simulation, the heterogeneity of the study designs, varying sample sizes, and methodological limitations reduced the overall confidence in the findings. In addition, the presence of publication bias, as indicated by asymmetrical funnel plots, further impacts the reliability of the results. Future studies with standardized methodologies and rigorous reporting are necessary to enhance the certainty of evidence in this area.