Missing data can often occur in psychological research, whether due to dropouts in longitudinal studies, skipped questions in surveys, or equipment limitations (e.g., eye-trackers failing to capture certain eye movements). Missing data can also result from the strategic use of a planned missing data design (Graham et al., 2006). For example, in studies with prohibitively lengthy questionnaires, researchers may want to administer only a subset of the questions to each participant, and treat the rest as missing, in order to reduce survey fatigue. Traditionally, psychological researchers who encounter missing data would most typically apply ad-hoc missing data techniques, such as listwise deletion or pairwise deletion. These methods are advantageous in their expedience, but will often lead to inconsistent estimates and power loss, resulting in bad inference. In the past decade or so, modern missing data techniques with better statistical properties, such as full information maximum likelihood (FIML; Allison, 1987; Muthén et al., 1987; Arbuckle, 1996) and multiple imputation (MI; Rubin, 1987), have become increasingly accessible via computer programs like lavaan (Rosseel, 2012) and mice (van Buuren and Groothuis-Oudshoorn, 2011), available in R (R Core Team, 2019).

Rubin (1976) defined three types of missing data mechanisms: missing completely at random (MCAR), when the reason data are missing is independent of any variable in the dataset, whether missing or observed; missing at random (MAR), when the reason data are missing is dependent only on observed data known as conditioning variables; and missing not at random (MNAR), when the reason data are missing is dependent on data that are not observed, even after conditioning on observed data. When data are missing in psychological studies, researchers typically assume ignorable missing data, which refers to MCAR or MAR missing mechanism, plus an additional technical assumption that the parameters describing the missing mechanism are independent of the model parameters (Little and Rubin, 2002). This article will focus on the impact of ignorable MAR mechanism on FIML parameter estimates. When describing the possible impact of missing data, researchers typically report only the rate of missing data per variable, in the dataset as a whole, or the proportion of incomplete cases. However, the missing rates alone are insufficient to determine the efficiency of the estimates under a general MAR mechanism.

The MAR assumption is very general, and there is a wide range of specific missing mechanisms that can generate MAR data. For example, the relationship between the conditioning variable and the missing probability can be monotonic or not monotonic (linear vs nonlinear MAR; see Collins et al., 2001; Yoo, 2009; Savalei and Rhemtulla, 2017). Further, the missing mechanism can be more deterministic or more probabilistic (strong vs weak MAR; see Yucel et al., 2011; Sullivan et al., 2018; Chen et al., 2020). When MAR mechanisms differ, the statistical properties of the FIML parameter estimates can differ even when the missing rates are the same. Although MCAR, a special case of MAR, does not involve any conditioning variables, thus avoiding some of these issues, it could still lead to parameter estimates with varying efficiency when the number of missing data patterns differ (e.g., Savalei and Bentler, 2005; Savalei and Falk, 2014).

Differential performance of FIML estimates under different MAR mechanisms can translate into differences in efficiency, and therefore different power when used to perform null hypothesis tests. In light of the replication crisis, there has been a renewed call for researchers to better understand the power of their analyses. The impact of missing data on power can only be understood if we adopt a more informative diagnostic measure than missingness rates. In this article, we argue that the appropriate diagnostic measure is a quantity known as the fraction of missing information (FMI). We evaluate the performance of several estimates of FMI in the context of FIML estimation within the structural equation modeling (SEM) framework. This framework is quite general and includes regression models, path analysis, confirmatory factor analysis, and general linear models involving any combination of observed and latent variables.

SEMs are usually fit as mean and covariance structure models. Given a set of p variables, a particular model implies that the mean and covariance structure of these variables is given by μ(θ) and Σ(θ), where θ is a q × 1 vector of model parameters. Under the assumptions that the missing data mechanism is MAR and that data are multivariate normally distributed, the FIML parameter estimates θ^FIML can be obtained by maximizing the observed data normal-theory log-likelihood logL(θ|Y), given by (Little and Rubin, 1987):

where N is the sample size, p_i is the number of variables observed for case i. μ_i(θ) is the p_i × 1 model-implied vector of means, and Σ_i(θ) is the p_i × p_i model-implied covariance matrix, both for the variables that are not missing in case i.

Parameter estimates θ^FIML are asymptotically fully efficient if the distributional assumptions are met. However, they are not as efficient as they would have been had there been no missing data. If for the same sample of N observations, we write X = (Y, Z), where Y is the observed data, Z is the missing data, and X is the complete data, then the hypothetical complete data parameter estimates θ^ML obtained by maximizing L(θ|X) would be more efficient than the FIML parameter estimates θ^FIML to the extent that the missing data Z contains information about θ.

While the idea for how to obtain sample FMIs following FIML estimation was published a number of years ago (Savalei and Rhemtulla, 2012), we are not aware of any simulation studies that have been conducted to evaluate the performance of sample FMIs. Furthermore, it was not until recently that FMI estimates with better properties (using the same type of standard errors in both numerator and denominator), have become automated in lavaan. In order to evaluate the performance of these sample FMI estimates, we conducted a simulation study with two commonly used models in psychology, a simple regression model and a two-factor model. The main objective was to determine whether the population FMI values can be estimated with reasonable accuracy under a realistic sample size. The three estimates, δ^1,j, δ^2,j, and δ^3,j, were expected to be practically equal in the saturated regression model. It was unclear, however, how they would differ in the two-factor model. Therefore, another objective for the study was to compare their performance.

Here we provide an example on how to obtain FMIs from the lavaan package (version 0.6-7 and up) in R, using the Holzinger and Swineford (1939) dataset and simulated MCAR missing data. The example is adapted from Savalei and Rosseel (ress). The dataset, available through the lavaan package, contains cognitive performance test scores from 301 school children. The data for this example can be loaded into the R workspace using the following code.

For the purpose of this demonstration, we will conduct a confirmatory factor analysis with three correlated factors: visual skills, verbal skills, and mental speed. Each factor is measured by three tests, with variable names x1 to x9 in the datasets, following a model given by the lavaan model syntax below.

As the dataset does not contain missing data, we will introduce MCAR missingness by randomly removing 61 values from each variable independently, resulting in an overall missing rate of 20%.

With the example data and model syntax prepared, we fit the data to the CFA model and obtain the FMIs with the following code.

The option std.lv = TRUE fixes all variances of the latent factors to 1, allowing all loadings to be freely estimated, while the missing = ~ml~ asks lavaan to handle the missing data using FIML. The function parameterEstimates extracts the results from the model fit, where the option fmi = TRUE requests FMI estimates from lavaan alongside the parameter estimates. The remove.nonfree = TRUE option omits parameters that are not freely estimated from the output—In this case, the latent factor variances are not printed in the output table, as they were fixed to 1. By default, lavaan uses Hessian numeric estimation of the observed information matrix, which yields δ^1 for the FMI estimates. To produce δ^2, the observed.information input must be specified to request the analytic approximation of the information.

By default, the analytic approximation is based on structured information, to obtain δ^3, the h1.information input must be provided to request the unstructured information.

The three FMI estimates provide largely similar values (see Table 6). They agree on which loading estimates have the highest FMIs, namely the loadings of X₁ (δ^1=0.29, δ^2=0.27, δ^3=0.29), X₂ (δ^1=0.26, δ^2=0.25, δ^3=0.28), X₃ (δ^1=0.28, δ^2=0.28, δ^3=0.31), and X₇ (δ^1=0.27, δ^2=0.27, δ^3=0.28). The three estimates also agree on which factor correlation has the highest FMI, namely the correlation between visual skill and mental speed, δ^1=0.23, δ^2=0.24, δ^3=0.29. The FMI of the factor correlation between visual skill and mental speed also shows the largest difference among the FMI estimates, between 0.23 and 0.29. The three estimates disagree on which factor correlation has the lowest FMI, but the differences are small: For the correlation between visual and verbal skills, δ^1=0.18, δ^2=0.15, δ^3=0.18; for the correlation between verbal skill and mental speed, δ^1=0.18, δ^2=0.18, δ^3=0.20.

Overall, the highest FMI estimate in the model is 0.31. The associated WIF is 1/1-0.31=1.2, which indicates an estimated 20% increase of the width of the confidence interval due to missing data. Reporting these FMIs alongside analyses of empirical data will provide readers with a better sense of how much the presence of missing data has affected the efficiency of the parameter estimates, especially when comparing across studies where FMIs may differ even under the same missing rates. The full R code of this example is provided in Appendix A, and a summary of the lavaan options of the three estimates is given in Supplementary Table 1.

The current simulation study suggests that a relatively large sample size may be necessary for the estimation of the FMIs. Sample FMI estimates were largely unbiased, even in very small samples with N = 50, which are typical of regression analyses. However, at such a small sample size, the estimates were imprecise, varying greatly from sample to sample, especially when the missing rate was high. When the missing rates were reasonably low (π_mis = 0.2, 0.4; corresponding to 10–20% overall missing rate), sample sizes of several hundreds, which are typical in structural equation models, were able to produce reasonably efficient estimates. However, at an overall missing rate of 30%, it would require sample sizes exceeding 1,000 to produce precise estimates. It is worth noting that we used very strong selection mechanisms in the MAR-L and MAR-NL conditions to contrast the results from MCAR. In applied settings, the MAR selection mechanism would typically be weaker, which would lead to sample FMI estimate performances that are closer to better performances we saw in the MCAR conditions.

The three estimates are identical for saturated models, such as regression. However, the choice makes a difference when the model is not saturated. In the two-factor model, δ^1,j, the estimate via numeric Hessian, showed a distinct disadvantage as it was more likely break down when the sample size was small, or when the missing rate was high. In contrast, δ^3,j, the analytic estimate based on the unstructured model, was much less likely to break down in all cases, and was more precise than δ^2,j. Although δ^3,j occasionally showed a slightly higher bias than δ^2,j, which was based on the structured model, its performance was overall more favorable in these simulations.

The simulation study investigated FMI in FIML, but the results should generalize to FMI computed from MI with a large number of imputations. In MI, the FMI is conceptually given by the ratio of the between-imputation variance over the sum of the within- and between- imputation variances. As the number of imputations approaches infinity, this ratio becomes equivalent to the ratio of variance increase due to missing data over variance in the observed data as estimated from FIML. For simulation studies, FMI can be more computationally expensive in MI, as the estimate is produced in the final pooling stage of the analysis, and often requires a large number of imputations (more than 100) to achieve an acceptable level of accuracy (Harel, 2007). For substantive research, the researcher may simply choose between FIML and MI as the estimation method of FMI based on the missing data technique they are already using to produce the estimates of the model parameters.

As far as we are aware, this study is the first to look at the properties of sample FMIs computed using FIML. As such, we focused on two relatively simple and commonly used models, with three missing data mechanisms selected to contrast the impact of the specific mechanism on the FMI values and to stress that these values are not the same as the rates of missing data. Future research may wish to expand on the study conditions, for example, by controlling for the number of missing patterns, examining how changing the values of parameters in the model (such as the regression coefficient) would change the properties of the FMI estimates. It would also be worthwhile to investigate the relative performance of the three FMI estimates under incorrect models. When the model is wrong, the Hessian-based estimate, δ^1,j, is theoretically superior, as it is the only consistent estimate. However, whether this theoretical advantage would translate into a practical advantage needs to be examined in simulation studies. It would also be helpful to develop bootstrap SE/CI for the sample FMIs, so that researchers would have a better sense of the precision of the FMI estimates in their particular sample.

While our focus was on evaluating the properties of sample FMI estimates in terms how well they served as estimates of the corresponding population FMIs, the properties of population FMIs themselves may be of interest, and is a topic we are exploring in other work. In this ongoing work, we are finding that information loss can occur in unintuitive and unpredictable ways, and patterns in population FMIs observed in one context do not always generalize to other context. For example, based on the population-level FMI values we obtained for the conditions in this study, one may be tempted to conclude that the population FMIs of factor correlations are, in general, insensitive to missing data mechanisms (e.g., the middle rows of Table 1). However, this was not always the case. In conditions not reported here, when the indicators of F₁ were all completely observed, and the indicators of F₂ contained missingness conditioned on the indicators of F₁, the FMI of the factor correlation became more sensitive to the missing data mechanism. Although the population quantities estimated by the sample FMIs may exhibit different patterns, we do not expect the sample FMI estimates themselves to show drastically different properties in these alternative scenarios.

The FMI estimates via FIML are available in most recent releases of lavaan (version 0.6-7 and up), and example R code for how to retrieve them is given in Appendix A. We recommend empirical researchers to routinely examine and report FMIs for key parameters in substantive analysis. FMIs capture the complex interplay between numerous factors such as missing rates, missing data mechanisms, and model parameters, sometimes in unintuitive ways. They can provide critical additional insights into how the standard errors, confidence intervals, and hypothesis tests may have been impacted by the presence of missing data. For methodologists conducting simulation studies with missing data, the population FMIs in the different study design conditions should be computed and reported, in order to provide better context of the performance of missing data techniques being studied.

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://osf.io/xyzt8/.

VS developed the equations, statistical properties, and details of implementation for FMI estimates. LC designed and carried out the simulation study and wrote the initial draft of the manuscript. All authors contributed to the revisions of the manuscript.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.