Measuring Model Fit

Measuring Model Fit

Fit refers to the ability of a model to reproduce the data (i.e., usually the variance-covariance matrix). It should be noted that a good-fitting model is not necessarily a valid model. There are now literally hundreds of measures of fit.

Moreover, a model all of whose parameters are zero is of a "good-fitting" model. This page includes some of the major ones, but does not pretend to include all the measures. Though a bit dated, the book edited by Bollen and Long (Testing structural equation models. Newbury Park, CA: Sage, 1993) explains these indexes and others.

Chi Square: X2
For models with about 75 to 200 cases, this is a reasonable measure of fit. But for models with more cases, the chi square is almost always statistically significant. Chi square is also affected by the size of the correlations in the model: the larger the correlations, the poorer the fit. For these reasons alternative measures of fit have been developed. (A website for computing p values for chi square.)

Chi Square to df Ratio: X2/df
There are no consistent standards for what is considered an acceptable model.

Transforming Chi Square to Z
Z = √(22) - √(2df - 1)

where df refers to the degrees of freedom of the model.

Bentler Bonett Index or Normed Fit Index (NFI)
Define the null model as a model in which all of the correlations or covariances are zero.

The null model is referred to as the "Independence Model" in

X2(Null Model) - X2(Proposed Model)
-----------------------------------
X2(Null Model)

A value between .90 and .95 is acceptable, and above .95 is good. A disadvantage of this measure is that it cannot be smaller if more parameters are added to the model. Thus, the more parameters added to the model, the larger the index. It is for this reason that this measure is not used much anymore, but rather one of the next two is used.

Tucker Lewis Index or Non-normed Fit Index (NNFI)
A problem with the Bentler-Bonett index is that there is no penalty for adding parameters. The Tucker-Lewis index does have such a penalty. Let X2/df be the ratio of chi square to its degrees of freedom

X2/df(Null Model) - X2/df(Proposed Model)
-----------------------------------------
X2/df(Null Model) - 1

If the index is greater than one, it is set at one. It is interpreted as the Bentler-Bonett index. Note than for a given model, a lower chi square to df rati (as long as it is not less than one) implies a better fitting model.

Comparative Fit Index (CFI)
This measure is directly based on the non-centrality measure.
Let d = X2 - df where df are the degrees of freedom of the model.

The Comparative Fit Index equals
d(Null Model) - d(Proposed Model)
---------------------------------
d(Null Model)

If the index is greater than one, it is set at one and if less than zero, it is set to zero. It is interpreted as the previous indexes. If the CFI is less than one, then the CFI is always greater than the TLI. CFI pays a penalty of one for every parameter estimated.

Root Mean Square Error of Approximation (RMSEA)
This measure is based on the non-centrality parameter.
Its formula can be shown to equal:

√[X2/df - 1) /(N - 1)]

where N the sample size and df the degrees of freedom of the model.
(If X2 is less than df, then RMSEA is set to zero.) Good models have an RMSEA of .05or less. Models whose RMSEA is .10 or more have poor fit.

A confidence interval can be computed for this index. First, the value of the non-centrality parameter is determined by X2 - df. The confidence interval for non centrality parameter can be determined for X2, df, and the width of the confidence interval. (One can use the function "CNONCT" within SAS to compute these values. Also a website for computing p values for the non-centrality parameter.) Then these values are substituted for X2 - df into the formula for the RMSEA.

Ideally the lower value of the 90% confidence interval includes or is very near zero and the upper value is not very large, i.e., less than .08.
p of Close Fit (PCLOSE)

The null hypothesis is that the RMSEA is .05, a close-fitting model. The p value examines the alternative hypothesis that the RMSEA is greater that .05. So if the p is greater than .05, then it is concluded that the fit of the model is "close."

Standardized Root Mean Square Residual (SRMR)
This measure is the standardized difference between the observed covariance and predicted covariance. A value of zero indicates perfect fit. This measure tends to be smaller as sample size increases and as the number of parameters in the model increases. A value less than .08 is considered a good fit.

Akaike Information Criterion (AIC)
This measure indicates a better fit when it is smaller. The measure is not standardized and is not interpreted for a given model. For two models estimated from the same data set, the model with the smaller AIC is to be preferred.

X2 + k(k - 1) - 2df

where k is the number of variables in the model and df is the degrees of freedom of the model. Note that k(k - 1) - 2df equals the number of free parameters in the model. The AIC makes the researcher pay a penalty of two for every parameter that is estimated. The absolute value of AIC has relatively little meaning; rather the focus is on the relative size, the model with the smaller AIC being preferred.

GFI and AGFI (LISREL measures)
These measures are affected by sample size and can be large for models that are poorly specified. The current consensus is not to use these measures.

Hoelter Index
The index should only be computed if the chi square is statistically significant. Its formula is:
(N - 1)X2(crit)
--------------- + 1
X2

where N is the sample size, X2 is the chi square for the model and X2(crit) is the critical value for the chi square. If the critical value is unknown, the following approximation can be used:
[1.645 + √(2df - 1)]2
---------------------- + 1
2X2/(N - 1)

where df are the degrees of freedom of the model. For both of these formulas, one rounds down to the nearest integer value. The index states the sample size at which chi square would not be significant, i.e., that is how small one's sample size would have to be for the result to be no longer significant. Hoelter recommends values of at least 200. Values of less than 75 indicate very poor model fit.