Calculating effect size and associated SE

[Kiel]

My questions pertain to estimating effect size and the 95%CI of the effect size when model averaging is used.

The example for calculating effect size given in the Markbook (Chapter 7, pp. 7-38) is based on a single estimate of beta1 and beta2. When multimodel inference is necessary, is it appropriate to model average the beta1 and beta2 terms and then used these in the formula for calculating effect size?

Provided that the above procedure is correct, for calculation of the SE of the effect size, I've model averaged the SE for the respective beta terms and then square this value and applied these in the formula [SQRT(var(A)+var(B)-2cov(A, B))]. Similarly, I've model averaged the covariance between A and B and applied this number in the formula. Is this the correct approach when basing the estimate of effect size on several models?

Last, if the coefficient of the covariance is negative, inclusion of the covariance inflates the SE rather than having the desired effect of making it smaller. Is this what is supposed to happen? Or, should the absolute value of the covariance be used instead?

Cheers, Kiel

[bmitchel]

There were a couple of questions raised in Kiel's post:

1) When the interest is in an effect size rather than beta terms, what is the appropriate way to incorporate model averaging?

It is possible to model average the effect size based on modeled covariates, provided that you use the model-averaged covariance matrix for the calculation. I don't know if MARK can calculate the model averaged covariance matrix. If not, it can be done using formulas in Burnham and Anderson 2004 (Sociological Methods and Research 33(2):261-304); the formula in their 2002 book is not correct. The alternative is to go model by model, and compute the effect size for that model, then model average the effect sizes based on their weights.

2) How do you calculate the effect size given a set of beta terms and their variances and covariances?

The general equation for this calculation is too convoluted for this format... but for the specific example of:

y-hat = x0b0 +x1b1 + x2b2

with the following variance-covariance matrix:

b0 b1 b2
b0 f
b1 g i
b2 h j k

the estimated variance of y-hat =
f + ((x1)^2)*i + ((x2)^2)*k + 2*x1*g + 2*x2*h + 2*x1*x2*j

The formula in Kiel's post is incorrect (the covariance term should be added). In this case, a positive covariance inflates the variance estimate, while a negative one decreases it. While we would all like to decrease the variance in our estimated effect sizes, it is completely inappropriate to alter the formula for estimating variance; the idea of the formula is to honestly apply uncertainty in parameter estimates to our estimate of uncertainty in the effect sizes (in other words, this is error propagation). In some cases (negative covariance) we are blessed with a smaller variance; in others we are stuck with a larger variance.

Anyone who wants to see the general equation or a formatted version of the above equations can write to me at brian.mitchell@uvm.edu

Brian

[cooch]

2) How do you calculate the effect size given a set of beta terms and their variances and covariances?

The extended expression Brian is referring to, which accounts for sampling covariance, is in fact found in Chapter 7, in the box on page 39.

[bmitchel]

This is just to follow up on a post from a few week's ago about calculating effect sizes; I recently compared the formula (3rd from the bottom) on page 7-39 of Evan Cooch's book with the formula I usually use for calculating effect sizes, and I noticed that the formula I use is more general. Evan's formula works fine for indicator or dummy variables (that take a value of 1 or 0). But, if you are using continuous variables, you need a different formulation:

variance (y) = sigma(x-sub-j-squared * variance(beta-sub-j)) + sigma sigma (2 * x-sub-j * x-sub-k * covariance(beta-sub-j, beta-sub-k)).

In other words, your effect size is valid only for a specific set of x values (a typical approach is to use the mean of each x, e.g. the mean temperature), and these x values must be squared in the variance term and multiplied in the covariance term of Evan's formula. For indicator variables these terms vanish, but they are important if you are using continuous variables.

Brian Mitchell
Postdoctoral Research Associate
University of Vermont

[cooch]

<snip>

In other words, your effect size is valid only for a specific set of x values (a typical approach is to use the mean of each x, e.g. the mean temperature), and these x values must be squared in the variance term and multiplied in the covariance term of Evan's formula. For indicator variables these terms vanish, but they are important if you are using continuous variables.

Brian Mitchell
Postdoctoral Research Associate
University of Vermont

Agreed - very useful followup. I should probably add more text to the book to reflect the more general case.