calculate confidence intervals for model averages

[cebert]

Hello,
when model averaging (unfortunately, I am working with a quite sparse data set), I get some really exorbitant confidence intervals (e.g. lower CI -425 (!), upper CI 2354). Now I read in the MARk handbook (14-18 to 14-22) that it is recommended to calculate CI's 'by hand' for model averages. I did this (using the method presented in the handbook) and now the CI's are surprisingly small (lower CI 443, upper CI 526) compared to before. Even though this of course looks very nice, I am a little bit puzzled When I calculate CI's only for single models - also 'by hand' using the same method - they also are very much smaller compared to the 'regular' CI's calculated via MARK. Is this normal and can I really use these CI or am I in danger of creating a 'false precision'? And can you give me some hint what is the reason for this huge differences?
This would be of great help for me...!
Greetings,
cebert

[cooch]

Hello,
when model averaging (unfortunately, I am working with a quite sparse data set), I get some really exorbitant confidence intervals (e.g. lower CI -425 (!), upper CI 2354). Now I read in the MARk handbook (14-18 to 14-22) that it is recommended to calculate CI's 'by hand' for model averages. I did this (using the method presented in the handbook) and now the CI's are surprisingly small (lower CI 443, upper CI 526) compared to before. Even though this of course looks very nice, I am a little bit puzzled When I calculate CI's only for single models - also 'by hand' using the same method...

The 'method' described on those pages applies only to model averaging - not to the calculation of the CI for individual models. So, you're already going down the wrong track. The conditional CI's reported for individual models are correct as reported. The unconditional SE for estimated average N is also correct. The only thing you need to do 'by hand' is calculate the 95% CI for the model averaged abundance. This is noted in several places in Chapter 14 (make sure you're looking at the most recent version).

[cebert]

Hello again,
thanks a lot for your answer. In fact, I did not intend to really 'use' the CI I calculated for the individual model (I only needed a reasonable CI for the model average). But since I was so astonished about the huge difference between the regular MARK CI for the model average and the 'by hand' CI, I just calculated the CI for one individual model to see if there would be a similarly large difference.
There are just two more aspects I still do not really understand: Is the CI I calculated by hand for the model average then comparable to the CI's of all individual models (because it is much smaller compared to all individual models, even though I would have expected the opposite)? Or is it not possible to look at the result of the model average (and its CI) in comparison with those of the individual models it is derived from? And what is the reason for the huge difference between the 'regular MARK CI' and the 'by hand' CI?
With best regards,
cebert

[cooch]

Hello again,
thanks a lot for your answer. In fact, I did not intend to really 'use' the CI I calculated for the individual model (I only needed a reasonable CI for the model average). But since I was so astonished about the huge difference between the regular MARK CI for the model average and the 'by hand' CI, I just calculated the CI for one individual model to see if there would be a similarly large difference.

Which you now know is incorrect.

There are just two more aspects I still do not really understand: Is the CI I calculated by hand for the model average then comparable to the CI's of all individual models (because it is much smaller compared to all individual models, even though I would have expected the opposite)? Or is it not possible to look at the result of the model average (and its CI) in comparison with those of the individual models it is derived from? And what is the reason for the huge difference between the 'regular MARK CI' and the 'by hand' CI?

The model CI's are conditional - the model averaged CI is based on the unconditional variance. Not comparapable. Apples and oranges. If you're sure you did the calculation correctly, then you're fine. The only caveat is that occasionally the CI's for individual models indicate that some of the models in your candidate model set are clearly 'wrong'. In such cases, it is reasonable to drop those few models, renormalize the AIC weights, and calculate the model averaged value and CI from this subset of models.

[cebert]

Hello once again,
and again thanks for the reply and the advice. This helped me a lot, but I now have one additional question concerning this topic: When using the (unconditional) 'by hand'-calculated CI's for my model averages, what about the corresponding SE? Shall I use the 'unconditional' SE calculated by program MARK in the regular model averaging function or do I have to calculate the SE also 'by hand' in some manner? Or am I again galloping off in some wrong direction? I did not find anything concerning this question in the corresponding section of chapter 14.
All the best,
cebert

[cooch]

Hello once again,
and again thanks for the reply and the advice. This helped me a lot, but I now have one additional question concerning this topic: When using the (unconditional) 'by hand'-calculated CI's for my model averages, what about the corresponding SE? Shall I use the 'unconditional' SE calculated by program MARK in the regular model averaging function or do I have to calculate the SE also 'by hand' in some manner? Or am I again galloping off in some wrong direction? I did not find anything concerning this question in the corresponding section of chapter 14.

If you didn't find it, you might want to look harder next time. Its right in the book (chapter 14), in 2 different places (thus doubling the chances someone would find it - or, in the spirit of things, increasing the latent detection probability by 2 - all other things being equal).

1. p. 33 - first paragraph after the equation at the top of the page:

In fact, MARK handles this calculation of the unconditional variance for you - you would simply
need to take the reported unconditional SE and square it to get the unconditional variance. But - you
need to calculate the CI by hand.

2. p. 34 (about half-way down the page):

Note that if we were to fit these models in MARK, the unconditional SE for the model averaged
abundance would be reported as 26.9045 - if we square this value, we get (26.9045)^2 = 723.901. Again,
the unconditional SE - and thus the variance - reported by MARK is correct (i.e., you do not need to
calculate the SE - or variance - by hand). However, the CI as reported by MARK is not correct - this,
you need to do by hand.

So, to finish this off - the unconditional SE reported by MARK for the SE of the model-averaged abundance is correct. You can use it as is, or square it to get an estimate of the unconditional variance. But, if you want the correct 95% CI for the model-averaged estimate, you need to do it by hand, as discussed at length in Chapter 14.

[phettinga]

Is there any way to cite this method of calculating confidence intervals following model averaging? Ch. 14 of MGI doesn't seem to contain any outside references in this regard. Thanks

[cooch]

Is there any way to cite this method of calculating confidence intervals following model averaging? Ch. 14 of MGI doesn't seem to contain any outside references in this regard. Thanks

Most commonly used citation is Rexstad & Burham (1991) (the CAPTURE manual), and/or Burnham et al. (1987) (the Wildlife Monograph). If either of those don't satisfy, you could cite Williams, Nichols & Conroy (the big book), although their equation for the calculation of C is not correct as written.