Lack of agreement between bootstrap and median c-hat results

[Moore]

Hello,

I am running a multistrata analysis with 2 states and 52 occasions. The data come from small mammal trapping on several sites. Not all sites were sampled during every occasion so I have fixed p for a number of sites during a number of occasions. There are also some losses on capture (42 out of 1642 total captures). A fully time-varying model does not converge well and I am not interested in a fully time-varying model anyway. I tested GOF on the most highly parameterized model from the set of models of interest. This model included groups and time-varying covariates (e.g., season) but no time-varying individual covariates. I tested GOF using the median c-hat and bootstrap approaches. The median c-hat estimate was 9.01 with SE 0.33. From 1000 bootstrap simulations, the observed c-hat / mean bootstrapped c-hat was 1.09. None of the 1000 bootstrapped deviances were greater than the observed model deviance.

I realize the Fletcher's c-hat is not well suited for models with fixed parameters and losses on capture, but in case it may be of interest, the Fletcher's c-hat was 0.99.

My question is what to make of these results. One of the GOF tests indicates very poor fit and other suggests pretty good fit. I understand that the various GOF tests will not give the same result but I thought they would at least point in the same direction. I am not certain how to proceed from here. If anyone has any thoughts or advice I would really appreciate it.

Thanks in advance for any suggestions.

[cooch]

Short of 'seeing the results', my suspician is that something has 'gone wrong' with your median c-hat derivation (which is based on an interpolation from a logistic regression applied to observed frequencies...), whereas the bootstrap and Fletcher c-hat (which are somehwhat concordant) are more likely to me correct (although 2with 52 occassions, and missing observations, anything is possible - when you say 'fixed p', you mean, fix it to 0 for occasions you didn't sample?). Did your attempt at the median c-hat at least generate a smooth 'logit function' in the plot that get's generated? If not, then this is diagnostic of the problem -- you need to iteratively 'home in' on the range over which you want to run the simulations.

[Moore]

Thank you for the quick reply.

when you say 'fixed p', you mean, fix it to 0 for occasions you didn't sample?).

Yes, p was fixed to 0 for sites during sessions when we didn't sample.

Did your attempt at the median c-hat at least generate a smooth 'logit function' in the plot that get's generated? If not, then this is diagnostic of the problem -- you need to iteratively 'home in' on the range over which you want to run the simulations.

The plot wasn't particularly smooth. The y values were approximately 0 for simulated c-hat values from 1 up to about 7. For simulated c-hat values > ~7, the P(<observed deviance c-hat) were somewhat erratic but generally increased rapidly.

The handful of times I've run median c-hat simulations for other analyses, I've started by doing a rough run using bounds from 1 to just over the model observed c-hat with only a handful of deign points and repetitions (with ~20-25 simulations). Then, after getting a rough c-hat, I've run it again using narrower bounds around that first estimate and more design points and repetitions (with ~150 simulations). I assumed part of the reason for this 2-tiered approach was to save time. For the c-hat results described in my original post however, I just did one big run with many design points and repetitions equaling ~450 simulations. This was over a holiday weekend and I figured more was better and I didn't mind that it would take a long time to run. Is there something problematic about a "one big run" approach?

Regarding your suggestion to itertiavely home in on a simulation range - would you recommend homing by starting over and using something more like the 2-tiered approach described above or would it be better to home in on the range of the simulations I already ran where the plot becomes erratic?

Thank you again for the help.