McKenzie & Bailey GOF test in Presence 2.2

[darryl]

I can see why you might be concerned! Jim was trialling some pooling algorithms at one point to try and make the test more robust to sparse data sets, but I'm not sure in which builds that was/was not included. In either of the outputs does it say anything about pooling?

Jim, what's currently implemented in PRESENCE?

[jhines]

Darryl,

The different results are due to the fact that I had PRESENCE pool small (<2.0) observed/expected values for a while, then after talking with you, changed it back to no pooling. This is a case where pooling seems to make a fairly big difference in the result (OK fit w/ pooling vs poor fit w/o pooling), although it's based on a relatively small data set (48 obs, 5 surveys), and the psi(.),p(.) model.

I'm not sure what to make of it. From simulations I've done, I can't say that one method (pooling vs no pooling) is any better than the other. In one 'real' data-set, pooling seemed to make sense to me as one or two very small expected values seemed to change the fit statistic from a reasonable fit to very poor fit. Looking back now, maybe since we cannot know truth, we should expect weird cases like this once in a while and just stick with no pooling. What do you think?

Cheers,

Jim

[darryl]

Generally people talk about having expected values >2.0 in goodness of fit tests so that the assumption of the test statistic having a chi-square distribution is more reasonable. The logic for using the parametric bootstrap in this situation is so that we don't have to assume a chi-square distribution so the test should be less sensitive to those small expected values (not to say that it won't be sensitive to them at some point though).

The psi(.)p(.) model seems pretty simplistic, are there any more complex models that you're willing to consider? General rule of thumb is to use the GOF results from the most complex model and apply that to the whole model set. Not sure if it's biologically reasonable for the way the data has been collected and formatted, but I tried a psi(.)p(t) model in the most recent version of PRESENCE (so no pooling) and the p-value became 0.1479 and c-hat = 1.4
Darryl

[jota]

This dataset is used for the practicals of a course and we start with the simplest model. When rerunning that material I noticed the change in the GOF results from previous PRESENCE versions and that is what triggered my questions. I was not particularly worried in fitting the best model but rather in understanding the methods that are implemented in PRESENCE.

So we should be cautious with trusting the test when working with histories with low expected values... is there any rule of thumb (e.g. in terms of expected values) that would provide an indication on whether the data is such that one can consider this parametric bootstrap test robust? Is from your point of view the no pooling option better for the test?

Thanks again!

[darryl]

I haven't looked at this specifically in detail, so can't say for certain, but in the original MacKenzie and Bailey 2004 paper, we conducted simulations and in some scenarios some of the expected values were getting really small (<0.005) and most were <2. Given there was no lack of fit, the test procedure had the correct size (alpha level), so it would seem that without using pooling the bootstrap procedure with small expected values doesn't give us too many false rejections of the null hypothesis.

Whether this still holds when there is a lot of missing values (which we didn't assess in the paper) or really sparse data, I don't know. My gut feel is that the bootstrap should do a reasonable job of it provided you have enough bootstrap samples to adequately sample those really rare instances, but perhaps there are occasions when pooling does help. If anyone out there is looking for a Honours project in this general area, this might be a good one.