GOF testing with detection near 1

[mrodtka]

Hi,

I am running a GOF (10k bootstraps) test on my global (single-season, spatial dependance) model. Detection data was collected along a stream reach, detection in the first 50 m segment was allowed to differ from subsequent detections as all surveys commenced at the head of a riffle section, subsequent detection data was broken into 50 m segments. Consequently the p1 detection parameter is ~0.95 (p2+~0.75) and the GOF output indicates a c-hat of 0.0000 (chi-square around 0.3)! Distribution of simulated test statistics indicates a very skewed distribution with a high proportion of results in the lower categories but a few massive test statistics that are pulling up the average test stat. When I remove the p1 detection parameter from the global model, test seems to run fine with a more balanced test distribution and very little evidence of over/under dispersion (c-hat 1.02, p = 0.37).

Is it possible the high detection probability in p1 is causing this problem? How would you handle this situation?

Thanks in advance,

Mike

[jhines]

Mike,

The most likely problem is small sample sizes. When sample sizes are small, the estimates of psi and p can differ from one simulation to the next by quite a bit. This can cause the fit statistic to vary a lot. If you have a large number of surveys, the expected history probabilities for each site can get quite small, resulting in very high chi-square values.

What do you mean by 'remove p1 from the global model'? Are you setting it equal to p2? Your p1 and p2 values are high, but what values of psi and theta does the global model use? (I'm thinking that if these are low, it could cause low expected history probabilities mentioned above.) Another possibility is that a particular combination of covariates is causing certain sites to have low expected history probabilities. Does the global model you're doing the goodness-of-fit test for include covariates for p, psi or theta?

Jim

[mrodtka]

Analysis includes 92 sites with 3-10 spatial replicates per site (average around 5). Sorry, I should have said setting p1 to equal p2 (still learning model speak!). Global model psi = 0.3, theta1 = 0.22, theta2 = 0.84, all are (.) with no covariates; p includes 5 covariates related to habitat features thought to influence capture efficiency in the stream (undercuts, substrate type, etc.).

Thanks for your help Jim.

[jhines]

As with any gof test, bad things happen when sample sizes are small. Some gof tests have arbitrary pooling algorithms to avoid problems with small expected values, but the test in PRESENCE does not. PRESENCE tries to account for small expected values by running lots of simulations so that the average overall test statistic is not dominated by a few sites with small expected values.

PRESENCE computes the gof test statistic by simulating data with the estimated parameters, then computing new estimates from the simulated data, then computing chi-square values comparing the expected frequencies versus the simulated frequencies. With small sample sizes, the estimates from the simulated data can be very inconsistent, which can produce very small expected frequencies. This results in a very high test statistic for that simulation. By running 10,000 simulations, a few high test statistics will not affect the mean much. However, the combination of small sample sizes and the combination of real data estimates can sometimes produce enough simulations with high test statistics to overwhelm the average test statistic. I think this is what's happening in your case (with p1 not = p2). The other test (with p1=p2) seems to have produced more consistent test statistics, resulting in a more 'reasonable' c-hat.

I don't think it makes sense to use a c-hat near zero. In fact, I don't think a c-hat of less than 1.0 should be used. In the cases where c-hat is less than 1.0, the recommendation (from folks with a lot more stats background than me) is to use c-hat=1.0, which means that variances are not adjusted/inflated to account for underdispersion. I think this makes sense in your case as the p1=p2 model gives a c-hat near 1.0, and the p1 not = p2 model should fit at least as well as the p1=p2 model.

Jim

[darryl]

A c-hat of near 0 can sometimes be indicative of overfitting the data (ie too many parameters in the model). Do you have good precision (small standard errors) on your theta estimates?
Cheers
Darryl

[mrodtka]

Not that precise: Untransformed Beta's Theta0 = -1.22 SE 1.15; Theta1 = 1.68 SE 0.95.

Thanks for the perspectives.

Mike